博客园_Safe3 Network Center

Using Internet Explorer from .NET

2012-04-24T03:51:00Z

5.0 Introduction

Earlier in this book we have looked at how to read HTML from websites, and how to navigate through websites using GET and POST requests. These techniques certainly offer high performance, but with many websites using cryptic POST data, complex cookie data, and JavaScript rendered text, it might be useful to know that you can always call on the assistance of Internet Explorer browsing engine to help you get the data you need.

It must be stated though, that using Internet Explorer to data mine web pages creates a much larger memory footprint, and is not as fast as scanning using HTTP requests alone. But it does come into its own when a data mining process requires a degree of human interaction. A good example of this would be if you wanted to create an automated test of your website, and needed to allow a non-technical user the ability to follow a sequence of steps, and select data to extract and compare, based on the familiar Internet Explorer interface.

This chapter is divided into two main sections. The first deals with how to use the Internet Explorer object to interact with all the various types of web page controls. The second section deals with how Internet explorer can detect and respond to a user interacting with web page elements.

5.1 Web page navigation

The procedure for including the Internet Explorer object in your application differs depending on which version of Visual Studio .NET you are using. After starting a new windows forms project, users of Visual Studio .NET 2002 should right click on their toolbox and select customize toolbox, click COM components, then select Microsoft Web Browser. Users of Visual Studio .NET 2003 should right click on their toolbox and select Add/Remove Items, and then follow the same procedure as mentioned above. In Visual Studio .NET 2005, you do not need to add the web browser to the toolbox, just drag theWebBrowser control to the form.

An important distinction between the Internet Explorer object used in Visual Studio .NET 2002/03 and the 2005 version is that, the latter uses a native .NET class to interact with Internet Explorer, whereas the former uses a .NET wrapper around a COM (Common Object Model) object. This creates some syntactic differences between how Internet Explorer is used within .NET 2.0 and .NET 1.x. The first example in this chapter will cover both versions of .NET for completeness. Further examples will show .NET 2.0 code only, unless the equivalent .NET 1.x code would differ substantially.

The first thing you will need to know when using Internet Explorer is how to navigate to a web page. Since Internet Explorer works asynchronously, you will also need to know when Internet Explorer is finished loading a web page. In the following example, we will simply navigate to www.google.com and popup a message box once the page is loaded.

To begin this example, drop an Internet Explorer object onto a form, as described above, and call it WebBrowser. Now add a button to the form and name it btnNavigate. Click on the button and add the following code

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync("http://www.google.com");

MessageBox.Show("page loaded");

}

VB.NET

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

NavigateToUrlSync("http://www.google.com")

MessageBox.Show("page loaded")

End Sub

We then create the NavigateToUrlSync method. Note how the C# version differs in version 1.x and 2.0. This is because the COM object is expecting four optional ref object parameters. These parameters can optionally define the flags, target frame name, post data and headers sent with the request. They are not used in this case, yet since C# does not support optional parameters they have to be passed in nonetheless.

C# 1.x

public void NavigateToUrlSync(string url)

{

object oMissing = null;

bBusy=true;

WebBrowser.Navigate(url,ref oMissing,ref oMissing,ref oMissing,ref oMissing);

while(bBusy)

{

Application.DoEvents();

}

C# 2.0

public void NavigateToUrlSync(string url)

{

bBusy=true;

WebBrowser.Navigate(url);

while(bBusy)

{

Application.DoEvents();

}

VB.NET

Public Sub NavigateToUrlSync(ByVal url As String)

bBusy = True

WebBrowser.Navigate(url)

While (bBusy)

Application.DoEvents()

End While

End Sub

The while loop is polls until the public bBusy flag is cleared. The DoEvents command ensures that the application remains responsive whilst waiting for a response from the web server.

To clear the bBusy flag, we handle either the DocumentComplete (.NET 1.x) or DocumentCompleted (.NET 2.0) thus:

C# 1.x

private void WebBrowser_DocumentComplete(object sender, AxSHDocVw.DWebBrowserEvents2_DocumentCompleteEvent e)

{

bBusy = false;

}

C# 2.0

private void WebBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)

{

bBusy = false;

}

VB.NET 1.x

Private Sub WebBrowser_DocumentComplete(ByVal sender As Object, _

ByVal e As AxSHDocVw.DWebBrowserEvents2_DocumentCompleteEvent) _

Handles WebBrowser.DocumentComplete

bBusy = False

End Sub

VB.NET 2.0

Private Sub WebBrowser_DocumentCompleted(ByVal sender As Object, _

ByVal e As WebBrowserDocumentCompletedEventArgs) _

Handles WebBrowser.DocumentCompleted

bBusy = False

End Sub

To finish off the example, don抰 forget to declare the public bBusy flag.

public bool bBusy = false;

VB.NET

public bBusy As Boolean = false

To test the application, compile and run it in Visual Studio, then press the navigate button.

5.2 Manipulating web pages

An advantage of using Internet Explorer over raw HTTP requests is that you get access to the DOM (Document Object Model) of web pages, once they are loaded into Internet Explorer. For developers familiar with JavaScript, this should be an added bonus, since you will be able to control the web page in much the same way as if you were using JavaScript within a HTML page.

The main difference however, between using the DOM in .NET versus JavaScript, is that .NET is a strongly typed language, and therefore you must know the type of the element you are interacting with before you can access its full potential.

If you are using .NET 1.x you will need to reference the HTML type library, by clicking Projects > Add Reference. Then select Microsoft.mshtml from the list. For each of the examples in this section you must import the namespace into your code thus:

using mshtml;

VB.NET

Imports mshtml

If you then cast the WebBrowser.Document object to an HTMLDocument class, many of the code examples shown below should word equally well for .NET 1.x as .NET 2.0

5.2.1 Frames

Frames may be going out of fashion in modern websites, but oftentimes, you may need to extract data from a website that uses frames, and you need to be aware how to handle them within Internet Explorer. In this section, you will notice that the code differs substantially between version 1.x and 2.0 of .NET, therefore source code for both are included.

To create a simple frameset, create three files, Frameset.html, left.html and right.html, these files containing the following HTML code respectively.

Frameset.html

<html>

</frameset>

</html>

Left.html

<html>

This is the left frame

</html>

Right.html

<html>

This is the right frame

</html>

In the following example, we will use Internet Explorer to read the HTML contents of the left frame. This example uses code from the program listing in section 5.1, and assumes you have saved the HTML files in C:\

VB.NET 1.x

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

NavigateToUrlSync("C:\frameset.html")

Dim hDoc As HTMLDocument

hDoc = WebBrowser.Document

hDoc = CType(hDoc.frames.item(0), HTMLWindow2).document

MessageBox.Show(hDoc.body.innerHTML)

End Sub

VB.NET 2.0

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

NavigateToUrlSync("C:\frameset.html")

Dim hDoc As HtmlDocument

hDoc = WebBrowser.Document.Window.Frames(0).Document

MessageBox.Show(hDoc.Body.InnerHtml)

End Sub

C# 1.x

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"C:\frameset.html");

HTMLDocument hDoc;

object oFrameIndex = 0;

hDoc = (HTMLDocument)WebBrowser.Document;

hDoc = (HTMLDocument)((HTMLWindow2)hDoc.frames.item(

ref oFrameIndex)).document;

MessageBox.Show(hDoc.body.innerHTML);

}

C# 2.0

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"C:\frameset.html");

HtmlDocument hDoc;

hDoc = WebBrowser.Document.Window.Frames[0].Document;

MessageBox.Show(hDoc.Body.InnerHtml);

}

The main difference between the .NET 2.0 and .NET 1.x versions of the above code is that the indexer on the frames collection returns an object, which must be cast to an HTMLWindow2 under the COM wrapper in .NET 1.x. In .NET 2.0 the indexer performs the cast internally, and returns an HtmlWindow object.

To test the application, compile and run it from Visual Studio .NET, press the navigate button, and a message box should pop up saying this is the left frame.

5.2.2 Input boxes

Input boxes are used in HTML to allow the user enter text into a web page. Here we will automatically populate an input box with some data.

Given a some HTML, which we save as InputBoxes.html as follows

<html>

My Name is :

</form>

</html>

We can get a reference to the input box on the form by calling getElementById on the HtmlDocument. In .NET 1.x this should be then cast to an IHTMLInputElement.

C# 2.0

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"C:\InputBoxes.html");

HtmlElement hElement;

hElement = WebBrowser.Document.GetElementById("myName");

hElement.SetAttribute("value", "Joe Bloggs");

}

VB.NET 2.0

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

NavigateToUrlSync("C:\InputBoxes.html")

Dim hElement As HtmlElement

hElement = WebBrowser.Document.GetElementById("myName")

hElement.SetAttribute("value", "Joe Bloggs")

End Sub

In order to enter the text into the input box, we call the SetAttribute method of the HtmlElement, passing in the property to change, and the new text. In .NET 1.x we would set the value property of the IHTMLInputElement to the new text.

To test the application, compile and run it from Visual Studio .NET, then press the navigate button.

5.2.3 Drop down lists

In HTML, drop down lists are used in web pages to allow users input from a list of pre-defined values. In the following example, we will demonstrate how to set a value of a drop down list, and then read it back.

We shall start off with a HTML file, which we save as DropDownList.html

<html>

My favourite colour is:

</select>

</form>

</html>

We can get a reference to the drop down list by calling getElementById on the HtmlDocument. In .NET 1.x this should be then cast to anIHTMLSelectElement.

C# 2.0

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"C:\dropdownlists.html");

HtmlElement hElement;

hElement = WebBrowser.Document.GetElementById("myColour");

hElement.SetAttribute("selectedIndex", "1");

MessageBox.Show("My favourite colour is:" + hElement.GetAttribute("value"));

}

VB.NET 2.0

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

NavigateToUrlAsync("C:\dropdownlists.html")

Dim hElement As HtmlElement

hElement = WebBrowser.Document.GetElementById("myColour")

hElement.SetAttribute("selectedIndex", "1")

MessageBox.Show("My favourite colour is:" + hElement.GetAttribute("value"))

End Sub

Here, we can see that in order to set our selection we pass ?/span>selectedIndex? and the selection number to SetAttribute. We then pass ?/span>value? to GetAttribute in order to read back the selection. In .NET 1.x, we achieve the same results by setting the selectedIndex property on theIHTMLSelectElement and reading back the selection from the value property.

To test the application, compile and run it from Visual Studio .NET, press the navigate button, and you should see a message box appear saying my favorite color is: Red.

5.2.4 Check boxes and radio buttons

Check boxes and radio buttons are generally used on web pages to allow the user to select between small numbers of options. In the following example, we shall demonstrate how to toggle check boxes and radio buttons.

We shall start off with a HTML file, which we will save as CheckBoxes.html

<html>

<input type="checkbox" name="myCheckBox">Check this.<br>

<input type="radio" name="myRadio" value="Yes">Yes

<input type="radio" name="myRadio" checked="true" value="No">No

</form>

</html>

As before we can get a reference to the checkbox by calling getElementById. However, since the two radio buttons have the same name, we need to use

Document.All.GetElementsByName and then select the required radio button from the HtmlElementCollection returned.

In .NET 1.x, we would use a call to getElementsByName on the HTMLDocument. This returns an IHTMLElementCollection. We can then get the reference to the IHTMLInputElement with the method item(null,1).

C# 2.0

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"C:\checkboxes.html");

HtmlElement hElement;

HtmlElementCollection hElements;

hElement = WebBrowser.Document.GetElementById("mycheckBox");

hElement.SetAttribute("checked", "true");

hElements = WebBrowser.Document.All.GetElementsByName("myRadio");

hElement = hElements[0];

hElement.SetAttribute("checked", "true");

}

VB.NET 2.0

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

NavigateToUrlSync("C:\checkboxes.html")

Dim hElement As HtmlElement

hElement = WebBrowser.Document.GetElementById("mycheckBox")

hElement.SetAttribute("checked", "true")

hElement = WebBrowser.Document.All.GetElementsByName("myRadio").Item(0)

hElement.SetAttribute("checked", "true")

End Sub

As before, we set the property of the HtmlElement using the SetAttribute method. In .NET 1.x, you need to set the @checked property on theIHTMLInputElement

To test the application, compile and run it from Visual Studio, then press the navigate button. You should see the check box and radio button toggle simultaneously.

5.2.5 Buttons

Submit buttons and standard buttons are generally used to submit forms in HTML. They form a crucial part in navigating any website.

Given a simple piece of HTML, which we save as Buttons.html as follows:

<html>

</form>

</html>

We can get a reference to the button on the form by calling getElementById on the HtmlDocument. In .NET 1.x this should be then cast to anIHTMLElement.

C# 2.0

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"C:\buttons.html");

HtmlElement hElement;

hElement = WebBrowser.Document.GetElementById("btnSubmit");

hElement.InvokeMember("click");

}

VB.NET 2.0

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

NavigateToUrlSync("C:\buttons.html")

Dim hElement As HtmlElement

hElement = WebBrowser.Document.GetElementById("btnSubmit")

hElement.InvokeMember("click")

End Sub

In the above example, we can see that after we get a reference to the button, we call the click method using InvokeMember. Similarly, if we wanted to submit the form without clicking the button, we could get a reference to myForm and pass ?/span>submit? to the InvokeMember method.

In .NET 1.x, there is no InvokeMember method of IHTMLElement, so therefore you must call the click method of theIHTMLElement. In the case of a form, you should cast the IHTMLElement to an IHTMLFormElement and call it submit method.

To test this application, compile and run it from Visual Studio .NET, and press the navigate button. The form should load and then automatically forward itself to a google.com search result page..

5.2.6 JavaScript

Many web pages use JavaScript to perform complex interactions between the user and the page. It is important to know how to execute JavaScript functions from within Internet explorer. The simplest method is to use Navigate with the prefixjavascript: then the function name. However, this does not give us a return value, nor will it work correctly in all situations.

We shall start with a HTML page, which contains a JavaScript function to display some text. This will be saved asJavaScript.html

<html>

<span id="hiddenText" style="display:none">This was displayed by javascript</span>

function jsFunction()

{

window.document.all["hiddenText"].style.display="block";

return "ok";

}

</script>

</html>

We can then use the Document.InvokeScript method to execute the JavaScript thus:

C# 2.0

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"C:\javascript.html");

string strRetVal = "";

strRetVal = (string)WebBrowser.Document.InvokeScript("jsFunction");

MessageBox.Show(strRetVal);

}

VB.NET 2.0

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

NavigateToUrlSync("C:\javascript.html")

Dim strRetVal As String

strRetVal = WebBrowser.Document.InvokeScript("jsFunction").ToString()

MessageBox.Show(strRetVal)

End Sub

In .NET 1.x, we would call the parentWindow.execScript method on the HTMLDocument. Not forgetting to add empty parenthesis after the JavaScript function name. Unfortunately execScript returns null instead of the JavaScript return value.

To test the application, compile and run it from Visual Studio .NET, then press the Navigate button.

5.3 Extracting data from web pages

In order to extract HTML from a web page using Internet Explorer, you need to call Body.Parent.OuterHtml in .NET 2.0 orbody.parentElement.outerHTML in .NET 1.x. You should be aware that the HTML returned by this method is different to the actual HTML content of the page.

Internet Explorer will correct HTML in the page by adding <BODY>, <TBODY> and <HEAD> tags where missing. It will also capitalize existing HTML Tags, and make other formatting changes that you should be aware of.

Techniques for parsing this textual data are explained later in the book under the section concerning Regular Expressions.

5.4 Advanced user interaction

When designing an application which uses Internet Explorer as a tool for data mining, it comes of added benefit, that the user can interact with the control in a natural fashion, in order to manipulate its behavior. The following sections describe ways in which a user can interact with Internet Explorer, and how these events can be handled within .NET

5.4.1 Design mode

If you wanted to provide the user with the ability to manipulate web pages on-the-fly, there is no simpler way to do it, than using the in-built design mode?in internet explorer. This particular feature is not supported with the managed .NET 2.0WebBrowser control. However, it is possible to access the unmanaged interfaces, which we were using in .NET 1.x through the Document.DomDocument property. This can be then cast to the HTMLDocument in the mshtml library (Not to be confused with the managed HtmlDocument class). Therefore, in the case, you will need to add a reference to the mshtml library and add a using mshtml? statement to the top of your code.

In this example, we will create a simple rich text editor based on Internet Explorer design mode. Within design mode the user can perform a wide variety of tasks using intuitive actions, for example, you can insert an image by right clicking on the browser, or convert text to bold by pressing CTRL+B. Many of these tasks can be further automated using the execCommandmethod of the HTMLDocument object. In the following example, we will demonstrate how to set fonts using this method.

Open a new project in Visual Studio .NET, and drag a WebBrowser control onto the form, followed by a button, namedbtnFont. Also Add a FontDialog control named fontDialog. Click on the form and type the following code for the form load event.

C# 2.0

private void Form1_Load(object sender, EventArgs e)

{

string url = "about:blank";

webBrowser.Navigate(url);

Application.DoEvents();

HTMLDocument hDoc = (HTMLDocument)webBrowser.Document.DomDocument;

hDoc.designMode = "On";

}

VB.NET 2.0

Private Sub Form1_Load(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles MyBase.Load

Dim url As String = "about:blank"

webBrowser.Navigate(url)

Application.DoEvents()

Dim hDoc As HTMLDocument = webBrowser.Document.DomDocument

hDoc.designMode = "On"

End Sub

In .NET 1.x, we would cast the Document to an HTMLDocument, rather than referencing the DomDocument property, and also, the Navigate method would be as described in section 5.1.

Now click on the font button and enter some code as follows

C# 2.0

private void btnFont_Click(object sender, EventArgs e)

{

fontDialog.ShowDialog();

HTMLDocument hDoc = (HTMLDocument)webBrowser.Document.DomDocument;

IHTMLTxtRange selection = (IHTMLTxtRange)hDoc.selection.createRange();

hDoc.execCommand("FontName", false, fontDialog.Font.FontFamily.Name);

hDoc.execCommand("FontSize", false, fontDialog.Font.Size);

selection.select();

}

VB.NET 2.0

Private Sub btnFont_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnFont.Click

fontDialog.ShowDialog()

Dim hDoc As HTMLDocument = webBrowser.Document.DomDocument

Dim selection As IHTMLTxtRange = hDoc.selection.createRange()

hDoc.execCommand("FontName", False, fontDialog.Font.FontFamily.Name)

hDoc.execCommand("FontSize", False, fontDialog.Font.Size)

selection.select()

End Sub

From the above code we can see that we get a reference to the text currently highlighted by the user using theselection.createRange method. We then execute two commands on this selection, FontName and FontSize. Other commands that could be used would be ForeColor Italic, Bold and so forth.

To test the application, compile and run it from Visual Studio .NET, enter some text into the space provided, then highlight it. Click on the font button and choose a new font and size. The text should change to the selected font.

5.4.2 Capturing Post data

When a user is navigating through web pages, it may be necessary to keep track of what URLs they are going to, and post data sent between Internet Explorer and the web server. Although there are ways and means of doing this using packet sniffing, or third party tools, these do sometimes tend to listen to too much data and record traffic from other applications. Due to a bug in the .NET wrapper for Internet Explorer (see Microsoft Knowledge base 311298), the beforeNaviate event will not fire as you move between pages.

In order to subscribe to this event, we need to know a little about how COM events work under the hood? Every COM object which generates events will implement the IConnetionPointContainer interface. A client wishing to subscribe to events from this COM object must call the FindConnectionPoint method on this interface, passing the IID (Interface ID) of the required set of events.

Some COM objects support multiple sets of events or connection Points? for example, Internet Explorer supports theDWebBrowserEvents connection point, and the DWebBrowserEvents2 connection point. Herein lies the problem, the .NET wrapper will by default attach to the DWebBrowserEvents2 connection point, which contains a version of BeforeNavigatewhich is incompatible with .NET due to unsupported variant types.

If you open the ILDASM utility, then click file open, and select Interop.SHDocVw.DLL.
From the information in Figure 5.8 we can see that the Dispatch ID is set to 64 Hex (100 decimal). While using ILDASM we can also find the IID of the DWebBrowserEvents connection point by double clicking on class interface?that is, eab22ac2-30c1-11cf-a7eb-0000c05bae0b. At this point we have everything we need to create an interface in C# for this event.

[Guid("eab22ac2-30c1-11cf-a7eb-0000c05bae0b"),

InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]

public interface IWebBrowserEvents

{

[DispId(100)]

void RaiseBeforeNavigate(String url, int flags, String targetFrameName,

ref Object postData, String headers, ref Boolean cancel);

}

To put this all together, create a new project in Visual Studio .NET, and drop in a Web Browser control (not the .NET 2.0 version, but the COM version). Add a reference to Microsoft Internet Controls under COM references. You will need to include both SHDocVw and System.Runtime.InteropServices in the using list at the head of your code. Now add the code for the interface listed above.

private void Form1_Load(object sender, System.EventArgs e)

{

UCOMIConnectionPointContainer icpc;

UCOMIConnectionPoint icp;

int cookie = -1;

icpc = (UCOMIConnectionPointContainer)axWebBrowser1.GetOcx();

Guid g = typeof(DWebBrowserEvents).GUID;

icpc.FindConnectionPoint(ref g, out icp);

icp.Advise(this, out cookie);

}

What this code does, is that it obtains a reference to Internet Explorer underlying IConnectionPointContainer, by calling the GetOcx method, that axWebBrowser1 has inherited from AxHost. From this, we can then obtain a reference to the required connection point by passing its GUID / IID to theFindConnectionPoint method. To subscribe to events, we call the Advise method. To unsubscribe we should call the unAdvise method, if required.

To handle the event, we shall simply pop up a message box immediately before the page navigates. We shall also display any post data being sent.

public void RaiseBeforeNavigate(String url, int flags, String

targetFrameName, ref Object postData, String headers, ref Boolean cancel)

{

string strPostData="";

if (postData!=null)

{

strPostData= System.Text.Encoding.UTF8.GetString((byte[])postData);

MessageBox.Show(strPostData);

}

Since we have specified that our class should implement IWebBrowserEvents, this dictates that the post data must be received as an object. This object should then be cast to a byte array, and then to a UTF8 string for readability.

To finish off the example, add a button to the form, and attach some code to it, to allow it to navigate to some website with a post-form on it, in this example, Amazon.com

private void button1_Click(object sender, EventArgs e)

{

object o = null;

this.axWebBrowser1.Navigate("http://www.amazon.com",ref o,ref o,ref o,ref o);

}

To test the application, run it from Visual Studio .NET, press the navigate button, enter something in the Amazon search box, and press go. You should see a message box appearing, containing the post data which you sent to the web server.

5.4.3 Capturing click events

Although you can capture events such as DocumentComplete to determine when a user navigates to a new page, it is a little trickier to trap events which do not involve page navigation, such as entering text into a text box for instance.

The event trapping technique differs substantially between .NET 1.x and .NET 2.0. In the latter version, you need to implement the default COM interop method, this an entry point in your application which is marked as Dispatch ID 0, which COM uses to call back whenever your application subscribes to an event. In order to use COM interoperability, you need to include a using System.Runtime.InteropServices statement at the top of your code, in .NET 1.x.

In .NET 2.0, it is a little more straightforward. Here, we attach an HtmlElementEventHandler delegate to the Document.Clickevent, and implement it in our own event handler.

Basing this example on the sample code in section 5.1, we shall now add some extra event handling capabilities to pop up a message box whenever the user clicks a HTML element in the web browser.

C# 2.0

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"http://www.google.com");

WebBrowser.Document.Click += new HtmlElementEventHandler(Document_Click);

}

C# 1.x

private void btnNavigate_Click(object sender, System.EventArgs e)

{

NavigateToUrlSync(@"http://www.google.com");

HTMLDocument hDoc = (HTMLDocument)WebBrowser.Document;

hDoc.onclick = this;

}

At this point we have now subscribed to the click event, and in the case of .NET 2.0, supplied a call back delegate namedDocument_Click. For demonstration purposes, we shall simply display the tag name of the element clicked, and the event type (which should always be click?in our case).

C# 2.0

public void Document_Click(object sender, HtmlElementEventArgs e)

{

string sTag = WebBrowser.Document.GetElementFromPoint(e.MousePosition).TagName;

MessageBox.Show("Object: " + sTag + ", type:" + e.EventType);

}

C# 1.x

[DispId(0)]

public void DefaultMethod()

{

HTMLDocument hDoc = (HTMLDocument)WebBrowser.Document;

HTMLWindow2 hWin = (HTMLWindow2)hDoc.parentWindow;

MessageBox.Show("Object: " + hWin.@event.srcElement.tagName +

", Type: " + hWin.@event.type);

}

In order to get a reference to the element clicked, we have used a different technique for each version of .NET. In .NET 2.0, the GetElementFromPoint method is used to determine the element from the mouse location. In .NET 1.x, we can get the reference to the element via the Document.parentWindow.@event.srcElement property.

To test the application, compile and run it from Visual Studio .NET, press the navigate button, then click anywhere on the screen. You should see a message box appear with the tag name of the HTML element that you clicked on.

5.5 Extending Internet Explorer

The examples so far have dealt with embedding Internet Explorer in our applications, rather than embedding our applications in Internet Explorer. This may not be ideal for all users, as we loose the familiar interface that users are accustomed to. This section deals with how build applications around running instances of Internet Explorer.

5.5.1 Menu extensions

When you right click on a web page, you can see a context menu, which you can extend with a simple registry tweak. In this example, you can add a link to 揝end to a friend?in the context menu, which will link to a website that allows you to send emails. Firstly create the following registry key:

HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\MenuExt\Send to a friend

Then set the default value to a location on your hard drive, say c:\SendToAFriend.html, which would contain the following HTML:

window.open("http://www.pop3webmail.info/reply.aspx?url=" + external.menuArguments.document.URL);

</script>

After making the change to the registry, close all browser windows.

5.5.2 Spawning a new instance of Internet Explorer

A simple way of controlling instances of Internet Explorer is to create them yourself using COM. In this example I use COM late binding, which differs from the early-bound examples used earlier in this chapter, specifically in the .NET 1.x examples. Early bound COM objects are compiled into the application at design time. Late bound COM objects are loaded dynamically at run time.

The benefit of early bound objects is that the development environment will be aware of the object model of the component, and Intellisense will assist you determine which methods you can call. We do not have such a luxury with late bound objects. However, there is an advantage that we can bind to COM objects hosted as executables, such as in the following example.

To start off, create a new windows forms application in Visual Studio .NET, drop a button on the form, and attach the following code to it.

public Type tIE;

public object oIE;

private void btnNavigate_Click(object sender, EventArgs e)

{

object[] oParameter = new object[1];

tIE = Type.GetTypeFromProgID("InternetExplorer.Application");

oIE = Activator.CreateInstance(tIE);

oParameter[0] = (bool)true;

tIE.InvokeMember("Visible", BindingFlags.SetProperty, null, oIE, oParameter);

oParameter[0] = (string)"http://www.google.com";

tIE.InvokeMember("Navigate2", BindingFlags.InvokeMethod,

null, oIE, oParameter);

}

VB.NET

Public tIE As Type

Public oIE As Object

Private Sub btnNavigate_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles btnNavigate.Click

Dim oParameter(0) As Object

tIE = Type.GetTypeFromProgID("InternetExplorer.Application")

oIE = Activator.CreateInstance(tIE)

oParameter(0) = CType(True, Boolean)

tIE.InvokeMember("Visible", BindingFlags.SetProperty, Nothing, oIE, oParameter)

oParameter(0) = CType("http://www.google.com", String)

tIE.InvokeMember("Navigate2", BindingFlags.InvokeMethod,

Nothing, oIE, oParameter)

End Sub

You should also add references to System.Threading and System.Reflection at the top of your code.

The above code retrieves a reference to the COM object model for the Internet Explorer application by inspecting the ProgID?/span>InternetExplorer.Application? It then creates an instance of this COM object. It sets its Visible property to true, then calls the Navigate2 method, passing the URL www.google.com as a parameter.

Unfortunately it is not trivial to subscribe to events from this late bound object, so therefore, if it is necessary to detect navigation between pages, it may be necessary to poll on the LocationURL property of the browser.

To test this application, compile and run it from Visual Studio .NET, then press the button on the form. You should see a new browser window open.

5.5.3 Browser Helper Objects

When you need to get really tight integration with Internet Explorer, in cases where you want code to execute completely transparently to the user, and yet have full control of the browsers document model and be able to subscribe to events, Browser Helper Objects (BHO) is the way to go.

BHO technology is widely associated with Spyware applications, which silently run in the background, as a user is browsing websites. Since the BHO would have access to the HTMLDocument object of the Internet Explorer instance hosting it, it would be possible to read the text of the webpage being visited, and duly display context-sensitive advertisements.

Internet Explorer expects the BHO object to be COM based, not a .NET assembly. Therefore it is necessary to create a CCW (Com Callable Wrapper) for our assembly. This CCW has a unique Class ID, which we store in the registry at the following location:

HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\
Browser Helper Objects

When Internet Explorer (or Windows Explorer) starts, it reads all the Class ID抯 listed in at the registry location listed above, and creates instances of their respective COM objects, and in our case, the underlying .NET assembly. It then interrogates the COM object to ensure that it implements the IObjectWithSite interface. This interface is very strictly defined and implemented as follows:

using System;

using System.Runtime.InteropServices;

namespace BrowserHelperObject

{

[ComVisible(true),

InterfaceType(ComInterfaceType.InterfaceIsIUnknown),

Guid("FC4801A3-2BA9-11CF-A229-00AA003D7352")]

public interface IObjectWithSite

{

[PreserveSig]

int SetSite([MarshalAs(UnmanagedType.IUnknown)]object site);

[PreserveSig]

int GetSite(ref Guid guid, out IntPtr ppvSite);

}

Internet Explorer uses the two methods listed above to interact with the BHO. The SetSite method is called by Internet Explorer whenever it starts up, or shuts down. This is to update the BHO with the status of any internal references it may hold to the instance of Internet Explorer which is hosting it. GetSite may be called by Internet Explorer to query the reference a BHO holds to it. Every BHO must implement both of these methods, and handle requests to and from the hosting instance correctly.

To demonstrate Browser Helper Objects, we shall go though a simple example, where we attach a BHO to Internet Explorer, which will append the current date to every page visited by the user.

Start a new class library project in Visual Studio, add a reference to the Microsoft.mstml .NET assembly, and also to the COM object named Microsoft Internet Controls? Add a new class file containing the definition of IObjectWithSite as listed above. Then you can create the skeleton of your BHO thus:

using System;

using System.Runtime.InteropServices;

using SHDocVw;

using Microsoft.Win32;

using mshtml;

namespace BrowserHelperObject

{

[ComVisible(true),

Guid("F839CC51-A6D8-4e9c-ACE5-F05071AD0C74"),

ClassInterface(ClassInterfaceType.None)]

public class DateStamp : IObjectWithSite

{

WebBrowser webBrowser;

}

What you can see from the code above is that the class implements the IObjectWithSite interface, which is a pre-requisite of any BHO. It also has a GUID (Genuinely Unique Identifier), - this is used to uniquely identify the CCW, and can be chosen arbitrarily, using the GuidGen.exe tool or similar. The WebBrowser class in the code does not refer to the familiarWebBrowser class as used in .NET 2.0, but instead is a class defined within SHDocVw. It is this object which will contain a reference to the hosting instance of Internet Explorer.

As mentioned previously, it is necessary for every BHO to implement both the GetSite and SetSite methods. In most cases, there is little need to perform any custom actions within GetSite, so therefore its implementation would remain standard for most types of BHO. A typical implementation would be as follows:

public int GetSite(ref Guid guid, out IntPtr ppvSite)

{

IntPtr punk = Marshal.GetIUnknownForObject(webBrowser);

int hr = Marshal.QueryInterface(punk, ref guid, out ppvSite);

Marshal.Release(punk);

return hr;

}

What this code does, is that it firstly obtains a pointer to the IUnknown COM interface for our reference to the hosting instance of Internet Explorer. It then queries the IUnknown interface with a GUID issued internally by Internet Explorer. This returns a pointer to another object, as required by Internet Explorer. The code then frees the resources associated with theIUnknown pointer, and returns a HRESULT in the event that an error occurred whilst trying to query the interface.

What is of more interest is the SetSite method. This is where we have the opportunity to attach custom event handlers to the hosting web browser. In this case, we attach the DocumentComplete event handler.

public int SetSite(object site)

{

if (site != null)

{

webBrowser = (WebBrowser)site;

webBrowser.DocumentComplete += new

DWebBrowserEvents2_DocumentCompleteEventHandler(

this.OnDocumentComplete);

}

else

{

webBrowser.DocumentComplete -= new

DWebBrowserEvents2_DocumentCompleteEventHandler(

this.OnDocumentComplete);

webBrowser = null;

}

return 0;

}

As mentioned earlier, Internet Explorer may also call this method as it is shutting down, therefore, in which case the passed parameter is null. It is required that we should detach event handlers and free any associated resources at the point when the host closes.

At this point we are in a position to add our own custom logic, which we place within the OnDocumentComplete function thus:

private void OnDocumentComplete(object frame, ref object urlObj)

{

if (webBrowser != frame) return;

IHTMLDocument2 document = (IHTMLDocument2)webBrowser.Document; document.body.innerHTML = document.body.innerHTML +

DateTime.Now.ToShortDateString();

return;

}

In the above code, we retrieve a reference to the HTMLDocument contained within the hosting Internet Explorer instance, and simply add the current date to the HTML of the page.

Before we are ready to try out our new BHO, we should add some extra plumbing to enable the assembly to store the Class ID of it CCW in the registry with the other Browser Helper Objects installed on the system.

public static string BHOKEYNAME = "Software\\Microsoft\\Windows\\CurrentVersion\\Explorer\\Browser Helper Objects";

[ComRegisterFunction]

public static void RegisterBHO(Type t)

{

RegistryKey key = Registry.LocalMachine.OpenSubKey(BHOKEYNAME, true);

if (key == null) key = Registry.LocalMachine.CreateSubKey(BHOKEYNAME);

string guidString = t.GUID.ToString("B");

RegistryKey bhoKey = key.OpenSubKey(guidString);

if (bhoKey == null) bhoKey = key.CreateSubKey(guidString);

key.Close();

bhoKey.Close();

}

The above code is called whenever we create a CCW from the assembly. It inserts a new key containing the Class ID, at the registry location as specified.

Similarly, as we un-register the CCW, we will want to remove that key from the registry. This would be implemented thus:

[ComUnregisterFunction]

public static void UnregisterBHO(Type t)

{

RegistryKey key = Registry.LocalMachine.OpenSubKey(BHOKEYNAME, true);

string guidString = t.GUID.ToString("B");

if (key != null) key.DeleteSubKey(guidString, false);

}

You will find that whilst developing a BHO, you may need to recompile and test the code several times to perfect your application. Every time that you attach a BHO to Internet Explorer, it will also attach itself to Windows Explorer, and the assembly will be locked for the duration of the lifetime of these two processes. When the assembly is locked you will not be able to delete it, or modify it by building a new version of the BHO over it.

To unlock the BHO, you will need to un-register it using regasm /unregister then stop all iexplore.exe and explore.exe processes, through either task manager, or by logging off and logging back in again.

To test the above application, compile the above code, then open up the Visual Studio .NET command prompt and navigate to the folder that contains the output DLL. then run the command

Regasm /codebase browserHelperObject.dll

Now open up an Internet Explorer window and you should see the date written at the bottom of the page.

If you receive the following warning ?do not panic, as long the GUID you used in your assembly is unique, it will not cause a problem

RegAsm warning: Registering an unsigned assembly with /codebase can cause your assembly to interfere with other applications that may be installed on the same computer. The /codebase switch is intended to be used only with signed assemblies. Please give your assembly a strong name and re-register it.

5.6 Conclusion

This chapter has demonstrated how to control Internet Explorer from within a .NET application. It should pave the way for automating data mining processes using this versatile component.

With the added benefit of enabling a user to interact with web pages in a natural fashion, and being able to trap events from within Internet Explorer, it should be possible to implement data mining training tools, and website test automation utilities with the examples shown in this chapter.

本文链接

Safe3 WEB安全网关 v3.9.1

2011-11-30T02:53:00Z

Safe3WAF是国内第一款免费的Linux轻量级的反向代理Web安全网关，采用类似nginx的占有内存少、高并发架构。作为网页服务器的前置，不但可以抵御各种黑客攻击，还可以高速内存cache服务器缓存相关请求来提高Web服务器的速度，并且具备网站集群负载均衡等功能。目前中国大陆使用Safe3WAF网站有游久网、联合早报等。

主要功能:

1.拦截GET sql注入

2.拦截POST sql注入

3.拦截Cookie sql注入

4.拦截XSS跨站攻击

5.拦截web溢出攻击

6.拦截网站信息泄露攻击

7.拦截非法http请求方法攻击

8.web负载均衡功能

9.拦截上传web后门功能

10.拦截webserver漏洞

黑客攻击日志记录： /usr/local/safe3waf/log/attack.log

v3.9.1更新说明

1.新增自定义拦截跳转url

2.新增拦截IIS脚本解析执行漏洞

3.新增svn、htaccess、mdb等信息泄露拦截

4.新增struts2框架XSLTResult本地文件代码执行漏洞拦截

本文链接

scrapy结合webkit抓取js生成的页面

2011-10-19T10:34:00Z

1 scedule

scrapy 作为抓取框架，包括了spider,pipeline基础设施

2 webkit

scrapy 本身不能作为js engine,这就导致很多js生成的页面的数据会无法抓取到，因此，一些通用做法是webkit或者xmi_runner(firefox)。通过这个手段可以对于js生成的数据进行抓取。需要安装的包有

python-webkit (相关依赖自行解决)

Xvfb (用于非Xwindow环境)

3 开发downloader middleware

from scrapy.http import Request, FormRequest, HtmlResponse

import gtk
import webkit
import jswebkit
import settings

class WebkitDownloader( object ):
def process_request( self, request, spider ):
if spider.name in settings.WEBKIT_DOWNLOADER:
if( type(request) is not FormRequest ):
webview = webkit.WebView()
webview.connect( 'load-finished', lambda v,f: gtk.main_quit() )
webview.load_uri( request.url )
gtk.main()
js = jswebkit.JSContext( webview.get_main_frame().get_global_context() )
renderedBody = str( js.EvaluateScript( 'document.body.innerHTML' ) )
return HtmlResponse( request.url, body=renderedBody )

4 配置

在scrapy的settings.py中加入：

#which spider should use WEBKIT
WEBKIT_DOWNLOADER=['ccb']

DOWNLOADER_MIDDLEWARES = {
'rate_crawler.dowloader.WebkitDownloader': 543,
}

import os
os.environ["DISPLAY"] = ":0"

5 使用

启动 Xvfb (假设DISPLAY=:0)

要与settings.py中的DISPLAY对应（本例中是:0)。

scrapy crawl xxx

本文链接

快速构建实时抓取集群

2011-10-19T10:31:00Z

定义：

首先，我们定义一下定向抓取，定向抓取是一种特定的抓取需求，目标站点是已知的，站点的页面是已知的。本文的介绍里面，主要是侧重于如何快速构建一个实时的抓取系统，并不包含通用意义上的比如链接分析，站点发现等等特性。

在本文提到的实例系统里面，主要用到linux+mysql+redis+django+scrapy+webkit，其中scrapy+webkit作为抓取端，redis作为链接库存储，mysql作为网页信息存储，django作为爬虫管理界面，快速实现分布式抓取系统的原型。

名词解析：

1. 抓取环：抓取环指的是spider在存储中获取url，从互联网上下载网页，然后将网页存储到数据库里面，最后在从存储里面获取下一个URL的一个流程。
2. Linkbase：链接库的存储模块，包含一般的链接信息；是抓取系统的核心，使用redis存储。
3. XPATH：一门在 XML 文档中查找信息的语言，XPath 可用来在 XML 文档中对元素和属性进行遍历，是 W3C XSLT 标准的主要元素。使用XPATH以及相关工具lib进行链接抽取和信息抽取。
4. XPathOnClick：一个chrome的插件，支持点击页面元素，获取XPATH路径，用于编辑配置模板。
5. Redis：一个开源的KV的内存数据库，具备很好的数据结构的特征和很高的存取性能。用于存储linkbase信息
6. Django：爬虫管理工具，用于模板配置，系统监控反馈。Django在这里主要是用来管理一个数据库，使用Admin功能。
7. Pagebase：页面库，主要是存储网页抓取的结果，以及页面抽取的结果，和dump交互，使用mysql实现。
8. Scrapy：一个开源的机遇twisted框架的python的单机爬虫，该爬虫实际上包含大多数网页抓取的工具包，用于爬虫下载端以及抽取端。
9. 列表页：指的商品页面之外的所有页面
10. 详情页：比如商品B2C的抓取中，特指商品页面，比如：http://item.tmall.com/item.htm?id=10321272374

系统架构

一：存储 redis+mysql

链接库（linkbase）是抓取系统的核心，基于性能和效率的考虑，本文采用基于内存的redis和磁盘的mysql为主，对于linkbase主要是存储抓取必须的链接信息，比如url，anchor，等等；对于mysql，则是存放抓取的网页，便于后续的抽取和处理。

a) PageBase：使用Mysql分库分表，存放网页，如下图：

b) Linkbase 使用Redis集群，存储linkbase信息。

几个基本的数据结构：
1：抓取队列 (candidate list)
分为待抓取的url队列和更新的url队列；队列存放urlhash，使用redis的list数据结构，对于新提取的url，push到对应的列表里面，对于spider抓取模块，从list pop得到。对于一个站点而言，抓取队列有两种类型：列表页抓取队列和详情页抓取队列。
2：链接库 (linkbase)
链接库实际上是存储链接信息的DB；Key是urlhash，Value是linkinfo，包含url，purl，anchor，xpath。。。；在redis使用hash存储，直接存放在redis的里面。KV链接库，不区分页面类型。
3：已抓取集合（crawled_set）
已抓取集合指的是当前已经下载的页面的urlhash，存放已经抓取的网页，使用redis的set实现，set的key是urlhash，score是时间戳，已抓取集合主要是用来记录哪一些页面已经抓取和抓取的时间，用于后续的更新页面调度以及抓取信息的统计。同抓取队列一样，每一个站点有两种类型的已抓取集合，详情页和列表页

二：调度模块：

调度模块是抓取系统的关键，调度系统的好坏决定了抓取系统的效率；这块是主要是在redis linkbase之上的数据结构，主要有抓取队列、抓取集合、抓取优先级等等数据结构组成；对于一个抓取循环来说：获取URL，提交到抓取模块的待抓取队列，启动抓取，抓取完成之后对新链接进行抽取，最后进入等待抓取的队列里面。

调度系统的基本配置：
a) 频率（间隔多少秒）
b) 各个抓取列表的选取比例：get_detail，mod_detail，get_list，mod_list
链接抽取：抽取页面的链接，进行除重，对于新的链接，插入到待抓取列表里
内容抽取：按照模块的配置XPATH，抽取页面信息，并写入到pagebase中。
离线调度：按照更新的比例，从crawled_set里面，定期选取url进入Mod队列里面进行刷新。

三：抓取模块：

抓取模块是抓取的必要条件，抓取模块来说，重要的是应付互联网上各式的问题，以及如何实现对对方站点的ip平衡，当然，这块是和调度系统的紧密结合的，对于抓取模块而言，本文主要使用scrapy工具包里面的下载模块。

首先，抓取模块从linkbase获取对应站点的抓取url，进行页面下载，然后将页面信息写回到pipeline中，并完成链接抽取和页面抽取，同时调用调度模块，插入到linkbase和pagebase中。
下载端设计：
IP：每台机器需要配置多个物理公网IP，下载的时候，随机选择一个IP下载
抓取频度调整：读取配置文件，按照配置文件的抓取频率进行选取url

四：配置界面：

配置界面主要是对抓取系统的管理和配置，包括：站点feed、页面模块抽取、报表系统的反馈等等。

类似于通用的抓取架构，本文提到的抓取系统架构如下图：

一个完整的抓取数据流：

1：用户提供种子URL

2：种子URL进入linkbase中新URL队列中

3：调度模块选取url进入到抓取模块的待抓取队列中

4：抓取模块读取站点的配置文件，按照执行的频率进行抓取

5：抓取的结果返回到pipeline接口中，并完成连接的抽取

6：新发现的连接在linkbase里面进行dedup，并push到linkbase的新URL模块里面

7：调度模块选取url进入抓取模块的待抓取队列，goto 4

8：end

系统扩展

本文提到的抓取系统，核心是调度和存储模块；其中，抓取，存储，调度都是通过数据进行交互的，因此，模块之间可以任意平行扩展，对于系统规模来说，只需要平行扩展mysql和redis存储服务集群以及抓取集群即可。当然，简单的扩展会带来一些问题：比如垃圾列表页的泛滥，链接库的膨胀等等问题，这些问题后续在讨论吧。

本文链接

Safe3 Web漏洞扫描系统 v9.6免费版

2011-08-05T01:44:00Z

Safe3 Web漏洞扫描系统是安全伞网络推出的网站安全检测工具，传统的方法往往依靠渗透测试（黑箱、白箱测试），往往局限于测试人员的技术水准高低。

软件界面截图

目前，大多是采用一系列已知攻击手段进行手工检测，且工作量巨大，由于时间关系以及各类网站系统的复杂性程度不同，通常得不到真正有效的评估，国内能从事此类工作的技术人员往往较少，用户最终得到的评估报告往往仅是找到几个系统已知漏洞、某个注入点或者跨站脚本攻击漏洞等常规漏洞。由于评估人员的知识面局限性使得整体评估不够全面，且深度不足。网站的应用逐步增多，更新较快，每隔一段时间应做一次全面检测，若采用传统渗透测试方法，花费昂贵，且往往得不到真正意义上的风险报告。Safe3 Web Vul Scanner使用领先的智能化爬虫技术及SQL注入状态检测技术，使得相比国内外同类产品智能化程度更高，速度更快，结果更准确。

系统适用领域：

国内金融、证券、银行、电子政务、电子商务、教育、网游、综合行业门户、IDC等网站必备检测工具。

技术优势：

SQL注入网页抓取
网页抓取模块采用广度优先爬虫技术以及网站目录还原技术。广度优先的爬虫技术的不会产生爬虫陷入的问题，可自定义爬行深度和爬行线程，网站目录还原技术则去除了无关结果，提高抓取效率。并且去掉了参数重复的注入页面，使得效率和可观性有了很大提高。

SQL注入状态扫描技术
不同于传统的针对错误反馈判断是否存在注入漏洞的方式，而采用状态检测来判断。所谓状态检测，即：针对某一链接输入不同的参数，通过对网站反馈的结果使用向量比较算法进行比对判断，从而确定该链接是否为注入点，此方法不依赖于特定的数据库类型、设置以及CGI语言的种类，对于注入点检测全面，不会产生漏报现象。并且具备绕过WAF、IPS、IDS检测功能，扫描到隐藏的注入点。

超线程和内存回收技术

市面上web漏洞扫描系统通常不能完成腾讯、搜狐等类似门户网站的扫描，原因是第一扫描进度非常慢，第二随着扫描占用系统内存非常高，最终因为系统内存不够而崩溃退出，Safe3 web漏洞扫描系统在这方面表现尤为出色，软件不仅采用线程池等技术保证了很低的CPU占用，还使用自动内存回收功能回收无用的内存，另外软件内部采用独特存储算法，可以在存储千万url地址时扫描速度依然不减，所以如果你的网站非常大，那么Safe3 web漏洞扫描系统是你的首选。

企业扫描报表截图

软件运行需要Microsoft .NET Framework v2.0

新版功能：

v9.6

1.新增错误模式扫描功能 (适合有报错显示的网站)

2.增强企业报表显示

3.增强Form智能识别提交功能

下载地址： http://www.safe3.com.cn/safe3wvs.rar （免费版支持GET型sql注入和XSS漏洞扫描）

本文链接

Varnish+Xcache构建高性能WEB构架初探

2011-07-22T02:00:00Z

本文主要讲述web优化方案和缓存工具的调研及使用。根据目前的测试结果来看，采用varnish+xcache作为 apache和

php缓存这种架构具有高并发、高稳定性，易扩展等优点，服务器的动态请求处理能力是之前的7倍之多。通过分析发现，

目前对服务器的负载主要是在cpu使用方面，随着流量的增加瓶颈也将出现在cpu方面，而内存和IO方面都不是问题。

针对这样的情况，我们就要研究怎么去降低cpu的负载，消除或降低系统的瓶颈。

业务特点分析
U服务采用的是LAMP(Linux Apache mysql php)架构，而服务本身的逻辑比较简单，就是根据不同的url返回特定的页面内容，而这些页面内容基本是不会变的。整个过程是由php动态完成的，不需要和其他服务器交互。在一个请求的响应过程中，系统cpu的消耗基本都在php处理上面。我们要做的就是尽量减少php的动态处理。

优化方案调研
性能优化通常的方法是采用缓存策略。根据U服务的业务特点，优化主要从两个方面进行，php缓存和前端缓存，原理图如下：

Php缓存调研
每次HTTP请求PHP页面时，PHP代码都会被解析和翻译为操作码（PHP 引擎直接执行的原语指令--类似于汇编语言）再执行。在要求很低或可忽略的情况下，服务器看上去能立即执行这个复杂的解释过程。但是一旦处理的页面增加，重复工作就会对服务器造成很大的负担。在某些情况下，“编译”PHP代码的时间会远远超过执行该代码所需的时间，并且会对服务器负载造成很大压力。Php缓存主要是缓存opcode，避免重复编译PHP代码。目前针对php的内存缓存策略主要有 memcache 、 eAccelerator 、 xcache 和 APC 。Memcache是一种基于网络的缓存策略，是一个高性能的分布式的内存对象缓存系统，通过在内存里维护一个统一的巨大的hash表，它能够用来存储各种格式的数据，包括图像、视频、文件以及数据库检索的结果等。从memcache的特点来看，更适合用于分布式数据库和分布式计算领域。光从页面缓存和加速角度来说远不及eAccelerator、 xcache和APC。eAccelerator、 xcache 和 APC 差不多，都是自由开放源码php加速器，优化和动态内容缓存，能够大大提高php脚本性能。它使得PHP脚本在编译的状态下，对服务器的开销几乎完全消除。其中xcache是一种比较新的加速器，是eAccelerator的替代产品，更加稳定和高效，目前shifen前端已在使用。

Http前端缓存调研
针对HTTP的内存缓存策略使用得比较多就是 squid 和 varnish 。Squid 是一种在Linux下使用得比较多的优秀的代理服务器，针对web用户来说，它还是一个高性能的代理缓存服务器，它将数据缓存在内存中，同时也缓存DNS的查询结果，可以加快网络浏览的速度，提高客户机的访问命中率。Squid是目前使用的最广的HTTP加速器之一，目前在百度使用得还很少。Varnish 是另一种高性能的开源 HTTP加速器。Varnish 的作者Poul-Henning Kamp是FreeBSD的内核开发者之一，他认为现在的计算机比起1975年已经复杂许多。在1975年时，储存媒介只有两种：内存与硬盘。但现在计算机系统的内存除了主存外，还包括了CPU内的L1、L2，甚至有L3快取。硬盘上也有自己的快取装置，因此Squid Cache自行处理物件替换的架构不可能得知这些情况而做到最佳化，但操作系统可以得知这些情况，所以这部份的工作应该交给操作系统处理，这就是 Varnish cache设计架构。目前国内有些门户网站如新浪，腾讯正在使用。

缓存工具性能对比及选择
测试平台： HOST：tc-un-ct00.tc OS： Linux 2.6.9_5-4-0-2 #1 SMP CPU: Intel(R) Xeon(R) CPU 5150 @ 2.66GHz × 4 MEM: 8GB 测试数据： jx-rcv00 20081028日18:00 – 19:00 间的数据压力工具： pfmtest
Php缓存工具对比测试
几个主流php缓存工具性能对比测试如下：

前端缓存工具对比测试
apache，squid+apache，varnish+apache 性能对比如下：

测试结论
Php缓存工具测试中，xcache在响应时间，cpu占用的方面均有最佳的表现；另一方面，从前端缓存测试结果来看，varnish更适合U服务的全动态服务。
Varnish及xcache的使用
Varnish的使用
varnish 工具的介绍及详细使用方法参见http://varnish.projects.linpro.no/
Xcache的使用
Xcache的介绍及详细使用方法参加http://xcache.lighttpd.net/

本文链接

IIS监控请求脚本

2011-07-17T00:55:00Z

{3a2a4e84-4c21-4981-ae10-3fda0d9b0f83} 0 5 IIS: WWW Server
{06b94d9a-b15e-456e-a4ef-37c984a2cb4b} 0 5 IIS: Active Server Pages (ASP)
{dd5ef90a-6398-47a4-ad34-4dcecdef795f} 0 5 Universal Listener Trace
{a1c2040e-8840-4c31-ba11-9871031a19ea} 0 5 IIS: WWW ISAPI Extension
{AFF081FE-0247-4275-9C4E-021F3DC1DA35} 0 5 IIS: ASP.NET
{d55d3bc9-cba9-44df-827e-132d3a4596c2} 0 5 IIS: Global
{3b7b0b4b-4b01-44b4-a95e-3c755719aebf} 0 5 IIS: Request Monitor
{DC1271C2-A0AF-400f-850C-4E42FE16BE1C} 0 5 IIS: IISADMIN Global

以上内容保存为 iistrace.guid

del summary.txt
del workload.txt
C:\windows\system32\logman start "NT Kernel Logger" -p "Windows Kernel Trace" (process,thread,disk) -ct perf -o krnl.etl -ets
C:\windows\system32\logman start "IIS Trace" -pf iistrace.guid -ct perf -o iis.etl -ets
@echo 取样分析建议10分钟以内，请及时点击 “停止分析并生成报告” 命令...
pause

保存为start.bat

C:\windows\system32\logman stop "IIS Trace" -ets
C:\windows\system32\logman stop "NT Kernel Logger" -ets
C:\windows\system32\tracerpt iis.etl krnl.etl -o -report -summary
del dumpfile.csv
del iis.etl
del krnl.etl
notepad.exe workload.txt

保存为stop.bat

执行start.bat 十分钟后执行stop.bat 就能获取 10分钟内所有的IIS请求并且有统计结果包括响应速率 cpu使用率请求次数等

本文链接

酒店宾馆IP冲突解决办法

2011-07-13T06:29:00Z

客人在我所供职的酒店上网的时候，经常会弹出一个对话框，显示一些提示，如上网的注意事项和消费标准等信息;并且有自己的电影和歌曲服务器，DHCP-server也是其中的一台服务器，宾馆、酒店就是用这台机器，为客户分配IP地址提供上网功能，即客户把自己的计算机连上网线，网卡配置自动获取IP地址，就会从动态主机配置协议(DHCP)服务器分配到一个IP地址;采用DHCP server可以自动为用户设置网络IP地址、掩码、网关、DNS、Wins 等网络参数，简化了用户网络设置，提高了管理效率。
那么我们的问题也出现了:常见的，很多用户抱怨用这种方法上不了网，但不是所有客户都上不了网。经过调查发现，住宾馆、酒店的人绝大多数是商务人员和工程师，他们携带的手提电脑一般安装的是Windows server版本，server版本默认启动了DHCP server功能，当一台这样的计算机连入网络，在他之后的计算机就会把他当成DHCP服务器，并被分配了不正确的IP地址，从而上不了网。

DHCP服务器地址分配方式

DHCP是一种用于简化主机IP配置管理的协议标准。通过采用DHCP标准，可以使用DHCP服务器为网络上所有启用了DHCP的客户端分配、配置、跟踪和更改(必要时)所有TCP/IP设置。此外，DHCP还可以确保不使用重复地址、重新分配未使用的地址，并且可以自动为主机连接的子网分配适当的IP地址。当一个网络中，有2个或2个以上的DHCP服务器时，提醒切勿将DHCP地址池定义的过大，以免多个地址池之间出现"包含于"的关系，或者是部分客户端手工指定的IP地址包含于DHCP服务器的地址池中，从而造成DHCP的一些异常故障。

针对不同的需求，DHCP服务器有三种机制分配IP地址:

自动分配 DHCP服务器给首次连接到网络的某些客户端分配固定IP地址，该地址由用户长期使用;

动态分配 DHCP服务器给客户端分配有时间限制的IP地址，使用期限到期后，客户端需要重新申请地址，客户端也可以主动释放该地址。绝大多数客户端主机得到的是这种动态分配的地址;

手动分配由网络管理员为客户端指定固定的IP地址。

三种地址分配方式中，只有动态分配可以重复使用客户端不再需要的地址。

每项技术都是有利有弊的，DHCP也不例外，由于DHCP有着配置简单，管理方便的优点，问题也随之产生，由于DHCP的运作机制，通常服务器和客户端没有认证机制，如果网络上存在多台DHCP服务器将会给网络造成混乱。由于用户不小心配置了DHCP服务器引起的网络混乱非常常见，足可见此问题的普遍性。

本人在从事网络工作的几年里，遇到过很多问题，其中有关DHCP-server冲突的不在少数，在解决问题的同时也总结了一些经验，在这里简单介绍一下，与大家分享，希望给在解决此类问题的同行一些帮助，也希望广大高手指出其中的不足和需要改进的地方。

DHCP服务器冲突的解决方法

使用DHCP snooping技术来解决

针对这种DHCP服务器冲突的解决方法有很多，最直接的方法就是贴告示，让入住的客户在上网时关闭Windows的DHCP网络服务，这个选项在‘控制面板'，‘管理工具'里的‘DHCP网络服务'，进入关闭即可。这里要注意的是，非server版的Windows不用关闭，并且不要把‘控制面板'，‘管理工具'，‘服务'中的DHCP client给停止了，这样是分配不到地址的。

当然上面的方法比较被动也不合常理，更不便于我们网络的管理，所以还是应该从我们网络本身出发来解决问题。

既然是DHCP的问题，那么我们就用DHCP的技术来解决问题，比较有代表的就是DHCP snooping技术。DHCP snooping技术是DHCP安全特性，通过建立和维护DHCP snooping绑定表过滤不可信任的DHCP信息，这些信息是指来自不信任区域的DHCP信息。DHCP snooping绑定表包含不信任区域的用户mac地址、IP地址、租用期、vlan-id接口等信息。

首先定义交换机上的信任端口和不信任端口，其中信任端口连接DHCP服务器或其他交换机的端口;不信任端口连接用户或网络。不信任端口将接收到的DHCP服务器响应的DHCP ack 和DHCP off报文丢弃;而信任端口将此DHCP报文正常转发，从而保证了用户获取正确的IP地址。具体配置如下:
配置中的命令都是以CISCO的设备为基础，但不管是哪个公司的设备，总体设计思想是一致的，不同的可能在命令格式上略有差异，工作人员应该根据具体的实际情况来解决相应的问题。

在全局模式下启动DHCP snooping功能，这个默认是关闭的，而且不是所有设备都支持这个功能，最好先看使用说明。

switch(config)#ip dhcp-snooping

如果有vlan就使用下面的命令来监测具体的vlan

switch(config)#ip dhcp-snooping vlan vlan-id

然后定义可信任的端口，默认情况交换机的端口均为不信任端口，通常网络设备接口， TRUNK 接口和连接DHCP服务器的端口定义为可信任端口。

switch(config)#int f0/x

switch(config-if)#ip dhcp snooping trust

使用PVLAN技术来解决

有很多二层的技术可以防止DHCP-server冲突的，PVLAN就是其中一个运用比较广的技术。

PVLAN私有局域网(private vlan)，在PVLAN的概念里，端口有3种类型:Isolated port，Community port, Promiscuous port;它们分别对应不同的vlan类型:Isolated port属于Isolated PVLAN，Community port属于Community PVLAN，而代表一个Private vlan整体的是Primary vlan，前面两类vlan需要和它绑定在一起，同时它还包括Promiscuous port。在Isolated PVLAN中，Isolated port只能和Promiscuous port，彼此之间不能访问;在Community PVLAN中，vlan与vlan之间都不能访问，同一Community vlan的接口可以互相访问，并且所有Community vlan的接口都可以与Promiscuous port进行通信。利用这项技术，我们可以把上连或连接DHCP服务器的接口定义为Promiscuous port，其他接口分配到Isolated vlan里，这样所有接口都只能与上连或DHCP服务器进行通信，即使有一台机器设为DHCP服务器，其他机器也不会与它产生流量，把它做为服务器。

利用这个技术解决DHCP-server冲突的方法有很多，也很灵活，下面介绍一种比较简单的方法，也是用的比较多的:

首先把交换机配置成transparents模式:

switch(config)#vtp mode transparent

顺便可以打开端口的保护功能，它的意思是打开端口保护的端口之间不能访问，但打开保护的端口可以与没有开启此项功能的端口通信，可以根据自己的需求来打开保护功能:

switch(config)#int range f0/124

switch(config-if-range)#switchitchport protected

建立isolated vlan和primary vlan，把isolated vlan定义为primary lan的附属vlan，因为要与primary互相访问:

switch(config)#vlan 14

switch(config-vlan)private-vlan isolated

switch(config)#vlan 44

switch(config-vlan)#private-vlan primary

switch(config-vlan)#private-vlan association 14

本文链接

Safe3 Web应用防火墙14.1版评测

2011-06-23T06:31:00Z

四年前的今天，一款名为“Safe3 Web应用防火墙”的网站安全防护软件华丽面世，从这时开始国内服务器安防领域开始进入到全新时代。昨天安全伞网络科技公司正式发布了Safe3 Web应用防火墙 14.1企业版，版本号也由之前的13.X升级为14.1 周年预览版。
一般来说如此大幅度的版本升级常常都意味着重大功能的加入，那么这次Safe3 Web应用防火墙又会给我们带来哪些惊喜？下面就让我们一同来看一看。

软件名称：	Safe3 Web应用防火墙
软件版本：	14.1 周年版预览版
软件大小：	4.9M
软件授权：	试用版
适用平台：	Win2000 Win2003 Win2008 （32、64位）
下载地址：	Safe3 Web应用防火墙

一、安装与界面
安装过程一如既往地简洁，不过由于新功能的加入，体积较上一版略有增加，达到了4.9MB。整个过程依旧由解压、选择32位或64位安装包、“安装路径”等几个关键步骤组成，除了版本号略有调整外，其余基本上与13.X版无异。
Safe3 Web应用防火墙一向不在安装过程中捆绑其他软件，我们大可放心地一路点击“下一步”完成安装，即便没有太多的电脑应用基础也不用发愁。

图1 安装截图

首次启动后软件将出现功能面板，其中左侧可以直接链接到对应的功能，能够帮助用户更快捷地掌握新功能的位置与属性。

图2 新版主界面

二、实测软件功能

如何提高安全软件的黑客防御能力，一直都是很多厂商都感头疼的问题，虽然web应用防火墙逐渐普及，可大多数厂商仍在不断提高自家产品的拦截性能。在Safe3 Web应用防火墙14.1中，笔者终于见识了什么叫“韩信点兵、多多益善”。

除了传统的web安全防护以外，Safe3 Web应用防火墙加入了web杀毒、网站监控、篡改检查等诸多实用功能，从而让Safe3 Web应用防火墙成为了一款真正意义上的全功能web信息安全产品。当然正所谓“无图无真相”，究竟实用功能能否为Safe3 Web应用防火墙创造奇迹，我们决定还是通过一个真实的评测看一下！

图3 “传说”中的sql注入保护功能

和很多专业web应用防火墙一样，Safe3 web应用防火墙也加入了GET和Cookie防注入功能，但是除了这两种保护还有POST sql防注入功能，这个是微软官方开发的产品UrlScan所不具备的。另外软件还内置了安全伞科技多年积累的超强sql注入防护规则，而市面上很多类似产品黑客可轻易绕过防注入，软件形同虚设。对于高级web管理员，对sql注入和正则表达式比较熟悉的还可以自定sql防注入规则。

图4 功能强大的网站监控功能

网站被黑客上传web后门在平时并不鲜见。而由于web程序员水平参差不齐，很难找出黑客的web后门和漏洞所在网页，而要想杜绝这种问题，光靠web程序员的大海捞针显然不足。这时你就可以使用网站监控功能，它可以监控记录指定类型文件的修改、删除、重命名等所有改动，让黑客对网页的一举一动都在我们的监视之下。我们还可以勾选删除指定类型的新建文件，比如asp文件，这样黑客上传的asp网站后门就可以轻松被干掉，从而杜绝网页上传漏洞带来的进一步危机。

图5 强力的网站后门扫描功能

Safe3 web应用防火墙率先在国内推出第一款第一款能完全扫描网站后门的全功能web杀毒软件。由于软件采用智能脚本解析扫描引擎，查杀率在国内同类产品中摇摇领先。

图6 人性化的日志查看功能

图7 详尽的报表功能

Safe3 web应用防火墙具有企业化的防火墙日志查询功能，并且可以导出详细的日志查看报表，这不仅方便网站管理员分析查看黑客攻击，还可以生成报表便于领导查看。

总结

作为一整套完整的企业web信息安全产品，Safe3 web应用防火墙集成了黑客攻击防御、后门查杀、网站监控、篡改检查、报表生成等多项安全防护及企业报表功能，已经发展成为一款复合型的优秀网站安全产品。此次发布的Safe3 web应用防火墙14.1则是安全伞网络科技发展史上一个里程碑式的作品，可以说，Safe3 web应用防火墙是一款融入健康理念的安全产品，它不仅仅以保护网站安全为使命，更把保护网络健康为责任，相信Safe3 web应用防火墙一定能够成为每一位站长的必备软件！

本文链接

SQL Injection

2011-06-03T13:12:00Z

SQL injection is an attack in which malicious code is inserted into strings that are later passed to an instance of SQL Server for parsing and execution. Any procedure that constructs SQL statements should be reviewed for injection vulnerabilities because SQL Server will execute all syntactically v.