During the development of our DataVault™ service we came across some areas of the code that did not seem well suited to TDD. These were areas of the code which had dependancies on external sites that we needed to access using the HttpWebRequest / Response objects. As anyone who’s attempted to write code using these objects is aware they are intrinsically difficult to mock; HttpWebResponse has a private constructor and it leaves you with two options:

  1. You can use an intercepting proxy to fake the presence of the real web server
  2. You can create an abstraction of the HttpWebRequest/Response classes and use IoC to allow us to inject stubbed variants so that the method under test remains largely intact.

Whilst both of the above methods will work they both have inherent issues. A is clumsy; in that we need an external process to intercept the calls and we’re not truly ‘unit’ testing the code; it’s turning into a little bit of an integration test. B will allow us to unit test the original method body; however it’s at the cost of introducing a further level of abstraction and it still requires us to come up with some method of testing the additional class(es) that we will inevitably introduce (or trusting that they are trivial enough not to need tests – risky!)

Luckily for us; Moles allows us to take a third option – we can Mock the HttpWebResponse objects directly.

Enter Microsoft Moles!

Download Microsoft Moles

Download Microsoft Moles


Let’s describe a trivial problem that we may construct as an example:

  • Before we attempt to crawl a site we need to ensure that we can connect to the web server and obtain responses from our requests.

This may seem slightly simplistic; but it will demonstrate the method which we can use to accomplish our goals.

Initial thoughts would lead us to this simple method declaration:

1
2
3
4
5
6
7
public class Crawler
{
   public bool CanWeConnect(string Url)
      {
         return false;
      }
}

Simple enough… we pass in a Url; and expect to get true or false depending on our ability to connect to the target site.

Ok; now the first thing we need to do is to enable Moles for the System dll. This gets us access to the Moled variants of the system assembly – If you are not familiar with this process then create a new test project and (assuming you have Moles installed) right click on the System assembly within that project and select ‘Add Moles Assembly’ – then compile the test project (this last step is important; the moles assembly is generated at compile time so you won’t have access to the new assembly until your test project has been compiled)

Next; we can construct an initial test to ensure that we call WebRequest.Create – note the use of Moles in the HostType attribute.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[TestMethod()]
[HostType("Moles")]
public void CanWeConnect_ShouldCallCreateWebRequest()
{
   var urlToTest = "http://www.google.com/";
   var targetUnderTest = new Crawler();
   var createWebRequestCalled = false;
 
   MWebRequest.CreateString = (x) =>
   { 
      createWebRequestCalled = true; 
      return null; 
   };
 
   targetUnderTest.CanWeConnect(urlToTest);
   Assert.IsTrue(createWebRequestCalled);
}

Obviously; if we run this test it will fail; so we need to add code to the main method to ensure this test passes:

   var request = (HttpWebRequest)WebRequest.Create(Url);
   return false;

This will run; however we are returning null from our delegate – In practise we will be using the getRequest object throughout the code – and we don’t really want to have to make sure we actually got an object back from the system before we use it. But HttpWebRequest has a private constructor – so what can we return ? The answer is simple: our Moled System assembly exposes a new object to use – MHttpWebRequest; and this can be used as a direct replacement for HttpWebRequest – and most importantly it’s got an anonymous constructor. Our test then becomes this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[TestMethod()]
[HostType("Moles")]
public void CanWeConnect_ShouldCallCreateWebRequest()
{
   var urlToTest = "http://www.google.com/";
   var mockedRequest = new MHttpWebRequest();
   var targetUnderTest = new Crawler(); // Instantiate the class under test
   var createWebRequestCalled = false;
 
   MWebRequest.CreateString (x) => 
   { 
      createWebRequestCalled = true; 
      return mockedRequest; 
   };
 
   targetUnderTest.CanWeConnect(urlToTest);
   Assert.IsTrue(createWebRequestCalled);
}

Now; we need to make sure that our code attempts to get a response from the server; so again another test method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[TestMethod()]
[HostType("Moles")]
public void CanWeConnect_ShouldCallGetResponse()
{
   var urlToTest = "http://www.google.com/";
   var mockedResponse = new MHttpWebResponse();
   var targetUnderTest = new Crawler();
   var getResponseCalled = false;
 
   MHttpWebRequest.AllInstances.GetResponse = (x) =>
   {
      getResponseCalled = true; 
      return mockedResponse;
   };
 
   targetUnderTest.CanWeConnect(urlToTest);
   Assert.IsTrue(getResponseCalled);
}

You’ll see that we don’t need to mock the whole HttpWebRequest object at this stage; we’ve already got a test to explicitly ensure we are creating it – you could fairly easily combine the two tests at this point – it depends on how closely you like to stick to the ‘One test should test one thing’ mantra.

So; we’ve another failing test – let’s fix that by adding some more code to our method:

   var request = (HttpWebRequest)WebRequest.Create(Url);
   var response = request.GetResponse() as HttpWebResponse;
   return false;

We can see that we have successfully created a pass in the second test; however we have also caused the first test to now fail. If you inspect the test result we can see that this is because the code in the method under test is attempting to excercise some un-mocked methods on the mocked web response. There are two options to deal with this in Moles; the easiest way is to call mockedRequest.BehaveAsDefaultValue() – this will (in principle) simply pass the call through to the original class; however – remember we have instantiated our mock with an anonymous constructor and so this method isn’t available to us. We will have to include the WebResponse mocking we have used in the second test. So now; our original test method looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
public void CanWeConnect_ShouldCallCreateWebRequest()
{
   var urlToTest = "http://www.google.com/";
   var mockedRequest = new MHttpWebRequest();
   var mockedResponse = new MHttpWebResponse();
   var targetUnderTest = new Crawler();
   var createWebRequestCalled = false;
 
   MWebRequest.CreateString = (x) =>
   {
      createWebRequestCalled = true; 
      return mockedRequest;
   };
 
   MHttpWebRequest.AllInstances.GetResponse = (x) => 
   { 
      return mockedResponse; 
   };
 
   targetUnderTest .CanWeConnect(urlToTest);
   Assert.IsTrue(createWebRequestCalled);
}

We now have both tests working. In practice; the shared setup of the mock objects could be moved into a shared method and either called once before all tests (to set up the method moling) or before each individual test (if there is some data that needs to be reset to a known state prior to each test)

Now finally; we need to make sure that we react to both a successful web response and to an unsuccessful response so we will create a pair of tests at this point:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
[TestMethod()]
[HostType("Moles")]
public void CanWeConnect_ShouldReturnTrueWithOkResponse()
{
   var mockedWebResponse = new MHttpWebResponse();
   var urlToTest = "http://www.google.com";
   var targetUnderTest = new Crawler();
 
   MHttpWebRequest.AllInstances.GetResponse = (x) =>
   {
      return mockedWebResponse;
   };
 
   mockedWebResponse.StatusCodeGet = () => 
   { 
      return HttpStatusCode.OK; 
   };
 
   Assert.IsTrue(targetUnderTest.CanWeConnect(urlToTest));
}
 
[TestMethod()]
[HostType("Moles")]
public void CanWeConnect_ShouldReturnFalseWithHttpListenerException()
{
   var mockedWebResponse = new MHttpWebResponse();
   var urlToTest = "http://www.google.com";
   var targetUnderTest = new Crawler();
 
   MHttpWebRequest.AllInstances.GetResponse = (x) =>
   {
      return mockedWebResponse;
   };
 
   mockedWebResponse.StatusCodeGet = () => 
   { 
      throw new HttpListenerException(); 
   };
 
   Assert.IsFalse(targetUnderTest.CanWeConnect(urlToTest));
}

We can now update the method to pass the final two tests:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public bool CanWeConnect(string Url)
{
   var request = (HttpWebRequest)WebRequest.Create(Url);
   var response = request.GetResponse() as HttpWebResponse;
   try
   {
      if (response.StatusCode == HttpStatusCode.OK)
         return true;
   }
   catch (HttpListenerException ex)
   {
      Debug.WriteLine(String.Format("Unable to connect to {0}", Url));
   }
 
   return false;
}

You will notice that we have once again ‘broken’ the first two tests we wrote by utilising an un-mocked property in the method – You can fix that by either adding the StatusCodeGet mocks to the original tests or by refactoring the code to share the setup as mentioned earlier.

Tags: , , , , , ,

4 Responses to “Mocking HttpWebResponse with Moles”

  1. Albert Sterlott March 17, 2011 at 1:38 pm #

    Whoa, just what I’ve been looking for!

  2. C#Cowboy
    Twitter:
    March 17, 2011 at 1:57 pm #

    Thanks, I found this post really useful in my implementation of a web scraping object. My code is below:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    
    [TestMethod]
    [HostType("Moles")]
    [Description("Tests that the default scraper returns the correct result")]
    public void Scrape_KnownUrl_ReturnsExpectedValue()
    {
    	var mockedWebResponse = new MHttpWebResponse();
     
    	MHttpWebRequest.AllInstances.GetResponse = (x) =>
    	{
    		return mockedWebResponse;
    	};
     
    	mockedWebResponse.StatusCodeGet = () => { return HttpStatusCode.OK; };
    	mockedWebResponse.ResponseUriGet = () => { return new Uri("http://www.google.co.uk/someRedirect.aspx"); };
    	mockedWebResponse.ContentTypeGet = () => { return "testHttpResponse"; }; 
     
    	var mockedResponse = "<html> \r\n" +
    					 "  <head></head> \r\n" +
    					 "  <body> \r\n" +
    					 "     <h1>Hello World</h1> \r\n" +
    					 "  </body> \r\n" +
    					 "</html>";
     
    	var s = new MemoryStream();
            var sw = new StreamWriter(s);
     
    	   sw.Write(mockedResponse);
               sw.Flush();
     
               s.Seek(0, SeekOrigin.Begin);
     
            mockedWebResponse.GetResponseStream = () => s;
     
    	var scraper = new DefaultScraper();
    	var retVal = scraper.Scrape("http://www.google.co.uk");
     
    	Assert.AreEqual(mockedResponse, retVal.Content, "Should have returned the test html response");
    	Assert.AreEqual("http://www.google.co.uk/someRedirect.aspx", retVal.FinalUrl, "The finalUrl does not correctly represent the redirection that took place.");
    }
  3. Stefan June 29, 2011 at 9:24 am #

    Hi Supernerd

    Really good reading. But have you tryd this with VS2010 .Net fw 4.0.

    I cant Mole System assambly in any way. Need a Mole for the HttpWebRequest and HttpWebResponse but it wont work. Seems like other people have the same issue.

    Microsoft says that Pex and Moles fully supports fw 4.0 but I dont think it does.

    Nice post anyway, this would have solved ny problem.

  4. Stefan June 29, 2011 at 12:17 pm #

    Now its working. But I had to revert the test project to framework 3.5. Do the System.Moles, and then change back the target framework to 4.0.

    The moles console runner complained a little when compiling. Its allways something with the stubs that the System.Mole.dll contains. Anyway, after building the hole solution it seems like Moles are accepted and I now have the System.Net.Moles.MHttpWebRequest available.

    Big fuzz for such a small thing.

Leave a Reply