WebClient, HttpWebRequest or HttpClient to perform HTTP requests in .NET?

The most common reasons for performing HTTP requests from code are consuming HTTP APIs that have no wrapper library available, and scraping web content. Without resorting to third-party libraries, there are basically three managed ways to perform an HTTP request from .NET:

The post is supposed to be both a reference, as it contains many links to documentation and Q&A sites, as well as an evaluation of which of the three APIs is the most versatile and streamlined. To prove the latter, I attempt to perform exactly the same operations on all implementations, such as “upload a string as request body with the PUT method and download the response body as a statically-typed model instance”, and compare how tedious and transparent the resulting code for each implementation is.

In this post, I will compare the three classes on the following properties:

Availability and portability

HttpClient was introduced in .NET 4.5, and it’s available as a NuGet package for .NET 4. HttpWebRequest and WebClient are available in all .NET versions. The WebClient class is not available in the Portable Class Library, nor is it for Store apps, but the other two are (albeit partially).

You can verify these claims under ‘Version Information’ at their respective MSDN pages linked to at the top of this post. For individual  methods or properties, check the icon to their left ( Portable Class Library icon (MSDN) for Portable Class Library, Windows Store App icon for Store apps).

Retrieving the response stream

From the response to an HTTP request, you will usually want to get the response body (if any) as a Stream, in order to pass it to other classes as transparently as possible.

This and following code uses an instance of the class RequestParameters, made up by me, which contains what its name suggests. The properties show are also named obviously, like Uri to contain the Uri to make the request to. Additional properties are introduced along the way when required. Unless mentioned otherwise, the methods shown don’t do any error handling, and most of them should not be used without understanding what can be wrong with the demonstrated code, which I also try to explain. At first the code will be synchronous.

I’ll show the easiest way to perform a GET request with the HttpWebRequest class first. Remember this is the most basic code that fails in many common cases, especially for the HttpWebRequest, so don’t bluntly copy-paste this code but read on to learn about proper use of these APIs.

System.IO.Stream GetResponseStream(RequestParameters parameters)
{
  HttpWebRequest request = 
    (HttpWebRequest)WebRequest.Create(parameters.Uri);

  WebResponse response = request.GetResponse();
 
  return response.GetResponseStream();
}

Note that though the returned WebResponse (actually an httpWebResponse) implements IDisposable, its documentation states:

You must call either the Stream.Close or the HttpWebResponse.Close method to close the response and release the connection for reuse. It is not necessary to call both Stream.Close and HttpWebResponse.Close.

In fact, when the above method would wrap response in a using block, actually using the returned stream would throw the exception: "Stream was not readable".

Which means it is up to the caller of this method to dispose the response stream, as you should always do when a method returns a Stream that won’t be needed afterwards.

Returning the response stream for a request made using WebClient is done like this:

System.IO.Stream GetResponseStream(RequestParameters parameters)
{
  using (var client = new WebClient())
  {
    return client.OpenRead(parameters.Uri);
  }
}

Since WebClient uses HttpWebRequest internally, and OpenRead() remotely resembles the first block of code shown, it is safe to dispose the client here.

The same call using HttpClient (made synchronous by accessing Task<T>.Result) looks alike, but after performing the request we have to call EnsureSuccessStatusCode() on the response in order to achieve the same behavior as the two methods shown above, which is throwing an exception upon receiving an HTTP error response code for the request:

System.IO.Stream GetResponseStream(RequestParameters parameters)
{
  using (var client = new HttpClient())
  {
    var result = client.GetAsync(parameters.Uri).Result;
    result.EnsureSuccessStatusCode();
    return result.Content.ReadAsStreamAsync().Result;
  }
}

Later in this post I’ll handle more on asynchronous use and error handling.

As you can see, all of this is pretty straight-forward. You can now for example feed this stream to parsers, or write it to a storage mechanism.

Reading the response as string

The result body of a web request can sometimes be more useful as a string. The HttpWebRequest has no built-in method for this, so we’ll have to wrap its response stream in a StreamReader and read that till the end:

string GetResponseString(RequestParameters parameters)
{
  using (var reader = new StreamReader(GetResponseStream(parameters)))
  {
    return reader.ReadToEnd();
  }
}

The WebClient doesn’t need a separate call to get the stream:

string GetResponseString(RequestParameters parameters)
{
  using (var client = new WebClient())
  {
    return client.DownloadString(parameters.Uri);
  }
}

And again, the equivalent HttpClient calls requires explicit error handling, but also accommodates a built-in way of returning the response as a string:

string GetResponseString(RequestParameters parameters)
{
  using (var client = new HttpClient())
  {
    var result = client.GetAsync(parameters.Uri).Result;
    result.EnsureSuccessStatusCode();
    return result.Content.ReadAsStringAsync().Result;
  }
}

So reading a response as string also doesn’t seem too hard. All three ways can however cause encoding problems, which are explained later in this article.

Reading machine-readable responses

When consuming XML or JSON, you’ll want to be able to use an object representing that response. For demonstration purposes I created the Friend class:

public class Friend
{
  public int ID { get; set; }

  public string Name { get; set; }

  public DateTime BirthDate { get; set; }
}

And when some URL returns a JSON object representing such a friend, you can easily deserialize that into an instance of that class, for example using NewtonSoft.Json (also available on NuGet):

var jsonString = httpClient.GetResponseString(parameters);
Friend friend = JsonConvert.DeserializeObject(jsonString);

Where httpClient is an instance of a class that implements any of the three GetResponseString() methods shown above.

XML is just as easy, given the earlier shown calls are performed on an ASP.NET WebAPI controller, which can also returns an XML serialization of the output. Note that using the following API, the XmlSerializer, we have to specify the namespace returned in the XML by the Web API. Other .NET classes and libraries may offer different approaches. You can omit the namespace for the XmlSerializer if you place the proper attributes on the Friend class, but that’s quite intrusive if you want to be able to reuse that class.

So the code to read and deserialize XML with the HttpClient looks like this:

var xmlStream = httpClient.GetResponseStream(parameters);
string nsUrl = 
  @"http://schemas.datacontract.org/2004/07/CodeCaster.HttpRequests.Models";

var serializer = new XmlSerializer(typeof(Friend), nsUrl);
Friend friend = (Friend)serializer.Deserialize(xmlStream);

Now in both cases, using XML or JSON, the result is a Friend instance that you can use statically-typed, so you can ‘dot into’ an HTTP response body like it is an object:

if (friend.Name == "...") { ... }

But as you can see, once you’re able to abstract reading a string or stream out of the client, the implementations of GetResponseStream() in HttpWebRequest, WebClient and HttpClient as shownn above are interchangeable.

Can you do the same thing just as easily for sending data?

Sending data

In order to access request or response data using HttpWebRequest, you’ll have to access the respective streams. There are no more convenient methods available, so uploading data is achieved the same way as downloading it:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(parameters.Uri)
{
  Method = parameters.RequestMethod,
  ContentType = parameters.ContentType,
};

byte[] bytes = parameters.GetFriendBytes();
Stream stream = request.GetRequestStream();
stream.Write(bytes, 0, bytes.Length);
stream.Close();

return request.GetResponse();

The WebClient has various ways to upload data, the most reusable one being UploadData():

using (var client = new WebClient())
{
  client.Headers[HttpRequestHeader.ContentType] = parameters.ContentType;
  return client.UploadData(parameters.Uri, 
                           parameters.RequestMethod, 
                           parameters.GetFriendBytes())
}

HttpClient is pretty explicit, so it’s easy to read what’s going on:

using (var client = new HttpClient())
{
  var request = new HttpRequestMessage
  {
    RequestUri = parameters.Uri,
    Method = parameters.RequestMethod,
  };
  
  request.Content = 
    new ByteArrayContent(parameters.GetFriendBytes());

  request.Content.Headers.ContentType = 
    new MediaTypeHeaderValue(parameters.ContentType);

  var result = client.SendAsync(request).Result;
  result.EnsureSuccessStatusCode();
  return result;
}

Custom request methods

The property parameters.RequestMethod used above is a string, so it’s very easy to create requests using custom request methods (or ‘verbs’), illustrated here through the use of “PROPFIND”:

// HttpWebRequest
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(parameters.Uri)
request.RequestMethod = "PROPFIND";

using (var response = request.GetResponse())
{
  ...
}

// WebClient
using (var client = new WebClient())
{
  client.UploadData(..., "PROPFIND", ...);
}

// HttpClient
using (var client = new HttpClient())
{
  var request = new HttpRequestMessage
  {
    Method = "PROPFIND",
  };

  var result = client.SendAsync(request).Result;
  result.EnsureSuccessStatusCode();
}

There are certain overloads available which accept enums, or you could use constants, as in HttpMethod.

Reading and writing default and custom headers

You can access the most common headers for HttpWebRequest through properties named like the header, such as Accept before issuing the request. See the HttpWebRequest Properties documentation for all its properties.

Custom headers can be set in the Headers collection of the request:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(parameters.Uri)

request.Accept = "application/xml";
request.Headers["Foo"] = "Bar";

When you try to set a header through the Headers collection for which a property exists, such as:

request.Headers["Accept"] = "application/json";

The code will throw an exception: "The ‘Accept’ header must be modified using the appropriate property or method". The properties for which that happens are listed here.

Reading headers from an HttpWebResponse is just as easy, with common headers in the HttpResponseHeader enumeration or any header using a string:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(parameters.Uri);

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
  string date = response.Headers[HttpResponseHeader.Date];
  string contentType = response.Headers["content-type"];
}

When a header is not found, as in the result of response.Headers["whargarbl"], null is returned.

The WebClient allows for three ways to set a common request header. The last two can be used for custom headers also, by passing a string instead of the HttpRequestHeader enum:

using (var client = new WebClient())
{
  client.Headers[HttpRequestHeader.Accept] = "text/html";
  client.Headers["accept"] = "application/xml;q=1";
  client.Headers.Add("Accept", "application/json;q=0.8");
	
  return client.UploadData(...);
}

As you notice, the header name is case-insensitive, just as in the HTTP protocol. The second expression overwrites the value set in the first (and the same happens when they are in reverse order), while the third statement adds the value to the header if it is already present and creates it otherwise.

Given both the name and the value are plain strings, you can put anything in there you like, as demonstrated by the qvalue weights added to the values:

Accept: application/xml;q=1,application/json;q=0.8

Because most of WebClient‘s methods return a string or Stream, the response headers are set in the client instance itself after calling a request method, in its ResponseHeaders property:

using (var client = new WebClient())
{
  string responseBody = client.DownloadString(parameters.Uri);
  
  string date = client.ResponseHeaders[HttpResponseHeader.Date];
  string contentType = client.ResponseHeaders["content-type"];
  string willBeNull = client.ResponseHeaders["whargarbl"];
}

The HttpClient‘s HttpRequestMessage.Headers property, of type HttpRequestHeaders, allows for accessing a variety of object-oriented headers.

Just like the HttpWebRequest, the headers class has properties like Accept and UserAgent, and you can again set headers by string, even if they have a property counterpart.

var request = new HttpRequestMessage
{
  var xml = new MediaTypeWithQualityHeaderValue("application/xml");
  var json = new MediaTypeWithQualityHeaderValue("application/json", 0.8);
  
  request.Headers.Accept.Clear();
  request.Headers.Accept.Add(xml);
  request.Headers.Accept.Add(json);
  request.Headers.Add("Accept", "text/html;q=0.5");
}

When this request is executed, the resulting Accept-header that will be sent is:

Accept: application/xml, application/json; q=0.8, text/html; q=0.5

Note that the collection of field values are separated by a comma followed by whitespace (", "), which is done by the HttpHeaders class, while the whitespace is optional. Content-specific headers can be set on the request.Content.Headers property, such as ContentType.

Using the HttpClient‘s HttpResponseMessage it is pretty awkward to read headers. The logic is contained in the System.Net.Http.Headers. Every header is supposed to support multiple values, so any header value is returned as a string collection. There is a convenience method you can use to pry out the header values, TryGetValues(), to re-establish the behavior shown by the HttpWebResponse and WebClient with their Headers dictionary (you could also use FirstOrDefault() which would result in the same amount of code):

private string GetHeaderValue(HttpHeaders headers, string needle)
{
  IEnumerable<string> values;
  if (headers.TryGetValues(needle, out values))
  {
    return string.Join(",", values);
  }
  return null;
}

You can also use headers.GetValues(), but that will throw an InvalidOperationException when the requested header is not present: "The given header was not found". Content headers, such as content-type, are missing from the response.Headers collection: you’ll have to dive into response.Content.Headers. The HttpClient code equivalent to the header reading code shown for the HttpWebResponse and WebClient, using the method above, looks like this:

private void ReadHeaders(HttpResponseMessage response)
{
    string date = GetHeaderValue(response.Headers, HttpResponseHeader.Date.ToString());
    string contentType = GetHeaderValue(response.Content.Headers, "content-type");
    string willBeNull = GetHeaderValue(response.Headers, "whargarbl");
}

Handling errors

Now you know how to make any HTTP (<2) request using HttpWebRequest, WebClient and HttpClient: you can use any request method, add arbitrary headers and read the response either as stream or string. So what could possibly go wrong? A lot. First there's user error, for example caused by providing the wrong values or calling certain methods at inappropriate times. This will raise exceptions that carry extremely helpful, albeit sometimes somewhat cryptic messages, explaining the details of the cause of the error. I will not go into any detail on this kind of error. Use your Google-fu, and ye shall find. The kind of error you are interested in are errors like connection failures or timeoutss before or during transport (uploading or downloading data), and HTTP errors in the 400 and 500 ranges that are returned by the server.

For HttpWebRequest, the HTTP and transport related errors occur during the call to request.GetResponse(). So in order to catch HTTP errors, you catch the WebException first. Then you might want to consider catching all other exceptions, but that depends on how well layered your code is. The calling code in this example expecting a JSON string in return to deserialize it and display some of its values on a web page, so that seems like the proper place to catch the exceptions in this case in order to just display the error, but in other cases you might want to create an application-specific way to handle a failed request appropriately.

string body;

try
{
  var jsonString = implementation.GetResponseString(parameters);
  Friend Friend = FriendSerializer.DeserializeJson(jsonString);

  body = string.Format("Friend: [ID: {0}, Name: {1}, BirthDate: {2}]", 
                             Friend.ID, Friend.Name, Friend.BirthDate);
}
catch (WebException wex)
{
  body = "(!) A WebException occurred: " + wex.Message;
}
catch (Exception ex)
{
  body = "(!) An Exception occurred: " + ex.Message;
}

Again, implementation can be any of the three implementations of the GetResponseString() method shown above. The only difference is for HttpClient, on which you can call EnsureSuccesStatusCode in order to make it throw an exception (i.e. to behave uniform with the other two, older APIs). Note this will be an Exception, not a WebException.

The WebException is quite helpful when caught. First there is the Status property which tells you the source of the error. Common values are ConnectFailure and ProtocolError. When you cast the Response Property to an HttpWebResponse, you can inspect what the server returned, if anything.

When a non-protocol related issue such as a network error is the cause of the exception the Response may be null. So, to catch and print a WebException, you can use this code:

catch (WebException wex)
{
  body = string.Format("(!) A WebException occurred, status '{0}'", wex.Status);

  var response = (HttpWebResponse)wex.Response;
  if (response != null)
  {
    body += string.Format(", response: {0} {1}: {2}", 
              (int)response.StatusCode, response.StatusDescription, wex.Message);
  }
}

The result of a call made with the HttpClient, a HttpResponseMessage can be inspected like this:

using (var client = new HttpClient())
{
  var result = client.GetAsync(parameters.Uri).Result;

  if (!result.IsSuccessStatusCode)
  {
    HttpStatusCode code = result.StatusCode;

    if (code == HttpStatusCode.NotFound)
    {
      // ...
    }
  }
}

Apart from the status code, which may be useful for debugging during development and for logging, you can also read the response headers and body. For HttpWebRequest and WebClient, it’s as easy as calling response.GetResponseStream() and passing that on to the appropriate processor, like a StreamReader to read the response as string, or an XML or JSON deserializer of choice, making the code to handle and read a WebException using HttpWebRequest or WebClient like this:

catch (WebException wex)
{
  body = string.Format("(!) A WebException occurred, status '{0}'", wex.Status);

  var response = (HttpWebResponse)wex.Response;
  if (response != null)
  {
    if (response.ContentLength > 0)
    {
      using (var reader = new StreamReader(response.GetResponseStream()))
      {
        string responseBody = reader.ReadToEnd();

        body += ", Response body: " + responseBody;

        // Or (from: http://stackoverflow.com/a/2053222/266143):
        reader.BaseStream.Position = 0;
        reader.DiscardBufferedData();

        using (var textReader = new JsonTextReader(reader))
        {
          body += ", Response JSON: ";

          dynamic responseJson = new JsonSerializer().Deserialize(textReader);
          body += "Message: " + responseJson.Message;
        }
      }
    }
    else
    {
      body += string.Format(", response: {0} {1}: {2}",
	    (int)response.StatusCode, response.StatusDescription, wex.Message);
    }
  }
}

In this case the reponse is dynamic, but when consuming an API you might want to model its documented exception models in according classes and deserialize the reponse body to an instance thereof.

Accessing the response headers looks familiar:

string errorHeader = response.Headers["ErrorHeader"];

if (!string.IsNullOrEmpty(errorHeader))
{
    body += string.Format(", ErrorHeader value: '{0}'", errorHeader);
}

HttpClient offers a more streamlined access to the response, whether there was an HTTP error or not, so you don’t have to use exceptions in your higher layer of code:

using (var client = new HttpClient())
{
  var request = new HttpRequestMessage
  {
     RequestUri = parameters.Uri,
  };

  var response = client.SendAsync(request).Result;

  string responseBody = response.Content.ReadAsStringAsync().Result;

  if (response.IsSuccessStatusCode)
  {
    responseBody = "Friend object received: ";
    responseBody += FriendSerializer
              .DeserializeJson(responseBody)
              .ToString();
  }
  else
  {
    var textReader = new JsonTextReader(new StringReader(responseBody));
    dynamic responseJson = new JsonSerializer().Deserialize(textReader);
    responseBody = "Deserialized JSON error message: " + responseJson.Message;
  }
  
  return responseBody;
}

Instead of writing directly to the responseBody variable and returning a string, you may of course consider to return the appropriate deserialized object to a view engine, or return the error in a format that the program can handle or display. Alternatively you can decide to throw a custom exception upon receiving an HTTP error, containing properties you can use to inspect and/or log the error the higher layer of your program, or you could return null to indicate an error. In all cases it depends on what you want to return to the caller.

Preliminary conclusion

This post was in my Drafts for way too long, I’ve now posted it as-is. It may be outdated in some constructs, but I hope it conveys the message. Other topics I wanted to address, and maybe will in a future post include: encoding, redirection, cookies, proxy settings and asynchronous use of the various classes.

This entry was posted in Tech. Bookmark the permalink.

Leave a Reply