How to Retrieve and Cache HTTP Data in ASP.NET

Introduction

Retrieving Data from a Remote Server via HTTP

数据挖掘交友

Caching Data from a Remote Server via HTTP

Conclusion

数据挖掘交友

数据挖掘工具

Now that you′ve seen how easy it is to cache objects in ASP.NET I hope you′ll use the powerful built-in caching for lots of things, but at the very least you really should use it to cache HTTP requests. It not only makes your site run faster, but it also cuts down on network traffic and causes less strain on the remote server as well. The only question left is how long should something be cached? Well that depends on what it is and the answer will be different in each case, but caching something for even just a minute can make a huge difference on a busy site. 数据挖掘工具

I first learned how big a deal caching is when I was trying to upgrade an old computer. I had just gotten a new hard drive and had installed it along with my old one in order to copy over all my data. I booted off of a plain old boot floppy and started using XCopy to transfer the files from one drive to the other. It was painstakingly slow. So slow that I thought there must be a problem with the new drive. Well after several hours of troubleshooting and a few phone calls later, I discovered just how big a difference SMARTDrive can make. I′ve been a huge fan of caching ever since.

So why is caching of HTTP requests so important. Because they′re one of the slowest things you can ask a web server to do. HTTP requests are even slower then most database requests. At least your database is usually located on a local network. HTTP requests can be sent anywhere on the Internet and there are plenty of things that can go wrong before you get a response. First you need to resolve the IP address of the server you′re trying to contact via DNS. Then your request needs to get to the server. The server then needs to respond, and finally you need to receive the response. We take for granted a lot of the steps that go into making a successful HTTP request because of how reliable the Internet has become, but there are a number of things that need to happen in order for you to get a response and they don′t always all happen quickly. 数据挖掘论坛

http_cached.aspx

<%@ Page Language="VB" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>
<script runat="server">
    Sub Page_Load(Src as object, E as EventArgs)
        Dim strRssFeedText As String

        ′ Retrieve the entry from the cache
        strRssFeedText = Cache("15SecsRssFeedText")

        ′ If the data retrieved is null then we
        ′ get it from the remote server and insert
        ′ it into the cache for future calls.
        If strRssFeedText = Nothing Then

数据挖掘实验室

Notice the lines in red in the listing above. They are the key lines that handle the caching. The first one retrieves the cached data. We then check to see if any data was returned on the second red line. If it was, then we use that data to build the page and never make the HTTP request to the remote server. If nothing was returned from the cache then we request the data from the remote server. The third red line then stores a copy of the data in the cache for future calls to the page.

数据挖掘论坛

The ASP.NET caching engine is quite robust and allows you to specify when a cached item should expire in a number of ways. That′s why the Cache.Insert method looks so confusing. The version I′m using has the following syntax: 数据挖掘实验室

Insert(key As String, value As Object, dependencies As CacheDependency, absoluteExpiration As DateTime, slidingExpiration As TimeSpan)

Key and value are easy enough. The key is how you identify the data you′re caching. And the value is the actual data you want to cache. The dependencies are used if you want to make the cache dependent on other objects (usually a file). I′m not using it so I pass in Nothing. In the example above I′m using the absolute expiration parameter to make the cached data expire 15 minutes from the time it was cached. Sliding expiration allows you to say don′t remove this from the cache unless it isn′t used for a certain amount of time. It′s like a countdown timer that keeps getting reset each time the cache is used. If it doesn′t get reset before it reaches zero then the cache is flushed. 数据挖掘论坛

For more information on the Insert method of the Cache object see Cache.Insert Method in the documentation on MSDN. The method is overloaded, but I′m using the 3rd version which allows you to set expiration policies.

When you run the cached version of the script you′ll once again see the 15 Seconds RSS feed retrieved. This time however, the timestamp at the bottom doesn′t change with every refresh. That′s because you′re seeing the version from the cache instead of retrieving a new copy every time. However, if you wait fifteen and then refresh the page you′ll get a new timestamp which will then be returned for the next 15 minutes. 数据挖掘交友

I′d been using the same script to retrieve HTTP data in classic ASP for years without it ever giving me any trouble. So, when I needed to write a .NET version, I figured I′d just convert my existing script and keep using it.

For those who are interested in seeing the classic ASP version, I wrote up it up in a short article a few years back: A Simple Method for Caching HTTP Requests

Well it turned out that there wasn′t much in the script that didn′t need to change. Since we′re now using the .NET Framework instead of COM Components, the whole HTTP section needed to change, and since .NET now has built-in caching facilities, I no longer needed to resort to using an Application variable in order to get in memory caching. So much for converting the old script... it was time to start fresh.

The first step is obviously to just retrieve the data from a remote server via HTTP. The process really isn′t that complex, but it can look that way because of the number of objects involved.

http.aspx 数据挖掘研究院

<%@ Page Language="VB" %>
<%@ Import Namespace="System.Net" %>
<%@ Import Namespace="System.IO" %>
<script runat="server">
    Sub Page_Load(Src as object, E as EventArgs)
        Dim strRssFeedText As String

        ′ Call the GetWebPageAsString function to retrieve
        ′ the 15 Seconds RSS feed as text.
        strRssFeedText = GetWebPageAsString( _
            "http://www.15seconds.com/dyna/15secs_rss.asp")

        ′ HTMLEncode the RSS feed and display it in the browser.
        litTextRetrieved.Text = Server.HTMLEncode(strRssFeedText)

When run, the script above makes a request to "http://www.15seconds.com/dyna/15secs_rss.asp", which just happens to be the address of 15 Seconds′ RSS feed. At the bottom of that feed is a comment indicating the current time at the server... and yes... it′s usually a few minutes off. 数据挖掘交友

In today′s world of Web services and RSS feeds, it seems like every application you build needs to pull in data from somewhere. Luckily, ASP.NET makes retrieving data from a remote server via HTTP extremely simple... maybe too simple. Many applications are brought to their knees because for every request that comes in, multiple requests go out. 数据挖掘交友

The first time I found the need to display data from a remote server on a web page was when a client I was building a web site for wanted to put a stock quote on their home page. Not wanting to disappoint them (they were a big client after all -- they had stock!), I set out to figure how exactly I was going to do it. When I finally got the script working (which was no easy task -- anyone else remember the Internet Transfer Control and WinInet?), I was disappointed at how slow it was. Between the slowness, the fact that the web server had a tendency to crash when it made requests, and the fact that I was still doing a lot of my work via dial-up, I made sure that those stock quotes were cached for a good long while!

数据挖掘论坛