Redirect caching issue with IE9 and FireFox

If you have questions about using mojoPortal, you can post them here.

You may want to first review our site administration documentation to see if your question is answered there.

This thread is closed to new posts. You must sign in to post in the forums.
3/6/2012 3:20:06 PM
Gravatar
Total Posts 21

Redirect caching issue with IE9 and FireFox

I ran into an unusual problem the other day.  I think it may require some support from mojoportal to fix it.

Here's the trouble...I had a menu entry on my mojoportal site whose URL was redirected to another site.  As it happens, I didn't do it through mojoPortal but rather through IIS6.  However, I might have easily done it with the redirect manager within mojoportal.

What happens is that when the redirect is removed, neither IE9 nor FireFox (5+) learn of the change.  They both keep going to the old redirected page. Huh?

What is happening is that both of these browsers introduced caching of redirects in order to enhance performance.  When either of them sees a 301 redirect, they cache it and never consult the original server again to see if it is no longer redirecting.  While technically that's in the spirit of a "permanent" redirect, it fails to take into account any margin for error.  That is, once the redirect is cached, there is no way for the original server to undo the change.

Let's say you tried to tell the browser to go back to the original url.  You could do this by setting a redirect on the target of the original redirect to point back to the original url.  Guess what, now all you get is a redirect loop!  You can't invalidate the original cached redirect this way.  If IE9 and FF were a little more intelligent, then they might take this as a sign that perhaps it should check on the status of the original redirect and see if the server still has it in force, but neither have such intelligence.

Attempting to use such a redirect to "fix" the problem also gets cached, therefore completely breaking both URLs permanently! (for that browser at least)

Ok, so there's no fix from the server side for one of these cached 301s.  Perhaps you're thinking, "You shouldn't have used a 301 in the first place if you expected it would change later."

Well, that's a very good point.  It's correct that, if there's any chance the redirect could be removed in the future, then it's a temporary redirect, isn't it?  It so happens that I did expect this redirect to be temporary, so that's my mistake.  I should have used a 302 in this case.  That way, when the redirect went away, they wouldn't have been cached by the browser.

There's a couple problems with that.

First, I made the redirect before IE9 came out, when 301s weren't cached.  So it would be reasonable but not technically accurate to use a 301.

Second, I would have made the redirect within mojoPortal had I known how, which would have resulted in a 301 since there's no 302 redirecting in mojo.  This is the path of least resistance for many, especially those who don't want to dive into IIS and the nuances of 301s and other types of redirects. (You mean there's more than one kind of redirect?)

Third, IIS6 doesn't even show the 301/302 terminology in its UI, so it's really easy to use the wrong one, especially since 301 is apparently the default.

So let's say you made the mistake I made.  Honest mistake.  Or, let's say you briefly renamed a page in mojoportal, then realized a bit later you wanted to rename it back the way it was (you still have to visit the redirect manager in mojoPortal to undo the first redirect).

So now, everyone who has visited your site since the mistake is now irretrievably unable to ever use that URL to reach you again!  You have no way to fix them in any sort of automated fashion.  Does that sound like reasonable behavior?  You essentially black out ever being able to use a URL again when you issue a 301.  That certainly doesn't seem like what most webmasters intend, it's not the way things used to be, and now it's just too bad.  I don't know about you, but all of a sudden I don't ever want to use a 301 again if I want to maintain control over my namespace.  There's a real difference between saying "hey google, I want you to start linking to this new url instead of the old one" (which is what 301s are good for) and saying "hey, I want to permanently and irretrievably invalidate this url from being used on my site again."  The latter is what it means now, and it didn't before.

All right, so I've slightly overdramatized the irretrievably lost url thing.  But only a little.  The point is, it's still lost to your control.  There is a way to get the browser to come back to your site.  Clear the cache, right?

Wrong.  IE9 clears the cache completely...of everything except the cached redirects.  What?  There's no way to clear the cache using the normal cache clearing controls.  You can even reset IE to total defaults...still no luck.

There are two solutions...either go into IE9 private mode (Ctrl-Shift-P) and visit the url, or use the great http debugger Fiddler to clear the Wininet cache (under Tools).  Now you can get back to the url in regular IE.

Not only is this a ridiculous oversight by the IE9 developers, even if it were a real solution, you'll be lucky if your users bother to ever call you to say they can't visit your important URL.  Telling them to clear the cache is a bad enough "solution".  Walking them through one of these other workarounds is a nightmare if you expect to do actual business over your site.

So, I propose some alternatives for solutions:

a) Stop using 301s on mojoportal and use 302s instead

This isn't a great solution since presumably 301s help your search indexing by telling Google that yes, this url really moved somewhere else.

b) Issue "Cache-Control: no-cache" headers on all 301s

This is probably more reasonable.  IE9 and FF will respect the no-cache directive and check back with the server every time the user visits the URL.  This is, at worst, equivalent to the old behavior.  The IE9 developers may see this as subverting a good user experience, but I see it as maintaining control over my namespace.  I've already lost one very valuable url ("downloads.aspx") to their "performance enhancement").

c) Issue "Cache-Control" headers with a cache timeout on all 301s

This is the middle-of-the-road solution.  The redirects will be cached, but will also expire.  The two problems I see with this are: first you have to educate the mojoportal admin that this setting even exists and what it does so they can have an opinion of what's a reasonable setting.  Second, you're not going to want to set this to anything more than a couple days, and even then, what are you going to tell your users for whom it breaks?  Wait a couple days, or go into private mode?  They're going to take the easiest alternative: don't bother trying to buy your product.

Here are the articles I found on this topic:

Ted

3/6/2012 7:21:22 PM
Gravatar
Total Posts 1203
Proud member of the mojoPortal team

Help support mojoPortal!
Add-on modules

Re: Redirect caching issue with IE9 and FireFox

Wow, that's an interesting post, and good food for thought. I can understand why browser makers would do this, since the main way they compete with each other is on overall speed. I'll definitely be more careful in the future about creating 301 redirects.

I think that perhaps an alternate version of "c" might work best for user experience and URL recoverability, but it would require an additional change in 301 redirect manager (maybe a checkbox) to turn a 301 redirect into a 302.

Scenario:

You create a 301 redirect of /downloads.aspx to /downloads_new.aspx, with a cache expire time of one week.

Some day in the future, you want to change everyone back to /downloads.aspx. In order to do that you'd:

  1. Change the 301 redirect to a 302.
  2. Wait a week.
  3. Remove the redirect.
3/7/2012 7:38:40 AM
Gravatar
Total Posts 18439

Re: Redirect caching issue with IE9 and FireFox

Hi Guys,

I think this is a rather obscure issue but it does highlight the fact that 301 redirects should be used with care, the need for them should be avoided because urls should not change in the first place but when urls do change it should be understood that 301 redirects are meant to be permanent therefore they should be used with care because they also should not change. 

I'm not interested in any complicated solution with checkboxes for 301 vs 302, but I'm willing to go along with web.config settings that would allow 

b) Issue "Cache-Control: no-cache" headers on all 301s

and/or

c) Issue "Cache-Control" headers with a cache timeout on all 301s

If either of you want to submit a patch the relavent code is in Web/Components/UrlRewriter.cs at the bottom in the Do301Redirect method.

I think this would also require updating the documentation so that people can know about these options before they setup 301 redirects, but the default behavior should probably remain as it is today.

Best,

Joe

3/7/2012 8:47:20 AM
Gravatar
Total Posts 73

Re: Redirect caching issue with IE9 and FireFox

I'm all in favor of the web.config setting for (b).  I don't want to lose any more URLs, intentionally or by accident.

The new semantics are like the post office saying yes, you can move to a new house and we'll forward your mail, but if you ever happen to want to move back to your old home, we're never going to deliver to you there again.

3/7/2012 9:31:53 AM
Gravatar
Total Posts 2253

Re: Redirect caching issue with IE9 and FireFox

Hi Ted,

I was on the side of "don't create a 301 if it's not going to be permanent" until your analogy with Post Office. That changed my mind, seriously.

I will say that I've created incorrect 301s before but I've never had those cached for so long that seemed like forever. Usually, they were working the way I wanted within a week (which isn't always acceptable). I was using at least FF 4, IE9, and Chrome 1546545 (or whatever version they're on now). I don't think this caching happens exactly the way the vendors are suggesting it does, but either way it is a nuisance if you make a mistake.

I do think the setting should be for issuing a cache timeout on 301s. No caching at all seems a bit extreme.

Just my 2¢

-Joe D.

3/7/2012 9:41:27 AM
Gravatar
Total Posts 18439

Re: Redirect caching issue with IE9 and FireFox

I understand wanting to see it like a post office but I don't really think it is a good analogy, people are expected to move urls are not and that analogy is just making seem like its ok for urls to move.

In reality urls do sometimes move and that is why 301s and 302s exist but I don't think we should let this post office analogy soften our stance that changing urls should be avoided as much as possible and changing 301 redirects also should be avoided as much as possible.

Neverltheless with config settings people make it do whatever they like.

Best,

Joe

3/7/2012 11:20:53 AM
Gravatar
Total Posts 1203
Proud member of the mojoPortal team

Help support mojoPortal!
Add-on modules

Re: Redirect caching issue with IE9 and FireFox

Hi Joe, I don't have time to test this (hopefully Ted will), but according to my research this should work for setting the cache time in hours. As a bonus, if you set the cache time to 0, it will be effectively the same as no-cache.

int cacheHours = ConfigHelper.GetIntProperty("301RedirectCacheHours", -1);
if (cacheHours >= 0)
{
   TimeSpan ts = new TimeSpan(cacheHours, 0, 0);
   app.Context.Response.Cache.SetMaxAge(ts);
}

I'll update the 301 Redirects document with the key if you decide to implement this.

Jamie

 

3/7/2012 11:23:51 AM
Gravatar
Total Posts 73

Re: Redirect caching issue with IE9 and FireFox

Thanks Jamie.  I'll see if I can give this a try tonight.  The first discovery was with someone else's browser, so it may take me a bit longer than that to get a test environment set up.  I don't use IE9 personally because I can't stand the new font antialiasing.

3/7/2012 11:37:34 AM
Gravatar
Total Posts 1203
Proud member of the mojoPortal team

Help support mojoPortal!
Add-on modules

Re: Redirect caching issue with IE9 and FireFox

Since the SetMaxAge method throws an exception on timeSpans over one year, for safety the if statement should be:

if (cacheHours >= 0 && cacheHours <= 8700)

3/7/2012 11:56:02 AM
Gravatar
Total Posts 18439

Re: Redirect caching issue with IE9 and FireFox

Hi Guys,

I've implemented these options in the source code repository, so the method now looks like this:

private static void Do301Redirect(HttpApplication app, string newUrl)
{
//add web.config options to allow setting a cache timeout?
//https://www.mojoportal.com/Forums/Thread.aspx?thread=9947&mid=34&pageid=5&ItemID=5&pagenumber=1#post41411

string siteRoot = SiteUtils.GetNavigationSiteRoot();

app.Context.Response.Status = "301 Moved Permanently";
if (WebConfigSettings.PassQueryStringFor301Redirects)
{
app.Context.Response.AddHeader("Location", siteRoot + "/" + newUrl + app.Request.Url.Query);
}
else
{
app.Context.Response.AddHeader("Location", siteRoot + "/" + newUrl);
}

if (WebConfigSettings.DisableCacheFor301Redirects)
{
app.Context.Response.Cache.SetNoStore();
app.Context.Response.Cache.SetCacheability(HttpCacheability.NoCache);
app.Context.Response.Cache.SetRevalidation(HttpCacheRevalidation.AllCaches);

}
else if (WebConfigSettings.SetExplicitCacheFor301Redirects)
{
int daysToCache = WebConfigSettings.CacheDurationInDaysFor301Redirects;
if (daysToCache > 365) { daysToCache = 365; }
TimeSpan cachDuration = TimeSpan.FromDays(daysToCache);
app.Context.Response.Cache.SetCacheability(HttpCacheability.Public);
app.Context.Response.Cache.SetExpires(DateTime.Now.Add(cachDuration));
app.Context.Response.Cache.SetMaxAge(cachDuration);

}

}

I don't think we need ot be able to use hours for something that is supposed to be permanent so I used days (default 30), and by default caching will be neither disabled nor explicitely set, ie the behavior is not changed by default.

I have not tested enabling these so if anyone cares to test it that would be good.

I don't plan to use these settings myself and my general advice would be to not use them, avoid changing urls and creating 301 redirects but when you do create them plan carefully and treat them as permanent as they were intended to be.

I've seen arguments on both sides of the issue in googling for "set cache on 301 redirect" and I think it is an issue where smart people can reasonably disagree which is why I added these options but myself after reading the arguments for and against I'm still on the side of 301 redirects are supposed to be permanent and should be cached for a long time as browsers are now doing.

Best,

Joe

You must sign in to post in the forums. This thread is closed to new posts.