Darn IIS and Service Packs!

I thought that I was getting pretty good at deciphering odd SharePoint errors. After all, most of the time they are not very friendly! After the subsite/site collection issue of yesterday, you would think I would be in for an easier day today!

But it was not meant to be…

I was out at a client site today, basically going through a SharePoint installation with the very nice infrastructure people. They had a single Win2003 Standard server all ready and waiting. I sat back while they took the driver’s seat. They installed while I passed on lots of insightful, brilliant and invaluable advice as all high class consultants like me do :-P. All was going swimmingly, IIS and SQL was installed easily, security was locked down, service accounts created and the SharePoint installation went through without a hitch.

(By the way, if you don’t want to read my tale of woe then skip to the end, but writing this is cathartic, so please indulge me 🙂 )

So, as I was saying, the install of SQL, IIS and SharePoint went fine. Ran the 9 step SharePoint configuration wizard and that was all good!

Then…

Then I started up "SharePoint Central Administration" on the SharePoint server and received an authentication prompt.

"Aha!", I confidently pronounced to said client, "I have not uninstalled the ‘Internet explorer enhanced security configuration’ option, nor have I added central admin to the trusted sites zone in MSIE".

So I did those things and  was *still* prompted for authentication.

"Aha", I exclaimed, "maybe it’s a DCOM issue. I blogged about this". I then confidently explained that sometimes when attempting a secure by default install, you do have to loosen up DCOM permissions here and there, especially when the server performs multiple roles. So I checked the event logs, and sure enough saw a few DCOM complaints. To satisfied nods all round, I corrected them with DCOMCNFG.

I was *still* prompted for authentication.

"Hmm, okay", I said, thinking about some of the other zillion stupid authentication issues you can have with IE/Zones/IIS/Impersonation and the likes. "It might be a Kerberos thing…" Even though I can’t see how, and the fact that I chose NTLM when I ran the configuration wizard, I thought that IIS might be trying to authenticate with Kerberos before NTLM as per KB215383. After all, the article says:

"By default, the NTAuthenticationProviders metabase property is not defined when you install IIS 6.0. IIS 6.0 uses the Negotiate, NTLM parameter when the NTAuthenticationProviders metabase property is not defined. Therefore, you do not have to configure IIS to use the Negotiate,NTLM property value unless the default value has been overwritten".

So, using adsutil.vbs I forced it to be NTLM only and retried…

I was *still* prompted for authentication.

At this point I said to the client, "Let’s go have lunch!". All agreed and I continued to ponder over a large juicy burger and a coke.

So, upon my return, the clients left me for half an hour to see if I could nail it. I noticed that when I entered valid credentials at the authentication prompt, even the domain admin account, it popped up again for credentials.

If I clicked cancel, after 3 cancel attempts, it would let me into Central Administration. If I entered credentials and clicked ok, it would also let me in eventually, but it would take more attempts than three.

"Aha, I have seen this before", I said to the empty room. "This sort of thing happened to me when the SharePoint configuration account didn’t have appropriate permissions to the ‘Temporary ASP.NET Files’ folder under the ‘C:\WINDOWS\Microsoft.NET\Framework\<version>\’ folder."

So, I checked the permissions here and it looked good. The config account was a member of all local IIS/WSS worker process groups that it needed to be. But to be sure, I enabled failure auditing on the C:\WINDOWS\Microsoft.NET\Framework and set the local security policy to log pretty much any audit event that was a failure.

Tried again, reproduced the problem and the security event logs showed no files access/create failure.

Decided to see if there was anywhere else on the filesystem causing it, I extended auditing to the whole filesystem.

Still no-go, this was not a filesystem permission issue. It was also not Kerberos/NTLM I reasoned, because if it were, why was I able to eventually get into central administration?

So, I decided to look at the IIS logs for the central admin web application. The first part is pasted below. (I’ve snipped some of the superfluous log into for readability)

#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2007-11-15 06:07:07
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
2007-11-15 06:07:07 W3SVC1117623206 192.168.6.11 GET / 401 2 2148074254
2007-11-15 06:07:07 W3SVC1117623206 192.168.6.11 GET / 401 1 0
2007-11-15 06:07:13 W3SVC1117623206 192.168.6.11 GET / 200 0 0

Now, the sequence above (the 401 2 error followed by a 401 1 and 200) is the normal integrated authentication sequence. As described in this KB article 264921, when the browser makes a request, it always considers the first request to be Anonymous. Therefore, it does not send any credentials. If the server does not accept Anonymous, the IIS server responds with an "Access Denied" error message (401 messages) and sends a list of the authentication types that are supported. If the browser supports the authentication type, it will use it and successfully retrieve the page (200 message).

But now, let’s look at the next sequence…

2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /default.aspx SOMESITE\Administrator 200 0 0
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /_themes/CENTRLAD/CENT1011-65001.css 401 2 2148074254
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /_themes/CENTRLAD/CENT1011-65001.css 401 1 0
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /_themes/CENTRLAD/CENT1011-65001.css 200 0 0

Now, what we see above is that /default.aspx was retrieved successfully. Then the browser tried to retrieve a CSS file from /_themes and we are back into the integrated auth sequence. This is probably because _themes is another web application (I’ve never checked). The point is, things are still going to plan.

Then, a little further down, we see this…

2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 2 2148074254
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 0
2007-11-15 06:07:25 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 2148074248
2007-11-15 06:07:34 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 0
2007-11-15 06:07:34 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 2148074248
2007-11-15 06:07:34 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 0
2007-11-15 06:07:34 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 2148074248

[snip]

2007-11-15 06:07:36 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 0
2007-11-15 06:07:36 W3SVC1117623206 192.168.6.11 GET /WebResource.axd 401 1 2148074248

So, what’s going on here? Instead of the 402 2, 401 1 and 200 sequence as it is supposed to be, we now have 401 2, 401 1 and then another 401 1 with a win32 code of 2148074248. And, what is this WebResource.axd anyway? That seems to be the source of this issue.

So, what did uncle google have to say about WebResource.axd and 2148074248?

Lot’s actually, but not quite what I was looking for. But, then I had a lucky break and found a newsgroup posting from Jan 07 to the microsoft.public.sharepoint.portalserver.development group.

David Schneider wrote this in answer to the original request:

"I have exactly the same problem on a customer site. The Central Admin of SharePoint use the httphandler webresource.axd to publish some javascript code. But when the browser try to load this javascript code via the webresource.axd, the current user isn’t unable to authenticate.

I did a lot of debugging (IIS log, file security, policies, fiddler, etc.) but I didn’t find a solution yet. Finally I opened a case at Microsoft Partner Support. And now I’m waiting for a response of these guys. "

Fortunately for me, he followed up…

"I had to re apply Windows 2003 SP1 and then the problem was gone. I had reinstalled IIS and the reinstallation has overwritten the SP1 DLLs with the ‘original’ DLLs"

Now, I checked with the client. The server was Windows 2003 SP2 and they used the Windows 2003 SP1 media to install IIS prior to the SQL Server install. My hopes jumped in nerdy anticipation! Could it be the same issue?

As it turned out it *was*, and I kicked myself for not including this requirement in my "insightful, brilliant and invaluable advice" to my client. After re-applying service pack 2, the problem was solved, the client was happy.

So, another day ends and another weird error sorted! One day I will actually not encounter this sort of issue and instead have a day where I can blog on governance topic (my actual area of interest!!).

Wouldn’t that be nice!