I was recently tasked to determine why the ASP.NET State Service would not start on a Windows 2003 Terminal Server. All I had to go by was the error message, “Error 5: Access is denied.”
Not a lot to go on…
In addition to the above error message a cryptic Event 532 was being logged in the security log of event viewer.
Asphinctersayswhat?
According to Microsoft the ASP.NET State Service provides support for out-of-process session states for ASP. ASP has a concept of session state. If this service is stopped or disabled, out of process requests will not be processed and subsequently the developers using this Terminal Server for their development work are out of business.
Ok, now what? As Donald Rumsfeld would say, “We also know there are known unknowns; that is to say we know there are some things we do not know….”
Researching either “Error 5: Access is denied” or “Event ID 532” yielded no useful results and in some cases pointed you in completely the wrong direction.
I recently watched Mark Russinovich’s on-line video titled, “Case of the Unexplained 2010,” which is an excellent tutorial on how to use the Sysinternals utility Process Monitor.
Note: Video of this webcast is listed at the end of this article.
So, what better time to put this knowledge to use and find out what is going on underneath the hood by firing-up Process Monitor.
Note: A link to the download for Sysinternals is at the end of this article.
After opening Process Monitor the first thing I did was reduce the noise by including only services.exe. After scrolling through the many results I finally hit paydirt when I saw “ACCESS DENIED” in the results column.
You can run but you can't hide from Process Monitor…
Ok, now we’re getting somewhere…
You can see in the above screenshot that the QueryOpen operation on aspnet_state.exe is successful but as soon as the operating system attempts the CreateFile operation it fails with the access denied error message.
I then opened Windows Explorer and saw that someone did something that they should not have done. A user modified the NTFS file permissions on the aspnet_state.exe file from its default permissions. You can see from the below screenshot that the user not only modified NTFS file permissions but he prevented inheritable permissions from the parent folder. Not good…
User = FAIL
This was quickly remedied by enabling inheritable permissions from the parent folder.
I then opened Services.msc and I was able to successfully start the ASP.NET State Service.
A mechanic is only as good as the tools he has at his disposal. The Sysinternals Suite is one of those must-have tools in any IT admins toolbox.
Incidentally, I e-mailed Mark Russinovich and he will be including this in his future Case of the Unexplained presentations and in his new Sysinternals book that he is co-authoring.
I recently ran into a situation where I was using the SysInternals tool ProcDump to write a dump file to be examined for a memory leak.
The problem started when trying to run ProcDump against the process oracle.exe. The error message was “Access denied.”
I was an administrator on the server so how could I become more powerful than an administrator?
The answer comes in the form of opening a command prompt as NT AUTHORITY\SYSTEM, which will then grant us the authority to access the oracle.exe process to create a dump file.
The first step is to download the Sysinternals tool PsExec from the below URL:
Extract PsTools.zip to a folder on your hard disk.
Launch a command prompt as administrator (right-click the command prompt shortcut):
In the command prompt navigate to the folder containing the PsTools.zip extracted data.
We will now launch PsExec.exe with the -i and -s switches to launch the program interactively using Local System.
psexec.exe -i -s %SystemRoot%\system32\cmd.exe
Type whoami at the newly opened command prompt and you will see that you are now running as NT AUTHORITY\SYSTEM:
You can now execute ProcDump against the process that you were previously denied access to and complete your work.
Note: If your system does not have whoami.exe, you can typically find this program as a separate download via the resource kit or support tools appropriate to your Microsoft operating system.
In early December 2010, I decided to give Vonage a try. Why would I want to switch from a service that I had been using since 1994 and had essentially treated me with no problems? The simple answer is the outrageous amount that AT&T charges for their service. Below is a screenshot of my local services plan from AT&T. The monthly service costs $54.03, which includes local and unlimited long distance. The surcharges and other fees totaled $10.95\month! This is more than the entire monthly fee for Vonage’s Lite service! Quite frankly I was tired of paying a hefty service, which I barely used.
AT&T Monthly Service
So, on December 6, 2010 I ordered Vonage’s Lite service. Their website was straightforward and easy to use and the ordering process was painless.
In the meantime, I had brushed-up on some Vonage forums looking at other issues people were having but dismissed those problems as I was confident that those problems would not affect me.
A few days later the Vonage hardware arrived and I proceeded to setup the service. The instructions wanted me to replace my DSL modem\router with their Vonage router (model VDV22). This was not possible since my AT&T 2Wire DSL router uses an RJ-11 jack that goes from the wall to the router. The Vonage router uses RJ-45 jacks only so it was not compatible with my setup. No problem. I’ll just setup my Vonage router behind my AT&T router, which many others have done with success.
I then connected the Vonage blue port (WAN) and the Vonage yellow port (LAN) to my switch. Since my computer has two LAN ports I’ll just connect both of those ports to the switch and configure Vonage via this method. After I got everything connected and I logged into the Vonage web page (192.168.15.1) and I could see that the status page kept saying, “Could not connect to configuration server.” I’ll spare you the gory details but after a call into Vonage tech support they determined that the unit was faulty. So, I spent the $11 to ship the unit back to Vonage and wait a few more days for a replacement unit to arrive.
I connect the replacement unit to my switched environment using the same configuration I used for the first Vonage device. I logged into the Vonage web page and the damn thing is giving me the same error (Could not connect to configuration server). At this point, I was pretty much ruling out another hardware error. The odds were stacked against that probability. It had to do something with the way I was connecting my Vonage device to my network. Instead of connecting both of the Vonage device’s ports to my network I decided to connect just the WAN (blue) to my switch. Sure enough the device worked. Nowhere in the documentation did it say that you could not have both the WAN and LAN ports connected to the same VLAN, which was causing some sort of conflict with the Vonage device looping-back to itself. One strike against Vonage.
Now that I have the device up and running I can begin my testing. At first the device seemed to work properly with an occasional audio dropout, which I didn’t think much of at the time. It wasn’t until I made a dentist appointment where the call audio was completely cutting out every 30 seconds or so. The audio dropout was consistent and pervasive. I opened a command prompt and initiated a persistent ping to the Vonage device to see what kind of latency times it was experiencing (see screenshot below).
Vonage audio dropouts
These are some of the highest latency times that I’ve ever seen to a single device. Even my wireless aircard from Sprint has latencies better than this and this Vonage device was sitting on my gigabit switched network! I would confirm that the observed latency coincided with the audio dropouts by making test calls and seeing the high latencies associated with complete audio cutouts. Since there was no network traffic on my LAN that could be attributed to this latency it felt like a bug in the Vonage device. So, I power cycled the device and the problem went away. Strike two against Vonage.
I was hoping that this was an isolated case and that Vonage would somehow start working better. As I would see this was not to be the case.
To continue my testing I forwarded my AT&T home phone number to my new Vonage number so that I could start real world testing. Every single call that I received or placed would experience audio dropouts during the call. This was amazing to me since my DSL service is 6 Mbps downstream and 768 Kbps upstream plenty enough to handle VoIP service. In addition, I had just changed my AT&T DSL profile from Interleaved to FastPath, which reduced my latency to my DSLAM from 47 ms to 9 ms making a huge difference in Internet response times. My Internet circuit is also barely utilized so the notion that my circuit was somehow saturated is not a valid argument.
I became so frustrated that I downloaded and installed a trial version of Solarwinds’ ipMonitor. I was familiar with ipMonitor as I used it for years in my capacity as a data center engineer in a hosted environment. I setup a monitoring session in ipMonitor so that it would send a ping to the Vonage device every second and e-mail me a daily report of the results. What I found was astonishing. For a device that is sitting idle most of the time I was seeing ping response times approaching 100 ms or more. Below is a screenshot of ipMonitor monitoring the device. The second row shows the tale of the tape. The large blue mountains show the latency to the device for which I had no explanation for. The only thing I could think of is that the device was so busy doing something. The problem though is that it wasn’t busy doing anything. No phone calls were made or received during this monitoring time. Strike three against Vonage.
VDV22 response time
It was time to cancel the service. I had spent enough time and energy troubleshooting this issue that I decided to be done with it. The Vonage account management rep on the phone tried to send out a technician to help me get it installed but I don’t see how a technician could resolve high ping times to the device, which had absolutely nothing to do with the way it was implemented. So I declined and the service was canceled on December 23, 2010.
The only other VoIP device out there that gets my attention is Ooma. The downside to this device is that you have to purchase it for a one-time cost of $199 and there is no trial period so once you buy it you’re stuck with it.
I may revisit Vonage in another year to see if they’ve gotten any better. For now, I’ll stick with the high-priced oversubscribed service that I barely use knowing that I’ll have good call quality.
The old adage of you get what you pay for rings true in this case…
A friend and colleague recently went through trials, tribulations, weeping and gnashing of teeth in an attempt to virtualize SBS 2003 to more easily provide an upgrade path to SBS 2008.
His first and subsequent attempts at using Disk2vd were unsuccessful resulting in a black screen at boot. I was surprised at this because I had great success in using Disk2vhd to virtualize every server that I had come across. Disk2vhd is a free utility from Windows Sysinternals.
His attempt then turned to using the P2V utility that comes with System Center Virtual Machine Manager 2008 (SCVMM). This attempt also failed. VMM requires that the server you want to virtualize be in the same domain that the VMM server is in. In addition, VMM requires you to download the NIC drivers for the server that you want to virtualize, which were unavailable from the server manufacturers website.
At this point his entire weekend was consumed and he had to abandon these approaches and look for another method.
The method that he finally came up with is brilliant and worked flawlessly. Here is the detailed procedure to successfully virtualize SBS 2003:
Download and install Acronis Backup & Recovery 10. This should be installed on the physical server that is to be virtualized.
Run a full backup of the physical server to your Hyper-V server using the default .tib backup file type.
Install Acronis on the Hyper-V server.
Run Acronis on the Hyper-V server and do a restore job to a VMWare machine using the above .tib file.
On the Hyper-V server download and install the free StarWind V2V Image Converter.
Convert the above VMWare image to a .vhd file.
Load the newly created .vhd file in Hyper-V and you are good to go.
On a 300+ GB system the above processes took approximately 20 hours to complete.
In the now virtualized copy of SBS 2003 you’ll then want to open the services applet and confirm that all of your services are running.
If you are having problems virtualizing a system using any of the myriad of virtualization tools out there this is a free and somewhat efficient method of getting it done.
In ISA Server 2004\2006 or Forefront Threat Management Gateway 2010 the “Proxy requests to published site” setting is one of the most confusing yet simple configuration options when publishing a server.
When you publish a website using the website publishing rule feature the default setting is to select the “Requests appear to come from the Forefront TMG computer” radio button available in the To tab of the website publishing rule’s properties dialog box. This is what I would call the “safe” option but as we’ll find out not necessarily the correct option for your environment (see image below):
Note: The default setting for publishing non-web server publishing rules is set to “Requests appear to come from the original client.” More on this specific setting at the end of this article.
TMG Listener
With the setting “Requests appear to come from the Forefront TMG computer,” the logic is as follows:
The request comes into TMG and TMG sends it back to the web server. When TMG does this it changes the “Source IP” address field in the IP Header to its Internal adapter IP address. When the web server sees this request (assume it’s on the same logical segment as TMG), it sees the source IP is TMG. The web server then checks its routing table and says “This address is directly accessible to me. All I have to do is ARP for this IP and if I get a ARP response, I’ll deliver it directly to that host.” The response is complete as far as the web server is concerned.
Now, with the setting “Requests appear to come from the original client,” the logic goes like this:
The request comes into TMG and TMG sends it back to the web server. When TMG does this it leaves the “Source IP” address field in the IP Header alone (i.e. the external client’s IP is maintained.) When the web server sees this request (assume it’s on the same logical segment as TMG) it sees the Source IP as someone on the Internet. The web server then checks it’s routing table and says “This address isn’t directly accessible to me. My routing table says that if I don’t have a more specific route I’ll just punt and send it to my default gateway. Let me ARP for the gateway and if I get a response, I’ll deliver it to that system”. The request is now done as far as the web server is concerned.
You can see this behavior by opening a command prompt and typing “netstat -ano” without the quotes. This command displays your current network connections. You will be able to see the internal TMG source IP address or the client source IP address depending on the setting you have in TMG.
With the setting for “Requests appear to come from original client” it is essential that the return path for the responses from the web server honor the incoming path of the request. In other words, if the request came in from TMG and this setting is enabled, then the response has to go out through TMG as well. In other words, if you select the option “Requests appear to come from the original client” you must have the web server’s default gateway set to the TMG’s internal IP address! This begs the question as to why can’t TMG’s response go out some other device for the response such as another internal router? There are two reasons for this:
1) If the other device is capable of stateful inspection, it will not have any state for the response from the web server since the original packet came in through TMG.
2) Even if that device doesn’t perform stateful inspection it most likley does NAT. As the response goes through that other device, the Source IP will get changed to the NAT device’s IP and by the time the original client gets the response, the Source IP is not who the client sent it’s request to and it will drop the response.
They say a picture is worth a thousand words and in this case a graphical representation of what is going on helps. Below are four graphs depicting the four possible scenarios.
Scenario 1: Requests appear to come from the TMG computer. Default Gateway is TMG.
The below graph shows a client request traversing through TMG with the default setting of “Requests appear to come from the Forefront TMG computer.” The default gateway of the web server is set to TMG. This configuration will work but the downside to this configuration is that the web log files will only contain the IP address of the TMG’s internal interface obfuscating the true source of the request. This is a surefire way to infuriate the marketing folks, which for some of you may actually be an enjoyable thing to do. This scenario is a safe option but totally unnecessary in my opinion. If the Web server’s default gateway is set to the TMG server then there is no reason to have the requests appear to come from the Forefront TMG computer unless you want to purposely obfuscate the IP addresses in the web log files.
Requests appear from TMG, DG=TMG
Scenario 2: Requests appear to come from the TMG computer. Default Gateway is a router.
The below graph shows a client request traversing through TMG with the default setting of “Requests appear to come from the Forefront TMG computer.” The default gateway of the web server is set to another router. This configuration will work but you have the same downside to this configuration as scenario 1 in that the web log files will only contain the IP address of the TMG’s internal interface. You will typically encounter this scenario in larger enterprises where they likely have a more complex routing infrastructure and they want the web server’s default gateway set to a router for policy reasons.
Requests appear from TMG, DG=Router
Scenario 3: Requests appear to come from the original client. Default Gateway is TMG.
The below graph shows a client request traversing through TMG with the setting of “Requests appear to come from the original client.” The default gateway of the web server is set to TMG. This is the most common scenario and the one that I use almost exclusively. This configuration will work and the web log files will contain the client’s Source IP address. I can hear the marketing folks jumping for joy.
Requests appear from client, DG=TMG
Scenario 4: Requests appear to come from the original client. Default Gateway is a router.
The below graph shows a client request traversing through TMG with the setting of “Requests appear to come from the original client.” The default gateway of the web server is set to a router. I’ve saved the worst scenario for last because this is the scenario where people get into trouble. This scenario will not work because the client’s source IP address is maintained all the way to the TMG server but the TMG server forwards its response out through its default gateway breaking the chain of communication.
Requests appear from client, DG=Router
The question then becomes how can you maintain the client’s Source IP address in the web logs in this scenario? The answer is to use the X-Forwarded-For HTTP header field. This is a de facto standard for identifying the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer. Checkout the below article that describes how to use Winfrasoft’s X-Forwarded-For client for ISA and TMG server:
At the beginning of this article I briefly mentioned that the default setting for publishing non-web server protocol publishing rules is for the request to appear from the original client. So, the question then becomes why is this default setting different from publishing standard website publishing rule? I came across a possible scenario which may explain the reason. When publishing the latest version of a Microsoft CRM website you must use HTTPS. Microsoft CRM uses forms-based (or claims-based) authentication so that you log in using a secure form to gain access to your CRM website. What if you want to securely publish CRM for multiple companies using a wildcard SSL certificate? You would purchase a wildcard SSL certificate (i.e. *.example.com) and then you would create a non-web server protocol publishing rule in TMG using the HTTPS protocol. This wildcard SSL certificate then allows you to host CRM for multiple companies such as company1.example.com, company2.example.com and so on using a single HTTPS web listener. Microsoft CRM has intelligence built into the form whereby it inspects the URL that you are sending it and then depending on the URL it will forward you to the proper CRM database on the back end. This behavior only appears to function properly when you have the HTTPS rule set to have requests appear to come from the original client. If you have the other setting applied where the TMG server replaces your source IP with its IP address you will no longer get a form to sign on with but instead you will be presented with a pop-up window to logon to the domain.
Are you confused about the functional differences between TMG (Threat Management Gateway) and UAG (Unified Access Gateway)? You’re not alone.
Thomas Schinder (MSFT) sums it up nicely by mentioning that UAG is designed to be an Enterprise solution for remote access into internal corporate resources without supplying any outbound access.
TMG is designed for both inbound and outbound access but at a more comfortable price point for SMB’s.
Here is a good TechNet thread where numerous Microsoft’ees chime in with their opinion:
Mike Howells's Blog 