A spectator’s Ready to eat meal.

In this scenario, TMG admin had published a web application through the TMG server, There was client side application as well on the clients, when users access the application using this client piece, they were getting error

 

“error- 20152–500 Internal server Error (Data is invalid.)” in a window apart from the name of the application in the error message.

 

TMG admins had noticed in the TMG live logs they were getting “status 13 – data invalid”

 

Data analysis and Troubleshooting.

We collected TMG data packager from the TMG server ,while trying to connect from the client. In the Data, I tracked the client’s request in the TMG live logs, where we were getting result code as 13, then using the corresponding Request id in the live logs ( i have explained this technique in this blog post of mine, http://blogs.technet.com/b/sooraj-sec/archive/2012/11/07/data-analysis-using-with-tmg-data-packager.aspx).

Then In the TMG trace found following

***********************************************************************************************************************************

Info:0000000038EEA540: the client ask for compression different than gzip. The filter remove it

Info:Filter called GetServerVariable: ‘IS_CHUNKED_REQUEST’, Context:0000000038EEA540

Info:[GetHeader]:’Transfer-Encoding:’ – the requested header wasn’t found.

Info:[GetServerVariable]:’IS_CHUNKED_REQUEST’ – Returned value:’0′, Context:0000000038EEA540

Entering CalcRequestContentLength

Info:Filter called GetServerVariable: ‘REQUEST_CONTENT_LENGTH’, Context:0000000038EEA540

Info:[GetHeader]: ‘Content-Length:’, Returned Value: ‘xxxx,xxxx’.

Info:[GetServerVariable]:’REQUEST_CONTENT_LENGTH’ – Returned value:’xxxx,xxxx’, Context:0000000038EEA540

ERROR:pContentLength->FromDecimalString(xxxx,xxxx) failed, hr=0x8007000d(ERROR_INVALID_DATA).

ERROR:CalcRequestContentLength() failed dwError = 13(ERROR_INVALID_DATA)

ERROR:OnClientPreProcHeaders() failed dwError = 13(ERROR_INVALID_DATA)

Info:Rejecting the request

Info:WPPISAPUBLIC:Returning error text "The data is invalid. " for error code 13(ERROR_INVALID_DATA)

Noise:WPPISAPUBLIC:(x.x.x.x:xxxx <== x.x.x.x:xxxx), xxxx bytes, "HTTP/1.1 500"

——————————————————–

Info: WPPISAPUBLIC:Context property:Cache info = 0x0

Info: WPPISAPUBLIC:Context property:Error info = 0x200

Info: WPPISAPUBLIC:Context property:Filter info = Req ID: xxxxxx

Info: WPPISAPUBLIC:Context property:Result code = 13

Info: WPPISAPUBLIC:Context property:Response source = 0x00000000(fpcSrcUnknown)

*******************************************************************************************************

It seemed that the client application was asking for compression method other then gzip and then TMG was not able to understand and was failing to calculate the content-length, Once I explained that to the TMG admin, they removed the content-length header from the client application side and then tested and after that issue did not happen.

Ideally they should have configured the client application with gzip compression but they went with removal of content-length header. But i guess they were happy with the outcome.

My Goa Visit in year 2010- The road trip-change of destination. Part 1.

This is one of the most interesting cases ,I worked on recently , So I thought of sharing that with all to provide different angles to a problem we usually work and troubleshoot for resolution. This case was opened with a description as explained in the title i.e. DA clients were not able to connect to the internal network through UAG DA server and we had a mysterious warning about "A client certificate was not provided".

Interesting part is, that lot of other third party consultants had already worked with customer,  Lot of troubleshooting had happened before it came to me. They had changed CA(Certificate Authority) and its Chain, as well client and UAG certificates multiple times, still DA clients would not connect to the internal network through UAG DA server. Another important fact , customer had native IPv6 on his internal network.

 

Troubleshooting Approach 

I started with basics ,Since we were getting warning about certificate,  checked the certificates on the client and the UAG server and found them correct , then checked the Main mode Security Associations using command netsh advfirewall monitor show mmsa  I found that we did not have Security Association for the corpnet tunnel. I found SA with ComputeCcert and UserNTLM but I could not find SA with Computerkerb and UserKerb.

Example of  output of the command from my own client machine

netsh advfirewall monitor show mmsa

Note : I have removed some personal details from following example

Main Mode SA at 08/11/2012 04:05:17
———————————————————————-

Auth1:                                ComputerCert
Auth2:                                UserCert
Cookie Pair:                          xxxxxxxxxxxxxxx

Main Mode SA at 08/11/2012 04:05:17
———————————————————————-
Auth1:                                ComputerKerb
Auth2:                                UserKerb
Cookie Pair:                          xxxxxxxxxxxxxxx

My first suspicion was on the Kerberos tickets, as they are critical for this SA (Security Association) to happen. Question was, is the client machine able to get it from the internal domain controllers or not? So i used my favourite tool, Network monitor and captured network traffic, on the internal NIC of the UAG DA server and Domain controller, while restarting IP helper service on the DA Client machine, as client would try to initiate the Main Mode SAs and would also try to get Kerberos tickets from domain controller through UAG DA server.

Then in the network monitor traces i found that the Client’s requests was forwarded by the UAG DA server, to the domain controller i.e. Syn frame, to initiate the TCP handshake,  but we were not getting any Syn Ack, Back from the domain controller i.e. Domain controller was not answering to UAG DA server back.

When I looked at the Network traces, on the domain controller , I found the root cause of the problem. In the traces on the Domain controller , I found that when domain controller gets the Syn frame from the UAG DA server. It sends the Syn Ack Back,  not to UAG DA server but to another device, When I looked at the Mac address in the network captures for that device , I found that device was another firewall. After looking at that asked customer to check the default gateway on the domain controller and he informed that it was his firewall. I explained him that we need to route this request coming from UAG DA server back through same route, in which it came. We then put a route on domain controller that would route the requests coming from UAG DA server back same route.

 

 

1

 

This resulted in successful SA establishment of Corpnet access as well. DA clients were able to connect to the internal network as expected.

One simple learning, from this scenario, As discussed before ,Customer was trying to get this issue resolved for a long time , along with some brilliant folks and they checked almost every aspect of the UAG DA deployment. But their focus was coming back to the certificate warning repeatedly,they believed it was somehow  behind this issue.

In such scenarios we can go back to basics and make sure all parts of the chain are connected together.  Never forget the Networking basics and use our best friend Network monitor , It tells you almost everything , Well  at application level, if traffic is encrypted you may not see anything e.g. IPSEC traffic But at network layer level lot of things can be understood.

One of the quotes from Einstein “Doing the same thing over and over again and expect different results”  can inspire us to think differently in similar scenarios.

Welcome to WordPress.com. After you read this, you should delete and write your own post, with a new title above. Or hit Add New on the left (of the admin dashboard) to start a fresh post.

Here are some suggestions for your first post.

  1. You can find new ideas for what to blog about by reading the Daily Post.
  2. Add PressThis to your browser. It creates a new blog post for you about any interesting  page you read on the web.
  3. Make some changes to this page, and then hit preview on the right. You can alway preview any post or edit you before you share it to the world.