Tuesday, March 2, 2010

Random SOAP Faults in application (Part II - Finding the Cause of problem)

This is in continuation to my last post where some of our end users running low b/w were encountering random SOAP faults in the flex application.

To debug the problem i have requested access to one of the end users machines to see what was going on. Spending good amount of time trying to work with the end users machine didnt give any clue though. I have found that while the end user recieves the SOAP Fault in the flex application, the response sent by the server for that specific request returned just fine with HTTP 200 OK with SOAP body and header looking just perfect in fiddler (http debugging tool). I didnt have a clue on what was going wrong when the response looks perfect.

Next, I took the SOAP body from the fiddler and started looking into the Flex framework, SOAP Decoder to analyze if something is wrong with the response body content, content lenght etc. After spending more time, Everything went through fine. The response was indeed getting decoded fine and everything looks perfect. Since these SOAP Faults were random and cannot be reproducible often, it was making the client frustrating putting us through difficult situations. Time is ticking without any solution yet.

I thought of the idea of trying to deploy a debug build for one release where i enable flex framework logging. I really thought this would give hints to what is happening internally.

Call this function in your creation complete of your flex application and it would enable debugging all the messaging, rpc logs in the framework.

This link would give you complete details on what to do next to enable logging in flashlog file. After I deployed the debug build into a custom environment, I have installed debug version of flash player on couple of end users machines, configured the flashlog settings and asked them to run their tests on this custom environment swf. I should say i was really lucky to get helpful hand from the end users who were willing to nail down this problem as there was no pattern for reproducing the problem.

After couple of days, I got this error SOAP Fault hit once by the end user running the debug flex build. Without any delay i got the flashlog.txt on their machine and looked for the traces of the problem.

Without much help from flexcoders@yahoogroups or the adobe-flex-forums, I spent a lot of time trying to debug through the flex framework to identify the problem. After going through the framework code i have bottomlined it to the fact that the culprit was FIDDLER. The message is not reaching the flash player properly.

This link on the msdn gives how fiddler works. Since fiddler acts like a proxy, messages go and come via fiddler. Unfortunately Client was running old version of fiddler...In some scenarios, even though fiddler recieves the response message fine HTTP 200 OK, it doesnt throw it back to IE properly (so FP doesnt recieve it fine). One of our QA person, running old version of fiddler encountered the problem and the cause was identified by running the same scenario 20 times, with fiddler on, Flex shows SOAP Fault events vs Without fiddler on, it never shows SOAP Faults.

When the end users removed fiddler things started to work since last one week without any random SOAP Faults in the application. It looks to be problem with older version of fiddler 1.x and not with newer versions as we could not reproduce the problem in newer versions. I hope this saves someone's time. I would encourage running network sniffers instead of HTTP Debuggers for debugging purposes for these reasons



PS: The thoughts shared in this blog post are purely experimental based on what we encountered inhouse and may differ

0 comments: