Debugging PluginProcess for an NPAPI Plugin

I ran into an interesting problem lately working on an NPAPI plug-in. How on earth do you debug this thing properly? Thinking on it more the answer was fairly trivial, though I thought I would share.

If your plug-in uses the Objective-C runtime, you may end up facing scenarios where you need to troubleshoot common memory problems, like double releasing an object, or sending a message to a dealloc’ed instance. Typically the way you handle this in Objective-C is to set the NSZombieEnabled environment variable to YES for the process when it starts. Though how do you do this for the PluginProcess, if you have no control over how that process starts.

The easiest thing for you to do is start Safari itself with NSZombieEnabled, and of course any child processes that Safari spins off, including PluginProcess, will inherit that environment variable.

You can do this from Terminal:

set env NSZombieEnabled YES
killall quit Safari
open /Applications/Safari.app/

The killall is there just to ensure the process doesn’t get reused when starting Safari again. You can combine this with other very useful debugging environment variables, such as CFZombie and MallocHelp. This will work for other out-of-browser plugin models too, such as Chrome’s if you need to debug an NPAPI plugin from Chrome.

Kernel Debugging With Parallels

Occasionally, I get to do something that I haven’t done in such a long time, that I don’t know how to do it using my current toolset.

Such is an example of kernel debugging.

Kernel debugging isn’t entirely new to me, but also one of those things that I do so infrequently that it takes a little bit of ramp up time to get back in the swing of it. Additionally, I had since moved to OSX. My objective was to diagnose a GINA problem with a component. GINA hooks allow you to add customization to the Windows Login screen. In Windows Vista+, the API changed to a Credential Provider, but this particular case was Windows XP. GINA hooks run in kernel space, so we need to attach a kernel debugger.

We need to troubleshoot this problem as Windows is starting up, so we need another machine. Typically the way I’ve done this before is to have two workstations hooked together via a Serial Port. The Kernel debugger works across a serial port.

My first problem was, I don’t have a physical workstation anymore. All of my Windows work is done in Virtual Machines now. Fortunately, what I discovered is that Parallels for Mac can do that just fine. Like the workstations, we will need two virtual machines: the debugger and the debugee.

My debugee is Windows XP SP3.

In Parallels, open up the Configuration for the Virtual Machine and switch to the Hardware Tab. Click the “+” to add a serial port. For the Source, select “New Socket”, and give your socket a name. Set the Mode to “Server” and then you are all set. Make sure the “connected” checkbox is checked as well.

Screen Shot 2012-04-17 at 10.16.54 PM

That’s all it takes from Parallels for the serial port. Now would be a good time to start the debugee to enable debugging.

From here, it’s all pretty straight forward as if you were doing it on a real machine. Open up “msconfig” go to BOOT.INI, check “/DEBUG”, and set the debug port to COM1 and the baud rate to 115200. Click OK, and OK again. It may prompt you to restart, but lets not do that yet. Shut it down for now.

Now for the debugger. I’m using Windows 7 x64 for debugging, and WinDbg as my debugger tool. Before we get there, we need to add a serial port to the debugger Virtual Machine.

Shut down the debugger instance, open up the VM configuration and go to the “Hardware” tag. Click “+” and add a Serial Port. For the source, select the port that you created for the debugee instance, and set the mode to “Client”, and make sure it’s Connected.

Screen Shot 2012-04-17 at 10.27.07 PM

Start the debugger instance, and get WinDbg fired up. Make sure to get your symbol source fixed up, and start Kernel debugging over COM1.

Screen Shot 2012-04-17 at 10.36.06 PM

Once it’s waiting, go ahead and start the debugee operating system.

Screen Shot 2012-04-17 at 10.42.44 PM

Success! We have a kernel debugging session across virtual machines in Parallels.

Writing a Managed Internet Explorer Extension: Part 4 – Debugging

Picking up where we left of with Writing a Managed Internet Explorer Extension, debugging is where I wanted to go next. I promise I’ll get to more “feature” level stuff, but when stuff goes wrong, and it will, you need to know how to use your toolset. .NET Developers typically write some code and press F5 to see it work. When an exception, the debugger, already attached, steps up to the plate and tells you everything that is wrong. When you write an Internet Explorer Extension it isn’t as simple as that. You need to attach the debugger to an existing process, and even then it won’t treat you like you’re use to. Notably, breakpoints aren’t going to launch the debugger until the debugger is already attached. So we have a few options, and some tricks up our sleeves, to get the debugger to aide us.

Explicit “Breakpoints”

The simplest way to emulate a breakpoint is to put the following code in there:

System.Diagnostics.Debugger.Break()

Think of that as a breakpoint that is baked into your code. One thing to note if you’ve never used it before is that the Break method has a [Conditional(“DEBUG”)] attribute on it – so it’ll only work if you are compiling in Debug. When this code gets hit, a fault will occur. It will ask you if you want to close, or attach a debugger. Now is your opportunity to say “I want a debugger!” and attach.

It’ll look like just a normal Internet Explorer crash, but if you probe at the details, “Problem Signature 09” will tell you if it’s a break. When working on a BHO, check this every time IE “crashes” – it’s very easy to forget that these are in there. It’s also important that you compile in Release mode when releasing to ensure none of these sneak out into the wild. The user isn’t going to look at the details and say, “Oh it’s just a breakpoint. I’ll attach and hit ‘continue’ and everything will be OK”. Once that’s done, choose Visual Studio as your debugger of choice (more on that later) and you should feel close to home.

Iebreak

This is by far one of the easiest ways to attach a debugger, the problem with it is it requires a code change to get working, meaning you need to change the code, close all instances of IE, drop in the new DLL, restart Internet Explorer, and get it back into the state it was. A suggestion would be to attach on SetSite when the site isn’t null. (That’s when the BHO is starting up. Refresher here.) That way, your debugger is always attached throughout the lifetime of the BHO. The disadvantage of that is it’s get intrusive if you like IE as just a browser. You can always Disable the extension or run IE in Safe Mode when you want to use it as an actual browser. If you take this approach, I recommend using Debugger.Launch(). I’ll leave you to the MSDN Documents to understand the details, but Launch won’t fault the application, it will skip straight to the “Which debugger do you want to use?” dialog.

Attaching to an Existing Process

Attach

You can just as well attach to an existing process like you normally would, but there is one drawback: “Which process do I want to attach to?” In IE 8 that is a question that can be difficult to answer. Each tab has it’s own process (a trend in new generation browsers – IE was the first to support it). You will have at minimum of two IE processes. One for each tab, and one per actual instance of IE acting as a conductor for the other processes. Already, with just a single tab open, you have a 50/50 chance of getting it right if you guess. Visual Studio can give us some help though. If you pull up the Attach to Process Dialog, you should see your two instances of IE. The “Type” column should give it away. We want the one with Managed code in it (after all, the title of this blog series is “Writing a Managed Extension”).

Once you’re attached, you can set regular breakpoints the normal way and they’ll get hit. Simple!

Bphit

It isn’t quite as easy when you have multiple tabs open – sometimes that’s required when debugging a tricky issue. You have a few options here:

  1. When building a UI for your BHO (It’s a catch 22 – I know I haven’t gotten there yet) have it display the PID of the current process. That’s easy enough to do using the Process class. You can dumb it down a little more though and write a log file in a safe location (IE is picky where BHOs write to the File System Refresher here).
  2. Attach to all tab processes. That can lead to a lot of confusion of which tab you are currently in, because if you have two tabs open – and a breakpoint gets hit – which tab did it? The Threads Window should help you there if that is the route you choose.
  3. Always debug with a single tab, if you can.

Power Debugging

There is a trick you can do in Visual Studio to gain access to some additional debugging features. Hopefully this isn’t brand new material to everyone, but for some I would suspect it is. If you manually choose what you want to attach, include managed code and Native. Attaching to Native is very helpful if you are trying to debug a COM Marshaling issue, and plenty of other issues. We can start a the Managed Debugger Extension to diagnose issues at the CLR level, and even poke at the CLR’s native memory and objects. Once attached in Visual Studio with Native code, get to a breakpoint or pause, and launch the Immediate Window. Type .load sos and hit enter. If it worked, you should get a message like extension “C:\Windows\Microsoft.NET\Framework\v4.0.30319\sos.dll loaded”. There are many blogs out there about SOS (Son of Strike). I may blog on it later. Some useful commands are:

  • !help
    Pretty self explanatory. Shows you some available commands.
  • !dso (!DumpStackObjects)
    Does a dump of all CLR objects that are in the stack.
  • !dumpheap
    Dumps the entire heap. Careful! – That really means the entire heap. A more
    generally useful use of dumpheap is to specify the -stat flag (!dumpheap
    -stat
    ). This gives you general statistics of the heap. It will tell you which
    objects are in memory, and how many of them there are. This is useful starting point
    if you believe there is a memory leak – this can at least tell you what you are
    leaking.
  • !soe (!StopOnException)
    Again, I feel that the name of this is pretty self explanatory. The usage of it is a
    little tricky to beginners. A simple example would be, “I want to stop whenever
    there is an OutOfMemoryException”. This is useful for some types of exception,
    OOM is a good example. The problem with debugging an OOM in a purely managed way is
    the CLR is dying by the time the exception happens, so you will get limited
    information by slapping a debugger on the managed code. For an OOM, a !dumpheap
    -stat
    is a good place to start. Other examples where this is useful are Access
    Violations (more common when doing Marshaling or Platform Invoke), Stack Overflows,
    and Thread Aborts. The usage is !soe -create
    System.AccessViolationException
    .
  • !CLRStack
    This will dump the CLR’s stack only. The Native stack is left out. This is
    the same as normal managed stacks that you have looked at. It has some cool
    parameters though. The -p parameter will tell you what the values of
    the parameters that were passed into it. Often, it will be the address of what was
    passed in. Use !DumpObject and pass in the address to figure out exactly
    what it was. The -l flag dumps the locals in the frame, and
    a dumps both parameters and locals.
  • !DumpStack
    This is like CLRStack but on steroids. It has the managed stack like CLRStack, but
    also has the Native stack. It’s useful if you use Platform Invoke. This command
    is best used outside of Visual Studio and instead in something like WinDbg –
    more on that further down.

That’s the tip of the iceberg though. The complete documentation on MSDN is here. That lists commands that !help doesn’t list – so have a look. However, you’re not getting your money’s worth by doing this in Visual Studio. Visual Studio is great for managed debugging and using SOS, but when you want to use the Native commands, such as !analyze Visual Studio falls short. In addition, SOS is limited to the debugging functionality that Visual Studio provides it – you may often see a message like this: “Error during command: IDebugClient asked for unimplemented interface” Visual Studio doesn’t fully implement these features that SOS is asking for.

Other debuggers, like WinDbg, are significantly more powerful at the cost of they aren’t as simple to use. If there is demand for further details, I’ll post them. Using WinDbg is fairly similar, once you are attached, run .load C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos.dll. In WinDbg, you need to specify the full path. In addition, you will want to get symbols from Microsoft’s Symbol Server. There are symbols for mscoree and mscorwks. Having symbols for these can significantly help diagnose native to managed (and vice-versa) transitions.

Happy Debugging!

More of this series

  1. Writing a Managed Internet Explorer Extension: Part 1 – Basics
  2. Writing a Managed Internet Explorer Extension: Part 2 – DOM Basics
  3. Writing a Managed Internet Explorer Extension: Part 3
  4. Writing a Managed Internet Explorer Extension: Part 4 – Debugging
  5. Writing a Managed Internet Explorer Extension: Part 5 – Working with the DOM
  6. Writing a Managed Internet Explorer Extension: Part 6 – Regrets