Catching float- and struct-returning messages to nil

Wim came up with a neat trick a while back that we've used to find and fix several bugs in our software, and to file a bunch of Radars. There are several messenger dispatch functions in the Objective-C runtime. Of particular interest here are objc_msgSend_fpret and objc_msgSend_stret. These are used by the compiler when calling a method that returns a float or struct, respectively.

Depending on your architecture, the result of such a message can be undefined when sent to nil. Messaging nil is very useful most of the time, but you can introduce rarely manifesting bugs in this case.

Looking at the disassembly for these two functions in gdb, though, gives us an easy way to catch them. Under 10.4.8/x86, we see the following:

(gdb) x/50i objc_msgSend_fpret

0x90a573c0 : mov 4(%esp),%eax

0x90a573c4 : test %eax,%eax

0x90a573c6 : je 0x90a57420

0x90a573c8 : mov 0(%eax),%eax

...

That is, load the first argument, check for zero, if so jump to 0x90a57420.

Likewise, in objc_msgSend_stret:

(gdb) x/50i objc_msgSend_stret

0x90a57340 : mov 8(%esp),%eax

0x90a57344 : test %eax,%eax

0x90a57346 : je 0x90a573a0

0x90a57348 : mov 0(%eax),%eax

...

In our ~/.gdbinit we can have:

# Nil-handling path for objc_msgSend_fpret.

b *0x90a57420

comm

p (char *)$ecx

end

# Nil-handling path for objc_msgSend_stret.

b *0x90a573a0

comm

p (char *)$ecx

end

(where the print command shows the selector).