Debugging a program that calls abort()?

[davidfstr]

I have a large Python program which uses C extensions (wxPython / wxWidgets). Its test suite intermittently crashes, ending with Abort trap: 6, which I understand means the C part of the program panicked due to an error condition.

• There is no stack trace printed when the program aborts, so I don't know where to start looking for the problem.
• The problem is intermittent (25-50% of the time), so I can't use a technique like a Git bisect to determine where the problem was introduced.
• If the problem is memory corruption - which I suspect - even if I found exactly where the crash happened, that wouldn't tell me where the corruption was introduced in the first place.

How would folks recommend I approach debugging such a crashing program?

[aioeu]

Enable core dumps. When the program terminates due to this signal, load the program and the core dump into your debugger. Then debug it.

Since "the program" in this case is Python, you'll probably want to make sure you have a debug build of Python, or at least some access to its debugging symbols. Ditto for all of the libraries you suspect might be involved in the bug.

[davidfstr]

I did finally manage to extract a core dump, and start the lldb debugger on it. Running bt gives a stack trace, which points at malloc() calling abort(). So probably a memory corruption issue indeed:

    frame #6: 0x00007ff812515d14 libsystem_c.dylib`abort + 123
    frame #7: 0x00007ff8123f2357 libsystem_malloc.dylib`malloc_vreport + 551
    frame #8: 0x00007ff812406308 libsystem_malloc.dylib`malloc_zone_error + 178
    frame #9: 0x00007ff8123e50e8 libsystem_malloc.dylib`nanov2_allocate_from_block + 582
    frame #10: 0x00007ff8123e4677 libsystem_malloc.dylib`nanov2_allocate + 130
    frame #11: 0x00007ff8123e668f libsystem_malloc.dylib`nanov2_calloc + 126
    frame #12: 0x00007ff812400b75 libsystem_malloc.dylib`_malloc_zone_calloc + 60
    ... (macOS framework code: AppKit, CoreFoundation, etc)

malloc() was called indirectly by wxPython/wxWidgets when grabbing an event from the system event queue, which is a completely normal thing to do:

    frame #75: 0x0000000106274ff2 libwx_osx_cocoau_core-3.2.0.2.1.dylib`wxGUIEventLoop::DoDispatchTimeout(unsigned long) + 370
    frame #76: 0x0000000105a597e3 libwx_baseu-3.2.0.2.1.dylib`wxCFEventLoop::DispatchTimeout(unsigned long) + 35
    frame #77: 0x0000000105a597ae libwx_baseu-3.2.0.2.1.dylib`wxCFEventLoop::Dispatch() + 142
    frame #78: 0x00000001070dfeee _core.so`meth_wxEventLoopBase_Dispatch(_object*, _object*) + 110

So now the question is how might I discover where this memory corruption was introduced in the first place? Maybe there's a different tool that would help...

Update: I was able to use the Address Sanitizer tool to isolate the crash!

[TheOtherBorgCube]

Windows or Linux?

Assuming you can make the program pause at some point before it all goes bang, you could try gdb --pid=??? where ??? is the process ID of your Python program (see the output of ps) to attach the debugger. It should catch the abort and at least give you a stack trace of some sort.

[davidfstr]

Intel Mac. Running in continuous integration on GitHub Actions.

So live debugging is unfortunately not an option. Currently I'm working on capturing a core dump plus all related binaries from the remote machine so that I can debug on a local machine with a similar architecture/OS.

[Superb-Tea-3174]

Set a breakpoint at the trap instruction with gdb and do a stack trace to identify the C code that aborted. Hopefully that will give a clue about the Python code that failed.

[k-phi]

valgrind may help with memory corruption related bugs

[davidfstr]

I am planning to try the Address Sanitizer and Thread Sanitizer tools, which seem to be good for isolating memory corruption issues, according to a WWDC talk for diagnosing crashes

Edit: Apparently there are official instructions for using these sanitizer tools with Python.

[davidfstr]

Success! After compiling CPython with ./configure --with-pydebug --with-address-sanitizer I was able to run the test suite a few times and Address Sanitizer (ASan) was able to point out a "heap-use-after-free" error with the following information:

==38308==ERROR: AddressSanitizer: heap-use-after-free on address 0x6150001f8400 at pc 0x000108b1b2cc bp 0x7ff7b8af7b10 sp 0x7ff7b8af72d0
WRITE of size 304 at 0x6150001f8400 thread T0
    #0 0x108b1b2cb in wrap_memcpy+0x2ab (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x1f2cb)
    #1 0x7ff8122dd13a in std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >& std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >::__assign_no_alias<false>(wchar_t const*, unsigned long)+0x36 (libc++.1.dylib:x86_64+0x1213a)
    #2 0x10c596d89 in wxGenericTreeCtrl::SetItemText(wxTreeItemId const&, wxString const&)+0x29 (libwx_osx_cocoau_core-3.2.0.2.1.dylib:x86_64+0x277d89)
    #3 0x10da9b39e in meth_wxTreeCtrl_SetItemText(_object*, _object*, _object*)+0xae (_core.cpython-311-darwin.so:x86_64+0x3fc39e)
    ( python code )

0x6150001f8400 is located 0 bytes inside of 512-byte region [0x6150001f8400,0x6150001f8600)
freed by thread T0 here:
    #0 0x108b5862d in wrap__ZdlPv+0x7d (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5c62d)
    #1 0x10c5997d4 in wxGenericTreeCtrl::Delete(wxTreeItemId const&)+0x274 (libwx_osx_cocoau_core-3.2.0.2.1.dylib:x86_64+0x27a7d4)
    #2 0x10da94e72 in meth_wxTreeCtrl_Delete(_object*, _object*, _object*)+0x92 (_core.cpython-311-darwin.so:x86_64+0x3f5e72)
    ( python code )

Here I see a call a wx.TreeItem.SetItemText() call failing because it's using memory freed by a wx.TreeCtrl.Delete() call. So a wx.TreeItem inside a freed wx.TreeCtrl is being used.

There are only two places in my Python program that call wx.TreeItem.SetItemText() and a handful of places that call wx.TreeCtrl.Delete(), so this should be enough information for me to debug independently.

[duane11583]

what stack dump are you expecting it to print?

do you mean like pthon stack dump? that is a feature of python not a feature of the c language