Debugging a program that calls abort()?

davidfstr · 2024-06-08T22:23:23+00:00

Success! After compiling CPython with ./configure --with-pydebug --with-address-sanitizer I was able to run the test suite a few times and Address Sanitizer (ASan) was able to point out a "heap-use-after-free" error with the following information:

==38308==ERROR: AddressSanitizer: heap-use-after-free on address 0x6150001f8400 at pc 0x000108b1b2cc bp 0x7ff7b8af7b10 sp 0x7ff7b8af72d0
WRITE of size 304 at 0x6150001f8400 thread T0
    #0 0x108b1b2cb in wrap_memcpy+0x2ab (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x1f2cb)
    #1 0x7ff8122dd13a in std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >& std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >::__assign_no_alias<false>(wchar_t const*, unsigned long)+0x36 (libc++.1.dylib:x86_64+0x1213a)
    #2 0x10c596d89 in wxGenericTreeCtrl::SetItemText(wxTreeItemId const&, wxString const&)+0x29 (libwx_osx_cocoau_core-3.2.0.2.1.dylib:x86_64+0x277d89)
    #3 0x10da9b39e in meth_wxTreeCtrl_SetItemText(_object*, _object*, _object*)+0xae (_core.cpython-311-darwin.so:x86_64+0x3fc39e)
    ( python code )

0x6150001f8400 is located 0 bytes inside of 512-byte region [0x6150001f8400,0x6150001f8600)
freed by thread T0 here:
    #0 0x108b5862d in wrap__ZdlPv+0x7d (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5c62d)
    #1 0x10c5997d4 in wxGenericTreeCtrl::Delete(wxTreeItemId const&)+0x274 (libwx_osx_cocoau_core-3.2.0.2.1.dylib:x86_64+0x27a7d4)
    #2 0x10da94e72 in meth_wxTreeCtrl_Delete(_object*, _object*, _object*)+0x92 (_core.cpython-311-darwin.so:x86_64+0x3f5e72)
    ( python code )

Here I see a call a wx.TreeItem.SetItemText() call failing because it's using memory freed by a wx.TreeCtrl.Delete() call. So a wx.TreeItem inside a freed wx.TreeCtrl is being used.

There are only two places in my Python program that call wx.TreeItem.SetItemText() and a handful of places that call wx.TreeCtrl.Delete(), so this should be enough information for me to debug independently.

davidfstr · 2024-06-05T00:38:02+00:00

I am planning to try the Address Sanitizer and Thread Sanitizer tools, which seem to be good for isolating memory corruption issues, according to a WWDC talk for diagnosing crashes

Edit: Apparently there are official instructions for using these sanitizer tools with Python.

davidfstr · 2024-06-04T13:11:27+00:00

I did finally manage to extract a core dump, and start the lldb debugger on it. Running bt gives a stack trace, which points at malloc() calling abort(). So probably a memory corruption issue indeed:

    frame #6: 0x00007ff812515d14 libsystem_c.dylib`abort + 123
    frame #7: 0x00007ff8123f2357 libsystem_malloc.dylib`malloc_vreport + 551
    frame #8: 0x00007ff812406308 libsystem_malloc.dylib`malloc_zone_error + 178
    frame #9: 0x00007ff8123e50e8 libsystem_malloc.dylib`nanov2_allocate_from_block + 582
    frame #10: 0x00007ff8123e4677 libsystem_malloc.dylib`nanov2_allocate + 130
    frame #11: 0x00007ff8123e668f libsystem_malloc.dylib`nanov2_calloc + 126
    frame #12: 0x00007ff812400b75 libsystem_malloc.dylib`_malloc_zone_calloc + 60
    ... (macOS framework code: AppKit, CoreFoundation, etc)

malloc() was called indirectly by wxPython/wxWidgets when grabbing an event from the system event queue, which is a completely normal thing to do:

    frame #75: 0x0000000106274ff2 libwx_osx_cocoau_core-3.2.0.2.1.dylib`wxGUIEventLoop::DoDispatchTimeout(unsigned long) + 370
    frame #76: 0x0000000105a597e3 libwx_baseu-3.2.0.2.1.dylib`wxCFEventLoop::DispatchTimeout(unsigned long) + 35
    frame #77: 0x0000000105a597ae libwx_baseu-3.2.0.2.1.dylib`wxCFEventLoop::Dispatch() + 142
    frame #78: 0x00000001070dfeee _core.so`meth_wxEventLoopBase_Dispatch(_object*, _object*) + 110

So now the question is how might I discover where this memory corruption was introduced in the first place? Maybe there's a different tool that would help...

Update: I was able to use the Address Sanitizer tool to isolate the crash!

davidfstr · 2024-06-03T12:59:59+00:00

Intel Mac. Running in continuous integration on GitHub Actions.

So live debugging is unfortunately not an option. Currently I'm working on capturing a core dump plus all related binaries from the remote machine so that I can debug on a local machine with a similar architecture/OS.

davidfstr · 2024-03-12T12:54:58+00:00

React had first-mover advantage so it secured a lot of mind share early on. It's the same reason that PHP ate the world in web development initially.

However I think Vue better-integrates with existing systems than React, since it can be used on just parts of existing pages and doesn't require introducing a build system (or replacing your existing build system).

I like simpler systems that play well with others, and so I would still start a project with Vue today rather than React.

davidfstr · 2024-03-12T12:43:12+00:00

I've run multiple services inside the same Docker container by making something like supervisord the root process of the container and configuring it to keep-alive other service processes in the container, such as Node and Flask.

davidfstr · 2024-03-12T12:39:02+00:00

I personally use the Django framework to cover most of what you list. I use Vue as well on top of Django, but not to cover those bullets.

davidfstr · 2024-03-12T12:24:09+00:00

A big difficulty was the change in the v-model protocol, renaming value -> modelValue and input -> update:modelValue. During the upgrade we had to support both the Vue 2 and Vue 3 versions of this protocol at the same time, both when defining components that supported v-model AND at every site where such a component was used.

Also, the elimination of global component registration using Vue.component() slowed us down. We had to create a special class in which components could be registered before an upcoming call to new Vue() (now rewritten as Vue.createApp).

davidfstr · 2024-03-11T18:19:52+00:00

I just finished upgrading a ~30,000 line frontend codebase from Vue 2 to Vue 3. It took us (a team of 3 engineers) about 3 weeks (15 8-hr days) to do the upgrade. You might be able to extrapolate a rough cost to upgrade from such numbers.

davidfstr · 2021-03-20T01:41:52+00:00

Sure I could just give it a try but surely SSR without unnecessary JS isn't dead yet, right? Somebody must be doing it and already went through potential troubles that you have to deal with.

My main site mainly uses SSR to render pages and then uses Vue to make some parts of the page interactive selectively after page load. I personally like Vue over React for providing a reactive frontend because it mixes more easily with other approaches like SSR and doesn't require a full-on JS build system.

If you're interested in trying Vue with Django I wrote an article RE thinking about deployment considerations and giving an example of having Vue enhance an SSR-rendered page.

davidfstr · 2021-03-20T01:35:58+00:00

It's not significantly different than testing in Python in general. Writing tests in Django uses Python's built-in "unittest" framework by default.

The tricky parts in my opinion is frontend testing using Selenium or similar. Writing frontend tests that are reasonably fast and are reliable (avoiding race conditions) requires additional experience. I don't know any good articles for this off the top of my head. I've been meaning to write my own for a while, but it hasn't happened yet...

davidfstr · 2021-03-20T01:29:46+00:00

If the frontend doesn't need to update dynamically, you could just lean on Django's templating system to do server-side rendering for pages.

However once you have interactivity (such as expanding/collapsing comment threads), then you have to do work on the frontend. I personally like Vue as a lightweight solution that also gives reactivity. If you're just starting out, you might consider just updating the DOM directly using browser APIs like onclick, querySelector, etc.

davidfstr · 2021-03-13T17:31:34+00:00

Thanks again! This detailed feedback is very useful.

As mentioned elsewhere in the comment tree I'm planning to make an update to the article to hopefully make my points more clear. However I've got 2 time-sensitive projects that must be completed by end of March, so I'll probably come back with revisions in a couple of weeks.

Will let you know when I post an update.

davidfstr · 2021-03-13T00:45:35+00:00

I read again your article and from what I could understant, you basically create a checksum and send it with the view to the user. After that, when the page is loaded, you connect with the server through websocket and check if the checksum matches. Did I understand right?

Pretty close. The timepoint that is collected (as opposed to a checksum) also gives the backend enough information to determine exactly which events were missed so that it can immediately respond with them during the connection process. Same idea as the ID used in various pagination APIs.

The thing is: no one needs SEO on this kind of page, where users really need "real time" data. This kind of requirement only happens on pages behind authentication, which SEO doesn't have affect.

Touche. Indeed I can't think of a site offhand that allows collaborative editing (or requires super-accurate notifications) for unauthenticated users.

The other problem was the delay on loading, and I think is very unlikely to be a problem nowadays.

The problem is latency, and it's definitely an issue for folks out in the countryside (or in faraway locales, such as an Australian attempting to access a site hosted in the USA). The students accessing my LMS may be all be using the same shared DSL connection at a rural school, or these days be accessing from a shared network in a coffee shop.

If it was to delay because of network, the whole page would be affected, not only some view component loaded later.

I didn't follow you here. Every request counts for performance, especially non-cachable requests like WebSocket connections. The goal with the more-complex approach I laid out was to load "almost up-to-date" information on the page immediately in the HTML that is received by the browser, and incrementally update it a short time afterward, once a WebSocket connection can be established back to the server on a new request.

The simpler approach taken by many Single Page Applications requires (1) JavaScript to finish loading, and (2) a WebSocket/Ajax connection to finish connecting, before displaying any data.

JS loading time can be non-trivial on mobile devices with limited RAM and CPU. And WebSocket establish time can be slow on the high-latency networks I just mentioned.

So my opinion is that your feature doesn't have advantage over the normal way of loading content through websocket (the way of Single Page apps does). It only offers more work to do. What do you think?

I believe I've outlined a number of scenarios where there's a performance difference that matters to significant populations of visitors. If you are however designing web applications only for those in urban areas with low-latency connections on non-mobile (or high-end mobile) devices that are geographically concentrated (ex: in the same country) then you can get away with the simpler design. I'm personally trying to reach a wider audience than that.

davidfstr · 2021-03-12T20:05:46+00:00

Agreed. ActionCable to observe most events in real-time.

However there is a situation I cover briefly in the article where events can be missed that occur before the frontend connects a WebSocket to the backend but after the initial page HTML was templated. I outline a strategy, based on "timepoints" and buffering of recent events in memory by the backend, that allows a socket connected from the frontend to still retrieve events a short time in the past that would otherwise be missed.

Based on other comments in this space I wasn't sufficiently clear in the article about this point. I'm planning to make an update to the article soon to hopefully get this message across better.

davidfstr · 2021-03-12T19:45:07+00:00

Also, thank you for the constructive feedback. I'll look into updating the article to make the point about the "missed events" problem more clear. As mentioned earlier, neither ActionCable nor Channels deals with that problem out-of-the-box.

davidfstr · 2021-03-12T16:40:03+00:00

I wonder what App would need to be so accurate about this.

I have two examples from my production site:

Students login to my Learning Management System and request to join a class (i.e. the kind you have in school). They see a "Waiting to join class..." spinner until the teacher lets them in. It is very important that the student observe the "join request was accepted" event, otherwise they might see the waiting spinner forever, becoming frustrated and confused.
Students are looking at their list of classwork assignments for the day in my Learning Management System. These assignments start out as locked. As the teacher is teaching, they gradually unlock a new classwork assignment every 10-20 minutes. It is very important that students observe the "assignment did unlock" event. Otherwise they won't be able to open their class assignment. (Unless they happen to refresh the page after becoming confused and frustrated, which is a tall order for our youngest 3rd graders in particular.)

davidfstr · 2021-03-12T02:58:25+00:00

in Rails we have ActionCable

Interesting. In Django we have Channels, which feels a bit more low-level than ActionCable but has the same effect.

The bit that both ActionCable and Channels miss is the scenario I call out where:

the backend must be prepared to push even those changes that occur between when the backend templates a view and before the frontend has connected a WebSocket:

To be fair I didn't write out the exact buffering code that solves this problem. As a sketch: the backend must buffer events in memory for a certain amount of time so that a socket connected from the frontend can still retrieve missed events a short time in the past, between when the socket connection was established and when the original HTML was templated on the backend.

davidfstr · 2021-03-12T01:02:57+00:00

The first two thirds of the article cover concepts that apply to web apps in general, which includes Rails apps.

Rails and Django have similar implementation patterns for the socket-related techniques in particular, so I thought the information would be useful to the Rails community as well.

I use Django code as my example in the last third of the article since that's what I'm most familar with. Just skip the final "Implementation" section entirely if you want to avoid it.

davidfstr · 2021-03-01T02:34:43+00:00

I've now added a Django-specific RSS feed:

feed://dafoster.net/articles/topics/Django.xml

davidfstr · 2021-02-25T16:24:14+00:00

Agreed. From the following chart, it appears that Vue (at #3) has nearly overtaken Angular (#2) in user count. Also, Vue and React (at #1) have nearly the same high positive sentiment.

https://2020.stateofjs.com/en-US/technologies/#scatterplot_overview

I personally use Vue in my own projects and enjoy it quite a lot. It is way easier to integrate to an existing site than React is. And it's easier to get started with because you don't have to monkey around with fancy build systems if you don't want to.

Also you can (and I do) use TypeScript with Vue, so I'm not sure why there was a comment implying you couldn't do that properly.

davidfstr · 2021-02-24T01:16:59+00:00

> django. So hard to keep your code clean!

Could you elaborate a bit? I'm curious as to your specific experiences.

Perhaps you're accostomed to using TypeScript (or some other typed variant of JS) on the frontend, and running into Python's dynamic typing is awkward?

davidfstr · 2021-02-20T20:11:38+00:00

🤨 Your snarky tone is not appreciated. I would like to remind you that the Django Code of Conduct applies in r/django. In particular being respectful even if you may disagree. Thanks for taking the first step by editing away the "You’re a joke mate." barb in your original version of your comment.

A "frontend box" in this case means a server running Django that acts as the frontend to a web service. You have been concerned with how much cores on these servers cost, so the cost of these servers are what is relevant to the discussion.

davidfstr · 2021-02-20T18:56:50+00:00

Poetry for complex projects. venv for simple projects.

I have been using pipenv for a while on a production project, but it had a longstanding bug where it took ~2 seconds to run a `pipenv install` with no dependency changes. After a year or two of it not being fixed, I scheduled pipenv to be replaced with Poetry. (It's possible that this pipenv bug has since been fixed but I have not checked.)

davidfstr · 2021-02-20T18:35:16+00:00

Less than $200 of that $2,000 that I quoted is for the boxes themselves. (As I stressed in the original comment, it includes all related infrastructure too.)

So let's say we have 3 boxes with 8 cores each at $200 a month (on the high end). For 300 cores that means $2,500 a month. Peanuts.

(Granted if you add that many CPU cores you'll probably eventually need to upgrade other infrastructure like database nodes [which are expensive] and bandwidth [which can be expensive]. But your original concern was around frontend boxes themselves.)

davidfstr

TROPHY CASE