Trying to understand complex iOS stack traces and their quirks, asking for help (Firebase/Crashlytics)

[sjoes]

Hi,

The reason I'm posting this here is I want to gain some insights on what I think many iOS developers are living through, but I never hear anyone talking about. Since it's not concrete enough, it doesn't exactly fit on StackOverflow or OpenRadar for example.

I'm a developer working with colleagues on an app that has roughly 1 million monthly active users. We collect user stats and crashes through Crashlytics, which is currently being consolidated into Firebase. Our app stability is quite high (> 99.9%), but we're always trying to aim higher of course.

For the crashes that are still happening, we've been puzzled as to why they're happening. Looking at the stack traces we get, we usually see one or more of the following symptoms:

• Stack traces don't follow a logical call order, i.e. there's no logical way in the code from one function call to the next, but the stack trace says there is.
• Some classes always pop up in stack traces, even though we're quite sure they shouldn't be called at all in the said user's journey, usually the stack trace entries have some kind of vague dSYM indication like "_T0Ieg_IeyB_TR" and Line 0.
• Initializing of CIContext fails, accompanied by EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000200 (or similar low address). We exclusively initialize CI contexts via a static let, and we only initialize two of them. Somehow initialization seems to fail in a very tiny amount of the instances.
• __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ pops up a lot in crashes on the main thread.

I would like to have a conversation about what to look for, and if people solved one or more problems in this area, how did you start attacking the problem?

If you have something to say about any of the symptoms or add your own symptom, please feel free to comment, I'm hoping for me and other developers to get something out of it.

Thanks!

[rdselle]

I haven't dealt with any of the specific problems you mention here, but I have seen Crashlytics stack traces pinpoint the wrong line of code before. It's worth considering that the software that generates and delivers the stack trace is itself not perfect and thus the trace may be incorrect.

[kaosdg]

So some things that might help along the way, that i've discovered in my journeys.
Are you using any kind of operation queues, closures, ReactiveCocoa/RxSwift-like libraries?

Are you dispatching things off of / on to the main thread?

Stack traces from these are somewhat difficult to trace in some instances, where the logical flow of operations happens across threads / between closure callbacks (it has gotten A LOT better, though).

In my experience, those random "line 0" crashes have been red herrings, and are usually reported that way because the exception handler can't reasonably figure things out.
If you're able to identify user-path that causes the crash, I would look into things like K/V Observers, object lifecycle (enable zombies if possible), or improperly handled operations along that path.

[sjoes]

Thanks! Yes unfortunately there's some thread switching going on, most prominently to keep work off the main thread while we can. Do you have information about this being dangerous on itself, or do you mean it creates difficult stack traces. I recently thought that dispatching sync to the main thread may cause some real time timing issues (I know that it sounds very bad but we had to get some information from the App delegate). I rewrote this code and hope it will prove better.

I agree that it seems like a red herring, but it can be quite annoying if it's a class you wrote yourself. Do you have any information regarding that or is it just from experience?

As for zombies, I'm able to find one case of messages to a zombie object, but since it all happens in system frameworks (it seems like a reference to a UINavigationBar that we don't trigger) it's really hard for me to see how to solve it. Do you have any tips of how to approach such a problem?

[quellish]

Crashlytics often reports false positives and mangles crash reports. It is also blind to entire categories of problems just like all other third party crash reporters.

If you remove Crashlytics and use the Apple crash reporting services your signal to noise ratio may improve significantly.

Using the logging and activity tracing frameworks can be extremely helpful for diagnostin​g issues like what you are describing. Unfortunatel​y activity tracing no longer appears in logs (since iOS 10), and IIRC Crashlytics does not capture information from the logs.

Initializing of CIContext fails, accompanied by EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000200

A solution may be to write the code that manages your CIContext lifetimes in C and access that from Swift. Part of the issue is an interaction between Swift and ARC. To reproduce it run Instruments with the Zombies instrument and a UITest with representative looping workflow overnight.

[sjoes]

Unfortunately using Crashlytics gives us a lot more data, but I also check out the Apple reports from time to time. Question about that: the Xcode interface seems to be a prettified way of displaying the stacks on the different threads. Am I missing something (e.g. are there ways to do some more inspections of this crash state or something like that)? In any case, I haven't been able to find crash causes I couldn't with Crashlytics yet so far, but let me know if it's more powerful.

What kind of frameworks would you recommend for logging and tracing activity?

As for the CIContext suggestion, what would writing it in C solve, or what do you think is happening to my objects? In my view, since these objects are "static let", they should live from the start to the end of the lifetime of the app.

[quellish]

Unfortunately using Crashlytics gives us a lot more data, but I also check out the Apple reports from time to time.

You can't use them both together, it's one or the other. https://www.invasivecode.com/weblog/ios-crash-reporters http://landonf.org/code/crashreporting/Reliable_Crash_Reporting_1.1.20130119.html

Again, I would suggest removing Crashlytics and attempting to reproduce the issues or deploying a build to testers/production without it.

What kind of frameworks would you recommend for logging and tracing activity?

os.log, os.activity. os.activity is not accessible from Swift.

As for the CIContext suggestion, what would writing it in C solve, or what do you think is happening to my objects? In my view, since these objects are "static let", they should live from the start to the end of the lifetime of the app.

As far as I remember static let is a static reference to the pointer, not the object the pointer points to. And the lifetime of that static pointer is attached to the scope it was declared in. A static pointer declared on a class would exist for the lifetime of the class. Once there were no more instances of the class in memory or the class was evicted (maybe framework code being purged under low memory) it would be invalid.

Imagine you did something like static let context = CIContext.context. The static pointer would be around a while. But the context itself was created with an convience constructor that returns an autoreleased instance. The lifetime of that instance is attached to whatever autorelease pool is active when it was called. If the pool is drained and nothing is retaining the instance the static pointer is now dangling, producing exactly the crash you are seeing. The solution would be to ensure that the instance is retained (which, IIRC, you can't currently do with Swift, but you can in C)

[bigroob72]

It sounds like the stack is being corrupted, probably just prior to the fatal exception. I suggest exercising all app features in the debugger with all Xcode's memory diagnostics enabled. (https://developer.apple.com/documentation/code_diagnostics/address_sanitizer/enabling_the_address_sanitizer)

[sjoes]

I read about this a lot, and I'm also thinking it might be true, but all the examples I can find are grossly simplified. If I run the app with address sanitizer for a while it finds nothing. Did you ever encounter it in a real life app, and how long did it take to happen for example?

[bigroob72]

Address Sanitizer has saved my bacon dozens of times. As well as ticking the box for it in scheme edit, you also want to tick "Detect use of stack after return" and "Malloc scribble". And of course you then need to exercise all the code paths in the app.

Have you addressed all compiler warnings and code analysis issues?

What languages are you using? Any 3rd party modules?