Presented by HockeyApp

Crash report data
from multiple services on multiple CPU architectures.

We analyzed all of the reports
and categorized them as follows:

No crash report created or the crash report did not contain the method that caused the crash.
The crash report has missing frames, wrong class names, wrong methods, or shows wrong line numbers.
The crash report is 100% accurate and matches the control flow of the app.

Introduction

There are many crash reporting services and all claim to provide the best crash reports. So we started wondering which service actually holds up to their claims?

CrashProbe is a crash reporting test suite presented by HockeyApp that benchmarks and compares popular iOS and Mac OS X crash reporting services. In our effort to ensure the best possible and most accurate crash reports, we created a set of test cases that we use to verify PLCrashReporter and improve HockeyApp.

Two components are involved in the crash reporting process:

  1. A crash reporting framework that can be embedded into apps to capture crashes and exceptions.

  2. A server-side process that symbolicates the crash reports to add details like class names, filenames, line numbers, and additional information like the reason.

Both components in this process are important, so even if a service is using the same framework as another service, the crash report can still be quite different.

CrashProbe is open source, so you can perform all tests yourself with any crash reporting framework. Compare the quality of the symbolicated crash reports yourself on iOS and OS X. The source code is available under the MIT license on GitHub.