Thread by @gabrielesvelto

Over the past few months @mozilla we worked on improving #Firefox stability on #Linux. This effort quickly became an example of good synergies between #FOSS projects. A short thread 👇

When Firefox crashes the user can send us a crash report which we use to analyze the problem and hopefully fix it. However to extract useful information from the report we need the debug information from the affected build and its dependencies

Most #Firefox users on #Linux use the build that comes packaged with their favorite distribution rather than our builds. Because of that we couldn't do much with their reports as we lacked the debug information for those builds (with some exceptions)

To make matters worse we didn't have the debug information for #Firefox dependencies which include a lot of different projects: @GTKtoolkit, #Mesa, @FFmpeg, #sqlite, etc...

To address this issue we started scraping debug information for #Firefox builds and its dependencies from the package repositories of multiple distrubtions: @ubuntu, @debian, @fedora, @Linux_Mint, @ArchLinuxEn and @openSUSE

With that information now available we were able to analyze >99% of the crash reports we received from #Linux users, up from less than 20%

This greatly sped up our response time to #Linux-specific stability issues, and often allowed us to identify problems in #Nightly and #Beta versions of #Firefox before they reached users on the release channel

We could also quickly identify issues in bleeding-edge components such as #WebRender, #WebGPU, #Wayland and #VAAPI video acceleration; oftentimes providing a fix within days of the change that triggered the issue

But we didn't stop there: we could now identify distro-specific stability issues and regressions. We started informing package maintainers of the problems and often had them resolved quickly

Some distributions which were using their own crash reporting system for #Firefox decided to switch to ours given the improved results

We could also identify regressions and issues in our dependencies. We communicated the issues upstream and sometimes contributed fixes, benefiting both our users and theirs

Additionally we identified a number of elusive security-sensitive issues in upstream projects thanks to #Firefox large user base and crash analysis tools such as our probabilistic heap checker

This also exposed some limitations in the tools we use for crash analysis, so we decided to rewrite them in #Rust largely relying on the excellent crates developed by @getsentry

The resulting tools were dramatically faster than our old ones, used a fraction of the memory and produced more accurate results

Patches flowed both ways: we contributed improvements to their crates (and their dependencies) while they expanded their APIs to address our new use-cases and fixed the issues we discovered

This goes on to show how collaboration between #FOSS projects not only benefits their users but ultimately improves the whole ecosystem and the broader community that relies on it

Special thanks to @calixteman @nnethercote @jan_auer and all the others that contributed to this effort!