Mozilla has apologised for the fiasco earlier this month when an expired certificate disabled most users' Firefox add-ons.
"We strive to make Firefox a great experience. Last weekend we failed, and we're sorry," Joe Hildebrand, the recently named head of Firefox engineering, wrote in a post to a company blog. "We let you down and what happened might have shaken your confidence in us a bit, but we hope that you'll give us a chance to earn it back," Hildebrand concluded.
The gaffe began just after 9 p.m. ET on Friday, May 3, when a certificate used to digitally sign Firefox extensions expired. Because Mozilla had neglected to renew the certificate, Firefox assumed add-ons could not be trusted - that they were potentially malicious - and disabled any already installed. Add-ons could not be added to the browser for the same reason.
As users went ballistic, Mozilla rushed a stop-gap fix to the browser via its Studies system, infrastructure normally responsible for pushing test code to small groups or collecting data on reactions to sponsored content. Because the Studies approach did not reach everyone, on May 5 and May 7 Mozilla shipped two Firefox updates - 66.0.4 and 66.0.5 - that corrected the certificate chaining error.
Although Hildebrand didn't describe the debacle in detail, Eric Rescorla, Firefox's chief technology officer, did. Rescorla explained how Firefox add-ons are digitally signed, what caused the incident and why some users didn't notice that their add-ons were crippled until much later than others. He also narrated the steps Mozilla engineers took off the bat, and those implemented later, producing what read as a forthright postmortem.
"We weren't sure that either of these ((approaches)) would work, so we decided to pursue them in parallel and deploy the first one that looked like it was going to work," Rescorla said. "At the end of the day, we ended up deploying the second fix, the new certificate."
Most importantly, Rescorla laid out his ideas on the steps Mozilla should take to insure against a reoccurrence. "We clearly need to adjust our processes both to make this and similar incidents less likely to happen and to make them easier to fix," he wrote.
"First, we should have a much better way of tracking the status of everything in Firefox that is a potential time bomb and making sure that we don't find ourselves in a situation where one goes off unexpectedly," Rescorla said, clearly referring to the oversight leading to the expired certificate. "Second, we need a mechanism to be able to quickly push updates to our users even when - especially when - everything else is down." Calling the Studies system "an imperfect tool," Rescorla also noted that Firefox's update mechanism needs to be more responsive.
Mozilla will publish a more thorough evaluation of the incident this week, Rescorla promised, as well as a list of intended changes.