Understanding Google's Bug Bounty Program

Some people have taken Google’s idea of offering security bug bounties, and taken them to their logical conclusion: why stop at security bugs? Why not incentivize reporting of ALL software bugs with bounties? Aren’t other companies cheap for not offering bug bounties?

Questions along these lines misunderstand how software development works. Engineers don’t sit on their hands and surf Reddit after shipping a product. They’re already working on bugs. All sufficiently complex software ships with known bugs; more reporting isn’t likely to change whether they get fixed or not.

So the premise that “reporting more bugs will improve software quality” is speculative, at best. Software quality is determined by what the market will bear. The market usually rewards buggy-but-good-enough software that solves a problem now, rather than perfect solutions that are late to the party. This is partly because of the time value of software, and partly because chasing defects offers diminishing returns.

But more to the point: everyday users are not equipped to report bugs. They don’t have the training, tools, or motivation to do it properly.

At Microsoft (and other software companies), crash dumps already include as much information as can be legally collected, based on the user’s consent. So bug information for crashes and many other issues (uncaught exceptions, for example) are already being collected in an automated, accurate way. So really, any well-supported software already has built-in reporting for most high-impact bugs.

In addition, you run into other problems with bug bounties:

  • Common bugs will be over-reported, wasting everyone’s time. Customers are already motivated to report these since they want them fixed.
  • Unexpected, but by-design behavior will be incorrectly reported as bugs.
  • Problems will be exacerbated by the bounties, increasing volumes and decreasing quality. This actually creates noise around what is really important to fix (e.g., the defects users would report even if there was no bounty).
  • A bug bounty program isn’t free. Someone has to triage the input and it’s a zero sum game: either a developer does less productive work sorting through bounty submissions or headcount grows. It’s not like you can fire the testing team.

The only exception to this rule is security bugs, which operate a little bit differently than run-of-the-mill defects:

  • There is already a market for security bugs, which can be sold to hackers. The developer is simply trying to outbid them to keep the product secure.
  • This means there’s already a set of professionals who are hunting for such bugs; professionals are much more likely to find bugs on account of understanding how software is designed and implemented.
  • Users are unlikely to notice or report security bugs since they generally don’t obstruct functionality, meaning there are fewer dupes to wade through, and bug reports will be of higher quality on average.

So in my opinion, paying bounties for security bugs can be effective, but its not likely bounties for functionality bugs in the general case would be particularly productive.

Leave a Reply

Your email address will not be published. Required fields are marked *