Today I want to talk about how hard drive prices have plunged dramatically since 2000 (most data from John C. McCallum's disk prices page):

We are talking about a roughly 97% decrease in price per GB over 20 years. In fact it's kind of hard to see on a linear scale, so let's switch to logarithmic for this time period:

Yup. Definitely logarithmic. Let's zoom in to the past 10 years or so (linear scale):

For fun we can also superimpose a chart from BackBlaze, a backup service that shared their average price per hard drive through 2Q 2017:

The 2011 Thailand monsoons left a noticeable blip on prices in early 2012, but otherwise the trend is pretty clear: there's been a deceleration of progress, with gains getting a lot slower starting around 2016. To date there's been a lot of resistance getting below $0.02 per GB. So the era of just assuming storage will get significantly cheaper every year is over, at least for now. In fact, prices don't seem to have really dropped at all since mid 2019 when some drives started selling for $0.019 / GB.

However, there's a nuance we're missing if we look at only price per GB without accounting for density. Let's look at a snapshot of hard drive prices at various capacities from December 2020 on Newegg:

The prices for 3.5" desktop drives range from $40 for a 500GB drive to ~$645 for an enterprise 18TB. We can conclude that at $40 the cost of the drive is almost entirely dominated by fixed costs, since the price doesn't actually change at all between 250GB and 1TB, and barely changes at 2TB (the cheapest 2TB drive is roughly $50).

Because of this fixed cost issue, it's very hard to get good "bang for buck" at lower drive capacities. Here's the cost per GB as a function of drive capacity:

And here's the same chart on a log scale:

For many years the sweet spot was a 4TB drive, but now it looks like the absolute best deal might be the 8TB drives. However, I'm surprised to see some 14TB and 16TB drives get within striking distance ($0.021 per GB vs 0.019 per GB). If the larger capacity drives are roughly as reliable, then going for 16TB drives is a no-brainer, since they'll consume less space and power than two 8TB drives.

Because of this, and despite $/GB not really dropping much, big datacenters and cloud providers are likely to continue seeing marginal decreases in total cost of ownership and operational costs per GB-month, but absent new breakthrough technology, not that much progress.

What are the implications of this?

A number of startup darlings have succeeded by betting on trends in underlying technologies to make their business viable (e.g., YouTube was founded 15 years ago and would've remained hopelessly unprofitable if not for the massive drop in bandwidth prices). And many a startup capitalized on the drop in storage prices to build attractive data warehousing and database solutions, replacing technology that assumed a GB cost $15. But if you're betting on the price of say, AWS S3 pricing to go down significantly in the near future to make your startup profitable, well, bad news:

Amazon literally hasn't reduced prices for S3 since 2016, almost 5 years now.