Category: Software

One of the things about the software world is that things are constantly changing, and we can feel that change, but can we quantify it?

I’m a big fan of Google Trends, so let’s use it to identify some (mostly arbitrary) software trends and see how they compare to our intuitions.

Let’s start with a few selected programming languages from the past 5 years:

Note the fact that Javascript and Python have overtaken PHP, and somewhere in 2015 Javascript became the most popular open source language out of the ones listed. However, if we zoom out and add one more language, we can see a broader trend:

Namely, that Java has dominated for the past decade+ but has been on a long slow decline (along with PHP) for as far back as the data goes. There are probably a few reasons for each, none of which are particularly original: Java for much of its history has been overly verbose and “feels” heavyweight, because of relatively long boot times. It’s use in the enterprise has branded it an “enterprise” language.

PHP, meanwhile, benefited from being lightweight enough that it could be run on shared hosting with decent-ish security and sandboxing, as well as being loosely-typed enough and amazingly simple to deploy that it was the hobbyist platform of choice pretty much since the dawn of its existence.

But the lack of good package management plagued the Java and PHP ecosystems. These aren’t problems for Ruby, Python, or Javascript, and I personally believe they made a huge difference: people could benefit from, and add incremental value to, the ecosystem very easily, instead of siloing solutions to the same problems in repositories everywhere.

But back to the charts. If we look only at the search terms (as opposed to the topic name in any language), we see a slightly different interpretation:

Mainly, one where Ruby, PHP, and Python have more relative volume overall (but with unchanged slopes) –  perhaps a quirk of the overly aggressive categorization of searches. Which graph you trust depends on how often you think people search the term “ruby” or “python” without intending to search for the programming languages by those names.

Since I can only chart 5 terms at once, I will superimpose C++ on as the 6th line (in orange) for the curious:

It turns out C++ is the most popular search term, and has held up quite well compared to Java’s trendline. Since most software is eventually C++ or C, this is not extremely surprising, even if it is a lazy observation.

Next up is the slow death of coffeescript next to the skyrocketing popularity of es6:

Not much deep introspection needed here – Coffeescript paved the way for ES6 and was never meant to be its successor. I personally think its conciseness and tolerance leads to sloppy and sometimes buggy code. What’s more surprising to me is this:

…In which Typescript interest seems to be outpacing that of ES6. I like the concept of Typescript. It’s nice to see some respect for type safety again (all you hooligans that grew up knowing nothing but NodeJs will come around eventually).

I think this isn’t a perfect comparison, though: including “Javascript” in this chart would be meaningless, for example, because ES6 represents a subset of “Javascript” proper, whose search volume would dominate all the others. Typescript/Coffeescript are languages of their own, even if they compile to JS. However, I still find the trends interesting and somewhat unexpected. This however makes sense in light of how popular AngularJS is:

AngularJS seems to be healthy, whereas EmberJS is many times smaller and has stopped growing. I’ve thrown in jQuery and React (both libraries, not frameworks) because they both fit into the larger story: jQuery’s mission today is less relevant because modern browsers behave well and are standards compliant. This has in turn allowed React (as a library, and as a concept) to take off in the front-end.

And here, a few web servers from the past 13 years:

No surprise here: declining share of Apache and IIS is being eaten by Nginx and Node.js, with Node.js in the lead by far. Overall volumes are also down over the years, as fewer people manage their own servers. The success of Nginx at the expense of Apache doesn’t surprise me – I myself have seen good performance increases from the switch to an evented model. And Microsoft long ago lost the war for web server market share, where the open HTTP protocol and the proprietary .NET ecosystem hindered adoption by both repelling users looking for free alternatives and those drawn to open source stacks.

A similar story plays out when looking at web frameworks:

Microsoft’s stack is the only one to have lost significant market share, with CakePHP dwindling from “small” to “insignificant”. (note I couldn’t search Express.js or the like; I used Node.JS in its place). We should expect .NET and IIS adoption to move in tandem, and they do.

Finally, my favorite:

MySQL has dominated for many of the same reasons PHP has dominated for a long time: ease of setup and use, even at the expense of lax / insecure defaults, or unpure/technically inferior architecture and design.

MySQL’s relative decline is probably to the benefit of the many NoSQL solutions available today – many of which aren’t even listed.

There are other forces at play in this chart. Cassandra is almost certainly benefiting from overloaded search terms. I’m pretty sure it’s less popular than MongoDB, and it has always been difficult to deploy compared to the others (another success story along the lines of “build it fast and easy, then worry about making it reliable”). Heroku has probably done more than any other company on the planet at increasing PostgreSQL adoption.

The growing popularity of Redis does not surprise me – it has always been high value to offer “a block of memory in the cloud.”

What other trends are interesting to chart?

Some corrections and clarifications from this week’s talk on hashes in Ruby 2.4.

On converting hashval to index:

I gave an example of using modulus on a hashval (which might be huge) to an index in an array (like `152`), so it can be inserted in a table. Ruby does not actually do modulus for performance reasons; it actually takes some of the lower bits of the hashval instead (discarding the other bits). This is generally not a good idea, but apparently the alternatives are worse.

On collision resolution:

In MRI 2.4, collisions result in a secondary hash operation to find a new index, but I was incorrect when I said that Ruby did the equivalent of:

new_index = Hash(index)

This would result in all collisions continuing to collide until a free index was found. All else equal, this would almost certainly still be better than closed addressing, but we can do better. The algorithm is more accurately summarized as:

new_index = Hash(index + lower_five_bits(hashval))

which is much better than the behavior I described. When two keys collide (in bucket 152, for example), the next index to be generated via double hashing is likely to be distinct, because the hashvals of the colliding keys are probably distinct (after all, the conversion from hash to index involves discarding bits).

On guaranteeing that empty entries are found:

The mathematical theorem I was looking for was “Hull-Dobell theorem”, which guarantees a cycle linear congruential generator. Ruby uses this theorem to guarantee that during a collision the secondary hash function will eventually find an empty index, if it exists in the table.

On load factor:

In ruby trunk the load factor is currently 0.5, so if half of the table fills up, it should trigger a rehash.

We noticed some interesting behavior recently while upgrading to Rails 5.0.1 and Ruby 2.4. Between MRI 2.3x branch and 2.4.0, constant unloading no longer affects variable references to the unloaded class constant:

ref = Object.const_set(“ASD”, Class.new)
=> ASD
2.4.0 :014 > Object.send(:remove_const, “ASD”)
=> ASD
2.4.0 :015 > ref.name
=> “ASD”

In previous versions of Ruby, ref.name would return nil. In fact, in 2.4 the reference doesn’t appear to be unloaded in nearly any sense of the word – I can do ref.new and it will create a new instance of the ASD class, and calling instance.class on it will return ASD. The only change appears to be that directly trying to reference ASD now results in a NameError.

At first I thought this had something to do with threadsafety, but now I’m not so sure; Ruby lets you do all sorts of things that can silently affect behavior in other threads (like redefining methods and classes), so this reasoning would be inconsistent with that design.

Anyways, there are 3,392 commits between 2.3.0 and 2.4.0. Should be fun to track down.

We just raised a round of funding led by the amazing guys at Operator.vc. Yoav Shapira (1st VPE, Hubspot) and Mike Volpe (founding CMO, Hubspot) are swell guys in addition to being towering figures in Boston, and I look up to them a lot more than they probably know.

The new financing also includes participation from Bill Cohen, Managing Partner and Todd Breeden, Principal at KiwiVenture Partners II. With this new round, Operator.vc and KiwiVenture Partners II join our existing investors (Hubspot, Accomplice, 500 Startups and more).

Around the interwebs:

• Announcing Lead Investment in Privy – Operator.vc
• My Latest Investment – SMB MarTech
• 60,000 marketers and our new round of funding

On a related note, check out our job openings.

In nuclear weapons design, there is a two-man rule that prevents any single individual from accidentally — or maliciously — launching nuclear weapons. Each step requires knowledge and consent from two individuals to proceed. Even when the President initiates a launch order, he must jointly authenticate with the Secretary of Defense (they’re given separate codes, even though the President has sole authority).

When the order reaches the launch control center, two people are required to authenticate and initiate the launch, for example by (vastly simplifying…) turning two keys simultaneously.

The benefits are at least twofold. First, it’s much harder to compromise or impersonate two people simultaneously than it is to compromise one. Second, it also provides error correction. When two people are involved in a process, it’s much more likely that if someone is about to make an oversight or error, it will be caught. This works better when the roles are asymmetric, because then they won’t both be on the same “wavelength.” Most good processes of this type seem to be asymmetric in some way.

There are many contexts where we want error correction and extra security: executing large financial transfers, preparing patients for surgery, performing space shuttle launch checks, or running nuclear reactors. It also comes up a lot in software development, which is what got me thinking about this. Let’s count the ways we implement the two man rule:

Code review: Everyone is either doing this or making bad excuses for why they shouldn’t. But it’s the clearest and most accessible example of a two-man rule in software engineering.

Spec review: An essential part of any sizable project is a review of the specification to make sure, in particular, that 1) the right thing is being built in the right way, and 2) the right people and teams are aware of any impact the work might have on them.

Continuous integration: The branch built on your machine, but does it build on another one? This turns up countless “oh right I added this config variable/package and forgot to propagate the change” incidents before they become blocking.

Pair programming: I think of this as just real-time code review. It has all the same benefits and more, with the downside that it can’t be done asynchronously.

Deployments: I wish we did this closer to 100% of the time, but it has definitely been helpful to have a second person on hand for deployments in addition to the primary engineer. This is especially critical during complex deployments that happen in phases or involve many moving parts. Ideally the role is relegated to going through the checklist one last time (“says there are database migrations, are we expecting downtime or can we keep pre-boot on, and if so is the config correct?”), and in the event of an issue, helping to investigate or doing the checklist in reverse to roll back.

Mind the Gap

As we continue to grow, there are a few areas where I think a more consistent two-man rule will lead to high return on effort in the future:

• manually rebooting servers, changing server counts or container types
• adding/scaling services
• running one-off commands against the production database

And yes, every once in a blue moon we deploy tiny changes to production without full code review, or force a failing build onto staging — something that is intentionally difficult and unwieldy to do. This has gone from rare to extremely rare, and I expect this trend to continue. But I like processes to be developed and enforced bottom-up if possible, and prefer values over inflexible rules. So far this tenet hasn’t failed us, and we still trust each other with good judgment above all else.

However, as the stakes get higher every day, the cost/benefit equation will eventually tip towards a standard operating procedure that can be summarized as “trust, but verify.” If that doesn’t sound like a good proverb to live by, maybe a second opinion is in order?

 

I have been at Privy for a year. I’m proud of the team and product we’ve built, and I was excited to sit down and make a list of some of the new things I learned during my time here. Then I realized that most of these “lessons” would’ve been covered if I had just re-read everything ever written by Fred Brooks, Martin Fowler and Eric Ries…but that doesn’t make a good blog post.

So that got me thinking about the things I already sorta-knew that had been validated. Perhaps there was some pattern there. And so I made my first order list, which I present below.

I have learned virtually nothing about…

1. Using a stack in the middle of the adoption curve: Ruby on Rails.
   • Ruby/MRI is between 2 and 50x slower than running a static language on the JVM, but even a slight increase in developer productivity more than makes up for the operations cost.
   • The advantage of using a really fancy stack (more cool factor for recruiting, etc) really doesn’t seem to compare favorably to the disadvantages (more uncertainty, smaller pool of technical talent).
   • The evidence that startups regularly die due to technology stack is vanishingly flimsy, so no need to dwell here.
2. Building a local team.
   • Geographically distributed teams and getting on the bandwagon of “work anywhere cuz we have Slack lol” seems all the rage today, but the early team is more important than the early product, and the best teams are in the same place every day.
   • Resisting the urge to go remote has been something of a useful filtering mechanism: does this individual believe enough in our vision to consider moving here for the job?[1]
3. Having some really solid cultural values (or aspirations, as they may be) that aren’t totally groundbreaking.
   • It’s more important that we live up to great values than come up with amazing ones. I’ll leave the latter to the management consultants.
4. Using traditional engineering management.
   • We basically do agile: there are weekly-ish sprints; we do higher level planning on a monthly basis; a couple times a year we work on a strategic roadmap. We write software specifications before we code, and we ship daily with continuous integration and lower test coverage than I’d like to admit. Yawn.
   • We don’t use “flat” organizations or Holacracy or whatever trendy hipster management structure is in vogue. What the hell kind of problem is this trying to solve anyway? My theory is it’s got something to do with cool factor for recruiting, but I have a feeling the people trying this are no more certain than I am.

What’s the big meta lesson here?

If anything, it probably goes a little bit like this: the available levers to pull in a startup are numerous, but there are only a few that make a measurable difference. The things that are most likely to kill us are the things that kill most startups: having a subpar team, building a product that nobody wants, executing poorly on feedback loops, that kind of thing.

These are the things that, in Paul Graham terminology, make you “default dead” until you figure out how to get them right. And it’s critically important to realize that things like “what do we build?” and “who do we sell it to?” are the things that startups are doing “wrong by default” and need to diagnose and fix as quickly as possible.

But then there are the other things, like “how do we write a scalable system to respond to HTTP requests?” or “how should we manage engineering teams?” in which there are essentially no forced errors, and where (barring a well-articulated exception[2]) the correct answer is the default one. So almost all of the risks here seem to be to the downside, and any upside is probably insignificant compared to the scale and difficulty of the hard problem: building a novel product under uncertainty.

There are certainly going to be exceptions to this. There are going to be teams that have figured out how to deviate from orthodoxy and are reaping benefits from it. I’m OK with this, and my theory is that it either doesn’t matter (e.g., they were going to be a success anyway) or it won’t rescue them (they’re doomed and they didn’t differentiate in a way that mattered).

And so it must follow that the majority of our iterating and tweaking is on the thing that will make us a great company: what do we build? Who do we sell it to? There are enough variables in there that I don’t really have any brainpower left over to do anything except reach for Generic Ruby/Python/JavaScript framework and using engineering/recruiting/management techniques that were old 30 years ago.

 

[1] This isn’t all roses, since it biases us significantly towards younger folks who don’t have as many attachments, the net effect of which is…debatable, but obviously not lethal in a vibrant tech city like Boston.

[2] Example: One excuse I’ve used to provision real hardware in a real datacenter as opposed to just spinning up an EC2 instance is “I’ve done the math and TCO in AWS is literally 25X more expensive.”

If you’re like me, you’ve been enjoying Windows 10 for quite some time now. Couple things annoy me:

1. I accidentally changed all my file associations to the new default Windows apps, because the (intentionally) misleading firstrun experience presented fine print I glossed over.
2. I don’t like searching the web from the Windows Start menu, because I’d rather not transmit everything I type there over the network. Call me old fashioned.

Remove default apps

Open up a powershell prompt and run this to remove most of the default apps:

Get-AppxPackage *onenote* | Remove-AppxPackage
Get-AppxPackage *zunevideo* | Remove-AppxPackage
Get-AppxPackage *bingsports* | Remove-AppxPackage
Get-AppxPackage *windowsalarms* | Remove-AppxPackage
Get-AppxPackage *windowscommunicationsapps* | Remove-AppxPackage
Get-AppxPackage *windowscamera* | Remove-AppxPackage
Get-AppxPackage *skypeapp* | Remove-AppxPackage
Get-AppxPackage *getstarted* | Remove-AppxPackage
Get-AppxPackage *zunemusic* | Remove-AppxPackage
Get-AppxPackage *windowsmaps* | Remove-AppxPackage
Get-AppxPackage *soundrecorder* | Remove-AppxPackage

Turn off Web Search

Next, open up Group Policy Editor (gpedit.msc) and navigate to:

Computer Configuration -> Administrative Templates -> Windows Components -> Search. Enable the policies:

• Do not allow web search
• Don’t search the web or display web results in Search
• Don’t search the web or display web results in Search over metered connections

Finally, open up “Cortana and Search Settings” and disable “Search online and enable web results”.

Couple of quick points on Heroku’s pricing changes which I’ve been meaning to get out:

• Its not an across-the-board price cut. While the dyno pricing has decreased, they also got rid of the free $36ish/month in free dyno credits.
• New free tier replaces the free dyno credit. Minimum 6 hours of sleep per day means no more abusing the free tier by pinging your app every few minutes to keep it from sleeping. Seems a lot of people were doing this to run production apps for free; good riddance.
• New $7/month hobby tier is a great new option for people who were previously hosting production apps for free and need them live 24/7. This is a great deal since you can even have worker/background dynos for the same price. Makes sense for Heroku too – they’ll derive a good deal of long tail revenue from folks who would’ve previously just stuck with the free tier (maybe using the ping hack to prevent idling). Honestly I think the revenue is not the point – it’s more just preventing people from abusing the free tier while giving enough folks a no-excuses carrot to use the platform so it’ll be a no-brainer when they “go pro.”
• Professional dyno pricing drop is great, but it’s going to be a wash for the majority of paying users because the free credit is going away. Basically there’s no more big cliff where you go from free->paid any more, but the steepness of the pricing increases is somewhat lower. My intuition is the winners are the 4-5 figure/month customers, makes sense since that’s around the time they start thinking about moving to AWS directly for cost savings. More of them will just consider staying.

Performance has always been an obsession of mine. I enjoy the challenge of understanding why things take as long as they do. In the process, I often discover that there’s a way to make things faster by removing bottlenecks. Today I will go over some changes we recently made to Privy that resulted in our production application sending emails 150x faster per node!

Understanding the problem

When we starting exploring performance in our email queueing system, all our nodes were near their maximum memory limit. It was clear that we were running as many workers as we could per machine, but the CPU utilization was extremely low, even when all workers were busy.

Anyone with experience will immediately recognize that this means these systems were almost certainly I/O bound. There’s a couple obvious ways to fix this. One is to perform I/O asynchronously. Since these were already supposed to be asynchronous workers, this didn’t seem intuitively like the right answer.

The other option is to run more workers. But how do you run more workers on a machine already running as many workers as can fit in memory?

Adding more workers

We added more workers per node by moving from Resque to Sidekiq. For those who don’t know, Resque is a process-based background queuing system. Sidekiq, on the other hand, is thread-based. This is important, because Resque’s design means a copy of the application code is duplicated across every one of its worker processes. If we wanted two Resque workers, we would use double the memory of a single worker (because of the copy-on-write nature of forked process memory in linux, this isn’t strictly true, but it was quite close in our production systems due to the memory access patterns of our application and the ruby runtime).

Making this switch to Sidekiq allowed us to immediately increase the number of workers per node by a factor of roughly 6x. All the Sidekiq workers are able to more tightly share operating system resources like memory, network connections, and database access handles.

How did we do?

This one change resulted in a performance change of nearly 30x (as in, 3000% as fast).

Wait, what?

Plot twist!

How did running more workers also result in a performance increase of 500% per worker? I had to do some digging. As it turns out, there’s a number of things that make Resque workers slower:

• Each worker process forks a child process before starting each job. This takes time, even on a copy-on-write system like linux.
• Then, since there are now two processes sharing the same connection to redis, the child has to reopen the connection.
• Now, the parent will have to wait on the child process to exit before it can check the queue for the next job to do.

When we compounded all of these across every worker, it turns out these were, on average, adding a multiple-seconds-long penalty to every job. There is almost certainly something wrong here (and no, it wasn’t paging). I’m sure this could’ve been tuned and improved, but I didn’t explore since it was moot at this point anyway.

Let’s do better – with Computer Science^TM

In the course of rewriting this system, we noticed some operations were just taking longer than felt right. One of these was the scheduling system: we schedule reminder emails to be sent out in redis itself, inserting jobs into a set that is sorted by time. Sometimes things happen that require removing scheduled emails (for example, if the user performs the action we were trying to nudge them to do).

While profiling the performance of these email reminders, I noticed an odd design: whenever the state of a claimed offer changes (including an email being sent), all related scheduled emails are removed and re-inserted (based on what makes sense for this new state). Obviously, this is a good way to make sure that anything unnecessary is removed without having to know what those things are. I had a hunch: If the scheduled jobs are sorted by time, how long would it take to find jobs that aren’t keyed on time?

O(n). Whoops!

It turns out that the time it took to send an email depended linearly on how many emails were waiting to be sent. This is not a recipe for high scalability.

We did some work to never remove scheduled jobs out of order – instead, scheduled jobs check their validity during runtime and no-op if there is nothing to do. Since no operations depend linearly on the size of the queue any more, its a much more scalable design.

By making this change, we saw an increase in performance of more than 5x in production.

Summing up

• Moving from process-based to thread-based workers: ~6x more workers per node.
• Moving from forking workers to non-forking workers: 5x faster.
• Removing O(n) operations from the actual email send job: 5x faster.
• Total speedup: Roughly 150x performance improvement.