Everyone seems to know what internal tools are, but they are poorly understood on a broader level. How do the world’s best companies think about internal tools? How much time should engineering spend on them? What should you build internally, and what’s worth buying? We’re profiling engineers, product managers, and designers who build world class internal tools at their companies to get some answers.
Jeremy Edberg has been working as an engineer in the Bay Area since 1998. He’s worked on great internal tools at eBay, as the first engineering hire at Reddit, and as the first reliability engineer at Netflix.
I started my post-college career at eBay / Paypal, and I worked there for about 4 years, the first 3 on security tools for fraud detection (identifying things like phishing websites) and remediation. My last year there, I worked on “Area 51” – the lab where we certified hardware and software for use in our backend systems.
After eBay I joined as the first DevOps engineer at Reddit, before that was a thing, and at least 50% of our time there was spent on internal tooling: mostly focused on identifying and preventing spam and cheating. From there I went to Netflix as their first reliability engineer, and built a whole bunch of tools related to monitoring and reliability.
Basically, I’ve worked mostly on internal tools my whole career. Things weren’t all business though – when we were at Reddit, we had a fish tank in the office, and we needed a way to feed the fish when we weren’t there; so we built a script that would eject the CD drive of a server and activate the fish feeder.
It depends on what the company is using. At eBay, we used PHP because that’s what the tech lead wanted to use. When I got to Reddit, everything was in Python, so I learned Python and ended up writing everything in Python. At Netflix, I was the first person in our group so I got to choose the language that we built our tools in – so I picked Python, and then hired people for whom that was their preferred language.
Internal tools can be lower quality, but otherwise you start the same way. You talk to your customers, who in the case of internal tools happen to work at your company, gather their requirements, design and build to satisfy their needs, iterate, etc. The whole process is the same – it’s just easier because it’s internal.
Ideally, you’d build these tools with the same quality as user facing features. In practice though, you need to cut corners. And because your customers are internal, you won’t have a PR disaster if things break.
It depends on the tool. At Netflix, the metric was: if more people use our tool than work around it, we’ve succeeded. At Reddit, we were so small that it was even simpler: are we using this or not. At eBay, it was: are we showing value? We were able to categorize in dollar terms how much money we were saving the company. When we could show that we were saving many times our salaries, we didn’t need to justify our existence.
Internal tools are important because they get the work of the business done. Product is probably more important, but you need these internal tools to get the product done. And that’s why these tools are often viewed as second class, because they don’t drive revenue directly, kind of like IT. But much like IT, if you didn’t have it, you wouldn’t be able to do the work you do to drive revenue.
I’ve been lucky to work at organizations that value great internal tooling. At Reddit, the same engineers were working on internal tools as the external product, so we were setting our own priorities. At Netflix, there was big buy in on these internal tools, so it was not hard to get resources for them; and it was the same thing at eBay. Everyone appreciated the value of what we were doing.
The biggest frustration is that internal tools tend to be bespoke, so it’s hard to find good examples of how other people solve these problems. It’s rare to find a best practice on building a type of tool, even if you’re able to get small nuggets here and there.
A really good example is Netflix: their open source site is full of stuff written for Netflix, and worked beautifully at Netflix. But unless you’re using everything that Netflix does, it’s not going to do a lot for you.
Everywhere that I worked, it was build or buy. At Netflix, it was very standard to say:
- Is there a SaaS product that can get us most of the way there,
- How much would it cost us to pay for the SaaS product, integrate the SaaS product, and maintain that integration,
- vs. how much would it cost us to build and maintain our own bespoke tool
People often forget to include the cost of maintaining an internal tool when doing the buy vs. build calculation.
We would always make this calculation, but our default was SaaS. If there was a SaaS tool that made sense, we’d use it. We weren’t in the business of building whatever it is they were building; we were in the business of delivering movies on the internet. But a lot of times there was no SaaS product that would meet our needs (scale, integration, etc.) so we ended up building a lot of it ourselves.
Generally, you don’t get to see other companies’ internal toolsets! You hear about the stuff that’s open source, but what about all of the stuff that Facebook and Google don’t put out? To an extent, I’d put Google in here: we had a bunch of people who came from there and would usually say something like “oh man, we had a great tool at Google that did this for us.”
Treat internal tools like you’d treat external tools: talk to your users, create an MVP, iterate, and find a metric for success. And find a place where there’s a gap; where having something automated would be super beneficial, and then automate that. Find a place where automation would make a 10x improvement.
Reader
