TL;DR – You should purchase and use external autonomous reliability software if you:
- Don’t have (maybe want) to develop the expertise to create and more importantly maintain a system;
- Have a large or growing number of incidents, or complexity in your production environment;
- Need to manage your costs so they are reliable and predictable.
Should I build or buy autonomous reliability software?
Introduction
Everyone knows production management is hard, and the hardest of the hard is incident management. Dealing with a complicated, critical service when it’s gone down is like being stuck in a collapsed mineshaft – everyone’s screaming in the dark, no-one knows what’s really going on, and even figuring out what’s happened is a major achievement. After a few of these situations, it’s natural to start to wonder if doing production management really has to involve so much darkness and screaming.
If you’ve decided to try to improve things, there’s a large number of ways to do that, but what actually happens usually depends on who’s in charge of production and what their background is. There are folks who naturally turn to fixing processes: they pick up a copy of the SRE book(s), look at incident management protocols, or institute professionalisation to fix things previously fixed by a combination of luck and “everybody knows only Sam can do this.” (Hopefully Sam already knows this.)
But these days, an increasingly popular choice for helping you manage production is investing in software. It can be used in a thousand ways to help your production environment stabilize: as of the time of writing, we have everything from artisanal scripts living in someone’s home directory through to extremely sophisticated monitoring systems feeding ML models combing through terabytes of data for anything that seems anomalous.
Right now, lots of people are starting to look at getting software support for one key thing that SREs do – production investigations, or figuring out why something is happening and what to do about it. Obviously this happens a lot in incidents, and is very important in that context, but it happens outside that too; many’s the time in my career where I’ve spent a long while debugging an apparently isolated small weird behaviour, to discover that it means something has been misconfigured literally for years.
The great news is that today’s software systems are actually capable of helping with this. They don’t get it entirely right all the time, of course, but they usually do enough of the ground work of investigation for you that you can get rid of a lot of messing about with only a small risk of going off the rails.
As a result, it’s increasingly practical to use these emerging and sometimes jaw-droppingly accurate technologies, though one of the main decisions you need to make is extremely traditional: i.e., build-versus-buy.
Compare and Contrast
Build-versus-buy – which is really build-and-operate-versus-buy – is a question lots of companies ask themselves all the time, and the set of things you have to think about is pretty well understood. Let’s look at the main concerns now.
Is building this core to my business?
Not every company is a giant multinational, needing or wanting to create their own version of everything they touch in their production flow. Business success often flows from focusing on the things that actually matter to your business model, rather than keeping a side project going for years. Answer: Could be both, but most likely in favour of outsourcing.
Will we build it successfully?
Though AI-inflected software engineering has made it easier than ever to get code written, there are still many problems with doing that well. There is also a growing realisation that we have pushed the difficulty of making good software towards the categories of deciding what software to write, how to test it, and how to make sure it doesn’t misbehave, rather than actually constructing the software. It turns out that it is harder to do this than you might think, and if you don’t believe us, look at Orca-bench, a paper by professionals in the field showing that it’s very hard to get otherwise untuned models to give reliable results. Why would you spend your time on this rather than your actual business? The counter argument is that very large companies, with a substantial production estate, highly custom tech stacks, and skilled staffing to match might well be able to do this, but we expect for everyone else, it will be peripheral. Answer: Could be both, but most likely in favour of outsourcing.
How much will it cost?
As with most things in the world of software, it could be quite complicated, and there’s a large amount of detail to understanding it fully, but the basics are reasonably straightforward.
The classic trade-off is that going with a commercial provider gives you relatively bounded costs, and going in-house gives you the opposite. To be clear, it can be cheaper in-house – particularly if the salaries and token resources are being paid in some other way – but it can also be a lot more expensive, particularly for maintenance. The worst outcome is thinking you can do it cheaply yourself, and discovering, expensively, that you can’t.
There’s a couple of dimensions to this.
The simplest dimension is of course the cost of construction, but as per the start of this section, we shouldn’t forget the cost of maintenance: ultimately, maintaining something non-core, even with AI support, is going to cost something that you could have spent on your main thing, whether it’s salary or tokens. For example, if an engineer paid $250k a year spends a month to build it and 10-15% of their time on maintaining it, that’s roughly $50k without adding product, project, or other management costs, never mind the unanticipated surprises that never happen in software development (coughs gently).
Another dimension is the running costs: though token economics have very much their own dynamics, vendors are generally hugely incentivized to be efficient and provide value for that spend. It’s certainly possible for you to go out and source tokens at arbitrary prices, but it’s a very different question as to how relevant and useful that token spend is. Vendors learn things across multiple customers and every customer benefits as a result: responsible vendors work hard on making sure every customer benefits from every other customer’s experiences.
A yet further dimension is the overall cost of keeping up with advances in AI. Anyone touching the space even a little bit knows how fast it’s moving, and how having to master today’s hot topic guarantees precisely nothing about your system ever “being done”, yet each trend is definitional at the time. RAG, context engineering, loop engineering, and whatever will be in the future have all had their moment and the only thing we know for sure is that there will be more to come. Keeping up with advances is a full-time job and it is easy to fall behind. Finally, though it’s a little more indirect that the above, in-house projects are often overfitted to the in-house situation (an understandable situation and outcome), and as a result generalises to the future less cheaply, perhaps discontinuously so if there is a substantial change in the production environment.
Answer: in the best possible case, it can be cheaper to do internally, but outsourcing gives you useful cost certainty.
What are the risks on both sides?
In general, the risks when you build in-house are around a lack of execution, and an inaccurate understanding of costs. With the best will in the world, we often see execution – particularly on the second 90% of the work required for any software system – suffer for non-core systems. (Especially risky if the development priorities change at short notice.) We’ve already covered cost control above.
The risks when you outsource are typically either a product misfit, or business continuity risks on the provider side. That is, the provider could run out of money, have a significant technical problem or be unreliable, get sued out of existence, etc. For a product misfit, the product might not end up doing the things you want, or the provider might take the tool in a direction you don’t agree with.
A full accounting for all these risks is tricky, but ultimately you are balancing the (assumed) increased control you have over things in your local scope, versus external risks in the provider, coupled with their obvious motivation to make the core product a success. Answer: could go either way, depending very much on the client situation, though outsourcing probably edges it given the wide availability of similar products.