Everyone wants them, many claim to have used them with success, few in reality have done it right and a minuscule percentage understand the trade offs involved. They’re all the rage these days - if you don’t have a 'Microservices architecture' then someone higher on the totem pole is bound to pontificate that the approach is outdated/legacy. :)
Of late, I’ve come across customers & teams where there’s a rush to 'do microservices' and I’ve spent a good amount of time laying out the trade offs and helping them evaluate their own readiness (or lack of it).
In Greek mythology, the Sirens (Greek singular: Σειρήν Seirēn; Greek plural: Σειρῆνες Seirēnes) were dangerous creatures, who lured nearby sailors with their enchanting music and voices to shipwreck on the rocky coast of their island.
Why Microservices?
Some of the promised benefits of using microservices architecture:
-
Be agile and respond to business needs quickly.
-
Scale out each service of the whole individually
-
Clear and explicit boundaries of subsystems - helps keep things cohesive and devolving into a Big Ball of Mud over time
-
Scale out from a human standpoint - it’s easier to bring on new people if they have to understand a smaller part of the whole at a time than the entire 800 pound gorilla.
-
Avoid technology/platform lock in.
-
Avoid having to make either/or technology decisions in the beginning when you/your team may not understand the problem well enough yet.
What do you lose?
Microservices are distributed systems. And distributed systems are complex and hard. Period. You have to solve (or be prepared to deal with) distributed systems' problems. Compared to a monolith, you lose a lot of simplicity.
-
Deployment is a lot more complex - more moving parts all around and the failure to deploy a single service properly could lead to your qa (or any dependent teams) twiddling thumbs waiting for the system to be available.
-
Development is more complex - esp when you have a change that isn’t actually limited to one or two services
-
NO ACID - no transactions (in the traditional sense) and building systems that compensate for failure is not the same as a saying BEGIN TRANSACTION/COMMIT TRANSACTION
-
Understanding the system is more complex - since locally observing a single service doesn’t convey the full picture.
-
Debugging is harder since all you’re going to see is a bunch of api calls at a service level. To understand the entire big picture you need to build/instrument good tracing.
-
It’s harder to understand the performance profile of your application as a whole. A single service’s minor failure/increased latency could have a snowball effect upstream. Gaining a clear understanding of this and solving these problems isn’t going to happen without building good instrumentation in your services.
-
Testing the system is harder - While you can test individual api calls, you need to be able to orchestrate entire flows across the system. Doing this in a distributed environnment is harder.
You need to be this tall…
I’d almost say that you should start applying microservices when you have an impeccable monolith (or, at least one that’s well understood). If you don’t think a team has the chops to implement a good monolith, then they sure as hell won’t be able to build a MSA system. In fact, it’s much more likely that soon they’ll be chasing their own tails trying to find out what happened to a user request as service calls trigger other service calls.
Hmm.. so what do you need to be reasonably sure that going the microservices way gives you some of the promised benefits? Here’s a starter:
-
Proven devops skills - your team does automated nightlies with unit and integration tests already? Good - you’re going to be setting up a lot more of those.
-
Can apply DDD - You/your team understands the problem space well enough. Well, the picture’s never going to be full or complete but you need to have a good core team (5-10%?) who understand and can apply domain driven design practices to break the problem into services that make sense.
-
Understand distributed systems challenges - your team has enough folks who understand distributed systems' challenges. If you have to explain at-least-once vs. exactly once delivery semantics to more than a handful of your core team, then kiss microservices goodbye while you still can.
And what happens in the real world?
If I hadn’t seen it first-hand time and again, I’d have thought it was an exaggeration. However, the number of times it hits home is amazing.
Stuff I’ve seen done in the name of microservices:
-
Using microservices as an excuse to not design the system at all.
-
No strategic view - architecture/design in sprints in the name of agile! :)
-
Coupling all microservices with a single database
-
Deploying services by hand/no CI/CD in place.
-
Zero monitoring
-
No log aggregation
-
No ability to trace calls through the system
-
No tests - either at a service level (forget system tests)
-
Building everything as synchronous service calls.
Way too many other horror stories to recount :)
So stay away from microservices?
Now, you might be thinking that I’m all glum on microservices and in essence, my rant amounts to "stay away from microservices" - but believe me, I’m actually a microservices believer! And the reason is that we shipped and operated services at scale for more than 2 years. Nothing beats real world experience, I say.
When we built Scheduler and Brewmaster, they were actually really well structured services and we were 'doing' microservices - just that no one cared. We cared a lot more about how our REST API footprint looked like, we cared about how our domain was structured but we didn’t care if we were 'doing' microservices.
As an engineering team, we focused on the domain first, then we made sure that we designed the subsystems to be loosely coupled so that they could all be evolved individually. We wrote unit tests along with our code and then integration and system tests. We had a good CI and CD pipeline on day 1 when setting up the solution structure.
In fact, our first release was a monolith. We just didn’t have reasons to split services enough at the outset without some sort of market validation.
More importantly, we built the team with good engineers who were willing to learn - every one was interviewed by everyone else on the team, you had to pass a hands-on coding test and anyone could review anyone else’s code. Even then, when someone didn’t work out on the team, we were quick to take corrective action and part ways amicably. We were our own Dev team and Ops team and everyone was responsible for their code in production. Engineering culture’s important!
As proof that microservices really worked for us, here’s how quickly we evolved:
-
Public beta launch of the first service - Scheduler in 6 weeks.
-
Feedback and validation
-
Azure marketplace launch
-
Refactor User and Tenant into their own services - 4 weeks
-
Payments integration - 4 weeks
-
Conceptualize the second service - IaaS automation on Azure - Brewmaster.
-
Brewmaster Beta in 8 weeks
-
Template SDK in 6 weeks supporting user uploadable templates.
-
Redo Payment provider since the 3rd party provider we were using jacked up their processing fess - 4 weeks.
At our peak we were releasing to production more than 5 times everyday.
Summary
This post has already grown far longer than what I’d initially thought. The hardest thing is breaking down a system into components with well-designed boundaries. That is hard to do and is something you have to do irrespective of whether you choose microservices or not.
Arguably, MSA architecture’s biggest gain is that you have small, independent teams that can evolve and release each service independently. However, this is extremely dependent on having clear boundaries of responsibility between the services. If you end up having a bunch of dependent services, then you’re going to be much slower than with a monolith as your teams are all interdependent and deadlock on each other.
Microservices are harder to debug, trace, monitor and otherwise reason about the system behavior and it comes with it’s fair share of work to enable those. So as a first step it’s much more prudent, in my opinion, to follow DDD and design a well structured monolith. Once you grow beyond a monolith (team size, customers, revenue etc., you can tease apart services and have a much more realistic chance of benefiting from microservices.
