A Lever Long Enough...
We cannot solve our problems with the same thinking we used when we created them. So what are our options?
“ Give me a lever long enough and a fulcrum on which to place it, and I shall move the world.
Archimedes
Every now and then I come across a paper that is both brilliant and effortless to absorb. One such rare pieces is Donella Meadows' Leverage Points - Places to Intervene in a system. As we build systems, we keep looking for places to improve existing infrastructure whether to drastically improve our reactance to anomalous load patterns or to tighten the range of operations for controlling costs. Yet the sheer plethora of knobs forces us to collapse into a state of decision-fatigue. Overwhelmed, we tend to look outwards for a one-size-fits-all model to magically whisk our operational complexity away. And sure, there is real merit in leaning on battle-tested architectures. But it certainly helps to have a mental model of the way our tools can be ranked in order for us to make decisions that are both more informed and more attuned to the domain we’re working in.
Leverage points in a system are exactly what we're thinking - points in the system where small changes can cascade into an avalanche of consequences in the overall state of the system.
The Search for our Hercules
Just as every lore has that hero who swoops in to rescue humanity from untold misery, our own search space spans twelve tiers of increasingly potent interventions(couldn't escape that Hercules reference 😜).
- Constants, parameters and numbers
- Size of buffers relative to flows
- Structure of material stocks and flows
- Length of delays relative to the rate of change
- Strength of negative feedback loops relative to the impacts they are correcting against
- The gain around driving positive feedback loops
- Structure of information flows
- The rules of the system
- The power to add, change, evolve or self-organize system structure
- The goals of the system
- The mindset or paradigm out of which the system arises
- The power to transcend paradigms
As the paper describes, this is no "sure-fire" recipe to finding these leverage point but a good starting point to begin searching. I've tried compiling these directives into what it might mean for designing computing systems.
Params and constants
These lie at the very bottom of our list. And yet these consume a bulk of our mindspace in terms of tweaking system parameters.
Size of Buffers Relative to the Flows
Ingress Buffer | ⬜ ⬜ ⬜ ⬜ ⬜ ⬜ ⬛ ⬛ ⬛ |
Egress Buffer | ⬜ ⬜ ⬜ ⬜ ⬛ ⬛ ⬛ ⬛ ⬛ |
Connection Pool | 🟥 🟥 🟥 🟥 🟥 🟥 🟥 🟥 🟥 |
Structure of material stocks and flows
Observe the path linking various system components. What is on the critical path? Is there a fast path for serving requests? Are our services loaded equitably? What about our availability zones?
Length of delays relative to the rate of change
Strength of negative feedback loops relative to the impacts they are trying to correct against
We have a feedback loop that scales our deployment when a one-minute latency metric breaches a configured threshold. Now consider a highly bursty workload. Requests arrive in sharp spikes, but Prometheus only observes the service at scrape intervals and KEDA evaluates those metrics periodically. As a result, our system won't respond quickly enough to the load characteristic lagging behind in such cases (not to mention the length of delays while starting a pod).
The gain around driving positive feedback loops
The above illustration is pretty instructive. Our users abandon their requests after waiting for 1 seconds and retry. This leads to a retry storm and ultimately our problem compounds as more users join the party. A similar reason for expecting ALL flows in computer networks to be TCP "friendly" (otherwise we're penalizing TCP flows for being benevolent).
The structure of information flows
Lets take an application developer. They build an application, define the Helm chart used to deploy the Kubernetes resources it needs and create a PR.
What might the infra team do in order to make the developer aware of the impacts of the service they're creating on the cluster they'll deploy on? Perhaps they add a GitHub workflow that, alongside the usual tests, reports the CPU, memory, and disk being requested and estimates the additional infrastructure cost. The infrastructure team itself might use a tool such as Infracost to expose the expected cost of infrastructure changes before they're provisioned.
Nothing here prevents either team from making the change. The consequences of a decision are surfaced directly to the person making it, at the moment they are making it. Making consequences impossible to ignore can change behavior without imposing a single new rule.
The rules of the system
These are the hard boundaries. Let's say an application developer is trying to deploy an application without resource requests/limits or requests resources that are beyond the quota of the namespace, they'll be blocked from doing so. No pod may run with escalated security privileges and inter-pod communication can be clamped with the aid of network policies.
The power to add, change, evolve or self-organize system structure
Most human-built systems are fallible. We build for an operating range beyond which the rules and assumptions we based our system on need to change. Systems that remain rigid won't stand the test of time. On the other hand, if there are no rules to abide by, we'll land up with an extremely unmaintainable system.
The interesting question, then, is whether we can give a system the ability to evolve within useful constraints.
Lets take the case of our application developer again. Assume there are a group of applications that need access to an S3 bucket. A naive way for configuring such access would be for the developers to ask the platform engineers to allow an IAM role with appropriate accesses to be assumed by a k8s service account. But thats going to create a flurry of requests for the platform team. Rather let the platform team define a crossplane composition (aka XRD) that auto-creates the IAM role and service account with the appropriate annotation. The application need only instantiate an XR and boom, their application has access.
The platform team hasn't surrendered control. It has changed where the power to create structure resides. Rather than manually modifying the system for every new application, it has created a mechanism through which application teams can safely cause the system to evolve.
The goals of the system
“ When a measure becomes a target, it ceases to be a good measure.
- Goodhart's Law
This is the objective underpinning all the activity that goes into building the system - the raison d'être.
Take the case of a CI/CD pipeline design. Suppose the goal is "Deploy as quickly as possible". We would start tracking metrics like deployments per developer per week and optimizing build times, pipeline parallelism, reducing gates and slow checks and so on.
Now lets say we slightly tweak this objective to say "Get changes safely into our user's hands as quickly as possible". Notice that even though at a cursory glance the previous objective might appear to have the same effect the metrics we would start measuring would drastically change. Because a change that is escalated to prod rapidly but causes a service outage isn't valuable anymore.
Remember that the metrics we track aren't equivalent to the goal. They're a path towards achieving that goal (till they aren't and a new measure is needed). For example, a stable user experience might lead us to an SLO of 99.9% availability. But a service that hits this but fails on latency/usefulness isn't valuable.
The mindset or paradigm out of which the system arises
What do we believe is true that led us to design the system in this way in the first place?
Lets take the CI/CD pipeline for example. The stated goal is "Get changes safely into our user's hands as quickly as possible". The paradigm that might produce such a goal could be "Small, frequent changes are safer than large, infrequent ones" or the continuous delivery paradigm. We don't wait to accumulate a certain critical mass of changes before release but rather continuously release, get feedback and adapt.
My point is, this literally guides our goals which in turn informs what we measure.
The power to transcend paradigms
Even the very paradigm that produced the system must be questioned from time to time to evaluate its suitability.
For example, does our belief that small, frequent changes are safer apply equally to a critical substrate of our application where stability and an extremely low rate of change may be more valuable?
As engineers, we quickly want to formalize best practices and standardize everything. But this might lead to unwarranted dogmas and the all-too-familiar "This is how we've always done things" moment. The original context fades and the paradigm remains. And that is how we become cultists from technologists.
Conclusion
What we just did was a quick skim of the paper from my POV of the things I've faced at work 😅. At low leverage points, we tune the knobs exposed by the system while at high leverage points we question the very reasons that led to the system in the first place. And between this we ensure that our systems remain functional and useful. Till next time....
