Before Observability: Start With the Business, Not the Tools
Observability works in your favour when it starts from what the business needs, and costs you dearly when it starts from the tools. Before you collect a single signal, answer a few business questions first.
Everyone wants observability to fix it
Every observability programme starts with a room full of people who each want something different from it:
Leadership wants fewer outages.
Operations wants less noise.
Engineering wants faster fixes.
Finance wants lower tool spend.
The business wants to see when customers are being hurt.
These are fair expectations, and observability can help with every one of them. The risk is the rush to get there.
Rush in, and it gets costly. Switch everything on and collect everything, and you pay to ingest and store data nobody uses. The alert volume buries the few signals that matter. A year later the team is re-platforming and trying to win back the confidence it lost.
Get it right, and it works in your favour. Spend follows business value. Alerts mean something when they fire. Answers come faster, and cost comes down over time instead of climbing with every new data source.
The tools and the budget can be identical in both cases. The difference is where you start.
We became experts at collecting evidence, not at understanding it
Over the last few decades the industry got extraordinarily good at collecting evidence. We can instrument almost anything, store petabytes and query billions of data points in milliseconds. Yet when something breaks at 3 AM, we still rely on a person staring at dashboards to work out what is actually happening.
That gap between evidence and understanding is the expensive part. It comes from the order in which most programmes are built.
| Bottom-up: the usual way | Top-down: what works | | --- | --- | | 1. Collect metrics, logs and traces | 1. Agree what the business must achieve | | 2. Build dashboards for every component | 2. Name the services that deliver it | | 3. Alert on technical thresholds | 3. Define what “working” means | | 4. Ask what it means for the business | 4. Only then choose the signals |
Bottom-up leaves the most important question until last, after the money has been spent. Top-down asks it first, so every signal collected afterwards has a reason to exist.
A real example: 120 minutes to 3 minutes
Mehdi Daoudi of Catchpoint describes an enterprise customer that took around 120 minutes, on average, to detect production issues. They were not short of telemetry or dashboards. What they lacked was a way to turn signals into a recognised state that demanded action.
Detection dropped to about 3 minutes once they validated a known journey against an expected outcome: a known journey, tested from a known vantage point, at a known frequency.
It wasn’t a telemetry gap. Nothing told the signals which outcome they were there to protect.
The important detail is what came first. Someone had to know which journey mattered and what a good outcome looked like. Those are business questions, not tooling questions.
Six questions to answer before any telemetry
These six questions come before any conversation about tools. Each one narrows the next, from what the business wants down to who is allowed to act.
What is the business trying to achieve? Answer in the business’s own terms: revenue, reputation, customer trust, regulatory standing. If the answer is “99.9% uptime”, keep asking why.
Which business services deliver it? Checkout, onboarding, claims, payments, booking. These are the things customers and the board would notice if they stopped.
What does “working” mean to the customer? State it as an outcome, not a server metric: an order placed in under a minute, a claim paid in five days, a motor service booked for 3pm. This becomes the expected outcome you test against.
Which journeys and processes make it happen? Map the steps a request takes, and who owns each step. Ownership matters as much as the path, because someone has to act when a step fails.
What does each step depend on? Applications, infrastructure, third parties, the network and the internet. This is where the dependency map, and later the telemetry, comes from.
Who acts, and what are they allowed to do? Capability, authority and guardrails decide which actions are safe.
The last question matters more every year. As actions move to automation and AI agents, an action is only useful if the actor has both the capability and the authority to take it safely. A runbook that nobody is permitted to run is not a plan.
Where observability fits
Observability is the bridge between the business and its technology, not the destination. It helps to picture the business as a pyramid.
At the top sits the business itself, judged on revenue and reputation. It delivers through business services. Those services run as journeys and processes, which travel through applications, infrastructure, the network, the internet and third parties.
Two things flow through that pyramid in opposite directions:
Intent flows down. Business outcomes decide which services, journeys and dependencies actually matter. The six questions are how you capture that intent.
Evidence flows up. Observability gathers signals from the lower layers and should carry them back up as an answer the top can act on.
Skip the top-down part and observability has nothing to aim at. You end up with plenty of evidence about the bottom of the pyramid and very little about the top.
Where to focus effort and spend
Spend most in the top left and least in the bottom right. Draw the whole business as one box and two axes appear.
Across the box, services and processes run from most critical on the left to least critical on the right:
Customer and revenue: checkout and payments, search and account, the marketing site.
Core operations: order fulfilment, billing and CRM, reporting.
Internal support: identity, payroll and HR, email and collaboration, the intranet.
Non-production: pre-production, staging, shared test, dev and sandboxes.
Down the box run the service tiers, each with its own support expectation:
| Tier | Support expectation | | --- | --- | | Platinum | 24/7, response in minutes | | Gold | 24/7, response in hours | | Silver | Business hours | | Bronze | Best effort |
The top-left corner holds platinum, production, customer-facing services. That is where the deepest observability belongs: full tracing, longer retention, tighter objectives, the fastest response and the most engineering time. Move right or down and the investment tapers off. In the bottom-right corner, dev and sandboxes, basic monitoring on a best-effort basis is enough.
The same map shapes cost allocation. The observability budget should be weighted heavily towards the top left, not spread evenly across every host and environment. It is also how to defend the bill: if most of the spend sits in the bottom right, it is going to the wrong place.
The services and their placement here are examples. Every organisation will place its own services and set its own split; the point is that coverage and cost follow the map.
Where the tiers come from
Knowing which services are platinum and which are bronze typically comes from a CMDB. Use it where it helps, but do not make your observability deployment dependent on it. A complete, accurate CMDB is a long project, and waiting for one stalls everything else.
For this map you only need to see the services and their tier, not every asset related to them. A short list of business services, each with a tier and an owner, is enough to start tagging and prioritising. The asset detail can follow later, and observability itself will often discover it for you.
Going live: let tags decide the lane
Route every alert on its tags: platinum production goes in the fast lane, everything else in the slow lane. The moment you start onboarding, alerts arrive from everywhere: infrastructure and cloud, applications and APM, the network, logs and security, synthetics, third parties and SaaS.
Treat every one of them the same and the event management platform fills up. Costs climb with volume, and the alerts that matter get lost in the noise. The answer is context, and the cheapest way to carry context is tags.
Every event should carry a small, agreed set of tags that the platform validates on arrival, for example:
tier = platinum
env = production
service = business
lob = digital
If all of them are true, the event goes in the fast lane. Anything else goes in the slow lane.
| Lane | Fast lane: platinum and production | Slow lane: everything else | | --- | --- | --- | | Share | A small share of the volume, most of the attention | Most of the volume, a fraction of the cost | | Handling | Correlate and enrich with business context | Deduplicate, batch and suppress | | Response | Page on-call, open an incident | Daily digest or a ticket, no paging | | Data | Full fidelity, longer retention | Sampled, shorter retention, cheaper tier |
An event that arrives with no tags goes in the slow lane too, and raises a ticket to fix its tagging. That way the gap closes over time instead of quietly growing.
This is the focus matrix put into practice at the point where events arrive. Routing on tags keeps event-management volume, licence and on-call cost in line with business value.
Observability is one rung of the ladder
Most observability tooling stops three rungs short of where the business needs to get to. William Louth of Humainary, building on Mica Endsley’s situation awareness model from aviation, describes a chain that runs from signal to action:
| Rung | What it is | Who usually does it today | | --- | --- | --- | | Signal | A raw measurement | Tooling | | Sign | A judgement drawn from a pattern | Tooling | | Status | The state of one thing | Tooling | | Situation | What it means for my goal | A person, in their head | | Projection | Where it goes next | A person, in their head | | Affordance | What I can safely do | A person, in their head | | Action | The intervention | A person |
Each rung reduces uncertainty about what matters and moves closer to a purposeful intervention. The chain applies equally to a thermostat, an autoscaler or a human operator. Tooling is very good at the first three rungs, then hands over to a person who assembles the situation, projects what happens next and decides what is safe to do.
A metric is not a situation
Here is the same incident described four ways:
Signal: “Latency exceeded 200 ms.”
Sign: “Saturated.”
Status: “The queue is saturated.”
Situation: “Checkout capacity is exhausted during a product launch.”
Only the last line tells you what to do. It names a business service, a business moment and, by implication, what is at stake.
None of that can be derived from the telemetry alone. You have to know that checkout is a critical service, that a launch is happening, and what good looks like. A situation is always relative to someone’s goal, and that goal comes from the six questions. A system with rich telemetry but a poor vocabulary of situations will misclassify, confidently.
Questions first, telemetry second
Work in this order: outcome → service → journey → dependency → signal. Start with the outcome the business cares about, then the services that deliver it, the journeys that make those services happen, and what each journey depends on. Only then choose the signals.
In practice that means:
Answer the six questions before you buy, onboard or instrument anything.
Place every service on the criticality and tier map, and weight effort and budget towards the top left.
Tag everything at source, and let the tags decide which lane an alert takes.
Aim the platform at situations, not just statuses, so it climbs further up the ladder.
If you can’t say which decision a signal informs, you’re not ready to collect it.
Observe. Automate. Operate with confidence. If you want to know where your organisation stands today, take the free Observability maturity assessment at visibilityplatforms.com.
Sources
William Louth, Observability is a rung, Humainary
Mehdi Daoudi, From Telemetry to Situation: Why Observability Needs a New Unit, LinkedIn
Comments