When You Can't Instrument the Black Box: Monitoring for COTS-Only Estates

Most monitoring guidance assumes you own the application code.

Most monitoring guidance assumes you own the application code. You can add instrumentation, adopt OpenTelemetry and trace a request from the user interface through to the database. For organisations whose applications are entirely commercial off-the-shelf (COTS), that assumption can be wrong from the outset.

ERP, CRM, HR, document management, claims and payment systems may be closed products. Their vendors can restrict access to internal components and require customers to use embedded monitoring tools. You cannot recompile the application, add tracing wherever you choose or necessarily inspect its database.

That does not mean the service has to be a black box. It means the monitoring approach has to start somewhere different: with what the business needs the application to do, and with the signals available around it.

Accept the boundary, then monitor what crosses it

A closed application has real limits. Teams may not be able to see its internal code paths, private components, embedded queues, caching behaviour or vendor-managed dependency logic. The vendor’s own monitoring may offer useful application health information, but it cannot always provide the same depth or access as instrumentation in software you control.

It is important to be clear about those limits. Otherwise, teams risk buying an observability platform or commissioning a full-stack implementation that cannot collect the data they expect.

But the boundary of an application still exposes valuable evidence. Depending on the product and contractual permissions, teams can monitor:

• Hosts and infrastructure: utilisation, availability and resource pressure on cloud VMs, Kubernetes nodes and other infrastructure supporting the application.

• The vendor’s monitoring: dashboards, APIs, status feeds and event exports, where available.

• Network behaviour: connectivity, DNS, flows, packet data and the path between users, services and the application.

• User experience: browser-based synthetic checks, digital experience monitoring and real user signals where the product permits them.

• Integration points: middleware, API gateways, identity services, message brokers and other dependencies.

• Accessible logs and database activity: when the vendor exposes them and the licence permits their use.

The aim is not to claim visibility into internals that remain private. It is to build the fullest possible picture of the application’s perimeter: what users can reach, which interactions succeed, what dependencies are involved, and how the service behaves over time.

That perimeter matters in modern estates, too. A COTS product may run on public cloud services, containers or Kubernetes; it may depend on serverless workloads and managed platform services. The application’s internals may belong to a vendor, but the infrastructure and dependencies around it still generate telemetry, change and operational signals. Monitoring needs to connect those signals to the service without implying that every layer is equally accessible.

Start with the work the application enables

The most important monitoring questions in a COTS estate are often not about code. They are about business meaning.

Does payroll complete on time? Can a claims handler submit and retrieve a case? Can a customer make a payment? Does the system support the month-end close within the required window? A green status indicator alone cannot answer those questions.

To define what “good” looks like, monitoring teams need structured conversations with the people who use and depend on the application. Business users can explain which transactions matter, where delays become harmful, how workarounds affect operations and what the cost of failure is. They can also identify rhythms that infrastructure charts may miss: payroll runs, quarterly peaks, claims surges, term starts and other periods when demand changes sharply.

This work takes time. It is not a shortcut around technical monitoring; it is how technical signals acquire context.

Together, business and technology teams can map critical journeys and then connect each one to the signals that can test or explain it. A journey might include sign-in through an identity provider, a search or transaction in the COTS product, an API call to another service and a downstream update. Some steps may be visible only through a synthetic check; others may have logs, network data or vendor events.

The distinction between monitoring vs observability is useful here. Monitoring can tell us whether a defined condition or journey is healthy. Observability aims to help teams understand system behaviour from available outputs. In a COTS-only estate, those outputs are constrained, so business-process knowledge becomes an essential part of the model.

Make synthetic and network evidence first-class

When internal instrumentation is unavailable, synthetic monitoring is not a nice-to-have. It can test whether a representative user can complete a meaningful interaction, from a relevant location and at a useful interval. A check might validate sign-in, search, document access or a non-destructive transaction flow.

Design these checks carefully. Use test accounts with appropriate access, protect credentials, avoid actions that create real business consequences, and agree suitable intervals and maintenance windows. A single probe failure may be transient; repeated failure across locations can point to a wider service issue. Alert on business-relevant failure, not every noisy measurement.

Network evidence adds a different kind of independent view. DNS, connection time, route and flow data can help distinguish a vendor-side problem from an identity, local network, cloud or connectivity issue. That matters during an incident. Instead of exchanging assumptions (“your system is slow” versus “our network is slow”) teams can compare evidence from both sides of the application boundary.

The vendor’s built-in monitoring remains valuable, but it reflects the data the vendor chooses to expose and the product scope it covers. Independent checks verify the experience from the user’s side and can give incident teams a shared factual starting point. This is a governance benefit as well as a technical one: it can help shorten disputes and make vendor conversations more precise.

Where logs, APIs or database telemetry are available, teams should bring them into the same operational view when practical. An observability platform can only correlate signals it receives, so clarify how vendor data can be exported, how frequently it updates, and whether licensing or technical limits apply. A telemetry pipeline may help normalise and route those feeds, while event-management capabilities can help correlate vendor events with external monitoring signals.

Treat access as a design and procurement decision

COTS monitoring should be considered before a product is purchased or renewed, not after an incident exposes a blind spot. Ask vendors what the licence actually allows: agent deployment, API access, log export, SNMP, status feeds, database visibility, synthetic testing and use of data in third-party tools.

These are operational requirements. Monitoring access belongs in procurement criteria, alongside availability commitments and support arrangements. A product with limited telemetry may still be the right choice, but the organisation should understand the consequence and plan for independent evidence around it.

Use the vendor’s own dashboards where they are useful. Then assess whether their events and measures can be exported or integrated into a central platform, where they can be considered alongside infrastructure, network and user-experience signals. Maintain change awareness as well. A vendor-controlled upgrade is still a change to the service: release timing, configuration updates and platform migrations may alter behaviour even when the customer does not control the deployment.

For decision support, our Observability Periodic Table can help compare capabilities across areas such as network monitoring, digital experience, synthetics, logs and cloud. The Telemetry Pipeline Periodic Table is relevant when assessing how vendor exports, logs and API feeds can be ingested, processed and routed. The Event Management & AIOps Periodic Table can help evaluate how events from different tools may be correlated across a fragmented estate. The Network Observability Matrix compares features across the leading network observability vendors, which matters here because network evidence is often the most independent signal available in a COTS estate.

And if you are unsure where your organisation stands today, our free Observability Maturity Score asks 27 questions across 11 areas and shows where you sit on the path from monitoring to observing.

These tables are decision-support tools, not universal rankings. Use them to compare features, identify overlap, explore deployment and licensing considerations, and test vendor claims against your requirements. The right choice depends on the signals your products expose and the outcomes your teams need.

Monitoring depends on more than a tool

There is a lesson in this for estates of every shape, not only COTS ones. Monitoring depends on more than a tool. The instinct is to reach for a platform first and assume capability will follow. In practice, the questions come first. A platform can only amplify the understanding behind it.

The proof is in this approach itself. With a closed application, the decisive work is not which product was purchased. It is the structured effort to establish what good looks like, map the critical journeys, and identify which evidence can test each one. Any capable tool applied to those questions can reach the same place, because the questions carry the value, not the logo on the dashboard.

This is why two organisations can run the same platform and get very different outcomes. One asks which business transactions matter and how each signal connects to them. The other collects everything the tool supports and searches for meaning afterwards. The first will keep getting better; the second will keep getting noisier.

We stand by a simple ladder for framing this, articulated by William Louth at Humainary in "Observability is a Rung": a strict capability hierarchy in which each rung answers one question and yields one product that the rung above consumes. It runs from instrumentation (what can be observed?) through telemetry (what was observed?), monitoring (did a defined condition occur?) and diagnostics (what explains what we observed?) to observability proper (what state or operating regime is the system in?), and continues upward through situational awareness, decision and control (how do we influence what happens next?). Two things in that model matter to us. First, most products marketed as observability are actually delivering diagnostics. Second, observability is a means, not an end: understanding system state has no purpose unless it is converted into operational context and used for action.

Seen through that ladder, the COTS approach in this article is not a workaround; it is a natural fit. A closed product constrains the lower rungs: instrumentation and telemetry belong to the vendor. But the higher rungs, situational awareness, decision and control, belong to the organisation. The business-user conversations described above are how those upper rungs get built: they supply the significance model that raw evidence alone cannot.

The principle is not new either. Go back decades, to mainframe operations, and the approach was the same: the mechanics were opaque, but operators held a model of what the system did, watched the outputs that crossed the boundary, and judged its condition from the signals and the business rhythm around it. Monitoring was never achieved by the tool alone. It was always achieved by the questions, the model and the people using them.

None of this diminishes the importance of choosing well. Coverage, deployment models, licensing and cost all matter, and the comparison resources below exist for exactly that purpose. But capability is the multiplier, not the origin. Start with the questions, qualify the tools against them, and the platform becomes an amplifier for understanding rather than a substitute for it.

Build the picture you can use

There is no single workaround that makes a closed application transparent. The practical route is to combine business-process understanding with every signal the vendor and surrounding environment permit. Start by agreeing what success means; map the critical journeys; inventory available telemetry; and test how independently the service can be monitored from the user and network boundary.

This approach may be unconventional compared with instrumenting software you own. It is also realistic. A disciplined monitoring strategy can give teams meaningful evidence, clearer incident conversations and a stronger basis for prioritising investment, even when the application itself remains inaccessible.

At Visibility Platforms, we help organisations build monitoring for estates they cannot fully instrument. Our vendor-neutral, use-case-first approach combines structured engagement with business users, hands-on engineering and strategic roadmap guidance. We can work flexibly alongside your teams to assess what each product exposes, connect the available signals and focus monitoring on the journeys that matter.

If your estate is built around COTS applications, we can help you define what good looks like, and establish the best monitoring approach available within the real constraints of your vendors.

Observe. Automate. Operate with confidence.