Building Observable Backend Systems: The Monitoring Setup That Catches Bugs Before Users Do
Build observable backend systems with structured logging, Prometheus metrics, and OpenTelemetry distributed tracing.
The worst production incident I ever experienced was not the one that woke me up at 3 AM. It was the one that ran silently for three weeks. A memory leak in our order processing service was slowly degrading performance — response times crept up from 200ms to 800ms, then to 2 seconds. No alerts fired. No errors appeared in the logs. Customers just quietly stopped completing purchases, and we did not notice until the monthly revenue report showed a 15% drop.
That experience fundamentally changed how I think about backend systems. I stopped asking “is the server running?” and started asking “is the system behaving correctly?” That shift — from monitoring to observability — is the difference between discovering problems when users complain and discovering them before users even notice.
This guide covers the observability stack I have built and refined over several years of production backend work, including the specific tools, configurations, and hard-won lessons that made it effective.
TL;DR — The Observability Stack
| Pillar | What It Answers | Tool I Recommend | Alternative |
|---|---|---|---|
| Logs | What happened? | Grafana Loki | ELK Stack |
| Metrics | How is the system performing? | Prometheus + Grafana | Datadog |
| Traces | Where did time go? | OpenTelemetry + Jaeger | Zipkin |
| Alerts | When should I care? | Alertmanager | PagerDuty |
If you only implement one thing from this guide, make it structured logging with request IDs. It is the single highest-leverage observability improvement you can make.
The Three Pillars of Observability
Observability is not a single tool — it is a practice built on three complementary signal types. Each answers different questions, and you need all three to understand what your system is actually doing.
Pillar 1: Logs — What Happened?
Logs tell you the story of individual events. But not all logs are created equal. When I started my career, our logs looked like this:
[2026-03-15 14:23:01] INFO: Processing order
[2026-03-15 14:23:01] ERROR: Something went wrong
[2026-03-15 14:23:02] INFO: DoneThese are essentially useless for debugging production issues. No context, no identifiers, no structure. After years of iterating, I have settled on structured JSON logging as the non-negotiable standard:
{
"timestamp": "2026-03-15T14:23:01.456Z",
"level": "error",
"service": "order-service",
"request_id": "req_a1b2c3d4",
"user_id": "usr_789",
"method": "POST",
"path": "/api/v1/orders",
"status_code": 500,
"duration_ms": 1234,
"error": "Connection pool exhausted",
"stack_trace": "..."
}The request_id field is the single most important thing in that log entry. It lets you trace a request across every service it touches. I generate it at the API gateway and propagate it through every downstream call.
Setting Up Structured Logging in Node.js
const pino = require('pino');
const logger = pino({
level: process.env.LOG_LEVEL || 'info',
formatters: {
level: (label) => ({ level: label }),
},
timestamp: pino.stdTimeFunctions.isoTime,
redact: ['req.headers.authorization', 'body.password', 'body.credit_card'],
});
function requestLogger() {
return (req, res, next) => {
const requestId = req.headers['x-request-id'] || crypto.randomUUID();
req.requestId = requestId;
req.log = logger.child({ request_id: requestId });
const start = Date.now();
res.on('finish', () => {
req.log.info({
method: req.method,
path: req.path,
status_code: res.statusCode,
duration_ms: Date.now() - start,
user_id: req.user?.id,
});
});
next();
};
}Notice the redact option — I learned the hard way that logging raw request bodies can expose passwords and API keys. One of our early systems accidentally logged credit card numbers in plaintext. That discovery led to a very unpleasant security audit. Always redact sensitive fields at the logger level.
Pillar 2: Metrics — How Is the System Performing?
Metrics are aggregated numerical measurements over time. Where logs tell you about individual events, metrics tell you about trends and patterns. The four metrics that I track for every backend service are known as the RED method plus saturation:
| Metric | What It Measures | Why It Matters |
|---|---|---|
| Rate | Requests per second | Traffic volume and trends |
| Errors | Error rate (%) | Reliability and correctness |
| Duration | Response time (p50, p95, p99) | User experience |
| Saturation | Resource usage (CPU, memory, connections) | Capacity limits |
Prometheus Metrics in Practice
const promClient = require('prom-client');
const httpRequestDuration = new promClient.Histogram({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'route', 'status_code'],
buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});
const httpRequestsTotal = new promClient.Counter({
name: 'http_requests_total',
help: 'Total number of HTTP requests',
labelNames: ['method', 'route', 'status_code'],
});
const activeConnections = new promClient.Gauge({
name: 'active_connections',
help: 'Number of active connections',
});
function metricsMiddleware() {
return (req, res, next) => {
const end = httpRequestDuration.startTimer();
activeConnections.inc();
res.on('finish', () => {
const route = req.route?.path || 'unknown';
const labels = {
method: req.method,
route: route,
status_code: res.statusCode,
};
end(labels);
httpRequestsTotal.inc(labels);
activeConnections.dec();
});
next();
};
}
// Expose /metrics endpoint for Prometheus
app.get('/metrics', async (req, res) => {
res.set('Content-Type', promClient.register.contentType);
res.end(await promClient.register.metrics());
});One mistake I made early on was using high-cardinality labels. I added user_id as a Prometheus label, which created millions of unique time series and nearly crashed our Prometheus server. The rule of thumb: labels should have low cardinality (method, route, status code — not user ID or request ID). Use logs for high-cardinality data.
Pillar 3: Traces — Where Did the Time Go?
Distributed tracing follows a request as it flows through multiple services. Each service adds a “span” with timing information, and the trace shows you exactly where time was spent.
This is the pillar I resisted the longest. Logs and metrics felt sufficient until I spent two days debugging a slow endpoint that turned out to be caused by a downstream service making a synchronous call to a third service that was timing out. Traces would have shown me the bottleneck in five minutes.
OpenTelemetry Setup
OpenTelemetry is the industry standard for instrumentation. Here is a basic setup:
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { ATTR_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');
const sdk = new NodeSDK({
resource: new Resource({
[ATTR_SERVICE_NAME]: 'order-service',
}),
traceExporter: new OTLPTraceExporter({
url: 'http://jaeger:4318/v1/traces',
}),
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();The auto-instrumentation is what makes OpenTelemetry practical. It automatically traces HTTP requests, database queries, Redis calls, and message queue operations without you adding manual spans to every function. I add manual spans only for business-critical operations:
const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('order-service');
async function processOrder(order) {
return tracer.startActiveSpan('process-order', async (span) => {
span.setAttribute('order.id', order.id);
span.setAttribute('order.total', order.total);
try {
await validateInventory(order);
await chargePayment(order);
await sendConfirmation(order);
span.setStatus({ code: SpanStatusCode.OK });
} catch (err) {
span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
throw err;
} finally {
span.end();
}
});
}Alerting Without the Fatigue
The monitoring stack is only useful if it tells you when something is wrong. But bad alerting is worse than no alerting — I have worked on teams where developers muted Slack channels because alerts fired so often that nobody read them anymore.
Alert on Symptoms, Not Causes
This is the most important alerting principle I have learned. Alert on what the user experiences, not on internal system metrics.
| Bad Alert (Cause) | Good Alert (Symptom) |
|---|---|
| CPU usage > 80% | Request latency p99 > 2s |
| Memory usage > 90% | Error rate > 1% for 5 minutes |
| Disk usage > 85% | Order completion rate dropped 20% |
| Pod restarted | Health check failing for 3 minutes |
CPU at 80% might be perfectly fine if response times are normal. But if response times spike even while CPU is low, that is a problem worth waking someone up for. I keep cause-based alerts as informational dashboards, not paging alerts.
Alertmanager Configuration
# alertmanager.yml
route:
receiver: 'slack-notifications'
group_by: ['alertname', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: 'pagerduty-critical'
- match:
severity: warning
receiver: 'slack-warnings'
receivers:
- name: 'pagerduty-critical'
pagerduty_configs:
- service_key: '<key>'
- name: 'slack-warnings'
slack_configs:
- channel: '#backend-alerts'
title: '{{ .GroupLabels.alertname }}'
text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'Prometheus Alert Rules
# alert_rules.yml
groups:
- name: backend-alerts
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status_code=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "Error rate above 1% for 5 minutes"
- alert: HighLatency
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)
) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "P99 latency above 2 seconds"
- alert: HighSaturation
expr: |
active_connections / max_connections > 0.8
for: 10m
labels:
severity: warning
annotations:
summary: "Connection pool above 80% capacity"The for clause is crucial. It prevents alerting on brief spikes that resolve themselves. Before I added for: 5m to our error rate alert, we got paged every time a single batch of retries caused a momentary spike. Now alerts only fire if the condition persists, which dramatically reduces noise.
Building Dashboards That People Actually Use
I have built dozens of Grafana dashboards. Most of them were never looked at after the first week. The ones that survived share a common trait: they answer a specific question that someone asks regularly.
The Three Dashboards Every Backend Needs
1. Service Overview Dashboard: The first thing anyone looks at during an incident. Shows request rate, error rate, latency percentiles, and resource saturation for each service. One row per service, four panels per row.
2. Endpoint Performance Dashboard: Drill-down into individual endpoints. Shows the slowest endpoints, highest error rate endpoints, and traffic distribution. I use this weekly to identify optimization opportunities.
3. Infrastructure Dashboard: Node-level CPU, memory, disk, and network metrics. This is the “is the hardware okay?” dashboard. Useful for capacity planning but rarely the first thing I check during incidents.
Dashboard Design Tips
After iterating through many failed dashboard designs, here is what I have found works:
- Top-to-bottom priority: Put the most important panels at the top. During incidents, people do not scroll.
- Use consistent colors: Green for healthy, yellow for warning, red for critical. Across every dashboard, every panel.
- Show rate of change, not absolute values: “Errors per second” is more useful than “total errors” for understanding whether a problem is getting worse.
- Include the query: Add a text panel or annotation explaining what each panel shows. Six months from now, nobody will remember what
rate(http_requests_total{status_code=~"5.."}[5m])means at a glance.
Connecting the Pillars: Correlation
The real power of observability comes when you can jump between pillars. You see a latency spike in your metrics, click through to the traces from that time window, find the slow span, and then pull up the logs for that specific request.
Request ID as the Glue
The request_id I mentioned earlier is what makes this possible. Generate it at the edge, propagate it through every service, include it in every log entry, and attach it as a trace attribute. When something goes wrong, one ID gives you the complete picture.
function correlationMiddleware() {
return (req, res, next) => {
const requestId = req.headers['x-request-id'] || crypto.randomUUID();
const span = trace.getActiveSpan();
if (span) {
span.setAttribute('request.id', requestId);
}
req.requestId = requestId;
res.set('X-Request-ID', requestId);
next();
};
}When a customer reports an issue, I ask for the request ID from their response headers. That single string lets me find every log entry, every trace, and every metric associated with their request across our entire system. Before we had this, debugging customer issues was a guessing game that could take hours. Now it takes minutes.
Common Observability Mistakes
Having built and torn down several monitoring setups, here are the mistakes I see most often:
Logging everything: More logs is not better observability. I once worked on a system that logged every database query with full parameters. The log volume was so high that our log aggregator could not keep up, and we missed actual error logs buried in the noise. Log at the right level: errors always, request summaries at info, debug details only when needed.
Ignoring log retention costs: Storing months of logs at debug level gets expensive fast. I set retention tiers — 7 days for debug, 30 days for info, 90 days for errors and warnings. Most debugging happens within the first few days anyway.
Not testing alerts: An alert that has never fired might not work when you need it. I schedule monthly “alert fire drills” where we intentionally trigger each critical alert to verify the notification chain works end to end.
Dashboard sprawl: Every team member creating their own dashboards leads to dozens of unmaintained dashboards with conflicting information. I maintain a small set of canonical dashboards and review them quarterly.
Frequently Asked Questions
What Is the Difference Between Monitoring and Observability?
Monitoring tells you whether a system is working — it checks predefined conditions and alerts when thresholds are crossed. Observability tells you why a system is not working — it gives you the data to investigate novel, unexpected failures. Monitoring asks “is the database up?” Observability asks “why are 3% of queries to the users table taking 10x longer than yesterday?” You need both, but observability is what saves you during complex incidents.
Do I Need All Three Pillars From Day One?
No. Start with structured logging (the highest leverage improvement), then add metrics when you need trend analysis and alerting, then add distributed tracing when debugging latency issues across services becomes painful. For a monolithic application, logs and metrics are usually sufficient. Tracing becomes essential when you move to microservices.
How Much Does an Observability Stack Cost to Run?
Self-hosted stacks (Prometheus, Grafana, Loki, Jaeger) cost primarily in compute and storage — typically $200-500 per month for a small-to-medium system. Managed services like Datadog or New Relic charge per host and per data volume, and costs can escalate quickly to $1,000+ per month. In my experience, start self-hosted to learn the concepts, then evaluate managed services when operational overhead becomes a bottleneck.
How Do I Avoid Alert Fatigue?
Three rules that have worked for me: (1) alert on symptoms, not causes — user-facing impact, not internal system metrics; (2) use the for clause in every alert rule to ignore transient spikes; (3) review and prune alerts quarterly. Every alert should have a clear owner and a documented response playbook. If nobody acts on an alert for a month, delete it.
Should I Use OpenTelemetry or a Vendor-Specific SDK?
OpenTelemetry is the correct default choice in 2026. It is vendor-neutral, widely adopted, and supported by every major observability platform. Vendor-specific SDKs may offer deeper integration with their platform, but they create lock-in. I have migrated between observability vendors twice in my career, and having vendor-neutral instrumentation made those migrations dramatically easier.
The Bottom Line
Observability is not about tools — it is about building systems that tell you their own story. The three-week silent memory leak that kicked off this article would have been caught in hours with the setup I described here: structured logs with request IDs, Prometheus metrics with sensible alerts, and distributed traces that show you exactly where time is being spent.
Start small. Add structured logging to your application today — it takes less than an hour and immediately improves your debugging capability. Then add Prometheus metrics and a basic Grafana dashboard. Finally, introduce OpenTelemetry tracing when your architecture grows complex enough to need it. Each layer compounds the value of the others, and the investment pays for itself the first time you catch a production issue before your users do.
Product recommendations are based on independent research and testing. We may earn a commission through affiliate links at no extra cost to you.
