Data engineering playbook
Catch Data Problems Before Your Users Do
A Practical Data Observability Playbook
Know whether important data is ready before someone uses it. Choose useful checks, expose gaps in the evidence, and give every important alert an owner and a next action.

What this guide helps you do
- Choose: start with one critical flow and work back from its business deadline.
- Check: distinguish job completion, data health, and consumer readiness.
- Respond: give useful alerts a named owner, escalation path, and permitted action.
- Prove: rehearse correction, downstream refresh, and acceptance before release.
Includes a worked replenishment example, a reusable monitoring-and-response worksheet, and a practical 30-day pilot.
Get the full 10-page guide
Complete the form to access the full playbook, worked example, reusable worksheet, and pilot plan.
Data observability
The job finished. Is the business output ready?
KCM Solutions | Updated September 11, 2026
A pipeline can finish on time while delivering data the business should not use. Yesterday's transactions can arrive in a new file. One location can disappear while higher activity elsewhere keeps the total row count looking normal. A report can refresh successfully and still support the wrong decision.
Data observability means gathering enough evidence about a data flow to notice problems, understand the likely impact, and respond. Its value depends on what happens after a warning: who investigates, which consumers are affected, and what they are permitted to do.
Start with one question: If this data becomes late, incomplete, or wrong, will someone act before the business relies on it?
Separate three questions that a green job status cannot answer
Execution health: Did the expected job run and complete? This confirms something about the delivery machinery. It does not prove the source supplied the right records.
Data health: Did the expected records and values arrive? Evidence might include event timestamps, completeness by location, key validity, and reconciliation to a source control total.
Consumer readiness: Can the report, planning process, or AI workflow use the data within its agreed conditions? This includes the decision deadline, relevant exceptions, downstream refresh, and the person authorized to accept or hold the output.
Keep these states visible separately. A successful job should not silently turn an untested business output green.
Choose the decision before choosing the checks
Select a recurring output that matters: a replenishment recommendation, cash forecast, service report, or operational dashboard. Name its business owner and the time at which someone acts on it.
Map the source, transformations, dataset, refresh, and consumer. Identify critical segments such as locations, business units, currencies, or product groups. Validate this map with the people who operate it; available lineage may omit manual files or external steps.
For each check, record the required evidence, acceptable threshold, frequency, maximum result age, owner, and blind spots. Choose timing that leaves a useful response window before the decision deadline.
If the meaning of the metric itself is disputed, settle that separately. KCM's metric-contracts guide addresses definitions and ownership. Monitoring can assess whether today's data meets those agreed conditions.
Measure the data the decision actually needs
Freshness requires a precise clock. Event time records when the business event occurred. Ingestion time records when it arrived. Table modification time records a change to the table. Check time records when the monitor evaluated it. A recent table modification can contain old events.
Use event-time coverage appropriate to the flow. The newest timestamp alone may conceal older records or a stale segment. Define how late arrivals, time zones, weekends, and expected periods affect the rule.
Check completeness at the level where an omission matters. An overall volume range may pass while a distribution centre is absent. Compare expected segments and, where feasible, use independently established control totals for counts or quantities.
Add validity checks for required fields, keys, ranges, and relationships. Treat unexpected changes as reasons to investigate: a promotion or genuine demand shift may explain an anomaly. Document legitimate exceptions and their expiry rather than quietly weakening the rule.
Show when the monitoring evidence is unavailable
A healthy result needs a completed check, current evidence, and the intended coverage. Give other states explicit labels:
| State | What it means |
|---|---|
| Data issue | The check completed and found a rule breach. |
| Missing result | An expected result has not arrived. |
| Stale result | The last result is too old for the decision. |
| Check failed | The monitor could not finish its evaluation. |
| Unconfigured | No suitable check exists for this requirement. |
These states need different investigation paths, but none is evidence of health. Monitor the checks themselves: a stopped scheduler or expired permission can otherwise remove the warning system without fixing the data.
Turn related warnings into one useful incident
An actionable alert names four things: affected output, evidence, responsible owner, and next action.
Include the check and dataset, observed value, expected condition, event period, result age, affected segments, and a link to investigation evidence. Route it to a named responder with an acknowledgment expectation and an escalation route.
Consolidate related symptoms where they share evidence or impact. Preserve the individual findings so consolidation does not hide a second problem. Suggested causes are investigation leads; confirm them before describing a cause as established.
A worked replenishment example
Consider this process. The inventory load finishes at 8:00 a.m., ahead of a 9:00 a.m. ordering decision. The timing sets the ordering decision for this example.
One distribution centre's file contains yesterday's records. Another centre is missing, but higher volume elsewhere keeps the aggregate row count within its usual range. The report refreshes and generates a recommendation.
Three checks provide different evidence. Event-time checks flag the stale centre. Expected-segment checks flag the absent centre. Reconciliation confirms that the extra rows at the other centre match its source count; those rows cannot substitute for the missing centre, and other readiness checks still apply.
The responder opens one incident naming the replenishment recommendation, affected centres, check results, source owner, and response deadline. Under this example's agreed procedure, the business owner places the affected recommendation on hold and tells buyers which locations are affected.
The monitor raises the signal; the hold requires a separate, implemented control or manual action. If a dated fallback is allowed, the owner approves its scope and limitations. Otherwise, the recommendation remains withheld.
After correction, the team reruns the affected checks and refreshes the consumer output. The owner records acceptance evidence and explicitly releases the hold. A green source table alone does not complete that handoff.
Prove a small monitoring service before expanding it
Use a 30-day pilot to select the flow, establish checks, and rehearse a controlled incident in a test environment. Verify the responder receives the alert, consumers receive the warning, and the agreed hold and release process works.
Start with existing checks where they provide sufficient coverage and useful routing. Consider additional tooling when specific gaps justify its access requirements, integration effort, cost, and maintenance.
Review missed incidents, stale or missing results, actionable alerts, detection time, acknowledgment time, and operating cost. Record what each measure includes. Expand when the team can explain both its coverage and its limits.
For broader recovery planning, use Can Your Analytics Recover?. For release controls before a new pipeline goes live, see AI Can Build the Pipeline. Is It Ready for Production?.
Build your monitoring-and-response plan
The companion 10-page guide includes the worked example, a reusable worksheet, and a practical pilot plan. Use it to map the checks, people, and response steps needed for one critical flow.
KCM can help review the coverage and response process for one critical data flow. Explore KCM Data Engineering.
Get the full 10-page guide and worksheet
Turn the ideas into a monitoring-and-response plan for one important data flow.
Common questions
Is this tied to a particular monitoring platform?
No. Use existing platform checks or additional tooling according to the coverage, integration, security, cost, and response requirements of the flow.
Does a recent table update prove that business data is fresh?
No. A new load can contain old events. Distinguish event, ingestion, table-modification, and check times, and check the critical business segments.
Will an alert automatically stop a report or AI workflow?
Only if an appropriate downstream control has been deliberately implemented and tested. Treat the monitoring signal and the consumer hold or release action separately.
Where should we start?
Choose one recurring output with a clear owner and decision deadline. Map its delivery path, define useful checks, and rehearse a controlled incident in a test environment.
Review one critical data flow.
Explore KCM Data Engineering