Observability for daily data syncs
A sync that fails loudly is a good day. The dangerous one succeeds while quietly returning nothing, and only monitoring built for that case will catch it.
Danny Starr
Co-founder, Backline · 5 June 2026 · 3 min read
In short
- The failure that matters is the partial one: the run succeeds, most projects are fine, and one source quietly returns nothing.
- Record a run per execution and granular events per project and per integration. Run status alone cannot see a partial failure.
- Buffer events in memory and write them once at the end, so monitoring never slows the thing it monitors.
- Track freshness per project per source, not just run success. Freshness is what a customer experiences.
- A degraded run, one that finished but did much less than usual, deserves an alarm of its own. Nothing else catches a slow collapse.
A daily sync that throws an exception is easy. Something alerts, somebody looks, it gets fixed.
The one that costs you a customer is the run that completes successfully, processes 40 of 44 projects normally, and returns nothing for four because a token expired. Status green, charts flat, nobody knows for three weeks.
Monitoring for that specific failure is a different design from monitoring for crashes.
Two levels of record
A run record per execution. Job name, start, end, duration, status, summary counts, and the error if there was one. This answers whether the job ran and how long it took.
Granular events inside the run. One per project per integration, at least on failure and as a summary of what was ingested. This is what answers whether the run actually did anything for a given project.
Without the second level, a run that silently did nothing for a subset is indistinguishable from a run that worked.
What each level of record can tell you
Illustrative| Run record | Per-project events | Freshness view | |
|---|---|---|---|
| Did the job run | Yes | No | No |
| How long it took | Yes | No | No |
| Did one project get nothing | No | Yes | Yes |
| Which vendor failed and why | Only if the whole run failed | Yes | No |
| What a customer experiences | No | Indirectly | Yes |
| Is this run smaller than usual | With history | Yes | No |
Buffer the events
The obvious implementation writes an event row as each one occurs, which adds a database round trip inside the hot loop of a job already fighting a time limit.
Buffer them in memory and flush once in a batch at the end. Two properties follow: monitoring cost is one write regardless of event count, and monitoring can never be the reason a run times out.
The tradeoff is that a hard crash loses the buffer. That is acceptable, because a hard crash is the failure mode that already alerts.
Freshness is what the customer feels
Run status is about your jobs. Freshness is about their data, and they are different questions.
A freshness view answers: for each project and each source, when did we last successfully store something? That surfaces the partial failure immediately, because one project sitting at nine days when everything else is at one day is visible at a glance in a way that a list of green run statuses is not.
It is also the right thing to put in front of a support question. "This looks wrong" becomes "that source last synced four days ago and here is why", in one query.
Alarm on degraded, not only on failed
The subtle failure is the run that succeeds while doing much less than usual. Half the projects skipped because a shared credential expired. A third of tracks failing to resolve after a provider changed a requirement.
The check is comparing this run's counts against the recent normal for that job, and alarming when it is substantially below even though the status is success. It is the only monitoring that catches a slow collapse, and it needs the run history you are already storing.
Failure modes by how long they go unnoticed
IllustrativePractical rules
Guard the auth check against an unset secret. A scheduled job protected by comparing a header against an environment variable will accept everything if that variable is empty. Check that it is set as well as that it matches.
Continue past a failing project. One project's broken token must not stop the other forty-three. Catch per project, record, continue.
Record before every return. Including the early error returns, which are the paths most likely to be missed and most valuable to see.
Prune on a schedule. Granular events are the largest table in this design. Keep them for a few months, keep run records longer, and delete in the application rather than hoping someone remembers.
Keep one list of jobs and schedules. The job names in your monitoring and the schedule in your deployment configuration have to agree, or a job silently stops being scheduled and the monitoring shows nothing because nothing ran.
What to look at weekly
Three things, and it takes two minutes.
Did every job run yesterday. Is any project's freshness materially worse than the rest. Is any job's duration trending upward, which is the early warning for a timeout that will eventually start truncating work.
That last one is worth stating plainly: a job that grows past its time limit does not usually fail cleanly. It gets killed part way through, which looks exactly like a partial failure, which is the thing all of this exists to catch.
Common questions
- How do you monitor a data sync that fails silently?
- Record two levels: one run per execution with status, duration and counts, and granular events per project and per integration inside it. Then track freshness per project per source, which is what a customer actually experiences, and alarm when a run succeeds while doing substantially less than its recent normal.
- Should monitoring events be written as they happen?
- No. Buffer them in memory and flush once at the end of the run. That makes monitoring cost one write regardless of event count and means it can never be the reason a job exceeds its time limit. A hard crash loses the buffer, which is acceptable because a crash already alerts.
- What is a degraded run and why alarm on it?
- A run that completes with a success status while processing far less than usual, for example because a shared credential expired or a provider changed a requirement. Nothing else catches a slow collapse, because every conventional status stays green while the data quietly stops arriving.
Sources
- 1Google, Reviewed August 2026. Search Console API: Search Analytics query reference
- 2Meta, Reviewed August 2026. Graph API insights reference
Danny Starr
Co-founder, Backline
Danny Starr is a co-founder of Backline and builds the platform. He writes about the data engineering behind music analytics: ingestion, identity, honesty in charts, and the AI layer on top of it.
Backline does this for the projects you run
Streaming, audience, social, advertising, website, search, ticketing and press data in one dashboard per project, with an AI assistant that answers questions about your own connected data. Invite-only.
Keep reading
Data engineering
Designing an API layer over a dozen music data vendors
Streaming, social, advertising, ticketing, email and search all speak differently, fail differently and rate limit differently. The patterns that keep an integration layer from becoming a liability.
Danny Starr · 3 min read
Data engineering
Cumulative counters, daily deltas, and the data you can lose forever
Music data arrives as a running total, so daily numbers have to be derived. The three bugs that derivation produces, one of which destroys data permanently.
Danny Starr · 3 min read
Analytics foundations
Anomaly detection that managers do not mute
Most alerting in music tools gets switched off within a fortnight. The design rules that keep an alerts feed credible: real baselines, absolute floors, and a bias towards silence.
Danny Starr · 4 min read
