Reporting lag: the two days nobody accounts for

Ask a third-party music data API for a track's stream total today, and you get a number. What you do not get is the date that number describes.

Comparing those totals against a platform's own reporting shows the gap. For Spotify specifically, the totals reaching third parties have been running about two days behind ground truth in our checks. A total read on the 10th is closer to the true position on the 8th.

This is invisible in every way that matters, because the response looks current.

What it breaks

Release day lands late. A Friday release's surge appears on the chart on Sunday. Everyone notices and someone invents an explanation for why the weekend was strong.

Correlations compare the wrong days. Line up same-day ad spend against a two day lagged streaming series and each comparison is offset by 48 hours. On a two week campaign that is enough to reverse the finding, while producing a confident-looking coefficient.

Windows are off by the lag at both ends. A 30 day window ending today actually ends two days ago, so its baseline should also be shifted or the window is 30 days of the wrong 30 days.

Two systems disagree. If one place in a codebase corrects and another does not, the same project shows two different daily figures, offset by exactly the lag. We shipped that too, and it was mystifying until we noticed the difference was always the same number of days.

Why it has to be a manual constant

We looked for a way to derive it. There is no as-of or data-through field in the responses we work with, and we probed for dated history endpoints across several likely shapes without finding one. Without a stated reference date, the lag can only be established by comparing against a platform's own reporting, by hand, per source.

So it lives as a named constant: a map from source to a number of days. Currently that is two days for Spotify and zero for the others, which reflects what we have been able to verify rather than what is true everywhere.

Two obligations come with that.

Document it as a manual correction. Someone reading the code in a year needs to know this is a measured constant, not a computed one, and that it needs revisiting if the upstream pipeline changes.

Verify it periodically. A correction that was right in March and wrong in September is worse than no correction, because it is confidently wrong.

Apply it once, from one place

The rule that matters operationally: every consumer that builds a dated series imports the same constant.

Ours is applied in two places, the sync that emits dated events and the route that builds the chart series. Those originally had separate implementations and drifted, which is exactly the bug above. One shared constant, imported, never copied.

Stamp the output

Every emitted point carries the lag that was applied to it. Two reasons.

Auditability. When somebody queries a number in six months, the answer to "was this corrected" is in the data rather than in someone's memory of what the code did at the time.

Change safety. If the constant changes, you can tell which stored points were produced under the old value, which turns a data migration question into a query.

Lag against cadence

Worth separating, because they get confused.

Lag is the offset between when activity happened and when the number describing it becomes available. It shifts a series along the time axis and is fixed with an offset.

Cadence is how often the number updates at all. A source that only advances once a week is not lagged, it is coarse, and shifting it does nothing useful. That needs weekly bucketing instead, which is a different problem.

Applying a lag correction to a weekly source, or weekly bucketing to a lagged daily one, makes both worse. Work out which you have by looking at the change dates in your own stored snapshots: a lagged source changes most days but describes an earlier date, and a coarse source only changes on one weekday.