Designing an API layer over a dozen music data vendors

A music analytics platform is mostly an integration layer. The interesting parts are the derivations, and the volume of work is in talking to a dozen services that agree on nothing.

These are the patterns that have held up.

Normalise at the edge

One module per vendor, whose only job is to speak that vendor's dialect and return your shapes. Nothing downstream knows which vendor a number came from.

Two payoffs. Swapping a vendor is a rewrite of one module rather than a search across the codebase. And the vendor's peculiarities, its pagination, its nulls, its idea of a date, are quarantined in a file that can be tested against recorded responses.

The discipline that makes it work is that the shared module has no framework or database imports, so it can be unit tested directly. Once an integration module imports a database client, its logic stops being testable and starts being a job for a staging environment.

Verify the shape before writing the types

Read the documentation, then make the request and look at what comes back.

APIs that look similar nest their payloads differently. One provider we use returns a status as a plain string with the payload as a sibling key at the top level, which is not what a similar-looking API next to it does. Assuming the wrong nesting produces types that compile and code that fails at runtime, on a schedule, in production.

Recording a real response and building the types from it costs ten minutes and removes a class of bug entirely.

Fail open, everywhere

The rule: one vendor being unavailable costs one section, never a page and never a sync run.

Concretely, every read path returns an unavailable marker rather than throwing, every section renders its own empty state, and a cron continues to the next client after a failure rather than aborting the run.

The same applies to schema. A read path that selects a column added by a migration should retry without that column if it errors, so the code can ship before the migration is applied. It looks defensive to the point of paranoia until the first time a deployment lands ahead of a migration and nothing breaks.

Concurrency limits are real and undocumented

Some vendors cap concurrent requests per project, and exceeding it does not always produce a clean error. One we work with returns rate limit responses that, handled naively, appear as empty result sets rather than failures, so the symptom is missing data instead of an exception.

Two mitigations. A wrapper around every outbound call that retries on rate limiting with backoff, and a bounded concurrency helper used for any fan-out, so a project with 200 tracks does not open 200 connections.

The general lesson is that unbounded parallelism over third-party APIs eventually produces silent data loss, and silent is the expensive part.

Credentials expire and grants get revoked

Every long-lived integration eventually breaks through no fault of yours. A token expires, a customer removes an app, a partner rotates a key, a permission gets withdrawn during an unrelated audit.

So the integration includes refresh, rotation and status. Persist the refreshed token every time, since some providers rotate the refresh token itself on use and dropping the new one means the next refresh fails. And surface a connection status the customer can see, because the alternative is a chart quietly going flat and nobody noticing for a month.

Secrets stay on the server

Credentials belong in server-side storage and never reach a browser. A read endpoint returns a boolean saying whether a secret is set, never the value, and a write treats an empty field as leave unchanged so an interface that cannot display a secret can still edit around it.

Related and worth stating: a key that is safe in a browser and a key that is not are different things, and mixing them up is a common failure. An ingestion key designed for client-side use is fine in a public page. An export key with read access to everything is not, and they often look similar.

Log per vendor, per project

When a sync covers a dozen integrations across many projects, "the sync failed" is not a diagnosis.

Every run records what it did per vendor and per project: successes, failures, counts, durations. That turns a support question into a query, and it makes the difference between a customer telling you a number looks wrong and you telling them a grant was revoked four days ago.