Tool design for an analytics agent

We have twelve tools sitting over a music project's connected sources. Rewriting several of them taught us more about answer quality than any prompt change.

1. Shape tools around questions, not tables

The instinct from API design is small, composable primitives. For an agent it is usually wrong, because every join the model has to do is a chance to do it differently than the dashboard did.

Compare two designs for live shows. Primitives would be list-shows, get-capacity and get-ticket-sales, leaving the model to compute sell-through. One tool returns each show with its ticket pacing summary already computed: latest sold, sell-through against capacity, first and last data date, number of points.

The second is better on three counts. The arithmetic matches the dashboard because it is the same function. The model spends its attention on the answer rather than the join. And it takes one call instead of three, which matters when a loop is bounded.

2. Make the empty case structured

The most common state for any given source on any given project is not connected. That is not an error and it is not silence.

A good empty response says which source is missing, that this is expected rather than broken, and what connecting it would add. The model then writes something useful, and the manager learns something about their setup.

3. Put the caveats in the response

Tool descriptions are read once when the model plans. The response is what it has in front of it while writing.

So units, cadence and warnings belong in the returned data. Our ticket sales tool returns a notes field explaining that bulk box-office uploads and per-show pacing are two different sources that must never be silently combined. Our short-form tool returns a note saying own-account metrics are not the same as videos using the song. Those notes materially change the answers.

4. Return a refusal when ambiguity would mislead

Covered in the assistant article, and it is a tool design rule: when a project has several ad accounts and the call does not say which, the tool returns the account list and no figures.

The general form is to identify the cases where a plausible answer would be wrong and make the data for that answer unavailable, rather than adding another sentence to a prompt.

5. One computation, two consumers

Every number a tool returns should come from the same function the interface uses. When they diverge, the customer sees two numbers and stops trusting both.

The practical discipline is that the shared function lives in its own module with no framework or database imports, so it is unit-testable and can be called from a page, an API route and a tool without duplication.

6. Scope at the boundary, not in the tool

The project identifier comes from the authenticated session and is passed in. A tool must never accept a project list from the conversation, because the conversation contains whatever the user typed.

For token-authenticated access, the same rule applies one level up: the token grants access, and every read is scoped by it. A tool is not an access control layer, it is a reader operating inside one.

7. Bound everything

A hard iteration cap on the tool loop. Timeouts on every outbound call. Caps on rows returned. A model asking for a year of daily data across five sources will happily fill a context window with numbers it cannot use, and the answer gets worse rather than better.

8. Name tools for what they answer

get_ticket_sales beats query_sales_table. The name is read by the model as documentation, and a name describing the question it answers gets chosen correctly more often than one describing a data structure.

The same goes for fields. sell_through_pct is unambiguous. ratio is not.

9. Test the tools without the model

Tool logic belongs in pure functions tested directly against synthetic inputs: a project with no data, a project with one day, a project with a gap, a project with several accounts, a project where one source errors.

Testing through the model is slow, non-deterministic and tests the wrong thing. The model's job is judgement about which tool to call. The tool's job is to be correct, and correctness is testable.

What we would do differently

We started with tools shaped like our database and moved towards tools shaped like questions, and every one of those moves improved answers. If we were starting again, the first design question would be: what does a manager actually ask on a Monday, and what would one call have to return to answer it completely.