Tools

Measuring the answer layer

Two small tools I built because the question I needed answered was not one the subscriptions answer, and the APIs that do answer it are public. Both are here to read and to run.

The problem

Semrush will tell you an AI Overview appeared on a keyword. It will not tell you whether your domain was named inside the answer. Those are different questions, and only the second one says whether you are winning the surface.

That gap is not academic. A brand can hold a keyword, watch an AI Overview take the click, and have no idea whether the answer that replaced the click recommended them or their competitor. The platforms that do measure it sit behind subscriptions. But every engine worth measuring already publishes its own citations through a documented API, so the measurement can be built directly, and run for cents.

citation_meter.py

Sends a fixed prompt set to four answer engines, reads each engine's own cited sources, and reports how often you appear against named competitors.

Each engine runs only if its key is present, so it degrades gracefully. Set OPENROUTER_API_KEY alone and all four run through OpenRouter. Native keys win where both exist, because they return richer citation payloads.

python3 citation_meter.py --config example.json --dry-run   # shape only, no keys needed
python3 citation_meter.py --config example.json             # the real thing
python3 citation_meter.py --config example.json --trend     # read the series, spend nothing

Three decisions in here are the ones that make the number worth trusting.

The parser is deliberately dumb. Response shapes drift between API versions, so rather than couple to a schema it walks the whole JSON and collects anything that looks like a URL, then matches on registrable domain. Hard to break, and subdomains count, so docs.example.com matches example.com.

The denominator is hashed. A citation share only means something if the prompt set holds still. Every run records a hash of its prompts, and changing the prompts splits the trend at that point rather than quietly comparing different questions to each other.

Engines get weighted by who actually uses them. Four separate engine percentages do not tell you how much of a buyer's AI surface you hold. Given audience data, the meter orders engines by that buyer's real usage and reports one reach-weighted number.

Arms: holding everything still except one thing

A config can carry named arms instead of a flat prompt list. Each arm asks the same buyer about the same buying criteria in a different framing, so any difference in who gets cited is attributable to the framing and to nothing else.

python3 citation_meter.py --config example.json --all-arms

The point is control. Ask "which SOAR platforms" and you will get SOAR vendors back. That tells you nothing until you have also asked "which workflow automation platforms" with the buyer and the criteria held constant. Each arm keeps its own history series and its own prompt hash, so arms never contaminate each other's trend.

One trap worth naming, because it is easy to fool yourself with: an arm that names a competitor in every prompt has primed that competitor's number. Read your own domain in that arm, not theirs. The tool works this out from the prompts and prints the warning next to the comparison table.

sparktoro.py

The meter answers who gets cited. It cannot answer the two questions that come before that, and for a while I was answering both by intuition: which engines should we meter for this buyer, and which questions should we meter them on?

SparkToro answers both from clickstream panel data. Two endpoints per audience. /v3/apps/ai gives the assistants that audience actually uses against the national baseline, which is the engine list and its weighting. /v3/prompts gives the prompt topics they carry affinity for, which is what the prompt set gets written against.

python3 sparktoro.py --describe "security engineers who evaluate automation" --slug acme
python3 sparktoro.py --slug acme --report
python3 sparktoro.py --slug acme --sync-config example.json

It reports topics, not verbatim prompts, and assistant usage, not citations. It tells you where the buyer is and what they ask about. Only the meter can tell you whether you are named there. The prompts it suggests are templates over the topics, marked as derived, and a human rewrites them before they count as a measurement.

Read the source

citation_meter.py sparktoro.py example.json

Runs land in ./runs/ relative to the working directory, not next to the scripts, so configs and their history stay together wherever you keep them. --dry-run writes nothing and adds nothing to the series.

I built this same instrumentation at OnPay before a vendor sold it. If you want it pointed at your category, that is a conversation I enjoy having.

CONTACT RON