# Measuring the answer layer

Two free Python tools by Ron Kagan. The citation meter measures API answers to a fixed
buyer-question set. SparkToro supplies audience usage and topic inputs. Google Search
AI Overviews require separate Search measurements; API results do not reproduce the
consumer ChatGPT, Claude, Gemini or Perplexity apps.

## Start with three questions

Download [citation_meter.py](citation_meter.py) and [example.json](example.json) into the
same private working folder. Use Python 3.10 or later. There are no third-party Python
package dependencies. Replace the fictional domain, competitors, brand aliases and
questions before making a real measurement.

```bash
python3 citation_meter.py --config example.json --dry-run
python3 citation_meter.py --config example.json
python3 citation_meter.py --config example.json --trend
```

The dry run spends nothing and writes nothing. The real run calls whichever providers
have keys. Supply keys through the process environment, never the config or source:

| Provider | Environment variable | Default model |
| --- | --- | --- |
| OpenAI | `OPENAI_API_KEY` | `gpt-5.2` |
| Anthropic | `ANTHROPIC_API_KEY` | `claude-sonnet-5` |
| Gemini | `GEMINI_API_KEY` | `gemini-2.5-flash` |
| Perplexity | `PERPLEXITY_API_KEY` | `sonar-pro` |
| OpenRouter fallback | `OPENROUTER_API_KEY` | Provider-qualified versions of the four above |

A native key takes priority for that engine. OpenRouter explicitly requests native
search; it does not silently select an external search plugin. Confirm that your account
supports the selected model and native search. Model availability can change.

Use `"engines": ["openai"]` in the config to run just OpenAI, including through
OpenRouter. Supported names are `openai`, `anthropic`, `gemini` and `perplexity`;
omitting the list attempts all configured providers. Empty, duplicate or unknown names
are rejected before any request.

In the [10 September 2026 trial](skilltrade-sample-2026-09-10.json), OpenAI through
OpenRouter produced three scorable answers. The selected Perplexity and Gemini
native-search routes returned HTTP 404; Anthropic returned zero-position annotations
that could not be reliably attached to answer passages. Those routes were unavailable,
not measured zeros. A valid key alone does not establish citation compatibility. The
meter deliberately rejects invalid spans rather than counting every retrieved URL.

The optional `models` object pins native models by engine name, for example
`{"openai": "gpt-5.2"}`. Requested and returned model IDs are stored with the run.
Provider charges depend on tokens, searches, questions and repeats. Check current
pricing and set a spending limit; the tools do not promise a fixed cost.

## Inspect the real Skill Trade example

The [worked example](https://www.skilltrade.marketing/tools/#worked-example) includes
all 12 attempts from one trial of three unbranded questions across four model routes.
Only OpenAI completed three scorable answers: no Skill Trade mention or owned-site
citation in the problem and category answers, and both in the decision answer.

Download the [text report](skilltrade-sample-2026-09-10.txt) and
[JSON evidence](skilltrade-sample-2026-09-10.json). The export preserves full generated
answers and URL annotations from returned responses, including unreadable ones. It
omits retrieved source excerpts, reasoning and provider request metadata. Full raw
responses remain private. A parser replay verified the recorded outcomes without new
API requests. Additional diagnostic requests are not pooled into this sample.

```bash
python3 citation_meter.py --report skilltrade-sample-2026-09-10.json
```

This single observation per question is not a general citation benchmark or evidence
of uplift. The page's proposed guide experiment has not been run.

## Define the measurement

Each question is an object with `id`, `stage` and `text`. IDs must be unique. Stages are
`problem`, `category`, `decision` or `unspecified`. Legacy string prompts still work;
their stage is `unspecified` rather than guessed. These are separate requests, not a
multi-turn buyer conversation.

`brand_aliases` maps each domain to an explicit list of names to find in answers.
Matching is case-insensitive and respects word boundaries. A mention establishes only
that the text names a brand, not that the answer recommends it.

| Outcome | What counts |
| --- | --- |
| Mention | An answer contains the configured domain or brand alias. |
| Owned-site citation | A documented answer citation links to that domain or a subdomain. |
| Third-party candidate | A citation's attached passage names the brand but links elsewhere. Inspect the source before calling this coverage or support. |
| Uncited mention | An answer names the brand and has no citations at all. |

These categories overlap. `mention_without_owned_citation` is also retained in each
brand's evidence: it can include third-party citations, so it is not called an uncited
mention. Missing or narrow annotation passages can miss third-party candidates; a human
can inspect the preserved complete answer and source links.

## Where the citation comes from

- [OpenAI Responses](https://developers.openai.com/api/docs/guides/tools-web-search):
  `url_citation` annotations on assistant answer text. Search action sources stay separate.
- [Anthropic Messages](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool):
  `web_search_result_location` citations on text blocks, not every search-result URL.
- [Gemini generateContent](https://ai.google.dev/api/generate-content#GroundingSupport):
  `groundingSupports` link answer segments to referenced `groundingChunks`. Google's
  grounding redirect URLs are resolved before matching; both URLs are retained.
- [Perplexity](https://docs.perplexity.ai/api-reference/sonar-post): numbered answer markers
  select entries from `citations`. Unmapped citation lists are unavailable for scoring.
- [OpenRouter](https://openrouter.ai/docs/guides/features/plugins/web-search):
  standardized `url_citation` annotations attached to the returned answer.

Plain links elsewhere in a response, unused search results, images and tool inputs do
not earn citation credit. A citation establishes an attached source, not that the source
supports every claim. Unknown shapes, failed searches, refusals, truncated responses and
unresolved grounding links are errors, not measured absences.

## Inspect the evidence and coverage

Runs are JSON files in `./runs/`, or a directory supplied through `--history`. Keep
private runs outside your public-site checkout. Each attempted question retains its
status, stage, timestamp, provider, requested model and raw response when one returned.
Parsed answers, citation passages, original/resolved URLs and retrieved URLs remain
separate. Request errors retain a safe error category, not credentials or error bodies.

```bash
python3 citation_meter.py --report /path/to/saved-run.json
python3 citation_meter.py --config example.json --history /path/to/private-runs --trend
```

A complete-run rate requires every planned answer for that engine. The file also retains
`observed_share` for the successful subset, clearly separate from the complete-run rate.
Stage tables show completed/planned counts; incomplete stages have no rate. A successful
answer with no citation is a measured zero. A failed request is unavailable.

Question text, IDs, stages, aliases, parser version, provider, requested/returned models
and request settings identify a series. Changes restart deltas, including a return to an
older prompt set. Legacy files from the original URL-harvesting parser remain readable
as unverified results; they cannot establish deltas against corrected runs.

## Test framing with repeated observations

Download [framing-example.json](framing-example.json). It contains two category framings
with matching question IDs and stages. Keep the buyer and buying criteria constant.

```bash
python3 citation_meter.py --config framing-example.json --all-arms --repeats 3 --seed 0
python3 citation_meter.py --config framing-example.json --arm category-a --trend
```

`--all-arms` defaults to three repeats; ordinary runs default to one. Within each repeat,
question blocks and arm order are shuffled reproducibly by the seed. Versions of the
same question run next to one another for each engine. The full schedule is saved.

The comparison uses matching engine/question/repeat observations that succeeded in every
arm. It reports matched/planned coverage and the rate range across repeats. Missing pairs
can bias this comparison. Separately run arms and mid-experiment model changes cannot
produce a comparison. These observations can suggest framing sensitivity; they do not
prove causation. Naming a brand in any question is reported as prompted visibility.

## Add audience inputs

Download [sparktoro.py](sparktoro.py) and supply `SPARKTORO_API_KEY`. SparkToro requires an
account and credits. `--credits` reads the balance; `--describe` creates an audience report.

```bash
python3 sparktoro.py --describe "small teams evaluating project management software" --slug sample
python3 sparktoro.py --slug sample --report
python3 sparktoro.py --slug sample --sync-config example.json
```

`/v3/apps/ai` supplies assistant usage; `/v3/prompts` supplies topic affinity, not exact
buyer questions. New pulls retain a timestamp and report ID. Existing questions are
preserved. When the script generates a new question list, it marks `prompts_status` as
`draft`: rewrite it and set that field to `approved` before measurement.

The optional audience-weighted citation index uses usage weights among metered engines
with complete results. People may use several assistants, so it is not unique reach,
traffic or revenue. Valid usage can be retained when a national baseline is missing;
that row's lift is unavailable. Older caches without timestamps remain undated.

## Verify changes without API calls

The published `test_citation_meter.py` and `test_citation_experiments.py` use synthetic
protocol fixtures, explicitly separate from real measurements. Run from the site root:

```bash
python3 -B -m unittest discover -s tools -p 'test_*.py' -v
```

CI runs those regressions and both example dry runs without credentials.