Measuring Useful Work, Not Tokens
Token prices are visible, comparable, and incomplete. The research problem is to convert them into the cost of accepted work.
Tokens are the most convenient price unit in AI. They are also a poor final unit for management decisions.
Two systems can consume the same number of tokens and produce different economic results. One may require almost no review. Another may generate plausible errors that take longer to find than the work would have taken in the first place.
This means the useful price is not dollars per token. It is dollars per accepted unit of work.
A basic measurement loop
A simple experiment can start with a defined task, a fixed source packet, a model, and an evaluation rubric.
For each run, record:
- model and version;
- inference settings;
- input and output tokens;
- API cost;
- elapsed time;
- human setup time;
- human review time;
- quality score;
- corrections required;
- failure status;
- notes on error type.
The calculation then becomes more economic than technical. What did it cost to get an output that a human decision-maker would actually accept?
Why accepted work matters
A cheap model that produces work requiring heavy review may be rational for low-risk tasks and irrational for high-risk tasks. A strong model may be overkill for classification and underpriced for complex research.
The point is not to crown one model. The point is to understand where additional intelligence earns a return.