Every transaction enrichment vendor says their categorization accuracy is high. The number is usually somewhere between 85% and 97%, and it is presented with confidence on the marketing page. What these numbers rarely say is: accurate compared to what? Measured on which dataset? Evaluated against whose labeling standard?
This is the ground-truth problem in transaction categorization, and it is worth understanding before you evaluate any vendor, including us.
The problem is not that vendors are lying. It is that there is no neutral, universally agreed-upon answer to the question "what category does this transaction belong to?" Accuracy metrics in machine learning require a labeled test set to measure against, and in transaction categorization, the construction of that labeled test set involves subjective decisions at almost every step.
Why Ground Truth Is Not Obvious
Consider a transaction from a large pharmacy chain, say CVS or Walgreens. The raw string is something like CVS/PHARMACY #7841 BOSTON MA. What is the correct category?
Options a reasonable person might choose: Health and Wellness, Personal Care, Shopping (General), Household Supplies, Groceries (for pharmacy locations that sell food), or Medical (for prescription pickups specifically). All of these are defensible depending on your category taxonomy and what you believe the user is buying.
Now imagine your labeled test set has this transaction marked as "Health and Wellness" and your model assigns it "Personal Care." Is the model wrong? Maybe. But maybe "Personal Care" is a sensible subcategory under "Health and Wellness" in a different taxonomy design. The "error" depends entirely on how you drew the category boundary.
This kind of boundary ambiguity exists across hundreds of merchant types. Big-box stores like Walmart and Target sell groceries, electronics, clothing, and household supplies in the same transaction. Gas station purchases include fuel, snacks, and sometimes automotive parts. Online marketplaces aggregate across almost every category. The transaction string alone does not tell you what was purchased.
How Vendors Construct Test Sets (and Why It Matters)
When a vendor reports "94% categorization accuracy," there are at least four things you need to know to evaluate that claim.
First, what is the test set composition? A test set heavy on top-100 merchants will always score higher because those merchants have unambiguous strings and well-established categories. A test set that accurately reflects the long tail of merchant diversity (which is where real user transactions actually live) will score lower for any system.
Second, who did the labeling? Human labelers have their own category intuitions. Two labelers looking at the same transaction will disagree on category more often than you might expect, particularly for ambiguous merchants. Inter-annotator agreement in transaction labeling tasks is typically in the 75% to 85% range before adjudication, which means there is substantial inherent uncertainty in any labeled dataset.
Third, whose category taxonomy was used? Accuracy of 94% against taxonomy A might be 78% against taxonomy B if the category definitions differ. A vendor whose model was trained on one schema will score best when tested against that same schema.
Fourth, when was the test set collected? Merchant strings change over time. A test set from two years ago may not reflect current payment processor formats, new merchant entrants, or changes in how banks report transaction data.
How We Think About Ground Truth at Openaggr
We have made a few specific choices about how we handle this problem, and I want to be transparent about what they are rather than just claiming a number.
Our primary accuracy metric is not a single number against a fixed test set. We track category confidence distributions and monitor for confidence drops on specific merchant types over time. A confidence score of 0.92 means the model is highly certain about its assignment. A confidence score of 0.61 means the assignment is a best guess and the calling application should handle it differently, perhaps by deferring to user correction or surfacing a secondary candidate category.
We maintain labeled test sets at three coverage levels. The high-coverage tier covers the top 2,000 merchants by transaction volume across our user base; these have been labeled by multiple annotators with adjudication on disagreements. The mid-coverage tier covers merchants ranked 2,001 through 15,000 and is labeled by a single annotator with spot-check review. The long-tail tier is a random sample of transactions from merchants outside the top 15,000; this tier is deliberately kept at limited size because systematic labeling of the long tail would require resources that would distort our team's priorities.
We measure separately across all three tiers because the numbers are genuinely different. Accuracy on the high-coverage tier is higher. Accuracy on the long tail is lower. Reporting a blended number without disclosing the tier composition would be misleading.
The Feedback Loop Problem
The other half of the ground-truth problem is ongoing: how do you keep your labeled set current as merchant strings change and new merchants appear?
User corrections are the most direct signal. When a user recategorizes a transaction in a PFM app, that is a labeled example. But user correction data is noisy: users apply personal logic (treating a purchase at a pet supply store as "Household" rather than "Pets" because that is how they budget), they make typos when typing custom categories, and correction rates vary heavily by user engagement level.
We use correction signals as one input, weighted against their consistency. A correction that conflicts with 200 prior consistent assignments for the same merchant type gets a low weight. A correction that matches a pattern we see across many users for a specific merchant type gets a higher weight and triggers a review of that merchant's category assignment.
We are not saying user corrections are wrong to use. We are saying that raw correction counts are not equivalent to ground-truth labels, and treating them as such leads to model drift toward idiosyncratic user preferences rather than toward more accurate categories.
A Practical Framework for Evaluating Categorization Quality
When a developer or product team is evaluating an enrichment API, here is a more useful evaluation framework than asking for a top-line accuracy number.
Run your own test set. Take 500 to 1,000 real transactions from your user base (anonymized, obviously), label them according to your own category taxonomy, and run them through the API. Compute accuracy yourself against your own labels. This tells you how the system performs on your specific user population's spending patterns, which is the only number that actually matters for your product.
Check coverage separately from accuracy. What percentage of transactions come back with a category at all (coverage), versus what percentage of those categorized are correct (precision)? A system that only categorizes when highly confident might have impressive precision but poor coverage. A system that categorizes everything might have higher coverage but more errors. Neither is universally better; the right tradeoff depends on your use case.
Look at the confidence score distribution. If a system returns uniformly high confidence scores on everything, that is a red flag. Confidence scores should reflect genuine uncertainty. On genuinely ambiguous transactions, a well-calibrated system should return lower confidence, and you should be able to use those scores to decide how to handle the output in your UI.
Test on your long tail. Pick 100 transactions from merchants that appear only once or twice in your dataset. These are the most realistic test of how a system handles novelty. Any lookup-table-based system will fail here. A system that generalizes from description signals should do better, but that quality should be verifiable, not assumed.
What We Report and What We Do Not
We will tell you our weighted accuracy across our test tiers, broken down by coverage level. We will tell you the inter-annotator agreement rate on our labeled set, which gives you a sense of the inherent ceiling on what is measurable. We will tell you the average confidence score on different transaction types.
What we will not do is claim a single top-line number without context, because that number would obscure more than it reveals. The ground-truth problem in categorization is real, and the right response is to be specific about methodology rather than to paper over the ambiguity with a marketing statistic.
If you want to run your own evaluation against your transaction data, the sandbox environment supports batch enrichment with confidence scores returned on every transaction. That is the most honest way to assess fit for your use case.