How do you grade the work fairly when nobody had the requirements in advance? How do you build a common rubric that evaluates the range of solutions delivered when there are whole sections of responses you couldn’t have anticipated?
I thought about that feeling a lot during my time on the COUNTER working group as we tackled AI usage.
The first post of this blog series concluded that the assumptions about content usage in our industry were no longer reliable in the age of AI. This post is about how COUNTER is trying to rebuild the rubric while the ground is still moving.
Why this work was needed
Release 5.1 of the COUNTER Code of Practice, which took effect in January 2025, was approved for publication before the agentic AI discovery boom took off. Every existing COUNTER metric assumes a "usage" event means a human requested and viewed something, but this is an assumption that no longer holds. Chat interfaces for discovery on publisher platforms, assistive search, lay summaries, and retrieval tools that pull in source material to generate a response all produce activity that somebody is counting somehow (hopefully).Without a shared standard for reporting this new type of usage, the consequences are already visible in reports: libraries are seeing their human usage numbers drop even as publishers see raw usage spike. Everybody using COUNTER data to make real decisions is affected, because the numbers may quietly mix human and machine activity with no reliable way to pull them apart.
I joined the working group as a voice for technology providers. AI tools were already deployed. Usage was already being counted, in whatever way each platform had decided on its own. Our job was to write the rubric after the assignments had been turned in.
What the guidance does
The best practice published this month does a few important things at a conceptual level.First, it creates a dedicated lane for on-platform AI tool activity. Platforms that elect to report AI usage can now do so on its own channel, distinct from the human traffic it was quietly impacting.
Second, it defines new metrics that mirror familiar COUNTER concepts but describe what AI systems actually do: generate responses, pull content chunks to synthesize answers, return summaries on request. The parallels are intentional, yet the behaviors are genuinely different.
Finally, it draws careful lines around what doesn't count. Content that an AI system briefly considered and then discarded doesn’t count. Unprompted pre-generated summaries served from a cache don't produce a new event every time someone views one. Only content the on-platform AI tool actually used in its response counts. These exclusions create the difference between a standard that measures something real and one that measures noise.
The guidance is neither perfect nor finished. We are writing definitions for behaviors that are still evolving. But we now have a rubric everyone can be graded against, and comparability is what makes usage data mean anything.
What this means for publishers
Every stakeholder in scholarly publishing is still after the same things they've always wanted. Researchers want relevant, high-integrity content at the right time. Librarians want to buy what their communities need and help them find it efficiently. Publishers want to attract great authors from trusted institutions and make their content discoverable. None of that has changed.What's changing is that the metrics we've used to power those conversations can't carry the full weight anymore. That’s unsettling, but it's also a real opportunity. Shifting what we measure is how we keep doing the things that matter. The COUNTER community was overwhelmingly clear in consultation that reporting AI usage separately and clearly is urgent.
Silverchair's participation in this work is one way we try to be useful to our clients. The analytics we build are only as trustworthy as the standards underneath them, and we’re proud to help shape those standards rather than inherit them.
Solving the harder problem is still to come
The current COUNTER AI best practice is Phase 1 and handles reporting usage for on-platform AI tools. Phase 2 takes on a harder question: when a researcher asks an external AI tool a question and gets a synthesized answer drawn from scholarly sources, how do we measure their usage when they never visit the publisher's site at all?The groundwork from Phase 1 combined with COUNTER's existing guidance on syndicated usage gives us a real foundation. Standards can't solve everything on their own, but they make the next conversation possible.
If you have a perspective, COUNTER runs open community consultations, and I'd encourage you to get involved. Silverchair clients with questions about how AI usage affects their reporting can reach out to their account manager, and we'll carry your perspective forward.
We don't have the rubric finished yet. But we're writing it in public, with everyone in the room. That's a better way to do it than the alternative.
Be the first to hear news by subscribing to our email newsletter.