The prototype delivered its subtitles ten seconds after the voice, and it sometimes invented sentences no one had said. On air, ten seconds is an eternity: the presenter has finished the sentence, the weather has been and gone, and the subtitle is still talking about the previous topic. That is the state I found the project in at an SME that subtitles live broadcasts for major national channels. A project run with energy and good intentions, with everything of a serious project except a usable result: too much delay, too expensive to run, a codebase no one wanted to open again.
The intelligence had been put in the wrong place
The previous project set out to automate what an operator does by hand: condensing a transcript and reshaping it so it fits on screen. The work had therefore gone into that adaptation layer, handed over to generative AI.
The problem was upstream. The transcript feeding that layer was of mediocre quality, and nothing downstream makes up for a sentence that was misheard. Diarisation picks the wrong speaker because the sentence is wrong. The segmentation segments an error. The adaptation neatly rephrases something that was never said, and sometimes fills in the gaps by itself. Garbage in, garbage out, in the most literal sense.
The wow effect is a poor architect
That choice is understandable. In early 2025, generative AI was the most impressive thing you could put on a screen, in that company as everywhere else. A young technology produces an immediate demo effect, and that effect weighs on architecture decisions far more than anyone admits. The question "what does this change for the user" comes afterwards, when it comes at all.
An architecture, though, is decided on constraints: what has to hold in production, what is allowed to fall over, what will be expensive to change later. The newest technology has no reason to sit at the most critical point of the chain, and yet that is where it lands most often, because that is where it shows.
In this project, the generative brick was in the right place on paper: it did the editorial work, the part with the most value. It simply depended on an input no one had qualified. A chain is never worth more than its first link.
Changing the method before changing the technology
My first recommendation was not about a technology. It was about how to go about it: rebuild nothing, stand up a small POC alongside the existing system, and test a single hypothesis. Is an off-the-shelf speech recognition layer, well chosen and well surrounded, enough to produce something clean?
Rebuilding that brick yourself, at the scale of an SME, is an illusion. The specialist players put resources into it that no company of that size will ever gather. The value of a technical SME lies in what it builds around the brick, not in the brick itself.
Four vendors, one decimal place and nothing useful
That left choosing which one. Four solutions in the running, all serious, all with an impeccable spec sheet and a word error rate quoted to the decimal place.
Word error rate is the industry's standard metric. It counts the words that are wrong, missed or added, then divides by the number of words. Simple, comparable, published everywhere. It was unusable here, for a very concrete reason: my quality reference was not transcripts, it was the subtitles the company had actually sent to the control room. A subtitle is two to three times shorter than a transcript, because a viewer reads more slowly than the presenter speaks. Comparing a raw transcript word for word with an adapted subtitle produces nothing usable.
An LLM put in charge of grading the others
So I wrote down what counts on screen, with the people who do the job: proper nouns and figures, the punctuation that carries the meaning, speaker changes, and the severity of each error, because an error that undermines the credibility of what is said doesn't weigh the same as a misplaced comma. Delay is measured separately.
To apply that grid to thirty hours of real streams, thousands of sentences have to be graded. By hand, that is a month of work for a team that already has a channel to keep on air. So I used an LLM as a judge, on that grid, with two safeguards: the criteria are written by humans and for the use, not by the judge, and a sample of its grades was reviewed by hand to check that it graded the way their operators did.
That test bench would have been a project in itself a few years ago. I coded it in a few days, with AI assistance for the test harness, the replay of the streams and the aggregation of the scores. That is what makes this method accessible to an SME: building what you need to test a hypothesis now costs little enough that there is no longer any reason to do without it.
The two results the spec sheets didn't give
The first is about latency. The vendor advertising the best theoretical latency turned out to be the slowest once measured on real streams, by more than a factor of two. A latency quoted per word under laboratory conditions says nothing about how a continuous stream with interruptions behaves.
The second is more interesting. The ranking flips depending on the use. To go straight to air with no intervention, what counts is stability: no visible artefact, no absurd string of digits, a continuous flow. To serve as raw material for an operator who adapts it, what counts is richness: capturing the spoken word finely, the exchanges, the overlaps, even if it has to be cleaned up afterwards. Those are not the same winners. A benchmark returning a single ranking would have led to the wrong decision on one of the two uses.
A primary, a backup, four seconds
The final choice kept a primary vendor and a backup vendor, with automatic failover: a live broadcast gives no warning when it falls over, and subtitling that stops on air is noticed immediately. Latency came down to around four seconds, in assisted mode, with the AI proposing and an operator validating before broadcast.
The invented sentences were fixed by removing what produced them. The setup that was kept chains speech recognition with a correction layer based on business rules, with no generative adaptation layer. A transcript can still get a word wrong, it no longer invents statements.
The real work started after that. Proper nouns are the first pain point of live subtitling, and that is where the differentiation settled: a dictionary prepared upstream on the topics of the programme, enriched continuously from the news wires. The recognition engine itself is swapped by changing one line of configuration.
What this changes in your tool choices
The lesson goes beyond subtitling, and I have applied it to every tool choice since.
A public benchmark tells you who is best in general. It never tells you who is best at your place, on your data, with your constraints. Nor does it tell you for which of your uses, when the same tool can win on one and lose on the other. As long as your criteria are not written down, it is the vendor's spec sheet that decides for you, and it was written to sell.
Four questions before choosing any AI tool, whatever it is:
- What data was it tested on: yours, or the demo's?
- What are the three criteria that matter for your use, and are they written down anywhere?
- Do you have a single use for it, or several that don't ask for the same thing?
- What happens the day it falls over?
That is exactly what the AI & process diagnostic does: one or two use cases tested on your real data, with your criteria, before deciding. If you have a tool choice under way, write to me.
