What 71 AI Configurations Taught Me About My News Pipeline

🔍 Read the full analysis: What 71 AI Configurations Taught Me About My News Pipeline on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get hardware and tech essentials delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says he tested 71 AI configurations on real news sources between Sept. 8 and Oct. 13, finding that GPT-6 Astra scored highest while the much cheaper GPT-6 Luna was the model he chose for production. The test also led him to remove two fallback models after they produced invented quotations or unusable articles, a finding relevant to publishers relying on automated news workflows.

The test used the same production prompt Meyer says his news pipeline uses, with two source texts: a 6,340-character product filing and a 619-character political news item. Articles were generated in English and German. Two AI judges, GPT-6.1 Sol and Claude Opus 5.5, assessed source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, along with German language quality.

Meyer reports that GPT-6 Astra received 4.00 out of 5 in both languages in the standard test, with a perfect source-fidelity score there. In a follow-up robustness run covering 12 articles, it averaged 3.93. But he estimates Astra costs about $0.12 per article, compared with roughly $0.0012 for GPT-6 Luna. Luna scored 3.67 in English and 3.68 in German in the initial test; in the robustness run, it scored 3.67 and every article was valid under the pipeline’s output requirements.

The report says raising Luna’s reasoning effort did not produce a reliable quality gain. At the maximum setting, an article took 211 seconds and cost 1.43 cents, compared with 17 seconds and 0.12 cents at the no-effort setting. Meyer says the score differences were within the test’s noise range. He also reports that GPT-4.1 nano scored 1.62 in English and 1.68 in German, with a source-fidelity score of 1.00, while locally run gpt-oss-120b produced zero valid articles out of four.

Cost and Reliability in Production

The results point to a practical trade-off for publishers: the model with the highest writing score may not be the best fit for a high-volume workflow. Meyer says Astra’s roughly 100-times-higher per-article cost led him to use Luna instead, whose reported quality was lower but consistent enough to meet his production needs. Those figures are specific to his setup and should not be treated as a universal ranking of models or prices.

The fallback results highlight a separate operational risk. A primary model can perform well while a backup quietly introduces unsupported quotations or fails formatting checks. Such errors can make automated content unreliable even if the main model is capable. Meyer’s account suggests that testing the full route, including failure handling, matters alongside comparing headline model scores.

The report also cautions against assuming that strong performance in one task carries over to another. Meyer says Claude Opus 5.5, which he uses for coding and which served as one of the judges, scored 3.46 in English and 3.39 in German as a news writer. His finding is not an independent industry-wide evaluation, but it illustrates why publishers may need tests built around their own sources, prompts and editorial requirements.

How the Pipeline Was Tested

Meyer’s report describes a test intended to resemble his daily article-production task, rather than a general benchmark of model intelligence. It covered API services, subscription models, free-tier options accessed through OpenRouter and open-weight models run on his hardware. Each configuration generated both English and German versions from the same source material.

The source set was small in the first round: one long filing and one short political item. Meyer says short sources make up much of his production work, so he considered the 619-character item the more representative challenge. He then ran the strongest configurations on 10 additional real sources, mostly short ones. The reported scores include 95% bootstrap intervals, and Meyer says differences below about 0.3 points across the two initial articles should be treated as noise; he considers the larger follow-up run more useful for ranking.

For an article to count, it had to produce valid output after the same repair step used in production. Meyer says the test prompted a route change on Oct. 7: he removed GPT-4.1 nano and gpt-oss-120b as fallbacks. The report’s final sentence about the replacement route is incomplete, so it does not establish which backup, if any, now follows Luna.

“Public leaderboards tell you which model is smartest. They don’t tell you which model should write your articles.”

— Thorsten Meyer, in the report

Limits of the Model Comparison

The reported results come from one publisher’s pipeline, prompt, two initial sources and selected follow-up sources. They do not establish how the configurations would perform for other newsrooms, topics, languages, source formats or editorial standards. The judges were also AI systems, and the supplied report does not describe an independent human scoring panel or provide article-level results for every configuration.

It is not clear from the report whether the cost estimates include all operational expenses, such as subscriptions, infrastructure, retries or editorial review, or how prices may change over time. The source also ends mid-sentence while describing the new production route, leaving the current fallback arrangement unconfirmed. The findings are Meyer’s account of his own test, not a broader certification of any model’s reliability.

Further Testing in Production

Meyer says GPT-6 Luna remains in production and reports a median generation time of 17 seconds per article over the past week, at a cost of about $0.0013 per article. His next practical step is continued use of that configuration and monitoring its output against real sources; the report does not announce a further test date or a planned expansion of the benchmark.

For readers evaluating similar systems, the reported method points to checks that could be repeated: test on representative short and long sources, generate every target language, verify quotations against source text and measure whether the output is usable after normal repair steps. Any later results, changes to the fallback route or evidence from a larger human-reviewed sample would help clarify how well these findings generalize.

Key Questions

Which model scored highest in Meyer’s test?

Meyer reports that GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across 12 articles in the robustness run.

Why did Meyer choose GPT-6 Luna for production?

He says Luna delivered consistent, usable articles at about $0.0012 per article, roughly one-hundredth of Astra’s reported $0.12 cost. Those prices reflect his setup and may not apply to other users.

What happened when reasoning effort was increased?

Meyer reports no reliable improvement across Luna’s six effort settings. The maximum setting took 211 seconds and cost 1.43 cents per article, versus 17 seconds and 0.12 cents at the no-effort setting.

Why were two fallback models removed?

According to Meyer, GPT-4.1 nano received a low source-fidelity score and generated unsupported quotations flagged by the judges. The locally run gpt-oss-120b produced no valid articles in four attempts and also generated named quotations he says were invented.

Can these results rank models for every newsroom?

No. They describe a test using Meyer’s prompt, pipeline, sources and judges. The report does not establish how the models would perform under other editorial workflows, source types or cost arrangements.

Source: ThorstenMeyerAI.com