📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems are rapidly automating core engineering tasks in AI research, with benchmarks reaching saturation. Research remains less automated, but progress suggests it may also become increasingly mechanized soon.
Recent empirical data confirms that AI systems are now capable of automating most core engineering tasks involved in AI research, with benchmarks approaching saturation. This shift significantly reduces the role of human engineering in AI development, leaving research as the residual challenge, according to Thorsten Meyer’s analysis of Jack Clark’s recent essay.
Thorsten Meyer’s review of Jack Clark’s recent essay highlights that three independent benchmarks—CORE-Bench, MLE-Bench, and kernel design—are nearing saturation, with AI systems achieving performance levels previously thought to require human expertise. For example, CORE-Bench, which measures the reproduction of research code and experiments, has reached 95.5%, with its author declaring it ‘solved.’ Similarly, MLE-Bench, evaluating Kaggle competition performance, has hit 64.4%, surpassing mid-tier human performance in some cases.
These advances imply that reproducing existing research or performing engineering tasks at scale is now an engineering problem rather than a research challenge. The bottleneck has shifted from capability to application, with AI systems handling dependencies, code execution, and optimization routines at levels comparable to or exceeding human experts. Meanwhile, progress in kernel design—such as automated GPU kernel generation—further indicates that AI is moving toward production-grade automation in infrastructure tasks.
Clark’s analysis suggests that while AI can automate the engineering side of research, the more creative, hypothesis-driven aspect of research remains less automated. However, the structural pattern of rapid, overlapping progress across multiple domains raises questions about how long research itself will remain a residual challenge before it, too, becomes mechanized.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
AI Tools for Finance and Accounting Professionals: Automate Tasks, Save Hours, Work Smarter
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
GPU kernel automation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
AI code reproduction software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
automated machine learning platforms
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications for AI Development and Research Workforce
The rapid automation of core engineering tasks in AI R&D implies that the traditional bottleneck—manual engineering—may soon be eliminated, drastically changing the landscape of AI development. This shift could reduce the need for large engineering teams, accelerate research cycles, and lower costs. However, it also raises questions about the future role of human researchers and the nature of scientific innovation, which may become more dependent on creative and hypothesis-driven work that AI has yet to fully automate.
Progress in AI R&D Benchmarks and Infrastructure
Over the past 18 months, multiple benchmarks measuring AI capabilities in research reproduction, Kaggle competitions, and kernel design have shown consistent progress toward saturation. Notably, CORE-Bench, which tests the ability to reproduce research experiments, has reached near-complete performance levels, with some experts declaring it ‘solved.’ Similarly, MLE-Bench, assessing competitive ML performance, has surpassed mid-tier human levels, prompting the leaderboard to pause submissions for fairness adjustments.
These developments reflect a broader pattern of AI systems increasingly handling tasks that once required human expertise, particularly in engineering and infrastructure design. The progress in kernel design, including automated GPU kernel generation, further signals that AI is moving toward automating production-grade infrastructure work.
“The pattern across multiple benchmarks indicates that AI has automated the core engineering skills necessary for AI research, leaving research as the residual challenge.”
— Thorsten Meyer
Unresolved Questions About Research Automation
It remains unclear how much of the research process—hypothesis generation, creative problem-solving, and scientific insight—can be automated in the near term. While engineering tasks are approaching full automation, the residual challenge of research may also diminish faster than expected, but this is not yet confirmed.
Next Milestones in AI R&D Automation
Expect continued rapid progress in automation benchmarks, with potential breakthroughs in automating hypothesis generation and scientific discovery. Researchers and institutions may need to adapt to a landscape where engineering is fully mechanized, shifting focus toward managing AI-driven research processes and exploring the remaining creative aspects. Monitoring updates from ongoing benchmark evaluations and infrastructure development will be key over the next 18-24 months.
Key Questions
What does the automation of engineering mean for AI research teams?
It suggests that many traditional engineering tasks—code reproduction, infrastructure optimization, and experiment replication—can now be handled by AI, potentially reducing the size and cost of research teams and accelerating development cycles.
Are there limits to what AI can automate in research?
While engineering tasks are nearing full automation, the ability of AI to generate novel hypotheses, interpret results creatively, and drive scientific innovation remains less certain and is likely the next frontier.
How reliable are the current benchmarks as indicators of progress?
The benchmarks are increasingly saturated and are widely regarded as reliable indicators of AI’s capabilities in specific research and engineering tasks, though they may not fully capture the creative or conceptual aspects of research.
What are the potential risks of fully automating AI research tasks?
Risks include reduced human oversight, potential loss of scientific diversity, and challenges in ensuring AI-generated research aligns with ethical standards and long-term goals.
Source: ThorstenMeyerAI.com
