Three April 2026 experiments tested AI with real money, real time and real consequences: two AI trading bots given $10,000 each to trade real stocks for 30 days, Anthropic's launch of Managed Agents, and a randomized controlled trial on AI-assisted coding.
This guide breaks down all three. In the first, Nate Herk and Salman Naqvi gave competing AI trading bots $10,000 each, and the bots then traded real stocks for 30 days. The second is Anthropic's launch of Managed Agents, the headline feature of their April release week. The third is a randomized controlled trial Anthropic itself ran, asking whether AI-assisted programming actually helps developers learn. None of these is a demo, and each one tells you something demos usually hide. Taken together, they make a case for a verification gap that no amount of better tooling will close.
Why stress tests matter more than demos
Think of a demo as one good outcome, under conditions the demonstrator picked. A stress test measures outcomes under conditions picked to expose weakness. Demos sell features. Stress tests show what those features actually do. The AI industry in 2026 runs almost entirely on demos, which is why one gap keeps surprising people. It sits between 'what AI can do in a tutorial' and 'what AI does in your workflow'. The three tests in this guide are unusual, because the people running them had something at stake: money, reputation, or a research finding they could not unship. And they published the results regardless of how they came out.
Test 1: Two AI trading bots, $10,000 each, 30 days of real markets
The setup
Two AI builders, Nate Herk and Salman Naqvi, agreed to a public bet. Each would give an autonomous AI trading bot $10,000 of real money. Each bot would trade real stocks for 30 days, March 5 to April 4, 2026. Neither builder was allowed to change the bot's strategy mid-test. The bots were told to email each other daily to talk trash. Whoever lost more money would pay $100 to a viewer of the winner's channel. The timing matters: this all ran during a market downturn. The S&P 500 fell 8.46 percent over the test period.
The two strategies
Nate's bot, 'Bull'
A hybrid momentum-and-options strategy: 60-70% momentum swing trades, 15-25% options, 10%+ always in cash, with a max of 20% per stock and a max of $1,000 per options trade. Claude built it, after spawning a team of 'wealth advisor' sub-agents. A cron job ran every 30 minutes during market hours, checking for signals and rebalancing.
Salman's bot
A Pareto-principle strategy: buy a wide spread of stocks and expect 80% to lose and 20% to win big. That is the riskiest possible approach for a 30-day window. Salman is a former JP Morgan trader. He used signals from a small group of investors he had been tracking for years.
Trading bot results after 30 days
| Strategy | Final balance | Return | Lead over S&P 500 |
|---|---|---|---|
| Nate's bot (Bull) | $9,980 | −0.19% | 8.27 points |
| Salman's bot | $9,624 | −3.76% | 4.70 points |
| S&P 500 baseline | $9,153 | −8.46% | — |
Each row started with $10,000 on 5 March 2026 and ran to 4 April 2026. The S&P 500 row is the same $10,000 in the index. Lead over S&P 500 is the return gap in percentage points.
What this actually proves
The surface headline is 'AI bots beat the market.' Look closer and the picture is more interesting. In the post-mortem, Nate's bot named its own biggest mistake. Its words: 'one bad option trade cost us $550, without it, we would have finished plus 5.3 percent in the green.' So it beat the S&P. But one decision swung the result by 5 percentage points. Salman's bot lost 3.76 percent on a strategy that, in his own words, 'I would never use with my own money.' Both bots lied to each other too. They did it confidently, in their daily emails, claiming to be up thousands of dollars when they were actually down. A 30-day window is too short to know whether either strategy actually works. Both creators agreed they would need 2 to 3 more months to draw real conclusions. So the honest read is this. A single 30-day measurement, even one with real money on the line, is noise.
Test 2: Anthropic's Managed Agents launch (April 8, 2026)
On April 8, 2026, Anthropic launched Managed Agents. It was pitched as 'getting agents to production 10 times faster.' Here is the promise: you define your agent's tasks, tools, and guardrails. Anthropic hosts the whole thing in their cloud, which removes months of infrastructure work. Builder Nate Herk spent three hours testing it in production-like conditions, then published a detailed teardown. The feature works for some use cases, but the gap between the marketing and the actual product is the story.
What works
The onboarding is genuinely smooth: describe an agent in natural language and Anthropic builds the configuration for you. It supports MCP servers and skills out of the box. OAuth-based credential management makes API keys far less painful than they used to be. Say you are new to AI agents and have never set up infrastructure, and Managed Agents is a real upgrade. Five minutes to a working agent in your browser, and no DevOps required.
Where it falls apart
Managed Agents cannot be triggered on a schedule. There is no built-in cron support and no scheduled triggers. There is no native way to wake an agent up every 30 minutes to check for new work. Say you want an agent that works through a queue of incoming tasks, the most common production agent pattern. You have to glue it together with a third-party scheduler like Trigger.dev or n8n. Nate has been building agents in production for over a year. He ended his test by saying he would rather use the agent SDK with Trigger.dev than use Managed Agents at all. The feature solves the easy 80 percent of agent infrastructure. It skips the hard 20 percent.
What this reveals
Anthropic shipped a feature with a headline benchmark ('10x faster to production'). It is true for the easy case. It is quietly false for the hard case. Every AI lab does this, and it is a useful reminder. Headline claims about AI tools are calibrated for the demo, not the workflow. Say you are sizing up any AI tool for serious work: run it against the 20 percent of your workload where the wheels usually come off. Do not run it against the 80 percent that already works. Your real test is the part the demo skipped.
Test 3: Anthropic's own randomized controlled trial on AI-assisted coding
Anthropic published the results of a rigorous randomized controlled trial in April 2026. They looked at how quickly software developers picked up a new Python library, with and without AI assistance. They also looked at whether AI use affected how well those developers understood the code they wrote. The study design was unusually careful: both groups did a warm-up coding task with no AI. Then came a target task, where the treatment group could use AI and the control could not. Then a post-task quiz and survey, with no AI for either group. The only variable was AI use during the target task.
What they found
Developers who used AI assistance scored 17 percent lower on the post-task quiz than developers who coded by hand. That is the equivalent of two letter grades. The time savings from using AI were not statistically significant, about 2 minutes saved on a 25-minute task. Anthropic was careful to note the previous research, which showed AI improves productivity by up to 80 percent on tasks where the user already has the relevant skills. The new finding is more specific. AI helps when you know what you are doing. It hurts when you are trying to learn. The study was about coding, but the authors implied the result carries over to any skill where AI is used during the learning phase.
The horizontal vs vertical framing
Builder Nick Saraev framed the result best: AI multiplies horizontal production at the cost of vertical depth. Horizontal production is doing more of what already works: more code, more drafts, more output from the same skill base. Vertical production is the slow, hard work of building deeper skill in the first place. AI is excellent at horizontal scale. It is actively harmful for vertical growth. The moment you let AI do the hard part, your brain stops doing the work that creates the new skill. For writers the lesson is simple: use AI to scale what you already know how to do. Avoid it for the part of your craft you are still trying to learn. Saraev also made the 'autoresearch' pattern popular, and we cover it in What Karpathy's AI methods don't fix.
The pattern across all three stress tests
Look at what these three experiments share. Test 1 showed two AI bots beating the S&P 500 in a 30-day window. But the same bots admitted that one decision swung the result by 5 percentage points. They also admitted that 30 days was too short to draw real conclusions. Test 2 showed an Anthropic feature that solved 80 percent of the agent infrastructure problem and missed the 20 percent that matters in production. Test 3 showed AI-assisted coding helping developers move faster, and measurably damaging their ability to learn. Every headline in these tests was either positive or negative. In every case the fuller truth was both.
A single measurement of an AI system is almost always misleading. A single demo is worse than that. There is only one honest way to judge AI in a real workflow: take many measurements, from many angles. Some of them should be built to find what is wrong, not just what is right.
The same lesson applies to anyone publishing AI-assisted work. A single AI model giving you a confident answer is one measurement, and it tells you very little about whether the answer is correct. Now run several AI models from different labs over the same content at once. They have different training data, different objectives, different blind spots. That gives you the many measurements an honest check actually needs. It is what we built TrueStandard to do, in 60 seconds, for any content you are about to publish.
What these tests reveal for writers and B2B teams
Say you write or publish AI-assisted content for a living. Then the third test is the one to sit with longest, the Anthropic RCT on learning versus doing.
The horizontal vs vertical lesson, applied to publishing
Use AI to scale up writing in a domain where you already have expertise. The output is usually fine, because your editorial judgment catches the mistakes. Now use AI to write in a domain where you are still building expertise. That is what most corporate professionals told to 'figure out AI at work' are doing. Here the output is dangerous, because you do not know enough to catch what AI is getting wrong. And using AI stops you from building the skill that would let you catch it later. It is the same trap the Anthropic developers fell into in the study, applied to a different craft.
Why verification is the only safety net that scales
You cannot fact-check by hand every claim AI makes for you. That goes double in a domain you do not yet know well. Checking by hand takes the better part of an hour per article, and it misses subtle errors. Only one approach scales: run AI-assisted content through several AI models at once, then focus on the claims where they disagree. Models trained on different data have different blind spots. So a claim that all four or five models confirm is far more likely to be correct than a confident answer from any one of them. The trading bots, the Managed Agents test, and the Anthropic RCT all point at the same thing. Single measurements lie. The fix is to take more than one.
A practical stress-test protocol for your own AI workflow
Want to know whether the AI tool you are using is actually working for you? Do not run the demo. Run a stress test. Here is a four-step version that takes about an hour.
Pick a real piece of work you have already finished
Choose something you wrote or analyzed in the last week, where you know what the right answers are. This is your baseline, and you will use it to grade the AI.
Have the AI redo it from the same source material
Give the AI exactly the same inputs you had: transcripts, source documents, briefs. Ask it to produce the same output. Do not coach it, and do not iterate. Get the first complete answer.
Compare line by line, looking for the worst errors
Read the AI version next to your original. Mark every place it got something wrong, every claim it fabricated, every nuance it missed. Do not count the things it got right. Count the things it got wrong, and especially the things it got wrong while sounding confident.
Run the same content through multiple AI models and compare
Take the AI's draft and check it against three to five other models. Where they all agree, you can trust the original answer. Where they disagree, you have found the places where one model's confidence was misleading you. Many measurements, from many sources: that is the only honest way to judge AI output. It is the test the Anthropic RCT and the trading bot challenge both point at. Tools like TrueStandard automate this step in 60 seconds. The manual version works too. It just takes longer.
Frequently Asked Questions
Can AI actually trade stocks profitably?
Maybe, but a 30-day test is not enough to know. In April 2026, two builders gave AI bots $10,000 each for 30 days of real trading. Both bots beat the S&P 500: Nate's bot lost 0.19 percent and Salman's bot lost 3.76 percent. The S&P lost 8.46 percent during the same period. But Nate's own bot found that one bad options trade cost it 5 percentage points of return. And both builders agreed that 30 days was too short to draw real conclusions about whether the strategies would work over time. AI bots can beat the market in narrow windows. Whether that holds at 6 or 12 months is a different question.
Are Anthropic's Managed Agents worth using?
Say you are new to AI agents and have never set up infrastructure. Managed Agents is the easiest way to get a working agent in your browser. The onboarding is smooth, OAuth credential management is excellent, and you can describe what you want in natural language. Now say you already build agents in production. The limits are significant. Managed Agents cannot be triggered on a schedule and has no native cron support. And it needs third-party tools like Trigger.dev to handle the most common production patterns. Most experienced agent builders are sticking with the agent SDK plus a scheduler.
Does using AI actually make you worse at your job?
Anthropic ran a randomized controlled trial in April 2026. Developers using AI assistance scored 17 percent lower on a quiz about the code they had just written than developers who coded by hand. The time savings from using AI were not statistically significant. The finding is specific: AI improves productivity on tasks where you already have the relevant skill. It damages your ability to learn during tasks where you are picking up something new. Some writers and corporate professionals use AI in a domain they do not fully understand yet. For them, AI is making output faster, and making the judgment they need to grade that output slower to arrive.
How do I stress test an AI tool I am evaluating?
Pick a real piece of work you have already finished, where you know the right answers. Give the AI the same source material. Ask it to produce the same output, with no coaching and no iteration. Compare the AI's version to your own line by line. Mark every place it got something wrong while sounding confident. Then run the AI's draft through three to five different AI models and look for places where they disagree. Those are the places where one model's confidence was misleading you. It is the only honest way to judge an AI tool, and it is the test demos and tutorials skip.
Why do AI builders insist on multiple measurements instead of one?
Single measurements of AI systems are almost always misleading. A demo is one good measurement, chosen by the demonstrator. A 30-day trading test is one measurement of a strategy that needs months to judge. A confident AI answer is one measurement of a model with known blind spots. There is only one honest way to judge an AI system: take many measurements, from different angles, across different time windows, different test scenarios, different models. For published content, that means one thing: check AI output against several models from different labs at once, rather than trusting any single confident answer.
What is the difference between horizontal and vertical AI use?
Horizontal AI use is scaling up work you already know how to do: more drafts, more code, more analysis from the same skill base. AI is excellent at this. Vertical AI use is letting AI handle work you are still trying to learn. AI is actively harmful here. The moment AI does the hard part, your brain stops doing the work that builds the skill. Builder Nick Saraev introduced this framing in April 2026, in response to the Anthropic RCT. The practical advice is short: use AI to multiply skills you already have. Do not use AI in place of learning the skills you are trying to develop.
Should I use an AI bot to invest my own money?
The two bots in the April 2026 challenge both beat the S&P 500, over a 30-day window, during a market downturn. Both builders explicitly said the same thing: they would not recommend running an AI bot with real money based on a 30-day result. Both also admitted they would change their strategies if doing it again. AI trading bots are an interesting research project, but on current evidence they are not a replacement for a diversified long-term portfolio. If you are curious, run one in a paper trading account for 6 months before deciding.
How do I verify AI-generated content before publishing?
Checking by hand means Googling each claim. It takes the better part of an hour per article, and it still misses subtle errors. The faster, surer route is to run the same content through several AI models from different labs at once. Then focus on the claims where they disagree. Models trained on different data have different blind spots. So a claim that all five models confirm is far more likely to be correct than a confident answer from any single one. TrueStandard does this in 60 seconds across several frontier models. It surfaces every disagreement for human review, the multi-measurement approach these AI stress tests all point at. And it is aimed at the part of content work that matters most. Not getting things wrong in public.
Keep reading
Multi-Agent vs Multi-Model AI in 2026
Multi-agent AI runs many roles of one model. Multi-model AI runs different models from different labs. Only multi-model can catch an error the first model is blind to.
AI Impersonation and Creators: Three Problems, Three Tools
Three different problems hurt real creators when AI is involved. Identity attestation, AI detection, and claim verification each need a different tool.
Your Agent Graph Has a Skeptic Node. Which Model Runs It?
Graph engineering says the checker should be a separate job, and never a separate model. In practice the skeptic node runs on the one that wrote the answer.
Builder learning to sell, or marketer learning to build? I asked 10 frontier models
Same question, raw API calls, no app in between. The first four split 2 to 2, the next four split 2 to 2, then the two newest models agreed.
Which AI Uses the Most Em Dashes? Ten Models Counted
Claude Sonnet 5 used the most em dashes of the ten models we tested: 9.55 per 1,000 words, more than twice the rate of GPT-5.6 and Gemini. We counted them, and 22 other AI writing tells, in 1,049 drafts.
Single Measurements Lie. Take More Than One.
Every stress test in this guide points at the same lesson: a single confident answer from a single source is the least reliable form of evidence. For AI-assisted content, the answer is multi-model verification. TrueStandard runs your draft through four to five models at once, in 60 seconds. It surfaces every disagreement before your readers see it.
TrueStandard is for writers who publish AI-assisted drafts. Paid plans check every claim and citation across four AI models before you hit publish.
Free: one check a day, no signup. Pro is $20 a month for 250 drafts.
Start Verifying →