The hypothesis

The claim is everywhere: AI can write better email subject lines than humans. Tools promise higher open rates. Case studies quote impressive numbers. I wanted to know if that held in a simple, controlled test — not enterprise-scale with sophisticated AI tools, but the kind of setup a solo marketer or small team can actually run.

The test: same email content, same list, same send time. Two subject lines — one written by Claude following a structured prompt, one written by me manually. Sent to a split list. Winner determined by open rate. 60 days, two sends per week.

📋 Test Parameters

Duration: 60 days, January–March 2026. Send frequency: 2× per week. Total sends: 120 (60 per treatment). List size: small — under 800 subscribers. AI tool: Claude. Prompt used: "Write an email subject line for [topic]. Under 50 characters. Make it specific, curiosity-driven, and avoid spam trigger words." No editing of AI output before sending.

The headline numbers

120
Total sends tested
34
AI wins
22
Human wins
4
Too close to call

AI won 34 of 60 A/B tests. Human won 22. 4 were within margin of error. That's a 57% win rate for AI — meaningful, but not dominant. The average open rate delta when AI won: +4.1 percentage points. When human won: +6.3 percentage points.

The most important finding: when the human-written subject line won, it won bigger. When AI won, the margins were smaller.

What AI did better

Consistency. My manually written subject lines had wider variance — occasional big wins, occasional embarrassing underperformers. AI's results clustered tighter. The floor was higher; the ceiling was lower. For a marketer who wants reliable, predictable performance, AI is genuinely useful here.

Speed-to-competent. Writing a good subject line manually takes 5–10 minutes of deliberate work. Getting a competent AI subject line takes 30 seconds. At scale, that's not a marginal saving.

Specific content types. AI outperformed on newsletter sends with data-driven angles. Prompts like "write a subject line for a roundup of 5 AI tools tested this week" produced consistently strong results — better than what I'd write under time pressure.

Where human won

Anything requiring cultural context. Sends timed to specific events, news cycles, or moments in the reader's professional calendar produced my best-performing subject lines. AI doesn't know what just happened in marketing Twitter this week. It can't leverage the specific cultural moment. I can.

Personality and voice. My single highest-performing subject line of the 60 days was deliberately weird — a two-word line that only made sense if you'd read last week's edition. AI won't take that risk unprompted.

ScenarioWinnerAvg delta
Data/stats-led contentAI+4.8pp
Tool roundupsAI+3.9pp
News-reactive sendsHuman+7.2pp
Personal story / experimentHuman+5.8pp
Evergreen how-to contentTie<1pp

The practical recommendation

Use AI for subject lines on data-led and tool-review sends. Write them yourself for news-reactive, story-driven, and audience-specific sends. The hybrid approach almost certainly outperforms either in isolation.

The other takeaway: brief quality matters as much here as anywhere else. The prompt I used improved steadily over 60 days. The AI subject lines in week 8 were noticeably better than week 1 — same model, better briefing.

Frequently Asked Questions

Does AI really write better email subject lines than humans?
On average in this test: slightly yes, but not dramatically so. AI won 57% of A/B tests on a small list. The real advantage is consistency and speed — AI's floor is higher than an average human writing under time pressure. The ceiling, however, belongs to humans who can leverage cultural context and genuine voice.
What AI tool should I use for subject lines?
Claude, ChatGPT, and Gemini all produce competent subject lines with good prompts. Dedicated tools like Anyword offer predictive performance scoring — useful at volume. For a small list, free-tier Claude or ChatGPT with a good prompt will match any paid tool.
What's the right prompt for AI email subject lines?
The prompt that worked best: "Write 5 email subject lines for [topic]. Under 50 characters each. Angle: [curiosity / data-led / direct]. Audience: marketing professionals. No spam trigger words. No clickbait. Make the benefit or insight explicit." Then pick the strongest of the 5.
How large does a list need to be for A/B testing to be valid?
For statistically significant results, most email platforms recommend a minimum of 1,000 subscribers per variant. This test used under 800 total — meaning the results are directionally useful but not statistically robust. Larger lists will show cleaner signal. The patterns observed here are consistent with broader industry data on AI subject line performance.
Should I A/B test every email subject line?
Not necessarily. On a small list, A/B testing produces noisy data. A better approach for small lists: run AI and human subject lines for 8–10 sends each, compare averages, then use whichever performs better for your content types going forward. Continuous A/B testing produces better learning on lists above 2,000 subscribers.