Working with AI

Why AI Still Needs a Human: How I Use and Check AI in Client Work

Six months on, the tools are sharper and more often wrong, and checking their work has become a bigger part of mine. How I use, and check, AI in client work.

1 Aug 2025 · 5 min read · By Sophie

Why AI Still Needs a Human: How I Use and Check AI in Client Work

Six months on, the tools are sharper and more often wrong, and checking their work has become a bigger part of mine.

Earlier this year I wrote a piece asking whether AI would replace Digital Operations Partners. My answer was no, with the caveat that it would change the job. Several months later the tools have improved in some ways and got worse in others, and the part of the work that doesn't transfer to AI has grown.

The newer models are more capable, and more often wrong

Some of the newer AI tools are less reliable than the ones they replaced. They hallucinate more often, with more confidence and in better prose.

OpenAI's own April 2025 system card for its reasoning models, o3 and o4-mini, showed o3 hallucinating on 33% of PersonQA questions and o4-mini on 48%. On the SimpleQA benchmark, o4-mini hallucinated 79% of the time. The previous reasoning model, o1, hallucinated on 16% of the same PersonQA test. OpenAI said that "more research is needed" to understand why.

I use four tools across a working week, and each fails in its own way. Claude is strong on long-form work and structured analysis, and I've caught it inventing statistics plausible enough to believe if you didn't already know the answer. ChatGPT is fast and good for first drafts but unreliable with numbers and percentages. Omni in Airtable spots patterns across structured data and gets the actual calculations wrong. I used Perplexity for research because it showed its sources, until I moved away from it (the ethical AI alternatives piece explains why).

In the last six months I've seen AI produce project summaries that were polished, confident and almost entirely fabricated, client reports with invented data points, and email sequences that caught a brand voice exactly while proposing strategies the client would never approve. Someone still has to ask, every time, whether it is true, whether the client would put it that way, whether it matches what was agreed, and whether the numbers add up.

The work has shifted from doing to checking

What I do now is closer to editing and direction than to execution. The model produces a draft. I bring the brand context, the client knowledge and the operational judgement, and I say "this is wrong, try again" more often than most people would guess.

What clients pay for is someone who understands how their business works and won't send them a confident mistake. The draft can come from a machine; direction, voice, accuracy and the decision about whether it is good enough still come from a person.

What's tripping people up

These are patterns I've watched colleagues and clients run into, and a few I've run into myself.

Over-trusting AI numbers

Tools fudge data more often than you'd expect, especially across large spreadsheets or layered logic. Claude once told me, confidently, that a client's email open rates had improved by 23% when they had dropped. Check anything involving figures, percentages or comparisons.

The AI tell

Perfect grammar, a slightly Mid-Atlantic tone, no contractions, sentences opening with "It's important to note". The output reads correctly and doesn't sound like the person who is meant to have written it. The fix is unglamorous: read the draft aloud, rewrite anything that sounds like a press release, and leave some texture in.

Forgetting what it was meant to remember

I load client preferences and project context into ChatGPT's memory and Claude's projects, and neither is reliable enough for ongoing complex work. Mid-conversation I still have to remind both about a client's tone, their audience or the buzzwords they hate. The memory helps, and re-briefing is still part of the job.

The context cliff

Both ChatGPT and Claude have conversation limits that interrupt complex work, and hitting a session cap halfway through a long strategy document, with no warning, can cost an afternoon. Save the context somewhere else as you go so you can hand it back.

What AI has made easier

Transcription is the clearest gain. Otter and similar tools save hours, though I still scan the transcripts for the odd mangled technical term. First drafts of blog posts, social copy and awkward emails come faster, and the more specific the brief, the better the draft. Reformatting one piece for LinkedIn, Instagram and an email subject line takes under a minute. Make and n8n, combined with AI, have replaced several hours of weekly manual work in my own practice: invoice summaries pulled from Xero, project updates generated from ClickUp, files routed from MailerLite to storage.

What I don't hand to AI

Anything emotionally sensitive stays with me, from difficult emails to conversations where someone is stressed and needs to feel heard before they need help. So does anything that depends on instinct, like reading between the lines on a client call or noticing what a client hasn't said. Strategic decisions that affect a client's business are mine to make, nothing reaches a client without my eyes on it first, and anything AI drafted gets checked twice. Hand over that judgement and you become a prompt operator, which is a different and lesser job.

Two questions to ask before you hire

If you're considering a Digital Operations Partner, a VA or the local equivalent, ask how they use AI in their work. "I don't" means they're behind. Then ask how they check what AI gives them. If the answer is "I trust it", or there isn't a clear answer, that is the bigger problem, because most of the value lies in catching what AI gets wrong.

Where this is going

The framing that has settled across most of the research I trust is "collaborative intelligence": AI as a fast, slightly unreliable colleague. That matches my experience. The people who do well from here will pair fluency with the tools with the brand judgement and operational instinct that made the work worth paying for in the first place.

If someone has told you AI is about to replace your work, ask who told you and why. From inside the work, I see more becoming possible, less of it mindless, and a sharper line between the parts that need a person and the parts that don't.

Back to the Journal