Nowadays it feels like everyone has an opinion on AI. It’s not just chit chat. These conversations are shaping real policies and procedures inside agencies. Our co-founder and principal, Tony Welz, recently wrote an opinion piece in PRWeek about ‘PR’s AI Originality Dilemma.’ His point was simple but powerful: AI can make us faster, but it should never replace the originality we bring. In an industry built on creativity and trust, outsourcing our thinking undermines the value we deliver.
What I like most about Tony’s perspective is that he respects the power of AI without glorifying or fearing it, and I completely agree! Some people treat AI like a shortcut and use it for everything, while others won’t touch it because they see it as a threat. There are also companies putting out policies outlining what employees can and can’t do with AI or even building their own custom version so employees can use them safely without leaking confidential information.
For me – and my colleagues here at W2 Communications – AI isn’t a threat or a replacement. It doesn’t do my job for me. Instead, it makes me faster and sharper. It helps me refine my thoughts, look at angles I might have missed and elevate the way I deliver an idea. It is not a magic button, but more like having an extra consultant in the room, extending what I already bring to the table.
The Challenges with Detection
Let’s be honest, almost all of us have used AI at some point. But, as Tony mentioned in his article, our publication partners are using tools to detect AI-generated content. Our goal is to find a balance in using AI to help us, while ensuring that our content is not flagged as AI-generated (see Tony’s recent blog on our commitment to originality). But – like many AI tools – we’re seeing challenges arise – A significant one is a system deciding if your work looks “human enough.”
You write something, drop it into a detector and wait for the tool to tell you if you are “in the clear,” even when you know for a fact you 100% wrote it yourself. This raises the question: How accurate are these tools?
Real-World Examples of AI Detection Gone Wrong
Some of our team participated in a live demo for an AI detection tool. The rep drafted a quick article using AI, copied it over and then ran it through their detector. Lo and behold, the result was not what any of them expected. The tool gave an output of 100% human. An awkward and unfortunate experience. What was supposed to be a proof point turned into the perfect example of the flaw, live for all to see.
I’ve experienced the same thing myself. I’ve taken things I have written, no AI involved, and plugged them into different detectors. The results were scattered. One said 100% human. Another said 13% AI. Another flagged it at 80% AI. Same exact text. Three totally different results. The screenshots here are examples of these varying results. That’s not a content problem; that’s a detection problem!
How AI Detection Tools Work and Why They’re Flawed
The varying results made me even more curious, so I went to the source. I asked AI what these detectors actually measure. Most of these companies don’t actually publish how their systems work. What they do share is pretty vague. A lot of these tools lean on how predictable the language is. In AI terms, it’s called “preplexity” (how predictable the text is) and “burstness” (the variation between sentences). Clean, polished sentences get flagged as AI, while more irregular or unpredictable writing is labeled as “human.”
Some tools also factor in extra signals like typing patterns or paste detection, though not all of them do. To be clear: none of these tools are actually detecting AI. They’re making a statistical guess based on how closely your text matches patterns they’ve seen before. Above all of that, what’s even more concerning is how easily these probability scores are treated as hard evidence.
While putting this blog together, I also asked AI to explain the flaws of these detection tools. The response was almost identical to what I had already experienced myself: bias, inaccuracies and inconsistent logic. Even the technology admitted the same gaps we already see in practice.
Here are some considerations to bear in mind when using these tools:
- These tools are trained on limited samples of human and AI writing, with false positives being documented, particularly against students and non-native writers.
- Accuracy rates are low and inconsistent.
- Inconsistent results across different detectors analyzing the same text.
AI text can often bypass detectors entirely with simple edits, which shows how fragile these systems really are.
And the truth is, this isn’t new. Detection tools have been wrong for years. We’ve had plagiarism checkers that flagged original essays because some phrases matched a database. Email spam filters still bury real emails. Lie detectors aren’t even trusted in court. AI detection is just another version of an old problem, tools that overpromise and underdeliver.
Why Does This Matter to Brands?
The part that really matters is where these tools are being used. Classrooms. Businesses. Publishing platforms. Even online forums. And in the agency world, we hear it from clients too. They are worried about whether they should be running detectors, or what happens if their content gets flagged. The irony is that the same issues I run into personally – inconsistency, over-flagging, no clear explanation – are the ones we warn clients about professionally. It does not matter if you are a student, a writer or a company protecting its brand. If you are relying on these tools, you are relying on an imperfect solution.
The Questions We Should Be Asking
Earlier in this blog, I shared screenshots showing how different AI detectors scored the text. That inconsistency isn’t a fluke. OpenAI actually shut down its own AI detection tool in 2023 because the accuracy was so low.
I ran this blog through those same tools, and the results were just as inconsistent. That doesn’t prove whether something is AI written or not. It just proves these systems aren’t 100% reliable.
So, similar to Tony’s recommendation, I encourage everyone to explore AI tools and explore how they can help you, but remember that these tools are flawed. They can be useful in some cases, but their scores aren’t the final word, and they shouldn’t be treated like they are. They should not replace an SME’s authentic thoughts or expertise. Human review and input is still critical to ensure your content answers key questions like: Does the content add value? Is it accurate? Does it stand on its own?