Resource
The questions that separate someone who can build a demo from someone who can build an operation. Use these before you sign anything.
April 2026
The AI consulting market is growing faster than the expertise in it. That means a lot of people are selling AI services who built their first system six months ago, watched a few tutorials, or repackaged general consulting with AI branding.
These questions are designed to help you tell the difference. They're not trick questions. They're the questions that a competent AI architect should be able to answer clearly, specifically, and without deflecting.
About Their Approach
Why this matters
Anyone who jumps to solutions before understanding your specific workflows, data, and team is selling a template, not an architecture. The assessment should involve talking to the people who actually do the work, not just leadership.
Red flag
They propose a solution in the first meeting.
Why this matters
This is the core architectural question. If they don't have a clear framework for separating reasoning tasks from mechanical tasks from judgment calls, they're going to build something where AI does everything, which means it will be slow, expensive, and fragile.
Red flag
They describe everything as an "AI solution" without distinguishing between layers.
Why this matters
Every AI system will produce errors. The question is whether the architecture is designed to catch them, flag them, and recover. If there's no answer for error handling, there's no production-ready system.
Red flag
"The AI is very accurate" with no discussion of error handling or human review points.
About Their Experience
Why this matters
Demos prove the concept. Production proves the architecture. A system that's been running means they've dealt with edge cases, maintenance, scaling, and the reality of messy data over time.
Red flag
They only show demos or proof-of-concept work.
Why this matters
Someone who's built real systems has battle scars. They know what breaks. If the answer is vague or optimistic, they haven't been through enough real implementations to know what actually goes wrong.
Red flag
"AI is pretty reliable now" or they can't name specific failure patterns.
Why this matters
This is the question that separates architects from tool integrators. The demo always works. The question is what they do when documents come in wrong formats, data has gaps, and edge cases appear that nobody anticipated.
Red flag
They don't acknowledge the gap exists.
About Governance and Handover
Why this matters
If the system depends on the person who built it, it's not infrastructure. It's a dependency. Documentation, governance frameworks, and team enablement should be part of the engagement, not an afterthought.
Red flag
They suggest an ongoing retainer as the only maintenance option.
Why this matters
If AI is processing client data, financial information, or internal communications, governance isn't optional. This should be designed in from day one, not addressed after legal raises a flag.
Red flag
"We can figure that out later" or they defer to your legal team without offering a framework.
Why this matters
A system without documentation is a black box. You should receive architecture specifications, workflow documentation, governance frameworks, and maintenance procedures. If they can't tell you exactly what you'll get, you're buying a service, not a system.
Red flag
Vague promises about "knowledge transfer" without specifics.
About Cost and Value
Why this matters
AI systems have running costs: API calls, compute, maintenance, monitoring. If they can't estimate this, they haven't thought through the operational economics. You need to know the total cost of ownership, not just the implementation fee.
Red flag
They only quote the build cost and can't estimate operational costs.
Why this matters
"It uses AI" is not a success metric. Time saved, error reduction, process speed, cost per transaction. There should be specific, measurable outcomes defined before the work begins.
Red flag
Success metrics are vague or entirely qualitative.
Why this matters
Anyone who promises a full AI operation in two weeks is either building something trivially simple or skipping the architectural work that makes it last. Honest timelines include assessment, design, iterative implementation, and handover.
Red flag
"We can have something running by next week."
A good AI consultant won't be threatened by these questions. They'll welcome them, because they know the answers, and because they've been burned by the same problems these questions are designed to surface.
If someone can't answer these clearly, they're not ready to build something your business depends on.
I hold myself to these same standards. Ask me these questions too.
Read More
Ready to ask these questions? Start with me.
Get in touch