What to ask before hiring an AI shop

The AI services market has a problem: it's very easy to look credible without being good. A polished deck, a GPT demo, and a few buzzwords can get an agency through a first call with most small business owners.

We know because we've seen what clients come to us with after a bad experience — tools that never quite worked, contracts that locked them in, "custom agents" that were just off-the-shelf chatbots with a logo on them.

These twelve questions will help you tell the difference. Ask every vendor you speak with. The answers — and how they give them — will tell you everything.

1. Can I see a live example that's similar to my business?

Not a demo. Not a recording. A live agent currently running for a real client. If they can't show you one, ask why. A competent shop has references and is proud of them.

Green flag: They pull up a live example and walk you through it in real time.

Red flag: "We can build you a demo based on your specs."

2. What happens if the agent makes a mistake?

It will. Every agent does. The question is how they've designed for it.

Good shops have fallback paths, escalation protocols, and monitoring in place. Bad shops say "the AI learns from mistakes" and leave it at that.

Green flag: Clear description of fallback logic, human handoff paths, and how errors get reported and fixed.

Red flag: "The model improves over time" with no specifics.

3. Who owns the agent when we're done?

Some vendors retain full ownership of what they build. You're renting access, not buying a system. If you leave, you lose everything.

You should own your agent's training data, your conversation logs, and ideally the configuration. Ask explicitly.

Green flag: Clear IP assignment. Your data is yours. They provide the infrastructure; you own the asset.

Red flag: Vague answers, "proprietary system," or ownership buried in the contract.

4. What's the ongoing monthly cost, and what does it cover?

Get the total cost of ownership, not just the setup fee. Ask what the monthly fee includes — hosting, support, updates, model costs. Ask what triggers additional charges.

Green flag: Flat monthly fee with a clear scope. Honest about what could add cost.

Red flag: Low setup fee, complex monthly billing with usage tiers you can't predict.

5. How long have you been building these specifically for small businesses?

AI agencies are appearing fast. Many are run by people who've built one or two tools and are now selling the process. That's not necessarily disqualifying, but you should know.

Ask for a portfolio. Ask how many agents they've built and for what kinds of businesses. Ask how many are still running.

Green flag: Specific examples, honest about failures, clear track record.

Red flag: "We've been in AI for years" without specifics, or a portfolio that's all mockups.

6. How do you handle data privacy?

Your agent will handle customer names, phone numbers, email addresses, and potentially payment or health information depending on your business. Where does that data go? Who can access it? How long is it stored?

Green flag: Clear data handling policy, can point to their sub-processors, compliant with PIPEDA or relevant regulation.

Red flag: "We take privacy seriously" without specifics.

7. What does the build process look like, week by week?

A good vendor has a defined process. They can tell you what happens in week one, what you need to provide, when you'll see a first draft, how testing works, what sign-off looks like.

Green flag: Clear timeline with milestones and defined inputs they need from you.

Red flag: "It depends" without a framework, or a very fast timeline with no discovery phase.

8. How do I update the agent when my business changes?

Prices change. Services change. Staff change. Policies change. What happens when yours do?

Green flag: Clear update process. Monthly check-ins included. Easy mechanism for you to flag changes.

Red flag: "Open a support ticket" or updates billed hourly with no included maintenance.

9. What metrics will you report on, and how often?

If an agent is handling your calls or emails, you should know how it's performing. Ask what you'll measure, how often you'll review it, and what a "bad month" looks like.

Green flag: Specific KPIs, monthly reporting, baseline measurement before build.

Red flag: "The AI handles it, you'll notice the difference" — no metrics, no accountability.

10. What can your agent not do?

Every tool has limits. A vendor who answers this question confidently and specifically is one who knows their product. One who deflects or gets vague is selling you something they don't fully understand.

Green flag: Clear list of limitations, edge cases they've encountered, types of tasks they'd recommend handling differently.

Red flag: "It can handle anything" or an answer that sounds like the marketing page.

11. What's your cancellation policy?

If the agent isn't working, can you leave? Is there a minimum term? What happens to your data when you cancel?

Green flag: Month-to-month after the build period. Clear offboarding process. You keep your data.

Red flag: 12-month contracts, cancellation fees, vague data retention.

12. Can I talk to one of your current clients?

The most reliable signal of all. Not a testimonial. Not a case study they wrote. A real business owner you can call and ask: "Did it work? Were they good to work with? Would you do it again?"

Green flag: They offer a reference without you asking, or respond immediately with a name and number.

Red flag: "Our clients prefer to stay private" or a long delay while they "check."

Take this list to your next demo call. Good vendors will enjoy the questions. They've thought about all of them.