AI automation agencies have multiplied fast, and quality varies wildly — from teams that build serious, integrated systems to resellers wrapping a template in a markup. We're an agency ourselves, so read this knowing our bias. But these are the questions we'd ask anyone, including us, before signing.
1. 'What happens when the AI can't handle something?' The right answer involves specific escalation rules, human-transfer paths, and what the handoff looks like. If the answer is 'the AI handles everything,' walk away — it doesn't, and they know it.
2. 'Can I see a live deployment, not a demo?' Demos are rehearsed. Ask to interact with something they've actually shipped, or talk to a client running one. We put live demos of every bot on our site for exactly this reason — kick the tires before any call.
3. 'What does the first 30 days after launch look like?' Automation needs tuning against real-world usage. An agency that disappears at go-live is selling you a launch, not a system. Look for monitoring, refinement rounds, and a named person you can reach.
4. 'What exactly integrates with what?' Get specific: which CRM, which calendar, which phone system, and what data flows in each direction. 'It integrates with everything' means they haven't checked your stack.
5. 'Who owns the system and the data if we part ways?' You want clear answers on data export, account ownership, and what keeps working if the relationship ends. Vendor lock-in disguised as convenience is the most common trap in this space.
6. 'What will this NOT do well?' Every honest builder has a ready answer, because every system has limits. A vendor with no limits to disclose hasn't deployed enough to find them.
7. 'How do you measure whether it's working?' Look for baseline-versus-after numbers: answer rates, response times, recovered leads, hours saved. If success is 'you'll love it,' you're buying a feeling.
8. 'What do you need from my team, and when?' Real deployments need your input — call scripts reviewed, pricing logic explained, edge cases flagged. An agency that needs nothing from you is building something generic.
9. 'What's the total cost in year one — setup, monthly, and changes?' Get the full number including the inevitable adjustments. Cheap setup with expensive change orders is a common pricing trick.
10. 'Why this automation first?' The best agencies will sometimes talk you out of what you came in asking for, because something else pays back faster. A vendor who agrees with everything is selling; a partner who re-prioritizes is consulting.
The Pattern Behind All Ten
Every question above probes the same thing: has this agency actually deployed systems into messy, real businesses — or just demos into pitch decks? Deployment experience shows up as specificity. Listen for the specifics.