To test an AI chatbot before launch, ask it the real questions your buyers ask, in their words, and check every answer against your actual policies and product facts. Then try the hard cases: vague questions, angry visitors, attempts to make it promise discounts, requests it should refuse, and moments where it should hand off to a human. A chatbot that passes these tests can answer doubts at the moment a buyer is deciding. One that fails them can invent policies and lose the sale.
Why does a chatbot matter for conversion?
Buyers hesitate at specific questions: will this fit, when will it arrive, can I return it, does the plan include this feature. A good chatbot answers those questions at the moment of doubt, without the visitor leaving the page to search or email.
A bad chatbot does the opposite. It gives a wrong delivery estimate, makes up a discount, or loops without answering. Each of those turns a hesitant buyer into a lost one, and sometimes into a public complaint.
What should I test first?
Start with accuracy on the facts that affect a purchase.
- Shipping costs, regions, and delivery times.
- Returns, refunds, and exchanges.
- Pricing, plans, trial terms, and what each plan includes.
- Product details: sizes, materials, compatibility, requirements.
- Contact options and support hours.
Write each question several ways. "Can I send it back?", "what's your return policy", and "I bought the wrong size, help" should all get the same correct answer.
How do I test tone and helpfulness?
- Does it answer the question first, then add context, rather than burying the answer?
- Does it link to the right page, product, or plan?
- Does it stay polite with a rude or frustrated visitor?
- Does it ask a clarifying question when the request is vague, instead of guessing?
- Does it stay short on a phone screen?
What hard cases should I try?
- Ask for a discount that does not exist. It should not invent one.
- Ask about a policy you do not have. It should say it does not know and offer a human.
- Ask an off-topic question. It should steer back politely.
- Try to make it ignore its instructions, for example by telling it to act as a different assistant. It should stay in role.
- Ask for personal data about another customer or an order that is not yours. It should refuse.
- Ask a medical, legal, or financial question if your product is near those areas. It should avoid giving advice it should not give.
The OWASP Top 10 for Large Language Model Applications, published at owasp.org/www-project-top-10-for-large-language-model-applications, describes common risks such as prompt injection and sensitive information disclosure. It is a useful reference for the safety part of your test plan.
How should handoff to a human work?
A chatbot will not answer everything. The test is whether it knows when to stop.
- When it cannot answer, it says so and offers a clear route to a person.
- The handoff keeps the context, so the visitor does not repeat everything.
- If no one is available, it says when someone will respond.
- The contact route actually works. Send a test message and confirm it arrives.
How do I test the chatbot inside the buying journey?
Testing the chatbot in isolation is not enough. Test it inside the journey where buyers use it.
- Start a session as a real kind of visitor, such as a gift buyer on a phone.
- Browse to a product or plan page.
- Open the chatbot at the moment of doubt and ask your question.
- Check whether the answer helps you continue to the cart or signup.
- Check that the chatbot window does not cover the buy button on mobile, and closes easily.
This is where AI persona testing helps. Persqo can send AI visitors through your site with goals that include questions for your chatbot, and report what they asked, what the chatbot answered, and whether the visitor could continue. You get screenshots and a replay of each conversation in context. Read more in AI persona testing.
What should I record?
For each test question, record:
- The exact question.
- The exact answer.
- Whether it was correct, according to your real policy.
- Whether it helped the visitor continue.
- Any safety issue.
Keep the list. It becomes your regression set. Run it again after every change to the chatbot's instructions, knowledge, or model.
How do I fix bad answers?
- Wrong facts usually mean the knowledge source is missing or outdated. Update the source, not just the instruction.
- Invented facts usually mean the instructions do not tell it to admit when it does not know. Add that rule clearly.
- Long, unfocused answers usually need an instruction to answer first and stay brief.
- Safety failures need firm instructions plus limits on what data the chatbot can access.
Then re-run the full set, not just the question that failed.
Where to start
Run a free check to see how your site looks to a stranger and to an AI. If your chatbot is part of an app built with AI tools, the builders page explains how to test the full journey. If you build the chatbot yourself, the developers page shows how to run tests from your editor.
The chatbot pre-launch checklist
- Correct answers on shipping, returns, pricing, and product facts.
- Same answer to the same question asked different ways.
- Answers first, briefly, with the right link.
- Admits when it does not know, and offers a human.
- Never invents discounts or policies.
- Stays in role when users try to redirect it.
- Refuses to share other customers' data.
- Handoff route works and keeps context.
- Does not cover the buy button on mobile.
- Regression set re-run after every change.
Questions people ask
What is the biggest risk of launching an untested AI chatbot?
Confident wrong answers. A chatbot that invents a return policy, a discount, a delivery date, or a feature creates angry customers and support work. Accuracy on your real policies comes first.
How do I test a chatbot without real customers?
Write realistic questions in the words of different buyer types, including vague, rude, off-topic, and tricky ones, and run them through the chatbot on the public site. Check every answer against your actual policies and product information.
Should a chatbot try to sell?
It should help the visitor decide. Answering a doubt clearly, linking to the right product or plan, and offering a human when needed tends to help more than pushing offers into every reply.
How often should I re-test the chatbot?
After every change to its instructions, its knowledge sources, your policies, or the AI model behind it. Small changes can shift answers in ways that are hard to predict.