How to measure chatbot performance and what counts as a conversation
A conversation counter is easy to inflate. It picks up people who only opened the window and your own tests from the launch period, so if one month you count everything and the next only real messages, the comparison means nothing. Pick one rule and stick to it: a conversation is a session with at least one customer message, excluding your tests.
Chat conversion is the share of conversations linked to at least one order among all conversations in the period. Ten such conversations out of 200 give 5%. If you count orders instead of conversations with a purchase, name the metric differently, since a single thread can produce several purchases.
Write down how long after a conversation you still count a purchase. A result counted to the end of the session and a result that covers the next few days are two different measures, so do not mix them on one chart. With low traffic, show the count and the denominator next to the percentage, because two out of twenty and twenty out of two hundred are the same 10% with very different certainty.
Does revenue from chat mean extra sales?
Not directly, because revenue from conversations is only the value of orders that the system linked to the chat. CartBot distinguishes two situations here. In the first, the customer added a product to the cart from the chat and later bought it. In the second, the bot recommended a product in the conversation and that product later appeared in an order. When the shop integration does not pass on order line items, the calculation rests on an estimate, so check the labels in the report before you add up amounts.
Some of those people would have bought without the bot. It works the other way too. Someone who asked in the chat on a phone and ordered in the evening from a computer may never be recognized as the same person, even though the bot's answer is what convinced them. Whatever tool you use, ask the vendor exactly what it calls an order from a conversation and what time window it uses.
So compare the amount from the report with your shop panel and subtract canceled orders and returns, remembering that a cash-on-delivery order is not yet money in your account. To judge profitability, take your margin and the cost of handling the order, and only then put that result next to the bot's invoice.
Conversations resolved without a human
In CartBot a conversation counts as resolved by the bot when no human joined it and the customer did not give a negative rating. That is a useful signal, but a customer who closed the window without rating confirmed nothing. They may have got an answer or given up.
Before you treat this share as a measure of quality, read a sample of such threads. Only then will you see how many were settled matters and how many were quiet surrenders.
Handovers to a human and contacts left
More handovers to a human does not have to mean the bot is breaking down. In November there may simply be more complaints or questions about individual quotes, which the bot should not settle on its own anyway. Before you change anything in the configuration, compare the topics of handed-over conversations with the previous month.
For service businesses you have a left contact instead of a cart, but that is only an enquiry. The road to money still runs through a booked visit and a paid service, and each of those stages can break, so count them separately.
Measure time saved on a sample. Note how long a dozen or so similar cases take you today and how long they took before the launch, instead of converting every contact into an imaginary sale.
Which chatbot KPIs to put in a monthly summary
One page a month is enough, as long as it is calculated the same way every time. Otherwise you are comparing two different metrics under one name.
- Number of conversations under a fixed definition.
- Conversation conversion in percent, together with the count and denominator, so you can see how many conversations the result stands on.
- Revenue from chat, separately for cart additions from a conversation and for recommendations, with estimates marked.
- The same amount after cancellations and returns, and next to it the margin after order handling costs, since that is the result worth comparing with the bot's invoice.
- Share of conversations without a human, with a note from the sample you read.
- Handovers to a human grouped by topic, with a short note on what changed since last month and why.
- Handling time measured on a sample.
A weekly review of conversations
Numbers alone will not tell you what to fix, so block a short, regular slot in your calendar, 20 minutes to start with, and read a few threads handed over to a human and a few that ended without one. If there is more to clear up, make the review longer.
In the conversations you look for two things: questions the bot had no good answer to, and places where the customer gave up. For a wrong answer, write down the customer's question and the information missing from the catalog or the knowledge base. After the fix, ask the bot the same question. The following week, come back to the case and check that the change really reached customer conversations.