How to write effective chatbot scripts

Why Chatbot Analytics? Part 1/2

This blog introduces the topic of chatbot analytics. Chatbot Analytics encompasses the process from defining the relevant metrics, setting up tracking and analyzing to optimizing bot systems. Before we dive into the depths, we would like to give you an overview of the topic in our first part:

  • Why does it need analytics for chatbots and voice assistants?
  • Which key performance indices are important for chatbots?
  • How do you properly analyze a chatbot?
  • Which chatbot analytics tools are there?

Why does it need chatbot analytics?

Without a database, no correct basis for decision-making - whether the chatbot actually achieves its goal and thus contributes to the overall business goal can only be seen if the right key figures are defined on the one hand and these are correctly measured and interpreted on the other.

Analytics as a basis for decision-making in innovations

Unfortunately, innovation projects are often difficult to justify to management because they are often viewed as resource-intensive, require new knowledge from employees or are often classified as “not necessary” by the operational organization. That is why it is all the more important to show management the advantages and benefits of innovations on the basis of facts. If your manager sees that the new Shopbot has carried 20% more orders, or that the company website has 30% more traffic than before due to the new Voicebot, then these are clear facts that justify the innovation project.

Similar to business intelligence or web analytics projects, it is important that the corresponding key performance indices are derived and defined with the project objective. In the case of web analyzes, so-called “actionable metrics” should be defined, with which effective actions and thus improvements can be achieved. The information “The chatbot has an average of 3000 users per month” is less useful than the statement “3% of 3000 bot users have converted to a lead” (conversion rate). The number of users can be increased, but whether a bot is really useful can be measured more effectively through the effective generation of leads. Last but not least, a bot system can only be improved and further developed in the right direction if its benefits - especially through the conversations - are measured.

Chatbot Analytics as a monitoring tool

Chatbot Analytics makes it possible to determine whether the bot is working properly or whether errors, downtimes or, in the worst case, attacks are occurring. The NLP model can be monitored and the bot's understanding can be optimized. If this monitoring is missing, this is extremely suboptimal for the operation of a digital assistant.

Chatbot versus web analysis: what are the differences?

For example, the BLS study was able to use the analysis data to identify which customer inquiries were made most frequently. In the six-month study, for example, we recognized that many users had asked about topics relating to the “car transport at the Göschenen Tunnel”. Such data insights support product or customer service managers in optimizing the offer or customer service.

Which key performance indices are important for chatbots?

We have summarized the most important key figures for you here:

  • Active users: A user is then counted as soon as a person communicates with the computer. The active users provide information about the usage rate of the bot and thus also how successful it is, piece by piece. If a user accesses the bot with a different device or browser, this counts as a new user. With Voicebots, the identification of the user often depends on whether he has activated “Voice Match” and the “Personalized Settings”. Otherwise the user will be counted as a “guest user” and repeat users will not be identified as one user or “counted twice”.
  • Conversations: Conversations are calls by the bot, which often contain several bot and user messages (so-called “messages”). If there are no more messages within a certain period of time, the conversation is considered completed. This “session duration” is limited to five to n minutes, depending on the tool. The number of “message turns”, ie how many bot and human messages were exchanged during the conversation, are of particular interest. Many message turns indicate a very committed user. The user seems to be interested in the bot.
  • Conversation duration: The conversation duration in minutes shows how long a user has been talking to the chatbot. Long conversation times are not necessarily an indication of a meaningful conversation. The number of message turns based on the conversation must also always be considered, because it is possible that a user answers only very slowly or the session duration of n minutes has led to a long conversation with no real substance. If the duration of the conversation is not satisfactory, this may be due to the texts that need to be optimized.

For companies, high interaction rates and long conversation times are very relevant, because they show that a user deals with the subject matter of the company and could be interested in the offer.

How do you properly analyze a chatbot?

In order for the chatbot to be properly analyzed, the KPIs must first be correctly defined. There are a few things to consider:

Functional vs. experience-oriented chatbot

With a functional bot, the user wants to solve a problem or question as quickly as possible. For example, as a user of the SBB voice assistant, I don't want to have long small talks, but rather quickly query my timetable. On the other hand, a quizzes bot should ask exciting questions and answers and it is important to me as a user that I can have long and eventful conversations. Companies strive for high interaction rates and a high degree of simplicity in conversation, because they increase the likelihood of a lead or purchase in online marketing. Conclusion: The KPIs can be defined from the chatbot goals and measured accordingly.

KPI examples:

  • Functional bot: The average conversation time for a query of the weather without additional information should be a maximum of 10 seconds.
  • Experience-oriented bot: The average number of messages per user is at least 8 per user.

Measure abstract constructs: How do I measure whether the user was satisfied with the chatbot?

Satisfaction can be measured in different ways in practice. Various factors (so-called “items”) can be measured, satisfaction can be queried in the chat or forecasts can be calculated using sentiment analysis and machine learning methods. However, studies show that sentiment analysis is not ideal for measuring satisfaction with chatbots and that common tools do not yet provide reliable results. Here it is useful to take a closer look at the conversations and also to analyze the users' swearwords as well as the misunderstandings and fallbacks (e.g. “I didn't understand you”). The Retention rate - it shows how often the same user comes back on average in the same week - provide information. It is often assumed that loyal users are satisfied with the chatbot and want to reuse it. A quick one Bounce rate At the beginning or at special steps in the conversation (e.g. transaction) of the chatbot conversation can also provide information on whether the bot was useful for the user.

Which chatbot analytics tools are there?

The best-known tools used on the market are Dashbot, Chatbase or the Analytics interface of the Google Action Console. The problem with common tools is that they are costly, have performance problems, do not adequately map the business goals or do not provide any insight into the transcripts. Often the data is also stored on unknown or international servers, which has consequences for data protection and security concerns. We also advise against hastily writing analytics data into databases, retrieving it from powerful analysis software or even developing such a tool yourself without first thinking about the purpose, the visualization and the budget. For database logs, correct database queries must first be written, which can be very time-consuming. The connection to BI software such as Qlikview, Tableau or Elasticsearch also requires a lot of resources and know-how and often has to be implemented in a separate project. A simple first chatbot analytics dashboard gives you an overview of the most important data and saves a lot of time and money.

Conclusion

The article shows that it is worthwhile to reserve enough time for analytics even before the chatbot project, because the subject is complex. Paixon offers the Paixon Analytics Tool, which covers the most important components that are necessary in the everyday life of a chatbot owner and voice project manager.

Paixon Analytics at a glance

  • Dashboard with the most important key performance indices and visualizations
  • Overview and search for conversations
  • Filter functions: By time period, by bot platform and version
  • All data is logged in a database
  • Paixon Analytics runs on a Swiss cloud infrastructure, which guarantees high security and data protection standards

Would you like to learn more about Chatbot Analytics? No problem, we are happy to help you at: Contact