How to Start with Conversational AI

Since a few years, chatbots are here, and they will not go away any time soon. Facebook popularised the chatbot with Facebook Messenger Bots, but the first chatbot was already developed in the 1960s. MIT professor Joseph Weizenbaum developed a chatbot called ELIZA. The chatbot was developed to demonstrate the superficiality of communication between humans and machines, and it used very simple natural language processing. Of course, since then we have progressed a lot and, nowadays, it is possible to have lengthy conversations with a chatbot. For an overview of the history of chatbots, you can read this article.

Chatbots are a very tangible example where humans and machines work together to achieve a goal. A chatbot is a communication interface that helps individuals and organisations have conversations, and many organisations have developed a chatbot. There are multiple reasons for organisations to develop a chatbot, including obtaining experience with AI, engaging with customers and improving marketing, reducing the number of employees required for customer support, disseminating information and content in a way that users are comfortable with and, of course, increasing sales.

Chatbots offer a lot of opportunities for organisations, and they can be fun to interact with if developed correctly. But how do you start with conversational AI and how do you build a good and engaging chatbot? To answer that, I researched 20 organisations from around the globe who developed a chatbot. As part of my PhD, I wanted to understand how organisations can get started with conversational AI and be successful.

Seven Steps to Develop a Chatbot

Starting with a chatbot is not easy, as there are many different variables to take into account.

1. Define the reason for a chatbot;
2. Create the conversation flow;
3. Determine to develop the chatbot in-house or using third-party tools;
4. Integrate the chatbot and conversation flow in the front-end and back-end of your organisation for context awareness;
5. Test the chatbot and obtaining approval of relevant stakeholders;
6. Analyse the conversations and the data derived from it;
7. Improve the chatbot based on the analytics received.

Let’s discuss each step briefly:

1. Define the problem

First of all, it is important to decide why you would need a chatbot. A chatbot should be a means to an end, but not an end itself. It should alleviate a pain point or increase your customer engagement, but it cannot, yet, replace your entire customer support department. Understanding the objective of your chatbot will help define the conversation flow as well as determine the type of chatbot you need. After all, there are different types of chatbots ranging from simple FAQ bots, so-called ‘on rails’ bots to chatbots that allow the input of free text. The more you allow the user to determine the direction of the conversation, the less the chatbot is in control.

2. Create the conversation flow

Designing the conversation within a chatbot is challenging. Not only should you develop a persona that matches your brand personality, but the conversational interface should also be clean, and the chatbot should aim for a positive experience. Therefore, the conversation should not be developed by the developer, but by a copywriter in collaboration with the marketing or communication department. It is important to create the right conversation flow for the right objective. For some conversation, people feel more comfortable with a chatbot than with a human. For example, an Australian financial services company noticed that customers feel more comfortable cancelling with a machine than they do with a human. Therefore, when developing a chatbot, you should pay attention to the conversational strategy and know that the platform itself is not standalone, but should be integrated with all the other elements of the business.

3. Selecting the chatbot platform

There are many different chatbot platforms, ranging from platforms that enable simple FAQ chatbots to more advanced chatbots that take into the context. Such context-aware chatbots can offer a lot of added value because they can offer a positive experience to the end user.

Once you have decided what platform to use, it is important to decide whether to outsource or not. There are plenty of chatbot developers out there that can help you, but not every developer might offer the right solution. Therefore, it is important to investigate and ensure that you work with the right chatbot developer.

4. Integrating the chatbot

Building a chatbot is the easy part, among others because of the many platforms and developers out there. Integrating the chatbot into your systems is a lot more difficult, but that’s when the added value is achieved. If the chatbot is connected to your system your CRM or database), and when someone wants to change, for example, an address and the chatbot can say: ‘sure, give me your address, and I will update the system for you’. This is where you see operational efficiency, satisfaction and the NPS going up.

One American chatbot developer created a chatbot that is person-aware, meaning that the chatbot knew who the person is in the chat, as the chatbot is linked to internal systems. As a result, it is a lot smarter because it has a better understanding of the context and can service the customer faster better.

5. Testing the chatbot

Developing is only one part, as with any software development project, the testing is a crucial aspect of the project. Fortunately, most of the organisations I spoke, test the code of the chatbot. Especially the chatbot developers have rigorous testing practices in place. These processes include a testing environment, an acceptance environment and a live environment to ensure that everything can be properly tested. Not only should you test the code of your own projects, but you should also test the software that you use. Unfortunately, many organisations did not test the third-party tools they implemented and sort of trusted the third party that their tool and the code in that tool was correct and did not have any bugs. There is a strong reliance on and confidence in the third-party tools. However, it is important to have proper controls in place. One American chatbot developer went as far as never to allow third-party developers access the code. And another option is to spend time on reverse engineering what you have built to ensure that the code is indeed correct.

6. Analysing the conversations

A conversation is by its nature data-driven and leads to more data. This data can be analysed, and the insights of the analytics can be used to improve the conversational flow of the chatbot. However, to enable that any output text can be used to train the chatbot, thorough testing processes have to be in place. Such as any text that should be written by copywriters not developers and especially large organisations require some sort of governance structure to be in place around the content that is said by the chatbot.

Since all conversations are data, it is possible to extract valuable information from the conversations, both actively and passively to capture and feed that data into the overall reporting mechanisms. So that’s on a micro level of an individual conversation for an individual user, and it is at a macro level for the questions that are being asked and answered. This is called conversational analytics: what was said, how was it said, what was the intent, what is the sentiment, did we accomplish the goal, what was the goal. Where does it fit in the larger context? Without conversational analytics, it is impossible to develop an engaging and successful chatbot.

It is also possible to add the capability to jump in and intervene in any conversation, but that sort of deceits the purpose. However, it can be useful because often machines still don’t understand the full context and then human intervention is required.

7. Improve the chatbot

Of course, all those analytics can offer valuable insights to improve the chatbot. Reviewing the transcripts looking at places where the chatbot did not understand what people are asking helps to build up any datasets so to retrain the chatbot or to look at places where the chatbot thinks it got it right, but actually got wrong and so that the information can be rectified and the conversations can be improved. Such supervised learning helps improve the chatbot, while it prevents problems such as Microsoft’s Tay, which learned unsupervised. The objective should be to continuously improve the chatbot, make it increasingly context-aware and better at understanding the intent of the conversation.

Conclusion

Chatbots offer a great way for organisations to improve their business, make it more efficient and increase the customer experience. However, it is important that the chatbot learns in a supervised way and is bound by certain rules that drive your conversation if you wish to prevent examples such as Microsoft’s Twitter bot Tay. Natural language processing is getting better, and in due time, it will become possible to have engaging conversations with a machine.