A New Framework for Designing and Assessing Conversational AI Agents

In today's competitive business landscape, creating a pleasant customer experience (CX) is crucial for organizations aiming to gain a competitive advantage. As customer behavior continues to evolve due to factors like the Covid-19 pandemic and advancements in artificial intelligence (AI), firms need to adapt their capabilities to meet changing customer expectations. Autonomous service systems, including service robots and chatbots, have become increasingly prevalent and have revolutionized customer-firm interactions.

One area that has seen significant growth is conversational AI, which encompasses technologies like chatbots and virtual agents. These AI agents simulate human interactions, understand speech and text inputs, and can translate meaning across multiple languages. Conversational AI has become one of the most widely used AI applications in enterprises, with a projected consumer retail spend of over $142 billion by 2024. Conversations in customer service play a pivotal role in shaping the overall customer experience, as language and tone can convey emotions and significantly impact customer satisfaction.

However, the effectiveness of conversational AI agents in delivering a positive customer service experience depends on their design and management. While there are existing approaches to select the technical capabilities of chatbots or classify individual avatars, there is currently a gap in the literature when it comes to designing and assessing conversational AI agents specifically for better customer service experiences.

To address this challenge, our recent conference paper proposes a comprehensive framework that combines the academic literature on customer experience management and technological advancements in conversational AI. The framework aims to enable CX managers to design and assess personalized conversational agents based on customer intent and context. By identifying contextual factors that influence customer intent, the framework bridges the gap between managing customer service experiences and the capabilities of conversational AI.

The paper's contribution can be summarized in three main points. Firstly, it integrates the knowledge from customer experience management and conversational AI to merge the requirements for managing customer service experiences with the capabilities of AI agents. Secondly, it provides CX managers with a practical framework to assess and design personalized conversational agents that align with customer intent and context. Lastly, the paper applies the framework to evaluate existing conversational AI agents deployed in various industry sectors, resulting in a typology of different approaches.

The framework comprises six dynamic stages: sense, adapt, assign, delegate, orchestrate, and train. These stages involve continuously considering contextual factors to match customer intent at each design stage of conversational AI agents. The framework was refined through the assessment of nine conversational AI agents deployed by established firms in different sectors. The assessment involved interacting with the agents as prospective customers, evaluating their responses to various pre-purchase, purchase, and post-purchase intents.

The study emphasizes the importance of sensing customer intent as a vital stage in designing conversational AI agents. Agents should be able to adapt dynamically to changing contexts and assess intent throughout the conversation. Infusing tone and emotional elements into conversations is also crucial for managing conversational AI agents, as it contributes to a more personalized and engaging customer experience.

Furthermore, the framework highlights the need for seamless delegation between AI agents and human agents. Finding the right balance between AI substitution and augmentation is critical, and frontline service employees must be equipped with relevant context when interacting with customers who have been previously engaged with AI agents.

Two additional dimensions, which remain invisible to customers, are identified as critical elements for fulfilling the customer experience on conversational AI agents. The first dimension is the ability to autonomously automate underlying processes to fulfill customer requests during conversations. The second dimension involves continuous training of conversational AI agents using customer journey data and analytics. These dimensions contribute to the agent's capability to provide efficient and effective service.

While the framework offers valuable insights into designing conversational AI agents, the study also acknowledges some limitations and suggests avenues for future research. One limitation is that the framework primarily focuses on the design and assessment of conversational AI agents for customer service experiences. It does not delve into other potential applications, such as sales or technical support. Future research could explore the extension of the framework to these areas. Additionally, the framework assumes a certain level of customer familiarity and comfort with interacting with conversational AI agents. However, not all customers may be accustomed to or prefer this mode of interaction. It would be worthwhile to investigate how customer acceptance and adoption of conversational AI agents can be enhanced, and whether there are demographic or cultural factors that influence their preferences.

In conclusion, the framework proposed by Jan and Gautam provides a valuable contribution to the design and assessment of personalized conversational AI agents for enhancing customer service experiences. By integrating knowledge from customer experience management and conversational AI, the framework enables CX managers to align the capabilities of AI agents with customer intent and context. Further research in this area can lead to the development of more effective and customer-centric conversational AI agents, ultimately improving overall customer satisfaction and loyalty.