“I felt like I was talking to a robot"
Shout Texter
A common reason for being dissatisfied with the service is finding the conversation robotic, scripted, or impersonal.
Interestingly, tracking how this feedback evolved over time revealed that it peaked shortly after the initial ChatGPT release (end of November 2022), and has been increasing since. This suggests that, at least in part, this sentiment may reflect the public awareness of AI chatbots capable of carrying out highly human-sounding, and often intentionally validating conversations.
Monthly percentages of conversations reported as robotic in the texter feedback. The green logo shows the date of ChatGPT release (30 November 2022), which coincides with a spike in the ‘roboticism’ feedback received at Shout.
Shout does not use AI to carry out conversations, and our main aim is to provide mental health support through human connection.
To understand what volunteer behaviours led to conversations being described as ‘robotic’, we analysed the contents of conversations that received this feedback.
Users who voiced this complaint often mentioned the use of ‘pre-written messages’, stating they could tell the difference between these and more personalised responses.
Guided by these comments, we measured how closely volunteer replies matched the example messages available on the Shout platform. These example messages are intended to:
We found that the more volunteers relied on these example messages during a conversation, the more likely texters were to:
Based on this insight, we ran a series of instructional videos, talks, and workshops to help volunteers:
To build on this further, we also designed an automatic system that highlights conversations likely to be rated as robotic to the Volunteer Support team. This allows conversations to be reviewed and targeted feedback to be provided to volunteers.