Conversational Survey Frontends: How Can Chatbots Improve Online Surveys?

Abstract

Relevance & Research Question:
Even though online chats have been around for a long time, the tremendous success of WhatsApp and Facebook messenger have fundamentally changed how people interact and exchange information. With the appearance of “intelligent”, machine-learning based chatbots we assume that the areas of application will become even more versatile which leads to the question how we can utilize chatbots for market research purposes.
Chatbots consist of two components: (a) the frontend for the user as the point of interaction and (b) a backend, often based on Natural Language Processing algorithms that handles the user requests and sends appropriate responses back to the user.
Focusing on the frontend, we wondered how a chat interface impacts answer behavior. We were especially curious to understand the effect on response rates, data quality and survey fatigue by engaging in a conversational manner.
Methods & Data:
We developed a Chatbot interface which delivers survey questions to the user. Our aim was to create a non-obstructive, responsive frontend that feels familiar to users of messenger services. A sample of 600 participants from a commercial online panel was randomly assigned to either a traditional online questionnaire or a Chatbot interface. Both questionnaires included exactly the same questions covering topics concerning media consumption and mobility, as well as questions on the perception of the questionnaire itself. Different answer types were presented to the respondent, such as open questions, Likert scales and Multiple Choice questions.
(The study was pre-registered at http://aspredicted.org/blind.php/?x=cb5cqk)
Results:
Results will be available in early January 2017.
Added Value:
Chatbots as survey frontends do not only offer a new but familiar interface for respondents. They allow the integration within different contexts using developer APIs for Facebook Messenger or other messaging services. This allows to recruit and survey participants very easily without the need to redirect them to a separate questionnaire.
Thus, it is important to evaluate benefits and potential pitfalls gained from using such frontends. Especially, when future applications of chatbots in online surveys include AI capabilities to analyze responses and adapt questions in real-time.

Full text

Christopher Harms, SKOPOS GmbH & Co KG
Sebastian Schmidt, SKOPOS GmbH & Co KG
Conversational Survey Frontends: How Chatbots Can
Improve Online Surveys
Contact: christopher.harms@skopos.de
General Online Research Conference (GOR 17)
15-17 March 2017
HTW Berlin University of Applied Sciences
Berlin, Germany
This work is licensed under a Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/)
Suggested citation: Harms, Schmidt. 2017. “Conversational Survey Frontends: How Chatbots Can Improve
Online Surveys” General Online Research (GOR) Conference, Berlin.

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Conversational Survey
Frontends
How Chatbots Can Improve
Online Surveys
Christopher Harms, Consultant Research & Development
Sebastian Schmidt, Senior Research Executive
GOR 2017, Berlin
March 17th, 2017

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Hello GOR 2017! Hello Berlin!
Somebody there?
Hello, friendly chatbot!
Nice to meet you! We want to talk with you about the
future of online surveys. In particular, how you can help us!
That sounds great! It’s gonna be legen –wait for it! –dary!
It will be… right?

But … wait! Why is that even relevant?
Well, let’s see…
Backend
Frontend
Well… that’s awkward. I feel so naked!

Conversational Survey Frontends –How Chatbots can improve Online Surveys
So, what did you do?
186 199
191 186
n = 377 385
We made a 2x2-design online study with 762 participants.
We even pre-registered the study!

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Nice! And how did the user frontends look like?
The traditional questionnaire was like this:
Stylish and responsive. #like

Conversational Survey Frontends –How Chatbots can improve Online Surveys
And the chat interface?
Does it look familiar to you?
Very cool! But you can see prior answers…

Conversational Survey Frontends –How Chatbots can improve Online Surveys
So, I’ve got a little sister!
Several topics, actually:
Okay, now I’m hooked. What did you find out?
But what were your participants asked about?
1. Holiday and travel habits
2. Political Attitudes
3. Nutrition and Supermarkets
4. Social Desirability
5. Demographics
6. Survey Evaluation
This took roughly 15 minutes.

Conversational Survey Frontends –How Chatbots can improve Online Surveys
First question:
Do participants with a chat interface have a higher
completion rate?
No, surprisingly they did not. On the contrary:
81% 93%
     
We suspect technical problems as primary reason.

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Okay, next, what about data quality?
Were there any differences with regard to response
styles?
No:
0%
25%
50%
75%
100%
Score ±SE
Extreme Response Style Tendency to the middle
           

Conversational Survey Frontends –How Chatbots can improve Online Surveys
What about the frequency of giving no answer?
No differences as well…
0
1
Average Number of „No answer“
responses ±SE
Main Effect Group:        
Main Effect Device Type:        
Interaction Group x Device Type:       

Conversational Survey Frontends –How Chatbots can improve Online Surveys
And the average text length of open questions?
Differences between devices were expected.
0
5
10
15
20
25
30
Average Length of answers to open
questions ±SE
  ,   
Main Effect Group:        
Main Effect Device Type:        
Interaction Group x Device Type:        
Post-hoc we can also explain the difference between
groups: Chat interfaces showed only single-line text fields
while the questionnaire had multi-line fields.

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Were there any differences regarding completion time?
No, no median differences were detected.
779 sec 771 sec
     
Cool. So, if I understand you puny humans correctly: There
are no differences in data quality just because participants
answer questions in a chat interface.
Yep.

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Here’s another thought:
If people feel like they are having a conversation with
someone in the chat interface, they might be responding
more socially desirable.
Good idea! We’ve checked that too. No differences though.
3
5
7
9
11
13
15
Average Score ±SE
Self Deception Impression Management
           

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Honest humans… more or less… ok!
I wonder what participants think about this way of
answering a survey…
Well… we asked them.
Responses were very positive. In fact, for both groups.
There were only differences regarding the novelty of the
approach.
In an open question, participants were very positive about
the chat frontend. They compared it to a messenger or a
conversation. While not clearly visible in the comparative
statistics, participants seem to have liked the style.
Hooray! So… that’s it?

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Now, let’s summarize:
Do participants with a chat interface have a higher
completion rate?
Nope.
Is the data quality higher for participants in a chat?
Nope.
Does the conversational style influence social desirability?
Nope.
But participants like the survey frontend of a chat?
Yep.

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Well, that’s great! People fancy me!
Only chatbot surveys from now on until the end of times!
If it is not superior in any way, then why waste time on the
development of such an interface? Better stick to what is
known, proven and readily available.
Moreover, participants are familiar with questionnaires
and know what is expected of them.

Conversational Survey Frontends –How Chatbots can improve Online Surveys
Okay, okay… So what’s next?
Easy, pal!
Chat interfaces can be useful. But we have work to do:
1. Explain higher drop-out rates.
2. Test the chatbot in different areas/situations.
3. Apply Machine Learning techniques for dynamic
questions and responses
Sounds great! I’m all in!

Conversational Survey Frontends –How Chatbots can improve Online Surveys
And who are you, guys?
Christopher Harms
Consultant Research & Development
christopher.harms@skopos.de
+49 (0) 2233 99 88 - 513
Sebastian Schmidt
Senior Research Executive
sebastian.schmidt@skopos.de
+49 (0) 2233 99 88 - 504
This work is licensed under a Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/)
Suggested citation: Harms, C. & Schmidt, S. (2017). “Conversational Survey Frontends: How Chatbots Can Improve Online
Surveys”. General Online Research (GOR) Conference, Berlin.