Conference PaperPDF Available

On Utilizing Player Models to Predict Behavior in Crowdsourcing Tasks

Authors:

Abstract

Player Modeling is a research field that studies player characteristics by analyzing in-game behavior. We aim to develop independent models, which are transferable and useful beyond a game’s context. We shall demonstrate the feasibility of this approach by applying player models to crowdsourcing to predict workers’ task completion effectiveness. Specifically, we model a user’s Need for Cognition based on in-game behavior, and based on that try to assign appropriate tasks to workers.
On Utilizing Player Models to Predict Behavior in
Crowdsourcing Tasks
Carlos Pereira Santos1,2, Vassilis-Javed Khan2, and Panos Markopoulos1
1 Eindhoven University of Technology, Eindhoven, The Netherlands
{c.a.pereira.santos, p.markopoulos}@tue.nl
2 NHTV Breda University of Applied Sciences
{santos.c, khan.j}@nhtv.nl
Abstract. Player Modeling is a research field that studies player characteristics
by analyzing in-game behavior. We aim to develop independent models, which
are transferable and useful beyond a game’s context. We shall demonstrate the
feasibility of this approach by applying player models to crowdsourcing to pre-
dict workers’ task completion effectiveness. Specifically, we model a user’s
Need for Cognition based on in-game behavior, and based on that try to assign
appropriate tasks to workers.
1 Introduction
Every single player interaction within a video-game has the potential of generating
data. Monitoring in-game data to gain a better understanding of player behavior has
become common-practice in the industry with the intent of improving game develop-
ment, content and revenue. Each player’s in-game behavior can be tracked, for exam-
ple, the sequence of pressed buttons, the usage of specific words, the response time
for a certain part of the game, or the frequency of game-specific actions such as walk-
ing, picking up items or purchasing items.
By using data analysis techniques like simply counting the occurrences of specific
events [1], or more complex clustering techniques [2] it is possible to obtain general
information about the player, like: personality traits, age groups and skill sets. Player
modeling is a recent research field under the general umbrella of Computational Intel-
ligence directly applied to games, which recent research studies show to attain not
only player skill levels but also predict specific players’ traits.
Those models are currently used to improve video-game experiences, during new
development cycles like associating specific play styles with player age [3], or using
avatars to profile personality traits [4], or even to reduce a game’s difficulty and im-
prove player experience [5].
In a recent chapter [6] defined four different purposes for using player models: i)
game balancing, i.e., adapt the game experience based on the player skill, ii) personal-
ized content, i.e., procedural generation of more content, iii) game authoring, i.e.,
playtest to determine whether it provides the desired player experience and iv) mone-
tization, i.e., increase revenue, popular in free-to-play games.
However, we think that there is a greater potential for player models. The work
we have been developing is focused on creating Context Independent Player Models
(CIPMs), by utilizing existing methodologies to collect and analyze in-game player
data and build player models which can be applied for other than game systems. More
specifically we aim to investigate whether and to what extent player models can be
utilized to identify more suitable workers in crowdsourcing systems. This aim re-
quires a two-step process. First one needs to extract CIPMs and second one would
need to check the effectiveness of it in crowdsourcing task completion.
2 Prior Art
Crowdsourcing tasks are making an important contribution to society by providing
jobs, while still allowing enterprises to maintain quality and cut costs. Apart from the
obvious motivation of financial benefits there are more reasons for people to engage
on crowdsourcing such as: altruism, enjoyment, reputation and socialization [7].
There are already some examples that illustrate the potential of our approach. The
Malaria Training Game [8] is a first example combining gaming aspects and crowd
intelligence. It utilizes learning capabilities of human crowds to conduct reliable mi-
croscopic analysis of biomedical samples. A second example is Quizz [9]. It is a
crowdsourcing game, which assesses a user’s knowledge within a specific field and
enrich the system’s knowledge base. In both systems, the game results (answers) are
used to improve or validate knowledge. The difference in our approach is that these
and other similar systems can be improved if the in-game player behavior is analyzed,
providing a richer knowledge about the player, not only using the game results but
also the way game is played. Hence the need to create CIPMs, which would be valid
not only within the game context, but also, as a cross-domain profiling tool.
Most of crowdsourcing task platforms do not consider individual workers abilities,
personality, and context when assigning tasks. Recent studies have showed that user
profiling tools could improve quality and workers satisfaction [10, 11]. Integration of
CIPM to profile crowdsourcing workers would bring advantages such as: i) keep
workers more motivated, ii) it would be more difficult for workers to cheat in self-
reported profiling tools, iii) it would be an unobtrusive way of creating a player model
and iv) can complement existing services or be integrated within the task.
3 Case Study: Need for Cognition as CIPM
Most of player modeling studies are performed based on data-mining techniques
which try to statistically correlate (large) set of collected variables about player in-
game behavior. In the research line we propose a different approach: to design game
mechanics, which are able to determine specific player models.
For this reason we focused our research on a specific personality trait Need for
Cognition (NC). NC is a personality trait associated with the extent a certain person is
inclined to perform cognitive activities. NC researches the link between personality
and behavior, and can be reliable measured through a self-report questionnaire [12].
NC is a simple and stable personality model, and it has been applied across con-
texts to make predictions of user behavior, e.g., how to effectively advice based on
whether someone will likely to follow the specific persuasion techniques [13]. High
NC defines a personality which is more inclined to mental problem solving, assess-
ment of situations with higher degree of elaboration; persons with lower NC are prone
to less elaboration and follow more heuristic/empirical strategies. The dualism of this
personality trait creates an area to explore inference using CIPM. Although, the NC
scale has been used within the video-games as a tool to determine players’ profiles
[14, 15], there is no reference of being inferred through in-game behavior.
NC is also important for specific crowdsourcing tasks. Studies on profiling labor
compared crowdsourcing workers to other recruiting methods and concluded that
workers are representative of the general population and substantially less expensive
to recruit. With regard to NC, workers are very similar, but, there is a slight skew to
having more individuals with higher NC within crowdsourcing workers [16].
To our knowledge NC is being used to serve as profiling tool within both contexts
(games and crowdsourcing), but, never being inferred through game behavior or being
used to predict and influence crowd tasks; this makes NC a good and credible per-
sonality trait for us to study as CIPM.
Based on the aforementioned analysis of existing studies, the key research ques-
tions we are interested in addressing are:
Can Need for Cognition predict people’s behavior in crowdsourcing applications?
Can certain game mechanics be designed to reliably infer Need for Cognition?
3.1 Method and Current study
We are currently in the exploratory phase, where we are evaluating the possibility of
inferring NC based on users’ gameplay. We have already developed a game which
integrates a mechanic to specifically to measure NC (Fig. 1). This mechanic is based
on the work of Boatman [15] which states that individuals with a high NC are pro-
cess-oriented as opposed to outcome-oriented. More specifically, in our game we ask
players to control the movement of a set of units. We embedded a hint system (using
game AI) which provides one possible (non-optimal) movement to support the deci-
Fig. 1. The mobile game we developed to integrate mechanics to infer a player’s NC
sion making process. In theory, higher NC players are driven by the cognition and the
process, not the outcome, hence, we should observe that high NC players will use and
follow less hints, unlike lower NC which are goal oriented and opt to take mental
shortcuts, consequently, follow hints more frequently. The in-game data will be com-
pared to NC measured using the self-report questionnaire [12]. The game is available
online
1
and we are gathering data.
References
1. Drachen, A., Canossa, A.: Towards gameplay analysis via gameplay metrics. Proc.
MindTrek ’09. pp. 202–209. ACM Press (2009).
2. Mahlmann, T., Drachen, A., Togelius, J., Canossa, A., Yannakakis, G.N.: Predicting
player behavior in Tomb Raider: Underworld. Proc. CIG2010. pp. 178185 (2010).
3. Tekofsky, S., Spronck, P., Plaat, A., van den Herik, J., Broersen, J.: Play style:
Showing your age. Proc. CIG2013. pp. 18. IEEE (2013).
4. Yee, N., Ducheneaut, N.: Introverted elves & conscientious gnomes: the expression of
personality in world of warcraft. Proc. CHI2011 pp. 753762 (2011).
5. Zook, A.E., Riedl, M.O.: A temporal data-driven player model for dynamic difficulty
adjustment. Proc. AIIDE 2012. pp. 9398 (2012).
6. Yannakakis, G.N., Spronck, P., Loiacono, D., André, E.: Player Modeling,
http://drops.dagstuhl.de/opus/volltexte/2013/4335/, (2013).
7. Quinn, A.J., Bederson, B.B.: Human Computation: A Survey and Taxonomy of a
Growing Field. Proc. CHI2011 pp. 14031412. ACM (2011).
8. Luengo-Oroz, M.A., Arranz, A., Frean, J.: Crowdsourcing malaria parasite
quantification: an online game for analyzing images of infected thick blood smears. J.
Med. Internet Res. 14, e167 (2012).
9. Ipeirotis, P.G., Gabrilovich, E.: Quizz: targeted crowdsourcing with a billion
(potential) users. Proc. 23rd Int. Conf. World wide web. 143154 (2014).
10. Difallah, D.E., Demartini, G., Cudré-Mauroux, P.: Pick-a-crowd: tell me what you
like, and i’ll tell you what to do. 367–374 (2013).
11. Sarasua, C., Thimm, M.: Microtask Available, Send us your CV! 2013 International
Conference on Cloud and Green Computing. pp. 521524. IEEE (2013).
12. Cacioppo, J.T., Petty, R.E., Kao, C.F.: The efficient assessment of need for cognition.
J. Pers. Assess. 48, 306307 (1984).
13. Kaptein, M., Markopoulos, P., Ruyter, B. de, Aarts, E.: Can You Be Persuaded?
Individual Differences in Susceptibility to Persuasion. Proc. INTERACT 2009.
14. Hoffman, B., Nadelson, L.: Motivational engagement and video gaming: a mixed
methods study, (2010).
15. Boatman, P.R., Day, E.A., Kowollik, V., Espejo, J., McEntire, L.E.: Modeling the
links between need for cognition and the acquisition of a complex skill, (2007).
16. Berinsky, A.J., Huber, G.A., Lenz, G.S.: Evaluating Online Labor Markets for
Experimental Research: Amazon's M.T.. Polit. Anal. 20, 351368 (2012).
1
http://imi.nhtv.nl/nanobots/
ResearchGate has not been able to resolve any citations for this publication.
Conference Paper
Full-text available
Crowdsourcing allows to build hybrid online platforms that combine scalable information systems with the power of human intelligence to complete tasks that are difficult to tackle for current algorithms. Examples include hybrid database systems that use the crowd to fill missing values or to sort items according to subjective dimensions such as picture attractiveness. Current approaches to Crowdsourcing adopt a pull methodology where tasks are published on specialized Web platforms where workers can pick their preferred tasks on a first-come-first-served basis. While this approach has many advantages, such as simplicity and short completion times, it does not guarantee that the task is performed by the most suitable worker. In this paper, we propose and extensively evaluate a different Crowdsourcing approach based on a push methodology. Our proposed system carefully selects which workers should perform a given task based on worker profiles extracted from social networks. Workers and tasks are automatically matched using an underlying categorization structure that exploits entities extracted from the task descriptions on one hand, and categories liked by the user on social platforms on the other hand. We experimentally evaluate our approach on tasks of varying complexity and show that our push methodology consistently yield better results than usual pull strategies.
Conference Paper
Full-text available
We describe Quizz, a gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz operates by asking users to complete short quizzes on specific topics; as a user answers the quiz questions, Quizz estimates the user's competence. To acquire new knowledge, Quizz also incorporates questions for which we do not have a known answer; the answers given by competent users provide useful signals for selecting the correct answers for these questions. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging the targeting capabilities of existing, publicly available, ad placement services. Quizz quantifies the contributions of the users using information theory and sends feedback to the advertisingsystem about each user. The feedback allows the ad targeting mechanism to further optimize ad placement. Our experiments, which involve over ten thousand users, confirm that we can crowdsource knowledge curation for niche and specialized topics, as the advertising network can automatically identify users with the desired expertise and interest in the given topic. We present controlled experiments that examine the effect of various incentive mechanisms, highlighting the need for having short-term rewards as goals, which incentivize the users to contribute. Finally, our cost-quality analysis indicates that the cost of our approach is below that of hiring workers through paid-crowdsourcing platforms, while offering the additional advantage of giving access to billions of potential users all over the planet, and being able to reach users with specialized expertise that is not typically available through existing labor marketplaces.
Article
Full-text available
We examine the trade-offs associated with using Amazon.com's Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples—the modal sample in published experimental political science—but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples.
Conference Paper
Full-text available
Age has been shown to influence our preferences, choices, and cognitive performance. We expect this influence to be visible in the play style of an individual. Player models would then benefit from incorporating age, allowing developers to offer an increasingly personalized game experience to the player. To investigate the relationship between age and play style, we set out to determine how much of the variance in a player's age can be explained by his play style. For this purpose, we used the data from a survey ('PsyOps') among 13,376 'Battlefield 3' players. Starting out with 60 play style variables, we found that 45.7% of the variance in age can be explained by 46 play style variables. Furthermore, similar percentages of variance in age are explained when the sample is divided along gaming platform: 31 play style variables explain 43.1% on PC; 30 play style variables explain 53.9% on Xbox 360; 28 play style variables explain 51.7% on Playstation 3. Our findings have a high external validity due to the large and heterogeneous nature of the sample. The strength of the relationship between age and play style is considered 'large' according to Cohen's classification. Previous research indicates that the nature of the relationship between age and play style is likely to be based on life-time developments in cognitive performance, motivation, and personality. All in all, our findings merit a recommendation to incorporate age in future player models.
Conference Paper
Full-text available
User-oriented research in the game industry is undergoing a change from relying on informal user-testing methods adapted directly from productivity software development to integrating modern approaches to usability-and user experience testing. Gameplay metrics analysis form one of these techniques, being based on instrumentation methods in HCI. Gameplay metrics are instrumentation data about the user behavior and user-game interaction, and can be collected during testing, production and the live period of the lifetime of a digital game. The use of instrumentation data is relatively new to commercial game development, and remains a relatively unexplored method of user research. In this paper, the focus is on utilizing game metrics for informing the analysis of gameplay during commercial game production as well as in research contexts. A series of case studies are presented, focusing on the major commercial game titles Kane & Lynch and Fragile Alliance.
Article
Full-text available
There are 600,000 new malaria cases daily worldwide. The gold standard for estimating the parasite burden and the corresponding severity of the disease consists in manually counting the number of parasites in blood smears through a microscope, a process that can take more than 20 minutes of an expert microscopist's time. This research tests the feasibility of a crowdsourced approach to malaria image analysis. In particular, we investigated whether anonymous volunteers with no prior experience would be able to count malaria parasites in digitized images of thick blood smears by playing a Web-based game. The experimental system consisted of a Web-based game where online volunteers were tasked with detecting parasites in digitized blood sample images coupled with a decision algorithm that combined the analyses from several players to produce an improved collective detection outcome. Data were collected through the MalariaSpot website. Random images of thick blood films containing Plasmodium falciparum at medium to low parasitemias, acquired by conventional optical microscopy, were presented to players. In the game, players had to find and tag as many parasites as possible in 1 minute. In the event that players found all the parasites present in the image, they were presented with a new image. In order to combine the choices of different players into a single crowd decision, we implemented an image processing pipeline and a quorum algorithm that judged a parasite tagged when a group of players agreed on its position. Over 1 month, anonymous players from 95 countries played more than 12,000 games and generated a database of more than 270,000 clicks on the test images. Results revealed that combining 22 games from nonexpert players achieved a parasite counting accuracy higher than 99%. This performance could be obtained also by combining 13 games from players trained for 1 minute. Exhaustive computations measured the parasite counting accuracy for all players as a function of the number of games considered and the experience of the players. In addition, we propose a mathematical equation that accurately models the collective parasite counting performance. This research validates the online gaming approach for crowdsourced counting of malaria parasites in images of thick blood films. The findings support the conclusion that nonexperts are able to rapidly learn how to identify the typical features of malaria parasites in digitized thick blood samples and that combining the analyses of several users provides similar parasite counting accuracy rates as those of expert microscopists. This experiment illustrates the potential of the crowdsourced gaming approach for performing routine malaria parasite quantification, and more generally for solving biomedical image analysis problems, with future potential for telediagnosis related to global health challenges.
Article
We examine the trade-offs associated with using Amazon.com's Mechanical Turk (MTurk) interface for subject recruitment. We first describe MTurk and its promise as a vehicle for performing low-cost and easy-to-field experiments. We then assess the internal and external validity of experiments performed using MTurk, employing a framework that can be used to evaluate other subject pools. We first investigate the characteristics of samples drawn from the MTurk population. We show that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples-the modal sample in published experimental political science-but less representative than subjects in Internet-based panels or national probability samples. Finally, we replicate important published experimental work using MTurk samples. © The Author 2012. Published by Oxford University Press on behalf of the Society for Political Methodology. All rights reserved.
Article
Many computer games of all genres pit the player against a succession of increasingly difficult challenges such as com-bat with computer-controlled enemies and puzzles. Part of the fun of computer games is to master the skills necessary to complete the game. Challenge tailoring is the problem of matching the difficulty of skill-based events over the course of a game to a specific player's abilities. We present a ten-sor factorization approach to predicting player performance in skill-based computer games. Our tensor factorization ap-proach is data-driven and can predict changes in players' skill mastery over time, allowing more accurate tailoring of chal-lenges. We demonstrate the efficacy and scalability of ten-sor factorization models through an empirical study of human players in a simple role-playing combat game. We further find a significant correlation between these performance ratings and player subjective experiences of difficulty and discuss ways our model can be used to optimize player enjoyment.
Article
Need for cognition (NFC) refers to the tendency for individuals to engage in and enjoy effortful cognitive activity (Cacioppo & Petty, 1982). Much of the research on NFC has been concerned with information-processing in situations involving persuasion and argumentation. Despite conceptual underpinnings suggesting NFC could be a potential determinant of complex skill acquisition, little empirical research has examined the role of NFC in training or skill acquisition contexts. Therefore, the purpose of this laboratory study was to test the proposition that NFC is a trait-like motivational construct that distally influences complex skill acquisition through more proximal motivational states. Accordingly, 411 young adult males trained for 5h on a novel and complex computer task that involved strong cognitive and psychomotor demands. Performance feedback was provided throughout training with monetary incentives for high performance. In addition to NFC, participants completed measures of state goal orientation, self-efficacy, and general cognitive ability (g). The results supported our proposition and showed that NFC was associated with, but distinguishable from g and linked to skill acquisition through the mediating roles of learning orientation and self-efficacy.