Conference PaperPDF Available

Abstract and Figures

This paper describes our research investigating the perception of difficulty in video games, defined as players’ estimation of their chances of failure. We discuss our approach as it relates to psychophysical studies of subjective difficulty and to cognitive psychology research into the overconfidence effect. The starting point for our study was the assumption that the strong motivational pull of video games may lead players to become overconfident, and thereby underestimate their chances of failure. We design and implement a method for an experiment using three games, each representing a different type of difficulty, wherein players bet on their capacity to succeed. Our results confirm the existence of a gap between players’ actual and self-evaluated chances of failure. Specifically, players seem to underestimate high levels of difficulty. The results do not show any influence on difficulty underestimation from the players gender, feelings of self-efficacy, risk aversion or gaming habits.
Content may be subject to copyright.
From Objective to Subjective Difficulty
Evaluation in Video Games
Thomas Constant, Guillaume Levieux, Axel Buendia, and St´ephane Natkin
Conservatoire National des Arts et M´etiers, CNAM-C´edric
292 Rue St Martin, FR-75141 Paris Cedex 03
Abstract. This paper investigates the perception of difficulty in video
games, defined as the players’ estimation of their chances of failure. We
discuss our approach with regard to the psychophysical studies of subjec-
tive difficulty and to the cognitive psychology research on overconfidence
bias. We assume that the strong motivational pull of video games may
lead players to be overconfident and underestimate their chances of fail-
ure. Our method is tested within three games related to three types of
difficulty, where the players have to bet on their capacity to win each
challenge. Results confirm the existence of a gap between the players
actual and self-evaluated chances of failure. More precisely, players seem
to strongly underestimate high levels of difficulty. Results do not show
any influence of the players gender, feeling of self-efficacy, risk aversion
and gaming habits on the difficulty estimation error.
Keywords: User Modelling ·Affective HCI, Emotion, Motivational As-
pects ·Tools for Design, Modelling, Evaluation ·Fun / Aesthetic Design
1 Introduction
Jesper Juul proposes to define a video game as “a rule-based formal system
with a variable and quantifiable outcome, where different outcomes are assigned
different values, the player exerts effort in order to influence the outcome, the
player feels attached to the outcome, and the consequences of the activity are
optional and negotiable” [1]. In this definition, the player exerts effort to influence
the outcome, which emphasizes the fact that a game must have a certain level
of difficulty to be considered as such.
Many authors acknowledge challenge as one of the most fundamental aspect
of video games’ inherent appeal. Malone proposes three features of computer
games that make them so captivating: challenge, curiosity and fantasy [2]. In his
model, challenge is directly related to the game’s difficulty and corresponds to
the uncertainty for the player to reach the game’s goals. Lazzaro proposes a four
factor model, where Hard Fun is related to the feeling of overcoming difficult
tasks [3]. Sweetser et al see also challenge as one of the most important part
of their Game Flow framework [4]. Their work stems from Mihaly Csikszent-
mihalyi’s Theory of Flow [5], who has been trying to figure out the properties
2 Constant et al.
of activities showing a strong, intrinsic ability to motivate. Csikszentmihalyi re-
search states that these activities provide perceived challenges, or opportunities
for action, that stretch (neither overmatching nor underutilizing) existing skills
[5]. The study of large population of players in two commercial games confirm
that players prefer specific levels of difficulty[6] Ryan et al study as well intrin-
sic motivation and apply their Self-Determination Theory to video games. They
show how enjoyment is related to the feeling of competence, which relies on an
optimal level of challenge, and thus, to the game’s difficulty [7]. Jesper Juul
provided insight on how failure, and thus difficulty, is one of the core aspects of
video game enjoyment and learning progression [8,9].
In order to foster and maintain the players’ motivation, it is thus fundamental
to correctly set the difficulty of a video game. One can provide different difficulty
settings for the player to select, or use an algorithm that adapts the gameplay’s
difficulty in real time to match the game designer’s theoretical difficulty curve
with regard to the current player skills [10–12].
It requires beforehand to evaluate the game’s difficulty. The game designer
might provide a heuristic which may or may not express the game’s difficulty.
We may use sensors to estimate workload or affective state, but currently only in
a lab setting and this question is still by itself a research topic [13,14]. We could
also try to estimate the players’ chances of failure [15]. All these approaches
provide insight on a specific aspect of a game’s difficulty.
The point is, difficulty is by itself a complex notion. We may draw distinctions
between skill-based difficulty, effort-based difficulty [16], and between sensory,
logical and motor difficulty [15, 17]. Moreover, video games are created for an
aesthetic purpose, evoking specific emotion in the player [18]. Thus, we must
draw a fundamental distinction between objective difficulty and subjective dif-
ficulty. Objective difficulty is to be directly estimated by observing gameplay
variables and events, while subjective difficulty is a psychological construct of
the player. When adapting a game’s difficulty, especially when using a dynamic
difficulty adjustment (DDA) algorithm, we rely on an objective estimation of
difficulty, which may be quite different to what the player actually feels while
playing the game.
The main objective of this paper is to study the relationship between sub-
jective and objective difficulty, in the context of video games. We thus review
different studies on both subjective and objective difficulty estimation in the
next sections. First, we present the psychophysical approach of perceived diffi-
culty. Second, we report cognitive psychology studies on overconfidence. Then,
we introduce our method for measuring objective and subjective difficulty. Ob-
jective difficulty is modelized using a logit mixed effect regression to estimate
the player’s actual chances of failure for a given challenge. Subjective difficulty is
considered to be the player’s estimation of his chances of failure, that we gather
using an in-game bet system. We then present the three games we developed
for this study, allowing us to separate logical, motor and sensory gameplays. We
detail and discuss our results in the last sections.
From Objective to Subjective Difficulty 3
2 Psychophysical approach to subjective difficulty
Many studies have tried to clarify the link between subjective and objective
difficulty in various tasks: Raven’s progressive matrices, digits memorization, vi-
sual search of letters, wire labyrinth [19, 20], Fitts’ tapping task [21, 22], dart
throwing on a moving target [23], rock climbing [24] or reaction time, even while
riding a bike [19, 25]. All these experiments take a psychophysical approach, try-
ing to estimate the link between objective difficulty as a stimulus, and subjective
difficulty as a perception or evaluation of this stimulus.
These studies use various techniques to estimate objective difficulty, and
often tend to draw a distinction between objective difficulty and performance.
For all the Fitts’s tapping task, authors use the Fitts’s law [26] as a measure
of objective difficulty, and time as a measure of performance. When such a law
is not available, they rely however solely on performance, e.g. response time or
success frequency [23, 20], or select a variable highly correlated with perceived
difficulty like electromyographic data from a specific muscle in the rock climbing
experiment [24]. Also, in these studies, objective difficulty is never assessed with
regard to each subject abilities, but across all or a few subgroups of subjects. In
our research, we do not rely on any specific objective difficulty estimation but
follow a more generic approach that allows cross-game comparisons. We estimate
a mapping between the challenge’s variables and the player’s probability to lose
this challenge. We also use a mixed effect model that can take into account each
player’s abilities.
In these studies, subjective difficulty is assessed by the subject using a free
scale. Very often, a reference value is given to the subject, e.g. subjective difficulty
of 10 for a specific task [21]. Deligniere has proposed the DPE-15 scale, a 7 points
Likert scale with intermediate values, for a more convenient and comparable
measure [23]. In our experiment, we integrate the measure to the gameplay
and will use a specific 7 points scale, described in section 4. To avoid personal
interpretation of the notion of difficulty evoked in the previous section, we will
concentrate on the success probability, as estimated by the player.
Except in Slifkin & Grilli [21], all subjective evaluations are done at the end
of the challenge, often after having repeated the challenge many times. We think
that to understand what the player feels while playing a video game, and not
when thinking back about a past game session, it might be interesting to have
a look at the player’s evaluation of the current difficulty, during each one of his
attempt to overcome a challenge. As our measure of subjective difficulty is an
estimation of failure chances, it can be integrated to the gameplay and thus be
repeated more often without pulling the player out of the game (see section 4).
3 Overconfidence and the Hard/Easy effect
We define subjective difficulty as the players’ own evaluation of their chances
of failure. This evaluation is a complex cognitive process, often rushed, based
on the interpretation of incomplete information about the game state, on in-
game performance feedback, and on assessments of the players own knowledge
4 Constant et al.
and skills with respect to a specific challenge. Cognitive psychology research on
judgmental heuristics studies how such a reasoning can be biased, and can help
us understanding how players may have a wrong evaluation of their chances of
Heuristic approach to judgment and decision-making has opened a vast field
of research to explain human behavior in a context of uncertainty. Kahneman
& Frederick [27, 28] consider that, confronted to a complex decision, people sub-
stitute one attribute of the decision to a simpler one, more available, to reduce
cognitive effort. In some cases, the use of judgmental heuristics can lead to fun-
damental errors, called cognitive biases by Kahneman & Tversky [29].
The overconfidence effect is one of those. Well-studied in the financial field,
this behavior relies on a surrealistic evaluation of our own knowledge and skills,
leading to an overestimation of our abilities or of those of others [30–34]. Over-
confidence seems particularly interesting to study with regard to video games as
those are essentially built to motivate their players. Self-efficacy theory of moti-
vation states that having a strong confidence in his future chances of success is
a key aspect of motivation [35]. Video games, using for instance a well crafted
difficulty curve, may manipulate the players’ perception of their success chances
to keep them motivated.
Overconfidence has already been studied in many games. During games of
bridge, beginners or amateurs players can misjudge both their performances and
play outcomes [36]. The same results were noticed about poker’s players, where
beginners show inferior capacity to predict their odds of winning [37], during
game tournament of poker and chess game [38], or associated with gambling
games [39, 40]. In brief, the overconfidence effect appears when the players have
a limited knowledge of the game and for any type of game, whether it is a pure
game of chance, like fruit machine, or a skill based game, like chess game.
There exist many situations and cognitive biases that influence the overesti-
mation or underestimation of one chances of success, like the level of expertise
[41, 42], the gambler’s fallacy [43, 44], the hot hand bias [45, 44], the illusion of
control [46, 47] or the hard/easy effect. All these aspects of overconfidence are
worth studying in the context of video games but for this research, we chose to
focus on the hard/easy effect. The hard/easy effect specifies that for low and high
level of difficulty, decision-makers cannot estimate the real difficulty of the task
[48]. For low levels, they will underestimate their chances of success, whereas for
high levels, they will overestimate them [41, 33].
Starting from the hard / easy effect, our research focuses on two main points.
First, from a methodological point of view, we want our experiment to be as close
as possible to a real video game. Thus, we use a dynamically adjusted difficulty
that starts from a low level of difficulty, which is a key element of game design.
We also do not evaluate players’ confidence by directly asking them if they
feel confident on a percentage scale, but instead we use a betting mechanism
integrated to the gameplay, avoiding to break the player’s immersion. Second,
our research distinguishes between three types of difficulty. We use three different
From Objective to Subjective Difficulty 5
games, each one of them focusing on a specific kind of difficulty. We fully describe
our experiment in the next section.
4 Experimentation
As stressed out in the previous sections, there exist different types of difficulties in
video games. In this experiment, we want to assess them separately, in an attempt
to distinguish between various facets of video games. We follow Levieux et al [17,
15] approach and consider three categories of difficulty in games: sensory, logical
and motor. Sensory difficulty relates to the effort needed to acquire information
about the game state. Logical difficulty corresponds to the effort needed to induce
or deduce, from the available information, the solution of a problem in terms
of action(s) to perform. Lastly, motor difficulty is related to the physical agility
needed to perform these actions. To realize an accurate analysis of the player’s
behavior for each of these types, the experiment is split between three specifically
designed games.
In this experiment, we choose a general, practical approach and estimate the
probability that a player has to fail at a specific challenge, with respect to his
current skills [15]. As the player will be asked to evaluate his chances of success,
there is no risk for the player to hesitate between effort-based and skill-based dif-
ficulty. Moreover, our definition of difficulty directly follows Malone’s definition
of challenge, as a source of uncertainty in video games [2]. Indeed, uncertainty
about success or failure is what Costikyan calls uncertainty of outcomes [49].
Also, to distinguish between logical, motor and sensory, we designed three dif-
ferent games which are described in section 4.3. In order to maximize the player
motivation and propose an experience as close as possible to an actual game,
the system adapts dynamically the difficulty, analyzing the player’s successes
or failures. Many games use a dynamic difficulty adaptation, like racing games
(mostly rubber banding like in Mario Kart series), RPG (e.g. Fallout series) or
FPS (e.g. Unreal Tournament series) where the difficulty to defeat an opponent
depends on the players level. Games without dynamic difficulty adaptation can
use a predetermined difficulty curve, based on the mean level of the players.
On the opposite, few games use a totally random difficulty, and for those (e.g.
FTL,The Binding of Isaac), there is still a global progression. Thus, randomness
would be more convenient for statistical analysis, but also highly questionable
from a game design perspective.
Furthermore, to avoid any memory bias on the past challenge and to better
monitor the actual feeling of the player, we measure the subjective difficulty
during the game session and not with post-experiment questionnaires [50]. To
be able to do so without pulling the player out of the game, we propose to use
a bet system, described in the next section.
4.1 Measuring Subjective difficulty
Our proposition adopts the cognitive psychology tools to measure overconfidence,
and integrates them to the gameplay. Our goal is to avoid any disturbance dur-
6 Constant et al.
ing the game session, to keep a high level of engagement and motivation. The
measure is made before the players choices, as a pre-evaluation, but after giving
them all the elements useful to perform their judgment. We use a bet system
based on a 7 points Likert scale, which is integrated into the game progression
and, thus, is related to the players’ score. If the player wins, the amount of the
bet is added to his score; and if he loses, this amount is subtracted, improving
players concentration on their own evaluation. An in-game question is used as
instruction for betting and as a reminder for the players to strictly assess their
own confidence.
Measurement of the subjective difficulty is based on the players’ bet, noted
Dsubj . With bbeing the bet value we use the formula Dsubj = 1 b1
6to get
the estimated chances of failure.
4.2 Measuring Objective difficulty
The objective difficulty of a challenge is estimated from the players’ failures
and successes for that challenge, as Levieux et al defined it [17, 15]. In order
to take into account personal differences, we estimate for each challenge the
objective difficulty by fitting a logit mixed-effects model [51]. Time and difficulty
parameter of each challenge (e.g. cursor speed, number of cells...) are used as
fixed effect parameters, and we add random intercepts. We use a mixed model
through repeated evaluations of the same subject. The random intercepts give us
a coefficient for each players that we use as a global evaluation of each player’s
level. The gap between the players’ objective difficulty and their evaluation of
odds of winning is called the difficulty estimation error. The design of the three
games, each one based on a difficulty type - logical, motor or sensory -, is detailed
in the next section.
4.3 Game’s description
The experiment is based on the observation of the players’ bet for the three
dimensions of difficulty. Each dimension is represented by a specific game, de-
scribed as follows, for which all the adjustment variables for the challenges are
pre-established and common for all players. A first series of playtests was also
conducted with the target audience, in the same settings used during the exper-
iments, for gameplay calibration.
A brief story was included in order to improve the player’s motivation and
to have a narrative justification of the bet system. In the game universe, the
players have to save citizens of a mysterious kingdom, transformed into sheep by
the local sorcerer. The players challenge the sorcerer during three tests, one for
each kind of difficulty. The players’ unique objective is to save as many sheep as
possible. Then, each game is an opportunity for the player to save doom citizens,
by betting one to seven sheep on their odds of winning.
All games show a common user interface, except for the central frame that
depends on each sub-game (figure 1). All important information is displayed
at the bottom of the central frame, like the number of remaining turns, the
From Objective to Subjective Difficulty 7
global score, and remaining number of actions for the logical game. Directives
are placed just below the main title, on a colored banner, blue for directives,
red for corrective feedbacks. A rules reminder is accessible at the bottom of the
Feedbacks are provided all along the player’s progression, right after the
results presentation. Positive (on green background) and negative ones (on red)
are displayed on the two sides of the screen, allowing the players to constantly
follow their amount of saved and lost sheep. Sounds are associated to them;
one bleating for a saved sheep, one sorcerer’s mocking laugh for a lost one.
Animations are used to aim for a more stimulating in-game interface.
For each game, we modify the difficulty using a difficulty parameter. This
parameter varies from 0 to 1 and is used to interpolate gameplay parameters,
defined in the next section. For all games, the difficulty parameter starts at 0.2,
and increases or decreases by 0.1 step after each turn based on players’ success
or failure.
(a) Logical task (b) Sensory task (c) Motor task
Fig. 1: Game interface for the logical, sensory and motor tasks. Logical task is shown
with the whole user interface, while we only show the center frame for the motor and
sensory task. Screenshots were taken for the easiest levels of difficulty.
Logical difficulty The logical task is based on a well-known sliding-puzzle
game. The players have to restore the numerical order of a 300 pixels wide
grid composed of 9 squares. The fifth square, originally placed on the middle of
the grid, is the only one that can be moved. This square can only be moved by
switching position with another, adjacent square (figure 1a). At the beginning of
each turn, before displaying the grid, the fifth square is randomly moved several
times, and the mixed up grid is displayed for 20 seconds before disappearing.
The players have all the information required to place a bet: the remaining
time is visible and the number of inversion is specified. After betting, the grid
will reappear and the players can begin to move the fifth square to restore the
numerical ordering. The difficulty parameter allows us to adapt the difficulty
8 Constant et al.
by changing the number of steps during the randomization of the grid, linearly
from 1 to 11 steps.
Sensory difficulty For the sensory difficulty, we designed a 300 pixels wide
grid composed of multiple squares (figure 1b). At the end of a countdown timer,
five of them will fade out during a limited time, that we can approximate as
follows, with tbeing the fade out time and das the difficulty parameter: t=
d20.24d+ 1.21. Then, the players have to find them back, by clicking on
the grid. The selected squares are displayed in a blue color, whereas the other
remains in a gray one, to avoid any color perception bias. The winning squares
will be shown after making a bet, over the players’ selection. By doing so, we
want to induce a near-miss effect, allowing the players to see if they selected
all, some or none of the winning squares. The countdown timer is blocked on 3
seconds. The number of squares will vary with the difficulty of the task: when
the player wins, the grid will gain one square on each side. But the surface of
the grid constantly remains the same, implying that the squares will become
smaller after a winning round. The maximum difficulty levels is a grid composed
of 11 squares on a side; the minimum levels is one composed of 4 squares. This
values are linearly interpolated for in-between difficulties using the difficulty
parameter. Winning squares’ random location is chosen to avoid the most simple
patterns, and thus, to minimize pattern-induced variations of difficulty for a
specific difficulty parameter value. For example, for a 5 squares sided grid, any
adjacent winning squares are forbidden.
Motor difficulty The motor difficulty is a basic and common reflex-based
task. A cursor goes back and forth along a horizontal segment, with a linear
speed. The players have to stop the cursor when it covers a black mark at the
center (figure 1c). They can only stop the cursor by clicking on a button. Before
that, the players have to bet on their chances to success. This evaluation is not
timed. The difficulty of the game is based on the cursor’s speed, ranging linearly
from 100 to 400 pixels per seconds. The sliding area is 320 pixels wide, the cursor
is 15 pixels wide, and the black target 2 pixels wide.
Protocol consistency These three tasks, although different in nature, do
share a similar protocol and always provide the player with the elements needed
to evaluate the difficulty. For the motor task, players can observe the moving
cursor before betting. For the logical task, the game displays the number of
inversions and let the player look at the problem for a fixed duration. For the
sensory one, where the visual memory is crucial, the player select tiles to solve the
problem, but without any feedback, before betting. Previous playtests showed
that the task was really frustrating if the player had to stop focusing on the
grid for betting without selecting the tiles. Each game has a specific gameplay,
as each one focuses on a specific dimension of difficulty. Results can thus be
compared between games, while carefully discussing the gameplay differences.
1This equation is a quadratic regression of the fade out time. In the game, the color is
incrementally modified in the game loop, but plotting this equation is much clearer
than reading the color update code.
From Objective to Subjective Difficulty 9
4.4 Procedures
Experiments were conducted in Paris at the Cit´e des sciences de l’industrie, a
national museum dedicated to science and critical thinking, during ten days of
the All Saint’s vacations. The target audience was composed of young gamers
and non-gamers population, and they were free to participate. A few declined
the invitation, judging their experience about video game as too weak, or by
lack of interest about participating in a science experiment.
Nine laptops, with the same configuration, were dispatched in an isolated
room. Each one had a mouse and a headset. The main program runs on a
web browser, and was developed with JS, HTML5 and CSS. Participants were
informed that the game’s goal was to save as much sheep as possible, and of the
duration of the experiment, approximately 40 minutes, questionnaire included.
They were not allowed to communicate between them during the session. Before
starting to play, the participants had to fill an online questionnaire employed to
realize several user profile:
A gaming habits profile, based on the amount of time that participants
spent playing board games, video games (including social games) and gam-
bling games.
– A self-efficacy profile, based on General Self-Efficacy scales [52, 53] and
adapted to video games situations. This part of the questionnaire is only
accessible for the participants who answered yes to the question ”Do you
consider yourself as a video games player?” in the gaming habits section. The
purpose of it is to verify any negative or positive effect of the participant’s
gaming capacities self estimation on his confidence.
– A risk aversion profile, based on Holt and Laury Ten-Paired Lottery-
Choices [54] in order to evaluate the impact of risk incentive on the players’
The game macro-progression is the same for all players:
Aprologue introduces the story before a random selection of the three
A specific page presents the rules of the mini-game, just before playing it.
Players can take as much time as they want to understand them.
Each task, or mini-game, lasts 33 turns. The first three turns are used as a
practice phase. At the end of this practice phase, the score is reset to zero.
The turn progression is identical for all the mini-games. First, players
have to observe the current game state in order to evaluate the difficulty.
Then, they have to bet from 1 to 7 sheep about their confidence to succeed.
The same question is always asked to the player: ”How much sheep are you
betting on your chances of winning?”. This question allows us to estimate
the players’ perception of his chances of failure. By validating the bet, the
system unlocks the game and players can try to beat the challenge. The result
is presented on screen and the score is updated at the same time. Then, a
new turn begins, with the adequate modification of the difficulty level: when
players win, the difficulty increases; when they lose, the difficulty decreases.
10 Constant et al.
After each task, a screen allows the player to check his/her progression and
score, and works as a game hub, enabling the connection to an other task.
Then, by completing the three tasks, a brief epilogue of the story announces
the player final score, the sum of all sheep won and lost.
To avoid any order effect, the selection of the task is randomized. The best score
of the day was written on a board, visible by the players. At the end of each turn,
the designed difficulty of a challenge, the players’ bet, their score, are recorded
on CSV files.
5 Results
A total of 80 participants have played the games. Some of them have left the
experiment before the end, but we keep results for all completed games, giving
us a total of 6990 observations. For each task, we remove outliers, such as players
who did not use the bet to perform a self-assessment, placing always the same
bet, or players with outlying performance. A very low score may reflect some
user experience issues, and some players took advantage of the adaptive difficulty
system in order to maximize their score, by deliberately losing with a low bet,
then by placing a high bet on the next easier challenge and so on. Nine outliers
have been removed: one from the motor task, three from the perceptive task,
and six for logical one. We thus removed 300 observations from the dataset.
5.1 Modeling objective difficulty
As explained in section 4.2, we perform a logit mixed effect regression to evaluate
the objective difficulty. For each task, we report the conditional R2, i.e. using
both fixed and random effects [55] and evaluate the model by performing a
10-fold cross-validation, using our model as a binary predictor of the challenge
outcome (figure 2).
Parameters / Tasks Logical Motor Sensory
Difficulty parameter 4.88
(p < 2e16)***
(p < 2e16)***
(p < 2e16)***
(p= 2e6)***
(p= 0.0051)**
(p= 0.0454)*
σ(random intercepts) 1.24 0.83 0.76
R20.48 0.28 0.42
Cross Validation 0.66 0.61 0.69
Fig. 2: Modeling objective difficulty for each task: logit mixed effect regression results
for difficulty and time over failures.
From Objective to Subjective Difficulty 11
As can be seen in figure 2, the difficulty parameter is always highly significant,
and has the strongest effect on failure probability, especially for the sensory
task, which means that we were indeed manipulating the objective difficulty by
changing this parameter.
The effect of time is always negative and significant. This means that, if
the difficulty parameter stays constant, objective difficulty seems to decrease
with time. This might indicate that players are actually learning as their success
rate improves with time for a given difficulty parameter value. Time effect is the
strongest for the logical task (1) which is coherent with the fact that the player
should learn more from a logical problem than from a purely sensory motor one
(respectively, 0.46 and 0.37). Also, it may be noted that we have the highest
standard deviation of random intercept for the logical task, which means that
inter-individual differences are the highest for this task.
The link between the difficulty parameter and the objective difficulty of the
game can be plot to better understand each challenge difficulty dynamics. We
choose to plot objective difficulty over difficulty parameter at time t= 0. We
also used the random intercept to separate the player in three groups of levels
using k-means (fig. 3).
(a) Logical task (b) Motor task (c) Sensory task
Fig. 3: Objective difficulty for each task at t= 0. Blue dashed line is the median player,
red dashed lines show first and last quartiles. The less efficient players are in yellow,
medium players in cyan and best players in green.
Curves in figure 3 give us information about our design of each task’s dif-
ficulty. We can see that the logical task is the most balanced, with objective
difficulty being the closest to the difficulty parameter value. The motor task
is is a bit too hard for low difficulty levels: objective difficulty is around 0.25,
when the difficulty parameter is 0. Also, sensory task should vary more slowly :
objective difficulty is maxed when the difficulty parameter is only 0.5.
Figure 4 shows the progression of objective difficulty during the game. The
curves confirm the balancing of each task and the efficiency of the difficulty
adaptation system, as the players reach the average objective difficulty level (0.5)
in all cases. The logical task starts at 0.2 for medium players and goes up. The
motor task is too hard at the beginning, and thus bad players see a decrease of
difficulty with time. Sensory task shows a ”wavy” pattern which might be related
to the fact that the difficulty is less stable for this game. Indeed, the difficulty
12 Constant et al.
Logical task Motor task Sensory task
All playersGood playersMedium playersBad players
Fig. 4: Progression of objective difficulty with time for all tasks and players during the
whole play session. Blue line is the median players, dots represent the observations for
each turn.
parameter varies by 0.1 step for all tasks, but as the maximum objective difficulty
is already reached at 0.5, it varies approximately twice faster than for the logical
Overall, the objective difficulty model is the weakest for the motor task with
a low conditional R2(0.28) and the lowest prediction accuracy (0.61). R2and
prediction accuracy are higher for the logical (R2= 0.48, accuracy = 0.66) and
sensory task (R2= 0.42, accuracy = 0.69).
5.2 Differences between objective and subjective difficulty
To investigate the differences between objective and subjective difficulty, we sep-
arate the data into 16 equally sized bins using the objective difficulty estimated
by the mixed effect model. In each bin, we compute, for each player, the mean
subjective difficulty. We thus have only one value by player in the bin, and each
From Objective to Subjective Difficulty 13
observation is thus independent from the others. Then, for each bin, we test the
null hypothesis that the bin’s median subjective difficulty’s is equal to the objec-
tive difficulty at the center of the bin’s interval. We use a Wilcoxon Signed Rank
Test and compute the 95% confidence interval (red bars) and pseudo median
(black dot and triangles), plotted in figure 5. We only show the pseudo median
and confidence intervals for bins with enough samples to run the Wilcoxon signed
rank test, and blue line represents our null hypothesis, where objective difficulty
equals subjective difficulty. Results allow us to safely reject the null hypothesis
for each median represented by an empty triangle in the plots, where Wilcoxon
signed rank test p-value is lower than 0.05.
Logical task Motor task Sensory task
All players
¯n= 53.3 (σ=20.6) ¯n= 48.8 (σ=28.1) ¯n= 42.8 (σ=10.9)
Good players
¯n= 20.8 (σ=6.51) ¯n= 15.8 (σ=4.44) ¯n= 5.46 (σ=1.39)
Medium players
¯n= 19.8 (σ=5.73) ¯n= 36.2 (σ=16.7) ¯n= 23.7 (σ=6.57)
Bad players
¯n= 17.5 (σ=6.93) ¯n= 7.57 (σ=2.3) ¯n= 13.9 (σ=5.12)
Fig. 5: Subjective and objective difficulty for all tasks and players. ¯nis the mean (sd)
number of players in each bin for each task and level.
14 Constant et al.
There seems to be a strong hard effect for both logical and motor tasks.
For the sensory task, players seem to be slightly overconfident for all objective
difficulties. When split by levels, the effect seems stable for the motor task,
but the relatively low number of bad (¯n= 7.57) and good players (¯n= 15.8)
might lead to the non significance of the results. The same can be seen in the
sensory task where pseudo medians are always under the calibrated evaluation,
but results lose significance with the decrease of subjects number. However, for
the logical task, bins size are equivalent for the three conditions, but medium
players seem better calibrated. This result should be investigated further in a
specific experiment to provide more insightful results.
5.3 Influence of participants’ profiles on the subjective difficulty
We conduct several tests in order to analyze whether gender, gaming habits,
assessment of self efficacy and risk aversion have an impact on players’ levels
and difficulty estimation error. We take the random intercept of the objective
difficulty model as each player’s level.
Out of 80 participants 57 are males and 23 are females. 49 of them daily
play video games, and 12 weekly. 31 play board games monthly, and 36 almost
never. 58 are risk-averse, and for the 46 of the participants who answered the
self-efficacy questionnaire, 28 tend to see themselves as efficient players and all
esteem themselves as superior to a medium player.
First we tested gender influence on players’ levels and difficulty estimation
error with a Wilcoxon rank sum test. The null hypothesis is that both of them are
pulled from the same distribution for each gender. The test was only significant
for players’ level. Females players seem to be performing less on the motor game
(W= 255, p= 2.6e5), with a difference in location of -0.67, and on the logical
game (W= 341, p < 0.01), with a difference in location of -0.82.
We tested how gaming habits, self efficacy and risk aversion impact on level
and difficulty estimation error using the Kendall’s rank based correlation test.
The test was only significant for the influence of risk aversion on players’ level,
for the sensory game (z= 3.3093, p < 0.001) with τ= 0.29 and for the logical
game (z= 3.2974, p < 0.001) with τ= 0.28, meaning that for both these games,
risk averse players tend to perform better. Thus, in our experiment, we did not
detect any impact of gender, playing habits, assessment of self efficacy and risk
aversion on difficulty estimation error, only on player’s actual performance.
6 Discussion
6.1 Influence of difficulty and hard effect
We observe that the players estimation of difficulty is always below the actual
objective difficulty, except for the logical and motor tasks on the easiest difficulty
levels. More precisely, motor and logical tasks show the existence of a strong hard
effect, namely, an overestimation of the players’ chances to success for the hard-
est levels of difficulty (figure 5). Contrarily to the cognitive psychology studies
From Objective to Subjective Difficulty 15
related to overconfidence, in addition to the hard effect, nothing seems to indi-
cate any easy effect, namely an underestimation of the chances of success for the
easiest tasks [48, 56].
The presence of a hard effect and absence of easy effect might be explained by
the players’ confidence towards the game designers: games are rarely impossible
to finish. What makes games different from many other tasks is that difficulty is
artificial created for entertainment, players know that given enough time, they
are almost always supposed to eventually win. This may drive us to be over
confident in their chances of success
Moreover, players’ overconfidence and hard effect may be stronger in our
games than in previous cognitive psychology studies because of player progres-
sion. Indeed, our games allow players to experiment, to learn from their failures
and, thus, to increase their performance. This feeling of progression and mas-
tery may help players to become more confident on their chances of success. In
cognitive psychology studies, where general culture questionnaires are very often
used, this might not be the case.
The player’s global confidence towards the game and their feeling of progres-
sion and mastery are also enhanced by the use of DDA algorithm. By presenting
the players challenges that are adapted to their current level, the game is neither
too boring nor too frustrating, allowing them to stay motivated and to believe
in the fairness of the game.
Also, during our experiment, it is to note that objective difficulty starts
below 0.5, and thus that players face more easy challenges at the beginning of
the game, when they do not know the game, than at the end. Previous studies
on the hard/easy effect relies on general culture questions and thus players may
be able to assess their knowledge and their chances to win from the very first
question. Thus, we may postulate that players assessment of easy challenges is
biased by their ignorance relative to the games’ procedures. However, the motor
task has an almost flat progression curve, and this task has no sign of an easy
effect. Also, for sensory and logical task, both have more easy challenges at the
beginning of the session, but for the logical task, we seem to be close to a small
easy effect, and for the sensory, it’s the opposite and player’s are overconfident for
easy challenges. There thus do not seem to have a clear impact of oversampling
easy challenges at the beginning of the session on easy effect.
We may explain the results’ differences between sensory against both the
logical and motor tasks by the nature of the subjective difficulty the players
have to assess. As defined in our method in section 4.3, the bet system focuses
on the players’ estimation of their performance, and this estimation is not always
performed in the exact same conditions. For the sensory game, the players can
select the squares before betting, making the play effort before interpreting their
chances of failure. Of course, they do not know about their actual performances,
but they go one step further toward the completion of the challenge than for
the two other games. Thus, they assess their chances of failure after having
realized the exercise, and thus may have a more accurate feeling of the quality
of their answer. For the two other games, they perform no manipulation and
16 Constant et al.
have to guess all the next steps. This design choice for the sensory task was
made because we did not want to focus on memorization, but on the sensory
aspect of detecting blinking squares.
It is to note that our result are different from those of psychophysical studies
on subjective difficulty, where perceived difficulty seems to never reach a plateau
and have a more linear or exponential aspect. We think that this is mainly the
case because we ask player to predict the difficulty of a challenge, not to evaluate
it after many repetitions. We think that our approach, closer tho the cognitive
psychology approaches, may be closer to what a player really feels while playing.
Motor game is the task where the quality of our model is the lowest (R2=
0.28). It is by far the fastest game to play, where participants can complete
quickly one turn after the other which may explain the higher objective difficulty
variability. However, such design is representative of action games. Slowing the
game’s pace may produce more stable results, but the experiment will be less
6.2 The player’s profile impact
We do not find any evidence of the influence of the players’ profile on their
estimation of difficulty. It seems to be contradictory to the studies about over-
confidence. Some aspects of the experiment may explain such results.
In a financial analysis field study, Barber & Odean [57] study overconfidence
in order to explain the difference of trading’s performance gender, concluding
that men have a tendency to be more overconfident and less risk averse than
women. We do not observe such behaviors, and we explain it mainly because our
experimentation’s protocol differ from Barber & Odean’s study. First, the median
age of our participants is 15 while their is 50. Then, their participants have a
certain degree of expertise about investment, whereas ours do not know the
content of the games before playing them. Moreover, we may postulate that our
tasks are very abstract and less prone to culturally induced gender differences.
Risk-aversion is also a determinant of an excessive confidence [57, 34]. How-
ever we do not find any influence of the risk on the difficulty estimation error.
Contrary to these studies, our participants’ age is quite young. Also, the ques-
tionnaire relies on mental calculus and probabilities and might be less efficient
on adolescents.
Stone [58] shows that initial and positive self-efficacy assessment may rein-
force the participants’ confidence and modify their performance, which is not
the case in our study. However, in Stone’s experiment, self-efficacy is assessed
with regard to the given task, i.e. participants are asked to estimate their per-
formance. In our study, we estimated self-efficacy using a general self efficacy
questionnaire [52, 53]. However, if we use players’ mean bet as a measure of self-
efficacy, there is a clear relationship between self-efficacy, i.e. how high the mean
bet is, and overconfidence, i.e. how high mean bet minus mean actual result is.
This is not surprising, as objective difficulty is adapted to 0.5 for each player.
Also, we do not find a link between mean bet and player performance.
From Objective to Subjective Difficulty 17
6.3 Experiment’s limitations
There are some limitations relative to our approach, and in particular to the bet
The bet system Our approach is based on the use of a bet system to
measure the players’ difficulty estimation error. First, this approach is limited to
specific tasks, where the interaction rhythm can be combined with a recurrent
question addressed to the player. Also, it is important to note that the bet
is not strictly related to the confidence, as measured in cognitive psychology
studies. For our games, the optimal strategy is to bet 7 when Dobjective >0.5,
and 1 when Dobjective <0.5. Thus, our evaluation might be less accurate than
confidence scales. Moreover, as we said in section 6.1, the bet system does not
allow us to clearly distinguish between effort-based and skill-based subjective
difficulties. New experiments can improve the separation between them.
Dynamic Difficulty Adjustment DDA is representative of how video
games are designed, and must have a notable impact on the hard / easy ef-
fect. Such an adjusted curve should allow players to be more confident in their
chances of success, and we should thus observe a weaker easy effect and a stronger
hard effect than in a pure random experiment. Our experiment shows than when
using DDA, players do develop a strong feeling of confidence in two of the three
tasks. Nevertheless, be able to point out DDA as responsible of this overconfi-
dence, we need and A/B experiment to compare our results to others based on
a random difficulty system.
Motivational influences The actual performance of a player is both de-
pendent on the task difficulty and on the players’ effort. If the player is not
motivated enough, he may correctly assess the difficulty but be less efficient
because he does not want to make the effort. Video games’ players experience
various states of emotion [59,8], including sometimes boredom or anxiety. As
such, these emotions have to be taken into account in future experiment. We
also have to note that only sensory and motor tasks induce a near-miss effect,
while the players do not know if they were almost successful during the logical
task. The near-miss effect may convict the players that they were almost win-
ning, leading them to overestimate their chances of success for the next turn [39,
7 Conclusion and Perspectives
In this paper, we investigate the players’ perception of difficulty. We extend
previous psychophysical and cognitive psychology studies by proposing a method
to evaluate objective difficulty and focusing on video games.
First, results demonstrate the efficiency of our objective difficulty estima-
tion. The mixed effect model allows us to easily take into account the difference
between players. Results show a predictive accuracy ranging from 61% for the
motor task, to almost 70% for the other tasks. Estimated objective difficulty is
coherent with DDA, showing a convergence of objective difficulty to 0.5, for all
18 Constant et al.
groups and levels. We are also able to measure a learning effect, as a negative
effect of time on objective difficulty for a give difficulty parameter value. This
learning effect is coherent with the nature of the tasks, with a higher learning
effect for the logical task.
Then, results confirm the existence of an unrealistic evaluation of the players’
actual chances of failure. More specifically, players are always overconfident,
except for low level of difficulty in the motor and logical task. Results show a
strong hard effect for the motor and logical tasks, and no significant easy effect
for all tasks.
We postulate that players’ strong overconfidence might be explained by the
fact that our tasks are video games. First, players know that games are made to
be eventually beaten. Second, games allow players to get better and develop a
sense of progression and mastery. Even more, the use of DDA should reinforce
both of these experiment’s aspects.
The absence of a hard effect on the sensory task may be clarified by the
design of the sensory task itself, as the difficulty evaluation performed after
players actually tried to solve the problem, and thus, they may have a better
feeling of their performance.
New experiments will be conducted in order to improve our understanding
of difficulty perception in video games. We want to explore the use of a random
difficulty curve, to validate the impact of DDA on the hard / easy effect by com-
paring both experiments. Moreover, we plan to investigate the influence of the
previous turns on the players’ perception of difficulty. DDA creates a temporal
relationship between the difficulty of subsequent turns and thus prevented us to
realize the analysis on this experiment.
Then, we plan to verify the impact of feedbacks on the players’ assessment of
difficulty. Constant feedbacks about the decision process leads the participants to
re-evaluate their judgments during the task, reaching a more accurate level [60].
Continuous feedbacks about the user’s progression is a main feature of human
computer interaction and, in particular, video games. It implies to distinguish
between positive and negative feedbacks, or to test the influence of feedbacks’
accuracy. Video games adopts various types of feedback in order to affect the
players, to generate more uncertainty of the outcome and to improve enjoyment
[59, 49].
It is to note than from a game design perspective, the presence of a hard
effect has pro and cons. Hard effect is a good consequence of the game’s mo-
tivational mechanics : if the player believes in his chances of success, then he
may be motivated to play. However, having players believe that a challenge is
easier than it is, especially in high level of difficulty, may also lead players to
frustration because they will lose at challenges they thought they could win.
The motivational aspects of the discrepancies between subjective and objective
difficulty seems thus worthy of further investigation.
Finally, we plan to expand our approach with other measures of mental effort
like eye-tracking methods that have been used to assess cognitive load related to
computer interface [61], specially about memory and logical related tasks [62].
From Objective to Subjective Difficulty 19
Authors would like to thank Daniel Andler, Jean Baratgin, Lauren Quiniou, and
Laurence Battais & H´el`ene Malcuit from Carrefour Numerique.
1. Juul, J.: The game, the player, the world: Looking for a heart of gameness. In
Raessens, J., ed.: Level Up: Digital Games Research Conference Proceedings. Vol-
ume 1. (2003) 30–45
2. Malone, T.W.: Heuristics for designing enjoyable user interfaces: Lessons from com-
puter games. Proceedings of the 1982 conference on Human factors in computing
systems (1982) 63–68
3. Lazzaro, N.: Why we play games: Four keys to more emotion without story. In:
Game Developers Conference. (March 2004)
4. Sweetser, P., Wyeth, P.: Gameflow: a model for evaluating player enjoyment in
games. Computers in Entertainment (CIE) 3(3) (2005) 3–3
5. Nakamura, J., Csikszentmihalyi, M.: The concept of flow. In: Flow and the foun-
dations of positive psychology. Springer (2014) 239–263
6. Allart, T., Levieux, G., Pierfitte, M., Guilloux, A., Natkin, S.: Difficulty influence
on motivation over time in video games using survival analysis. In: Foundation of
Digital Games. (2017)
7. Ryan, R.M., Rigby, C.S., Przybylski, A.: The motivational pull of video games: A
self-determination theory approach. Motivation and emotion 30(4) (2006) 344–360
8. Juul, J.: A Casual Revolution: Reinventing Video Games and Their Players. Mit
press edn., Cambridge, USA (2009)
9. Juul, J.: The Art of Failure. 1 edn. The MIT Press, Cambridge, USA (2013)
10. Hunicke, R.: The case for dynamic difficulty adjustment in games. In: Proceed-
ings of the 2005 ACM SIGCHI International Conference on Advances in computer
entertainment technology, ACM (2005) 429–433
11. Andrade, G., Ramalho, G., Santana, H., Corruble, V.: Extending reinforcement
learning to provide dynamic game balancing. In: Proceedings of the Workshop on
Reasoning, Representation, and Learning in Computer Games, 19th International
Joint Conference on Artificial Intelligence (IJCAI). (2005) 7–12
12. Vicencio-Moreira, R., Mandryk, R.L., Gutwin, C.: Now you can compete with
anyone: Balancing players of different skill levels in a first-person shooter game. In:
Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing
Systems, ACM (2015) 2255–2264
13. Rani, P., Sarkar, N., Liu, C.: Maintaining optimal challenge in computer games
through real-time physiological feedback. In: Proceedings of the 11th international
conference on human computer interaction. Volume 58. (2005)
14. Afergan, D., Peck, E.M., Solovey, E.T., Jenkins, A., Hincks, S.W., Brown, E.T.,
Chang, R., Jacob, R.J.K.: Dynamic Difficulty Using Brain Metrics of Workload.
In Jones, M., Palanque, P., eds.: CHI ’14 Proceedings of the SIGCHI Conference
on Human Factors in Computing Systems, Toronto, Ontario, Canada, ACM New
York, NY, USA (2014) 3797–3806
15. Aponte, M.V., Levieux, G., Natkin, S.: Difficulty in Videogames: An Experimental
Validation of a Formal Definition. In Rom˜ao, T., ed.: Proceedings of the 8th
International Conference on Advances in Computer Entertainment Technology,
ACE 2011, Lisbon, Portugal, ACM New York, NY, USA (2011) 1–18
20 Constant et al.
16. Passyn, K., Sujan, M.: Skill-based versus effort-based task difficulty: A task-
analysis approach to the role of specific emotions in motivating difficult actions.
Journal of Consumer Psychology 22(3) (2012) 461–468
17. Levieux, G.: Mesure de la difficult´e dans les jeux vid´eo. Th`ese, Conservatoire
National des Arts et M´etiers CNAM Paris (2011)
18. Hunicke, R., LeBlanc, M., Zubeck, R.: MDA: A Formal Approach to Game Design
and Game Research. In: Proceedings of the AAAI Workshop on Challenges in
Game AI, San Jose, CA, USA, AAAI Press (2004)
19. Deligni`eres, D., Famose, J.: Perception de la difficult´e et nature de la tˆache. Science
et motricit´e 23 (1994) 39–47
20. Borg, G., Bratfisch, O., Dorni’c, S.: On the problems of perceived difficulty. Scan-
dinavian journal of psychology 12(1) (1971) 249–260
21. Slifkin, A.B., Grilli, S.M.: Aiming for the future: prospective action difficulty,
prescribed difficulty, and fitts law. Experimental Brain Research 174(4) (2006)
22. Deligni`eres, D., Famose, J.P.: Perception de la difficult´e, entropie et performance.
Science & sports 7(4) (1992) 245–252
23. Deligni`eres, D., Famose, J.P., Genty, J.: Validation d’une ´echelle de cat´egories pour
la perception de la difficult´e. Revue STAPS 34 (1994) 77–88
24. Deligni`eres, D., Famose, J.P., Th´epaut-Mathieu, C., Fleurance, P., et al.: A psy-
chophysical study of difficulty rating in rock climbing. International Journal of
Sport Psychology 24 (1993) 404–404
25. Deligni`eres, D., Brisswalter, J., Legros, P.: Influence of physical exercise on choice
reaction time in sports experts: the mediating role of resource allocation. Journal
of Human Movement Studies 27(4) (1994) 173–188
26. Fitts, P.M.: The information capacity of the human motor system in controlling
the amplitude of movement. Journal of experimental psychology 47(6) (1954) 381
27. Kahneman, D., Frederick, S.: A model of heuristic judgment. In Holyoak, K.J.,
Morrison, R.G., eds.: The Cambridge Handbook of Thinking and Reasoning. 1
edn. Cambridge University Press, Cambridge, UK (2005) 267–293
28. Shah, A.K., Oppenheimer, D.M.: Heuristics made easy: an effort-reduction frame-
work. Psychological bulletin 134(2) (mar 2008) 207–22
29. Kahneman, D., Tversky, A.: Judgment under Uncertainty: Heuristics and Biases.
Science (New York, N.Y.) 185(4157) (sep 1974) 1124–31
30. Russo, J.E., Schoemaker, P.J.H.: Managing overconfidence. Sloan Management
Review 33(2) (1992) 7–17
31. Bessi`ere, V.: Exc`es de confiance des dirigeants et d´ecisions financi`eres: une
synth`ese. Finance Contrˆole Strat´egie 10 (2007) 39–66
32. Moore, D.A., Healy, P.J.: The Trouble with Overconfidence. Psychological review
115(2) (apr 2008) 502–17
33. Griffin, D., Tversky, A.: The weighing of evidence and the determinants of confi-
dence. Cognitive psychology 411435 (1992) 411–435
34. Johnson, D.D.P., Fowler, J.H.: The evolution of overconfidence. Nature 477(7364)
(sep 2011) 317–20
35. Bandura, A.: Self-efficacy: Toward a Unifying Theory of Behavioral Change. Psy-
chological Review 84(2) (1977) 191–215
36. Keren, G.: Facing uncertainty in the game of bridge: A calibration study. Organi-
zational Behavior and Human Decision Processes 39(1) (feb 1987) 98–114
37. Linnet, J., Gebauer, L., Shaffer, H., Mouridsen, K., Møller, A.: Experienced poker
players differ from inexperienced poker players in estimation bias and decision bias.
Journal of Gambling Issues (24) (2010) 86–100
From Objective to Subjective Difficulty 21
38. Park, Y.J., Santos-Pinto, L.: Overconfidence in tournaments: Evidence from the
field. Theory and Decision 69(1) (2010) 143–166
39. Sundali, J., Croson, R.: Biases in casino betting : The hot hand and the gambler
’ s fallacy. Judgment and Decision Making 1(1) (2006) 1–12
40. Parke, J., Griffiths, M.: The psychology of the fruit machine: The role of structural
characteristics (revisited). International Journal of Mental Health and Addiction
4(2) (2006) 151–179
41. Lichtenstein, S., Fischhoff, B.: Do those who know more also know more about
how much they know? Organizational Behavior and Human Performance 20 (1977)
42. Klayman, J., Soll, J.B.: Overconfidence: It depends on how, what, and whom you
ask. Organizational Behavior and Human Decision Processes 79(3) (1999) 216–247
43. Kahneman, D., Tversky, A.: Subjective probability: A judgment of representative-
ness. Cognitive Psychology 3(3) (1972) 430–454
44. Croson, R., Sundali, J.: The gambler’s fallacy and the hot hand: Empirical data
from casinos. Journal of Risk and Uncertainty 30(3) (2005) 195–209
45. Gilovich, T., Vallone, R., Tversky, A.: The hot hand in basketball: On the misper-
ception of random sequences. Cognitive Psychology 17(3) (jul 1985) 295–314
46. Langer, E.J.: The illusion of control. Journal of personality and social psychology
32(2) (1975) 311–328
47. Goodie, A.S.: The role of perceived control and overconfidence in pathological
gambling. Journal of Gambling Studies 21(4) (2005) 481–502
48. Pulford, B.D., Colman, A.M.: Overconfidence: Feedback and item difficulty effects.
Personality and Individual Differences 23(1) (jul 1997) 125–133
49. Costikyan, G.: Uncertainty in Games. 1 edn. MIT Press, Cambridge, USA (2013)
50. Lankoski, P., Bj¨ork, S.: Game Research Methods: An Overview. 1 edn. ETC Press
51. Bates, D., M¨achler, M., Bolker, B.M., Walker, S.C.: Fitting linear mixed-effects
models using lme4. Journal of Statistical Software 67(1) (2015) 1–48
52. Chen, G., Gully, S.M., Eden, D.: Validation of a New General Self-Efficacy Scale.
Organizational Research Methods 4(1) (2001) 62–83
53. Bandura, A.: Guide for constructing self-efficacy scales. In Urdan, T., Pajares,
F., eds.: Self-efficacy beliefs of adolescents. 1 edn. Information Age Publishing,
Charlotte, USA (2006) 307–337
54. Holt, C.A., Laury, S.K.: Risk aversion and incentive effects. The American Eco-
nomic Review 92(5) (2002) 1644–1655
55. Nakagawa, S., Schielzeth, H.: A general and simple method for obtaining r2 from
generalized linear mixed-effects models. Methods in Ecology and Evolution 4(2)
(2013) 133–142
56. Keren, G.: On The Calibration of Probability Judgments: Some Critical Comments
and Alternative Perspectives. Journal of Behavioral Decision Making 10(3) (sep
1997) 269–278
57. Barber, B.M., Odean, T.: Boys will be boys: Gender, overconfidence, and common
stock investment. Quarterly Journal of Economics 116(1) (2001) 261–292
58. Stone, D.N.: Overconfidence in Initial Self-Efficacy Judgments: Effects on Deci-
sion Processes and Performance. Organizational Behavior and Human Decision
Processes 59(3) (1994) 452–474
59. Caillois, R.: Les jeux et les hommes : le masque et le vertige. 2 edn. Gallimard,
Paris, France (1958)
22 Constant et al.
60. Arkes, H.R., Christensen, C., Lai, C., Blumer, C.: Two methods of reducing over-
confidence. Organizational Behavior and Human Decision Processes 39 (1987)
61. Goldberg, J.H., Kotval, X.: Computer interface evaluation using eye move-
ments : Methods and constructs Computer interface evaluation using eye move-
ments : methods and constructs. International Journal of Industrial Ergonomics
24(November 2015) (1999) 631–645
62. Klingner, J., Tversky, B., Hanrahan, P.: Effects of visual and verbal presentation
on cognitive load in vigiolance, memory, and arithmetic tasks. Psychophysiology
48 (2011) 323–332
... [...] On peut ainsi dire que dans un jeu vidéo, un challenge d'une certaine difficulté demande une certaine quantité d'attention pour être résolue, et cette quantité d'attention dépend directement de la complexité du problème. ( [39], page 24) La différenciation entre complexité et difficulté a notamment été illustrée dans la suite de ces recherches sur la mesure de la difficulté d'un jeu de taquin face à un jeu de réflexe [20]. Le premier est un jeu de difficulté logique, mesurée par l'effort indispensable au joueur pour réaliser une induction ou une déduction à partir des informations qu'il a en sa possession, où le joueur doit replacer une grille mélangée aléatoirement dans son ordre numérique. ...
... De son côté, Levieux et al. déterminent la difficulté objective d'une tâche dans un jeu vidéo sur ces critères analogues, à différencier d'une difficulté perçue qui tient compte, elle, de l'appréciation du joueur. Cette difficulté objective est calculée pour chaque niveau la probabilité d'échouer, considérant le paramètre de difficulté du jeu et les compétences du joueur, en tenant compte des variations de performance d'un joueur à un autre [5,20]. Si cette mesure n'isole pas non plus un calcul indépendant de la curiosité, elle est néanmoins plus réaliste dans son appréciation la difficulté moyenne d'un challenge, ou d'une tâche, sans avoir à passer par une valeur de base donnée par un expert, dont la sélection pose aussi problème. ...
... De plus, il a été montré que la perception de la difficulté d'une tâche dans un jeu diffère grandement de sa difficulté objective, et qu'elle est de plus particulièrement sensible aux systèmes d'ajustement de la difficulté [20,19]. Il a été observé que les joueurs ont tendance à surestimer leurs chances de succès lorsque la difficulté du jeu s'adapte à leur niveau de compétence. ...
Technical Report
Full-text available
Ce rapport décrit les problématiques de conception relatives à la variabilité de l’expérience de jeu, et en particulier comment la perception de la variabilité permet de garder un joueur motivé d’une session de jeu à une autre. L’approche décrite dans ce document concerne la perception de la variabilité à l’échelle de l’architecture d’un niveau, et donc de l’influence de certaines pratiques de level design sur la motivation du joueur. L’attrait pour la nouveauté a été décrit au travers de l'émergence de l'état de curiosité, en particulier dans un contexte d'apprentissage, pour lequel la curiosité va permettre l'acquisition de connaissances et aider au développement de compétences. La curiosité peut être définie comme un état cognitif afférent à la volonté d'expérimenter se traduisant par une importante motivation conduisant à la recherche d'information et de connaissances sur le fonctionnement d'une situation. L'état de curiosité est ainsi central aux théories de la motivation intrinsèque, favorisant l'apparition d'un comportement d'exploration de l'environnement, où la personne va se confronter à des problèmes complexes, parfois hors de sa portée a priori. Les jeux vidéo mettent en scène des environnements virtuels complexes, détaillés, réalisés dans l'optique de transmettre une expérience, un récit, un discours, et susciter des émotions. Manipuler le comportement du joueur, attirer son attention, le motiver à explorer l'environnement est fondamental pour que les concepteurs puissent travailler leurs mises en scène et en mesurer les effets. Comprendre comment, dans le cadre de jeux vidéo, il est possible de susciter la curiosité du joueur apparaît comme nécessaire pour améliorer l'expérience de jeu, et au travers, la motivation du joueur.
... Our experiment shows that we can reach error rates lower than 20% in less than two minutes of playtime, and our actual player failure rates will be close to our targeted failure rates ( fig. 2). Our experiment shows what our δ-logit approach is able to achieve in the context of a shooter game, but it is to note that the required accuracy for a DDA system is still an open question, as perception of difficulty is a very complex matter [7,6]. ...
... Their approach allows to predict spell effectiveness for a specific player at a specific time but does not provide a more generic measure of difficulty like failure probability. Allart and Constant used a mixedeffect logistic regression to evaluate both commercial and experimental games difficulty [6,7,1]. Logistic regression seems indeed well suited to predict a failure probability from few samples and binary outcomes. ...
... That way, if a designer asks for a difficulty of 0.2, the model will estimate the value of θ for a difficulty randomly picked between 0.15 and 0.25. This allows us to have values of θ that always vary and we consider that the player's perception of difficulty is not accurate enough to perceive such a subtle difference [7]. However, difficulty perception is a complex matter and in further studies, we should investigate the impact of this parameter on both perception of difficulty and difficulty estimation accuracy. ...
Full-text available
Difficulty is a fundamental factor of enjoyment and motivation in video games. Thus, many video games use Dynamic Difficulty Adjustment systems to provide players with an optimal level of challenge. However, many of these systems are either game specific, limited to a specific range of difficulties, or require much more data than one can track during a short play session. In this paper, we introduce the δ-logit algorithm. It can be used on many game types, allows a developer to set the game’s difficulty to any level, with, in our experiment, a player failure error prediction rate lower than 20% in less than two minutes of playtime. In order to roughly estimate the difficulty as quickly as possible, δ-logit drives a single metavariable to adjust the game’s difficulty. It starts with a simple +/- δ algorithm to gather a few data points and then uses logistic regression to estimate the players failure probability when the smallest required amount of data has been collected. The goal of this paper is to describe δ -logit and estimate its accuracy and convergence speed with a study on 37 participants playing a tank shooter game.
... Perception of difficulty is fundamental: as Nakamura puts it, the most important aspect of difficulty, when it comes to it's impact on game experience, is the player's perception of this difficulty, not the actual one [45]. The thing is that player's perception of difficulty is absolutely not straightforward and can be strongly biased [17]. Furthermore, player's estimation of the useful effort to overcome a challenge depends on multiple factors. ...
... The current experimentation is investigating a potential influence of DDA systems on players' perception of their chances of success, and thus, on players' confidence. Following the results of our previous research showing the existence of a gap between the objective difficulty of a challenge and the perceived difficulty of the same challenge [17], we want to study the influence of DDA systems on this gap, called the difficulty estimation error. As we suggest that balanced difficulty should boost player's confidence, our hypothesis is that the gap is widened when difficulty is adapted to players skills. ...
... (figure 4). Following results of previous research [17], we do not find such easy effect for the other games. One explanation is that the logical game may be closest to the task used in cognitive psychology studies, implying pure logical challenge like mathematical problems or general knowledge questionnaires. ...
Conference Paper
Full-text available
Difficulty is one of the major motivational pull of video games, and thus many games use Dynamic Difficulty Adjustment (DDA) systems to improve the game experience. This paper describes our research investigating the influence of DDA systems on player's confidence, evaluated using an in-game bet system. Our hypothesis is that DDA systems may lead players to overconfidence, revealed by an overestimation of their success chances when betting. This boost of confidence may be a part of the positive impact of DDA systems on the quality of game experience. We explain our method to evaluate player's confidence and implement it into three games related to logical, motor and sensory difficulties. We describe two experimental conditions where difficulty is either randomly chosen or adapted using a DDA algorithm. Results show how DDA systems can lead players to high level of overconfidence.
... Subsequently, other factors (i.e., past individual experience) would be incorporated into the subjective judge (i.e., win possibility prediction) for the final decision or real action. Such conditions had been observed in determining the game's level of difficulty [17] and play performance during competitive decision-making [24]. ...
... A significant difference was found between the objective and the subjective recognition in a player's evaluation of the game [17]. The author also points out that motivational aspects of the objective and the subjective recognition warrant further investigation. ...
Full-text available
The gaming industry had rapidly been expanding globally, where it encompasses more than the purpose of recreational and increasingly becomes more immersive and engaging, and potentially leads to pathological gaming behaviors that lead to addiction. Such experience of engagement and addiction involves understanding the fundamental functions of the human mind’s dynamic state. This study uncovers the mind’s underlying physics via the analogy of motion (i.e., mass, velocity, etc.) using games as the source of information. This study also conjectures that the law of conservation in mind occurred in games where momentum and energy were conserved over time, where game-playing experience relative to the gambling psychology and perceptive force were identified from the objective and subjective perspectives. It was found that momentum conservation provides new engagement measures while energy conservation, considering several other factors, provides the necessary components of understanding the addiction mechanism in the game-playing context. This measurement is examined in various domains, such as popular boards and sports games, and public gambling, where its effectiveness is determined.
... Confidence is critical for motivating players in gameplay, as the self-efficacy theory of motivation [37] says that confidence in completing a task is a crucial aspect of motivation. Researchers have been studying confidence in relation to playing digital games; Constant et al. [39] conducted an experiment on the deviation between players' estimation of their own success rates and their actual game performance. They determined that participants overestimated their chance of success at the hardest level of a game [39], which was in agreement with related findings in cognitive psychology [40]. ...
... Researchers have been studying confidence in relation to playing digital games; Constant et al. [39] conducted an experiment on the deviation between players' estimation of their own success rates and their actual game performance. They determined that participants overestimated their chance of success at the hardest level of a game [39], which was in agreement with related findings in cognitive psychology [40]. ...
Full-text available
Researchers have been investigating ways to improve users’ spatial perception in virtual environments. Very limited studies have focused on the context of virtual reality (VR) games. Tutorials with practices, a common element in games, are good opportunities to implement measures that improve players’ spatial perception. Using an experiment, this paper investigates how two types of practices (real-world and virtual-world practices) influence players’ spatial perception, game performance, and immersion in VR games. Given that spatial perception is viewed as an essential aspect of VR applications, the moderating role of spatial perception on the effect of practices in game performance is also explored. The results demonstrate that virtual-world practice is effective in improving players’ spatial perception of the virtual environment of VR games. Real-world practice is suggested to be effective in enhancing spatial perception when it is averaged over multiple sessions. The results also suggest that spatial perception moderates the effects of practices on game performance. The results imply that practices in game tutorial can be a transitional environment for new players to enter a VR game.
... Several approaches to defining difficulty in games already exist, ranging from measuring the difficulty of video games [33], [34] to assessing the difficulty of educational games [11], [35]. ...
Full-text available
In a world where algorithms are ubiquitous, the development of computational thinking competencies is becoming progressively important among students, technology professionals, and 21st-century citizens in general. Educational games as a means of promoting computational thinking skills have gained popularity in recent years. Offering efficient educational games that promote computational thinking competencies requires personalized learning paths through adaptive difficulty. The research presented herein is a first attempt to define a difficulty function for maze-based programming challenges using log data obtained from Kodetu, which is a block-based maze game. Specifically, we conducted three studies with 9- to 16-year-old students who were asked to solve sequences of maze-based programming challenges. Using log data from these studies, we investigated the maze characteristics and the coding limitations that affect performance in the challenges and calculated the performance obtained by the participants using a fuzzy rule-based system. The results showed that the turns in a maze, the number of total steps of a maze, and the blocks provided affect student performance. Using regression analysis, we defined a difficulty function for maze-based programming challenges that considers the weights of these factors and provides a first step towards the design of adaptive learning paths for computational thinking-related educational games.
... C'est d'autant plus problématique si nous souhaitons évaluer l'impact de la diculté sur la motivation au cours du temps, car nous pouvons dicilement nous er à un questionnaire post-expérimentation pour évaluer précisement l'évolution de la diculté au cours du jeu. Il est possible d'intégrer cette évaluation dans le gameplay, comme Constant et al l'ont fait pour la diculté subjective (Constant et al. (2017)). Mais ce type d'évaluation est limité aux jeux basés sur le tour par tour et nécessite une modication du gameplay. ...
Cette thèse s'intéresse à l'analyse des données longitudinales, potentiellement grandes selon les trois axes suivants : nombre d'individus, fréquence d'observation et nombre de covariables. A partir de ces données, éventuellement censurées, nous considérons comme facteur d'étude le temps d'apparition d'un ou plusieurs évènements. Nous cherchons dans des classes de modèles à coefficients dépendant du temps à estimer l’intensité d’apparition des événements. Or les estimateurs actuels, ne permettent pas de traiter efficacement un grand nombre d’observations et/ou un grand nombre de covariables. Nous proposons un nouvel estimateur défini via la vraisemblance complète de Cox et une pénalisation permettant à la fois la sélection de variables et de forcer, quand c’est possible, les coefficients à être constants. Nous introduisons des algorithmes d'optimisation proximaux, permettant d'estimer les coefficients du modèle de manière efficace. L'implémentation de ces méthodes en C++ et dans le package R coxtv permet d'analyser des jeux de données de taille supérieure à la mémoire vive; via un streaming du flux de données et des méthodes d'apprentissage en ligne, telles que la descente de gradient stochastique proximale aux pas adaptatifs. Nous illustrons les performances du modèle sur des simulations en nous comparant aux méthodes existantes. Enfin, nous nous intéressons à la problématique du design des jeux vidéo. Nous montrons que l'application directe de ce modèle, sur les grands jeux de données dont dispose l'industrie du jeu vidéo, permet de mettre en évidence des leviers d'amélioration du design des jeux étudiés. Nous nous intéressons d'abord à l'analyse des composantes bas niveau, telles que les choix d'équipement fait par les joueurs au fils du temps et montrons que le modèle permet de quantifier l'effet de chacun de ces éléments de jeu, offrant ainsi aux designers des leviers d'amélioration direct du design. Enfin, nous montrons que le modèle permet de dégager des enseignements plus généraux sur le design tels que l'influence de la difficulté sur la motivation des joueurs.
Conference Paper
Full-text available
Challenge, and thus difficulty, is one of the main factors of enjoyment and motivation in video games. To enhance the players’ motivation, many studies rely on Dynamic Difficulty Adjustment model in order to follow a difficulty curve. However, few authors worked on the shape of the difficulty curve itself. Our goal in this paper is to evaluate how players react to different difficulty curves. We use four different difficulty curves, including two flat curves and two curves with different baseline and peak levels. We test those curves on 67 students of a video games school while playing a First-Person Shooter game. Our study shows that curves with peaks have the strongest impact on players’ motivation.
Full-text available
Difficulty is the personal experience of a subject facing resistance that prevents them from reaching a goal or desired state. It is an experiential part of everyone’s existence. In digital games, difficulty is strongly linked with designed challenges and obstacles that must be overcome by physical effort, manual skills, coordination, and dexterity. But this widespread perspective is a reductionist categorization of the expressive possibilities of difficulty. Because as experiential, difficulty is aesthetic expression and therefore it is much more than the mere skill challenge. The difficulty experience that emerges from an opposing force between object and subject, between game and player, can be interpretive, poetic, narrative, ethical or atmospheric among other expressive forms. Understanding difficulty from these broad parameters, we pose it as an aesthetic expression, which forges multiple experiences at the intersection between mechanics, fiction, and the player’s performance. This study analyses, drawing from philosophy, postphenomenology, and game studies, some aspects of two contemporary games, The Last of Us Part II and Death Stranding from the view of difficulty as aesthetic experience perspective, considering the significant and discursive tensions beyond purely ludic and mechanical elements.
Conference Paper
Full-text available
In this paper, we study the link between difficulty and player's motivation in two games developed by Ubisoft®: Rayman®Legends and Tom Clancy's The Division®. We describe a method to estimate players' difficulty over time and link it's time varying effect with players retention. Results confirm flow and self-efficacy theory. Also, for the first hours of playtime, results differ between the two games. We explain that discrepancy with regard to attribution theory : in Rayman Legends, failure can be mainly attributed to the player skills, while in Tom Clancy's The Division, avatar's strength plays a fundamental role and can always be relatively quickly improved.
Full-text available
Presents an integrative theoretical framework to explain and to predict psychological changes achieved by different modes of treatment. This theory states that psychological procedures, whatever their form, alter the level and strength of self-efficacy. It is hypothesized that expectations of personal efficacy determine whether coping behavior will be initiated, how much effort will be expended, and how long it will be sustained in the face of obstacles and aversive experiences. Persistence in activities that are subjectively threatening but in fact relatively safe produces, through experiences of mastery, further enhancement of self-efficacy and corresponding reductions in defensive behavior. In the proposed model, expectations of personal efficacy are derived from 4 principal sources of information: performance accomplishments, vicarious experience, verbal persuasion, and physiological states. Factors influencing the cognitive processing of efficacy information arise from enactive, vicarious, exhortative, and emotive sources. The differential power of diverse therapeutic procedures is analyzed in terms of the postulated cognitive mechanism of operation. Findings are reported from microanalyses of enactive, vicarious, and emotive modes of treatment that support the hypothesized relationship between perceived self-efficacy and behavioral changes. (21/2 p ref)
Conference Paper
When player skill levels differ widely in a competitive First-Person Shooter (FPS) game, enjoyment suffers: weaker players become frustrated and stronger players become less engaged. Player balancing techniques attempt to assist the weaker player and make games more competitive, but these techniques have limitations for deployment when skill levels vary substantially. We developed new player balancing schemes to deal with a range of FPS skill difference, and tested these techniques in one-on-one deathmatches using a commercial-quality FPS game developed with the UDK engine. Our results showed that the new balancing schemes are extremely effective at balancing, even for players with large skill differences. Surprisingly, the techniques that were most effective at balancing were also rated as most enjoyable by both players -- even though these schemes were the most noticeable. Our study is the first to show that player balancing can work well in realistic FPS games, providing developers with a way to increase the audience for this popular genre. In addition, our results demonstrate the idea that successful balancing is as much about the way the technique is applied as it is about the specific manipulation.
Many decisions are based on beliefs concerning the likelihood of uncertain events such as the outcome of an election, the guilt of a defendant, or the future value of the dollar. Occasionally, beliefs concerning uncertain events are expressed in numerical form as odds or subjective probabilities. In general, the heuristics are quite useful, but sometimes they lead to severe and systematic errors. The subjective assessment of probability resembles the subjective assessment of physical quantities such as distance or size. These judgments are all based on data of limited validity, which are processed according to heuristic rules. However, the reliance on this rule leads to systematic errors in the estimation of distance. This chapter describes three heuristics that are employed in making judgments under uncertainty. The first is representativeness, which is usually employed when people are asked to judge the probability that an object or event belongs to a class or event. The second is the availability of instances or scenarios, which is often employed when people are asked to assess the frequency of a class or the plausibility of a particular development, and the third is adjustment from an anchor, which is usually employed in numerical prediction when a relevant value is available.