Conference PaperPDF Available

Explaining and exploring job recommendations: a user-driven approach for interacting with knowledge-based job recommender systems

Authors:

Abstract and Figures

The dynamics of the labor market and the tasks with which jobs are being composed are continuously evolving. Job mobility is not evident, and providing effective recommendations in this context has also been found to be particularly challenging. In this paper, we present Labor Market Explorer, an interactive dashboard that enables job seekers to explore the labor market in a personalized way based on their skills and competences. Through a user-centered design process involving job seekers and job mediators, we developed this dashboard to enable job seekers to explore job recommendations and their required competencies, as well as how these competencies map to their profile. Evaluation results indicate the dashboard empowers job seekers to explore, understand, and find relevant vacancies, mostly independent of their background and age.
Content may be subject to copyright.
Explaining and Exploring Job Recommendations: a User-driven
Approach for Interacting with Knowledge-based Job
Recommender Systems
Francisco Gutiérrez Sven Charleer Robin De Croon
Dept. Computer Science, KU Leuven Dept. Computer Science, KU Leuven Dept. Computer Science, KU Leuven
Leuven, Belgium Leuven, Belgium Leuven, Belgium
francisco.gutierrez@cs.kuleuven.be sven.charleer@gmail.com robin.decroon@cs.kuleuven.be
Nyi Nyi Htun Gerd Goetschalckx Katrien Verbert
Dept. Computer Science, KU Leuven VDAB Dept. Computer Science, KU Leuven
Leuven, Belgium Brussels, Belgium Leuven, Belgium
nyinyi.htun@cs.kuleuven.be gerd.goetschalckx@vdab.be katrien.verbert@cs.kuleuven.be
ABSTRACT
The dynamics of the labor market and the tasks with which jobs
are being composed are continuously evolving. Job mobility is not
evident, and providing eective recommendations in this context
has also been found to be particularly challenging. In this paper, we
present Labor Market Explorer, an interactive dashboard that en-
ables job seekers to explore the labor market in a personalized way
based on their skills and competences. Through a user-centered
design process involving job seekers and job mediators, we devel-
oped this dashboard to enable job seekers to explore job recom-
mendations and their required competencies, as well as how these
competencies map to their prole. Evaluation results indicate the
dashboard empowers job seekers to explore, understand, and nd
relevant vacancies, mostly independent of their background and
age.
CCS CONCEPTS
Information systems Recommender systems
; Decision
support systems; Personalization;
Human-centered computing
Human computer interaction (HCI).
KEYWORDS
Recommender systems, user control, personal characteristics, ex-
planations, actionable insights.
ACM Reference Format:
Francisco Gutiérrez, Sven Charleer, Robin De Croon, Nyi Nyi Htun, Gerd
Goetschalckx, and Katrien Verbert. 2019. Explaining and Exploring Job Rec-
ommendations: a User-driven Approach for Interacting with Knowledge-
based Job Recommender Systems. In Thirteenth ACM Conference on Recom-
mender Systems (RecSys ’19), September 16–20, 2019, Copenhagen, Denmark.
ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3298689.3347001
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
1 INTRODUCTION
Dierent technical solutions have been proposed to support better
job seekers in nding relevant jobs from an often very abundant
overload of vacancies [
29
]. Recommendation techniques have been
extensively used to lter out relevant vacancies for a job seeker [
2
].
Although the approaches have been shown to perform well based on
both implicit and explicit preference indicators of job seekers [
29
],
the dynamics of the labor market also demand exploration support
to enable potential job mobility. Typical recommender systems use
information about an individual to suggest relevant items, but often
lack support for exploration and user control [15].
Interactive data visualization [
6
,
7
] is a second approach that has
been proposed to allow users to lter and get an overview of jobs.
Although these interactive visualizations have been proposed, to
the best of our knowledge they have not been co-designed with
job seekers or job mediators, and there may be a gap between
actual needs of end-users and the needs that these tools address.
Moreover, user studies that assess the utility of visualizations with
a heterogeneous user group, including users with low technical
skills and non-native speakers, have not yet been conducted.
In this paper, we present “Labor Market Explorer", an interactive
dashboard that enables job seekers to explore a diverse set of job
recommendations. Through a user-centered design process involv-
ing job seekers and job mediators, we developed this dashboard to
enable job seekers to explore current vacancies and gain actionable
insights by engaging with both an overview visualization as well
as diverse kinds of lters. Note that our design tries to complement
existing systems, such as job or career nders, not to replace them.
This raises key research questions whether enabling job seekers to
interact with visualization techniques empowers them to explore,
understand, and nd job recommendations (RQ1), and whether per-
sonal characteristics, such as age and background, impact the user
perception and user interaction with such an interface (RQ2). The
contributions of this paper are two-fold: rst, we present results
of an iterative design process involving both job seekers and job
mediators with diverse backgrounds. The resulting dashboard en-
ables job seekers to interact with a rich set of job recommendations
RecSys ’19, September 16–20, 2019, Copenhagen, Denmark
responding to their user prole, including competences and skills.
© 2019 Association for Computing Machinery.
Design implications for job recommendation dashboards include
ACM ISBN 978-1-4503-6243-6/19/09.. . $15.00
https://doi.org/10.1145/3298689.3347001
60
RecSys ’19, September 16–20, 2019, Copenhagen, Denmark Gutiérrez et al.
the importance of overview components that explain recommenda-
tions and required competencies, as well as a wide variety of lter
and sorting features to support user control and actionable insights.
Second, we present the results of an elaborate user study (n=66)
that assesses the perceived eectiveness and user experience of our
dashboard with a heterogeneous user group of job seekers. Results
indicate that the dashboard was perceived as an eective tool by dif-
ferent user groups, mostly independent of their age and background.
Dierent actionable insights are obtained through the use of our
dashboard, including a greater autonomy, a better understanding
of needed competencies and potential location-based job mobility.
The dashboard also triggers job seekers to update their user prole,
which, in turn, will result in better job recommendations.
2 RELATED WORK
2.1 Interactive Recommender Systems
In recent years, several interactive visualizations have been elabo-
rated to support both explanations and user control over recommen-
dations [
15
]. Besides, interaction with visualizations can strongly
inuence users’ understanding of complex data [
34
]. Since visual
representation of information can help reduce complex cognitive ef-
forts [
20
], the use of such visualizations has been argued to be very
crucial, particularly in high-risk domains such as health-care and
nancing [
17
]. EpistAid [
11
] is a prominent example that supports
the screening process of physicians by using an interactive visu-
alization to understand the relevance feedback algorithm. Several
other visualizations focus on improving the accuracy of recom-
mendations with both explanations and support for user control.
PeerChooser [
23
] and SmallWorlds [
13
] support interaction with
collaborative ltering recommender engines.
Many visualizations have also been developed to interact with hy-
brid recommender systems. TasteWeights [
8
] is a system that allows
users to control the impact of dierent algorithms as well as vari-
ous input data sources on the recommendation results. SetFusion
[
24
] is an interface that relationships between recommendations
and allows users to ne-tune weights of a hybrid recommender
system. MoodPlay [
5
] is a hybrid music recommender system that
integrates dierent techniques to support explanation and control
of aective data. The system allows the exploration of a music
collection through latent emotional dimensions, thereby improving
acceptance and understanding of recommendations. Yucheng et
al. [
19
] also explored user control in recommendations, creating a
visualization on top of the Spotify recommender system.
Some tools focus on exploration of recommendations using a
multi-perspective approach [
10
,
33
], where dierent recommender
engines that provide diverse machine-produced recommendations
and data sources are used to increase the potential of nding rele-
vant items. It has also been shown that interactive recommender
systems have the potential to increase the diversity of content
[
31
,
32
]. In this sense, diversity-enhanced visual interfaces might
signicantly reduce the exploration eorts, serving as explanatory
mechanisms for rening the user prole and assisting users to
perceive the diversity of the recommended items [9, 32].
Increasing resources of digitized labor market information and
job postings enable better job recommendations for job seekers [
29
].
However, such increasing volumes of data induce also additional
complexity for job seekers. As suggested by previous research on
interactive recommender systems, novel interactive visualizations
of job recommendations can be helpful for job seekers to support
better exploration, understanding and user control. In our work,
we build on these visual approaches to recommender systems to
improve job recommendations.
2.2 Tools to Support Job Recommendation
Novel approaches for job recommendations include improving the
algorithmic side [
1
,
4
,
14
,
30
] or the ltering for exploration [
16
].
RésuMatcher [
14
] is a job search tool that uses a similarity index
based on the job seekers experience, academic, and technical quali-
cations. CASPER [
28
] uses among others a collaborative ltering
technique to support nding relevant jobs. Absolventen.at [
18
] uses
hybrid user proling and makes use of both explicit and implicit
relevance feedback. Proactive [
21
] is a comprehensive job recom-
mender system developed for dierent categories of target users:
users with a broad range of preferences and interests and users
with clear career goals and narrow interests.
Bakri et al. [
6
] designed a system with interactive visualizations
to allow users to lter and get an overview of all posted jobs that
meet their criteria. The system designed by Baneres and Conesa [
7
]
is another example related to knowledge and missing skills for a
job position. A Sankey diagram is used to visualize the connections
between educational programs and needed skills. LinkedVis [
9
] is
another interesting example in this domain and has been deployed
to enable user control over job recommendations. In our work,
we focus on job seekers and use a similar interactive visualization
approach to support job seekers in nding jobs. Other interesting
approaches use conversational recommendation techniques [3].
As in the work of Bakri et al. [
6
] and Baneres and Conesa [
7
],
we focus on exploration of job information, knowledge and skills,
as well as potential competence gaps. In contrast to these earlier
approaches, we adopt a user-centered design process, involving
both job seekers and job mediators, to elaborate a design that meets
the needs of a heterogeneous user group, ranging from non-native
speakers to highly technically skilled users, in all age ranges.
3 TECHNICAL ARCHITECTURE
We designed the application following a client-server architecture,
see Figure 1. The server includes the following components that
provide diverse services to the application: the MyCareer API con-
tains information about the prole of the job seeker in XML format.
This prole consists of information regarding their desired jobs,
education, and detailed competencies and skill related information
that is entered by job seekers in the MyCareer system of the Public
Employment Service VDAB. The ELISE
1
component provides a ser-
vice to match the prole of the user with open vacancies registered
in the platform that was developed by VDAB, using a knowledge-
based recommendation technique to suggest relevant vacancies to
job seekers [29].
To overcome the typical lter bubble problem of such a recom-
mender system, we retrieve a much larger set of vacancies from
ELISE than the default top-N approach based on competences. This
set is then used to enable job seekers to narrow down relevant
1https://www.wcc-group.com/software/elise-software-platform
61
Explaining and Exploring Job Recommendations RecSys ’19, September 16–20, 2019, Copenhagen, Denmark
vacancies with various lters. A maximum of 2000 vacancies is
returned.
Figure 1: Overview of the technical architecture
GeoServices is used to provide information about the location of
the vacancies and also to calculate the distance between the address
of the job seeker and the vacancy. The MySQL database is used to
store information about previous searches.
4 USER-CENTERED DESIGN PROCESS
To design the Labor Market Explorer, a user-centered methodology
was applied to gradually improve the initial design of the Labor
Market Explorer. After every evaluation, feedback was addressed
in the next design, which was then again evaluated. An overview
of the dierent studies is presented in Figure 2. The nal design is
presented in Figure 5.
4.1 Focus Groups
We conducted two focus groups: a rst focus group with job seekers
and a second focus group with job mediators. The rst focus group
consisted of nine participants (2F), following training at VDAB in
dierent elds, including book keeping, welders, electricians, and
ICT support. Four participants were non-native speakers. The sec-
ond focus group consisted of nine job mediators (8F), with diverse
specializations, including helping customers in the eld of industry
and non-native speakers.
The list of parameters used by the knowledge-based recom-
mender system of VDAB [
29
] was presented and explained to the
participants. The list is detailed in Table 1. The list was distributed
on paper and participants were asked individually to rank them by
personal importance/preference. Afterwards, the importance of the
dierent parameters was discussed.
Results are presented in Figure 3. Each cell in this gure repre-
sents how often a particular parameter was ranked on a specic
position. Job seekers ranked distance to a job and the job title most
frequently rst. These two parameters were also ranked rst fre-
quently by job mediators. Job seekers indicated that one of the key
aspects for them is to know whether they can easily reach the loca-
tion of a job. Postal code was for them also an important parameter,
Figure 2: Timeline that shows the studies and the partici-
pants in each iteration. The main goal of each study and the
most important outcomes are summarized. The results of
the nal evaluation are described in Section 5.
Table 1: Parameters of the knowledge-based recommender.
Parameter Values
Job title Job description in the job oer.
Commute Distance between place of residence and work place.
Type of contract E.g., open-ended contract, temporary, student job, etc.
Employee status Full-time or part-time contract.
Work regime Working hours, e.g., day job, night shift, shift work, etc.
Work experience
Experience required, e.g., no experience, less than two years,
at least ve years, or not important.
Studies
Minimum degree required, e.g., none, elementary, high school,
bachelor, master.
Language
The languages and their level required for the job oer.
prociency
Certicates Additional certicates that might be requested.
Driver’s license Driver’s licenses recognized by the European Union.
Job related
List of competencies based on the ROME/Competent sys-
competencies tem [26].
Additional List of competencies based on the ROME/Competent sys-
competencies tem [26].
Postal code Code for the postal address.
Region code Based on the European NUTS nomenclature [12]
as some cities are much easier to reach then others. This parameter
was ranked second by four out of nine job seekers. Type of contract,
time arrangement, languages, study, competencies, work regime
and work experience were also ranked frequently, starting on the
second position. A few job mediators ranked study on the rst
place, but job seekers ranked this parameter less frequently and
lower. Certicates and driver’s license were often ranked beyond
the top-10. In addition, region code was considered less important.
The more specic postal code was considered to be more useful.
In addition to the ranking of parameters, job seekers and media-
tors reected on their current issues and needs when searching for
jobs. A key issue that was highlighted was the lack of explanations:
the MyCareer system of VDAB ranks job recommendations based
on a matching score, but this score is not explained to end-users. In
62
RecSys ’19, September 16–20, 2019, Copenhagen, Denmark Gutiérrez et al.
Figure 3: Ranking of parameters as voted by participants.
addition, job seekers requested more advanced exploration options.
Jobs are recommended based on distances and matching compe-
tences, but no support is provided to tweak these parameters. Job
seekers requested the support to tailor recommendations to their
current interests and to enable more ne-grained search.
4.2 Co-design sessions
Based on the parameter ranking and input of the focus groups, we
elaborated nine dierent designs that present information about
jobs. Figure 4 presents three of these designs. Design (a) presents the
dierent parameters that were deemed important in four dierent
clusters: location, work organization, competencies, and diploma.
Work organization combines the type of contract, time arrangement,
and work regime parameters. Location is presented in the rst tab
and shown by default as it was deemed important in the focus
groups. The table below the map shows the list of vacancies with
their distance as well as the required competencies and languages.
Design (b) presents the activities of a job seeker on the left side. In
the middle component, the vacancies are shown. Green dots indicate
that the job seeker masters the required competence and empty dots
indicate competencies that are required, but not listed in the prole
of the job seeker. Yellow stars are used to highlight competencies
that are in high demand. Red empty stars are used to highlight
competencies that are in high demand, but that are not yet mastered
by the job seeker and may be interesting training opportunities.
Distance is here presented on the right side, with a slider that can
be adjusted. Design (c) combines the idea of presenting activities
with dots (green, red) and stars, as well as the map for detailed
location information.
The digital sketches were printed and distributed to participants.
The mediator co-design group consisted of seven participants (7F).
The job seeker co-design group consisted of eight participants (3F).
Participants were rst asked to individually think about dierent
Figure 4: Three of the dierent designs that emerged from
the discussions that took place in the focus groups.
aspects that they liked or disliked in the designs. These aspects
were then discussed in group.
The four dierent categories as presented in design (a) were in
general well received. The group with mediators did indicate that a
tooltip may be helpful to further explain the categories. They also
indicated that they would rename a number of elements with more
intuitive terms (work organization
type of contract, location
place). There was also the question to show the name of the
location next to the title of a vacancy. Further suggestions included
showing the total number of job recommendations responding to
the user prole and sorting by distance.
Job seekers asked about the possibility to save lters. In a few
designs, a red star was used to indicate that a specic activity is often
requested (“hot item"). The concept was considered interesting, but
the current representation was not intuitive enough.
4.3 Formative user evaluations
We conducted two think aloud studies with a rst and a second
prototype of the dashboard. Design (a) was selected as the main
starting point for the rst prototype, as the other designs were too
limited for representing actual job recommendations, with often
very long titles. Based on the input of the focus group and co-design
63
Explaining and Exploring Job Recommendations RecSys ’19, September 16–20, 2019, Copenhagen, Denmark
sessions, the central focus of the map was also retained. Additional
features requested were added, including adding the name of the
city next to the distance, functionality to save searches, and various
other lter and sorting options.
A rst formative user evaluation was conducted with a group of
seven job mediator (7F) and six job seekers (3F). The think aloud
protocol was used to capture feedback from participants. The screen
was recorded as well as their voice when interacting with the in-
terface. Afterwards, general feedback was discussed. Overall, the
interface was well received: participants interacted frequently with
the location components and the table with competencies and lan-
guages. They indicated to like the map component and considered
distance as an important steering wheel. One of the key challenges
that participants identied was the comparison of jobs and their
competencies: the list with competencies was deemed too long. The
suggestion was made to group competencies in clusters. In addition,
several participants struggled with the overview of competencies:
although the visual dots enabled a visual comparison, a tooltip was
used in this prototype to show which competence a specic dot
represents. The approach turned out to be dicult to use: several
participants indicated that the text of the dierent competencies
needs to be visible on the screen to enable meaningful comparison.
The second formative user evaluation was conducted with 11 job
seekers and ve job mediators and used a new version of the proto-
type that addressed several issues of the rst prototype. Column
headers were added to the table overview with the names of the
competencies and several other features such as lters to narrow
down the scope of vacancies based on distance were added. Overall,
the interface was again well received. The overview was considered
to be a great improvement compared to the earlier version. Job
seekers also indicated that some of their competencies were miss-
ing and were “triggered" to update their prole. Feature requests
included a “favorites" option that would move competencies to the
front of the table to enable easier comparison of vacancies with
specic competencies of interest. In addition, the concept of “hot
items" was requested (stars in previous designs) that shows insight
into the current demand of specic competencies.
4.4 Design goals and nal prototype
Based on the feedback gathered from the previous iterations, we
dened the following design goals for our nal prototype:
(DG1)
Exploration/control: job seekers should be able to control job
recommendations and lter out the information ow coming
from the recommender engine.
(DG2)
Explanations: recommendations and matching scores should
be explained, and details should be provided on-demand.
(DG3)
Actionable insights: the interface should provide actionable
insights to help job-seekers nd new or more job recommen-
dations from dierent perspectives.
Following our design goals, we have dened a set of user inter-
face components. We describe the components according to our
goals in the following paragraphs.
4.4.1 Map and filtering components. We designed a map and vari-
ous ltering components to explore and control
(DG1)
a broad set of
job recommendations. The map shows the location of the vacancies
recommended to the job seeker that respond to the competence
prole of the job seeker, see Figure 5c. The map represents available
jobs with a blue dot. A grey dot is used to show unavailable jobs
(ltered out). Moreover, on the top of the map, the type of contract
and diploma tab components contain check-boxes to lter out and
control the information ow of recommendations. Job seekers can
also save the state of the application, including the conguration
state of the dierent lters and favorites, see Figure 5b, for later
consultation.
4.4.2 Vacancies table component. The table shows the list of all
the job recommendations, together with visualization components
that explain
(DG2)
the recommended jobs, see Figure 5e. Instead
of showing a typical matching score, the table explains this score
with dierent dots: blue dots indicate that the job seeker masters
a required competence, an empty blue dot represents a required
competence that is not mastered, see Figure 5f. In addition, the
table shows the distance, location, and name of the vacancy. To-
gether, this information explains why the job seeker receives the
recommended items and how well it matches with his/her prole.
Moreover, the competencies overview can be expanded by clicking
on the “view details" button, further explaining in detail the re-
quired competencies in column headers (Figure 5, right). Inspired
by the UpSet visualization technique [
22
], the histogram (Figure 5h)
shows a blue bar at the top of each column indicating the number of
times that the competence is required, providing actionable insights
(DG3) into the demand of particular competences.
Moreover, each of the competencies columns also includes a Fa-
vorites star to control
(DG1)
the recommendation output by giving
priority to particular competences, see Figure 5i. The favorites star
button moves the competence to the beginning of the table as a
means to help job-seekers gain insight into job recommendations
that require a particular set of competences of interest.
5 FINAL EVALUATION
5.1 Participants
The nal evaluation was conducted with 66 job seekers (age 33.9
±
9.5, 18F), recruited from eight dierent training programs. Ta-
ble 2 shows an overview of the dierent groups and their training
programs. A heterogeneous composition was an important objec-
tive and a purposeful sampling strategy [
25
] was used to create a
heterogeneous group of participants. The aim is to create a group
of participants with a wide diversity. Factors that are taken into
account were age, gender, and background. Participation was vol-
untary and not compensated.
To arrange the data analysis, we organized the participants ac-
cording to their background and age. Four groups were created that
cluster participants according to their background: the technical
group consists of job seekers participating in training programs for
learning the programming languages Java and C#. The construction
group consists of participants in training programs for welding,
residential installation, and maintenance electricians. The sales
group consists of participants in training for sales and commercial
assistance. The fourth group non-native speakers are participants
of a vocational training for non-native speakers. We also grouped
participants by age. The groups were the following: 15-24, 25-34,
35-44, 45+ years old and participants that did not say.
64
RecSys ’19, September 16–20, 2019, Copenhagen, Denmark Gutiérrez et al.
Figure 5: Two screen shots of the Labor Market Explorer. Left: Overview of competencies. a) Navigation tabs, b) Save button,
c) Map of Vacancies, d) Distance-to-job slider, e) Vacancies Table (collapsed), f) competencies Overview. Right: Detail of com-
petencies. g) Vacancies Table (expanded), h) Competencies Histogram, i) Favorite Competence.
Table 2: Distribution of the participants over the eight train-
ing programs.
Training program # Px Age Group
Commercial assistants
Sales
Employee training non-
native speakers
ICT, C# training
Maintenance electrician
Residential installer
ICT, Java training
Welding
9 (8f)
2 (1f)
10 (7f)
12
4
11
9
9
P1-P9
P10-P11
P12-P21
P22-P33
P34-P37
P38-P48
P49-P57
P58-P66
35.7 ± 11.4
33.5 ± 12.5
38.4 ± 7.9
30.9 ± 6.9
28 ± 0
30.5 ± 9.6
36.8 ± 11.7
31 ± 4.2
Sales
Sales
Non-native
Technical
Construction
Construction
Technical
Construction
5.2 Experimental setup and data collection
A mixed methods methodology was used. Users were asked to freely
interact with the Labor Market Explorer and use the interface to
nd relevant recommendations based on their own competence
prole. Afterwards, they were asked to ll out a questionnaire
that measured dierent aspects using the ResQue framework [
27
].
Additional feedback was captured with two open questions. Logging
was used to capture interactions with the various elements of the
dashboard. A one hour slot was used for each group of participants.
job recommendations. As language is not trivial for this group, also
working with an interface in a dierent language as well as nding
suitable jobs is evidently much more dicult. Perceived usefulness
(Q2) was a bit lower for the construction group. The wide range of
very specic competencies in their domain was one of the reasons
they highlighted for this lower score.
The construction and sales groups indicated more positively to
nd more jobs than expected (Q7). The median score dropped to
“neutral" for both the technical and non-native speaker categories.
In general, we observed in these two user groups that the number of
vacancies that was provided was smaller than in the other groups.
In general, we do not see many dierences across the dierent
age categories (Figure 6c). Overall, all questions were answered pos-
itively in the dierent age group. The explorer was slightly better
perceived by older participants (45+) with respect to overall satis-
faction (Q5) and use intention (Q4). Condence (Q6) was slightly
higher for younger participants.
Additional feedback was captured with two open questions. The
rst question inquired whether participants would recommend the
explorer to friends or family, and why. Seventy-ve percent of the
participants would recommend the explorer. Eighteen participants
explicitly mention the competence-based explanations (
DG2
) as a
key enabler for nding job recommendations. P17 indicates that “It
helps me to see which competencies I need for a certain job. It is an
easy way to nd a job. P48 indicates that “nowadays there are so
5.3 User feedback
Figure 6 presents the results of the ResQue questions. Overall, par-
ticipants perceived the Labor Market Explorer as an eective tool,
as all questions had a median score of 4 (“agree"). This positive trend
was maintained across the dierent backgrounds and age groups,
see Figure 6b. Technical users were a bit more negative with respect
to overall satisfaction (Q5). They gave very detailed and concrete
suggestions of how particular components could be further im-
proved, including re-sizability of the table. Perceived accuracy (Q1)
was a bit lower for non-native speakers: sixty percent of non-native
speakers were neutral as to whether the explorer helps nding good
many requirements that people do not know if they have the needed
competencies. P21 comments on similar needs: “we can see our own
competencies and see whether they are sucient or not. P1 mentions
“the diverse sub competencies that are visible and can be selected.
The diverse set of lters seems to be another important aspect
(
DG1
), as it was mentioned by twelve other participants, and partic-
ularly the option to lter on combinations (P24, P55). P4 indicates
that “you can nd tting job oers thanks to the broad lters". Partici-
pants also indicate that the overview and lter combination enables
to “easily explore the job market" (P39) and enables “analysis" (P30).
Such actionable insights
(DG3)
were also reected on by P60: “with
65
Explaining and Exploring Job Recommendations RecSys ’19, September 16–20, 2019, Copenhagen, Denmark
Figure 6: Results ResQue questionnaire: a) Overview, b) Re-
sults by participant category, c) Results by age group.
this tool, I can understand what is important. The location-based
overview is another aspect that is often highlighted, as it helps to
see how far the job recommendations are (P18, P37, P40, P58, P59)
and the regions where jobs are oered (P33), supporting potential
location-based job mobility.
Responses to the second open question (tips for the developers)
include a wide range of additional feature requests, including dese-
lecting of groups of lters, which is only implemented for location
in the current interface. Participants also requested a button to undo
actions (P4, P23, P29) and the functionality to update their prole
from within the dashboard (P28). Six participants indicated that the
application overall seems crowded. Suggestions for improvement
were selecting the 10 most relevant competencies instead of show-
ing all competencies (P1), clustering the competencies (P5), a clear
tutorial (P1) and more explanations (P18).
5.4 Interaction patterns
To better understand the use of the tool, we logged the clicks of
participants through the dierent visualization components. We
summarize the overview of the interaction traces by their respective
groups in Figure 7. In general, participants in the technical group
interacted most with the dashboard
(M =
146
.
63
)
; this behavior
was signicantly dierent
(p < .
05
)
compared to the other groups,
see Figure 7c.
Participants with technical background engaged the most with
the job vacancies table, where they performed 69% of the interac-
tions, see Figure 7a, in a signicant dierent way compared to the
other groups
(M =
53
.
89
, p < .
05
)
, Figure 7d. Moreover, when using
the table component, these participants frequently used the multiple
Figure 7: Traces of the participants using the tool, percent-
ages and means of clicks by visual component: a) Overview
of interactions. b) Job Vacancies Table. c) Overall interface
interaction (mean clicks), d) Mean clicks in components, e)
Mean clicks inside Job vacancies table components.
features of the table, including the favorites, sorting, and ltering
features, with a signicant dierence
(M =
32
.
94
, p < .
05
)
com-
pared to the other groups, see Figure 7b,e. Although competence-
based sorting was not used frequently by the other groups, we do
see that the sales group also engaged frequently with the favorites
option.
Participants in the construction group and non-native speakers
engaged less with the job vacancies table (
M =
10
.
92 and
M =
13
.
75, respectively). We can observe that these groups engaged
more with the map component. When interacting with the table,
participants from the construction group also mainly interacted
with the distance and title sorting of job vacancies
(M =
25
.
47
)
. In
general non-native speakers registered the lowest activity
(M =
78
.
3
)
compared to the other groups, and engaged less with the
interactive components such as the map
(M =
40
)
and job vacancy
table (M = 13.75).
66
RecSys ’19, September 16–20, 2019, Copenhagen, Denmark Gutiérrez et al.
6 DISCUSSION
6.1 Answering the research questions
In this section, we discuss the results of the questionnaire and link
them to the interaction patterns to answer our research questions.
User empowerment (RQ1)
. Responses to the questions indi-
cate that job seekers value the use of interactive visualization tech-
niques to nd relevant job vacancies. A key objective of the inter-
face is supporting greater autonomy. The labor market is a rapidly
evolving and dynamics may be dicult to understand for dierent
end-users. In general, the approach is perceived as eective to ex-
plore job recommendations
(DG1)
. The interaction patterns also
indicate that participants engaged well with the interface, with a
mean number of clicks above 78 for all user groups. Most partici-
pants indicated that they will use the explorer again. Also important
is that they feel condent using the tool.
Also explanations
(DG2)
seem to contribute well to better sup-
port of user empowerment. Many of the responses hint to better
understanding of the job recommendations by being able to see
which competencies are required. A diverse set of actionable in-
sights
(DG3)
were also mentioned by participants. Participants
indicate that the overview of competencies enables them to explore
the labor market and understand whether they have the needed
competencies. They also gain insight into regions where most jobs
are oered, supporting potential location-based job mobility. Other
actionable insights include the observation that their user prole
is not up to date. Such insights are key as well, as they trigger job
seekers to update their prole and, in turn, receive better recom-
mendations.
Personal characteristics (RQ2).
In general, we can observe
only few dierences with respect to the age and gender of dierent
participants. The explorer was slightly better perceived by older
participants (45+), who all indicated that the explorer is a good tool
and that they would use the explorer again. We do see dierences in
some of the interaction patterns and subjective responses in relation
to the background of participants. Participants in the technical
group engaged more with all the dierent features of the dashboard,
including the options to lter and sort by competencies and to
favorite specic competencies. Although the perceived eectiveness
scores in general are not very dierent, this group makes the most
eective use of the dierent features of the interface.
The interaction patterns as well as subjective data of non-native
speakers, sales and construction groups indicate that they engage
more with the map component. The sales group also interacts well
with the table overview. The construction group mostly focuses on
sorting by distance and job title. The non-native speakers engaged
also more with the map, but we see a very wide spread in the number
of clicks in this group. Some participants may have struggled too
much with the language. The table overview was perceived as very
useful by all user groups, but the interaction may need further
simplication for some users.
6.2 Design implications
The user-centered design process involving both job seekers and job
mediators identied the importance of dierent parameters used
by job recommendation and search services. Based on this rank-
ing, a dashboard was designed with a map component that shows
the location where jobs are oered, as well as a table component
that provides explanations of the job recommendations, including
needed competencies of jobs that match the prole of the job seeker.
The iterative design process identied several key features for in-
teracting with such an overview, including the option to favorite
competencies that are of interest. In addition, a visual represen-
tation of labor market related information such as the number of
times a specic competence is demanded was deemed interesting
by both job seekers and job mediators, and was conceptualized
as a miniature bar chart in the dashboard, inspired by the UpSet
visualization technique [22].
The competence-based explanations as well as the diverse set of
lters, together with location-based search and overview features,
were highlighted as important. Interaction patterns indicate that
all users engaged well with the interface, although some users did
not use all features to interact with the table overview. The reasons
may be two-fold: for some users, the table is less crowded and still
manageable without rearranging the view or narrowing down the
scope. For other users, and particularly the non-native speaking
user group, the interface may need further simplication. Another
option is to use the dashboard in collaboration with job mediators
to fully exploit the dierent features.
6.3 Limitations
There are some limitations to this work that need to be articulated.
First, the nal prototype used for evaluation was a fully working
prototype, but the link to the actual vacancies did not yet work. The
limitation may have impacted the perceived utility of the dashboard.
Second, the dashboard was evaluated with job seekers of diverse
trainings at VDAB. While we reached a very diverse user group in
terms of technical skills, language and age, a wider audience would
be useful to assess the utility of the dashboard. Third, the interface
was evaluated in the very specic domain of job recommendations.
To generalize the ndings, it would be interesting to replicate the
design process in a dierent domain.
6.4 Future work
Future work will focus on a “simulation mode" of the dashboard,
where users will be able to enter information from scratch, includ-
ing the location and activities/competencies they would like to
do instead of starting from the user prole. The approach will be
researched to further investigate job mobility scenarios. We expect
that such a simulation mode will trigger not only job seekers but
citizens in general to explore further “what-if" scenarios, such as
changing locations and adding skills that they can acquire. A follow-
up study will assess whether such an approach provides further
insights into potential job mobility, and the impact on autonomous
exploration of the labor market.
ACKNOWLEDGMENTS
Part of this research has been funded by the European Social Fund
(ESF Vlaanderen) - under grant agreement no. 7099 (Arbeidsmark-
tverkenner). We thank all the job seekers and job mediators who
participated in the user studies for their interesting feedback and
input.
67
Explaining and Exploring Job Recommendations RecSys ’19, September 16–20, 2019, Copenhagen, Denmark
REFERENCES
[1]
Shibbir Ahmed, Mahamudul Hasan, Md Nazmul Hoq, and Muhammad Abdullah
Adnan. 2016. User interaction analysis to recommend suitable jobs in career-
oriented social networking sites. In Data and Software Engineering (ICoDSE), 2016
Int. Conf. on. IEEE, 1–6.
[2]
Shaha T Al-Otaibi and Mourad Ykhlef. 2012. A survey of job recommender
systems. International Journal of Physical Sciences 7, 29 (2012), 5127–5142.
[3]
Oznur Alkan, Elizabeth M. Daly, Adi Botea, Abel N. Valente, and Pablo Pedemonte.
2019. Where Can My Career Take Me?: Harnessing Dialogue for Interactive
Career Goal Recommendations. In Proceedings of the 24th International Conference
on Intelligent User Interfaces (IUI ’19). ACM, New York, NY, USA, 603–613. https:
//doi.org/10.1145/3301275.3302311
[4]
Nikolaos D Almalis, George A Tsihrintzis, and Nikolaos Karagiannis. 2014. A
content based approach for recommending personnel for job positions. In In-
formation, Intelligence, Systems and Applications, IISA 2014, The 5th Int. Conf. on.
IEEE, 45–49.
[5]
Ivana Andjelkovic, Denis Parra, and John O’Donovan. 2016. Moodplay: Interac-
tive Mood-based Music Discovery and Recommendation. In Proceedings of the
2016 Conference on User Modeling Adaptation and Personalization. ACM, 275–279.
[6]
M. Bakri, Siti Z. Z. Abidin, and Amal Shargabi. 2017. Incremental Filtering
Visualization of JobStreet Malaysia ICT Jobs. In Soft Computing in Data Science,
Azlinah Mohamed, Michael W. Berry, and Bee Wah Yap (Eds.). Springer Singapore,
Singapore, 188–196.
[7]
David Baneres and Jordi Conesa. 2017. A Life-long Learning Recommender Sys-
tem to Promote Employability. Int. Journal of Emerging Technologies in Learning
(iJET) 12, 06 (2017), 77–93.
[8]
Svetlin Bostandjiev, John O’Donovan, and Tobias Höllerer. 2012. TasteWeights:
A Visual Interactive Hybrid Recommender System. In Proceedings of the Sixth
ACM Conference on Recommender Systems (RecSys ’12). ACM, New York, NY, USA,
35–42. https://doi.org/10.1145/2365952.2365964
[9]
Svetlin Bostandjiev, John O’Donovan, and Tobias Höllerer. 2013. LinkedVis:
exploring social and semantic career recommendations. In Proceedings of the 2013
international conference on Intelligent user interfaces. ACM, 107–116.
[10]
Bruno Cardoso, Gayane Sedrakyan, Francisco Gutiérrez, Denis Parra, Peter
Brusilovsky, and Katrien Verbert. 2019. IntersectionExplorer, a multi-perspective
approach for exploring recommendations. International Journal of Human-
Computer Studies 121 (2019), 73 92. https://doi.org/10.1016/j.ijhcs.2018.04.008
Advances in Computer-Human Interaction for Recommender Systems.
[11]
Ivania Donoso-Guzmán and Denis Parra. 2018. An Interactive Relevance Feedback
Interface for Evidence-Based Health Care. In 23rd Int. Conf. on Intelligent User
Interfaces (IUI ’18). ACM, New York, NY, USA, 103–114.
[12]
Eurostat. 2011. Regions in the European Union, Nomenclature of territorial units
for statistics NUTS 2010/EU-27. Technical Report. Eurostat.
[13]
Brynjar Gretarsson, John O’Donovan, Svetlin Bostandjiev, Christopher Hall,
and Tobias Höllerer. 2010. Smallworlds: visualizing social recommendations. In
Computer Graphics Forum, Vol. 29. Wiley Online Library, 833–842.
[14]
Shiqiang Guo, Folami Alamudun, and Tracy Hammond. 2016. RésuMatcher: A
personalized résumé-job matching system. Expert Systems with Applications 60
(2016), 169 182.
[15]
Chen He, Denis Parra, and Katrien Verbert. 2016. Interactive recommender
systems: A survey of the state of the art and future research challenges and
opportunities. Expert Systems with Applications 56 (2016), 9–27.
[16]
Islam A. Heggo and Nashwa Abdelbaki. 2018. Hybrid Information Filtering
Engine for Personalized Job Recommender System. In The International Confer-
ence on Advanced Machine Learning Technologies and Applications (AMLTA2018),
Aboul Ella Hassanien, Mohamed F. Tolba, Mohamed Elhoseny, and Mohamed
Mostafa (Eds.). Springer International Publishing, Cham, 553–563.
[17]
Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. 2000. Explaining
collaborative ltering recommendations. In CSCW ’00: Proceedings of the 2000
ACM Conference on Computer Supported Cooperative Work. ACM, New York, NY,
USA, 241–250.
[18]
Matthias Hutterer. 2011. Enhancing a job recommender with implicit user feedback.
Citeseer.
[19]
Yucheng Jin, Bruno Cardoso, and Katrien Verbert. 2017. How do dierent levels
of user control aect cognitive load and acceptance of recommendations?. In Proc.
4th Joint Workshop on Interfaces and Human Decision Making for Recommender
Systems co-located with ACM Conf. on Recommender Systems (RecSys 2017). CEUR-
WS, 35–42.
[20]
Daniel A Keim, Florian Mansmann, Jörn Schneidewind, and Hartmut Ziegler.
2006. Challenges in visual data analysis. In Information Visualization, 2006. IV
2006. Tenth International Conference on. IEEE, 9–16.
[21]
Danielle H Lee and Peter Brusilovsky. 2007. Fighting information overow with
personalized comprehensive information access: A proactive job recommender. In
Third International Conference on Autonomic and Autonomous Systems (ICAS’07).
IEEE, 21–21.
[22]
Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and
Hanspeter Pster. 2014. UpSet: visualization of intersecting sets. IEEE transactions
on visualization and computer graphics 20, 12 (2014), 1983–1992.
[23]
John O’Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and
Tobias Höllerer. 2008. PeerChooser: visual interactive recommendation. In Pro-
ceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM,
1085–1088.
[24]
Denis Parra and Peter Brusilovsky. 2015. User-controllable Personalization. Int.
J. Hum.-Comput. Stud. 78, C (June 2015), 43–67. https://doi.org/10.1016/j.ijhcs.
2015.01.007
[25]
Michael Quinn Patton. 2014. Qualitative Research & Evaluation Methods: Integrat-
ing Theory and Practice. SAGE Publications.
[26]
Pôle emploi. [n.d.]. Competent - Les prols et les standards. http://www.
competent.be/competent-fr/main.html
[27]
Pearl Pu, Li Chen, and Rong Hu. 2011. A user-centric evaluation framework for
recommender systems. In Proceedings of the fth ACM conference on Recommender
systems. ACM, 157–164.
[28]
Rachael Rafter, Keith Bradley, and Barry Smyth. 2000. Personalised retrieval
for online recruitment services. In The BCS/IRSG 22nd Annual Colloquium on
Information Retrieval (IRSG 2000), Cambridge, UK, 5-7 April, 2000.
[29]
Michael Reusens, Wilfried Lemahieu, Bart Baesens, and Luc Sels. 2018. Evaluating
recommendation and search in the labor market. Knowledge-Based Systems 152
(2018), 62–69.
[30]
Walid Shalaby, BahaaEddin AlAila, Mohammed Korayem, Layla Pournajaf, Khal-
ifeh AlJadda, Shannon Quinn, and Wlodek Zadrozny. 2017. Help me nd a job:
A graph-based approach for job recommendation at scale. In Big Data (Big Data),
2017 IEEE Int. Conf. on. IEEE, 1544–1553.
[31]
Nava Tintarev. 2017. Presenting diversity aware recommendations: Making
challenging news acceptable. (2017), 4.
[32]
Chun-Hua Tsai and Peter Brusilovsky. 2018. Beyond the Ranked List: User-Driven
Exploration and Diversication of Social Recommendation. In 23rd International
Conference on Intelligent User Interfaces (IUI ’18). ACM, New York, NY, USA,
239–250. https://doi.org/10.1145/3172944.3172959
[33]
Katrien Verbert, Denis Parra, and Peter Brusilovsky. 2016. Agents vs. users: visual
recommendation of research talks with multiple dimension of relevance. ACM
Transactions on Interactive Intelligent Systems (TiiS) 6, 2 (2016), 11.
[34]
Elena Zudilova-Seinstra, Tony Adriaansen, and Robert van Liere. 2009. Overview
of Interactive Visualisation. Springer London, London, 3–15.
68
... Behavioral interactions, e.g., [79,113,122,132,136] Explicit preferences [61,125] Preference model [12,[16][17][18]60] Dynamic preferences Neural architectures (e.g. LSTM) [50,88,104] Time-dependent features [88] Time-dependent loss function [88] Fig. 4. ...
... Another way that user preferences are taken into consideration in recommendation is by using explicit preferences specified in the user profile (e.g., interests, location, etc.) or in a dashboard. Gutiérrez et al. [61] designed a dashboard for job seekers to visualize and explore available vacancies based on their preferences. A fuzzy-based recommendation was proposed by Slama and Darmon [125] that matches job seekers with job postings based on their fuzzy preferences in their profiles. ...
... In other studies, explainability is provided using explicit relations in data. In [61] a dashboard was provided to view the job seekers' affinity with the required skills for the jobs that are recommended. In [127], recommendations were generated using a knowledge graph together with a template for explainability, where the template was then completed using the nodes in the knowledge graph. ...
Preprint
Full-text available
E-recruitment recommendation systems recommend jobs to job seekers and job seekers to recruiters. The recommendations are generated based on the suitability of the job seekers for the positions as well as the job seekers' and the recruiters' preferences. Therefore, e-recruitment recommendation systems could greatly impact job seekers' careers. Moreover, by affecting the hiring processes of the companies, e-recruitment recommendation systems play an important role in shaping the companies' competitive edge in the market. Hence, the domain of e-recruitment recommendation deserves specific attention. Existing surveys on this topic tend to discuss past studies from the algorithmic perspective, e.g., by categorizing them into collaborative filtering, content based, and hybrid methods. This survey, instead, takes a complementary, challenge-based approach, which we believe might be more practical to developers facing a concrete e-recruitment design task with a specific set of challenges, as well as to researchers looking for impactful research projects in this domain. We first identify the main challenges in the e-recruitment recommendation research. Next, we discuss how those challenges have been studied in the literature. Finally, we provide future research directions that we consider promising in the e-recruitment recommendation domain.
... An advantage of using job ontologies is that this also simplifies the implementation of keyword-based search engines and simplifies filtering. Gutiérrez et al. [50] consider such an approach, in which an interactive recommender system was built on top of the knowledge-based recommender ELISE [39], where users are able to filter jobs based on travel distance or the type of contract. The recommender system also explains why the recommendation is (not) given, e.g., by indicating that a required skill is not mastered by the job seeker. ...
Preprint
Full-text available
This paper provides a review of the job recommender system (JRS) literature published in the past decade (2011-2021). Compared to previous literature reviews, we put more emphasis on contributions that incorporate the temporal and reciprocal nature of job recommendations. Previous studies on JRS suggest that taking such views into account in the design of the JRS can lead to improved model performance. Also, it may lead to a more uniform distribution of candidates over a set of similar jobs. We also consider the literature from the perspective of algorithm fairness. Here we find that this is rarely discussed in the literature, and if it is discussed, many authors wrongly assume that removing the discriminatory feature would be sufficient. With respect to the type of models used in JRS, authors frequently label their method as `hybrid'. Unfortunately, they thereby obscure what these methods entail. Using existing recommender taxonomies, we split this large class of hybrids into subcategories that are easier to analyse. We further find that data availability, and in particular the availability of click data, has a large impact on the choice of method and validation. Last, although the generalizability of JRS across different datasets is infrequently considered, results suggest that error scores may vary across these datasets.
... Typically, a combination of techniques from information retrieval and recommender systems are used that are trained on training data supplied by human recruiters, which represent their past actions and assessments. The goal of the AI algorithm is then to learning a proper scoring function which represents the preferences of the individual job seeker [6,7,11,12,21]. ...
Conference Paper
Full-text available
We explore the information seeking behaviour of the recruiters while they search to find candidate job seekers that match the open positions posted by the organizations. We perform a set of contex-tual inquiries with recruiters at one of Scandinavia's largest job portals and recruitment agencies by using its job database system. We aim to better understand short-term (matching) and long-term (recruitment) information seeking behaviour of the recruiters and their interaction with the search engine based on Solr. Based on the conducted contextual inquiries, we list a set of design implications to be used for better matching systems that can assist the recruiters to find more relevant candidates.
... However, similarly to the above, collecting and analyzing such data did not fit in the scope of our present work. In summary, our paper revolves around studying algorithmic methods that aim to help both jobseekers and recruiters find a better match between individuals and occupations, we consider studies with actual end-users out of scope [13]. ...
Preprint
Full-text available
The labor market is constantly evolving. Occupations are changing, being added, or disappearing to fit the needs of today's market. In recent years the pace of this change has accelerated, due to factors such as globalization, digitization, and the shift to working from home. Different factors are relevant when selecting employment, e.g., cultural fit, compensation, provided degree of freedom. To successfully fulfill an occupation the gap between required (by the job) and possessed (by the job seeker) skills needs to be as small as possible. Decreasing this skill-gap improves the fit between a job candidate and occupation. In this paper we propose a custom-built Skills & Occupation Knowledge Graph (KG) that fits the above described dynamic nature of the labor market, by leveraging existing skills and occupation taxonomies enriched with external job posting data. We leverage this KG and explore several applications for skills-based matching of jobs to job seekers. First, we study link prediction as a means to quantify relevance of skills to occupations, which can help in prioritizing learning and development of employees. Next, we study node similarity methods and shortest path algorithms for career pathfinding. Finally, we leverage a term weighting method for identifying which skills are most "distinctive" for different (types of) occupations.
... The system allows the exploration of a music collection through latent emotional dimensions, thereby improving acceptance and understanding of recommendations. Labor Market Explorer (Gutiérrez et al. 2019) is another example of the interactive recommendation system, which enables job seekers to explore the labor market in a personalized way based on their skills and competencies. Those works propose a good solution for dealing with cold start problem and provide guidance in educational course selections. ...
Article
Full-text available
The abundance of courses available in a university often overwhelms students as they must select courses that are relevant to their academic interests and satisfy their requirements. A large number of existing studies in course recommendation systems focus on the accuracy of prediction to show students the most relevant courses with little consideration on interactivity and user perception. However, recent work has highlighted the importance of user-perceived aspects of recommendation systems, such as transparency, controllability, and user satisfaction. This paper introduces CourseQ, an interactive course recommendation system that allows students to explore courses by using a novel visual interface so as to improve transparency and user satisfaction of course recommendations. We describe the design concepts, interactions, and algorithm of the proposed system. A within-subject user study (N=32) was conducted to evaluate our system compared to a baseline interface without the proposed interactive visualization. The evaluation results show that our system improves many user-centric metrics including user acceptance and understanding of the recommendation results. Furthermore, our analysis of user interaction behaviors in the system indicates that CourseQ could help different users with their course-seeking tasks. Our results and discussions highlight the impact of visual and interactive features in course recommendation systems and inform the design of future recommendation systems for higher education.
Article
Full-text available
E-commerce development has not bypassed the labor market, and employers seeking to hire are posting vacancies on specialized online platforms. Operators of such online platforms and employers are assessing the attractiveness of job vacancies, as they are interested in making online job vacancies (OJVs) more attractive to job seekers, i.e. to increase the number of views of the posted job ad, the number of times job seekers submit an application form and conversions (the ratio of the two). However, given that job types and job seekers are extremely heterogeneous, it is difficult for employers and specialized online platform operators to design attractive job ads. In collaboration with a leading OJV platform in Slovakia, we study whether machine-learning methods can improve the predictions of OJV attractiveness on a sample of 32482 OJVs that offer as many as 883 job features. Our study shows that, as opposed to various linear models, considerable prediction improvements can be achieved using random forest, which exploits possible nonlinear relationships between combinations of explanatory variables and measures of attractiveness. Based on this insight, we perform a statistical evaluation of key variables of importance. We find that the job classification, job benefits, and variables related to a simple morphological description of the job and job title are relevant. The results of this study can help the operators of specialized job vacancy platforms and employers improve their job ads to attract more job seekers.
Conference Paper
Full-text available
p>With the increase in data volume, velocity and types, intelligent human-agent systems have become popular and adopted in different application domains, including critical and sensitive areas such as health and security. Humans’ trust, their consent and receptiveness to recommendations are the main requirement for the success of such services. Recently, the demand on explaining the recommendations to humans has increased both from humans interacting with these systems so that they make an informed decision and, also, owners and systems managers to increase transparency and consequently trust and users’ retention. Existing systematic reviews in the area of explainable recommendations focused on the goal of providing explanations, their presentation and informational content. In this paper, we review the literature with a focus on two user experience facets of explanations; delivery methods and modalities. We then focus on the risks of explanation both on user experience and their decision making. Our review revealed that explanations delivery to end-users is mostly designed to be along with the recommendation in a push and pull styles while archiving explanations for later accountability and traceability is still limited. We also found that the emphasis was mainly on the benefits of recommendations while risks and potential concerns, such as over-reliance on machines, is still a new area to explore.</p
Article
Recommender systems are increasingly used in more high risk application domains, including in the domain of Human Resources (HR). These recommender systems help end-users find relevant vacancies out of an abundant overload of available vacancies, but also support other important objectives such as job mobility. Despite the use in industry applications, there are several research challenges associated to such objectives that have not yet been addressed in detail in this context, such as supporting end-users to steer the recommendation process with input and feedback and increasing diversity of recommendations. The goal of our workshop is to build a strong research capacity around recommender systems for HR to address these challenges. This paper describes the goal and activities of the Workshop on Recommender Systems for Human Resources (RecSys in HR 2021), organized at the 15th ACM Conference on Recommender Systems. Date : 10--14 September, 2021. Website : https://foo2021.net.
Chapter
Full-text available
The recommendation system, also known as recommender system or recommendation engine/platform, is considered as an interdisciplinary field. It uses the techniques of more than one field. Recommender system inherits approaches from all of machine learning, data mining, information retrieval, information filtering and human-computer interaction. In this paper, we propose our value-added architecture of the hybrid information filtering engine for job recommender system (HIFE-JRS). We discuss our developed system’s components to filter the most relevant information and produce the most personalized content to each user. The basic idea of recommender systems is to recommend items for users to suit their interests. Similarly the project tends to recommend relevant jobs for job-seekers by utilizing the concepts of recommender systems, information retrieval and data mining. The project solves the problem of flooding job-seekers with thousands of irrelevant jobs which is a frustrating and time-wasting process to let job-seekers rely on their limited searching abilities to dig into tons of jobs for finding the right job.
Conference Paper
Full-text available
The beyond-relevance objectives of recommender systems have been drawing more and more attention. For example, a diversity-enhanced interface has been shown to associate positively with overall levels of user satisfaction. However, little is known about how users adopt diversity-enhanced interfaces to accomplish various real-world tasks. In this paper, we present two attempts at creating a visual diversity-enhanced interface that presents recommendations beyond a simple ranked list. Our goal was to design a recommender system interface to help users explore the different relevance prospects of recommended items in parallel and to stress their diversity. Two within-subject user studies in the context of social recommendation at academic conferences were conducted to compare our visual interfaces. Results from our user study show that the visual interfaces significantly reduced the exploration efforts required for given tasks and helped users to perceive the recommendation diversity. We show that the users examined a diverse set of recommended items while experiencing an improvement in overall user satisfaction. Also, the users' subjective evaluations show significant improvement in many user-centric metrics. Experiences are discussed that shed light on avenues for future interface designs.
Article
Full-text available
Online job boards are one of the central components of modern recruitment industry. With millions of candidates browsing through job postings everyday, the need for accurate, effective, meaningful, and transparent job recommendations is apparent more than ever. While recommendation systems are successfully advancing in variety of online domains by creating social and commercial value, the job recommendation domain is less explored. Existing systems are mostly focused on content analysis of resumes and job descriptions, relying heavily on the accuracy and coverage of the semantic analysis and modeling of the content in which case, they end up usually suffering from rigidity and the lack of implicit semantic relations that are uncovered from users' behavior and could be captured by Collaborative Filtering (CF) methods. Few works which utilize CF do not address the scalability challenges of real-world systems and the problem of cold-start. In this paper, we propose a scalable item-based recommendation system for online job recommendations. Our approach overcomes the major challenges of sparsity and scalability by leveraging a directed graph of jobs connected by multi-edges representing various behavioral and contextual similarity signals. The short lived nature of the items (jobs) in the system and the rapid rate in which new users and jobs enter the system make the cold-start a serious problem hindering CF methods. We address this problem by harnessing the power of deep learning in addition to user behavior to serve hybrid recommendations. Our technique has been leveraged by CareerBuilder.com which is one of the largest job boards in the world to generate high-quality recommendations for millions of users.
Conference Paper
Full-text available
As intelligent interactive systems, recommender systems focus on determining predictions that fit the wishes and needs of users. Still, a large majority of recommender systems research focuses on accuracy criteria and much less attention is paid to how users interact with the system, and in which way the user interface has an influence on the selection behavior of the users. Consequently, it is important to look beyond algorithms. The main goals of the IntRS workshop are to analyze the impact of user interfaces and interaction design, and to explore human interaction with recommender systems from a human decision making perspective. Methodologies for evaluating these aspects are also within the scope of the workshop.
Conference Paper
Career goals represent a special case for recommender systems and require considering both short and long term goals. Recommendations must represent a trade off between relevance to the user, achievability and aspirational goals to move the user forward in their career. Users may have different motivations and concerns when looking for a new long term goal, so involving the user in the recommender process becomes all the more important than in other domains. Additionally, the cost to the user of making a bad decision is much higher than investing two hours in watching a movie they don't like or listening to an unappealing song. As a result, we feel career recommendations is a unique opportunity to truly engage the user in an interactive recommender as we believe they will invest the cognitive load. In this paper, we present an interactive career goal recommender framework that leverages the power of dialogue to allow the user interactively improve the recommendations and bring their own preferences to the system. The underlying recommendation algorithm is a novel solution that suggests both short and long term goals through utilizing the sequential patterns extracted from career trajectories that are enhanced with features of the supporting user profiles. The effectiveness of the proposed solution is demonstrated with extensive experiments on two real world data sets.
Article
In this paper, we advent a novel approach to foster exploration of recommendations: IntersectionExplorer, a scalable visualization that interleaves the output of several recommender engines with human-generated data, such as user bookmarks and tags, as a basis to increase exploration and thereby enhance the potential to find relevant items. We evaluated the viability of IntersectionExplorer in the context of conference paper recommendation, through three user studies performed in different settings to understand the usefulness of the tool for diverse audiences and scenarios. We analyzed several dimensions of user experience and other, more objective, measures of performance. Results indicate that users found IntersectionExplorer to be a relatively fast and effortless tool to navigate through conference papers. Objective measures of performance linked to interaction showed that users were not only interested in exploring combinations of machine-produced recommendations with bookmarks of users and tags, but also that this “augmentation” actually resulted in increased likelihood of finding relevant papers in explorations. Overall, the findings suggest the viability of IntersectionExplorer as an effective tool, and indicate that its multi-perspective approach to exploring recommendations has great promise as a way of addressing the complex human-recommender system interaction problem.
Article
This study evaluates the most popular recommender system algorithms for use on both sides of the labor market: job recommendation and job seeker recommendation. Recent research shows the drawbacks of focusing solely on predictive power when evaluating recommender systems, which become especially prominent in job- and job seeker recommendation, where aspects such as reciprocity and item spread are two other vital performance metrics for the quality of recommendations. Besides evaluating using these extra metrics, we compare recommendation with search using free text search engines. We measure what is gained, and what is lost when consuming items (jobs and job seekers) retrieved using search versus items presented via a recommender system. Based on insights in date recommendation literature, we propose changes to rating matrix construction aimed at mitigating the drawbacks of recommendation in the labor market. Our results, obtained from extensive experimentation on three datasets gathered from the Flemish public employment services, show that popular recommender algorithms perform significantly worse than user search in terms of reciprocity. Furthermore, we show that by swapping the rating matrices between two sides of a reciprocal recommender context, we can outperform user search in terms of reciprocity with limited trade off in predictive power. The insights from this research can help actors in the labor market to better understand the positioning of recommendation versus search, and to provide better job recommendations and job seeker recommendations.
Conference Paper
We design, implement and evaluate EpistAid, an interactive rel- evance feedback system to support physicians towards a more efficient citation screening process for Evidence Based Health Care (EBHC). The system combines a relevance feedback algorithm with an interactive interface inspired by Tinder-like swipe interaction. To evaluate its efficiency and effectiveness in the citation screening process we conducted a user study with real users (senior medicine students) using a large EBHC dataset (Epistemonikos), with around 400,000 documents. We compared two relevance feedback algorithms, Rocchio and BM25-based. The combination of Rocchio relevance feedback with the document visualization yielded the best recall and F-1 scores, which are the most important metrics for EBHC document screening. In terms of cognitive demand and effort, BM25 relevance feedback without visualization was perceived as needing more physical and cognitive effort. EpistAid has the potential of improving the process for answering clinical questions by reducing the time needed to classify documents, as well as promoting user interaction. Our results can inform the development of intelligent user interfaces for screening research articles in the clinical domain and beyond.
Conference Paper
In Malaysia, JobStreet is one of the most popular job search engines for job seekers. There are thousands of available positions posted on the website by numerous companies. Due to many choices available for the job seekers, ability to quickly find the jobs that met their criteria such as location, industry, job role and specialization will be a help. Currently, JobStreet only allows job filtering by specialization and state. To get more details for the job, they need to spend significant amount of time reading the descriptions. Thus, in order to assist the users in their job seeking process, this project proposes a system which will use interactive visualization with incremental filtering that consists of heat map and barcharts. The developed system allows users to filter and get an overview of all posted jobs that met their criteria in less time.