Content uploaded by Jo Ann Oravec
Author content
All content in this area was uploaded by Jo Ann Oravec on Jun 17, 2024
Content may be subject to copyright.
Evaluating the Impact of
Artificial Intelligence Versus Human Management
on Modifying Workplace Behavior
Benny Traub, Inspira AI Corp.
Izzy Traub, Inspira AI Corp.
Dr. Paul W. Thurman, Professor, Columbia University
Dr. Phil Peper, Research Associate, Arizona State University
Dr. Jo Ann Oravec, Professor, University of Wisconsin at Whitewater
Special thanks to Dr. Adam Waytz, Professor, Kellogg School of Management for his input into study design
Overview
This pilot study was a behavior change experiment, where a small sample of employees of a
digital marketing company was asked to form new habits related to their work schedules.
Three groups of remote employees were studied over 14 weeks. One group was coached by a
human manager, another group was coached by an AI system consisting of robotic process
automation and a specially trained conversational chatbot. The final group was coached by both
humans and AI.
All groups of employees had previously been in the habit of working whenever they felt like it,
with no fixed schedule. Starting in November of 2023, without disclosing that they were to be
part of a pilot study, employees were asked to change their behavior, specifically:
Plan their work schedule at least a few days in advance, start work each day at their planned
start time and work during the hours that had been scheduled
The focus of the pilot was to create a study design that is capable of comparing human to AI
management systems and to draw initial conclusions regarding the efficacy of the current state
of AI management technology developed by Inspira AI Corp.
The results of the pilot study (starting on page 5) demonstrate that the AI management system,
without any human involvement, created behavior change on par with the human manager.
Readers should note that at least some of the data indicated that a combination of AI and
human managers may deliver even better results, under certain circumstances, than AI or
human managers on their own. Larger sample sizes are needed to verify this hypothesis and
the conditions under which it may be true.
1
Electronic copy available at: https://ssrn.com/abstract=4739430
Introduction
Managers have often found it difficult to inspire, nudge, or coerce changes in the everyday
working habits of employees (Li et al., 2023; Schoch et al., 2023)
1
. These difficulties were
compounded recently as many modifications to workplace processes were put into place
because of the COVID pandemic. Employees who never used flextime before were given the
opportunity to set their own schedules, often with little direction from managers. Now that
concerns about the pandemic have lessened considerably, many employees who were recently
utilizing flextime scheduling and freeform work habits are now often asked to conform to
planned scheduling modalities, even if they are still working on a remote basis away from direct
oversight of managers and colleagues. The research described in this paper was conducted in
this challenging context of workplace flux to compare the efficacy of human, artificial intelligence
(AI), and human-AI combination managerial systems in assisting employees to shift into more
predictable work habits. This research is part of a larger set of studies in which the applicability
of AI-enhanced management in various settings is being examined (Traub et al., 2023)
2
. These
studies are inspired by the potential for human managers to have an array of AI tools for the
support of their employees in making shifts in work habits and becoming more in sync with the
larger objectives of their organizations. The possibility that synthetic AI ‘management’ systems
can offload some of the more intensive day-to-day support that employees could need, presents
potential savings of managerial time and attention. Synergetic AI and human strategies may
provide even more effective support for employee work habit change and thus stabilize and
enhance future workplaces.
Literature Review
Introductions of AI technologies into workplace settings can be complicated, facing legal and
human-resource related challenges (Arslan et al., 2022; Rodrigues, 2020)
3,4
. These issues
include employees’ concerns about fairness as well as managers’ uncertainties about their own
future activities and statuses. The use of workplace AI systems for “coaching,” “co-piloting,” and
other forms of mentoring was given considerable stimulus by the Covid pandemic but has
continued as a managerial trend as remote work options have persisted in many organizations
(Schoch et al., 2023)
5
. Among the various characterizations of these mentoring systems is the
notion of “Digital Productivity Assistants” or DPAs (Cranefield et al., 2023)
6
, systems designed to
enhance employee production through behavioral change efforts. A commonly accepted
vocabulary for characterizing AI management systems has not yet emerged, however, so
whether the systems are identified as AI coaches, co-pilots, mentors, or DBAs can vary in
workplace and research efforts.
Behavioral change modification strategies in the workplace such as habit formation are often
rooted in the notion of “nudging” (Thaler and Sunstein, 2009; Mele et al., 2021)
7,8
, in which the
choice architecture is designed in ways that can shape human behavior. Applications of AI to
2
Electronic copy available at: https://ssrn.com/abstract=4739430
behavioral change extend nudging in a “smart” manner through the individualization of choice
architectures and customization of human-computer interaction. The positive and even
“friendly” employee-AI interactions that are integral to the design of some AI management
systems also play roles in their effectiveness in stimulating workplace habit formation (Traub et
al., 2023)
2
. Gkinko and Elbanna (2022)
9 describe how desirable outcomes such as increased
social connection and emotional dynamics among employees (such as “empathy, forgiveness,
tolerance, fairness and closeness”) can be engendered by the use of AI management systems.
Background
Remote employees at a digital marketing company were accustomed to setting their own work
hours.
The lack of structure often results in workers not working at all, working fewer hours than
verbally promised, or working during hours that do not overlap with other team members’,
producing lag times in communication with direct impact upon the production pipeline.
All employees use a semi-automated time tracker to indicate which project they are working on,
which enabled the researchers to verify the actual hours worked.
Management (whether human or AI) can coach remote workers to construct well-considered
schedules that take into account the obstacles the workers face in their work settings as well as
fulfilling the obligations that are mapped out in those schedules. To this end, management must
convey to the remote workers that their schedule-related choices and habits will affect the
success of the entire operation.
This experiment evaluated the use of a ‘flex-planned’ program, where remote workers may set
their own schedules, but must plan it in advance. In addition, they are expected to adhere to the
schedule that they created.
Our primary objective is the formation of habits for the following behaviors:
1. Creating a carefully-considered work schedule in advance
2. Showing up on time, reasonably near the scheduled start time of their shift
3. Working the full shift as per the schedule by measuring the ratio of time worked within
the start/end times of their planned schedule
Despite the priming, the change the remote employees were asked to make were signinificant,
as the full-flex habits were well established. We expected a degree of failure, with the possibility
of some team members not surviving the transition.
This experiment compared the resulting behavior change when the policy is managed entirely
by Inspira’s autonomous ‘machine suite’, compared to a control group who were managed
3
Electronic copy available at: https://ssrn.com/abstract=4739430
solely by a human manager. A potential benefit of being managed with the help of an
autonomous entity is its capability to inspire personal reflection in a non-judgmental setting and
encourage openness and careful consideration.
Methods
Participants
Twenty-six remote employees working at a digital marketing company were asked to change
their work habits by using a calendar app to plan their intended work schedules at least several
days in advance, then adhere to those schedules. This was a single-blind study with the
participants unaware that they would be participating in an experiment.
Design and Procedure
Prior to the experiment, all employees had been working ‘full-flex’ hours that gave them freedom
to work any hour of the day/week as long as they completed their assigned workloads.
Participating employees were asked to adopt a ‘flex-planned’ program where they could set
their own weekly schedules, but must plan it in advance. All team members had been informed
of this policy change, first announced in September of 2022 and repeated numerous times
since. The data presented in this paper covered all work days between and including November
6 of 2023, through February 11 of 2024.
Participating employees were asked to, 1) set their schedule for the following week by no later
than Sunday of each preceding week, 2) check in for work on time, and 3) work during the
planned hours. How well employees adhered to these goals served as our critical outcome
variables. The employees fell into three groups. Eight were coached by a human manager,
thirteen by an AI coaching system and five by both human and AI coaches.
Both the human and AI manager served to coach and assist the participating employees toward
adhering to the new flex-planned program. All groups received weekly reminders to set their
schedule, because past research has found reminders are particularly beneficial when an
intended goal is complex (Peper et al., 2023)
10
.
When the human was triggering the reminders, they do so through a realtime chat interface, and
included personalized encouragement to do better.
When the AI was triggering the reminders, they were either automated (email and push
notifications), or they were by realtime chatbot with an encouragement to do better. The AI
system could detect when the employee started work. Under these conditions, a chatbot would
automatically launch on the user’s computer and infrequently initiate a ‘coaching’ conversation
when the employee was either ontime, early or late for work. The AI system personalized the
conversations, praising when the employee was early or on time, and gently coaching those
4
Electronic copy available at: https://ssrn.com/abstract=4739430
where were late. The coaching bot referenced the employee’s planned start times, actual start
times, next shift date/time and their historical adherence to the new policies.
Participating employees successfully set their schedule if they planned each day of next week
by Sunday of the current week. When a participating employee scheduled a day to work, they
also specified a time window (e.g., 1pm - 6pm). Participating employees were deemed checked
in for work on time if they signed in through a designated desktop software called TrueTime. All
employees had used this software previously and were familiar with how it operated. Checking
in for work was considered on-time if employees signed on to TrueTime within a 15-minute
window of their scheduled start time (e.g., for a 1pm scheduled start time, a window of 12:50pm
- 1:05pm would be considered on time). An employee who worked four of the scheduled five
hours (e.g., 2pm - 6pm on Monday) would have worked 80% of the scheduled hours that day.
Results
Prior to the pilot study there was the expectation of high rates of failure due to anticipated
challenges remote employees may face in changing their habits. The resulting changes in
observed behavior were higher than anticipated across all three groups, with the exception of
how well the employees were able to adhere to the planned hours of any given day.
Planning of Work Schedules
Not surprisingly, employees sometimes forgot or neglected to plan their work schedules in
advance, and therefore ended up working more days than they had planned. The ratio of
worked days that were planned to those that were unplanned was very similar between
employees who were coached by Human or AI, with success ratios of 0.45 and 0.44
respectively. Those coached by both human and AI were more successful, obtaining an average
planned to unplanned work days ratio of 0.72 (72%).
5
Electronic copy available at: https://ssrn.com/abstract=4739430
Starting Work on Time
Employees in all three groups had similar success in starting work at the intended start time.
Those coached by a human manager adhered to their plans 44% of the time, those within the AI
group had an adherence of 42%, and those being coached by both human and AI showed up
46% of the time.
Advance Planning
Employees in all three groups had insignificant differences with regards to how far in advance
they planned their workdays. Those coached by a human manager planned 5.79 days in
advance. Those within the AI group planned 5.13 days in advance, and those being coached by
both human and AI came in 4.82 days in advance. The results were so similar across the three
groups that any difference could easily be explained by statistical anomalies associated with the
small sample sizes, as is the risk with any small-sample study such as this one.
6
Electronic copy available at: https://ssrn.com/abstract=4739430
Worked during scheduled hours
Employees also in all three groups had similar results in working during their planned shifts, with
a high degree of failure across the board. Those coached by a human manager adhered to their
planned hours only 7% of the time, those within the AI group had an adherence of 6%, and
those being coached by both human and AI came in at 10%. The remote workers who
participated in the study had been in the habit of breaking their work day up with frequent and
long breaks. The interventions utilized in this pilot study were not sufficient to break these
habits, illustrating the difficulties in changing behavior when entrenched habits exist, at least
with remote workers.
Summary
Although the results described below are preliminary and are based on a small sample, they
present some interesting topics for further study.
Our findings suggest that AI-based management may indeed be able to influence changes in
behavior at a pace comparable to that of human managers. Therefore, we continue to support
the theory that certain usecases of AI management systems may currently be as effective as
human management in improving reliability and modifying other workplace behaviors.
Furthermore, under certain conditions a combination of AI and human managers may
outperform either AI or human managers on their own.
7
Electronic copy available at: https://ssrn.com/abstract=4739430
Future Research
The findings in this pilot study substantiates the possibility that AI management systems may
replace some of the duties traditionally performed by humans, specifically that of monitoring and
analyzing behavior, then implementing contructive feedback loops designed to modify behavior
or even improve the work experience. For example, by offering more consistent praise for work
well-done, which would normally fall under the radar of a human manager. Or by coaching
employees during early signals of poor performance, before the issue grows and results in a
human confrontation. Essentially, this enables employees to outperform what would normally be
possible. It sets them up for more favorable performance reviews, which may lead to higher job
security or economic benefits such as raises or bonuses.
A synergy between AI and human management might offer organizations substantial benefits,
serving as a compelling reason to develop a blend of both human and AI management instead
of shifting entirely to AI solutions.
To confirm these hypotheses, further research involving larger and randomly selected samples
is necessary.
The kind of research portrayed in this paper is highly sensitive to cultural, economic, and
disciplinary differences in workplace settings (Lewis et al., 2021; Li et al., 2023)
11
. Habit
formation in complex environments involves an assortment of psychological and social
dimensions (Rebar, Rhodes, & Verplanken, 2023)
12
; changes made in some arenas could have
an impact on others. Some modifications of the content and style of AI management systems
might be necessary for various audiences and contexts. Employees who are reasonably
familiar with technology and comfortable with AI concepts may be more amenable to the
interaction provided by the AI management system described in this paper. However, as
knowledge about and comfort with AI capabilities increase on the part of many working-aged
individuals, these kinds of concerns may decrease. The apparent ease-of-use of the system
discussed in this paper (Inspira’s autonomous Machine Suite) increases the prospects for
successful implementation in a wide variety of workplace settings.
Future examination of the capabilities of chatbots for eliciting sensitive and revelatory
information from human interactors could also extend the research described in this paper. This
information could increase the effectiveness of chatbots in the support of workplace habit
formation, possibly improving results over those of human managers’ efforts in dramatic ways.
For example, employee reflections about why they are having trouble changing their work habits
can be used to create more specific and possibly effective human-AI interactions; in many
contexts, the employees may not want to reveal such information to a human manager because
of personal embarrassment or other kinds of reticence. Development and use of employee
profiles in habit formation protocols could also enhance the capabilities of chatbots to foster
behavioral change.
8
Electronic copy available at: https://ssrn.com/abstract=4739430
In a follow-up phase for those who struggle to adhere to their schedule, both human and
autonomous system participants can identify obstacles to the fulfillment of their planned
schedule (such as “my child started soccer practice last week” or “I needed to give my dad a
ride to the clinic”). This investigation process can reveal whether the remote workers are more
straightforward and open with their human bosses or with the autonomous entities. With
increased self-reflection, the remote workers may submit more accurate schedules as well. A
deliberate and careful approach to schedule development will produce better results.
Conclusion
For centuries, human managers have worked with employees to help create and reinforce
appropriate work habits. The initiative described in this paper is an effort to ascertain how an AI
management system compares with human managers in habit formation assistance and related
forms of workplace mentoring. The remote context of these experiments provides additional
obstacles and complications for behavioral modification, presenting settings in which employees
have many distractions and competing obligations for their attention and energies. The results
of the experiments conducted with HARRi versus human managers show that AI systems can
indeed enhance efforts to improve work habits in remote work environments. The habit
formation friction involved with remote work provides challenges to both modes of employee
habit formation, AI-assisted as well as human-assisted. The results described in this paper
present tantalizing prospects for future AI systems that are designed to have a wider set of
behavior modification objectives and that draw from even more extensive informational profiles
of employees.
References
1. Li, C., Zhang, Y., Niu, X., Chen, F., & Zhou, H. (2023). Does Artificial Intelligence Promote or
Inhibit On-the-Job Learning? Human Reactions to AI at Work. Systems , 11 (3).
https://doi.org/10.3390/systems11030114
2. Traub, B., Thurman, P., Kay, C., & Oravec, J. A. (2023). Perception of Human Characteristics in
an AI Chatbot acting as on-the-job Coach. Available at SSRN 4647255
3. Arslan, A., Cooper, C., Khan, Z., Golgeci, I., & Ali, I. (2022). Artificial intelligence and human
workers interaction at team level: a conceptual assessment of the challenges and potential HRM
strategies. International Journal of Manpower , 43 (1), 75-88
https://doi.org/10.1108/IJM-01-2021-0052
4. Rodrigues, R. (2020). Legal and human rights issues of AI: Gaps, challenges and vulnerabilities.
Journal of Responsible Technology , 4 . http://dx.doi.org/10.1016/j.jrt.2020.100005
9
Electronic copy available at: https://ssrn.com/abstract=4739430
5. Schoch, M., Gimpel, H., Maier, A., & Neumeier, K. (2023). From broken habits to new intentions:
how COVID-19 expands our knowledge on post-adoptive use behaviour of digital communication
and collaboration. European Journal of Information Systems , 32 (6), 989-1010
6. Cranefield, J., Winikoff, M., Chiu, Y. T., Li, Y., Doyle, C., & Richter, A. (2023). Partnering with AI:
The case of digital productivity assistants. Journal of the Royal Society of New Zealand , 53 (1),
95-118. https://doi.org/10.1080/03036758.2022.2114507
7. Thaler, R. H., & Sunstein, C. R. (2009). Nudge: Improving decisions about health, wealth, and
happiness . New York: Penguin
8. Mele, C., Spena, T. R., Kaartemo, V., & Marzullo, M. L. (2021). Smart nudging: How cognitive
technologies enable choice architectures for value co-creation. Journal of Business Research ,
129 , 949-960. https://doi.org/10.1016/j.jbusres.2020.09.004
9. Gkinko, L., & Elbanna, A. (2022). Hope, tolerance and empathy: employees' emotions when
using an AI-enabled chatbot in a digitalised workplace. Information Technology & People , 35 (6),
1714-1743. https://www.emerald.com/insight/content/doi/10.1108/ITP-04-2021-0328/full/html
10. Peper, P., et al. (2021) Individual differences in working memory capacity predict benefits to
memory from intention offloading
https://www.tandfonline.com/doi/abs/10.1080/09658211.2021.1991380
11. Lewis, R., Liu, Y., Groh, M., & Picard, R. (2021). Shaping Habit Formation Insights with Shapley
Values: Towards an Explainable AI-system for Self-understanding and Health Behavior Change.
Proc. Realizing AI Healthcare, Challenges Appearing Wild CHI
https://francisconunes.me/RealizingAIinHealthcareWS/papers/Lewis2021.pd f
12. Rebar, A. L., Rhodes, R. E., & Verplanken, B. (2023). Habits and behavioral complexity–dynamic
and distinct constructs. Health Psychology Review , 17 (3), 485-489
10
Electronic copy available at: https://ssrn.com/abstract=4739430
