Conference PaperPDF Available

A Process Perspective on the Evaluation of Enterprise Social Software

Authors:

Abstract and Figures

Many organizations use enterprise social software (ESS) to support, for example, communication, knowledge and innovation management. Companies are increasingly realizing benefits and competitive advantages from using ESS. However , as with any other type of information systems (IS), assessing this value on an organizational level is an extensive challenge. Several models have been proposed to measure IS success. Many of these models disregard that IS success is not a static concept, but rather a temporary status which varies depending on its stage in the IS lifecycle. However, considering different success definitions at certain times in the lifecycle is crucial to produce significant results. In addition, IS theories, such as models for success measurement, are not one-to-one transferrable to ESS. The peculiarities of ESS have to be taken into account when it comes to develop valid measurement instruments. Consequently, the development of an approach with which to evaluate ESS, considering different stages of the ESS lifecycle on the basis of well-founded evidence still lies in the future. With our research we want to bridge this gap. Theoretically founded on the process theory of Soh and Markus (1995) and data gained from two qualitative studies our results are able to explain ESS success depending on the time within the ESS lifecycle. This process perspective assumes that ESS investments lead to organizational performance through a chain of three processes: (1) Within the conversion process ESS expenditures are transformed into usable ESS assets. (2) The potential of assets to change work procedures is exploited in the ESS use process. (3) In the competitive process the ESS impacts that materialize during the ESS use process are exploited and, eventually, transformed into improved organizational performance. Our results can help to gain a deeper understanding of ESS success and its meaning in every phase of the ESS lifecycle. Indicators are identified with which to evaluate an ESS in a concrete organizational setting. Furthermore, measurement barriers are examined so that they can be considered in the planning. This study also has implications for future research since we were able to show how the process theory can be applied in an ESS context. The indicators identified in the study might serve as measures to quantitatively test its validity.
Content may be subject to copyright.
A Process Perspective on the Evaluation of Enterprise Social Soft-
ware
Melanie Steinhueser1, Christian Herzog1, Alexander Richter2 and Uwe Hoppe1
1University of Osnabrueck, Osnabrueck, Germany
2University of Zurich, Zurich, Switzerland
melanie.steinhueser@uos.de
christian.herzog@uos.de
arichter@ifi.uzh.ch
uwe.hoppe@uos.de
Abstract: Many organizations use enterprise social software (ESS) to support, for example, communication, knowledge and
innovation management. Companies are increasingly realizing benefits and competitive advantages from using ESS. How-
ever, as with any other type of information systems (IS), assessing this value on an organizational level is an extensive chal-
lenge. Several models have been proposed to measure IS success. Many of these models disregard that IS success is not a
static concept, but rather a temporary status which varies depending on its stage in the IS lifecycle. However, considering
different success definitions at certain times in the lifecycle is crucial to produce significant results. In addition, IS theories,
such as models for success measurement, are not one-to-one transferrable to ESS. The peculiarities of ESS have to be taken
into account when it comes to develop valid measurement instruments. Consequently, the development of an approach
with which to evaluate ESS, considering different stages of the ESS lifecycle on the basis of well-founded evidence still lies in
the future. With our research we want to bridge this gap. Theoretically founded on the process theory of Soh and Markus
(1995) and data gained from two qualitative studies our results are able to explain ESS success depending on the time within
the ESS lifecycle. This process perspective assumes that ESS investments lead to organizational performance through a chain
of three processes: (1) Within the conversion process ESS expenditures are transformed into usable ESS assets. (2) The po-
tential of assets to change work procedures is exploited in the ESS use process. (3) In the competitive process the ESS impac ts
that materialize during the ESS use process are exploited and, eventually, transformed into improved organizational perfor-
mance. Our results can help to gain a deeper understanding of ESS success and its meaning in every phase of the ESS lifecycle.
Indicators are identified with which to evaluate an ESS in a concrete organizational setting. Furthermore, measurement bar-
riers are examined so that they can be considered in the planning. This study also has implications for future research since
we were able to show how the process theory can be applied in an ESS context. The indicators identified in the study might
serve as measures to quantitatively test its validity.
Keywords: enterprise social software, IS success measurement, system lifecycle, qualitative research, process theory
1. Introduction
More and more companies are using Enterprise Social Software (ESS) such as social networking sites, wikis or
weblogs in order to enhance internal communication and collaboration (Faraj et al., 2011). Although initial stud-
ies show that ESS usage can produce positive effects, such as increased productivity, efficient knowledge sharing
(Chui et al,. 2012) or greater innovation (Gray et al., 2011) it is currently being controversially debated whether
and how to evaluate these effects reliably at company level.
Measuring the success of information systems (IS) in general, is a major challenge for both researchers as well
as practitioners. A number of models have been developed that are able to explain IS success with different
perspectives to be considered, e.g., DeLone & Mclean (1992), Grover et al. (1996), Seddon et al. (1999). How-
ever, a broad gap exists between theory and practice, which is reflected by the models’ limited practicality
(Rosemann & Vessey, 2005). The common practice of companies aligning their success measurement to the
project or system lifecycle is not captured by most of the scientific models (Neumann et al., 2011). Pertaining to
the interests, the purpose and the design of a success measurement however, this is of great relevance (Hitt &
Brynjolfsson, 1996). Only very few models take this into account (Markus & Tanis, 2000). In a study that investi-
gated the practicality of different models (Keyes-Pearce, 2005), Soh and Markus’ process theory (Soh & Markus,
1995) was identified as one of the few approaches that enable a lifecycle-oriented success measurement and
therewith meets practical demands.
Although it is difficult to draw a clear line between ESS and other types of IS (Kane et al., 2014), there are some
characteristics which distinguish them from each other. The term ESS refers to web-based technologies that
support users’ contributions of persistent objects to a shared pool and that enable responses to these objects.
429
Melanie Steinhueser et al.
ESS comprises functionalities that visualize profile information and link users with one another (e.g., dis-
cover/subscribe/follow/friend). Potentially, these distinguishing characteristics require researchers to adapt es-
tablished theories, or possibly develop new ones (Majchrzak, 2009). Consequently, recent research has pro-
duced some approaches that address ESS success measurement, e.g. Raeth et al. (2012) or Herzog et al. (2013).
However, none of these approaches meets the practical requirement of being easily applicable by integrating
the system lifecycle.
This motivated us to develop an approach on the basis of well-founded evidence that enables companies to
measure ESS success according to their individual needs. For this purpose, it was necessary to identify success
indicators and, at the same time, to examine measurement barriers to be included in the planning. The process
theory of Soh and Markus (1995) serves us as theoretical basis and framework for our empirical research. Its
process-theoretical point of view allows meeting the practical requirements and at the same time represents a
probate theoretical basis. The constructs of this process theory are configured with relevant ESS indicators,
which were collected in two qualitative studies. Hence, our results can also serve as a starting point for a quan-
titative assessment of the process model’s relationships by testing the theory in the ESS field. Chapter 2 provides
an overview of the process theory. Then, in section 3 we explain our methodological approach leading to the
results that are presented in chapter 4 and discussed in chapter 5. A conclusion and an outlook on further re-
search conclude the article.
2. Soh and Markus‘ process theory
Soh and Markus’ (1995) process theory illustrates how IS success can be defined, and thus measured, depending
on the lifecycle stage a system is in. Their approach explains how investments in a system contribute to organi-
zational value creation. It assumes that IS expenditures contribute to organizational performance through a
chain of three processes, as shown in Figure 1. In contrast to variance theories (see e.g. Poole et al., 2000) the
process theory is characterized by necessary but not sufficient conditions that impact the organizational perfor-
mance. Other in- and external factors that are not (immediately) visible and therefore not (immediately) corri-
gible can decide about success or failure, at any time (Markus & Tanis, 2000).
Figure 1: Process theory of Soh und Markus (1995)
The first process model, the conversion process, describes how IS expenditures convert to IS assets when a
system is implemented and can be applied. These assets comprise resources, which are available to a company,
in order to develop or offer its products / services. They are formed from the system as well as the users’ capa-
bilities to take advantage of it. Investments in an IS represent a necessary but not a sufficient condition to gen-
erate assets.
The potential of IS assets to change work procedures is exploited in the subsequent use process. The impacts
depend on the nature and the extent of use, which is appropriate to a certain extent in a particular situation or
context. These impacts show up in organizational changes in terms of productivity, flexibility or changes in pro-
cesses. However, an increasing use does not necessarily mean that impacts also change (to the same direction
or extent). For this reason, the authors prefer the concept of appropriate use, which allows for different inter-
pretations. Existing assets are therefore necessary, but not sufficient to achieve impacts. Furthermore, an ap-
propriate system usage is also required.
In the competitive process, finally, the impacts affect organizational performance which is reflected by profit,
revenue or market share, for example. This phase is particularly affected by external factors: even if significant
impacts occur from using an IS, these are not sufficient for enhanced organizational performance. Rather, many
other in- and external factors, such as competitors’ behaviors, play a major role.
430
Melanie Steinhueser et al.
3. Methodology
Soh and Markus (1995) do not restrict their process theory to any particular IS. We, therefore, assume that the
cause and effect relations on the level of expenditures, assets, impacts and organizational performance are also
true for ESS. The model serves as theoretical foundation and helps us identifying and structuring indicators and
barriers of ESS success measurement. Considering that success is not a fix construct but rather depends on the
ESS’ stage of implementation.
The data collection took place between 2009 and 2014. Two studies, which aimed at exploring and analyzing ESS
success, provided us with relevant data sets. In both studies we adopted a qualitative approach (Walsham,
2006). In the first study, a large interview study, we asked companies which metrics and methods they apply for
ESS success measurement. We furthermore asked for difficulties that may appear in this process. Results of this
study offer valuable implications for all phases of the process theory. A deep understanding of ESS success in
the specific business context could be obtained in the second study, where data were collected in five case
studies. The focus was on the analysis of success factors and impacts, including the identification of related
measures and dimensions. With regard to the process theory the second data set particularly provides infor-
mation on indicators of the competitive process. Table 1 provides an overview of both data sets.
Table 1: Overview of data sets
Period
Method
Data
Focus
Data
set 1
12/2011 -
04/2014
Qualitative cross-
sectional analysis
31 semi-structured interviews with
ESS experts in 29 companies
Methods, metrics and barri-
ers
of ESS success measure-
ment
Data
set 2
04/2009 -
01/2010
Case study re-
search
5 case studies (with 10 interviews,
about 60 documents and w
ebsites,
usage statistics)
Dimensions and measures of
ESS success
In both studies, we used a semi-structured interview guide to support the conversation with the interviewees
(Bryman & Bell, 2007). The interview guide contained questions in different categories including questions about
the person and the company, about the experience with ESS, about the understanding of ESS success and also
about measuring ESS success. This enabled us to get an idea of the participant’s experience and simultaneously
to detect different contexts of statements. The interview guide allowed for a meaningful comparison of the
interviews and at the same time it left sufficient room for comprehensive statements and additional questions
from the interviewer (Bryman & Bell, 2007). We recorded and transcribed the interviews and subsequently
coded the text documents. We analyzed the two data sets by identifying indicators that were relevant for our
purpose and allocated them to the phases, accordingly. This was done by content analysis (Mayring, 2000). The
citations isolated from the interviews either contained
temporal information (e.g., "before installing ..."),
mention of activities (e.g. "requirements analysis") or
mention of measurement objects (e.g. "license fees").
This helped us to logically match interview statements with each phase’s contents. E.g., hardware costs can
evidently be assigned to expenditures. We discussed the results in the research group. Uncertainties regarding
interpretation and classification were discussed and resolved. The barriers have been identified and assigned in
the same way.
Our results are described in the following chapter where indicators and possible barriers for each lifecycle phase
are first presented in a table and then described. The indicators are represented by variables to measure ex-
penditures, assets, impacts and organizational performance. The barriers help identifying potential difficulties
that may occur at different times. This primarily addresses the evaluation’s practicality and allows for early con-
sideration and inclusion of possible barriers in the planning.
4. Success measurement in the ESS lifecycle: Presentation of results
The analysis of the data confirms that companies actually measure ESS success at different times. Various prob-
lems arise in the course of the lifecycle which can limit the evaluation or even prevent it. Our study reveals that
companies are particularly focused on the use process, which is due to different reasons. Firstly, ESS is charac-
431
Melanie Steinhueser et al.
terized by a multitude of user-generated content which serves different communicative and collaborative pur-
poses. This means that the evaluation of the use and its purpose is gaining importance. Secondly, the collection
of usage data is relatively easy and inexpensive. Compared to the conversion or the competitive process, meas-
urement barriers can be overcome easily in the use process. While in the initial phase in the conversion process,
especially organizational barriers may hinder the ESS success measurement, difficulties of the key figures or
target definitions as well as a lack of methodological knowledge for collecting and analyzing the data lead to
problems in the competitive process. In the following, the results of the studies are described and discussed in
the individual phases of the ESS lifecycle.
4.1 The ESS conversion process - from expenditures to assets
The ESS conversion process includes activities for project initiation and implementation where the expenditures
are converted into assets. This conversion is seen as a fundamental step and a prerequisite for the use of the
ESS which in turn is necessary to influence the impacts and finally the organizational performance positively.
Subject of the measurements in this process are therefore costs and potential benefits of a future ESS system in
a defined quality, which is determined by defined requirements. These requirements are necessary for the op-
timal system design, although not sufficient to lead to ESS assets.
Table 2: Results for the measurement of ESS expenditures
Indicators
Barriers
hardware and software costs; personnel expenses; costs
per user; benefit estimations; ESS requirements
no persons responsible; no official project, no compara-
tive data; invalid assumptions
Our interviews reveal that ESS expenditures are calculated ex ante based on the collection and assessment of
hardware and software costs as well as the work costs of the (project) employees. The expenditures are related
to the system design requirements. Furthermore, potential benefits are estimated. Objective of the measure-
ment at this time is the legitimacy and procurement of the project budget by a sound economic analysis, e.g. in
the form of a business case, a cost-benefit analysis or a present value calculation. The stakeholders of the success
measurement are hereby the person in charge of the costs, the management or other decision makers. The
result could be a decision for or against a (specific) ESS system. If the decision is made for the ESS introduction,
the next steps are the design and implementation of the project.
Our data shows that an initial evaluation is often skipped because in many cases the ESS is introduced bottom-
up and there is no official responsible instance. But even if the responsibility was clearly defined, it would not
be easy to estimate potential costs and benefits carefully. By a lack of comparative data, the improvement can-
not be easily identified at the appropriate current state. Since at this stage still no data, such as costs incurred
and by difficulties in estimating the effects precisely, only (invalid) assumptions can be made.
Table 3: Results for the measurement of ESS assets
ESS assets
Indicators
Barriers
user satisfaction; usability; applicability; integration; in-
formation quality; affinity; user know-how and skills;
functionality; target/actual performance comparison
survey limitation (language, length); lack of methodologi-
cal know-
how; difficulties to find the right people for in-
terviews; limited decision
-
making power of the platform
owner; regulations by the works council; complexity of
the system analysis
ESS assets are generated when the budget is approved and converted into a running system which can be ade-
quately used by the employees. It is necessary to examine the system requirements and to promote the im-
provement of the ESS. In our interviews it would appear that the users’ experiences with existing systems play
an important role as well as their expectations. Indicators are usability, applicability but also the skills of em-
ployees who use the ESS and their affinity. The addressee of an evaluation is the platform owner or project
manager, who aims to develop the ESS. Difficulties that may occur in this phase are primarily due to the data
collection itself. Barriers would include the difficulty to identify and gain the right people for interviews, the lack
of knowledge of methods or the limitations of interviews regarding the language or the length. Regulation by
the works council as well as lack of freedom of the project or platform managers can cause delays in the perfor-
mance measurement and increase their effort.
432
Melanie Steinhueser et al.
4.2 The ESS use process – from assets to impacts
The use process enables ESS assets to produce ESS impacts. The successful use of an ESS is to be interpreted as
positive confirmation of the previous conversion process and therefore must be ensured.
Table 4: Results of measuring the ESS use
ESS use
Indicators
Barriers
usage statistics; usage statistics in relation to other
systems (e.g. e-mail); degree of crosslinking; use
types; use cases; use behaviour; text length; number
of hyperlinks, images, contact requests, direct mes-
sages, comments; sentiments
technical limitations of data collection and analysis; bound-
aries of surveys (language, length); lack of significance of
usage statistics; privacy policy; regulations by works coun-
cil; protection of personal data; complexity of content
analysis
Due to the huge amount of user generated content, this phase is of particular importance in this context. This is
reflected in the frequent use of usage analysis of the first study’s participants (23 of 29 companies from Study 1
collect usage statistics). The evaluation goal in this phase is to obtain an understanding of the nature and inten-
sity of the ESS use. The indicators are hereby usage statistics that can be collected through log-file analysis. In
addition, use types and sentiments can be obtained by content analysis or the degree of crosslinking of employ-
ees may be charged by network analysis. In addition, the interviewees used surveys and interviews to evaluate
use behavior.
The barriers which are identified in this phase include privacy regulations or regulations by the works council
that are necessary though, but increase the measurement costs (e.g. the fact that personal data must be anon-
ymized or agreements with the works council are time consuming). Normally, the collection of usage statistics
can be implemented inexpensively and partly by the statistic tools of the ESS. However, some platform owners
encounter limitations in the technical possibilities to collect comprehensive and detailed data. Another challenge
is the interpretation of usage data, as usage statistics alone have only a limited significance for success. In a
second step, an assessment of the usage data is necessary to bring them into context (e.g. the page impressions
have increased by x% with the same amount of users, which means that the information range has increased).
The appropriate use of the system provides a positive transition into the next phase, the competitive process.
4.3 The ESS competitive process - from impacts to organizational performance
In the competitive process, after an appropriate use an effect (impact) on the employees and the company arises
that eventually pays off in organizational performance. It was shown that errors in the initial phases of the lifecy-
cle have a negative effect in this phase, inter alia by a negative cost-benefit ratio or a negative ROI. The objective
of the evaluation is thus on the one hand a gain in knowledge about the impact and subsequent effects on
organizational performance, and on the other hand a control of the achieved results.
Table 5: Results for the measurement of ESS Impacts
ESS impacts
Indicators
Barriers
created ideas; workload for tools; intensity of coopera-
tion; employee satisfaction with the ESS;
validation of
use cases; time spend for reading, writing and re-
sponses; learning; awareness; effectiveness of the deci-
sions; productivity; serendipity; fun; networking; em-
ployee engagement
difficulty in identifying and interpreting effects; no ac-
ceptable benefit / cost ratio; lack of capacit
y; boundaries
of surveys (language, length); difficulty to attract the right
people for interviews; lack of knowledge of methods; com-
plexity of content analysis
In our studies ESS impacts are determined by qualitative and quantitative indicators, such as the number of
created ideas, employee satisfaction with the ESS and the resulting accessibility, the intensity of cooperation,
but also the workload arising from working with the tools. Furthermore, various use cases, which are mapped
by ESS, can be verified and validated. A main evaluation barrier in this phase is the difficulty of identifying and
interpreting effects which is exacerbated by a lack of methodological knowledge and limitations of employee
surveys. This can be attributed to the ESS characteristic of malleability which means that ESS often not serve
predefined purposes. To obtain meaningful results, a high effort has to be made for which often no budget or
timeframe is available. Furthermore the efforts can be in a negative ratio to the potenti al benefits of a survey,
whereby the evaluation is not or only minimally implemented. The analysis of the effect delivers knowledge to
assess the subsequent organizational performance.
433
Melanie Steinhueser et al.
Table 6: Results for the measurement of organizational performance
Organizational performance
Indicators
Barriers
number of implemented ideas; cost per user; cost-ben-
efit ratio; cost savings (travel expenses, IT costs, costs
for services); ROI; additional revenue; number of newly
acquired customers; perceived performance; ROI of ap-
plications; opportunity revenues of projects
no objectives; inaccurate (not measurable) objectives;
complexity; difficulty of defining indicators; invalid as-
sumptions; unacceptable benefit-
cost ratio; lack of capac-
ity; missing comparative data;
difficulty in quantifying ef-
fects; difficulty to identify and allocate costs and revenues;
lack of knowledge of methods
To verify a change in organizational performance and thus to justify the project ex ante to the management, it
was shown that there are a number of quantitative monetary as well as non-monetary indicators available which
flow directly into the objectives of increasing market share, profits or revenues. Examples include travel cost
savings through the use of ESS, costs savings in comparison to other (former) systems, the number of imple-
mented ideas, additional revenue, the ROI or opportunity revenues (revenue which I would lose when I had not
used the ESS). The assignment of monetary effects or the calculation of ROI is often very difficult (sometimes
impossible) for the platform owners. Whereas infrastructure costs can still be calculated relatively easily, an
allocation of revenues is difficult to carry out. Furthermore, missing or inaccurate objectives of the ESS is a bar-
rier that hinders the evaluation. The malleability complicates the definition of key figures and makes the exact
objective essential for checking the system's success. The cost and complexity of the evaluation in the competi-
tive process are much higher compared with the conversion and use process. Hence, an accurate planning in the
first phase is necessary to ensure a smooth and targeted evaluation.
5. Discussion
The identified indicators enable us to explain ESS success depending on the lifecycle stage. The process theory´s
underlying assumption, however, should always be taken into consideration explicitly: The results of each phase
are not only dependent from the ESS-specific indicators but also from other internal and external factors which
are partly unpredictable. The ESS success measurement does not allow for conclusions about direct correlations
in the following phases, but explains the situation at a given time.
As practice shows, IT projects often fail at an early stage. Although large financial budgets are available, compa-
nies are not able to turn their investments into a well-operating system. On the other hand, many examples
show that no- or low-budget ESS projects that are carried out in a bottom-up manner can lead to the successful
creation of ESS assets. It is therefore important to note that, in the conversion process, not only ESS expenditures
but also the management’s commitment and experience as well as structural factors (e.g. company size, indus-
try) influence the creation of ESS assets (Weill, 1992). As a large part of an ESS’ content typically grows and
improves during the use process, it has furthermore to be considered that ESS assets may change over the whole
lifecycle (Raeth et al., 2012) and therefore, should be re-evaluated at a later time.
Similarly, in the competitive process, the ESS impacts are exploited and, eventually, transformed into improved
organizational performance and may contribute to competitiveness (Markus & Robey, 1988). However, espe-
cially in this phase, a number of other factors than ESS impacts affect the performance of an organization. These
are not only different internal resources (Wernerfelt 1984) but also the ability to dynamically reconfigure these
resources as needed (Teece et al., 1997) and thereby taking into account external forces (Porter, 2008).
The use process with the corresponding collection of usage statistics is of particular relevance when it comes to
measure ESS success. The large amount of user-generated content and the possibility to simply collect the data
at low cost lead to a special interest in the creation of usage statistics.
Appropriately using an ESS is a necessary condition to generate positive ESS impacts. However, the term “ap-
propriate use” leaves room for interpretation. Not only the extent of use (e.g. how many users must have read
a blog post so that it is "successful"?) is relevant but also the quality of use (e.g. how does a blog post affect the
ESS impacts?). Studying the use construct as an interface between humans and computers has been much-de-
bated in recent years (Burton-Jones & Straub, 2006; Petter et al., 2008) with controversial opinions (Straub &
Guidice, 2012). It is therefore essential, especially in in the case of ESS, not only to collect use statistics, but
rather to define in advance what is meant by an appropriate use depending on the particular context.
434
Melanie Steinhueser et al.
6. Conclusions
This paper presents indicators and barriers of ESS success measurement taking into account different meanings
of success depending on the phase of implementation. We admit that ESS success is not a static concept, but
rather a temporary status which varies depending on the lifecycle stage. Considering different success defini-
tions at certain times in the lifecycle is crucial to produce significant results. Various methods such as surveys,
interviews, requirements and log file analysis, financial calculations or process and use observations are suitable
for the measurement of the identified indicators in the specific business context.
Our results provide practical orientation and guidance on ESS success measurement in different phases of the
lifecycle. In addition, they support practitioners in planning their success measurement by creating awareness
for the barriers that may occur.
Besides the practical relevance our results have also implications for research. We have shown how Soh and
Markus’ process theory can be applied in an ESS context. To what extent the results can be transferred to other
types of IS and what adjustments need to be made has not been addressed in this work . The identification of
the indicators is a prerequisite to quantitatively test the validity of the process theory in the ESS context. Our
results are therefore an important basis for future research that examines the causal relationships between ESS
investment, ESS assets, and ESS impacts toward organizational performance.
References
Bryman, A. and Bell, E. (2007) Business Research Methods, Oxford University Press, New York.
Burton-Jones, A. and Straub, D. W. J. (2006) Reconceptualizing system usage: An approach and empirical test. Information
Systems Research, Vol. 17, No. 3, pp 228-246
Chui, M., Manyika, J., Bughin, J., Dobbs, R., Roxburgh, C., Sarrazin, H., Sands, G. and Westergren, M. (2012) The Social Econ-
omy: Unlocking Value and Productivity through Social Technologies, McKinsey Global Institute.
DeLone, W. and Mclean, E.R. (1992) Information Systems Success: The Quest for the Dependent Variable, Information Sys-
tems Research, Vol. 3, No. 1, pp 60–95.
Faraj, S., Jarvenpaa, S.L. and Majchrzak, A. (2011) Knowledge Collaboration in Online Communities, Organization Science,
Vol. 25, No. 5, pp 1224–1239.
Gray, P.H., Parise, S. and Iyer, B. (2011) Innovation Impacts of Using Social Bookmarking Systems, MIS Quarterly, Vol. 35,
No. 3, pp 629–644.
Grover, V., Jeong, S. and Segars, A.H. (1996) Information Systems Effectiveness: The Construct Space and Patters of Appli-
cation, Information & Management, Vol. 31, No. 4, pp 177–191.
Herzog, C., Richter, A., Steinhueser, M., Hoppe, U. and Koch, M. (2013) Methods and Metrics for Measuring the Success of
Enterprise Social Software - What we can Learn from Practice and Vice Versa, ECIS Proceedings, Utrecht, The Nether-
lands.
Hitt, L.M. and Brynjolfsson, E. (1996) Productivity, Business Profitability, and Consumer Surplus: Three Different Measures
of Information Technology Value, MIS Quarterly, Vol. 20, No. 2, p 121.
Kaganer, E. and Vaast, E. (2010) Responding to the (almost) Unknown: Social Representations and Corporate Policies of
Social Media, ICIS 2010 Proceedings.
Kane, G.C., Alavi, M., Labianca, G. J. and Borgatti, S. (2014) What’s Different about Social Media Networks? A Framework
and Research Agenda, MIS Quarterly, Vol. 38, No. 1, pp 275–304.
Keyes-Pearce, S. (2005) IT Value Management in Leading Firms: The Fit Between Theory and Practice, University of Sydney,
Australia.
Majchrzak, A. (2009) Comment: Where is the Theory in Wikis? MIS Quarterly, Vol. 33, No. 1, pp 18–20.
Markus, M.L. and Robey, D. (1988) Information Technology and Organizational Change: Causal Structure in Theory and Re-
search, Management Science, Vol. 34, No. 5, pp 583 – 598.
Markus, M.L. and Tanis, C. (2000) The Enterprise System Experience - from Adoption to Success, in R. W. Zmud, ed. Fram-
ing the Domains of IT Research: Glimpsing the Future through the Past. Cincinnati, OH: Pinnaflex Educational Re-
sources, pp 173–207.
Mayring, P. (2000) Qualitative Content Analysis, Forum Quality Social Research, Vol. 2, No. 1.
Neumann, M., Sprenger, J., Gemlik, A. and Breitner, M. H. (2011) Untersuchung der praktischen Anwendbarkeit des IS-
Erfolgsmodells von DeLone und McLean, Wirtschaftsinfomatik Proceedings, Zurich, Swizerland.
Petter, S., DeLone, W. and McLean, E. (2008) Measuring Information Systems Success: Models, Dimensions, Measures, and
Interrelationships, European Journal of Information Systems, Vol. 17, No. 3), pp 236–263.
Poole, C.F., Gunatilleka, A.D. and Sethuraman, R. (2000) Contributions of Theory to Method Development in Solid-Phase
Extraction, Journal of Chromatography A, Vol. 885, No. 1-2, pp 17–39.
Porter, M. (2008) The Five Competitive Forces that Shape Strategy. Harvard Business Review.
Raeth, P., Urbach, N., Smolnik, S. and Butler, B. (2012) Corporate Adoption of Social Computing: A Process-Based Analysis,
Journal of Information Technology Case & Application Research, Vol. 14, No. 2, pp 3–27.
435
Melanie Steinhueser et al.
Raeth, P., Kügler, M. and Smolnik, S. (2011) Measuring the Impact of Organizational Social Website Usage on Work Perfor-
mance: A Multi-level Model, Proceedings of the 32nd International Conference on Information Systems.
Rosemann, M. and Vessey, I. (2005) Linking Theory and Practice: Performing a Reality Check on a Model of IS Success, Pro-
ceedings of the 13th European Conference on Information Systems.
Seddon, P. B., Staples, S., Patnayakuni, R. and Bowtell, M. (1999) Dimensions of information systems success, Communica-
tions of the AIS, Vol. 2, No. 1.
Soh, C. and Markus, M.L. (1995) How IT Creates Business Value: A Process Theory Synthesis, Proceedings of the Sixteenth
International Conference on Information Systems, Amsterdam, The Netherlands.
Straub, D. W. and del Guidice, M. (2012) Editor's Comments: Use. MIS Quarterly, Vol. 36, No. 4, pp iii-viii.
Teece, D.J., Pisano, G. and Shuen, A. (1997) Dynamic Capabilities and Strategic Management, Strategic Management Jour-
nal.
Walsham, G., (2006) Doing Interpretive Research, European Journal of Information Systems, Vol. 15, No. 3, pp.320–330.
Weill, P. (1992) The Relationship Between Investment in Information Technology and Firm Performance: A Study of the
Valve Manufacturing Sector, Information Systems Research, Vol. 3, No. 4, pp 307–333.
Wernerfelt, B. (1984) A resource-based view of the firm, Strategic Management Journal, Vol. 5, No.2, pp 171-180.
436
... The key theme with the largest number of publications is the (1) measurement of system usage. General usage figures, such as the number of users, logins, events or objects are frequently discussed (Shami, Muller and Millen, 2011;Herzog et al., 2013;Herzog, Richter and Steinhueser, 2015;Steinhueser, Herzog, Richter and Hoppe, 2015). Such metrics provide a high-level overview on platform usage. ...
... Log file analysis requires carefully designed metrics, which provide new information and insights on system usage. It is argued, however, that there is a lack of meaningful metrics (Otjacques et al., 2006;Meske et al., 2014;Steinhueser et al., 2015). The interpretation of these metrics is a further and directly related issue (Otjacques et al., 2006;Windhager, Zenk and Federico, 2011;Meske et al., 2014). ...
... However, in some cases policies forbid to analyse activities of users by these external parties (Friedman, Burns and Cao, 2014). Several authors argue that privacy laws and privacy policies constrain or even forbid system log file analyses because of a possible violation of the employees' privacy (John and Seligmann, 2006;Lin et al., 2012;Muller, 2012;Cao, Gao, Li and Friedman, 2013;Herzog et al., 2013;Riemer, Finke and Hovorka, 2015;Steinhueser et al., 2015). The authors consistently argue that privacy laws are one of the biggest obsta-6 cles for conducting system measurements. ...
Conference Paper
Full-text available
This paper aims to identify and document the status quo and the perceived barriers of Social Collaboration Analytics in practice. The study is part of a publicly funded longitudinal research project on Enterprise Collaboration Systems (ECS) involving 29 early-adopter user companies. Our paper draws on and extends findings from previous research. The key themes identified from a structured literature review and the documented findings from previous workshops with practitioners were used to develop an online questionnaire. Longitudinal case studies on the same companies helped to inform and interpret the findings. The survey shows that most of the participants have implemented some form of Collaboration Analytics, however, the outputs are mostly on a high abstraction level and the methods quite simplistic. Complex analyses that could help in assessing the degree of cooperation, structures or dynamics in the use of ECS are not (yet) applied. We identified three important reasons for this situation: (1) The analytics tools provided by standard collaboration software do not provide sufficient functionality, (2) there is a lack of awareness and knowledge about more complex forms of analyses and (3) some concepts from the academic literature are not perceived as relevant by practitioners. The survey also confirmed that there are financial, regulatory, data and HR barriers for Social Collaboration Analytics.
... number of blog posts, and unique visitors), which can be used to measure ESS success. Following a similar idea, Steinhueser et al. [13] propose measuring ESS use by considering usage statistics, the degree of crosslinking, the number of hyperlinks, direct messages and comments. ...
... In the more traditional research stream, analytics are conducted for deriving design implications [11,16] (improving or changing the system). In recent years, a new research stream concerned with evaluation and success measurement [5,6,13] has emerged, with the objective of providing decision makers with information on platform usage and success. ...
... Wu et al. [38] analyse the closeness of relationships between employees based on interactions in ESS by computing behavioural metrics for a subject's behavioural factors, friends' behavioural factors, subject-friend interaction factors, mutual connection factors and company direction factors. Steinhueser et al. [13] and Hacker et al. [17] suggest analysing the connectedness of users as an indicator of networking behaviour and general platform usage. While the before mentioned researchers focus on analysing user-centric networks, Nasirifard et al. [39] suggest analysing document-centric networks. ...
Conference Paper
Full-text available
Over the last years, research on benefits and success measurement of Enterprise Social Software (ESS) has gained momentum. Literature reviews in this topic area discuss methods for measuring ESS success and benefits and demonstrate that Social Collaboration Analytics is gaining increasing importance in the context of benefits research for ESS. This paper provides an overview of the literature on Social Collaboration Analytics for ESS. The selected literature can be categorised in seven key themes: measuring system usage, identification of usage patterns, identification of types of users, analysis of groups, identification of expertise, network analysis and organisational impacts. The paper concludes with a discussion of the importance of Social Collaboration Analytics for other relevant research streams.
... In the first iteration, application areas for SCA were identified. Most of the analyzed studies used SCA for providing an overview of ECS use by using general metrics such as the number of users or activities in a system [25,26]. The analysis of the studies showed that social network analysis is often applied for analyzing networks and relations between users in an ECS [5]. ...
Article
Full-text available
Enterprise collaboration systems (ECS) have become the core of the digital workplace in many organizations. More and more companies have introduced this new business software for supporting computer-mediated collaboration among employees. The emergence of ECS in the technical and strategical landscape of companies leads to new challenges in business intelligence and reporting. In contrast to traditional business software such as ERP systems, there are no established methods for decision-making in ECS projects. The collaboration professionals responsible for managing the ECS require insights into the use of an ECS to understand how employees use these systems. By developing the Social Collaboration Analytics Framework, this work provides a framework for establishing the analysis of collaboration activities in the digital workplace (social collaboration analytics) as part of organizations' business intelligence. The Social Collaboration Analytics Framework consists of distinct phases and guides the analysis of ECS data. It includes working steps, recommendations, and guiding questions developed based on the findings from longitudinal research in a university-industry collaboration context and a comparison of established data mining process models.
... Classical approaches are well tested methods for measuring JS, technology acceptance, IS quality, usability and user experience with a solid background. They are the only ones that can be used during the initial stages of the development of an IS project [28]. However, their application in practice is not easy. ...
Article
Full-text available
Evaluating how an Information Systems (IS) intervention in the workplace affects daily work and impacts on workers and organizations is a challenge that requires a very broad research approach. IS researchers have derived several models to explain and measure IS success, taking various perspectives and system types into account. This study presents an evaluation framework for measuring the impacts of an IS intervention especially at the shop floor in production environments. In this framework, we take a broad scope of examination and apply an integrated model that comprises elements from several methods for analyzing the acceptance and the impact of the new solutions. Thus, the aim is to further develop and enhance the existing methods and models for measuring the acceptance and the impacts of the sociotechnical interventions in production environments.
... Classical approaches are well tested methods for measuring JS, technology acceptance, IS quality, usability and user experience with a solid background. They are the only ones that can be used during the initial stages of the development of an IS project[28]. However, their application in practice is not easy. ...
Conference Paper
Full-text available
Evaluating how an Information Systems (IS) intervention in the workplace affects daily work and impacts on workers and organizations is a challenge that requires a very broad research approach. IS researchers have derived several models to explain and measure IS success, taking various perspectives and system types into account. This study presents an evaluation framework for measuring the impacts of an IS intervention especially at the shop floor in production environments. In this framework, we take a broad scope of examination and apply an integrated model that comprises elements from several methods for analyzing the acceptance and the impact of the new solutions. Thus, the aim is to further develop and enhance the existing methods and models for measuring the acceptance and the impacts of the sociotechnical interventions in production environments.
... Steinhueser et al [14] propose a set of indicators and barriers to measure ESS expenditures, assets, use and organizational performance impacts. A McKinsey study has presented a matrix highlighting the increase in the value added for different types of Enterprise 2.0 technologies for different levels of penetration of the technologies in the organization. ...
Conference Paper
In a globalized world, where companies operate across different locations and work becomes increasingly complex, collaboration in a diversity of ways is required among employees to perform tasks more effectively. Following a case study methodology that involved six interviews across three different country locations, this research addresses the phenomenon of Enterprise Social Networks (ESN) in a multinational corporation with a focus on the assessment of ESN success.
Chapter
Aufgrund des großen Erfolges von sozialen Medien im privaten Umfeld beschäftigen sich auch Organisationen zunehmend mit dem Potenzial sogenannter Enterprise Social Software (ESS). Im Organisationskontext kann ESS dazu beitragen, dass die Kommunikation transparenter und schneller, die Zusammenarbeit effizienter und die Unternehmenskultur partizipativer wird. Gleichzeitig bringt ESS Herausforderungen mit sich, die ihrem nutzbringenden Einsatz entgegenstehen können. Diese Effekte sind bedingt durch die besonderen Eigenschaften von ESS, mit denen sich Mitarbeiter*innen auf allen Unternehmensebenen und Funktionen konfrontiert sehen. Der Beitrag fasst verschiedene Studien zum Einsatz von ESS in Organisationen zusammen und erläutert aus der Perspektive der Wirtschaftsinformatik insbesondere Randbedingungen für die Einführung sowie Ansätze zur Erfolgsmessung.
Chapter
In diesem Kapitel zeigen wir praxiserprobte Lösungen für typische Kompetenzprobleme. Lesen Sie, wie zum Beispiel die Entwicklung einer Kompetenzstrategie, die Identifikation und Vernetzung wichtiger Experten, die kompetenzbasierte Besetzung von Teams, die Weitergabe von Kompetenzen über Generationen, die kompetenzbasierte Vergütung oder auch die individuelle Neuorientierung mit den geeigneten Instrumenten des Kompetenzmanagements effektiv bewältigt werden können.
Chapter
Aufgrund des großen Erfolges von sozialen Medien im privaten Umfeld beschäftigen sich auch Organisationen zunehmend mit dem Potenzial der Enterprise Social Software (ESS). Im Organisationskontext können Werkzeuge wie Weblogs und Wikis dazu beitragen, dass die Kommunikation transparenter und schneller, die Zusammenarbeit effizienter und die Unternehmenskultur partizipativer wird. Gleichzeitig bringt die ESS Herausforderungen mit sich, die ihrem nutzbringenden Einsatz entgegenstehen können. Diese Effekte sind bedingt durch die besonderen Eigenschaften von ESS, mit denen sich Mitarbeiter auf allen Unternehmensebenen und Funktionen konfrontiert sehen. Der Beitrag fasst verschiedene Studien zum Einsatz der ESS in Organisationen zusammen und erläutert aus der Perspektive der Wirtschaftsinformatik insbesondere Randbedingungen für die Einführung sowie Ansätze zur Erfolgsmessung.
Article
Full-text available
Digital natives — the generation for whom the Internet has always existed — have embraced the medium as one of choice. They use social computing applications as a medium for many activities, including receiving and giving product advice (ratings, comments), meeting with and talking to friends (social networking, chat), organizing events (social networking), learning (Wikipedia, weblogs), and communication with the general public (Youtube, weblogs). Facebook, a social networking service, has reached an audience of approximately 800 million users; young Iranians organize protests against their government via social networks, while amateur journalists and artists are viewed by millions on Youtube and weblogs. Inspired by these developments, corporations now seek to adopt social computing applications and derive similar benefits for their organizations. However, despite their growing interest, many firms report significant problems with the implementation and acceptance of social computing applications. We describe how three companies solved significant communication and collaboration issues by incorporating social computing applications into their intranets. Particular attention is paid to the deployment and adoption processes. Based on these case analyses, a common understanding of corporate social computing adoption is outlined and recommendations are derived.
Article
Full-text available
Recent years have witnessed the rapid proliferation and widespread adoption of a new class of information technologies, commonly known as social media. Researchers often rely on social network analysis (SNA) in attempting to understand these technologies, often without considering how the novel capabilities of social media platforms might affect the underlying theories of SNA, which were developed primarily through studies of offline social networks. This article outlines several key differences between traditional offline social networks and online social media networks by juxtaposing an established typology of social network research with a well-regarded definition of social media platforms that articulates four key features. The results show that at four major points of intersection, social media has considerable theoretical implications for SNA. In exploring these points of intersection, this study outlines a series of theoretically distinctive research questions for SNA in social media contexts. These points of intersection offer considerable opportunities for researchers to investigate the theoretical implications introduced by social media and lay the groundwork for a robust social media agenda potentially spanning multiple disciplines.
Article
Full-text available
In "The Shoemaker's Children: Using Wikis for Information Systems Teaching, Research, and Publication," Kane and Fichman argue that wikis will enable the IS discipline. The Wikipedia database will enable IS research, the transparency of wikis will enable peer-reviews of manuscript submissions to our IS journals, and the tracking and open contributions on wikis will enable IS teaching. I completely agree with an underlying assumption of the paper: for the IS field to advance, IS researchers should try innovative technologies and practices. Trying innovative technologies not only provides closer connections to the students, but often stimulates ideas for research as well as an appreciation of the challenges faced by users in working with new technologies. While the authors focus on wikis, I would argue that this assumption should apply to all Web 2.0 tech-nologies. Web 2.0 consists of much more than wikis: social networking tools, mashups, blackberries, MMORPGS (mas-sively multi-player online role-playing games), online predic-tion markets, tagging, folksonomies, jam sessions, and virtual worlds. While I am quite supportive of the notion of a Wiki-pedia project on IS, I would suggest that IS researchers not be constrained to Wikipedia for creating publicly accessible IS projects. IS researchers should consider starting a topic on the Programmable Web for research on mashups; create a Facebook profile for IS academics or IS research topics; grow a Linked-In network that connects IS academics to IS professionals; contribute to the IS island on Second Life; hold electronic jam sessions instead of (or in addition to) ICIS; create a mashup hub on IS World where mashups can be posted and statistics on their usage can be tracked; and host online prediction markets where the IS academic community votes on important future options that might affect our lives and research, such as salaries, class enrollments, where our students will be placed next year, new technologies that might have an impact on our theories, new theories that might have an impact on technologies, etc. Apparent from these com-ments is that Kane and Fichman's suggestions that we, as IS scholars, should try new technologies will clearly be helpful to the IS community, especially if the IS community expands upon their suggestions to consider the vast array of Web 2.0 technologies beyond wikis.
Article
Full-text available
Online communities (OCs) are a virtual organizational form in which knowledge collaboration can occur in unparalleled scale and scope, in ways not heretofore theorized. For example, collaboration can occur among people not known to each other, who share different interests and without dialogue. An exploration of this organizational form can fundamentally change how we theorize about knowledge collaboration among members of organizations. We argue that a fundamental characteristic of OCs that affords collaboration is their fluidity. This fluidity engenders a dynamic flow of resources in and out of the community— resources such as passion, time, identity, social disembodiment of ideas, socially ambiguous identities, and temporary convergence. With each resource comes both a negative and positive consequence, creating a tension that fluctuates with changes in the resource. We argue that the fluctuations in tensions can provide an opportunity for knowledge collaboration when the community responds to these tensions in ways that encourage interactions to be generative rather than constrained. After offering numerous examples of such generative responses, we suggest that this form of theorizing— induced by online communities— has implications for theorizing about the more general case of knowledge collaboration in organizations.
Article
Full-text available
Recent years have witnessed the rapid proliferation and widespread adoption of a new class of information technologies, commonly known as social media. Researchers often rely on social network analysis (SNA) in attempting to understand these technologies, often without considering how the novel capabilities of social media platforms might affect the underlying theories of SNA, which were developed primarily through studies of offline social networks. This article outlines several key differences between traditional offline social networks and online social media networks by juxtaposing an established typology of social network research with a well-regarded definition of social media platforms that articulates four key features. The results show that at four major points of intersection, social media has considerable theoretical implications for SNA. In exploring these points of intersection, this study outlines a series of theoretically distinctive research questions for SNA in social media contexts. These points of intersection offer considerable opportunities for researchers to investigate the theoretical implications introduced by social media and lay the groundwork for a robust social media agenda potentially spanning multiple disciplines.
Chapter
The dynamic capabilities framework analyzes the sources and methods of wealth creation and capture by private enterprise firms operating in environments of rapid technological change. The competitive advantage of firms is seen as resting on distinctive processes (ways of coordinating and combining), shaped by the firm's (specific) asset positions (such as the firm's portfolio oj difficult-to-trade knowledge assets and complementary assets), and the evolution path(s) it has afIopted or inherited. The importance of path dependencies is amplified where conditions oj increasing returns exist. Whether and how a firm's competitive advantage is eroded depends on the stability oj market demand, and the ease of replicability (expanding internally) and imitatability (replication by competitors). IJ correct, the framework suggests that private wealth creation in regimes of rapid technological change depends in large measure on honing internal technological, organizational, and managerial processes inside the firm. In short, identifying new opportunities and organizing effectively and efficiently to embrace them are generally more fundamental to private wealth creation than is strategizing, if by strategizing one means engaging in business conduct that keeps competitors off balance, raises rival's costs, and excludes new entrants.
Article
The dynamic capabilities framework analyzes the sources and methods of wealth creation and capture by private enterprise firms operating in environments of rapid technological change. The competitive advantage of firms is seen as resting on distinctive processes (ways of coordinating and combining), shaped by the firm's (specific) asset positions (such as the firm's portfolio of difftcult-to- trade knowledge assets and complementary assets), and the evolution path(s) it has aflopted or inherited. The importance of path dependencies is amplified where conditions of increasing retums exist. Whether and how a firm's competitive advantage is eroded depends on the stability of market demand, and the ease of replicability (expanding intemally) and imitatability (replication by competitors). If correct, the framework suggests that private wealth creation in regimes of rapid technological change depends in large measure on honing intemal technological, organizational, and managerial processes inside the firm. In short, identifying new opportunities and organizing effectively and efficiently to embrace them are generally more fundamental to private wealth creation than is strategizing, if by strategizing one means engaging in business conduct that keeps competitors off balance, raises rival's costs, and excludes new entrants. © 1997 by John Wiley & Sons, Ltd.