Content uploaded by Miriam Wilmsen
Author content
All content in this area was uploaded by Miriam Wilmsen on May 21, 2018
Content may be subject to copyright.
INTERNATIONAL DESIGN CONFERENCE - DESIGN 2018
https://doi.org/10.21278/idc.2018.0341
AGILE METHOD DEVELOPMENT: A LIVE-LAB
CASE STUDY ON PRODUCT PROPERTIES FOR
PROCESS PLANNING
A. Albers, N. Bursac, C. M. Eckert, B. Walter, M. Wilmsen and J. Heimicke
Abstract
Developing design methods can be described as a process similar to product development processes.
Following agile approaches is worthwhile in order to identify relevant requirements of potential users
at early stages of method's maturity. Live-Labs, as a controllable research environment located between
laboratory and field studies, provide an environment whose results are neither too specific nor too
generic. In this paper, an agile development of a method for process planning depending on product
attributes is presented. The research took place in a Live-Lab with industrial participation.
Keywords: agile development, Live-Labs, design methodology, research methodologies and
methods, design methods
1. Introduction
The development of methods and tools to support industry is one of the main objectives of engineering
design research (Blessing and Chakrabarti, 2009), however in practise industry seems to use few of the
design methods proposed by academia in the way that was intended by the creators of the methods.
Many (e.g. Birkhofer et al., 2002; Jagtap et al., 2014) comment on the lack of uptake of methods in
industry. This does not mean that engineers do not work in a systematic or methodical way but they
employ fragments of methods that they have come across during their studies or past professional
practise (Gericke et al., 2017). Introducing methods in industry can be problematic, because “methods
tend to be too complex, abstract and theoretical”; “too much effort is needed to implement them”; “the
immediate benefit is not perceived”; “methods do not fit the needs of designers and their working
practices”; and “little or no training and support are provided” (Wallace, 2011). One of the underlying
reasons is that researchers don’t consider the needs of users adequately (Daalhuizen, 2014) and do not
actively involve them in the development of methods. There can also be a lack of method evaluation
and communication of the value of the methods (Jagtap et al., 2014). This paper proposes to apply an
agile development approach to method development, where graduate students try out methods in
projects, to gain frequent and rapid feedback on the methods and direct refinements of the method
regarding strengths and weaknesses. Evaluating methods in industrial practise can be challenging.
Vermaas, (2016b) argues that many design methods are derived from what is perceived best practise
and validated through the experience of experts, who approve the methods. However, this is only
sufficient of corroborate the effectiveness of the method (i.e. when I applied the method, it was useful)
rather than the efficiency the method (i.e. it works fester for addressing my issue / solving my problem
than other methods or current practice). Furthermore, both: the applicability of methods to different
problems would need to be assessed and alternative methods as well as no method would need to be
used to address the same problem (Vermaas, 2016a). Most methods are developed by researchers at
DESIGN PROCESSES
713
universities, who have limited access to industry. Either methods are developed based an analysis of
literature or on a small number of industrial case studies. While this has led to excellent methods, it has
a limited potential for evaluating methods. Trying out immature methods might be a distraction to
industry and potentially carry the risk of misleading results for industry. Even if it is possible to persuade
a research team to use a method under development, it needs to have sufficient maturity that it is easy
and efficient to use to avoid losing the good will of busy engineers. It is rarely possible to get exposure
to different projects to test the scope of applicability of methods before they have reached a considerable
degree of maturity. In consequence research efforts often stop with proposing a potential method rather
than something that could be taken up by industry. This paper reports on repeated testing of methods in
agile sprints, as used in the Live-Lab at the IPEK – Institute for Product Engineering at the KIT, which
has been set up with the dual purpose of training product development master students how to work in
industry projects and providing an environment, in which the IPEK researchers and the students
themselves can develop and test methods (Albers et al., 2016a). The students are given project
challenges from industry and work with industry experts in small teams on innovative solutions for
industry. The students follow an agile development approach, whereby they develop up to four
prototypes of their design. For each round of development, they have to plan, manage and document
their activities. This provides a testing ground for new tools and methods, provided they have a level of
maturity that they can be deployed with support by the method creators. In this context it is not enough
to have an idea for a method, but it need to be supported with appropriate representations and processes.
This gives the method researchers immediate and intuitive feedback on the applicability and benefit of
their method as well as explicit feedback on the method. In this paper, we use the example of a method
that scores product characteristics in order to plan design processes, based on a theoretical analysis of
drivers and constraints as drivers of design processes (see Eckert and Stacey, 2014).
2. Methods development
The term method has been used since the beginning of systematic design research in the 19th century in
Germany. Even though there is consensus that a method is a form of repeatable procedure that helps
design engineers in carrying out their work, there has been little consensus over the precise meaning
and nature of a method. E.g. Pahl and Beitz (Pahl et al., 2007, German version p. 784) define a method
as a “systematic procedure with the intention to reach a specific goal” (in German “planmäßiges
Vorgehen zum Erreichen eines bestimmten Ziels”). The meanings range from a loose collection of
heuristics to a recipe for a particular activity. What is called a method also varies enormously in scale
and ambition from all-encompassing approaches to product development like QFD to very specific
methods for a specific task. Methods exist for all aspects of the product development process, some that
are specific to a particular phase or a particular kind of problem (see Kumar, (2013) for general design
methods; Cross and Roy (1989) and Albers et al. (2015) for a collection of classic engineering design
methods; or Phadke, (1995) for specific robust engineering methods). The degree of specificity is a
challenge associated with design methods, because it is rarely stated clearly as Vermaas (2016b) has
pointed out. These books are a collection of successful methods, that have received sufficient
recognition to be considered as worth including in a method collection, however many more methods
are proposed in publications or PhD theses, which never get anywhere near introduction in industry. In
the evaluation of a method it is also important to understand why and where the method failed. While
this can arise from an inappropriate use or a context outside the scope, there can also be aspects of the
method itself where not developed adequately. Considering this Gericke et al. (2017) broke a method
down into the constituent components that work together for a method to be successful (see Figure 1).
However, many methods have dedicated tools. To be useable, each method needs to come with a
description that covers the following aspects (definitions from Gericke et al. (2017)):
Core idea: The basic principle, technique or theory that the method employs.
Representation: An object or other artefact that shows and stands for a system of objectives, i.e.
intermediate results and deliverables created by using the method.
Procedure: A description of the actions required to apply a method, for enabling the user of the
method to do something more easily or with a sufficient guarantee of correctness, focusing on the
sequence of actions and their completeness.
714
DESIGN PROCESSES
Intended use: A description of scope of a method, the coverage within, scope and expected
benefit from using the method, informing the user about suitability of the method for a particular
design task in a specific context.
In engineering design research, a number of research methodology have been proposed such as
Figure 1. Elements of a method from Gericke et al. (2017)
Method development is a core element of several design research methodologies, which connect method
development to other research activities. Design Research Methodology (DRM) (Blessing and
Chakrabarti, 2009) proposes to identify a problem and criteria by which a solution to that problem is
evaluated. Descriptive study I is then carried out to understand the problem and its context better, than
an intervention, which can be a tool or method, is developed in the prescription phase and evaluated
through a further case study in descriptive study II. In practice it is rarely possible to go through the
entire process and if it is possible, it can be difficult to determine without doubt that an improvement is
actually brought about by the new tools or method or if it was produced by other factors. The Spiral of
Applied Research (Eckert et al., 2003) therefore splits the development of tools and methods into four
phases: empirical studies, theory and method development, prototype development and introduction in
the industry. Each of these are evaluated separately and don’t need to be carried out in that order as
insights can lead to the need for more empirical studies or theory development or move forward as an
existing tool or method is identified as a solution to a current problem. Both approaches are based on an
analysis of the current situation and a synthesis, in which a form of improvement is proposed and
acknowledge that there can be considerable iteration, as new problems or insights require revisiting
previous steps. More recently Marxen (2014) developed the integrated Design Support Development
Model based on the iPeM - integrated Product engineering Model (Albers et al., 2016b). This approach
separates research activities from a chronological sequence and recommends a selection of research
methods for individual activities.
2.1. Live-Lab environment
Live-Labs are a half-way house between laboratory studies and field studies and help to resolve the
conflict between too specific and too generic research results (Walter et al., 2016). Results from field
studies are usually very company-specific and thus have restricted scope, while findings from classical
laboratory environments are too general to be directly applied to industrial companies with an added
value. The Live-Lab at the KIT employs an agile product development philosophy (Albers et al., 2018).
2.2. Agile product development
To assure that evolving customer requirements can be met and costs from late changes can be avoided,
companies increasingly develop products on the basis of agile product development processes. Agile
processes are characterized by a sequence of iterative sprints (for example 1 month), in which
parallelized activities are undertaken. At the end of each sprint an incremental prototype of the product
or various subsystems is created so that during the entire process the degree of maturity of the product
is continuously and successively increased. Each incremental prototype increases the functionality or
features or confidence of the previous prototype generation. This enables early evaluation of subsystems
or components. Incremental development allows also for early feedback of customers. Agile
development teams need to possess all the necessary skills to develop and validate the product. Agile
approaches are well established in software development processes, and receive increasing attention in
Intended use
Design
Method
Description(s)
Procedure
Representation
Core
idea
Tool
DESIGN PROCESSES
715
the development of mechatronic products, however as the boundary conditions are different (e.g. virtual
vs. physical prototypes) modified forms established themselves. In this article the focus is set on the
processes support in different phases of the typical overall process (Initiate, Discover, Deliver and
Release). Popular agile approaches are briefly listed as a knowledge base for this goal:
Design Thinking, which focusses on gaining deepest possible understanding of the end-user's
needs, and aims at a rapid conception of solutions, an implementation in prototypes and early
testing for further knowledge gain (Brown and Wyatt, 2010).
Human Centered Design, which work with the users to gain a holistic understanding of the
design problems that is addressed to develop systems, which are useful, usable and desirable for
humans with the consistent focus on higher levels of humans' needs (Zhang and Dong, 2009).
Scrum is a framework to support the improvement of project management and development
practices to develop and sustain complex products. It has been used by leading software
companies for object-oriented software and in product engineering as an agile process of trial and
error to develop new systems within a team of non-experts (Schwaber and Sutherland, 2017).
ASD - Agile Systems Design is a holistic, structuring approach for the agile development of
mechatronic systems, the associated product strategy, validation systems and production systems,
consisting of principles, methods and processes of PGE - Product Generation Engineering. It
focuses on the consequent integration of PGE into the development process and creates a
situation-specific balance between structuring and agile elements (Albers et al., 2018).
2.3. Agile product development in Live-Labs by Co-Creation
The traditional meaning the term labs was associated with environments in which controlled experiments
can be carried out under manageable boundary conditions and controlled access., but is now also applied
to much freer organisational forms such as Media Labs, City Labs, Living Labs and Green Labs. In these
labs stakeholders from science (e.g. scientists from different disciplines), civil society (citizens from non-
governmental organisations, …), economy (small, medium and large companies) as well as government
representatives come together to focus on factors such as closeness to reality, practical application,
flexibility, creativity, exploration, rapid gain of knowledge and openness. Several universities have
started engineering labs (e.g. Design Lab at Griffith University). To focus on the integration of future
customers and users in the different phases of the development process and often make use of methods
of the User Centred Design and of Co-Creation. Often the goal is to generate new product profiles and
product ideas, to have relevant customers test functionalities of products early and realistically, to
increase the use of products and minimize development risks (Albers et al., 2018). In addition to other
lab approaches, the Live-Lab at the KIT also enables new methods, processes and tools to be evaluated
at an early stage and under realistic conditions to mature them sufficiently to be suitable for use in a
company. The Live-Lab enables exploration and evaluation of methods and processes in a realistic
environment, while at the same time being able to control the boundary conditions. Engineering students
in the later stages of their degrees can work on real development tasks set by a company in a joint
development project. They use various methods, such as creativity methods and valuation methods, and
tools, e.g. software tools in the context of a structured product development process. Process data and
insights are collected through observation, questioning, measurement and assessment of work results,
etc. in accompanying studies about the methods and tools and the procedural, social and personal aspects
of their usability. If a sufficiently number of subjects are involved in the Live-Lab, comparative studies
with control groups are also possible. In contrast to traditional laboratory environments, in which subjects
work on tasks under strictly controlled boundary conditions, the participants work on industry focussed
engineering project and see themselves less as test subjects than as product developers, which makes their
behaviour very realistic. The also work under tight deadlines and have to convince industry partners of
the merits of their results. This makes them more critical of immature methods and processes, but happy
to embrace methods that are beneficial to the success of the project. This ensures that the results of the
experiments have significantly increased practical relevance compared to laboratory environments. The
KIT developed three Live-Labs in product development: ProVIL - Product development in a Virtual Idea
Lab, IP - Integrated Product development, and AIL - Agile Innovation Lab. All follow the ASD - approach
and are offered as modules in the Master's degree in Mechanical Engineering (Albers et al., 2018). In all
716
DESIGN PROCESSES
three Live-Labs, students work on a product development task, which is defined by a partner company
with the objective of generating usable results (product concepts, mock-ups, prototypes, ...). The Live-
Labs lasts between 12 to 19 weeks. The students work in teams of four to six people. After completing
the two to five week sprints the students present their interim results in mile stone reviews to
representatives of the partner company, who decide on the direction of the next spring. The students also
attend lecturers and workshop. The grading largely focuses on learned process- and method knowledge
in the field of product engineering, so that the actual project work is driven by the students’ intrinsic
motivation. All these measures serve to make the industrial development project as realistically as
possible. The study presented in this paper was carried as part of ProVIL in the summer semester 2017.
48 students worked in eight teams from the beginning of May to the end of July on “Mobility Solutions
for Future Sharing Economies”, set as a development challenge by the project partner Centro Ricerche
Fiat (CRF). The development process of ProVIL includes a weekly online project survey, which all
students have to fill in between Friday afternoon and Sunday evening. The project survey includes weekly
recurring questions (for example, motivation, use of tools, etc.) as well as questions about the methods,
processes and tools used. This provides a corpus of data that can be analysed.
3. Research methodology
As Figure 2 illustrates the research methodology can be seen on three levels: the overall methodology
of agile method development, the development of a specific method and its application in the context of
a specific project.
Figure 2. Overview of the research and structure the paper (Section 3)
The role of the participants changes on these level, while the participants can be seen as product
development researchers on the first two level, they become engineers in a project, which is embedded
its context. This highlighted in the diagram through a reference to product generation engineering which
emphasise the existence of similar product insight or outside of the immediate context of the project.
The results of the previous level can be applied in the next level. The Live-Lab acts as project context
in our study. The sections that explain individual aspects are referenced in the diagram.
3.1. The example method: Product characteristics
The research started with the fundamental assumption that the design processes of products with similar
characteristics also follow processes with similar characteristics and pose similar challenges to the
developers. Eckert and Stacey (2014) argue that product constraints and characteristics could be used as
a starting point to predict the behaviour of design processes and to inform process planning, because
some high-level characteristics of the product lead to many of the activities. For example, all products
that are safety critical have to go through activities of rigorous and often prescribed testing, which has
to be considered in the development and determines many of the internal project deadlines. All products
with a certain degree of complexity require a significant coordination effort and are likely to go through
multiple rounds of convergent iteration and run the risk of churn, i.e. unproductive activities during
additional iteration rounds (see Yassine et al., 2003). There is a clear link between product properties
and the process by which they are being generated. Even though this is fundamental to product
development, in that we aim at developing products that meet customer and user requirements in an
appropriate way with the least effort and resources, we don’t have a theoretical way of describing the
DESIGN PROCESSES
717
link between products and processes. This method aims to develop a list of product characteristics from
the key drivers of a wide variety of products to act as an input to the product planning process at the
beginning of the process, based on the expected properties of the candidate design. This is
complementary to current planning approaches which tend to start with either gateway processes
working backwards from the expected deadline or the starting product in generational design and the
activities expected to be required to update the product.
3.2. ProVIL task 2017: Solutions for shared mobility
This ProVIL cohort was working on the development of solutions for shared mobility with Fiat as a
partners as part of the EU Project: Science2Society. While shared mobility is widely promoted multiple
issues have to be addressed. It is still uncertain which future vehicle sharing concepts will be developed
to meet a wide range of individual needs. The different student teams developed solutions ranging from
business models for a car sharing franchise to a devise that sterilizes the insight of the car, as the students
found more germs on the samples taken on steering wheel than in the universities restroom.
3.3. The authors and roles
The team of authors covers all three layers outlined in Figure 2:
Level A – New Research Methodology: four of the authors work on establishing new research
methodologies for design research. Thus, two authors already proposed research methodologies
and three of them developed the ideas of Live-Labs for early evaluation of design research.
Level B – New Engineering Method: four of the authors work on developing a new engineering
method by using product properties for process planning. One author is working on product
property methods. Four authors research on process modelling and planning.
Level C – New Product Generation: two of the authors are working in ProVIL 2017. One author
is running the ProVIL course and has been moderating the ProVIL sessions and therefore had an
in-depth understanding of the teams and their knowledge needs. Another author was a member
of the ProVIL student cohort and developed the spreadsheets.
3.4. The steps of the research for this paper
To develop and refine the method, the authors carried out the following steps:
1. The authors developed a list of product characteristics starting with a list provided in Eckert and
Stacey (2014). This list was then expanded during a brainstorming meeting. It was decided to
score the characteristics on a scale of 1 to 10
2. Twice, two authors got together and scored two products based on a list of properties. It proved
very useful to compare two products getting a sense of the scale of the score. Again the list was
refined. It was decided to use a product everybody was familiar with as reference product.
3. The list was then discussed in a meeting with 5 additional engineers, who discussed products they
were familiar with and scored them in comparison to a car.
4. All participants found the discussion helpful in thinking through the products they were familiar
with. At this point it was decided to apply the method in the context of the ProVIL project, which
was running at this point and had one last iteration cycle to go through.
5. To make the method assessable to participants in ProVIL, the list was presented as excel
spreadsheet to them. One of the authors, who herself was taking the ProVIL class, developed a
first draft of the spreadsheet. This was discussed with all the co-authors and further developed.
To make the scoring clearer to the ProVIL members, for each characteristic some well-known
products were scored as calibration, see Figure 3. In addition to just scoring the factors, the
participants would be asked to identify the 5 most important aspects of their product.
6. The method was presented to the students in a workshop two weeks before the end of their project
to support them in the planning of the last activities. The majority of the authors took part in the
workshop and observed groups of students.
7. In the last step a questionnaire was sent to the students to evaluate the method.
8. Evaluations of the questionnaire and lessons learned.
718
DESIGN PROCESSES
Figure 3. Example line of the spread sheet
4. Case study on product attributes
There is an intuitive link between the characteristics of a product and its development process. Well
known methods like QFD (e.g. Akao, 2004), develop product characteristics from the user requirements
and identify characteristics that add value in the first stage and use them to identify critical issues in the
following stage. This method is intended as a light touch method at the beginning of a product
development process to predict aspects of the process and inform product planning. The method could
therefore be used to select design alternatives according to the effort they require. In an industrial process
it is anticipated that the method would be deployed at the beginning, as key product characteristics are
decided very early in the process. The ProVIL students are working on high level product ideas and do
not have a predecessor product, unless they choose one deliberately. Their challenge is to fit their
ProVIL activities in with their busy student lives during exam season. As they were free to choose a
product, their solutions differ widely from mechatronic solutions to pure service offerings and they
therefore had to think through their development processes. They were provided with the classical iPeM
to support their process planning (Albers et al., 2016b). The workshop was implemented at the beginning
of the phase Specification, therefore the students already conducted successfully the phases Analyze,
Identifying Potentials and Conception. For them the method was aimed at bridging the gap between
their team’s product idea and the process they needed to follow.
4.1. Product attributes to enable / support process planning
The method that the students followed consisted of three steps. The focus was on step 1.
1. Scoring the product on the basis of a list of product properties (supported by a spreadsheet)
2. Identifying the time each team member had before the end of ProVIL
3. Planning the remaining activities
The method included 51 characteristics covering different aspects of the product:
1 to 8 broadly characterized the product, its name, product class, industry sector; whether the
product interacted directly with an end customers or supplied another company and its relation in
the supply chain; the contribution of the product to the system it will be part of; the life cycle of
the product and the overall life cycle of the system it is part of. These were scored with regards
to example products, see Figure 3.
11 to 18 focusses on how much was already known about the product from previous products or
known solution principles in terms of principle variation and embodiment variation, the
percentage of carry over parts, to which extend the system architecture, the parametrization, the
key equations, the functional architecture and the interfaces are known. These were scored as a
percentage in 10% increments.
19 to 23 addresses the percentage of effort required by different engineering disciplines: software,
mechanics, electronics and hydraulics
24 and 25 address the complexity of the product and the amount of subparts on a 1 to 10 scale
26 to 28 investigates the criticality of weight and geometry in the design
29 to 32 addresses the level of integration with other systems and its implications for the
maintenance and retrofit of the product
33 to 36 addresses the interfaces with other elements of the product
37 to 42 covers the importance of different end user issues, such as intuitiveness, ergonomics and
aesthetics
43 to 45 were concerns with influence on the validation, such as worth case scenarios or misuse
of the product
46 to 51 covers a range of issues around the flexibility of the product.
DESIGN PROCESSES
719
The participants were also asked to select the 5 most important characteristics and score them,
concerning the probability of occurrence of problems regarding the team-specific system of resources
and the product engineering task. Based on this, the participants identified the most critical problems
defined for each alternative solution. Furthermore, the students selected the most suitable solution
regarding their boundary conditions. Finally, they developed a plan for the following project phase based
on the necessary tasks and available resources. Table 1 describes the elements of the method in terms
of the elements of a method described by Gericke et al. (2017), see Figure 1.
Table 1. Product-property-method in terms of a Gericke et al. (2017) model
Core idea A product is scored based on a list of potential properties to support planning the develop-
ment process
Representation A list of product characteristics and a score sheet
Procedure A series of steps was outlined in the power point presentation
Use context All engineering systems
Tool An excel spreadsheet was developed to enable designers to score their product on a scale
of 1 to 10 and prioritize the issues that need to be addressed. Examples were also provided.
4.2. Case study results
To evaluate the design method, the case study was analysed from different points of view. The method
representation, procedure, description and the intended use could be evaluated through direct feedback
from the participants and through observations during the ProVIL session. In the course of the weekly
survey within the ProVIL project, a questionnaire with in total 7 questions regarding the usefulness and
applicability of the workshop and the possibilities of improvement for the future application of the
workshop was conducted. 14 students participated in the survey, which also conducted the workshop. 7
of 8 teams were represented through at least one participant.
4.2.1. Core idea
The participants considered the core idea of the method in general to be useful and helpful regarding
the identification of relevant product attributes of their product in development. The systematic analysis
of the product properties was on average considered less useful then the ability to prioritise the
activities of the current phase. This could have different reasons, one was that the participants were
already in the final sprint and availability of resources to complete the project had become a major
issue. However overall, they rated scoring product attributes and identifying available resources as
useful in the project by making the planning of the phase more transparent to the whole team. The
participants rated the clear assignment of the project tasks/ synchronisation within the team,
prioritisation of the activities and the planning of the phase as the three most helpful results of the
method. As the project ideas of the teams varied from mechatronic solutions to pure service offering,
not all properties applied to each product.
4.2.2. Representation
The participants indicated that they found some properties difficult to score. This could have been
because they were not applicable to their particular project ideas or they were ambiguously phrased.
For example, the characteristics concerning the geometric size and the weight were marked as not
rateable by 3 of the 8 participating teams, because they developed software solutions. In a future
iteration it would make sense to make some of the properties depend on product categories. For
example, for a product which scores 100% on software development amount, it might make sense to
make some characteristics invisible. In the analysis of the characteristic software development stood
out as having the highest impact on the planning. This might be, because three projects were purely
software. Another reason might be, that nearly all participants had a mechanical engineering
background, so software development is not one of their core competencies, which leads to an
uncertainty on the project planning.
720
DESIGN PROCESSES
4.2.3. Procedure
Based on observations during the case study and the feedback, it become clear that none of the teams
could finish all tasks of the method within the given time of 2 hours. More time was the most
frequently named improvement suggestion (6 of 11) for the application of the method in the next
year’s course. The teams also mentioned that the procedure forced them to carry out very detailed
planning (2 of 11). For some team rough planning might have been sufficient to leave greater
flexibility to the members.
4.2.4. Description
The method description was given to the participants orally backed up with slides. For a more formal
evaluation of the method, it will be necessary to standardise description further.
4.2.5. Intended use
In the evaluation 21,4% of the participants rated the method as unhelpful or very unhelpful for the
planning of the current project phase. 43% of the participants evaluated the prioritisation of product
properties as neutral. Some participant commented that most of the teams had already planned the
current phase a few days earlier. This leads to the conclusion, that the method was applied too late in
the project.
4.2.6. Tool
During the case study, hardly any questions about the tools arose and each team was able to work
with the tool easily. While that the current tool was suitable for this case study, it could not be used
as a standalone tool in industry, since it needs explanation of a moderator, who would need to be
specially trained or hired. Both require investment, which a company is at this stage of method
development usually not willing to pay, since there is no experience or best practices so far to show
the clear benefit of the method to a company. Thus, the design of the tool is strongly dependant of the
intended use.
5. Implication for agile method development
Methods can be seen as products in their own right, so that the development of methods can also
follow the philosophy of agile development with the same analysis and synthesis steps as product
development. This changes when synthesis and analysis is carried out, as the goal is to validate at an
early stage and go the several refinement and improvement cycles. Using the classical approach for
the development of methods as advocated in DRM (see Section 2), a comprehensive literature study
must first be carried out often followed by an empirical study of design practise, then it has to be
ensured that there is an actual need for the method and from that a method is developed. Only now
the validation can take place by applying it in a real industrial process, ideally the one in which the
need was identified. For such a validation all the elements of a method need to be present including
a detailed description of the methods is necessary and a functioning tool. After this effort, the
scientist naturally hopes for positive feedback during method validation to complete his work. This
introduced a confirmation bias, and the work tends to be corroborated rather than validated, and
relies very heavily on experts as pointed by Vermaas (2016a). Validation thereby becomes a
necessary evil instead of the hoped-for gain. The agile approach for method development aims to
carry out a first validation very early after only a few days or weeks have been invested to generate
knowledge and insights for the next iteration step. It is important to focus on the findings of the
validation study in each sprint. Logically, in such an approach, the degree of maturity of the methods
at the beginning is lower than with a conventional method developing approach. For this, however,
the activities of the method development have been completed in explicit iteration loops which
means the repeating after the validation study and detailing in the next step. This can be illustrated
by the framework introduced in Section 2, where the iterative procedure of the method development
in the phase model can be seen.
DESIGN PROCESSES
721
Figure 4. Activities of Design Support Development according to Marxen (2014)
Each sprint reflects different degrees of maturity of the elements of methods as illustrated in Table 2.
The elements of the method might not mature at the same speed. For example, in the case study method
the existing tool was working well after the first iteration, but the documentation required additional
work. The logical steps of the individual increments of the elements are outlined in Table 2.
Table 2. Logical steps for the development of different elements of methods
The rapidly implemented and in many places still immature results can lead to negative feedback, but
also provide useful input for the next iterations. For example, as the sprint sessions are moderated, the
moderator can offer support if the method is not adequately described and specify unclear steps. The use
of posters and post-it notes also allows a spontaneous adaptation of the tools, if the users have problems
with particular aspects of the method. The quick validation brings out, if particular aspects of the method
are particularly useful in the method, for example the participants really liked the ability to prioritise
characteristics that needed particular work. This interactive approach has the advantage that the users of
the method can take up a Co-Creation role, because they can influence the design of the method. The idea
of iterations of improvement also makes both the users and the creators of the method more willing to
put up with partial failure of the method, because the investment is less for both, but the influence that
they have is greater. Using the method with multiple groups in parallel, also allows an evaluation of the
methods in different application contexts which enables an assessment of the scope of the method. For
the case study method, it becomes clear that for purely software or service projects, a different list of
properties would be required, or at a least the characteristics would need to be filtered by product type.
This approach can also have the disadvantage that participants might feel rushed, because they have to
invest time into multiple trial rounds. As the case study in this paper shows, methods as not applicable
equally well to all stages of a development process, so that iterations of the methods might co-insight
with activities where they add no value. For example, the characteristics of the product would remain the
same after the concept has been selected and the participants might gain much benefit from the method.
However, the method can be applied modified next year for the Live-Lab. Furthermore, iteration loops
have to be integrated in a coherent description in order to of scientific texts, such as dissertations or
papers, is not suitable for presenting this iterative procedure in concise text sections. To evaluate design
methods with the presented agile approach, there are just a few questions which work for the evaluation
of all types of design methods as e.g. the question whether the user likes the method and whether they
can apply it easily. Methods support people and therefore processes, thus is the effect on the product
quality often indirect through the process as e.g. reduction of iterations, more time to work on a product
or greater maturity of the product. Although these aspects are also relevant for the method examined here,
722
DESIGN PROCESSES
it was not possible to analyse this criterion, because of the very early evaluation of the method in the
Live-Lab. Issues like less resources or shorter lead-time only apply to some methods and therefor the
Live-Lab study needs to be adapted to examine these and other criteria in detail. As the human being is
the centre of product development, all design processes, methods and tools should focus on the
applicability and usability in the course of the validation of the design methods. Therefore it is sufficient
for the very early validation of a method to focus on the criterion of the user appreciation and to use this
as a filter for methods in an early stage. This stands in contrast to DRM, which wants to start with fixing
the criteria against which success is measured. The presented approach uses graduate students
specialising product development as representative for young engineers in companies without specific
training. There are aspects of methods for which students are excellent predictors, e.g. usability of tools,
logic coherence steps. As mentioned before, the criterion user appreciation is essential for a design
method and needs to be addressed before the method is tested in industry. Through the real-world project
character of Live-Labs, the students have real project pressure to complete the required results up to a
defined deadline to present these to industrial representatives. Because of the mentoring of the students
through experienced design researchers, the students are willing to learn new methods, but they also
critically question the application of used methods and tend to adapt them to their problem situation.
Thus, Live-Labs work as real-world validation environments to evaluate design methods in a very early
stage before spending too much time into the elaboration of methods.
6. Conclusion
The example of evaluating a method which supports the planning of product development processes
based on product attributes within a Live-Lab shows the big potentials of agile design research. We argue
that the painful gap between methods based on knowledge acquired from literature, pure laboratory
studies and observation of practice and its application in industry can be partly closed by an agile
development of design methods using empirical studies accompanying product development projects at
universities. Through this approach, the user is actively involved in the process of method development
in the sense of Co-Creation, whereby important requirements are identified at an early stage that serve to
align and adapt methods. On the one hand this requires courses which function as teaching courses,
product development projects and research environments at the same time leading to the need for
harmonizing the dimensions teaching, research and innovation. On the other hand, (and more important)
it requires a mind-set change as it does often not follow a “clean”, waterfall or cycle like research process
but is more spontaneous, chaotic and less mature in its context of discovery. In return, results from agile
empirical Live-Lab research can be generated much faster and under much better controllable conditions
compared to field studies. It is evident that following agile research processes doesn’t imply that quality
criteria of empirical research have less importance or that the accuracy of description of the research
methodology in use is rated lower. It just means that empirical research in total can be much quicker and
mature when allowing intermediate results with lower maturity and a more explorative character as long
as design researchers can justify what they did according to research standards. Insofar, agile design
research and Live-Lab studies in particular shall not replace existing approaches, but can add specific
value, when it ensures to comply quality criteria of empirical research.
Acknowledgements
We want to thank the EU for their support in the Live-Lab ProVIL within the project Science2Society.
References
Akao, Y. (2004), Quality function deployment: Integrating customer requirements into product design,
Productivity Press, Cambridge Mass.
Albers, A., Bursac, N., Heimicke, J., Walter, B. and Reiß, N. (2018), “20 years of co-creation using case based
learning. An integrated approach for teaching innovation and research in Product Generation Engineering”,
Proceedings of the 20th ICL Conference, Springer, pp. 636-647. https://doi.org/10.1007/978-3-319-73204-6_69
Albers, A., Bursac, N., Walter, B., Hahn, C. and Schröder, J. (2016a), “ProVIL – Produktentwicklung im virtuellen
Ideenlabor”, Entwerfen, Entwickeln, Erleben Werkzeuge und Methoden in Produktionsentwicklung und Design
- Dresden, pp. 185–198.
DESIGN PROCESSES
723
Albers, A., Reiß, N., Bursac, N. and Richter, T. (2016b), “iPeM – Integrated Product Engineering Model in Context
of Product Generation Engineering”, Procedia CIRP, Vol. 50, pp. 100–105,
https://doi.org/10.1016/j.procir.2016.04.168
Albers, A., Reiß, N., Bursac, N., Walter, B. and Gladysz, B. (2015), “InnoFox – Situationsspezifische
Methodenempfehlung im Produktentstehungsprozess”, Proceedings of Stuttgarter Symposium für
Produktentwicklung.
Birkhofer, H., Kloberdanz, H., Berger, B. and Sauer, T. (2002), “Cleaning up Design Methods - Describing
Methods Completely and Standardised”, Proceedings of DESIGN 2002 / 7th International Design Conference,
Dubrovnik, Croatia.
Blessing, L.T.M. and Chakrabarti, A. (2009), DRM, a Design Research Methodology, Springer, Dordrecht,
London. https://doi.org/10.1007/978-1-84882-587-1
Brown, T. and Wyatt, J. (2010), “Design Thinking for Social Innovation IDEO”, Development Outreach, Vol. 12
No. 1, pp. 29–43. https://doi.org/10.1596/1020-797X_12_1_29
Cross, N. and Roy, R. (1989), Engineering Design Methods, 4th ed., Wiley, New York.
Daalhuizen, J. (2014), Method Usage in Design – How methods function as mental tools for designers, PhD thesis,
Technical University of Delft.
Eckert, C.M. and Stacey, M.K. (2014), “Constraints and Conditions. Drivers for Design Processes”, In:
Chakrabarti, A. (Ed.), An anthology of theories and models of design, Springer, London, pp. 395–415
https://doi.org/10.1007/978-1-4471-6338-1_19
Eckert, C.M., Stacey, M.K. and Clarkson, P.J. (2003), “The spiral of applied research. A methodological view on
integrated design research”, Proceedings of the 14th International Conference on Engineering Design (ICED
03), The Design Society, Glasgow.
Gericke, K., Eckert, C.M. and Stacey, M.K. (2017), “What do we need to say about a design method?”,
Proceedings of the 21st International Conference on Engineering Design (ICED 17), The Design Society,
Glasgow, pp. 101-110.
Jagtap, S., Warell, A., Hiort, V., Motte, D. and Larrson, A. (2014), “Design methods and factors influencing their
uptake in product development companies: A review”, Proceedings of DESIGN 2014 / 13th International
Design Conference, Dubrovnik, Croatia, The Design Society, Glasgow, pp. 231–240.
Kumar, V. (2013), 101 Design Methods: A Structured Approach for Driving Innovation in Your Organization,
John Wiley & Sons, Hoboken NJ.
Marxen, L. (2014), A Framework for Design Support Development based on the integrated Product Engineering
Model iPeM, PhD thesis, Karlsruher Institut für Technologie.
Pahl, G., Beitz, W. and Feldhusen, J. (2007), Konstruktionslehre: Grundlagen erfolgreicher Produktentwicklung
Methoden und Anwendung, 7th ed., Springer-Verlag Berlin Heidelberg, Berlin Heidelberg.
Phadke, M.S. (1995), Quality engineering using robust design, Prentice Hall PTR, Englewood Cliffs, NJ.
Schwaber, K. and Sutherland, J. (2017), The Scrum Guide. [online] Scrum. Available at:
https://www.scrum.org/resources/scrum-guide (accessed n.d.).
Vermaas, P.E. (2016a), “A logical critique of the expert position in design research. Beyond expert justification
of design methods and towards empirical validation”, Design Science, Vol. 2 No. e7.
https://doi.org/10.1017/dsj.2016.6
Vermaas, P.E. (2016b), “Towards Precedence that Justifies the Knowledge Claims of Design Methods”, The
Design Journal, Vol. 19 No. 2, pp. 195–204. https://doi.org/10.1080/14606925.2016.1129144
Wallace, K. (2011), “Tranferring Design Methods into Practice”, In: Birkhofer, H. (Ed.), The Future of Design
Methodology, Springer-Verlag London Limited, London, pp. 239–248. https://doi.org/10.1007/978-0-85729-
615-3_21
Walter, B., Albers, A., Haupt, F. and Bursac, N. (2016), “Produktentwicklung im virtuellen Ideenlabor –
Konzipierung und Implementierung eines Live-Lab”, Proceedings of the 27th Symposium Design for X, pp.
283-295.
Yassine, A., Joglekar, N., Braha, D., Eppinger, S. and Whitney, D. (2003), “Information hiding in product
development. The design churn effect”, Research in Engineering Design, Vol. 14 No. 3, pp. 145–161
https://doi.org/10.1007/s00163-003-0036-2
Zhang, T. and Dong, H. (2009), Human-centred design: An emergent conceptual model, Royal College of Art.
Dr.-Ing. Nikola Bursac
Karlsruhe Institute of Technology, Institute of Product Engineering
Kaiserstr. 10, 76131 Karlsruhe, Germany
Email: Nikola.Bursac@kit.edu
724
DESIGN PROCESSES
