BookPDF Available

Trial Guidance Methodology Handbook

February 2020

Publisher: DRIVER+ (Driving Innovation in Crisis Management for European Resilience)

Authors:

Chiara Fonio

Vrije Universiteit Amsterdam

Adam Widera

University of Münster

Crisis management (CM) organisations often face difficulties in assessing the potential impact of a change in their sociotechnical setup for several reasons, for instance the lack of adequate methodological know-how to assess innovative solutions. Investments in new, but inappropriate solutions, not only produce significant costs, but also have negative impacts for the operational performance of response organisations. Changes may be brought about by different types of solutions, such as new software or new training or workflow processes, each adopted with the aim to improve certain functions or activities. For example, the use of an app for managing volunteers (compared to legacy systems and procedures) can be assessed in a trial on the basis of key performance indicators. Assessing the impact of any kind of change is not a trivial task, as it points to both capability development and to the identification of innovation. This is why we need trials. Trials are of interest for people dealing with research and innovation who would like to test some new solutions, for practitioners in the field who have identified a problem in daily operations and are motivated to initiate the process of assessing solutions, for experts working in coordination centres who consider to participate in trial-like activities. Furthermore, in trials solution providers can collect user feedback to improve their solutions. A trial has a well-defined objective and needs to be structured It also implies a co-creative approach and an open mind. Workshops and tools are essential, as several iterations (especially for preparation) are usually needed. Trials are evolving processes: they grow “in the making”, like a handcrafted artefact. Time should be devoted to adjust the design. Key decisions must be taken in agreement with different stakeholders that need to be identified. The success of a trial then clearly depends on its design: a robust design will lead you to find appropriate answers to your needs. This trial guidance methodology provides step-by-step guidelines, a list of roles and responsibilities, tools and methods to perform a trial through a clear, structured and co-creative approach.

Content uploaded by Adam Widera

Content may be subject to copyright.

Version 1.1  February 2020

TRIAL GUIDANCE

METHODOLOGY

HANDBOOK

driver-project.eu

The handbook would not exist without the highly valu-

able contributions, intense discussions, and tireless com-

mitment of the methodological core team: Many thanks

to Dirk Stolk, Kees van Dongen, Lisette de Koning from

TNO; Konstanze Lechner and Alexander Scharn weber

from DLR; Stine Bergersen and Bruno Oliveira Mar-

tins from PRIO; Tom de Groeve and Funda Atun from

JRC, Hanneke Vreugdenhil from HKV and Nicola Rupp

and Michael Middelho from WWU. Having the whole

DRIVER+ Pan-European Test-bed embedded in the

methodological approach we further would like to thank

all contributors in the network-, tools- and methods

sections: Georg Neubauer, Dennis Havlik, Drazen Ignija-

tovic, Gerald Lichtenegger from AIT, Cor-Jan Vermeulen

and Job Verkaik from HKV, Agnese Macaluso and Niels

van Wanrooij from ECORYS, Michael Löscher, Stéphanie

Albiero, Andreas Seipelt, Laure Dodin, Marion Bonlieu,

Myriam Ben Ammar from ARTTIC, Andrzej Adamczyk,

Magdalena Karczewicz-Lamparska, Karolina Pieniowska

from ITTI, Joanna Tymińska and Emil Wrzosek from SRC

PAS, Thomas Obritzhauser and Ludwig Kastner from

Frequentis, Frédérique Giroud and Alice Clemenceau

from Valabre, Thomas Seltsam from Palacio Camilo from

ARC, Marcin Smolarkiewicz and Tomasz Zwęgliński from

SGSP, Andre de Rond and Regis Flohr from SRH, Steven

van Campen, Maurice Sammels, and Tinus Hendriks from

XVR, and Erik Vullings from TNO.

Last but not least, our special thanks go to the interdisci-

plinary and highly committed design team which gifted

us with a variety of perspectives on the methodology

itself. Big thanks to Michael Middelho, Nicola Rupp,

Johann Rolshoven, Felix Hummel, Sebastian Henke,

Niclas Rotering, and Stefan Tenhagen (WWU) for an

unforgettable design project including the iterative im-

provements and adjustments. Additionally, many thanks

to Hugo Vivier (RIKKA), Santiago Duque (Squareclouds

Design), and especially Uwe Wältring, Eske Lübbers and

Fabian Holtrup (GUCC graﬁk & ﬁlm) for their great ideas

and highly professional ﬁne tuning of the design results.

The time from the DRIVER+ kick-o in May 2014 to

the release date of the current edition of the handbook

in February 2020 was -for many of us- an impressive,

instructive and intense journey of learning, experienc-

ing and growing. Writing sessions, design workshops,

intense discussions and continuous exchanges have

paved the ground to a mature version that we are happy

to share, in the hope that it will result into a stimulating

reading.

We have come to realise that the journey has just began:

we sincerely hope that the readers and future applicants

of the TGM will enjoy their experience and understand

the handbook as being a living book, which can and

should grow with every single trial. We are deeply grate-

ful for having been -literally- on the road with DRIVER+.

TGM Handbook editors

Chiara Fonio

European Commission, Joint Research Centre

Adam Widera

University of Muenster,

ERCIS Competence Center Crisis Management

ACKNOWLEDGMENTS

The Trial Guidance Methodology (TGM) handbook is

a joint eort of the whole DRIVER+ consortium. The

DRIVER+ project (Driving Innovation in Crisis Manage-

ment for European Resilience) has received funding from

the European Union’s 7th Framework Programme for

Research, Technological Development and Demonstra-

tion under Grant Agreement (GA) N° #607798.

First, we would like to thank the Project Ocer, Guil-

laume Lapeyre, our review team from the EC Research

Executive Agency, as well as the DRIVER+ Advisory

Board for believing in the underlying approach and the

continuous provision of highly relevant and constructive

feedback and inputs. Next to this, the DRIVER+ internal

review board deserves sincere gratitude for an always

timely, meticulous, and precious input. Thank you so

much, Julia Zillies and Tim Stelkens-Kobsch (DLR, Quality

Managers), Marcel van Berlo (TNO, Technical Coordina-

tor), Peter Petiet and Marijn Rijken (TNO, Project Direc-

tors), David Lund (PSCE), Marcin Smolarkiewicz (SGSP),

Rob Munro (ARTTIC) and Francisco Gala (ATOS). We are

very glad about the amount of interest the handbook

has raised and would also like to thank our “volunteer

reviewers”, who shared their important thoughts with

us: Angela Schmitt (DLR), Anna Foks-Ryznar (SRC PAS),

Edith Felix (Thales Group), Tomasz Zwęgliński (SGSP).In

addition to that, we have received many encouraging and

highly useful comments from outside the consortium.

Given the iterative design approach, the handbook has

been publicly available between March and September

2019: Our gratitude goes to all those who have down-

loaded the handbook, and in particular to those who

shared their positive and constructive feedback with us.

ABOUT CRISIS MANAGEMENT TRIALLING

WHY TRIALS?

THE TRIAL GUIDANCE METHODOLOGY

WHY A METHODOLOGY?

THE HANDBOOK

WHY THIS GUIDE?

Crisis management (CM) organisations often face dif-

ﬁculties in assessing the potential impact of a change

in their sociotechnical setup for several reasons, for

instance the lack of adequate methodological know-how

to assess innovative solutions. Investments in new, but

inappropriate solutions, not only produce signiﬁcant

costs, but also have negative impacts for the operational

performance of response organisations. Changes may

be brought about by dierent types of solutions, such

as new software or new training or workﬂow processes,

each adopted with the aim to improve certain func-

tions or activities. For example, the use of an app for

managing volunteers (compared to legacy systems and

procedures) can be assessed in a trial on the basis of key

performance indicators.

Assessing the impact of any kind of change is not a triv-

ial task, as it points to both capability development and

to the identiﬁcation of innovation. This is why we need

trials. Trials are of interest for people dealing with re-

search and innovation who would like to test some new

solutions, for practitioners in the ﬁeld who have identi-

ﬁed a problem in daily operations and are motivated to

initiate the process of assessing solutions, for experts

working in coordination centres who consider to partici-

pate in trial-like activities. Furthermore, in trials solution

providers can collect user feedback to improve their

solutions.

A trial has a well-deﬁned objective and needs to be struc-

tured It also implies a co-creative approach and an open

mind. Workshops and tools are essential, as several itera-

tions (especially for preparation) are usually needed. Trials

are evolving processes: they grow “in the making”, like a

handcrafted artefact. Time should be devoted to adjust

the design. Key decisions must be taken in agreement

with dierent stakeholders that need to be identiﬁed.

The success of a trial then clearly depends on its design:

a robust design will lead you to ﬁnd appropriate answers

to your needs. This trial guidance methodology provides

step-by-step guidelines, a list of roles and responsibil-

ities, tools and methods to perform a trial through a

clear, structured and co-creative approach.

A methodology is one thing. A good practical guide

under your arm anytime to quickly ﬁnd any clue of this

methodology is another! This handbook shall guide you

during the whole journey of the trial experience. You

don’t have to memorize it. Instead, having it next to you

when working on the trial allows you to ﬁnd speciﬁc

answers to your current questions. It can be considered

as a “cookbook” helping you step by step to execute a

speciﬁc recipe by telling you the ingredients you need

and how to use them. Enjoy!

PREFACE

WHY THIS HANDBOOK?

WHAT IS IN THIS HANDBOOK?

TABLE OF CONTENTS

ACKNOWLEDGMENTS 2

PREFACE 4

INTRODUCTION

A BIRD’SEYE VIEW ON THE TGM 6

HOW TO READ 8

RISK TABLE 10

ROLES AND RESPONSIBILITIES 14

THE PANEUROPEAN TESTBED 18

PHASES & STEPS

STEP ZERO 24

PREPARATION 30

EXECUTION 50

EVALUATION 64

METHODS & TOOLS 80

SPACE FOR NOTES 124

IMPRINT 131

A BIRD’SEYE VIEW ON THE TGM

THREE PHASES

The TGM consists of three main phases:

•Preparation

•Execution

•Evaluation

In this handbook, you will ﬁ nd a detailed explanation of

the preparatory six step approach and the execution

and evaluation phases. Before you start reading, you

may want to have an overview of the methodological

approach.

The preparation phase consists of two tasks:

TASK 1 is the so-called step zero (S0), the prerequisite

for all trials. It involves the identiﬁ cation and the speciﬁ -

cation of gaps relevant in your context. To highlight the

importance of S0, it is depicted separately in the right

bar at the descriptions of the steps.

TASK 2 is the design of your trial. The design follows an

iterative and non-linear six step approach. Identify the

trial objectives ﬁ rst and then formulate one or more

research questions. In the trial you should address your

questions. The goal is not to elaborate a research paper,

but to generate robust results regarding the added value

of solutions, which are relevant for your speciﬁ c con-

text. To do this, you need to put in place an appropriate

data collection plan as well as having in mind evaluation

approaches and metrics to analyse the data collected

during your trial. To conduct the trial, realistic scenarios

must be developed and solutions to be trialled selected

to allow you to ascertain whether they could be

innovative.

Once the trial design has been developed, you are ready

for the execution phase, which starts with the trial in-

tegration meeting (TIM). The TIM is crucial to align the

perspectives of relevant stakeholders involved in the

trial before the arrangements are tested at the location

where the trial takes place (dry run 1). The full rehearsal

of the trial is called dry run 2. After dry run 2, you are

ready to run your trial.

After having executed your trial, the data collected

can be analysed and disseminated. The main evaluation

activities deal with checking and analysing the collect-

ed data according to the predetermined evaluation

approaches. When the analysis is done, you are ready

to synthetise the results providing you evidence on the

impact of your solutions of interest and to disseminate

the results within and beyond your community.

If you are ready to dig deep into the TGM, turn the page

and start your journey.

A BIRD’SEYE VIEW ON THE TGM

TGM’S ANATOMY

HOW TO READ

USING THIS HANDBOOK

THE THREE PHASES

A BIRD’S-EYE VIEW ON THE METHODOLOGY

ROLES

THE PEOPLE YOU NEED

METHODS & TOOLS

THE TOOLS YOU NEED

TRIAL LOCATIONS

THE PLACE YOU NEED

The TGM is split in three phases that anyone willing to run a trial should follow: preparation (designing the trial),

execution (performing the trial), and evaluation (assessing the results). Each of these phases is divided into steps.

By ﬂ ipping pages and using the

vertical bar at the right of the book,

you can quickly and easily reach a

speciﬁ c step in a given phase.

At the end of each phase section,

you’ll see examples of how this

phase has been implemented in pre-

vious trials.

Section page

“Evaluation” Phase, step 4

Tool/method page

Steps where

this tool/method

is useful

Phase example page

Trial locations page

Step page

Roles page

Going through all phases of a trial is a team e ort. The

Roles section presents the main human functions need-

ed for a trial. Multiple roles can be covered by more than

one person who can deal with several responsibilities.

Tip: On the “Steps” and “Methods & Tools” pages, you

can ﬁ nd the roles that should be involved in this speciﬁ c

part of the trial.

Tools and methods are meant to help you executing the

various tasks of a trial. They are described in a dedicated

section (one page for each tool or method).

By ﬂ ipping pages and using the vertical bar at the right

of the book, you can quickly and easily see which tools

are used in which steps and phases.

The TGM includes trial locations, which are the place you

need to perform your trial. Trial locations are presented

in a dedicated section at the end of this handbook. They

consist in physical, methodological and technical infra-

structure elements to systematically conduct trials and

evaluate solutions within an appropriate environment.

They are places where trials can be run. Please contact

them in case you consider to organise a trial.

THE AIM OF THIS HANDBOOK IS TO LET YOU PROMPTLY FIND WHAT YOU ARE ACTUALLY LOOKING FOR

WHEN CARRYING OUT A TRIAL. HERE ARE A FEW TIPS TO NAVIGATE THIS GUIDE AND USE IT EFFECTIVELY.

PREPARATION

LINK

ABOUT

WHAT THIS TOOL IS FOR

•TRIAL OWNER, TRIAL HOST

•SOLUTION COORDINATOR

•SOLUTION PROVIDER

•PRACTICIONER PARTICIPANT

THE MAIN PURPOSE OF THE PORTFOLIO OF

SOLUTIONS IS TO STORE AND PROVIDE ALL

RELEVANT INFORMATION ABOUT INNOVATIVE

SOLUTIONS IN CRISIS MANAGEMENT

TOOL: PORTFOLIO OF SOLUTIONS

POS

EVALUATION

CHECKLIST

•

IN DEPTH

ALL YOU NEED TO KNOW ABOUT THIS STEP

METHODS TOOLS

INPUT OUTPUT

IN A NUTSHELL

WHAT THIS STEP IS ABOUT

TRIAL OWNER (LEAD)

PRACTITIONER COORDINATOR

2 DAYSTO MAKE SURE THE GAINED

KNOWLEDGE IS SUSTAINED

EVALUATION

DISSEMINATE RESULTS

The second part of the preparation phase is the six step

approach. After having thought carefully about the con

textualisation of your gap(s) - step zero - you are now

ready to start designing your trial.

Again the starting point is being on the same page about

your goal with everyone involved: the trial objective.

This is a very important step as the trial objective(s) is/

are pointing the way ahead. Speciﬁc information is pro

vided on the following pages.

Based on this you will also formulate research questions.

The aim of formulating a research question is that it

increases the incentive to ﬁnd an answer, right? Further

more, by stating research question(s) you make it clear

to everyone that you are not going to just play around

with some nice new “toy” and only “to ﬁnd out if people

like it or not”. Your goal is to assess potentially innovative

solutions that may/will be a “game-changer” in your or

ganisation.

Because you aim at a structured assessment that will

bring you concrete data to prove whether a new solu

tion will bridge your gap, you need to think about those

data. What exactly do you need to measure? Which

is the Key Performance Indicator that is your “game

changer”? Will improve everything by increasing or de

creasing? All this you pin down in a data collection plan.

You have to be clear on how to collect those data. It is

up to you to decide and to write it down in your eval

uation approach. Is it something that can be measured

using the test-bed technical infrastructure, can it be

observed and captured through a questionnaire?

When you know what to measure and how, you know

what speciﬁc situations you have to create, in order to

trigger the gap. You know all involved roles, their activ

ities and the information exchanged. Based on this in

formation you can create a dedicated trial scenario, that

will make sure all needed “gap behaviour” is triggered in

a way that enables the application of a new sociotechni

cal solution and to related measurement.

And ﬁnally you know exactly what you need – and can

now choose a solution for trialling it that does not only

claim to bridge your gap, but is ready to prove how and

to what extent it can do this. Now you can make an

informed decision at the solution demonstration and

selection meeting.

The above mentioned process is an iterative one. Every

time your information changes, you might want to up

date other parts of this cycle. For example, if you have

chosen a particular solution, you have to update your

data collection plan to the speciﬁc characteristics of this

solution.

TRIAL OBJECTIVE 32

RESEARCH QUESTION 34

DATA COLLECTION PLAN 36

EVALUATION APPROACHES AND METRICS 38

SCENARIO FORMULATION 40

SOLUTION SELECTION 42

EXAMPLE TRIALS 1, 2 & 3 44

THE SIX STEP APPROACH

PREPARATION

STEP ZERO

PREPARATION

EXECUTION

EVALUATION

CHECKLIST

•

Cross-checked whether every gap is covered by (at least

one) research question

•

Checked that each research question meets the above

mentioned research question criteria

•

Checked whether each research question is updated with

the newest information (while following the iterative,

co-creative six step approach)

While your trial objective(s) might seem a little general, now you can go into

detail. If you are e.g. interested in a communication problem between

hierarchical levels during construction ﬁres, you can now dive deeper into the

problem by identifying the underlying gap: Is it a connectivity problem? Do they

use dierent languages (phrases, words)? In an interactive discussion with your

CM practitioners, you will naturally formulate questions. This will help you to

identify the data that must be collected. For example When? means you need to

measure time. How? might lead to intensive observations in combination with

some data logged by the test-bed technical infrastructure.

The wording can also help you to select the functionality you are actually

looking for in an innovative solution. For example: Do you need an ampliﬁer

or a vocabulary trainer or something entirely dierent?

Here you can ﬁnd a list of criteria to formulate good research questions:

Needs to be a question

Needs to address a distinct gap of the trial

Needs to cover the three dimensions of trials

•

Trial dimension

•

Crisis management dimension

•

Solution dimension

Must not be scenario-driven

Needs to be answered and measurable by the trial

Needs to be understood and approved by all trial stakeholders

Scenario and evaluation are directly related to the research-question

Can be organised in a multi-level hierarchical structure

Is formulated simple (but is not always easy to answer)

IN DEPTH

ALL YOU NEED TO KNOW ABOUT THIS STEP

METHODS

Workshop, discussions, societal impact

assessment, research ethics 3 dimensions

& KPI’s

TOOLS

Physical meeting, teleconferences, mind

maps, pen & paper, trial guidance tool, trial

action plan, knowledge base

INPUT

Trial context, CM gaps,

SMART, trial objective(s)

OUTPUT

One or more research questions

By formulating a SMART objective you have deﬁned

“what” you want to achieve/investigate in your trial.

Now you need to formulate research questions that

address what you are trying to ﬁnd out in your trial.

The aim of this step is to identify the proper mix of

research methods and data analysis techniques, taking

the trial conteext into account.

IN A NUTSHELL

WHAT THIS STEP IS ABOUT

EVALUATION COORDINATOR

(LEAD)

TRIAL OWNER

2 HOURSTO FOCUS ON SPECIFIC AS-

PECTS AND DETERMINE YOUR

EVALUATION APPROACH

PREPARATION

RESEARCH QUESTION

STEP ZERO

PREPARATION

EXECUTION

EVALUATION

CHECKLIST

•

Lessons Learnt Library ﬁlled in

•

Knowledge base updated

•

Portfolio of Solutions updated

•

Internal documentation done

•

Internal dissemination done

•

External documentation done

•

External dissemination done

•

Consider legal restrictions or limitations with regards to the

solutions when you communicate results. Always interpret

and consider the evaluation results in the trial context.

Do some good and talk about it! A lot of people were involved in preparing

and conducting the trial. The evaluation on the other hand was most likely

done only by a few people. So now go ahead and let all the others know what

you found out. What was it that they contributed to? Did it help that they

spent their time working on it?

You could organise a meeting to talk about the results with your practi

tioners and discuss a way forward - in the end you still have your gap but

now maybe also a solution. Include the outside world. crisis management is

a local, a European and also global task. So share your knowledge and inspire

others (who might also have that same or a similar gap). Here you can up

date the lessons learnt library, the DRIVER+ knowledge base and also the

portfolio of solutions.

Your solution providers are very important. Let them know what you think

of their “products”- they will be very thankful for any bit of information that

helps them to go forward in their development! And don’t forget about re

searchers. Sitting in an ivory tower is not nice, so help them in see the real

world!

IN DEPTH

ALL YOU NEED TO KNOW ABOUT THIS STEP

METHODS

Meeting, social media, website, newspaper

article, conferences, societal impact

assessment, research ethics

TOOLS

Lessons learnt framework, portfolio of solu

tions, trial guidance Tool (knowledge base),

lesson learnt library

INPUT

Answers

OUTPUT

Tweets, newspaper article, website content,

journal paper, updated lessons learnt library

etc.

At the end of the trial you want to create something

sustainable. Therefore spread the word: Let people know

what you learnt. About your gaps and how to bridge

them but also about trials. Furthermore: Write down

what lessons you learnt with regards to trials etc. - for

conducting trials, for crisis management, for your organ

isation etc.

IN A NUTSHELL

WHAT THIS STEP IS ABOUT

TRIAL OWNER (LEAD)

PRACTITIONER COORDINATOR

2 DAYSTO MAKE SURE THE GAINED

KNOWLEDGE IS SUSTAINED

EVALUATION

DISSEMINATE RESULTS

EXAMPLE TRIAL 1 – PL

PREPARATION PHASE

THIS EXAMPLE PRESENTS AN EXCERPT OF THE PREPARATION PHASE IN THE FIRST DRIVER+ TRIAL

HOSTED IN POLAND. IT DEMONSTRATES THE SIX STEP APPROACH OF THE PREPARATION PHASE START-

ING FROM ONE OF THE TRIAL OBJECTIVES AND FOLLOWS ONE GAP, AS WELL AS ONE RESEARCH QUES-

TION. ACCORDINGLY, THE LATER STEPS OF FORMULATING THE DATA COLLECTION AND EVALUATION

PLAN, SCENARIO FORMULATION AND SOLUTION SELECTION WILL ALSO FOCUS ON THIS NARROWED

SCOPE FOR ILLUSTRATION PURPOSES.

Objective

The overall objective was to simulate coordinated actions

at the local, regional, national and international level with

the purpose of counteracting the eects of the disaster

eects and to trial selected solutions for their applica-

bility in addressing current crisis management gaps. The

sub-objective relevant for this example is to improve the

eectiveness of identifying the needs of aected people

trapped in buildings in the chemical spill area through:

• Shortening the time to indicate/point on the map the

location of the residents in need.

• Improving the accuracy of the identiﬁcation of the

type of needs.

Gap

Among others, one of the identiﬁed gaps was the

insuciency in terms of resource management (human

resources, hardware, etc.) during multi-stakeholder

long-term rescue operations.

Research Question

A research questions was formulated speciﬁcally for the

gap mentioned above. Gap speciﬁc research question:

How can cross-border resource management be support-

ed through sociotechnical solutions during multi-stake-

holder long-term rescue operations? Accompanying with

this research question, an assumption was formulated,

which is to be assessed through the data collection and

evaluation plan. Such an assumption is not required by the

methodology, but it might help in guiding further actions.

Data Collection Plan

The trial was executed as a simulated table top and ﬁeld

experiment, which motivated the use of dedicated ob-

servers, who recorded and documented the actions. For

the evaluation purposes of this part of the trial, the data

below was collected, evaluation questionnaires ﬁlled in

by the observers and aimed at recording operational

decision time slots (from achieving the data collected

during the drone ﬂight to the end of counting or mea-

surements).

Evaluation questionnaires on three dimensions (crisis

management, trial and solution dimensions) ﬁlled in by:

• Practitioners: providing feedback (data) regarding

quality of the trial as well as usability, innovation,

user friendliness and other aspects of the solution.

• Observers: providing feedback (data) regarding

observed organisational diculties of the trial

conduction, external constraints that may inﬂuence

the trial results.

Besides overall satisfaction and usability scores from

questionnaires, further KPIs have been deﬁned to as-

sess the potential improvement in crisis management

achieved by applying new solutions.

KPI 1 – Number of identiﬁed needs in total indicated

by coloured ﬂags.

KPI 2 – Time for decision-making.

KPI 3 – Types of identiﬁed needs indicated by the

correct identiﬁcation of coloured ﬂags.

KPI 4 – Location of the needs.

Evaluation

In order to enable the assessment of improvements,

multiple sessions have been executed to compare

the current mode of operation in the baseline to

the innovative solutions in the Innovation Line. This

enabled a comparison between these sessions. The

combined observations support the assessment of

the results in light of the speciﬁc trial execution

considering diculties and constraints as well as

the three evaluation dimensions crisis management,

trial and solution.

Plan Scenario

The scenario of the trial includes a massive release

of liquid toxic substances because of a maintenance

failure in a reservoir collecting chemical waste.

A valve failure means that the pumps, pumping

chemical waste liquid to the reservoir, cannot be

switched o. Due to this, there is a rapid inﬂow of a

signiﬁcant amount of a liquid, mud-like toxic

chemical to the retention reservoir. The dikes of

the reservoir are weakened after prolonged rainfall

during past few days. Due to increased pressure,

the dikes break.

Selected Solutions

Drone rapid mapping - The solution enables very

fast generation of orthophoto maps based on

imagery acquired by a drone (RPAS) available to

rescue or crisis management actors. The resulting

maps could be viewed and analysed in the dedi-

cated geoportal or any GIS environment already

utilised by crisis management institutions. The

additional product was a 3D model of the terrain,

enabling better and more intuitive understanding

of the area of interest.

STEP ZERO

PREPARATIONEXECUTIONEVALUATION

In this Handbook, there is a section

for each phase and a dedicated page

for each step:

ROLES AND RESPONSIBILITIES

ACTORS AND STAKEHOLDERS 1. TRIAL OWNER

2. TECHNICAL COORDINATOR

The “owner” of a trial is the CM organisation which is

mainly responsible for the trial itself. While, on the one

hand, trials are collective eorts, there should be one

organisation that takes up the responsibility for planning,

executing and evaluating the activities. This important

role encompasses the following responsibilities:

A Developing a proper scenario so that the gaps and

needs of the main stakeholder are captured in the

trial (scenario development);

B Hosting the trial itself using one or more locations

and ensuring that the chosen location is apt to the

purpose of the trial (trial host);

C Directing the trial. The director has a prominent role

in all phases and, as the name suggests, he or she

gives the right directions: for instance, the director

initiates the trial during the actual execution and is

entitled to stop it any time, in case of problems and/

or to put in place mitigation actions;

D Managing the trial-event in terms in logistics (e.g.

rooms and equipment), safety (e.g. make sure that

the people involved in the trial are not in danger),

media (e.g. dealing with the media before and after

the event) and participants (from active to passive

actors: players, observers and guests).

The technical coordinator is responsible for a proper

technical set-up of the trial scenario, so that an ade-

quate assessment of the selected solutions is ensured.

Speciﬁcally, the following three responsibilities should

be covered by the technical coordinator:

A The ﬁrst aspect is the application of the technical

test-bed infrastructure. The technical coordinator

makes sure that the test-bed technical infrastructure

is adjusted according to the decisions taken in the

preparation phase and to the lessons learned during

the rehearsal and that all components work together

smoothly with the trialled solutions. During the trial,

the technical coordinator oversees all technical as-

pects (e.g. integration with legacy tools at the trial

location, data exchange etc).

B This is why the technical coordinator is also in charge

of a proper solution providers management. Solution

providers are actively involved in the development

of the trial, as they know how to best integrate their

solutions in the trial scenarios. Therefore, solution

providers need to participate in relevant meetings

prior to the execution phase so that they can get a

comprehensive overview of the activities. The role of

the technical coordinator does not end at the end of

the trial execution. In fact, the technical coordinator

works closely with the evaluation coordinator to pro-

vide insights on the overall test-bed application.

C Another key responsibility is the training manage-

ment to be provided to the trial participants. The

technical coordinator takes decisions with regards to

the training needs by deciding how to train the play-

ers who actively use the selected solutions during the

trial. To do this, solutions providers must be instruct-

ed and involved in the overall trial design from the

onset.

REGARDLESS OF THE SIZE OF YOUR TRIAL, IT IS VERY IMPORTANT TO AGREE ON "WHO IS DOING WHAT"

BEFOREHAND. IT'S EASIER TO SAY SO THAN TO PUT IT INTO PRACTICE THOUGH, AS THERE ARE SEVERAL

ASPECTS TO BE CONSIDERED. HOWEVER, THERE ARE SOME MINIMUM STANDARDS, MEANING SOME

KEY ROLES AND RESPONSIBILITIES YOU DON'T WANT TO SKIP TO ENSURE THAT THINGS GO SMOOTHLY.

PLEASE CONSIDER THAT ONE PERSON CAN FULFIL ONE OR MORE RESPONSIBILITIES: BASED ON YOUR

TIME AND YOUR RESOURCES, YOU CAN DECIDE TO HAVE A TRIAL OWNER WHO IS RESPONSIBLE FOR

HOSTING AND DIRECTING THE TRIAL AS WELL AS FOR FOLLOWING THE DEVELOPMENT OF THE SCENAR-

IO AND MANAGING THE EVENT AS SUCH. WHAT IS IMPORTANT TO BEAR IN MIND IS TO NOURISH THE

BRAIN POWER NEEDED FOR THE TRIAL RELYING ON AT LEAST ON FOUR MAIN ROLES: TRAIL OWNER,

TECHNICAL COORDINATOR, EVALUATION COORDINATOR AND PRACTITIONER COORDINATOR.

A SHORT EXPLANATION IS PROVIDED IN THE FOLLOWING PAGES.

STEP ZERO

PREPARATION

EXECUTION

EVALUATION

LINK

https://pos.driver-project.eu/PoS/solutions

The portfolio of solutions provides the possibility of

describing a solution in a standardised way. The solu

tion owner is able to state in which innovation stage

the solution is currently in, what readiness level it has,

which crisis cycle management phase is targeting,

and which crisis size it covers. It also gives the oppor

tunity to provide information on which standards are

supported by the solution, and to upload and store all

documentation regarding the solution, such as man

uals, installation/ conﬁguration guides etc. Solution

providers can also describe use cases in which CM

functions are addressed. Other than that, PoS allows

references to be added to both internal DRIVER+

trials and external experiments, to give additional in

formation on how the solution performed in real-life

situations.

For the trial owners and CM practitioners, the PoS’s

search function allows easy discovery of relevant

solutions by ﬁltering all information provided by the

solution owner and by clearly stating which CM func

tions are being addressed. The solution overview page

of the PoS is based on search API which implements

deep search algorithms that allow searching through

all components of the described solution for relevant

terms, delivering fast, user-speciﬁed search and also

gives the possibility to ﬁlter the solutions by CM func

tions, allowing easy matching with trial gaps. The PoS

also implements a PDF export function to allows easy

information extraction for further usage. This func

tionality can be combined with the ﬁltering function

that the tool oers to generate PDFs containing us

er-speciﬁed information, that being a description of a

single solution, or for example, description of all solu

tions that address the same CM functions. Integrated

help functionality is designed to help both solution

owners in describing their solution in the best possible

way and to help trial owners in selecting relevant solu

tions to be benchmarked in a trial.

The future goal of the PoS is to propose a market

place where the next generations of CM practitioners

will be able to ﬁnd information related to solutions to

ﬁll the existing gaps in crisis management, and also to

discover new innovative solutions provided by solu

tion owners for arising problems.

The portfolio of solutions is a web-based online platform

that aims to document all relevant information regard

ing the solutions in the crisis management across Europe

in such a way that dierent stakeholders can easily ac

cess this information. It also aims to standardise the lan

guage through the use of shared vocabulary of pre-de

ﬁned taxonomies, so that for example, CM professionals,

solution owners, CM practitioners and trial owners can

work on the same level, and use the same terms, making

the collaboration much easier. The trial guidance meth

odology describes a six step approach - an iterative pro

cess for trial preparation, where the last step includes

selection of trial relevant solutions. The main role of the

PoS in this step is to allow trial owners, and CM practi

tioners to select solutions that are going to be used and

evaluated in the trial and that are related to the deﬁned

trial gaps, which are linked to CM functions. In other

words, the PoS aims to help in the solution selection

process, by oering the information on which CM func

tions are addressed by the solutions, so that they can be

matched with the deﬁned gaps.

Another important function of the PoS, is to propose a

marketplace where providers can advertise their inno

vative solutions in the ﬁeld of crisis management, and

improve the chance of them being selected for a trial, or

being used by CM practitioners. It also allows descrip

tion of potential use cases, to give more insights on the

actual use of the solutions.

The search functionality of the PoS enables an easy

search through a large number of solutions, maintaining

the high level of relevancy, by applying the correct ﬁl

ters that narrow the search results. A goal for the future

is to make the PoS project independent, so that infor

mation about potential solutions for ongoing real-life

crisis management problems is always available when

needed.

ABOUT

WHAT THIS TOOL IS FOR

•TRIAL OWNER, TRIAL HOST

•SOLUTION COORDINATOR

•SOLUTION PROVIDER

•PRACTICIONER PARTICIPANT

THE MAIN PURPOSE OF THE PORTFOLIO OF

SOLUTIONS IS TO STORE AND PROVIDE ALL

RELEVANT INFORMATION ABOUT INNOVATIVE

SOLUTIONS IN CRISIS MANAGEMENT

TOOL: PORTFOLIO OF SOLUTIONS

POS

SZKOŁA GŁÓWNA SŁUŻBY POŻARNICZEJ

MAIN SCHOOL OF FIRE SERVICE

ENTENTE POUR DE LA FORÊT MÉDITERRANÉE

VALABRE

The Main School of Fire Service (SGSP) is a state services national technical

university supervised by the Minister of Interior and Administration with al-

most 100 years of history. It consists of two faculties: Civil Safety Engineer-

ing (incl. topics: crises and risk management, civil protection, civil emergen-

cy planning and coordination, internal security, CBRN, CIMIC, rescue and

logistic, etc.) and Fire Safety Engineering (incl. topics: ﬁre engineering, ﬁre

and rescue operations, command and control, incident commanding, etc.).

Besides being a university, SGSP is also an operational unit of the State

Fire Service, which runs its own professional ﬁre station and forms national

rescue reserves ready to be deployed country wide by General Director for

Civil Protection in the event of a major disaster.

To enable the most eective training, SGSP has not only a very good IT in-

frastructure, which is focused on didactic and oce work, but also a training

ground that allows for various scenarios (incl. USAR, water rescue etc.).

Valabre is a governmental organisation for the protection of the forest and

the environment against ﬁres. This organisation coordinates the eorts of

the 14 departments most aected by forest ﬁres of the South of France

covering 4 regions: Provence Alpes Côte d’Azur, Occitanie, Corsica, and

Auvergne-Rhône-Alpes, to ﬁght forest ﬁres.

The ﬁre ﬁghter ocer’ speciality training school (ECASC) is one department

of the VALABRE organisation. Within its various pedagogical means, it uses

simulation, notably in its new facility Centre Euro-méditerranéen de Sim-

ulation des Risques (CESIR). CESIR is a facility specially focused on virtual

simulation environment, with an area of 600 m² fully customisable for any

organisation. It contains a conference room with 150 seats and multi-source

displays. Several meeting rooms and classrooms are also available.

Simulation capability is deployed in CESIR, enabling the immersion of partic-

ipants in a virtual scenario. A large number of rooms allows scenarios to be

planned with a lot of dierent actors from ﬁeld actors to upper hierarchical

levels. Such rooms are connected via internet and radio communication.

Contact

Prof. Dr. Marcin M. Smolarkiewicz

Słowackiego 52/54

01-629 Warsaw, Poland

+22 (0) 561 7569

marcin.smolarkiewicz@

projectdriver.eu

www.sgsp.edu.pl

Contact

Alice Clemenceau

Domaine de Valabre

13120 Gardanne, France

+33 (0) 4 4260 8683

alice.clemenceau@ProjectDriver.eu

www.valabre.com

RISK TABLE

HOW TO MITIGATE RISKS?

TECHNOLOGY

ORIENTATION

INVOLVEMENT OF

SOLUTION PROVIDERS

REFERENCE DATA COPARTICIPATION AND

COMMUNICATION

REALISM OF TRIALS

EXPLANATION

RISK AREA

EXPLANATION

Once a solution is pre-selected, trial participants

tend to develop the trial scenario according to

the functionalities of the solutions. By doing so,

often practitioners’ realities are neglected. In

consequence, the gathered data might become

irrelevant for the practitioners and the ultimate

goal of providing a practitioner-driven evalua-

tion can be missed.

The experiences collected in trials, highlighted

an active involvement of solution providers

during the actual execution. Especially when

complex solutions were used for the ﬁrst time.

Assessing innovative solutions can be done in

many dierent ways. Running a trial according

to the TGM is one speciﬁc approach, which com-

bines traditional approaches with a new way of

investigating the impact of solutions on the CM

performance at the center of the assessment. It

may happen that TC members are more familiar

with traditional approaches which might limit

their willingness to spend additional eorts es-

pecially on providing reference data needed to

measure the impact of new solutions.

It was often observed that a participatory ap-

proach was used internally but not externally.

Meaning that players, observers or the solution

providers missed the full picture. The CM-re-

lated participants might get lost as soon as the

scenario does not reﬂect their realities or if the

execution of the trial is not explained properly

(i.e. what happens when, why and how). On the

other side, also the involved solution providers

might get confused or even frustrated if the

scenario and the way how their solution was

integrated into it, was not reﬂected with and

communicated to them.

Don’t design the trial scenario following the log-

ic of technical solutions. The interest of the CM

practitioners is at the centre of a trial. Before

taking major decisions, always check that the in-

terest(s) expressed by the main stakeholder (CM

practitioner) does not get lost. The key recom-

mendation is to put enough emphasis on draw-

ing the base- and innovation lines and freezing

the scenario design as soon as possible.

Ensure that training is appropriate to minimise

an active involvement of solution providers

during trials. In case of the use of very complex

solutions, solution providers should be allowed

to guide practitioners during the execution

phase, providing that roles and responsibilities

are clariﬁed from the onset.

The main mitigation measure is to start each trial

with a proper presentation of and agreement on

the TGM. When it comes to generating refer-

ence data it is key to keep in mind the implica-

tions it might have on the required eorts. If you

have the opportunity to re-play past scenarios

for which data is already stored, then use this.

In doing so, you will reach a high level of realism

and the execution of the trial comes along with

less costs. If this is not the case, the best answer

to ensure a comparison to the perceived perfor-

mance in the innovative trial scenario is to exe-

cute baseline runs. This will double your eorts

during the execution phase, but it is key to carry

out appropriate comparisons.

Have an inclusive approach with all the stake-

holders involved in a trial, including those who

join “only” the execution phase. Explain how

data is collected to the participants. Commu-

nicate key results to practitioners so that that

they can learn from the experience. A trial does

not end at the execution phase! Also, make sure

that the solution providers are not afraid of the

results. Communicate clearly that a trial is only

showing the potential contribution in one par-

ticular scenario. The results are not about saying

something is good or bad, but how it did con-

tribute to one speciﬁc simulated operation.

In DRIVER+ trials there was the tendency to

come up with complex scenarios to make sure

that all requirements were met (address all gaps

and trial all solutions). A negative side eect is

the inability to communicate the scenario and

the trial objectives, which causes confusion

among the CM practitioners, observers and the

solution providers. In turn, misunderstandings

and confusion amomg trial participants aects

badly the analysis of the trial results.

Scenarios should cover all gaps but they should

-for and foremost - be as much realistic as pos-

sibile. Scenarios must reﬂect practitioners’ re-

alities: this is a minimum requirement. Complex

scenarios are not necessarily better ones. Avoid

getting lost in details and stick to overall vision

and to the requests of the main stakeholder(s)

involved in the trial. A good approach to check

the degree of complexity and level of realism is

to ask the main stakeholders (CM practitioners)

for their feedback on the data collection plan in

relation to the ﬁnal scenario.

MITIGATION MEASURE

Before taking a deep dive into the TGM, you would

perhaps be interested in reading about some risks

which might occur in a trial. Actually, these risks did

not come out of the blue: we have some hands-on

experience. In the risk table you will ﬁnd risks cate-

gorized per topic, with an explanation and potential

mitigation measures. You might come up with better

ones but please, take ﬁve minutes of your time to

have a look at the table.

RESPONSIBILITY DIFFUSION

EXPECT THE UNEXPECTED TIMING AND TIME PRESSURELANGUAGE

EXPLANATION EXPLANATION

The TGM is a highly scalable approach. Trials can

be “simple” by investigating one particular solu-

tion in a modest scenario but trials can also be

used to assess several solutions at the same time

in a complex scenario. Depending of the overall

setup the size of TCs can vary signiﬁcantly. While

small TCs might cause higher workloads, the

risk of big TCs is more complex. Next to a neg-

ative eect on decision-making time, a tricky

challenge has been identiﬁed in the assignment

and fulﬁlling the responsibilities. In case of an

unclear, multiple or overlapping distribution of

responsibilities among the TCs it might happen

that important tasks are not taken up, executed

inappropriately or cause serious delays.

No matter how precise and detailed you are

during the preparation phase and in rehearsals:

hiccups can always happen during the actual

trial. For instance, data exchange between solu-

tions can go wrong with a detrimental impact on

the data collection or CM practitioners invited

as players might not show up because of a real

crisis they have to deal with

In collaborative projects in general, every proj-

ect member has the tendency to get things

done fast. Given the nature of dedicated roles

and responsibilities the importance of a deci-

sion depends on the role each member has. This

causes conﬂicts of interest with the allocation of

time to dierent decisions. In turn, group dy-

namics might lead to impatience within the trial

committee.

There are many reasons why during the TGM

application it is suggested to use English as the

trial language (e.g. because of an international

trial team or the available early-stage solution).

However, CM practitioners are regularly using

their native language which is part of their stan-

dard operating procedures. Ignoring the practi-

tioners’ realities has a serious impact on how the

potential added value of innovative solutions is

perceived and assessed.

To overcome a potential diusion of respon-

sibilities it is important to (1) not overload the

number of TC roles, (2) to clearly deﬁne and

dierentiate the responsibilities as well as (3)

to communicate regularly the state of the trial

development structured along the roles and

the responsibilities. These mitigation measures

might be percieved overwhelming in the very

beginning of a trial . Remember that assigned

responsibility does not mean that no additional

support can be requested. It is actually quite the

opposite, as the assigned roles will be empow-

ered by a lower decision making complexity and

an explicit area of responsibility.

Having plans B with regards to organisers and

participants: always have more than one person

appointed for a speciﬁc role/responsibilities.

During the trial: have a small group of deci-

sion-makers, problem solvers and pre-deﬁned

workarounds speciﬁcally appointed to tackle

problems as soon as they arise.

Haste makes waste. It is important to be patient

within the TC, while being realistic with schedul-

ing and setting deadlines during the trial devel-

opment. It is also possible to adjust and change

your plans, even during the execution phase.

Enter each phase with an open mind: it’s better

to change things when you can, instead rushing

into decisions you might regret during the actual

trial. Inappropriate decisions can cause serious

limitations to reaching the overall goal of a trial.

Try to use the native language of the involved

practitioners as much as possible. The more famil-

iar the practitioners get with the new solutions,

the more relevant the trial results might be. This

principle might cause additional eorts, e.g. by

providing new language packs of the solutions, but

these costs allow for a better assessment of the

solutions. In case of dedicated scenarios, which

include e.g. cross-border operations, using non-na-

tive languages can be appropriate. All other cases

call for careful considerations of pros and cons.

MITIGATION MEASURE MITIGATION MEASURE

TRIAL, NOT EXERCISE

Due to the nature of the TGM, the innovative

solutions are trialled under as much realistic as

possible circumstances. This implies that the

participating practitioners are requested to re-

spond to the dierent events as they would do

in reality – except the agreed changes given by

a realistic implementation of the new solution to

the standard operating procedures. By doing so,

the actual solution moves to the background, as

most of the legacy systems are used intuitively.

As an unintended consequence, the actual use

of new solutions might decrease as opposed to

the use of legacy systems.

Even though, this dilemma between using the

solutions and solving the crisis will be always

part of trials, there are several measures to avoid

the non-utilization of the solutions: (1) it is key to

design the scenarios in a way that the use of the

solutions is enforced, e.g. by emphasizing the

deviations from the standard operating proce-

dures; (2) implement various elements remind-

ing the attendees about the actual goal of trials

(e.g. time jumps, recaps between the sessions, or

reduction of stress); the more the trial scenario

is designed as an exercise, the more the prac-

titioners turn to their standard procedures and

refuse the use of the solutions.

RISK AREA RISK AREA

RISK TABLE

HOW TO MITIGATE RISKS?

ROLES AND RESPONSIBILITIES

ACTORS AND STAKEHOLDERS 1. TRIAL OWNER

2. TECHNICAL COORDINATOR

The “owner” of a trial is the CM organisation which is

mainly responsible for the trial itself. While, on the one

hand, trials are collective eorts, there should be one

organisation that takes up the responsibility for planning,

executing and evaluating the activities. This important

role encompasses the following responsibilities:

A Developing a proper scenario so that the gaps and

needs of the main stakeholder are captured in the

trial (scenario development);

B Hosting the trial itself using one or more locations

and ensuring that the chosen location is apt to the

purpose of the trial (trial host);

C Directing the trial. The director has a prominent role

in all phases and, as the name suggests, he or she

gives the right directions: for instance, the director

initiates the trial during the actual execution and is

entitled to stop it any time, in case of problems and/

or to put in place mitigation actions;

D Managing the trial-event in terms in logistics (e.g.

rooms and equipment), safety (e.g. make sure that

the people involved in the trial are not in danger),

media (e.g. dealing with the media before and after

the event) and participants (from active to passive

actors: players, observers and guests).

The technical coordinator is responsible for a proper

technical set-up of the trial scenario, so that an ade-

quate assessment of the selected solutions is ensured.

Speciﬁcally, the following three responsibilities should

be covered by the technical coordinator:

A The ﬁrst aspect is the application of the technical

test-bed infrastructure. The technical coordinator

makes sure that the test-bed technical infrastructure

is adjusted according to the decisions taken in the

preparation phase and to the lessons learned during

the rehearsal and that all components work together

smoothly with the trialled solutions. During the trial,

the technical coordinator oversees all technical as-

pects (e.g. integration with legacy tools at the trial

location, data exchange etc).

B This is why the technical coordinator is also in charge

of a proper solution providers management. Solution

providers are actively involved in the development

of the trial, as they know how to best integrate their

solutions in the trial scenarios. Therefore, solution

providers need to participate in relevant meetings

prior to the execution phase so that they can get a

comprehensive overview of the activities. The role of

the technical coordinator does not end at the end of

the trial execution. In fact, the technical coordinator

works closely with the evaluation coordinator to pro-

vide insights on the overall test-bed application.

C Another key responsibility is the training manage-

ment to be provided to the trial participants. The

technical coordinator takes decisions with regards to

the training needs by deciding how to train the play-

ers who actively use the selected solutions during the

trial. To do this, solutions providers must be instruct-

ed and involved in the overall trial design from the

onset.

Regardless of the size of your trial, it is very important

to agree on "who is doing what" beforehand. It's easier

to say so than to put it into practice though, as there

are several aspects to be considered. However, there

are some minimum standards, meaning some key roles

and responsibilities you don't want to skip to ensure that

things go smoothly. Please consider that one person can

fulﬁl one or more responsibilities: based on your time

and your resources, you can decide to have a trial owner

who is responsible for hosting and directing the trial as

well as for following the development of the scenario

and managing the event as such. What is important to

bear in mind is to nourish the brain power needed for

the trial relying on at least on four main roles: trail

owner, technical coordinator, evaluation coordinator

and practitioner coordinator.

A short explanation is provided in the following pages.

4. EVALUATION COORDINATOR

Similar to the practitioner coordinator, the evaluation

coordinator requires a dedicated role because of the im-

portance of executing trials. The overall goal of trials is

a robust assessment of potentially innovative solutions.

In turn, the actual evaluation calls for neutrality, inde-

pendence and an adequate degree of decision-making

power. Therefore, it is recommended to conﬁde the fol-

lowing responsibilities to someone, who is not in charge

for the activities of the other roles.

A In order to ensure a high evaluation quality, the eval-

uation coordinator needs to carefully question and

verify the overall test-bed application from the very

beginning up to the end of a trial. To do so, a close

interaction with the practitioner coordinator is im-

portant. As a next task, an alignment between the

practitioners’s inputs and the trial owner decisions

is needed and should be secured by the evaluation

coordinator. These results need to be communicated

continuously to the technical coordinator, who in

turn should feedback the alignment checks on a reg-

ular basis. In an ideal setup, this might lead to a highly

robust assessment of innovative solutions in realistic

setups. However, reality implies several limitations

like the partial availability of practitioners, an insu-

cient length of the trial execution or inadequate de-

piction of real scenarios in virtual simulations. There-

fore, trade-os need to be done and the evaluation

coordinator plays a key role in balancing costs and

beneﬁts of dierent setups.

B The next responsibility covers the trial evaluation

management. Here, the evaluation coordinator is in

charge of translating the agreed objectives and side

restrictions of the trial dimension into proper metrics

and target values. This task requires a strong collabo-

ration with the trial owner.

C The same applies to the Solution evaluation man-

agement. In this area, the evaluation coordinator is

tasked to transform the solution speciﬁcations, ex-

pressed as solution functions or features according

to the CM taxonomy, into the solution dimension

of the data collection plan. The main collaboration

takes place with the technical coordinator, who

should align the suggested metrics with the involved

solution providers. Their feedback should be prop-

erly incorporated, so that the solutions are assessed

according to what they are supposed or intended

to support. In turn, the evaluation coordinator is in

charge of an adequate feedback of the assessment

results to the solution providers.

D Probably, themost challenging responsibility refers

to the CM evaluation management. Here, the evalu-

ation coordinator relies on a proper input on how the

practitioners perceive the eectiveness of CM op-

erations simulated during the trial. Those deﬁnitions

are key to elicit the “real” impact of a solution on the

CM performance. In consequence, the required CM

practitioner proﬁles need to be communicated in ad-

vance to the practitioner coordinator in order to have

access to this tremendous important basis of a trial.

Another important step during the preparation phase

is to communicate the scenario-related metrics to

the trial owner, in order to ensure an adequate depic-

tion of the actual work practices in the scenario. Last

but not least, the technical coordinator needs to be

informed about which data is required from the test-

bed, so that the relevant data will be collected and

stored in a proper quality, format and amount. Finally,

during the evaluation phase the main task is to relate

the results in the CM dimension to the results in the

trial and solution dimensions. Changes in the CM

performance have to be explained through a proper

sense-making regarding a potential cause-eect rela-

tionship.

3. PRACTITIONER COORDINATOR

the TGM revolves around a practitioner-driven

approach, which is by-design reﬂected in every phase

and step. The term “practitioners” stands for all relevant

CM stakeholders. Starting the selection of potential

solutions with the gap assessment in a speciﬁc CM

practitioner context up to the ﬁnal assessment of the

potentially innovative solutions, it is the practitioner

who has the last word about what should be assessed, in

which context, how and what the results mean from the

practitioners perspective. In order to ensure the prac-

titioner-driven nature of the TGM, a dedicated practi-

tioner coordinator shall serve as a proper guard.

A The ﬁrst responsibility covers the (co-)participation

of CM practitioners in the respective phases and

steps of the TGM application. Here it is key to iden-

tify relevant stakeholders for each trial context. Ide-

ally, the practitioner coordinator should have a CM

background. This would facilitate the identiﬁcation

of the right proﬁles of CM practitioners needed to

develop an as much realistic as possible trial scenario.

Moreover, it would facilitate the identiﬁcation of the

main metrics for the CM dimension. Additionally, a

clear communication of expectations needs to be

ensured, so that all practitioners are aware that their

participation is also needed after the trial execution

to contribute to the sense making and dissemina-

tion of the trial results. The practitioner coordinator

should be very sensitive to eectively request a min-

imum commitment of CM practitioner’s involvement

while respecting the tight side restrictions practi-

tioners have with regards to their daily duties. At the

same time, this role will be regularly confronted with

rather high expectations from the other roles in the

TC, so that a proper translation and communication

of practitioners’ realities becomes vital.

B The second responsibility targets a well-balanced CM

practitioner relationship management. This rather

management oriented task goes beyond the con-

tent-related (co-)participation of CM practitioners,

because it refers to the establishment and mainte-

nance of a pool of practitioners as direct trial partic-

ipants and (indirectly participating) trial observers.

The main functions cover contact management, com-

munication, and reporting tasks.

THE

PANEUROPEAN

TESTBED

TRIAL LOCATIONS

SZKOŁA GŁÓWNA SŁUŻBY POŻARNICZEJ 20

ENTENTE POUR DE LA FORÊT MÉDITERRANÉE 21

VEILIGHEIDSREGIO HAAGLANDEN 22

TRAININGSCENTRE IN ERZBERG 23

One of the DRIVER+ objectives is the development

of a european test-bed for crisis management capa-

bility development. This test-bed consists of physical,

methodological and technical infrastructure elements

to systematically conduct trials and evaluate solutions

within an appropriate environment. In the context of

the project, an “appropriate environment” is a testing

environment where the trialling of solutions is carried

out using a structured, all-encompassing and mutual

learning approach.

The DRIVER+ trials have been conducted at four dier-

ent locations within Europe:

• Szkoła Głowna Służby Pożarniczej (SGSP) -

in Warsaw, Poland

• Centre Euro-méditerranéen de Simulation des Risques

(CESIR) of VALABRE - in Aix-EnProvence, France

• Veiligheidsregio Haaglanden - Safety Region The

Hague County - in The Hague, Netherlands

• Erzberg-Trainingszentrum of the Austrian Red

Cross - in Erzberg, Austria

The vision of DRIVER+ is to create, a pan-European arena

of virtually connected facilities and crisis labs (so called

Centres of Expertise) where users, solution providers,

researchers, policy makers and citizens jointly and iter-

atively can progress on new approaches or solutions to

emerging issues. The Centres of Expertise will be the ﬁnal

depositories and service managers of the DRIVER+ out-

puts. They will act as primary contact points at the nation-

al/regional level for all practitioner-driven organisations

operating in the ﬁeld of crisis management and disaster

risk reduction (or a speciﬁc domain under the latter) in-

terested in using one of more of the DRIVER+ outputs,

supporting them in their capability development and

innovation management. They will make sure local organ-

isations have easy access to such outputs and will provide

guidance and support on how to use them. The Centres

of Expertise can be found and approached via a dedicated

group on the Crisis Management Innovation Network

Europe (CMINE) website: https://www.cmine.eu/topics.

This network is intended to not only facilitate innovation

in CM, but also to generate a European CM culture and

more shared understanding of CM across Europe.