Conference PaperPDF Available

Development of a new evaluation framework for urban and regional mobility projects

Authors:

Abstract and Figures

Transport planners and authorities across Europe use several different methodologies and tools to evaluate mobility investments or measures. The existing approaches, whilst clear in their objectives, demonstrate a wide variation in procedures, coverage of transport modes, comprehensiveness, ease-of-use and flexibility. This paper discusses the development of a new evaluation framework for urban mobility projects. The framework allows policy makers and transport planners to execute an impact assessment in a comprehensive, user-friendly and flexible way. The developed framework consist of three evaluation strands: the assessment of sustainability by a multi-criteria analysis (MCA), the assessment of stakeholder preferences (MAMCA) and the assessment of policy success by target monitoring. Besides these evaluation tools, the framework also consists of a criteria and indicator framework. These are all used within the three evaluation tools to assess the impact.
Content may be subject to copyright.
BIVEC/GIBET Transport Research Day 2015
1
Development of a new evaluation framework
for urban and regional mobility projects
BULCKAEN Jeroen
1
KESERU Imre
2
DONOVAN Ceri
3
DAVIES Huw
4
MACHARIS Cathy
5
Abstract: Transport planners and authorities across Europe use several different
methodologies and tools to evaluate mobility investments or measures. The existing
approaches, whilst clear in their objectives, demonstrate a wide variation in procedures,
coverage of transport modes, comprehensiveness, ease-of-use and flexibility. This paper
discusses the development of a new evaluation framework for urban mobility projects. The
framework allows policy makers and transport planners to execute an impact assessment in a
comprehensive, user-friendly and flexible way.
The developed framework consist of three evaluation strands: the assessment of sustainability
by a multi-criteria analysis (MCA), the assessment of stakeholder preferences (MAMCA) and
the assessment of policy success by target monitoring. Besides these evaluation tools, the
framework also consists of a criteria and indicator framework. These are all used within the
three evaluation tools to assess the impact.
Keywords: sustainable urban mobility, evaluation framework, criteria, NISTO
1. Introduction
The concept of sustainability and sustainable development are evident in essentially all
aspects of life. Sustainable development was first defined in the Brundtland report in 1987 as
“meeting the needs of the present without compromising the ability of future generations to
meet their own needs” (WCED, 1987). Since this concept was first discussed, it has evolved,
resulting in a three-dimensional framework considering economic development,
environmental preservation and social development (Jeon and Amekudzi, 2005; Vermote,
2014). This has proven to be an enduring and compelling concept since it guides policy in a
clear way and it is flexible enough to adapt to new social aspirations, technological
improvements and economic conditions (Goldman and Gorham, 2006).
The ultimate goal many local governments strive for is transport sustainability in their city or
region. Transportation decisions should thus be made in the service of larger and long-term
policy goals, both on local and global aspects (Joumard and Nicolas, 2010). However,
sustainable urban and regional mobility planning is a challenging and complex task. Faced
with a choice of different transport projects, measures and ideas, many decision makers and
transport planners should be given assistance concerning making the right investments. They
must be informed on which projects contribute the most to sustainable development.
1
Vrije Universiteit Brussel, Research Group MOBI
2
Vrije Universiteit Brussel, Research Group MOBI
3
Cardiff University, School of Engineering
4
Cardiff University, School of Engineering
5
Vrije Universiteit Brussel, Department BUTO, Research Group MOBI
BIVEC/GIBET Transport Research Day 2015
2
Several frameworks, methodologies and approaches are currently being used by transport
planners and authorities across Europe to evaluate mobility investments. These tools allow
them to address the complexity of transport sustainability in a comprehensive and structured
way. However, an online survey with 214 responses from stakeholders showed that the
current practices are not always known, easy to use or flexible (Keseru et.al., 2013).
This paper introduces a new evaluation framework for urban and regional mobility projects
and discusses the five steps that underpinned its development. These steps are undertaken in
the NISTO project (New Integrated Smart Transport Options) which is co-funded by the
European Union’s Interreg IVB programme. Firstly, a state of the art analysis of existing
evaluation approaches is described. The next section gives more details on the expectations of
a new framework by future users and how these were met in the development process. The
third section discusses the development of a list of criteria and indicators to execute the
evaluation. Thereafter the organisation of regional workshops is discussed. Section 5
describes the test phase of the framework on pilot and demonstration projects.
2. A state of the art analysis of existing tools
To identify the evaluation approaches that contain suitable methods and metrics to assess the
impact of urban mobility projects, 18 EU and US evaluation schemes were reviewed (see
Table 1). We examined the advantages and disadvantages to check if these schemes or their
structure could be used in the new framework. The schemes were analysed on several aspects.
First the general overview described what the approach was and what final outcome is
generated when using the scheme. Thereafter the data that is needed to execute the evaluation
and the main impact areas were analysed. The experience with the use of the tools was
subsequently also looked up, with focus on the scale of evaluation (city, regional, national,
international), the easiness to apply the tool on other projects or in other North-Western
countries, and whether planners benefit from using these schemes.
APPROACH
TYPE
IMPACT AREAS
EASY TO
APPLY ON
DIFFERENT
PROJECTS?
BESTUFS (Best Urban
Freight Solutions) 1
Framework
Economy, Environment,
Society
No
CALTRANS Smart
Mobility Framework 2
Framework
Location efficiency, Reliable
mobility, Health and Safety,
Environmental stewardship,
Social equity, Robust economy
Yes
CIVITAS (Cleaner and
better transport in cities) 3
Framework
Economy, Energy,
Environment, Society,
Transport
Yes
Collection of Cycle
Concepts 2012 4
Plan/Study
Time costs, Accident costs and
the health effect,
Socioeconomic impact
No
CONDUITS (Coordination
of network descriptions for
urban intelligent
transportation 5
Framework
Traffic Efficiency, Safety,
Social Inclusion, Land Use
No
DfT Guidance on the
appraisal of Walking and
Framework
Environment, Safety,
Economy, Accessibility,
No
BIVEC/GIBET Transport Research Day 2015
3
Cycling schemes 6
Integration
EcoMobility SHIFT 7
Framework
Not fixed
Logic Mapping 8
Method
Depends on project context
London Freight Plan 9
Plan
Economy, Environment, Safety
LSTF Monitoring and
Evaluation Guidance 10
Framework
Primary: Economy, Reduce
carbon emissions
Secondary: Social and
Economic benefits, Safety, Air
Quality, Physical Activity
MAESTRO (Monitoring,
Assessment and Evaluation
of transport policy options
in Europe) 11
Guidelines
Transport System, Economic
Efficiency, Safety and Security
MAMCA (Multi-Actor,
Multi-Criteria Analysis) 12
Methodology
Depends on project context &
involved stakeholders
MAXSUMO (Successful
travel awareness campaigns
and mobility management
strategies) 13
Methodology
not stated
MCA (Multi-Criteria
Analysis) 14
Methodology
Depends on project context
PROSPECTS (Procedures
for recommending
sustainable optimal
planning of European City
transport systems 15
Guidebook
Economic Efficiency,
Liveability, Environment,
Equity, Social Inclusion,
Safety, Economic Growth
SCBA (Social Cost-Benefit
Analysis) 16
Methodology
Depends on project context
SUMMA (Sustainable
Mobility, Policy Measures
and Assessment) 17
Framework
Economy, Environment,
Society
Urban Freight
Consolidation Centre 18
Study
Logistics and supply chain
changes, Social and
environmental impact, goods
vehicle activity, Loading or
unloading activity
Table 1 : Overview of the methodological review of EU and US schemes (summary of the review reported
by Donovan et.al. (2013; 2014))
Sources: 1: Dasburg and Schoemaker (2008); 2: Californian Department for Transport (2010); 3: van
Rooijen and Nesterova (2013); 4: Andersen et. al. (2012) ; 5: Kaparias and Bell (2011); 6: Department for
Transport (2010); 7: Kodukula et. al. (2013); 8: Tavistock (2010) ; 9: Allen, Browne and Woodburn
(2012); 10: Department for Transport (2011); 11: Transport and Travel Research et. al. (1999); 12:
Macharis (2005) ; 13: Hyllenius et. al. (2009); 14: Department for Communities and Local Government
(2009); 15: May et. al. (2001); 16: De Witte et al (2011); 17: Ahvenharju et. al. (2004); 18: Browne et.
al. (2005).
The list of reviewed schemes contained many types of evaluation approaches such as
methodologies, frameworks, guidebooks, studies and plans. This also means that there are
different outcomes of evaluation. No consistency can be detected in what the final outcome of
the evaluation is. Some of the evaluation schemes provide scores, monetary values or
descriptions on what the effects might be. This difference is related to the aim of evaluation.
BIVEC/GIBET Transport Research Day 2015
4
Not all schemes have the same goal to evaluate a plan, measure or project. This diversity of
outcomes is however beneficial for transport planners and policy makers when they would
make use of more than one evaluation scheme. For this reason we consider a combination of
evaluation schemes in the framework. This creates benefits for the users since there is a huge
variety in specifications of transport and mobility projects on an urban and regional scale.
The difference in outcomes of evaluation is not only related to the expectations of the
evaluation. It is also dependent on the impact areas considered. Table 1 shows that no
consensus can be detected towards one standardised combination of impact areas. The
outcomes of evaluation tend thus to be based on various definitions of sustainability.
However, we see that the three pillars of sustainability (economy, environment and society)
are often part of the evaluation structure. This shows the potential to use the three pillar
concept as a breakdown to assess transport and mobility sustainability. Economy,
environment and society are hence considered as the three building blocks of the framework,
which can easily be understood by practitioners (Jeon and Amekudzi, 2005).
Looking at the data requirements of the schemes, we learned that there is a difference between
the use of qualitative and quantitative data. Not all approaches allow the combination.
However, when it is possible to use several sources of data, the evaluation scheme can be
easily adapted to other projects than those it was developed for. Another benefit of using
qualitative and quantitative data is the ease of data collection. For small-scale projects it is
often not possible to set up complex models which calculate impacts. Using a qualitative scale
makes it easier to collect data on a specific criterion and integrate the change in the
evaluation. For these two important reasons, which improve the adoption by practitioners, it
was decided that the NISTO framework should integrate both qualitative and quantitative data
sources.
3. An online survey to understand the needs and requirements of potential users
The new evaluation framework is being developed with stakeholder participation as a key
element in the process. Understanding the needs and requirements of potential users and adapt
it accordingly, enhances the possible adoptation. An online survey was set up since this makes
it possible to reach a large number of stakeholders at the same time. These are defined as
individuals from governments (national, state, regional or local level), public transport
operators, mobility organisations, NGO’s, research institutes, companies and other social
groups, living in North-West Europe (Belgium, France, Germany, the Netherlands and the
United Kingdom). The objective of the survey was to collect information about the experience
with and the knowledge of different evaluation methodologies and to understand how the
stakeholders think a new toolkit should look like and which features it should retain.
A response rate of 29.16% was achieved. The analysis of the 214 completed questionnaires
showed that the Social Cost-Benefit Analysis (SCBA) and the Multi-Criteria Analysis (MCA)
are the two most-known evaluation methods (40% and 38%). Besides the knowledge, they are
currently also being used by the stakeholders who completed the survey (20% use SCBA,
17% use MCA). Apart from the knowledge and usage of evaluation tools, the survey revealed
that the current practices are not always known or easy to use by practitioners. A new
evaluation framework should combine the strengths of existing evaluation tools and comply
with the following factors:
- it should allow non-experts to use it,
- it should allow the evaluation of different alternatives or scenarios,
- it should allow the use of a wide range of evaluation criteria and indicators,
BIVEC/GIBET Transport Research Day 2015
5
- it should be user-friendly,
- it should integrate qualitative and quantitative data,
- it should not be time-consuming,
- it should provide a balance between a flexible and standardized approach,
The stakeholders also recommended to involve several stakeholder groups in the evaluation
process as their preferences can be significantly different. Furthermore, the new evaluation
framework should be disseminated and explained in a clear way that the stakeholders will
understand it and will use it.
These prerequisites are considered in the development of the new framework. Since the
SCBA and the MCA are well-known and apply to most of the needs and demands, these
evaluation strands were further investigated for considering them as basic elements of the
NISTO framework. We wanted to simplify the SCBA to realign to the easy-of-use by
practitioners. Defining the evaluation period where effects are considered, fixed criteria to
valuate effects, standardised key values to monetize the impacts and attribute costs and
benefits would make it easier to execute a SCBA. Due to the loss of a qualitative outcome of
a simplified evaluation by SCBA, this was not anymore considered to be part of the
evaluation framework. Executing a simplified SCBA could mean that important impacts are
excluded from the evaluation and thus lead to misleading outcomes to support decision
makers. A second evaluation method is the MCA. This is a well suited approach for
evaluation as it allows for the consideration of conflictual, multidimensional,
incommensurable and uncertain effects in decision making (Martinez-Alier, Munda and
O’Neill, 1998). Macharis and Bernardini (2015) found that the use of MCA is also increasing
within the field of transport and mobility, because conventional tools like the Cost-
Effectiveness Analysis (CEA) or the Cost-Benefit Analysis (CBA) are complex and
inadequate to capture the full range of project impacts. The functionalities of a MCA meet
most of the requirements for a new framework, as stated by the respondents of the survey, and
is therefore considered as the first evaluation strand of the framework. To make sure the
structure of this method is understandable and easy to apply on projects with different scopes,
the concept of sustainability is considered. The three dimensions of sustainability (economy,
environment and society) are equally important. In that way the multi-dimensional impacts
are balanced and a clear view on the level of project sustainability can be generated. This
approach also avoids influence in the persuasion of sustainable development by governmental
bodies which finance the project.
Recently a shift towards more integrated and participative transport planning is found and
advised when setting up a Sustainable Urban Mobility Plan (SUMP)(Wefering et al., 2014).
The engagement and involvement of stakeholders has thus become increasingly important.
The respondents of the survey mentioned this as well . The stakeholders should be integrated
in all stages of the decision making process (Booth and Richardson, 2001). As shown by
Macharis et. al. (2009; 2012), a combination of multi-criteria decision analysis techniques and
participatory methods, like the Multi-Actor, Multi-Criteria Analysis (MAMCA) develop by
Macharis (2000; 2004) is well suited to involve stakeholders in the decision making process.
To provide an objective assessment of the impacts of certain projects or measures, preferences
from stakeholders should be integrated in the outcome of evaluation. For this reason the
NISTO framework will consider stakeholder preferences in the evaluation by including the
MAMCA in the framework.
Target monitoring is the third evaluation method that will be included in the framework. This
because policy makers often want to execute a quick “on the spot-check” whether a plan or
BIVEC/GIBET Transport Research Day 2015
6
measure will achieve the objectives and thus will be successful. This step is part of the
process of planning a SUMP. Based on the interim results or final outcomes and the
predefined targets, it can be easily detected whether a project was successful or not. Such a
quick evaluation method with no fixed structure meets the prerequisites of being flexible and
easy to use by policy makers across Europe. Hence it is integrated in the framework.
4. Developing a list of evaluation criteria and indicators
We reviewed 16 case studies from across North-West Europe to find best practices for the
definition and measurement of evaluation indicators. A spreadsheet was compiled with a list
of about 400 indicators. For each one a description and the data collection requirements such
as units and data sources were defined. The indicators were then reviewed. Similar indicators
were grouped and counted for each pillar of sustainability. It became clear which indicators
were mostly used. The case studies make the most use of biodiversity/habitats and CO2-
emissions for the environmental pillar. The economic activity in the city and the investments
are the two mostly used indicators to assess economic impacts while crime/security and safety
are used in most of the case studies.
For each pillar of sustainability the most frequently used indicators were reviewed on the
completeness of the definition, the data requirements and the applicable units. In most cases
indicators were chosen so that it should be feasible for small-scale projects to collect them.
Since it was not possible to attribute some indicators to one specific pillar of sustainability,
we created a fourth group of indicators. These are transport indicators which are used as
proxies to measure or calculate other criteria where data is unavailable. Some examples of
such transport proxy indicators are the distribution of passenger kilometres travelled by mode,
the number of freight vehicle movements, the number of journeys per mode and the level of
traffic congestion in the city during peak or off-peak hours.
This procedure resulted in an initial list of indicators that was reviewed by nine finished
transport and mobility projects and by stakeholders in the regional workshops. Based on their
feedback, this list was updated. It was possible to group some of the indicators under similar
criteria. In the end, the framework consists of 16 core criteria, which are subdivided under one
of the three pillars of sustainability (Table 2). The criteria are linked to 20 core and 15
optional indicators. The framework also considers a fourth group of transport proxy
indicators.
ECONOMY
ENVIRONMENT
SOCIETY
Economic activity
Land consumption
Safety
Cost effectiveness
Greenhouse gas emissions
Security
Reliability and travel time
Air quality
Health of citizens
Public funding of transport
Resource use
Liveability
Noise
Equity
Socio-political acceptance
Accessibility for people
with special needs
Table 2 : The NISTO evaluation criteria grouped under the three pillars of sustainability
This list of evaluation criteria and indicators is used to measure and calculate the impact of
mobility projects. The user of the framework can choose to a limited extend which criteria
and indicators he will integrate in the evaluation methods within the framework. This meets
BIVEC/GIBET Transport Research Day 2015
7
the need to be flexible and to have the possibility to use a wide range of criteria and
indicators.
5. Organising regional workshops to collect stakeholder feedback
In the development process regional workshops were organised to increase stakeholder
participation. In these events stakeholders from the five participating countries could give
feedback and suggestions. The first workshops took place in an early stage of the project,
when the concept of the framework was being developed. The second stakeholder workshop
was organised to get more feedback from stakeholder on their preferences for the evaluation
of the demonstration projects within NISTO.
The aim of the first regional workshops was to collect input for the development of the
toolkit. Participants discussed the existing evaluation tools on their benefits, disadvantages
and risks of using them for the evaluation of mobility projects. The stakeholders expressed the
need for more information about the existing tools. They are not experienced enough with
evaluation methods to choose the right method for the right project. In the workshops there
was also a demand for a flexible evaluation approach. The developed evaluation tool should
be adjustable to the complexity of the project and it should be possible to involve and define
several stakeholder groups within the evaluation. For these reasons, the framework will be a
combination of evaluation methods: MCA, MAMCA and Target Monitoring.
The second stakeholder workshops were focussed on the regional demonstration projects. The
stakeholders that were or will be affected or influenced by the project were invited. During
registration through an online survey, the participants agreed on or gave suggestions on the
specific preferences for their stakeholder groups, defined by the project managers. Citizens
for instance could mention that the air quality is important to them. In this way, a final list of
evaluation criteria was set up, specifically related to all stakeholder groups for the
demonstration project. At the workshop itself, the participants completed pairwise
comparisons for their evaluation criteria, for all possible combinations of two criteria. In these
paper questionnaires participants indicated which of the two compared criteria they found
more important. The weights of the criteria was collected by this method. These will be used
in the actual evaluation of the demonstration projects in summer 2015, as soon as monitoring
data is available.
6. Testing the new framework on pilot and demonstration projects
In an earlier stage of the development process, a list of evaluation indicators was defined. The
feasibility of measuring, quantifying and evaluating these indicators in the new framework
was tested on nine projects that have been completed. Project managers of these pilot projects
were asked to check if they were able to measure or calculate these indicators by their project
data. The feedback showed that the demonstration projects were able to measure most of the
indicators in a direct way. If it was not possible to measure the impact of the indicator in the
way as it was defined, mostly some basic calculations could be done to make sure the pilot
project data would meet the pre-set requirements (e.g. calculate the weekly amount of
passenger kilometres for bicycle travel). Based on the outcomes of the ability to measure the
impacts of a certain indicator, to use of the indicators in the framework was reviewed. In that
way a final list of evaluation indicators was set up, based on the performance of these pilot
projects.
BIVEC/GIBET Transport Research Day 2015
8
Currently, the framework is being tested on five demonstration project across Europe which
try to influence travel behaviour: mapping and influencing travel behaviour through a
smartphone app in Leuven (Belgium), a bicycle rental scheme in Boulogne-sur-Mèr (France),
an integrated tourist ticket in the region of Kassel (Germany), improving cycling connections
in the province of Noord-Brabant (the Netherlands) and intelligent information provision for
transport users in Wolverhampton (the United Kingdom). By monitoring and evaluating these
projects, the ability of the framework to deal with different kind of projects and data sources
is evaluated. For all the criteria we will find out if the demonstration projects are able to
measure the indicators. Based on these outcomes, the list of criteria and indicators is finalised
and will be considered in the development of an easy-to-use online toolkit, which can be used
be transport planners and authorities to evaluate their urban or regional mobility projects.
7. Discussion and conclusions
This paper describes the steps that are undertaken in the development of a new evaluation
framework for urban and regional mobility projects, which is shown in Figure 1. The
fundamentally new aspect is the usage of best practices and involving stakeholders in the
process to make sure that the framework is easy to use, flexible and meets the demands of
those future users.
The review of 18 EU and US schemes showed that many evaluation methods suit different
project types. In current practice, the choice to use a specific scheme is mainly based on the
expectations of the outcome. Depending on what a policy makers or transport planner wants
to know decides which scheme he would use. This concept is considered in the NISTO
framework since we chose to include three evaluation tools. Evaluators are allowed a
flexibility to choose from a Multi-Criteria Analysis to assess project sustainability, a Multi-
Actor, Multi-Criteria Analysis to understand stakeholder preferences and Target monitoring
to assess policy success.
Sustainability is achieved in this framework since the three pillars of sustainability (economy,
environment and society) are considered as the building blocks for the evaluation tools. It was
found that these often can be detected in existing evaluation tools and thus can be understood
by practitioners. When evaluating a project with one of these three defined tools, the user is
allowed a flexibility since he can combine the use of qualitative and quantitative data sources
to measure and calculate the impact. All of these features of the framework meet the
requirements of stakeholders, as derived from an online survey.
The evaluation criteria and indicator framework was built on best practices which are being
used across North-West Europe. Stakeholders had the chance to give feedback in regional
workshops. In these events they could express their prerequisites of a new tool. Including the
preferences on what practitioners expect from a new framework, should entail the usage of the
developed evaluation framework.
The new evaluation framework was tested on completed mobility projects and is now being
tested on ongoing projects which try to influence travel behaviour, from across North-West
Europe. This approach makes sure that the framework will be able to execute an evaluation of
projects with different scopes and goals. When the testing on the demonstration projects is
finished, the framework will be adjusted if needed and integrated in an online toolkit.
Herewith, transport planners, authorities from different countries, mobility organisations,
transport operators, transport consultants or other individuals or organisations will be able to
select one or more of the evaluation methods to evaluate mobility-related projects across
BIVEC/GIBET Transport Research Day 2015
9
several transport modes and at different geographical levels. In this way decision making and
strategy building is supported by a time and resource efficient way.
Figure 1 : The NISTO evaluation framework
8. Acknowledgements
The development of this evaluation framework was realized with the financial support of the
Interreg IVB programme, the Flemish Government and the Vrije Universiteit Brussel. The
authors wish to thank the NISTO partnership for the collaboration.
Bibliography
Ahvenharju, S., Könnölä, T., van Grol, R., Walker, W., Klautzer, L., Röhling, W., Burg, R. et
al., SUMMA Operationalising Sustainable Transport and Mobility: The System
Diagram and Indicators. Deliverable 3, 2004
Allen, J., Browne, M. and Woodburn, London Freight Data Report: 2012 Update, Transport
for London, 2012
Andersen, T., Bredal, F., Weinreich, M., Jensen, N., Riisgaard-Dam, M. and Kofod Nielsen,
M., Collection of Cycling Concepts 2012. Cycling Embassy of Denmark, 2012
Booth, C. and Richardson., T., Placing the Public in Integrated Transport Planning,
Transport Policy 8 (2), pp. 141149, 2001
Browne, M., Sweet, M., Woodburn, A. and Allen, J., Urban Freight Consolidation Centres,
Department for Transport, 2005
Dasburg, N. and Schoemaker, J., Best Urban Freight Solutions II. Quantification of Urban
Freight Transport Effects II. BESTUFS II Deliverable 5.2, 2008
Department for Communities and Local Government, Multi-Criteria Analysis: A Manual,
London: Department for Communities and Local Government, 2009
BIVEC/GIBET Transport Research Day 2015
10
Department for Transport, Guidance on the Appraisal of Walking and Cycling Schemes.
Transport Analysis Guidance Unit 3.14.1, 2010
———Creating Growth, Cutting Carbon. Making Sustainable Local Transport Happen,
2011
Donovan, C.A., Keseru, I., Bulckaen, J., Macharis, C., Van Raemdonck, K. and Bernardini,
A., Framework Development and Indicator Analysis Based on Best Practice. NISTO -
Work Package 1 Action 2. Final Report, 2014
Donovan, C.A., Macharis, C., Bulckaen, J., Van Raemdonck, K., Phillips, D., Mercier, D.,
Keseru, I., Glaser, N., Bernardini, A. and Davies, H., Evaluation of Approaches and
Recommendations. Project Deliverable for the NISTO Work Package 1 Action 1, 2013
Goldman, T. and Gorham, R., Sustainable Urban Transport: Four Innovative Directions,
Sustainable Cities 28 (12), pp. 261273, 2006
Hyllenius, P., Smidfelt Rosqvist, P., Haustein, S., Welsch, J., Carreno, M. and Rye, T.,
MaxSumo: Guidance on How to Plan, Monitor and Evaluate Mobility Projects, 2009
Jeon, C. M. and Amekudzi. A., Addressing Sustainability in Transportation Systems:
Definitions, Indicators, and Metrics, Journal of Infrastructure Systems 11 (1), pp.
3150, 2005
Joumard, R. and Nicolas, J-P., Transport Project Assessment Methodology within the
Framework of Sustainable Development, Ecological Indicators 10 (2), pp. 136142,
2010
Kaparias, I., and Bell, M.G.H., Key Performance Indicators for Traffic Management and
Intelligent Transport Systems, 2011
Keseru, I., Bulckaen, J., Macharis, C., Franchois, E., Van Raemdonck, K., Dobruszkes, F. and
Bernardini, A., NISTO New Integrated Smart Transport Options: Report on the
Results of the Online Stakeholder Survey, 2013
Kodukula, S., Carreno, M., Vleugels, I., Backx, K., Clarck, A., Neergard, K., Evanth, K.,
Ivanova, E., Hudak, K. and Kirkels, M. EcoMobility SHIFT. Manual for Auditors and
Advisors, 2013
Macharis, C. Strategische Modellering Voor Intermodale Terminals. Socio-Economische
Evaluatie van de Locatie van Binnenvaart/weg Terminals in Vlaanderen. PhD
Dissertation, Vrije Universiteit Brussel, 2000
Macharis, C. and Bernardini, A., Reviewing the Use of Multi-Criteria Decision Analysis for
the Evaluation of Transport Projects: Time for a Multi-Actor Approach, Transport
Policy 37, pp.177186, 2015
Macharis, C., de Witte, A. and Ampe, J., The Multi-Actor, Multi-Criteria Analysis
Methodology (MAMCA) for the Evaluation of Transport Projects: Theory and
Practice, Journal of Advanced Transportation 43 (2), pp. 183202, 2009
Macharis, C., Turcksin, L. and Lebeau, K., Multi Actor Multi Criteria Analysis (MAMCA) as
a Tool to Support Sustainable Decisions: State of Use, Decision Support Systems 54
(1), pp. 610620, 2012
Macharis, C., Verbeke, A. and De Brucker, K. The Strategic Evaluation of New Technologies
through Multicriteria Analysis: The Advisors Case, Research in Transportation
Economics, edited by E. Bekiaris and Y. Nakanishi, 8, Economic Impacts of
Intelligent Transportation Systems: Innovations and Case Studies, pp. 443462,
2004
Martinez-Alier, J., Munda, G. and O’Neill, J., Weak Comparability of Values as a Foundation
for Ecological Economics, Ecological Economics 26 (3), pp. 277286, 1998
May, T., Jarvi-Nykanen, T., Minken, H., Ramjerdi, F., Matthews, B. and Monzón, A.,
Procedures for Recommending Optimal Sustainable Planning of European City
Transport Systems: Deliverable 1 - Cities’ Decision-Making Requirements, 2001
BIVEC/GIBET Transport Research Day 2015
11
Transport and Travel Research, ARISE, VTT, NEI, BPV, DITS, GSF, et al., Monitoring
Assessment and Evaluation of Transport Policy Options in Europe, Transport and
Travel Research Ltd., 1999
Van Rooijen, T. and Nesterova, N., CIVITAS WIKI Applied Framework for Evaluation in
CIVITAS PLUS II, 2013
Vermote, L., Sustainable Mobility and Decision-Making. Participatory Assessment of
Mobility Policies in Belgium and Palestine, PhD Dissertation, Vrije Universiteit
Brussel, 2014
WCED, Our Common Future. World Commission on Environment and Development, Oxford
University Press, 1987
Wefering, F., Rupprecht, S., Buhrmann, S. and Böhler-Baedeker, S. Developing and
Implementing a Sustainable Urban Mobility Plan, European Commission, 2014
... Several stakeholders' groups were engaged in the process, and although their preferences varied, most groups considered state funding and subsidies as the most effective policy, along with traffic demand management. Bulckaen et al. (2015) proposed a framework for ranking three small-scale urban and regional mobility projects that include policies different in theme, country, and objectives. The framework was a combination of MCDA to assess the sustainability of the projects and MAMCA to assess stakeholder preferences. ...
Chapter
Full-text available
Transport and tourism are rising economic sectors based on their mutual growth on reliable technological tools, affordable energy sources in relatively peaceful decades. This growing trend, faced along years of sudden slowdown caused mainly by the financial and health crisis; one of the most severe and recent episodes was the COVID-19 pandemic. The pandemic outbreak is representing a global turning point for both international markets of tourism and transport which can be addressed with innovative solutions looking to retake the growing trends. In such a changing framework CIVITAS DESTINATIONS European project addressed the main challenges of tourism and resident mobility at local (rural, urban area, large city) and regional levels, focusing on six European islands (Malta, Cyprus, Elba, Crete, Gran Canaria,Madeira) aiming to implement and evaluate 83 different sustainable mobility measures. The project proves to have a cumulative relevant impact on environmental indicators such as CO2 emission avoided, fossil fuel saved and energy saved. Additionally, the project findings focus on the quality of life and economic development to grant long-run environmental and financial sustainability of tourism and transport at the local and regional levels. The main aspects observed were related to public participation and governance models in touristic mobility integrated with the residents’ needs, touristic mobility the island, role of technologies to focus on final user needs, growing trends of elder and disabled people, new integrated and innovative business in tourism and mobility markets, health at the core of future trends.
... Several stakeholders' groups were engaged in the process, and although their preferences varied, most groups considered state funding and subsidies as the most effective policy, along with traffic demand management. Bulckaen et al. (2015) proposed a framework for ranking three small-scale urban and regional mobility projects that include policies different in theme, country, and objectives. The framework was a combination of MCDA to assess the sustainability of the projects and MAMCA to assess stakeholder preferences. ...
Chapter
Full-text available
This chapter presents the assessment of 11 sustainable urban mobility measures according to 10 criteria for European medium-sized touristic cities, using multi-criteria decision-making. The study includes the viewpoint of six different European stakeholder groups, identifying their interests and comparing their ranking on appropriate mobility measures. It was found that the majority of stakeholders give the highest priority to the wellbeing of local communities and the quality of life, despite the economic implications of services and the potential impact on incoming tourism. Mostly they emphasise on at least two out of five criteria categories: Society and Environment or Society and Mobility. Tourism stakeholders showed a high preference for environmental criteria, demonstrating the continuously raising awareness on the links of tourism and environment. “Mobility management and travel plans” policy was the most popular policy amongst all groups, indicating that the provision of information, personalised plans, and smart applications can increase the use of sustainable mobility modes and have a significant positive impact in all examined categories. Overall, the multi-criteria analysis performed in this study can be a valuable tool for decision-makers during the shaping of future policies for sustainable mobility in urban tourist destinations, considering numerous parameters and stakeholders’ viewpoints. Moreover, it can be further developed and adapted to specific needs.
... The aim was to ensure that all relevant evaluation criteria for sustainability for urban and regional mobility are included covering all three sustainability pillars. (Bulckaen et al., 2015a). These criteria were weighted by 93 representatives of decision makers in the mobility domain from Belgium, France, Germany, the Netherlands and the United Kingdom. ...
Article
Full-text available
While sustainability of transport projects is of increasing importance, the concept of sustainability can be understood in many different ways by the stakeholders that are involved in or affected by mobility projects. In this paper, we compare the outcomes of the assessment of sustainability of projects through a multi-criteria analysis (MCA) and the appraisal of stakeholder preferences through the multi-actor multi-criteria analysis (MAMCA). Evaluating projects with both tools and comparing the outcomes can provide insight into the stakeholder support of sustainable solutions and the sustainability of alternatives preferred by stakeholders. The sustainability of projects is assessed through 16 criteria grouped under the three pillars of sustainability. They were selected by in-depth review of 16 case studies of mobility projects, 18 transport evaluation schemes and the ranking of potential criteria by 214 stakeholders in North-West Europe. These criteria were weighted by 93 representatives of decision makers in the mobility domain. Stakeholder preferences were appraised through the criteria identified for each stakeholder group. We illustrate the framework by evaluating alternative solutions to improve cycling connections between the towns of Tilburg and Waalwijk in the Netherlands. The results of the comparison show that stakeholder preferences are biased towards one or two of the sustainability pillars (economy, environment, society) in three ways: through the selection of the criteria by the stakeholders, the weights of each criterion by each stakeholder group and differences in the final ranking of alternatives between the stakeholder groups and the MCA.
... This list was derived from (i) a review of existing evaluation approaches and best practices, (ii) a stakeholder survey, (iii) feedback from stakeholders across North-West Europe in regional workshops and (iv) the analysis of assessment procedures for nine completed mobility projects (pilot projects). In the first step, 18 existing evaluation approaches and 16 case studies were analysed on which indicators were commonly used and how these were defined (see Bulckaen, Keseru, Donovan, Davies, & Macharis, 2015;Donovan et al., 2014). In the second step, we asked stakeholders in an online survey which criteria and indicators should be integrated in a framework for small-scale mobility projects (see Keseru et al., 2013). ...
Article
Public participation is becoming more and more important in transport planning and decision-making. However, the concept of sustainable mobility can be understood in different ways by the stakeholders and the current practice of participation is often limited. This paper presents a framework which makes it possible to evaluate, compare and understand the ranking of the alternatives for small-scale urban and regional mobility projects, assessed on sustainability and stakeholder preferences. We apply rank correlation to detect synergies in evaluation outcomes, in order to understand if there is concordance between the rankings. We also analyse if the stakeholder preferences, calculated by the multi-actor, multi-criteria analysis (MAMCA) were similar to the sustainability assessment by multi-criteria analysis (MCA). The results of the application of the framework on three demonstration projects indicate that most stakeholders have similar preferences compared to the project alternatives' ranking of sustainability and that any of the interventions would be more sustainable compared to the business as usual.
Conference Paper
Full-text available
In recent research work (FP7 CONDUITS) a performance evaluation framework for traffic management and Intelligent Transport Systems was developed. The new framework consists of a set of Key Performance Indicators (KPIs) for the strategic themes of traffic efficiency, safety, pollution reduction and social inclusion, and the last stages of the project saw its validation through its application to four case studies. Following up from this work, this paper presents the extension of the framework for use as a prediction tool enabling urban transport authorities to assess the impacts of relevant policies and technologies before implementing them. Focussing on pollution reduction, a tool (CONDUITS-DST) integrating the respective KPIs with microsimulation modelling is developed. The paper describes the integration process, including the model chosen for calculating the emissions levels of a number of scenarios, presents the results of the application to a case study in the city of Brussels, and outlines future developments targeted at broadening the integration of the KPIs into decision-making
Article
Full-text available
Addressing the sustainability of transportation systems is an important activity as evidenced by a growing number of initiatives around the world to define and measure sustainability in transportation planning and infrastructure provision. This paper reviews major initiatives in North America, Europe, and Oceania. The purpose is to characterize the emergent thinking on what constitutes transportation sustainability and how to measure it. While there is no standard definition for transportation system sustainability, it is largely being defined through impacts of the system on the economy, environment, and general social well-being; and measured by system effectiveness and efficiency, and the impacts of the system on the natural environment. Frameworks based on important causal relationships between infrastructure and the broader environment, infrastructure impacts on the economy, environment, and social well-being; and the relative influence of agencies over causal factors, are largely being used to develop and determine indicator systems for measuring sustainability in transportation systems. Process-based approaches involve community representatives and other stakeholders in planning and present opportunities to educate the public and influence collective behaviors. These frameworks can be used collectively to help agencies refine their visions as well as develop policies, planning procedures, and measurement and monitoring systems for achieving sustainable transportation systems.
Article
Full-text available
The main argument of this paper is that weak comparability of values should be seen as one characteristic feature of ecological economics. The formal properties of the concepts of strong comparability (implying strong or weak commensurability) and weak comparability (implying incommensurability) will be clarified. Multicriteria evaluation offers the methodological and mathematical tools to operationalize the concept of incommensurability at both macro and micro levels of analysis. The concept of incommensurability of values already has a long tradition in economics; moreover, we will show that analytic philosophy, theories of complexity, post-normal science and the recent theories of rationality lead with different trajectories to a non-algorithmic approach which, in our view, could be implemented by some forms of multicriteria evaluation.
Article
Full-text available
This paper examines the concept and implementation of sustainable transport. It traces efforts to define and operationalize the notion of sustainable transport in the urban context, noting that these efforts have tended to fall into two broad clusters of work: those that envision sustainable transportation as a policy pathway, and those that envision it as a policy end-point. The authors argue that to be successful, sustainable transport policy must avoid the common transportation policy pitfall of ignoring the larger systems in which transportation activity is embedded. The goal of sustainable transportation may be better served by a number of the organic innovations in transportation practice that are occurring in the field. The authors identify four emerging areas of innovation: New Mobility, City Logistics, Intelligent System Management, and Livability. Finally, the authors discuss the extent to which these innovations represent a more systems-oriented approach, and the institutional challenges inherent in these proposals.
Article
In this paper we give an overview of the use of Multi-Criteria Decision Analysis (MCDA) for transport project appraisal. The aim of this review is to provide an outline of the increasing use of MCDA methods in the evaluation of transport projects. We investigate for which kind of transport decisions the MCDA methods are applied. The review consists of identifying the transport related subjects, the interconnected arising decision problems and the kind of representative MCDA method(s) used for transport project evaluations. This review allowed deriving a general frame for the evaluation of transport projects. One of the conclusions resulted in the importance of integrating stakeholders in the decision process not yet very common in the transport projects that were reviewed. The Multi-Actor Multi-Criteria Analysis (MAMCA) approach is suggested as a direction for further research. The MAMCA methodology has already proven its usefulness in several transport related decision problems enabling to involve the stakeholders explicitly in the decision process.
Article
In this contribution, the multi actor multi criteria analysis (MAMCA) to evaluate transport projects is presented. This evaluation methodology specifically focuses on the inclusion of the different actors that are involved in a project, the so-called stakeholders. Like the traditional multi criteria decision analysis (MCDA), it allows including qualitative as well as quantitative criteria with their relative importance, but within the MAMCA they represent the goals and objectives of the multiple stakeholders. As such, the stakeholders are incorporated in the decision process. The theoretical foundation of the MAMCA method is shown, together with several applications in the field of transport appraisal.
Article
Faced with a choice of different transport projects, such as road or rail infrastructure projects, which project is the most sustainable? We suggest a relatively simple and transparent evaluation method for such projects. First, transport issues within the sustainable development framework must be addressed, bearing in mind the strong meaning of the term, which is the only concept allowing environmental issues to be taken seriously into account. It also means linking local and global aspects, long and short terms, and thus to specify the time and geographical scales of projects and their impacts. Secondly, we put forward these main principles as evaluation criteria enabling the sustainable development concept to be made operational. We suggest three economic criteria, four social criteria and eleven environmental criteria, in addition to an aggregation method for these criteria integrating the social or political preferences of decision-makers or their representatives. A concern that is often significant in transport projects is the future of traffic mobility, whose main parameters we have analysed in order to put forward probable scenarios, which form the basis for applying the criteria listed above.
Article
In this paper the multi-actor multi-criteria analysis (MAMCA) method to evaluate transport projects is presented. This evaluation method specifically focuses on the inclusion of qualitative as well as quantitative criteria with their relative importance, defined by the multiple stakeholders, into one comprehensive evaluation process in order to facilitate the decision making process by the different stakeholders. The MAMCA methodology is introduced by an overview of other evaluation methods for transport projects in the past and is illustrated by means of two practical cases. The introduction will lead us to the theoretical conception of the MAMCA method where we draw the attention to the proven usefulness of the MAMCA for the evaluation of transport projects and the inclusion of different kinds of stakeholders, individuals as well as groups, into the evaluation process.
Article
The authors of this paper, economic experts participating in the ADVISORS project, a large scale, pan-European study co-funded by the European Union (2000-2002), have developed a strategic evaluation methodology building upon multicriteria analysis (MCA), to assess advanced driver assistance systems (ADAS). ADAS is a set of new technologies intended to fundamentally improve road safety in the European Union. This paper includes a description of this new MCA methodology as it was applied to ADAS. This new MCA methodology is based on the analytic hierarchy process (AHP) of Saaty and synthesises information from both stakeholder analyses and technical performance studies.