Content uploaded by Dmitry O. Gorodnichy
Author content
All content in this area was uploaded by Dmitry O. Gorodnichy on Apr 12, 2015
Content may be subject to copyright.
Video Analytics technology: the foundations,
market analysis and demonstrations
Prepared by:
Dmitry O. Gorodnichy, Jean-Philippe Bergeron, David Bissessar, Ehren Choy, Jacques Sciandra
Canada Border Services Agency
Ottawa ON
Canada K1A 0L8
Contract Scientific Authority: Pierre Meunier
DRDC Centre for Security Science
613-992-0753
The scientific or technical validity of this Contract Report is entirely the responsibility of the
Contractor and the contents do not necessarily have the approval or endorsement of the
Department of National Defence of Canada.
Contract Report
DRDC-RDDC-2014-C251
September 2014
IMPORTANT INFORMATIVE STATEMENTS
PROVE-IT (FRiV) Pilot and Research on Operational Video-based Evaluation of
Infrastructure and Technology: Face Recognition in Video project, PSTP 03-401BIOM,
was supported by the Canadian Safety and Security Program (CSSP) which is
led by Defence Research and Development Canada’s Centre for Security Science, in
partnership with Public Safety Canada. Led by Canada Border
Services Agency partners included: Royal Canadian Mounted Police, Defence
Research Development Canada, Canadian Air Transport Security Authority,
Transport Canada, Privy Council Office; US Federal Bureau of Investigation,
National Institute of Standards and Technology, UK Home Office; University of
Ottawa, Université Québec (ÉTS).
The CSSP is a federally-funded program to strengthen Canada’s ability to anticipate,
prevent/mitigate, prepare for, respond to, and recover from natural disasters, serious
accidents, crime and terrorism through the convergence of science and technology with
policy, operations and intelligence.
© Her Majesty the Queen in Right of Canada (Department of National Defence), 2014
© Sa Majesté la Reine en droit du Canada (Ministère de la Défense nationale), 2014
Science and Engineering
Directorate
Border Technology Division
Division Report: 2014- 36 (TR)
July 2014
Video Analytics technology:
the foundations, market
analysis and demonstrations
Dmitry Gorodnichy,
Jean-Philippe Bergeron,
David Bissessar,
Ehren Choy,
Jacques Sciandra
This page left intentionally blank
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 3 of 70
Abstract
This report provides the history and background information related to the PROVE-IT(VA) project
conducted by the CBSA under the funding from Defence Research and Development Canada (DRDC)
Centre for Security Science (CSS) Public Security Technical Program (PSTP). The key outcomes from
the Interdepartmental Video Technology for National Security (VT4NS) meetings that led to establishing
the project are presented, the key concepts behind automated recognition in video are summarized,
survey of Video Analytics (VA) market offerings is developed, and the technology demonstration
developed for the projects are described.
Keywords: video surveillance, video analytics, recognition in video, intelligent video, technology
readiness, performance evaluation, data-sets.
Community of Practice: Border and Transportation Security
Canada Safety and Security (CSSP) investment priorities:
1. Capability area: P1.6. Border and critical infrastructure perimeter screening technologies/
protocols for rapidly detecting and identifying threats.
1. Specific Objectives: O1. Enhance efficient and comprehensive screening of people and cargo
(identify threats as early as possible) so as to improve the free flow of legitimate goods and
travellers across borders, and to align/coordinate security systems for goods, cargo and
baggage;
2. Cross-Cutting Objectives CO1. Engage in rapid assessment, transition and deployment of
innovative technologies for public safety and security practitioners to achieve specific objectives;
3. Threats/Hazards F. Major trans-border criminal activity e.g. smuggling people/material
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 4 of 70
Acknowledgements
This work is done within the project PSTP-03-402BTS “PROVE-IT(VA)" funded by the Defence
Research and Development Canada (DRDC) Centre for Security Science (CSS) Public Security
Technical Program (PSTP) .
The feedback from project partners: University of Ottawa, CRIM, ETS, RCMP, TC, CATSA, DRDC, UK
HomeOffice, FBI is gratefully acknowledged.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 5 of 70
Release Notes
Context: This document is part of the set of reports produced for the PROVE-IT(VA) project. All
PROVE-IT(VA) project reports are listed below.
1. Dmitry Gorodnichy, Jean-Philippe Bergeron, David Bissessar, Ehren Choy, Jacque Sciandra,
“Video Analytics technology: the foundations, market analysis and demonstrations", Border
Technology Division, Division Report 2014-36 (TR).
2. Dmitry O. Gorodnichy, Diego Macrini, Robert Laganiere, “Video analytics evaluation: survey of
datasets, performance metrics and approaches", Border Technology Division, Division Report
2014-28 (TR).
3. D. Macrini, V. Khoshaein, G. Moradian, C. Whitten, D.O. Gorodnichy, R. Laganiere, “The
Current State and TRL Assessment of People Tracking Technology for Video Surveillance
applications", Border Technology Division, Division Report 2014-14 (TR).
4. M. Lalonde, M. Derenne, L. Gagnon, D. Gorodnichy, “The Current State and TRL Assessment
of Unattended and Left-Behind Object Detection Technology", Border Technology Division,
Division Report 2014-13 (TR).
Jointly with the PROVE-IT(FRiV) project (PSTP-03-401BIOM):
5. D. Bissessar, E. Choy, D. Gorodnichy, T. Mungham, ``Face Recognition and Event Detection in
Video: An Overview of PROVE-IT Projects (BIOM401 and BTS402)'', Border Technology
Division, Division Report 2013-04 (TR).
6. D. Gorodnichy, E. Granger J.-P.Bergeron, D.Bissessar, E.Choy, T. Mungham, R. Laganiere, S.
Matwin, E. Neves C. Pagano, M. De la Torre, P. Radtke, “PROVE-IT(FRiV): framework and
results". Border Technology Division, Division Report 2013-10. Proceedings of NIST
International Biometrics Performance Conference (IBPC 2014), Gaithersburg, MD, April 1-4,
2014. Online at http://www.nist.gov/itl/iad/ig/ibpc2014.cfm
The PROVE-IT(VA) project took place from August 2011 till March 2013. This document was drafted
and discussed with project partners in March 2013 at the Video Technology for National Security
(VT4NS) forum. The final version of it was produced in July 2014.
Appendices: This report is accompanied by several appendices, which include the presentations from
Video Technology for National Security (VT4NS) confrences related to better understanding of Video
Analytics, its evaluation, and the Government of Canada efforts in researching and leveraging this
technology for national security.
Contact: Correspondence regarding this report should be directed to DMITRY dot GORODNICHY at
CBSA dot GC dot CA.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 6 of 70
Table of Contents
1. BACKGROUND ............................................................................................................ 7
2. INTELLIGENT SURVEILLANCE MARKET ANALYSIS ......................................................... 7
2.1 HOW INTELLIGENT IS “INTELLIGENT SURVEILLANCE” ? ................................................................ 7
2.2 MAIN FUNCTIONALITIES DEVELOPED BY THE COMPANIES .............................................................. 8
2.2.1 List of Video Analytics tasks ........................................................................................................... 9
2.2.2 Variations in Definition of Video Analytics tasks ..........................................................................10
2.2.3 Standardization of the Video Analytics tasks definition ...............................................................10
2.2.3 Mapping of companies per tasks ...................................................................................................11
2.3. Example case study..........................................................................................................................13
2.4. Conclusions .......................................................................................................................................13
3. TECHNOLOGY DEMONSTRATIONS .............................................................................. 14
3.1 CAMERA TAMPERING DETECTION AND TRAFFIC STATISTICS: VAP PILOT AT 14 COLONNADE .............. 14
3.2 LOW-BANDWIDTH REMOTE SURVEILLANCE WITH PEOPLE DETECTION ALARM ...................................... 17
3.3 MEASURING PIL PROCESSING TIME AND COUNTING PEOPLE ............................................................. 17
2.4 DETECTION OF EVENTS IN SIMPLE ENVIRONMENTS WITH COTS VA-ON-THE-EDGE CAMERAS ............. 18
4 VT4NS’13 DEMONSTRATIONS .......................................................................................................... 21
REFERENCES ............................................................................................................... 24
APPENDIX A: FIFTH INTERDEPARTMENTAL CONFERENCE ON VIDEO TECHNOLOGIES FOR
NATIONAL SECURITY (VT4NS 2013), MARCH 2013 ............................................................ 25
AGENDA ............................................................................................................................................ 25
FOREWORD PRESENTATION ............................................................................................................... 25
“PROVE-IT() FRAMEWORK” ............................................................................................................. 25
APPENDIX B: FOURTH INTERDEPARTMENTAL CONFERENCE ON VIDEO TECHNOLOGIES FOR
NATIONAL SECURITY (VT4NS 2011), SEPTEMBER 2011 ..................................................... 33
AGENDA ............................................................................................................................................ 33
FOREWORD PRESENTATION ............................................................................................................... 33
APPENDIX C: THIRD INTERDEPARTMENTAL CONFERENCE ON VIDEO TECHNOLOGIES FOR
NATIONAL SECURITY (VT4NS 2010), MAY 2010 ................................................................. 40
“VIDEO ANALYTICS: TECHNOLOGY MATURITY, DEPLOYMENT CHALLENGES, AND ROADMAP” .......... 40
APPENDIX D: THIRD INTERDEPARTMENTAL CONFERENCE ON VIDEO TECHNOLOGIES FOR
NATIONAL SECURITY (VT4NS 2008), OCTOBER 2008 ........................................................ 45
BACKGROUND, PROGRAM AND REFERENCES ..................................................................................... 45
“INTELLIGENT SURVEILLANCE: EXAMPLES, MYTHS AND LESSONS” ..................................................... 45
APPENDIX E: “RECOGNITION IN VIDEO”, THE IDENTITY, PRIVACY AND SECURITY INSTITUTE,
UNIVERSITY OF TORONTO PUBLIC LECTURE SERIES, NOVEMBER 30, 2009. ...................... 58
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 7 of 70
1. Background
Over the past five years the Government of Canada (GoC) has been intensively exploring how to
leverage the advances made in video technology for the national security needs. Starting from 2008
and built on the original collaboration between the National Research Council of Canada’s Institute for
Information Technology (NRC-IIT) and the Computer Vision section of the USA Director of National
Intelligence’s Intelligence Advanced Research Projects Activity (DNI IARPA), formerly USA Disruptive
Technology Office’s Video Analysis and Content Extraction program (DTO-VACE), the Science and
Engineering Directorate of the Canada Borders Services Agency (CBSA-S&E) took the lead in bringing
together all GoC stakeholders working in the area Video Surveillance and Analytics in what has
become known as the Video Technologies for National Security (VT4NS) initiative. Annual one-day
long VT4NS workshops have been organized for the GoC stakeholders, where GoC stakeholders could
exchange information about their needs in video surveillance, learn more about the state of the art in
the field, and finally discuss the next steps for addressing their needs.
A dedicated VT4NS SharePoint portal https://partners.drdc-rddc.gc.ca/css/Portfolios/Biometrics/VT4NS
accessible to GoC clients and partners has been created to facilitate the exchange of information on
the topic and where the proceedings from the past five VT4NS workshops have been archived. The key
excepts from the VT4NS proceedings archived at the VT4NS site are provided as Appendices in this
report. As summarized in the Appendices, the key outcomes from the VT4NS initiative and its affiliated
workshops were 1) raising the awareness of GoC video surveillance stakeholders on the technical
challenges of deploying Video Analytics, and 2) preparing the foundation for developing and conducting
– in partnership with all major GoC stakeholders (CBSA, RCMP, TC, CATSA, DTDC) and International
and Academic partners (University of Ottawa, CRIM, ETS, UK HomeOffice, FBI) - two major Research
and Development (R&D) studies called PROVE-IT(FRiV) and PROVE-IT(VA) dedicated to examining
and improving, where possible, the Technology Readiness Level (TRL) of video surveillance
technologies that can be potentially deployed by the GoC.
The current report presents the key outcomes of the PROVE-IT(VA) study. The key concepts behind
automated recognition in video are summarized, survey of Video Analytics market offerings is
developed and the technology demonstration developed for the projects are described.
2. Intelligent Surveillance Market Analysis
2.1 How Intelligent is “Intelligent Surveillance” ?
There are over 5000 companies registered in Canada doing business in video surveillance.[NUANS
Search Results with keywords “video, surveillance” in 2008 (www.nuans.com)]. Any company doing
business in video surveillance can potentially call its surveillance technology “Intelligent” or “Smart” by
simply performing naïve “pixel brightness comparison” (More detail on the difference between pixel
brightness comparison and real object motion detection as well as other key concepts related to the
automated recognition in video are provided in Appendices D-E and [1-5]).
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 8 of 70
It is also expected that most companies that implement many video analytics applications, do so by
using public domain computer vision and image processing libraries such as Intel Open CV. These
libraries can be used at no charge for commercial applications and contain functions and primitives
such as many low-level image processing libraries (edge detection, colour segmentation, motion/optical
flow computation) and more complicated codes such as for OCR, Face Detection, and even some Face
Recognition and Classification.
The use of these libraries is very common in the academic environment and many graduates from
computing science and engineering departments can use them. As a result, there are many companies
that are now capable of doing video processing and in doing so they can claim that they can provide
many Video Analytic (VA) functionalities (as in listed in Table 1). Clearly, some of these functionalities
are much harder than others, if possible at all. The key question therefore is to know which of these
functionalities are really ready for deployment and which are not.
Table 1. Non-inclusive list of VA functionalities reported by the companies as ``possible`` according to
their documentation and websites. Left column lists technologies that are of much higher difficulty than
those listed in the right column.
Harder
Easier
Human / Object Recognition and Tracking
Object Classification
People Counts
Vehicle recognition
People recognition / Face recognition
Unattended Baggage Detection
Object Removal Detection
Loitering Detection
Tail-gating
[Waiting] Line Control, Crowd management
Special Attribute Detection
Advanced Behaviour Analysis
Intrusion Detection / Virtual Tripwire
Autonomous PTZ Tracking
Stopped Vehicle Detection
Camera Tampering Detection
Congestion detection
Counter Flow
Automatic Licence Plate Recognition
Object Alteration Detection
Audio and Sound Classification
Face Detection / Face Tracking
Graffiti / Vandalism detection
Highway (vehicle) count
Below we provide a more detailed overview of VA functionalities offered by the industry. The
assessment of readiness of these functionalities is provided in separate PROVE-IT(VA) reports.
2.2 Main functionalities developed by the companies
The video surveillance market has many players that provide a range of commercial products using VA
capabilities. Some companies are specialised in real time Video Analytics while others focused on a
post processing mode.
This survey was based on Video Analytics running on the server side as its technology is more mature
in terms of the reliability and the power of the hardware as well as the efficiency of the operating
system. At the same time, we note that Video Analytics on the edge (i.e. performed on a camera or
encode hardware) is becoming also possible, due to the recent increase in camera power and
processing capabilities.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 9 of 70
Video Analytics commercial products can be deployed in many security applications. The figures below
provide illustrations of some commonly listed Video Analytics functionalities.
Figure 1 – Area detection Figure 2 – Any flow detection
Figure 3 – Loitering detection Figure 4 – Flowing Traffic direction detection
2.2.1 List of Video Analytics tasks
In the following we summarize the list of the most offered Video Analytics tasks, as provided in the
commercial products description, based on the overview of 44 companies working in this area.
Unattended/Left-Behind Baggage Detection
Description: refers to a person who carries a baggage and left it in the scene while he/she exits
the area of interest.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 10 of 70
Person Tracking in non-crowded and crowded environments
Description: refers to a person isolated and tracked while he/she stays in the scene.
Person-baggage Tagging (Association)
Description: refers to a couple person/baggage where the person left behind the baggage
while he/she stays in the area of interest. The Video Analytics keeps an association between
the person and his/her baggage.
Object Removal Detection
Description: refers to object that was previously present in the scene and is removed from his
location.
Loitering Detection
Description: refers to a person who stays in a same area for a certain period of time.
Tail-gating Detection
Description: refers to a person or a vehicle that follows closely on the tail of another to gain
access and attempts to circumvent access control.
Tamper Detection
Description: refers to an object put in front of the camera closing the field of view of the lens.
The above list is not exhaustive: other Video Analytics functions are provided by companies but these
functions appear more specific.
2.2.2 Variations in Definition of Video Analytics tasks
There is no consensus on the VA terminology across the industry, so it is difficult to fairly compare the
products. Similar functionality can be referred to in multiple manners. Additionally, the same term may
be used to refer to different capabilities.
For example, “Object Put” and “Object Left Behind” may be similar or very different, depending on the
scenario and definition chosen. In the first case, we only consider the status of the object in the scene
as in the second case we consider the actor that put the object in order to know if the object is left
(behind). In both cases the object can be considered as “put” in the scene and may create some
ambiguity in the decision making.
2.2.3 Standardization of the Video Analytics tasks definition
Choosing a Video Analytics technology to solve a problem for a client is challenging since there is no
standard in the level and the way of defining the event.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 11 of 70
The National Institute of Standards and Technology (NIST) put an effort to standardise some Video
Analytics events terminology in the TRECVID VA competition conferences that it organizes every year.
It can be observed however that the list of Video Analytics events proposed by the NIST is not well
aligned with the lists found in commercial product descriptions.
NIST describes events in a business-oriented manner, whereas industry uses inconsistent definitions
with varying levels of business and technical language.
Ex1. “Person Run” is more business related whereas “Object Speed Change” or “Object Size Change”
is more technically related.
Ex2. “Cross Fence detection” is more business related whereas “Cross Line detection” is more
technically related.
2.2.3 Mapping of companies per tasks
Table 2 presents 44 Video Analytics vendors surveyed over the Internet. The column headings present
some of the main common Video Analytics functionalities described above. For each company, the
second, third and fourth column of the table puts emphasis on the existence of the VMS manufacturer
partnership, while the other columns show (by `X`) the Video Analytics functions provided.
From Table 2 we can see that about 26 companies out of 44 (59.1%) surveyed in the study partner with
an VMS manufacturer. This shows that VMS environment is a strategic criteria in choosing Video
Analytics in order to have a full integration of the solution and to be able to run all capabilities of the VA
with operational surveillance system.
Table 3 shows the number of companies that perform each of listed VA tasks . It can be seen from the
table that the functions that are most commonly offered by industry are the following.
Loitering Detection (68.2%)
Unattended/Left-Behind Baggage Detection (63.6%)
Object Removal Detection (54.5%)
Only around a quarter of them offer the following functions:
Tamper Detection (43.2%)
Tail-gating Detection (34.1%)
Person tracking in non-crowded and crowded environments (31.8%)
At the time of this writing none of the surveyed vendors offered “Person-baggage tagging
(Association)”.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 12 of 70
Table 2- List of VA companies mapped on VA tasks
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 13 of 70
Table 3- VA tasks that are most commonly offered.
2.3. Example case study
In 2006 at the Madrid’s Barajas International Airport a terrorist attack killed 2 people, injured 26 others
while destroying 60% of the parking lot building and part of the terminal. Following this event Iberia
Airline Company decided to improve their security level not only by installing more cameras but also by
using Video Analytics technology. They selected the VI-System and its Agent VI’s real-time video
analytics solution. Analytics rules applied to the cameras in Liberia Airlines facilities include:
Person/vehicle moving in an area,
Person/vehicle crossing a line.
Security manager at Liberia Airlines commented that “moving to an all IP-Based solution required that
we look for a highly-scalable video analytics offering that would allow as to deploy hundreds of cameras
while eliminating the need for additional server hardware and keeping operational costs down. Analytics
have performed extremely well despite the challenging outdoor environment near a busy airport”.
2.4. Conclusions
There is substantial evidence showing the use of some of Video Analytics technologies in video
surveillance of protected or limited access premises. There however very little evidence showing the
use of Video Analytics technologies in monitoring public places.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 14 of 70
3. Technology demonstrations
In this section the VA technologies developed by the CBSA for the demonstration of the highest
achievable TRL are described. The development of technologies is done by CBSA-S&E Video
Surveillance and Biometrics (VSB) section research scientists and engineers based on the
commercially available products and the in-house developed Video Analytic Platform (VAP) [1-2].
3.1 Camera tampering detection and traffic statistics: VAP Pilot at 14 Colonnade
Installation details:
Entrance Camera: A camera (Entrance Camera) is installed on a
tripod at the entrance of 14 Colonnade Room 210 (as shown in the
figure at left) to view the people traffic through the entrance door.
This camera is connected via IP experimental “orange” network to
an experimental “orange” desk-top computer which will run VAP
software (VAP pilot computer). The information from this camera is
processed by VAP in real-time and the processing results is shown
on a screen (VAP Pilot Screen) using VAP Browser and the
commercial Video Management System software Milestone.
The extracted and saved information (meta-data) include timestamp of the detected person motion and
the timestamp of the camera mal-functioning (including tampering). No images are stored from this
camera. For debugging purpose, the program may save a tiny black-and-white image from the video
(not more than 16x12 pixels) which if required can be deleted after not more than 24 hours.
Lab Cameras: In addition to the Entrance camera, several cameras from the VSB Lab (Lab Cameras)
is also connected to the VAP Pilot Computer and the results obtained from those cameras is also
displayed on VAP Pilot Screen.
With the consent from all VSB members, the images from the Lab cameras are saved in addition to the
extracted meta-data.
VAP Screen and VAP Pilot computer: The data captured by VAP are stored on VAP Pilot computer,
which is a desktop machine located in the Room that is accessible and can be logged only by VSB
members.
This computer runs both modules of the VAP: the video analytic module called VAP Capture, which
processes the video and extracts metadata, and the visual analytic module called VAP Browser which
displays the extracted data and images (when available).
VAP Screen, located at the entrance of 14 Colonnade, is connected to the VAP Pilot computer using
the 100ft VGA cable, as a second monitor. The control of all displayed information will be done from the
VAP Computer.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 15 of 70
The type of displayed information is shown in figures below. It can be switched from Event View to
Time-line View, to Visual Statistics View.
Figure 5:
Event View ( Entrance Camera ).
Figure 6:
Event View ( Lab Camera).
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 16 of 70
Figure 7:
Visual Statistics View.
Figure 8:
Timeline View.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 17 of 70
3.2 Low-bandwidth remote surveillance with people detection alarm
The VAP software has been applied to address the operational need to view remotely who is visiting
the premises after work hours.
The VAP software was tuned to capture and transfer the images of moving objects via a secure IP
channel. Because video data was not transmitted, but only the JPG images were transmitted, the
system offered a solution to viewing remote locations, which have limited bandwidth connection.
3.3 Measuring PIL processing time and counting people
The VAP software has been used to address one of the operational needs of the CBSA related to the
analysis of traveller processing time at Primary Inspection Lane (PIL), where the passport check is
performed.
A COTS face detection SDK has been integrated with VAP and combined with object tracking modules
in order to compute the time spent by travellers in front of the PIL.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 18 of 70
2.4 Detection of events in simple environments with COTS VA-on-the-edge
cameras
VA from Bosch COTS on-the-edge camera/encoder was tested in a variety of settings with in mock-up
datasets and video recorded from operations .
The detection of the following events has been analyzed:
object put
object removed,
idling
loitering,
person run
wrong direction
It has been found that detection of these event is possible (TRL=5 or higher) in simple environments
with little traffic and controlled lighting and traveller motion patterns
However, the detection of the same events in more challenging settings such as those observed with
cameras installed in airports has not showed the readiness of the technology for these settings, unless
an intensive manual filtering of false alarms is performed by a human operator, similar to the technique
described in TRECVID evaluation, where VAP Browse is used to remove false alarms.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 19 of 70
Figure 9: Testing of person loitering.
Figure 10: Testing of object removed.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 20 of 70
Figure 11: Testing of put object.
Figure 12: Testing of wrong direction.
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 21 of 70
4 VT4NS’13 demonstrations
The following VA technology demonstrations have been prepared for sharing with the project partners :
1. Measuring PIL processing time and counting people (tested in operational environment) – in-house
developed VA + COTS Face Detection integrated on VAP.
2. Low-bandwidth remote surveillance with people detection alarm (tested in ops. environment) – in-
house developed VA integrated on VAP.
3. Camera tampering detection and traffic statistics (Pilot on Colonnade) – in-house developed VA
integrated on VAP.
4. CBSA-uOttawa participation at NIST TRECVID Interactive Surveillance Event Detection evaluation:
“Person run and other event detection in complex environment using Visual Analytics” – see [6] for
more details.
5. Detection of abandoned objects in simple environments with COTS VA software – evaluation of
COTS VA products by CRIM.
6. Detection of events (object put/removed, idling/loitering, person run/wrong direction) in simple
environments with COTS VA-on-the-edge cameras - evaluation of COTS VA products CBSA
In addition, in synergy with PROVE-IT(FRiV) project, the following technology demonstrations have
been prepared on Face Recognition in Video:
1. Post-event face tagging, grouping and search– integration of COTS face recognition built with
PittPatt FR SDK with VAP
2. Still-to-video watch-list screening: binary decision making – in-house built solution using
Cognitec FR SDK
3. Still-to-video watch-list screening: triaging -– in-house built solution using Cognitec FR SDK
4. Video-to-video face recognition and triaging decision-making – developed by project partner
ETS and TAMALE
VAP refers to the Video Analytic Platform that is developed by CBSA-S&E for integration of custom-
made and third party VA and FRiV codes into operational CCTV IP-based video surveillance networks.
It consists of two modules: VAP Capture, where VA and FRiV codes are executed, and VAP Browser,
which is the state-of-art Visual Analytic s GUI data mining / filtering end-user tool. For more information
on VAP see [1-3] and Appendices A-C.
These demonstrations have been recorded as AVI files and archived at VT4NS portal
(https://partners.drdc-rddc.gc.ca/css/Portfolios/Biometrics/VT4NS) and on CBSA-S&E o:drive, as
shown below:
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 22 of 70
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 23 of 70
O:\_Section-VSB\_Outputs\VT4NS-Proceedings\VT4NS'13\Demos
├───FR
│ │ Watchlist.png
│ │
│ ├───1-postEventFaceTaggingAndSearch(PittPatt,onCBSAType2mockup)
│ │ CBSA-demo-FR-1-PostEvent-FaceSearch-VAPBrowser.avi
│ │ CBSA-demo-FR-1-PostEvent-FaceSearch-VAPBrowser2.avi
│ │ Clustering.PNG
│ │
│ ├───2-WatchList-Binary(PittPatt-onCBSAType2mockup)
│ │ CBSA-demo-FR-2-2-WatchList(CBSA+ChokePoint+VSB)-EventBrowser.avi
│ │ CBSA-demo-FR-2-WatchList(CBSA+ChokePoint+VSB).avi
│ │ Vap-Pittpatt-img.png
│ │
│ └───3-WatchList-TriaggingTool(CognitecOLD-onChokepoint, CognitecNEW-onCBSAType2)
│ CBSA-demo-FR-3-WatchList-GUI-tool.avi
│ red.png
│ yellow.png
│ NB: CognitecOLD-does not work onCBSAType2 data
└───VA
├───1-pil procesing time
│ 1_img.png
│ 5_img.png
│ CBSA-demo-VA-1-1-FaceCounting-airport-close-eye-level.avi
│ CBSA-demo-VA-1-2-FaceCounting-airport-far-above-eye-level.avi
│
├───2-remoteViewing
│ Abnormal-22h break in.png
│ CBSA-demo-VA-2-PeopleDetection-VAPBrowser-Timeline.avi
│
├───3-colonadePilot
│ CBSA-demo-VA-3-1-VAPPilot-VisualStatitics.wmv
│ CBSA-demo-VA-3-2-VAPPilot-TamperingDetection.avi
│ CBSA-demo-VA-3-3-VAPPilot-TamperingDetection.avi
│ VAP-colonnade-pilot.PNG
│
├───4-trecvid
└───5-cots
CBSA-demo-VA-6-1-COTS-ObjectPut.avi
CBSA-demo-VA-6-2-COTS-ObjectRemoved.avi
CBSA-demo-VA-6-3-COTS-Loitering.avi
CBSA-demo-VA-6-4a-COTS-WrongDirection-1person.avi
CBSA-demo-VA-6-4b-COTS-WrongDirection-2persons-fail.avi
Loitering.png
ObjectRemoved.png
PutObject.png
WrongDirection.png
Figure 13: The archival structure of the VT4NS proceedings and the PROVE-IT()
reports and demonstrations on the DRDC SharePoint Portal (previous page) and
CBSA internal hard-drive (this page).
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 24 of 70
References
[1] Dmitry O. Gorodnichy and Elan Dubrofsky. "Automated extraction of intelligence from video using
Video Analytics Platform (VAP)". (Extended Abstract & Demo), Justice Institute of British Columbia
and the U.S. Department of Homeland Security's Center of Excellence VACCINE Workshop on
"Visual Analytics for Public Safety Professionals", September 20 -21, New Westminster, BC, 2010
[2] Dmitry O. Gorodnichy and Elan Dubrofsky. VAP/VAT: Video Analytics Platform and Testbed for
testing . Proceedings of SPIE Conference on Defense, Security, and Sensing, DS226: Visual
Analytics for Homeland Defense and Security track. 5 - 9 April 2010, Orlando
[3] Dmitry O. Gorodnichy, Tony Mungham, "Automated video surveillance: challenges and solutions.
ACE Surveillance (Annotated Critical Evidence) case study", NATO SET-125 Symposium "Sensor
and Technology for Defence against Terrorism", Mainheim, April 2008.
[4] Dmitry O. Gorodnichy, Mohammad A. Ali, Elan Dubrofsky, Kris Woodbeck. Zoom on the evidence
with ACE Surveillance. International Workshop on Video Processing and Recognition
(VideoRec’07). May 28-30, 2007. Montreal , QC , Canada . [Pdf and Poster]
[5] Dmitry O. Gorodnichy, ACE Surveillance: The Next Generation Surveillance for Long-Term
Monitoring and Activity Summarization. First International Workshop on Video Processing for
Security (VP4S-06), June 7-9, Quebec City, Canada. [Pdf]
[6] C. Whiten, R. Laganiére, E. Fazl-Ersi, F. Shi G. , Bilodeau, D. O. Gorodnichy, J. Bergeron, E.
Choy, D. Bissessar . “VIVA-uOttawa / CBSA at TRECVID 2012: Interactive Surveillance Event
Detection”. On line at http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.12.org.html (http://www-
nlpir.nist.gov/projects/tvpubs/tv12.papers/viva-uottawa.pdf)
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 25 of 70
Appendix A: Fifth Interdepartmental Conference on Video
Technologies for National Security (VT4NS 2013), March 2013
Agenda
Foreword Presentation
“PROVE-IT() Framework”
By D. Gorodnichy
UNCLASSIFIED
Gov’t departments and By invitation only
Fifth Interdepartmental Conference on
Video Technologies for National Security (VT4NS 2013)
Theme: Presentation and review of PROVE-IT (FRiV/VA) projects results.
PSTP BIOM-401: Real-time Face Recognition Technologies for Video-surveillance Applications
PSTP BTS-402: Video Analytics for Border and Transportation Security in Indoor and Outdoor Environments
When: Wednesday, 27 March, 2013. 8:30 – 16:30
In Person: DRDC-CSS, 222 Nepean Str
Via Telecom: Call-in toll-free number: 1-877-413-4788 / Call-in number: 1-613-960-7513
Attendee access code: 947 417 6
Portal: https://partners.drdc-rddc.gc.ca/css/Portfolios/Biometrics (Human ID Systems)/VT4NS
(will be provided in separate email)
Organized by:
Defence R&D Canada’s Centre for Security Science (DRDC-CSS), and
Canada Borders Services Agency’s Scientific and Engineering Directorate (CBSA-S&E)
Chairs:
Dmitry O. Gorodnichy: dmitry.gorodnichy@cbsa-asfc.gc.ca
Pierre Meunier: pierre.meunier@drdc-rddc.gc.ca
Facilitator:
David Bissessar: david_bissessar@hotmail.com
Scope and Objectives:
As in the past, the primary objective of this conference is to bring together GoC Border and Transport Security (BTS) and
Biometrics (Biom) Communities of Practice (CoP) in) for the exchange of information on the activities, needs, and
interests related to deploying video surveillance technology within GoC - in particular, in the area of Video Analytics (VA)
and Face Recognition in Video (FRiV).
The secondary objective is to present and discuss the findings from the DRDC-CSS-funded PROVE-IT() projects led by
CBSA-S&E in partnership with Academia and federal partners related to TRL assessment of VA and FRiV technologies.
UNCLASSIFIED
Attendees:
In person: CBSA, DRDC, DND, RCMP, PCO, CATSA, Public Safety, TC, DFAIT
Dial-in: HomeOffice, FBI, NIST, AFP
Tentative Program (time indicated as guideline only)
8:30 Introductions, Foreword – CSS, CBSA-S&E
9:00 PROVE-IT() objectives, framework and findings - CBSA-S&E
Main objective: Find technologies/applications/settings { Xi: TRL(Xi)>4 }
Tools: VAP, Visual Analytics, COTS products, academia partners work
Main results: TRL(VA/FRiV) assessment tables and technology demonstrations
10:00 Presentations from project contributors:
PROVE-IT(FRiV) – ETS and uOttawa-TAMALE
PROVE-IT(VA) - uOttawa-VIVA and CRIM
Presentations from other project partners (TBC)
12:00 Technology demonstrations
12:30 Lunch (not provided)
For Government attendees only:
13:30 Technology demonstrations (continued)
15:00 Round table discussion, review of results, next steps
Presentations from other project partners (TBC)
16:30 Adjurn
Technology demonstrations
(most demonstrations will be pre-recorded and will be uploaded to the Portal along with all presentations)
VA:
1. Measuring PIL processing time and counting people (tested in ops. environment) – VA+COTS FD on VAP
2. Low-bandwidth remote surveillance with people detection alarm (tested in ops. environment) – VA on VAP
3. Camera tampering detection and traffic statistics (Pilot on Colonnade) – VA on VAP
4. CBSA-uOttawa participation at NIST TRECVID Interactive Surveillance Event Detection evaluation:
Person run and other event detection in complex environment using Visual Analytics – by VIVA and CBSA
5. Detection of abandoned objects in simple environments with COTS VA software – by CRIM
6. Detection of events (object put/removed, idling/loitering, person run/wrong direction) in simple environments with
COTS VA-on-the-edge cameras - by CBSA
FRiV:
1. Post-event face search and retrieval – with PittPatt SDK and VAP
2. Still-to-video watch-list screening: binary decision making – with Cognitec SDK
3. Still-to-video watch-list screening: triaging - with Cognitec SDK and new end-user interface
4. Video-to-video face recognition – ETS / TAMALE
VAP refers to the Video Analytic Platform that is developed by CBSA-S&E for integration of custom-made and third
party VA and FRiV codes into operational CCTV IP-based video surveillance networks. It consists of two modules:
VAP Capture, where VA and FRiV codes are executed, and VAP Browser, which is the state-of-art Visual Analytic s
GUI data mining / filtering end-user tool.
Fifth Interdepartmental Workshop on
Video Technologies
for National Security
(VT4NS’13)
March 27, 2013, Ottawa
Organized by CBSA-S&E and DRDC-CSS
Gov’t:
• DRDC-CSS
• CBSA NHQ S&E: VSB
• CBSA NHQ Solutions
• CATSA
• TC
• DFAIT
• RCMP
• Public Safety
• Office of the Privacy Commissioner
Welcome!
Academia / Industry:
• ETS
• uOttawa TAMALE Lab
• uOttawa VIVA Lab
• CRIM
International:
• HomeOffice
• FBI
• NIST
• AFP
VT4NS: outcomes
2007: by NRC with US IARPA / VACE:
– first (live) demo of VA (inc. Face detection and annotation in video)
– first population of issues / first call for coordination of efforts
2008: by CBSA-S&E with DHS, IARPA
– Theme: “Building future-proof video surveillance systems (VSS)”
– Addressing CBSA’s need in VSS upgrades
Outcomes: Recommendation on Open Architecture systems
• which allows integration of VA & in-house testing / pilot
2010: by DRDC-CSS & CBSA-S&E
– Theme: “Moving Video Analytics into operational environment.” /
“Faces in video”
– Presentation: “Video Analytics: inside the Black Box”
– (re-) Introducing TRL for VA testing
– Demonstration of CBSA’s Video Analytics Platform (VAP)
– Brain-storming: in search for “killer” (best) VA applications
Outcomes: SoW for two new PSTP studies
VT4NS: outcomes (cntd)
2011 Sept: by DRDC-CSS & CBSA-S&E
– Theme: Kick-off of PSTP BIOM401 & BTS402 Studies
– Presentation: “Face Recognition: inside the Black Box”
– Demos of FR COTS systems for “watch list” screening (issue)
– Discussion on other FR deployment issues
2013 March: by DRDC-CSS & CBSA-S&E
– Theme: Closing of PSTP BIOM401 & BTS402 Studies
– Presentations from CBSA and partners
– Demos of FRiV and VA systems with TRL > 4
– Discussion in TRL assessment results
VT4NS portal
Presentations, demo videos, and other documents posted at
https://partners.drdc-rddc.gc.ca/css/Portfolios/
Biometrics (Human ID Systems)/VT4NS
USER = “VT4NS”, Password =“password”
Biometrics Community of
Practice – Study No. 2:
Real-time Face Recognition
Technologies for Video-
surveillance Applications
Duration: 1 + 1year
Funds: $200K + 80 K
In-kind: $500K
Objectives
• investigate and elevate, if possible, the maturity of face
recognition technologies that are applicable for real-time
identification of individuals using video cameras.
– To encourage focused national and, potentially, international
collaborative efforts … for identity/accreditation management and
access control.
– To recommend technological solutions to border surveillance and
interdiction challenges that blend with current doctrine and
technologies.
FRiV applications considered
Scope of the study: to assess how face recognition (FR)
technologies can be applied to facilitate real-time
recognition of individuals captured by video cameras in
applications such as:
I. Triaging of faces according to their resemblance to a Wanted List;
II. Fusion of face recognition from different cameras while tracking
III. Face recognition-assisted tracking;
IV. Matching a face/person across several video feeds;
V. Multimodal recognition, e.g., face and voice recognition;
VI. Soft-biometric based tracking/recognition techniques.
Study deliverables
1. Identify the applications and scenarios where FR
technology can be applied with existing video-surveillance
(CCTV) infrastructure for instant (real-time) extraction and
recognition of faces in unconstrained surveillance-type
applications.
2. Investigate the applicability and feasibility of multimodal
biometric recognition by combining face recognition with
other modalities that are collectable in surveillance
applications.
1. Such biometric modalities may include hard biometrics modalities,
such as voice, and iris, and
2. softer biometrics modalities such as height/weight, face shape, and
gait.
Study deliverables (cntd)
3. Examine the technology readiness levels (TRLs) of the FR
technologies for the SIX FRiV applications listed in the scope, and
identify those that have TRL 5 and higher, i.e., the technologies have
been successfully piloted or tested in relevant or operational
environments.
4. Conduct an analysis of the FR technologies listed in the scope
1. by examining reports from trusted unbiased sources (such as government
and academia organizations) or
2. by hands-on testing of recognition technologies embedded into a
common/operational video surveillance system.
5. Execute a demonstration of a video surveillance system
that performs live instantaneous analysis of the observed
faces.
6. Organize and host a combined workshop/information
session for the CoP (approximately 30 people) to review
findings.
Border and Transportation
Security – Study No. 2:
Video Analytics for Border
and Transportation
Security—Indoor and
Outdoor Environments
Duration: 1+1 year
Funds: $200K+80K
In-kind:$300K
VA applications considered
Scope of the study: focus on the analysis of TRLs for Video
Analytic technologies applied to the following border
security challenges:
1.Unattended/left-behind baggage detection;
2.Person tracking in non-crowded and crowded
environments,;
3.Person-baggage tagging (association) in crowded
environments;
4.Object removal detection;
5.Loitering detection;
6.Tail-gating detection; and
7.Camera tampering detection.
PSTP BIOM-401, PSTP BTS-402
PROVE-IT()
Framework
Dmitry Gorodnichy
PROVE-IT (X) key objective
find
applications/technologies/scenarios/settings
Xi: {TRL(Xi)>4 }
If the technology does not work for the application …
… there are two ways to proceed:
Find
another
application!
Find
another
technology!
“Wizard Oz” video-clip
with faces detected and
clustered by COTS FR.
With and without VA / FRiV
Area of Unexploited
Information!
DATA
COLLECTED
DATA USED with VA
DATA USED without VA
PROVE-IT (X) framework
In Search for High-TRL Applications, using :
– on Public datasets and in Mock-up setups
– with traditional and new applications
– in different Surveillance setups / scenarios
– with commercial and academic products
+ surveys (end-users, media, companies, academia)
Tools:
1.5-grade TRL assessment scale
2.Taxonomy of video-surveillance setups
3.End-user software (eg. VAP)
4.TRL assessment table
Technology Readiness Level
9. Actual system 'flight proven' through successful
mission operations (over 30)
8. Actual system completed and 'flight qualified'
through test and demonstration.
7. System prototype demonstration in operational
environment. PILOT
6. System prototype demonstration in relevant
environment. MOCK-UP
5. Component validation in relevant environment.
4. Component validation in laboratory
environment.
3. Analytical and experimental critical function -
characteristic proof of concept.
2. Technology concept / application formulated
1. Basic principles observed and reported.
TRL-based evaluation requires access to real environments,
real end-users, and real operational needs!
1. Five-point TRL assessment scheme
Years to deploy TRL
Additional Applied R&D
requirement
++
0 (can be deployed
immediately by any
operational agency with no
R&D capacity)
TRL=8
-9,
complete
COTS
system deployed and
proved useful by many
users
no development effort is required
to deploy it
+:
<1 (by most operational
agencies with minimal
Applied R&D capability)
TRL=7
-8, compete
COTS system deployed
somewhere
some minor development effort is
required to fit business
requirements
oo
1
-2 (only by operational
agencies that have
substantial Applied R&D
capability)
TRL=5
-6, system
validation in mock
-
up or
pilot
solid development effort is
required
o
2
-3 (only by operational
agencies that have access to
major to Applied R&D)
TRL=4, component
validation in relevant
24/7 environment
major development effort is
required
-
>3 (not foreseeable for
deployment in near future)
TRL=1
-3
significant academic / industry
R&D required
Other technology maturity criteria
• Producibility – Manufacturing Readiness
• Readiness to Receive (If Goal Is Transition from
Technology Developer and Technology Receiver)
• Practice Based Technology Maturity (Emphasis on
Community of Users) –
• User Readiness Program Readiness (program needs and
constraints).
• R&D Readiness, which measures the in-house R&D
capacity Required to deploy the technology (such as
required for Customization & tuning of the technology
Biometric vs. Surveillance: objectives
Biometrics
Real-time applications
1. “Access/Border Entry Control” –
in cooperative mode
facilitate entry to “Travellers”
– Iris: www.NEXUS.gc.ca)
– Faces: ePassports
2. “Screening applications” –
in non-/un-cooperative mode
prevent entry to “Criminals”
– Faces: “Wanted by CBSA”
Other applications
1. “Imposter” problem
2. For Intelligence & Enforcement
Video Surveillance
Four operational objectives:
1. Observe
2. Detect (abnormality/event)
3. Recognize (event/incident)
4. Identify (individuals)
Three modes of operation:
• Real-time Active (24/7 live
viewing)
• Real Time Passive (in
conjunction with other duties)
• Archival (post-event analysis,
thru recording)
Faces captured in controlled environment are much
easier to recognize (as in e-Gates with e-Passport)
Photos taken in a controlled environment provide:
Canonical face model adopted by ICAO’02 for
passport-type documents
high resolution - 60 pixels between eyes
face “nicely” positioned (front-faced, eye-level)
neutral facial expression
no occlusion (eye-glasses, scarf)
high quality
No motion – no blur
In focus
Good (best possible) illumination
No compression artifacts
Biometric setup (Type 0)
Face resolution in surveillance video
• Basic capture principle: either blur or lack of focus !
– The image quality of the moving object depends: aperture, exposure.
Face resolution (pixels between eyes)
• Sensor resolution: 24 – 200 pixels (1/32-1/8 frame width)
However:
• Informative resolution (aka Actual, Shannon index, Entropy-
based): 10 – 46 pixels
2. Taxonomy of Surveillance setups
Type 0: Cooperative Biometric setup (access control, eGate)
Type 1: constrained setup
– Eg. Primary Inspection Lane (PIL)
Type 2: unconstrained free-flow, one-at-time
– Eg. Port of Entry / Chokepoint entry
Type 3: unconstrained free-flow, many-at-time
– Eg. in Airport
Type 4: Outdoor (no lighting or structural constraints)
1
2
3
3. Testing using end-user software (VAP)
If VA / FRiV tool works “well” 24/7
over long period of time, and
the Officer finds it “Useful”,
only then
the TRL of this tool is 7 or more!
Archival mode
of operation
Real-time mode
of operation
VAP
ALARMS
+details
VAP: Video Analytic Platform software
Developed in house for end-users in the field (to be used with
existing cameras and Video Management Systems)
Consists of :
• VAP Event Capture:
Video Recognition modules
(includes camera tampering,
people/object tracking,
face detection, face matching)
• VAP Event Browser:
Visual Analytics GUI
(end-user interface)
GUI tools for the officer:
For Real-time mode:
1. Last Event view
with Alarm code
For Archival mode:
2. Event summary view
3. Timeline summary view
4. Smart Search tool
D.O. Gorodnichy and Elan Dubrofsky. VAP/VAT: Video Analytics Platform and Testbed for testing and deploying video
analytics. SPIE Conference on Defense, Security, and Sensing, Visual Analytics for Homeland Defense and Security track.
2010, Orlando
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 33 of 70
Appendix B: Fourth Interdepartmental Conference on Video
Technologies for National Security (VT4NS 2011), September 2011
Agenda
Foreword Presentation
UNCLASSIFIED
Gon’t departments and By invitation only
Registration is required.
Those planning to attend in person or via telecom should provide their names to Pierre Meunier.
Forth Interdepartmental Workshop on
Video Technologies for National Security (VT4NS 2011)
Theme: Kick-off of PSTP projects PROVE-IT (VA) and PROVE-IT(FRiV)
When: Friday, 23 September, 2011. 9:00 – 16:00
In Person: Large Boardroom on 11
th
floor, DRDC-CSS, 222 Nepean Str.
Via Telecom: Numbers to be provided.
Portal: https://partners.drdc-rddc.gc.ca/css/Portfolios/Biometrics (Human ID Systems)/VT4NS
(Gov’t organizations will receive login/password to the portal in a separate email)
Organized by:
Defence R&D Canada’s Centre for Security Science (DRDC-CSS), and
Canada Borders Services Agency’s Scientific and Engineering Directorate (CBSA-S&E)
Chairs:
Dmitry O. Gorodnichy: dmitry.gorodnichy@cbsa-asfc.gc.ca
Pierre Meunier: pierre.meunier@drdc-rddc.gc.ca
Facilitator:
David Bissessar: david_bissessar@hotmail.com
Scope and Objectives:
As in the past, the primary objective of this workshop is to bring together GoC Border and Transport
Security (BTS) and Biometrics (Biom) Communities of Practice (CoP) in) for the exchange of information
on the activities, needs, and interests related to deploying video surveillance technology within GoC - in
particular, in the area of Video Analytics (VA) and Face Recognition in Video (FRiV).
The secondary objective is to present an updated on two DRDC-CSS-funded projects that CBSA-S&E
has launched this year in partnership with Academia and its federal partners to address the VA and
FRiV needs of the GoC BTS and Biom CoP: PROVE-IT (VA) and PROVE-IT(FRiV).
UNCLASSIFIED
Tentative Program
(time is indicated as a guideline only)
8:45 Welcome and Introduction to the workshop - Pierre / Dmitry
9:00 Intro to PROVE-IT(FRiV) project: Objectives & Logistics – David
Assess applicability of FRiV technologies for various video surveillance tasks
Investigate, develop and test required Face Processing (FP) components
1. Pre-processing, Post-processing, Fusion
2. Face Detection, Face Tracking, Face Tagging
Intro to PROVE-IT(VA) project Objectives & Logistics – David
Analyze Technology Readiness Level (TRL) of various VA technologies:
Common objectives:
Develop evaluation methodology (including data-sets, mockups, and pilots).
9:30-12:30
BIOM 401 PROVE-IT(FRiV) project:
Presentations from Project’s main contributors -Tasks, proposed solutions, results obtained to date
9:30 CBSA: Dmitry
Survey of commercial products; companies; deployments; media reports;
FR Projects in other countries / govn’ts / FISWG & Intern’l Face Collaboration Meeting
FR Software SDK selected / acquired for testing
FR Datasets selected / acquired for testing
Integration of commercial FD / FR software into VAP & In-house code development
Other: R&D investment priorities
10:00 ETS: Eric
Survey of public domain knowledgebase in FRiV: academic papers, patents, codes
Survey of datasets of faces in video
Evaluation methodology & benchmarking
Video-based FR using incremental learning neural networks
11:00 uOttawa TAMALE LAB: Stan
Post-processing for face triaging problem: feature engineering, feature selection, data
fusion, classifier selection, and performance evaluation, esp. with noise and limited set of
the training images
11:15 tentative – uOttawa VIVA Lab: Robert
Pre-processing for FRiV: Face/Person Tracking, best facial image selection
11:30 Presentations from partnersother (potential) contributors:
What FRiV problem keeps you awake at night?
What would you like to get out of this study for your organization? Which of the discussed VA
tasks are most interesting to your organization?
How can you contribute to this study ? (datasets, software, testing methodologies/setups,
cameras/equipment, opportunities for pilots?)
12:00 – 12:30:
Live demonstrations – with live feed from IP-camera, with datasets:
1. Face Detection / Tagging
2. “Watch List” screening against CBSA WANTED LIST
3. Matching faces across several video feeds
UNCLASSIFIED
4. Combing face and iris images
5. Tamper detection
12:30-13:30 Lunch (not provided)
13:30 Focus group discussion (roundtable): feedback, recommendations, next steps
14:30 – 16:30
BTS 402 PROVE-IT(VA) project:
Presentations from Project’s main contributors -Tasks, proposed solutions, results obtained to date
14:30 CBSA: Dmitry
Survey of commercial products; companies; deployments; media reports;
Milestone VMS VA functionality and partners
VA Projects in other countries / govn’ts
VA Software selected / acquired for testing
VA Datasets selected / acquired for testing
Integration of commercial VA software into VAP; In-house code development
Other: R&D investment priorities
15:00 uOttawa VIVA Lab – Robert
Survey, Selection and development of techniques, datasets and evaluation practices for VA.
people tracking problem : in crowded and non-crowded environments, dealing with occlusions,
applications to object dropping/removal, person-object tagging and loitering detection ;
experimental methodology & Benchmarking
15:15 CRIM -
Unattended / Left-Behind Baggage Detection problem: Survey, Selection and development of
techniques, datasets ; experimental methodology & Benchmarking
15:30 Presentations from other (potential) contributors:
What VA problem keeps you awake at night?
What would you like to get out of this study for your organization? Which of the discussed VA
tasks are most interesting to your organization?
How can you contribute to this study ? (datasets, software, testing methodologies/setups,
cameras/equipment, opportunities for pilots?)
16:00 - 16:30
Focus group discussion (roundtable): feedback, recommendations, next steps
Invited Attendees:
CBSA, DRDC-O, DRDC-T, DRDC-V, RCMP, PCO, CATSA, Public Safety, TC, FAITC, DND,
HomeOffice, FBI
)RXUWK,QWHUGHSDUWPHQWDO:RUNVKRSRQ
9LGHR7HFKQRORJLHV
IRU1DWLRQDO6HFXULW\
9716¶
6HSWHPEHU2WWDZD
2UJDQL]HGE\&%6$6(DQG'5'&&66
)25(:25'
'PLWU\*RURGQLFK\
*RY¶W
'5'&&66
&%6$1+46(96%
&%6$1+46ROXWLRQV
&%6$3URJUDPV
&%6$125
&%6$*7$$
&$76$
')$,7
5&039LGHRDQG$XGLR$QDO\VLV8QLW
5&037HFKQLFDO6HFXULW\%UDQFK
5&03%LRPHWULFV6HFWLRQ
3XEOLF6DIHW\
5&031DWLRQDO9LGHR6\VWHPV,QWHJUDWLRQ
6HFWLRQ
'5'&21HWZRUN,QIRUPDWLRQ2SHUDWLRQV
'5'&7RURQWR
'5'&9DOFDUWLHU
'1''LUHFWRUDWHRI*HRVSDWLDO,QWHOOLJHQFH
+RPH2IILFH
:HOFRPH
$FDGHPLD
(76
X2WWDZD7$0$/(/DE
X2WWDZD9,9$/DE
&5,0
5HJUHWV
)%,
1,67
2IILFHRIWKH3ULYDF\
&RPPLVVLRQHU
7&
9716KLVWRU\RXWFRPHV
/HGE\15&,,7ZLWKVXSSRUWIURP86,$53$
± ILUVWJHWWRJHWKHU!*R&GHSWV
± ILUVWOLYHGHPRRI9$LQF)DFHGHWHFWLRQDQGDQQRWDWLRQLQYLGHR
± ILUVWSRSXODWLRQRILVVXHV
± ILUVWFDOOIRUFRRUGLQDWLRQRIHIIRUWV
/HGE\&%6$6(ZLWKSDUWLFLSDWLRQIURP'+6,$53$
± 7KHPHʊ%XLOGLQJIXWXUHSURRIYLGHRVXUYHLOODQFHV\VWHPV966)”
± +LJKHUDWWHQGDQFH!GHSDUWPHQWV
± 0RUHOLYH9$GHPRV
± $GGUHVVLQJ&%6$¶VQHHGLQ966XSJUDGHV
2XWFRPHV
± 5HFRPPHQGDWLRQRQ2SHQ$UFKLWHFWXUHV\VWHPV
ZKLFKDOORZVLQWHJUDWLRQRI9$LQKRXVHWHVWLQJSLORW
± &'5203URFHHGLQJVSXEOLVKHGZLWKSUHVHQWDWLRQV
± &%6$9$3LORWV
9716¶
/HGE\&%6$6(DQG'5'&&66
7KHPHV
± 0RYLQJ9LGHR$QDO\WLFVLQWRRSHUDWLRQDOHQYLURQPHQW
± )DFHVLQYLGHR
7XWRULDOʊ9LGHR$QDO\WLFVLQVLGHWKH%ODFN%R[”
UH,QWURGXFLQJ75/IRU9$WHVWLQJ
'HPRQVWUDWLRQRI&%6$¶V9LGHR$QDO\WLFV3ODWIRUP9$3
± :LWK2SHUDWLRQDO,3FDPHUDV
± ,QFOXGHGIDFHGHWHFWLRQ
%UDLQVWRUPLQJLQVHDUFKIRUʊNLOOer”EHVW9$DSSOLFDWLRQV
2XWFRPHV
3URFHHGLQJVSXEOLVKHGDW'5'&SRUWDO
7ZR3673&)3RQSURSRVHG6R:
7HFKQRORJ\5HDGLQHVV/HYHO6RXUFH'2'
$FWXDOV\VWHPIOLJKWSURYHQWKURXJKVXFFHVVIXO
PLVVLRQRSHUDWLRQVRYHU
$FWXDOV\VWHPFRPSOHWHGDQGIOLJKWTXDOLILHG
WKURXJKWHVWDQGGHPRQVWUDWLRQ
6\VWHPSURWRW\SHGHPRQVWUDWLRQLQRSHUDWLRQDO
HQYLURQPHQWÅ3,/27
6\VWHPSURWRW\SHGHPRQVWUDWLRQLQUHOHYDQW
HQYLURQPHQWÅ02&.83
&RPSRQHQWYDOLGDWLRQLQUHOHYDQWHQYLURQPHQW
&RPSRQHQWYDOLGDWLRQLQODERUDWRU\HQYLURQPHQW
$QDO\WLFDODQGH[SHULPHQWDOFULWLFDOIXQFWLRQ
FKDUDFWHULVWLFSURRIRIFRQFHSW
7HFKQRORJ\FRQFHSWDSSOLFDWLRQIRUPXODWHG
%DVLFSULQFLSOHVREVHUYHGDQGUHSRUWHG
75/9$)5L9
75/9$3 a
9716¶
/HGE\&%6$6(DQG'5'&&66
7KHPHV
± 0RYLQJ9LGHR$QDO\WLFVLQWRRSHUDWLRQDOHQYLURQPHQW
± )DFHVLQYLGHR
7XWRULDOʊ9LGHR$QDO\WLFVLQVLGHWKH%ODFN%R[”
UH,QWURGXFLQJ75/IRU9$WHVWLQJ
'HPRQVWUDWLRQRI&%6$¶V9LGHR$QDO\WLFV3ODWIRUP9$3
± :LWK2SHUDWLRQDO,3FDPHUDV
± ,QFOXGHGIDFHGHWHFWLRQ
%UDLQVWRUPLQJLQVHDUFKIRUʊNLOOer”EHVW9$DSSOLFDWLRQV
2XWFRPHV
3URFHHGLQJVSXEOLVKHGDW'5'&SRUWDO
7ZR3673&)3RQSURSRVHG6R:
9$3
9LGHR$QDO\WLFV3ODWIRUP
&RQWLQXRXV9LGHR
(YHQW%DVHG9LGHR6WDWLVWLFV
'(L
«
«
(PEHGGLQJ9$LQWR2SHUDWLRQDO&&79
2SHQ$UFKLWHFWXUH
$OORZVWRGLUHFWO\
DFFHVVWKHYLGHRIHHG
$OORZVWRFRQWUROWKH
YLGHRIHHGTXDOLW\VL]H
Î9$3,3&DP&DSWXUH
&ORVHG$UFKLWHFWXUH
'RHVQRW«
'RHVQRW«
Î9$36FUHHQ&DSWXUH
,3,QWHUQHW
VAP
ISV
VAP
,3,QWHUQHW
ISV
,3,QWHUQHW
9716¶
/HGE\&%6$6(DQG'5'&&66
7KHPHV
± 0RYLQJ9LGHR$QDO\WLFVLQWRRSHUDWLRQDOHQYLURQPHQW
± )DFHVLQYLGHR
7XWRULDOʊ9LGHR$QDO\WLFVLQVLGHWKH%ODFN%R[”
UH,QWURGXFLQJ75/IRU9$WHVWLQJ
'HPRQVWUDWLRQRI&%6$¶V9LGHR$QDO\WLFV3ODWIRUP9$3
± :LWK2SHUDWLRQDO,3FDPHUDV
± ,QFOXGHGIDFHGHWHFWLRQ
%UDLQVWRUPLQJLQVHDUFKIRUʊNLOOer”EHVW9$DSSOLFDWLRQV
2XWFRPHV
3URFHHGLQJVSXEOLVKHGDW'5'&SRUWDO
7ZR3673&)3RQSURSRVHG6R:
9$)5IRU$LUSRUW32(
ʊ)DFH7ULDJLQJDW3,/”: cRYHUWVFUHHQLQJDJDLQVW
ʊ:DQWHG/LVW”
ʊ)DFH7DJJLQJDW([LW32(´DUFKLYDORIDOOIDFHV
VHHQLQ&%6$FDPHUDV±WLPHVWDPSHGDQQRWDWHG
&DPHUD7DPSHULQJGHWHFWLRQ
2WKHUH[WUDFWHGLQWHOOLJHQFH
$EQRUPDOPRWLRQV6WDWV7UHQGV:DLWLQJWLPH
9$)5IRU/DQG32(
¾ )RUDXWRPDWHG
HYLGHQFHH[WUDFWLRQ
”NonUHSRUWHr”
ʊ3RUWUXQQHU”
H[WHQGLQJ/35
6WDWLVWLFV7UHQGV
9716¶
/HGE\&%6$6(DQG'5'&&66
7KHPHV
± 0RYLQJ9LGHR$QDO\WLFVLQWRRSHUDWLRQDOHQYLURQPHQW
± )DFHVLQYLGHR
7XWRULDOʊ9LGHR$QDO\WLFVLQVLGHWKH%ODFN%R[”
UH,QWURGXFLQJ75/IRU9$WHVWLQJ
'HPRQVWUDWLRQRI&%6$¶V9LGHR$QDO\WLFV3ODWIRUP9$3
± :LWK2SHUDWLRQDO,3FDPHUDV
± ,QFOXGHGIDFHGHWHFWLRQ
%UDLQVWRUPLQJLQVHDUFKIRUʊNLOOer”EHVW9$DSSOLFDWLRQV
2XWFRPHV
3URFHHGLQJVSXEOLVKHGDW'5'&SRUWDO
7ZR3673&)3RQSURSRVHG6R:
PSTP 2010 CFP
• Biometrics – Study No. 2:
Real-time Face Recognition Technologies for Video-
surveillance Applications
Winning bid - BIOM401: PROVE-IT (FRiV) by CBSA
• Border and Transportation Security – Study No 2.
Video Analytics for Border and Transportation Security—
Indoor and Outdoor Environments
Winning bid - BTS402: PROVE-IT (VA) by CBSA
VT4NS 2011
• Led by CBSA-S&E and DRDC-CSS
Themes :
– Update on PSTP BIOM401 Study
– Update on PSTP BTS402 Study
• Tutorial: ―Face Recognition: inside the Black Box‖
• Presenting studies work plan & results to the community
• More demonstrations of extended VAP
– With several COTS Face Recognition SDK
– Using in-house developed FRiV
• Soliciting feedback & Brain-storming: on next steps
Expected outcomes:
• Feedback from everyone
• Feedback for next round of PSTP CFP
VT4NS portal
presentations and other documents will be posted at
https://partners.drdc-rddc.gc.ca/css/Portfolios/
Biometrics (Human ID Systems)/VT4NS
…
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 40 of 70
Appendix C: Third Interdepartmental Conference on Video
Technologies for National Security (VT4NS 2010), May 2010
“Video Analytics: Technology Maturity, Deployment Challenges, and Roadmap”
By D. Gorodnichy
9716
'U'PLWU\2*RURGQLFK\
9LGHR6XUYHLOODQFH%LRPHWULFV6HFWLRQ
6FLHQFHDQG(QJLQHHULQJ'LUHFWRUDWH
9LGHR$QDO\WLFV
7HFKQRORJ\0DWXULW\
'HSOR\PHQW&KDOOHQJHV
DQG5RDGPDS
2XWOLQH
:KDWLV9LGHR$QDO\WLFV9$
7HFKQRORJ\0DWXULW\DGDSWDWLRQIURP'R'
¾ 7HFKQRORJ\5HDGLQHVV/HYHO75/9$ «
¾ 2WKHU7HFKQRORJ\0DWXULW\FULWHULD
9$GHSOR\PHQWFKDOOHQJHV
7KUHH3KDVH5RDGPDSIRU9$GHSOR\PHQW
¾ 2XU
VW
REMHFWLYHWRWHVW75/9$
¾ 2XU
QG
REMHFWLYHWRUDLVH75/9$WR
:KDWLV9LGHR$QDO\WLFV"
¾ 9LGHR7HFKQRORJ\97RIWKH¶VZDVSULPDULO\FRQFHUQHG
ZLWKYLGHRFDSWXUH&DPHUDVDQG5HFRUGHUV
¾ 7RGD\97LV9LGHR&DSWXUH9LGHR$QDO\WLFV
WK
F
VW
F
VW
F
9LGHR$QDO\WLFVDND,QWHOOLJHQW9LGHR6PDUW&DPHUD9LGHR
5HFRJQLWLRQ&RPSXWDWLRQDO$QDO\VLVRI9LGHR'DWDWKDW
GHDOVZLWK$XWRPDWHG([WUDFWLRQRI,QWHOOLJHQFHIURP9LGHR
+LJKUHVROXWLRQV
,3FDPHUDV
'LJLWDO
$QDORJ
ÅAnimated image
example:
The entire 17:00- 24:00
activity is summarized into
a few annotated
snapshots
(with NRC’s ACE
Surveillance™).
:LWKRXW9LGHR$QDO\WLFVDVLWLVQRZ
$UHDRI8QH[SORLWHG
,QIRUPDWLRQ
DATA
COLLECTED
DATA USED
:LWK9LGHR$QDO\WLFVZKDWZHFDQGR
$UHDRI8QH[SORLWHG
,QIRUPDWLRQ
DATA
COLLECTED
DATA USED
&DPHUDVHWXS$QQRWDWHG&(6
$&(GDLO\VXPPDUL]DWLRQ
2XWGRRUZLUHOHVVH\HOHYHO
2XWGRRUZHEFDPRYHUYLHZ ,QGRRUZLWKVXQOLJKW&&79
,QGRRUZRVXQOLJKW&&79
$&(6XUYHLOODQFH9$3LORW>*RURGQLFK\1$72@
9LGHR7HFKQRORJ\FRPSRQHQWVFRVW
9LGHR'DWD&DSWXUH
7UDQVIHU
6WRUDJH
3URWHFWLRQDQGVHFXULW\
,QWHJUDWLRQZLWKRWKHUVHQVRUVVRIWZDUH
0RWLRQKHDWVHQVRUDXGLR9LGHR$QDO\WLFV
9LGHRGDWDPDQDJHPHQW
,QGH[LQJYLVXDOL]DWLRQUHWULHYDORIGDWD
'DWD YLGHRDVVRFLDWHG0HWDGDWD
$QQRWDWLRQVREWDLQHGZLWK9LGHR$QDO\WLFV
9LGHRDQDO\WLFVIRUDXWRPDWLRQDQGILOWHULQJ
5HDOWLPHHYHQWGHWHFWLRQUHFRJQLWLRQ
$QDO\VLVRIDUFKLYHGYLGHRGDWD
¾ FDPHUDV
¾ HQFRGHVGHFRGHUV
¾ WUDQVPLWWHUVDQGUHFHLYHUV
¾ URXWHUVDQGPXOWLFDVWVZLWFKHV
¾ QHWZRUNYLGHRUHFRUGHUV
¾ VWRUDJHPHGLD
7HFKQRORJLFDOO\
VROYHG
%HLQJVROYHG±
UHTXLUHVH[SORULQJ
HYDOXDWLRQWXQLQJ
LQF5),3LORWV
7RWDOFRVW +DUGZDUH6RIWZDUH7HVWLQJ7XQLQJ
9LGHR$QDO\WLFVH[SHUWLVHPLQLPL]HVWKHFRVWRIERWKFRPSRQHQWV
2SHUDWLRQDO&RVW9DOXH
:KDW9LGHR$QDO\WLFVLV127"
¾ ³%ULJKWQHVVFKDQJHGHWHFWLRQ´PDUNHWHGE\LQGXVWU\DV³PRWLRQ
GHWHFWLRQ´ZKLFKLV127
¾ 127D³0DJLFEXOOHW´
-XVWDVZLWK$1<LPDJHUHFRJQLWLRQLQF%LRPHWULFVWKHUHZLOOEH
³)DOVH+LWV´DQG³)DOVH0LVVHV´
Î+RZHYHUWKHLU5DWHVFDQEHPLQLPL]HGWRDFFHSWDEOHIRURSHUDWLRQDO
QHHGV±E\HYDOXDWLRQDQGFXVWRPL]DWLRQ
¾ 12³RQHVL]HILWDOO´VROXWLRQHVSLQ1RQFRRSHUDWLYHVFHQDULRV
'LIIHUHQW9$FRGHVUHTXLUHGIRUHDFKVHWXSHQYLURQPHQWWDVN
Î+RZHYHUH[SHUWVPD\XVHWKHVDPHOLEUDU\WRZULWHWKHVHFRGHV
¾ ³+LJKUHVROXWLRQTXDOLW\´GR127DVVXPH³KLJKLQWHOOLJHQFH´
¾ ,WLV127H[SHQVLYHZLWKSURSHUXQLTXHVNLOOVDQGSODQQLQJ
,QIDFWLWVLJQLILFDQWO\UHGXFHVWKHHQWLUHFRVWRSHUDWLRQWKRXJKWKH
RSWLPL]HGHTXLSPHQWEXLOGXSDQGHIILFLHQWGDWDDQDO\VLV
0RQLWRULQJ7DVNV3HUIRUPHGE\+XPDQ6WDWXVTXR
7ZRPRGHVDUHDOWLPHPRQLWRULQJESRVWHYHQWDQDO\VLV
9LGHRFOLSV7E
$UFKLYDOPRGH
RIRSHUDWLRQ
5HDOWLPHPRGH
RIRSHUDWLRQ
0RQLWRULQJ7DVNV3HUIRUPHGE\+XPDQ6RIWZDUH
7ZRPRGHVDUHDOWLPHPRQLWRULQJESRVWHYHQWDQDO\VLV
9LGHRFOLSV7E
$UFKLYDOPRGH
RIRSHUDWLRQ
5HDOWLPHPRGH
RIRSHUDWLRQ
,QVWDQW
$QDO\VLV
$/$50
0HWDGDWD*E
&DSWXUH
ILOWHUVPRGXOHV
5HWULHYDO
PRGXOHVVRIWZDUH
0DLQFRQGLWLRQ2SHQ$UFKLWHFWXUH
7REHDEOHWRWDSLQWRYLGHRVLJQDOLQSXWDQGGDWDRXWSXW
&ULWLFDODQDO\VLVRI9$0DUNHWDQGUHDGLQHVV
0DLQTXHVWLRQ
:KDWLVSRVVLEOH
ZKDWLVQRW
DQG
KRZWRGLVWLQJXLVKRQHIURPWKHRWKHU"
,QVLGHU¶VYLHZ±KRZLQGXVWU\GRHVLW
¾ 7KHUHDUH!FRPSDQLHVUHJLVWHUHGLQ&DQDGDGRLQJEXVLQHVVLQ³6XUYHLOODQFH´
>18$166HDUFK5HVXOWVLQ@
¾ $Q\RQHZKRGRHV6XUYHLOODQFH´FDQDGG³,QWHOOLJHQW´RU³6PDUW´MXVWE\GRLQJ³SL[HO
EULJKWQHVVFRPSDULVRQ´
¾ 6RPHJRIXUWKHU
E\FUHDWLQJPDQ\³KHXULVWLFV´
QR8QLYHUVLW\NQRZOHGJHUHTXLUHG
WXQHGIRURQHVHWWLQJGRHVQRWZRUNIRUDQRWKHU
XVH3XEOLFGRPDLQFRGHVRIZKLFKWKHUHPDQ\
2&5)DFH'HWHFWLRQ6RPH)DFH5HFRJQLWLRQ
0DQ\ORZOHYHOLPDJHSURFHVVLQJOLEUDULHVHGJHGHWHFWLRQFRORXU
VHJPHQWDWLRQPRWLRQRSWLFDOIORZFRPSXWDWLRQ
¾ +RZHYHUYHU\IHZJRIXUWKHU±KLUH06F3K'LQ&RPSXWHU9LVLRQ3DWWHUQ
5HFRJQLWLRQWRGRKLJKHUOHYHOVHPDQWLFSURFHVVLQJRILPDJHV±ZKLFKLVVWLOORQH
WKHPRVWFKDOOHQJLQJUHVHDUFKDUHDVHJ6HH&59FRQIHUHQFH
HJKWWSZZZFRPSXWHUURERWYLVLRQRUJWXWRULDOBGD\KWPO
¾ $VDUHVXOWPDQ\FRPSDQLHVOLVWPDQ\9$WDVNVWKH\³FDQ´GR
9$WDVNVFRPSDQLHV³FDQ´GR
)URPZZZLGYUFDZZZLQWHOOLYLHZFDZZZYLVXDOFRUWHNFRPZZZPLRYLVLRQFRP
ZZZPDUFKQHWZRUNVFRPZZZLRLPDJHFRPZZZQLFHFRPZZZLQGLJRYLVLRQFRP
ZZZLRPQLVFLHQWFRPZZZREMHFWYLGHRFRPDQGPDQ\PRUH
¾ +XPDQ2EMHFW5HFRJQLWLRQDQG7UDFNLQJ
¾ 2EMHFW&ODVVLILFDWLRQ
¾ 3HRSOH&RXQWV
¾ 9HKLFOHUHFRJQLWLRQ
¾ 3HRSOHUHFRJQLWLRQ)DFHUHFRJQLWLRQ
¾ 2EMHFWOHIWEHKLQG8QDWWHQGHG%DJJDJH'HWHFWLRQ
¾ 2EMHFW5HPRYDO'HWHFWLRQ
¾ /RLWHULQJ'HWHFWLRQ
¾ 7DLOJDWLQJ
¾ >:DLWLQJ@/LQH&RQWURO&URZGPDQDJHPHQW
¾ 6SHFLDO$WWULEXWH'HWHFWLRQ
¾ $GYDQFHG%HKDYLRXU$QDO\VLV
¾ 6OLSDQG)DOO'HWHFWLRQ
¾ ,QWUXVLRQ'HWHFWLRQ9LUWXDO7ULSZLUH
¾ $XWRQRPRXV37=7UDFNLQJ
¾ 6WRSSHG9HKLFOH'HWHFWLRQ
¾ &DPHUD7DPSHULQJ'HWHFWLRQ
¾ &RQJHVWLRQGHWHFWLRQ
¾ &RXQWHU)ORZ
¾ $XWRPDWLF/LFHQFH3ODWH5HFRJQLWLRQ
¾ 2EMHFW$OWHUDWLRQ'HWHFWLRQ
¾ $XGLRDQG6RXQG&ODVVLILFDWLRQ
¾ )DFH'HWHFWLRQ)DFH7UDFNLQJ
¾ *UDIILWL9DQGDOLVPGHWHFWLRQ
¾ +LJKZD\YHKLFOHFRXQW
¾ $XWRPDWHGVXPPDUL]DWLRQRIDUFKLYHG
YLGHR
,QVLGHU¶VYLHZ±KRZ$FDGHPLDGRHVLW
¾ 9$LVRQHRIWKHOHDVWUHVROYHGDQGPRVWUHVHDUFKHG
DUHDV
(YHU\\HDU!PDMRUFRQIHUHQFHV!SDSHUVSXEOLVKHG
(YHU\\HDUSUDFWLFDOO\HYHU\XQLYHUVLW\KDV06F3K'
VWXGHQWVZRUNLQJRQXQUHVROYHGSUREOHPVLQFRQWHQW
VHPDQWLFVUHWULHYDOIURPLPDJHVYLGHR
¾ 6RPH9$WDVNVDUHLQGHHGQRZSRVVLEOH
%XWDOOKDYHOLPLWDWLRQV
¾ 0RVWRIUHSRUWHG9$VROXWLRQKDYHQHYHUEHHQWULHGLQOLYH
RSHUDWLRQDOHQYLURQPHQWV
7HFKQRORJ\0DWXULW\IURP'R'1$6$
0DLQ7HFKQRORJ\5HDGLQHVV/HYHO75/
¾ LQWHQVLYHO\XVHGE\'R'1$6$'+6'1,
3URYLGHVD&RPPRQ8QGHUVWDQGLQJRI67([LW&ULWHULD
$5LVN0DQDJHPHQW7RRO$OORZVWRHVWLPDWH
&RVW,QYHVWPHQWUHTXLUHG
¾ +RZHYHULWLVRQO\RQHGLPHQVLRQRI7HFKQ0DWXULW\
75/
5LVNVRU8QNQRZQV
8VLQJ75/VWR&RQWURO5LVNRI7HFKQRORJ\7UDQVLWLRQ
/RZULVNIRUWUDQVLWLRQ
+LJKULVNIRUWHFKQRORJ\WUDQVLWLRQ
5HTXLUHPHQWV
2WKHU7HFKQRORJ\PDWXULW\FULWHULD
$GRSWHGIURP$LU)RUFH5HVHDUFK/DE
¾ 3URGXFLELOLW\±0DQXIDFWXULQJ5HDGLQHVV
¾ 5HDGLQHVVWR5HFHLYH,I*RDO,V7UDQVLWLRQIURP
7HFKQRORJ\'HYHORSHUDQG7HFKQRORJ\5HFHLYHU
¾ 3UDFWLFH%DVHG7HFKQRORJ\0DWXULW\(PSKDVLVRQ
&RPPXQLW\RI8VHUV±8VHU5HDGLQHVV
¾ 3URJUDP5HDGLQHVVSURJUDPQHHGVDQGFRQVWUDLQWV
DGGHGE\96%
¾ 5HTXLUHG,QKRXVH5'FDSDFLW\5'5HDGLQHVV
&XVWRPL]DWLRQWXQLQJUHTXLUHG
7HFKQRORJ\5HDGLQHVV/HYHO6RXUFH'2'
%DVLFSULQFLSOHVREVHUYHGDQGUHSRUWHG
7HFKQRORJ\FRQFHSWDSSOLFDWLRQIRUPXODWHG
$QDO\WLFDODQGH[SHULPHQWDOFULWLFDOIXQFWLRQ
FKDUDFWHULVWLFSURRIRIFRQFHSW
&RPSRQHQWYDOLGDWLRQLQODERUDWRU\HQYLURQPHQW
&RPSRQHQWYDOLGDWLRQLQUHOHYDQWHQYLURQPHQW
6\VWHPSURWRW\SHGHPRQVWUDWLRQLQUHOHYDQW
HQYLURQPHQW
6\VWHPSURWRW\SHGHPRQVWUDWLRQLQRSHUDWLRQDO
HQYLURQPHQW
$FWXDOV\VWHPFRPSOHWHGDQGIOLJKWTXDOLILHG
WKURXJKWHVWDQGGHPRQVWUDWLRQ
$FWXDOV\VWHPIOLJKWSURYHQWKURXJKVXFFHVVIXO
PLVVLRQRSHUDWLRQVRYHU
75/PRVW9$
75/$&(6XUYHLOODQFH a
75/9$3 a
'RLQJ9$75/DVVHVVPHQW
¾ '1,9$&(±VHOHFWHGVXFFHVVIXO9$SURMHFWV
¾ 6HYHUDO9LGHRGDWDVHWVGHYHORSHGWRWHVW9$
1,6775(.9,'±WKHPRVWFRPSUHKHQVLYH
&/($5XVHGE\9$&(
)URP+RPH2IILFH8.
+RZHYHU7KH\DOORZWRWHVWDFKLHYH75/XSWRRQO\
LHSURWRW\SHLVVXFFHVVIXOLQUHOHYDQWHQYLURQPHQW
%HVLGHVWKHSDUWLFLSDQWDUHDOOIURP$FDGHPLDQRWIURP
,QGXVWU\
6RKRZZLOO\RXNQRZWKH75/RIWKH9$WKDW\RXQHHG"
9LGHR$QDO\WLFV7HFKQRORJ\5HDGLQHVV
7<3(5HDOWLPHPRQLWRULQJWDVNV
&XVWRPL]DWLRQ
WHVWLQJUHTXLUHG
7HFKQLFDO
UHDGLQHVV
3URJUDP
UHDGLQHVV
)DFHH[WUDFWLRQWDJJLQJ´
/LWWOH 5HDG\IRU3LORW
:URQJGLUHFWLRQGHWHFWLRQ5XQDZD\DODUP³
/LWWOH 5HDG\IRU3LORW
/RLWHULQJDODUP
0DMRU
2EMHFWOHIWEHKLQGRUDEXQGDQWREMHFWDODUP
0DMRU
7ULSZLUHWUHVSDVVLQJDODUP
/LWWOH 5HDG\5)3
2WKHUHYHQWVGRRURSHQLQJFDUSDUNLQJHWFDODUP
0DMRU
*HQHUDO7UDFNLQJ'HWHFWLRQRISHRSOHLQPXOWLSOHVWUHDPV
7<3(3RVW(YHQW$UFKLYDOPRQLWRULQJWDVNV
6XPPDU\RIGHWHFWHGHYHQWVVWDWLVWLFVWUHQGV
/LWWOH0HGLXP
6HDUFKLQJIRUDREMHFWSHUVRQLQVWRUHGVWUHDPV
/LWWOH0DMRU
*HQHUDO6XPPDU\6HDUFKLQXQVWUXFWXUHGHQYLURQPHQW
6SHFLDOFDVHWDVNV
/35/LFHQVH3ODWH5HFRJQLWLRQ
1RQH
)DFH5HFRJQLWLRQ
/LWWOH0HGLXP
7UDGLWLRQDOO\SHUIRUPHGE\+XPDQVPDQ\RIWKHVH0RQLWRULQJ7DVNV
FDQQRZEHIDFLOLWDWHVZLWK9$VRIWZDUH
5 – ready for pilot, 4 – requires Evaluation, 3/2– requires further Refining/Exploration, 1 – not ready yet
0DLQREMHFWLYHRI5'SULRUWRGHSOR\PHQW
2XUJRDOZLWK9$39$7
¾ WRWHVW75/DFKLHYH75/9$ LQ
RSHUDWLRQDOHQYLURQPHQW
DQG
¾ SRVVLEO\WRDWWDLQ75/DQGIRUVRPH
9$WHFKQRORJLHV
¾ ZRUNZLWK
2SHUDWLRQV3URJUDPV,QGXVWU\WR
H[DPLQHRWKHUGLPHQVLRQVRI9$
7HFKQRORJ\PDWXULW\
9$GHSOR\PHQWFKDOOHQJHV5RDGPDS
3KDVH)RXQGDWLRQ%XLOGLQJ%XVLQHVV,QIUDVWUXFWXUH
IRXQGDWLRQV5'FDSDFLW\
)URP.QRZLQJWKH$UWRI3RVVLEOH75/WR0DNLQJLW3RVVLEOH
'HDOLQJZLWK6WHUHRW\SHV0LVFRQFHSWLRQV
1RWWRRYHUHVWLPDWHRUXQGHUHVWLPDWHZKDW9$FDQGR
,QYHVWLJDWLQJ75/ULVNVFDSDFLW\WRGR5'
3KDVH'HYHORSPHQW5HVROYLQJ7HFKQRORJLFDO&KDOOHQJHV
5'SURJUDPPLQJ.QRZOHGJHRI2SHUDWLRQDO7DVNV
'HDOLQJZLWK³&ORVHG$UFKLWHFWXUHV´
6HOHFWLQJ%XLOGLQJDVROXWLRQ
3KDVH3LORWV7HVWLQJDQGGHSOR\LQJWKHVROXWLRQLQWKH³ILHOG´
.QRZLQJ&OLHQWVQHHGV(GXFDWLQJ7UDLQLQJWKH&OLHQW
&XVWRPL]LQJVWUHVVWHVWLQJLQ0RFNXSDQGUHDOVHWXSV
%XLOGLQJ9$6ROXWLRQ7HFKQLFDOFKDOOHQJHV
'LIIHUHQWWDVNVDQGVFHQDULRVUHTXLUHGLIIHUHQW9$FRGHVWREHZULWWHQDQG
WKHFXVWRPL]DWLRQRIWKH9$FRGHVFDQEHSURSHUO\GRQHRQO\E\D9LGHR
5HFRJQLWLRQH[SHUW$WWKHVDPHWLPH9$FXVWRPL]DWLRQUHTXLUHVVWURQJ
NQRZOHGJHRIRSHUDWLRQDOWDVNVDVZHOODVFRQVWDQWFRPPXQLFDWLRQZLWK
WKHLQYROYHGUHJLRQV$VDUHVXOWDVROXWLRQFRPLQJIURPRXWVLGHLVRIWHQ
YHU\H[SHQVLYHDQGLQPDQ\FDVHVQRWUHOLDEOH
,3FDPHUDVFRQWDLQYHQGRUVSHFLILFFRGLQJHQFRGLQJPHFKDQLVPVDQG
JHWWLQJDYLGHRIHHGIURPWKHVHFDPHUDVUHTXLUHVFXVWRPL]DWLRQLQ
SURJUDPPLQJFRGHVVSHFLILFWRHDFKYHQGRU)XUWKHUPRUHZKLOHVRPH
YHQGRUVSURYLGHIXQFWLRQDOLW\WRSHUIRUPGLUHFWFDSWXUHRIWKHYLGHRIHHG
IURPWKHLUFDPHUDVWKURXJKWKHXVHRIDGHGLFDWHG6'.2SHQ
$UFKLWHFWXUHFDPHUDVRWKHUVGRQRW&ORVHG$UFKLWHFWXUHFDPHUDV
6HOHFWLQJDJRRG9$SURGXFWUHTXLUHVWKHWHVWLQJRIGLIIHUHQWSURGXFWVIRU
WKHSXUSRVHRIPHDVXULQJDQGFRPSDULQJWKHLUSHUIRUPDQFHIRUDJLYHQ
WDVN0RVWDJHQFLHVFDQQRWDIIRUGWRSHUIRUPVXFKWHVWLQJDQGKDYHWRIXOO\
UHO\RQWKHYHQGRUV¶FODLPVLQVWHDG
7KHVHFKDOOHQJHVDUHEHLQJUHVROYHGE\
¾ &UHDWLQJ&%6$6(96%VHFWLRQ
¾ 'HYHORSLQJ9$39$7WHFKQRORJ\
«SURFHHGWRQH[WSUHVHQWDWLRQ
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 45 of 70
Appendix D: Third Interdepartmental Conference on Video
Technologies for National Security (VT4NS 2008), October 2008
Background, Program and References
“Intelligent Surveillance: examples, myths and lessons”
By D. Gorodnichy
UNCLASSIFIED
Second Interdepartmental Meeting on
Video Technologies for
National Security (VT4NS’08)
CENTURION CONFERENCE & EVENT CENTER
170 Colonnade Road South, Ottawa, K2E 7J5
23 October 2008
Theme: Deploying "future-proof" video technology
Goals | Participants | Presentations* | Repository | Next steps
Organized by: CBSA LSSD (Canada Borders Services Agency, Laboratory and Scientific Services Directorate).
In coordination with: US DHS (Department of Homeland Security), and US DNI-IARPA (Director of National
Intelligence, Intelligence Advanced Research Projects Activity), Computer Vision section, formerly DTO-VACE
(Disruptive Technology Office, Video Analysis and Content Extraction).
Coordinator: Dmitry O. Gorodnichy, CBSA LSSD (Dmitry.Gorodnichy@cbsa-asfc.gc.ca)
Background:
In the context of enhancing security, Video Technology (VT) is one of the most demanded technologies of the 21st
century.
Expanding and improving the capacity and functionality of VT is one of the current priorities for many federal
departments:
o US: fibre optic network along both borders, cams, motion/heat sensors deployed (2002-03) / General
purpose monitoring cams with remote monitoring deployed / Secure Border Initiative (2005, SCONSAD)
o CBSA, a major VT user at Points of Entry (POE): launched several VT-related initiatives and business
cases (2007-2008):
1. Port Runner (2007) - focus on visitors: capturing intelligence/evidence related to port
violators (faces, license plates).
2. Interview Rooms (2007) - focus on personnel/visitors monitoring
3. Port Integrity (part of Agency Integrity, 2008) - focus on personnel: deals with violations
within the agency.
4. Call for increased deployment of face recognition and other biometric technologies at border
crossings and ports of entry (inc. related to NEXUS).
5. Video-based identification of cargo containers (related to RADNET)
6. Video-based statistics measurements.
Intra-departmental camera working group (CWG) created (Mar 2008),
Plans to create a new LSSD section on Video Technology to guide and support VT deployment and
usage by the agency (inc. Biometric Surveillance technologies)
o TC: extensive funding for VT announced in 2007, released CCTV manual.
o RCMP: extra funding for VT, focus on Media Management System, face recognition in video
o DRDC: face recognition from video biometrics RFP
While VT of the past was primarily concerned with video capture and storage, VT of the future is more
concerned with Video Data Management, Video Analytics and its integration with other sensor systems.
IARPA (DTO-VACE): conducts evaluation and selection of the next-generation video analytics technologies. A
history of relationship with IARPA (DTO-VACE) since 2005.
The first VT4NS meeting was organized by NRC-IIT Video Recognition Systems, held in June 2007.
Goals:
The VT4NS workshops bring together project leaders and principle investigators working on deploying and
using Video Technology (VT) within the government of Canada. The workshops consist of several invited
oral presentations, followed by focus-group discussions on the raised issues.
Objectives:
1. To synchronize the effort in developing solutions for Video and related (Audio, and Biometrics) Technologies for
the new century.
2. To connect VT users and VT experts with each other .
o VT experts present SOTA (state of the art) in the area;
o VT users present Pilot projects, case studies, wish list and Action items.
3. To populate a list of interests and expertise, and to create a common knowledgebase.
4. To address action items and specific technical issues from the following topic list.
Topics:
1. Quality of video capture (equipment internal & external factors)
2. Camera locations and applications: Port Runner vs. Port Integrity
3. "Procedural" adjustments to ensure proper and efficient use of VT
4. Video data retention and storage (inc. review, disclosure and destruction)
1. Video data viewing rights, sharing and transfer.
5. Video for person identification (with iris and face recognition)
6. Use of visual data in courts as evidence.
7. Automated video recognition and analytics:
1. In real-time: alert enhancement, and meta-data insertion
2. In post-event investigation: video data analysis and content retrieval.
8. Combining video surveillance systems with audio and other sensors
9. Selecting video management systems
Participants:
IARPA / Computer Vision section (formerly, DTO-VACE)
DHS / Science and Technology Directorate / Suspicious Behavior Detection Programs
CRIM (Computer Research Institute of Montreal), L'equipe Vision et Imagerie
CRC (Industry of Canada, Communications Research Centre)/Advanced Video Systems
CBSA / LSSD / Advanced (Surveillance & Biometric) Technologies, Electronics and Computer Systems, Network
and Computer project
CBSA / Architecture division / Concepts and Consultations
CBSA / Travelers Projects and Systems, People Systems
CBSA / Operations / Professional Standards
CBSA / Camera Working Group (inc. Enforcement, Corporate Security, Regions)
RCMP / Surveillance Technology Section / Covert Vide(CV), Remote Sensing Technologies (RST) and Special
Purpose Vehicle (SPV) units
RCMP / Video/Audio Analysis Unit, Technical Investigation Services Branch
RCMP / Technical Security Branch
Transport Canada / Security Technology / Security and Emergency Preparedness
Office of the Privacy Commissioner of Canada
DRDC / Automated Intelligent Systems/UAV
DRDC / Network Information Operations Section
DND / Forces
In total, 45 people representing over 30 different federal groups/departments participated in the workshop. The participants
contact information can be requested from the workshop coordinator.
List of participants [PDF]
Program and Proceedings
See also PDF files for printing
23 October (Thusday) CENTURION CONFERENCE & EVENT CENTER
8:30-9:00 Breakfast, Coffee/Tea*
9:00
Welcome message from CBSA-LSSD Director General [PPT, 5-year strategy]
Foreword from workshop chair [PPT]
Round table introduction names/groups and related projects/interests [PPT]]
9:45
Introduction to related DHS and IARPA activities and interests:
o DHS: "Behavior-Based Surveillance" [PDF]
o IARPA: "Overview of the IARPA Computer Vision Program" [PPT]
11:15 Break*
11:30
CRC: "Overview of CRC Advanced Video Systems activities" [PPT]
DRDC: Automated Intelligent Systems projects [PPT, demos]
DRDC: Video-based Facial Verification System for Information Security [PDF]
RCMP: "Overview of the RCMP Audio & Video Analysis Section" [PPT]
12:30-13:30 Lunch Buffet*
13:30
LSSD: "Intelligent Surveillance: examples, myths and lessons" [PPT,demos]
CRIM: CRIM projects related to Video (and Audio) Analysis [PDF]
RCMP: Video Management Software Evaluation Project [PPT]
15:00 Break*
15:15
VT at CBSA and discussions [PPT]
o Related projects, activities, action items:
vision statement, survey, policy issues, RFPs and Regional requests, interim recommendations / Best
Practices documents.
o Cameras for physical security at CBSA
o Criminal Investigations Program
The relationship between Access to Information, Privacy (ATIP) and Disclosure Policy Division and Video
Technology
17:00 Adjourn
* Breakfast, lunch, snacks courtesy of CBSA-LSSD.
24 October (Friday) Laboratory and Scientific Services Directorate
9:00-11:00 LAB TOUR (Address: 79 Bentley Ave, Ottawa, ON, K1A 0L5)
Repository of related documents
See also Repository directory
IARPA : http://www.iarpa.gov
DNI InfoX, VACE programs:
o vace_brochure.pdf, DNI-VACE Presentation at NRC (August 2006).
o infox_brochure.pdf, DNI-InfoX Presentation at NRC (June 2007).
NRC Video Recognition Systems (2002-2007):
o bilingual brochure, Presentation for VACE (2006)
o Papers on ACE Surveillance, Face Recognition from video technologies
Evaluation of Intelligent Video solutions: from SAWER, VACE, UK Home Office
o Home Office CCTV and Imaging Technology i-LIDS dataset (copy available from LSSD)
CBSA:
o "Automated video surveillance: challenges and solutions. ACE Surveillance (Annotated Critical
Evidence) case study" (D.Gorodnichy, T.Mungham, NATO symposium "Sensor and Technology for
Defence Against Terrorism", April 2008).
o Port Runner Cornwall Pilot project: Report, Recommendations & Images (August 2008).
o CBSA Camera Working Group (CWG) interim documents and drafts (May-Nov 2008):
Policy drafts,
List of identified issues (for broad policy)
Recommendations for regions.
Survey questions
Video Surveillance Vision statement
o CBSA Physical security:
Guides:
Security Design of CBSA Facilities, Ch11, Appendix A, "Closed Circuit Television",
Comptrollership Manual (Protected A).
RCMP Threat and Risk Assessment (TRA).
Doctrinal sources
Government Security Policy (GSP)
Operational Standards for Physical Security
RCMP Technical Services Branch
CSE Information Technology Security Guides
Professional organizations and testing bodies for some elements (ASIS, CSIS, ULC, etc)
Concrete Sources
Results of a Threat and Risk Assessment then baseline requirements if no additional threat or
risk identified above general environment
From RCMP AVAS: Useful References on Video Evidentiary Requirements:
o Scientific Working Group on Imaging Technology (SWGIT) www.theiai.org/guidelines/swgit/index.php
o Technical Support Working Group (TSWG) http://tswg.gov/
Flipbook "Best Practices for the Retrieval of Video Evidence from Digital CCTV Systems" (2007).
Email: Pubs@TSWG.gov
Related PWGSC RFPs:
o DRDC "Video-based Facial Recognition: Algorithm and Demonstration." (Dec. 2008)
o CBSA "Advanced Video Management System" (Sep. 2008)
o CBSA "Access Control & Video Management" (Nov. 2008)
Standards:
o UL 2044 standard, Commercial Closed Circuit Television Equipment,
o IS23000-10 2008 (Copyright) Video Surveillance Application Format
o ONVIF (Open Network Video Interface Forum)
o PSIA ( The Physical Security Interoperability Alliance)
Media on Video Surveillance:
o < L'Actualite > (Oct. 15, 2008, p.14) article : "Big Brother est aveugle" ("Big Brother is blind")
Related links:
o http://ipvideomarket.info/companies
o www.canadiansecuritymag.com
o Video analytics:
CRV International Workshop on Video Processing for Security (VP4S'06).
IEEE-published Workshops on Video Processing and Recognition, research forums and labs in
Canada: www.computer-vision.org
Follow-up items:
Create a common Repository of Documents/references
o Share the Workshop Presentations
o Create on-line site for sharing the unclassified, public domain documents and links
o Interim internal documents (List of Issues, Survey Question etc) can be sent by request
Create an Interdepartmental Working Group (or Council) on Video Technology (based on the parties presented at
the workshop)
Contact Interdepartmental Working Group on Biometrics.
o Suggest organizing a similar academia-meets-operations workshop on Biometrics/Pattern Recognition.
Establish closer relationship with DRDC Center for Security Science
Have follow-up close-group focused meetings
o Video Management System process at RCMP
o Interim Recommendation documents/Criteria by CBSA
o Share comments/experience on currently used equipment (RCMP/CBSA)
o Face Recognition from Video (with DRDC,RCMP,CBSA)
Prepare for next VT4NS workshop
o Within SSC-PSTP
o US FBI Video Surveillance section, UK Home Office expressed interest in participating.
Last updated: 22.XII.2008
Intelligent Surveillance:
examples, myths and lessons
Dr. Dmitry Gorodnichy
Laboratory & Scientific Services Directorate
Canada Border Services Agency
VT4NS’08 Meeting
23 October 2008
Ottawa
2
Outline
Problems with status-quo Video Surveillance
Real-time and archival problems
Operational considerations
Demo
Next generation solution - Video Analytics based
“Motion detection” myth and problem
“Object detection” as example of real intelligence
Demos
Special interest - Faces
Conclusions
Other myths
What you can do with Faces
3
How it started:
Surveillance System = CCTV system
Basic architecture:
Camera + TV monitor (+ storage) + Agent
Hence, the name Closed Circuit Tele-Vision
3 modes of operation:
1. Active - personnel watch video at all times
2. Passive - in conjunction with other duties
3. Archival - for post-event analysis
Actively Used for:
monitoring protected/strategic areas
(by Physical Security, Enforcement, Infrastructure)
Demo
4
Problems in real-time modes
1. An event may easily pass unnoticed .
due to false or simultaneous alarms,
lack of time needed to rewind and analyse all video
streams.
2. No time to react in real-time (efficiently)
Because no intelligence is available (until Agent
extracts it from video data)
5
Problems in Archival mode:
Due to temporal nature of data:
1. Storage space consumption problem
• Typical assignment:
2-16 cameras, 7 or 30 days of recording, 2-10 Mb / min.
1.5 GB per day per camera / 20 - 700 GB total !
2. Data management and retrieval problem
• London bombing video backtracking experience:
“Manual browsing of millions of hours of digitized video from
thousands of cameras proved impossible within time-
sensed period”
[by the Scotland Yard trying to back-track the suspects]
[L’Actuality, 15 oct. 2008]
6
The only solution to these problems –
computerize Agent’s work!
Known as
Intelligent Video,
Video Analytics
Video Recognition,
Smart Cameras
Video Analysis & Content Extraction
Intelligent Filters / Adds-on / Modules
.H\LGHD±
&RPSXWHU6RIWZDUHLVDGGHGWRGRWKHMRE
0DLQFRQGLWLRQ±WREHDEOHWRWDSLQWRYLGHRVLJQDO
%RWKDDWFDSWXUHWLPHDQGEWRDUFKLYHGGDWD
9LGHRFOLSV7E
$UFKLYDOPRGH
RIRSHUDWLRQ
5HDOWLPHPRGH
RIRSHUDWLRQ
/DVWFDSWXUHG
&(6
$/$50
&(6*E
&DSWXUH
ILOWHUVPRGXOHV
5HWULHYDO
PRGXOHVVRIWZDUH
:KDWH[DFWO\ZHFDQGR"
z 6RPHUHVXOWVIURP15&9LGHR5HFRJQLWLRQ
6\VWHPVSURMHFW
z 'HPR/HWVFRQQHFWRXU79WRDODSWRS
0RUHGHPRV
6RPHWKLQJZKLFKLVTXLWHSRVVLEOH
z %HWWHUDODUPVUHVSRQVH
z $XWRPDWHGVXPPDU\DQQRWDWLRQV
z 'HWHFWLRQ5HFRJQLWLRQRIVSHFLILFREMHFWVGLUHFWLRQV
HYHQWV
z )DFHH[WUDFWLRQDUFKLYDOWDJJLQJ
z $XWRPDWHGWHOHRSHUDWRU
z 6WRUHGTXDOLW\TXDQWLW\GHSHQGVRQWKHFRQWHQW
z +LJK5HVVQDSVKRSVRUYLGHRRQO\ZKHQQHHGHG
z $QGPDQ\PRUHIRUDUFKLYHGWRQHVRIGDWD«
«EXWWKLVLVRQ5HVHDUFKVLGH
2QRSHUDWLRQDOVLGH«
z 9LGHRGDWDFDPHUDVEHFDPHDSSUHFLDWHG«
z QRWRQO\IRUWULJJHULQJDODUPVRUUHPRWHYLHZLQJ
z EXWDOVRSURYLGLQJVRXUFHRILQWHOOLJHQFHXVHIXOGDWD
z LWLV3XEOLFO\DFFHSWHG
Î([SORVLRQ³7VXQDPL´RI
z YLGHRWHFKQRORJ\SURMHFWVVHHSUHVHQWDWLRQLQ30
z LQIUDVWUDFWXUHEXLOGXSVDQG
z FDSWXUHGDQGVWRUHGGDWD
Î &RQVXOWDQWV3URYLGHUVDUHFRQWDFWHGIRUDGYLVH
«DQGWKLVLVZKDWZHJHW
%HWWHUHTXLSPHQW%87WKHVDPHPHQWDOLW\
&DPHUD79PRQLWRUVWRUDJH$JHQW
$YLJLORQ)URP3RUW
5XQQHU3LORWSURMHFW
/66'
6LPLODUVROXWLRQV
&RYL,QGLJR9LVLRQ
DQGPDQ\RWKHUV«
$QGWKHSUREOHPLV«
,I«
z DV\VWHPGRHVQ¶WVXSSRUW2SHQ$UFKLWHFWXUHDQG
z WUDQVPLWVDQGVWRUHVYLGHRGDWDXVLQJSURSULHWDU\IRUPDWV
\RX«
z ZRQ¶WEHDEOHWRDGG,QWHOOLJHQW9LGHRSURFHVVLQJPRGXOHV
RI\RXURZQ
z ZLOOKDYHWRUHO\HQWLUHO\RQWKHSURYLGHG³,QWHOOLJHQW´RSWLRQV
«ZKLFKLQPDQ\FDVHVDUHIDUIURPEHLQJ³,QWHOOLJHQW´
%HVLGHV\RXFDQ¶WHDVLO\UHSODFHXSJUDGHFRPSRQHQWV«
13
… and it is left to Agent
In real-time mode,
to figure out what happened (or didn’t happen) in ,
which may take long – demo
…because of limited “motion detection”
…no annotations / meta data (or possibility to add them)
In Archival mode,
to convert each “potentially useful” clip from
proprietary to standard format.
You can’t just take HD. – You need Provider’s software!
14
15
16
17
Status-quo “video intelligence”
Transport Canada CCTV Reference Manual for
Security Application .
Australian Government National code of practice for
CCTV applications in urban transport
USA Government :recommended security Guidelines
for Airport Planning, Design and Construction.
…. refer to “Motion-based” capture as Intelligent
Surveillance Technology, and make their
recommendations based on thereon.
18
More on “Motion-detection” myth
Term “Motion-based” is coined to make people believe that
video recognition is happening, which is not!
It’s actually illumination-change-based, as it uses simple
point brightness comparison: | I (pixel at t) – I (pixel at t+1) | > T
Which often happens not because of motion!
Changing light / weather (esp. in 24/7 monitoring)
Against sun/light, out of focus, blurred, thru glass
Reflections, diffraction, optical interferences
Image transmission, compression losses
19
“Object-detection” is intelligent …
… but few can do it, since necessary advances in video
recognition theory became possible only recently (>2002).
Example:
A 7-hour activity from day to
night (17:00 - 24:00) is
summarized into 2 minutes
(600Kb) of Annotated Critical
Evidence snapshots.
Note illumination changes! - Watch tree
shadows and sun light.
20
Adds on top of existing infrastructure
21
Status quo “Motion-based” capture
(Courtesy: NRC-IIT Video Recognition Systems project)
1. Many captured snapshots are
useless: either noise or
redundant
2. Without visual annotation,
motion information is lost.
3. Hourly distribution of
snapshots is not useful
22
ACE Surveillance “Object-based” capture
(Courtesy: NRC Video Recognition Systems project)
1. Each captured shot is useful.
2. Object location and velocity
shown augmentent.
3. Hourly distribution of shots is
indicative of what happened in
each hour, provides good
summarization of activities over
long period of time.
VT within CBSA 23
Camera / setup Annotated CES
ACE daily summarization
Outdoor, wireless, eye-level
Outdoor, webcam, overview Indoor with sunlight, CCTV Indoor w/o sunlight, CCTV
24
Conclusions
Affordable Intelligent Video Surveillance (IVS) is
possible!
However:
It is possible, only if Video data can be directly accessed
by 3
rd
party software (module)
Other:
Requires extra training from security officers.
Requires new protocols to handle automatically extracted
evidence.
- From forensic prospective, data that are not original and have
been processed by a computer can not be considered as evidence.
Requires new privacy policies.
- Surveillance data are normally not kept for a long period of time
(<1 month), due to their size. AVS allows to store on local machine
many months (even years) of evidence data.
25
Common myths
Naive “Motion detection” not to confuse with Object
Detection and Object Tracking.
The “one-fit-all” myth.
Extra video analytics expertise will always be required to set
and operate IVS.
Solutions are to be provided with SDK / API which allows to
customize them.
The more (data), the better.
“30 days or more, but it’s the data quality and ability to
efficiently retrieve the desired data that is also very
important!”
26
Special Interest – Face in Video
Video:
Taken in unconstrained
environment.
(in a “hidden” camera - like setup)
People
- don’t not look into camera
- even don’t face camera
Poor illumination
Blurriness, bad focus
Individual frames of poor quality
Video can also be MPEG compressed (for storage & transmitting)
Result from Pilot Project at
Cornwall (2007)
27
Conclusions wrt Face Recognition
Inner square part of the face is most important
12 i.o.d. , which is most frequent case, is sufficient for
recognition!
These face can be automatically extracted.
If less 10 i.o.d., body/gait biometrics should be used instead.
Technology is (almost) ready for
- For limited number of faces (limited access premises)
- Multiple-camera tracking
- Can be combined with hi-res photo-camera capture
Bottleneck:
- angle of view, quality of video
- 1 to Many is still a problem
- Demo
28
What is possible – best applications
1. Automated Recognition from ICAO-conformed passport
photographs - as good as finger or iris recognition.
2. Human assisted Recognition From Video (not biometrics):
Face is automatically extracted from video (e.g. to be linked
with boarding pass or vehicle plate number)
3. Face image and geometry automatically extracted from
video is used together with other modality (eg. Iris)
recognition.
4. Automated Recognition From Video only – is possible, if
procedural constraints are imposed (to make video
snapshot image quality closer to that of passport image
Currently, no business case is made. But we’ve collected
enough data/expertise (inc. from Pilot projects) to be ready!
Dmitry Gorodnichy et al, “Video Analytics technology: the foundations, market analysis and demonstrations"
Page 58 of 70
Appendix E: “Recognition in Video”, The Identity, Privacy and
Security Institute, University of Toronto Public Lecture Series,
November 30, 2009.
By Dmitry Gorodnichy
1.
The Identity, Privacy and Security Institute
University of Toronto Public Lecture Series
November 30, 2009
Dr. Dmitry O. Gorodnichy
Video Surveillance and Biometrics Section
Applied Research and Development Division
Science and Engineering Directorate
Recognition in VIDEO
2.
Outline
Part 1. History & Background
• Video, Recognition, in GoC
• Recognition taxonomy & main dilemma
• [ How brain does it ]
Part 2. Surveillance Recognition
• Objectives and Modes of operation
• Problems & Solutions
• Faces in Video: where Surveillance meets Biometrics
Part 3. Biometric Recognition
• Five Biometric Recognition Tasks
• Key Issue: Recognition Confidence Performance Evaluation
Discussion on Privacy Issues
3.
Part 1. History and Background:
Video, Recognition, in GoC
4.
Main Entry:
1
sto·ry
Etymology: Middle English storie, from Latin historia
Date: 13th century
narrative: a message that tells the particulars of an act or
occurrence or course of events;
tale: a piece of fiction that narrates a chain of related events;
history, account: a record or narrative description of past
events; "a history of France"; "he gave an inaccurate
account of the plot";
report: a short account of the news; "the report of his speech";
Invention of Movie(Video) made Story-telling much easier!
5.
What makes a story ?
• NOUNs
• VERBs
But how do you know that the story is true ?!
Is what you see on TV all true?..
Is what you see on CCTV sufficient to put someone in jail?..
6.
Recognition taxonomy & dilemma
Heisenberg‟s uncertainty principle:
Mother Nature does not allow both – temporal & spatial detail !
Recognize
what?
Automated By Human Requires:
Noun
(Identity)
Biometrics Forensic
examination
Spatial
details
Photo
Verb
(Activity)
Video
Analytics
CCTV
surveillance
Temporal
details
Video
D. Gorodnichy © 2009
:KDWLV9LGHR"
Æ $VHTXHQFHRIVWLOOLPDJHV
Æ&RQWLQXRXVVWUHDP
Æ &DSWXUHG'LVSOD\HG³IDVW´
Æ!IUDPHVSHUVHFRQG
Æ )URPWKHVDPHSRLQWRIYLHZ
&RQWLQXLW\LQWLPH±WKLVLVZKDW9,'(2LVDOODERXW
0DLQ(QWU\YLGāHRPRYāLH
0DLQ(QWU\
YLGāHR
(W\PRORJ\/DWLQYLGƝUHWRVHHRDVLQDXGLR
'DWH
0DLQ(QWU\(W\PRORJ\PRYLQJSLFWXUH
'DWH
0DLQ(QWU\PRWLRQSLFWXUH
)XQFWLRQQRXQ
'DWH
DVHULHVRISLFWXUHVSURMHFWHGRQDVFUHHQLQUDSLG
VXFFHVVLRQVRDVWRSURGXFHWKHRSWLFDOHIIHFWRID
FRQWLQXRXVSLFWXUHLQZKLFKWKHREMHFWVPRYH
DUHSUHVHQWDWLRQDVRIDVWRU\E\PHDQVRIPRWLRQSLFWXUHV
+LVWRU\RI9LGHR7HFKQRORJ\
¾
VW
F9,'(25(&2*1,7,21EHFDPHSRVVLEOH
¾
WK
F027,213,&785(9,'(2EHFDPHSRVVLEOH
WK
F
VW
F
VW
F
9LGHR$QDO\WLFV&RPSXWDWLRQDO$QDO\VLVRI9LGHR'DWDDQG
([WUDFWLRQRI,QWHOOLJHQFHIURP9LGHR
%HWWHUTXDOLW\PRUHDIIRUGDEOH
,3FDPHUDV
'LJLWDO
$QDORJ
ÅAnimated image
example:
The entire 17:00- 24:00
activity is summarized into
a few annotated
snapshots.
(with our VAP technology)
'HPDQGIRU9LGHR6XUYHLOODQFH
,QWKHFRQWH[WRIHQKDQFLQJVHFXULW\9LGHR6XUYHLOODQFHLVRQH
RIWKHPRVWGHPDQGHGWHFKQRORJLHVLQWKH
VW
FHQWXU\
0DQ\IHGHUDOGHSDUWPHQWV&DQDGDZRUOGZLGHDQQRXQFHG
PDMRUIXQGLQJIRUH[SDQGLQJ9LGHR6XUYHLOODQFHFDSDFLW\
6HYHUDOSURMHFWVLQLWLDWLYHVFXUUHQWO\DW&%6$«
$VDUHVXOW
³7VXQDPL´RIFDPHUDLQIUDVWUXFWXUHEXLOGXSDQGYLGHRGDWDWKDW
QHHGWREHUHFRJQL]HG±WRFUHDWHD758(6725<
)URP9LGHR$QDO\WLFVWR%LRPHWULFV
%HVWV\QHUJ\LPSDFWLVDFKLHYHGE\WUHDWLQJ9LGHR6XUYHLOODQFH
WRJHWKHUZLWK%LRPHWULFV
«EHFDXVHPRVW%LRPHWULFVDUHLPDJHEDVHG
6DPHXQLTXHVHWRIVNLOOV
,PDJH$QDO\VLV&RPS6FL</