Forty Years of Multiple Hypothesis Tracking - A Review of Key Developments

Full text

Forty Years of Multiple Hypothesis Tracking -
A Review of Key Developments
21st International Conference on Information Fusion, Cambridge UK
10 – 13 July 2018
1
Chee-Yee Chong
Independent Researcher
Los Altos, CA USA
Shozo Mori
Systems & Technology
Research
Sacramento, CA USA
Donald B. Reid
Independent Consultant
San Jose, CA USA

Outline
• Measurement-oriented MHT
• Track-oriented MHT
• Distributed MHT
• Graph-based association
• Relationship to random finite sets
• Applications and research directions
2

Multiple Hypothesis Tracking (MHT)
• MHT defers data association decisions when
association is difficult due to target density/
maneuvers, measurement errors, false alarms,
missing detections, etc.
• Maintains multiple data association hypotheses
• Makes decisions when sufficient good data are
available
• Need to use multiple frames/scans of data was
recognized before, but MHT considers association
hypotheses on all measurements, not just
individual tracks
• Key concepts (Morefield 1977)
• Track – sequence of measurements (indices)
hypothesized to originate from same target
• Data association hypothesis – set of consistent tracks
3
C. L. Morefield, “Application of 0-1 integer programming to multi-target tracking problems,”
IEEE Trans. Autom. Control., 1977
Truth and measurements
Best association with 2 scans
Best association with 3 scans

Measurement-Oriented MHT (MOMHT)
• MOMHT recursively generates and evaluates hypotheses on origins of
measurements (Reid 1979)
• Hypothesis evaluation (Mori et al 1986)
• False alarm likelihood
• Likelihood of measurement from a previously detected track
• Likelihood of previous detected track being undetected
• Likelihood of measurement from a newly detected target
4
D. B. Reid, “An algorithm for tracking multiple targets,” IEEE Trans. Auto. Control. 1979.
S. Mori, C. Y. Chong, E. Tse, and R. P. Wishner, “Tracking and classifying multiple targets
without a priori identification,” IEEE Trans. Auto. Control, 1986.
1
||
(|) () (|) ( |) ( | )
FA
kk kk
P
ZCZP ZLZ LZZ









()
[|] ()
FA
FA
FA j
kk FAk
jJ
LZ e z








||
( | ) ( |) () (| )( )
jj
kk Mkk Dk k
X
Lz Z p z xp xpxZ dx




||
(| ) (1 ()) (| )( )
kDk
X
LZ pxpxZdx




|
(
|
)(
|
)()()()
NT
jj
kk Mkk Dk NT
X
Lz Z e p z xp x x dx







MOMHT Processing
• Hypothesis management is used
to control combinatorial growth of
hypotheses
• Pruning low probability hypotheses
• Combining similar hypotheses
• Decomposing measurements and
tracks into clusters that can be
processed independently
• MOMHT requires efficient ways of
generating high probability
hypotheses, e.g., finding ranked
assignments (Cox and Miller
1995) with Murty’s algorithm
5
Data Frames
a
b
c
d
e
f
g
h
i
Association
Hypotheses
ad, be, cf
a, b, c
ad, bf, ce
ae, bd, cf
ae, bf, cd
af, bd, ce
af, be, cd
adg, beh, cfi
adg, bei, cfh
adh, beg, cfi
adh, bei, cfg
adh, bei, cfg
af, be, cd
ad, be, c, f
ad, be, c, f
afg, bdh, cei
afg, bdi, ceh
afh, bdg, cei
afh, bdi, ceg
adh, bei, cg, f
Frame 1 Frame 2 Frame 3
aeh, bdi, cfg
cfg
bei
I. J. Cox and M. L. Miller, “On finding ranked assignments with application to multitarget
tracking and motion correspondence,” IEEE Trans. Aerosp. Electron. Syst., 1995.

Track-Oriented MHT (TOMHT)
• Relies on batch hypothesis evaluation first introduced in Morefield 1977
and generalized in Chong et al 1989
• Hypothesis probability
• Track likelihood
• Measurement likelihood
6
C. Y. Chong, S. Mori, and K. C Chang, "Distributed multitarget multisensor tracking." in
Multitarget-Multisensor Tracking: Advanced Applications, Chapter 8, Y. Bar-Shalom, Ed.
Artech House, 1989
1
1: 1: 1: 1:
(| ) ( ) ( |) (, )
K
FA
K
KKKKKKK
PZ CZlZ lZ









1: | 1: |
1
(, ) (
|
)(
|
)( )
K
KK kk k k
k
lZ
g
ZxpxZ dx

 


(|) () if
(|) 1() if
MD
kD
pzxpx z
gzx px z









TOMHT Processing
• TOMHT maintains multiple tracks
over multiple data frames
• There is no explicit generation /
maintenance of association (global)
hypotheses
• Best (MAP) association hypothesis
over multiple frames can be found
by integer linear programming of
multi-dimensional assignment, e.g.,
Poore
• Best hypothesis can be used to
prune tracks after given window
• k-best solutions can be found by
generalization of Murty’s algorithm
7
Tracks
a
b
c
d
e
f
d
e
f
d
e
f
g
h
i
a
b
c
d
e
f
g
h
i
g
h
i
g
h
i
g
h
i
g
h
i
g
h
i
Frame 1 Frame 2 Frame 3
Association
Hypotheses
h
Best Hypo
2nd Best Hypo
4th Best Hypo
3rd Best Hypo
Pruning Window
A. B. Poore, “Multidimensional assignment formulation of data association problems arising
from multitarget and multisensor tracking,” Computational Optimization and Applications, 1994.

Distributed Processing
• Distributed processing (including HPC)
was recognized very early as means to
handle processing load of MHT
• Multi-stage MHT for single sensor
• Front end tracker compresses sensor
measurements
• Back end tracker processes compressed
measurements with MHT, e.g., in track
stitching
• Distributed MHT for multiple sensors
• Local sensor tracker generates tracks
• Track association/fusion associates local
sensor tracks using MHT and generates track
state estimates
• Requires de-correlation of local sensor tracks
into independent tracklets for processing
8
Front End
Tracker
Back End
Tracker
Track
Association
Fusion
Sensor 1
Tracker
Sensor N
Tracker
Multi-stage MHT for single sensor
Distributed MHT for multiple sensors

Graph-Based Association – Markov Association Likelihood
• Association graph (Chong 2012) provides
efficient implicit representation of possible
associations
• Nodes: measurements or tracklets
• Edges: possible associations
• Paths: tracks
• Path cover: association hypothesis
• Markov association likelihood
Implied by
• Then best association hypothesis can be
computed in polynomial time by
• Bipartite matching
• Minimum cost network flow (Castanon 1990)
9
a
a
b
c
g
h
i
d
e
f
b
c
d
e
f
g
h
i
C. Y. Chong, “Graph approaches for data association,” Fusion 2012
D. A. Castanon, “Efficient Algorithms for Finding the KBest Paths Through a Trellis,” IEEE
Trans. Aerosp. Electron. Syst., 1990.
11 1
(
|
,...., ) (
|
)
iiii
p
yy y
p
yy


1
(
|
,...., ) (
|
)
ii
px
yy
px
y

1
11
(|) ' (, )
NN
ijij
ij
PZC lyyx





Graph-Based Association – Non-Markov Likelihoods
• Many tracking problems have non-Markov or path-dependent likelihoods
•Examples
• Single sensor (e.g., radar) tracking - recursive filtering is useful
• Feature aided tracking – association likelihood depends on history
• Multi-sensor track association – association feasibility depends on history
• Graph-based association for non-Markov problems is active research area,
e.g., Coraluppi et al 2016
10
11 1
( | ,...., ) ( | )
iiii
py y y py y


1
( | ,...., ) ( | )
iiii
px y y px y

- - - -
- - - -
-
-
S1_1
S1_2
S2_1
S1_2 can associate with S2_1 but
with not (S1_1, S2_1)
S. Coraluppi, C. Carthel, W. Kreamer, and A. Willsky, “New graph-based and MCMC
approaches to multi-INT surveillance,” Fusion 2016..

Random Finite Sets (RFS) and MHT
• Multitarget Bayes filter
• Multitarget pdf from MHT
• Some common (mis-)beliefs
• RFS is the only correct approach for multitarget tracking
• Multitarget filter does not need data association
• MHT is an heuristic algorithm
11
1: 1 | 1 1| 1 1: 1 1
(X | Z ) = (X | X ) (X | Z ) X
kk kk kk k k k
fff




1
1: 1: 1: ( 1) 1: 1
(X
|
Z) [(Z
|
)] (Z
|
X) (X
|
Z)
kk k k Mkk kk
ffZff




1: 1:
(, )
#( ) #( )
(X | ) ( | ) ( ( ) | )
ˆ
Kk
k
kK kK k
aA X
X
fZ PZ pa
  











MHT and RFS Are Two Ways of Solving Same Problem
• Rigorous derivation of MHT (Mori et 1986) recognizes random set nature of
target state and measurements, e.g.,
• Explicit representation of number of targets and measurements in data set
• Symmetric or permutable joint target pdf
• MHT like trackers have been derived using RFS formalism
• Gaussian mixture CPHD filter is equivalent to MHT for single targets
•Multi-Bernoulli (MB) RFS filter has structure similar to MHT (Williams et al 2015)
• Labelled MB (LMB) RFS has structure similar to MHT (Meyer et al 2018)
• MOMHT can be derived from multi-Bayes filter by suitable interpretation of new
target density (Brekke/Chitre 2017, 2018)
12
J. L. Williams, “Marginal multi-Bernoulli filters: RFS derivation of MHT, JIPDA and association-
based MeMBer,” IEEE Trans. Aerosp. Electron. Syst., 2015.
F. Meyer, T. Kropfreiter, J. Williams, R. Lau, F. Hlawatsch, P. Braca, and M. Win, “Message
passing algorithms for scalable multitarget tracking,” Proc. IEEE, 2018
Edmund F. Brekke and M. Chitre, “The multiple hypothesis tracker derived from finite set
statistics,” Fusion 2017.
Edmund F. Brekke and M. Chitre, “Relationship between Finite Set Statistics and the Multiple
Hypothesis Tracker,” IEEE Trans. Aerosp. Electron. Syst., 2018.

Applications
• The conceptual foundation for MHT was established in late 1970s
• Research over the past 40 years has focused on efficient implementation
and applications and mostly outside academia
• 1980s
• MHT was proposed for many tracking problems, from trivial to impossible
• Success was limited due to computing hardware
• 1990s
• TOMHT and K-best MOMHT addressed combinational problem
• MHT was used for tracking framework that can adapt to computing resources
• 2000s
• MHT was tested on large tracking problems with real data
• Issues such as hypothesis hopping, hypothesis confidence, and scalability were
recognized
• 2010s
• MHT was used in real systems in many domains
• Graph-based association was developed to handle very large tracking problems
13

Possible Research Directions
• Relax current assumptions on MHT to handle
• Extended targets, merged measurements
• Known number of targets
• Dependent targets
• Predict MHT performance
• Cost benefit tradeoff
• Real-time performance, e.g., hypothesis probability
• Adaptive hypothesis management with real-time performance monitoring
• Solve general association graphs without path-independent assumption
• Feature-aided tracking
• Multi-sensor tracking
• Research on other MHT approaches
• Probabilistic MHT
• Retroactive MHT
• MCMC MHT
14