Reconstruction of a road by local image matches and global 3D optimization

Abstract

A method is presented for reconstructing a 3-D road from a single
image. It finds the images of opposite points of the road. Opposite
points are points which face each other on the opposite sides of the
road; the images of these points are called matching points. For points
chosen from one side of the road image, the algorithm finds all the
matching point candidates on the other side, based on local properties
of a road. However, these solutions do not necessarily satisfy the
global properties of a typical road. A dynamic programming algorithm is
applied to reject the candidates which do not fit the global road. A
benchmark using synthetic roads is described. It shows that the roads
reconstructed by the proposed method match the actual roads better than
those reconstructed by two other road reconstruction algorithms.
Experiments with 50 road images taken by the autonomous land vehicle
(ALV) showed that the method is robust with real-world data and that the
reconstructions are fairly consistent with road profiles obtained by
fusion between range images and video images

Full text

Page 1

Daniel DeMenthonandLarryS?Davis
ComputerVision Laboratory
Centerfor AutomationResearch
Univ
V?
ersit
w
y of
the
Maryland
metho
CollegePark?MD?????
Abstract
Anew methodis presentedforreconstructinga ?Droadfromasingleimage? It
?nds theimagesofoppositepoin ts oftheroad?oppositepointsarepoints whichface
eachotheronthe opposite sidesof the road?theimagesof thesepointsarecalled
matching points??Forpoin tschosen fromoneside oftheroadimage?the prop osed
algorithm?ndsallthe matc hingpoin tcandidatesonthe other side?based onlocal
propertiesofaroad?Ho w everthese solutionsdonot necessarilysatisfy the global
propertiesofatypicalroad?Adynamic programmingalgorithm isappliedtoreject
the candidateswhichdo not?t the globalroad?
Abenchmark usingsynthetic roadsisdescribed?whichshows thatthe roads re?
constructedby theprop osedmethodmatch theactualroadsbetterthantwoother
roadreconstruction algorithms? Experiments with??road imagestakenby theAu?
tonomousLandVehicle?ALshothatdisrobustwith realw orld data?
andthatthe reconstructionsarefairly consistent withroadpro?lesobtained byfusion
betweenrangeimagesandvideoimages?
?

Page 2

roads?????? ???? ???Forrobustnessinavarietyofconditions?these systemscan be
driv enbya sup ervisor systemreasoning ab outinformation pro videdbyseveralalgorithms?
suchasstereoalgorithms?stereomotionalgorithms?algorithms usingsinglevideo frames
or combiningvideo framesandrangeimages? Kahlmanpredictorscombining information
obtainedfrom several vehiclepositions?etc????Some algorithmsmay monitortheroadover
ashort distance oralongasingleedge?forinput toafaststeering controlloop ????Other
algorithmsmay attempttoextendtheir analysis tothe mostdistantavailabledata infront
ofthevehicle? forinputtolonger termreasoning mo dules?
Thispap erpresents a newalgorithm abletoreconstructtheroadshape froma single
image? providing thethreedimensional pro?leofthe roadinfrontofthe vehicle?often
uptothepointwhere theroadbecomeshidden?Reconstructingtheroadoveralarge
distance presentsseveraladvantages?Thereconstructionsfromseveral videoframes can
beov erlapped?andtheevidencefrom each reconstructioncan becombined for added
reliability?It isalsoclearly usefulforaroadfollowing systemtomakeestimationsof turns
wellinadvance?andadjust itssp eedaccordingly?Thislong rangeobservationoftheroad
doesnot precludethe useofashortrangeroadanalysisinthecontrolloopof the vehicle
steering?
Roadreconstructionfromasingleimageisa ?shape?from?contour?problem? Itis ob?
viouslyunder?constrained?yieldinganin?nit yofpossibleroadshapes unlessconstrain ts
abouttheroad structureinthe?Dscenearein troduced? Thus aroad modelhastobe
assumed?whic hprovidesareasonable set ofadditional constraints?
The simplestmo delwhichhasbeenapplied ????is theFlat?Earthge ometry model?the
roadis assumedplanarand inthesameplanewhichsupportsthevehicle?andtheroad
imageisback?projectedontothisplane?Themethodisfastanddoesnotdeterioratewhen
imageanalysisgivesraggedroadimageedges?Butitisvery sensitiveto thedi?erence
betweentheassumedand actualcameratilt anglewithrespectto the ground?Figure ???
F oracamera mounted onav ehicleat ???metersab ovethe ground?aw orldp ointat??
metersin frontof thev ehiclewillbe reconstructedat??meters ifthegroundplaneangle
isoverestimatedby?degreesandat??metersifitisunderestimatedby?degrees? an
errorrangeofmorethan?????Consequen tly?theFlat?Earthalgorithmismoresuitable
forreconstructingtheroadjustinfrontofthevehiclethanforalongrangeanalysis?
Moresophisticatedalgorithmshaveattemptedtoutilizetheconstraintthataroadgener?
allykeepsanapproximatelyconstantwidth ?????Theproblemwithapplyingthisconstraint
isthatonemust?ndthepairsofpointsseparatedbyadistanceequaltotheroadwidth?
?

Page 3

The constantroad widthconstraintisnotsu?cient?Another constraintmustbeadded
for thereconstructiontobepossible? Weha vec hosenthe zer o?bankconstraint? specifying
that theroaddo es nottilt sideways? Aroad modelcombiningconstantwidth andzero
bank wasoriginally suggestedin?????
In previouswork?wedevelopedanincrementalroadreconstructionmethod basedon
theseconstraints ?????inwhichanewpairofedgep oints couldbe found ifwehadalready
foundaneighborpairofedgepoints?theroadedgeswerereconstructedincremen tallyfrom
edge poin tsclosetothe vehicle toedge p ointsin thedistance?Thismethod isfragile
because anyincrementof constructiondep endsonthe previouselementsinthechain?Any
failureof theroad reconstructionatanypoin tcan befataltothefurtherprogressof the
reconstruction?
Thisincrementalmetho dusedadiscreteapproach?Roadreconstructions basedona
di?eren tial approachcanbefoundin??? ??? Aninteresting alternativetothe globaldynamic
programmingoptimizationproposed inthepresen tpapercan befound in????
? Summary
Theprop osedalgorithm canbedecomp osedintothefollowingsteps?
??Inapreliminarystep? notdetailedhere?appropriateimagepro cessing techniqueshave
isolated thetwo curvesof the edgesinthe image?andapolygonalapproximationhas
beenfound foreach edgecurv e?
??Pickingimagepointsanywhereononeimageedgecurve?weareableto?ndthepoints
whicharecandidatesforbeingmatchingpointsontheotherimageedgecurve??Two
imagepoin tsarecalled matchingpoints if theyare imagesof theendpoints ofcross?
segments ofthe ?Droad??This matc hing is madepossible by makingreasonable
hyp otheseson theshape of theroad? whichaddenoughconstraintsto makethe
problem solvable?Sp eci?cally?theroad ismodelledasaspaceribbonde?nedbya
centerline spineandhorizontalcross?segmentsofconstantlength cuttingthe spineat
theirmidpointsatanormaltothespine?Wefurtherassume thattangen tstothe
ribb onedgesatendpointsofcross?segmentsareapproximatelyparallel?Section???
We?ndanexpressionthatmustbe satis?edbythetwoimagepointslocatedonthe
facingimageedgecurvesandthetangentstotheedgeimagesinorderforthetwo
p ointstobematchingpoints?Section???Ifa
?
anda
?
arematchingpointsand?a
?
?
?

Page 4

where
?
Vis thevertical direction?F oredge curv es approximatedbypolygonal lines?
thematc hingpointa
?
canbeonaline segment?anditsposition betweentheend
points ofthelinesegmentcan beexpressedbyanumberbetween?and?? whereas
itstangentvector?a
?
?
isconstant?orthe matchingpointa
?
canbe atanendpoint
ofalinesegmen t?withaconstantp ositionbut witha tangentanglewhichcan be
expressedbyan umber between?and? withintherangeofanglesofthetwoadjacent
linesegmen ts ?Section ???F oreac hpointchosen fromone imageedge?wecheckfor
eachofthe linesegmentsofthe otherimage edgeif a matchingpoin tb elongs tothat
line segment?i?e?? ifourexpressiongives alinearcoordinatebet w een?and?forthis
line segment?Then welook for matchingpointsatthe nodesof thepolygonalline
byc heckingiftheexpressiongivesanumberbetween? and? forthe tangen tangle?
?? Foreachpointchosenfromoneedgeimage?thepreviousstepmaygiveseveralmatch?
ingpointsontheotheredgeimage?Oneofthereasonsisthattheimagesoftheedges
canbevery roughandwiggly? Anotherreason isthatthecondition usedisonly a
necessary conditionfor twop oints tobe matchingpointsintheimageof theroad?
This conditionislo cal andwemuststill c ho osethematc hingpoin tspairswhichare
themostgloballyconsisten t? and discardtheotherpairs?Thecriteriaofoptimization
arethree?dimensionalcriteria?Section???thusatthisstepofthealgorithm?fromthe
pairsofmatchingpoints?thecorrespondingthree?dimensionalcross?segmentsmust
befound?This correspondence isunique ifthe cross?segmentsareassumedhorizontal
andofknownconstantlength?Section???Theconstant lengthisthewidthofthe
road?andcannotbede?nedbythismethod?Theassumedroadwidthisascaling
factorinthereconstruction?whereastheoptimizationisbasedonangularconsider?
ations?whichareindependentofscaling?Fordrivingavehicletheroadwidthmust
even tuallybeobtainedfromothermethods?suchasstoreddataabouttheroad?the
?Flat?Earth?method?orclose?rangemethodssuchasstereoscopyortime?of??ight
ranging?
??Thegroupofmatchingpointpairscorresponding toasinglepointchosenononeedge
istheimageofagroupofworldcross?segmentsobtainedattheprevious step?and
theworldroadcango throughatmostoneofthesecross?segments?Section???Ifa
sequenceofpointsalongoneroadedgeistaken?asequenceofgroupsofcross?segments
isobtained?andtheworldroadmustgothroughatmostoneofthecross?segments
ofeachgroup?inthesameorderasthesequenceofpointschosenonthe?rstroad
?

Page 5

whichcharacterizesa?goodroad?? Thetotalev aluationfunctionis thesum of the
functionsofeachof thearcs ofthe graph?The evaluationfunctionfor anarc isthe
sumofweigh tedcriteria?which gradethec hoices ofindividualcross?segmen tsand
theneighb orhoodofconsecutivecross?segments? basedonangular considerations?
?Thematchingpoin t problem
Considerthe imageofarailroadtrackand itsrailroadties?and assumethatsomeappro?
priate image processing techniqueshavereducedthe images ofthe rails tocurvesand the
images of theties to linesegmentsb etweenthesecurves?Figure ???Thepositions ofthe
end points ofthe tiesegmentsonthe curvesofthe railarethe matchingpointsinthe
image?Thereconstruction oftheshape oftherailroadtrackin?Dspaceuses thematching
pointsandisstraigh tforwardifthreeh yp othesesare made?
??The widthwoftherailroadtrac kis constantand known?
??Thecoordinates of theverticalunitvector
?
V arekno wn in thecamera coordinate
system?
??Therailroad tiesare approximately horizon tal?
Obviously? the lasth ypothesisdoes notconstrain therailroaditself tobe horizon tal?Simi?
larly?thestairs inaspiralstaircasehavehorizontalstepedges?buttheruledsurfacede?ned
bythesestepedgesisfarfromhorizontal?
Considertwomatchingpointsa
?
anda
?
?theendp oints ofthe image ofatie?The
correspondingvectors from theviewp ointO to theseimagep ointswillbe denotedby?a
?
and?a
?
? The correspondingw orldp oints A
?
andA
?
arede?nedby
?
A
?
??
?
?a
?
?
?
A
?
??
?
?a
?
sinceworldpoin ts and theirimages are on thesameline ofsigh t?
Thew orldlinesegment isassumedhorizontal? thetwoparameters?
?
and?
?
arethen
relatedby
?
?
?m?
?
with
m?
?a
?
?
?
V
?a
?
?
?
V
?