Page 1

DanielDeMenthon and LarryS? Davis

Computer VisionLab oratory

Center forAutomation Research

Univ

V?

ersit

w

yof

the

Maryland

metho

CollegePark?MD ?????

Abstract

A new method ispresentedforreconstructinga?Droadfroma singleimage?It

?ndstheimages ofoppositepoints ofthe road?opp ositepointsarep ointswhichface

eachotheron theopp osite sides of theroad?the imagesofthesepoin tsarecalled

matc hingp oints??Forpointsc hosenfrom onesideof theroad image?theprop osed

algorithm?nds allthe matchingpoint candidatesonthe otherside?basedon local

propertiesofa road?How ever thesesolutions do notnecessarilysatisfytheglobal

prop erties ofat ypicalroad?Adynamicprogramming algorithm isapplied toreject

the candidates whichdonot?tthe globalroad?

Abenchmarkusingsyn theticroadsis described?whichshowsthat theroadsre?

constructedby the proposedmethodmatch theactual roadsb etter thantwoother

road reconstructionalgorithms?Experimentswith??roadimages takenbythe Au?

tonomousLandVehicle?ALshothatdisrobustwithrealworlddata?

and thatthe reconstructionsare fairlyconsistentwith roadpro?lesobtained by fusion

between range imagesandvideo images?

?

Page 2

roads ?????????? ???Forrobustnessinav ariety ofconditions?these systemscanbe

drivenbya supervisorsystemreasoning about information pro videdby severalalgorithms?

such asstereoalgorithms? stereomotionalgorithms? algorithms usingsinglevideoframes

or combiningvideo framesand rangeimages? Kahlmanpredictorscom bininginformation

obtained fromseveralvehiclepositions?etc????Some algorithmsmaymonitor the roadov er

a shortdistanceoralonga single edge?for inputtoafast steeringcon trollo op????Other

algorithmsmay attempttoextendtheiranalysisto themostdistantavailabledatainfront

ofthev ehicle?for inputto longerterm reasoningmo dules?

This paperpresen tsanewalgorithm able to reconstructthe roadshape fromasingle

image?providingthethreedimensional pro?leof theroadinfront of thev ehicle?often

uptothepoint wherethe roadbecomeshidden? Reconstructingtheroadov eralarge

distancepresentsseveral adv antages?The reconstructionsfromsev eralvideoframescan

beov erlapped?andtheevidencefromeach reconstructioncanbecombined foradded

reliability?It isalsoclearlyusefulforaroadfollowing systemto makeestimationsofturns

w ellinadvance?and adjustits speed accordingly?Thislongrangeobservationof theroad

do esnotprecludethe useofashortrangeroadanalysisin the controlloop ofthevehicle

steering?

Roadreconstruction froma singleimageisa?shape?from?contour? problem?Itisob?

viouslyunder?constrained? yieldinganin?nityofp ossibleroadshapesunless constraints

ab outtheroadstructureinthe ?Dsceneare introduced? Thusaroadmodelhasto be

assumed?whichprovidesareasonable set of additionalconstraints?

Thesimplest model whichhasbeenapplied ????istheFlat?Earth geometrymodel?the

roadisassumedplanarand inthesameplanewhichsupportsthevehicle?andtheroad

imageisback?projectedontothisplane?Themethodisfastanddoesnotdeterioratewhen

imageanalysisgiv esraggedroad imageedges?Butit isverysensitivetothedi?erence

betweentheassumedand actualcamera tiltangle withrespecttotheground ?Figure ???

For acamera mountedonavehicleat ???metersabove theground?aw orldp oint at??

metersin front ofthev ehiclewillbe reconstructed at??metersifthe groundplane angle

isoverestimatedby?degrees andat?? metersifit isunderestimatedby?degrees? an

errorrange of more than?????Consequently? theFlat?Earth algorithm ismore suitable

forreconstructing theroadjustinfrontofthev ehiclethanfora long rangeanalysis?

More sophisticated algorithms haveattempted toutilizetheconstraintthataroad gener?

allykeepsanapproximately constantwidth ????? Theproblemwithapplyingthis constraint

isthatonemust?ndthepairsofpointsseparatedbya distanceequaltotheroadwidth?

?

Page 3

Theconstantroad widthconstraintisnot su?cient?Another constraintmustbe added

forthereconstruction tobep ossible?Wehavec hosen the zero?b ank constraint? specifying

that theroaddo es nottiltsideways?Aroadmodel combining constantwidth andzero

bankwasoriginallysuggested in?????

Inpreviouswork?wedevelop edanincremen talroadreconstructionmethod based on

these constraints?????inwhicha newpairofedgepoin ts couldbefound ifwehad already

foundaneighb orpairof edgep oints?theroadedgesw erereconstructed incrementallyfrom

edgep oints closetothev ehicle toedgep ointsinthedistance?This methodis fragile

because anyincrementof constructiondep endsonthepreviouselemen tsinthec hain? Any

failureoftheroadreconstructionatanypoint canbe fataltothe furtherprogress of the

reconstruction?

This incrementalmethodusedadiscreteapproach?Roadreconstructionsbased ona

di?eren tialapproach canbefound in?????? An interesting alternativetotheglobaldynamic

programming optimizationproposedin thepresentpaper canbe foundin????

? Summary

Theproposedalgorithm canbedecomposedintothe followingsteps?

?? Ina preliminary step?notdetailed here? appropriateimagepro cessingtechniqueshave

isolated thetwo curvesof theedgesin theimage?andap olygonalapproximationhas

beenfoundforeachedgecurv e?

??Pickingimagepointsan ywhereononeimageedgecurve?weareableto ?ndthep oints

whicharecandidatesforbeingmatchingpointsontheotherimageedgecurve??Two

imagepointsarecalledmatchingpointsiftheyareimagesoftheendp oints ofcross?

segmentsofthe ?Droad??Thismatchingismadep ossiblebymakingreasonable

hypotheseson theshapeof theroad?whichaddenoughconstraints tomake the

problem solvable? Speci?cally? the roadis modelledasaspace ribb onde?nedbya

centerlinespine and horizon talcross?segmen ts ofconstant lengthcuttingthespine at

their midpoin ts atanormal to thespine?Wefurtherassume thattangen tstothe

ribb on edgesat endp oints of cross?segments are appro ximatelyparallel ?Section???

We?ndanexpression thatmustbe satis?edbythetwo imagepoin tslocated onthe

facingimageedge curvesandthetangentstotheedgeimagesin orderfor thetwo

pointstobematchingpoints?Section??? Ifa

?

anda

?

arematc hingp oin tsand?a

?

?

?

Page 4

where

?

Visthev erticaldirection?F or edgecurv esapproximatedbypolygonallines?

thematc hingpointa

?

can beonalinesegment?and itsp ositionbetweenthe end

p oints of thelinesegment canbeexpressedbyan umb erbet ween?and ?? whereas

itstangentvector?a

?

?

isconstant? orthematc hingpointa

?

canbeat anendp oint

ofaline segmen t?with aconstantp ositionbut withatangentanglewhich canbe

expressedbyan umberbetween? and? withintherangeof anglesofthetwoadjacent

linesegments?Section???Foreachpointchosenfromoneimage edge?wec heckfor

eac hofthelinesegmentsoftheother image edgeifamatchingp ointb elongs tothat

linesegment? i?e??ifour expressiongiv esa linearcoordinatebet ween ? and?for this

line segmen t?Then welookformatc hing p oin tsatthe no desof thep olygonalline

byc hecking iftheexpression givesan umberb etween?and?forthetangentangle?

??F or eachpointchosen from oneedgeimage?thepreviousstep maygivesev eralmatc h?

ingpoints onthe otheredgeimage?Oneofthereasonsisthattheimagesoftheedges

canbevery roughandwiggly?Anotherreasonisthatthe conditionusedis onlya

necessaryconditionfortwopoin ts tobematc hingpointsintheimageof theroad?

Thisconditionis local andwemust stillchoose the matc hingpointspairswhichare

the most globally consistent?anddiscardthe otherpairs?Thecriteriaof optimization

arethree?dimensionalcriteria?Section???thusatthisstepofthe algorithm?from the

pairsofmatc hingpoin ts?thecorresp ondingthree?dimensional cross?segmen tsm ust

befound?Thiscorrespondence isuniqueifthecross?segmen ts are assumedhorizontal

and of kno wnconstant length ?Section???Theconstant lengthisthewidthofthe

road?andcannotbede?nedbythismetho d?Theassumed roadwidthisascaling

factorinthereconstruction?whereasthe optimizationis basedon angularconsider?

ations?which areindependentofscaling?Fordrivingav ehicletheroadwidthmust

eventuallybe obtained fromother methods? suchasstoreddataabouttheroad? the

?Flat?Earth?method?orclose?range methodssuchasstereoscopyortime?of??ight

ranging?

?? Thegroupof matc hingpointpairscorresp ondingto asinglep ointc hosenononeedge

is theimageofagroup of world cross?segments obtainedat thepreviousstep?and

thew orldroad cangothroughatmostone of thesecross?segmen ts?Section??? Ifa

sequence ofpoin ts alongoneroadedge istaken?asequence of groupsofcross?segmen ts

isobtained? andthe world road must gothroughatmostoneofthe cross?segments

ofeac hgroup? inthesame order asthesequenceofpointschosenon the?rst road

?

Page 5

whichcharacterizesa?good road??The totalev aluationfunctionisthesum ofthe

functionsof each ofthe arcsofthegraph?Theev aluationfunctionforanarc isthe

sumofweighted criteria?whichgradethec hoicesofindividualcross?segmentsand

theneighb orhoodofconsecutivecross?segments? based onangularconsiderations?

?Thematc hingpointproblem

Considerthe imageofa railroad track and itsrailroadties?and assume thatsome appro?

priateimage processingtechniqueshave reducedtheimages oftherailstocurvesand the

images ofthetiestolinesegmentsb etween thesecurv es?Figure??? Thepositions of the

endpoin ts ofthe tiesegmen ts onthecurv esof therail arethematchingp oin ts in the

image?The reconstructionoftheshapeoftherailroad track in ?D spaceuses the matc hing

p oin tsandisstraigh tforwardifthreeh ypothesesare made?

?? Thewidthwoftherailroadtrackis constantandkno wn?

?? Thecoordinates ofthe vertical unitvector

?

Vare knownin thecameraco ordinate

system?

??Therailroadties areapproximatelyhorizontal?

Obviously?the lasth ypothesisdoes notconstraintherailroad itselftobehorizontal?Simi?

larly? thestairs inaspiralstaircasehave horizon talstepedges?but theruled surfacede?ned

by thesestep edges is far fromhorizontal?

Considertwomatchingpointsa

?

anda

?

?theendpointsoftheimageofatie?The

correspondingvectorsfromtheviewpoin tOtotheseimagepoin tswillbe denotedby?a

?

and?a

?

? ThecorrespondingworldpointsA

?

andA

?

arede?nedby

?

A

?

??

?

?a

?

?

?

A

?

??

?

?a

?

sinceworldp ointsand theirimages areonthesamelineofsight?

The worldlinesegmentisassumedhorizontal?thetwo parameters?

?

and ?

?

arethen

relatedby

?

?

? m?

?

with

m?

?a

?

?

?

V

?a

?

?

?

V

?

Page 6

?

?

?

w

??a

?

?

?m

?

?a

?

?

??m?a

?

??a

?

?

?

?

???

Th usthetwo curvesoftherailsinthescene canbeingeneraluniquelyreconstructed

fromtheirimagesuptoascale factor? ifthe tiesare assumed horizontaland ofconstant

length?Problemsoccur onlyif the railroadimage crossesthehorizon? as noted in????In

thiscasetheties arehorizontal onthehorizonline and theirrangecannotbedetermined?

as canbeseenfrom theequationsabov e?

Consider nowthe problemofreconstructingaro ad fromitsimage? oncesomeappropriate

imagepro cessing techniqueshave isolatedthecurv escorresponding to theroadedgesinthe

image? Thistime ofcoursewe do not have theimages ofrailroad tiesegmen ts tohelp us?

The methodwe proposeisth usto?ndasa?rststeptheendp ointsof linesegmen tswhich

corresp ond toimages of railroadtiesegments?andthendothe?Dreconstructionofthe

endpoin ts of theimagesof thesesegmentsbythe methodjustdescrib edfortherailroad?

We callthesew orldsegments corresponding to railroadties?cross?segmen ts??andtheir

endp oints ?oppositep oints?? Theimages ofthesepoin tsarethe?matchingp oints??The

mainproblemofthereconstructionofaroadfromitsimagecanthenbestatedas?Given

apointononeedgeoftheroad image?whereisthematchingpointontheotheredge?

Wechoosearoadmodel similartotherailroad model? theroad ismodelled asa space

ribb ongeneratedbyacenterline spineandhorizon tal cross?segments ofconstan tlength

cutting thespineat theirmidp ointsat anormal tothespine? This modellinggiv escross?

segmen ts the properties ofrailroad ties?

?Cross?segmen ts arehorizontal? i?e?p erpendicular tothev ertical ?on the ALV thevertical

w asdetectedbytrimsensors??

? Cross?segments haveconstant length ?the roadwidth??

?Cross?segmen tsareperpendicular tobothroadedges?i?e? locallyp erp endicular tothe

centerline ofthe road?

Saying thatcross?segmen tsarenormaltob othroadedges meansthatthey arenormal

to thetangen tstotheedges attheirendp oints?Note thatthisdoesnotgenerallymean

thatthetangents tooppositep oin tsare parallel?In????how ever?we showthatassuming

thetangents toopp ositep ointsto beapproximatelyparallelisareasonableassumption

inmostcon?gurations?Thisassumptionconsiderablysimpli?estherecoveryofopposite

p oints fromthe image?Itis addedtoour roadmodeland usedinthefollo wingsection?

?

Page 7

Consideraworldroad de?nedbytwo ?Dcurv esE

?

andE

?

?andtheroad imagede?ned

bytwo imagecurv ese

?

ande

?

? Assumethattwoopp ositepoin tsA

?

andA

?

on roadedges

E

?

andE

?

havebeen found? Theirimages area

?

anda

?

?Figure???andthefollo wing

propertiesfollow fromtheworldroad model?

??ThesegmentA

?

A

?

ishorizontal?

??The tangentstotheroadedgesatA

?

andA

?

are p erp endiculartoA

?

A

?

?

??ThetangentstoA

?

andA

?

areapproximately parallel?

??The tangent?a

?

?

tothe image edgee

?

ata

?

isthe imageofthetangen t

?

A

?

?

tothe

w orldedgeE

?

atA

?

? thetangent?a

?

?

tothe image edgee

?

ata

?

istheimage ofthe

tangent

?

A

?

?

toE

?

inA

?

?This isageneral property of projectedcurv es andtangents?

Inderivingthe followingconsequences?we makeuse ofthepropertythat thedirection of

the intersection oftwoplanesisp erpendicular tothenormalsofeachplane?andcanbe

obtainedbythecross?product ofthetwonormals?

??? Directions

?

of tangentstooppositep oints

If a

?

anda

?

arematc hingpoints and?a

?

?

and?a

?

?

arethetangentsto theimage edges at

thesep oints? the directionofthecorrespondingworldtangents is

??a

?

??a

?

?

????a

?

??a

?

?

?

Pr o of? Ifa

?

anda

?

areimages ofopp ositepoin ts?theworldtangents to theworld edgesare

parallel? Since theimagesofthew orldtangen tsare?a

?

?

and ?a

?

?

?thew orldtangentslieon

theplanes?

?

Oa

?

??a

?

?

? and?

?

Oa

?

??a

?

?

?respectiv ely?These planesarenotparallel sincethey

sharethep ointOandthey donotcoincide? Since thetangen tsare parallel?theym ustbe

parallel to the intersection ofthese planes?The directionofthisintersection is givenby

the previous expression?

???Direction ofacross?segment

Ifa

?

anda arematchingpoin tsand

?

Vthev erticalvector? thedirection of theworld

cross?segmen tis

?

V???a

?

??a

?

??

Pr oof? A

?

A

?

belongstoahorizon tal planesinceit ishorizontal?Sincea

?

a

?

istheimage

ofA

?

A

?

?A

?

A

?

alsobelongs to theplane?Oa

?

?Oa

?

??This planeisgenerally nothorizontal?

?

Page 8

Th us the directionofA

?

A

?

is givenby

?

V???a

?

??a

?

??

??? Matc hingcondition

Ifa

?

anda

?

are matchingp oin ts and?a

?

?

and?a

?

?

arethetangent directionstothe image

edges inthesep oin ts?the following relation holds?

?

?

V???a

?

??a

?

??? ???a

?

??a

?

?

????a

?

??a

?

?

???????

Pro of? Ifa

?

anda

?

are imagesof oppositep oints? thedirectionofthe cross?segmentA

?

A

?

isp erpendicular to thedirection oftheparalleltangen ts?

??? Localnormal totheroad

Ifa

?

anda

?

arematc hingp oin tsand?a

?

?

and?a

?

?

arethetangents totheimageedges at

thesep oints? the localnormal tothew orldroadhas thedirectiongivenby

?

N??

?

V???a

?

??a

?

??? ???a

?

??a

?

?

????a

?

??a

?

?

?? ???

Pro of?The local planarpatch of theworldroad isde?nedbyA

?

A

?

andbytheparallel

tangents atA

?

andA

?

?The directionof thenormal tothis plane is thecross?product of

thedirections of thecross?segment andofthetangen ts?

Tosummarize?whenapointa

?

and thetangent?a

?

?

totheroadimage aregiven? Equa?

tion?becomesan equationwhichmustbe satis?edbythe coordinates ofa

?

andtheslope

of thetangentto theedge ina

?

inorder fora

?

tobeamatc hingp oint toa

?

?We canalso

?nd thedirection of the normalalong thecorrespondingworldcross?segmentA

?

A

?

?

? Search foramatchingp ointofagivenimagepoint

Ifapointa

?

isc hosenononeedge image?andiftheotheredge imageisap olygonalline?

thematchingp ointa

?

canbelocatedononeofthelinesegmentsof thepolygonalline?

or atoneofthe v erticesbet weenthesegmen ts? Allthe linesegmen tsandall thevertices

arec hecked? b ecauseasinglepoin ta

?

can haveseveralmatchingpointcandidates due?for

example?toedgeirregularities?Otherreasonsareconsidered in????For eachlinesegment

andforeac hv ertex?the equations develop edinthenexttwosubsections areapplied?

?

Page 9

on thissegment if

?a

?

??p

?

???p

?

q

?

with?b etween?and?? Thepointa

?

mustalso?with itstangent tothe edge?satisfy

Equation??Thetangent?a

?

?

totheedgeimage ina

?

isapproximatedbythevector?p

?

q

?

?

We replace?a

?

?

??a

?

bytheirv alues?p

?

q

?

? and?p

?

???p

?

q

?

inEquation?? andtransformcross?

productcom binationsinto dotproductsbythew ell?kno wn identity

?a??

?

b??c????a??c?

?

b???a?

?

b??c

Theresultingvalue for? is

?

??

??

or

?

?

V??a

?

??

?

K??p

?

???

?

K??a

?

??

?

V??p

?

?

?

?

V??a

?

??

?

K??p

?

q

?

???

?

K??a

?

??

?

V??p

?

q

?

??

???

where

?

K???a

?

??a

?

?

????p

?

??q

?

?? If? isb etween?and??thein tersectionisbet ween the

endpoin tsoflinesegmentp

?

q

?

?and thev alueof?speci?es theposition ofa

?

onp

?

q

?

?The

search alsotak es placeamongthev erticesbetweenthe line segments?

???Searchfora matchingpoint atav ertex

We canthinkofapointq

?

linkingtwolinesegmentsp

?

q

?

andq

?

r

?

asap ointat whichthe

slopeof thetangent tothe edgechangesfromtheslopeofthesegmentp

?

q

?

totheslopeof

thesegmentq

?

r

?

?Anapproachsimilartotheprevioussubsection isfollow ed?Amatc hing

pointa

?

isatthevertexq

?

if

?a

?

?

??p

?

q

?

????q

?

r

?

??p

?

q

?

?

with?b etween?andFthisp oint tobeamatching point toa

?

?it mustalso satisfy

Equation?? Thisproduces thefollowing valuefor ?

???

?

?

M??q

?

???n

?

??p

?

r

?

????n

?

??q

?

??

?

M??p

?

r

?

?

?

?

M??q

?

???n

?

???q

?

r

?

??p

?

q

?

?????n

?

??q

?

??

?

M??q

?

r

?

??p

?

q

?

??

???

where?n

?

??a

?

??a

?

?

?

?

M?

?

V???a

?

??q

?

?

If theresultingvalue of?isbetween?and?? amatching p ointa

?

tothepointa

?

is

locatedat the v ertexq

?

?

?

Page 10

search formatc hingp ointsa

?

isdoneforboththeline segmentsoftheotheredgeimageand

the verticesbetw eenthesegments? producingsev eralmatching candidatesa

?

?Wec hoose to

representbothedgeimageswithalistoflinkedlinesegments?Thep ointsa

?

are tak enat the

midpoints ofthe linesegments? andthetangents?a

?

?

areinthe direction ofthelinesegments

themselv es?For eachpointa

?

?generally oneto threepoints a

?

arefound?sometimes more

forv erynoisyimages? butgenerallyonep ointa

?

isa more?correct?matchingp oint forthe

pointa

?

in terms ofcompatibilitywiththe globalroad reconstruction?Itisalsop ossible

thatbecause ofnoise intheimage? the correctmatch is notamongtheresults?in which

i

case allpointsa

?

are badmatc hingpoin ts?Th usa methodforchoosingmatc hingpoin

?

ts

compatible

?

witharealisticworldroad isrequired?andis now discussed?

Whenapointisc hosen onone road imageedge? the exhaustive searchmatc hes this

pointwithsev eralpoints ontheotherroadimageedge?Thisgr oupofmatchingp oin ts

pairsis theimageofa groupofw orld cross?segments? butthew orldroad canpass through

at most one of these cross?segments?Ifasequenceofp oin ts alongoneroadedge is taken?

asequence of groupsofcross?segmen tsisobtained? andtheworld roadmust gothrough at

mostone of thecross?segmen tsof each group?in thesame order asthe sequence ofpoints

c hosenon the?rstroad imageedge? Eachcross?segmentcanberepresentedbya nodeofa

graph?Figure ???The graphismadeupofgroupsofno desandapathm ustbe foundwhich

visitseach groupintheprop ersequenceandgo esthroughatmostonenodeofeach group?

Thispathm ust alsomaximize anevaluationfunctionwhichcharacterizes the?goo dness?

of the road? Thetotalevaluationfunctionisthe sum oftheev aluation functionsofeach of

the arcsofthe graph?The ev aluationfunctionfor an arc isthesum ofw eighted criteriaC

whicharec hosentoc haracterizeagoodpairing oftwoneighbor cross?segments A

?

Aand

BB

?

?The followingcriteriawerechosen

?The localnormal

?

Nforthecross?segment ?Equation?? shouldbeclosetovertical?

C

?

?

?

N?

?

Vshouldbe closeto ??

?The slopeofthepatc hof twosuccessiv ecross?segmen ts shouldbe closetovertical?

C

?

????

?

A

?

B

?

?

?

A

?

B

?

???

?

Vshouldbeclose to???istheconstantwhichnormalizes

thevectorwhichfollows??

?The av erageofthe directionsofthetwocross?segmen tsshould bep erpendicularto

thelinejoiningtheirmidpoints ?trapezoid constraint?see ?????

C

?

?????

?

A

?

A

?

?

?

B

?

B

?

????

?

A

?

B

?

?

?

A

?

B

?

?shouldbe closeto??

??

Page 11

Itw ouldalsobedesirable toin troduce constraints suchasarequirementfor small

di?erencesof slopebetween successive patches?butthist ype ofrelation inv olv esthree

successive cross?segments andcomplicates the interactiongraph? The prop osedunary and

binarycriteriaseem tobesu?cien t fordiscarding unw anted no des?

Dynamic programmmingis anappropriatetechniqueforthist ype ofpath optimization

???? It isbased onthe following ideas?

?When extendingpathsfromone groupofmutuallyexclusivenodes to thenextgroup?

keepthe maximumnumber of usefulpaths up tothe end?usefulnessb eingde?ned

inthenext paragraph?? sincethe b est pathalongsev eralarcs couldgothrough the

w orst arcslaterand loseitschance?and inourcase couldeven reachadeadend? If

wehave thec hoiceof several pathswhen the lastgroup isvisited?choose thepath

whichhasthelargestcumulated evaluationfunction?

? Ifsev eralpathshavebeenextendeduptoseveral nodesofagroupkand thetotal

ev aluationfunctionisknownfor eac h?whenextending thepathsfrom thegroupk

tothe groupk?? itis notuseful toextendtwopathstomeet atoneno de ofgroup

k??? Among thetwo paths?thepath toextend isthe pathwhich gives the largest

totalev aluationfunction whenthe new arcis added? Ifpathshavebeenbuilt up

togroupk? thealgorithm atstepk?? involv esexaminingthe nodesofthis group

onebyone? and foreach no deextendingonlythepathwhichgiv es thehighest total

evaluationfunctionwhenthe newarc isadded?

Thefollowing rulesw ere addedtothe basic dynamicprogrammingalgorithm?

?A no deof groupk??may have noacceptable arcextending thepathswhich reached

groupk? Anarc is labelledunacceptablewhen any ofthe criteriawhich make up the

arc evaluationfunctionisundera giv enthreshold? Thisunconnected no deismarked

as unusable tothenodesof thenextgroup?

? Ifingroupk?? nono decanbelink edtoanyofthe nodesofgroupk? groupk??

isdiscarded altogether andthe groupofnodesk?? isconsideredinstead?and so

on? un tila groupis foundtoextendthe pathswhichreac hedgroup k?Ifnofurther

group succeedsinextending thepaths? thepathsterminateatgroupk? andamong

themthe pathwiththe largestev aluation functionischosen?

??

Page 12

width? and onalargen umb erofroad imagesfrom theDARPAAutonomousLandV ehicle?

Theresults forthesyntheticroad arepresented?and theresults fromthe realw orld data

arediscussed?

???Syntheticroadimage

The road pro?lefrom which syntheticimageswerecreated isshown in Figure ??Theroad

centerline pro?le?side view?isanelement ofasinusoidfromacresttoa trough?slope

do wnward? orfroma trough toacrest?slopeupw ard?? and theroad slopecanbe modi?ed

byvaryingthesin usoidamplitude?Whatwe callroad slopeinthe follo wingis theslope

at themidpoint of the straightpiece b etweenthetwoturns?In thissyn theticroadit is

foundequal toH??????whereH isthe di?erenceoflevel inmetersbetweenthelow estand

highestpoint of theroad?Intopviewand goingawayfrom the camera?theroad hasa

shortstraightstretc h?then takesa?? degreeleftturnanda similar rightturn? separated

byashort straight line?Thecamerap osition?orien tationandparameters?alsolistedin

that?gure?w ere takenequaltothevalueswhichdescrib e the camera oftheAL V?

Abenchmark was developed formeasuringtheperformanceof theproposed matching

p ointsalgorithmand otheralgorithms? A reconstructedroadislab elled?na vigable? ifthe

trac ksofatwo meter?widev ehicle follo wingthe centerlineof thereconstructed road stay

b etw eentheedgesof theactual roadov erthe wholereconstruction anddonot cut these

edges?Figure ???No cross?segment canbe shiftedsidewa yswithresp ectto theactualroad

bymorethanaquarter ofits length? Notice? how ever?thatnon?na vigablereconstructions

from analgorithm are stillusable if they are

??

not too far from theactualroad? Indeeda

fastcomputer couldpro ducereconstructions ina ?xed coordinatesystem atframe rate?

andaccum ulate theevidence obtainedfrom each reconstructiontoproduceacomp osite

reconstructed roadofhigher reliability? Withthis idea inmind?areconstructed roadis

lab elled?usable? if thecenterlineof thereconstructed roadstaysbetw een the edgesof

theactualroadanddo esnotcut theseedges?Inotherwords?ausable reconstruction

isareconstructionwhichis navigablebyamotorbik e?ora reconstruction inwhich no

cross?segmentis o? theactualroadbymorethanhalfits length?Whenalargen umb er

of roadswithrandomv ariationsareconsidered?percentages of na vigableandusableroads

arecalculated?

Toobtainthebenchmarkvalues described abov e?randomv ariations areintro duced

aroundthe nominalv aluesof theroad width??m? andtheroadbank?? degrees?? The

random widthandbankvariations aregiv enGaussiandistributions ofprede?nedstandard

Page 13

werestudiedforeac h ofthe roadslopes?

??m?? degrees???????m???degree??????? m???degrees??????? m??? degrees??

?????m???degrees??

F ortyroadsw erepro duced for eachofthese ??com binationsof slopesand standard

deviations?andtheresultsfor these forty roadsw ereaveraged toyieldthepoin ts shown in

the followinggraphs?

Figure?sho wstwoexamples ofreconstructions forimagesof downw ardslopeswith

largev ariationsof widthand bank?Noticethat themaximumconsidered widthandbank

v ariations?????m???degrees?bottomleft?correspondtov erylargedistortionsinthe

images? Thenavigablelength isalmost????forthetopexample?but lessthan??? for

theb ottomexample?partlyb ecausethe actualroadbecomesvery narro w?howeverthe

usablelengthisalmost???? forbothexamples?b ecausethe cen terline stays withinthe

actualroad?

In Figure??threealgorithmsarecompared?the present matchingpointsalgorithm? the

step?by?step?incremental?zero?bank algorithm ???? ??andtheFlat?Earth algorithm?A

globalmeasure ofp erformance isobtainedbyaveraging theresultsobtained forthe? slope

con?gurations? asiftheresultsw ereaveraged fromtestsov eraterraincomprisingequal

proportionsofS?turns atslopes ????? ???? ?????? and ????The Flat?Earth algorithm

giv es reconstructed roadswhich are????usable if theslope iszero? and unusable all

theway down iftheslope isnot zero?independent of width andbankv ariations?th us

theav erageproportion ofusableroad pro ducedby the Flat?Earthalgorithm is ???? The

av eragingov ervarious slopes ismorenatural for theothertwo algorithms?b ecausethe

resultsarefound tobe almost independent ofthe slop e?

Figure? sho wsthat the matchingp ointalgorithm givesb etter resultsthan theother

two algorithmsintermof usablereconstruction?More detailscanbefound in ????

???Real Imaging

Experimentsw erealsop erformedwith actualroadimagesobtainedwiththe ALV when

itw asop erationalatMartin Marietta?Denver? The?groundtruth?waspro videdbya

fusion algorithmcom bining range dataandvideodata ????Reconstructionswereproduced

for about??roadcon?gurations includingcombinations ofturnsand slopechanges? Both

methodsprov edquite robust?giving plausibleandconsistentroad reconstructionsforall

these tests? how ever?theERIM laserrangerhasalimitedrangeof action?Only the?rst ??

meters ofthe roadcould bereconstructed bythefusion metho d? Thereconstruction bythe

??

Page 14

gridinperpective istheplane ofthe Flat?Earthalgorithm?an extensionofthe plane ofthe

p oints ofcontact ofthev ehiclewheelswith the ground? Elevations above thisplanegiv en

by theotheralgorithms aredisplay ed asvertical linesegmen ts? Figure??a?sho wsthe tick

marksof the roadedgesreconstructedbypro jectingtheroad imageon to theFlat?Earth

plane?and theroad edgeelev ationsgiv enby therange?videofusion? Theseelevations also

appear in Figure??b?? along withthecross?segmen ts androad edgesconstructedby the

presentmatc hingp oin tsalgorithm ?called ?mo di?edzero?bankalgorithm? in thecaption??

The result ofy et anothermethod? thehill?and?dalealgorithm???? isalso sho wn inFigure??

Itw as notasp erformant as thepresent algorithmin sharperroadturns?Di?erences of

elevations app earedinsome experimen ts insideviewb etween thereconstructions of the

range?videofusion andthepresentalgorithm?although thedi?erencew ould probablynot

have resulted indi?erentsteerings ofthevehicle? Thisseems tobe duetothe fact that

we obtainedtheroadwidth fromtheFlat?Earthappro ximationapplied to the closestroad

segment?a methodwhich is sensitive tolocal bumps under the wheelsof thevehicle? In

suchsituations of course therange?videofusionalgorithmstill pro ducesa correctroad

pro?le?F urtherdiscussions about these experimen tswithactual videoandrange datacan

be found in????

?Conclusions

In thiswork?we have deriv ed an analyticalcondition forp oin ts taken on theimage of

roadedges tobematc hingp oin ts? i?e?images of opp ositeedgep oin ts?T aking onep oint

on one roadimage edge?we generally?nd more than onecandidatematc hingp oint? All

candidatesw erebac k?projected to?D? andadynamic programmingoptimizationbuilta

physicallyacceptable roadthrough the appropriate cross?segments?

Ab enc hmarkw asapplied tocompare this algorithmwithtwo others? the step?b y?step

zero?bankalgorithmandthe Flat?Earth algorithm? Theprop osedmatchingpointsalgo?

rithmw as foundde?nitely moree?ective than theother algorithmsov erall theconsidered

v ariationsin width andbank?

Experimen ts withrealw orlddataand comparisons withroad reconstructionsobtained

byfusionb etweenvideo data andrangedatashow eda good agreement intheshortrangefor

which rangedataareav ailable?provided the scalingfactor leftunde?nedbythealgorithm

isw ellchosen?The proposed algorithm canusetheinformationprovidedby the rangedata

tocalculatethisscaling factor?and extendtheroad reconstructionto theroad partswhich

??

Page 15

Ac knowledgemen ts

Theauthorswould like tothankPeterMeer?BehroozKamgar?Parsi andAzrielRosenfeld

forhelpfulcommen tsand discussions?The support of theDefenseAdvanced Research

Projects Agencyandthe U?S?ArmyEngineerT op ographicLab oratoriesunderCon tract

???C??????D ARPA Order?????isgratefullyackno wledged?

References

???D?H?BallardandC?M?Brown??ComputerVision??Pren tice?Hall?Inc?

??? D?DeMen thon? ?Inv ersePersp ective ofaRoad fromaSingleImage??CenterforAu?

tomationResearchT echnicalReportCAR?TR????? Univ ersityof Maryland?July?????

???D?DeMen thon??AZero?Bank Algorithmfor InverseP ersp ective ofaRoadfroma

SingleImage?? inProc?IEEE In t?Conf? RoboticsandAutomation? Raleigh?NC? April

????? pp???????????

??? D?DeMenthon??Reconstruction ofaRoadby Matching EdgeP oin ts in theRoad

Image?? Center for AutomationResearchT echnical ReportCAR?TR?????University

ofMaryland?June ?????

??? E?D?Dic kmannsand V?Graefe??Dynamic Monocular MachineVision? and ?Appli?

cations ofDynamic Monocular Mac hineVision??Machine VisionandApplications?

InternationalJournal?Springer?V erlagInternational? ?????

??? K? Kanatani? K?W atanabeand C? Koyama?? ?DReconstruction ofa RoadShape

fromImagesforAutonomousLandVehicles??Proc? Int?Workshop onIn telligent Robot

Systems?Octob er?Nov emb er?????Toky o? pp? ????????

??? K?Kanatani andD?DeMenthon? ?Reconstruction of?DRoadShape from Images?A

ComputationalChallenge?? Cen ter forAutomationResearchT echnical Rep ort CAR?

TR????? Decemb er ?????

??? K?Kanatani andK?Watanab e??Computing ?DRoadShap e fromImagesfor ALV?

AChallenge toanIll?PosedProblem?? ComputerScienceT echnical Report CS??????

GunmaUniv ersity? June?????

??

Page 16

???? S? Ozawa and A?Rosenfeld??Syn thesisofaRoadImage asseen fromaVehicle??

P atternRecognition? ????????pp?????????

????C? Thorp e?M?H?Heb ert?T? Kanade?andS?A?Shafer? ?Vision andNavigation forthe

Carnegie?Mellon Navlab?? IEEET rans?Pattern Anal?Mac hine Intell??v ol?P AMI????

no??? May ?????pp?????????

???? M?A?T urk? D?G?Morgen thaler?K?D?Grem ban?andM?Marra??VITS?AVision

SystemforAutonomous LandV ehiclenavigation??IEEETrans?Pattern Anal?Machine

In tell??vol?P AMI???? no??? May ?????pp?????????

???? A?M?W axman? J?LeMoigne?L?S? Davis? B?Sriniv asan? T?R? Kushner?E? Liang?and

T?Siddalingaiah? ?A VisualNavigation System forAutonomousLandV ehicles?? IEEE

J? RoboticsandAutomation? v ol?RA???no???April?????pp?????????

??