Page 1
Extending CLP?FD?withInteractiveData
Acquisitionfor?DVisualObject
Recognition
M?Gavanelli?E?Lamma?P?Mello?M?Milano
DEIS?Univ?Bologna?
Viale Risorgimen to? I??????Bologna?Italy?
E?mail?felamma?mgavanelli?pmello?mmila nog?dei s?u nibo?it
M?PiccardiR?Cucchiara
Dip?Ingegneria?Univ?Ferrara?Univ?Mo dena?
ViaSaragat????????Ferrara?Italy?ViaCampi??????????Mo dena?Italy?
E?mail?mpiccardi?ing?unife?it
E?mail?
rita?cucchiara?unimo?it
Abstract
Thispaperaddresses the?D object recognitionproblemmodelledasa Con?
strain tSatisfactionProblem? Inthissetting?eachobjectviewcanbemodelledas
aconstraintgraphwhereno desare objectpartsandconstrain ts aretopological
andgeometricalrelationshipsamongthem? By modellingtheproblem asaCSP?
wecanrecognize anobjectwhen allconstrain ts aresatis?edbyexploiting results
from theCSP?eld? How ev er?inclassicalCSPsv ariabledomains have tobe
staticallyde?nedatthebeginningoftheconstraintpropagationprocess?Thus?
notonlyfeatureacquisitionshouldbecompletedbeforetheconstraintsolving
processstarts?butallimagefeaturesshouldbeextractedevenif notbelonging
tosigni?cantimage parts?Invisualapplications?thisrequirementturns outto
beine?cientsince visualfeatures acquisitionisav erytimeconsumingtask?We
present an
Inter active ConstraintSatisfaction
modelfor problemswherev ariable
domainsmaynotbecompletelyknown atthebeginningofthecomputation?
andcanbe interactiv elyacquired during thecomputational process?Somevari?
abledomainvaluesmaybealreadyatdisposal whentheconstraintpropagation
process starts?while others canbe dynamicallyacquired during thecomputa?
tiononlywhen needed ?ondemand?? Theconstraint propagationprocessworks
on alreadyknowndomainvalues andadds newconstrain tsonunkno wndomain
parts?These newconstrain ts canbeused toincrementallyprocess newinfor?
mation withoutrestarting theconstraint propagationprocess fromscratcheach
time new informationisavailable?Inaddition?theseconstraintscan guidethe
featureacquisition process? thusfo cussingattentiononsigni?cant imageparts?
Wepresent theInteractive CSPmo del anda propagationalgorithm forit?We
prop oseanimplementation ofthe frameworkinConstraintLogicProgramming
on Finite Domains?CLP?FD??
? INTRODUCTION
ConstraintSatisfaction systemsprovidea simplebut powerful framew orkfor solving
avariety ofArti?cialIntelligence?AI?problems?ConstraintSatisfactionProblems
?CSP?forshortinthefollowing?arede?nedona?nite setofvariables each rangingon
a?n umericalorsymbolic? domainandaset ofconstraints?We assumevariable domains
Page 2
tobe ?nite?Asolutiontoa CSPis anassignment ofvalues tovariableswhichsatis?es
theconstrain ts?Propagation algorithms ???? ?e?g?? forw ardchecking? lo okaheadetc??
haveb eenprop osedbasedon the active use ofconstrain tsduring thesearch process?
The idea istoremov e? duringsearc h?bymeans ofconstraintpropagation?combination
ofassignmentswhich cannotapp ear in anyconsistentsolution?
ACSP?basedinference enginehasbeensuccessfullyused in manyapplications?In
thispap erwe focus ona ?D objectrecognitionapplication wherealo wlev elsystem
providesalargeamount of?constrained?data tobe processed?i?e?? visualfeatures
ofobje ctsin an image? Severalexamples ofCSP?basedreasoning systemshaveb een
proposed forobject recognition?see forinstance ??????????????? Visualapplications
usuallyrequire someform of interaction betw eenalowlevelmodule?imageproces?
sorandfeature extractor?providing ?constrained?data? andaconstraint solver? In
classicalCSPs?v ariabledomainshave tobe completely knownbefore theconstraint
propagationprocess starts?Data acquisitionand itspro cessing aresequentiallyp er?
formedth us leadingto anine?cientbehaviour of thewhole system especiallywhen
thedataacquisitionprocess iscomputationallyexpensiv e?Forexample?aCSPmodule
in teractingwithalow level visual system should?rst acquireallthe visual features
inthescene?thusrequiringthe lowlev elsystem to process thewholeimage? inorder
tocreatevariable domains?and thenstarttheconstraintpropagation processforob?
jectrecognition? In avision systemthe data extractionphase isv erytime?consuming?
usuallyitism uch morecomputationallyexpensivethan theconstraint solvingphase?
Wearguethatinterleavingtheacquisitionofdomainvaluesandtheirprocessing
couldgreatlyincrease thep erformancesof the object recognitionprocess?Domainvalue
acquisition canbep erformedon demandonlywhenv aluesaree?ectivelyneeded? This
approach canbeseen asakind oflazy domainevaluation?Lazy evaluation ??? is known
asaparameter evaluation mechanismwhichavoidsa computationif itsresultingvalue
will nev erbe used?Similarly?weavoidtoconsidervaluesforconstraint propagation
ifthey arenotneeded?Thisidea hasb eenalreadyexploited in the?eldofconstraint
satisfaction in ??????whereas soon asoneconsistentvalueisfound? thepropagation
stops in ordertop erforma minimaln umb er ofconstraintc hecks?
F urthermore?afundamentalpointwhichcanbeexploitedinourframeworkis that
theacquisitionprocesscanbeguidedbyconstraints?calledinteractiveconstraints?thus
leadingtoretrieveonlyconsistentvaluesandminimizinguselessdataacquisitions?Ina
visualsystem?thisfeatureallo wstofo custheattention of thefeature extractionmodule
ona restrictedpart ofthe scene?bypropagatingspatial andtopologicalconstrain ts?
second? toconstrainthe featurespaceand assist thecomputation ofvisualfeatures?
Therefore?aCSP systemshouldbeable notonlyto prunethe dataset afterit has
b eencomputed? butalso toguide thedataacquisitionpro cess?
For thispurp ose?wepresen t anInteractiveConstraintSatisfactionmodelwhere
domains canbepartially knownwhen theconstraintsatisfaction process startsand
aredynamicallyacquiredduringthecomputation?Weapplythis frameworktoa?D
objectrecognitionapplication?
ApowerfullanguageformodellingandsolvingCSPsisConstraintLogicProgram?
ming?CLP??????InrecentyearsCLPhasbeensuccessfullyusedforsolvinghardcom?
binatorialproblems????????modeledasConstraintSatisfactionProblems?CSPs??In
thispaper?wefocusonConstraintLogicProgrammingon?nitedomains?hereinafter
referredtoasCLP?FD??Wehave implementedthe interactiveframeworkontopof
Page 3
the?nitedomainlibraryoftheECL
i
PS
e
language???? Theextension of theconstraint
solv er isaimed at copingwith interactiveconstrain ts andpartially orcompletelyun?
knowndomains?
Thepaperis organizedas follows? insection?we show how to mo del theobject
recognition problemasa CSP? In section? the interactive CSP framework is described
alongwith theinteractivev ersionof theF orwardCheckingalgorithm presen tedin
section?? Theextension of theECL
i
PS
e
CLP?FD?library issk etchedin section??
Finally? section? shows theapplicationof theframework toa ?Dobject recognition
problem and experimen talresults?A discussion anda mention of futurew orkconclude
thepaper?
?MODELLING OBJECTSTHROUGH CONSTRAINTS
?Dobjectrecogitioninvolv estwo steps?object mo delbuilding andmo delsolving?We
adoptaconstrain tbasedapproach toob ject recognition?Th us?ob jectscanbe modeled
bymeansofconstraints?andrecognizedbymeansofconstraintsatisfaction?
Inthis section?we fo cusonmo delbuildingby de?ninga generalandreliableobject
model?inordertoresultpossiblyinvariant toroto?translation in?Dspace?to distance
ob ject?camera andother environmentfactors? Asaconsequence? the object model
cannotbe limited toa matc hingofmeasuredfeatures toabsolutevaluespreviously de?
?ned?but shouldbebased ongeometric andtopologicalrelationshipsbetweenfeatures
??????? Each objectwew ant torecognize canberepresen tedby meansofaconstraint
graphwhere eachob ject partorcharacterizingprimitive featureis modeledbya node
?variable? of thecorresponding CSPandspatial orshape relationsamong object parts
canberepresentedbyarcs?constraints??
Speci?c aspects of thesingleprimitiv es maybe modeled asunaryconstraints? such
asthe minimum length ofan objectpart? itscolor? theplanarity ofa surfaceandso
on?whilegeometricand topologicrelationshipsb etw eenthem canberepresen tedby
binaryconstrain ts?e?g?? angularrelationshipsb etw eencontours? lines?orsurfaces?or
spatial relationships?suchasisconnectedto?touch?iscontained??
Asanexample?ifwewanttomodeltheL?shapedobject shown inFigure??aweshall
haveano decorrespondingto each of thesurfaces ?named respectively
X
?
?
X
?
????? and
will imposetheconstrain tsdepicted inFigure??b?Notethat in ?Dobject modelling?
wehave theproblemof multiple views?Weare dev elopinganobserver?independent
mo delfor ?Dob ject?byusingasingle object graphand exploitinghidingconstraints
fornon visible parts?This aspectwillbe brie?ydescribedin section???? and itis
subject ofcurrentresearch?
Someoftheseconstraintsareredundant? Thisredundancyisusefulwhencon?
straintsarepropagatedbymeansofanincompletealgorithmsuchasforexample
arc?consistency?
Supposethevisionsystemprovidesalltheinformationaboutthesurfacesinthe
scene?Therefore?eachvariablerangesonadomaincontainingallthesurfaces?We
canperformastraightforwardbutine?cientbacktrackingalgorithminorderto?nd
asolution? Sincewemo delobjectsbymeans ofconstraints?wecanexploitconstraint
propagationinordertoprune thesearchspace?Therefore?we intertwinea propagation
processthatremov escombinations ofassignmen tswhic hcannotappearinan yconsis?
tentsolution?i?e?? whichcannotformanL?shapedobject?and alab elingstrategythat
Page 4
Figure??
Mo del ofan
L?shaped
object
assignstoeachvariableav alue?apossiblesegmen t??Ifasolution isfound?we have
identi?ed thesearc hedobject in thescene?i?e??values forvariables?surfacescomposing
themodel?which areconsistentwithconstrain ts?Notethat in thissp eci?cexample
thereare symmetriesintheproblem?i?e?? somep ermutationof thevariablesmapsa
solution onto anothersolution??????
???Multiple viewgraph
RITA
? INTERACTIVECSP
In theprevioussection?we havepresen ted how to model objectsby meansofcon?
straints?Th us?we canuseconstraint satisfaction techniquesinorderto recognizean
object?Theproblemwith thiskind ofrepresentationconcernsthefactthatwe have
toacquirefromthevisionsystemalltheinformationaboutsurfacesinthescenebe?
foreprocessingthem?Thefeatureextraction? speciallyin?Dvisualapplications?isa
computationallyexpensivetaskifcomparedwith theconstraintsatisfaction process?
Constraints expressinglocality ?such as touch? canbe exploitedin orderto fo custhe
wholesystem?s attention ona limitedimagepart? th usavoiding featureextraction on
mostof theimagesurface?Selectiv eattention isafundamen talskill ofbiologic vi?
sionsystems?since everyimage contains hugequantitiesof dataandpro cessing allof
themwouldbeunwise?Weprop osean alternative approach?based onwhatwecall
InteractiveConstraintSatisfactionframework?thatintert winesthevisionfeatureac?
quisitionwiththeconstraintsatisfactionprocess th usreducingthefeatureextraction
computational cost?
We ?rststartbygivingsomepreliminaries onCSPs?ACSPisde?ned onaset
ofvariables
X
?
?????X
n
rangingrespectivelyon?nitedomains
D
?
?????D
n
?A
constraint
c?X
i
?
?????X
i
k
?de?nes asubset ofthe cartesianproductof
D
i
?
?????D
i
k
?
i?e??asetofcon?gurations ofassignmen tswhich may appear inaconsistentsolution?
Inthispaper?wefo cusonbinary CSPs?AbinaryCSPcanberepresentedbymeans
of aconstraintnetworkwhereeac hnodeisavariableandarcsareconstraints?
Astandardconstraintsolverneedsalltheinformationandtheknowledgeonthe
problematthebeginningofthecomputation?Thenitpropagatesconstraintsbyremov?
Page 5
ingassignmentswhich cannotappear in anyconsistentsolution? Theinteractionwith
alowlev elsystem?andtheconsequentpropagation?requiresadataacquisition process
which lastsduring thewhole computational process?Therefore?we havetoc hange the
classical CSPmodel?and allowthe propagationalgorithm towork onpartiallyknown
domains?
To thispurp ose?we de?neanInter activeCSP?ICSP? mo delwhich hastocope
withincomplete domains?Domainscanbepartially knownin thesensethatsome
domain elemen ts canbe alreadyatdisposal forpropagation?while other domainel?
emen tsha ve tobeacquiredfromalow lev el systemin thefuture?Thestrength of
this approachconcerns the factthattheICSPsystem canguidedataacquisitionby
meansofconstraints? andincrementally processnewinformation withoutrestartinga
constraintpropagation processfromscratch eachtime newdataareavailable?
On thebasis of theserequiremen ts?wede?ne theICSP modelasfollows?
De?nition?
AnInteractive CSP ?ICSP?isde?ned ona?niteset ofvariables
fX
?
?????X
n
g
each rangingonapartiallyknowndomain
fD
?
?????D
n
g
whereeach
D
i
?
fKnown
i
?
U nK nown
i
g?
K nown
i
representsthe known part?i?e??the set ofav ailablevalues?while
U nKnown
i
representsinformation whichisnotyetavailable? i?e?? thesetofv alueswill
be retriev ed in thefuture?Both
K nown
i
and
UnK nown
i
canbepossibly empty
?
?
Also?foreach
i?
Known
i
?
U nKnown
i
?
?? Aninter activeconstraintamong vari?
ablesde?nesa?possiblypartiallyknown?subsetofthecartesianproduct ofvariable
domains?AsolutiontotheICSPis?asforstandardCSPs?anassignmentofvaluesto
variableswhichisconsistentwithconstraints?
Constraintpropagationisquitedi?erentfromthestandardcase?Consider?forthe
sakeofclarity?onlybinaryconstraints
c?X
i
?X
j
??Inthemostgeneralcase?both
X
i
and
X
j
domainscontaina nonempty kno wnandunkno wnpart? Inordertopropagate
theconstraint
c?X
i
?X
j
?wehave topropagatefourkinds ofconstrain ts?respectively
betweenthe knownand unkno wn partsofv ariabledomains?
c?Known
i
?K nown
j
??
c?Know n
i
?UnKnown
j
?
c?UnK nown
i
?Known
j
??
c?UnKnown
i
?UnK nown
j
?
andcollectthepropagationresults
?
?While the constraintc heckonknownpartscanbe
performedasusual?thecheckonatleastone unknownpartrequiresadataacquisition
inordertoacquirenewinformation?Inaddition?thedataacquisitioncanbeguided
bymeans of in teractive constrain tsinthesensethat dataacquisitionretriev esvalues
whichareconsistentwithconstraints?
Letusseea simpleexample inthedomainofintegers? Considertwo domain
variables
X
and
Y
rangingrespectivelyonthefollowingdomains
f??
?g?
X
?and
f???
?g?
Y
??Thekno wnpart of thedomain of
X
containstwov alues
f??
?g? while
its unkno wn part?
X
??representsnotyetavailablevaluesfor
X
?Similarly? theknown
andunknownpartofthedomains of
Y
are
f???
?g
and
Y
?respectiv ely?Aconstraint
between
X
and
Y
?say
X
?
Y
?issatis?edif andonly ifvariables
X
and
Y
assume
consistentvaluesintheirknownpart?e?g??
X
??and
Y
??or
X
?? and
Y
???or
ifthedata acquisitionprocessprovidesconsistentvaluesforthevariables?
?
Whenboth areemptyaninconsistencyarises?
?
We refer?with abuseofnotation?todomainsinstead ofvariables?However?the meaningis
straigh tforward?
Page 6
We nowpresentintuitiv ely how theconstraint propagationalgorithmworks on the
above example?Then?we sketchandexplainthe pseudoco de thatimplements it?
Constraint propagationbetw een thekno wn partoftheconstrain tscanbep erformed
asusual? Notethat novalueshow evercanberemov ed ifbothvariabledomains are
partiallykno wnsincetheycouldbesupportedbyfuture acquisition?The ICSPcom?
putation startswitha labelingstep? as in CSPsearc h?Instead? theconsistencyc heck
b etw eenv alues? and?forv ariable
X
and theunknownpartofv ariable
Y
?i?e??
Y
??
calls fora data acquisitionthat isaimed tocollectatleast onev alue for
Y
which is
consistentwith the kno wnpart ofthedomain of
X
? Inotherw ords?thesystem collects
forv ariable
Y
atleast onevaluewhichis greater orequal to? or ??This is equivalent
toposeaconstraint onthe unkno wnpartof
Y
?e?g???
?
Y
?
?
?
?
Y
?andguide
thedataacquisitionbymeansoftheseconstraintsbyaskingthelowlev elsystemfor
thosev alues thatsatisfythe abovemen tionedconstrain ts?Similarly?theconstraint
propagation actsonthe unknown partof
X
andtheknown partof
Y
? Finally? the
constraintb etw eenunkno wnpartswillc hecknewacquiredvalues assoonastheywill
beav ailable?
In Figure?we sk etch thepseudo?co de forbinaryconstraintpropagation inthe
interactiveframework? Thepro cedurepropagate
constraintsworks onaconstraint
c
in orderto reducev ariabledomains
D
?We have todistinguish threecases?the ?rst case
concernstheclassicalconstraintpropagationwhenbothv ariablespresentanon?empty
known part?procedurepropagate
known??The second caseregards thepropagation
betweenan unkno wnpartandakno wn one?pro cedurepropagate partiallykno wn??
Inthiscase? anacquisitionshouldbeperformedwhich canbeguidedbytheconstraint
itselfonthebasisofvaluescontainedintheknownpartof thedomain? Procedure
propagate partiallykno wnqueries thelowlevelsysteminorder toretrieveonlyvalues
consistentwith thekno wn part of
X
forv ariable
Y
? Thelast caseconcernsaconstraint
propagation ontwo unkno wnparts ?pro cedurepropagateunknown??Inthiscase the
constrain tis delayedsincetheknowledgeacquisitioncouldnotbeguided by anyknown
value?Thedelay edconstrainton
UnKnown
domainpartsguaranteesthatfuture
acquisitionswillbe consistentwithconstraints?
Notethatconstrain tson at leastone unknownpartintensionallyrepresentpotential
solutionsonnotyetacquireddata?
Aninterestingpointconcernsdataacquisition?Dataacquisitionisguidedbycon?
straintscontainingonevariable?wewill callit guidingvariable?which hasa partially
orfully kno wn domainandonevariable?wewillcallit guidedvariable?whose domain
iscompletelyunkno wn?We haveto answ ertobasicallytwo questions?
?
how manyvaluesoftheguidingvariabledomainto useinordertoguide data
acquisition?
?
howmanyvaluesof theguidedv ariable toacquire?
Basically?alazyapproachtendsto minimizedata acquisitions? sinceit?soften the most
expensive task?inmanyproblemsitinvolv eshardsignal processing??Th us?thedata
acquisitionisstopp edas so on asoneconsistentvaluehasbeenretrievedfortheguided
variableandisguidedbymeansofonevalueinthedomainof theguidingvariable?On
thecon trary?aneagerapproachcollectsallconsistentvalues fortheguidedvariable
andguidesdataacquisitionbymeans ofallkno wnvalues?
Page 7
pro cedurepropagateconstrain ts?D
?c?A?B
??
begin
D
A
?
K nown
A
?
U nKnown
A
?
D
B
?
Known
B
?
UnKnown
B
?
propagate
kno wn?Know n
A
?Known
B
??
propagate
partiallyknown?Known
A
?U nKnown
B
??
propagate
partially known?K nown
B
?U nK nown
A
??
propagate
unkno wn?U nK nown
A
?U nKnown
B
??
end
procedurepropagatepartially known?K nown
X
?U nK nown
Y
??
begin
guided
acquisition?c?Known
X
?U nK nown
Y
???
end?
procedurepropagateunkno wn?U nK nown
X
?U nKnown
Y
??
b egin
dela y?c?UnKnown
X
?UnKnown
Y
???
end?
Figure??
Theinteractiveconstraintpropagation
Theb estchoice dependson theapplicationwe have tosolv e? Inthe?eld of ob ject
recognition?domainv aluescomefrom animage pro cessing anda feature extraction
system?b othofwhich are computationallyexpensive tasks?F or?Dob jects images
arerange ?ordepth?images andextractedfeaturesaresurfaces? This activity greatly
bene?ts oftheexploitationoflocalit ycriteriathatrestrict theimagepart tobe pro?
cessed? As soonasarestrictedpartofanimage isselected? retrievingone or all features
b elongingtothatimage partiscomputationally comparablewith theretrievalof only
onefeature?Thus?inthe visualapplication?we decided toimplementasearch al?
gorithmcapable ofexploitingthisprop erty?Dataacquisitionwillbep erformedby
meansof interactiveconstrain tswith only onevaluefor the guidingv ariable?i?e? the
guidingv ariablewillbe instantiated? andby retrievingall theconsistentvalues forthe
guidedvariable? Thisideais similar totheForw ardCheckingalgorithm? and is widely
explained inthenextsection?It?sworthnotingthatthe ICSPframeworkcomprises
theclassicalCSPenvironment?sowearenotforced toimplementeveryconstraint
theinteractiveway? Someconstrain tsareuseful forinteraction?e?g? theconstraints
expressing lo cality inthevisionsystem?Otherconstrain tsmaybeused only forprun?
ingthesearchspace? as in theCSP case?Whenanon?interactiveconstraint hasto
dealwithincomplete knowledge? it can simplysuspend andwait fortheinteractive
constrain tstoacquirethe neededinformationitems?
?INTERA CTIVEF ORWARDCHECKING
One ofthew ell kno wnand widelyacceptedpropagation algorithms forsolvingCSPsis
theforw ardc hecking ?FC?tec hnique ????TheFC algorithmin tert winesalab elingstep?
Page 8
whereav ariable
X
isinstan tiated toavalue
v
initsdomain?anda prop agationstep
where domainvariableslink edwith
X
bymeans ofconstrain ts arecheck ed inorder to
removevalueswhichare notcompatible with
v
?
Inour framework?wehave tocope withpartially knowndomains?Therefore? the
operationalb ehaviourof theFCalgorithmshould bechangedaccordingly? In tuitively?
the ?rstlabelingstepinstantiatesav ariable
X
toav alue
v
belongingtothe known
part ofthe domain if any?Otherwise?a dataacquisition isperformedretrievinga
value
v
whichis successiv elyassignedto
X
? The prop agation stepconsiders domain
v ariables
X
?
?????X
k
link edwith
X
by means ofconstrain ts?Thisstepremov esfrom
theknownpart of
X
?
?????X
k
domainthosevalueswhich arenotconsistentwith
v
?
and?ev en tually?acquiresconsistentvalues forthe unknownpart
?
?
Notethat inthe algorithmpresen ted inFigure?? only the?rsttwo pro cedures
?propagate
kno wnandthe?rstpropagatepartiallykno wn?areperformedsincefor the
forwardcheckingstrategy onevariableis alwaysinstantiated?thuscompletely kno wn??
Let usconsider anexample in the?eldof ?Dshaperecognition? Ifwew ant to mo del
arectanglewe can identifyfour nodescorresponding to the fouredges composing the
rectangle ?n umb ered respectiv ely
X
?
?
X
?
?
X
?
and
X
?
? andweimp osethefollowing
symmetricconstrain ts?
allD iff erent??X
?
?X
?
?X
?
?X
?
???
touch?X
?
?X
?
??
touch?X
?
?X
?
??
touch?X
?
?X
?
??
touch?X
?
?X
?
??
no
touch?X
?
?X
?
??
no touch?X
?
?X
?
??
sameleng ht?X
?
?X
?
??
samel enght?X
?
?X
?
??
perpendicular
?X
?
?X
?
??
perpendicular
?X
?
?X
?
??
perpendicular
?X
?
?X
?
??
perpendicular
?X
?
?X
?
??
parall el
?X
?
?X
?
??
parall el
?X
?
?X
?
?
?
?
We consideronlythe touchconstraintasinteractive?becauseitexpresses locality
andallo wsselectiveatten tion?Thetask is torecognize the ?rstrectangle in thescene
depictedinFigure ??a?Variabledomainsaresegmentsretriev edfrom theimage?
Initially?thevariabledomainsarecompletelyunknown?
X
?
??
UnKnown
?
?
X
?
??
UnKnown
?
?
X
?
??
UnKnown
?
?
X
?
??
U nKnown
?
?
Figure ??bsho wsan IFCcomputation un tilsuccess is reached?TheIFCalgorithm
startswithalabelingstepon variable
X
?
?Sincethe domainof
X
?
doesnotcon tain
any acquiredvalue?thelabelingstepperformsafeatureacquisitionprocess ?p ossibly
guidedby unaryconstraintson
X
?
??Asegment
a
intheimageisretriev edandassigned
to
X
?
?i?e??
X
?
?
a?
Now?theplainFCalgorithmcollectsallthev ariableslink edto
X
?
by meansof
constrain tsandremov esfrom theirdomainsallv alueswhichareinconsistentwith
a?
Since thenonin teractivealgorithmrequiresall thesegmen tstobeinthedomainsat
thebeginning ofthecomputation?it hastocheckvalue
a
againstallthesegments
in
X
?
?sdomain?i?e?
fa?b?????l
g??before?ndingthatnovalueisconsisten t?The
IFC?instead?doesnotknowthevaluesinthedomainsof thenextvariables?so it
requestsv aluesconsistentwith the interactiveconstraint
touch?X
?
?X
?
??Notethatthe
featureacquisitionprocessis guidedbythe abovementionedconstraint exploitingthe
localitycriteria embedded in theconstrainttouch? Therefore?theunderlying visual
?
Thisdata acquisitionisperformed ornotonthebasisofaeager orlazyacquisitionpolicy?
?
Theconstraints
p erpendicular?
parallel
and
samelength
havean ob vious seman tics? The constraint
touch
?resp ectively
no touch? means thatoneendingpointof the?rstsegmen tmusttouch anending
pointofthesecondone ?resp? theendingpoints mustnottouch eachother??
Page 9
Figure??
ExampleofICSPcomputation
systemlo oksfor segments touching
a
th usfocusingatten tionaround it? Onlysegment
a
touc hesitself? so thedomainofv ariable
X
?
becomes
fag?thisv alueisdeletedby
thenoninteractiveconstrain tsandaninconsistency israised?Inthiscase?the IFC
algorithmperforms onlyoneconstraintc hecktodetectinconsistency?
Computationsucceedswhen
X
?
is assignedthev alue
d? assho wninFigure ??b?
Asconcernsthenumber ofv aluestobe retriev ed?intheforward chec kingalgorithm
we can decidetoacquire allv alues consistentwith thecurrentlyinstantiatedvariable
or onlyone v alue?In the ?rst case?thedomains of
X
?
and
X
?
become completely
known andthey arethe same asthe onesresulting fromthe classicalforwardchecking
algorithm aftertheinstantiationof
X
?
to
d? In the secondcase?thedomains of
X
?
and
X
?
areleftpartially kno wn andconstrain tsareimposed on thev ariables representing
theunknownparts?This secondapproach is similar toperformingaminimal forw ard
c heckingalgorithm ????
Ina visualsystem environmen t?we decide tocollect allvalues forvariables since
featureextractionexploiting lo calitycriteriaisalmostindependentfromthenumber
offeaturesextracted?Th us?the system collectsfoursegments
d?
e?
g
and
l
?
whichare
putin thedomainsof
X
?
and
X
?
?
Theunknownpartsof
X
?
and
X
?
are nowcompletely speci?ed andthedomain
of
X
?
and
X
?
con tain allvalues whichtouch
d?Value
d
isremov edfromb oth these
domainsby noninteractive constrain ts?Again? thenonin teractiveFC has tocheck
thevalue
d
assignedto
X
?
againstallthevaluesintheotherdomains?i?e?allthe
segmen tintheimage? toget to the samesituation?Thein teractive?instead?usesthe
constrainttouchtoguideacquisitionandtheotherconstraintshaveonlytocheck the
fouracquiredvaluesforconsistency?
Thenextlab elingphaseassigns
e
to
X
?
?thisc hoice leads to failure and the next step
sets
X
?
?
g
?A secondconstraint propagationprocess startsbyconsideringallv ariables
in volvedinaconstraintwith
X
?
?Consider theconstrain t? describedinsection ??
b etween
X
?
and
X
?
stating thatthetwovariables representtouc hing segmen ts?anda
constraintwith
X
?
statingthatthetwovariablesshouldhavethesamelength?The
?
Notethatconstraintsdescribingtherectanglearesymmetric?Symmetriesshouldbeavoided
inaconstraintsatisfactionprocedure?????Therefore?inpractice?we donotputalltheacquired
segmentsinbothdomains?Intheexample?howev er? for thesakeofsimplicity?weomitthetreatment
ofsymmetries?
Page 10
?rst constraint propagationpro cessresultsinafeatureacquisition? collectingallv alues
thattouch
X
?
?i?e?
d?
f
and
g
?Notethat?again?featureacquisitionisfocussed around
g
anddoesnotneed toconsider thewholeimage?Thesesegmen tsareput inthe
?kno wnpartof the?domainof
X
?
while theunknown partis deleted?Thepropagation
stepbetw een
X
?
and
X
?
istheusualforwardc hec kingconstraint propagationsince the
domain of
X
?
iscompletely known? TheFC algorithm continuesthelab elingand the
constraintpropagation processasusualsince allthedomains now areknown?
The purp oseof theinteractive framework isto forcethelow levelsystem toretrieve
anumber of segmentswhich issigni?can tlysmallerthanthose retrievedbya non?
in teractivesystem which ?rstcollects allsegmen tsin an imageand thenstarts the
constraintpropagation process? Then umb er ofextractionsm ustbek ept assmall
aspossible? since theextraction pro cessisthemost expensivetask?usuallytheCSP
solving processemplo ysanegligiblecomputationtime w?r?tthefeatureextractiontime?
Inordertoavoidre?extraction offormerlyacquiredvalues?due? e?g??tobacktracking??
each extractedelement isstored inmemory afterinteraction? insuccessivecomputation?
ev ery segmentwill?rstbe searchedfor in memoryandthen requested to the low?
lev elsystem? Notethat theamount of memoryrequired for thisstructure isonly
prop ortional tothen umb er ofelemen tsina domain?which is negligiblewith resp ect
to the structuresneededby theFC algorithm????
InFigure??we have sk etched thebasic InteractiveFC algorithm?It?rst selects
avariable
Var
tobeinstantiated?thenitp erforms an interactivelab eling?in terac?
tive
label??Thisproceduretakesavaluein theknownpartoftheselectedvariable
domainif itexists?otherwiseitacquiresonevaluefortheselectedvariable?procedure
acquisitionv ar?? Thepro cedure collectconstrain tscollectsallconstrain ts containing
theselected?instantiated?variable?Then?foreachcollectedconstraint?theconstraint
propagationalgorithmpresentedinFigure?starts? Notethatb eing
V ar
instan tiated?
theprocedurepropagateconstrain tsalways?nds the ?rstvariable completely known
andeitherperformsthe usualpropagationifotherv ariablescon tainedin
Constr
are
alsoknown? or retrievesv alues with the propagatepartiallyknownprocedure?
??? Interactive MinimalForwardChec king
TOGLIAMO?
Whentheconstraintcheckisacomputationally costlyop eration?theF orward
Checkingalgorithmis outperformedby theMinimalF orw ardCheking ?MFC? tec h?
nique????FCchecks forconsistency ev eryelement inthe forw ardconnecteddomains
allowing to detectfailure inearlystagesofcomputationandtolimitthenumb erof
backtracking mov esneededduringsearch? Ifa domainbecomesemptyduringsearch?
FC detectsinconsistencyand backtracks?An yway?inordertoavoidbacktracking?only
oneconsistentdomainvalueisneeded?
TheMFCalgorithm looksforaconsistentv alue in ev eryfutureconnected domain?
the otherelemen tsareleftunc heck ed? Ifthe onlyconsistentvalueisremoved?another
candidatemustbefoundandcheck edfor consistencywith respecttoallthepastcon?
nectedvariables?Thisreducesthenumb erofconstraintchecksperformedonaverage
duringsearc h?butrequirescomplexdata?structuresmanagingand?inaCLPframe?
work?the needfora modi?edbac ktrackingmec hanism?Aspoin tedoutin????this
algorithmis notsogoodwhenconstraintchec ksare notcomputationally exp ensive?
Page 11
pro cedure IFC?C
?
D
?
begin
forallvariablesdo
begin
select
v ariable?V ar
?D
??
interactive
lab el?V ar
?D
??
collect
constrain ts?C
?D
?Var
?C
???
foreachconstraint
Constr
in
C
? do
propagateconstraints?D
?Constr
??
end?
end?
procedureinteractivelabel?Var
?D
?
b egin
ifunkno wn?Var
?
thenacquisition
v ar?Var
?D
V ar
??
lab el?V ar
?
end?
Figure??
Theincremen talforw ardc hec kingalgorithm
The Interactive MinimalForw ardCheckingkeepsonlyone consistentvalue in the
kno wnpart ofdomains? while theotherv aluesareleft unextracted?Iftheconsistent
value isremov edbyconstraint propagation?anotherv aluem ustbeacquired? This
technique isveryusefulwhenminimizing thenumb erofextractedv alues isan issue?
anyway? ina visionsystem it isnot?becauseextractingall theconsistentvalue is
performed asa single operation? Sinceforthevision systemextractingallthe features
in anareais nearlyas costly asextracting only one feature?the IFCalgorithm ism uch
morepromising?
?EXTENSIONS OF THECLP?FD? LIBRARY
We haveimplemen ted the In teractive ConstraintSatisfactionframew ork ontopof
the?nitedomainlibraryof theECL
i
PS
e
language ????We havechosentoexploit
ConstraintLogicProgramming ????on?nite domains?CLP?FD?? sinceitisav ery
e?ective programmingparadigm forsolvingCSP????????The CLP?FD?solv erhas
b eenextendedbymeansofuserde?ned constraints inorder tocopewith partially
known domains?
Inparticular?theimplementationhasconcerned?
?
anextensionoftheconstraintsolv erin ordertocopewithin teractiveconstrain ts
and partiallyor completely unkno wndomains?
?
someuser?de?ned interactiveconstraintsperformingdataacquisition?whenwork?
ingonunknown domainvariables?andclassicalconstraintpropagation?when
workingon knowndomainvariables??
Page 12
Asconcernstheextension oftheconstraintsolv er?we have implementedasetoflow?
levelpredicateswhichallowtheusertoprocess partially knowndomainvariables?
mo difythemand writenewconstraintpredicates?Inparticular?we have extended the
followingECL
i
PS
e
predicates
?
dvar
remove element
which remov esan element fromavariabledomain ?this
predicatehasb een implemented alsofor remo vingthe greatest orthesmallest
domainelemen t??
?
dvar
update
which up datesa domainv ariable?
?
dom
member
whic h selectsa domain element?
Inaddition?a newpredicate
specifydomain
hasbeen de?ned inordertoper?
formdataacquisition and in troduceinthedomain newv aluesacquired during the
computation?
Onthe basisof thissolver extension?wehave implemented somein teractive con?
strain tsperformingthepropagationexplainedinthe previous sections?As anexample?
we sk etchheretheco de ofaninteractiveconstraint
itouch
b etw eentwovariablesrep?
resentingtwosegments?
itouch?S??S?? ??
?dvar?domain?S????
?? ?dvar?domain?S????
??touch?S??S??
?itouch?propagate?S??S???
? ?dvar?domain?S????
??itouch?propagate?S??S??
?make?suspension?itouch?S??S ?????Susp??
insert?suspension??S??S???Susp ?specifyofdom?pd?dom?pd?
???
itouch?propagate?S??S????
?nonvar?S??
??dvar?domain?S??Dom???dom?to?list?Dom??L???
acquisition?L??L??? specify?domain?S??L??
?make?suspension?itouch?S? ?S?????S usp??
insert?suspension?S??Susp ?anyoffd?fd??
insert?suspension?S??Susp?spe cifyofdom?pd?dom?pd? ??
Ifb othvariablesareknown? thenon?interactive constraint
touch?S??S??
is called?
Otherwise? if one ofthetwovariables isunkno wnadataacquisitionstartsfor the
unkno wnvariable onthe basisof theknownone? Ifb othvariables are unknownthe
constraint is susp ended? Notethat inthe case offorw ardc hec kingstrategy? thislatter
casenever happ enssinceconstrain tsarec heck ed withonevariableb oundtoavalue?
Theextension of theCLP?FD? solv er doesnot a?ectthe declarativeseman tics of
CLP itself? butonlyits op erationalb ehaviour? in fact?the in teractive constrain tsand
thenon in teractive onehold forthe samepairsofv alues?
Page 13
??DOBJECT RECOGNITIONVIAICSP
Inthissection?we present theapplicationof ICSPtotheproblemof ?Dobjectrecog?
nitioninimages? The?D ob jectrecognitionisaveryhard open problemincomputer
vision sinceitssolutionrequiresaformalde?nitionof di?eren telemen ts? the object
model? the visualfeaturesused intherecognition process ?i?e?the visualelements ex?
tractedfrom theimages?? and??nally?theinferenceengineexploited in therecognition
pro cess?An e?cientc hoice ofvisualfeatures fordescribing andrecognizing?D objects
?especially CAD?madeob jects?consists inextractingfromimages thesetofvisible
surfaces?andthen computingsome geometric andtop ologicalrelationsbetweenthem
????
Usingsurfacesasfeatures?amodel
M
ofa ?Dob ject canbeformalizedasapair
?X?R
?where
X
??X
?
?????X
n
?isthesetof surfacesand
R
isasetof unary? binary or
n?aryrelationsb etw eensurfaces? Usingonly unary and binary relations?a ?Dob ject
model canbedescrib edby meansofa constraintgraph whereobjectsurfaces areno des
?variables? andrelationsb etweensurfacesarearcs?Agraphrepresentationof object
mo delshasb eenused in manydi?erentcon texts of ?Dshapes?????andextended to
the?Dscenerecognition????????
Unaryconstraints refertovisualpropertiesofsinglesurfaces? suchascolours?tex?
ture?orshape?Binaryconstrain tsare geometricrelationshipsb etw eensurfaces? the
mostcommonlyused istheadjacency constraintthat holdsiftwo surfaceshaveacom?
mon edge?Graphs exploitingsurfaceadjacency areknown asSurfaceAdjacency Graph
?SA G???? ??
Therefore?solving the?Dob ject recognitionproblemasa standardCSP canbe
seen asaresultofthe followingprocess?
?
extractfromthe imageall surfaces
S
?
?????S
m
?
?
createv ariabledomains
D
?
?????D
n
inthe de?nedgraph?
?
?ndanassignmentofsurfacestovariablessatisfying allunaryandbinarycon?
straints?
Intherecognitionsystem?weusedthe shapepropriety
is
axyz?X
i
?
?where
xyz
is
a speci?c shape?rectangle? trap ezium?L
shape etc?asunaryconstrain t?theadjacency
constraint
touch?X
i
?X
j
??andgeometric relations
isnormal?X
i
?X
j
??
isparallel?X
i
?X
j
?
asbinaryconstrain ts?Notethat allthese constraintsareindependentfrom rotations?
translationsanddistance b etw eentheobserverandtheobject?
Figure??ashowsanL?shaped?Dobjectand its corresp ondingSAG? Figure??b
containsthe modelwritten inECL
i
PS
e
?Note that?fromthecomputer visionpointof
view?thisapproachshoulddealwithsomeopen questions??rst theconsideredSAG
referstoasingleobject view?thusamodelforeachview oran objectcentered model
isrequired?Second?themodelshouldhavethepossibilityofhandlinguncertainand
imprecisedataduetoover?segmentation?miss?segmentationorunder?segmentation
problems thatnormally arisewhenreal imageare processed?Both problemshavebeen
facedinthecompletevisionsystem extensivelydescrib edin??? ?? but arenot considered
inthispap erforbrevity?
TheCSP?basedapproach forobject recognition su?ersofa severe limitationinterm
ofe?ciencywhen applied ofrealvisionapplications? theknowledgeacquisition step?
Page 14
that isthe surfaceextraction? isav erytimeconsumingstepw orkingonrange images
?i?e?? imageswhereeac h pixelcon tains theinformationofthe?Ddistancefrom the
p ointofview??
The timespent forextracting allsurfaces mayresulttoo hea vyin many robotic
orindustrialapplications? for exampleFigure? represen tsa ?D scenewith several
partiallyoverlappedobjects?theproblemisto?ndanobjectthatsatis?es the de?ned
mo del?the L?shap edofFigure ?? i?e?theblock in themiddle?leftpart of theimage??
The?rstframeofFigure?showsthe original image?the secondframethe segmented
image where all surfacesaresegmen tedandthe thirdtheextracted informationon edges
andcornersinorder toc heckadjacency and geometricconstraints?Theseoperations
take some min utesonstandardworkstations?
Figure??
Thebloc k?image
In this framework theadoption ofan interactiveversion ofCSP? asde?ned inprevi?
oussections allowedastrongperformance improv ement?TheICSP?based recognition
executesthe followingsteps?
?
theconstraintsolv er interactswiththelo w?level imageprocessing systemand
calls for aninitialunconstrainedsurface ?S???
?
the in teractingconstraintpropagation starts?andwheneveravariabledomainis
not kno wn?newvariablevaluessatisfyingconstrain ts arerequested?
Inparticular?insteadofusinga function
get
allsurfaces??
at theb eginningof
CSP? the ICSP?duringthepro cedure propagate partiallykno wnexecutesthe function
get surfacetouching?Xi?
sothatonly adjacent surfacesare directlycomputed? This
allo wsto improve thesp eedof theICSP sincevariabledomains turn tobe smaller and?
more importan t?thevisual systemfo cusses onlyon signi?cantimageparts? Th us? the
guided interactionprev en ts thelow?levelsystemof acquiring manyuseless surfaces?
Sev eral testsareperformed onimages ofa speci?cdata?base ofrangeimages?we
havecreateda mo di?edversion of theWashingtonStateUniversitydatabase????by
assemblingsev eralimagesinorder toobtainnewimagescontainingmanydi?erent
possiblyoverlapped objects?Resultsareobviouslydatadependentsincethespeed
achiev edin?ndingthe?rstobjectsatisfyingthemodeldep endsonthep ositionof the
objectinthe imageinfunction ofthe searchdirection?
Page 15
ImageICSPCSPSpeedup
BLOCK?????x????? ????????????????
BLOCK?????x????????????????? ????
BLOCK?????x?????????? ??????????
BLOCK?????x????????? ??????????
BLOCK?????x?????????????????????
BLOCK?????x???? ????? ??????????
BLOCK?????x?????????? ?????? ????
BLOCK?????x?????????????????????
BLOCK?????x????????????????????
Table?? Computationalresults ??no L?shapeobjectintheimage?
ResultsinT able?refertoadatabase of?images and describethetime ?in seconds?
sp ent for extractingthe?rst object withL?shap e? theCSP andtheICSP approaches
arecompared? Other resultsare described in ?????In some images?marked with?? the
objectw asnotpresent and insuch casesallimagesmustbe exploredand allsurfaces
m ustbe computedinb othtechniques? Whenallsurfacesareextracted intheimage
the gainusing an in teractiveapproachisnot particularly high?Iftheobject isnot
presen t? thelabeling procedure tries toinstan tiatethe?rstv ariablesto allthep ossible
surfaces?sothelo w?levelsystemmustscanthe wholeimage?Anyway? surfacedata
are recorded?soa featureis never extractedt wice? Moreov er?eachv ariabledomain
isprobably smaller?duringcomputation? thaninthe CSPcase?in factitcontains
onlyvaluesconsistentwith interactiveconstrain ts?For thisreason? even when the
object is notpresent?wecan haveagain inperformance? In othercases?such as in
imageBLOCK
??theICSPsu?ersalittlep enalty?duetothehighernumberofc hoice
pointsadded?On theotherhand?whentheobjectispresent? greatimpro vementcan
beobtained? sinceextraction stopsasso on asonesolution is found? Thisextraction
avoidance?com binedwiththefactthat domainsarek eptsmaller? dealsa speedup of
? to ?? Thisperformanceimprovement prov es the e?cacyofan interactiveapproach
ofrecognition thatallo ws ofusingandmanagingallthe actualknowledge requiredby
the process?
Theadv antage of theapproach istwofold?from the visualsystem viewpoin t?on
av erage?weacquireasmallern umber offeaturessinceweguidetheextractionby
means ofconstraints?F romtheconstraint solverp ointofview?wework withsmaller
domainsthus increasinge?ciency? Notealsothatthiskindofacquisitioncorresponds
toana?prioriapplicationofconsistencytechniquessincethe visualsystempro vides
onlyconsistentvalues with constrain ts?
? CONCLUSIONANDFUTUREWORK
Wehavepresen tedamodelforinteractiveCSPwhichcanbeusedwhendataon the
domainisnotcompletely knownattheb eginningofthecomputation? but canbe
dynamicallyacquired ondemandbya low lev elsensorsystem?More important? itis
usedin orderto guidethe searchbygenerating newconstraintsateach step?
Wehaveimplemented theframeworkbyextendingthe ECL
i
PS
e
CLP?FD? library
andapplied it toa case studyofobjectrecognitionand identi?cationinavision
system? Objects are modeledby meansof constraintsandconstraintpropagationis the
Download full-text