Conference PaperPDF Available

A delay prediction approach for teleoperation over the Internet

Authors:
Tissaphern Mirfakhrai at Eaton
Tissaphern Mirfakhrai
Shahram Payandeh at Simon Fraser University
Shahram Payandeh
Download full-text PDF
Read full-text
Download citation

Abstract and Figures

Based on the notion of wave variables and the idea of wave-integral transmission, a new method is suggested to match the system parameters with changes in the delay. An autoregressive model is used as a predictor to forecast the future values of the delay. The predictions are used with a lookup table to tune the gain with which the wave integrals are to be fed to the system. This gain scheduling and tuning improves the system performance and decreases the mismatch between forces and velocities at the master and slave sides.
Figures - uploaded by Shahram Payandeh
Author content
All figure content in this area was uploaded by Shahram Payandeh
Content may be subject to copyright.
Content uploaded by Shahram Payandeh
Author content
All content in this area was uploaded by Shahram Payandeh on Aug 19, 2014
Content may be subject to copyright.
1
ADelayPredictionApproachforTeleoperation
over theInternet
TissaphernMirfakhraiand ShahramPayandeh
Abstract|Basedonthenotionofwavevariables,and
theideaofwave-integraltransmission,anewmethodis
suggestedtomatchthesystemparameterswithchanges
inthedelay.Anautoregressivemodel isusedasapre-
dictortoforecast thefuturevaluesofthedelay.The
predictionsareusedwithalook-up tabletotunethe
gainwithwhichthewaveintegralsaretobefedto
thesystem.Thisgainschedulingand tuningimproves
thesystemperformanceand decreasesthemismatch
betweenforcesand velocitiesat themasterand slave
sides.
Keywords|Teleoperation,Force-feedback,Internet.
I.Introduction
THEconceptofteleoperation beenaround fora
while[3]. Designand developmentoftelerobotic
systemswerefurthermotivatedwithintroductionofa
newubiquitousmeansofcommunication, i.e.theIn-
ternet.Alongwithgeneralproblemscaused bytime
delays,usingtheInternetasthemeansofcommuni-
cationcanresultinmoredi±cultiesinteleoperation
becauseofthevariabletimedelay.Thevariabletime
delayscanresultinwrongcommand order,decreased
control latencyand evenlostcommand packets.
Ithasbeenshown byAndersonand Spong[2]thatif
theforce and velocitysignalsaretransmittedasthey
arefromthemastersidetotheslavesideinthepres-
ence of force feedback,thesystemwill becomeunsta-
ble evenwiththesmallestdelays.Theysuggesteda
modi¯edcontrol law,whichwasbasedonthetransfer
functionofapasivetransmissionline,and guaranteed
thesystemstability.
BasedonAnderson'sideas,Niemeyerand others[5]
proposedthe conceptofWave variables.Usingthe
wavevariableswill resultinatwo-portcommunica-
tionlinethatispassivefromoutside.Thereforethe
stabilityofthesystemduringteleoperationisguaran-
teed.
Thewavevariablesareintroduced byrede¯ningthe
systempower°ow.LetFbetheforce appliedto a
systemand _xbethevelocityofmotioninthatpartof
thesystem.Usually,thepower°owisde¯nedasthe
productofane®ortand °owpairsuchas:
TissaphernMirfakhraiand ShahramPayandeh,Experimen-
talRoboticsLaboratory,SimonFraserUniversity,Burn-
abyBC,CanadaV5A1S6,Email: tmirfakh@cs.sfu.ca,
shahram@cs.sfu.ca.
P=_xTF(1)
Tointroduce thewavevariablesuand v,weassume
twostreamsofpowermovinginoppositedirectionsin
thesystem.Thismeanswehavedividedthepower
°owto a streamgoingfromthemastersidetowards
thesalveside(positivedirection) (1
2uTu)and astream
goingfromslavetomaster(1
2vTv).Inotherwords,we
assumethat themastersideisalwaysgivingenergyto
thesystem.Thisgivenenergymightbecomenegative
atinstants,meaningthat thepowertransferisactually
fromslavetomaster.
Therefore,tointroduce thewavevariables,we can
rede¯nethepower°owas:
P=1
2uTu¡1
2vTv(2)
Weassumethatuand varelinearcombinationsof
_xand F.Usingequation1 and 2,thewavevariables
can be calculatedintermsof_xand Fas:
u(t)=b_x(t)+F(t)
p2b
v(t)=b_x(t)¡F(t)
p2b(3)
Thetuningparameterbactsasaweightfunction
and changestherelativemagnitudeof_xand Fwith
respect toeachother.Anypairoftheabovevari-
ables(u;v;_x;F),can beselectedasinputoroutput
variablestothewave-transformerblock.Itispossible
toshowthat the communicationlineispassiveifthe
energystoredintheoutgoingwaveofvislimitedto
the energyofincomingwaveofu.Now, ifthesewave
variablesaretransmittedinsteadoftheactualforce or
velocitysignals,theoverall systemwould bepassive
and noinstabilitywill happen.
Figure1showstheoverall blockdiagramofthe
wave-basedteleoperationsystem.Theoperatorap-
pliesaforce Fhtothemastermanipulatorwiththe
mass ofMm.ThemastercontrollerisaPDcontroller
withthedampingfactorofBmand springfactorof
Km.Afeedbackforce isgenerated bythemastercon-
trollerbasedonthefeedbackwavevariables.Thisin-
putforce alongwiththefeedbackforce,will determine
thevelocityofmotionofthemastermanipulator.The
masterforce (Fm)and mastervelocity(_xm)arethen
codedintothewavevariableumusingequation3.This
2
wavevariableisthentransmittedthroughthe commu-
nicationlinetotheslaveside.
Ontheslavesidethewavevariableisreceivedafter
goingthroughthetimedelay.Thenthewavevariable
isdecodedagain usingequation3.Thedesiredslave
velocity(_xsd)isgiventotheslave controller,which
generatestherequiredforce ontheslavemanipulator.
ThiscontrollerisaPDcontrollerwiththedamping
factorofBsand thespringconstantofKs.Theslave
manipulatorhasamass ofMs,and isreceivingthe
force Fethroughcontactwiththe environment.
Fig.1.Theoverall systemblock diagram.
Inthispaperwesuggestusing a delaypredictorto
predict thefuturevalueofdelay.Introducing a delay
predictorinawave-basedteleoperationsystemcanim-
provetheperformance throughfeedbackingtheinte-
gralsofthewavevariables.Bytuningthevalueofthe
feedbackgainfortheintegrals,someofthelostprop-
ertiesofthesignalcan berestoredtheoverall delay
can bereducedforforce-precisionorvelocity-precision
tasks.
Thepaperisorganizedasfollows:SectionII ex-
plainstheidea oftransmittingthewavevariableinte-
gralsalongwiththeoriginalwavevariablestoimprove
systemperformance.Theintegrals somehowrepresent
the energyofthewavevariables.Soifthe energydif-
ference isfed backtothesystemwithyacertaingain,
theperformance can beimproved.Tuningthisgain
accordingtothemagnitudeofthetimedelayhelps
theperformance,butrequiresaforecastknowledgeof
thedelay value.SectionIII discussesoursuggested
methodtopredict thefuturevaluesofthedelay.In
sectionIV,weusethesepredictionstotunethesys-
temthroughteleoperation.SectionVincludes some
concludingremarks.
II.WaveIntegralTransmission
Depending onthetaskin hand,and whetherwe
want tomatchforcesorvelocities,therewill always
besomemismatch betweenforcesand velocitiesat the
masterand slavesides.Evenifthemismatch between
thevelocitiesis small, the errorwill causethepositions
ofthetwomanipulatorstodriftapartgraduallyas
timepasses.
Oneperformance improvementstrategyistransmit-
tingwaveintegrals.Asthewavevariablesthemselves
carryinformationabout theforcesand velocitiesof
thetwomanipulators,theirintegralswill encodemo-
mentumand position.Thustransmittingthewave
integralsmeans someinformationabout theposition
ofthemanipulators.Althoughtheoreticallythepo-
sitioninformationcan bedecodedfromtheoriginal
wavevariablesbyintegratingthevelocities,theposi-
tionerrorwill increasegraduallybecauseofnumerical
integrationmethodsused.Transmittingthewavein-
tegralswill solvethisproblem.
Niemeyerand Slotine[6], hadsuggestedtotransmit
thewaveintegralsU(t)=Rudtand E(t)=Ru2dt
alongwiththewavevariablesthemselvestoimprove
theteleoperation performance.E(t)isthewave en-
ergy,meaningthe energythatisbeingtransmitted by
thewavestreaminacertain direction.
Theysuggested using a ¯lterto obtainacorrected
versionofus,denotedasuout,fromtheintegralof
thewavevariableU(t)=Rudtand theintegralofthe
squareofthewavevariableE(t)=Ru2dt(Figure2).
Fig.2.The con¯gurationofthe communicationlinkusinga
reconstruction ¯lteras suggestedbyNiemeyerandSlotine
The¯lterisde¯nedas
uout(t)=(®E(t)
U(t)ifU(t)6=0
0ifU(t)=0(4)
where®isaconstant thatcanmodifytheshape
ofthe¯lter response.The¯lterimpulseresponsesfor
®=1,®=1:5 and ®=2 areplottedin ¯gure3.
Theidea oftransmittingwaveintegralswasfurther
investigated byYokokohji, Imaida,and Yoshikawain
[7]. Theysuggestedthat theintegralofthemasterside
wavevariable(um)should be calculated numerically
up tothetimeofeachtransmissionand thenshould
besentalongtotheslaveside.
At theslaveside,theintegralofthereceivedwave
variable^us(t)iscalculatedsimilarlyand isthencom-
paredwiththenumericvalueoftheintegralreceived
3
Fig.3.Theimpulseresponseofthereconstruction ¯ltersug-
gestedbyNiemeyerandSlotinefor®=1,®=1:5and®=2
fromthemasterside.Thedi®erence (¢),can be cal-
culatedas:
¢(t)=Zum(t)¡Z^us(t)
¢can beinterpretedasameasureofchangeinen-
ergyofthesignal, and will befed backto^ustore-
storethelostenergy(Figure4).¢(t)isaddedtothe
receivedwavevariable,^us(t),ampli¯ed byacertain
gain,hereaftercalled¾.Finallywe calculatetheslave
wavevariableas:
us(t)=^us(t)¡¾¢(t)=us(t)¡¾(Zum(t)¡Z^us(t))
(5)
Fig.4.The con¯gurationofthe communicationlinkusinga
reconstruction ¯lteras suggestedbyNiemeyerandSlotine
Yokokohjietal. [7]had noticedtheimportance of
thegainofthisfeedback(¾)and hadmentionedthat
thevalueofthegainshould be chosensuchthat the
systemiswell compensated,butat thesametimenot
tosensitivetodisturbances.However,theyhad not
suggestedapracticalwaytotunethisgain.
Oursimulations showthat theoptimalvalueofthis
gain,to obtainthesmallesterrorvariesasthemag-
nitudeofthetimedelaychanges.Thismeansthatat
every valueofT(t),the errorinforce/velocitycan be
minimizedifthemagnitudeof¾ischosenaccordingly.
Thereforeifthefuturetimedelayisknown,alook-
up table can beformedforthepurposeof¯nding a
¾valuethatresultsinthesmallestmismatch between
theforcesorvelocitiesat themasterand slavesides.
Thisideawill bediscussedin detail insectionIV
Thequestion nowishowto obtainanaprioriknowl-
edgeoftherandomtimedelayoftheInternet.Inother
wordsweneedtopredict thefuturevalueofthedelay
tobeabletotunethegaintoitsoptimalvalue.The
nextsectionaddressesthemethodofprediction.
III.ModelingandPrediction
TheInternet time-delaysarerandomprocesses.A
randomprocess isarandomvariablethevalueofwhich
alsodependsontime.Althoughrandomprocessesare
randomand thereforeunpredictable,modelscan be
created basedontheirpastvaluestopredict thefu-
turevaluesoftheprocess withsome error.Themost
widelyusedmodelsfor randomprocessesaretheMov-
ingAverage(MA)Modeland theAutoregressive(AR)
model. Themoving averagemodelactsbasicallyas
alowpass ¯lterontheincomingsignal, and predicts
thefuturevalueofthesignaltobetheaverageofits
pastvalues.However,averagingdisregardsall ofthe
highlystochasticbehavioroftheprocess.ThusanMA
model isnotadequateforahighlystochasticprocess
suchastheInternet timedelay.Therefore,theautore-
gressivemodelhastobeusedforthedelayprediction
application.Forastudyofalternativedelayprediction
approaches see [10].
A.Modelderivation
Anautoregressive(AR)model isamodelthatre-
latesthevalueofthesignalat timentoitsvaluesat
timen¡1through
x[n] =
N
X
i=1
aix[n¡i] + w[n](6)
wherex[n] isthesignalwewant tomodeland Nis
theorderofourARmodel, whichcan beselectedas
adesign parameter[4]. w[n] iswhitenoisewithauto-
correlation
E[w[n]w¤[n]]=¾2
w±[n](7)
where¤indicatescomplexconjugate,¾2
wisthevari-
ance ofthenoiseand theoperatorE[] takesthe ex-
pectedvalueoftheparameterappearingbetweenits
brackets.
4
Itisdesiredto¯nd the coe±cientsa1;a2;a3; :::aN
suchthatifthelastNvaluesofxareknown(x[n¡
1];x[n¡2]; :::x[n¡N]),thenextvalueofx(x[n])can be
predicted usingequation6withthesmallesterrorcom-
paringtoitsactualmeasuredvalue.Itcan beshown
[4]that theoptimalvaluesofaisto givethesmallest
errorbetweenthepredictedvalueand theactualmea-
suredvalueofxarethesolutionsoftheYule-Walker
matrixequation[1].
WemeasuredthedelayfromtheSimonFraserUni-
versity(SFU),BC,Canadatoanumberofdi®erent
destinationsaround theworld. [1]. Theresultsillus-
tratedinthepresentpaperweremeasuredfromthe
communicationlinktotheDataCommunicationIn-
corporation(DCI)inIran,whichliesalmostonthe
exactoppositesideoftheglobetoSFU.Asthedelays
oftheInternetareknowntohaveaquasi-periodicpro-
¯leoveraweek[9], we createourmodelbasedonthe
delay valuesobserved duringaweek.Thismodel is
then usedtopredict thebehaviorofthedelayover
theweekaftertheobservations.Delaysaremeasured
severaltimeseverydayand apiecewisemodel iscon-
structedfromall setsofresults.
Thesamplingtimeswerearbitrarilyselectedtobe
12:00AM,4:00AM,8:00AM,12:00PM,4:00PMand
8:00PMeveryday.Theseparticulartimeswerese-
lectedsuchthat theyfall in di®erent timesoftheday,
representingverylow,veryhighand averagedatatraf-
¯cload.Starting ateachoftheabovetimes,thedelay
wasmeasured24 timesusingthepingutility.The
experimentswere continuedfor2weeks.
Foreachexperiment,theRootMean Square(RMS)
errorwascalculatedfordi®erentmodelorders,topick
anorderthatgivesusthesmallesterror.The errors
versusthemodelorderNareplottedin ¯gures5.a
and 5.b.The errorseemstobedecreasingwhenthe
modelorderNisincreased.To avoid unnecessarily
large computation,N=24 wasacceptedasbeinglarge
enoughforourpurpose.
Figure6shows6setsofour24 modelparameters,
calculatedat thesixmeasurement timesduringacer-
tain day.
B.Predictingdelays
Once themodel iscreated basedonthedatafrom
measurements,thevaluesobtainedfromthismodeling
process areusedtopredict thebehaviorofthesystem
duringthesecond week.Thisprocess ofmeasuring,
modeling and predictingwill continueduringteleoper-
ation.In ¯gure7,thepredictionsare comparedwith
theactualvaluesofthedelayobtainedthroughmea-
surement.Thepredictionand therealmeasuredval-
uesareless then%20 di®erent,exceptat timesof fatal
crashesatoneofthemasterand slave computers.
Fig.5.ModelparametersforconnectionwiththeDCIsystem
on Saturday,plotted at the6measurement times.
Fig.6.ModelparametersforconnectionwiththeDCIsystem
on Saturday,plotted at the6measurement times.
Fig.7.Predicted delays(dashed)comparedtotheactualvalues
ofdelays(solid)oftheDCIseverduringthesecondweek.
IV.WaveIntegralGainScheduling
Inourscenario,theoperatorappliesaforce inthe
formofasquarepulsewiththemagnitudeof2Nto
5
amastermanipulatorwith unitmass (Mm=1Kg).
Thismanipulatoriscontrolled byaPDcontrollerwith
sti®ness factorof0.5N=m(P)and dampingfactorof
1N=m=s(D).Thiscontrollermovesthemasterma-
nipulatorbyrespondingtothevelocitydictated by
theoperatorand thevelocitythatisobtained bythe
feedbacksystemthroughthenetwork.
Themastersidewavetransformerconvertstheforce
and velocitysignalstoum,whichistobetransmitted
throughthe communicationline.Thewaveimpedance
bischosentobe equalto 1,to give equalweight to
theforce and velocitiesatbothmasterand slavesides.
Thedelayofthe communicationline(T(t)) isassumed
tobevariableand will bediscussedfurtherbelow.
Theslavesidesystemisassumedtobe exactly
similartothemasterside.Sotheslavesidewave
transformerworkswithawaveimpedance alsoequal
to 2;theslavemanipulatoralsohasaunitmass
(Ms=1Kg)and the controllergainsare exactlythe
sameasthoseofthemasterside.Theslavema-
nipulatorisassumedtobeinteractingwithanen-
vironment,consisting ofaspring(k=1N/m)and a
damper(B=0.5N/m/s).Themanipulatorispushing
thespring againstasolidwall, whilebeingheld bya
damper.
ThedelayoftheslaveÃmastercommunicationlink
is supposedtobe equaltothatofthemaster!slave
link, i.e.T(t).Wealso assumethat thereisnoscaling
betweenthemasterand slavesides.
Letusde¯netheforce errortobethemaximum
mismatch betweentheforcesat themastersideand
theslaveside.
Ferr =maxfFm(t)¡Fs(t¡T(t))g(8)
Thevelocityerroris similarlyde¯nedasthemax-
imum mismatch betweenthevelocitiesat themaster
sideand theslaveside.
_xer r =maxf_xm(t)¡_xs(t¡T(t))g(9)
Oursimulationstudies showthatforevery valueof
T,thereisavalueof¾tominimize the error.
Figure8showsthebehavioroftheabovementioned
errorswithchangesinTand ¾.Itcan beseenthat
forevery valueofT,thevalueof¾can be chosensuch
that the errorisminimized.ForeachTin ¯gure8,the
topsurface ofoneofthe errorbarsispainted blackto
showtheoptimalvalueof¾.Inthesame¯gure,the
hatchedsquares showsomelimitations,wherethesys-
tembecomesunstableduetothe choice ofanimproper
¾.At thosevaluesof¾theamountofenergyfedto
thesystemforcompensationismorethan necessary
and thatmakesthesystemnon-passiveand unstable.
Whenoperating,thedelaypredictorwill estimate
thefuturevalueofthedelay.Thisestimatedvalueof
Fig.8.Force andvelocityerrorsatdi®erentestimated delays
fordi®erentvaluesof¾.
Tisthen used bythegainschedulertosearchina
look-up tablelike¯gure8to¯nd theoptimalvalue
ofthegain¾.Figure9,comparesthesystemperfor-
mance withtheoptimalvalueof¾withthesystem
performance with nocompensation.Itisvisiblethat
thesalveforce followsthedesiredforce more closely
whencompensationisadded.
Itshould benotedthat thevalueof¾tominimize
theforce error, isnotalwaysthesameasthevalueof¾
tominimize thevelocityerror.Thiscan beseenfrom
¯gure8,wheretheblackpainted pathontheforce
errorplot(8.b), isdi®erentfromtheblackpainted path
onthevelocityerrorplot(8.a).
Figure10 showsthevelocitiesonthemasterand
slavesides,withand withoutacompensationgainthat
minimizestheforce error.Itcan beseenthatalthough
applyingthe compensationgainof¾=0:5hasthebest
compensatione®ectontheforce, itactuallyincreases
thedi®erence between_xmand _xs.
Also atlargedelays, ifthe¾value exceedsacertain
valuethesystemgoesunstable.Thehatchedsquares
in ¯gure8representsuchcases.Thisinstabilitycan
be explained bynotingthataddingthe¾-path,we
areviolatingthe conditionsofpassivityas statedin
Niemeyer'scalculationsin[8]. Thusifweadd too
muchfeedbacktothesystem,wemightend up with
positivefeedback,whichtendstodestabilize thesys-
tem.
6
Fig.9.ForcesatE(T(t))=500 msandE(T(t))=700 ms
Fig.10. _xm,_xswithoutcompensationand_xswith¾=0:5
thatminimizestheforce error.
Therefore,whenthefuturevalueofthedelayispre-
dicted,we can usealook-up tablesimilartotheblack
pathin ¯gure8toretunethesystem.Thiswaychoos-
ingtheoptimalvalueof¾canminimize ourerroron
force orvelocity,orboth.When dealingwithlarge
delaysthevalueof¾hastobeset tozero,to guaranty
thepassivityofthesystemand tokeepstableopera-
tion.Forexample, ifthepredictedvalueforT(t)is
1second, itisoptimaltoset¾=0:5basedon ¯gure
8.aifaminimalforce errorisintended.Also¾=0:2
will resultinminimalvelocityerrorasisvisiblefrom
¯gure8.b.
V.Conclusions
Amethodis suggestedtodecreasethe errorbetween
forces/velocitiesofthemasterand slavemanipulator.
Thefuturevalueofthevariabletimedelayispredicted
basedonitspastvaluesusing anAutoregressive(AR)
model. Alook-up tableisthen usedto¯nd avalueof
thewaveintegralfeed-forwardgain¾tominimize the
errorbetweentheforce/velocitiesofthemasterand
slavemanipulators.Bytuningthegaintothisopti-
malvalue,someofthelostpropertiesofthesignalcan
berestored,the errorbetweenthemasterand slave
forces/velocitiesisdecreasedand theoverall teleoper-
ationsystemperformance isimproved.
References
[1]TissaphernMirfakhraiand ShahramPayandeh,AModel
forTime-DelaysforTeleoperationovertheInternet,Proc.
IEEE SymposiumofComputationalIntelligence inRobotics
and Automation(CIRA2001),pp.236-241.
[2]RobertJ.Andersonand MarkW.Spong,"Bilateralcon-
trolofTeleoperatorswithtimedelay,IEEE transactionson
AutomaticControl, 34(5):494-501,1989.
[3]ThomasB.Sheridan,Space teleoperationthroughtimedelay
reviewand prognosis,IEEE TransactionsonRoboticsand
Automation,9(5):592-606,October1993.
[4]HenryStark,John W.Woods,Probability,Random
Processes,andEstimationTheoryforEngineers,January
1994,Prentice-Hall Canada,ISBN0137287917.
[5]GÄunterNiemeyerand Jean-JacquesE.Slotine,StableAdap-
tive TeleoperationIEEE JournalofOceanicEngineering,
Vol.16,No.1,pp.152-162,1991.
[6]GunterNiemeyerand Jean-JacquesE.Slotine,Towards
Force-Re°ectingTeleoperationOvertheInternet,Proc.
IEEE Int.Conf.onRoboticsand Automation,pp.1909-
1915,1998.
[7]YasuyoshiYokokohji, TakashiImaida,and Tsuneo
Yoshikawa,BilateralControlwithEnergyBalance Moni-
toringUnderTime-VaryingCommunicationDelay,Proc.
IEEE Int.Conf.onRoboticsand Automation,pp.2684-
2689,2000.
[8]GÄunterDieterNiemeyer,Usingwave variablesintimede-
layedforce re°ectingteleoperation,PHDthesis,MIT,MA.
[9]RobertoOboeand PaoloFiorini, ADesignandControl
EnvironmentforInternet-BasedTelerobotics,TheInter-
nationaljournalofRoboticsResearch,vol.17,No.4,April
1998,pp.433-449.
[10]Q.P.Wang,D.L.Tan,NingXiand Y.C.Wang,TheControl
OrientedQoS:AnalysisandPrediction,Proc.ofthe2001
IEEE InternationalConference onRobotics&Automation,
pp.1897-1902.
... In terms of safety alerts for signal delay, current delay modelling methods [10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25] predominantly focus on specialised network types and transmission links, such as Software-Defined Networking (SDN) networks, Generalised Stochastic Petri Netbased Voice over IP (GSPN-based VoIP) networks, and Time-Sensitive Networking (TSN) networks. However, these methods generally lack universality and do not address realtime applications. ...
... The calculation for the proportional coefficient γ i between discrete time points is the same as in Equation (14). Then, we can determine the parameter P x (t i -t delayi ) before the delay interval, expressed as Equation 18. ...
View
... For example, Munir and Book included predictive techniques in the wave variable method to handle the time-varying delays encountered on the Internet [14]- [17]. Other work on controlling teleoperators experiencing variable time delay includes [3], [13], [20], [26]. ...
... Substituting in the wave variable relationships (4) and (5) and rearranging [1] one obtains (13) and where ...
View
... Furthermore, predictive techniques are introduced to model and mitigate the random time delay with fuzzy adaptive control methods by Lu et al. (2017) and Mirfakhrai and Payandeh (2002) where they used an autoregressive model. Ye et al. (2002) and Shen et al. (2019) used nonlinear time series analysis to understand the delay behavior and estimate time delay. ...
... Several early approaches for delay mitigation used time series prediction. Mirfakhrai and Payandeh (2002) employed an autoregressive model to forecast future values of time delay. The predictions were used with a look-up table to tune the system by adjusting the gain and decreasing the mismatch between forces and velocities at the master and slave sides. ...
A Brief Survey of Telerobotic Time Delay Mitigation
Article
Full-text available
  • Dec 2020
There is a substantial number of telerobotics and teleoperation applications ranging from space operations, ground/aerial robotics, drive-by-wire systems to medical interventions. Major obstacles for such applications include latency, channel corruptions, and bandwidth which limit teleoperation efficacy. This survey reviews the time delay problem in teleoperation systems. We briefly review different solutions from early approaches which consist of control-theory-based models and user interface designs and focus on newer approaches developed since 2014. Future solutions to the time delay problem will likely be hybrid solutions which include modeling of user intent, prediction of robot movements, and time delay prediction all potentially using time series prediction methods. Hence, we examine methods that are primarily based on time series prediction. Recent prediction approaches take advantage of advances in nonlinear statistical models as well as machine learning and neural network techniques. We review Recurrent Neural Networks, Long Short-Term Memory, Sequence to Sequence, and Generative Adversarial Network models and examine each of these approaches for addressing time delay. As time delay is still an unsolved problem, we suggest some possible future research directions from information-theory-based modeling, which may lead to promising new approaches to advancing the field.
View
... Nuño et al. [42] reviewed work on passivity-based controllers for non-linear bilateral teleoperation. Using autoregressive models to predict future control inputs from available past inputs, Lu et al. [43] and Mirfakhrai and Payandeh [44] developed predictive techniques to model and mitigate the random time delays due to internet-based teleoperation with fuzzy adaptive control methods. While these control-based approaches are robust to small time delays and improve stability, there are some drawbacks. ...
Time Delay Mitigation in Aerial Telerobotic Operations Using Heterogeneous Stereo-Vision Systems
Article
Full-text available
  • Jun 2023
This paper investigates the use of a heterogeneous stereo-vision system to mitigate the effects of time delays in a drone-based visual interface presented to a human operator. Time delays in the display for a telerobotic interface refer to the time difference between the operator’s input action and the corresponding visible outcome. In human/machine interfaces, time delays can arise due to computation, telecommunication, and mechanical limitations. These delays can degrade the performance of the human/machine system. A heterogeneous stereo-vision predictive algorithm is presented that can reduce the negative effects of time delays in the operator’s display. The heterogeneous stereo-vision system consists of an omnidirectional camera and a pan/tilt/zoom camera. Two predictive display setups were developed that modify the delayed video imagery that would otherwise be presented to the operator in a way that provides an almost immediate visual response to the operator’s control actions. The usability of the system is determined through human performance testing with and without the predictive algorithms. The results indicate that the predictive algorithm allows more efficient, accurate, and user-friendly operation.
View
... It is well known that the unknown time delay can be regarded as known in advance through time-delay prediction [30]. Sparse multivariate linear regression (SMLR) [30,31], autoregressive (AR) models [32,33], neural networks (NNs) [34], and maximum entropy methods [35] have been proposed for time-delay prediction. However, many of these methods have low precision when the time delay exhibits large fluctuations. ...
View
... According to the application the user can vary the scale factor at will [2]. The third factor, prediction, constitutes a great help when teleoperation relies on visual feedback in systems with significant delays [3] In robotics, having a human in the loop implies an improvement as the person provides the robots lacking intelligence. However, humans fail in that they have some inherent limitations, they introduce imprecision to the system, being either related to movement, as the physiological tremor, or related to the misinterpretation of some relevant perceptive information such as distance or time. ...
View
... The proposed scheme combines the predictive EBA [23] and the projection method [5] in order to utilize couple of major advantages of both methods: (1) the predictive EBA provides robust stability regardless of time delays as well as time-varying nonlinear dynamics of slave and environment. In order to implement variable time delays, there are two options either predict the time delay [24] or regulate the time delay through one of the many buffering techniques such as [25]. Since the SP can cancel regulated constant delay in a better way as compared to the variable one, the time-varying delays are implicitly considered by making the time-varying delay to be a large constant time delay in this paper. ...
Projected predictive Energy-Bounding Approach for multiple degree-of-freedom haptic teleoperation
Article
  • Oct 2016
This paper presents a geometrically and dynamically transparent projected predictive Energy-Bounding Approach (EBA) for multiple degree-of-freedom (m-DOF) haptic teleoperation with time delays. The straightforward extension of a single-DOF teleoperation to m-DOF teleoperation suffers from a bad force distortion problem that becomes worse with the increase in communication time delays between master and slave sites. Due to this, geometry perception of remote m-DOF objects at master site is severely deteriorated. To solve this problem, a projected predictive m-DOF EBA is proposed by combining the m-DOF Predictive EBA and the projection method that can compensate the force distortion problem. The proposed approach also includes steps to get stably the information about the contact locations (geometry) of m-DOF remote objects without geometry detection sensors. In order to validate the proposed approach, some experiments performing surface contour-following are conducted using two Phantoms: (i) 2-DOF object (bobbin), and (ii) 3-DOF semi-spherical object in the presence of large time delays. © 2016 Institute of Control, Robotics and Systems and The Korean Institute of Electrical Engineers and Springer-Verlag Berlin Heidelberg
View
Network Delay Forecast and Master–Slave Consistency Enhancement for Remote Surgical Robots
Article
  • Feb 2025
  • INT J MED ROBOT COMP
Background The inevitable network delay can directly impact the process of remote surgeries and affect the master–slave motion consistency, and sudden changes in delay can compromise surgical safety. Methods Firstly, real‐time calibration of unidirectional network delays is performed. Subsequently, the network delay is forecasted with a real‐time training parallel recurrent neural network for safety warnings, and the real‐time forecast of slave manipulator position is performed to enhanced the master–slave motion consistency. Finally, the forecast accuracy across multiple scales is assessed to provide feedback. Results The programme can operate on standard computers at distances of at least 630 km. Our forecast method meets the real‐time requirement, demonstrates strong generalisation capabilities and reduces the impact of network delay on master–slave motion consistency to approximately 20%–80% of its original level. Conclusions The proposed forecast method enables real‐time delay forecast for remote surgeries, reducing the impact of delay on master–slave motion consistency.
View
View
Eine vorausschauende Anzeige zur Teleoperation von Straßenfahrzeugen
Thesis
Full-text available
  • Mar 2016
Die Teleoperation von Straßenfahrzeugen bietet die Möglichkeit, die Rentabilität von Car-Sharing-Systemen zu erhöhen und die Ladestellenproblematik für Elektrofahrzeuge zu entschärfen. Dabei wird das Fahrzeug vom Fahrer über eine Mobilfunkverbindung ferngesteuert. Die Umgebung des Fahrzeugs wird dafür mit Hilfe von Kameras erfasst. Um die zu übertragende Datenmenge zu verringern, werden die Bilddaten mit dem Videokompressionsalgorithmus H.264 komprimiert, bevor sie an den Fahrer übertragen werden. Auf Basis der empfangenen Bilddaten tätigt dieser über Pedale und Lenkrad die primären Steuereingaben, die zurück an das Fahrzeug übertragen werden. Bei der Verarbeitung und der Übertragung der Video- und Steuerdaten treten Zeitverzögerungen auf. Diese bringen eine zusätzliche Totzeit in den Fahrer-Fahrzeug-Regelkreis mit ein und beeinträchtigen damit die Fahrzeugsteuerung stark. Diese Arbeit stellt ein System vor, mit dessen Hilfe die negativen Auswirkungen der Zeitverzögerungen verringert werden können. Die Bewegungen des eigenen Fahrzeugs und die Bewegungen der anderen Verkehrsteilnehmer werden dafür vorausberechnet und als vorausschauende Anzeige dem Videobild überlagert. Nachdem das vorausberechnete Fahrzeug unmittelbar auf die Steuereingaben reagiert, erhält der Fahrer dadurch unverzögert Rückmeldung über deren Auswirkungen. Für die Vorausberechnung der eigenen Fahrzeugposition wird ein Einspurmodell verwendet, das in vielfacher Echtzeit ausgeführt wird. Die Anzeige dieser Position erfolgt durch Einzeichnung der Fahrspuren der Vorderräder und eines stehenden dreidimensionalen Rahmens auf Höhe der Fahrzeugfront. Dadurch wird zum einen der Bewegungszustand des Fahrzeugs erkennbar. Zum anderen kann der Fahrer anhand des Rahmens die Fahrzeugabmessungen erfassen, ohne dass große Teile des Bilds durch Einzeichnung verdeckt werden. Die aktuellen Positionen und Bewegungsvektoren anderer Verkehrsteilnehmer werden mittels Laserscanner erfasst. Über eine Extrapolation der aktuellen Bewegung werden daraus die zukünftigen Positionen vorausberechnet. Diese werden durch virtuell auf den Boden projizierte Rechtecke bzw. Ellipsen visualisiert. In mehreren Probandenstudien wird gezeigt, dass mit diesem System die mentale Beanspruchung des Fahrers stark verringert und die Fahrleistungen gesteigert werden können. Dies gilt sowohl für die Umsetzung in der Simulation, als auch am Realfahrzeug.
View
The Control Oriented QoS: analysis and prediction.
Conference Paper
Full-text available
  • Jan 2001
Now a number of advanced control systems take advantage of networks such as Ethernet, Fieldbus and Internet as their communication channel. Transmission time delay of these networks may lead unstable problems to system. In this paper, the prediction of network delay using neural network is presented first. Then the analysis of the control oriented quality of service (CO-QoS) specially with the Internet-based teleoperation is considered and fuzzy inference is used to construct a quantification of CO-QoS. At last, the simulation results show that this method provides a practical way for the judgment of the network's characteristics.
View
Probability, Random Processes, and Estimation Theory for Engineers
Book
Full-text available
  • Jan 1994
1. (Note: Each chapter concludes with Problems and References)2. Introduction to Probability3. Random Variables4. Functions of Random Variables5. Averages6. Vector Random Variables7. Estimation and Decision Theory I8. Random Sequences9. Random Processes10. Mean-Square Calculus11. Stationary Processes and Sequences12. Estimation Theory II13. Appendix: Review of Relevant Mathematics14. Index15. Information for Using the Program Disk
View
Space Teleoperation Through Time Delay: Review and Prognosis
Article
Full-text available
  • Nov 1993
A 30-year history of research on dealing with the effects of time delay in the control loop on human teleoperation in space is reviewed. Experiments on the effects of delay on human performance are discussed, along with demonstrations of predictive displays to help the human overcome the delay. Supervisory control is shown to offer a variety of options, from switching to local impedance control upon contact with the environment to higher-level local automation. Wave transformation techniques to ameliorate the effects of delay are also described. Space teleoperations have tended to deal with the problem of time delay by avoiding it and not attempting to teleoperate from the ground. It is suggested that the US space effort might have gotten further ahead and at a lower cost with a greater is commitment to space teleoperation from the ground through the delay. Predictive display works well for free positioning. Local impedance control is recommended for control in contact with the environment, possibly accompanied by wave transformation techniques. Higher level supervisory control is always an option for sufficiently predictable tasks and will continue to improve with better sensors and task models
View
View
Using Wave Variables in Time Delayed Force Reflecting Teleoperation
Article
  • Aug 2005
Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Aeronautics and Astronautics, 1996. Includes bibliographical references (p. 195-198). by Günter Niemeyer. Ph.D.
View
A model for time-delays for teleoperation over the Internet
Conference Paper
  • Feb 2001
The stochastic nature of the communication delay in teleoperation systems makes it hard to predict the behavior of such systems. Although achieving a general-purpose model for the Internet is not easy, if possible at all, stochastic models for specific cases enable us design predictors, which can help in stabilizing the teleoperation systems. In this paper we have developed an initial notion to create an autoregressive model for delays over the Internet. By probing the system behavior at different times of the day for a period of time, the general behavior of the delay can be comprehended and so partially predicted. Introducing a predictor in the teleoperation system can improve the system behavior through for example gain scheduling at the master controller.
View
Bilateral control with energy balance monitoring under time-varying communication delay
Conference Paper
  • Feb 2000
In previous work, we (1999) have proposed a control scheme based on wave variables. Unlike other methods, the proposed method could minimize the performance degradation due to the fluctuation of time delay. With this method however the system may potentially generate infinite energy in some special situations and the system stability could not be ensured rigorously. In this paper, we modify the method by introducing an energy input/output balance monitoring mechanism, which limits the energy that the system can generate. We conducted some simulation studies with a one DOF system. Simulation results show the validity of the proposed scheme
View
Bilateral Control of Teleoperators with Time Delay
Conference Paper
  • Sep 1988
A control law for teleoperators is presented which overcomes the instability caused by time delay. By using passivity and scattering theory, a criterion is developed which shows why existing bilateral control systems are unstable for certain environments, and why the proposed bilateral control law is stable for any environment and any time delay. The control law has been implemented on a single-axis force-reflecting hand controller, and preliminary results are shown. To keep the presentation clear, a single-degree-of-freedom (DOF) linear time-invariant (LTI) teleoperator system is discussed. Nevertheless, results can be extended, without loss of generality, to an n-DOF nonlinear teleoperation system
View
Towards force-reflecting teleoperation over the Internet
Conference Paper
  • Jun 1998
This paper extends earlier results on stable force reflecting teleoperation in the presence of significant time-delays to the case, frequent in practice, where the transmission delays are themselves varying with time in an unpredictable fashion. It shows that stability can be preserved through the systematic use of specially designed wave-variable filters. The resulting performance of the teleoperation system is illustrated in simulations, and is consistent with reasonable expectations on “ideal” behavior. The results may provide a practical tool for implementing force-reflecting teleoperation over the Internet
View
Stable Adaptive Teleoperation
Article
  • Feb 1991
A study is made of how the existence of transmission time delays affects the application of advanced robot control schemes to effective force-reflecting telerobotic systems. This application best exploits the presence of the human operator while making full use of available robot control technology and computing power. A physically motivated, passivity-based formalism is used to provide energy conservation and stability guarantees in the presence of transmission delays. The notion of wave variable is utilized to characterize time-delay systems and leads to a configuration for force-reflecting teleoperation. The effectiveness of the approach is demonstrated experimentally. Within the same framework, an adaptive tracking controller is incorporated for the control of the remote robotic system and can be used to simplify, transform, or enhance the remote dynamics perceived by the operator.< >
View