Content uploaded by Liao Wu
Author content
All content in this area was uploaded by Liao Wu on Aug 14, 2023
Content may be subject to copyright.
1
Optimal Vision-Based Orientation Steering Control for a 3D Printed
Dexterous Snake-Like Manipulator to Assist Teleoperation
Andrew Razjigaev, Ajay K. Pandey, David Howard, Jonathan Roberts Senior Member IEEE,
Anjali Jaiprakash, Ross Crawford, and Liao Wu Member IEEE
Abstract—Endoscopic cameras attached to miniaturized snake-
like continuum robotic arms can improve dexterity, accessibility,
and visibility in minimally invasive surgical tasks. This steerable
camera can expand the field of view and enhance the surgical
experience with additional degrees of freedom. However, it also
creates more control options that complicate human-machine
interaction. This challenge presents an opportunity to develop
novel human-machine control strategies using visual sensing
to complement the dexterous actuation of a steerable surgical
manipulator. This study presents a semi-autonomous controller
to assist teleoperation by steering a camera such that it keeps an
arbitrary target in the centre of the field of view, thus assisting
in surveying different orientations about the target with image-
based information. Two controllers are proposed using classical
image-based visual servoing techniques and optimal visual pre-
dictive control techniques. These techniques are simulated and
validated on our robot SnakeRaven: a 3D-printed patient-specific
end-effector attached to the RAVEN II surgical platform. Both
systems, most notably the visual predictive approach, operated
successfully with robustness to a lack of information about the
target. A video demonstrating the main results of this paper can
be found via https://youtu.be/fiUM9qYdl1U and code via GitHub.
Index Terms—Surgical Robotics, Teleoperation, Human-
Computer Interaction, Visual Servoing
I. INTRODUCTION
With the use of rigid arthroscopes and tools in knee
arthroscopy, it is surveyed that 98.9% of surgeons report diffi-
culty in visualising some sections of the knee [1]. In addition,
surgeons have reported frustration with the rigid design of
current surgical equipment and utilising multiple tools in a
confined space [2]. The limitation of a rigid arthroscope is its
lack of accessibility and dexterity in the confined space of the
knee joint and, therefore, a limited field of view (FOV) for
observing features inside the knee [3].
Continuum robotic manipulators present a promising solu-
tion for novel arthroscopic instruments. They resemble bio-
logical systems such as elephant trunks, tentacles, and snakes
This work was supported in part by the Australian Research Council
under Grant DP210100879, in part by CSIRO’s Active Integrated Matter
Future Science Platform ID TB04, and in part by the Queensland University of
Technology (QUT) Centre for Biomedical Technologies Industry Engagement
Grant 2021
A. Razjigaev, A. K. Pandey, J. Roberts and A. Jaiprakash are with the
School of Electrical Engineering and Robotics at the QUT Centre for Robotics,
Brisbane, Australia. andrew.razjigaev@csiro.au
D. Howard is with the Cyber Physical Systems Program, Data61,
CSIRO. david.howard@csiro.au
R. Crawford is with the Department of Orthopaedic
Surgery, Prince Charles Hospital, Chermside QLD, Australia.
r.crawford@qut.edu.au
L. Wu (corresponding) is with the School of Mechanical and Manu-
facturing Engineering, University of New South Wales, Sydney, Australia.
liao.wu@unsw.edu.au/dr.liao.wu@ieee.org
BA
C
LED
muC103A
Image Sensor
3D printed
rolling joints
Distal Module Proximal Module
Distal Tendons
Proximal Tendons Proximal Tilt
Distal Pan
direction
Distal Tilt
direction
LED
Image Sensor
D
Proximal Pan
18mm 4mm
Fig. 1: A. SnakeRaven - a 3D printed steerable arthroscope
attached to the RAVEN II telerobotic system. B. the end-
effector and its illumination inside a phantom knee C. a close-
up view of the end-effector and camera sensor D. a diagram
of the end-effector structure with the integrated camera, distal
components in red and proximal components in blue
that can be steered to traverse confined spaces, reach difficult-
to-access surgical sites, and complete tasks with dexterity [4].
With 3D printing, these robots can become patient-specific,
maximising the benefits of robot-assisted surgery and improv-
ing patient outcomes by minimising the risk of unintended
damage to the patient [5]. However, an ongoing challenge
with continuum robots is the development of efficient human-
machine interactions in teleoperated control with image guid-
ance [4].
In our previous work [6], we developed the design and
closed-loop kinematic control algorithms to teleoperate a 3D-
printed patient-specific continuum robotic manipulator called
2
1 2 3
𝒗𝒄
𝝎𝒄
𝒗𝒄
𝑻𝒄𝒂𝒎 𝑻𝒄𝒂𝒎
𝑇 𝑇
Camera
Field of
view Central
Field of
view
A
𝑥𝑥
𝑧𝑧
𝒛𝟏
𝑦
𝒗𝒄
𝑻𝒄𝒂𝒎
𝑇
𝑥
𝑧
𝑦
𝒛𝟐𝒛𝟑
𝑦
+
o
1
o
o
o
𝒔𝒊= 𝑥 , 𝑦
(𝑥 , 𝑦)
𝑟
2
𝒆+
o
o
o
o
+
3
o
o
o
o
B
𝑻𝒕𝒂𝒓𝒈𝒆𝒕
𝑧
𝑥𝑦
𝒛𝟏= 𝒛𝟐
𝒛𝟑
ℎ
𝜑
ℎ, ℎ
Unit
Sphere
13
Camera Tip Orientation Trajectory on Service Sphere
Orientation
Trajectory
2
CImage
Plane
Key points
Desired
Region
(Central Field
of View)
𝒆𝟏+
o
o
o
o
𝑥
𝑦
(𝑥, 𝑦)
(𝑥, 𝑦)
(𝑥 , 𝑦)
𝑟
𝑟
𝑟
𝜃
𝜃
𝜑, 𝜑
𝒆𝟐
D
+
𝒔𝒊(𝑘)
o
o
𝒔𝒊(𝑘 + 1)
𝒅𝒔
𝒅𝒔∗𝒔𝒊
∗(𝑘)
𝒔𝒊
∗(𝑘 + 1)
𝑑𝜃
𝒅𝒔𝒏
𝒅𝒔𝒕
𝒅𝒔𝒕
∗𝒅𝒔𝒏
∗
𝑟
𝑑𝑡 → 0
𝒅𝒔𝒏
∗→ 0
∴ 𝒅𝒔∗≈ 𝒅𝒔𝒕
∗
∴ 𝒅𝒔∗
𝒅𝒔𝒕
≈𝒓
𝒓𝒊
𝑟(𝑘 + 1)
Velocity Relationship
between 𝒔and 𝒔∗
As 𝒔𝒊
∗is always the closest point
from 𝒔𝒊to the desired region
Fig. 2: Visualisation of the control problem where the human operator can control the camera translation vcwhile the vision-
based controller regulates the orientation ωcsuch that the central field of view (FOV) is fixated at a target and thus can observe
it from different orientations. Subfig. A. shows the starting point at state 1where the square target is inside the central FOV
represented as a red circle in the image view below it, state 2where the camera translation in the y-axis vccauses the target to
drift from the circle causing an error vector e, and state 3where a controller responds with an ωcaction that re-positions the
target back into the central FOV. Subfig. B. shows how the camera orientations from motions 1to 3are mapped as a trajectory
(in red) on the surface of a service (unit) sphere relative to the target. This mapping takes z-axis vectors z1,z2, and z3and
converts them into longitude (in angle) and latitude (in height) coordinates ϕand h. Subfig. C. shows how the feature error
eis measured from a keypoint (xi, yi)deviating from the desired circular region criteria (red circle) centred on (xc, yc)with
radius rwhich are values chosen by the operator for the application. Subfig. D. shows how the feature velocity ˙s and desired
set-point velocity ˙s∗are related to each other in this problem
SnakeRaven. The snake-like end-effector consists of a prox-
imal and distal set of pan and tilt rolling joints actuated
by tendons pulled by the 4 instrument degrees of freedom
(DOF) of the RAVEN II which has 7 DOF in total (See
Fig. 1). We optimized the patient-specific design parameters
of the end-effector to achieve high dexterity considering the
constraints of the knee anatomy from a patient scan [7].
Compared to a rigid instrument, SnakeRaven has the dexterity
and accessibility to increase the FOV of an arthroscope and
can observe features inside the knee using its additional DOF.
This dexterous camera can collect multiple views of the same
features, which is beneficial for many applications, including
visual inspection, scanning and 3D reconstruction in occluded
task spaces inside the knee [8]. In this study, we integrate
a vision system for SnakeRaven, as shown in Fig. 1, and
design vision-based controllers that assist the operator during
teleoperation for inspection or scanning tasks by steering the
camera’s orientation such that it keeps visual targets within
the centre of the FOV (see Fig. 2).
A. Related Work
Applications of visual servoing in assisting the control of
continuum robots have been particularly prevalent in overcom-
ing modelling problems and satisfying task constraints. In the
work of concentric tube robots (CTRs) we have seen model-
free visual servoing approaches to track a surgical target [9],
singularity avoidance [10], and eye-in-hand visual servoing
for accurate motion [11]. Model Predictive Control (MPC),
also referred to as Visual Predictive Control (VPC) in visual
servoing literature, has been utilized to avoid singular configu-
rations [12]. In tendon-driven robots, VPC is used to optimize
control performance, satisfy constraints, account for depth
[13], and achieve robustness [14]. Other teleoperation assistive
controllers have been applied to image stabilisation [15],
minimising the motion in confined spaces [16], respecting
physical limits and remote centre of motion (RCM) constraints
[17], robust motion planning [18], and for scanning surfaces
[19]. However, no task-oriented approach is applicable to the
problem presented in Fig. 2 where the controller assists the
operator in steering the tool orientation about surgical targets.
B. Contribution
Continuum snake-like manipulators are mechanically com-
plex devices that have not received sufficient attention on
vision-based control strategies with human-machine interac-
tion, particularly in a task concerning orientation steering
during teleoperation. This gap presents an opportunity to de-
velop novel human-machine interactions using visual sensing
to complement the dexterous actuation of a steerable surgical
manipulator. We present a novel approach to applying vision-
based control strategies to assist in the teleoperation of a
dexterous continuum robot for an application that requires
3
orientation steering about an arbitrary visual target. We present
two methods to solve this control problem using the classical
image-based visual servoing formulation and a visual pre-
dictive optimal control strategy. Both of these methods were
simulated, implemented, and validated experimentally.
II. PRO BL EM FO RM UL ATIO N
As visualized in Fig. 2a, the control problem is that a
camera, embedded inside the tip of SnakeRaven, must keep
a target in the centre of the FOV during teleoperation. Ini-
tially, SnakeRaven is deployed as a steerable arthroscope for
follow-the-leader deployment inside the knee where the human
operator teleoperates the robot from the camera coordinate
frame Tcam and navigates it towards an arbitrary target. Once
the operator selects the target to be in the central FOV as
in Fig. 2a state 1, they activate the assistive visual servo
controller and the camera orientation (with angular velocity
ωc) is automatically regulated to keep the target within the
central FOV. The operator can then survey the target through
teleoperation by applying a translational camera velocity vc
which would consequently cause the target to drift away from
the central FOV as in Fig. 2a state 2. The desired response
from the controller is to steer the camera with angular velocity
ωcsuch that the target returns to the central FOV as in Fig.
2a state 3. Essentially, the controller acts as an auto-aiming
function that keeps the target within the centre of the FOV
during teleoperation. Performing this task manually can be a
difficult manoeuvre with human-machine interfaces as it is
limited to the skill of the operator. Automation can assist
the operator by allowing seamless exploration of orientations
about a target. The resulting orientations reached by the
camera can then be measured on a service sphere in longitude
(in angle) and latitude (in height) coordinates (ϕ, h)as in Fig.
2b. The control task objectives, constraints, and assumptions
are summarized as follows.
•Objectives: The main objective is to have a semi-
autonomous controller based on vision sensing where the
camera orientation (through ωc) is automatically adjusted
to fixate on a specific target set by the operator during
teleoperation where the operator controls the camera
translation vc
•Constraints: This vision system must add to the design,
control and telemanipulation system of SnakeRaven from
our previous work [6] by using an embedded camera at
the tip (see Fig. 1d)
•Assumptions: The target the operator selects in our
arthroscopic application can be efficiently detected with
the state-of-the-art keypoint feature detection algorithms,
such as those recommended by Marmol et al. [20].
III. PROP OS ED VI SUAL-SERVO GUIDED TELEOPERATION
The proposed solution to this control problem is to cascade
the SnakeRaven controller from [6] with an image-based
visual servo (IBVS) controller on the outer loop, as shown
in the control diagram in Fig. 3. Cascading the controllers
allows compatibility between the two modules which operate
at different rates and preserves the fast responsiveness of the
SnakeRaven controller to disturbances.
1) Inner Control Loop: In our previous work [6], we
established a teleoperation system communicating between the
human operator at the console and the RAVEN II robot with
the Robot Operating System (ROS) in Linux programmed in
C++. The SnakeRaven Controller is a ROS node subscribed
to joint measurements jpos of the 7 DOF of the RAVEN II
control node and handles the kinematic modelling to capture
the complex geometries in SnakeRaven (refer to [6] for more
details). It determines the snake-like end-effector configuration
q(also 7 DOF) by estimation and minimising the observed
error through inverting the actuation Jacobian Ja
dˆq =Ja
−1(ˆq)(jpos −ˆ
j(ˆq)) (1)
The update of dˆq converges the estimate ˆq to the solution
of q. This result is then used to solve the forward kinematics
which is a chain of rigid transformations to get the end-
effector pose Tend. The input is the desired end-effector pose
Tdes which sends a velocity dxto the inverse kinematics
block which utilizes a weighted damped least-squares solver
to update q:
dq=JT(JJT+λ2
mI)−1Wdx(2)
where Jis the manipulator Jacobian, λmis a constant damping
factor used to minimize the joints to safely move through
singular configurations, Iis the identity matrix, and Wis
a diagonal matrix of weights used to prioritize orientation
errors in the controller. The joint velocity dqis then mapped
to the change in end-effector tendons which determines the
necessary motor velocity djpos to publish to the RAVEN
II control node such that it will maintain the position and
orientation of the end-effector. Motor velocities are published
at a fast rate of 1000Hz but the inner loop process actually
calculates at a rate of approximately 536Hz. The system
maintains safety and stability by limiting the joint velocity to
a slow magnitude during human teleoperation and restricting
the motion to respect joint limits in the robot workspace.
2) Outer Control Loop: As an extension into this study, an
endoscopic camera is rigidly glued to the tip of the robot as
shown in Fig. 1. A ROS node captures camera images Iby
serial communication at a rate of 30Hz and publishes them
as well as the camera calibration data to the visual servo
controller node. A feature tracker detects point features s,
using OpenCV, and compares them to a region criterion that
describes the desired location for the features as s∗. The error
between them is computed as eand is fed into the visual servo
block that computes the desired angular velocity of the camera
ωc. The camera translation vcis the teleoperation command
that comes from the console interface by the operator which
runs at 10Hz and is limited to a low value to maintain safety.
These vectors form the overall desired camera motion that is
published to the SnakeRaven controller and fed into the eye-in-
hand block, which maps the velocity to the end-effector frame.
This result is used to determine the desired end-effecter pose
Tdes for the input to the inner loop. The operator has full
control to activate the visual servo controller and deactivate
it, switching back to providing the angular velocity ωcby
teleoperation.
4
RAVEN II Robot
Image Sensor
SnakeRaven Controller
Visual Servo Controller
Human Operator
Region Criteria
Console Input
Visual Servo
Eye in Hand Inverse Kinematics
Forward Kinematics
Joint to Actuation
Actuation to joint
Joint Update
Joint Encoder
Camera
Feature Tracker
s∗e
vcTdes dxdqdjpos
qjpos
I
ωc
−
Tend
−
s
Fig. 3: Control diagram for Visual-Servo Assisted Teleoperation cascading the visual servo controller in the outer loop with
the SnakeRaven kinematic controller in the inner loop [6]
A. Feature Tracker: Detection and Matching
The function of the feature tracker is to identify the visual
features of the arbitrary target and then match them correctly
to the corresponding features of the target reference frame in
real-time. From a study on multi-view feature-based matching
in arthroscopic applications [20], Binary Robust Invariant
Scalable Keypoint (BRISK) [21] features offer the best trade-
off between accuracy and computation time, which was why
it was selected as the keypoint detector for recognising the
target in real-time for this study.
When the operator starts the visual servo control, BRISK
features are detected within the desired circular region of the
initial image and are set as the reference points of the target.
For every image afterwards, BRISK features are detected
and matched to the reference image by distance-based brute
force. Outliers are filtered with random sample consensus
(RANSAC). As the target is arbitrary, the number of features
used by the controller varies depending on the number of
features matched to the initial set of features in the reference
image. BRISK feature detection and matching algorithms are
openly available in OpenCV.
B. Region-reaching Criteria
Since the arbitrary target is composed of a variable amount
of keypoints, it is advantageous to define the objective of the
control problem as a region. This is because regions generalize
the task (as we cannot determine the desired view), reduces
reliance on correct feature correspondence and can result in
faster and less motion, consequently consuming less energy to
converge than conventional setpoint control [22].
As the operator teleoperates the camera with vc, a control
action ωcwould be applied that maintains a fixated view of the
target by keeping it closer to the centre of the FOV. Therefore,
the controller should uniformly draw any keypoint deviation
back towards the centre of the FOV. In this notion, a circular
region criterion is a logical choice where the centre point
(xc, yc)is the centre of the FOV and the radius rdetermines
its size which is selected by the operator for the application.
A visualisation of the image plane and this circular region is
shown in Fig. 2c.
1) Defining the Task function: The aim of this visual servo
control task is to minimize a task function e, a vertical stack
of error vectors ei, that draws the ith keypoint si= [xi, yi]T
towards s∗
i= [x∗
i, y∗
i]Tas defined by:
ei=si(mi(t),a)−s∗
i=xi−x∗
i
yi−y∗
i(3)
where siare points projected from 3D (Xi, Yi, Zi)to the
image plane as measurements mi(t)for the pixel coordinates
(ui, vi)and ais the camera intrinsic parameters from camera
calibration including the principal point (u0, v0), the focal
lengths fx, fyand pixel dimensions ρu, ρvin mm. We have,
(xi=Xi/Zi= (ui−u0)ρu/fx
yi=Yi/Zi= (vi−v0)ρv/fy
(4)
When considering region-reaching regulation tasks, the de-
sired setpoint, s∗
i= [x∗
i, y∗
i]T, could be any point inside the
desired region. To check if a keypoint si= [xi, yi]Tis within
the desired circular region, the radial distance from the centre
rican be measured with:
ri=p(xi−xc)2+ (yi−yc)2(5)
If the radial distance is greater than the circle radius, ri> r,
then the keypoint is outside the region, and a control action
should be applied to minimize eiand drive the point iback
to the circle. Otherwize, if the radial distance is less than the
circle radius, ri≤r, the keypoint is inside the desired circular
region satisfying the task objective, i.e. ei= [0,0]T.
As the desired setpoint s∗
i= [x∗
i, y∗
i]Tcould be any point
inside the desired region, this provides flexibility in selecting
it. The best choice is to pick the closest point which is on the
circular boundary raligned by angle θi= arctan yi−yc
xi−xc
ei=xi−x∗
i
yi−y∗
i=ricos θi−rcos θi
risin θi−rsin θi= (1 −r
ri
)xi−xc
yi−yc
(6)
This error vector eidescribes the shortest distance required
to return point iinto the circle.
C. Orientation Partitioned Image-Based Visual Servoing
In the classical IBVS formulation [23], the feature velocity
˙s = [ ˙x, ˙y]Tis affected by the six DOF camera velocity input
uc= [vx, vy, vz, ωx, ωy, ωz]Tthrough the interaction matrix
Lswhich is the time derivative of the projection equations (4):
5
˙s =Lsuc(7)
The interaction matrix Lsis a 2×6matrix that depends on
the feature point s= [x, y]Tand the depth of the feature Z:
Ls=−1
Z0x
Zxy −(1 + x2)y
0−1
Z
y
Z1 + y2−xy −x(8)
Given that the teleoperation command provides the trans-
lation component of the desired camera motion, vc=
[vx, vy, vz]T, the controller only needs to compute the camera
angular velocity ωc= [ωx, ωy, ωz]T. The interaction matrix
can be partitioned into translational and rotational parts [24]:
˙s =LvLωvc
ωc=Lvvc+Lωωc(9)
Where Lvis the translational partition and Lωis the orienta-
tion partition of the interaction matrix:
Lv=−1
Z0x
Z
0−1
Z
y
Z(10)
Lω=xy −(1 + x2)y
1 + y2−xy −x(11)
The strategy by Chaumette et al. [23] is to minimize the
task function eby ensuring a decoupled decrease in the error
˙e =−λewhere λis a constant. However, as s∗updates,
˙e resembles the feature trajectory problem ˙e =˙s −˙s∗[24].
Regarding the relationship between ˙s and ˙s∗in Eq. 6, any
component of ˙s acting normal to the circle (˙sn) does not affect
˙s∗. Therefore ˙en=˙snin the normal direction. Considering
the tangential component of ˙s, (i.e. ˙st), ˙s∗is proportionally
smaller by the ratio r/rias shown in Fig. 2d, therefore:
˙et=˙st−˙s∗≈˙st−r
ri
˙st= (1 −r
ri
)˙st(12)
As ri> r,˙s will always have a positive association to ˙e,
therefore, if we set the feature velocity as ˙s =−λe, it would
still ensure a decoupled decrease in the error e:
˙et≈ −λ(1 −r
ri
)et≡ −λe(13)
The angular velocity of the camera can then be solved by
substituting ˙s =−λeinto Eq. 9 and rearranging to get this
control law by taking the pseudo inverse of the orientation
partition of the interaction matrix:
ωc=−L+
ω(λe+Lvvc)(14)
Since the task involves a slow and infrequent teleoperation
input running at a rate of 10Hz compared to the 30Hz of
the vision system), vc= [0,0,0]Tfor most iterations in
between teleoperation commands making the computation of
Lvwith the estimate of depth Zunnecessary at times. We
propose assuming a simplification to the control law where
the controller has no knowledge of vc:
ωc≈ −λL+
ωe(15)
where a larger gain for λcan adequately compensate for the
lack of knowledge at executing the task as would be tested in
simulations in section IV. With either control law in Eq. 14 or
Eq. 15, the desired angular velocity ωccan be computed. With
this result, the eye-in-hand block of Fig. 3 uses the hand-eye
transformation Tend
cam to map the velocities vcand ωcto the
end-effector coordinate frame for the input of the inner control
loop.
D. Optimal Visual Predictive Control Approach
The previous method involving the classical IBVS approach
may have limitations such as difficulty in handling constraints
and unsatisfactory behaviour of the camera displacement [25].
As an alternative method of solving ωc, we propose to use
a predictive control strategy with Visual Predictive Control
(VPC) [25] where a model is used to predict the motion of
the features over a period, optimize a cost function given
constraints and solve the best input to provide to the system.
The intended advantage of this approach is to anticipate when
features would drift away from the desired region and apply
an early control action. Since the teleoperation input vccauses
features to drift, predicting camera motion can be adequately
solved with a simplifying local model of the outer loop.
Taking the orientation input from the controller as the
variable u=ωc, the discrete set of future control inputs can
be defined as Uwhere kis the current iteration of control
input, and Npis the limit to the extent of future predictions
known as the prediction horizon:
U=u(k),u(k+ 1), ..., u(k+Np−1)(16)
Considering the input sequence Uand assuming that the
current teleoperation input vc(k)and the depth Zare constant
throughout the prediction horizon, the motion of feature point
scan be predicted by discretisation of Eq. 9 using a sample
time ∆tat iteration k:
s(k+ 1) = s(k) + Lv(k)vc(k) + Lω(k)u(k)∆t(17)
Eq. 17 can be used to predict the motion of sfor several
iterations from k+ 1 up to k+Np. For every jth prediction
of sin this interval, the task function ecan be computed and
the following cost function can be evaluated:
C(U) =
k+Np
X
j=k+1 e(s(j))TQe(s(j)) + u(j)TRu(j)(18)
where Qand Rare positive definite symmetric weighting
diagonal matrices that penalize the control error eand the
input urespectively. Minimising the cost function C(U)with
constraints on the maximum tool-point velocities (−umax ≤
u≤umax)is a quadratic optimisation problem that can be
solved with quadratic programming techniques. Solving this
minimisation problem results in the optimal control sequence
U∗bounded by limits ¯
U
U∗= min
U∈¯
U
C(U)(19)
6
The first control action u∗(k)is selected as the angular
velocity output ωc:
u∗(k) = U∗[1] (20)
The rest of the sequence is discarded and the optimisation
problem is solved at every iteration. This can be computation-
ally expensive which is the primary drawback of this approach.
IV. SIMULATIONS OF VISION-BASE D CON TRO L
To test the assumptions made in the two proposed visual
servo control schemes, a simulation emulating the scenario
illustrated in Fig. 2a was created in MATLAB where SnakeR-
aven starts at a straight pose looking down at a 2mm wide
four-keypoint square target 7mm below it initialized in the
circular region positioned at (0,0) with ra sixth of the size of
the square image using the actual camera intrinsic parameters
(see Table I). The teleoperation input vcis only acting in the
positive y-axis i.e., vy= 3mm/s with vx= 0 and vz= 0. This
simulation runs for 100 iterations where the teleoperation will
cause features to move outside the circle so a control action
must be applied. For each of the two algorithms, Orientation-
Partitioned IBVS and Optimal VPC, the controller was tested
with either knowing the teleoperation input vcor assuming it
is vc= [0,0,0]T. Additionally, a single-point target with a
setpoint was tested as an alternative to solving this problem.
A. Orientation-Partitioned IBVS simulation results
The simulation results for Orientation-Partitioned IBVS for
knowing the teleoperation input, not knowing it, and knowing
it for a point target are in Fig. 4, 5, and 6 respectively. All
cases of the IBVS controller are stabilized by moving the
camera such that it keeps the target within or converging to
the central FOV. The most smooth trajectory was achieved
with knowledge of the teleoperation input (Fig. 4). Without
knowledge, a high λ= 10 gain adequately compensated
for the inaccuracies and only applies a control action when
features drift from the central FOV (Fig. 5). With a point
target, the camera trajectory is longer which could indicate
that less controller effort is achieved with more feature points
and region-reaching control in the task (Fig. 6).
B. Optimal Visual-Predictive simulation results
The simulation results for Optimal VPC for knowing the
teleoperation input, not knowing it, and knowing it for a point
target, are in Fig. 7, 8 and 9 respectively. In all cases, the
controller stabilizes moving the camera such that it keeps the
target within or converging to the central FOV. Determining
the ideal solution with MATLAB’s sequential quadratic pro-
gramming (SQP) algorithm with Np= 10, the controller with
knowledge was able to successfully anticipate teleoperation
and keep the target within the central FOV for the whole
simulation. Without knowledge, the target drifts but is moved
quickly back to the central FOV when required. With a point
target, the camera trajectory is long and the angular velocity
is saturated at limits which again indicates that less controller
effort is achieved with more feature points and region-reaching
control in the task. Further discussion of simulation results is
presented in section VI.
TABLE I: Visual Servo Control Experiment Parameters
Parameter Value Description
umax, vmax 384 ×384 Image size
r96 Circle radius (pixels)
xc, yc(192,192) Circle center (pixels)
ρu,v 2.344 Pixel size (µm)
fx,y (252.3,253.1) focal length (pixels)
u0, v0(207,205) Principal point (pixels)
λ10 IBVS control gain
∆t0.033 Sample time (s)
umax [1.5,1.5,1.5]TInput Limit (rad/s)
Np3 Prediction Horizon
Z30 Estimated Depth (mm)
Qdiag10 10 ... 10Feature weights
R I3Input weights
TABLE II: VPC Optimisation Computation Time
Np3 5 7 10 15
Iterations 7 8 8 9 13
Time (ms) 55 160 270 550 1500
V. EX PE RI ME NT: V ISION-BA SE D CON TRO L VALIDATION
A. Experiment Setup
The physical control task is set up in Fig. 10 to match the
simulated control task by using a small feature-rich target (an
Aruco marker) inside a red circular central FOV. This target
was placed on a featureless white background that ensured
feature-matching errors would be minimal. The algorithm
will use a variable number of features based on the BRISK
detector, but it would be limited to up to 10 points as a
memory constraint. OpenCV functions would indicate BRISK
points as white circles in the images. Feature points outside
the circle are indicated by a red line from the point to the circle
centre. An electromagnetic (EM) sensor from the NDI Aurora
tracking system is attached to the last rigid end-effector joint
of SnakeRaven to measure the orientations of the endoscopic
camera embedded into the fibre-optic channel.
As in the simulated task, the target features initially start
inside the central FOV of the endoscope. Two teleoperation
sequences are tested where the first sequence has the human
operator apply only a positive velocity in the x-axis (vx≈
3mm/s) while the second sequence is the negative velocity
(vx≈-3mm/s). This teleoperation is performed by repetitively
pressing the same keyboard button for the entire experiment
which is ’a’ for positive vxmotion and ’d’ for negative
vxmotion. The visual servo controller then must respond
to the feature motion deviating from the central FOV. Both
the Orientation Partitioned IBVS and VPC algorithms were
implemented on SnakeRaven with lacking knowledge of vcfor
IBVS and having knowledge in VPC. The experiment control
parameters are shown in Table I. Unlike the simulation, VPC
on the robot used the Levenberg Marquardt non-linear opti-
misation algorithm from the Eigen C++ library to minimize
the cost function. It also required a smaller prediction horizon
Np= 3 and reduced frequency to 15Hz to meet the real-time
7
(a) Camera pose trajectory
012345
Time [s]
-2
-1
0
1
2
3
Camera Velocity [mm/s] or [rad/s]
Camera Velocity over Time
vx
vy
vz
x
y
z
(b) Camera velocity over time
0 1 2 3 4 5
Time [s]
-40
-30
-20
-10
0
10
20
30
40
Error [Pixels]
Feature Error over Time
e1x
e1y
e2x
e2y
e3x
e3y
e4x
e4y
(c) Feature error over time
-100 0 100
x - pixels
-150
-100
-50
0
50
100
150
y - pixels
Feature Motion on Image Plane
Central FOV
Trajectory
Initial Points
Final Points
(d) Feature motion on image plane
Fig. 4: Simulation of SnakeRaven with teleoperation input: vy= 3mm/s and IBVS has knowledge of vc
(a) Camera pose trajectory
012345
Time [s]
-2
-1
0
1
2
3
Camera Velocity [mm/s] or [rad/s]
Camera Velocity over Time
vx
vy
vz
x
y
z
(b) Camera velocity over time
0 1 2 3 4 5
Time [s]
-40
-30
-20
-10
0
10
20
30
40
Error [Pixels]
Feature Error over Time
e1x
e1y
e2x
e2y
e3x
e3y
e4x
e4y
(c) Feature error over time
-100 0 100
x - pixels
-150
-100
-50
0
50
100
150
y - pixels
Feature Motion on Image Plane
Central FOV
Trajectory
Initial Points
Final Points
(d) Feature motion on image plane
Fig. 5: Simulation of SnakeRaven with teleoperation input: vy= 3mm/s and IBVS has no knowledge of vc
(a) Camera pose trajectory
012345
Time [s]
-2
-1
0
1
2
3
Camera Velocity [mm/s] or [rad/s]
Camera Velocity over Time
vx
vy
vz
x
y
z
(b) Camera velocity over time
0 1 2 3 4 5
Time [s]
-40
-30
-20
-10
0
10
20
30
40
Error [Pixels]
Feature Error over Time
e1x
e1y
(c) Feature error over time
-100 0 100
x - pixels
-150
-100
-50
0
50
100
150
y - pixels
Feature Motion on Image Plane
Central FOV
Trajectory
Initial Points
Final Points
(d) Feature motion on image plane
Fig. 6: Simulation of a point target for SnakeRaven with teleoperation input: vy= 3mm/s and IBVS has knowledge of vc
(a) Camera pose trajectory
012345
Time [s]
-2
-1
0
1
2
3
Camera Velocity [mm/s] or [rad/s]
Camera Velocity over Time
vx
vy
vz
x
y
z
(b) Camera velocity over time
0 1 2 3 4 5
Time [s]
-40
-30
-20
-10
0
10
20
30
40
Error [Pixels]
Feature Error over Time
e1x
e1y
e2x
e2y
e3x
e3y
e4x
e4y
(c) Feature error over time
-100 0 100
x - pixels
-150
-100
-50
0
50
100
150
y - pixels
Feature Motion on Image Plane
Central FOV
Trajectory
Initial Points
Final Points
(d) Feature motion on image plane
Fig. 7: Simulation of SnakeRaven with teleoperation input: vy= 3mm/s and VPC has knowledge of vc
8
(a) Camera pose trajectory
012345
Time [s]
-2
-1
0
1
2
3
Camera Velocity [mm/s] or [rad/s]
Camera Velocity over Time
vx
vy
vz
x
y
z
(b) Camera velocity over time
0 1 2 3 4 5
Time [s]
-40
-30
-20
-10
0
10
20
30
40
Error [Pixels]
Feature Error over Time
e1x
e1y
e2x
e2y
e3x
e3y
e4x
e4y
(c) Feature error over time
-100 0 100
x - pixels
-150
-100
-50
0
50
100
150
y - pixels
Feature Motion on Image Plane
Central FOV
Trajectory
Initial Points
Final Points
(d) Feature motion on image plane
Fig. 8: Simulation of SnakeRaven with teleoperation input: vy= 3mm/s and VPC has no knowledge of vc
(a) Camera pose trajectory
012345
Time [s]
-2
-1
0
1
2
3
Camera Velocity [mm/s] or [rad/s]
Camera Velocity over Time
vx
vy
vz
x
y
z
(b) Camera velocity over time
0 1 2 3 4 5
Time [s]
-40
-30
-20
-10
0
10
20
30
40
Error [Pixels]
Feature Error over Time
e1x
e1y
(c) Feature error over time
-100 0 100
x - pixels
-150
-100
-50
0
50
100
150
y - pixels
Feature Motion on Image Plane
Central FOV
Trajectory
Initial Points
Final Points
(d) Feature motion on image plane
Fig. 9: Simulation of a point target for SnakeRaven with teleoperation input: vy= 3mm/s and VPC has knowledge of vc
constraints of the 10Hz teleoperation. The computation time of
minimising the cost function in comparison to different values
of Npis presented in Table II.
B. Experimental Results
For both the Orientation Partitioned IBVS and the optimal
VPC algorithms, the physical robot was able to successfully
apply a control action ωcthat moved the features back into the
central FOV during both teleoperation sequences of vx≈ ±
3mm/s. During teleoperation in the negative x-axis with IBVS
assistance, the endoscopic camera views are shown from the
initial frame Fig. 11a to the final frame Fig. 11e where features
drift from the circle and return back into the central FOV.
OpenCV functions were used to draw the BRISK keypoints
as white circles, the central FOV as a red circle with a centre
crosshair and also show when features are outside the central
FOV by red bold lines from the keypoint to the centre as is
the case in frame Fig. 11c. The corresponding external view
of the robot motion is shown in Fig. 11f to 11j.
All the camera orientations the robot achieved about the
target have been projected to service spheres in Fig. 12a and
12b for IBVS control and in Fig. 12c and 12d for VPC in the
positive and negative teleoperation sequences. On the service
sphere, blue dots indicated the orientations the EM tracker
measured and the red dots indicated the orientations estimated
by the forward kinematics. The corresponding feature error
over time is plotted in Fig. 12e and 12f for IBVS control and
in Fig. 12g and 12h for VPC in the positive and negative tele-
SnakeRaven
EM Device
Endeffector
EM Sensor
Target Console
Endoscopic view
Desired Region
Target features
Endoscope
Fig. 10: Experiment setup of SnakeRaven, the EM system and
the vision-based guided teleoperation task
operation sequences. A longer test was conducted in Fig. 12g
where the motion ends with the robot reaching its workspace
limits and no longer being able to converge. While a shorter
test was conducted in Fig. 12h.
VI. DISCUSSION
Both of our algorithms solved our control problem by apply-
ing a control action that kept the target within or converging
to the central FOV. Simulation results demonstrate that the
two approaches are effective even with a lack of knowledge
about the teleoperation input (assuming that there is no input
vc= [0,0,0]T) and assuming a constant depth. Our simulation
results also demonstrated the benefits of using region-reaching
control and multiple feature points to describe the target
9
(a) Initial reference fea-
tures inside the circle
(b) Features just before
leaving the circle
(c) Features are outside the
circle (shown by red lines)
(d) Control action correc-
tion drives them back
(e) Features return to the
circle
(f) Initial neutral robot
pose
(g) Teleoperation towards
the negative x-axis
(h) Target moves too far
from central FOV
(i) Control action begins
corrective actuation
(j) SnakeRaven turns to
target in view
Fig. 11: Camera views of features moving out of the desired region and being corrected back into the region with the
corresponding external view of the robot motion during teleoperation in the negative x-axis (vx≈-3mm/s)
(a) IBVS control Service Sphere
orientations about the target tele-
operating a negative vx≈-3mm/s
(b) IBVS control Service Sphere
orientations about the target tele-
operating a positive vx≈3mm/s
(c) VPC control Service Sphere
orientations about the target tele-
operating a negative vx≈-3mm/s
(d) VPC control Service Sphere
orientations about the target tele-
operating a positive vx≈3mm/s
0 500 1000
Iterations
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Feature error [mm]
Feature error over time
(e) IBVS control feature error
over time while teleoperating a
negative vx≈-3mm/s
0 200 400 600 800 1000
Iterations
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Feature error [mm]
Feature error over time
(f) IBVS control feature error over
time while teleoperating a positive
vx≈3mm/s
0 500 1000 1500 2000 2500
Iterations
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Feature error [mm]
Feature error over time
(g) VPC control feature error over
time while teleoperating a nega-
tive vx≈-3mm/s
0 200 400 600
Iterations
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Feature error [mm]
Feature error over time
(h) VPC control feature error over
time while teleoperating a positive
vx≈3mm/s
Fig. 12: Validation results of the Orientation Partitioned classical IBVS and the optimal VPC algorithms reaching orientations
about the target represented as service spheres with measurements of the estimated forward kinematics and the EM sensor
measurement as well as the feature error (in x and y components) of all ten points over time in mm
10
because the camera trajectory was shorter with less control
action than the single set-point case. Our algorithms were also
successful in the validation experiment where the robot motion
is subject to disturbances caused by noisy feature tracking,
tendon backlash and hysteresis. However, we do observe the
advantages and disadvantages of the two algorithms and how
they are shown in the simulation and experiments.
Our Orientation Partitioned IBVS approach handled the
case without teleoperation knowledge in both simulation and
validation. With knowledge, the IBVS simulation produced
smooth motion converging feature errors while without knowl-
edge it provided an impulse control action every time the
features drift from the central FOV. To maintain stability, it
required a high λ= 10 in order to compensate for the lack
of teleoperation input knowledge but the resulting controller
only acts when features are outside the central FOV. When
validating the algorithm onto the robot, we found that our
IBVS approach without knowledge was feasible in practice
even though backlash would delay the control action and
cause features to drift further from the central FOV than the
simulated scenario. Despite those issues, IBVS was computa-
tionally quick, provided satisfactory camera motion and was
robust to the conditions presented without knowledge and with
disturbances in the physical system.
For the VPC approach, the predictive strategy showed
benefits in the cases with and without knowledge in simulation
and with knowledge under validation with the physical system.
With teleoperation knowledge in the simulation and an ideal
optimal solution from a long prediction horizon Np= 10, VPC
was able to eliminate feature error completely by anticipating
that features would drift from the central FOV and preventing
it with an early control action. Without knowledge of vc
in simulation, VPC could not anticipate the features drifting
but it was able to complete the task with fewer impulse
control actions than IBVS by saturating the input for short
periods. When validating the VPC approach onto the robot,
we required a shorter prediction horizon Np= 3 to meet
the real-time constraints of our teleoperation system. As a
result, we found that the anticipation behaviour was not as
prevalent particularly considering how backlash would delay
the control action and cause features to drift further from the
central FOV than the simulated scenario. We also observed
how the robot behaves when the teleoperation continues until
it reaches the workspace limit where the controller was unable
to converge the feature errors as in Fig. 12g. At such extreme
angles, occlusions can also occur reducing feature tracking
quality. Nevertheless, VPC can surpass the performance of
IBVS with the advantage of anticipating feature motion and
reducing feature error and control action but at the cost of
increased computation time which can be less responsive than
IBVS in real-time teleoperation.
Overall, the two algorithms were validated in our proposed
vision-guided teleoperation system and could successfully
steer the robot end-effector at different orientations about the
target. VPC could provide the best control performance with
increased computation time but IBVS can provide satisfactory
performance in real-time. Validation with the physical system
showed the robustness of the algorithms to a lack of knowledge
and to disturbances in feature tracking and tendon actuation.
However, performance could be improved with enhanced
feature tracking and reduced actuation delay caused by tendon
backlash and hysteresis.
VII. CONCLUSION
This study described the methodology and implementation
of an assistive vision-guided teleoperation system for SnakeR-
aven, our patient-specific snake-like manipulator attached to
the RAVEN II. The task was to regulate the orientation of
a steerable arthroscope to keep an arbitrary target within the
centre of the FOV during manual teleoperation of the cam-
era translation. We proposed two controllers for SnakeRaven
based on IBVS control and VPC. Both algorithms performed
the task successfully with robustness to a lack of knowl-
edge about the teleoperation input, depth and to disturbances
in the tendon actuation of the physical system. Validation
showed that the approaches are feasible in practice but with
different advantages and disadvantages. VPC can anticipate
the teleoperation motion and prevent feature errors but it
requires increased computation while IBVS provides adequate
performance but operates much faster. Future work would
investigate techniques to overcome the disturbances in feature
tracking and improve the accuracy of robot actuation through
appropriate compensation.
REFERENCES
[1] A. Jaiprakash, W. B. O’Callaghan, S. Whitehouse, A. Pandey, L. Wu,
J. Roberts, and R. Crawford, “Orthopaedic surgeon attitudes towards
current limitations and the potential for robotic and technological
innovation in arthroscopic surgery,” Journal of Orthopaedic Surgery,
vol. 25, 2017.
[2] J. Opie, A. Jaiprakash, B. Ploderer, M. Brereton, and J. M. Roberts,
“Towards surgical robots: Understanding interaction challenges in knee
surgery,” Proceedings of the 31st Australian Conference on Human-
Computer-Interaction, 2019.
[3] L. Wu, A. Jaiprakash, A. Pandey, D. Fontanarosa, Y. Jonmohamadi,
M. Antico, M. Strydom, A. Razjigaev, F. Sasazawa, J. Roberts, and
R. W. Crawford, “Robotic and image-guided knee arthroscopy,” 2020.
[4] J. Burgner-Kahrs, D. C. Rucker, and H. Choset, “Continuum robots for
medical applications: A survey,” IEEE Transactions on Robotics, vol. 31,
pp. 1261–1280, 2015.
[5] J. P. Desai, J. Sheng, S. Cheng, X. Wang, N. Deaton, and N. Rahman,
“Toward patient-specific 3d-printed robotic systems for surgical inter-
ventions,” IEEE Transactions on Medical Robotics and Bionics, vol. 1,
pp. 77–87, 2019.
[6] A. Razjigaev, A. K. Pandey, D. Howard, J. Roberts, and L. Wu,
“Snakeraven: Teleoperation of a 3d printed snake-like manipulator
integrated to the raven ii surgical robot,” in 2021 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS),
2021, pp. 5282–5288. [Online]. Available: http://dx.doi.org/10.1109/
IROS51168.2021.9636878
[7] ——, “End-to-end design of bespoke, dexterous snake-like surgical
robots: A case study with the raven ii,” IEEE Transactions on Robotics,
pp. 1–14, 2022. [Online]. Available: http://dx.doi.org/10.1109/TRO.
2022.3164841
[8] A. Marmol, P. Corke, and T. Peynot, “Arthroslam: Multi-sensor robust
visual localization for minimally invasive orthopedic surgery,” 2018
IEEE/RSJ International Conference on Intelligent Robots and Systems
(IROS), pp. 3882–3889, 2018.
[9] K. Wu, L. Wu, C. M. Lim, and H. Ren, “Model-free image guidance
for intelligent tubular robots with pre-clinical feasibility study: Towards
minimally invasive trans-orifice surgery,” 2015 IEEE International Con-
ference on Information and Automation, pp. 749–754, 2015.
[10] K. Wu, G. Zhu, L. Wu, W. Gao, S. Song, C. M. Lim, and H. Ren,
“Safety-enhanced model-free visual servoing for continuum tubular
robots through singularity avoidance in confined environments,” IEEE
Access, vol. 7, pp. 21 539–21 558, 2019.
11
[11] A. V. Kudryavtsev, M. T. Chikhaoui, A. Liadov, P. Rougeot, F. Spindler,
K. Rabenorosoa, J. Burgner-Kahrs, B. Tamadazte, and N. Andreff, “Eye-
in-hand visual servoing of concentric tube robots,” IEEE Robotics and
Automation Letters, vol. 3, pp. 2315–2321, 2018.
[12] M. Khadem, J. J. O’Neill, Z. Mitros, L. da Cruz, and C. Bergeles,
“Autonomous steering of concentric tube robots via nonlinear model
predictive control,” IEEE Transactions on Robotics, vol. 36, pp. 1595–
1602, 2020.
[13] M. M. H. Fallah, S. N. Ghazbi, A. Mehrkish, and F. Janabi-Sharifi,
“Depth-based visual predictive control of tendon-driven continuum
robots,” 2020 IEEE/ASME International Conference on Advanced In-
telligent Mechatronics (AIM), pp. 488–494, 2020.
[14] S. N. Ghazbi, A. Mehrkish, M. M. H. Fallah, and F. Janabi-Sharifi, “Con-
strained visual predictive control of tendon-driven continuum robots,”
Robotics Auton. Syst., vol. 145, p. 103856, 2021.
[15] G. Dwyer, F. Chadebecq, M. T. Amo, C. Bergeles, E. Maneas, V. Pawar,
E. V. Poorten, J. Deprest, S. Ourselin, P. De Coppi, T. Vercauteren, and
D. Stoyanov, “A continuum robot and control interface for surgical assist
in fetoscopic interventions,” IEEE Robotics and Automation Letters,
vol. 2, no. 3, pp. 1656–1663, 2017.
[16] X. Ma, C. Song, P. W. Y. Chiu, and Z. Li, “Autonomous flexible
endoscope for minimally invasive surgery with enhanced safety,” IEEE
Robotics and Automation Letters, vol. 4, pp. 2607–2613, 2019.
[17] X. Zhang, W. Li, P. W. Y. Chiu, and Z. Li, “A novel flexible robotic
endoscope with constrained tendon-driven continuum mechanism,” IEEE
Robotics and Automation Letters, vol. 5, pp. 1366–1372, 2020.
[18] H. Mo, R. Wei, B. Ouyang, L. Xing, Y. Shan, Y. hui Liu, and D. Sun,
“Control of a flexible continuum manipulator for laser beam steering,”
IEEE Robotics and Automation Letters, vol. 6, pp. 1074–1081, 2021.
[19] Z. Zhang, B. Rosa, O. Caravaca-Mora, P. Zanne, M. J. Gora, and
F. Nageotte, “Image-guided control of an endoscopic robot for oct path
scanning,” IEEE Robotics and Automation Letters, vol. 6, pp. 5881–
5888, 2021.
[20] A. Marmol, T. Peynot, A. P. Eriksson, A. Jaiprakash, J. Roberts,
and R. Crawford, “Evaluation of keypoint detectors and descriptors
in arthroscopic images for feature-based matching applications,” IEEE
Robotics and Automation Letters, vol. 2, pp. 2135–2142, 2017.
[21] S. Leutenegger, M. Chli, and R. Y. Siegwart, “Brisk: Binary robust
invariant scalable keypoints,” in 2011 International Conference on
Computer Vision, 2011, pp. 2548–2555.
[22] C. Cheah, D. Q. Wang, and Y. C. Sun, “Region-reaching control of
robots,” IEEE Transactions on Robotics, vol. 23, pp. 1260–1264, 2007.
[23] F. Chaumette and S. Hutchinson, “Visual servo control. i. basic ap-
proaches,” IEEE Robotics Automation Magazine, vol. 13, no. 4, pp. 82–
90, Dec 2006.
[24] ——, “Visual servo control. ii. advanced approaches [tutorial],” IEEE
Robotics Automation Magazine, vol. 14, no. 1, pp. 109–118, March
2007.
[25] G. Allibert, E. Courtial, and F. Chaumette, “Predictive control for con-
strained image-based visual servoing,” IEEE Transactions on Robotics,
vol. 26, pp. 933–939, 2010.
Andrew Razjigaev has received the Ph.D. de-
gree from the Queensland University of Technology
(QUT) in 2022 on continuum robots for medical
applications. He also received a B.S. degree in
Mechatronics Engineering from QUT in 2017. He
was affiliated with the Australian Centre for Robotic
Vision, an ARC Centre of Excellence, from 2017 to
2020. After the Ph.D., he joined the Commonwealth
Scientific and Industrial Research Organisation in
2023. His research interests include manipulator
design, vision-based control, kinematics, surgical
robotics, additive manufacturing and optimisation problems.
Ajay Pandey received the M.Sc in Physics and
M.Tech in Optoelectronics degrees in 2001 and
2003, respectively from some of the most prestigious
universities in India (University of Allahabad and
Devi Ahilya University, Indore). He was awarded a
PhD in Molecular Electronics from the University
of Angers, France in 2007.
He is currently an Associate Professor with the
School of Electrical Engineering and Robotics,
Queensland University of Technology (QUT), Bris-
bane, Australia. His research interests include exci-
tonic physics, integrated photonics, and quantum sensing, with a particular
focus on their applications in robotics and medicine.
David Howard was born in Grimsby, U.K., in 1984.
He received the B.Sc. degree in Computing and
M.Sc. degree in Cognitive Systems from the Uni-
versity of Leeds, Leeds, U.K., in 2005 and 2006, re-
spectively, and the Ph.D. degree Neuro-Evolutionary
Reinforcement Learning from the University of the
West of England, Bristol, U.K., in 2011.
He has been with the Commonwealth Scien-
tific and Industrial Research Organisation, Brisbane,
QLD, Australia, since 2013. His research interests
include embodied cognition, the reality gap, and soft
robotics.
Jonathan Roberts received a BEng Hons. in
Aerospace Systems Engineering (1991) and a PhD
in Computer Vision (1995), both from the University
of Southampton, UK. He is currently a Profes-
sor in Robotics with the Queensland University of
Technology (QUT), Brisbane, QLD, Australia, and
the Director with the Australian Cobotics Centre
(also known as the ARC Industrial Transformation
Training Centre for Collaborative Robotics in Ad-
vanced Manufacturing), Brisbane, QLD, Australia.
He is also the Technical Director with the Advanced
Robotics for Manufacturing (ARM Hub), Northgate QLD, Australia. He is a
Past President of the Australian Robotics and Automation Association Inc.,
Sydney, NSW, Australia. He is the Co-inventor of the UAV Challenge. His
main areas of research are field robotics, design robotics, and medical robotics.
Anjali Jaiprakash received the Med.Sc. degree.
She is currently a Life Sciences Scientist and an
Advance QLD Research Fellow of medical robotics
with the Australian Centre for Robotic Vision and
the Queensland University of Technology. She works
at the intersection of medicine, engineering, and
design developing medical devices for diagnosis and
surgery, including the patented light field retinal
diagnostic systems and vision-based robotic leg ma-
nipulation systems. She has experience in the fields
of orthopaedic research, optics, and design. She has
extensive research experience in the hospital and clinical setting and the ethical
conduct of research in compliance with the Australian Code for the responsible
conduct of research.
Ross Crawford received the Ph.D. degree from
Oxford University. He is currently a Professor of or-
thopaedic research with QUT. He undertakes private
clinical practice at The Prince Charles Hospital and
Holy Spirit Hospital. He has mentored over 30 Ph.D.
and M.Phil. students for the completion of their
degrees and has a wealth of experience in teaching
and leading researchers at all levels. He is currently
a member of numerous medical committees. He
has published more than 200 articles. As an Expert
Surgeon, he assists with cadaver surgery experiments
at the Medical Engineering Research Facility, QUT, Prince Charles Campus,
and brings significant knowledge of knee arthroscopy and the use of medical
robotics to this research.
Liao Wu is a Senior Lecturer with the School of
Mechanical and Manufacturing Engineering, Uni-
versity of New South Wales, Sydney, Australia. He
received his B.S. and Ph.D. degrees in mechani-
cal engineering from Tsinghua University, Beijing,
China, in 2008 and 2013, respectively. From 2014
to 2015, he was a Research Fellow at the National
University of Singapore. He then worked as a Vice-
Chancellor’s Research Fellow at Queensland Univer-
sity of Technology, Brisbane, Australia from 2016 to
2018. Between 2016 and 2020, he was affiliated with
the Australian Centre for Robotic Vision, an ARC Centre of Excellence.
He has worked on applying Lie groups theory to robotics, kinematic
modelling and calibration, etc. His current research focuses on medical
robotics, including flexible robots and intelligent perception for minimally
invasive surgery.
