Synchronization of PiCam Cameras for Three-Dimensional Study of Dynamic Multi-Domains Natural Scenes

Abstract

This poster illustrate the article of the same title published in ISPRS Annals 2020 by L. Avanthey, L. Beaudoin, C. Villard, S. Mellouk and R. Treglia. We study the interest of PiCam and its possibilities offered for the realization of a light payload (small and inexpensive) in order to perform the 3D reconstruction of dynamic scenes (underwater or aerial) in close-range remote sensing. We see that on these observation scales, movements of the scenes due to flora and fauna cannot be ignored if we want these objects to be part of the final model. We review the sensors used in the literature for 3D reconstruction and then present the arguments in favor of PiCam with regard to the constraints posed by the use of light and agile vectors. The main issue is the synchronization of these low cost sensors, which is not native: we explain the different steps to obtain a satisfactory synchronization rate with regard to the dynamism of the studied scenes and present the results obtained.

Full text

Synchronization of PiCam cameras for
three-dimensional study of dynamic
multi-domain natural scenes
L. Avanthey, L. Beaudoin
ACTE (Data Acquisition
and Processing) research
Team, LASTIG, IGN,
94160, Saint-Mandé,
France
SEAL (Sense, Explore, Ana-
lyse and Learn) research
team, EPITA, 94270 Le
Kremlin Bicêtre, France
Natural scenes are often dynamics What if we want to reconstruct them in 3D?
Centimetric GSD (Bulatov et al., 2011, Kng et al.,
2011, Rossi et al., 2017)
Acquisitions sensitivity to the dynamism of the scene increases with the GSD1
This concerns many aerial
surveys and most underwa-
ter surveys where viewing
distances are reduced due to
absorption
Visual representations of displacements observed
over 1s from a fixed position on scenes with favo-
rable conditions (low wind or current)
1 Ground Sample Distance
Sub-centimetric GSD: 50 mm (Aicardi et al., 2018),
< 5 mm (Skarlatos et al., 2012, Menna et al., 2018),
< 2 mm (Henderson et al., 2013, Burns et al., 2015, Sch-
midt, Rzhanov, 2012, Gracias et al., 2013, Germanese et
al., 2019)
On average, they are of the order of several
tens of pixels, (up to 10 cm given the GSD)
To reconstruct dynamic objects, motion must be frozen
The 3D reconstruction process assumes a rigid geometry
of the scene (a single matrix for an overlapping area)
An object that moves between the acquisitions of the
stereoscopic viewpoints of the scene will have a
local geometry different from
the global geometry
Moving divers disap-
peared during 3D re-
construction
It will be eliminated
during the geometric
filtering process
Global geometry
Shark local geometry
Fish local geometry
Let's freeze the scene!
Results on real scenes
High synchronization of PiCam sensors
The matching rate (number of inliers) is used to assess the adequacy of our synchronization
OpenMax
OmxcamRaspicam
MMAL
HARDWARE
VideoCore
Cam1
Board
Master
Cam2 Cam...
Board
Slave
Board
Slave
Synchronisation protocol to trigger the shootings
Qualification of synchronisation
This poster illustrates the article "Synchronization of PiCam cameras for three-dimensional studyof dynamic multi-domain natural scenes",
by L. Avanthey, L. Beaudoin, C. Villard, S. Mellouk, R. Treglia, published in 2020 at ISPRS Annals
A self-adaptive Harris point detector followed by
local statistical filtering on the vector flow formed
by the paired points is used (Avanthey et al., 2016)
~1 millisecond difference for sync pairs,
~1 second for out-of-sync pairs
No camera mouvements, fixed base
Loss greater than 40% of good pairings when moderate
movements occured (anemones, fish, birds, etc.)
Loss greater than 95% when fast
movements occured (caustics, etc.)
Sync (~1 ms)
Out-of-sync (~ 1s)
Average of 45% of good matches (inliers) on synchronized
pairs (close to results on static scenes)
RaspiCam library on a non real-time
OS: ~50ms
Omxcam Library on a non real-time
OS: ~8ms (~6x better)
Omxcam Library on a real-time OS
(PreemptRT): < 1 ms (no drift over
time)
The evaluation is carried out by taking a photo of a digital timer displayed on a screen
coupled with a LEDs bench
2 white LEDs surround the 10 LEDs and are lit every two cycles to distinguish them
The LEDS bench has 10 LEDs which light up alternately every 1 ms (1 cycle = 10ms)
The screen display refreshes at 60 Hz, so every 16.7ms
50 cm
~60,000 pts
~120,000 pts
30 cm
Freezing a school of mullets to catch them in 3D
The local relative displacement between overlaping shots
should not exceed the size of the GSD (Avanthey et al., 2016)
More than one sensor
Control of the shooting
and acquisition parameters
(on-board computer)
A full-duplex communication
protocol to trigger shots and
quickly exchange information
between master and slaves
PiCam V2 PiCam HQ Raspicam working principle Possible accesses to control a PiCam
The delay in launching a task, called jitter, means that:
Launching two tasks at the same time does not imply their execution at the same time
The more tasks, the more time gaps add up unpredictably
1.
PiCam sensors meet the criteria
mentioned and are suitable for pho-
togrammetric work (Venkataraman et
al., 2013, Santise et al., 2017, Piras et al.,
2017)
Its size, weight and price are com-
patible with close-range remote
sensing (operational flexibility and
high precision rather than spatial
coverage)
Control the jitter by using a real-time OS (RTOS) which guarantee a max jitter or great-
ly minimizes it: the jitter of the soft RTOS PREEMPT-RT is correct for our use (Arthur et
al., 2007, Dias et al., 2014)
2.
Reduce the number of tasks: the Omxcam wrapper (semi-open OpenMax API) offers
better and more minimalistic control than the Raspicam wrapper (proprietary MMAL
API) which is just an image grabber on a videostream (not suitable for synchronization)
Tried I2C (half-duplex), then UART (full-duplex)
and next test will be RS485 (more bandwidth)
Choice of sensor:
PiCam HQ allows to change the
optics