Recording and analyzing kinematic data in children and adults with SOLLAR: Sonographic & Optical Linguo-Labial Articulation Recording system

Abstract

Understanding the development of spoken language in young children has become increasingly important for advancing basic theories of language acquisition and for clinical practice. However, such a goal requires refined measurements of speech articulation (e.g., from the tongue), which are difficult to obtain from young children. In recent years though, technological advances have allowed developmental researchers to make significant steps in that direction. For instance, movements of the tongue, an articulator that is essential for spoken language, can now be tracked and recorded in children with ultrasound imaging. This technique has opened novel research avenues in (a)typical language acquisition, enabling researchers to reliably capture what has long remained invisible in the speech of young children. Within this context, we have designed an experimental platform for the recording and the processing of kinematic data: SOLLAR (Sonographic and Optical Linguo-Labial Articulatory Recording system). The method has been tailored for children, but it is suitable for adults. In the present article, we introduce the recording environment developed to record over 100 children and 30 adults within SOLLAR. We then describe SOLLAR’s data processing framework, providing examples of data visualization and a summary of strengths and limitations.

Full text

Noiray, A., et al. 2020 Recording and analyzing kinematic data in children and
adults with SOLLAR: Sonographic & Optical Linguo-Labial Articulation Recording
system.
Laboratory Phonology: Journal of the Association for Laboratory
Phonology
11(1): 14, pp. 1–25. DOI: https://doi.org/10.5334/labphon.241
lab
la
phon
Journal of the Association for
Laboratory Phonology
Laboratory Phonology
hon
JOURNAL ARTICLE
Recording and analyzing kinematic data in children
and adults with SOLLAR: Sonographic & Optical
Linguo-Labial Articulation Recording system
Aude Noiray1,2, Jan Ries1, Mark Tiede2, Elina Rubertus1, Catherine Laporte3 and
Lucie Ménard4,5
1 Linguistics department, University of Potsdam, DE
2 Haskins Laboratories, US
3 Department of Electrical Engineering, École de technologie supérieure, CA
4 Département de linguistique, Laboratoire de Phonétique, UQAM, CA
5 Center for Research on Brain, Language, and Music, CA
Corresponding author: Aude Noiray (anoiray@uni-potsdam.de)
Understanding the development of spoken language in young children has become increasingly
important for advancing basic theories of language acquisition and for clinical practice. However,
such a goal requires refined measurements of speech articulation (e.g., from the tongue), which
are difficult to obtain from young children. In recent years though, technological advances have
allowed developmental researchers to make significant steps in that direction. For instance,
movements of the tongue, an articulator that is essential for spoken language, can now be
tracked and recorded in children with ultrasound imaging. This technique has opened novel
research avenues in (a)typical language acquisition, enabling researchers to reliably capture
what has long remained invisible in the speech of young children. Within this context, we have
designed an experimental platform for the recording and the processing of kinematic data:
SOLLAR (Sonographic and Optical Linguo-Labial Articulatory Recording system). The method has
been tailored for children, but it is suitable for adults. In the present article, we introduce the
recording environment developed to record over 100 children and 30 adults within SOLLAR. We
then describe SOLLAR’s data processing framework, providing examples of data visualization and
a summary of strengths and limitations.
Keywords: Ultrasound imaging; speech kinematics; methodology; language acquisition
1. Introduction

        
     
         
 
          


developmental research because empirical questions remain that cannot be addressed
             
   
   

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 2 of 25

             





            

            
        

   
           




      
   
      
    
    
    
     



   

           
      
   
           









     
    

            

   



Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 3 of 25
   


     
  


      table      
         

   

           
             

         
            




         






             
         




2. Recordings within SOLLAR
2.1. Creating a child-friendly recording environment
     


        
     


         

        
        
    
            


Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 4 of 25







            
 

              


  


               
     
        
              

    


    




       


2.2. Role of experimenters in child recordings

          


          
    
                 


     
          

     




Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 5 of 25





period, she/he regularly encourages the child in completing the missions and monitors
            


2.3. Equipment used for SOLLAR’s recording platform

   table
         

   



Figure 1 table
    

table electrical table
          table 
      
to remain stable            







Figure 1: Left: Profile view of the ultrasound probe, probe holder, and pedestal when not integrated
into the spaceshift’s control panel, right: front view of the probe, probe holder, and car seat.

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 6 of 25

Figure 2   









            
              

2.4. Strategies for probe and head stabilization
              

       





increased constraints on head motion lead to more constrained and thus less natural




    
  
      


Figure 2: Profile (left) and frontal (right) views on the face of an adult speaker during a recording
with SOLLAR.

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 7 of 25


          
      
Figure 2 


Figure 2     




      









       







3. Description of SollarSuite
3.1. General description
      
            

   Figure 3 
  

table
      


       

Figure 5



Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 8 of 25







            

3.2. Data processing (SollarSynch)

 
 
, , , and  containing
             
       


Figure 3: Flowchart highlighting the main components of SollarSuite and how they integrate the
different data streams.

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 9 of 25
 and available ultrasound video





 




     
 

 



 and 
    , , and      


    

table


 to import all avail




3.3. Tongue surface contour detection (SollarContours)

 builds upon
     

         

   

 
Figure 4
 

   


Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 10 of 25
        











   


           
  is a publicly available


the video, 
            

  
       
           
            data, the contour

          
Figure 4: Display of SollarContours.

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 11 of 25

             
 table

 
    
           
 to reduce the time

3.4. Head and probe movement correction (SollarTrack)
   
Figure 5





 



    







Figure 5: Screenshot of the SollarTrack GUI, displaying a frame of the camera video with tracking
markers (yellow circles) and palate trace (green lines) superimposed. Smaller panels on the
right-hand side show template matching information (top) and the four reference templates.

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 12 of 25
 puts no constraints on the number or layout
            



            

      



      





is estimated


 

 
    
     



 


       
               
    


         


improves data quality by removing the variability introduced by head motion, either by

    







Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 13 of 25
3.5. Data exploration and export


    
             


              
      
     










tion in contour plots

the data into individual plots
           






     

   Figure 6  







     

              

        



Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 14 of 25





     





 


             
Figure 6: Scatter plot illustrating the front-back tongue body value position for the highest point
on the tongue body at the temporal midpoint of a vowel (x-axis) and previous consonant
(y-axis) in CV sequences produced repeatedly by an adult speaker.
-20 -10 01020
-20
-10
0
10
20
CONSONANT1: b | r2=0.80
CONSONANT1: d | r2=0.11
CONSONANT1: g | r2=0.46
C1_050
V1_050

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 15 of 25





4. Examples of data visualization



            

Figure 7
           

         
  
Figure 7: Averaged midsagittal tongue contours of an adult speaker created with SollarPlot. Left
side: anterior part of the tongue; right side: back of the tongue. Each colored tongue contour
represents the temporal midpoint of /g/ in CV syllables with various vocalic contexts. The black
line illustrates an estimate of the hard palate structure.

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 16 of 25
             




line in Figure 7




Figure 7 as covariance



 Figure 7

    











Figure 8           
           



            

      
      






5. Summary of strengths and limitations
              
    



Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 17 of 25


     
    






     



     

Table 1

6. Conclusion
           
           
   table        
            


         
          

Figure 8: Midsagittal tongue contours at the midpoints of four consonants for a seven-year-
old child. Left: highly variable contours prior to motion correction. Right: reduced variability
subsequent to corrective transformations applied and trials excluded, for which the head is
displaced more than 5 mm laterally above the probe.
-50 -40 -30 -20 -10 10 20 30
mm
0
10
20
30
40
60
70
80
mm
CONSONANT1: b
CONSONANT1: d
CONSONANT1: g
CONSONANT1: z
-50 -40 -30 -20 -10 10 20 30
mm
0
10
20
30
40
60
70
80
mm
CONSONANT1: b
CONSONANT1: d
CONSONANT1: g
CONSONANT1: z
FEXCL: -1 FEXCL: 0

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 18 of 25
Table 1: Summary table of SOLLAR’s strengths, limitations, and perspectives for improvements.
Strengths Limitations & known Problems Perspectives for improvement
Recording
platform
– Child-friendly, maintains children’s interest
and motivation to complete the task
– two experimenters needed to conduct the study (one
operating all devices, the other to monitor children)
– Multiple devices needed in addition to the ultrasound
device
– Using research-oriented ultrasound
devices that includes synchronized
audio signal recording, substantial
storage of high-quality ultrasound video
and potentially includes a video camera
Video
Camera
setup
– Inexpensive: two webcams and blue stickers – USB cameras and recording software not built for
accurate synchronization; most likely only correct to
within one or two frames
– Trade-off needs to be found between video file size,
image quality, and frame-exact image retrieval when
applying video codec settings
– Requires dedicated video recording machine with
sufficient power
– Replacing individually placed markers
with larger tracking marker could help
calibrate for mm distance, replacing the
obtrusive spectacles
– Separately recording the two webcams
and using audio stream synchronization
could improve temporal accuracy, but
complicates lab setup
SollarSync – Data synchronization by cross-correlating
audio is very reliable
– Shared .sllr data structure removes the need
for dealing with a multitude of files
– Video handling in MATLAB depends on the capabilities
of the computer and operating system and can produce
slightly different results from one platform to the next
when building a shared timecode
– Storing data in a single structure makes data slightly less
accessible
– Greater flexibility and adjustability
should be a main direction for future
updates
– Speed improvements and batch
processing capabilities could be added
Sollar-
Contours
– Expanded GUI makes main functionality of
GetContours more accessible to researchers
less familiar with MATLAB
– Import of  data can speed up tongue
contour tracing by relying on automatically
generated data
– Additional navigational functionality in GUI
improves its use as a tool for data inspection
– Tongue detection (or manual correction) is time
consuming
– Forked off of a previous version of GetContours, i.e.,
updates to GetContours have to be manually ported to
SollarContours
– Displays a fair amount of currently unused information,
e.g., spectrogram with formants
– Some assumptions about our specific setup are hard-coded
– GUI should be reworked to focus on
most-used elements
– Greater flexibility for differently sourced
ultrasound data
(Contd.)

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 19 of 25
Strengths Limitations & known Problems Perspectives for improvement
SollarTrack – Good flexibility regarding configuration of
tracking templates
– Combined approach of reference matching
between recordings and frame-by-frame
tracking within recording proved a reliable
method
– Chance of increasingly unreliable point tracking when
videos are long
– Performs time consuming pre-import of video frames (to
then greatly speed up tracking and motion calculation)
– Requires relatively large amounts of memory
– Relies on MATLAB’s parallel computing toolbox, which
might not be available to all users
– Handling of problems tracking should provide better
options for user manual intervention
– Feature to exclude partial segments of a
video, where tracking is interrupted, in
testing stage
– Development of tracking algorithms
to inform experimental setup and
emphasize importance of strict testing
protocol

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 20 of 25
Acknowledgements
    
          






Competing Interests

References
     
The Journal of the Acoustical
Society of America, 144

       
Clinical Linguistics & Phonetics, 25

      
        Advances in Speech
Language Pathology, 9
               
       
  Plos one, 15   

            
Clinical Linguistics & Phonetics, 19



    
      Proceedings of the National Academy of
Sciences, 112

Journal
of Speech, Language, and Hearing Research, 57   


          
7th International Conference on Speech Motor Control


Journal of Speech, Language, and Hearing Research, 62


       Clinical linguistics & phonetics,
29

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 21 of 25
           
     

            

Speech Communication, 93

       International Journal
of Language and Communication Disorders, 30   

          
J Speech Lang Hear ResApr; 42


Clinical linguistics & Phonetics, 31

             
 Clinical Linguistics & Phonetics, 19 


Journal of Speech, Language, and Hearing Research, 51


Journal of Speech, Language, and
Hearing Research, 43

 Journal of Speech, Language, and Hearing Research, 45 

              
 
Speech Communication, 52


         Speech Pathology Australia National
Conference: Evidence and Innovation

      
Proceedings of the 17th Meeting of Formal Approaches to
Slavic Linguistics
          
Language and Linguistics Compass, 14

      
The Journal of the Acoustical Society of America,
100
    Un poste “visage-parole” couleur: Acquisition et traitement
automatique des contours des lèvres. 
            
   Medical image analysis,
44

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 22 of 25
           
The Journal of the Acoustical
Society of America, 105
             
Clinical linguistics & phonetics, 32

      
Clinical linguistics & phonetics, 19

            
      Clinical linguistics & phonetics, 30

      
Journal of
the Acoustical Society of America
            
         Journal of Speech,
Language, and Hearing Research, 57

            
Faits de langues, 37


Journal of Speech,
Language, and Hearing Research
            
     
Journal of Phonetics, 35

            
Behavioral and Brain Sciences, 21


            
Journal of Speech, Language, and Hearing Research, 61

        
Speech motor control:
New developments in basic and applied research

              

Emergence of language Abilities

              
          
  The Journal of the Acoustical Society of America, 129 


Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 23 of 25


Proceedings of 8th International Speech Production Seminar, Strasbourg 

             
  Laboratory phonology, 5 


Eighth International
Conference on Spoken Language Processing
     
The Journal of the Acoustical Society
of America, 133

         Frontiers in
Psychology, 10
      
Journal of Speech,
Language, and Hearing Research, 62   


   
The Journal of the Acoustical Society of America, 92

           
Proceedings of the International Seminar on Speech
Production, Providence

Frontiers in Human Neuroscience,
10

       Language, speech, and hearing
services in schools, 50

         
   Laboratory Phonology  

            
      PloS one, 13


Laboratory Phonology, 11

    Proceeding International Speech Production Seminar,


          Conference
Phonetics and Phonology in Europe.

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLARArt. 14, page 24 of 25

Journal of Speech, Language, and Hearing Research, 41

       

The Journal of the Acoustical Society of America, 125

     
   The Journal of
the Acoustical Society of America, 131

             
         
         
Journal of Phonetics, 41 

Clinical
linguistics & phonetics, 19   


International Journal of Language
and Communication Disorders, 54 

              
          
   Journal of Communication Disorders, 44 


Proceedings of Ultrafest VII.

    International
Journal of Pediatric Otorhinolaryngology, 101  


           
Journal of Speech, Language, and Hearing Research

         
          
     Int J Lang Commun Disord Mar; 54



In Proceedings of the 9th International Seminar on Speech Production (ISSP)

Clinical
linguistics & phonetics, 29


QMU Speech Science Research Centre Working Papers, WP-13

Noiray et al: Recording and analyzing kinematic data in children and adults with SOLLAR Art. 14, page 25 of 25
How to cite this article: Noiray, A., Ries, J., Tiede, M., Rubertus, E., Laporte, C., & Ménard, L. 2020 Recording and analyzing
kinematic data in children and adults with SOLLAR: Sonographic & Optical Linguo-Labial Articulation Recording system.
Laboratory Phonology: Journal of the Association for Laboratory Phonology
11(1): 14, pp. 1–25. DOI: https://doi.org/10.
5334/labphon.241
Submitted: 25 October 2019 Accepted: 28 August 2020 Published: 13 October 2020
Copyright: © 2020 The Author(s). This is an open-access article distributed under the terms of the Creative Commons
Attribution 4.0 International License (CC-BY 4.0), which permits unrestricted use, distribution, and reproduction in any
medium, provided the original author and source are credited. See http://creativecommons.org/licenses/by/4.0/.
OPEN ACCESS
Laboratory Phonology: Journal of the Association for Laboratory Phonology
is a
peer-reviewed open access journal published by Ubiquity Press.

Clinical linguistics & phonetics, 31


    Motor Control,
15
     

Journal of the International Phonetic Association, 42   
