Left and right-hand guitar playing techniques detection

Full text

Left and right-hand guitar playing techniques detection
Loïc Reboursière
Numediart Institute, UMONS
TCTS Lab
Mons, Belgium
loicreboursiere@gmail.com
Otso Lähdeoja
Numediart Institute
TCTS Lab, UMONS
Mons, Belgium
otso.lahdeoja@gmail.com
Thomas Drugman
Numediart Institute
TCTS Lab, UMONS
Mons, Belgium
thomas.drugman@umons.ac.be
Stéphane Dupont
Numediart Institute
TCTS Lab, UMONS
Mons, Belgium
stephane.dupont@umons.ac.be
Cécile Picard-Limpens
Numediart Institute
TCTS Lab, UMONS
Mons, Belgium
ccl.picard@gmail.com
Nicolas Riche
Numediart Institute
TCTS Lab, UMONS
Mons, Belgium
nicolas.riche@umons.ac.be
ABSTRACT
In this paper we present a series of algorithms developed
to detect the following guitar playing techniques : bend,
hammer-on, pull-off, slide, palm muting and harmonic. De-
tection of playing techniques can be used to control exter-
nal content (i.e audio loops and effects, videos, light events,
etc.), as well as to write real-time score or to assist gui-
tar novices in their learning process. The guitar used is a
Godin Multiac with an under-saddle RMC hexaphonic piezo
pickup (one pickup per string, i.e six mono signals).
Keywords
Guitar audio analysis, playing techniques, hexaphonic pickup,
controller, augmented guitar
1. INTRODUCTION
Guitar has maintained a close relationship with technologi-
cal innovation throughout its history, from acoustic to elec-
tric and now to virtual [3]. The term ”augmented instru-
ment”is generally used to refer to a traditional (acoustic) in-
strument with added sonic possibilities. The augmentation
can be physical like John Cage’s sonic research on prepared
pianos, but nowadays the term has acquired a more com-
putational meaning: the use of digital audio to enhance the
sonic possibilities of a given instrument as well as the use of
sensors and/or signal analysis algorithms to give extra and
expressive controls to the player. Guitar playing techniques
detection are part of this last category and several of them
have alreday been investigated. In [13] and [10] focus has
been put on estimating the point where the string has been
plucked, i.e the plucking point. In [5], left-hand fingering of
a guitar player has been analyzed and characterized offline.
In [6] algorithms to detect plucking and expression styles
for bass guitar has been investigated. In [9] automatic note
transcription from an hexaphonic pickup has been achieved.
On the other hand, several studies focus more on the artis-
tic side using added sensors and/or analysis algorithms to
control synthetic sound parameters [7], [11], [12] or [4].
In this paper, we put our efforts on the audio signal anal-
ysis part, in order to detect guitar playing techniques. As
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
NIME’12, May 21 – 23, 2012, University of Michigan, Ann Arbor.
Copyright remains with the author(s).
they are closely linked to the guitarist it is a very natural
way for the instrumentalist to control any type of media or
effects and turn the guitar into a multi-media instrument /
controller.
Our system could be compared to existing pitch-to-midi
technologies like the 1Axon AX 50 or the 2Roland VG-99
systems but our approach is broader and more complete as
we include all major guitar playing techniques.
2. EXTRACTION OF PLAYING ARTICU-
LATIONS
Articulations used to play the guitar can vary a lot from one
guitarist to another as well as from one style to another. We
worked on the most common guitar articulations (defined
in [8]) listed as follows: hammer-on, pull-off, slide, bend,
harmonic, palm muting.
Our algorithmic approach for articulation detection is de-
scribed in Figure 1. The diagram emphasizes the discrimi-
nation between right-hand and left-hand attack as the first
important element leading to the detection of all the artic-
ulations. We did not develop any pitch detector algorithm,
as this question is a fairly well-known problem in signal
processing. The following algorithms have been used: YIN
(temporal), sigmund~ Max MSP object (spectral) and MIR
toolbox pitch detection algorithm (used with the autocor-
relation method).
Figure 1: Diagram of the articulations detection algorithm
1http://www.axon-technologies.net/
2http://www.roland.com

3. BUILDING THE DATABASE
A Godin Multiac guitar with an under saddle hexaphonic
RMC pickup and Alvarez Alliance HT classic nylon strings
mounted on it has been used to build the database. Two
styles of picking (finger and pick) have been recorded by two
different guitarists, leading to the building of two databases
for which: the range of the recorded notes is going from
the open string to the 16th fret; hammer-on, pull-off and
slide notes range from a half-tone to one and a half tone of
variation; slide notes have been recorded in both directions;
bended notes couldn’t hardly go above a half-tone of varia-
tion, as we were using nylon strings; the five first harmonics
have been recorded for each string; normal notes have been
played at three different positions: bridge, soundhole and
fretboard; all recordings have been hand-segmented with
3Sonic Visualizer software.
Each picking style database is made out of 1416 samples
(normal notes: 288, palm muted: 96, hammer-on: 234, pull-
off: 234, slide: 468, bend: 66, harmonic: 30).
The guitar database we recorded is available 4online.
4. ONSET DETECTION
As a general scheme for describing onset detection approaches,
one can say that they consist in three steps [2]. First, the
audio signal is pre-processed in order to accentuate certain
aspects important to the detection task, then, the amount
of data from the processed signal is reduced in order to
obtain a lower sample rate. Finally, thresholding and/or
peak-picking can be applied to isolate the potential onsets.
4.1 Evaluation
Here, we have been comparing a range of onset detection al-
gorithms which covers 18 variants of amplitude-based meth-
ods (including the energy, log-energy domains, and their
time derivatives) and short-time Fourier transform based
methods (including spectral flux in different domains, phase-
based methods and their variants using amplitude weight-
ing, and complex-based methods). Evaluation is performed
using the approach proposed in [1]. A tolerance of plus or
minus 50 ms on the timing of the detected onset has been
used to still be considered as valid because of the lack of
accuracy of the hand-annotated reference files. We have
not been using methods requiring a pitch estimation. The
monophonic recordings (sum of the 6 separate channels) of
the guitar signals were used and we optimized the detection
threshold for peak F-measure for each detection method in-
dividually. Table 1 presents the results for the four best
performing methods, and provide the peak F-measure on
the whole data set, as well as recall values for 4 categories
of attacks: right-hand attack (i.e normal notes), harmonic,
hammer-on, pull-off. Bends haven’t been considered for this
evaluation.
We observe that if the detection of right hand (including
harmonics) attacks is generally not a problem, the detection
of hammer-on and pull-off attacks is better achieved using
STFTs-based methods. Indeed, a significant part of those
left-hand attacks do not show any amplitude increase at
all. Finally, the best performing approach has a F-measure
above 96% with a recall close to 100% for right-hand at-
tacks, 98% for hammer-ons, and a moderate 88% for pull-
offs. Some further research would hence be necessary to
understand how pull-offs can be better detected.
4.2 Discrimination between left and right hand
attacks
3http://www.sonicvisualiser.org/
4http://www.numediart.org/GuitarDB/
When the string is plucked (by a finger or a pick), its vibra-
tory regime is momentarily stopped, resulting in a trough
just before the attack in the signal envelope. As opposed
to a plucked note, legato attack doesn’t stop the string’s
vibration leading to a shift in pitch without a significant
change in amplitude. After several testing with different
speeds of playing, it appears that the gap can vary from
20ms to 50ms. It has to be noticed that for faster playing,
e.g tremolo, the gap disappears and a dedicated algorithm
should be implemented to detect such audio event. Our
left-hand / right-hand attack discrimination system is thus
based on the observation of the very first milliseconds be-
fore the onset. A simple measure of the slope between the
minimum energy point preceding the attack and the maxi-
mum point at the attack. Best results were obtained when
computing the slope using two neighboring half-overlapping
frames. This lead to a 94.0% correct classification rate when
using a properly optimized threshold. We observed that
classification errors are often due to string noise preceding
right hand attacks, causing the energy slope to be smaller
than it could be. We are hence looking into improving ro-
bustness to these playing artefacts.
5. LEFT-HAND ARTICULATIONS
Once the distinction between right-hand and left-hand at-
tack is performed, it is thus possible to categorize the left-
hand articulations by inspecting into the pitch profile of the
note. The left hand works on the string tension and fretting,
thus affecting the pitch. Our method of left-hand playing
technique detection operates by measuring the pitch time
derivative.
Figure 2: Distribution of the number of transition half
tones with the maximal relative slopes for notes with left-
hand articulation: hammer-on (blue), pull-off (black), bend
(green) and slide (red).
Hammer-on (ascending legato) is characterized by an abrupt
change in pitch, as well as its counterpart, the pull-off (de-
scending legato). The bend shows a slower evolution in
pitch. The slide has a pitch derivation similar to the hammer-
on or pull-off, but with a ”staircase” profile corresponding
to the frets over which the sliding finger passes.
As a first step, two parameters are investigated: the num-
ber of half tones of the note transition, and the maximal
relative pitch slope, defined as the maximal difference of
pitch between two consecutive frames spaced by 10ms di-
vided by the open string pitch value. Figure 2 shows how
these two parameters are distributed for notes articulated
with a hammer-on, a pull-off, a bend or a slide. It can be
noted from this figure that the great majority of bended

Method F-measure Normal Right-hand Recall Harmonic Recall Hammer-on Recall Pull-off Recall
Spectral Flux 96.2% 99.6% 100% 97.9% 88.0%
Weighted Phase Divergence 95.8% 98.9% 100% 97.4% 87.6%
Amplitude 92.4% 99.6% 100% 92.3% 75.2%
Delta Amplitude 91.8% 99.6% 100% 91.9% 74.4%
Table 1: Results of the onset detection with four different techniques
notes (green points) are easily identified as they present a
very low pitch slope. Secondly, it can be observed that for
transitions with more than one half tone, a perfect determi-
nation of slide versus hammer-on or pull-off is achieved. As
a consequence, the only remaining ambiguity concerns the
distinction of slide with hammer-on/pull-off for a transition
of a half tone.
To address this latter issue, two parameters are extracted:
the energy ratio and the spectral center of gravity ratio.
Both of them are computed on two 40ms-long frames: one
ends at the transition middle, while the other starts at that
moment.
Based on the aforementioned approaches, a detection of
left-hand articulated notes has been proposed simply by us-
ing thresholding. The results of the ensuing classification
are presented in Table 2. It can be noticed that all bend
effects are correctly identified. Slides are determined with
an accuracy of 97.61%. Finally, hammers and pulloffs are
detected in more than 93%. The main source of errors for
these latter effects is the remaining ambiguity with slides of
one half tone.
Hammer Pull-off Bend Slide
Hammer 93.27% 0.45% 0% 6.28%
Pull-off 0% 93.69% 0% 6.31%
Bend 0% 0% 100% 0%
Slide 1.74% 0.43% 0.21% 97.61%
Table 2: Confusion matrix for detection of left-hand artic-
ulated notes
6. RIGHT-HAND ARTICULATIONS
In this section, two right-hand articulations are studied:
palm muting and harmonics notes.
6.1 Palm Muting
Palm muting is obtained by plucking the string with the
palm of the right hand slightly touching the string. The
produced sound is stifled, the sustain period of the palm
muted note is shorter and decreases faster than the one of a
normal note. Moreover high frequencies decrease faster than
other part of the spectrum when the note is palm muted.
As a consequence the spectrum was filtered out from 0 to
500Hz. Figure 3 shows the slopes (starting at the attack) of
the spectral envelopes of four notes: three notes are played
with a normal attack (at the bridge, soundhole and fret-
board) and one is palm muted. For the three normal notes,
one can see the longer sustain period as the curves stay
rather flat compared to the palm muted note whose slope
increase logarithmically.
Based on that behavior, our algorithmic approach calcu-
lates the value of the energy envelope slope at the attack
and compares it to a defined threshold. The slope is com-
puted as the energy ratio between two frames: one located
at the energy peak following note onsets, and the one that
follows. Table 3 shows the results of the algorithm which
has been run on 48 normal notes for each strings (16 per
position: bridge, soundhole, fretboard and 16 palm muted
Figure 3: Slope of the energy of four notes played at the
bridge (blue), at the soundhole (red), at the fretboard(green)
and palm muted (black)
notes). The misclassified notes are mostly due to imperfect
playing. 97.91% means that one note has been misclassified
and 93.75% and 87.5% respectively leads to one and two
misclassified notes.
String Normal notes Palm muted notes
1 (thresh -0.06 ) 97.91% 93.75%
2 (thresh -0.06 ) 100% 100%
3 (thresh -0.06 ) 100% 100%
4 (thresh -0.04 ) 100% 100%
5 (thresh -0.05 ) 100% 100%
6 (thresh -0.05 ) 97.91% 87.5%
Table 3: Palm muting detection results for the six strings
6.2 Harmonics
Harmonics are obtained by slightly fretting a note on a node
of the string with the left hand. Two techniques are inves-
tigated to achieve this detection. One operates in the time
domain, while the other one focuses on the spectral domain.
Both approaches only consider the note attack, which allows
for a real-time detection of harmonic notes. It has been at-
tested that, after the attack, the differentiation between an
harmonic and a normal note might become more difficult,
especially for the 7th and 12th fret. The two methods are
explained in the following.
•Time-domain approach: Figure 4 shows the wave-
form at the attack for a normal note and an harmonic.
Two parameters are proposed: the attack duration is
defined as the timespan during which the attack wave-
form remains positive; the relative discontinuity dur-
ing the attack is defined as Amin/Amax (right side
of Figure 4).
•Frequency-domain approach: Figure 5 shows the
magnitude spectrum of the attack for a normal note
(left) and an harmonic note (right) using a 40ms Han-
ning window. On these spectra, we extracted a single
parameter: the harmonic-to-subharmonic ratio is de-
fined as the difference in dB between the amplitude of

Figure 4: Attack during the production of a normal note
(left) and an harmonic note (right).
the first harmonic (at F0) and the amplitude of the
subharmonic at 1.5·F0.
Figure 5: Magnitude spectrum during the attack of a nor-
mal note (left) and an harmonic note (right).
Based on the two previous approaches, a detection of har-
monic notes has been proposed simply by using a thresh-
old. The results of the ensuing classification are presented
in Table 4. It can be noticed that all harmonic notes are
correctly identified, while for normal notes, the best results
are achieved for notes played on the fretboard (with only
0.52% of misclassification) and the worst ones are obtained
for notes played at the bridge (with a bit less of 95% of
correct detection).
Harmonic detection Normal detection
Harmonic 100% 0%
Fretboard 0.52% 99.48%
Soundhole 2.78% 97.22%
Bridge 5.38% 94.62%
Table 4: Confusion matrix for harmonics detection.
7. CONCLUSION AND PERSPECTIVES
This paper focused on features extraction from an hexa-
phonic guitar signal, in order to detect and recognize all
the playing techniques commonly used on the guitar. The
methodology was built on the successive detection and clas-
sification of 1) attacks/note onsets, 2) left-hand and right-
hand discrimination, 3) articulation types: normal, mute,
bend, slide, hammer-on, pull-off, harmonic and palm mut-
ing.
Playing techniques algorithm have been tested and imple-
mented separately in Matlab and Max/MSP with positive
feedback from informal live evaluations. However a global
algorithm needs to be implemented to gather all playing
techniques detection. This implementation should assessed
the processing power issues encountered when using at least
three detections in real-time. In addition, the defined algo-
rithms have to be tested by different guitarists as well as
with different types of guitar in order to build a relevant
user-study. Finally, a machine-learning method should be
considered to enhance the adaptation of the global algo-
rithm to these different users.
8. ACKNOWLEDGMENTS
Thomas Drugman is supported by FNRS and Nicolas Riche
by FNRS/FRIA Belgium. Lo¨
ıc Reboursi`ere, Otso L¨
ahdeoja,
St´ephane Dupont and C´ecile Picard-Limpens are supported
by 5numediart, a long-term research program centered on
Digital Media Arts, funded by R´egion Wallonne, Belgium
(grant N◦716631).
9. REFERENCES
[1] A.Holzapfel, Y.Stylianou, A.C.Gedik, and B. Bozkurt.
Three dimensions of pitched instrument onset
detection. IEEE Transactions on Audio, Speech and
Language Processing, 18 (6), August 2010.
[2] J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury,
M. Davies, and M. B. Sandler. A tutorial on onset
detection in music signals. IEEE Transactions on
Speech and Audio Processing,, 16 (5), September 2005.
[3] G. Carfoot. Acoustic, electric and virtual noise: The
cultural identity of the guitar. Leonardo Music
Journal, 16:35–39, 2006.
[4] R. Graham. A live performance system in pure data:
Pitch contour as figurative gesture. In Proceedings of
the annual pure data convention, 2001.
[5] E. Guaus and J. L. Arcos. Analyzing left hand
fingering in guitar playing. In Proc. of SMC, 2010.
[6] G. S. Jacob Abesser, Hanna Lukashevich.
Feature-based extraction of plucking and expression
styles of the electric bass. Proceedings of the ICCASP
Conference, 2012.
[7] O. L¨
ahdeoja. Une approche de l’instrument augment´e:
La guitare ´electrique. PhD thesis, ´
Ecole Doctorale
Esth´etique, Sciences et Technologies des Arts, Paris 8,
2010.
[8] J. C. Norton. Motion capture to build a foundation for
a computer-controlled instrument by study of classical
guitar performance. PhD thesis, CCRMA, Dept. of
Music, Stanford University, 2008.
[9] P. D. O’Grady and S. T. Rickard. Automatic
hexaphonic guitar transcription using non-negative
constraints. In Proceedings of the Irish Signal and
Systems Conference, 2009.
[10] H. Pettinen and V. V¨
alim¨
aki. A time-domain
approach to estimating the plucking point of guitar
tones obtained with an under-saddle pickup. In
Applied Acoustics, volume 65, pages 1207–1220, 2004.
[11] M. Puckette. Patch for guitar. In Pd-convention,
2007. Workshop Pd patches available here
http://crca.ucsd.edu/~msp/lac/.
[12] L. Reboursi`ere, C. Frisson, O. L¨
ahdeoja, J. I. Mills,
C. Picard, and T. Todoroff. Multimodal guitar: A
toolbox for augmented guitar performances. In Proc.
of NIME, 2010.
[13] C. Traube and P. Depalle. Extraction of the excitation
point location on a string using weighted least-square
estimation of comb filter delay. In Proceedings of the
Conference on Digital Audio Effects (DAFx), 2003.
5http://www.numediart.org