Speech Acoustics - Science topic

The key to speech emotion recognition is the feature extraction process. The quality of the features directly influences the accuracy of classification results. If you are interested in typically feature extraction, the Mel-frequency Cepstrum coefficient (MFCC) is the most used representation of the spectral property of voice signals as well as you can try energy, pitch, formant frequency, Linear Prediction Cepstrum Coefficients (LPCC), and modulation spectral features (MSFs).

According to your suggested IS09 and IS10 which one is better so both are working good and there is no big difference but I recommend trying high-level (DL) features, it will be defiantly better than low-level.

Abhijeet R Deshmukh Have a look: https://stats.stackexchange.com/questions/282987/hidden-markov-model-vs-recurrent-neural-network

Hi, I would recommend this website for checking emotional stimuli datasets https://airtable.com/shrnVoUZrwu6riP9b/tbljKUnVvikhzaNvF/viwlo7OvlHBG2q88P?blocks=hide

"KAPODI - the searchable database of free emotional stimuli sets."

Best regards,

Diogo Branco

I find your research interesting. Now bear in mind that one of the objectives of conducting researches is to prove or support existing theories as well as yield results which are dissimilar to the available published findings and resources. I believe you would want to choose either of these two options.

Isabel,

I know of many studies on when phoneme detectors develop but never thought about your question.

Are we even sure that people perform speaker normalization in the same sense that speech recognition systems do (or at least used to before deep learning)?

Since we do know that one of the early layers of human speech recognition processing produces a sequence of best phoneme guesses, and higher layers can force backtracking to try the next best guesses, it is possible to check psychophysically if there is such a normalization at lower layers by checking the processing speed, and if there is a change in processing speed after normalization takes place.

It is also an interesting question how the speaker recognition and speech recognition processes are related. Do we first recognize the speaker and then apply that speaker's phoneme recognizers? I remember reading a paper a few years ago about recognizing accents before recognizing speech (but can't find the reference).

Another clue is Hearing different accents at home impacts language processing in infants from U Buffalo (www.sciencedaily.com/releases/2017/12/171205104127.htm ) which found that infants exposed to multiple accents before 12 months develop different recognition strategies. See also the JASA reference there and

Linguistic processing of accented speech across the lifespan (www.frontiersin.org/articles/10.3389/fpsyg.2012.00479/full).

Finally Language Discrimination by English-Learning 5-Month-Olds: Effects of Rhythm and Familiarity (labfon.letras.ulisboa.pt/personal/sfrota/aeli/Nazzi_Jusczyk_Johnson_2000.pdf) may be of use.

Would love to hear what you learn!

Y(J)S

Please, try https://arxiv.org/ftp/arxiv/papers/1709/1709.07552.pdf.

May be, it will be usefull...

Nice Dear Kévin Vervier

There are so many varieties of tools for extracting pitch, but none of the fully automatic algorithms I know can guarantee accuracy and consistency of extracted f0, especially in terms of continuous f0 trajectories in connected speech. An alternative is to allow human operators to intervene where automatic algorithms helplessly fail. ProsodyPro (http://www.homepages.ucl.ac.uk/~uclyyix/ProsodyPro/) provides such a function. It is a script based on Praat—A program already with some of the best pitch extraction algorithms. But ProsodyPro allows human users to intervene with difficult cases by rectifying raw vocal pulse markings. It thus maximizes our ability to observe continuous f0 trajectories.

Yes, you can use RNN Encoder-Decoder to produce the phoneme embeddings, it means RNN maps each the phoneme to embedding space.

Segmentation of naturally spoken speech into words, even when there is a relatively small dictionary of words, is a harder problem than recognizing isolated digits.

People tend to think of spoken words as somehow isolated but "close" in time. This is not the case, unless you have a cooperating speaker (who helps the detection, or at least monitors it and repeats when it misdetects).

You can easily find in the literature the standard end-point detection mechanisms people use (mostly Viterbi based), and then run the isolated word detectors, but they are computationally expensive and don't really work very well for natural speech (the possible exception was flexible endpoint DTW, but I doubt that you are using DTW as a detector).

Y(J)S

It sounds simple, but unfortunately, it is not!

There are many confounding factors that make this process complicated. I give you some examples: consider you have a recording of your own voice recorded in a sound proof room saying "OPEN THE DOOR", and you would like to use that recording as the reference to which other voice commands are compared to take an action to open the door, for example.

Considering the formants of the two signals and comparing them using some similarity measures could be a very simple and quick solution. But unfortunately, they do not provide a good result since, for example, the similarity measure of two completely different sentences recorded in a particular acoustic environment can be relatively higher than two roughly similar sentences recorded in different environment, or if in the second recording the speaker utters the similar words than the reference recording but in different order.

To deal with these factors and variabilities, you might need a model (such as hidden Markov model, Gaussian mixture model) to capture the acoustical characteristics of the signals (in some relevant feature space such as cepstral domain or time-frequency domain) and to relate the segments of a signal to the language unites, and also you need a language model to link the unites to recognize the sentence. All these procedures are covered under the speech recognition field.

Most speaker verification databases like NIST ones are paid. But, there are a couple of freely available databases like Voxceleb (http://www.robots.ox.ac.uk/~vgg/data/voxceleb/) and SITW(http://www.speech.sri.com/projects/sitw/).

Hope that helps!

There is also this: http://www.oxfordwaveresearch.com

Hi Tohidul , there are quite a few:

What is the evidence that testing with English word lists is not valid. Patients and testers are naturally reluctant for the use of English lists on foreign speakers, but in my experience meaningful results can be obtained.

Here is another method: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2902000/

I personally prefer Praat scripts

web.stanford.edu/dept/linguistics/corpora/materi PRAAT_workshop_manual_v421.pdf

Good luck

It is one of the old issues in a speech recognition research field. That is on the relationship between any speech enhancement technique and the classification accuracy.

As far as I know, both the MSE and PSNR are frequently used for improving the quality of the input. They are known as useful in reducing WER. However, the relationship with recognition accuracy is not directly proportional. 

Enhancing a noisy signal in terms of MSE or PSNR means that you may have a good quality of the input but there is a risk. Sometimes, unexpected artifacts are produced by the speech enhancement techniques and WER can be increased in the worst case.

So, in phonemic classification task, matched condition is more crucial. And in the case of mismatched condition between train and test, MSE and PSNR are somewhat related to WER, but not directly. It is a case-by-case study.    

A.C.Gimson - An Introduction to the Pronunciation of English

Since DTW works only on pair of (potentially multidimensional) time-series, if you want to have all the three utterances aligned with DTW, you can only obtain that by aligning two signals separately to the third one. To choose the best reference signal, you may try each of them and select either the one which leads to smallest amount of time-warping (DTW outputs a cost value measuring the amount of time-warping) or the one which leads to the smallest error when comparing the aligned signals with the reference.

hope it helps

Your question is ill-posed.

It depends on a variety of factors: acoustic models, language models, recognition task, ASR front-end, ASR back end etc....

To simplify, if you have close-talking models, probably a RIR long 50ms (800 sample) won't affect much the ASR performance.

Dear Jim,

if you're also interested in experimental work on tone sandhi in Mandarin you may want to check out the papers below:

I agree with Harm.  Vocal cord/fold action affects Fundamental action; Fo action generates the harmonics; their relative strength is determined by state of folds; thicker, shorter has stronger more numerous harmonics that are filtered by vocal tract.

                               Richard Dean, Ohio University/Stanford U

Hi,

A member of my old research group investigated overt speech with fMRI. Check her publications: Brendel, Bettina.

E.g.:

Brendel, B., Hertrich, I., Erb, M., Lindner, A., Riecker, A., Grodd, W., et al. (2010). The contribution of mesiofrontal cortex to the preparation and execution of repetitive syllable productions: An fMRI study. NeuroImage, 50, 1219-1230

Hi Dina, 

Have a look at the study by Jan Svec and Svante Granqvist entitled "Guidelines for Selecting Microphones for Human Voice Production Research"

And also Sramkova et al 2015 entitled "The softest sound levels of the human voice in normal subjects" 

I believe this two articles will really help with your selection. 

Good luck

There have been some recognition experiments on Yoruba e.g. A Review of Yorùbá Automatic Speech Recognition, Yusuf et al, 2013 IEEE 3rd International Conference on System Engineering and Technology, 19 - 20 Aug. 2013, Shah Alam, Malaysia; CROSS-LANGUAGE MAPPING FOR SMALL-VOCABULARY ASR IN UNDER-RESOURCED LANGUAGES: INVESTIGATING THE IMPACT OF SOURCE LANGUAGE CHOICE, Vakil and Palmer, SLTU 2014. Typically the language model is used to help differentiate between acoustically similar but distinct language contexts. The IARPA Babel programme has been looking at recognition and keyword spotting of low resource languages which covers a wide range of types of languages and should provide some useful information for you. There have been a number of papers published at Interspeech, ICASSP, SLTU and ASRU. The Kaldi toolkit provides a recipe to build a state-of-the-art ASR system for Babel languages. Depending on your experience and the data set(s) available for training your acoustic and language models this may be a good place to start.

Hi Bettina,

if you have enough data from your subjects, z-scoring, but separately for each phoneme class, may be an option. I've used phoneme-specific z-scores for recognizing prosodic boundaries and pitch accented syllables. This takes into account Hartmut's finding, mentioned by Susanne above, that different phonemes are affected differently. Nick Campbell, by the way, came up with the term elasticity -- phonemes differ in their elasticity, and he introduced the phoneme-specific z-scores to model phoneme durations in synthesis in a paper in 1992.

Antje

For a cheap setup, for 2 by 2 cases, behringer ecm8000 measurement microphones, and any usb interface, in particular I have used an alesis io2 express 2 channel USB interface. You will need to adapt some stand to mount the microphones.

Use a notebook for recording, unplugged from the wall to avoid power noise. Get a good characterization of the room acoustic characteristics. You can measure impulse responses using TSP method or MLS method and then schroeder backintegration to estimate power decay curves. I have made some software for this, you can find it here: http://ldipersia.wikidot.com/software#revtime

If you have a sound proof room you can get better recordings. You can add drapery and carpets to reduce reverberation time, and you can use plywood panels to increase the reverberation time, so you can produce different environments to perform the experiments.

Dear Poornoma,

There are many mathematical equations that show the relationship between two signal. e.g

1) Correlation

2) Mutual Information

3) Euclidean distance 

etc...

and some  features that help to biuld some relationship between signals e.g

1) Pitch

2) Harmonics

3) Fundamental frequency

4) Energies

 and distribution of above features e.g standard deviation, skewness , kurtosis etc

Thank you Prof. Ivleva and  Prof. Prunescu. Prof. Ivleva, I very much like the historic insight given at the website. Prof. Prunescu, your point is well noted. It has to do with geographical locations. I am thankful to both of you. I am working on variations in Ewe (i.e. a local language) and your points are very useful to me.

Sir, You can write to John M Swales who was instrumental in developing MICASE. He responds to our queries. Generally we get access to transcripts only. The audio databases are not shared. There is Dr Claudia from Germany, Dresden . she collected a lot of samples from Indian users of English.  Her contact is also useful.

I would suggest to test prtools toolbox for matlab

For catching a particular frequency component Goertzel algorithm or EPLL technique can found useful. I found it while doing literature review. 

Hi Paulina,

Your best bet for a description of "Standard Scottish English" is:

Robinson, C. & Crawford, C. A. (2001). Scotspeak: A guide to the pronunciation of modern urban Scots. Perth: Scots Language Resource Centre.

The comparison with Received Pronunciation is something you'll have to do yourself!

Lachlan

Dear Kiran,

Check this paper out if it gives you an idea:

February 7, 2006 Feature

Resonance Disorders and Nasal Emissions

Evaluation and Treatment using "Low Tech" and "No Tech" Procedures

 by Ann W. Kummer

Thank you very much Pooya. I was told the same thing by the Department of Astronomy, but my investigations came up with very different results. I looked at your list of publications and noted that you must be in really advanced physics and mathematics (if I'm not wrong). I am looking at the tension issue from the standpoint of auditory psychophysics and not in terms of pure physics. The document I'm referring to is my paper The Tension Theory of Pitch Production and Perception. This is the work that led to all the questions I am asking.  if you are not in hearing research, the paper  may bot be relevant to you area of expertise...

Thank you once again for your help on the matter.

I know it is not the same problem, but a similar approach which might be relevant to you is that used to improve sound source separation results in:

Xabier Jaureguiberry, Gaël Richard, P. Leveau, Romain Hennequin et E. Vincent, (2013), Introducing A Simple Fusion Framework For Audio Source Separation, "Machine Learning for Signal Processing (MLSP)", Southampton, UK

A lot of thanks Marzena! I wish I could help you with something, too! :)

Best,

Juris

Hi Saameh,

It would be good if you check your formant results (as well as the f0 results) for their fit with values given for the respective vowels (see lit) and gender of the speaker (f: between ~150-400/500, m: ~70-150). This can be done automatically to exclude and reanalyse possible outliers. How many files do you have? If it is less than say a hundred you could also try Praats inbuilt formant and F0 tracker and check those results visually.  This also has the additional advantage that you can use Praat's scripting techniques to get your results into e.g. R, Julia or whatever prog u r going to use.

Hello Shakti,

we are running a European project in which this problem is being handled. A paper on a novel solution is in preparation by Johan Jansson, N. Cem Degirmenci and Jeannette Spuehler; the journal to be decided. You can read more about the project at www.eunison.eu.

Best regards,

Sten Ternström

EUNISON Coordinator

You could give ProsodyPro a try to see if it can handle some of your cases: http://www.phon.ucl.ac.uk/home/yi/ProsodyPro/

Apart from special issues, there are four particularly prominent accounts of 'speech acts':

(1) JL Austin's 'How to Do Things with Words' (1975),

(2) JR Searle's 'Speech Acts' (1969),

(3) Bach & Harnish's 'Linguistic Communication and Speech Acts' (1979) and

(4) WP Alston's 'Illocutionary Acts and Sentence Meaning'.

The notions given of what an 'illocutionary act' (or 'speech Act') actually is differ strongly in each case: your friend should be aware of the possible fact that these authors are dealing with different phenomena. Thus, Austin is concerned with institutional act which involve (what he calls) the "securing of uptake"; in the case of Searle it is rather unclear what identity conditions the acts he deals with actually have; Bach and Harnish are dealing primarily with acts of communication ("communicative speech act"), and secondarily with institutional acts ("cnventional speech acts"), while Alston's interest is directed at acts of 'saying something' (in a somewhat pecular sense of 'saying' not equal to any of the beforementioned kinds of act). One of the important first decisions your friend shall have to make is thus: Which of the different conceptions of 'speech acts' does he/she intend to deal with in his/her diss?

[Note: Austin's and Searle's accounts are not easy to detangle; my own diss. 'Illocutionary Acts -- Austin's Account and What Searle Made Out of It' (2006) sets out to give detailed analyses of both. Searle's account was commonly recognised as the leading account for decades; during the eighties, Bach and Harnish's account became widely 'accepted', too. Most authors endorse the outstanding significance of Austin's work on the matter, but only few seems to have actually read the relevants texts authored by him (in any meaningful sense of 'read'); here the work of Marina Sbisà contains particularly helpful analyses.]

Hi Katharina,

not really practical help, but a few psycholinguistic studies on the role of ambisyllabic consonants in Dutch and English. Zwitserlood et al. (1993) showed that Dutch CVC syllables primed both CVC.CVC syllables and also strings with an ambisyllabic consonant CV(C)VC. Also CVC syllables were detected faster than CV syllables in words with ambisyllabic consonants. In production, the results of syllable-reversal experiments by Schiller et al. (2007) also suggest that the ambisyllabic consonants are mostly seen as the coda of the first syllable. Similar findings would probably apply to German.

In work on tonal alignment, Atterer & Ladd (2004) and Ladd, Mennen & Schepman (2000) used target words in which some of the stressed syllables were closed by an ambisyllabic consonant. In these conditions, the ambisyllabic consonant was treated as belonging to the stressed syllable (Atterer & Ladd 2004: 184).

Hope this helps

Dutch:

Schiller, N.O. Meyer, A. & Levelt, W.J.M. 1997. The syllabic structure of spoken words: evidence from the syllabification of intervocalic consonants. Language & Speech 40, 103-140.

Zwitserlood, P., H. Schriefers, A. Lahiri & Wilmaar van Donselaar. 1993. The role of syllables in the perception of spoken Dutch. Journal of Experimental Psychology: Learning Memory, & Cognition 19. 260-271.

English:

Treiman, R. and Danis, C. 1998. Syllabification of Intervocalic Consonants. Journal of Memory and Language 27, 87—10.

Ladd, D.R., Mennen, I., Schepman, A. 2000. Phonological conditioning of peak alignment in rising pitch accents in Dutch. Journal of the Acoustical Society of America 107, 2685-96.

Well, it would be heavily dependent on the frequency - the lower the frequency, the longer (more visible) the wave length. It would probably be a very low sound (about 2-5 Hz) used by no animal known to man (at least not to me). Another thing is that it would depend heavily on the density of air or any other medium (liquid or solid). If you "talk to water" you can observe some waves that your speech is causing (some of them will be of course caused solely by the expulsion of air but others, the smaller and more numerous ones, will resonate to the vibrations of your voice). I'm not sure about the effect of the temperature or humidity.

All in all, human eye (and brain) can process images at maximum about 15 Hz, whereas the ear starts at at least about 20 Hz. So you will be able to see it but not to hear it. That's for sure. Do such things occur in nature? I don't know and I don't think we can see air distortion (unlike water distortion). If they do, do they depend on the environment? Yes, very much.

Shimmer and jitter give you information about stability in the phonation, while MFCC give information regarding articulators (it is still in dicussion in the state of the art). Normally, if you use MFCC you don't need to use LPC because the information provided by LPC is already in the MFCC. Respect to PLP or RASTA analyzes, all of that are related to perform spectral smoothing and are very useful for speaker identification.

For fussion you can use all of the features you want or you need. At the features level is just to put together all of the matrixes (be careful with the sizes) and depending on the classifier, you can fuse also at the "scores" level, i.e. if you use an SVM, you take the distances from the sample to the hyperplane as feature.