Speech Signal Processing - Science topic

Answers to some of your queries are as follows:

- It depends on network configurations, the way one creates training examples & datasets. A lack of systematic benchmarking of existing methods however creates confusion. There are several studies which shows that LSTM outperforms CNN for speech emotion recognition. Specially LSTM with attention mechanism helps to boost emotion recognition performance. Some other studies report the opposite and CNN seems to be a better choice. You can do a quick literature survey on latest papers published in last couple of INTERSPEECH, ICASSP & ASRU.

- MFCC is the default choice for most speech processing tasks including speech emotion recognition. However, MFCC is not the optimal one as it lacks prosody information, long-term information. That's why often MFCC is augmented with pitch (to be more specific log F0) and/or shifted delta coefficients. These additional information help to boost the emotion recognition performance. MFCC lacks phase information but the role of phase in emotion recognition performance is not much investigated. The parameters for MFCC computation such as the number of filters, the frequency scale are chosen experimentally and they are dependent on the dataset and the backend classifier.

Did you try to use different engines?

An intuition about the cepstral features can help to figure out what we should look for when we use them in a speech-based system.

- As cepstral features are computed by taking the Fourier transform of the warped logarithmic spectrum, they contain information about the rate changes in the different spectrum bands. Cepstral features are favorable due to their ability to separate the impact of source and filter in a speech signal. In other words, in the cepstral domain, the influence of the vocal cords (source) and the vocal tract (filter) in a signal can be separated since the low-frequency excitation and the formant filtering of the vocal tract are located in different regions in the cepstral domain.

- If a cepstral coefficient has a positive value, it represents a sonorant sound since the majority of the spectral energy in sonorant sounds are concentrated in the low-frequency regions.

- On the other hand, if a cepstral coefficient has a negative value, it represents a fricative sound since most of the spectral energies in fricative sounds are concentrated at high frequencies.

- The lower order coefficients contain most of the information about the overall spectral shape of the source-filter transfer function.

- The zero-order coefficient indicates the average power of the input signal.

- The first-order coefficient represents the distribution spectral energy between low and high frequencies.

- Even though higher order coefficients represent increasing levels of spectral details, depending on the sampling rate and estimation method, 12 to 20 cepstral coefficients are typically optimal for speech analysis. Selecting a large number of cepstral coefficients results in more complexity in the models. For example, if we intend to model a speech signal by a Gaussian mixture model (GMM), if a large number of cepstral coefficients is used, we typically need more data in order to accurately estimate the parameters of the GMM.

Dear Navdeep Kaur,

Look the link, maybe helpful.

Regards, Shafagat

I would like to recommand you these two articles:

What you actually want to change is the window size. For that you need to change the pitch settings and give a minimum pitch 4 times more than the rate you want rate you want to have (i.e. 400/4000 for a window size of 0.ü1/0.001). This will however not change the overall mean intensity much, and is more relevant to clinical research.

I use both: MFCCs and SDC. However, I recomand you to try also extencional features such as: RASTA PLP, Thomson MFCC and others...

It is one of the old issues in a speech recognition research field. That is on the relationship between any speech enhancement technique and the classification accuracy.

As far as I know, both the MSE and PSNR are frequently used for improving the quality of the input. They are known as useful in reducing WER. However, the relationship with recognition accuracy is not directly proportional. 

Enhancing a noisy signal in terms of MSE or PSNR means that you may have a good quality of the input but there is a risk. Sometimes, unexpected artifacts are produced by the speech enhancement techniques and WER can be increased in the worst case.

So, in phonemic classification task, matched condition is more crucial. And in the case of mismatched condition between train and test, MSE and PSNR are somewhat related to WER, but not directly. It is a case-by-case study.    

This is a good idea however it needs a lot of work, Firstly, we can employ computer vision to identify objects and in a simple case just convert object names into speech. In summary, the components of such a system are available namely; a module for object identification in images and speaking words module. Just integration is needed if not done somewhere.

If the recorded speech signal is degraded by Background noise, using wavelet tresholding would be more appropriate than low pass filters for vowels.

You can find measured distribution of the clean speech signal in time and frequency domain plus models in this paper: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Tashev_Acero_Statistical_Modeling_of_the_Speech_Signal.pdf.

Hi Yilun,

DET:

First, you need to compute the true score from genuine trials and false scores from imposter trials. Then you may use NIST DETware

or BOSARIS toolkit to get the DET plot/

Check the help sections/readme files for details.

Speaker identification (SI) v/s Speaker verification (SV): Check Page 5 of

Hope this helps.

Sahid

Dear Sushanta,

Here's some literature that might be relevant to your interest:

Also you could take a look at the tools we have developed for prosody analysis and synthesis: https://www.researchgate.net/project/Research-Tools-mostly-Praat-based

Frequency approach to music, speech and hearing research has never ceased to raise fundamental questions since Ohm's Acoustical Law (1843) and Helmholtz's Resonance Theory (1877). Hussein's question above is just one of millions unanswered questions. . Stephen's reference above to the Nyquist theorem provides adequate answer to the question at hand. If I record musical instruments at say 8 k/Hz sample rate, the sound quality is relatively poor in comparison to the same signal recorded at a higher sample rate. In speech, you might find that different phonemes (particularly fricatives/sibilants) are not well captured at low sampling rates as you'll lose the high frequency components which are critical distinctive features (acoustically speaking).  Thus, if you use a unique sample rate, you might limit yourself to a specific sound source. I adopt Xaver's procedure mentioned above. It is one wise way to discover what sample rate does to the quality of your recordings.

I have tried mfcc together with pitch feature, and it works well in Chinese ASR, because Chinese language is a strong tonal languange. However, for pitch to be effected, something should be dealt carefully:

1. mfcc frame shift and that of pitch should be the same, so that the total frames are the same

2. for tonal language ASR, the tonal information is rather saved in delta pitch and deltadelta pitch, so the static pitch can be removed;

3. be careful, for speech where pitch is not found (no resonance), do not try to amend it by averaging methods, this is harmful for speech recognition, just set delta pitch and delta delta pitch to zero

That's my experience to make these two features work.

1. recognition always work on correlation of data or you can say correlation of features in the data.

2. more number of samples will always help you to increase accuracy

3.accuracy is your case will depend on what type of features you use

4. if you want to enhance accuracy use both time and frequency domain features , it may slow down your algorithm but accuracy will improve.more the feature you include more will be the accuracy.

5.my advice work with very simple logic you need to increase correlation.it can be done by increasing samples if you cant increase samples you have to increase features for recolonization . 

This might be interesting

A hybrid phonotactic language identification system with an SVM back-end for simultaneous lecture translation

Sapan,

The human ear and brain are very good at speech recognition as are Siri and other computer-based algorithms. To spoof an person's speech, you must first have a good, lengthy sample of the speech and then develop what is essentially a vocal tract model for the human speaker. The model is a transfer function between the vocal cords, air supply, and air flow of the specific human vocal tract and the listener. Of course the model changes if the speaker is sick, has a cold, swollen vocal tract, etc. Once you have a vocal tract model and proper excitation sounds (like vocal cords), you can create speech that spoofs the speaker. This is not an easy task because you must learn a lot about how human speech is created.

Please ask another question if you need clarification.

Good luck,

Steve

The way I did it was inspired by an algorithm I found in a book titled DAFX by Zölzer (2002). Basically you calculate the cross-correlation between the end of the first phoneme and the beginning of the second and find the point where they correlate the most. This is where you want to concatenate the phonemes/diphones/units. Based on this point, you multiply the end of the first phoneme by a decreasing ramp and the beginning of the second phoneme by an increasing ramp and then just add them up. I hope you understand the idea.

Dear Suciati Anandes,

First, you have to make an inventory of certain targeted statements that carefully pose the suprasegmental features you want to investigate . Subsequently, ask the participants (the selected sample) to read them. Naturally, you record their voices in this particular stage. Finally, ask three native speakers of English to rate them based on the accuracy and appropriacy of their production of suprasegmentals. Good luck with your research.

Best regards,

R.Biria

In India, we are working on the Indian language (Tamil & Hindi) speech recognizer  based on the KALDI toolkit.

It is a very challenging task and highly depends upon the number of speakrs, the f0 interval and dynamics of the speakers.

Eg. if there are only one male and one female speakrs, you could probably use band-pass filter with the f0 range and then a standard pitch tracking algorithm (e.g. KALDI pitch tracker or SWIPE).

I think it may be usefool for you

As you know, I’m no expert in phonetics and phonology.  To be honest, I have no idea what the AS-unit coding framework is!

I was just wondering whether the AS-unit coding framework meant the same thing to all researchers.  It appears to me that in our research papers, we often need to explain how we define certain analytical concepts and how we understand certain analytical frameworks, and not only adopt but also adapt these concepts/frameworks.

You asked “if I should use another clause definition for break-down fluency (i.e., pause) measurements”.  The crux of the argument might be what clause definition serves best to answer your research questions.

Hope other people will be involved in our discussion!

Thank you, I got it Rajesh ji. I misinterpreted it. For ex, x= (0.100)₂ is equals to (4/8)_10,  if you truncate it to x=(0.1)₂ again it equals to (4/8)₁₀ = (1/2)_10. This holds for positive numbers so I assumed it is true for negative numbers to without even having a single thought about it . That's a big mistake, but I learned from it.

Formant frequencies vary depending on the the size and shape of the vocal tract. The first three formants (F1, F2, and F3) are the most important in speech. F1 is correlated with the height of the tongue, i.e. F1 increases as the tongue moves from high to low position. F2 is associated with the backness of the sound, i.e front sounds have higher F2 frequencies than back sounds do. F3 correlates with lip rounding; it decreases as the lips take rounding shape. 

Hi, I've not read the article you refer to, but the variance (2nd moment of a density function or distribution) is usually symbolised as sigma-squared. Standard deviation is often a more useful measure though. If calculated on audio data, SD is referred to as spectral spread which is a perceptually relevant predictor in many contexts. From spectral spread you'll get "spectral variance" by squaring. You can find scripts for Matlab for example in the excellent MIR toolbox (https://www.jyu.fi/hum/laitokset/musiikki/en/research/coe/materials/mirtoolbox)

Dear Yasir Rahmatallah ,Alessio Brutti ·,B. Tomas 

Thank you for the detailed answer. It is exactly what I was trying to do.

Maybe You can see chapter

 Determination of Spectral Parameters of Speech Signal by Goertzel Algorithm,B. Tomas

Speech Technologies 01/2011;

 

Hi Arpit,

Assuming that you are using your 2 microphones as a ULA (Uniform Linear Array), the separation of 6cm/0.06m will correspond to a wavelength of 0.12m.  Assuming the speed of sound is 343m/s, the wavelength of 0.12m will correspond to a frequency of 2.858 kHz.  I am thinking that if your are sampling a audio signal greater than 2.858 kHz, you may want to place the microphones nearer to each other (less than 6 cm, maybe 5 cm for up to 3 kHz) so as to prevent signal returns due to grating lobes from interfering with your intended audio signals. 

Also, when using MVDR, an array of 2 elements with only provide two degrees of freedom (DOF) for the constraints: (1) to prevent distortion of the signal in the desired direction of arrival and the (2) to suppress interference from other directions.  Thus, by adding a couple more of array elements (microphones), you will be able to produce better interference suppression from undesirable signals from other directions.

Dear Poornoma,

There are many mathematical equations that show the relationship between two signal. e.g

1) Correlation

2) Mutual Information

3) Euclidean distance 

etc...

and some  features that help to biuld some relationship between signals e.g

1) Pitch

2) Harmonics

3) Fundamental frequency

4) Energies

 and distribution of above features e.g standard deviation, skewness , kurtosis etc

Hi ... what is the problem with the ^reliablty ??. U use a safe value 44k

Further to Mikel's answer: 13*(100*s) is a 2D matrix of MFCC. Typically, various researchers take statistics such as mean, var, IQR, etc. to reduce the feature dimension. Some researchers model it using multi-variate regression, and some fit it to Gaussian mixture model. It depends on the next step in the processing pipeline.

if s is noisy and X is original and clear speech:

SNR=10*log10var(X)/var(s-X)

Also I think s= X+W,i.e, noise is additive.

I assume that you are trying to scale different sound files to an approximately equal loudness. If that is the case, you can select a Sound Object in Praat and under "Modify", you can use "Scale intensity" to set a specific dB SPL level (Drop me an email if you need a script that normalizes all files in a directory that way). To determine the actual SPL of the output you can use a sound level meter. 

There are two types of features of a speech signal:

I haven't worked on speech recognition, but maybe the CMU ARCTIC databases could be useful. 

Thank you very much for your suggestion mr. Koch. I'm going to try YAAPT as well, but I still need to test RAPT. For now, I'm going to test it in a few signals.

For TDM you need to simulate the encoding and decoding (G.711 or G.726 for international links). You can learn about acceptable BER from G.821 and G.826. Next you need to model the effects of line echo - see G.168 and the text of the relevant P series documents.

For packet based (e.g., VoIP) systems you need to do is to implement the desired compression and decompression (e.g., G.729, G.723, AMR), and then simulate the effects of packet loss on the compressed voice packets. You need to decide whether compressed frames are sent separately or several per packet, and then to model the packet loss (the accepted way is to use a GIlbert-Eliott state machine, as detailed in ETSI and ITU-T documents).

If you need to model delay and PDV as well, you could use real traces, or generate traces using a tool like NS2 or NS3.

If you need to simulate acoustic echo, then further work is involved. Read the relevant P series documents.

Y(J)S

Yes, it is compulsory. The group delay is, to certain extent, similar to the magnitude spectrum of the signal. Those spikes are due to wrapped phase and not actual, and it has to be avoided. 

There's some analysis in section 5.4 of this paper: http://www.researchgate.net/publication/222436066_NIST_and_NFI-TNO_evaluations_of_automatic_speaker_recognition

You can also find something here, where they use non-nativeness scores to improve performance: http://www.researchgate.net/publication/224762259_System_combination_using_auxiliary_information_for_speaker_verification

You could give ProsodyPro a try to see if it can handle some of your cases: http://www.phon.ucl.ac.uk/home/yi/ProsodyPro/

It is exactly in the frequencies which are weakened by the algorithm or in the places that noise has strong peaks and more power.

If the noise is white then the musical noise is distributed in all frequencies.

I think, you have to detect both, voicing level of speech signal and unvoiced portions and the time depency of the transition.

Compared with antenna array sp , microphone array sp often aims to the near field broadband signals,eg. acoustic signals. In addition , mutipath or echo is another difficulty. Especially for DOA and beamforming techniques, it 's almost a disaster.

http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=5477492&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D5477492 I hope this paper may help.

Robert Marks published the first textbook treatment I used, in 1991 "Introduction to Shannon Sampling and Interpolation Theory". I saw the material in graduate courses in 1981. His references for Fourier methods and the PSF go back to 1961, Landau, "Prolate spheroidal wave functions Fourier analysis and uncertainty", in the Bell Systems Technical Journal. Landau and others in IEEE in the mid 1960's.

Robert Marks' two books: "Introduction to Shannon Sampling and Interpolation Theory", and "Advanced Topics in Shannon Sampling and Interpolation Theory" have a *very* good bibliography.

Wavelet connections, I can't speak to directly. There may be interesting connections to be made, mathematically, but the only one obvious to me is the same advantage you get any time you recognize an orthogonal basis and a transform which provides a simpler analysis in one space than another.

I believe the question is much too general to yield reasonable answers.

What is the quality measure being sought?

What is the nature of the underlying sound that requires quality improvement?

For each sound there will be different methods to tackle the problem.

For example, if the sound signal is filled with high-intensity, high-frequency random noise then one might pursue quality improvement in some particular directions.

For a sound that is overcome by periodic sound interference, there would be another direction.

etc.

Dr. Salih Qaraawi, You will get some aspect of your question through article accessible via link below.

Dear Kuldeep Dhoot, The center of gravity of is a function of only of the first derivative of the phase spectrum at the origin.

It depends on what you need these features. For example you can calculate the average value for each MFCC coefficient . Than you will have 13 features. To calculate delta (first derivative) or delta delta (second derivative) you have to calculate the rate of change and acceleration changes basing on MFCC. Than you can also calculate the averge.

Dear Abu Dzarr,

why don't you use a coherency analysis? It can be used to estimate the causality between the input and output.

See for example: http://en.wikipedia.org/wiki/Coherence_(signal_processing)