Estimation Theory - Science topic

There are many estimation techniques that can be used to estimate system poles. Here are a few popular ones:

To apply an algorithm to estimate system poles, you can start with a simple transfer function and apply the algorithm to estimate the poles. You can then compare the estimated poles with the known poles of the transfer function to evaluate the accuracy of the algorithm. It may also be useful to test the algorithm on more complex systems to see how well it performs.

[1] Bazzi, Ahmad, Dirk TM Slock, and Lisa Meilhac. "Efficient maximum likelihood joint estimation of angles and times of arrival of multiple paths." 2015 IEEE Globecom Workshops (GC Wkshps). IEEE, 2015.

[2] Bazzi, Ahmad, Dirk TM Slock, and Lisa Meilhac. "On a mutual coupling agnostic maximum likelihood angle of arrival estimator by alternating projection." 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP). IEEE, 2016.

[3] Bazzi, Ahmad, Dirk TM Slock, and Lisa Meilhac. "On Maximum Likelihood Angle of Arrival Estimation Using Orthogonal Projections." 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018.

Dear Himali Singh,

I suggest you to see links and attached files on topic.

Best regards

During the modeling process, it may be important to consider the relations of (input and output variables) , the governing differential modeling principles (including simplifications and linearity assumptions, steady-state conditions…), and after the model is obtained, see the properties of the eigenvalues of the system martrix (sign of roots, complex roots, if roots are bounded or not…), and or the poles of the transfer function T may be seen.

Given some system with matrix A find the characteristics polynomial of A.

Apply the “ Routh-Hurwitz” stability criterion on the polynomial.

(1) Negative eigenvalues lead to stable system

(2) Positive eigenvalues lead to unstable ones.

(3) Complex eigenvalues (may) tell us oscillatory conditions….

There may be other cases of considerations...

Thanks

I think that your answer may vary depending on what you consider as better results. In your case, I will assume that you are referring to better results in terms of smaller bias and mean square error. As stated above, if you have poor knowledge and assume a prior that is very far from the true value, the MLE may return better results. In terms of Bias, if you work hard you can remove the Bias of the MLE using formal rules and you will get better results in terms of Bias and MSE. But if you would like to look at as point estimation, the MLE can be seen as the MAP when you assume a uniform distribution.

On the other hand, the question is much more profound in terms of treating your parameter as a random variable and including uncertainty in your inference. This kind of approach may assist you during the construction of the model, especially if you have a complex structure, for instance, hierarchical models (with many levels) are handled much easier under the Bayesian approach.

Hi, you can use the R library called Intsvy or BIFIsurvey. Also it's possible to work with IDB analyzer.

Please, explain the problem in details

Dear Ramin,

I offer you a variety of reference trough  links and attached files in topics.

-Mixture Models - CMU Statistics

-Binomial Gaussian mixture filter | EURASIP Journal on Advances in ...

https://asp.eurasipjournals.springeropen.com/.../s13634-015-0221-... 

Best regards

I would say that any method of measuring the excitation which does not involve the regressor matrix (at least indirectly) is but a pale imitation of one that does. The symmetric (noise-weighted) matrix [R^TWR]^-1 should map to the co-variance of your parameter estimates if you were to re-run the estimation with new data, and this is precisely what identification trajectory design seeks to control: bad trajectories are those which allow parameters of the model to have a low signal to noise ratio.

For an attempt at an intuitive idea, for simplicity assume you have a sinusoidal signal with some noise. Kalman Filter is a filter and like any filter, even a simple first order pole, will clean some of the noise, yet with some time delay, so its output will remain behind the desired signal.

Now, if you have the entire record and you go in the opposite direction, from the end to the start, you again clean some noise, yet the new "delay" is actually a time advance and so, it actually puts the output ahead of the actual signal in time.

Combining the two output signals, besides getting even better cleaning, also eliminates the delay and so, tells you what the real signal did at the real time. 

There seems to be a confusion here. The Delphi technique should not be used to develop ideas. It might be used to evaluate them. Also, there is no requirement that the Delphi technique be used with experts from a given field. Actually, a diverse group is recommended.

You are welcome. I would be interested to know in due course what you discover.

So the only interesting equation is the one describing the state dynamics, i.e. x(k+1) =...

There the difference is the index of the input, i.e. v(k) or v(k+1). Starting from a continuous time model and discretizing it with a first-order hold element, v(k) seems a good choice, but you're free in that.

Thanks Mr. Hugh... I have understood... :)

You can use an order parameter that measures the ratio between the time variance of the mean series and the mean of the time variances of the series. For time series S_i(t), i=1,...,N

R = Var_t ( Mean_i( S_i ) / Mean_t ( Var_t( S_i) )

Mean_i(S_i) is the mean over all time series

The order parameter R is between 0 and 1. If the times series are phased-synchronised, then Mean_i(S_i) is close to any S_i and the ratio R is close to 1. If the time series are out-of-phase, Mean_i(S_i) is more or less constant and the numerator is small compared to the denominator: R is small.

I don't remember the literature or the GMM method clearly but to use GMM T needs to be at least 3. You can also use GMM when you have a long T and small N but this may cause problems due to presence of many instrumental variables. But if T is very large for example 20, 30 or more and N is smaller I think you can or it may be better to use panel time series techniques. 

I have contacted someone and found  final result as " the convergence rate depends on the specific system under consideration (convergence assumes system is observable)"

The data are panel in nature. All cross-section units have the same number of observations for the same years (1990, 1995, 2000, 2004, 2009, 2011). Now, the gap for the first three observations is 5, but for the rest the gaps are 4, 5 and 2 respectively. Is it possible to apply GMM when data are observed in this fashion?

Nonlinear systems by Khalil is very good in Lyapunov theory!

I agree with Norma, in Delphi design, researcher should seek Experts opinion; otherwise if regular respondents are selected in the methods {either random or non-random} it turns to be a traditional sample.

The argument behind Delphi technique is to gather opinions/perspectives of experts from a wide geographical regions on a phenomenon with vaguness. Thus, you need the experts to share their ideas.

best luck

Hello. I think it depends on what you mean by unknown. Even the simplest estimators/filters make implicit assumptions about the errors in the system and measurement models, with most techniques requiring independent (white) noise. If you are after robust estimators, then the H-infinity class may be a good choice. I understand that this filter may not always be realisable depending on its tuning. Another class of estimator that does not make many assumptions about the data is set theoretic. In this case, you only need to bound the noise.

Hello. Your question seems to relate to error bounds for state estimation, in which case I think the answer is yes. Various bounds have been calculated, of which one of the more recent is the posterior Cramer Rao lower bound. This applies to nonlinear state space systems with white noise inputs. The main reference that I know of for the PCRLB is:

Tichavsky, P.,  Muravchik, C.H. and Nehorai, A., "Posterior Cramer-Rao bounds for discrete-time nonlinear filtering", IEEE Transactions on Signal Processing, vol 46, no 5, pp. 1386 - 1396, 1998.

You are right: there has been no great advance since the first predictor/corrector estimators (the Kalman filter and the Luenberger observer) were proposed. All other general recursive methods share the same structure but differ in the correction gain computation technique. Among them are the extended Kalman filter, iterated Kalman filter, unscented ("sigma-point") Kalman filter, cubature Kalman filter, particle filter, H-infinity filter, etc. None of them guarantees convergence for all nonlinear systems.

It is even possible to imagine a system (with a multimodal state vector distribution), for which the notion of a single "state estimate" is meaningless. For such systems, there exist more general methods based on the Fokker-Planck-Kolmogorov equation, but they are much more complex.

My personal experience is that it is often better to investigate the system in depth and to invent an ad hoc estimator which is very simple (though not general) and has a clear physical meaning.

Dear Huda

The maximum likelihood estimatior of the scale, is the average of the absolute deviation from the random variables and the median of the sample (the maximum likelihood estimatior of the location parameter). See http://en.wikipedia.org/wiki/Laplace_distribution.

I hope this help.

When analyzing individual (not aggregated) data such low values are not unusual - you have to decide is it practically useful and have the assumptions behind the analysis been met. Individuals are typically very heterogonous in their attitudes, actions and behaviours.

I am reminded of a famous clinical  trial of the effect of taking aspirin on heart attack - the odds ratio was so dramatic that the trial was stopped and placebo group advised to take aspirin. And yet the odds ratio of a heart attack for placebo compared to taking aspirin was, a rather a lowly 1.83 and the R2   was a puny 0.0011; yet this was sufficient for action.

Your arguments are strengthened if you testing a relationship and have not gone on a fishing expedition and you have tried to take account of theoretically relevant confounders. Epidemiology has gone to some extent from 'what are the causes of this outcome?' to 'does this potential cause have an effect?.

I would also  add if you are modeling binary(0 and 1) outcomes it is exceedingly difficult to achieve high R2 values as the predicted probability values are not very likely to be exactly 1 and 0!

Finally we have to accept that there are outcomes where chance does genuinely part a large part so we now have evidence that luck plays a bigger part in some cancers than genes and lifestyle; see

So for me it is theory, focused question and the size of the slope term and careful evaluation of the model rather than just R2.

Zhang this paper was published seventeen year ago. To my understanding author is trying to use filtering and smoothing technique to reduce noise which is produced by Recursive Newton Method. In Newton Method you can find some computational stability issues due to constraints and your calculation may deviate. But Author also mention that he is not considering computational efficiency. So numerical stability is not the issue then. In short, I suggest you to take a look on recent development on this topic. You can do this by either searching for articles or also try to track publications of Rodriguez, G.   

Hello!, you may find very useful information in the book of Jazwinski "Stochastic Processes and Filtering Theory". In the very fundamentals, both the Kalman Filter and the MHE are equivalent if the system under consideration is linear and there are no constraints. However, both approaches start to diverge once constraints are considered. In the same way, you may find some similarities between the EKF and the MHE, but again, once constraints are considered differences will come out. EKF is very useful when nonlinearities are not that strong. If this is the case, the UKF appears to be a better choice. MHE is an interesting filtering approach but, as Sajith mentioned above the solution of non-convex nonlinear optimization problems related to the MHE formulation is to be considered.

hi,

it will be good if robust the error in your model.

but the more important, model have to pass a bond test , sargan test and wald test.

if you uses GMM two step is better to robust standard error

while, with one step no need

use Stata will help you to robust standard error

if your model harm after robust but the model pass all previous tests no need for robust.

The set of all regular n times n matrices is open in the set of all n times n matrices (even more the set of invertible linear bounded operators in a Banach space X is open in the space of all linear bounded operators in X). So, if you have an approximating sequence T_n  of matrices that converge to an invertible matrix, then for some N one has that all T_n with n > N are invertible. Since the quadrature formula you use leads to convergent sequence of approximations, the job is done. 

Excuse me,sir.Could you say clearly?I don't understand.

Our experience suggests that cross-power spectral density measures (which are the Fourier transform of the cross-correlations between the signals received from two distinct electrodes) are much more indicative of brain state than just power spectral density measure derived from a single electrode.

Thank you Ramoni Adeogun

I need to solve that equation analytically, may you help me the detail of solution? because I have tried but  I can not proof that equation like in that journal.

Thank you for your attention.

I have recently viewed a paper titled

Mars Science Laboratory Launch-Arrival Space Study:

A Pork Chop Plot Analysis by

Alicia Dwyer Cianciolo , richard Powell, and Mary Kae Lockwood

Hope this would be useful

The upper bound is infinite. Take any unbiased estimator. Now add to it the result of a single independent drawing from the N(0, sigma^2) distribution. The estimator is still unbiased, the variance is increased by sigma^2. Now let sigma become arbitrarily large.

I apologize, my original answer is incorrect - I have edited it.

fa is from R(n+q) to R(n) - that is, it takes in explicit values for the noise variables and produces the state at the next time step considering those noise values. It does *not*, as I originally stated, produce values of the noise variables at the new time t+1. This is made clearer by "The unscented Kalman Filter" by Wan and van der Merwe, in the book "Kalman filtering and neural networks". Equation 7.40 table 7.3.1 is what I was missing.

I also made it clearer that the noise parameters include the process and the measurement noise, as shown, for example, in eq. 7.38 of that same reference.

There are many regularisation approaches that you can use, Ridge regression is not the only one. You could, for example consider the Lasso (http://www.jstor.org/stable/2346178), Partial Least Squares regression (http://epubs.siam.org/doi/abs/10.1137/0905052) or Continuum regression (http://www.jstor.org/stable/2345437). Then there are many other approaches including kernel learning methods like Support vector machines regression.

Which is best? it all depends what you want to do. In general no one method is uniformly best. Some methods, like the lasso are designed to produce sparse models (not many predictors)_, that is they include a degree of variable selection. Other models like Partial least squares regression or continuum regression integrate over many variables. Which is more appropriate in you context? Do you want to identify a small number of predictors for future study or simply get the best calibration you can?

In high dimensional work the processes you go through to keep the results honest (i.e. cross validation, hold out independent validation sets, avoiding pre filtering - or filtering within the cross validation loop only) are very very much more important than which model you use. The higher the ratio of the number of potential predictors to the number of observations the more chance you have of finding spurious relationships. Above all you need to be rigorous with your validation.

What I mean by information from time zero is that in the full information problem, the evolution of states from t=0 till t=T is estimated. My question is that why do we need to calculate from t=0 every time.

Dear Marzieh, your questions are too general. Do you mean in state space? If so, then: 1) The finite norm of the system state transition matrix quarantees stability. 2) To be sure that the estimated parameters are true, you need the system model. In state space, the estimation error covariance matrix consists of all the necessary statistical information about the difference between the model and the estimated parameters. If you are not talking about the state-space form, then you must be more certain.

The data reconciliation and Kalman filtering algorithms both solve the optimization problem by minimizing the MSE. Therefore, under the conditions claimed by Kalman, they are convertible. However, the conceptual steps are different. In applications, you may find hybrid structures with prefiltering and then reconciliation, visa versa, or even suggestions to use the reconciliation algorithm as an alternative to Kalman. But note that the Kalman filter is an elegant and widely recognized engineering solution. And that commonly generates a problem for other approaches to compete.