ArticlePDF Available

Abstract and Figures

In regression analysis least square (LS) estimator fails because of its sensitivity to unusual observations present in the data. Robust estimation provides alternative estimates which are insensitive and efficient, when the data are not normally distributed or polluted with distant observations usually called outliers. To cope with the problem of outliers is more challenging in nonlinear regression (NLRM) than linear regression. In this study, least trimmed absolute (LTA) estimator which is a robust and high breakdown estimator is adopted for nonlinear regression model fitting. Bias and mean square error is used to check the efficiency of the proposed estimator. The performance of the estimator is compared with LS and existing robust M-estimators using simulated data sets and real world problems. It has been observed that LTA is efficient in case of contaminated data sets as compared to LS and M estimator. Furthermore, it has been concluded that in case of 40% contamination LTA outperforms LS and M estimator, while in case of 20% outliers LTA and M estimators perform equally well. In regard to clean data sets, LS, LTA and M estimator performs equally well. Conclusion has been made on the basis of simulated data sets as well as real data sets.
Content may be subject to copyright.
© 2017 Pakistan Journal of Statistics 223
Pak. J. Statist.
2017 Vol. 33(3), 223-236
AN EFFICIENT AND HIGH BREAKDOWN ESTIMATION PROCEDURE
FOR NONLINEAR REGRESSION MODELS
Dost Muhammad Khan, Shumaila Ihtesham2, Amjad Ali2,
Umair Khalil1, Alamgir3, Sajjad Ahmad Khan1 and Sadaf Manzoor2
1
Department of Statistics, Abdul Wali Khan University Mardan,
KP, Pakistan
2
Department of Statistics, Islamia College Peshawar, KP, Pakistan
3
Department of Statistics, University of Peshawar, KP, Pakistan
§
Corresponding Author Email: dost_uop@yahoo.com
ABSTRACT
In regression analysis least square (LS) estimator fails because of its sensitivity to
unusual observations present in the data. Robust estimation provides alternative estimates
which are insensitive and efficient, when the data are not normally distributed or polluted
with distant observations usually called outliers. To cope with the problem of outliers is
more challenging in nonlinear regression (NLRM) than linear regression. In this study,
least trimmed absolute (LTA) estimator which is a robust and high breakdown estimator
is adopted for nonlinear regression model fitting. Bias and mean square error is used to
check the efficiency of the proposed estimator. The performance of the estimator is
compared with LS and existing robust M-estimators using simulated data sets and real
world problems. It has been observed that LTA is efficient in case of contaminated data
sets as compared to LS and M estimator. Furthermore, it has been concluded that in case
of 40% contamination LTA outperforms LS and M estimator, while in case of 20%
outliers LTA and M estimators perform equally well. In regard to clean data sets, LS,
LTA and M estimator performs equally well. Conclusion has been made on the basis of
simulated data sets as well as real data sets.
KEY WORDS
High Breakdown, nonlinear least squares, least trimmed square, leverage points,
outliers, M-estimator.
1. INTRODUCTION
The fundamental goal of nonlinear regression is identical to linear regression that is to
regress a response variable to a set of predictor variable . In nonlinear regression
model, the response variable is nonlinearly related to explanatory variable or parameters.
For example, the strengthening of concrete is a nonlinear process. The strength increases
rapidly at the start, and then flattens out. Similarly, marginal cost of production is
nonlinear function of the unit produced. Recently, the issue of analysis and model
estimation in nonlinear regression is, frequently, addressed by statisticians and scholars
because of its growing popularity and application.
An Efficient and High Breakdown Estimation Procedure
224
Consider the nonlinear regression model of the form
     
where is the explanatory variable and is the response variable, is a nonlinear
function, denotes the unknown dimensional vector of parameters, are independent
and identically distributed random variable with mean 0 and unknown variance . If the
model assumptions are satisfied, the parameters can be estimated through ordinary least
squares (OLS), which minimizes the sum of square residuals.
In general, there are two types of nonlinear models. In the first type, models can be
made linear through transformation. In this method, either independent variable,
dependent variable or the combination of both can be transformed. These models are
generally called intrinsically linear models or nonlinear in variable models. Usual least
square method can be applied to the transformed model in order to get estimates of
unknown parameters. Unluckily, this method gives an inaccurate result as the linear
regression is applied to transformed data, which may falsify the error term or completely
change the relationship between and variables. Furthermore, the confidence intervals
are no more symmetric. Therefore, this method has become obsolete and should not be
applied (Brown, 2001).
The second category includes intrinsically nonlinear models. Such models cannot be
transformed by linearization as these models are nonlinear in parameters. For example
Logistic Model:   
  
Asymptote regression model:    
Gompertz growth model:   
To fit nonlinear models in parameters, iterative optimization procedures are used to
compute the parameter estimates. Starting value of the parameters must be provided using a
guess or prior experience. Sum of Squares is computed in first iteration using starting
values then in the next iteration parameter values are changed by small amount and
recalculating the sum of squares, this process is repeated until converge (Brown, 2001).
For iteration, different algorithms are introduced by the researchers. The Marquardt
method is used frequently by pharmacological and biochemical researchers. Some other
algorithms include steepest descent, Nelder-Mead and Gauss Newton method. Another
method is simplex algorithm. In this method, initial value of the parameter and its
increment is provided. This method is fast, and easy but it does not provide standard
errors of the parameters (Motulsky & Ransnas, 1987).
The problem of model estimation becomes challenging when the data set is
contaminated with outliers or an influential observations. An outlier is a data point that is
not consistent with the mass of the observations. In a regression model, outlying
observations are often recognized as those data points whose residuals are much larger or
smaller than the remaining residuals (Gilbert, 2007). In other words, any observation
which is not according to the pattern of remaining data set is called an outlier. Such data
sets may also be called as contaminated data sets. A standard data set may contain the
following three types of outliers; i.e., vertical outliers (outliers in y direction), leverage
Dost Muhammad Khan et al.
225
point (outliers in x direction) and good leverage point (outliers in xy direction)
(Rousseeuw & Driessen, 2006; Tabatabai et al., 2014)
Least square criteria is not robust to outliers, hence the results of estimation and
hypothesis testing would be ambiguous and unreliable in nonlinear regression (Tabatabai
et al., 2014). Robust regression is a statistical method which tries to decrease or eliminate
the consequence of outliers so as to attain more reliable results from the bulk of data. In
case of multivariate contaminated datasets, robust estimation provides the most
appropriate substitute to the classical estimation procedures to balance its sensitivity to
outliers (Khalil et al., 2013). Some of the most general robust regression methods for
linear regression include M-estimator, S-estimator, redescending M estimator, MM
estimator, L estimator which include Least absolute deviation (LAD), least trimmed
square (LTS), least median of square (LMS) and least trimmed sum of absolute (LTA),
etc.
Most of the robust linear regression techniques are successfully adopted for nonlinear
setting. Ekblom and Madsen (1989) proposed Marquardt algorithm for Huber estimator
in nonlinear regression. Stromberg and Ruppert (1992) studied the breakdown properties
of LTS and LMS in nonlinear regression problem. Verboon (1993) applied M estimators,
particularly Huber and Biweight function, for nonlinear model fitting. In the same year
Stromberg extended LMS and MM estimator from linear to nonlinear setting. Tabatabai
and Argyros (1993) incorporated -estimators, originally proposed by Yohai and Zamar
for linear regression, to nonlinear regression analysis. Neugebauer (1996) has analyzed
the robustness properties of M estimator in nonlinear regression.
Sakata and White (2001) extended S-estimators of linear regression to nonlinear cross
section and time series regression. Cizek (2002) worked on LTS and its asymptotic
property in nonlinear fitting. Similarly, Hawkins and Khan (2009) proposed an algorithm
for LTS estimator in this regard. Abebe and Mckean (2013) used rank based estimator for
the said purpose. Rank based estimation was first proposed by Jureckova (1971) and
Jaeckel (1972) for linear regression problem. Most recently, a new M estimator based on
secant hyperbolic function was proposed for nonlinear case (Tabatabai et al., 2014). This
article considers the development of new algorithm for the use and estimation of the
parameters in nonlinear regression based on LTA approach.
2. PROPOSED ALGORITHMS IN THE LITERATURE
Least square method is generally adopted if the data is normally distributed. However
in the presence of outliers, it totally fails. To remedy this problem, alternative methods
have been developed in linear and nonlinear regression, which are more robust to
outliers. Among these, least trimmed sum of absolute residuals (LTA) estimator is rarely
been used in nonlinear regression despite the fact that LTA estimator is relatively easy as
compare to other high breakdown estimators.
Hawkins and Olive (1999) have found a connection between regression outliers and
case leverage. Least square estimator is affected by all types of outliers, irrespective of its
direction in space or space. As compare to this,  estimator is robust to regression
outliers on low leverage cases, but sensitive to outliers on high leverage cases. This
makes it a 0 breakdown estimator as OLS. For LTS, one has to choose suitably low
An Efficient and High Breakdown Estimation Procedure
226
values of h so that all outliers can be trimmed. While using LTA, it is usually enough to
trim outliers on high leverage case only. So, high values of can be used for LTA as
compared to LTS.
Few algorithms for the estimation of high breakdown estimators are discussed in the
next section.
2.1 Progress Algorithm
Rosseeuw and Leroy (1987) proposed PROGRESS (Program for Robust
reGRESSion) algorithm for the computation of LMS in linear regression case. This
algorithm takes into account a trial subset of p values, where p stands for number of
parameters, and fits a linear line passing through them. This method is continued many
times and the fit for which median of squared residual is minimum is retained. The
method is repeated
times. For small datasets it is feasible to consider all p subsets but
for large datasets it may become tedious. The steps for LMS estimator are given below:
Calculate exact fit to p points, where p stands for number of parameters, and
call it

Compute median of squared residuals at

Repeat these steps
times.
The value of
 for which median of the residual is lowest, is LMS estimate.
2.2 Feasible Solution Algorithm
Hawkins and Olive (1999) derived a feasible solution algorithm (FSA) for LTA in
linear regression. It produces perfect approximations to the exact LTA fit. For LTA
estimation,  is fitted to suitable half of the cases. One of its characteristic is that the
absolute residual of all cases that it does not include is greater than or equal to the
absolute residuals that it includes. The algorithm is as follows:
Generate random elemental sets of size, and calculate the residuals.
If the present elemental set provides the  fit to the cases with the smallest
absolute residuals, then it is a feasible solution.
If not, replace one of the cases in the elemental set with a better one.
Continue until a feasible solution is obtained.
Repeat it with the large number of random starts, say times.
Use the feasible solution with the smallest sum of absolute residuals.
2.3 FAST-LTS Algorithm
This algorithm was proposed by Rousseew and Van Driessen (2006), for the
computation of LTS estimator in linear regression estimation problem. Basically, it
involves two steps: Initial step and concentration C step. The fundamental part of this
algorithm is C-step. It states that starting from initial fit a new subset is taken for which
absolute residuals are the smallest, then applying least square to new subset to obtain new
fit, which guaranteed lower objective function. It can be summarized as follows:
Dost Muhammad Khan et al.
227
Initial step:
A starting value of
is generated.
Using starting values, OLS is applied to all cases and finding corresponding
residuals.
Order absolute residuals; identify those cases as best cases to cover for which
absolute residuals are the smallest.
C step:
Apply OLS to the set of covered cases, getting fresh
. Calculate residuals and
discover h smallest absolute residuals and calculate sum of square residuals.
If the sum of squared residuals has reduced from preceding step, then take these
cases and repeat OLS fit.
If the residual sum of square is unaffected from the preceding step, then the
concentration step ends, and LTS solution is obtained.
2.4 Direct Conversion to Nonlinear Regression
The aforementioned algorithms have shown their characteristics in linear regression
case and they can be adapted in the case of nonlinear regression. The main dilemma is
that the algorithms needed to be solved iteratively for a large number of optimization
problems for randomly selected subsets of data. Despite the fact, it does not matter at all
in linear regression case as the minimum of criterion function can be found easily and
uniquely in some cases. The circumstances are differing in nonlinear regression setting as
minimizing the objective function is time consuming, and convergence speed of different
algorithms might be different considerably with respect to the behavior of data.
Therefore, these algorithms can be used for LTA estimation in nonlinear regression, but it
may be comparatively slow and has most likely lower accuracy as compared to linear
regression (Cizek, 2001).
PROGRESS algorithm provides exact fit in case of simple linear regression with
slope and intercept when all possible subsets
are examined. But it fails when line
passes through origin as it does not provide exact LMS fit when intercept is zero. The
reason is that in PROGRESS algorithm intercept is adjusted given the best slope for exact
fit, but as the intercept is zero, this adjustment is not possible, and the resultant fit is local
minima rather than global minima (Barreto & Maharry, 2006). This failure of
PROGRESS algorithm in linear case can be encountered in nonlinear setting as well
since most of the nonlinear models may not include intercept term.
Furthermore, FSA algorithm can be adopted for LTA estimation in nonlinear
regression after some adjustment, but FSA in nonlinear case is not as favorable as in
linear regression. The weakness is that the number of random starts depends upon three
factors; the proportion of outliers, complexity of model and sample size. The value of t
will be increased as long as any one of the factors increases, but the amount of increase is
not obvious (Chen et al., 1997). According to Hawkins and Olive (1999), the
performance of FSA is superb for text book size problems, but for big data sets, its
performance is not satisfactory on efficiency grounds. Additionally, FAST algorithm may
take a lot of time in nonlinear setting investigating poor elemental fits as it requires
significantly more computation in nonlinear than linear setting (Hawkins & Khan, 2009).
An Efficient and High Breakdown Estimation Procedure
228
2.5 A Hybrid FAST Algorithm
As discussed above, there are many drawbacks of directly adapting these algorithms
to nonlinear case. For fitting LTS in nonlinear regression, Hawkins and Khan(2009)
proposed a Hybrid FAST algorithm. In this paper, the best properties of FAST-LTS and
PROGRESS algorithm are combined to get an efficient and fast algorithm. Their
algorithm is as follows:
Outer loop, executed I times
o Inner loop, executed J times
o Create an elemental set and compute elemental fit to get starting values.
o Use these initial values to compute residuals on all cases and find the sum of
squares of smallest absolute residuals. Keeping track of such smallest sum,
the cases giving the smallest absolute residuals, and corresponding elemental
fit.
Take this best-of-J elemental fits. Use its covered cases and elemental fit as
starting values for a concentration step using a usual nonlinear regression
procedure.
The final reported solution is the best of the resulting I outer loops.
3. PROPOSED ALGORITHM FOR ESTIMATION OF NLRM
The proposed algorithm is based on the idea of Hybrid FAST algorithm where the
best features of the PROGRESS and FAST-LTS are implemented in the nonlinear
regression setting. In the proposed algorithm, FAST-LTS approach is substituted with
FAST-LTA keeping in view the high efficiency of LTA as compared to FAST-LTS
approach. The steps for the proposed Hybrid FAST-LTA are as follows:
Generating M elemental sets and its elemental fits.
Using elemental fits for the calculation of residuals on all cases and find sum of
smallest absolute residuals.
Out of M times, identify h smallest absolute residuals, its corresponding cases and
elemental fits.
Using this elemental fit as starting values for concentration step.
Repeating step 1 to 4, N times.
Out of N, best reported candidate is obtained as Hybrid FAST-LTA solution.
The performance of the proposed Hybrid FAST-LTA will be assessed through
simulation study to be conducted in next section.
4. SIMULATION STUDY
Simulations study in this section is performed using R software. Michealis-Menten
model is used for three different sample sizes        and different
proportions of outliers (0%, 20%, and 40%) were considered. Each model is repeated 500
times, and mean square error and bias are calculated for LTA, LS and M estimate.
4.1 Michaelis-Menten Model
In biochemistry, Michaelis-Menten model is used to study the relationship between
reaction velocity V and concentration of substrate S as
Dost Muhammad Khan et al.
229
 

 
In this model,
 is a constant representing maximum velocity at saturation of
substrate concentration. Michaelis constant shows concentration of substrate at
which reaction time is 50% of
. It also offers substrate concentration measure
for significant catalysis. Furthermore, the parameter increases as the efficiency
between substrate and enzymes decreases (Berg et al., 2002).
Consider two  Michaelis-Menten model

  
The data has been simulated by taking a sample from Puromycin data set used in
Batts and Watts (1988), where ,   are the parameter values;
using these values a sample size of 50, 100 and 500 were generated with x values,
uniformly spaced over the range (0.02,1.1). Normal random errors with mean 10 and
standard deviation 5 were added to the regression model. The data generation process has
been adopted from Hawkins and Khan (2009). Then 0%, 20% and 40% outliers were
incorporated in y-direction. The results of least square LS, M estimate and proposed least
trimmed absolute residuals LTA are exhibited in Table 1.
Table 1
Mean Square Errors (MSE) and Bias for Michaelis-Menten Model
with Percentage Contamination
Sample
size
Procedure
MSE/BISE
20%
40%
n=30
Least
Squares
MSE
25.09
0.000016
993.77
0.0124
4800.79
0.068
BIAS
4.68
0.0023
31.26
0.11
69.02
0.26
M
Estimator
MSE
24.94
0.000016
54.62
0.00017
6744.67
0.097
BIAS
4.65
0.0024
6.92
0.0114
81.78
0.31
Hybrid
LTA
MSE
36.51
0.000055
36.38
0.000048
466.27
0.0077
BIAS
4.65
0.0024
4.83
0.0021
7.12
0.0085
n=50
Least
Squares
MSE
22.92
0.0000108
966.51
0.012
4736.42
0.067
BIAS
4.61
0.0023
30.92
0.109
68.66
0.2588
M
Estimator
MSE
22.93
0.000013
51.57
0.00015
6672.05
0.094
BIAS
4.54
0.0023
6.91
0.0114
81.47
0.307
Hybrid
LTA
MSE
30.77
0.000042
28.66
0.0000312
25.92
0.000025
BIAS
4.46
0.0025
4.54
0.0025
4.55
0.0025
n=100
Least
Squares
MSE
22.48
0.00001
962.97
0.0119
4739.75
0.06
BIAS
4.65
0.0023
30.96
0.108
68.75
0.25
M
Estimator
MSE
22.48
0.00001
50.875
0.00014
6698.97
0.093
BIAS
4.64
0.0023
7.011
0.115
81.72
0.31
Hybrid
LTA
MSE
28.32
0.000025
26.56
0.000022
24.19
0.000015
BIAS
4.65
0.0024
4.63
0.0023
4.59
0.024
An Efficient and High Breakdown Estimation Procedure
230
It can be seen clearly from Table 1 that with clean data LS, M and LTA estimator
give similar results for small as well as for large samples. As 20% outliers are
incorporated in direction, proposed method outperforms the least square estimator, but
produced the same results as that of existing robust methods. While increasing the level
of contamination up to 40%, our proposed estimator improves on its other counterpart.
4.2 Gompertz Model
Gompertz model was initially developed by Benjamin Gompertz in 1832 to fit human
mortality data (Missov et al., 2015). Later on, it was successfully used in predicting
cancer tumor growth. It is a three parameter model given by the following relationship
    
In this model, is the tumor size and is the fraction of breast cancer people with
metastases. The data has been simulated by the pattern of Tumor growth data used in
Tabatabai et al. (2014). Where  ,  and   are the parameter
values. Using these values a sample size of 30, 50, and 100 were generated with values
uniformly spaced over the range (20,160). Normal random errors with mean 1 and
standard deviation 0.5 were added to the regression model. Then 0%, 20% and 40%
outliers were incorporated in -direction. The results of least square LS, M estimator
and proposed Hybrid least trimmed absolute residuals LTA are exhibited in Table 2.
Table 2
Mean Square Errors (MSE) and Bias for Gompertz Model
with Percentage Contamination
N
Sample Size
0%
20%
40%
n=30
LS
MSE
1.12
0.0535
2.45e-6
32.09
0.43
0.00033
105.59
0.0749
0.00046
BIAS
1.042
0.196
0.00066
5.63
0.65
0.018
10.23
0.266
0.0214
M
Estimator
MSE
1.12
0.055
2.62e-6
3.49
0.29
6.49e-5
105.59
0.0749
0.00046
BIAS
1.04
0.198
0.00697
1.82
0.53
0.0077
10.23
0.266
0.0214
Hybrid
LTA
MSE
1.99
0.297
2.3e-5
1.71
0.22
1.45e-5
1.38
0.078
5.41e-6
BIAS
1.34
0.489
0.0038
1.26
0.40
0.0029
1.14
0.24
0.0013
n=50
LS
MSE
1.09
0.05
1.88e-6
97.496
0.22
0.00047
108.39
0.078
0.00046
BIAS
1.03
0.203
0.007
9.84
0.46
0.022
10.38
0.28
0.02
M
Estimator
MSE
1.10
0.05
1.93e-6
3.44
0.325
6.67e-5
108.39
0.078
0.00046
BIAS
1.03
0.203
0.0007
1.83
0.56
0.008
10.38
0.28
0.02
Hybrid
LTA
MSE
1.87
0.24
1.73e-5
1.38
0.106
6.96e-6
1.26
0.084
4.76e-6
BIAS
1.298
0.456
0.0034
1.13
0.284
0.0016
1.09
0.252
0.0013
n=100
LS
MSE
1.088
0.043
1.12e-6
33.71
0.45
0.00034
110.05
0.075
0.0005
BIAS
1.037
0.196
0.0006
5.79
0.667
0.018
10.47
0.27
0.02
M
Estimator
MSE
1.086
0.044
1.16e-6
3.45
0.317
6.55e-5
110.05
0.075
0.0005
BIAS
1.036
0.196
0.00065
1.84
0.56
0.008
10.47
0.27
0.02
Hybrid
LTA
MSE
1.65
0.204
1.27e-5
1.387
0.145
7.59e-6
1.175
0.069
3.16e-6
BIAS
1.25
0.425
0.0031
1.15
0.346
0.0022
1.065
0.233
0.001
Dost Muhammad Khan et al.
231
It is apparent from the results displayed in Table 2 that with clean data set LS, M
estimate gives minimum mean square error (MSE) and bias, while Hybrid LTA
 MSE and bias is a bit higher than both for small sample as well as for
increasing sample size. It is concluded that LS and M estimate are efficient in case of
clean data. Perturbing the data with 20% outliers, M estimate and Hybrid LTA shows
better performance than LS. While in case of 40% contamination, our proposed estimator
beats its other counterparts.
5. APPLICATIONS ON REAL DATASETS
The performance of Hybrid LTA is illustrated using nonlinear examples from the
field of biochemistry and medicines.
5.1 Michaelis-Menten Model
Michaelis-Menten model, used by Stromberg (1993) and Tabatabai et al. (2014),
expresses the reaction velocity as a function of concentration of substrate as

  
where response variable yi is velocity and predictor variable xi is substrate; the parameter
is the maximum reaction velocity and denotes concentration of substrate. The
Puromycin data, taken from Bates and Watts (1988) which was primarily used by Treloar
(1974), consists of 12 observations given below in Table 3.
Table 3
Concentration of Substrate versus Reaction Velocity
Concentration
Ppm
0.02
0.02
0.06
0.06
0.11
0.11
0.22
0.22
0.56
0.56
1.10
1.10
Velocity
Counts/min
76
47
97
107
123
139
159
152
191
201
207
200
The speed of an enzymatic reaction depends on the concentration of a substrate. As
outlined in Bates and Watts (1988), an experiment was conducted to inspect how a
treatment of the enzyme with an additional substance called Puromycin affects the
reaction speed. To check the performance of our proposed estimator, with least square
and M estimator, model is fitted on clean data as well as contaminated data. Twenty
percent outliers are incorporated in X space, Y space and both XY space. In X space,
shifting the value in observation 5,6,7 from 0.11, 0.11, 0.22 to 2, 2.01, 2.05. In the Y
space, changing the Y value in observation 10, 11,12 from 201, 207, 200 to 65, 50, 60.
And in both XY space, transforming Y value in observation 6, 8, 11 from 139, 191, 207 to
85, 100, 80 and X value in observation 6,8,11 from 0.11, 0.56, 1.10 to 0.65, 0.61, 0.6.
Table 3 shows standard error of estimate (SE) and residuals standard errors (RSE) for LS,
M and Hybrid LTA estimators. By examining the SE and RSE, it is clear that Hybrid
LTA is more efficient than its counterpart. The results are also presented through graphs
in figure 1 in which (a) presents clean data (b) shows the effect of outliers in X space
(c) demonstrates the pattern of contamination in XY space and (d) displays the
consequences of outliers in both Y space. From these graphs, it is apparent that in clean
An Efficient and High Breakdown Estimation Procedure
232
data set all three estimators perform well but when outliers enter in X,Y and XY space,
the least square becomes unacceptable whereas M estimator and Hybrid LTA execute
well.
Table 4
Standard Errors of Estimate and Residuals Standard Errors
Procedure
Clean
Data
Outliers
in X space
Outliers
in Y space
Outliers
in XY space
0
1
RSE
0
1
RSE
0
1
RSE
0
1
RSE
Least
Squares
6.94
0.008
10.93
12.2
0.015
29.2
20.0
0.014
46.7
19.20
0.018
41.3
M
Estimator
6.45
0.008
8.500
13.8
0.017
30.03
24.2
0.017
51.9
24.35
0.022
34.9
Hybrid
LTA
2.65
0.004
3.390
3.9
0.005
5.324
7.55
0.005
6.31
6.32
0.006
6.09
Figure 1: Michaelis-Menten Model; (a, b, c, d)
5.2 Gompertz Model
Gompertz model is a sigmoid function that is the slowest at the ends and fastest at the
middle. For the first time, it was successfully used by Laird (1964) to analyze the growth
of cancer tumor. Gompertz Model is of the form:
    
Dost Muhammad Khan et al.
233
It is frequently used for screening cancer regression and progression. The data in the
Table 5 comprise of 12 observations. This data is taken from Tabatabai et al. (2014)
which was primarily collected by Tubiana & Koscielny (1990). The given data is clean as
there is no outlier in it. Model is fitted on this data using Hybrid LTA estimator, M
estimator and least square. Then outliers are inserted in X direction, both XY direction and
in Y direction. In X space shifting the value in observation 12 from 90 to 9. In the Y
space changing the Y value in observation 6 from 0.55 to 3. And in both XY space
transforming Y value in observation 7 from 0.56 to 3 and X value in observation 12 from
90 to 2. Table 6 shows standard error of estimate (SE) and residuals standard errors
(RSE) for LS, M and Hybrid LTA estimators. By examining the SE and RSE, it is clear
that Hybrid LTA is more efficient than its other counterparts. The results are also
presented through graphs in figure 2 in which (a) presents clean data (b) shows the effect
of outliers in X space (c) demonstrates the pattern of contamination in XY space and
(d) displays the consequences of outliers in both Y space. From these graphs it is apparent
that in clean data set all three estimators perform well, but when outliers enter in
X,XY and Y space, the least square becomes unacceptable whereas M estimator and
Hybrid LTA executes well.
Table 5
Tumor Size Versus Fraction Metastasized Data
Tumor Size
x
12
17
17
25
30
39
40
50
60
70
80
90
Fraction
Metastasized y
0.13
0.20
0.27
0.45
0.42
0.55
0.56
0.66
0.78
0.83
0.81
0.92
Table 6
Standard Errors of Estimate and Residuals Standard Errors
Clean data
Outliers in X space
Outliers in XY space
Outliers in Y space
1
2
3
RSE
1
2
3
RSE
1
2
3
RSE
1
2
3
RSE
LS
0.05
0.40
0.007
0.044
2.3e6
3.03e2
4.1e-2
0.224
0.65
3.3
0.13
0.8
0.3
279
0.3
0.73
M
0.05
0.40
0.007
0.040
0.06
0.408
0.008
0.040
0.08
0.7
0.01
0.06
0.6
0.7
0.01
0.06
LTA
0.01
0.11
0.001
0.008
0.03
0.174
0.003
0.012
0.03
0.4
0.004
0.02
0.02
0.16
0.002
0.01
An Efficient and High Breakdown Estimation Procedure
234
Figure 2: Gompertz Model (a,b,c,d)
6. CONCLUSIONS
In this study, Hybrid FAST LTA estimator has been presented for univariate
nonlinear models. It is hoped that this study offers an introduction to Hybrid LTA
estimation in nonlinear regression. It makes the nonlinear robust fitting possible in usual
practice. It is expected that this algorithm may be adopted for other nonlinear robust
fitting procedures, for instance S estimators, M estimators and rank based estimators.
It has been concluded that Hybrid LTA estimator performs well in contrast to LS and
M estimator in case of contaminated data set, with level of contamination upto 40
percent. Through simulation study, it has been shown that Mean Square Error and Bias of
Hybrid LTA is less than LS and M estimator when the data is polluted with 40% outliers,
and for 20% outliers Hybrid LTA and M estimators are equally efficient to LS. In regard
to data sets, with 0% contamination LS performs well as compared to Hybrid LTA
estimator.
The introduction of robust techniques and its applications to real data has made high
breakdown estimation procedures more attractive to researchers rather than sensitive
procedures. Applications to real world data such as Puromycin data and Tumor growth
data show that Hybrid LTA estimator can also be used in diverse fields of medical
sciences.
Dost Muhammad Khan et al.
235
REFERENCES
1. Abebe, A. and McKean, J.W. (2013). Weighted Wilcoxon Estimators in Nonlinear
Regression. Australian & New Zealand Journal of Statistics, 55(4), 401-420.
2. Barreto, H. and Maharry, D. (2006). Least median of squares and regression through
the origin. Computational Statistics & Data Analysis, 50(6), 1391-1397.
3. Batts, D.M. and Watts, D.G. (1988). Nonlinear Regression Analysis and its
Application. John Wiley & Sons, New York.
4. Berg, J.M., Tymoczko, J.L., Stryer, L. and Stryer, L. (2002). Biochemistry. New
York: W.H. Freeman.
5. Brown, A. (2001). A step-by-step Guide to Non-linear Regression Analysis of
Experimental Data using a Microsoft Excel Spreadsheet. Computer Methods and
Programs in Biomedicine, 65(3), 191-200.
6. Chen, Y., Stromberg, A.J. and Zhou, M. (1997). The least trimmed squares estimate
in nonlinear regression. Technical Report Department of Statistics, University of
Kentucky, Lexington, KY, 40506.
7. Cizek, P. (2001). Nonlinear least trimmed squares. SFB Discussion Paper, Humboldt
University, 78-86.
8. Cizek, P. (2002). Robust estimation in nonlinear regression and limited dependent
variable models. CERGE-EI Working Paper, 189.
9. Ekblom, H. and Madsen, K. (1989). Algorithms for non-linear Huber estimation.
BIT Numerical Mathematics, 29(1), 60-76.
10. Gilbert, S. (2007). Using SAS Proc NLMIXED for Robust Regression. Statistics and
Data Analysis, 181, 1-9.
11. Hawkins, D.M. and Olive, D.J. (1999). Improved feasible solution algorithms for high
breakdown estimation. Computational Statistics and Data Analysis, 30(1), 1-11.
12. Hawkins, D. and Khan, D. (2009). A procedure for robust fitting in nonlinear
regression. Computational Statistics and Data Analysis, 53(12), 4500-4507.
13. Hawkins, D. and Olive, D. (1999). Applications and Algorithms for Least Trimmed
Sum of Absolute Deviation Regression. Computational Statistics and Data Analysis,
32(2), 119-134.
14. Jaeckel, L. (1972). Estimating Regression Coefficients by Minimizing the Dispersion
of the Residuals. The Annals of Mathematical Statistics, 43, 1449-1458.
15. Jureckova, J. (1971). Nonparametric estimate of regression coefficients. Annals of
Mathematical Statistics, 42(4), 1328-1338.
16. Khalil, A., Ali, A., Khan, S., Khan, D. M. and Khalil, U. (2013). A New Efficient
Redescending M-Estimator: Alamgir Redescending M-Estimator. Research Journal
of Recent Sciences, 2(8), 79-91.
17. Laird, A.K. (1964). Dynamics of Tumor Growth. British Journal of Cancer, 19, 278-
291.
18. Missov, T.I., Lenart, A., Nemeth, L., Canudas-Romo, V. and Vaupel, J.W. (2015).
The Gompertz force of mortality in terms of the modal age at death. Demographic
Research, 32, 1031-1048.
19. Motulsky, H. and Ransas, L.A. (1987). Fitting Curves to Data using Nonlinear
Regression: A practical and Nonmathematical Review. Official Publication of the
Federation of American Societies for Experimenal Biology, 1(5), 365-374.
An Efficient and High Breakdown Estimation Procedure
236
20. Neugebauer, S.P. (1996). Robust analysis of M-estimators of nonlinear models.
Unpublished Doctoral dissertation, Virginia Tech.
21. Rousseeuw, P. and Driessen, V. (2006). Computing LTS Regression for Large Data
Sets. Data Mining and Knowledge Discovery, 12(1), 29-45.
22. Rousseeuw, P. and Leroy, A. (1987). Robust Regression and Outlier Regression. New
York: Wiley.
23. Sakata, S. and white, H. (2001). S-Estimation of Nonlinear Regression Models with
Dependent and Heterogeneous Observation. Journal of Econometrics, 103(1), 5-72.
24. Stromberg, A.J. (1993). Computation of high breakdown nonlinear regression
parameters. Journal of the American Statistical Association, 88(421), 237-244.
25. Stromberg, A. and Rupert, D. (1992). Breakdown in Nonlinear Regression. Journal of
the American Statistical Association, 87(420), 991-997.
26. Tabatabai, M. and Argyros, I. (1993). Robust Estimation and Testing for General
Nonlinear Regression Methods. Applied Mathematics and Computation, 57(1),
85-101.
27. Tabatabai, M., Kengwoung-Keumo, J., Eby, W., Manne, U., Fouad, M. and Singh, K.
(2014). A New Robust Method for Nonlinear Regression. Journal of Biometrics &
Biostatistics, 5(5), 211.
28. Treloar, M.A. (1974). Effects of Puromycin on Galactosyltransterase in Golgi
Membranes. M.Sc. Thesis, University of Toronto.
29. Tubiana, M. and Koscielny, S. (1990). The Natural History of Human Breast Cancer:
Implications Fora Screening Strategy. International Journal of Radiation Oncology*
Biology* Physics, 19(5), 1117-1120.
30. Verboon, P. (1993). Robust nonlinear regression analysis. British Journal of
Mathematical and Statistical Psychology, 46(1), 77-94.
... It is equivalent to searching for finding w observations having smallest residuals and applying LS fit to these w observations. For the value of w � (n/2) + 1 the breakdown point for LTS estimator is 50%, i.e., BDP � 0.50 [9]. Even though the LTS estimator has more resistance to outliers, it suffers from the problem of very low relative efficiency [10]. ...
Article
Full-text available
Robust regression is an important iterative procedure that seeks analyzing data sets that are contaminated with outliers and unusual observations and reducing their impact over regression coefficients. Robust estimation methods have been introduced to deal with the problem of outliers and provide efficient and stable estimates in their presence. Various robust estimators have been developed in the literature to restrict the unbounded influence of the outliers or leverage points on the model estimates. Here, a new redescending M-estimator is proposed using a novel objective function with the prime focus on getting highly robust and efficient estimates that give promising results. It is evident from the results that, for normal and clean data, the proposed estimator is almost as efficient as ordinary least square method and, however, becomes highly resistant to outliers when it is used for contaminated datasets. The simulation study is being carried out to assess the performance of the proposed redescending M-estimator over different data generation scenarios including normal, t-distribution, and double exponential distributions with different levels of outliers’ contamination, and the results are compared with the existing redescending M-estimators, e.g., Huber, Tukey Biweight, Hampel, and Andrew-Sign function. The performance of the proposed estimators was also checked using real-life data applications of the estimators and found that the proposed estimators give promising results as compared to the existing estimators.
... LTS is more efficient than least median squares (LMS), but its computational procedure is more complicated as compared to LMS. It has a greater convergence rate [24,25]. e LTS procedure first identifies the outliers as the points with extreme positive or negative residuals. ...
Article
Full-text available
Consistent estimation techniques need to be implemented to obtain robust empirical outcomes which help policymakers formulating public policies. Therefore, we implement the least squares (LS) and the high breakdown robust least trimmed squares (LTS) regression techniques, while using econometric regression model based on a growth equation for the two countries, namely, India and Pakistan. We used secondary annual time series data which covers a long period of 41 years. The adequacy of the time series econometric model was checked through cointegration analysis and found that there is no spurious regression. Classical and robust procedures were employed for the estimation of the parameters. The empirical results reveal that the overall fit of the model improves in case of LTS technique, while the significance of the predictors changes significantly in cases of both countries due to the removal of outliers from the data. Thus, empirical findings exhibit that the results, obtained through LTS, are better than LS techniques.
Article
Full-text available
Regression analysis is widely applied in many fields of science to estimate important variables. In general, nonlinear regression is a complex optimization problem and presents intrinsic difficulties in estimating reliable parameters. Nonlinear optimization algorithms commonly require a precise initial estimate to return reasonable estimates. In this work, we introduce a new hybrid algorithm based on the association of a genetic algorithm with the Levenberg–Marquardt method (GALM) to adjust biological nonlinear models without knowledge of initial parameter estimates. The proposed hybrid algorithm was applied to 12 nonlinear models widely used in forest sciences and 12 databases under varying conditions considering classic hypsometric relationships to evaluate the robustness of this new approach. The hybrid method involves two stages; the curve approximation process begins with a genetic algorithm with a modified local search approach. The second stage involves the application of the Levenberg–Marquardt algorithm. The final performance of the hybrid method was evaluated using total fitting for all tested models and databases, confirming the reliability of the proposed algorithm in providing stable parameter estimates. The GA was able to predict the initial parameters, which assisted the LM in converging efficiently. The developed GALM method is effective, and its application is recommended for biological nonlinear analyses.
Article
Most algorithms for estimating high breakdown regression estimators in linear regression rely on finding the least squares fit to many p-point elemental sets, where p is the dimension of the X matrix. Such an approach is computationally infeasible in nonlinear regression. This article presents a new algorithm for computing high breakdown estimates in nonlinear regression that requires only a small number of least squares fits to p points. The algorithm is used to compute Rousseeuw's least median of squares (LMS) estimate and Yohai's MM estimate in both simulations and examples. It is also used to compute bootstrapped and Monte Carlo standard error estimates for MM estimates, which are compared with asymptotic standard errors (ASE's). Using the PROGRESS algorithm for a two-parameter nonlinear model with sample size 30 would require finding the least squares fit to 435 two-point subsets of the data. In the settings considered in this article, the proposed algorithm performs just as well with 25 as with 435 least squares fits, thus substantially reducing computation time. Using the new algorithm to compare standard error estimates for MM estimates reveals that, at least in the setting considered here, bootstrapped and to a lesser extent ASE's become increasingly unreliable as the percentage of outliers increases, In one example presented, the high breakdown estimates are useful in determining that a point appearing to be an outlier in the least squares analysis probably should not be considered an outlier. In the other example, the least squares analysis reveals no outliers and suggests that the model is inappropriate for the data. The residual plots using high breakdown estimates reveal that two points are outliers, and the model provides a reasonable fit to the remainder of the data.
Article
In this paper, we consider the estimation of parameters of a general near regression model. An estimator that minimises the weighted Wilcoxon dispersion function is considered and its asymptotic properties established under mild regularity conditions similar to those used in least squares and least absolute deviations estimation. As in linear models, the procedure provides estimators that are robust and highly efficient. The estimates depend on the choice of a weight function and diagnostics which differentiate between nonlinear fits are provided along with appropriate benchmarks. The behavior of these estimates is discussed on a real data set. A simulation study verifies the robustness, efficiency and validity of these estimates over several error distributions including the normal and a family of contaminated normal distributions.
Article
In this article, τ-estimates of parameters of general nonlinear models in finite dimensional spaces are derived. The method is highly insensitive to outliers. It can also be applied to solving a system of nonlinear equations. A probabilistic testing procedure is proposed and the method is supplemented by a numerical example.
Article
The breakdown point is considered an important measure of the robustness of a linear regression estimator. This article addresses the concept of breakdown in nonlinear regression. Because it is not invariant to nonlinear reparameterization, the usual definition of the breakdown point in linear regression is inadequate for nonlinear regression. The original definition of breakdown due to Hampel is more suitable for nonlinear problems but may indicate breakdown when the fitted values change very little. We introduce breakdown functions, which measure breakdown of the fitted values. Using the breakdown functions, we introduce a new definition of the breakdown point. For the linear regression model, our definition of the breakdown point coincides with the usual definition for linear regression as well as with Hampel's definition. For most nonlinear regression functions, we show that the breakdown point of the least squares estimator is 1/n. We prove that for a large class of unbounded regression functions, the breakdown point of the least median of squares or the least trimmed sum of squares estimator is close to ½. For monotonic regression functions of the type g(α + βx), where g is bounded above and/or below, we establish upper and lower bounds for the breakdown points that depend on the data.
Article
The estimation of regression models is often based on the least squares method, which is very sensitive to misspecification and data errors. An alternative estimation approach is based on the theory ofrobust statistics, which builds upon parametric specification, but provides a methodology for designing misspecification-proof estimators. However, this concept, developed in statistics, has so far been applied almost exclusively to linear regression models. Therefore, I adapt the least trimmed squares estimator to nonlinear regression models. This paper presents the adapted robust estimator and the proofofitsconsistency. Additionally,Iderivetheasymptotic distributionofthe nonlinear least trimmed squares including its asymptotic variance. Zusammenfassung. In der Regressionsanalyse werden hufig die Methoden, die auf dem Prinzip der kleinsten Quadrate beruhen, eingesetzt, obwohl sie sehr empfindlich gegen Datenfehler und Fehlspezifikation des Modells sind. Im Ge- gensatz dazu ermöglicht es die Theorie der robusten Statistik, robuste parame- trische Schätzer zu entwerfen (zum Beispiel, die Least Trimmed Squares (LTS) Methode). Diese robuste Methoden sind aber fast ausschlieslich nur fur lineare Modellegeeignet. UmdenLTSSchätzerfurdienichtlineareRegressionzuadap- tieren, verallgemeinere ich den LTS fur nichtlineare Modelle und beweise, dass der LTS Schutzer in diesen Follen konsistent ist. Schlieslich leite ich auch die asymptotische Verteilung des LTS inklusive der asymptotischen Varianz ab.