Logistic Regression - Science topic

A logistic model can be linear or non-linear. If you use a linear predictor function, then it's a "linear model". But you can also use a nonlinear predictor function, in which case it is a "non-linear model". The predictor function is linket to the expected value (µ) by a link function. In a logistic model this link function is (usually) the logit.

Example:

logit(µ) = β₀ + β₁*X₁ + ... β_k*X_k

is linear (in the parameters β - which are all simple coefficients here)

logit(µ) = sin( β₀ + β₁*X₁ )^β₃ + β₄*X₂

is not linear (not all parameters β can be expressed as simple coefficients).

The same relation between linear and nonlinear applies to other link functions (here it was the logit function), like the log ("log-linear" or "Poisson" model), the inverse, or the identity function (that is, the linear predictor represents µ directly; no transformation).

Hello Puneet,

Like Jamie Wallis , I don't attach any special meaning to "adjusted multivariate logistic regression" vs. "multivariate logistic regression;" they are the same.

Two points are pertinent here, and may help your understanding.

1. A number of studies will report both the crude (one IV, one DV) odds ratios and the adjusted odds ratioss (full set of IVs, one DV) for logistic regression models. Please note that the latter can differ from the former because of overlap among IVs, as well as the potential for suppressor variables.

2. It may seem trivial, but the correct term would be "multiple" or "multivariable" logistic regression here, not "multivariate." Among statisticians, multivariate refers to the case of having more than one DV in a model; univariate refers to the case of a single DV in a model.

Good luck with your work.

I'm not exactly sure what you are asking, but a logistic regression slope coefficient of 0 (corresponding to an OR of 1) indicates no relationship. Your regression slope is not significantly different from zero at the .05 level (p = .13), and your 95% CI for the OR therefore includes 1.0. Both these values thus indicate that the relationship is not statistically significantly different from zero.

If your regression slope coefficient were significant at the .05 level (i.e., p < .05), you would expect to see a 95% CI for the OR that does not include 1.0.

Two points before I can offer an answer:

1. Are you interested in how each of the 10 are associated with the response variable, or are you interest in how well some subset of the 10 can be combined to predict the response variable? Or do you have more specific research questions like does variable A predict Y after conditioning on variable B.

2. Say more about why including all ten does not work. Is the sample small? Are the predictor variables collinear? etc.?

You should use the model that makes more sense, practically and/or theoretically. A high R² is not in indication for the "goodness" of the model. A higher R² can also mean that the model makes more wrong predictions with a higher precision.

Do not build your model based on observed data. Build your model based on understanding (theory) and the targeted purpose (simple prediction, exptrapolation (e.g. forecast), testing meaningful hypotheses etc.)

Removing a variable from the model changes the meaning of the intercept. The intercepts in the two models have different meanings. They are (very usually) not comparable. The hypothesis tests of the intercepts of the two models test very different hypotheses.

PS: a "non-significant" intercept term just means that the data are not sufficient to statistically distinguish the estimated value (the log odds given all X=0) from 0, what means that you cannot distinguish the probability of the event (given all X=0) from 0.5 (the data are compatible with probabilities larger and lower 0.5). This is rarely a sensible hypothesis to test.

Why is this panel data and what do you mean by pooled? I am attaching what may be a similar study. Best, David Booth

Apologies for the sending..I don't have any ideas why the site did that. David Booth

I think you have answered your question: "It should be noted that when I plot the data, very very few participants chose yes in response to the neutral and positive images. Almost all yes responses were given in response to the negative images."

This is what you'd expect even in a simple 2x2 design. If the probability of a yes response in the positive condition is very high and the probability very low in the negative condition then the OR could be high as its the ratio of a big probability to a very low one.

This isn't unnatural unless the raw probabilities don't reflect this pattern. (There might still be issues but not from what you described).

I agree with David Eugene Booth here. I wrote this brief explainer as a supplement to a book a few years back. I think Nagelkerke's is probably the least useful in practice as it hasn't really got an intuitive interpretation.

Understand Forward and Backward Stepwise Regression – Quantifying Health

You should not select variables by their p-value. You should build a model based on theory. If you consider a variable interesting or important enough to include it in the model, then include it. Otherwise don't.

Could it simply be a result of the analysis that the overall explanatory power of these variables is low (or lower than expected)? Perhaps it just is what it is...

That error message says that for 2 of your DVs, everyone has the same value. Try this:

TEMPORARY.

SELECT IF NMISS(x1, x2, x3) EQ 0.

FREQUENCIES y1 y2.

Replace x1, x2, x3 with the list of explanatory variables in your model(s). And replace y1 y2 with the two DVs that are causing the error message to occur.

PS- Be sure to highlight and run all 3 lines together. Do not execute the SELECT IF line separately from TEMPORARY.

I don't understand what you want to do.in any case the attached may be helpful to you. Notice the validation step here works better than a single split etc Best wishes David Booth

Any new variable will only improve R^2, but one must not introduce variables that their theoretical model does not encompass. If you have no guesses (even better if others have already discussed it) on how a new independent variable might influence the dependent variable, it's best not to include it. The more variables, the more multicolliniarity. Any additional variation that you will explain adding the year variable will be misleading.

Adding time or space variables, for that matter, is almost always a bad idea. If you run LOO tests by year, they will show that adding year only worsens the actual predictive power of your model.

However, you might want to consider including some time-dependent macroeconomic indicators that impact bankruptcy. Although they will cause more multicolliniarity. I still strongly recommend running LOO cross-validation and perhaps target oriented cross-validation, if you can handle it, to show that such indicators do not spoil your model.

I believe Blaine Tomkins meant to describe the data as having a LONG format, not a wide format. Apart from that, I concur with his advice. SPSS can estimate that model. Look up the GENLINMIXED command:

A good resource is the book by Heck, Thomas & Tabata (if you can get your hands on it):

HTH.

Hello again Lisa,

It's not clear why Test 1 result list omits ORs/p-values for so many variables, and ditto for Test 2...that's one of the reasons for asking for clarification.

Looking at "Test 2" output,

1. Having ztc significantly improves odds of success;

2. Online course significantly improves odds of success (vs. traditional);

3. Hybrid course significantly reduces odds of success (vs. traditional);

4. ztc combined with online significantly reduces odds of success (more than you would expect from the related "main effects" directions, _and_ relative to traditional);

5. ztc combined with hybrid significantly increases odds of success (more than you would expect from the related main effects directions, _and_ relative to what happens with traditional classes).

If you look at the slopes of the Test 2 output plot, you'll see that ztc appears to have the sharpest increase in likelihood of success for the hybrid condition (but that the hybrid condition is still lower in odds of success than either of the other two methods). That is completely consistent with the significance test outcomes.

As well, ztc has less influence on success likelihood for online class than for traditional class. Again, consistent with the test results.

Good luck with your summary and presentation.

Plotting observed vs predicted is not sensible here.

You don't have observed probabilities; you have observed events. You might use "Temp", "Time", and "Conc.Log10" as factors (with 4 levels) and define 128 different "groups" (all combinations of all levels of all factors) and use the proportion of observed events within each of these 128 groups. But you have only 171 observations in total. No chance to get any reasonable proportions (you would need some tens or hundreds of observation per groups for this to work reasonably well).

Ok, that's corrected.

What ist the "Response"? Is this an indicator if the virus is detected? If so, then it's hard to estimate the half-time, as both, the concentration and the sensitivity of the assay are unknown. I also wonder how Response could be 0 at Time 0.

It would be possible to fit a Cox-model (as David suggested), you could also fit a binomial model and find the roots of the predition function (on the logit scale) which is the log odds of "Response = 1" vs. "Response = 0". But it's not clear if the lod odds = 0 corresponds to a 50% survival.

In binary logistic regression, the outcome is usually coded 1/0; then I am assuming you have an exposure of interest, and all other variables are potential confounders coded as indicator variables. Assuming a large sample size relative to the number of potential confounders, you can run a model with the exposure of interest and all confounders. The adjusted OR for the exposure-disease association (controlling for all confounders), is the most valid estimate of the exposure-disease association (assuming accurate measurements, etc.). You could stop there. Depending on the research question, you could consider simplifying the model by removing some of the variables which appear to be weak confounders. Remove one of these variables, then compared the adj OR for the exposure-disease association for this reduced model with the adj OR controlling for all variables (which is considered the "gold standard"). If the reduced model OR is similar to the gold std OR (within 10%), leave that variable out. Repeat this simplification approach always comparing to the "gold std" OR; if dropping one variable has an adj OR more than 10% away from the gold standard, place that variable back into the model and assess another variable. While not necessarily a perfect approach, this change-in-indicator approach is reasonable.

Dear Zuffain

It appears you are looking to draw counter-factual plots (one predictor changing while other predictors held at a constant value - usually average). Read from page 89 onwards (GELMAN, A., & HILL, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge, Cambridge University Press.) for more details.

Here is an example from my github page: https://github.com/uhkniazi/BRC_Allergy_Oliver_PID_18/blob/42ca5db2a0b755862da4d7ac0964a3c246ce9262/04_responseVsDiversity.R#L203

However I am using monte-carlo approximation instead of the glm function in R - but you can figure it out.

Best wishes

You may find this StackExchange thread useful:

HI Muhammad Badarnee ,

I think the p-value of 0.053 is the two-tailed result, I agree with professors that it is a significant level at p-value < 0.1 or an error at 10%. However, you may check for reliability and validity of the variables, if any item could be deleted, and Cronbach's Alpha and Factor Loading are over 0.7, the p-value could be accepted at p-value < 0.05.

Take a look at the attached screenshot information. PS stop collecting data before you know what you are going to do with it. David Booth

You seem to use R's logistf function (I guess from within SPSS). Therefore you need to fine-tune related parameters. There are functions like logistf.control, logistf.mod.control, logistpl.control which makes the tuning as described in the help document (https://cran.r-project.org/web/packages/logistf/logistf.pdf).

I don't know if from within SPSS these functions are accessible, but I presume so.

You can try fine-tuning parameters using these functions to push the procedure converge.

Or, alternatively you can try other software or other penalization procedure. But the latter requires broader computation skills.

Two questions. Have you done an interaction plot? the second one is are you using logistic regression? Best wishes David Booth

You need to interpret OR using Exponential of estimates.

You might find the search in the attachment to be helpful to you. Best wishes, David Booth

I guess Your task is to define factors that predict response b (No).

If so You need all data and lodistic regression for prediction the outcome a (yes) or (b (No).

If not You sholud define DV for group b (No).

This is McFadden's R^2 (Stata calls it Pseudo R squared) and is a measure of goodness of fit. Usually 0.2 - 0.4 indicates very good fit

Following my experience, if your maps are in the format of shape-file (boundary maps) you can transform it into a raster file (grid format). Then with this data structure, you can use the R program for logistic regression.

My paper in topic "Logistic Regression Model of Built-Up Land Based on Grid-Digitized Data Structure: A Case Study of Krabi, Thailand" was explained on it.

Nicole Walentek, Yes, You may check univariate logistics regression first. Also, you can try to remove the constant from the model :)

Hi Stanislava Klymova, by including this interaction term in your model, you're assuming that the effect of being born outside the US is different for those with/without PTSD. Another way of interpreting this coefficient is the 'extra effect' of being born outside the US while having PTSD (assuming you included these predictors as binary variables with yes = 1). It may be a good idea to first run a likelihood ratio test comparing two models - one with/one without the (PTSD*Born outside the US) interaction term to test whether including this effect modification improves the fit of your model.

Sam H.Bahreini I would also consider showing ROC curves and maybe the resultes of your Hosmer-Lemeshow goodness of fit tests. I have examples of each; DM me if interested.

Just like any other regression..If you want examples simply Google your question. Best wishes David Booth

OR = exp(logOR). Plugging in your two logOR values yields the ORs you reported. Using Stata, I got this:

logOR OR

1. 4.6821 107.9966

2. -4.6821 .0092595

Those two ORs show effects of exactly the same magnitude, just as the two logOR values do. HTH.

Could you simply compare the (pointbiserial) correlations between disease (binary variables) and psychological load (continuous/metrical variable) using statistical tests for comparing dependent correlations, that is, correlation coefficients obtained from the same sample for different variables?

bellow this link

https://stats.oarc.ucla.edu/sas/dae/multinomiallogistic-regression/

Cox Regression.

Hello Michelle,

When you say, multivariate (binary) logistic regression, do you simply mean using a model with more than one independent variable/predictor? If so, it should more correctly be multivariable LR, as the usual definition of "multivariate" is inclusion of two or more dependent variables in a model.

The Box-Tidwell test may be applied if your model has one continuous IV, or if your model has multiple continuous IVs. I would assess it only for IVs that you elect to include in your final model. Whether IVs you don't intend to add to the model do or don't show the linearity. Do the check for all included continuous IVs.

Finally, I'm not a fan of automated variable inclusion (e.g., step methods, such as forward inclusion). There's a host of technical reasons underlying this, but one of the chief ones is, you are not guaranteed to identify the best ensemble of IVs for a given situation.

Good luck with your work.

I think the following references may support you

1. Regression Modeling Strategies Frank E. Harrell, Jr. With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis

2. LOGISTIC REGRESSION MODELS FOR ORDINAL RESPONSE VARIABLES ANN A. O’CONNELL University of Connecticut

If your dependent (outcome) variable is an ordinal categorical type, then ordinal logistic regression is one regression technique you may consider using. However, interpretation of the ordinal regression can be confusing; this is because the distance (difference) between one level to another is not necessarily consistent. UCLA website below provides a tutorial on ordinal regression. Even if you don't use R, just read its output and its interpretation. https://stats.oarc.ucla.edu/r/dae/ordinal-logistic-regression/

Development of scoring system for risk stratification in clinical medicine: a step-by-step tutorial

Outliers can wreck an analysis.

There are various residual diagnostics for logit models that you can use to identify the effects of outliers in the predictor variables eg Dbetai will give you the change in a particular regression coefficient if that observation was dropped. This will tell you how sensitive your model is to particular observations. You can then investigate these observations further to understand what is going on.

see section on influential observations in this

The original development was by Pregibon, D. (1981) Logistic Regression Diagnostics, Annals of Statistics, Vol. 9, 705-724.

see also

They are quite widely implemented in software.

I am in agreement with both David and Jochen.

To be clear, for LR this is not only a matter of having a "large" sample; the issue of "rare events" and how that manifests in MLE regards the absolute size of the smallest group, not its relative size. That is, you will have the same problem with a 100/10 split and a 10000/10 split. This is why various rules-of-thumb regarding N for LR are in respect to the size of the smallest group for the number of predictors. Obviously, I have no knowledge of the data, but the applicability of this depends on what is meant by "unbalanced."

Firth's clever work, although often simply used to get LR estimates when they would otherwise be intractable with MLE (complete or quasi-complete separation), is a general attempt to penalize MLE weights in respect to the "smallness" of the sample, with those penalized weights being asymptotic to the MLE wights as sample size increases. As it has a Bayesian basis, Firth's penalization of MLE can be seen as conceptually parallel to the penalization of OLS in ridge or LASSO regression estimation.

John

Hello Guo-Hua,

The two approaches you propose are identical in impact (e.g., as tactics for "statistical control"). Once the data are collected, these are about your only options.

The usual array of methods by which to deal with nuisance / noise / extraneous / confounding variables in studies includes:

1. Randomization (doesn't eliminate variables, but tends to equalize their distribution across groups, over many trials);

2. Matching (on the target variable/s; this can be challenging with multiple variables);

3. Selecting cases with a fixed value (on the target variable/s; though this restricts generalizability);

4. Building the variables into the design (e.g., as a blocking factor);

5. Statistical control (e.g., as a covariate, in the ways you have proposed).

Good luck with your work.

Hello Tuhinur,

The implied question in your query is, can a study be overpowered (by huge N) so as to flag as significant models which don't account for a meaningful amount of the observed differences (e.g., trivial effects)? The answer, of course, is "Yes."

It's always a judgment call, however, as to whether an effect is trivial.

Now, let me try to address your query. First, I would not rely on pseudo-R-square values as the measure of model adequacy for a logistic regression model. The reason is three-fold: (a) the value does not truly represent variance accounted for (as in OLS regression), so trying to adapt guidelines you may have seen for OLS models may not make sense; (b) context matters: the variables involved, the target population, the perspective of the decision-maker, and the intended use(s) of the model; and (c) many times people choose to focus on the exponentiated regression coefficients for IVs (the "OR" or odds ratio estimates) and/or the classification accuracy of the model and/or AIC/BIC indicators. For me, I'd stick with OR and classification accuracy, or information, and try to interpret those in context of my research aims.

Here's a link you may find handy that compares a number of the common pseudo R-square values: https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faq-what-are-pseudo-r-squareds/

Good luck with your work.

The 2 tests are different. David Booth

In principle, confounding variables are often known as risk factors in the literature, whereas exposure variables are what we want to test in our work.

Si votre model est multivarié il faut ajuster pour les catégories de références sur d'autres variables indépendantes incluses dans ce model.

Try stat package in R programming

It would be illuminating to know why you categorize a continuous predictor variable? Why?

Well Done !!! interesting...

David Morse D. Eastern Kang Sim

Thanks greatly for your responses - they are very helpful and much appreciated!

Suppose we are interested in understanding whether a mother’s age affects the probability of having a baby with a low birthweight.

To explore this, we can perform logistic regression using age as a predictor variable and low birthweight (yes or no) as a response variable.

Suppose we collect data for 300 mothers and fit a logistic regression model. Here are the results:

📷

To obtain the odds ratio for age, we simply need to exponentiate the coefficient estimate from the table: e0.173 = 1.189.

This tells us that an increase of one year in age is associated with an increase of 1.189 in the odds of a baby having low birthweight. In other words, the odds of having a baby with low birthweight are increased by 18.9% for each additional yearly increase in age.

This odds ratio is known as a “crude” odds ratio or an “unadjusted” odds ratio because it has not been adjusted to account for other predictor variables in the model since it is the only predictor variable in the model.

But suppose we were interested in understanding whether a mother’s age and her smoking habits affect the probability of having a baby with a low birthweight.

To explore this, we can perform logistic regression using age and smoking (yes or no) as predictor variables and low birthweight as a response variable.

Suppose we collect data for 300 mothers and fit a logistic regression model. Here are the results:

📷

Use one of the following.

A- odds ratio,

B- relative risk,

C- risk differences.

Put group on the rows and outcomes on the columns.

on the SPSS software, you can easily change your reference to the end or first

I don't understand what you did. How did you resample cases? Can you show the code that you used? In general, if no one in your sample in one category, if you are sampling from the observed data then there would be also no one in that category in any of your bootstrap samples. You could sample in such a way that there is a probability of being in different categories. However, the norm in any analysis would be to remove that category. If you had been interested in that category would likely would have done your sampling differently so that a larger proportion were in that group.

That's a very poor method of selection because it just doesn't work. Here's two papers on the subject. First is about method doesn't work and second one is about how you might deal with this in a much better way.

Best wishes, David Boot

Maybe not useful for the original question anymore, but still good to know: a very detailed exposition on how to perform a matched cohort analysis is present in Kleinbaum's Logistic Regression 3rd edition. It gives the specifics on how to do it in SPSS using the Cox regression module in the Appendix. I used it some years ago, and it works (obviously). Hope this helps.

You have one continuous, independent variable and one dichotomous dependent variable, and an unpaired t-test is appropriate

Ahmed Waqas

I suggest reading this article:

This is a free online course that allows you to learn about ordinal logistic models and much else

These are the modules

Firth's logistic regression has become a standard approach for the analysis of binary outcomes with small samples. Whereas it reduces the bias in maximum likelihood estimates of coefficients, bias towards one-half is introduced in the predicted probabilities. The stronger the imbalance of the outcome, the more severe is the bias in the predicted probabilities. We propose two simple modifications of Firth's logistic regression resulting in unbiased predicted probabilities. The first corrects the predicted probabilities by a post hoc adjustment of the intercept. The other is based on an alternative formulation of Firth's penalization as an iterative data augmentation procedure. Our suggested modification consists in introducing an indicator variable that distinguishes between original and pseudo-observations in the augmented data. In a comprehensive simulation study, these approaches are compared with other attempts to improve predictions based on Firth's penalization and to other published penalization strategies intended for routine use. For instance, we consider a recently suggested compromise between maximum likelihood and Firth's logistic regression. Simulation results are scrutinized with regard to prediction and effect estimation. We find that both our suggested methods do not only give unbiased predicted probabilities but also improve the accuracy conditional on explanatory variables compared with Firth's penalization. While one method results in effect estimates identical to those of Firth's penalization, the other introduces some bias, but this is compensated by a decrease in the mean squared error. Finally, all methods considered are illustrated and compared for a study on arterial closure devices in minimally invasive cardiac surgery.

If your estimate is 0, you can’t have any other outcome. That's why wherever the estimate is zero, those entire rows show dots. :)

You need to understand the causal relationships that are implied by your hypothesis. I suggest that you visit http://www.dagitty.net which allows you to build a graphic representation of the relationships between your variables and, based on this, will allow you to build statistical models that test informative hypotheses.

Thanks Chris and David! Sorry I might not been clear in my question! I am aware of the fact that I cannot log 0 and that this is the reason why I loose cases. My qustion is how can I test the assumption of linearity of the logit for logistic regression then? Is there another approach than the Box-Tidewell Test?

Hello Max Pierini. In trying to find the best balance between Se and Sp, you are using Youden's Index (or some variation on it). Let me remind you of what Jochen Wilhelm said in his reply (with emphasis added):

The performance of the new method can be evaluated by a ROC analysis. You may decide on a useful combination of sensitvity and specificity you can get (standard indices like the Youden index or similar are ignoring all practical consequences of false positives and false negatives and should not be used).

The only change I would make to what Jochen said is that Youden's index (or similar) should only be used when false positives and false negatives are deemed equally costly. But you want to limit false negatives (because of lethality), so trying to strike a balance between Se and Sp does not make sense. Rather, you need a cut-point that guarantees whatever level of Se you deem necessary (IMO). HTH.

Hi Salvatore,

Thanks for your suggestion.