My (Chiffon) Nguyen

sensitivity analysis

4 min read

Examine how much the conclusions/results change as the result of changing assumptions/conditions . We focus on the results

In causal inference

Sensitivity analysis is a critical component of causal inference that helps assess the robustness of individual treatment effect to potential sources of bias and unobserved confounders. It involves systematically varying assumptions, model specifications, or parameters to examine how sensitive the causal inference is to different scenarios.

The sensitivity analysis asks how much hidden bias can be present before the qualitative conclusion of the study begin to change.

  • A result that WAS statistically significant when we do the analysis the usual way might STOP BEING STATISTICALLY SIGNIFICANT.
  • We could have a treatment effect that was non-zero, but it becomes 0 if we were controlling for that unobserved confounder.

Why do we need sensitivity analysis in causal inference

Assessing Assumption Robustness: Causal inference often relies on strong assumptions, such as the consistency, exchangeability, or positivity assumptions. Some assumptions are untestable due to unobserved confounders (they were never present nor measured). Sensitivity analysis allows researchers to test how sensitive their conclusions are to potential violations of these assumptions.

Controlling for Unmeasured Confounders: Confounders influence both treatment and outcome. Sensitivity analysis helps estimate the potential impact of unmeasured confounding on causal estimates.

  • How “strong” would a confounder (or some confounders) need to be to change the conclusion of a study?
  • In a worst case scenario, how vulnerable the study’s result is to all unobserved confounders acting together?
  • Are these confounders or scenarios plausible? How strong would they have to be relative to observed covariates in order to be problematic?
  • How can we present these sensitivity analysis results concisely

Understanding Potential Biases: Various biases, such as selection bias or measurement error, can impact causal inference. Sensitivity analysis can quantify the extent to which these biases may affect causal estimates.

Evaluating Heterogeneous Effects: Causal effects may vary across different subpopulations or settings. Sensitivity analysis can explore how causal estimates change for specific subgroups or under different contextual conditions.

What sensitivity analysis cannot do

  • Cannot identify which unobserved confounder - we cannot measure the unknowns
  • Sensitivity analysis can help assess the impact of known sources of bias and potential hidden biases under specified assumptions. However, it cannot identify or address all possible sources of hidden bias, especially those that are truly unobserved and unmeasurable.

R code

library(sensemakr)
data("darfur")
 
darfur.model <- lm(peacefactor ~ directlyharmed + female + village, data=darfur)
 
# investigate confounders that are 1,2,3 times stronger in relation to the treatment outcome, compared to observed covariate "female"
darfur.sensitivity <- sensemakr(model = darfur.model, treatment = "directlyharmed", benchmark_covariates = "female", kd = 1:3)
 
ovb_minimial_reporting(darfur.sensitivity, format = "latex")
 
plot(darfur.sensitivity, sensitivity.of = "t-value")

Example: Interpretation

  • the partial R squared of the treatment with the outcome
  • The robustness value for bringing the point estimate of directlyharmed exactly to zero is 13.9% . This means that unobserved confounders that explain 13.9% of the residual variance both of the treatment and of the outcome are sufficiently strong to explain away all the observed effect. On the other hand, unobserved confounders that do not explain at least 13.9% of the residual variance both of the treatment and of the outcome are not sufficiently strong to do so.
  • The robustness value for testing the null hypothesis that the coefficient of directlyharmed is zero falls to 7.6%. This means that unobserved confounders that explain 7.6% of the residual variance both of the treatment and of the outcome are sufficiently strong to bring the lower bound of the confidence interval to zero (at the chosen significance level of 5%). On the other hand, unobserved confounders that do not explain at least 7.6% of the residual variance both of the treatment and of the outcome are not sufficiently strong to do so.

If we actually had an unobserved variable that is 3-4 times as strong as female (observed), it would bring the treatment down to 0 (the red contour). It’s quite unlikely we were not controlling for such a confounder, that is uncorrelated with any observed covariate, so we should believe in the robustness of our conclusion.

However, if there was an unobserved variable that is twice as important as female, it would make our treatment statistically insignificant (gets beyond red contour).

Graph View

Backlinks