6
$\begingroup$

Here is classic linear mixed effects regression:

$$y_{ij} = \beta_0 + \beta_1 x_{ij} + u_j + \epsilon_{ij}$$

If the parameters of this model are estimated using frequentist method, a complex estimation procedure is needed involving GLS and RMLE to avoid biased estimation of variance components. This is because of problems with degrees of freedom.

If a Bayesian method is used to estimate the parameters of this model (e.g. MCMC, Gibbs Sampling), is the frequentist problem biased variance estimation avoided completely (since we are now dealing with posterior distributions vs point estimates)?

$\endgroup$

2 Answers 2

6
$\begingroup$

Bayesian estimators still have bias, etc.

Bayesian estimators are generally biased because they incorporate prior information, so as a general rule, you will encounter more biased estimators in Bayesian statistics than in classical statistics. Remember that estimators arising from Bayesian analysis are still estimators and they still have frequentist properties (e.g., bias, consistency, efficiency, etc.) just like classical estimators. You do not avoid issues of bias, etc., merely by using Bayesian estimators, though if you adopt the Bayesian philosophy you might not care about this.

There is a substantial literature examining the frequentist properties of Bayesian estimators. The main finding of importance is that Bayesian estimators are "admissible" (meaning that they are not "dominated" by other estimators) and they are consistent if the model is not mis-specified. Bayesian estimators are generally biased but also generally asymptotically unbiased if the model is not mis-specified.

$\endgroup$
4
  • $\begingroup$ I suppose that makes sense. The mean and variance of the posterior distributions will have bias. I was just confused about the following: in introductory texts on linear mixed effects regression, there is a huge emphasis on RMLE and the motivation is clearly explained (problem of biased variance estimation). But I have not encountered mentions of this biased estimation problem in Bayesian linear mixed effects regression. Perhaps this specific source of bias present in the frequentist for mixed effect regression setting is not there in bayesian, but there are instead other sources of bias? $\endgroup$ Commented yesterday
  • $\begingroup$ "you will encounter more biased estimators in Bayesian statistics than in classical statistics" I found this sentence interesting :) $\endgroup$ Commented yesterday
  • $\begingroup$ I never gave it much thought before. In frequentist, unbiasedness: E(thetha hat) = thetha , consistency: thetha_n -> thetha for n large, and efficiency is Var(thetha_hat). In Bayesian, there is a Posterior distribution of thetha-hat. I wonder how unbiasedness, consistency and effeciency are even defined in the bayesian setting? $\endgroup$ Commented yesterday
  • $\begingroup$ Consider Edwin Jaynes' sobering observation: "we demonstrate in Chapter 17, that a ‘biased’ estimate may be considerably closer to the truth than an ‘unbiased’ one, an ‘inadmissible’ procedure may be far superior to an ‘admissible’ one..." (page 508 of his book Probability Theory, the Logic Of Science). $\endgroup$ Commented 12 hours ago
3
$\begingroup$

In a sense, yes, you are making a lot of things simpler by being Bayesian in this particular case. If you manage (given the prior and the likelihood) to sample the posterior, then inference based on the posterior samples is simple. If everything worked well, it all ends up being pretty straightforward. It's a frequent (no pun intended) occurence that frequentist procedures for certain situations like repeated measures, multiple-rater-multiple-case studies, small sample confidence intervals, inference with zero counts etc. can get really complicated and need very case specific solutions. In contrast Bayesian solutions tend to look very similar for many different settings, just "write down likelihood, specify prior, get posterior".

Of course, it's not quite that simple. E.g. you need to pick priors in some way (and with flat improper priors you need to check the posterior even exists, also flat proper priors can have poor properties, too). Then, you need to be sure the typical MCMC machinery actually got you "good" samples of the posterior. E.g. using NUTS via Stan or something like that nowadays has better diagnostics for this than we used to have. Still, you might find out that it's hard to resolve issues or there can still be cases where you don't notice issues. There's various well-known tricks that can help (e.g. centered vs. non-centered parameterization for hierarchical models, how to map things like a simplex to an unconstrained parameter space in a way that behaves well, how to deal with discrete parameters by, say, integrating them out...), which you might argue are a similar thing as complicated special frequentist procedures.

$\endgroup$
2
  • $\begingroup$ The first paragraph you wrote is so true! If you look at the first question I asked, I asked about using weakly informative priors to avoid deriving the matrix algebra/calculus needed for a pure frequentist approach - this way, I get very similar parameter estimates to frequentist without the headaches of frequentist. It turns out that many people think this way :) $\endgroup$ Commented 11 hours ago
  • $\begingroup$ To be honest, this is the first time I have heard about centered vs non-centered parametrization in hierarchical models. Care to explain? $\endgroup$ Commented 11 hours ago

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.