Jeffreys prior
Introduction
In the realm of Bayesian statistics, the Jeffreys prior is a non-informative prior distribution that is invariant under reparameterization of the parameter space. Named after the British statistician Harold Jeffreys, it plays a crucial role in the objective Bayesian framework by providing a method to incorporate prior knowledge without introducing subjective bias. The Jeffreys prior is derived from the Fisher information matrix, which measures the amount of information that an observable random variable carries about an unknown parameter upon which the probability depends.
Mathematical Formulation
The Jeffreys prior is defined for a parameter \(\theta\) by the square root of the determinant of the Fisher information matrix \(I(\theta)\):
\[ \pi(\theta) \propto \sqrt{\det(I(\theta))} \]
where \(I(\theta)\) is the Fisher information matrix, given by:
\[ I(\theta) = -E\left[\frac{\partial^2 \log L(\theta)}{\partial \theta^2}\right] \]
Here, \(L(\theta)\) is the likelihood function of the parameter \(\theta\), and the expectation is taken with respect to the probability distribution of the data.
Properties
Invariance
One of the most significant properties of the Jeffreys prior is its invariance under reparameterization. This means that if \(\theta\) is transformed into another parameter \(\phi\) through a bijective function, the Jeffreys prior for \(\phi\) is given by:
\[ \pi(\phi) \propto \sqrt{\det(I(\phi))} \]
This invariance property ensures that the conclusions drawn from Bayesian analysis do not depend on the choice of parameterization.
Non-Informative Nature
The Jeffreys prior is considered non-informative because it does not favor any particular value of the parameter \(\theta\) a priori. It is designed to have minimal influence on the posterior distribution, allowing the data to speak for itself. This makes it particularly useful in situations where little or no prior information is available.
Scale Invariance
For scale parameters, the Jeffreys prior is proportional to the reciprocal of the parameter, \(\pi(\theta) \propto 1/\theta\). This property reflects the idea that the prior should be invariant to changes in the scale of the parameter.
Applications
The Jeffreys prior is widely used in various fields of science and engineering where Bayesian methods are applied. It is particularly useful in the following contexts:
Parameter Estimation
In parameter estimation problems, the Jeffreys prior provides a way to incorporate prior knowledge without introducing bias. It is often used in the estimation of parameters in normal distribution, Poisson distribution, and other common statistical models.
Hypothesis Testing
The Jeffreys prior is also used in Bayesian hypothesis testing, where it helps to define the prior probabilities of competing hypotheses. Its non-informative nature ensures that the test results are driven primarily by the data.
Model Selection
In Bayesian model selection, the Jeffreys prior can be used to assign prior probabilities to different models. This allows for an objective comparison of models based on their posterior probabilities.
Examples
Normal Distribution
For a normal distribution with unknown mean \(\mu\) and known variance \(\sigma^2\), the Jeffreys prior for \(\mu\) is uniform, reflecting the fact that all values of \(\mu\) are equally likely a priori. When both the mean and variance are unknown, the Jeffreys prior becomes more complex, involving both parameters.
Poisson Distribution
In the case of a Poisson distribution with parameter \(\lambda\), the Jeffreys prior is \(\pi(\lambda) \propto 1/\sqrt{\lambda}\). This prior reflects the scale-invariance property and is used in various applications involving count data.
Exponential Distribution
For an exponential distribution with rate parameter \(\lambda\), the Jeffreys prior is again \(\pi(\lambda) \propto 1/\lambda\), consistent with the scale-invariance property.
Criticisms and Limitations
Despite its widespread use, the Jeffreys prior is not without criticisms. Some of the main limitations include:
Improper Priors
In some cases, the Jeffreys prior can be improper, meaning that it does not integrate to one over the parameter space. This can lead to difficulties in interpreting the posterior distribution and requires careful handling in practical applications.
Sensitivity to Model Assumptions
The Jeffreys prior is derived based on the likelihood function, which depends on the assumed statistical model. If the model is misspecified, the resulting Jeffreys prior may not be appropriate, leading to biased inference.
Computational Complexity
Calculating the Jeffreys prior can be computationally intensive, especially for complex models with high-dimensional parameter spaces. This can limit its applicability in large-scale problems.