Bayesian inference
Introduction
Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Bayesian inference is an important technique in statistics, and especially in mathematical statistics. The Bayesian approach to statistics assigns probability distributions to both the hypotheses and the data, treating both "unknowns" and "random variables" in the same way.
Bayesian Statistics
Bayesian statistics is a subfield of statistics in which the evidence about the true state of the world is expressed in terms of 'degrees of belief' known as Bayesian probabilities. The degree of belief may be based on prior knowledge about the event, such as the results of previous experiments, or on personal beliefs about the event.
Prior Probability
In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken into account. For example, the prior could be the probability distribution representing the relative proportions of voters who will vote for a particular politician in a future election.
Likelihood Function
The likelihood function (often simply called the likelihood) measures the goodness of fit of a statistical model to a sample of data for given values of the unknown parameters. It is formed from the joint probability distribution of the sample, but viewed and used as a function of the parameters only, thus treating the random variables as fixed at the observed values.
Posterior Probability
The posterior probability is the probability of the parameters (unknowns) given the evidence. It contrasts with the likelihood function, which is the probability of the evidence given the parameters. The posterior probability is the result of applying Bayes' theorem, which is the fundamental underpinning of Bayesian statistics.
Bayesian Inference in Machine Learning
Bayesian inference has found application in a wide range of activities, including science, engineering, philosophy, medicine, sport, and law. In the context of machine learning, Bayesian inference is used to update the probability estimate for a hypothesis as more evidence or information becomes available. This approach is particularly useful in the field of machine learning, where the aim is to make predictions based on data.
Bayesian Networks
A Bayesian network, Bayes network, belief network, decision network, Bayes(ian) model or probabilistic directed acyclic graphical model is a probabilistic graphical model (a type of statistical model) that represents a set of variables and their conditional dependencies via a directed acyclic graph (DAG). Bayesian networks are ideal for taking an event that occurred and predicting the likelihood that any one of several possible known causes was the contributing factor.
Bayesian Linear Regression
Bayesian linear regression is an approach to linear regression in which the statistical analysis is undertaken within the context of Bayesian inference. When the regression model has errors that have a normal distribution, and if a particular form of prior distribution is assumed, explicit results are available for the posterior probability distributions of the model's parameters.
Bayesian Optimization
Bayesian optimization is a sequential design strategy for global optimization of black-box functions that doesn't require derivatives. It works by constructing a posterior distribution of functions (gaussian process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not.
Conclusion
Bayesian inference is a powerful tool in statistics and machine learning, allowing for the updating of predictions as new evidence is presented. It is based on the use of probability distributions and relies on Bayes' theorem to calculate these probabilities. The use of Bayesian inference in machine learning has led to significant advancements in the field, particularly in the areas of Bayesian networks, Bayesian linear regression, and Bayesian optimization.