Bernoulli Distribution

Introduction

The Bernoulli distribution is a discrete probability distribution for a random variable which takes the value 1 with probability \( p \) and the value 0 with probability \( 1 - p \). It is one of the simplest and most fundamental distributions in probability theory and statistics, often used to model binary outcomes or events that have two possible outcomes, such as success/failure, yes/no, or on/off.

Definition

Formally, a random variable \( X \) is said to follow a Bernoulli distribution with parameter \( p \) if it takes the value 1 with probability \( p \) and the value 0 with probability \( 1 - p \). The probability mass function (PMF) of \( X \) is given by:

\[ P(X = x) = \begin{cases} p & \text{if } x = 1, \\ 1 - p & \text{if } x = 0. \end{cases} \]

Here, \( 0 \leq p \leq 1 \) is the parameter of the distribution, representing the probability of success.

Properties

Mean and Variance

The mean (expected value) of a Bernoulli random variable \( X \) is given by:

\[ E(X) = p. \]

The variance of \( X \) is given by:

\[ \text{Var}(X) = p(1 - p). \]

These properties are derived from the definitions of expected value and variance for discrete random variables.

Moment Generating Function

The moment generating function (MGF) of a Bernoulli random variable \( X \) is:

\[ M_X(t) = E(e^{tX}) = (1 - p) + pe^t. \]

The MGF is useful for deriving moments of the distribution and for proving the distribution of sums of independent Bernoulli random variables.

Cumulative Distribution Function

The cumulative distribution function (CDF) of a Bernoulli random variable \( X \) is:

\[ F(x) = \begin{cases} 0 & \text{if } x < 0, \\ 1 - p & \text{if } 0 \leq x < 1, \\ 1 & \text{if } x \geq 1. \end{cases} \]

Entropy

The entropy of a Bernoulli random variable \( X \) is a measure of the uncertainty associated with the random variable. It is given by:

\[ H(X) = -p \log(p) - (1 - p) \log(1 - p). \]

Entropy is maximized when \( p = 0.5 \), indicating maximum uncertainty.

Applications

The Bernoulli distribution is widely used in various fields, including:

Coin Toss

A classic example of a Bernoulli distribution is the outcome of a coin toss, where the probability of getting heads (success) is \( p \) and the probability of getting tails (failure) is \( 1 - p \).

Quality Control

In quality control, the Bernoulli distribution can model the occurrence of defective items in a production process, where each item is either defective (success) or non-defective (failure).

Clinical Trials

In clinical trials, the Bernoulli distribution can model the outcome of a binary response variable, such as the success or failure of a treatment.

Reliability Engineering

In reliability engineering, the Bernoulli distribution can model the functioning or failure of a system component, where the component either works (success) or fails (failure).

Relationship with Other Distributions

Binomial Distribution

The Bernoulli distribution is a special case of the binomial distribution with a single trial (\( n = 1 \)). If \( X \) follows a Bernoulli distribution with parameter \( p \), then \( X \) can be seen as a binomial random variable with parameters \( n = 1 \) and \( p \).

Geometric Distribution

The geometric distribution models the number of trials needed to get the first success in a sequence of independent Bernoulli trials. If \( X \) follows a geometric distribution with parameter \( p \), then \( X \) represents the number of Bernoulli trials needed to achieve the first success.

Negative Binomial Distribution

The negative binomial distribution generalizes the geometric distribution to model the number of trials needed to achieve a fixed number of successes in a sequence of independent Bernoulli trials.

Poisson Distribution

While the Poisson distribution is not directly related to the Bernoulli distribution, it can be derived as the limit of a binomial distribution with a large number of trials and a small probability of success.

Estimation

Maximum Likelihood Estimation

The parameter \( p \) of a Bernoulli distribution can be estimated using the method of maximum likelihood estimation (MLE). Given a sample of \( n \) independent and identically distributed Bernoulli random variables \( X_1, X_2, \ldots, X_n \), the MLE of \( p \) is:

\[ \hat{p} = \frac{1}{n} \sum_{i=1}^n X_i. \]

This estimator is unbiased and consistent.

Bayesian Estimation

In a Bayesian framework, the parameter \( p \) can be estimated by specifying a prior distribution for \( p \) and updating it with observed data. A common choice for the prior distribution is the beta distribution, which is conjugate to the Bernoulli distribution. Given a beta prior \( \text{Beta}(\alpha, \beta) \) and a sample of \( n \) Bernoulli trials with \( k \) successes, the posterior distribution of \( p \) is:

\[ \text{Beta}(\alpha + k, \beta + n - k). \]

Simulation

Simulating a Bernoulli random variable is straightforward. Given a parameter \( p \), one can generate a uniform random variable \( U \) on the interval \([0, 1]\) and set:

\[ X = \begin{cases} 1 & \text{if } U \leq p, \\ 0 & \text{if } U > p. \end{cases} \]

This method is widely used in Monte Carlo methods and other simulation techniques.

History

The Bernoulli distribution is named after the Swiss mathematician Jacob Bernoulli, who studied the distribution in the context of Bernoulli trials and the law of large numbers. His work laid the foundation for the development of probability theory and statistics.

References