Gamma Function
The gamma function is defined as:
\[
\Gamma (z) = \int_{0} ^\infty t^{z-1}e^{-t} dt, \, Re(z) > 0.
\]
Theorem 1:
For any positive integer \(n\),
\[
\Gamma (n+1) = n!
\]
Proof:
Substituting \(z= n+1\) into the definition:
\[
\Gamma (n+1) = \int_{0} ^\infty t^{n}e^{-t} dt
\]
By integration by parts
(\(u = t^n \Longrightarrow du =ntdt\), and \(dv = e^{-t} dt \Longrightarrow v = -e^{-t}\))
\[
\begin{align*}
\Gamma (n+1) &= [-t^ne^{-t}]_{0} ^\infty + \int_{0} ^\infty nt^{n-1}e^{-t} dt \\\\
&= n\int_{0} ^\infty t^{n-1}e^{-t} dt \\\\
&= n \Gamma(n)
\end{align*}
\]
Thus, the gamma function has the recursive property like factorials hold.
Here, we assume that \(\Gamma(k+1) = k!\) for some \(k \in \mathbb{Z}^+\). By the recursive property,
\[
\begin{align*}
\Gamma (k+2) &= (k+1)\Gamma(k+1) \\\\
&= (k+1)k! \\\\
&= (k+1)! \\\\
\end{align*}
\]
Also, when \(n = 1\):
\[
\Gamma (2) = \int_{0} ^\infty te^{-t} dt
\]
By integration by parts
(\(u = t \Longrightarrow du =dt\), and \(dv = e^{-t} dt \Longrightarrow v = -e^{-t}\))
\[
\begin{align*}
\Gamma (2) &= [-te^{-t}]_{0} ^\infty + \int_{0} ^\infty e^{-t} dt \\\\
&= 1 \\\\
&= (2-1)!
\end{align*}
\]
Therefore, by mathematical induction, \(\Gamma(n+1) = n!\) or equivalently, \(\Gamma (n) = (n-1)!\).
Let's compute the most iconic value of the gamma function.
\[
\Gamma (\frac{1}{2}) = \int_{0} ^\infty t^{\frac{-1}{2}}e^{-t} dt
\]
Let \(t = u^2\), so \(dt =2udu\). Then
\[
\begin{align*}
\Gamma (\frac{1}{2}) &= \int_{0} ^\infty u^{2\cdot \frac{-1}{2}}e^{-u^2} 2udu \\\\
&= 2\int_{0} ^\infty e^{-u^2} du \\\\
&= \int_{-\infty} ^\infty e^{-u^2} du \tag{1} \\\\
&= \sqrt{\pi}
\end{align*}
\]
Note: in (1), we define the Gaussian function as
\[
f(x) = e^{-x^2}
\]
and then
\[
\int_{-\infty} ^\infty e^{-x^2} dx = \sqrt{\pi}.
\]
We will revisit this important result in the section of the normal distribution.
Technically, the factorial \(n!\) is defined only for non-negative integers, but "informally", we might consider
\(\Gamma (0.5) = (-0.5)!\). By the recursive property of factorial: \(n! = n(n-1)!\), we can compute non-integer
factorials:
\[\begin{align*}
&(0.5)! = 0.5(-0.5)! = \frac{1}{2}\sqrt{\pi} = \Gamma(1.5) \\\\
&(1.5)! = 1.5(0.5)! = \frac{3}{4}\sqrt{\pi} = \Gamma(2.5) \\\\\
&(2.5)! = 2.5(1.5)! = \frac{15}{8}\sqrt{\pi} = \Gamma(3.5)
\end{align*}
\]
(Formally, we should use the recursive property of the gamma function: \(\Gamma (n+1) = n \Gamma(n).\) )
BONUS: Volume of the \(n\)-dimensional sphere
\[
V = \frac{\pi^{\frac{n}{2}}}{\Gamma (\frac{n}{2} + 1)}r^n
\]
Gamma Distribution
A random variable \(X\) is said to have the gamma distribution with the shape parameter \(\alpha > 0\) and
the rate parameter \(\beta > 0\) if its p.d.f. is given by
\[
f(x) = \frac{\beta^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}, \, x \geq 0 \tag{2}
\]
It is denoted as \(X \sim \text{Gamma }(\alpha, \beta)\) and the mean and variance of the gamma distribution are given by
\[
\mathbb{E }[X] = \frac{\alpha}{\beta} \qquad \text{Ver }(X) = \frac{\alpha}{\beta^2}.
\]
The mean and variance of the gamma distribution:
Using definition of the mean,
\[
\begin{align*}
\mathbb{E }[X] &= \int_{0}^\infty x f(x)dx \\\\
&= \frac{\beta^\alpha}{\Gamma(\alpha)} \int_{0}^\infty x^{\alpha}e^{-\beta x} dx
\end{align*}
\]
Let \(t = \beta x\) and so \(x = \frac{t}{\beta}\) and \(dx = \frac{1}{\beta}dt\).
Substituting these into the equation:
\[
\begin{align*}
\mathbb{E }[X] &= \frac{\beta^\alpha}{\Gamma(\alpha)} \int_{0}^\infty (\frac{t}{\beta})^{\alpha}e^{-t} \frac{1}{\beta}dt \\\\\
&= \frac{\beta^\alpha}{\Gamma(\alpha) \beta^{\alpha +1}} \int_{0}^\infty t^{\alpha}e^{-t} dt \\\\
&= \frac{\Gamma (\alpha +1)}{\Gamma(\alpha) \beta} \\\\
\end{align*}
\]
Since \(\Gamma(\alpha +1) = \alpha \Gamma(\alpha)\),
\[
\begin{align*}
\mathbb{E }[X] &= \frac{\alpha \Gamma (\alpha)}{\Gamma(\alpha) \beta} \\\\
&= \frac{\alpha}{\beta}
\end{align*}
\]
We use the fact \(\text{Var }(X) = \mathbb{E }[X^2] - (\mathbb{E }[X] )^2\).
Compute\(\mathbb{E }[X^2]\) applying the same substitution as we did above:
\[
\begin{align*}
\mathbb{E }[X^2] &= \int_{0}^\infty x^2 f(x)dx \\\\
&= \frac{\beta^\alpha}{\Gamma(\alpha)} \int_{0}^\infty x^{\alpha+1}e^{-\beta x} dx \\\\
&= \frac{\beta^\alpha}{\Gamma(\alpha) \beta^{\alpha +2}} \int_{0}^\infty t^{\alpha+1}e^{-t} dt \\\\
&= \frac{\Gamma (\alpha +2)}{\Gamma(\alpha) \beta^2}.
\end{align*}
\]
Since \(\Gamma(\alpha +2) = (\alpha +1)\Gamma (\alpha +1) = (\alpha +1 )\alpha \Gamma(\alpha)\),
\[
\begin{align*}
\mathbb{E }[X^2] &= \frac{(\alpha +1 )\alpha \Gamma(\alpha)}{\Gamma(\alpha) \beta^2} \\\\
&= \frac{\alpha(\alpha +1 )}{\beta^2}
\end{align*}
\]
Substitute the results:
\[
\begin{align*}
\text{Var }(X) &= \mathbb{E }[X^2] - (\mathbb{E }[X] )^2 \\\\
&= \frac{\alpha(\alpha +1 )}{\beta^2} - (\frac{\alpha}{\beta})^2 \\\\
&= \frac{\alpha}{\beta^2}
\end{align*}
\]
Note: Often, the gamma distribution is parametrized such that
\[
X \sim \text{Gamma }(k = \alpha, \theta = \frac{1}{\beta})
\]
The Exponential distribution is a special case of the gamma distribution.
\[
X \sim \text{Exp }(\lambda) = X \sim \text{Gamma }(1, \beta = \lambda)
\]
So, p.d.f. (2) becomes
\[
f(x) = \lambda e^{-\lambda x} \qquad x \geq 0.
\]
The exponential distribution represents a process in which events occur continuously and independently at
an average rate \(\lambda\). For example, if a machine gets an error once every 20 years, then the time to
failure is represented by the exponential distribution with \(\lambda = \frac{1}{20}\).
The mean and variance of an exponential distribution are given by
\[
\mathbb{E }[X] = \frac{1}{\lambda} \qquad \text{Ver }(X) = \frac{1}{\lambda^2}.
\]
This implies the mean is equivalent to the standard deviation.
Also, its c.d.f. is give by
\[
F(x) = \lambda \int_{0} ^x e^{-\lambda u} du = 1 - e^{-\lambda x}
\]
Beta Function
The beta function is defined as:
\[
B(a, b) = \int_{0}^1 t^{a-1} (1-t)^{b-t} dt, \quad Re(a) > 0, \, Re(b) > 0.
\]
The beta function can be represented by the gamma function:
Theorem 2:
\[
B(a, b) = \frac{\Gamma (a) \Gamma (b)}{\Gamma (a+b)}
\]
Proof:
Consider the product of two distinct gamma functions with inputs \(a, b > 0\).
\[
\begin{align*}
\Gamma (a) \Gamma(b) &= \int_{0}^\infty u^{a-1}e^{-u}du \cdot \int_{0}^\infty v^{b-1}e^{-v}dv \\\\
&= \int_{0}^\infty \int_{0}^\infty u^{a-1} v^{b-1} e^{-(u+v)} dudv
\end{align*}
\]
Let \(s = u + v\) and \(t = \frac{u}{u+v} \). Then
\[
u = (u+v)t = st \Longrightarrow du = sdt
\]
and
\[
v = s - u = (1-t)s \Longrightarrow dv = ds.
\]
Substituting these into the product:
\[
\begin{align*}
\Gamma (a) \Gamma(b) &= \int_{0}^\infty \int_{0}^1 (st)^{a-1} ((1-t)s)^{b-1} e^{-s} s dt ds \\\\\
&= \int_{0}^\infty s s^{a-1} s^{b-1} e^{-s} ds \cdot \int_{0}^1 t^{a-1} (1-t)^{b-1} dt \\\\
&= \int_{0}^\infty s^{(a+b)-1} e^{-s} ds \cdot \int_{0}^1 t^{a-1} (1-t)^{b-1} dt \\\\
&= \Gamma (a+b) \cdot \text{B }(a, b)
\end{align*}
\]
Therefore,
\[
B(a, b) = \frac{\Gamma (a) \Gamma (b)}{\Gamma (a+b)}
\]
Beta Distribution
A random variable \(X\) has a beta distribution on the interval \([0, 1]\) with
parameters \(a\) and \(b\) if its p.d.f. is given by
\[
f(x) = \frac{1}{B(a, b)}x^{a-1} (1-x)^{b-1} \, x \in [0, 1] \tag{3}
\]
It is denoted as \(X \sim \text{Beta }(a, b)\) and the mean and variance of the beta distribution are given by
\[
\mathbb{E }[X] = \frac{a}{a+b} \qquad \text{Ver }(X) = \frac{ab}{(a+b)^2(a+b+1)}.
\]
The mean and variance of the beta distribution:
We can rewrite (3) using Theorem 2:
\[
f(x) = \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)}x^{a-1} (1-x)^{b-1} \quad x \in [0, 1].
\]
Using definition of the mean,
\[
\begin{align*}
\mathbb{E }[X] &= \int_{0}^1 x f(x)dx \\\\
&= \int_{0}^1 x \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)}x^{a-1} (1-x)^{b-1} dx \\\\
&= \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} \int_{0}^1 x^{a} (1-x)^{b-1} dx \\\\
&= \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} B(a+1, b) \\\\
&= \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} \frac{\Gamma (a+1) \Gamma (b)}{\Gamma (a+b+1)}
\end{align*}
\]
Since \(\Gamma (a+1) = a \Gamma (a)\) and similarly, \(\Gamma (a+b+1) = (a+b)\Gamma (a+b)\),
\[
\begin{align*}
\mathbb{E }[X] &= \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} \frac{a\Gamma (a) \Gamma (b)}{(a+b)\Gamma (a+b)} \\\\
&= \frac{a}{a+b}
\end{align*}
\]
We use the fact \(\text{Var }(X) = \mathbb{E }[X^2] - (\mathbb{E }[X] )^2\).
Compute\(\mathbb{E }[X^2]\):
\[
\begin{align*}
\mathbb{E }[X^2] &= \int_{0}^1 x^2 f(x)dx \\\\
&= \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} \int_{0}^1 x^{a+1} (1-x)^{b-1} dx \\\\
&= \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} B(a+2, b) \\\\
&= \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} \frac{\Gamma (a+2) \Gamma (b)}{\Gamma (a+b+2)}
\end{align*}
\]
Since \(\Gamma (a+2) = (a+1) \Gamma (a+1) = (a+1) a \Gamma (a)\), and similarly, \(\Gamma (a+b+2) = (a+b+1)(a+b)\Gamma (a+b)\),
\[
\begin{align*}
\mathbb{E }[X^2] &= \frac{\Gamma (a+b)}{\Gamma (a) \Gamma (b)} \frac{(a+1)a\Gamma (a) \Gamma (b)}{(a+b+1)(a+b)\Gamma (a+b)} \\\\
&= \frac{a(a+1)}{(a+b)(a+b+1)}.
\end{align*}
\]
Substitute the results:
\[
\begin{align*}
\text{Var }(X) &= \mathbb{E }[X^2] - (\mathbb{E }[X] )^2 \\\\
&= \frac{a(a+1)}{(a+b)(a+b+1)} - (\frac{a}{a+b})^2 \\\\
&= \frac{a(a+1)(a+b) - a^2 (a+b+1) }{(a+b)^2(a+b+1)} \\\\
&= \frac{ab}{(a+b)^2(a+b+1)}
\end{align*}
\]
The uniform distribution over the interval \([a, b]\) is a special case of the beta distribution.
\[
X \sim U[0, 1] = X \sim \text{Beta }(1, 1) \text{ then } f(x) = 1.
\]
In general, the p.d.f. of the uniform distribution \(X \sim U[a, b]\) is given by
\[
f(x) = \frac{1}{b - a} \qquad x \in [a, b]
\]
Note: Otherwise, \(f(x) = 0\).
The uniform distribution represents the situations where every value is equally likely over
the interval \([a, b]\). For example, this distribution is very popular for generating random
numbers in programming languages.
The mean and variance of the uniform distribution are given by
\[
\mathbb{E }[X] = \frac{a+b}{2} \qquad \text{Ver }(X) = \frac{(b-a)^2}{12}
\]
and its c.d.f. is given by
\[
F(x) = \frac{x-a}{b-a} \qquad x \in [a, b]
\]
Note: if \(x < a\), \(\, F(x) = 0\) and if \(x > b\), \(\, F(x) = 1\).
The mean and variance of the uniform distribution:
Using definition of the mean,
\[
\begin{align*}
\mathbb{E }[X] &= \int_{a}^b x \frac{1}{b-a}dx \\\\
&= \frac{1}{b-a} (\frac{b^2 - a^2}{2}) \\\\
&= \frac{a+b}{2}
\end{align*}
\]
Also,
\[
\begin{align*}
\mathbb{E }[X^2] &= \int_{a}^b x^2 \frac{1}{b-a}dx \\\\
&= \frac{1}{b-a} (\frac{b^3 - a^3}{3}) \\\\
&= \frac{a^2 +ab + b^2}{3}
\end{align*}
\]
and then
\[
\begin{align*}
\text{Var }(X) &= \mathbb{E }[X^2] - (\mathbb{E }[X] )^2 \\\\
&= \frac{a^2 +ab + b^2}{3} - (\frac{a+b}{2})^2 \\\\
&= \frac{4(a^2 + ab +b^2)-3(a^2 + 2ab + b^2)}{12} \\\\
&= \frac{(b-a)^2}{12}.
\end{align*}
\]
Interactive Visualization
Below is an interactive visualization to help you understand the Gamma and Beta distributions.
You can adjust the parameters using the sliders and observe how the probability density function changes.
Try the special cases to see important variants of these distributions.