Definition, equation, and example

Everyone has heard of a bell curve and has a general idea of what it is. Bell curves, regardless of their exact shape, are described by the normal distribution (aka "Gaussian distribution") equation, normalized to the number of individuals in the studied population. The normal distribution is a continuous probability density function (PDF) that is symmetrical around the mean. The normal distribution equation describes the probability of observing a value x, in a population with a mean \(\mu\) and a standard deviation \(\sigma\) (variance = \(\sigma^2\)). The equation is:

$$p(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{\frac{-1}{2}{(\frac{x-\mu}{\sigma})}^{2}}$$ $$\text{The Normal Distribution Equation}$$

• the mean, median, and mode are all equal in a normal distribution
• the standard deviation (or its square, the variance) is the measure of the spread of the graph
• In a PDF the probability of an observation p(x) is always positive
• In a PDF of all probabilities in a given interval adds up to 1
• For the normal distribution the sum of all probabilities from \(-\infty\) to \(+\infty\) equals 1.

$$\int_{- \infty}^{+\infty}{\frac{1}{\sigma\sqrt{2\pi}}e^{\frac{- 1}{2}{(\frac{x - \mu}{\sigma})}^{2}}dx = 1\ \ }$$ $$\text{ The Sum of All Probabilities}$$

The histograms in Figure 1 are superimposed with corresponding normal distribution curves. Actually, you can't fit the data to a normal distribution equation because, obviously, the men's and women's sample sizes are greater than one. Since the total area under a normal distribution curve is 1, if we want to fit the histograms to a normal distribution, we need to normalize it to the sample size, which amounts to multiplying the PDF by the respective sample sizes (n).

$$y=n\frac{1}{\sigma\sqrt{2\pi}}e^{\frac{- 1}{2}{(\frac{x-\mu}{\sigma})}^{2}}$$ $$\text{Normalizing to the Sample Size}$$ Once we do that we get the fitted plots shown in Figure 1. Well, we also need to plug \(\mu\) and \(\sigma\) into the PDF equation. If those values are unknown they can be calculated by fitting the PDF to the data, which is actually what we did (see the Python code that generated the figure right after the legend - data not included). Before we look at the definitions of the mean \(\mu\) and variance \(\sigma^2\) (the standard deviation is \(\sigma\) ), we must distinguish between a population and a sample. We usually don't have data for the entire population; rather, we have a sample or samples of the population from which we are trying to estimate, within a defined confidence interval, population parameters such as its mean and standard deviation.

Mean, variance/standard deviation: population and sample formulas

The formulas for the sample mean x̄ and the population mean \(\mu\) are the same. $$\overline{x} = \frac{\sum_{i=1}^{1=n}x_{i}}{n};\ \ \ \mu = \frac{\sum_{i=1}^{1=N}x_{i}}{N}$$ $$\text{Sample versus Population Mean}$$ But, the formulas for the sample variance s² and population variance \(\sigma^2\) are different, although the calculated values will get closer to each other as the sample size increases.

$$s^2=\frac{\sum_{i=1}^{i=n}{{(x_{i}-\overline{x}\ )}^{2}\ }}{n - 1};\ \ \ \sigma^2 = \frac{\sum_{i=1}^{i=N}{{(x_{i}-\mu)}^{2}}}{N}$$ $$\text{Sample versus Population Variance}$$

The \(\mu\) and the \(\sigma^2\) formulas for the theoretical normal distribution PDF are: $$\mu=\int_{-\infty}^{+\infty}{xp(x)dx}; \ \ \ \sigma^2=\int_{-\infty}^{+\infty}{(x-\mu)^2p(x)dx}$$ $$\text{Mean and Variance Expressed as Integrals}$$ The following example might help understand the correspondence between the integral and the summation formulas. Suppose we have a histogram with just five bins for heights in inches : 60", 65", 70", 75", 80", where the 60" bin accommodates people with heights 57.5" to 62.5", the next bin for 62.5" to 67.5", and so on, and that the numbers of people in those bins are, respectively 3, 10, 30, 12, 5. What is the mean height? Since the total number of people is 60. The mean is (3/60)x60" + (10/60)x65" + (30/60)x70" + (12/60)x75" + (5/60)x80" = 70.5". This calculation corresponds to the integral formula, where each term is p(x)dx. The summation formula further up is exactly the same calculation written as (3x60" + 10x65" + 30x70" + 12x75" + 5x80")/60 = 70.5".
Yet another way to define the mean and variance is by using the expected value operator \(\mathbb{E}\). $$\mu=\mathbb{E}[x]; \ \ \ \sigma^2 = \ Var(X) = \ \mathbb{E}[X-\mathbb{E}[X]^2]$$ $$\text{Mean and Variance Defined by the Expected Value Operator}$$

Derivation of the normal distribution equation

Next let's derive the normal distribution equation so that it is not a "black box". Although this is a long derivation, you should be able to follow it you have a basic understanding of calculus. I took the derivation from this web page The normal distribution: A derivation from basic principles, by Dan Teage, but will explain in more detail some of the steps that might not be obvious to non-mathematicians like me. Click on the arrow below to see the derivation.

Proof of the sample variance equation

Why do these formulas differ in the denominator (N versus n-1)?

The variance as defined by the equation on the left is valid only if we know the entire population's mean. However, we usually work with samples, in which case, we divide by n-1 instead of N. When writing about the mean and variance we should always use \(\mu\) and \(\sigma^2\) when referring to a population, and x̄ and s² when referring to a sample.
The standard way to explain the difference between these two formulas is to say that we lose one degree of freedom when sampling. Here is why: suppose I sample 3 of the 4 members of a family whose heights are respectively 3,4,5, and 6 ft. The true mean is 4.5 ft. But if I sample only 3 members my estimates of the mean are 4, 4.3, 4.7, and 5 ft, depending on which members I sample (3,4,5) (3,5,6) (3,4,5) or (4,5,6). Now when estimating the variance, I take the difference between each height and the estimated mean (x̄). Notice however that, because I already know x̄, after taking two of the numbers, the 3rd number is automatically determined. That is why we lose one degree of freedom, but it might not so obvious why this leads to an underestimate of the variance if we use n rather than n-1 in the denominator. This underestimate arises from the smaller spread of the sample when compared to the real spread of the population, which could be much larger. Therefore, dividing by n-1 will correct the underestimate by decreasing the denominator. To make sure let us prove this mathematically.

Confidence intervals

The probability of a random observation falling within an interval \( x_1 \rightarrow x_2 \) is called a confidence interval. For the normal distribution it is equal to the area under the curve, or the integral: $$\int_{x_1}^{x_2}\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\left( \frac{x-\mu}{\sigma}\right)^2}dx$$ $$\text{Confidence Interval}$$ This is a non-elementary integral, but approximated numerical solutions can be obtained by using power series methods. Before computers, people used tables of pre-calculated confidence intervals. These tables are still used today because for many people they are simpler to understand than computer programs. Typically, these tables (e.g., https://z-table.com/) give you the cumulative confidence interval, which is calculated using the cumulative distribution function: CDF. Mathematically this is the integral, or the area under curve, from \(-\infty \ to \ x\). Note however that you do not input x, but rather the z-score (z) which you calculate as follows: \(z=\frac{(x-\mu)}\sigma{}\). There is a very good reason to use z-scores rather than x. It is because this way you only a single number z to look up in the table, rather than 3 numbers (\(x, \ \mu, \sigma\)) in all sorts of combinations. Therefore, the CDF is more commonly stated as: $$ \text{CDF}(z) = \Phi(z) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{z} e^{-\frac{1}{2} t^2} \, dt $$ $$\text{Cumulative Distribution Function (CDF)}$$ Here is a quick derivation of what we just did: $$ \text{Start with the integral of the normal distribution:} \ \frac{1}{\sigma \sqrt{2\pi}} \int_{-\infty}^{x} e^{-\frac{1}{2} \left( \frac{x - \mu}{\sigma} \right)^2} \, dx $$ $$ Substitute \ z = \left( \frac{x - \mu}{\sigma} \right) \Rightarrow \frac{dz}{dx} = \frac{1}{\sigma} \Rightarrow dx = \sigma \, dz $$ $$\text{The upper integral limit} \ x \ becomes \ z = \left( \frac{x - \mu}{\sigma} \right) \text{or simply} \ z$$ $$ \Phi(z) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{z} e^{-\frac{1}{2} z^2} \, dz $$ But there is a confusing notation in the formula above. The \( z \) value in the upper limit of the integral is a constant corresponding to the \( x \) value that delimits our confidence interval, whereas the \( z \) inside the integral is a variable that can take any real number value \( x \in \mathbb{R} \). Therefore, in order to avoid this confusion, we change the name of the variable inside the integral from \( z \) to \( t \), which, of course, makes no difference mathematically. $$ \Phi(z) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{z} e^{-\frac{1}{2} t^2} \, dt $$ $$\text{Cumulative Distribution Function (CDF)}$$ As an example, Figure 4 illustrates a distribution with \(\mu\) = 2 and \(\sigma\) = 1. The following are 5 different ways one can ask the same question:

• how likely is it that a random observation will be a number smaller than 3?
• what is the probability that a random observation will yield a number smaller than 3?
• what is the confidence interval for x < 3?
• what is the area under the curve from \(-\infty\) to 3?
• given that the z-value corresponding to 3 is \(\frac{3-2}{1}=1\ ,what \ is \ \Phi(1) \ or \ cdf(1)\)?

Looking up Z = 1 on the table (https://z-table.com/) you see that the confidence interval cdf is 0.8413. If you prefer using computer programs you can make the same calculation with Python by using the stats.norm.cdf() method. If you click the arrow below you will see the Python code that generated Figure 4. Line 10, ci=stats.norm.cdf(1), defines the variable ci = 0.8413 as the confidence interval for Z = 1.

            import numpy as np
            import matplotlib.pyplot as plt
            import scipy.stats as stats
            m = 2
            s = 1
            x = np.linspace(-1,5,200)
            y = stats.norm.pdf(x,m,s)
            x_fill = np.linspace(-1,3,200)
            y_fill = stats.norm.pdf(x_fill,m,s)
            ci=stats.norm.cdf(1)
            fig,ax= plt.subplots(figsize=(12,5))
            ax.plot(x,y)
            ax.axvline(2,color='black',linestyle='--')
            ax.axvline(3,color='red',linestyle='--',ymin=0, ymax=0.7)
            ax.tick_params(axis='both',which='major',labelsize=18)
            ax.text(2,0.01,' μ=2',fontsize=18, color='black')
            ax.text(3,0.35,' σ=1',fontsize=18, color='red')
            ax.text(3,0.30,' z=1',fontsize=18, color='red')
            ax.text(0.5,0.05,f'c.i.={ci:.4f}',fontsize=18, color='black')
            ax.fill_between(x_fill,y_fill,color='lightblue',alpha=0.5)
            ax.set_ylim(bottom=0)
            ax.set_xlim(left = -1, right = 5)
            plt.show()
          

Now suppose you wanted a 95% confidence interval for the same distribution. You will find that the closest numbers to 0.95 on the table are 0.9495 and 0.9505, corresponding respectively to Z = 1.64 and Z = 1.65. Take 1.64. Thus, a 95% confidence interval corresponds to x < \(\mu\) + 1.64\(\sigma\) → x < 2 + 1.64\(\sigma\) → x < 3.64.

Again, if you click the arrow below you will see the Python code that generated Figure 5. Line 8, z = stats.norm.ppf(0.95), defines the variable z = 1.6449 as the Z value corresponding to a 95% confidence interval.

          import numpy as np
          import matplotlib.pyplot as plt
          import scipy.stats as stats
          m = 2
          s = 1
          x = np.linspace(-1, 5, 200)
          y = stats.norm.pdf(x, m, s)
          z = stats.norm.ppf(0.95)
          x_fill = np.linspace(-1, m + z * s, 200)
          y_fill = stats.norm.pdf(x_fill, m, s)
          fig,ax= plt.subplots(figsize=(12,5))
          ax.plot(x, y)
          ax.axvline(m, color='black', linestyle='--')
          ax.axvline(z * s + m, color='red', ymin = 0, ymax = 0.5, linestyle='--')
          ax.tick_params(axis='both', which='major', labelsize=18)
          ax.text(z * s + m, 0.25, f'z = {z:.4f}', color='red', fontsize=18)
          ax.text(0.5, 0.05, f'c.i.= 0.95', fontsize=18, color='black')
          ax.set_xlim(left=-1, right=5)
          ax.set_ylim(bottom=0)
          ax.fill_between(x_fill, y_fill, color='lightblue', alpha=0.5)
          plt.show()

        

Sometimes we want a two-tailed confidence interval (ci) for observations that fall equally to the right and to the left of the mean. For a 95% ci, this means we need to find the Z value corresponding to 97.5% since 2.5% of the remaining area under the curve is on each of the tail ends. Figure 6 illustrates how to do this calculation and compares the one-tail and two-tail ci possibilities.

          import numpy as np
          import matplotlib.pyplot as plt
          import scipy.stats as stats
          # GLOBAL VARIABLES
          adjst = 0.1  # Adjust arrow heads position
          m = 10
          s = 2
          z_ctr = stats.norm.ppf(0.975)  # 95% CI (2-tailed)
          z_cml = stats.norm.ppf(0.950)  # 95% CI (1-tailed cumulative)

          x1 = m - z_ctr * s
          x2 = m + z_ctr * s
          x3 = m + z_cml * s

          x_ = np.linspace(m - 3 * s, m + 3 * s, 1000)
          y_ = stats.norm.pdf(x_, m, s)

          xh_cml = np.linspace(m - 3 * s, x3, 200)
          yh_cml = np.full_like(xh_cml, 0.12)

          xh_ctr = np.linspace(x1, x2, 200)
          yh_ctr = np.full_like(xh_ctr, 0.1)

          x_fill_cml = np.linspace(m - 3 * s, x3, 200)
          y_fill_cml = stats.norm.pdf(x_fill_cml, m, s)

          x_fill_ctr = np.linspace(x1, x2, 1000)
          y_fill_ctr = stats.norm.pdf(x_fill_ctr, m, s)

          # PLOTS
          fig, ax = plt.subplots(figsize=(12, 6))
          ax.plot(x_, y_, label=r"pdf$(x,\mu=10,\sigma=2)$", color='blue')

          ax.axvline(x1, color='red', linestyle='--')
          ax.axvline(x2, color='red', linestyle='--')
          ax.axvline(x3, color='black', linestyle='--')
          ax.axvline(m, color='blue', linestyle=':')

          ax.plot(xh_cml, yh_cml, color='black')
          ax.plot(x3 - adjst, 0.12, marker='>', markersize=5, color='black')
          ax.plot(x1 + adjst, 0.1, marker='<', markersize=5, color='red')
          ax.plot(x2 - adjst, 0.1, marker='>', markersize=5, color='red')
          ax.plot(xh_ctr, yh_ctr, color='red')

          ax.fill_between(x_fill_cml, y_fill_cml, hatch='///', alpha=0.5, 
                          facecolor='white', edgecolor='black', linewidth=1.2,
                          label=r'$1-tail. \ ci=\int_{-\infty}^{\mu+\sigma z}p(x)dx=0.95$' + f' z=ppf(0.950)={z_cml:.3f}  ({m}+{s}x{z_cml:.3f})  x < {x3:.2f}')

          ax.fill_between(x_fill_ctr, y_fill_ctr, hatch='\\\\', alpha=0.5, 
                          facecolor='white', edgecolor='red', linewidth=1.2,
                          label=r'$2-tail. \ ci=\int_{\mu-\sigma z}^{\mu+\sigma z}p(x)dx=0.95$' + f' z=ppf(0.975)={z_ctr:.3f} ({m}'+r'$\pm$'+f'{s}x{z_ctr:.3f}) {x1:.2f} < x  < {x2:.2f}')

          # AXES AND GRID
          ax.set_ylim(bottom=0, top=0.3)
          ax.set_xlim(left=m - 3 * s, right=m + 3 * s)

          # TITLE, LABELS, AND ANNOTATIONS
          ax.set_title('One-Tailed (cumulative) vs Two-Tailed 95% Confidence Intervals', fontsize=18)
          ax.set_xlabel('x', fontsize=18)
          ax.set_xticks(range(4,17))
          ax.set_ylabel('pdf(x)', fontsize=18)
          ax.tick_params(axis='both', which='major', labelsize=16)

          # Manually adjust text positions
          ax.text(5.3, 0.003, '2.5%', fontsize=15, color='black')
          ax.text(x3 + 0.03, 0.003, '2.5%', fontsize=15, color='black')
          ax.text(14.1, 0.003, '2.5%', fontsize=15, color='black')

          ax.legend(loc="upper left", fontsize=14)

          # Show plot
          plt.show()     
        

The The Gauss Error Function \(\operatorname{erf}(x)\) is found in Excel, Google Sheets, Python, and on many scientific calculators. However, be very careful if you decide to use it because it is not the same as the cumulative density function (CDF). $$\operatorname{erf}(z) = \frac{2}{\sqrt{\pi}}\int_0^z e^{-t^2} dt, \quad \Phi(z) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{z} e^{-\frac{1}{2} t^2} dt$$ $$\text{ERF(z) versus CDF(z)}$$ The two functions are related as follows: $$\Phi(y) = \frac{1}{2} \left(1 + \operatorname{erf} \left(\frac{y}{\sqrt{2}}\right) \right)$$ $$\text{Relation Between CDF(z) and ERF(z)}$$

Using the Python cdf function (for the normal distribution): ci = stats.norm.cdf(1) yields ci=0.8413447460685429

Using the Excel erf function: =0.5*(1+ERF(1/SQRT(2))) results in: 0.841344746