Expectation

When we say ``expectation,'' we mean ``average,'' the average being roughly what you would think of (i.e., the arithmetic average, as opposed to a median or mode). For a discrete r.v. $X$, we define the expectation as

$$\boxed{E[X] = \sum_i x_i p_X(x_i)}$$

For a continuous r.v., we define the expectation as

$$\boxed{E[X] = \int_{-\infty}^\infty f_X(x) x dx}$$

Now a bit of technicality regarding integration, which introduces notation commonly used. When you integrate, you are typically doing a Riemann integral:

$$\int_a^b xf_X(x) dx = \lim_{\max_{1 \leq i \leq n-1}\vert x_... ...i\vert \rightarrow 0} \sum_{i=1}^n z_i f_X(z_i) (x_{i+1}-x_i)$$

$$a = x_1 < x_2 < \cdots <x_n = b \qquad\qquad z_i \in (x_i,x_{i+1})$$

In other words, we break up the interval into little slices and add up the vertical rectangular pieces.

Another way of writing this is to recognize that

$$z_i f_X(z_i) (x_{i+1}-x_i) \approx z_i P(x_i < X \leq X_{i+1}) = z_i(F_X(x_{i+1}) - F_X(x_i))$$

and that in the limit, the approximation becomes exact. Note, however, that this is expressed in terms of the c.d.f., not the p.d.f., and so exists for all random variables, not just continuous ones.

This gives rise to what is known as the Riemann-Stieltjes Integral:

$$E[X] = \lim_{\max_{1\leq i \leq n-1}(x_{i+1}-x_i) \rightarrow 0} \sum_{i=1}^n z_i[F_X(x_{i+1}) - F_X(x_i)].$$

We write the limit as

$$\int_a^b x dF_X(x)$$

This notation ``describes'' continuous, discrete, and mixed cases. That is,

$$E[X] = \int_{-\infty}^\infty x dF_X(x).$$

We have defined the Riemann-Stieltjes integral in a context of expectation. However, it has a more general definition:

$$\int_a^b f(x) dg(x) = \lim_{\max(x_{i+1}-x_i)\rightarrow 0} \sum_{i} f(z_i)[g(x_{i+1})-g(x_i)]$$

When $g(x) = x$, this reduces to the ordinary Riemann integral. Sufficient conditions for existence:

• $g(x)$ of bounded variation
• and $f(x)$ continuous on $[a,b]$

or

• $f(x)$ of bounded variation
• $g(x)$ continuous

The first case covers the case of expectation.

In a directly analogous way we define

$$\int_{-\infty}^\infty g(x) d F_X(x) = \lim \sum_{i=1}^n g(z_i) [F_X(x_{i+1}) - F_X(x_i)].$$

Now consider the r.v. $Y = g(X)$.

$$E[Y] = \int_{-\infty}^\infty y d F_Y(y)$$

Note that $dF_Y(y)$ is the representation of the limiting value

$$\begin{aligned} F_Y(y_{i+1}) - F_Y(y_i) &= Pr(y_i < Y \leq y_... ... \leq g^{-1}(y_{i+1})) = Pr(x_i < X \leq x_{i+1}) \end{aligned}$$

which, in the limit is equal to $dF_X(x)$, when $y = g(x)$. Thus

$$\boxed{E[Y] = \int_{-\infty}^\infty y d F_Y(y) = \int_{\infty}^\infty g(x) dF_X(x)}$$

Let us put this in more familiar terms: If $Y = g(x)$, then

$$\boxed{E[Y] = \int_{-\infty}^\infty g(x) f_X(x) dx}$$ (1)

One might think that finding $E[Y]$ would require finding $f_Y(y)$. However, as (1) shows, all that is necessary is to substitute $g(x)$ for $x$ in the expectation. This is sometimes called the law of the unconcious statistitian, since it can be done nearly thoughtlessly.

An interesting result is obtained through the use of indicator functions. Let $I : (\Fc,\Omega) \rightarrow \Rbb$ be defined by

$$I_A(\omega) = \begin{cases} 1 & \omega \in A \\ 0 & \omega \not \in A \end{cases}$$

In other words, the indicator function indicates which its argument is in the set which is the subscripted argument.

We define a simple function as one which is a linear combination of indicator functions: For some collection $A_1,A_2,\ldots,A_n \in \Fc$,

$$g(\omega) = \sum_{k=1}^n b_k I_{A_k}(\omega)$$

This gives us a piecewise-constant function on $\Omega$. It also defines a random variable.

Note that the collection need not be disjoint. However, we can shuffle things around to write the function as

$$g(\omega) = \sum_{k=1}^{n^*} b_k^* I_{A_{k}^*}(\omega)$$

where the $A_{k}^*$s are disjoint, and where the $b_k^*$s are unique. Note that

$$A_k^* = \{\omega \in \Omega : g(\omega) = b_k^*\}$$

Now note that

$$\boxed{E[I_A] = P(A)}$$

Based on this, and the disjointness of the $A_k^*$, we can write

$$E[g] = \sum_{i=1}^{n^*} b_k^* P(A_k^*)$$

There are many instances where indicator functions are used to get a ``handle'' on the probability of an event.

Let us examine the expectation in light of the Riemann-Stieltjes integral. We define

$$E[X] = \int_{-\infty}^\infty x dF_X(x) = \lim_{a \rightarrow-\infty, b \rightarrow \infty} \int_a^b xdF_X(x)$$

This is a stronger sense of the limit than, for example

$$\int_{-\infty}^\infty x dF_X(x) = \lim_{a \rightarrow \infty} \int_{-a}^a xdF_X(x)$$

For example, $\sin(x)$ has an integral in the latter sense (which is equal to 0), but not in the former sense.

Now we will consider an example of a density where the expectation does not exist.