Week 9: Predictability through the lens of Information Theory#
Perfect model experiments#
Information theory plays a central role in the field of predictability (and probably the most essential one in our entire class). Our goal today is to leverage information theory for finding the most predictable feature in a dynamical system. but first, let’s briefly walk through some game rules.
In week 1, we learn that the definition of predictability is that: given a forecast PDF (probability density function) and a climatological PDF, if we can no longer tell the difference between the two, we reach so-called predictability limit. However, there are many ways of testing the difference between two PDFs, the question is…can we reach any agreement on the definition of predictability? The answer is yes …as long as we agree on two a priori requirements for whatever measure we choose.
The measure of predictability is invariant under invertible linear transformations of the state.
The first two moments are the most important metrics for estimating predictability while higher moments are not considered.
Note that these two requirements are demands we place on the measure; they say nothing about the dynamics. We need a separate assumption about the system itself, and the simplest useful one is that the system is a Markov process, i.e., the future state only depends on the current state and the past doesn’t influence the future,
If we further assume the process is linear and Gaussian, it can be written as
where \(\mathbf{x_{t+\tau}}\) is the future state, \(\mathbf{x_{t}}\) is the current state and \(\mathbf{\epsilon}\) is the random white noise. \(\alpha\) is the memory passing from current time step to the next time step. Two conditions are implied: \(\mathbf{\epsilon}\) must be independent of \(\mathbf{x}_t\), and \(|\alpha|<1\) is required for the process to be stationary (otherwise the variance grows without bound and no climatological distribution exists). An alternative way of writing (160) is using a conditional PDF.
where (161) is read as “given the PDF of x at \(t\), the PDF of x at \(t+\tau\)”. (161) is also called transition probability. We know that the initial states change discontinuously when we observe the state. For example, the summer temperature around 2pm at National Taiwan University is around \(35\pm3\) Celsius from the climatological data. However, as soon as we take an observation at a given moment, the distribution collapses onto a much narrower one centered on what we measured. If the observation were perfect, it would collapse all the way to a delta function; in practice observations carry error, so what we are left with is a narrowed but still finite-width distribution. This new distribution of \(\mathbf{x_{t}}\) is defined as \(p(\mathbf{x_{t}}|\mathbf{\theta_t}) \) and named as analysis probability.
To reduce the notation burden, variables at the initial time, verification time and observation will be represented as
With (162), the analysis probability is written as \(p(\mathbf{i}|\mathbf{\theta})\) and transition probability is written as \(p(\mathbf{v}|\mathbf{i})\). The assumption that transition probability is known exactly ((161)) is called perfect model experiment. For simplification, we adopt the perfect model assumption for the later analysis.
We then define forecast distribution as
Physically, (163) says that we obtain the final state PDF by integrating the transition probability over all possible initial conditions, weighting each initial condition by how likely it is given the observation. (163) (the Chapman–Kolmogorov relation) holds from the theorem of compound probability, given that \(p(\mathbf{v}|\mathbf{i},\mathbf{\theta}) = p(\mathbf{v}|\mathbf{i})\) i.e., once the initial state is known, the observation carries no additional information about the future. It is worth emphasizing that this identity holds whether or not the system is predictable; what predictability controls is how narrow the resulting \(p(\mathbf{v}|\mathbf{\theta})\) is, not whether the integral is valid. In practice, the ensemble forecast is generated by repeatedly sampling data from the analysis distribution and computing the transition probability conditioned on drawn \(i\). The spread of the forecast distribution, \(\sigma_{v|\theta}^{2}\), is the noise, and the signal is the variance of the ensemble mean across different observations, \(\mathrm{Var}_{\theta}(\mu_{v|\theta})\). By the law of total variance the two add up to the climatological variance,
a decomposition we will put to work later.
For the reference state, we define a climatological distribution, where the specific observation is missing
One should notice that the climatological distribution should depend on time. For example, climatologically, we know the mid-night temperature is always lower than the afternoon temperature. or we know the winter temperature is on-average lower than the summer temperature.
Some studies (and most cases) have chosen the saturated forecast PDF (i.e., \(p(\mathbf{v}|\mathbf{\theta})\) in the limit \(\tau\rightarrow\infty\)) as the climatological PDF. This is perfectly legitimate whenever the system is stationary and ergodic. For the damped linear system with stochastic forcing that we use throughout this chapter, the saturated forecast PDF has variance \(\sigma_{\epsilon}^{2}/2\|a\|\) and is exactly the climatological PDF. It becomes a terrible choice when the system is non-stationary or non-ergodic. A seasonal cycle or a long-term trend means there is no single \(\tau\rightarrow\infty\) limit to converge to, and a system with multiple regimes means a forecast launched inside one regime may never sample the others, so the saturated forecast PDF ends up narrower than the true climatology. (A purely deterministic damped system is the degenerate example: its saturated PDF is a delta function, but so is its climatology, so the two still agree.)
Shannon Entropy and Predictability Components#
Now we have all the necessary ingredients to define predictability (forecast probability and climatology probability). To quantify the predictability, we still need some metrics. Usually, we define the predictability limit as the moment when \(p(\mathbf{v})\) and \(p(\mathbf{v}|\mathbf{\theta})\) are not statistically different from each other. In other words, when \(\mathbf{\theta}\) and \(\mathbf{v}\) has no connection, we hit the predictability limit.
Traditionally, we can compare two PDFs and estimate if their mean are significantly different from each other by calculating the t-scores. The t-score/z-score contains the information of both mean and standard deviations. However, the t or z-score only compares means; it doesn’t tell us how different the two PDFs are as distributions, and it discards everything above the second moment (which is harmless only if both PDFs are Gaussian, as we assumed at the very beginning). Therefore, we will try something more general here.
Recall that the Liouville equation ((84)) predicts the time evolution of PDF. If we rewrite (84), we can have
where \(d/dt\) is the material derivative following the flow and \(\mathbf{\Phi}\) is the phase-space velocity (\(d\mathbf{x}/dt = \mathbf{\Phi}\)). The right hand side is the divergence of that flow: it measures how fast a phase-space volume element is being stretched or squeezed. So (166) says that the log-density of a parcel of probability changes only because the parcel’s volume changes. Probability mass is conserved; the probability density is not. The question is how to physically interpret the left hand side of (166)?
Here are a few examples… If today, we cast a six-sided dice, as long as we know outcome of one side (let’s say…it’s 1), the probability for the rest are determined (i.e., 5 other sides have 0 probability). On the other hand, if we only have 2-sided coin, when we know the outcome of one side, we determine the outcome of the other side at the same time. From these two examples, we can find that the six-sided dice provides more information as long as the result of one certain side is determined. On the other hand, the probability of knowing the outcome of a single side in six-sided dice is also lower. Combining both quantities, Claude Shannon introduced Shannon entropy in 1948, which is defined as the expected value of information.
The first term in the integral is the probability of a given scenario and the second term, \(-\mathrm{log}[p(\mathbf{x})]\), is how much information is carried by that scenario (its self-information). Entropy is simply the probability-weighted average of the two.
With \(H\) in hand we can return to (166). Multiplying it by \(\rho\) and integrating over the whole domain gives the rate of change of entropy,
Note the density weighting: what controls \(dH/dt\) is the divergence as sampled by the PDF, \(\langle \nabla \cdot \mathbf{\Phi}\rangle\), not \(\int \nabla \cdot \mathbf{\Phi}\, d\mathbf{x}\). For a non-divergent (volume-preserving, e.g. Hamiltonian) flow, \(\nabla \cdot \mathbf{\Phi} = 0\) everywhere and the entropy is exactly conserved: reversible deterministic dynamics neither create nor destroy information, they only deform the PDF. For a dissipative flow \(\nabla \cdot \mathbf{\Phi} < 0\) and the entropy actually decreases as phase-space volume contracts onto the attractor. This is worth pausing on, because it means the entropy growth we associate with the loss of predictability does not come from (166) at all. It comes from the stochastic forcing \(\mathbf{\epsilon}\) in (160), which is precisely the piece of the dynamics that the Liouville equation leaves out.
In computational science, we mostly use binary information (i.e., 1 or 0) to transmit data. For a fair two-state variable the sum in (167) has two equal terms, \(H = -\sum_{i=1}^{2}\frac{1}{2}\mathrm{log}_{b}\frac{1}{2} = \mathrm{log}_{b}2\). When \(b=2\), this equals 1, which is the widely used unit in computational science, bit.
Note
When Claude Shannon proposed using (167) to quantify the dispersion of a PDF, he consulted John von Neumann about how to name this metric. von Neumann told him that using entropy is a good idea because similar concepts have been used in statistical mechanics and people don’t really know what entropy is.
Be careful not to confuse the two pieces of (167). An unlikely event carries more self-information, \(-\mathrm{log}\,p\), but that does not make the entropy lower. Entropy is a property of the entire distribution, and what makes it low is the distribution being sharply peaked (confident), not any individual outcome being rare. It’s like buying a lottery ticket. If someone tells you a certain combination of numbers will win, your distribution over outcomes collapses from “every combination equally likely” (high entropy) onto a single combination (zero entropy), and you would put your money on that one rather than spread it over the others. Knowing this information is equivalent to the probability changing when we have an observation, and the magnitude of that change is measured by the shift in \(-\mathrm{log} [p(\mathbf{x})]\).
By analogy with thermodynamics we expect the forecast distribution to broaden with lead time until it is indistinguishable from climatology, and the process to be irreversible. Two cautions are in order. First, for a Markov process the quantity guaranteed to be monotone is not the forecast entropy but the relative entropy relative to the stationary distribution, which decreases monotonically (the H-theorem, see (171) below); the forecast entropy itself can decrease if we start an ensemble whose spread is wider than climatology. Second, this irreversibility is a property of the forecast PDF, not of the underlying dynamics, which by (166) may be perfectly reversible. To measure how much information we still hold at lead time \(\tau\), we define
where
is the entropy of forecast distribution. The quantity \(P_{\theta}\) is the predictive information, which measures how much narrower the forecast distribution is than climatology, i.e. how much room we still have before we hit the predictability limit. It therefore decreases toward zero as the lead time grows. \(P_\theta\) depends on two things separately: on the lead time \(\tau\), and on which observation \(\theta\) we happen to have made. One caveat: \(P_\theta \geq 0\) is not guaranteed for an individual \(\theta\), since an unlucky observation can leave the forecast broader than climatology. Only the average over \(\theta\) ((172) below) is guaranteed to be non-negative.
An alternative metric for estimating the difference between two PDFs is the Kullback–Leibler distance, which is defined as
Technically, the Kullback–Leibler distance is not really a distance (it is not symmetric and does not obey the triangle inequality) but it still measures how different the two PDFs are. One key element of the Kullback–Leibler distance is the second term inside the integral, which is the ratio between two PDFs. Therefore, the Kullback–Leibler distance is also called relative entropy. For a Markov process, the relative entropy with respect to the stationary distribution decreases monotonically with lead time, which is the H-theorem mentioned above.
Remarkably, both the predictive information and the relative entropy are invariant with respect to invertible linear transformations of the state. This allows us to combine variables with different unit into a single measure of predictability without introducing weighting factors since such factors cannot affect final values. Relative entropy has the additional property of being invariant under invertible nonlinear transformations of the state. The reason for the difference is worth spelling out: under a change of variables the differential entropy picks up a Jacobian term \(\mathrm{log}|\mathrm{det}\,\mathbf{J}|\), which cancels between \(H(\mathbf{v})\) and \(H_\theta(\mathbf{v}|\mathbf{\theta})\) in (169) only when \(\mathbf{J}\) is constant, i.e. only for linear maps. In (171) the Jacobians cancel inside the ratio \(p(\mathbf{v}|\mathbf{\theta})/p(\mathbf{v})\) for any invertible map. (This is also why differential entropy on its own is a slippery quantity: it is not coordinate-invariant and can even be negative, whereas relative entropy is always non-negative and coordinate-free.)
Another remarkable point is that if we average both metrics over \(\mathbf{\theta}\), weighted by \(p(\mathbf{\theta})\), we get exactly the same value, which is the mutual information between \(\mathbf{v}\) and \(\mathbf{\theta}\). i.e.,
that means the estimated predictability limit will be the same regardless of which metric is used here. Note that the weight \(p(\mathbf{\theta})\) is essential: without it the two integrals are not averages and the identity below does not go through. We can take a detailed look of the first part of (172).
where the first term survives the integral because \(\int p(\mathbf{\theta})d\mathbf{\theta}=1\), and \(H(\mathbf{v}|\mathbf{\theta}) \equiv \int H_\theta(\mathbf{v}|\mathbf{\theta}) p(\mathbf{\theta}) d\mathbf{\theta}\) is the conditional entropy. (172) and (173) rigorously state that the predictability can be stated in two different ways: (1) the difference between the forecast and climatological distribution and (2) the statistical dependence between \(\mathbf{v}\) and \(\mathbf{\theta}\). Because when \(\mathbf{v}\) and \(\mathbf{\theta}\) lose their statistical connection, \(H(\mathbf{v}|\mathbf{\theta})\) will reduce to \(H(\mathbf{v})\) and the difference between \(H(\mathbf{v})\) and \(H_\theta(\mathbf{v}|\mathbf{\theta})\) vanishes.
To simplify the story, here we only focus on the normal distribution. The climatological distribution, observation and forecast distribution are then defined as
where \(\mathbf{\mu}\) is the mean vector and \(\Sigma\) is the covariance matrix. Because our metrics are invariant under invertible linear transformations, we are free to work in whichever linear coordinates are convenient. In particular we may project the state onto a single direction \(\mathbf{q}\) (we will choose \(\mathbf{q}\) shortly) and whiten it so that the climatological variance is one. The covariance matrices then collapse to scalar variances and (174) becomes
Now we will formulate a single equation, which reconciles the concepts over different metrics and can be used to calculated predictability.
But…first, let’s take a look of the old-school method. Traditionally, the measure of predictability is based on the mean square difference between the forecast and the truth. In a perfect model, truth is chosen from a randomly sampled ensemble (it’s like multi universes!). Let \(\epsilon\) be the difference between two independently drawn members of the forecast distribution, \(\mathbf{f_1}\) and \(\mathbf{f_2}\) (one forecast and one truth), whose common mean is \(\mathbf{\mu} \equiv \mathbf{\mu}_{\mathbf{v}|\mathbf{\theta}}\). Then the mean square error is defined as,
where the cross term drops out precisely because \(\mathbf{f_1}\) and \(\mathbf{f_2}\) are drawn independently.
As discussed previously, the measure of predictability quantifies the degree to which the forecast and the climatological distribution are different. We therefore normalize the forecast MSE by the same quantity computed from two independent draws of the climatological distribution, \(\mathbf{v_1}\) and \(\mathbf{v_2}\),
so that \(\Delta_\theta\) runs from \(0\) (a perfect forecast) to \(1\) (the predictability limit). Note that comparing against \(\overline{||\mathbf{v}-\mathbf{\mu}||^2} = \mathrm{Tr}[\Sigma_{\mathbf{v}}]\) instead would be off by a factor of two; the numerator and denominator must be the same kind of object.
The above metric has some limitations. First, the results depend on the chosen coordinate, which is used to represent the dynamical system. Second, if the chosen variable is not representative enough, it fails to tell the difference between forecast and the climatological PDFs. However, later we will show that with a certain linear transformation, the traditional method and the information theory-based method can give us the same result.
For the information theory based method, suppose there is a basis (spatial structure), \(\mathbf{q}\), which maximizes the signal at a given forecast lead time. Then the mean and variance of that basis’s time series can be written as
Then the metrics we used previously can be written as
The factors of \(\frac{1}{2}\) come from the Gaussian entropy \(H=\frac{1}{2}\mathrm{log}(2\pi e \sigma^2)\); the \(2\pi e\) cancels in every difference, leaving only the variance ratio. It is instructive to verify (172) directly here. Averaging \(R_\theta\) over \(\theta\) and using the signal–noise decomposition (164), we have \(\langle(\mu_{v|\theta}-\mu_v)^2\rangle = \sigma_v^2-\sigma_{v|\theta}^2\), so the last three terms inside the bracket become \(\frac{\sigma_{v|\theta}^{2}}{\sigma_{v}^{2}}-1+\frac{\sigma_v^2-\sigma_{v|\theta}^2}{\sigma_v^2}=0\) and \(\langle R_\theta \rangle = \langle P_\theta \rangle = M\) exactly, as promised.
We can find that across the four different metrics, they are essentially all controlled by one single parameter: \(\frac{\sigma_{v|\theta}^{2}}{\sigma_{v}^{2}}\). (Strictly, \(R_\theta\) for a single \(\theta\) also contains the signal term \((\mu_{v|\theta}-\mu_v)^2/\sigma_v^2\), but as we just saw that term vanishes on average, so the ranking of patterns is unaffected.) Given that the variable is whitened (standardized, or long-term variance = 1), \(\sigma_{v}^{2}\) will be 1. Thus, the only term that determines predictability is the whitened forecast error variance \(\sigma_{v|\theta}^{2}\), which is why that single quantity carries all the weight in what follows.
It’s not hard to find, for all four equations, if we can find a pattern that minimizes \(\sigma_{v|\theta}^{2}\), it will be the most predictable pattern. What we are minimizing is the Rayleigh quotient
whose stationary points solve a generalized eigenvalue problem,
which collapses to an ordinary eigenvalue problem \(\Sigma_{v|\theta}\mathbf{q}=\lambda\mathbf{q}\) once the state has been whitened so that \(\Sigma_{v}=\mathbf{I}\). The eigenvector with the smallest eigenvalue is the projection vector giving the longest predictability.
Note
Keep in mind that \(\mathbf{q}\) is the filter (the weights we apply to the field to extract the predictable time series), not the pattern we plot. The corresponding spatial pattern is obtained by regressing the field onto the time series \(\mathbf{q}^{T}\mathbf{x}\). The two coincide only when \(\Sigma_{v}=\mathbf{I}\).
Average Predictability Time#
From (179), we know that \(\sigma_{v|\theta}^{2}\) will approach \(\sigma_{v}^{2}\) (i.e. 1 after whitening) as \(\tau\) approaches \(\infty\). Because when forecast lead time is long enough, the forecast error will approach the variance of the climatological distribution. All four metrics then go to zero, so each of them defines a predictability limit. The awkward part is that “the limit” is whichever lead time we declare the ratio to be close enough to one, which is arbitrary. A cleaner summary is to integrate the entire decay curve, which is the idea behind the average predictability time (APT),
In addition, for a linear Markov (red noise) process the forecast signal decays exponentially with the increase of lead time, i.e. \(\mu_{v|\theta}(\tau)=\mu_{v|\theta}(0)\,\mathrm{e}^{a\tau}\), where \(\mu_{v|\theta}(0)\) is the ensemble mean at \(\tau=0\). In the multivariate case this scalar form deserves a word of care: projecting the matrix solution gives \(\mathbf{q}^{T}\mathrm{e}^{\mathbf{A}\tau}\mathbf{\mu}_{v|\theta}(0)\), which collapses to a single exponential only when \(\mathbf{q}\) is an eigenvector of \(\mathbf{A}\). To keep things transparent, the rest of this section works with the scalar red-noise case with \(a<0\).
Combining \(\mu_{v|\theta}(\tau)=\mu_{v|\theta}(0)\mathrm{e}^{a\tau}\) and the definition of \(\sigma_{v|\theta}^{2}\), we can have
Two steps deserve comment. The cross term between the initial-condition contribution and the noise integral vanishes because the forcing is independent of the initial state, and the double integral collapses to a single one because the forcing is white, \(\langle \epsilon(s)\epsilon(s')\rangle = \sigma_{\epsilon}^{2}\delta(s-s')\). The last line simply substitutes \(a=-\|a\|\), which holds for the red noise problem.
If we further specify \(\sigma_{v|\theta,\tau=0}^{2}=0\) (a perfectly observed initial state), we find \(\sigma_{v|\theta}^{2} = \frac{1-\mathrm{e}^{-2||a||\tau}}{2||a||}\sigma_{\epsilon}^2\). Letting \(\tau\rightarrow\infty\) identifies the climatological variance as \(\sigma_{v}^{2}=\sigma_{\epsilon}^{2}/2\|a\|\), exactly the saturated forecast variance we claimed earlier. The ratio that governs every metric in (179) is therefore remarkably simple,
and substituting into (182) gives
So the average predictability time is just the e-folding time of the damping, and \(\|a\|\) tells us how far we can still leverage \(v|\theta\) to inform the future states. (With an imperfect initial condition, \(\sigma_{v|\theta,\tau=0}^{2}>0\), the same calculation gives \(\mathrm{APT}=(1-\sigma_{v|\theta,\tau=0}^{2}/\sigma_{v}^{2})/\|a\|\), i.e. a worse analysis buys proportionally less predictability.)
Finally, note that maximizing APT over the projection vector \(\mathbf{q}\) leads to the same kind of generalized eigenvalue problem as (181), except that the forecast error covariance is now integrated over all lead times rather than evaluated at one. This is what makes APT attractive in practice: it ranks patterns by their predictability across the whole forecast range instead of at a lead time we had to pick by hand.
Note
In the homework assignment, you will use the ECMWF reforecast data to find the predictable components and the corresponding average predictability time. Do you find what pattern shows up over the Pacific- North America regions?