Stationarity¶

Introduction¶

• Previously, we learned that time series is data that is measured at equally-spaced intervals.
• We also learned that time series data can be decomposed into three parts: trend, seasonality, and residuals.
• Now we'll dig deeper by discussing an important concept called stationarity.
• Having a firm grasp of this concept will lay the foundation necessary to understand more advanced modeling strategies that we'll introduce later on.

In this lecture we will cover

• Definition of stationarity and examples of common stationary and nonstationary time series.
• How to detect stationarity.
• The most common ways to transform nonstationary time series into stationary ones.

---

Stationary Time Series¶

Why is stationarity important?¶

• Stationarity is a fundamental assumption in many time series forecasting models.
• Without it many basic time series models would break.
• Transformations can be applied to convert a nonstationary time series to a stationary one.
• There are advanced time series models that can handle nonstationary data (beyond the scope of this lecture).

What is stationarity?¶

• 💡 Intuition: a time series is stationary when there are no changes in the underlying system that generates it.
• In the following, we define:
  • Strict stationarity.
  • Weak stationarity.
  • Things to check in practice.

Strict stationarity¶

• A stochastic process $X(t): t \in \mathcal{T}$ is called strictly stationary if, for all $t_1, t_2, \dots, t_n \in \mathcal{T}$ and $h$ such that $t_1+h, t_2+h, \dots, t_n+h \in \mathcal{T}$, it holds that

$$\big(X(t_1),X(t_2), \dots, X(t_n)\big) \stackrel{\text{D}}{=} \big(X(t_1+h),X(t_2+h), \dots, X(t_n+h)\big)$$

• where $\stackrel{\text{D}}{=}$ indicates that the distributions are equal.
• In plain words, the joint distribution of any set of observations in the series remains the same regardless of the time at which the observations are taken.

• Strict stationarity is very difficult to verify, especially from a limited amount of data.
• For this reason, one checks the Weak stationarity.

Weak stationarity¶

• A time series $X$ is weakly stationary if:

  1. The mean is constant over time $\mathbb{E}[X(t)] = m, \;\; \forall t \in \mathcal{T}$
  2. The variance is finite $\mathbb{E}[X(t)^2] < \infty, \;\; \forall t \in \mathcal{T}$
  3. The covariance of $X(t)$ and $X(t+h)$ depends only on $h$, i.e., is shift-invariance in time.

• Strict stationarity + condition 2 $\implies$ weak stationarity

Things to check in practice.¶

1. Constant mean (no trend).
2. Constant variance (no heteroskedasticity).
3. Constant autocorrelation structure.
4. No periodic component (✋ careful here)

• Mean, variance, and periodic component (aka seasonality) should be familiar from the previous lecture.
• Autocorrelation may not be.
• Let's start with a simple example of a stationary time series.

• Let's create an utility function to make plots.

• First, we create a variable time that defines equally-spaced time intervals.
• Then we'll sample values from a Normal distribution with a mean of 0 and a standard deviation of 1.
• Essentially we're creating white noise, which is a stationary time series as it exhibits all the properties required of a stationary time series.

• We can easily check that the white noise has constant mean, constant variance, and no periodicity.
• What about the autocorrelation?

Autocorrelation¶

• Autocorrelation measures how much the current time series measurement is correlated with a past measurement.
• For example, today's temperature is often highly correlated with yesterday's temperature.
• To discuss these things we must introduce the idea of a lag, i.e., a delay.

• Say you wanted to know if today's temperature correlates more with yesterday's temperature or the temperature from two days ago.
• You could test this by computing the correlation between the original time series and the same series delayed 1 time step.
• The second value of the original time series would be compared with the first of the delayed time series.
• The third original value would be compared with the second of the delayed.
• And so on.

• Performing this process for a lag of 1 and a lag of 2, respectively, would yield two correlation outputs.
• This output would tell you which lag is more correlated.
• The process is usually repeated for $N$ different lags.
• Autocorrelation often results in a pattern, whereas a time series without autocorrelation looks random.

The covariance of $X(t_1)$ and $X(t_2)$ is called autocovariance and is slightly different from the autocorrelation.

• Autocorrelation:

$$R_{xx}(t_1,t_2)=\mathbb{E}[X(t_1)X(t_2)]$$

• Autocovariance:

$$C_{xx}(t_1,t_2)=\mathbb{E}[(X(t_1)−\mu_{t_1})(X(t_2)−\mu_{t_2})]=R_{xx}(t_1,t_2)−\mu_{t_1}\mu_{t_2}$$

• $\mu_{t_1}$ and $\mu_{t_2}$ are the means of the time series at times $t_1$ and $t_2$, respectively.
• If $X(t)$ has zero-mean, autocorrelation and autocovariance coincide.
• Since obtaining a time series with zero-mean is trivial, we'll use the two terms interchangeably.

• Assumption #3 in weak stationarity requires the time series to have constant (time-independent) autocovariance.
• If the autocovariance remains constant throughout the series, a simple transformation can be used to remove it.
• This will be useful for several future models.

Example: constant autocorrelation

• Time series of daily temperature in a specific location over a year.
• If this location has a very stable climate, the day-to-day changes in temperature might be minimal and consistent throughout the year.
• Then, the autocorrelation would be relatively constant.
• E.g., the correlation between today's temperature and tomorrow's temperature is roughly the same as the correlation between the temperature 30 days ago and 29 days ago.

Example: time-varying autocorrelation

• Time series representing the daily stock prices of a particular company.
• Stock prices can be influenced by market trends, company performance, economic indicators, world events, etc...
• Here, the autocorrelation might be time-dependent.
• E.g., during a period of market stability, the stock prices might show a consistent pattern over time (e.g., a steady increase).
• However, during a market upheaval or a significant company event (like a product launch or a scandal), the pattern could change dramatically.

Constant vs non-constant autocorrelation¶

• Consider two time series $X$ and $Y$.
  • In $X$ the autocorrelation is constant over time.
  • In $Y$ the autocorrelation changes after 100 steps.
• We will use an AR process for this example (more details on AR later on).

• Notice how the first half of $Y$ shows a stronger autocorrelation.
• Each value is heavily influenced by its immediate predecessor, leading to a smooth and predictable pattern.

• The second part is characterized by a weaker autocorrelation.
• The time series is more erratic and less influenced by its past values.

Common stationary and nonstationary time series¶

• If the analytic expression of the time series is known, it is possible to check the conditions for stationarity.
• In practice, the process generating the time series might be unknown and we'll need other tools to check stationarity.
• Starting the analysis by looking at the run_sequence_plot is an excellent starting point.

White noise¶

• We already looked at this time series and conclude that mean and variance are constant and there is no seasonal component.
• In addition, since the value of each time step is drawn independently, there is no correlation.

Random walk¶

• The random walk is one of the most important nonstationary time series.
• It is defined as:

$$X(t) = X(t-1) + \epsilon_t$$

• where $\epsilon_t$ are called innovations and are iid, e.g., $\epsilon_t \sim \mathcal{N}(0,\sigma^2)$.
• Let's create a random walk.

• Notice how the current value is highly correlated with the past values and fluctuates much less than the white noise.
• The random walk is nonstationary.
• Why 🤔?

Dependence on Initial Value

• The value of the series at any point in time depends on its initial value and the sum of all previous innovations.
• Clearly, this path dependence denotes nonstationarity.

Changing variance

• In a random walk, the variance changes over time.
• This violates condition #2 of weak stationarity.
• In particular, the variance increases linearly as time progresses because the effects of the innovations accumulate.
• Let's see this in detail.

1. We start at $t = 0$ and we let $X(0) = 0$ for simplicity.

2. At $t = 1$ we have:
   • $X(1) = X(0) + \epsilon_1$.
   • Variance: $\text{Var}(X(1)) = \text{Var}(\epsilon_1) = \sigma^2$.

3. At $t = 2$ we have:

   • $X(2) = X(1) + \epsilon_2 = X(0) + \epsilon_1 + \epsilon_2$.
   • Since $\epsilon_1$ and $\epsilon_2$ are independent we get :
   $$\text{Var}(X(2)) = \text{Var}(\epsilon_1 + \epsilon_2) = \text{Var}(\epsilon_1) + \text{Var}(\epsilon_2) = \sigma^2 + \sigma^2 = 2\sigma^2$$

4. At a general time $t$ we have:

   • $X(t) = X(t-1) + \epsilon_t = X(0) + \epsilon_1 + \epsilon_2 + \ldots + \epsilon_t$.
   • Again, using the independence of the $\epsilon$ terms and the fact that each has variance $\sigma^2$ we get:
   $$\text{Var}(X(t)) = \text{Var}(\epsilon_1) + \text{Var}(\epsilon_2) + \ldots + \text{Var}(\epsilon_t) = t\sigma^2$$

Time series with trend¶

• Consider the following data.

• Is this data stationary?

• No, the mean changes over time and violates condition #1.

Heteroscedastic time series¶

• Is this data stationary?

• No, the variance changes over time.

Time series with seasonality¶

• Is this time series stationary?

• The mean of a periodic signal over a complete cycle always assumes the same value (20 in our case).
• However, over an half-cycle or any period not equal to an integer multiple of the full cycle, the mean changes (is different from 20).
• This violates condition #1.

Time series with trend and seasonality¶

• Is this data not stationary?

• No, the mean changes over time because of the trend AND there is a periodic component.

Sinusoidal signal with random phase¶

• Consider the sinusoidal signal

$$X(t) = \mu + R \sin(\lambda t + \psi)$$

• where $\mu$ is the mean, $R$ is the amplitude, $\lambda$ is the frequency, and $\psi$ is the phase uniformly distributed over the range $[- \pi, \pi]$.

• Is this a periodic signal?
• Does it satisfy the conditions for weak stationarity?

• Due to the periodic and symmetric nature of the sine function, when $\psi$ is uniformly distributed over $[-\pi, \pi]$, every value of $\psi$ that leads to a positive value is balanced by a corresponding value of $\psi$ that leads to a negative value of the same magnitude.
• This balance means that $\mathbb{E}\big[ \sin(\lambda t + \psi) \big]=0$, therefore:

$$\begin{aligned} \mathbb{E}[x_t]&=\mu + R~\mathbb{E}[\sin(\lambda t + \psi)]= \mu + 0 = \mu \end{aligned}$$

• The mean of $\mu + R \sin(\lambda t + \psi)$ is $\mu$, which is constant.
• Condition #1 is satisfied ✌
• Let's verify it also with code.

• The variance of a random variable $X$ given by:

$$\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$$

• Let's first consider $\sin(\lambda t + \psi)$ for which we know $\mathbb{E}[\sin(\lambda t + \psi)] = 0$
• The variance simplifies to:

$$\text{Var}(\sin(\lambda t + \psi)) = \mathbb{E}[\sin^2(\lambda t + \psi)]$$

• Since $\psi$ is uniformly distributed over $[- \pi, \pi]$, the pdf of $\psi$ is:

$$p(\psi) = \frac{1}{2\pi}$$

• Therefore, the variance of $\sin^2(\lambda t + \psi)$ is:

$$\begin{align*} \mathbb{E}[\sin^2(\lambda t + \psi)] & = \int_{-\pi}^{\pi} \sin^2(\lambda t + \psi) p(\psi) d\psi\\ &= \frac{1}{2\pi} \int_{-\pi}^{\pi} \sin^2(\lambda t + \psi) d\psi \end{align*}$$

• For any $x$, we have that $\int_{-\pi}^{\pi} \sin^2(x) dx = \pi$

• This, gives us:

$$\text{Var}(\sin(\lambda t + \psi)) = \mathbb{E}[\sin^2(\lambda t + \psi)] = \frac{1}{2\pi}\pi = \frac{1}{2}$$

• It is easy to see that $\text{Var}[\mu + R\sin(\lambda t + \psi)]$ is $\frac{R^2}{2}$.
• We conclude that the variance is finite and constant.
• Also condition #2 is satisfied ✌

• The autocovariance function for a lag $h$ is defined as:

$$\text{Cov}(X(t), X(t+h)) = \mathbb{E}[(X(t) - \mu)(X(t+\tau) - \mu)]$$

• Since $X(t) = \mu + R \sin(\lambda t + \psi)$, the autocovariance becomes:

$$\text{Cov}(X(t), X(t+h)) = \mathbb{E}\left[\left(R \sin(\lambda t + \psi)\right)\left(R \sin(\lambda (t + \tau) + \psi)\right)\right]$$ $$= R^2 \mathbb{E}\left[\sin(\lambda t + \psi) \sin(\lambda (t + \tau) + \psi)\right]$$

• Now, we use the trigonometric identity:

$$\sin(A) \sin(B) = \frac{1}{2}[\cos(A - B) - \cos(A + B)]$$

• Applying this identity, we get:

$$\begin{align*} \text{Cov}(x_t, x_{t+h}) & = R^2 \mathbb{E}\left[\frac{1}{2}[\cos(\lambda t + \psi - \lambda (t + h) - \psi) - \cos(\lambda t + \psi + \lambda (t + h) + \psi)]\right]\\ & = \frac{R^2}{2} \mathbb{E}\left[\cos(\lambda h) - \cos(2\lambda t + 2\psi + \lambda h)\right]\\ & = \frac{R^2}{2} \mathbb{E}\left[\cos(\lambda h)\right] - \frac{R^2}{2} \mathbb{E}\left[\cos(2\lambda t + 2\psi + \lambda h)\right] \end{align*}$$

• The expectation is taken over the random variable $\psi$.
• Following the same reasoning we did for the mean, $\cos(2\lambda t + 2\psi + \lambda \tau)$ averages out to zero because $\psi$ is uniformly distributed over $[- \pi, \pi]$.
• Thus:

$$\text{Cov}(x_t, x_{t+h}) = \frac{R^2}{2} \cos(\lambda h)$$

• So, the autocovariance function depends only on the lag $h$ and is independent of time $t$.
• Also condition #3 is satisfied ✌.

In conclusion, the time series $X(t) = \mu + R \sin(\lambda t + \psi)$ is weakly stationary:

• its mean is constant,
• its variance is constant,
• its autocovariance depends only on the lag $h$.

How does $X(t) = \mu + R \sin(\lambda t + \psi)$ looks like?

📨 Take-away message

• A time series obtained from a periodic function might be weakly stationary.
• The presence of periodic functions is not explicitly stated in the 3 conditions.
• Be careful when you determine weak stationarity based on periodic components.

---

Detecting stationarity¶

Next, we will review practical techniques used to identify if a time series is stationary or not.

Specifically, we will cover:

1. Run-sequence plots
2. Summary statistics
3. Augmented Dickey-Fuller test

Run-Sequence Plots¶

• Run-sequence plots allows to visualize what's happening with the data.
• This should always be your starting point.
• Indeed, that is what we have been doing so far with our function run_sequence_plot.
• It often shows whether there is an underlying structure.
• Be on the lookout for trend, seasonality, and autocorrelation.

Summary Statistics¶

• An easy way to check for constant mean and variance is to chop up the data into separate chunks.
• Then, one calculates statistics for each chunk, and compare them.
• Large deviations in either the mean or the variance among chunks might indicate that the time series is nonstationary.
• It's not a rigorous method but gives a good hint of whether the data is approximately stationary or not.

This is how a stationary time series looks like.

This is how a non-stationary time series looks like.

• Let's look at an example based on the trend time series that we created before.

Summary statistics:

• While the variance is relatively constant over time, the mean is clearly changing, which means this data is not stationary.
• We do expect some fluctuation in values.
• It's unlikely that mean and variance will be exactly the same in all chunk, but they should be close.

• To see this better, it's a good idea to plot the means and variances.
• If you want to be more precise, you could run a statistical test to determine if the difference in means or the difference in variances is statistically significant.

Augmented Dickey-Fuller Test¶

• This is a statistical procedure to determine whether a time series is stationary or not.
• We will discuss more details about the test in the next lectures.
• For now, that's what we need to know:
  1. Null hypothesis: $H_0$: the series is nonstationary.
  2. Alternative hypothesis: $H_A$: the series is stationary.

• Like any statistical test you should set a significance level or threshold that determines whether you should accept or reject the null.
• The values $\alpha=0.05$ or $\alpha=0.01$, which yield a confidence of $95\%$ and $99\%$, respectively, are common choices.

Example: $p = 0.0001 \rightarrow$ reject $H_0 \rightarrow$ the time series is stationary

Example: $p = 0.43 \rightarrow$ fail to reject $H_0 \rightarrow$ the time series is non-stationary

• The ADF test might be inaccurate with small datasets or when heteroskedasticity is present.
• It is best to pair ADF with other techniques (run-sequence plots, summary statistics, histograms, etc..).

Stationary Data & ADF¶

• Let's see how to run this test in Python.

• Let's see and comment the main outputs from the ADF test.

• First, adf is the value of the test statistic.
• The more negative the value, the more confident we can be that the series is stationary.
• Here we see a negative value.
• That may not mean anything to you just yet but pvalue should.

• The pvalue is interpreted like any p-value.
• The pvalue should be compared with the confidence levels (e.g., $\alpha=0.05$ or $\alpha=0.01$).
• Based on the comparison, we reject or fail to reject $H_0$.
• Here pvalue is very close to zero so we reject the $H_0$ in favor of $H_A$ and conclude that the time series is stationary.

• The variable nobs is simply the number of observations in the time series.

• Finally, critical_values are the test statistic thresholds for common significant levels.
• Here we see a test statistic lower than -2.87 is sufficient to reject $H_0$ at a significance level of 5%.

• You may have noticed two outputs we didn't discuss.
• See statsmodels docs for a description of usedlag and icbest.

ADF on time series with trend¶

• Next, we run the ADF test on the trend time series.

• As expected, we fail to reject $H_0$ that the trend time series is nonstationary.

ADF on random walk¶

• Finally, we run the ADF test on the rand_walk time series.

• As we discussed before, the random walk is nonstationary because its variance $\sigma^2t$ is not constant over time.
• In the next section, we'll learn how to make this time series stationary.

• Good news 😎. At this point, we have several tools at our disposal to figure out if a time series is stationary or not.

---

Common transformations to achieve stationarity¶

• There are several ways to transform nonstationary time series data:

Transformation	Effect
Subtract trend	Constant mean
Apply log	Constant variance
Differencing	Remove autocorrelation
Seasonal differencing	Remove periodic component

• Often you will have to do several of these on one dataset.
• Note that this is just a small subset of the total possible ways to transform nonstationary time series.

Subtract estimated trend & seasonality¶

• Let's run the ADF test before transforming.

• Nonstationary as expected.
• Next, we apply the transformation.

• Let's look at the residual values.

• We'll have to remove the first $6$ and last $6$ values to run an ADF test since those are nan.
• The number 6 is due to the fact that the method applies seasonal differencing and we specified period=12.
• Here's how to do that and run the test all in one call.

• Now the time series is stationary ✌.

Stabilize variance with log transform¶

• This trick is useful when dealing with heteroskedastic data.
• The log squashes the parts of the time series where the variance is higher.
• In this way, the variance becomes more uniform.
• Let's plot again the heteroskedastic times series to remember how it looks.

• Let's run the ADF test.

⚠ Attention

• ADF is saying we reject $H_0$ (at least at $\alpha=0.05$ confidence level) in favor of the alternative that the time series is stationary.
• However, the plot clearly shows that the variance is not constant over time.

• This is an important lesson.
• Some statistical tests like ADF perform bad in some situations:
  • with few data,
  • when data is heteroskedastic.

• Let's now apply the log transformation.
• Before that, we subtract the minimum and add a small constant
• In this way we ensure that all values are positive.

• Look at the values on the $y$ axis.
• Notice how the variance has collapsed to a large extent.

📨 Takeaway message:

• The ADF test can fail to detect nonstationarity due to heteroskedasticity.
• Always plot the data.
• Do not rely on a single tool to determine stationarity.

Removing autocorrelation with differencing¶

• Remember that the ADF test (correctly) indicated that the rand_walk time series is nonstationary.
• We can transform rand_walk into stationary time series by applying differencing.
• This is done by subtracting a past value from the current value.

• We know that rand_walk was created with a lag of 1.
• So, a stationary time series can be obtained as:

$$\hat{X}(t) = X(t) - X(t-1)$$

• Let's run ADF on the differenced data.

• Now it's stationary!

• Let's see another example.
• We create a small dataframe from 1-10 (linear and squared).

• Make stationary the first time series (original) so that mean and variance don't change for sub-windows.

💡 NOTE: This is similar to taking a first-order derivative.

• Notice that we need to differentiate twice on the exponential time series.
• Every time we do differencing we lose a bit of data.

• We can also try to make stationary the second time series (squared) with log.

• Works somewhat but certainly not as well as differencing.

• In general we do not know the optimal lag to perform differencing.
• How to determine the optimal order of differentiation will be covered later on.

Seasonal differentiation¶

• Consider a time series with a seasonality of period $L$.
• There will be a strong correlation between $X(t)$ and $X(t+L)$.
• Seasonal differentiation can remove the seasonality and make the time series stationary:

$$\hat{X}(t) = X(t+L) - X(t)$$

• After seasonal differencing the time series will become $L$ steps shorter.
• Note that we can perform the standard differentiation in cascade to seasonal differentiation if we need to remove also the trend on top of the seasonality.
• Let's see this on the trend_seasonality time series.

---

Summary¶

In this lecture we learned:

1. What does it mean for a time series to be stationary.
2. A set of notable stationary and nonstationary time series.
3. Common tools to identify stationarity.
4. Useful nonstationary-to-stationary transformations.

---

Exercises¶

Exercise 1¶

• Load the two time series stationarity_ts1 and stationarity_ts2 by running the code below.

1. Plot the two time series with a run sequence plot.
2. Based on the plot, discuss if each time series seems to be stationary or not.

Exercise 2¶

Use the following tools to determine if the time series stationarity_ts1 and stationarity_ts2 are stationary or not.

1. Summary statistics on the chunks.
2. Histogram plots.
3. Augmented Dickey-Fuller test.

Discuss the result obtained with each method.

Exercise 3¶

If either or both datasets from exercises one and two are nonstationary, apply the transformations you learned in this section to make them so. Then apply the methods you learned to ensure stationarity.