Replicate Dynamic Return Dependencies Across Industries: A Machine Learning Approach by David Rapach, Jack Strauss, Jun Tu and Guofu Zhou.

1) Use industry returns from Ken French

2) Forecast (for example) this month's Chemical industry return using last month's returns from all 30 industries

3) Use LASSO for predictor subset selection over the entire 1960-2016 period to determine that e.g. Beer is predicted by Food, Clothing, Coal

4) Use LASSO-selected predictors and simple linear regression to predict returns

5) Generate portfolios and run backtests.

• Predictor selection - finds same predictors except 2 industries. Possibly use of AICc instead of AIC (don't see an sklearn implementation that uses AICc)

• Prediction by industry - R-squareds line up pretty closely

• Portfolio performance, similar ballpark results. Maybe AICc/AIC; Also paper standardizes predictors, which is pretty standard. Finally, for some reason their mean returns don't line up to geometric mean annualized, they seem to be calculating something different.

• Replicating exactly is hard but it does replicate closely and perform well

6) Run various sklearn regressors to see which performs best, understand metrics that predict performance. MSE does not predict Sharpe. Kendall's tau, i.e. correlation of predicted vs. actual rankings, performs better.

7) Tune ElasticNet to get slightly better performance than Lasso/OLS

8) Run Keras NNs. They don't improve on Lasso/OLS or ElasticNet.

In [1]:
import os
import sys
import warnings
import numpy as np
import pandas as pd
import time
import copy
import random
from itertools import product

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' #Hide messy TensorFlow warnings
warnings.filterwarnings("ignore") #Hide messy numpy warnings

import sklearn
from sklearn.model_selection import train_test_split, KFold
from sklearn.metrics import confusion_matrix, mean_squared_error, explained_variance_score, r2_score, accuracy_score
from sklearn.linear_model import LinearRegression, Lasso, lasso_path, lars_path, LassoLarsIC
from sklearn.ensemble.forest import RandomForestRegressor
from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.utils.testing import all_estimators
import xgboost

from scipy.stats import chisquare, kendalltau

import tensorflow as tf
tf.set_random_seed(1764)
print(tf.__version__)
# confirm GPU is in use
with tf.device('/gpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)

with tf.Session() as sess:
print (sess.run(c))

import keras
from keras.layers.core import Dense, Activation
from keras.layers import Input
from keras.models import Model

from keras.layers.recurrent import LSTM, GRU
from keras.regularizers import l1
from keras.models import Sequential

#import ffn
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

# this will make dataframe display as sortable widget
# commented out because sortable tables not viewable using nbviewer
from beakerx import *

import plotly
# print (plotly.__version__) # requires version >= 1.9.0
from plotly.graph_objs import *
import plotly.figure_factory as ff

init_notebook_mode(connected=True)

random.seed(1764)
np.random.seed(1764)

1.8.0
[[22. 28.]
[49. 64.]]

Using TensorFlow backend.


### 1. Replicate paper¶

In [2]:
print("Loading data...")
data = data.set_index('yyyymm')
industries = list(data.columns)
# map industry names to col nums
ind_reverse_dict = dict([(industries[i], i) for i in range(len(industries))])

rfdata = rfdata.set_index('yyyymm')
data['rf'] = rfdata['RF']

# subtract risk-free rate
# create a response variable led by 1 period to predict
for ind in industries:
data[ind] = data[ind] - data['rf']

#for ind in industries:
#    data[ind+".3m"] = pd.rolling_mean(data[ind],3)

#for ind in industries:
#    data[ind+".6m"] = pd.rolling_mean(data[ind],6)

#for ind in industries:
#    data[ind+".12m"] = pd.rolling_mean(data[ind],12)

for ind in industries:

data = data.loc[data.index[data.index > 195911]]
data = data.drop(columns=['rf'])
data = data.dropna(axis=0, how='any')

nresponses = len(industries)
npredictors = data.shape[1]-nresponses

predictors = list(data.columns[:npredictors])
predictor_reverse_dict = dict([(predictors[i], i) for i in range(len(predictors))])

responses = list(data.columns[-nresponses:])
response_reverse_dict = dict([(responses[i], i) for i in range(len(responses))])

print(data.shape)


Loading data...
(697, 60)

Out[2]:
yyyymm
195912 2.01 -4.49
196001 -4.49 3.35
196002 3.35 -1.67
196003 -1.67 1.17
196004 1.17 8.20
196005 8.20 5.39
196006 5.39 -2.11
196007 -2.11 4.57
196008 4.57 -3.88
196009 -3.88 1.02
196010 1.02 9.46
196011 9.46 4.51
196012 4.51 4.70
196101 4.70 4.21
196102 4.21 4.64
196103 4.64 -1.39
196104 -1.39 4.20
196105 4.20 -2.17
196106 -2.17 2.72
196107 2.72 4.92
196108 4.92 -0.62
196109 -0.62 3.73
196110 3.73 5.28
196111 5.28 -3.69
196112 -3.69 -6.67
196201 -6.67 -0.25
196202 -0.25 0.98
196203 0.98 -4.59
196204 -4.59 -11.25
196205 -11.25 -8.75
... ... ...
201507 4.03 -4.37
201508 -4.37 -1.19
201509 -1.19 5.81
201510 5.81 0.11
201511 0.11 1.96
201512 1.96 -1.67
201601 -1.67 0.95
201602 0.95 4.69
201603 4.69 0.63
201604 0.63 2.06
201605 2.06 4.75
201606 4.75 -0.51
201607 -0.51 -0.52
201608 -0.52 -2.92
201609 -2.92 -0.33
201610 -0.33 -4.41
201611 -4.41 4.43
201612 4.43 0.95
201701 0.95 1.71
201702 1.71 0.52
201703 0.52 0.76
201704 0.76 1.63
201705 1.63 -2.65
201706 -2.65 1.52
201707 1.52 -2.77
201708 -2.77 0.43
201709 0.43 0.71
201710 0.71 4.15
201711 4.15 -0.10
201712 -0.10 2.27

697 rows × 2 columns

In [3]:
# exclude 2017 and later to tie to paper
data = data.loc[data.index[data.index < 201701]]
data = data.loc[data.index[data.index > 195911]]
data

Out[3]:
yyyymm
195912 2.01 0.35 -3.02 1.64 7.29 0.67 1.87 -1.97 3.08 0.74 ... 0.62 -6.18 -7.93 -9.41 -4.31 -5.33 -6.09 -10.08 -4.68 -3.98
196001 -4.49 -5.71 -2.05 1.21 -5.47 -7.84 -8.53 -6.68 -10.03 -4.77 ... 8.07 9.13 5.09 3.00 -0.94 1.42 4.00 1.81 -0.98 6.32
196002 3.35 -2.14 2.27 4.23 2.39 9.31 1.44 -0.02 -0.74 0.32 ... -0.21 -0.31 3.34 -2.43 -4.99 -1.37 -0.13 -3.88 0.05 -2.43
196003 -1.67 -2.94 -0.18 -0.65 2.18 -0.56 -2.59 1.26 -2.75 -6.79 ... -1.24 7.14 1.77 0.41 -2.13 0.45 -0.53 8.86 -0.64 0.55
196004 1.17 -2.16 1.35 6.46 -1.17 -1.27 0.21 1.49 -5.53 -1.10 ... 3.05 -1.75 11.90 2.85 0.90 1.65 3.11 0.80 -0.45 1.02
196005 8.20 -0.52 2.44 7.28 11.67 7.74 1.74 13.50 3.40 2.10 ... -0.58 -8.07 2.39 3.50 2.17 5.96 3.41 1.03 3.72 6.41
196006 5.39 0.47 4.73 2.24 0.02 6.38 -1.59 -0.40 0.45 4.04 ... -0.03 2.84 -2.02 -4.10 -3.11 -6.16 -2.99 -1.25 0.09 -5.95
196007 -2.11 -0.79 4.60 -4.72 0.23 -0.60 -1.10 -3.99 -6.80 -3.14 ... 6.94 5.69 2.71 1.18 1.98 4.51 2.85 2.05 3.47 3.48
196008 4.57 3.24 5.20 7.16 3.63 5.09 3.34 2.29 1.17 -0.84 ... -6.07 -3.53 -7.61 -7.37 -7.07 -8.44 -8.57 -1.90 -5.78 -4.21
196009 -3.88 -5.00 -2.09 -2.33 -6.20 -9.18 -4.23 -8.87 -6.70 -5.25 ... -0.08 4.62 -3.40 -1.85 -1.02 -4.22 0.31 -4.54 -0.40 0.38
196010 1.02 0.54 3.87 0.11 2.38 6.48 -3.50 -3.71 -1.59 -3.06 ... 4.06 9.49 8.19 5.31 5.35 9.72 6.50 4.40 7.71 4.01
196011 9.46 6.57 5.44 13.91 10.11 9.13 3.15 3.91 4.25 2.04 ... 12.29 8.18 4.29 5.57 2.27 2.06 2.05 2.08 5.56 3.80
196012 4.51 -0.31 3.54 7.77 7.41 1.76 3.28 6.06 2.85 0.52 ... 7.70 4.29 5.08 4.56 8.35 7.93 2.28 4.08 7.12 8.23
196101 4.70 5.23 8.77 0.56 9.47 4.36 5.94 5.86 6.46 11.21 ... 0.61 0.20 4.54 6.83 4.22 3.31 4.82 8.23 7.00 6.00
196102 4.21 8.16 5.41 22.33 2.15 5.90 7.84 5.05 2.13 6.81 ... 7.23 -0.20 2.31 -0.69 0.86 4.45 5.76 4.06 4.34 7.08
196103 4.64 2.55 5.60 7.18 4.77 6.34 3.08 3.60 0.92 5.92 ... 0.63 -0.12 2.19 -0.37 -1.62 3.08 0.22 4.23 1.38 -3.67
196104 -1.39 1.40 -0.23 -2.21 -6.37 2.66 2.60 -0.47 -1.47 -5.31 ... -1.22 -0.70 1.57 1.39 4.74 -0.04 4.31 -1.90 4.00 3.32
196105 4.20 5.38 3.39 -3.91 2.71 -0.02 6.80 2.10 5.50 5.47 ... -4.19 0.13 -3.31 -4.46 -4.57 -4.90 0.80 -5.63 -2.88 0.37
196106 -2.17 -3.12 3.97 -5.87 -3.85 3.43 -5.50 -3.58 -1.32 -3.36 ... 6.25 -8.25 0.56 -0.50 -0.32 -0.01 2.45 2.69 3.35 5.37
196107 2.72 0.88 5.95 -1.21 -2.55 1.97 2.03 3.27 2.95 1.53 ... 0.12 8.99 5.11 5.37 3.52 3.09 3.03 0.46 8.65 1.64
196108 4.92 3.20 7.74 0.89 0.89 10.45 5.21 3.70 2.35 5.77 ... -2.94 -6.04 1.01 -2.74 -1.16 -4.22 0.66 -6.21 -0.40 3.14
196109 -0.62 -1.48 -0.07 1.24 0.75 -3.05 -1.14 -1.48 -4.45 -4.25 ... 0.00 2.24 6.53 1.74 2.16 4.30 9.35 0.71 2.02 0.39
196110 3.73 -0.84 7.05 -5.26 0.99 -0.67 8.28 3.33 0.05 3.11 ... 8.34 8.27 0.99 2.05 0.47 5.65 4.90 1.08 7.22 1.69
196111 5.28 4.47 8.03 0.25 3.75 4.51 5.30 3.12 2.49 7.37 ... 3.14 -0.68 -0.55 -2.65 -0.24 0.46 -0.63 -2.21 -4.44 -0.77
196112 -3.69 1.41 -6.12 1.97 -3.66 -3.78 0.32 -2.21 -0.16 -1.17 ... -6.32 -4.88 -6.91 -5.22 2.52 -0.79 -9.56 -3.90 -4.99 -3.62
196201 -6.67 -3.45 -4.28 -13.23 -3.44 -7.37 -5.89 -4.86 -4.76 0.57 ... 3.89 -1.94 -0.07 3.87 0.32 -0.09 1.58 -0.59 3.59 4.20
196202 -0.25 0.28 0.68 -2.02 -0.52 -0.90 2.01 3.56 3.30 1.93 ... -3.14 -1.41 -0.76 1.13 -1.68 -2.30 0.90 -4.07 -2.13 -1.83
196203 0.98 -0.34 -6.67 -5.34 0.41 4.31 -1.18 0.34 -2.72 -0.74 ... -4.93 -3.11 -12.01 -7.93 -6.27 -5.78 -4.61 -9.09 -7.69 -2.12
196204 -4.59 -3.59 -12.99 -11.04 -8.74 -7.03 -8.01 -11.23 -6.23 -7.53 ... -7.35 -10.97 -13.19 -11.03 -5.17 -11.34 -9.09 -7.46 -10.02 -11.83
196205 -11.25 -9.05 -14.14 -11.39 -14.87 -10.19 -10.01 -11.14 -8.25 -7.50 ... -8.72 -13.11 -12.59 -9.62 -7.81 -11.11 -10.43 -12.90 -11.01 -14.25
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
201407 -5.83 -2.92 -3.48 -3.70 -1.60 -2.62 -0.62 -0.33 -3.26 -5.70 ... 0.52 3.49 5.10 3.38 4.57 4.87 5.71 3.32 3.81 7.19
201408 6.38 5.46 5.08 -1.80 2.64 5.87 3.53 5.44 4.39 10.14 ... -1.70 -1.20 -1.91 -3.49 0.71 -2.29 -1.50 -0.60 -0.59 0.16
201409 -0.53 0.83 2.15 -4.38 -7.29 0.84 4.44 -0.08 -2.14 -2.38 ... 1.42 0.46 2.83 6.42 6.22 3.32 3.32 1.28 3.87 1.58
201410 1.70 3.28 6.13 0.58 3.28 3.71 1.57 5.77 -1.09 2.08 ... 3.03 2.05 6.47 4.80 5.88 1.83 8.91 5.57 1.93 4.55
201411 5.68 4.35 0.76 -0.40 1.11 3.23 8.89 2.67 2.21 7.53 ... -1.73 -0.60 -1.87 2.08 1.09 0.26 2.25 -0.43 2.08 0.17
201412 -2.48 -4.30 -3.11 -4.60 1.83 0.02 -0.82 -0.89 0.83 -0.31 ... -4.89 -4.44 -2.14 -2.43 -4.53 -2.37 -0.29 0.66 -7.61 -4.11
201501 -1.64 0.90 2.95 0.25 -2.18 -4.92 -3.75 1.56 -2.01 2.02 ... 9.12 7.92 8.43 6.68 3.10 5.37 5.58 6.82 7.93 4.63
201502 4.44 4.40 5.47 5.87 9.00 3.83 5.43 4.31 8.10 13.34 ... -2.20 -1.52 -3.12 -1.73 -3.62 0.57 1.00 -0.29 0.11 -1.77
201503 -0.72 -2.07 -8.82 -2.60 0.56 -1.98 1.23 0.86 -3.55 3.33 ... 3.40 2.25 0.10 -2.65 -1.14 -1.23 -2.88 0.51 0.76 -0.23
201504 -0.17 -0.52 5.94 3.75 -4.11 -2.41 -1.53 -1.39 1.11 -5.95 ... 0.81 -0.12 4.02 1.32 -3.06 1.44 0.54 1.56 3.09 0.92
201505 2.03 2.00 1.28 0.71 1.93 0.39 0.04 4.89 1.24 4.37 ... -0.48 -1.87 -4.92 -2.84 -3.25 -2.82 -0.49 0.29 1.34 -3.59
201506 -1.95 -1.71 -2.57 1.30 -0.84 -0.11 4.17 0.07 -2.72 4.09 ... 1.41 5.22 -0.91 -0.54 3.27 -1.32 5.79 4.17 1.97 3.18
201507 4.03 3.51 9.59 6.09 -2.90 0.71 5.96 3.66 -4.90 -0.72 ... -8.42 -5.29 -6.50 -5.69 -6.37 -4.13 -5.44 -6.48 -6.54 -5.20
201508 -4.37 -3.12 -4.06 -7.35 -8.61 -6.94 -3.86 -8.37 -7.15 -3.11 ... -2.63 -1.47 -1.72 -2.66 -0.71 -6.04 -1.75 0.44 -3.14 -1.87
201509 -1.19 2.58 2.37 -9.94 -5.32 -0.53 1.18 -7.28 -8.38 -5.92 ... 8.84 11.26 8.16 10.19 6.48 5.07 4.56 5.05 5.90 6.98
201510 5.81 8.06 10.90 14.61 12.21 5.81 0.98 7.74 16.62 7.96 ... -1.92 1.99 0.12 -0.02 -1.10 2.67 0.61 -1.01 2.16 0.05
201511 0.11 -0.71 -3.00 -0.41 -1.17 -1.10 -1.08 0.71 1.68 -2.59 ... -3.03 -1.19 -4.64 -3.75 -5.02 -1.88 0.82 -0.95 -2.92 0.25
201512 1.96 0.30 1.59 -1.70 -6.18 1.86 -4.38 0.39 -4.82 -2.65 ... -0.36 -5.09 -7.95 -5.27 -8.53 -8.68 -4.45 -0.94 -9.63 -3.20
201601 -1.67 -0.23 4.28 -8.15 -5.28 0.16 1.52 -9.43 -11.10 -5.33 ... 1.15 -2.45 1.45 2.99 6.89 3.85 -0.36 1.03 -2.85 2.71
201602 0.95 -2.34 0.93 4.25 -0.96 0.34 0.81 -1.09 6.79 0.63 ... 6.00 7.76 8.86 8.18 6.86 6.18 5.99 5.36 6.65 6.68
201603 4.69 5.61 5.04 8.61 6.88 4.65 2.30 2.90 8.33 3.58 ... 0.59 -2.55 -5.46 0.80 -1.08 0.49 -0.38 -2.38 3.96 0.67
201604 0.63 0.31 -0.25 -6.30 1.82 -0.42 -2.27 3.55 3.77 1.55 ... 0.30 4.67 5.64 1.79 -2.18 1.78 1.19 -1.48 2.15 -2.02
201605 2.06 -0.91 0.83 5.42 -0.53 -0.09 -4.96 2.46 -1.42 -1.70 ... 3.10 -2.12 -1.63 2.06 -2.53 1.81 0.71 1.16 -5.30 3.61
201606 4.75 5.31 6.87 -4.43 -0.34 3.16 1.63 0.11 -1.14 -4.67 ... 2.27 7.33 8.20 2.52 5.39 3.65 3.78 2.19 4.04 -0.21
201607 -0.51 1.82 -2.79 6.15 7.38 2.58 1.62 6.00 4.29 8.39 ... -3.56 1.19 2.38 2.46 1.09 -1.03 -1.69 -0.24 4.88 2.24
201608 -0.52 -0.90 -1.22 0.94 0.29 1.24 1.37 -3.24 2.48 1.03 ... 0.52 0.82 4.33 -0.64 2.86 -2.56 -0.18 -2.25 -1.45 -3.48
201609 -2.92 1.63 -2.78 4.62 -3.95 0.00 -6.92 0.35 -1.76 -4.87 ... -2.85 -0.55 -2.24 -5.49 -0.64 -8.18 -3.59 -1.96 1.40 -0.53
201610 -0.33 -1.65 4.59 5.59 -10.28 -2.96 -5.76 -7.45 -1.95 -4.17 ... 6.25 -0.01 2.39 4.21 12.75 9.29 2.99 8.47 12.84 8.29
201611 -4.41 -5.76 -5.12 3.87 8.15 -4.18 1.80 1.37 7.55 1.58 ... 4.65 -0.19 2.07 1.86 0.84 2.34 -0.98 0.58 3.80 2.57
201612 4.43 3.00 5.39 -3.36 1.98 1.43 -0.44 0.82 0.32 -1.27 ... 3.36 5.45 3.20 2.28 1.70 1.69 0.93 0.71 0.56 -0.87

685 rows × 60 columns

In [4]:
data.to_csv("data.csv")
desc = data.describe()
desc
# min, max line up with Table 1

Out[4]:
count 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 ... 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000 685.000000
mean 0.690715 0.710613 0.982321 0.701708 0.528277 0.554190 0.669460 0.650905 0.519781 0.667416 ... 0.520847 0.694234 0.584175 0.511241 0.582088 0.625562 0.662219 0.702730 0.609810 0.385620
std 4.339811 5.090215 6.061582 7.180918 5.809314 4.759874 6.386027 4.928072 5.518477 7.022552 ... 4.628520 6.527984 6.738979 5.055314 5.739306 5.605317 5.349341 6.104515 5.411766 5.815446
min -18.150000 -20.190000 -25.320000 -33.400000 -26.560000 -22.240000 -31.500000 -21.060000 -28.600000 -33.110000 ... -16.440000 -28.670000 -32.070000 -27.740000 -28.500000 -29.250000 -29.740000 -31.890000 -22.530000 -28.090000
25% -1.640000 -2.100000 -2.780000 -3.490000 -2.690000 -2.110000 -2.810000 -2.240000 -2.800000 -3.200000 ... -2.110000 -3.090000 -3.290000 -2.430000 -2.780000 -2.570000 -2.430000 -2.940000 -2.420000 -2.990000
50% 0.740000 0.710000 1.280000 0.890000 0.510000 0.750000 0.690000 0.750000 0.670000 0.630000 ... 0.610000 0.970000 0.560000 0.690000 0.860000 0.940000 0.470000 1.030000 0.820000 0.470000
75% 3.120000 3.660000 4.640000 5.310000 3.720000 3.550000 4.310000 3.560000 3.760000 4.490000 ... 3.360000 4.290000 4.590000 3.460000 4.060000 3.880000 4.000000 4.330000 4.000000 4.200000
max 19.890000 25.510000 32.380000 34.520000 33.130000 18.220000 31.790000 29.010000 21.680000 59.030000 ... 21.220000 23.380000 24.660000 21.000000 18.500000 17.530000 26.490000 27.380000 20.590000 19.960000

8 rows × 60 columns

In [5]:
# annualized returns don't match Table 1, oddly
# geometric mean, annualized
pd.DataFrame((np.prod(data/100 + 1)**(12.0/len(data))-1)[:30], columns=['Mean Ann. Return'])

Out[5]:
Mean Ann. Return
Food 0.074020
Beer 0.072005
Smoke 0.100147
Games 0.054031
Books 0.043953
Hshld 0.054098
Clths 0.057170
Hlth 0.065463
Chems 0.044917
Txtls 0.051888
Cnstr 0.041836
Steel 0.002802
FabPr 0.045615
ElcEq 0.062927
Autos 0.027963
Carry 0.063991
Mines 0.032527
Coal 0.026075
Oil 0.062748
Util 0.049564
Telcm 0.050868
Servs 0.057776
BusEq 0.042774
Paper 0.046776
Trans 0.051138
Whlsl 0.057056
Rtail 0.064258
Meals 0.063630
Fin 0.056748
Other 0.024894
In [6]:
# try this way, arithmetic mean then annualize (not very correct)
#print(pd.DataFrame(((desc.loc['mean']/100+1)**12-1)[:30]))
#nope

# same
pd.DataFrame(((1 + np.mean(data, axis=0)/100)**12 -1)[:30], columns=['Mean Ann. Return'])

Out[6]:
Mean Ann. Return
Food 0.086108
Beer 0.088687
Smoke 0.124460
Games 0.087532
Books 0.065268
Hshld 0.068568
Clths 0.083360
Hlth 0.080966
Chems 0.064188
Txtls 0.083096
Cnstr 0.064656
Steel 0.035833
FabPr 0.069639
ElcEq 0.087857
Autos 0.055859
Carry 0.089740
Mines 0.067862
Coal 0.091926
Oil 0.080976
Util 0.059601
Telcm 0.064437
Servs 0.085164
BusEq 0.071925
Paper 0.062952
Trans 0.072192
Whlsl 0.077332
Rtail 0.082704
Meals 0.088043
Fin 0.075605
Other 0.046240
In [7]:
#annualized volatility
pd.DataFrame((desc.loc['std']*np.sqrt(12))[:30].round(2))
# lines up with table 1

Out[7]:
std
Food 15.03
Beer 17.63
Smoke 21.00
Games 24.88
Books 20.12
Hshld 16.49
Clths 22.12
Hlth 17.07
Chems 19.12
Txtls 24.33
Cnstr 20.75
Steel 25.27
FabPr 21.18
ElcEq 21.52
Autos 23.17
Carry 21.80
Mines 25.82
Coal 35.31
Oil 18.53
Util 13.83
Telcm 16.04
Servs 22.61
BusEq 23.34
Paper 17.51
Trans 19.88
Whlsl 19.42
Rtail 18.53
Meals 21.15
Fin 18.75
Other 20.17
In [8]:
# Run LASSO, then OLS on selected variables

# skip last row to better match published r-squared
# looks like they forecast actuals 1960-2016 using 1959m12 to 2016m11
# not exact matches to Table 2 R-squared but almost within rounding error
X = data.values[:-1,:npredictors]
Y = data.values[:-1,-nresponses:]
nrows = X.shape[0]
X.shape

Out[8]:
(684, 30)
In [9]:
def subset_selection(X, Y, model_aic, verbose=False, responses=responses, predictors=predictors):

nrows, npreds = X.shape
nows, nresps = Y.shape
coef_dict = []

for response_index in range(nresps):
y = Y[:,response_index]
model_aic.fit(X, y)
predcols = [i for i in range(npreds) if model_aic.coef_[i] !=0]

#y_response = model_aic.predict(X)
# print ("In-sample LASSO R-squared: %.6f" % r2_score(y, y_response))
if verbose and responses:
print("LASSO variables selected for %s: " % responses[response_index])
print([predictors[i] for i in predcols])

if not predcols:
if verbose and responses:
print("No coefs selected for " + responses[response_index] + ", using all")
print("---")
predcols = list(range(npreds))

# fit OLS vs. selected vars, better fit w/o LASSO penalties
# in-sample R-squared using LASSO coeffs
coef_dict.append(predcols)
if verbose and responses and predictors:
print("Running OLS for " + responses[response_index] + " against " + str([predictors[i] for i in predcols]))
# col nums of selected responses
model_ols = LinearRegression()
model_ols.fit(X[:, predcols], y)
y_pred = model_ols.predict(X[:, predcols])
print ("In-sample OLS R-squared: %.2f%%" % (100 * r2_score(y, y_pred)))
print("---")

return coef_dict

#coef_dict = subset_selection(X, Y, LassoLarsIC(criterion='aic'))
coef_dict = subset_selection(X, Y, LassoLarsIC(criterion='aic'), verbose=True, responses=responses, predictors=predictors)
print(coef_dict)
# These subsets line up closely with Table 2
# except Clths, Whlsl, we get different responses

LASSO variables selected for Food.lead:
['Clths', 'Coal', 'Util', 'Rtail']
Running OLS for Food.lead against ['Clths', 'Coal', 'Util', 'Rtail']
In-sample OLS R-squared: 2.24%
---
['Food', 'Clths', 'Coal']
Running OLS for Beer.lead against ['Food', 'Clths', 'Coal']
In-sample OLS R-squared: 2.52%
---
['Txtls', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'Paper', 'Trans', 'Fin']
Running OLS for Smoke.lead against ['Txtls', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'Paper', 'Trans', 'Fin']
In-sample OLS R-squared: 6.55%
---
['Books', 'Clths', 'Coal', 'Fin']
Running OLS for Games.lead against ['Books', 'Clths', 'Coal', 'Fin']
In-sample OLS R-squared: 5.05%
---
['Games', 'Books', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Fin']
Running OLS for Books.lead against ['Games', 'Books', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Fin']
In-sample OLS R-squared: 6.30%
---
['Clths', 'Coal', 'Rtail']
Running OLS for Hshld.lead against ['Clths', 'Coal', 'Rtail']
In-sample OLS R-squared: 2.97%
---
['Clths', 'Coal', 'Oil', 'Servs', 'Rtail']
Running OLS for Clths.lead against ['Clths', 'Coal', 'Oil', 'Servs', 'Rtail']
In-sample OLS R-squared: 4.73%
---
['Books', 'Mines', 'Coal', 'Util']
Running OLS for Hlth.lead against ['Books', 'Mines', 'Coal', 'Util']
In-sample OLS R-squared: 2.68%
---
['Clths']
Running OLS for Chems.lead against ['Clths']
In-sample OLS R-squared: 0.78%
---
['Clths', 'Autos', 'Coal', 'Oil', 'Rtail', 'Fin']
Running OLS for Txtls.lead against ['Clths', 'Autos', 'Coal', 'Oil', 'Rtail', 'Fin']
In-sample OLS R-squared: 7.91%
---
['Clths', 'Coal', 'Oil', 'Util', 'Trans', 'Rtail', 'Fin']
Running OLS for Cnstr.lead against ['Clths', 'Coal', 'Oil', 'Util', 'Trans', 'Rtail', 'Fin']
In-sample OLS R-squared: 5.15%
---
['Fin']
Running OLS for Steel.lead against ['Fin']
In-sample OLS R-squared: 1.28%
---
['Trans', 'Fin']
Running OLS for FabPr.lead against ['Trans', 'Fin']
In-sample OLS R-squared: 1.57%
---
['Fin']
Running OLS for ElcEq.lead against ['Fin']
In-sample OLS R-squared: 0.80%
---
['Hshld', 'Clths', 'Coal', 'Oil', 'Util', 'BusEq', 'Rtail', 'Fin']
Running OLS for Autos.lead against ['Hshld', 'Clths', 'Coal', 'Oil', 'Util', 'BusEq', 'Rtail', 'Fin']
In-sample OLS R-squared: 6.13%
---
['Trans']
Running OLS for Carry.lead against ['Trans']
In-sample OLS R-squared: 2.32%
---
[]
No coefs selected for Mines.lead, using all
---
Running OLS for Mines.lead against ['Food', 'Beer', 'Smoke', 'Games', 'Books', 'Hshld', 'Clths', 'Hlth', 'Chems', 'Txtls', 'Cnstr', 'Steel', 'FabPr', 'ElcEq', 'Autos', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'BusEq', 'Paper', 'Trans', 'Whlsl', 'Rtail', 'Meals', 'Fin', 'Other']
In-sample OLS R-squared: 5.62%
---
['Beer', 'Smoke', 'Books', 'Autos', 'Coal', 'Oil', 'Paper', 'Rtail']
Running OLS for Coal.lead against ['Beer', 'Smoke', 'Books', 'Autos', 'Coal', 'Oil', 'Paper', 'Rtail']
In-sample OLS R-squared: 2.84%
---
['Beer', 'Hlth', 'Carry']
Running OLS for Oil.lead against ['Beer', 'Hlth', 'Carry']
In-sample OLS R-squared: 2.52%
---
['Food', 'Beer', 'Smoke', 'Hshld', 'Hlth', 'Cnstr', 'FabPr', 'Carry', 'Mines', 'Oil', 'Util', 'Telcm', 'BusEq', 'Whlsl', 'Fin', 'Other']
Running OLS for Util.lead against ['Food', 'Beer', 'Smoke', 'Hshld', 'Hlth', 'Cnstr', 'FabPr', 'Carry', 'Mines', 'Oil', 'Util', 'Telcm', 'BusEq', 'Whlsl', 'Fin', 'Other']
In-sample OLS R-squared: 7.86%
---
['Beer', 'Smoke', 'Books', 'Hshld', 'Cnstr', 'Autos', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Meals', 'Fin']
Running OLS for Telcm.lead against ['Beer', 'Smoke', 'Books', 'Hshld', 'Cnstr', 'Autos', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Meals', 'Fin']
In-sample OLS R-squared: 5.18%
---
['Smoke', 'Books', 'Steel', 'Oil', 'Util', 'Fin']
Running OLS for Servs.lead against ['Smoke', 'Books', 'Steel', 'Oil', 'Util', 'Fin']
In-sample OLS R-squared: 2.87%
---
['Smoke', 'Books', 'Util']
Running OLS for BusEq.lead against ['Smoke', 'Books', 'Util']
In-sample OLS R-squared: 2.75%
---
['Clths', 'Coal', 'Oil', 'Rtail', 'Fin']
Running OLS for Paper.lead against ['Clths', 'Coal', 'Oil', 'Rtail', 'Fin']
In-sample OLS R-squared: 3.24%
---
['Fin']
Running OLS for Trans.lead against ['Fin']
In-sample OLS R-squared: 1.32%
---
['Food', 'Smoke', 'Books', 'Carry', 'Coal', 'Oil', 'Util', 'Servs', 'Fin', 'Other']
Running OLS for Whlsl.lead against ['Food', 'Smoke', 'Books', 'Carry', 'Coal', 'Oil', 'Util', 'Servs', 'Fin', 'Other']
In-sample OLS R-squared: 6.79%
---
['Rtail']
Running OLS for Rtail.lead against ['Rtail']
In-sample OLS R-squared: 1.61%
---
['Smoke', 'Books', 'Clths', 'Steel', 'Carry', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Meals', 'Fin']
Running OLS for Meals.lead against ['Smoke', 'Books', 'Clths', 'Steel', 'Carry', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Meals', 'Fin']
In-sample OLS R-squared: 7.90%
---
['Fin']
Running OLS for Fin.lead against ['Fin']
In-sample OLS R-squared: 1.70%
---
['Clths', 'Fin']
Running OLS for Other.lead against ['Clths', 'Fin']
In-sample OLS R-squared: 2.69%
---
[[6, 17, 19, 26], [0, 6, 17], [9, 15, 16, 17, 18, 19, 20, 21, 23, 24, 28], [4, 6, 17, 28], [3, 4, 17, 18, 19, 21, 22, 26, 28], [6, 17, 26], [6, 17, 18, 21, 26], [4, 16, 17, 19], [6], [6, 14, 17, 18, 26, 28], [6, 17, 18, 19, 24, 26, 28], [28], [24, 28], [28], [5, 6, 17, 18, 19, 22, 26, 28], [24], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], [1, 2, 4, 14, 17, 18, 23, 26], [1, 7, 15], [0, 1, 2, 5, 7, 10, 12, 15, 16, 18, 19, 20, 22, 25, 28, 29], [1, 2, 4, 5, 10, 14, 15, 16, 17, 18, 19, 21, 22, 26, 27, 28], [2, 4, 11, 18, 19, 28], [2, 4, 19], [6, 17, 18, 26, 28], [28], [0, 2, 4, 15, 17, 18, 19, 21, 28, 29], [26], [2, 4, 6, 11, 15, 17, 18, 19, 21, 22, 27, 28], [28], [6, 28]]

In [10]:
# same predictors selected for all but 2 response vars
# use predictors from paper to match results
if True: # turn off/on
coef_dict_temp = {}
coef_dict_temp['Food.lead'] = ['Clths', 'Coal', 'Util', 'Rtail']
coef_dict_temp['Smoke.lead'] = ['Txtls', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'Paper', 'Trans', 'Fin']
coef_dict_temp['Games.lead'] = ['Books', 'Clths', 'Coal', 'Fin']
coef_dict_temp['Books.lead'] = ['Games', 'Books', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Fin']
coef_dict_temp['Clths.lead'] = ['Books', 'Clths', 'Chems', 'Steel', 'ElcEq', 'Carry',  'Coal', 'Oil', 'Util','Telcm', 'Servs', 'BusEq', 'Rtail']
# Running OLS for Clths against ['Clths', 'Coal', 'Oil', 'Servs', 'Rtail']
coef_dict_temp['Hlth.lead'] = ['Books', 'Mines', 'Coal', 'Util']
coef_dict_temp['Txtls.lead'] = ['Clths', 'Autos', 'Coal', 'Oil', 'Rtail', 'Fin']
coef_dict_temp['Cnstr.lead'] = ['Clths', 'Coal', 'Oil', 'Util', 'Trans', 'Rtail', 'Fin']
coef_dict_temp['Autos.lead'] = ['Hshld', 'Clths', 'Coal', 'Oil', 'Util', 'BusEq', 'Rtail', 'Fin']
coef_dict_temp['Coal.lead'] = ['Beer', 'Smoke', 'Books', 'Autos', 'Coal', 'Oil', 'Paper', 'Rtail']
coef_dict_temp['Util.lead'] = ['Food', 'Beer', 'Smoke', 'Hshld', 'Hlth', 'Cnstr', 'FabPr', 'Carry', 'Mines', 'Oil', 'Util', 'Telcm', 'BusEq', 'Whlsl', 'Fin', 'Other']
coef_dict_temp['Telcm.lead'] = ['Beer', 'Smoke', 'Books', 'Hshld', 'Cnstr', 'Autos', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Meals', 'Fin']
coef_dict_temp['Servs.lead'] = ['Smoke', 'Books', 'Steel', 'Oil', 'Util', 'Fin']
coef_dict_temp['Paper.lead'] = ['Clths', 'Coal', 'Oil', 'Rtail', 'Fin']
coef_dict_temp['Whlsl.lead'] = ['Food', 'Beer', 'Smoke', 'Books', 'Hlth', 'Carry', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'BusEq', 'Fin', 'Other']
# Running OLS for Whlsl against ['Food', 'Smoke', 'Books', 'Carry', 'Coal', 'Oil', 'Util', 'Servs', 'Fin', 'Other']
coef_dict_temp['Meals.lead'] = ['Smoke', 'Books', 'Clths', 'Steel', 'Carry', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Meals', 'Fin']

coef_dict_paper = []
for response in responses:
print(response, " -> ", coef_dict_temp[response])
coef_dict_paper.append([predictor_reverse_dict[jstr] for jstr in coef_dict_temp[response]])
print(coef_dict_paper)

Food.lead  ->  ['Clths', 'Coal', 'Util', 'Rtail']
Smoke.lead  ->  ['Txtls', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'Paper', 'Trans', 'Fin']
Games.lead  ->  ['Books', 'Clths', 'Coal', 'Fin']
Books.lead  ->  ['Games', 'Books', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Fin']
Clths.lead  ->  ['Books', 'Clths', 'Chems', 'Steel', 'ElcEq', 'Carry', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'BusEq', 'Rtail']
Hlth.lead  ->  ['Books', 'Mines', 'Coal', 'Util']
Txtls.lead  ->  ['Clths', 'Autos', 'Coal', 'Oil', 'Rtail', 'Fin']
Cnstr.lead  ->  ['Clths', 'Coal', 'Oil', 'Util', 'Trans', 'Rtail', 'Fin']
Autos.lead  ->  ['Hshld', 'Clths', 'Coal', 'Oil', 'Util', 'BusEq', 'Rtail', 'Fin']
Coal.lead  ->  ['Beer', 'Smoke', 'Books', 'Autos', 'Coal', 'Oil', 'Paper', 'Rtail']
Util.lead  ->  ['Food', 'Beer', 'Smoke', 'Hshld', 'Hlth', 'Cnstr', 'FabPr', 'Carry', 'Mines', 'Oil', 'Util', 'Telcm', 'BusEq', 'Whlsl', 'Fin', 'Other']
Telcm.lead  ->  ['Beer', 'Smoke', 'Books', 'Hshld', 'Cnstr', 'Autos', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Meals', 'Fin']
Servs.lead  ->  ['Smoke', 'Books', 'Steel', 'Oil', 'Util', 'Fin']
Paper.lead  ->  ['Clths', 'Coal', 'Oil', 'Rtail', 'Fin']
Whlsl.lead  ->  ['Food', 'Beer', 'Smoke', 'Books', 'Hlth', 'Carry', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'BusEq', 'Fin', 'Other']
Meals.lead  ->  ['Smoke', 'Books', 'Clths', 'Steel', 'Carry', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Meals', 'Fin']
[[6, 17, 19, 26], [0, 6, 17], [9, 15, 16, 17, 18, 19, 20, 21, 23, 24, 28], [4, 6, 17, 28], [3, 4, 17, 18, 19, 21, 22, 26, 28], [6, 17, 26], [4, 6, 8, 11, 13, 15, 17, 18, 19, 20, 21, 22, 26], [4, 16, 17, 19], [6], [6, 14, 17, 18, 26, 28], [6, 17, 18, 19, 24, 26, 28], [28], [24, 28], [28], [5, 6, 17, 18, 19, 22, 26, 28], [24], [], [1, 2, 4, 14, 17, 18, 23, 26], [1, 7, 15], [0, 1, 2, 5, 7, 10, 12, 15, 16, 18, 19, 20, 22, 25, 28, 29], [1, 2, 4, 5, 10, 14, 15, 16, 17, 18, 19, 21, 22, 26, 27, 28], [2, 4, 11, 18, 19, 28], [2, 4, 19], [6, 17, 18, 26, 28], [28], [0, 1, 2, 4, 7, 15, 17, 18, 19, 20, 21, 22, 28, 29], [26], [2, 4, 6, 11, 15, 17, 18, 19, 21, 22, 27, 28], [28], [6, 28]]

In [11]:
def predict_with_subsets(X, Y, create_model, coef_dict, verbose=False):
"""evaluate subset selection, pass a model function and subsets, compute avg R-squared"""
global responses

nrows, ncols = Y.shape
model = create_model()

scores = []
for response_col in range(ncols):
y = Y[:,response_col]

#        print("LASSO variables selected for %s: " % pred)
#        print(coef_dict[pred])

if not coef_dict[response_col]:
if verbose:
print("No coefs selected for " + responses[response_col])
#           print("---")
continue
# fit model vs. selected vars, better fit w/o LASSO penalties
# in-sample R-squared using LASSO coeffs
#print("Running model for " + pred + " against " + str(coef_dict[pred]))
# col nums of selected predictors
predcols = coef_dict[response_col]
model.fit(X[:, predcols], y)
y_pred = model.predict(X[:, predcols])
score = r2_score(y, y_pred)
scores.append(score)
if verbose:
print ("In-sample R-squared: %.2f%% for %s against %s" % (score*100, responses[response_col],
str([predictors[i] for i in coef_dict[response_col]])))
#        print("---")

if verbose:
print("Mean R-squared: %.2f%%" % (100 * np.mean(np.array(scores))))
return np.mean(np.array(scores))

predict_with_subsets(X, Y, LinearRegression, coef_dict_paper, verbose=True)

In-sample R-squared: 2.24% for Food.lead against ['Clths', 'Coal', 'Util', 'Rtail']
In-sample R-squared: 2.52% for Beer.lead against ['Food', 'Clths', 'Coal']
In-sample R-squared: 6.55% for Smoke.lead against ['Txtls', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'Paper', 'Trans', 'Fin']
In-sample R-squared: 5.05% for Games.lead against ['Books', 'Clths', 'Coal', 'Fin']
In-sample R-squared: 6.30% for Books.lead against ['Games', 'Books', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Fin']
In-sample R-squared: 2.97% for Hshld.lead against ['Clths', 'Coal', 'Rtail']
In-sample R-squared: 7.82% for Clths.lead against ['Books', 'Clths', 'Chems', 'Steel', 'ElcEq', 'Carry', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'BusEq', 'Rtail']
In-sample R-squared: 2.68% for Hlth.lead against ['Books', 'Mines', 'Coal', 'Util']
In-sample R-squared: 0.78% for Chems.lead against ['Clths']
In-sample R-squared: 7.91% for Txtls.lead against ['Clths', 'Autos', 'Coal', 'Oil', 'Rtail', 'Fin']
In-sample R-squared: 5.15% for Cnstr.lead against ['Clths', 'Coal', 'Oil', 'Util', 'Trans', 'Rtail', 'Fin']
In-sample R-squared: 1.28% for Steel.lead against ['Fin']
In-sample R-squared: 1.57% for FabPr.lead against ['Trans', 'Fin']
In-sample R-squared: 0.80% for ElcEq.lead against ['Fin']
In-sample R-squared: 6.13% for Autos.lead against ['Hshld', 'Clths', 'Coal', 'Oil', 'Util', 'BusEq', 'Rtail', 'Fin']
In-sample R-squared: 2.32% for Carry.lead against ['Trans']
In-sample R-squared: 2.84% for Coal.lead against ['Beer', 'Smoke', 'Books', 'Autos', 'Coal', 'Oil', 'Paper', 'Rtail']
In-sample R-squared: 2.52% for Oil.lead against ['Beer', 'Hlth', 'Carry']
In-sample R-squared: 7.86% for Util.lead against ['Food', 'Beer', 'Smoke', 'Hshld', 'Hlth', 'Cnstr', 'FabPr', 'Carry', 'Mines', 'Oil', 'Util', 'Telcm', 'BusEq', 'Whlsl', 'Fin', 'Other']
In-sample R-squared: 5.18% for Telcm.lead against ['Beer', 'Smoke', 'Books', 'Hshld', 'Cnstr', 'Autos', 'Carry', 'Mines', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Rtail', 'Meals', 'Fin']
In-sample R-squared: 2.87% for Servs.lead against ['Smoke', 'Books', 'Steel', 'Oil', 'Util', 'Fin']
In-sample R-squared: 2.75% for BusEq.lead against ['Smoke', 'Books', 'Util']
In-sample R-squared: 3.24% for Paper.lead against ['Clths', 'Coal', 'Oil', 'Rtail', 'Fin']
In-sample R-squared: 1.32% for Trans.lead against ['Fin']
In-sample R-squared: 7.43% for Whlsl.lead against ['Food', 'Beer', 'Smoke', 'Books', 'Hlth', 'Carry', 'Coal', 'Oil', 'Util', 'Telcm', 'Servs', 'BusEq', 'Fin', 'Other']
In-sample R-squared: 1.61% for Rtail.lead against ['Rtail']
In-sample R-squared: 7.90% for Meals.lead against ['Smoke', 'Books', 'Clths', 'Steel', 'Carry', 'Coal', 'Oil', 'Util', 'Servs', 'BusEq', 'Meals', 'Fin']
In-sample R-squared: 1.70% for Fin.lead against ['Fin']
In-sample R-squared: 2.69% for Other.lead against ['Clths', 'Fin']
Mean R-squared: 3.86%

Out[11]:
0.038622786316912544
In [12]:
# use all predictors - higher in-sample R-squared
coef_dict_all = []
for _ in responses:
coef_dict_all.append(range(len(predictors)))
predict_with_subsets(X, Y, LinearRegression, coef_dict_all, verbose=False)

Out[12]:
0.06637486888237351
In [13]:
# first iteration will train up to including 196911
# will use 196912 to predict 197001
# 1970101 will be first month of performance to use
# train on first 121 months up to 196912 (0:120), put first prediction in P[121] (122nd row)
# first month of performance will be 197002
FIRST_TRAIN_MONTHS = 121
FIRST_PREDICT_MONTH = FIRST_TRAIN_MONTHS # This is stupid but keeps my head straight

print(X[FIRST_TRAIN_MONTHS])
print(data.iloc[FIRST_TRAIN_MONTHS][:30])

[ -3.34  -1.95  -7.59  -7.76 -12.05  -7.5   -5.69  -7.71  -7.37  -5.26
-9.84  -6.31  -7.15  -6.89  -9.35 -12.49  -2.34  -0.77 -12.16  -4.83
-3.16 -11.17  -9.73  -8.89  -8.17  -8.28  -6.31 -13.12  -9.78  -6.2 ]
Food     -3.34
Beer     -1.95
Smoke    -7.59
Games    -7.76
Books   -12.05
Hshld    -7.50
Clths    -5.69
Hlth     -7.71
Chems    -7.37
Txtls    -5.26
Cnstr    -9.84
Steel    -6.31
FabPr    -7.15
ElcEq    -6.89
Autos    -9.35
Carry   -12.49
Mines    -2.34
Coal     -0.77
Oil     -12.16
Util     -4.83
Telcm    -3.16
Servs   -11.17
BusEq    -9.73
Paper    -8.89
Trans    -8.17
Whlsl    -8.28
Rtail    -6.31
Meals   -13.12
Fin      -9.78
Other    -6.20
Name: 197001, dtype: float64

In [14]:
class PredictWrapper():
"""Wrap an sklearn model e.g. LinearRegression to fit, predict all vars as a vector,
match the way our Keras model will do it"""

def __init__(self, create_model, coef_dict, fit_missing=None):
self.create_model = create_model
self.coef_dict = coef_dict
self.fit_missing = fit_missing
self.models = []

def fit(self, X_fit, Y_fit, verbose=False):

self.nrows, self.ycols = Y_fit.shape
self.models = []
# fit model for each column
for responsecol in range(self.ycols):
if self.coef_dict[responsecol]:
# column indexes to fit against each other
predcols = self.coef_dict[responsecol]
else: # no columns selected
if not self.fit_missing:
# default: don't fit if no predictors selected
self.models.append(None)
#print(self.models[-1])
continue
elif self.fit_missing == "mean":
# append a numeric value to use
self.models.append(np.mean(Y_fit[:,responsecol]))
#print(self.models[-1])
continue
elif self.fit_missing == "all":
# predcols = all columns
predcols = range(X_fit.shape[1])

model = self.create_model() # create 1 sklearn model for each column
model.fit(X_fit[:, predcols], Y_fit[:,responsecol])
if verbose:
print("fit on " + str(X_fit[:, predcols].shape) + str(predcols))
print(model.coef_)

self.models.append(model)

#debug
#print(responsecol)
#print(X_fit[:, predcols])
#print("=====")
#print(Y_fit[:,responsecol])
#print("=====")
#print(self.model.coef_)
#print(self.model.intercept_)
#print("=====")

def predict(self, X_predict, verbose=False):

predictions = []
nrows = X_predict.shape[0]

for responsecol in range(self.ycols):
#print (type(self.models[responsecol]))
if type(self.models[responsecol]) == np.float64:
pred = np.array([self.models[responsecol]] * nrows).reshape(nrows,1)
predictions.append(pred)
sys.stdout.write('\010#')
elif not self.models[responsecol]:
# don't predict
predictions.append(np.array([np.nan] * nrows ).reshape(nrows,1))
sys.stdout.write('\010N')
else:
predcols = self.coef_dict[responsecol]
y_pred = self.models[responsecol].predict(X_predict[:,predcols])
if verbose:
print("predict on" + str(X_predict[:, predcols].shape) + str(predcols))
print(y_pred)
predictions.append(y_pred.reshape(nrows,1))

return np.hstack(predictions)



### BacktestModel class to abstract walk-forward validation and backtesting workflow from modeling, portfolio construction, reporting¶

In [15]:
# typical pipeline
# backtestmodel = BacktestModel(X, # predictors
#                              Y, # responses
#                              create_model=LinearRegression, # create_model which returns a model (needed for 'timestep' which creates different model each timestep)
#                              # or model = someKerasModel, # initialized model that supports fit(X,Y), predict(X) , predicts an entire row,
#                              coef_dict_param=coef_dict_paper, # how to map predictors to responses ("all", "timestep", or a list of lists)
#                              startindex=FIRST_TRAIN_MONTHS, # initial training for backtest
#                              fit_missing='mean', # what to do when no predictors are selected in coef_dict_param - use all predictors, use historical mean, use np.nan
#                              scaler = None) # scaling function like MinMaxScaler

# backtestmodel.gen_predictions(verbose=False) # starting from startindex, step through X,Y month by month, fit up to current month, predict next month, store prediction in self.P
# backtestmodel.walkforward_xval(n_splits=5, verbose=True) # calls gen_predictions with a large step, fits and predicts one large fold at a time (useful to cross-validate quickly)
# backtestmodel.evaluate_predictions() # report metrics on prediction : MSE etc. #TODO: support custom calcs before/after/instead of default calcs
# backtestmodel.gen_returns(calc_returns, verbose=True) # takes a function that returns portfolio returns based on self.P, stores in self.R
# backtestmodel.report_returns(start_date=start_date_str, freq='M') # calc cumulative perf and report (TODO: allow it to take a reporting function to run before/after/in place of default report)
# backtestmodel.evaluate_quantiles(chart=True, verbose=True) # report quantile metrics # TODO: make this a custom calc passed into evaluate_predictions

class BacktestModel():

def __init__(self,
X, # predictors
Y, # responses
model=None, # model that supports fit(X,Y), predict(X) , predicts an entire row,
create_model=None, # or create_model which returns a model (needed for 'timestep' but slows down so pass model if dynamic not needed)
coef_dict_param="all", # mapping of predictors to responses ("all", "timestep", or a list of lists)
startindex=FIRST_TRAIN_MONTHS,
scaler=None,
fit_missing=None):

self.Xrows, self.Xcols = X.shape
self.Yrows, self.Ycols = Y.shape

if self.Xrows != self.Yrows:
raise(ValueError, "Shapes differ: X %s, Y %s" % (str(X.shape), str(Y.shape)))

self.X = X
self.Y = Y
self.Xscale = X.copy()
self.Yscale = Y.copy()

if scaler:
print("scaler: %s " %str(scaler))
# by rows
# MinMaxScaler: each row (min->0, max->1)
# StandardScaler: each row (mean->0, SD->1)
# self.Xscale = scaler().fit_transform(self.Xscale.transpose()).transpose()
# self.Yscale = scaler().fit_transform(self.Yscale.transpose()).transpose()
# by cols
# MinMaxScaler: each col (min->0, max->1)
# StandardScaler: each col (mean->0, SD->1)
self.Xscale = scaler().fit_transform(self.Xscale)
self.Yscale = scaler().fit_transform(self.Yscale)

self.model = model
self.create_model = create_model
self.coef_dict_param = coef_dict_param
self.startindex = startindex
self.fit_missing = fit_missing

def fit_predict(self, ntrain, npredict=1, verbose=False):
"""for backtest, train model using Y v. X
train on first ntrain rows. if ntrain=121, fit 0:120
predict following npredict rows
if npredict=1, predict row 121
if npredict=12, predict rows 121-132
"""

# fit first ntrain rows
X_fit = self.Xscale[:ntrain]  # e.g. 0:120
Y_fit = self.Yscale[:ntrain]
# predict npredict rows
X_predict = self.Xscale[ntrain:ntrain+npredict] # 121-122
X_predict = X_predict.reshape(npredict,self.Xcols)

# if no coef_dict select predictors into coef_dict
if self.coef_dict_param == "timestep":
msg = "Performing subset selection"
coef_dict = subset_selection(X_fit, Y_fit, LassoLarsIC(criterion='aic'))
# if coef_dict == "all" use all predictors for each response
elif self.coef_dict_param == 'all':
msg = "Using all predictors"
coef_dict = [range(self.Xcols) for _ in range(self.ycols)]
else: # should check valid dict
msg = "Using coef_dict predictors"
coef_dict = self.coef_dict_param
if verbose:
print(msg)

if self.create_model:
self.model = PredictWrapper(self.create_model, coef_dict, fit_missing=self.fit_missing)

self.model.fit(X_fit, Y_fit, verbose=verbose)
return self.model.predict(X_predict, verbose=verbose)

# predict all months
# initial train_months = 120 -> train first model on 120 rows
# first prediction will be in P[120] (121st row)
# step = 6 -> predict following 6 rows, then step forward 6 months at a time
# initialize predictions matrix self.P

# use either step or folds
# step, do range(self.startindex, nrows, step)
# folds, at each fold train 0:startfold, predict startfold+1:endfold
# store only out-of-sample predictions in P, calc out-of-sample MSE

# using a step > 1 or folds is quicker, for quicker xval, or to speed up by not estimating model at each timestep

def gen_predictions(self,
step=1,
splits=None,
verbose=False):

self.P = np.zeros_like(self.Y)

progress_i = 0
self.nrows, self.ycols = Y.shape

if splits:
month_indexes = splits[:-1] # last index is nrows
else:
# create list of steps
month_indexes = list(range(self.startindex, nrows, step))
steps = [month_indexes[i+1]-month_indexes[i] for i in range(len(month_indexes)-1)]
# last step -> end
steps.append(self.nrows - month_indexes[-1])

if verbose:
print ("Steps: " + str(month_indexes))

for month_index, forecast_rows in zip(month_indexes, steps):
if verbose:
print("Training on first %d rows (%d:%d), putting predictions in rows %s" % (month_index,
0, month_index-1,
str(range(month_index,month_index+forecast_rows))))
predictions = self.fit_predict(month_index, forecast_rows, verbose=False)

first_pred_row = month_index
for row_index in range(forecast_rows):
self.P[first_pred_row+row_index] = predictions[row_index]
sys.stdout.write('.')
progress_i += 1
if progress_i % 80 == 0:
print("")
print("%s Still training step %d of %d" % (time.strftime("%H:%M:%S"), progress_i, len(month_indexes)))
sys.stdout.flush()
print("")

def evaluate_predictions(self):

# evaluate prediction (can move to separate function)
msetemp = (self.P[self.startindex:]-self.Yscale[self.startindex:])**2
#remove nans
msetemp = msetemp[~np.isnan(msetemp)]
self.mse = np.mean(msetemp)
print("OOS MSE across all predictions: %.4f" % self.mse)
self.model.fit(self.Xscale, self.Yscale)
Y_pred = self.model.predict(self.Xscale)
self.in_sample_mse = np.mean((Y_pred - self.Yscale) ** 2)
print("In-sample MSE: %.4f" % self.in_sample_mse)

# force unpredicted ys to be nans, then remove nans
vartemp = self.Yscale[self.startindex:] - self.P[self.startindex:] + self.P[self.startindex:]
vartemp = vartemp[~np.isnan(vartemp)]
y_variance = np.var(vartemp[self.startindex:])
print("Variance: %.4f" % (y_variance))
print("R-squared: %.4f" % (1- self.mse/y_variance))

return(self.mse)

def evaluate_quantiles(self, chart=False, verbose=False):

self.P_quantiles = np.zeros_like(self.P)
self.Y_quantiles = np.zeros_like(self.Y)
self.kendalltaus = []
self.ktpvals = []
# compute score for predicted quantiles vs. actual (expected) quantiles
N_QUANTILES=5
for row in range(self.startindex, self.P_quantiles.shape[0]):
#print(self.P[row])
self.P_quantiles[row] = pd.qcut(self.P[row], N_QUANTILES, range(N_QUANTILES))
self.Y_quantiles[row] = pd.qcut(self.Y[row], N_QUANTILES, range(N_QUANTILES))
kt, p_val = kendalltau(self.P[row], self.Y[row])
self.kendalltaus.append(kt)
self.ktpvals.append(p_val)

self.kendalltau = np.mean(self.kendalltaus)
self.kendalltau_pvalue = np.mean(self.ktpvals)
print("Avg rank correlation (Kendall's tau): %.4f (Expected: 0)" % (self.kendalltau))
pred_quantiles = self.P_quantiles[self.startindex:]
true_quantiles = self.Y_quantiles[self.startindex:]

nrows, ncols = pred_quantiles.shape

pred_quantiles = pred_quantiles.reshape(nrows*ncols)
true_quantiles = true_quantiles.reshape(nrows*ncols)
self.quintile_accuracy = accuracy_score(pred_quantiles, true_quantiles)
print("5-quintile accuracy: %.4f (Expected: 0.2)" % (self.quintile_accuracy))

pred_direction = np.zeros(nrows*ncols)
true_direction = np.zeros(nrows*ncols)
for i in range(nrows*ncols):
if pred_quantiles[i] == 4:
pred_direction[i] = 1
elif pred_quantiles[i] == 0:
pred_direction[i] = -1
if true_quantiles[i] == 4:
true_direction[i] = 1
elif true_quantiles[i] == 0:
true_direction[i] = -1
self.directional_accuracy = accuracy_score(pred_direction, true_direction)
print("Long/short/flat accuracy: %.4f (Expected: 0.44)" % (self.directional_accuracy))

nrows = nrows * ncols

conf_mat_expected = np.array([[0.64, 0.16],[0.16, 0.04]])*nrows

myscores = []
for q in range(5):
temp_pred = pred_quantiles == q
temp_actual = true_quantiles == q
conf_mat5 = confusion_matrix(temp_pred, temp_actual)
diff_mat = conf_mat5 - conf_mat_expected
if verbose:
print("Confusion matrix for quantile %d" % q)
print(conf_mat5)
cstmp, cspvtmp = chisquare(conf_mat5.reshape(4), conf_mat_expected.reshape(4))
print("Chi-square: %.4f (p-value: %.8f)" % (cstmp, cspvtmp))

# probably no valid statistical interpretation but
# average of improvement in true positive % and true negative %
myscore = diff_mat[1][1]
myscores.append(myscore)

# sum of true positive for top and bottom quintiles
self.excess_tp = myscores[0] + myscores[4]
print("Excess true positive in quintiles 1 + 5: %f" % (self.excess_tp))

conf_mat = confusion_matrix(pred_quantiles, true_quantiles)
if chart:
fig, ax = plt.subplots(figsize=(10,10))
sns.heatmap(conf_mat, annot=True, fmt='d')
plt.ylabel('Actual Quintile')
plt.xlabel('Predicted Quintile')
plt.show()

return None

def walkforward_xval (self, n_splits=5, verbose=False):
"""quick and dirty genreturns, with a step"""
# generate k-folds
kf = KFold(n_splits=n_splits)
kf.get_n_splits(X)
last_indexes = []
for train_index, test_index in kf.split(X):
# use test_index as last index to train
last_index = test_index[-1] + 1
last_indexes.append(last_index)
print("%s Generate splits %s" % (time.strftime("%H:%M:%S"), str([i for i in last_indexes])))
#override startindex
self.startindex = last_indexes[0]
return self.gen_predictions(splits=last_indexes, verbose=verbose)

def gen_returns(self, port_returns_func, verbose=False):

self.R = np.zeros(self.P.shape[0])
first_pred_month=self.startindex

indcount = [0] * self.ycols
longcount = [0] * self.ycols
shortcount = [0] * self.ycols

for month_index in range(first_pred_month, nrows-1):
return_month = month_index + 1
port_return, long_indexes, short_indexes = port_returns_func(self.P[month_index],
self.X[return_month])
self.R[return_month] = port_return

for i in long_indexes:
indcount[i] += 1
longcount[i] += 1
for i in short_indexes:
indcount[i] += 1
shortcount[i] += 1
if verbose:
for i in range(len(responses)):
print("%s: long %d times, short %d times, total %d times" % (predictors[i],
longcount[i],
shortcount[i],
indcount[i]))
return self.R

def report_returns(self, start_date='01/01/1970', freq='M'):

first_pred_month=self.startindex
self.results = self.R[first_pred_month:]
nmonths = self.results.shape[0]
nyears = nmonths/12.0
index = pd.date_range(start_date,periods=nmonths, freq=freq)
perfdata = pd.DataFrame(self.results,index=index,columns=['Returns'])
perfdata['Equity'] = 100 * np.cumprod(1 + self.results / 100)
self.cumulative_return = perfdata['Equity']
self.mean_monthly_return_annualized = np.mean(1 + self.results/100) ** 12 - 1
self.mean_return = (self.cumulative_return[-1]/100) ** (1.0/nyears) - 1
self.annualized_vol = np.std(self.results/100) * np.sqrt(12.0)
self.sharpe = self.mean_monthly_return_annualized/self.annualized_vol
print("Mean return: %.3f%%" % (self.mean_return * 100 ))
#print("Mean monthly annualized return: %.3f%%" % (self.mean_monthly_return_annualized * 100 ))
#print("Monthly annualized volatility: %.3f%%" % (self.annualized_vol * 100))
print("Monthly Sharpe ratio: %.3f" % (self.sharpe))

In [16]:
# return calculation passed to gen_returns

NUMSTOCKS = 6 # top quintile (and bottom)

def calc_returns(prediction_row, return_row, numstocks=NUMSTOCKS, verbose=False):

# ensure nan sorts to top for shorts
short_sort_array = [999999 if np.isnan(x) else x for x in prediction_row]
# pick bottom numstocks
select_array = np.argsort(short_sort_array)
short_indexes = select_array[:numstocks]

# ensure nan sorts to bottom for longs
long_sort_array = [-999999 if np.isnan(x) else x for x in prediction_row]
# pick top numstocks
select_array = np.argsort(long_sort_array)
long_indexes = select_array[-numstocks:]

if verbose:
print("Longs: %s" %(str([(i,prediction_row[i]) for i in long_indexes])))
print("Shorts: %s" %(str([(i,prediction_row[i]) for i in short_indexes])))

# compute equal weighted long/short return
return np.mean(return_row[long_indexes])/2 - np.mean(return_row[short_indexes])/2, long_indexes, short_indexes

In [17]:
start_date_int = data.index[FIRST_TRAIN_MONTHS]
start_year, start_month = start_date_int // 100, start_date_int % 100
start_date_str = "%02d/%02d/%d" % (start_month, 1, start_year)
start_date_str

Out[17]:
'01/01/1970'
In [18]:
# test fit_predict
backtestmodel = BacktestModel(X, Y, create_model=LinearRegression, coef_dict_param=coef_dict_all, startindex=FIRST_TRAIN_MONTHS)
print(backtestmodel.fit_predict(121, npredict=3, verbose=False))

[[ -4.70219919  -5.3073358   -3.72029191 -12.42448745  -6.62775756
-2.90316148  -6.79960146  -2.23172984  -5.75101181  -6.66276641
-5.75265368  -7.71979359  -6.57864566  -4.32124795  -5.0355927
-5.92230241  -4.76675177  -1.96315925  -3.43369206  -2.30343164
-3.72794445  -2.91917105  -5.27753906  -6.40474001  -6.85223288
-10.16508667  -4.69016977  -8.8042513   -5.4481542   -6.40122815]
[ -1.09819574   0.33751522   1.47276575  -7.65291563  -0.40550754
-4.96855817  -1.82873168  -3.4663124   -2.05857985  -0.68158685
-1.54307573  -2.74859589  -2.04412571  -1.77673237  -3.39612781
-4.76047089  -0.65055072  -5.08513341   1.95082965  -2.60578312
-5.09137205  -1.72203771  -3.31591182  -1.94755286  -1.38046013
-2.75841651  -0.96353404  -6.81674253   1.34059086  -1.59466739]
[ -1.96144894  -0.87682739  -3.02833064   0.58137733  -0.02056567
1.67143038  -0.44895783  -1.70395859  -0.63283007  -0.4731534
-0.71228211  -3.61106494  -1.93572984  -0.64937617  -1.09751999
0.24726352  -0.37303973  -1.04141371   2.1679461   -0.55736749
0.76975122   0.67561178   1.62692312  -1.12079845  -1.07373008
-0.27905898   1.55099516  -3.06722504  -0.32196075   0.5859896 ]]

In [19]:
# test model on our calculated coef_dict
# selects LASSO vars over whole timespan
# uses selected vars to do OLS backtest
# included as an example of what not to do
backtestmodel = BacktestModel(X, Y,
create_model=LinearRegression,
coef_dict_param=coef_dict_paper,
startindex=FIRST_TRAIN_MONTHS,
fit_missing='mean',
scaler = None)
backtestmodel.gen_predictions(verbose=False)
backtestmodel.gen_returns(calc_returns, verbose=True)
backtestmodel.report_returns(start_date=start_date_str, freq='M')
backtestmodel.evaluate_predictions()
backtestmodel.evaluate_quantiles(chart=True, verbose=True)

################################################################################.
00:15:19 Still training step 80 of 563
################################################################################.
00:15:20 Still training step 160 of 563
################################################################################.
00:15:21 Still training step 240 of 563
################################################################################.
00:15:23 Still training step 320 of 563
################################################################################.
00:15:24 Still training step 400 of 563
################################################################################.
00:15:25 Still training step 480 of 563
################################################################################.
00:15:26 Still training step 560 of 563
###.
Food: long 93 times, short 37 times, total 130 times
Beer: long 122 times, short 94 times, total 216 times
Smoke: long 211 times, short 112 times, total 323 times
Games: long 147 times, short 125 times, total 272 times
Books: long 126 times, short 107 times, total 233 times
Hshld: long 81 times, short 72 times, total 153 times
Clths: long 195 times, short 172 times, total 367 times
Hlth: long 126 times, short 110 times, total 236 times
Chems: long 26 times, short 125 times, total 151 times
Txtls: long 160 times, short 128 times, total 288 times
Cnstr: long 92 times, short 124 times, total 216 times
Steel: long 4 times, short 180 times, total 184 times
FabPr: long 33 times, short 62 times, total 95 times
ElcEq: long 45 times, short 16 times, total 61 times
Autos: long 87 times, short 117 times, total 204 times
Carry: long 106 times, short 76 times, total 182 times
Mines: long 132 times, short 102 times, total 234 times
Coal: long 226 times, short 183 times, total 409 times
Oil: long 160 times, short 137 times, total 297 times
Util: long 161 times, short 182 times, total 343 times
Telcm: long 118 times, short 163 times, total 281 times
Servs: long 149 times, short 137 times, total 286 times
BusEq: long 106 times, short 123 times, total 229 times
Paper: long 50 times, short 107 times, total 157 times
Trans: long 39 times, short 56 times, total 95 times
Whlsl: long 168 times, short 147 times, total 315 times
Rtail: long 72 times, short 62 times, total 134 times
Meals: long 220 times, short 178 times, total 398 times
Fin: long 56 times, short 21 times, total 77 times
Other: long 61 times, short 117 times, total 178 times
Mean return: 6.667%
Monthly Sharpe ratio: 1.123
OOS MSE across all predictions: 39.4986
#In-sample MSE: 35.8279
Variance: 39.4097
R-squared: -0.0023
Avg rank correlation (Kendall's tau): 0.0589 (Expected: 0)
5-quintile accuracy: 0.2287 (Expected: 0.2)
Long/short/flat accuracy: 0.4706 (Expected: 0.44)
Confusion matrix for quantile 0
[[10953  2559]
[ 2555   823]]
Chi-square: 49.7107 (p-value: 0.00000000)
Confusion matrix for quantile 1
[[10884  2628]
[ 2628   750]]
Chi-square: 12.8020 (p-value: 0.00508507)
Confusion matrix for quantile 2
[[10850  2662]
[ 2656   722]]
Chi-square: 4.7384 (p-value: 0.19198761)
Confusion matrix for quantile 3
[[10851  2661]
[ 2666   712]]
Chi-square: 3.2442 (p-value: 0.35547827)
Confusion matrix for quantile 4
[[10994  2518]
[ 2523   855]]
Chi-square: 75.2761 (p-value: 0.00000000)
Excess true positive in quintiles 1 + 5: 326.800000


### Functions using Plotly to generate charts for performance, scatters, heatmaps¶

In [20]:
# chart performance

def mychart(args, names=None, title=""):
x_coords = np.linspace(1970, 2016, args[0].shape[0])
plotdata = []
for i in range(len(args)):
tracelabel = "Trace %d" % i
if names:
tracelabel=names[i]
plotdata.append(Scatter(x=x_coords,
y=args[i].values.reshape(-1),
mode = 'line',
name=tracelabel))

layout = Layout(
title = title,
autosize=False,
width=900,
height=600,
yaxis=dict(
type='log',
autorange=True
)
)

fig = Figure(data=plotdata, layout=layout)

return iplot(fig)


In [21]:
def myscatter(arg1, arg2, names=None, title=""):

plotdata = []

plotdata.append(Scatter(
x = arg1,
y = arg2,
mode = 'markers'
))

layout = dict(
title=title,
autosize=False,
width="600",
height="480",
yaxis=dict(
#            type='log',
autorange=True
)
)

#    py.iplot(data, filename='basic-scatter')

fig = Figure(data=plotdata, layout=layout)

return iplot(fig)

In [22]:
def plot_matrix(lossframe, x_labels, y_labels, x_suffix="", y_suffix=""):

pivot = lossframe.pivot_table(index=[y_labels], columns=[x_labels], values=['mse'])
#    print(pivot)
# specify labels as strings, to force plotly to use a discrete axis
#    print(pivot.columns.levels[1]).values
#    print(lossframe[x_labels].dtype)

if lossframe[x_labels].dtype == np.float64 or lossframe[x_labels].dtype == np.float32:
xaxis = ["%f %s" % (i, x_suffix) for i in pivot.columns.levels[1].values]
else:
xaxis = ["%d %s" % (i, x_suffix) for i in pivot.columns.levels[1].values]
if lossframe[y_labels].dtype == np.float64 or lossframe[y_labels].dtype == np.float32:
yaxis = ["%f %s" % (i, y_suffix) for i in pivot.index.values]
else:
yaxis = ["%d %s" % (i, y_suffix) for i in pivot.index.values]

#    print(xaxis, yaxis)
"""plot a heat map of a matrix"""
chart_width=640
chart_height=480

layout = dict(
title="%s v. %s" % (x_labels, y_labels),
height=chart_height,
width=chart_width,
margin=dict(
l=150,
r=30,
b=120,
t=100,
),
xaxis=dict(
title=x_labels,
tickfont=dict(
family='Arial, sans-serif',
size=10,
color='black'
),
),
yaxis=dict(
title=y_labels,
tickfont=dict(
family='Arial, sans-serif',
size=10,
color='black'
),
),
)

data = [Heatmap(z=pivot.values,
x=xaxis,
y=yaxis,
colorscale=[[0, 'rgb(0,0,255)', [1, 'rgb(255,0,0)']]],
)
]

fig = Figure(data=data, layout=layout)


### Compare LASSO, OLS models¶

In [23]:
perf_post_LASSO = backtestmodel.cumulative_return
mychart([perf_post_LASSO],["Post-LASSO"], title="Post-LASSO")

In [24]:
# do LASSO subset selection at each timestep
backtestmodel = BacktestModel(X, Y,
create_model=LinearRegression,
coef_dict_param="timestep",
startindex=FIRST_TRAIN_MONTHS,
fit_missing='mean',
scaler = None)
backtestmodel.gen_predictions(verbose=False)
backtestmodel.gen_returns(calc_returns, verbose=False)
backtestmodel.report_returns(start_date=start_date_str, freq='M')
backtestmodel.evaluate_predictions()
backtestmodel.evaluate_quantiles(chart=True, verbose=True)
perf_LASSO_each_timestep = backtestmodel.cumulative_return
mychart([perf_LASSO_each_timestep],["LASSO each timestep"], title="LASSO each timestep")

................................................................................
00:15:39 Still training step 80 of 563
................................................................................
00:15:52 Still training step 160 of 563
................................................................................
00:16:04 Still training step 240 of 563
................................................................................
00:16:17 Still training step 320 of 563
................................................................................
00:16:29 Still training step 400 of 563
................................................................................
00:16:41 Still training step 480 of 563
................................................................................
00:16:55 Still training step 560 of 563
...
Mean return: 3.545%
Monthly Sharpe ratio: 0.674
OOS MSE across all predictions: 41.4734
In-sample MSE: 35.7675
Variance: 39.4097
R-squared: -0.0524
Avg rank correlation (Kendall's tau): 0.0331 (Expected: 0)
5-quintile accuracy: 0.2163 (Expected: 0.2)
Long/short/flat accuracy: 0.4584 (Expected: 0.44)
Confusion matrix for quantile 0
[[10910  2602]
[ 2598   780]]
Chi-square: 24.8287 (p-value: 0.00001677)
Confusion matrix for quantile 1
[[10840  2672]
[ 2672   706]]
Chi-square: 2.1374 (p-value: 0.54439176)
Confusion matrix for quantile 2
[[10831  2681]
[ 2675   703]]
Chi-square: 1.6009 (p-value: 0.65918763)
Confusion matrix for quantile 3
[[10827  2685]
[ 2690   688]]
Chi-square: 0.4245 (p-value: 0.93512982)
Confusion matrix for quantile 4
[[10916  2596]
[ 2601   777]]
Chi-square: 24.2603 (p-value: 0.00002204)
Excess true positive in quintiles 1 + 5: 205.800000

In [25]:
# use all predictors at each timestep
backtestmodel = BacktestModel(X, Y,
create_model=LinearRegression,
coef_dict_param="all",
startindex=FIRST_TRAIN_MONTHS,
fit_missing='mean',
scaler=None)
backtestmodel.gen_predictions(verbose=False)
backtestmodel.gen_returns(calc_returns, verbose=False)
backtestmodel.report_returns(start_date=start_date_str, freq='M')
backtestmodel.evaluate_predictions()
backtestmodel.evaluate_quantiles(chart=True, verbose=False)
perf_all_preds = backtestmodel.cumulative_return
mychart([perf_all_preds],["All preds"], title="OLS all predictors")

................................................................................
00:16:58 Still training step 80 of 563
................................................................................
00:17:00 Still training step 160 of 563
................................................................................
00:17:02 Still training step 240 of 563
................................................................................
00:17:04 Still training step 320 of 563
................................................................................
00:17:06 Still training step 400 of 563
................................................................................
00:17:08 Still training step 480 of 563
................................................................................
00:17:11 Still training step 560 of 563
...
Mean return: 2.781%
Monthly Sharpe ratio: 0.506
OOS MSE across all predictions: 43.8692
In-sample MSE: 34.7467
Variance: 39.4097
R-squared: -0.1132
Avg rank correlation (Kendall's tau): 0.0246 (Expected: 0)
5-quintile accuracy: 0.2161 (Expected: 0.2)
Long/short/flat accuracy: 0.4657 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 225.800000

In [26]:
# pure LASSO (not LASSO followed by OLS on selected subset)
def create_model_lasso():
return LassoLarsIC(criterion='aic')

# use all predictors at each timestep
backtestmodel = BacktestModel(X, Y,
create_model=create_model_lasso,
coef_dict_param="all",
startindex=FIRST_TRAIN_MONTHS,
fit_missing='mean',
scaler=None)
backtestmodel.gen_predictions(verbose=False)
backtestmodel.gen_returns(calc_returns, verbose=False)
backtestmodel.report_returns(start_date=start_date_str, freq='M')
backtestmodel.evaluate_predictions()
backtestmodel.evaluate_quantiles(chart=True, verbose=False)
perf_LASSO_only = backtestmodel.cumulative_return
mychart([perf_LASSO_only],["LASSO only"], title="LASSO only")

................................................................................
00:17:23 Still training step 80 of 563
................................................................................
00:17:34 Still training step 160 of 563
................................................................................
00:17:45 Still training step 240 of 563
................................................................................
00:17:56 Still training step 320 of 563
................................................................................
00:18:07 Still training step 400 of 563
................................................................................
00:18:18 Still training step 480 of 563
................................................................................
00:18:29 Still training step 560 of 563
...
Mean return: 3.077%
Monthly Sharpe ratio: 0.553
OOS MSE across all predictions: 40.0227
In-sample MSE: 36.1107
Variance: 39.4097
R-squared: -0.0156
Avg rank correlation (Kendall's tau): 0.0331 (Expected: 0)
5-quintile accuracy: 0.2187 (Expected: 0.2)
Long/short/flat accuracy: 0.4599 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 198.800000

In [27]:
mychart([perf_LASSO_each_timestep, perf_all_preds, perf_LASSO_only],["LASSO / OLS", "OLS Only", "LASSO only"])


### Use walk-forward cross-validation to evaluate a variety of sklearn regression algos¶

In [46]:
# test walk-forward xval, get a baseline MSE for LinearRegression: ~41
backtestmodel = BacktestModel(X, Y,
create_model=LinearRegression,
coef_dict_param="timestep",
startindex=FIRST_TRAIN_MONTHS,
fit_missing='mean',
scaler=None)

backtestmodel.walkforward_xval(n_splits=5, verbose=True)
backtestmodel.gen_returns(calc_returns, verbose=False)
backtestmodel.report_returns(start_date=start_date_str, freq='M')
backtestmodel.evaluate_predictions()
backtestmodel.evaluate_quantiles(chart=False, verbose=False)

18:47:17 Generate splits [137, 274, 411, 548, 684]
Steps: [137, 274, 411, 548]
Training on first 137 rows (0:136), putting predictions in rows range(137, 274)
.Training on first 274 rows (0:273), putting predictions in rows range(274, 411)
.Training on first 411 rows (0:410), putting predictions in rows range(411, 548)
.Training on first 548 rows (0:547), putting predictions in rows range(548, 684)
.
Mean return: 2.861%
Monthly Sharpe ratio: 0.575
OOS MSE across all predictions: 41.3994
In-sample MSE: 35.7131
Variance: 39.6015
R-squared: -0.0454
Avg rank correlation (Kendall's tau): 0.0253 (Expected: 0)
5-quintile accuracy: 0.2149 (Expected: 0.2)
Long/short/flat accuracy: 0.4562 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 170.200000

In [47]:
# try all the sklearn regression estimators
estimators = []

for name, class_ in all_estimators():
if not issubclass(class_, sklearn.base.ClassifierMixin):
if hasattr(class_, 'predict'):
print(class_)
estimators.append(class_)

<class 'sklearn.linear_model.bayes.ARDRegression'>
<class 'sklearn.cluster.affinity_propagation_.AffinityPropagation'>
<class 'sklearn.ensemble.bagging.BaggingRegressor'>
<class 'sklearn.mixture.bayesian_mixture.BayesianGaussianMixture'>
<class 'sklearn.linear_model.bayes.BayesianRidge'>
<class 'sklearn.cluster.birch.Birch'>
<class 'sklearn.cross_decomposition.cca_.CCA'>
<class 'sklearn.mixture.dpgmm.DPGMM'>
<class 'sklearn.tree.tree.DecisionTreeRegressor'>
<class 'sklearn.linear_model.coordinate_descent.ElasticNet'>
<class 'sklearn.linear_model.coordinate_descent.ElasticNetCV'>
<class 'sklearn.tree.tree.ExtraTreeRegressor'>
<class 'sklearn.ensemble.forest.ExtraTreesRegressor'>
<class 'sklearn.mixture.gmm.GMM'>
<class 'sklearn.mixture.gaussian_mixture.GaussianMixture'>
<class 'sklearn.gaussian_process.gaussian_process.GaussianProcess'>
<class 'sklearn.gaussian_process.gpr.GaussianProcessRegressor'>
<class 'sklearn.linear_model.huber.HuberRegressor'>
<class 'sklearn.ensemble.iforest.IsolationForest'>
<class 'sklearn.cluster.k_means_.KMeans'>
<class 'sklearn.neighbors.regression.KNeighborsRegressor'>
<class 'sklearn.kernel_ridge.KernelRidge'>
<class 'sklearn.linear_model.least_angle.Lars'>
<class 'sklearn.linear_model.least_angle.LarsCV'>
<class 'sklearn.linear_model.coordinate_descent.Lasso'>
<class 'sklearn.linear_model.coordinate_descent.LassoCV'>
<class 'sklearn.linear_model.least_angle.LassoLars'>
<class 'sklearn.linear_model.least_angle.LassoLarsCV'>
<class 'sklearn.linear_model.least_angle.LassoLarsIC'>
<class 'sklearn.linear_model.base.LinearRegression'>
<class 'sklearn.svm.classes.LinearSVR'>
<class 'sklearn.neural_network.multilayer_perceptron.MLPRegressor'>
<class 'sklearn.cluster.mean_shift_.MeanShift'>
<class 'sklearn.cluster.k_means_.MiniBatchKMeans'>
<class 'sklearn.svm.classes.NuSVR'>
<class 'sklearn.svm.classes.OneClassSVM'>
<class 'sklearn.linear_model.omp.OrthogonalMatchingPursuit'>
<class 'sklearn.linear_model.omp.OrthogonalMatchingPursuitCV'>
<class 'sklearn.cross_decomposition.pls_.PLSCanonical'>
<class 'sklearn.cross_decomposition.pls_.PLSRegression'>
<class 'sklearn.linear_model.passive_aggressive.PassiveAggressiveRegressor'>
<class 'sklearn.linear_model.ransac.RANSACRegressor'>
<class 'sklearn.ensemble.forest.RandomForestRegressor'>
<class 'sklearn.linear_model.ridge.Ridge'>
<class 'sklearn.linear_model.ridge.RidgeCV'>
<class 'sklearn.svm.classes.SVR'>
<class 'sklearn.linear_model.theil_sen.TheilSenRegressor'>
<class 'sklearn.mixture.dpgmm.VBGMM'>
<class 'sklearn.linear_model.ridge._BaseRidgeCV'>
<class 'sklearn.gaussian_process.gpc._BinaryGaussianProcessClassifierLaplace'>
<class 'sklearn.multiclass._ConstantPredictor'>
<class 'sklearn.mixture.dpgmm._DPGMMBase'>
<class 'sklearn.mixture.gmm._GMMBase'>
<class 'sklearn.linear_model.ridge._RidgeGCV'>

In [48]:
# eliminate some that didn't work, gave errors, nans, maybe need correct parameters
estimators = [
sklearn.ensemble.bagging.BaggingRegressor,
sklearn.ensemble.forest.ExtraTreesRegressor,
sklearn.ensemble.forest.RandomForestRegressor,
#sklearn.ensemble.iforest.IsolationForest,
#sklearn.gaussian_process.gpr.GaussianProcessRegressor,
sklearn.kernel_ridge.KernelRidge,
sklearn.linear_model.base.LinearRegression,
#sklearn.linear_model.bayes.ARDRegression, # takes a couple of hours
sklearn.linear_model.bayes.BayesianRidge,
sklearn.linear_model.coordinate_descent.ElasticNet,
sklearn.linear_model.coordinate_descent.ElasticNetCV,
sklearn.linear_model.coordinate_descent.Lasso,
sklearn.linear_model.coordinate_descent.LassoCV,
sklearn.linear_model.huber.HuberRegressor,
sklearn.linear_model.least_angle.Lars,
sklearn.linear_model.least_angle.LarsCV,
sklearn.linear_model.least_angle.LassoLars,
sklearn.linear_model.least_angle.LassoLarsCV,
sklearn.linear_model.least_angle.LassoLarsIC,
sklearn.linear_model.omp.OrthogonalMatchingPursuit,
sklearn.linear_model.omp.OrthogonalMatchingPursuitCV,
sklearn.linear_model.passive_aggressive.PassiveAggressiveRegressor,
sklearn.linear_model.ransac.RANSACRegressor,
sklearn.linear_model.ridge.Ridge,
sklearn.linear_model.ridge.RidgeCV,
#sklearn.linear_model.theil_sen.TheilSenRegressor, very slow
#sklearn.mixture.bayesian_mixture.BayesianGaussianMixture,
#sklearn.mixture.gaussian_mixture.GaussianMixture,
sklearn.neighbors.regression.KNeighborsRegressor,
#sklearn.neural_network.multilayer_perceptron.MLPRegressor, slow, tried above
sklearn.svm.classes.LinearSVR,
sklearn.svm.classes.NuSVR,
#sklearn.svm.classes.OneClassSVM,
sklearn.svm.classes.SVR,
sklearn.tree.tree.DecisionTreeRegressor,
sklearn.tree.tree.ExtraTreeRegressor,
xgboost.XGBRegressor,
]


In [49]:
# run all the models
sharpe_list = []
mse_list = []
kendalltau_list = []
excess_tp_list = []
quintile_accuracy_list = []
directional_accuracy_list = []
keys = []

for estimator in estimators:
print(str(estimator)[8:-2])
backtestmodel = BacktestModel(X, Y,
create_model=estimator,
coef_dict_param="all",
startindex=FIRST_TRAIN_MONTHS,
fit_missing='mean',
scaler=None)
backtestmodel.gen_predictions(verbose=False)
backtestmodel.gen_returns(calc_returns, verbose=False)
backtestmodel.report_returns(start_date=start_date_str, freq='M')
backtestmodel.evaluate_predictions()
backtestmodel.evaluate_quantiles(chart=True, verbose=False)
keys.append(str(estimator)[8:-2])
sharpe_list.append(backtestmodel.sharpe)
mse_list.append(backtestmodel.mse)
kendalltau_list.append(backtestmodel.kendalltau)
excess_tp_list.append(backtestmodel.excess_tp)
quintile_accuracy_list.append(backtestmodel.quintile_accuracy)
directional_accuracy_list.append(backtestmodel.directional_accuracy)
print("---")


sklearn.ensemble.bagging.BaggingRegressor
................................................................................
18:48:08 Still training step 80 of 563
................................................................................
18:49:12 Still training step 160 of 563
................................................................................
18:50:38 Still training step 240 of 563
................................................................................
18:52:28 Still training step 320 of 563
................................................................................
18:54:48 Still training step 400 of 563
................................................................................
18:57:39 Still training step 480 of 563
................................................................................
19:01:13 Still training step 560 of 563
...
Mean return: 1.622%
Monthly Sharpe ratio: 0.348
OOS MSE across all predictions: 46.3171
In-sample MSE: 7.7592
Variance: 39.4097
R-squared: -0.1753
Avg rank correlation (Kendall's tau): 0.0093 (Expected: 0)
5-quintile accuracy: 0.2069 (Expected: 0.2)
Long/short/flat accuracy: 0.4517 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 95.800000

---
sklearn.ensemble.forest.ExtraTreesRegressor
................................................................................
19:01:56 Still training step 80 of 563
................................................................................
19:02:35 Still training step 160 of 563
................................................................................
19:03:23 Still training step 240 of 563
................................................................................
19:04:18 Still training step 320 of 563
................................................................................
19:05:22 Still training step 400 of 563
................................................................................
19:06:36 Still training step 480 of 563
................................................................................
19:08:01 Still training step 560 of 563
...
Mean return: 1.621%
Monthly Sharpe ratio: 0.348
OOS MSE across all predictions: 47.0459
In-sample MSE: 0.0000
Variance: 39.4097
R-squared: -0.1938
Avg rank correlation (Kendall's tau): 0.0105 (Expected: 0)
5-quintile accuracy: 0.2028 (Expected: 0.2)
Long/short/flat accuracy: 0.4509 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 92.800000

---
sklearn.ensemble.forest.RandomForestRegressor
................................................................................
19:08:51 Still training step 80 of 563
................................................................................
19:09:55 Still training step 160 of 563
................................................................................
19:11:20 Still training step 240 of 563
................................................................................
19:13:09 Still training step 320 of 563
................................................................................
19:15:28 Still training step 400 of 563
................................................................................
19:18:18 Still training step 480 of 563
................................................................................
19:21:48 Still training step 560 of 563
...
Mean return: 1.896%
Monthly Sharpe ratio: 0.426
OOS MSE across all predictions: 45.8743
In-sample MSE: 7.8784
Variance: 39.4097
R-squared: -0.1640
Avg rank correlation (Kendall's tau): 0.0230 (Expected: 0)
5-quintile accuracy: 0.2087 (Expected: 0.2)
Long/short/flat accuracy: 0.4512 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 108.800000

---
................................................................................
19:23:32 Still training step 80 of 563
................................................................................
19:25:28 Still training step 160 of 563
................................................................................
19:27:54 Still training step 240 of 563
................................................................................
19:30:45 Still training step 320 of 563
................................................................................
19:34:04 Still training step 400 of 563
................................................................................
19:37:53 Still training step 480 of 563
................................................................................
19:42:09 Still training step 560 of 563
...
Mean return: 2.007%
Monthly Sharpe ratio: 0.439
OOS MSE across all predictions: 46.7213
In-sample MSE: 12.0030
Variance: 39.4097
R-squared: -0.1855
Avg rank correlation (Kendall's tau): 0.0196 (Expected: 0)
5-quintile accuracy: 0.2123 (Expected: 0.2)
Long/short/flat accuracy: 0.4555 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 143.800000

---
................................................................................
19:44:57 Still training step 80 of 563
................................................................................
19:47:55 Still training step 160 of 563
................................................................................
19:51:22 Still training step 240 of 563
................................................................................
19:55:17 Still training step 320 of 563
................................................................................
19:59:41 Still training step 400 of 563
................................................................................
20:04:36 Still training step 480 of 563
................................................................................
20:09:49 Still training step 560 of 563
...
Mean return: 2.121%
Monthly Sharpe ratio: 0.455
OOS MSE across all predictions: 42.6787
In-sample MSE: 27.1000
Variance: 39.4097
R-squared: -0.0829
Avg rank correlation (Kendall's tau): 0.0186 (Expected: 0)
5-quintile accuracy: 0.2092 (Expected: 0.2)
Long/short/flat accuracy: 0.4545 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 117.800000

---
sklearn.kernel_ridge.KernelRidge
................................................................................
20:10:08 Still training step 80 of 563
................................................................................
20:10:10 Still training step 160 of 563
................................................................................
20:10:13 Still training step 240 of 563
................................................................................
20:10:18 Still training step 320 of 563
................................................................................
20:10:23 Still training step 400 of 563
................................................................................
20:10:29 Still training step 480 of 563
................................................................................
20:10:38 Still training step 560 of 563
...
Mean return: 2.553%
Monthly Sharpe ratio: 0.471
OOS MSE across all predictions: 44.0938
In-sample MSE: 35.0781
Variance: 39.4097
R-squared: -0.1189
Avg rank correlation (Kendall's tau): 0.0243 (Expected: 0)
5-quintile accuracy: 0.2175 (Expected: 0.2)
Long/short/flat accuracy: 0.4645 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 212.800000

---
sklearn.linear_model.base.LinearRegression
................................................................................
20:10:41 Still training step 80 of 563
................................................................................
20:10:43 Still training step 160 of 563
................................................................................
20:10:45 Still training step 240 of 563
................................................................................
20:10:47 Still training step 320 of 563
................................................................................
20:10:50 Still training step 400 of 563
................................................................................
20:10:52 Still training step 480 of 563
................................................................................
20:10:54 Still training step 560 of 563
...
Mean return: 2.781%
Monthly Sharpe ratio: 0.506
OOS MSE across all predictions: 43.8692
In-sample MSE: 34.7467
Variance: 39.4097
R-squared: -0.1132
Avg rank correlation (Kendall's tau): 0.0246 (Expected: 0)
5-quintile accuracy: 0.2161 (Expected: 0.2)
Long/short/flat accuracy: 0.4657 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 225.800000

---
sklearn.linear_model.bayes.BayesianRidge
................................................................................
20:10:59 Still training step 80 of 563
................................................................................
20:11:03 Still training step 160 of 563
................................................................................
20:11:07 Still training step 240 of 563
................................................................................
20:11:11 Still training step 320 of 563
................................................................................
20:11:15 Still training step 400 of 563
................................................................................
20:11:19 Still training step 480 of 563
................................................................................
20:11:23 Still training step 560 of 563
...
Mean return: 3.688%
Monthly Sharpe ratio: 0.617
OOS MSE across all predictions: 39.7450
In-sample MSE: 36.0953
Variance: 39.4097
R-squared: -0.0085
Avg rank correlation (Kendall's tau): 0.0358 (Expected: 0)
5-quintile accuracy: 0.2222 (Expected: 0.2)
Long/short/flat accuracy: 0.4631 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 225.800000

---
sklearn.linear_model.coordinate_descent.ElasticNet
................................................................................
20:11:26 Still training step 80 of 563
................................................................................
20:11:28 Still training step 160 of 563
................................................................................
20:11:30 Still training step 240 of 563
................................................................................
20:11:31 Still training step 320 of 563
................................................................................
20:11:33 Still training step 400 of 563
................................................................................
20:11:35 Still training step 480 of 563
................................................................................
20:11:37 Still training step 560 of 563
...
Mean return: 3.145%
Monthly Sharpe ratio: 0.530
OOS MSE across all predictions: 40.8226
In-sample MSE: 35.3600
Variance: 39.4097
R-squared: -0.0359
Avg rank correlation (Kendall's tau): 0.0332 (Expected: 0)
5-quintile accuracy: 0.2175 (Expected: 0.2)
Long/short/flat accuracy: 0.4623 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 208.800000

---
sklearn.linear_model.coordinate_descent.ElasticNetCV
................................................................................
20:14:07 Still training step 80 of 563
................................................................................
20:16:08 Still training step 160 of 563
................................................................................
20:17:57 Still training step 240 of 563
................................................................................
20:19:37 Still training step 320 of 563
................................................................................
20:21:10 Still training step 400 of 563
................................................................................
20:22:32 Still training step 480 of 563
................................................................................
20:24:02 Still training step 560 of 563
...
Mean return: 3.551%
Monthly Sharpe ratio: 0.632
OOS MSE across all predictions: 39.8834
In-sample MSE: 36.3646
Variance: 39.4097
R-squared: -0.0120
Avg rank correlation (Kendall's tau): 0.0349 (Expected: 0)
5-quintile accuracy: 0.2161 (Expected: 0.2)
Long/short/flat accuracy: 0.4572 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 196.800000

---
sklearn.linear_model.coordinate_descent.Lasso
................................................................................
20:24:08 Still training step 80 of 563
................................................................................
20:24:09 Still training step 160 of 563
................................................................................
20:24:11 Still training step 240 of 563
................................................................................
20:24:12 Still training step 320 of 563
................................................................................
20:24:13 Still training step 400 of 563
................................................................................
20:24:15 Still training step 480 of 563
................................................................................
20:24:16 Still training step 560 of 563
...
Mean return: 3.739%
Monthly Sharpe ratio: 0.639
OOS MSE across all predictions: 40.1141
In-sample MSE: 35.9242
Variance: 39.4097
R-squared: -0.0179
Avg rank correlation (Kendall's tau): 0.0356 (Expected: 0)
5-quintile accuracy: 0.2198 (Expected: 0.2)
Long/short/flat accuracy: 0.4645 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 237.800000

---
sklearn.linear_model.coordinate_descent.LassoCV
................................................................................
20:26:33 Still training step 80 of 563
................................................................................
20:28:42 Still training step 160 of 563
................................................................................
20:30:37 Still training step 240 of 563
................................................................................
20:32:20 Still training step 320 of 563
................................................................................
20:33:53 Still training step 400 of 563
................................................................................
20:35:20 Still training step 480 of 563
................................................................................
20:36:46 Still training step 560 of 563
...
Mean return: 3.323%
Monthly Sharpe ratio: 0.587
OOS MSE across all predictions: 39.9025
In-sample MSE: 36.3649
Variance: 39.4097
R-squared: -0.0125
Avg rank correlation (Kendall's tau): 0.0312 (Expected: 0)
5-quintile accuracy: 0.2182 (Expected: 0.2)
Long/short/flat accuracy: 0.4578 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 194.800000

---
sklearn.linear_model.huber.HuberRegressor
................................................................................
20:37:15 Still training step 80 of 563
................................................................................
20:37:38 Still training step 160 of 563
................................................................................
20:38:04 Still training step 240 of 563
................................................................................
20:38:30 Still training step 320 of 563
................................................................................
20:38:57 Still training step 400 of 563
................................................................................
20:39:24 Still training step 480 of 563
................................................................................
20:39:53 Still training step 560 of 563
...
Mean return: 1.166%
Monthly Sharpe ratio: 0.237
OOS MSE across all predictions: 43.6486
In-sample MSE: 35.1839
Variance: 39.4097
R-squared: -0.1076
Avg rank correlation (Kendall's tau): 0.0197 (Expected: 0)
5-quintile accuracy: 0.2088 (Expected: 0.2)
Long/short/flat accuracy: 0.4536 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 126.800000

---
sklearn.linear_model.least_angle.Lars
................................................................................
20:40:07 Still training step 80 of 563
................................................................................
20:40:18 Still training step 160 of 563
................................................................................
20:40:29 Still training step 240 of 563
................................................................................
20:40:40 Still training step 320 of 563
................................................................................
20:40:50 Still training step 400 of 563
................................................................................
20:41:00 Still training step 480 of 563
................................................................................
20:41:11 Still training step 560 of 563
...
Mean return: 2.210%
Monthly Sharpe ratio: 0.414
OOS MSE across all predictions: 44.4498
In-sample MSE: 35.0373
Variance: 39.4097
R-squared: -0.1279
Avg rank correlation (Kendall's tau): 0.0218 (Expected: 0)
5-quintile accuracy: 0.2121 (Expected: 0.2)
Long/short/flat accuracy: 0.4591 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 168.800000

---
sklearn.linear_model.least_angle.LarsCV
................................................................................
20:41:47 Still training step 80 of 563
................................................................................
20:42:20 Still training step 160 of 563
................................................................................
20:42:54 Still training step 240 of 563
................................................................................
20:43:27 Still training step 320 of 563
................................................................................
20:44:01 Still training step 400 of 563
................................................................................
20:44:35 Still training step 480 of 563
................................................................................
20:45:10 Still training step 560 of 563
...
Mean return: 2.888%
Monthly Sharpe ratio: 0.555
OOS MSE across all predictions: 39.8423
In-sample MSE: 36.4935
Variance: 39.4097
R-squared: -0.0110
Avg rank correlation (Kendall's tau): 0.0286 (Expected: 0)
5-quintile accuracy: 0.2165 (Expected: 0.2)
Long/short/flat accuracy: 0.4562 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 178.800000

---
sklearn.linear_model.least_angle.LassoLars
................................................................................
20:45:14 Still training step 80 of 563
................................................................................
20:45:15 Still training step 160 of 563
................................................................................
20:45:16 Still training step 240 of 563
................................................................................
20:45:17 Still training step 320 of 563
................................................................................
20:45:18 Still training step 400 of 563
................................................................................
20:45:20 Still training step 480 of 563
................................................................................
20:45:21 Still training step 560 of 563
...
Mean return: -1.129%
Monthly Sharpe ratio: -0.178
OOS MSE across all predictions: 39.8418
In-sample MSE: 37.1900
Variance: 39.4097
R-squared: -0.0110
Avg rank correlation (Kendall's tau): -0.0030 (Expected: 0)
5-quintile accuracy: 0.2106 (Expected: 0.2)
Long/short/flat accuracy: 0.4577 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 69.800000

---
sklearn.linear_model.least_angle.LassoLarsCV
................................................................................
20:45:59 Still training step 80 of 563
................................................................................
20:46:33 Still training step 160 of 563
................................................................................
20:47:08 Still training step 240 of 563
................................................................................
20:47:42 Still training step 320 of 563
................................................................................
20:48:17 Still training step 400 of 563
................................................................................
20:48:53 Still training step 480 of 563
................................................................................
20:49:29 Still training step 560 of 563
...
Mean return: 3.084%
Monthly Sharpe ratio: 0.594
OOS MSE across all predictions: 39.8093
In-sample MSE: 36.4704
Variance: 39.4097
R-squared: -0.0101
Avg rank correlation (Kendall's tau): 0.0301 (Expected: 0)
5-quintile accuracy: 0.2159 (Expected: 0.2)
Long/short/flat accuracy: 0.4557 (Expected: 0.44)
Excess true positive in quintiles 1 + 5: 177.800000