Step 2 - Cleaning step and sleep data and looking for trends

This notebook explains how Fitbit data may be cleaned by removing clear outliers. We will furthermore look for trends in the step and sleep data. Note that all analysis performed here is rather simple and can be seen as first steps in a more sophisticated workflow.

Getting and cleaning step data

Let's start with downloading and visualizing my daily step data - a process that was already discussed in Step 1 of this analysis. The following code snippet downloads my daily step data over the last three months and prints today's step count as an example.

In [1]:
import fitbit
from bin.parse_credentials import parse_client_credentials, parse_tokens

client_id, client_secret = parse_client_credentials('client_id_secret.txt')

!./gather_keys_oauth2.py $client_id $client_secret >access_refresh_tokens.txt 2>/dev/null

access_token, refresh_token = parse_tokens('access_refresh_tokens.txt')

authd_client = fitbit.Fitbit(client_id, client_secret, oauth2=True,
                             access_token=access_token,
                             refresh_token=refresh_token)

steps_ts = authd_client.time_series('activities/steps', period='3m')
steps_ts['activities-steps'][-1]
Out[1]:
{'dateTime': '2016-03-28', 'value': '14241'}

Next, we convert all dates to Python datetime objects.

In [2]:
import dateutil.parser
date_steps = [(dateutil.parser.parse(date_steps_dict['dateTime']), int(date_steps_dict['value']))
              for date_steps_dict in steps_ts['activities-steps']]
date_steps[-1]
Out[2]:
(datetime.datetime(2016, 3, 28, 0, 0), 14241)

As a last step prior to taking an initial look at the daily step counts, the step data are stored as a pandas DataFrame. This will simplify the downstream analyses and facilitates simple data visualizations. The last five rows of the resulting DataFrame looks like this:

In [3]:
import pandas

date_steps_df = pandas.DataFrame(date_steps, columns=('Date', 'Steps'))
date_steps_df.tail()
Out[3]:
Date Steps
88 2016-03-24 5210
89 2016-03-25 13451
90 2016-03-26 17618
91 2016-03-27 9304
92 2016-03-28 14241

We also import the seaborn library to make the plots look a little more attractive and then plot the daily steps for each day.

In [4]:
%matplotlib inline
import seaborn

date_steps_df.plot(x='Date')
Out[4]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fa43f490b00>

To clean the step data, let's remove all days with fewer than 5000 steps. This mostly removes days before I started tracking my steps and also an outlier day with very few steps where I may have taken off the wristband in the middle of the day.

We furthermore remove an outlier day with unusually high step count by restricting to days with 20000 steps or less.

In [5]:
keep_rows = date_steps_df.Steps >= 5000
keep_rows = keep_rows.combine(date_steps_df.Steps <= 20000, lambda x,y: x and y)
date_steps_df = date_steps_df[keep_rows]

date_steps_df.plot(x='Date')
Out[5]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fa43cec0cc0>

Getting and cleaning sleep data

We will now download and visualize my daily sleep data following the process explained in Step 1 of this tutorial.

In [6]:
sleep_ts = authd_client.time_series('sleep/minutesAsleep', period='3m')
sleep_ts['sleep-minutesAsleep'][-7]
date_sleep = [(dateutil.parser.parse(date_sleep_dict['dateTime']), int(date_sleep_dict['value']))
              for date_sleep_dict in sleep_ts['sleep-minutesAsleep']]
date_sleep_df = pandas.DataFrame(date_sleep, columns=('Date', 'Sleep'))
date_sleep_df.plot(x='Date')
Out[6]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fa43c989710>

We observe that the number of minutes asleep seems quite low on many days (or, equivalently, nights) over the last three months. For instance, there are many days with three hours of sleep, also there are only very few days with more than seven hours of sleep. Knowing my own sleep behaviour, this seems wrong.

What's going on? Let's look for duplicated dates first that could indicate that some of the dates are repeated hence breaking up the sleep data into several entries for the same day.

In [7]:
date_series = date_sleep_df.Date
len(date_series.unique()) == len(date_series)
Out[7]:
True

Nope, days are not broken up as no dates are duplicated.

Let's look at the detailed daily sleep logs, instead of sleep time series data, offered for download by the Fitbit API. Detailed sleep logs for all dates covered by sleep time series data are downloaded by the following command:

In [8]:
sleep_daily = [authd_client.sleep(date) for date in date_series]

We see that daily sleep logs actually offer a lot of additional information with minute resolution. Warning: long output ahead, just scroll down to the next code snippet.

In [9]:
sleep_daily[-1]
Out[9]:
{'sleep': [{'awakeCount': 1,
   'awakeDuration': 4,
   'awakeningsCount': 30,
   'dateOfSleep': '2016-03-28',
   'duration': 28920000,
   'efficiency': 91,
   'isMainSleep': True,
   'logId': 11239706727,
   'minuteData': [{'dateTime': '22:04:00', 'value': '1'},
    {'dateTime': '22:05:00', 'value': '1'},
    {'dateTime': '22:06:00', 'value': '1'},
    {'dateTime': '22:07:00', 'value': '1'},
    {'dateTime': '22:08:00', 'value': '2'},
    {'dateTime': '22:09:00', 'value': '1'},
    {'dateTime': '22:10:00', 'value': '1'},
    {'dateTime': '22:11:00', 'value': '2'},
    {'dateTime': '22:12:00', 'value': '1'},
    {'dateTime': '22:13:00', 'value': '1'},
    {'dateTime': '22:14:00', 'value': '1'},
    {'dateTime': '22:15:00', 'value': '1'},
    {'dateTime': '22:16:00', 'value': '1'},
    {'dateTime': '22:17:00', 'value': '1'},
    {'dateTime': '22:18:00', 'value': '1'},
    {'dateTime': '22:19:00', 'value': '1'},
    {'dateTime': '22:20:00', 'value': '1'},
    {'dateTime': '22:21:00', 'value': '1'},
    {'dateTime': '22:22:00', 'value': '1'},
    {'dateTime': '22:23:00', 'value': '1'},
    {'dateTime': '22:24:00', 'value': '1'},
    {'dateTime': '22:25:00', 'value': '1'},
    {'dateTime': '22:26:00', 'value': '1'},
    {'dateTime': '22:27:00', 'value': '1'},
    {'dateTime': '22:28:00', 'value': '1'},
    {'dateTime': '22:29:00', 'value': '1'},
    {'dateTime': '22:30:00', 'value': '1'},
    {'dateTime': '22:31:00', 'value': '1'},
    {'dateTime': '22:32:00', 'value': '1'},
    {'dateTime': '22:33:00', 'value': '1'},
    {'dateTime': '22:34:00', 'value': '1'},
    {'dateTime': '22:35:00', 'value': '1'},
    {'dateTime': '22:36:00', 'value': '1'},
    {'dateTime': '22:37:00', 'value': '1'},
    {'dateTime': '22:38:00', 'value': '1'},
    {'dateTime': '22:39:00', 'value': '1'},
    {'dateTime': '22:40:00', 'value': '1'},
    {'dateTime': '22:41:00', 'value': '1'},
    {'dateTime': '22:42:00', 'value': '1'},
    {'dateTime': '22:43:00', 'value': '1'},
    {'dateTime': '22:44:00', 'value': '1'},
    {'dateTime': '22:45:00', 'value': '1'},
    {'dateTime': '22:46:00', 'value': '1'},
    {'dateTime': '22:47:00', 'value': '1'},
    {'dateTime': '22:48:00', 'value': '1'},
    {'dateTime': '22:49:00', 'value': '1'},
    {'dateTime': '22:50:00', 'value': '1'},
    {'dateTime': '22:51:00', 'value': '2'},
    {'dateTime': '22:52:00', 'value': '1'},
    {'dateTime': '22:53:00', 'value': '1'},
    {'dateTime': '22:54:00', 'value': '1'},
    {'dateTime': '22:55:00', 'value': '2'},
    {'dateTime': '22:56:00', 'value': '1'},
    {'dateTime': '22:57:00', 'value': '1'},
    {'dateTime': '22:58:00', 'value': '1'},
    {'dateTime': '22:59:00', 'value': '1'},
    {'dateTime': '23:00:00', 'value': '1'},
    {'dateTime': '23:01:00', 'value': '1'},
    {'dateTime': '23:02:00', 'value': '1'},
    {'dateTime': '23:03:00', 'value': '2'},
    {'dateTime': '23:04:00', 'value': '1'},
    {'dateTime': '23:05:00', 'value': '1'},
    {'dateTime': '23:06:00', 'value': '1'},
    {'dateTime': '23:07:00', 'value': '1'},
    {'dateTime': '23:08:00', 'value': '1'},
    {'dateTime': '23:09:00', 'value': '1'},
    {'dateTime': '23:10:00', 'value': '1'},
    {'dateTime': '23:11:00', 'value': '1'},
    {'dateTime': '23:12:00', 'value': '1'},
    {'dateTime': '23:13:00', 'value': '1'},
    {'dateTime': '23:14:00', 'value': '1'},
    {'dateTime': '23:15:00', 'value': '1'},
    {'dateTime': '23:16:00', 'value': '1'},
    {'dateTime': '23:17:00', 'value': '1'},
    {'dateTime': '23:18:00', 'value': '1'},
    {'dateTime': '23:19:00', 'value': '1'},
    {'dateTime': '23:20:00', 'value': '1'},
    {'dateTime': '23:21:00', 'value': '1'},
    {'dateTime': '23:22:00', 'value': '1'},
    {'dateTime': '23:23:00', 'value': '1'},
    {'dateTime': '23:24:00', 'value': '1'},
    {'dateTime': '23:25:00', 'value': '1'},
    {'dateTime': '23:26:00', 'value': '1'},
    {'dateTime': '23:27:00', 'value': '1'},
    {'dateTime': '23:28:00', 'value': '1'},
    {'dateTime': '23:29:00', 'value': '1'},
    {'dateTime': '23:30:00', 'value': '1'},
    {'dateTime': '23:31:00', 'value': '2'},
    {'dateTime': '23:32:00', 'value': '1'},
    {'dateTime': '23:33:00', 'value': '1'},
    {'dateTime': '23:34:00', 'value': '1'},
    {'dateTime': '23:35:00', 'value': '1'},
    {'dateTime': '23:36:00', 'value': '1'},
    {'dateTime': '23:37:00', 'value': '1'},
    {'dateTime': '23:38:00', 'value': '1'},
    {'dateTime': '23:39:00', 'value': '1'},
    {'dateTime': '23:40:00', 'value': '1'},
    {'dateTime': '23:41:00', 'value': '1'},
    {'dateTime': '23:42:00', 'value': '1'},
    {'dateTime': '23:43:00', 'value': '1'},
    {'dateTime': '23:44:00', 'value': '1'},
    {'dateTime': '23:45:00', 'value': '1'},
    {'dateTime': '23:46:00', 'value': '1'},
    {'dateTime': '23:47:00', 'value': '1'},
    {'dateTime': '23:48:00', 'value': '2'},
    {'dateTime': '23:49:00', 'value': '1'},
    {'dateTime': '23:50:00', 'value': '1'},
    {'dateTime': '23:51:00', 'value': '1'},
    {'dateTime': '23:52:00', 'value': '1'},
    {'dateTime': '23:53:00', 'value': '1'},
    {'dateTime': '23:54:00', 'value': '1'},
    {'dateTime': '23:55:00', 'value': '2'},
    {'dateTime': '23:56:00', 'value': '1'},
    {'dateTime': '23:57:00', 'value': '1'},
    {'dateTime': '23:58:00', 'value': '1'},
    {'dateTime': '23:59:00', 'value': '1'},
    {'dateTime': '00:00:00', 'value': '1'},
    {'dateTime': '00:01:00', 'value': '1'},
    {'dateTime': '00:02:00', 'value': '1'},
    {'dateTime': '00:03:00', 'value': '1'},
    {'dateTime': '00:04:00', 'value': '1'},
    {'dateTime': '00:05:00', 'value': '1'},
    {'dateTime': '00:06:00', 'value': '1'},
    {'dateTime': '00:07:00', 'value': '1'},
    {'dateTime': '00:08:00', 'value': '1'},
    {'dateTime': '00:09:00', 'value': '1'},
    {'dateTime': '00:10:00', 'value': '1'},
    {'dateTime': '00:11:00', 'value': '1'},
    {'dateTime': '00:12:00', 'value': '1'},
    {'dateTime': '00:13:00', 'value': '1'},
    {'dateTime': '00:14:00', 'value': '1'},
    {'dateTime': '00:15:00', 'value': '1'},
    {'dateTime': '00:16:00', 'value': '2'},
    {'dateTime': '00:17:00', 'value': '2'},
    {'dateTime': '00:18:00', 'value': '1'},
    {'dateTime': '00:19:00', 'value': '1'},
    {'dateTime': '00:20:00', 'value': '2'},
    {'dateTime': '00:21:00', 'value': '2'},
    {'dateTime': '00:22:00', 'value': '1'},
    {'dateTime': '00:23:00', 'value': '1'},
    {'dateTime': '00:24:00', 'value': '1'},
    {'dateTime': '00:25:00', 'value': '1'},
    {'dateTime': '00:26:00', 'value': '1'},
    {'dateTime': '00:27:00', 'value': '1'},
    {'dateTime': '00:28:00', 'value': '1'},
    {'dateTime': '00:29:00', 'value': '1'},
    {'dateTime': '00:30:00', 'value': '1'},
    {'dateTime': '00:31:00', 'value': '1'},
    {'dateTime': '00:32:00', 'value': '1'},
    {'dateTime': '00:33:00', 'value': '1'},
    {'dateTime': '00:34:00', 'value': '1'},
    {'dateTime': '00:35:00', 'value': '1'},
    {'dateTime': '00:36:00', 'value': '1'},
    {'dateTime': '00:37:00', 'value': '1'},
    {'dateTime': '00:38:00', 'value': '1'},
    {'dateTime': '00:39:00', 'value': '1'},
    {'dateTime': '00:40:00', 'value': '1'},
    {'dateTime': '00:41:00', 'value': '1'},
    {'dateTime': '00:42:00', 'value': '2'},
    {'dateTime': '00:43:00', 'value': '2'},
    {'dateTime': '00:44:00', 'value': '2'},
    {'dateTime': '00:45:00', 'value': '1'},
    {'dateTime': '00:46:00', 'value': '1'},
    {'dateTime': '00:47:00', 'value': '1'},
    {'dateTime': '00:48:00', 'value': '2'},
    {'dateTime': '00:49:00', 'value': '1'},
    {'dateTime': '00:50:00', 'value': '1'},
    {'dateTime': '00:51:00', 'value': '1'},
    {'dateTime': '00:52:00', 'value': '1'},
    {'dateTime': '00:53:00', 'value': '1'},
    {'dateTime': '00:54:00', 'value': '1'},
    {'dateTime': '00:55:00', 'value': '1'},
    {'dateTime': '00:56:00', 'value': '1'},
    {'dateTime': '00:57:00', 'value': '1'},
    {'dateTime': '00:58:00', 'value': '1'},
    {'dateTime': '00:59:00', 'value': '1'},
    {'dateTime': '01:00:00', 'value': '1'},
    {'dateTime': '01:01:00', 'value': '1'},
    {'dateTime': '01:02:00', 'value': '1'},
    {'dateTime': '01:03:00', 'value': '1'},
    {'dateTime': '01:04:00', 'value': '1'},
    {'dateTime': '01:05:00', 'value': '1'},
    {'dateTime': '01:06:00', 'value': '1'},
    {'dateTime': '01:07:00', 'value': '1'},
    {'dateTime': '01:08:00', 'value': '2'},
    {'dateTime': '01:09:00', 'value': '1'},
    {'dateTime': '01:10:00', 'value': '1'},
    {'dateTime': '01:11:00', 'value': '1'},
    {'dateTime': '01:12:00', 'value': '2'},
    {'dateTime': '01:13:00', 'value': '1'},
    {'dateTime': '01:14:00', 'value': '1'},
    {'dateTime': '01:15:00', 'value': '1'},
    {'dateTime': '01:16:00', 'value': '1'},
    {'dateTime': '01:17:00', 'value': '1'},
    {'dateTime': '01:18:00', 'value': '1'},
    {'dateTime': '01:19:00', 'value': '1'},
    {'dateTime': '01:20:00', 'value': '1'},
    {'dateTime': '01:21:00', 'value': '2'},
    {'dateTime': '01:22:00', 'value': '1'},
    {'dateTime': '01:23:00', 'value': '1'},
    {'dateTime': '01:24:00', 'value': '1'},
    {'dateTime': '01:25:00', 'value': '1'},
    {'dateTime': '01:26:00', 'value': '1'},
    {'dateTime': '01:27:00', 'value': '1'},
    {'dateTime': '01:28:00', 'value': '1'},
    {'dateTime': '01:29:00', 'value': '1'},
    {'dateTime': '01:30:00', 'value': '1'},
    {'dateTime': '01:31:00', 'value': '1'},
    {'dateTime': '01:32:00', 'value': '1'},
    {'dateTime': '01:33:00', 'value': '1'},
    {'dateTime': '01:34:00', 'value': '1'},
    {'dateTime': '01:35:00', 'value': '1'},
    {'dateTime': '01:36:00', 'value': '1'},
    {'dateTime': '01:37:00', 'value': '1'},
    {'dateTime': '01:38:00', 'value': '1'},
    {'dateTime': '01:39:00', 'value': '1'},
    {'dateTime': '01:40:00', 'value': '1'},
    {'dateTime': '01:41:00', 'value': '1'},
    {'dateTime': '01:42:00', 'value': '1'},
    {'dateTime': '01:43:00', 'value': '1'},
    {'dateTime': '01:44:00', 'value': '2'},
    {'dateTime': '01:45:00', 'value': '1'},
    {'dateTime': '01:46:00', 'value': '1'},
    {'dateTime': '01:47:00', 'value': '1'},
    {'dateTime': '01:48:00', 'value': '1'},
    {'dateTime': '01:49:00', 'value': '1'},
    {'dateTime': '01:50:00', 'value': '1'},
    {'dateTime': '01:51:00', 'value': '1'},
    {'dateTime': '01:52:00', 'value': '1'},
    {'dateTime': '01:53:00', 'value': '1'},
    {'dateTime': '01:54:00', 'value': '1'},
    {'dateTime': '01:55:00', 'value': '1'},
    {'dateTime': '01:56:00', 'value': '1'},
    {'dateTime': '01:57:00', 'value': '1'},
    {'dateTime': '01:58:00', 'value': '1'},
    {'dateTime': '01:59:00', 'value': '1'},
    {'dateTime': '02:00:00', 'value': '1'},
    {'dateTime': '02:01:00', 'value': '1'},
    {'dateTime': '02:02:00', 'value': '1'},
    {'dateTime': '02:03:00', 'value': '1'},
    {'dateTime': '02:04:00', 'value': '1'},
    {'dateTime': '02:05:00', 'value': '1'},
    {'dateTime': '02:06:00', 'value': '1'},
    {'dateTime': '02:07:00', 'value': '1'},
    {'dateTime': '02:08:00', 'value': '1'},
    {'dateTime': '02:09:00', 'value': '1'},
    {'dateTime': '02:10:00', 'value': '1'},
    {'dateTime': '02:11:00', 'value': '1'},
    {'dateTime': '02:12:00', 'value': '1'},
    {'dateTime': '02:13:00', 'value': '1'},
    {'dateTime': '02:14:00', 'value': '1'},
    {'dateTime': '02:15:00', 'value': '1'},
    {'dateTime': '02:16:00', 'value': '1'},
    {'dateTime': '02:17:00', 'value': '1'},
    {'dateTime': '02:18:00', 'value': '1'},
    {'dateTime': '02:19:00', 'value': '1'},
    {'dateTime': '02:20:00', 'value': '1'},
    {'dateTime': '02:21:00', 'value': '1'},
    {'dateTime': '02:22:00', 'value': '1'},
    {'dateTime': '02:23:00', 'value': '1'},
    {'dateTime': '02:24:00', 'value': '1'},
    {'dateTime': '02:25:00', 'value': '1'},
    {'dateTime': '02:26:00', 'value': '1'},
    {'dateTime': '02:27:00', 'value': '1'},
    {'dateTime': '02:28:00', 'value': '1'},
    {'dateTime': '02:29:00', 'value': '1'},
    {'dateTime': '02:30:00', 'value': '1'},
    {'dateTime': '02:31:00', 'value': '1'},
    {'dateTime': '02:32:00', 'value': '1'},
    {'dateTime': '02:33:00', 'value': '1'},
    {'dateTime': '02:34:00', 'value': '1'},
    {'dateTime': '02:35:00', 'value': '1'},
    {'dateTime': '02:36:00', 'value': '1'},
    {'dateTime': '02:37:00', 'value': '1'},
    {'dateTime': '02:38:00', 'value': '1'},
    {'dateTime': '02:39:00', 'value': '1'},
    {'dateTime': '02:40:00', 'value': '2'},
    {'dateTime': '02:41:00', 'value': '2'},
    {'dateTime': '02:42:00', 'value': '1'},
    {'dateTime': '02:43:00', 'value': '1'},
    {'dateTime': '02:44:00', 'value': '2'},
    {'dateTime': '02:45:00', 'value': '1'},
    {'dateTime': '02:46:00', 'value': '1'},
    {'dateTime': '02:47:00', 'value': '1'},
    {'dateTime': '02:48:00', 'value': '1'},
    {'dateTime': '02:49:00', 'value': '1'},
    {'dateTime': '02:50:00', 'value': '1'},
    {'dateTime': '02:51:00', 'value': '1'},
    {'dateTime': '02:52:00', 'value': '1'},
    {'dateTime': '02:53:00', 'value': '1'},
    {'dateTime': '02:54:00', 'value': '1'},
    {'dateTime': '02:55:00', 'value': '1'},
    {'dateTime': '02:56:00', 'value': '1'},
    {'dateTime': '02:57:00', 'value': '1'},
    {'dateTime': '02:58:00', 'value': '1'},
    {'dateTime': '02:59:00', 'value': '1'},
    {'dateTime': '03:00:00', 'value': '1'},
    {'dateTime': '03:01:00', 'value': '1'},
    {'dateTime': '03:02:00', 'value': '1'},
    {'dateTime': '03:03:00', 'value': '1'},
    {'dateTime': '03:04:00', 'value': '2'},
    {'dateTime': '03:05:00', 'value': '1'},
    {'dateTime': '03:06:00', 'value': '1'},
    {'dateTime': '03:07:00', 'value': '1'},
    {'dateTime': '03:08:00', 'value': '1'},
    {'dateTime': '03:09:00', 'value': '1'},
    {'dateTime': '03:10:00', 'value': '1'},
    {'dateTime': '03:11:00', 'value': '1'},
    {'dateTime': '03:12:00', 'value': '1'},
    {'dateTime': '03:13:00', 'value': '1'},
    {'dateTime': '03:14:00', 'value': '1'},
    {'dateTime': '03:15:00', 'value': '1'},
    {'dateTime': '03:16:00', 'value': '1'},
    {'dateTime': '03:17:00', 'value': '1'},
    {'dateTime': '03:18:00', 'value': '1'},
    {'dateTime': '03:19:00', 'value': '1'},
    {'dateTime': '03:20:00', 'value': '1'},
    {'dateTime': '03:21:00', 'value': '1'},
    {'dateTime': '03:22:00', 'value': '1'},
    {'dateTime': '03:23:00', 'value': '1'},
    {'dateTime': '03:24:00', 'value': '1'},
    {'dateTime': '03:25:00', 'value': '1'},
    {'dateTime': '03:26:00', 'value': '1'},
    {'dateTime': '03:27:00', 'value': '1'},
    {'dateTime': '03:28:00', 'value': '1'},
    {'dateTime': '03:29:00', 'value': '1'},
    {'dateTime': '03:30:00', 'value': '1'},
    {'dateTime': '03:31:00', 'value': '1'},
    {'dateTime': '03:32:00', 'value': '1'},
    {'dateTime': '03:33:00', 'value': '1'},
    {'dateTime': '03:34:00', 'value': '1'},
    {'dateTime': '03:35:00', 'value': '1'},
    {'dateTime': '03:36:00', 'value': '1'},
    {'dateTime': '03:37:00', 'value': '1'},
    {'dateTime': '03:38:00', 'value': '1'},
    {'dateTime': '03:39:00', 'value': '1'},
    {'dateTime': '03:40:00', 'value': '1'},
    {'dateTime': '03:41:00', 'value': '1'},
    {'dateTime': '03:42:00', 'value': '1'},
    {'dateTime': '03:43:00', 'value': '1'},
    {'dateTime': '03:44:00', 'value': '1'},
    {'dateTime': '03:45:00', 'value': '1'},
    {'dateTime': '03:46:00', 'value': '1'},
    {'dateTime': '03:47:00', 'value': '1'},
    {'dateTime': '03:48:00', 'value': '1'},
    {'dateTime': '03:49:00', 'value': '2'},
    {'dateTime': '03:50:00', 'value': '1'},
    {'dateTime': '03:51:00', 'value': '1'},
    {'dateTime': '03:52:00', 'value': '1'},
    {'dateTime': '03:53:00', 'value': '1'},
    {'dateTime': '03:54:00', 'value': '1'},
    {'dateTime': '03:55:00', 'value': '1'},
    {'dateTime': '03:56:00', 'value': '1'},
    {'dateTime': '03:57:00', 'value': '1'},
    {'dateTime': '03:58:00', 'value': '1'},
    {'dateTime': '03:59:00', 'value': '1'},
    {'dateTime': '04:00:00', 'value': '1'},
    {'dateTime': '04:01:00', 'value': '1'},
    {'dateTime': '04:02:00', 'value': '1'},
    {'dateTime': '04:03:00', 'value': '1'},
    {'dateTime': '04:04:00', 'value': '1'},
    {'dateTime': '04:05:00', 'value': '1'},
    {'dateTime': '04:06:00', 'value': '1'},
    {'dateTime': '04:07:00', 'value': '1'},
    {'dateTime': '04:08:00', 'value': '1'},
    {'dateTime': '04:09:00', 'value': '1'},
    {'dateTime': '04:10:00', 'value': '1'},
    {'dateTime': '04:11:00', 'value': '2'},
    {'dateTime': '04:12:00', 'value': '1'},
    {'dateTime': '04:13:00', 'value': '1'},
    {'dateTime': '04:14:00', 'value': '1'},
    {'dateTime': '04:15:00', 'value': '1'},
    {'dateTime': '04:16:00', 'value': '1'},
    {'dateTime': '04:17:00', 'value': '1'},
    {'dateTime': '04:18:00', 'value': '1'},
    {'dateTime': '04:19:00', 'value': '1'},
    {'dateTime': '04:20:00', 'value': '1'},
    {'dateTime': '04:21:00', 'value': '1'},
    {'dateTime': '04:22:00', 'value': '1'},
    {'dateTime': '04:23:00', 'value': '1'},
    {'dateTime': '04:24:00', 'value': '1'},
    {'dateTime': '04:25:00', 'value': '1'},
    {'dateTime': '04:26:00', 'value': '1'},
    {'dateTime': '04:27:00', 'value': '1'},
    {'dateTime': '04:28:00', 'value': '1'},
    {'dateTime': '04:29:00', 'value': '1'},
    {'dateTime': '04:30:00', 'value': '1'},
    {'dateTime': '04:31:00', 'value': '1'},
    {'dateTime': '04:32:00', 'value': '1'},
    {'dateTime': '04:33:00', 'value': '1'},
    {'dateTime': '04:34:00', 'value': '1'},
    {'dateTime': '04:35:00', 'value': '1'},
    {'dateTime': '04:36:00', 'value': '1'},
    {'dateTime': '04:37:00', 'value': '1'},
    {'dateTime': '04:38:00', 'value': '1'},
    {'dateTime': '04:39:00', 'value': '1'},
    {'dateTime': '04:40:00', 'value': '1'},
    {'dateTime': '04:41:00', 'value': '1'},
    {'dateTime': '04:42:00', 'value': '1'},
    {'dateTime': '04:43:00', 'value': '1'},
    {'dateTime': '04:44:00', 'value': '2'},
    {'dateTime': '04:45:00', 'value': '1'},
    {'dateTime': '04:46:00', 'value': '1'},
    {'dateTime': '04:47:00', 'value': '1'},
    {'dateTime': '04:48:00', 'value': '1'},
    {'dateTime': '04:49:00', 'value': '1'},
    {'dateTime': '04:50:00', 'value': '1'},
    {'dateTime': '04:51:00', 'value': '1'},
    {'dateTime': '04:52:00', 'value': '1'},
    {'dateTime': '04:53:00', 'value': '1'},
    {'dateTime': '04:54:00', 'value': '1'},
    {'dateTime': '04:55:00', 'value': '1'},
    {'dateTime': '04:56:00', 'value': '1'},
    {'dateTime': '04:57:00', 'value': '1'},
    {'dateTime': '04:58:00', 'value': '1'},
    {'dateTime': '04:59:00', 'value': '1'},
    {'dateTime': '05:00:00', 'value': '1'},
    {'dateTime': '05:01:00', 'value': '1'},
    {'dateTime': '05:02:00', 'value': '1'},
    {'dateTime': '05:03:00', 'value': '1'},
    {'dateTime': '05:04:00', 'value': '1'},
    {'dateTime': '05:05:00', 'value': '1'},
    {'dateTime': '05:06:00', 'value': '1'},
    {'dateTime': '05:07:00', 'value': '2'},
    {'dateTime': '05:08:00', 'value': '1'},
    {'dateTime': '05:09:00', 'value': '2'},
    {'dateTime': '05:10:00', 'value': '2'},
    {'dateTime': '05:11:00', 'value': '2'},
    {'dateTime': '05:12:00', 'value': '2'},
    {'dateTime': '05:13:00', 'value': '3'},
    {'dateTime': '05:14:00', 'value': '3'},
    {'dateTime': '05:15:00', 'value': '3'},
    {'dateTime': '05:16:00', 'value': '3'},
    {'dateTime': '05:17:00', 'value': '2'},
    {'dateTime': '05:18:00', 'value': '1'},
    {'dateTime': '05:19:00', 'value': '1'},
    {'dateTime': '05:20:00', 'value': '1'},
    {'dateTime': '05:21:00', 'value': '1'},
    {'dateTime': '05:22:00', 'value': '1'},
    {'dateTime': '05:23:00', 'value': '1'},
    {'dateTime': '05:24:00', 'value': '1'},
    {'dateTime': '05:25:00', 'value': '2'},
    {'dateTime': '05:26:00', 'value': '2'},
    {'dateTime': '05:27:00', 'value': '1'},
    {'dateTime': '05:28:00', 'value': '2'},
    {'dateTime': '05:29:00', 'value': '2'},
    {'dateTime': '05:30:00', 'value': '2'},
    {'dateTime': '05:31:00', 'value': '1'},
    {'dateTime': '05:32:00', 'value': '1'},
    {'dateTime': '05:33:00', 'value': '1'},
    {'dateTime': '05:34:00', 'value': '1'},
    {'dateTime': '05:35:00', 'value': '1'},
    {'dateTime': '05:36:00', 'value': '1'},
    {'dateTime': '05:37:00', 'value': '1'},
    {'dateTime': '05:38:00', 'value': '1'},
    {'dateTime': '05:39:00', 'value': '1'},
    {'dateTime': '05:40:00', 'value': '1'},
    {'dateTime': '05:41:00', 'value': '1'},
    {'dateTime': '05:42:00', 'value': '1'},
    {'dateTime': '05:43:00', 'value': '1'},
    {'dateTime': '05:44:00', 'value': '1'},
    {'dateTime': '05:45:00', 'value': '1'},
    {'dateTime': '05:46:00', 'value': '1'},
    {'dateTime': '05:47:00', 'value': '1'},
    {'dateTime': '05:48:00', 'value': '1'},
    {'dateTime': '05:49:00', 'value': '1'},
    {'dateTime': '05:50:00', 'value': '1'},
    {'dateTime': '05:51:00', 'value': '1'},
    {'dateTime': '05:52:00', 'value': '1'},
    {'dateTime': '05:53:00', 'value': '1'},
    {'dateTime': '05:54:00', 'value': '1'},
    {'dateTime': '05:55:00', 'value': '1'},
    {'dateTime': '05:56:00', 'value': '1'},
    {'dateTime': '05:57:00', 'value': '1'},
    {'dateTime': '05:58:00', 'value': '1'},
    {'dateTime': '05:59:00', 'value': '1'},
    {'dateTime': '06:00:00', 'value': '1'},
    {'dateTime': '06:01:00', 'value': '2'},
    {'dateTime': '06:02:00', 'value': '2'},
    {'dateTime': '06:03:00', 'value': '2'},
    {'dateTime': '06:04:00', 'value': '1'},
    {'dateTime': '06:05:00', 'value': '2'}],
   'minutesAfterWakeup': 1,
   'minutesAsleep': 436,
   'minutesAwake': 45,
   'minutesToFallAsleep': 0,
   'restlessCount': 29,
   'restlessDuration': 42,
   'startTime': '2016-03-27T22:04:00.000',
   'timeInBed': 482}],
 'summary': {'totalMinutesAsleep': 436,
  'totalSleepRecords': 1,
  'totalTimeInBed': 482}}
In [10]:
sleep_minutes = [sleep_entry['summary']['totalMinutesAsleep'] for sleep_entry in sleep_daily]
sleep_minutes[-1]
Out[10]:
436

We now create another DataFrame summarizing my daily sleep data as obtained by the summary returned as part of the daily sleep logs request.

In [11]:
date_sleep_df = pandas.DataFrame({'Date': date_series, 'Sleep': sleep_minutes})
date_sleep_df.plot(x='Date')
Out[11]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fa43c9c15c0>

Now the sleep data looks a little bit more complete. In particluar since I seem to be sleeping around seven hours most nights which is close to my personal estimate (I would have guessed that I sleep about 7.5 hours each night).

Let's remove the remaining dates with less than 200 minutes of sleep (intercontinental flight? tracking error? low battery?).

In [12]:
keep_rows = date_sleep_df.Sleep >= 200
date_sleep_df = date_sleep_df[keep_rows]
date_sleep_df.plot(x='Date')
Out[12]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fa43c84edd8>

We will now look for trends in the step and sleep data using a sliding window approach: The mean step count over 10 subsequent days is computed, also known as the rolling mean, and then plotted.

For simplicity, let's choose a window size of 10 here and note that the window size could also be picked using an approach such as cross-validation. Furthermore, note that the rolling mean is a quite simple approach and more sphisticated strategies, e.g., weighted rolling means exist.

In [13]:
date_steps_df['RollingMeanSteps'] = date_steps_df.Steps.rolling(window=10, center=True).mean()
date_steps_df.plot(x='Date', y='RollingMeanSteps', title= 'Daily step counts rolling mean over 10 days')
Out[13]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fa43c4a8588>
In [14]:
date_sleep_df['RollingMeanSleep'] = date_sleep_df.Sleep.rolling(window=10, center=True).mean()
date_sleep_df.plot(x='Date', y='RollingMeanSleep', title= 'Daily sleep counts rolling mean over 10 days')
Out[14]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fa43c3db7f0>

Next steps

Before drawing conclusions about potential correlations between my daily step and sleep data, I would like to gather additional data over the next months and then return to this analysis. The reason is that I do not think that I can draw valuable conclusions from only two months of data right now. Once I collected more data, I would furthermore like to investigate whether my activity and sleep patterns depend on the day of the week.