Exploring your Home Assistant data

The goal of this page is to get you familiar with the data in your Home Assistant instance. The page you're reading right now is a Jupyter Notebook. These documents contain instructions for the user and embedded Python code to generate graphs and tables of your data. It's interactive so you can at any time change the code of any example and just press the ▶️ button to update the example with your changes!

To get started, let's execute all examples on this page: in the menu at the top left, click on "Run" -> "Run All Cells".

In [ ]:
from detective.core import db_from_hass_config
db = db_from_hass_config()

In the following example, we're going to explore your most popular entities and break it down per period of the day (morning/afternoon/evening/night).

We will do this by looking at which services are getting called and which entities they targeted. To make the results more relevant, we will filter out any service call that happened because of another service call. So if a user turns on a script which turns on a light, we only count the interaction with the script and not with the light.

In [ ]:
from collections import Counter, OrderedDict
import json

from detective.time import time_category, sqlalch_datetime, localize, TIME_CATEGORIES

# Prepare a dictionary to track results
results = OrderedDict((time_cat, Counter()) for time_cat in TIME_CATEGORIES)

# We keep track of contexts that we processed so that we will only process
# the first service call in a context, and not subsequent calls.
context_processed = set()

for event in db.perform_query("SELECT * FROM events WHERE event_type = 'call_service' ORDER BY time_fired"):
    entity_ids = None

    # Skip if we have already processed an event that was part of this context
    if event.context_id in context_processed:
        continue

    try:
        event_data = json.loads(event.event_data)
    except ValueError:
        continue

    # Empty event data, skipping (shouldn't happen, but to be safe)
    if not event_data:
        continue

    service_data = event_data.get('service_data')

    # No service data found, skipping
    if not service_data:
        continue

    entity_ids = service_data.get('entity_id')

    # No entitiy IDs found, skip this event
    if entity_ids is None:
        continue

    if not isinstance(entity_ids, list):
        entity_ids = [entity_ids]

    context_processed.add(event.context_id)

    period = time_category(
        localize(sqlalch_datetime(event.time_fired)))

    for entity_id in entity_ids:
        results[period][entity_id] += 1

print("Most popular entities to interact with:")

RESULTS_TO_SHOW = 5

for period, period_results in results.items():
    print()
    
    entities = [
        ent_id for (ent_id, count)
        in period_results.most_common(RESULTS_TO_SHOW)
    ]
    
    result = ', '.join(entities) if entities else '-'
    print(f"{period.capitalize()}: {result}")

Next up

Let's now use pandas to visualise the results.

In [ ]:
import pandas as pd
df = pd.DataFrame.from_dict(results).fillna(0)
df

View states

Detective makes it easy to view your state data as a pandas dataframe.

In [ ]:
%%time

db.fetch_all_data()

We now have the raw data in a Pandas dataframe on the master_df attribute of our db object. We must use another class to process this data into a format suitable for plotting. Detective provides helper classes for working with numerical senor data and binary sensor data.

Lets first assert that you have data in each class:

In [ ]:
assert 'binary_sensor' and 'sensor' in db.domains

If you received an AssertionError on the above cell, some of the following may not work for you.

Assuming no error was raised, let's splot some numerical sensor data.

In [ ]:
db.entities.get('sensor')
In [ ]:
from detective.core import NumericalSensors

numerical_sensors = NumericalSensors(db.master_df)

You can quickly see what numerical entities are available:

In [ ]:
numerical_sensors.entities

We can access the numerical data as a pandas dataframe on the data attribute and print out the first couple of rows of data.

In [ ]:
numerical_sensors.data.head()

Let's plot a random numerical sensor.

In [ ]:
import random

if numerical_sensors.entities:
    numerical_sensors.plot(random.choice(numerical_sensors.entities))

Let's now plot data of a binary sensor:

In [ ]:
from detective.core import BinarySensors

binary_sensors = BinarySensors(db.master_df)
In [ ]:
binary_sensors.entities
In [ ]:
if binary_sensors.entities:
    binary_sensors.plot(random.choice(binary_sensors.entities))

That concludes our introduction to getting started with Home Assistant data.