This notebook demonstrates the use of the MSTICPy entity graph visualization built using the Bokeh library and NetworkX.
You must have msticpy installed:
%pip install --upgrade msticpy
Often when investigating an incident you will want to keep track of the key events and entities that appear in the investigation, along with the key relationships between them. An effective way to do this is to create a graph of the key entities that represents those entities and thier connections.
The EntitiyGraph
feature has been created to allow a user to create a graph of Incidents, Alerts, and other entities during the course of an investigation in order to keep track of an visualize these interactions.
A graph can be initially created using either an Alert, Incident, SecurityIncident or other entity type, with additional entities, links between these entities, and notes added as an investigation progresses.
As well as creating a graph object this feature allows for the plotting of the graph, allowing for interactive exploration of the entities and their links.
# Imports
from msticpy.common.utility import check_py_version
MIN_REQ_PYTHON = (3,6)
check_py_version(MIN_REQ_PYTHON)
import pandas as pd
from msticpy import init_notebook
init_notebook(globals())
from msticpy.datamodel.soc.incident import Incident
from msticpy.datamodel.entities.alert import Alert
from msticpy.datamodel.entities.url import Url
from msticpy.nbtools.security_alert import SecurityAlert
from msticpy.vis.entity_graph_tools import EntityGraph
inc = {
"id": "123",
"name": "135a072e-77c3-4293-8c9a-7fdd53b5d620",
"etag": '"d4fce673-dd4b-4d22-a39f-bda55d6079f3"',
"type": "Microsoft.SecurityInsights/Incidents",
"properties.title": "Sample Incident",
"properties.description": "This is a sample incident to support unit tests",
"properties.severity": "Medium",
"properties.status": "Active",
"properties.owner.objectId": "0a70480d-b1cd-4466-9b75-3814e34579eb",
"properties.owner.email": "user@contoso.com",
"properties.owner.assignedTo": "A User",
"properties.owner.userPrincipalName": "user@contoso.com",
"properties.labels": [{"labelName": "Tests Label", "labelType": "User"}],
"properties.firstActivityTimeUtc": "2021-09-22T14:39:24.04Z",
"properties.lastActivityTimeUtc": "2021-09-22T14:39:24.04Z",
"properties.lastModifiedTimeUtc": "2021-09-22T15:09:34.376619Z",
"properties.createdTimeUtc": "2021-09-22T15:09:09.2786667Z",
"properties.incidentNumber": 123,
"properties.additionalData.alertsCount": 1,
"properties.additionalData.bookmarksCount": 0,
"properties.additionalData.commentsCount": 0,
"properties.additionalData.alertProductNames": ["Azure Sentinel"],
"properties.additionalData.tactics": ["PrivilegeEscalation"],
"properties.relatedAnalyticRuleIds": ["123"],
"properties.incidentUrl": "https://portal.azure.com/#asset/Microsoft_Azure_Security_Insights/Incident/subscriptions/",
"Entities": [
(
"Host",
{
"dnsDomain": "demo.contoso.com",
"hostName": "demo",
"omsAgentID": "ce7903cf-2d8f-47e9-a338-2259f02a9779",
"osFamily": "Windows",
"osVersion": "10",
"additionalData": {
"DataSource": "SecurityEvent",
"AzureResourceId": "/subscriptions/ce7903cf-2d8f-47e9-a338-2259f02a9779/resourcegroups/test/providers/microsoft.compute/virtualmachines/demo",
"SubscriptionId": "ce7903cf-2d8f-47e9-a338-2259f02a9779",
"ResourceId": "/subscriptions/ce7903cf-2d8f-47e9-a338-2259f02a9779/resourceGroups/test/providers/Microsoft.Compute/virtualMachines/demo",
"VMUUID": "ce7903cf-2d8f-47e9-a338-2259f02a9779",
"ShouldResolveIp": "False",
},
"friendlyName": "demo",
},
),
(
"Account",
{
"accountName": "auser",
"displayName": "CONTOSO\\auser",
"friendlyName": "CONTOSO\\auser",
},
),
],
"Alerts": [
{
"ID": "8b7d06d8-dbae-4b23-87ed-1a27b75437d5",
"Name": "User Added to Priviledged Group in CONTOSO Domain",
"Entities": None,
}
],
"Type": "incident",
}
incident = Incident(inc)
display(incident)
Creating a graph is as simple as instantiating an EntityGraph
object and passing it a incident object.
To display the graph as a visualization can be achieved by calling the plot
method.
graph = EntityGraph(incident)
graph.plot()
As you can see above this has displayed a visualization that shows the incident, the alert associated with that incident and the entities associated with the incident.
The same can be achieved by passing in an Alert or SecurityAlert entity:
sample_alert = {
"StartTimeUtc": "2018-09-27 16:59:16",
"EndTimeUtc": "2018-09-27 16:59:16",
"ProviderAlertId": "b6329e79-0a94-4035-beee-c2e2657b71e3",
"SystemAlertId": "2518642332435550951_b6329e79-0a94-4035-beee-c2",
"ProviderName": "Detection",
"VendorName": "Microsoft",
"AlertType": "RegistryPersistence",
"AlertDisplayName": "Windows registry persistence method detected",
"Severity": "Low",
"IsIncident": False,
"ExtendedProperties": {
"resourceType": "Non-Azure Resource"
},
"CompromisedEntity": "TESTHOST",
"Entities": [
{
"Type": "host",
"$id": "1",
"HostName": "TESTHOST",
"DnsDomain": "DOM.CONTOSO.COM",
"IsDomainJoined": True,
"NTDomain": "DOM",
"NetBiosName": "TESTHOST",
"OsVersion": None,
"OSFamily": "Windows",
},
{
"Type": "process",
"$id": "3",
"CommandLine": "",
"Host": {"$ref": "1"},
"ProcessId": "0x940",
"ImageFile": "test.exe",
},
],
"ConfidenceLevel": "Unknown",
"ConfidenceScore": None,
"ConfidenceReasons": None,
"Intent": "Persistence",
"ExtendedLinks": None,
"AzureResourceId": None,
"AzureResourceSubscriptionId": None,
"TenantId": "b6329e79-0a94-4035-beee-c2e2657b71e3",
"WorkspaceId": "b6329e79-0a94-4035-beee-c2e2657b71e3",
"AgentId": "b6329e79-0a94-4035-beee-c2e2657b71e3",
"SourceComputerId": "b6329e79-0a94-4035-beee-c2e2657b71e3",
"SystemSource": "Non-Azure",
"WorkspaceSubscriptionId": "b6329e79-0a94-4035-beee-c2e2657b71e3",
"WorkspaceResourceGroup": "test-east-us",
"TimeGeneratedUtc": "2018-09-27 16:59:47",
}
alert = Alert(sample_alert)
graph = EntityGraph(alert)
graph.plot()
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) ~\AppData\Local\Temp\ipykernel_66244\1263983688.py in <module> 53 alert = Alert(sample_alert) 54 ---> 55 graph = EntityGraph(alert) 56 graph.plot() e:\src\msticpy\msticpy\vis\entity_graph_tools.py in __init__(self, entity) 54 self.alertentity_graph = nx.Graph(id="IncidentGraph") 55 if isinstance(entity, (Incident, Alert)): ---> 56 self._add_incident_or_alert_node(entity) 57 elif isinstance(entity, pd.DataFrame): 58 self.add_incident(entity) e:\src\msticpy\msticpy\vis\entity_graph_tools.py in _add_incident_or_alert_node(self, incident) 352 self._add_incident_node(incident) 353 elif isinstance(incident, Alert): --> 354 self._add_alert_node(incident) 355 356 def _add_entity_node(self, ent, attached_to=None): e:\src\msticpy\msticpy\vis\entity_graph_tools.py in _add_alert_node(self, alert, incident_name) 365 if alert["Entities"]: 366 for ent in alert["Entities"]: --> 367 self._add_entity_node(ent, alert.name_str) 368 if incident_name: 369 self.add_link(incident_name, alert.name_str) e:\src\msticpy\msticpy\vis\entity_graph_tools.py in _add_entity_node(self, ent, attached_to) 356 def _add_entity_node(self, ent, attached_to=None): 357 """Add an Entity to the graph.""" --> 358 self.alertentity_graph = nx.compose(self.alertentity_graph, ent.to_networkx()) 359 if attached_to: 360 self.add_link(attached_to, ent.name_str) e:\src\msticpy\msticpy\datamodel\entities\entity.py in to_networkx(self, graph) 564 graph = graph or nx.Graph() 565 if not graph.has_node(self): --> 566 graph.add_node(self.name_str, **self.node_properties) 567 for edge in self.edges: 568 if not isinstance(edge.source, Entity) or not isinstance( F:\anaconda\envs\msticpy\lib\site-packages\networkx\classes\graph.py in add_node(self, node_for_adding, **attr) 517 if node_for_adding not in self._node: 518 if node_for_adding is None: --> 519 raise ValueError("None cannot be a node") 520 self._adj[node_for_adding] = self.adjlist_inner_dict_factory() 521 attr_dict = self._node[node_for_adding] = self.node_attr_dict_factory() ValueError: None cannot be a node
sec_alert = SecurityAlert(pd.Series(sample_alert))
graph = EntityGraph(sec_alert)
graph.plot()
As can be seen above creating a graph from an Alert or SecurityAlert produces the same graph output. Here we can see the relationship between the entities and the alert, as well as between the two entities.
Graphs can also be created with just an entity if that is the starting point of the investigation:
url_ent = Url({"Url": "www.contoso.com"})
graph = EntityGraph(url_ent)
graph.plot()
It is also possible to create graphs containing multiple alerts or incidents by passing a DataFrame containing incident or alert events to EntityGraph
this will then convert these to the relevant entity type and plot them all on the one graph.
df = pd.read_pickle("data/sent_incidents.pkl")
df_graph = EntityGraph(df)
df_graph.plot()
As we have seen above a visual representation of the graph can be produced by calling .plot
. However you can also choose to return but not display the plot by passing the hide=True
parameter when calling .plot
:
Its often useful to see not just the connection between enities in a graph but also how they relate to each other temporarily, for example to see if two entities of suspicion overlap in the times observed, or whether two similar incidents have an overlap in terms of times assocaited. This can be done by calling .plot_with_timeline
.
df_graph.plot_with_timeline()
During an investigation, you will want to expand or collapse the graph based on the outcomes of your investigations. The EntityGraph supports the ability to add and remove entities from the graph during the investigation.
Entities that are added with the add_entity
or add_incident
functions, depending on whether the item being added is an incident or an entity. Added entities can be attached to another entity in the graph by specifying the name of the entity to attach to with the attached_to
parameter.
url_ent = Url(Url="www.contoso.com")
graph = EntityGraph(incident)
graph.add_entity(url_ent, attached_to="demo")
graph.plot()
Removing a entity from the graph is done with remove_node
function, with the name of the entity to remove passed with the name
parameter:
As well as adding entities to the graph you will also want to update the links between them as an investigation progresses. This can be done with the add_link
and remove_link
functions:
graph.add_link("www.contoso.com", "Incident: Sample Incident")
graph.add_link("www.contoso.com", "CONTOSO\\auser")
graph.remove_link("CONTOSO\\auser", "Incident: Sample Incident")
graph.plot()
Entities are not the only elements that you might want to record as part of an investigation. To include a wide range of other items and observations the EntityGraph has the concept of Notes. Notes are nodes in the graph that have free form titles and descriptions, allowing the user to add anything they need - be it a comment on an entity on the graph, or a query used to find an event.
Notes area added with the add_note
function. As with the add_entity
function notes can be attached to an existing entity in the graph. In addition, you can adjust the color of the node added to the graph, and add a username associated with the note.
graph.add_note(name="This is a note",
description="Notes allow for free form additions to the graph",
attached_to="Incident: Sample Incident")
graph.plot()
As a graph has been built up during the course of the investigation you may want to access or export some of the key elements of the graph. This can easily be done with the to_df
function.
graph.to_df()
Name | Description | Type | TimeGenerated | EndTime | StartTime | |
---|---|---|---|---|---|---|
0 | Sample Incident | 2021-09-22T15:09:09.2786667Z - Sample Incident - 123 | incident | 2021-09-22 15:09:09.278666700 | 2021-09-22T14:39:24.04Z | 2021-09-22T14:39:24.04Z |
1 | User Added to Priviledged Group in CONTOSO Domain | User Added to Priviledged Group in CONTOSO Domain - ['8b7d06d8-dbae-4b23-87ed-1a27b75437d5'] | alert | NaT | None | None |
2 | CONTOSO\auser | None | entity | NaT | None | None |
3 | www.contoso.com | www.contoso.com | entity | NaT | None | None |
4 | This is a note | Notes allow for free form additions to the graph | analystnote | 2021-10-06 09:05:35.203699000 | None | None |