# %pip install --upgrade msticpy[azsentinel]
from msticpy.context.azure.sentinel_core import MicrosoftSentinel
import msticpy.nbwidgets as widgets
from msticpy.data import data_obfus as mask
The first step to be able to use the features is to call the AzureSentinel class and connect to it. Authentication uses the standardized Azure authentication options of using environment variables, Azure CLI credentials, Managed Identities, and interactive logons.
azs = MicrosoftSentinel()
azs.connect()
Attempting to sign-in with environment variable credentials...
Once connected we need to select an Azure Sentinel workspace to get details from. The easies way to do this is with the get_subscriptions() and get_sentinel_workspaces() functions to select the subscription and workspace you with to connect to. If you already know which workspace you wish to connect to you can skip straight to the other functions and enter these details.
# Query for our subscriptions
subs = azs.get_subscriptions()
subs = subs.mp_obf.obfuscate(column_map={"Display Name": "str"})
# Display subscriptions (masked names) in a pick list
print("Select a subscription:")
sub = widgets.SelectItem(
item_list=subs['Display Name'].to_list(),
auto_display=True
)
Attempting to sign-in with environment variable credentials... obfuscating columns: Display Name, done Select a subscription:
VBox(children=(Text(value='', description='Filter:', style=DescriptionStyle(description_width='initial')), Sel…
# Get the subscription ID
sub_id = subs[subs['Display Name'] == sub.value].iloc[0]['Subscription ID']
# Query for workspaces in that subscription
workspaces = azs.get_sentinel_workspaces(sub_id = sub_id)
# Display workspaces in a list
print("Select an Azure Sentinel Workspace:")
ws = widgets.SelectItem(
item_dict=workspaces,
auto_display=True
)
Finding Azure Sentinel Workspaces... Attempting to sign-in with environment variable credentials... Select an Azure Sentinel Workspace:
VBox(children=(Text(value='', description='Filter:', style=DescriptionStyle(description_width='initial')), Sel…
Now that we have selected our workspace we can call various functions to get details about content in the workspace. These are typically returned as DataFrames. Below we get a list of hunting queries configured in our workspace.
queries = azs.get_hunting_queries(ws.value)
queries.head().drop(columns=["id", "etag", "name"])
type | properties.Category | properties.DisplayName | properties.Query | properties.Version | properties.Tags | properties.FunctionAlias | properties.FunctionParameters | |
---|---|---|---|---|---|---|---|---|
2 | Microsoft.OperationalInsights/savedSearches | Hunting Queries | Powershell | SecurityEvent\r\n| where ParentProcessName con... | 2 | NaN | NaN | NaN |
5 | Microsoft.OperationalInsights/savedSearches | Hunting Queries | Anomalous AAD Account Creation | \nBehaviorAnalytics\n| where ActionType == "Ad... | 2 | [{'Name': 'description', 'Value': ''}, {'Name'... | NaN | NaN |
7 | Microsoft.OperationalInsights/savedSearches | Hunting Queries | Entropy for Processes for a given Host | \n// May need to reduce the number of days if ... | 2 | [{'Name': 'description', 'Value': ''}, {'Name'... | NaN | NaN |
9 | Microsoft.OperationalInsights/savedSearches | Hunting Queries | RareDNSLookupWithDataTransfer | \nlet lookbackint = 7;\nlet lookupThreshold = ... | 2 | [{'Name': 'description', 'Value': ''}, {'Name'... | NaN | NaN |
12 | Microsoft.OperationalInsights/savedSearches | Hunting Queries | Least Common Processes by Command Line | \nlet Allowlist = dynamic (['foo.exe', 'baz.ex... | 2 | [{'Name': 'description', 'Value': ''}, {'Name'... | NaN | NaN |
Hunting queries return the raw queries associated with them, this allows us to pass the query directly to a QueryProvider in order to get the results of the hunting query within the notebook.
from msticpy.data.data_providers import QueryProvider
from msticpy.common.wsconfig import WorkspaceConfig
qry_prov = QueryProvider('LogAnalytics')
wkspace = WorkspaceConfig()
qry_prov.connect(wkspace.code_connect_str)
qry_prov.exec_query(queries['properties.Query'].iloc[2])
TimeGenerated | EventID | Computer | SubjectUserSid | Account | Weight | AdjustedProcessEntropy | FullDecimalProcessEntropy | Process | NewProcessName | CommandLine | ParentProcessName | TotalProcessCountOnHost | ProcessCountOnHost | DistinctComputersWithProcessCount | timestamp | HostCustomEntity | AccountCustomEntity | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 2020-11-21 21:18:51.317000+00:00 | 4688 | WinAttackSim | S-1-5-18 | WORKGROUP\WinAttackSim$ | 10.743361 | 10.743361 | 0.001074 | Defrag.exe | C:\Windows\System32\Defrag.exe | C:\windows\system32\defrag.exe -c -h -k -g -$ | C:\Windows\System32\svchost.exe | 12688 | 1 | 1 | 2020-11-21 21:18:51.317000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
1 | 2020-11-23 01:31:49.930000+00:00 | 4688 | WinAttackSim | S-1-5-18 | WORKGROUP\WinAttackSim$ | 10.743361 | 10.743361 | 0.001074 | SppExtComObj.Exe | C:\Windows\System32\SppExtComObj.Exe | C:\windows\system32\SppExtComObj.exe -Embedding | C:\Windows\System32\svchost.exe | 12688 | 1 | 1 | 2020-11-23 01:31:49.930000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
2 | 2020-11-20 17:18:36.960000+00:00 | 4688 | WinAttackSim | S-1-5-18 | WORKGROUP\WinAttackSim$ | 10.743361 | 10.743361 | 0.001074 | makecab.exe | C:\Windows\System32\makecab.exe | "C:\windows\system32\makecab.exe" C:\windows\L... | C:\Windows\WinSxS\amd64_microsoft-windows-serv... | 12688 | 1 | 1 | 2020-11-20 17:18:36.960000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
3 | 2020-11-21 21:18:51.303000+00:00 | 4688 | WinAttackSim | S-1-5-18 | WORKGROUP\WinAttackSim$ | 10.743361 | 10.743361 | 0.001074 | rundll32.exe | C:\Windows\System32\rundll32.exe | C:\windows\system32\rundll32.exe Windows.Stora... | C:\Windows\System32\svchost.exe | 12688 | 1 | 1 | 2020-11-21 21:18:51.303000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
4 | 2020-11-21 21:18:51.310000+00:00 | 4688 | WinAttackSim | S-1-5-18 | WORKGROUP\WinAttackSim$ | 10.743361 | 10.743361 | 0.001074 | tzsync.exe | C:\Windows\System32\tzsync.exe | C:\windows\system32\tzsync.exe | C:\Windows\System32\svchost.exe | 12688 | 1 | 1 | 2020-11-21 21:18:51.310000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
5 | 2020-11-18 01:18:36.913000+00:00 | 4688 | WinAttackSim | S-1-5-18 | WORKGROUP\WinAttackSim$ | 39.820861 | 19.910430 | 0.001991 | lpremove.exe | C:\Windows\System32\lpremove.exe | C:\windows\system32\lpremove.exe | C:\Windows\System32\svchost.exe | 12688 | 2 | 1 | 2020-11-18 01:18:36.913000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
6 | 2020-11-21 03:18:49.223000+00:00 | 4688 | WinAttackSim | S-1-5-18 | WORKGROUP\WinAttackSim$ | 39.820861 | 19.910430 | 0.001991 | lpremove.exe | C:\Windows\System32\lpremove.exe | C:\windows\system32\lpremove.exe | C:\Windows\System32\svchost.exe | 12688 | 2 | 1 | 2020-11-21 03:18:49.223000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
7 | 2020-11-23 01:31:49.957000+00:00 | 4688 | WinAttackSim | S-1-5-20 | WORKGROUP\WinAttackSim$ | 39.820861 | 19.910430 | 0.001991 | slui.exe | C:\Windows\System32\slui.exe | "C:\windows\System32\SLUI.exe" RuleId=502ff3ba... | C:\Windows\System32\SppExtComObj.Exe | 12688 | 2 | 1 | 2020-11-23 01:31:49.957000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
8 | 2020-11-23 01:31:54.340000+00:00 | 4688 | WinAttackSim | S-1-5-20 | WORKGROUP\WinAttackSim$ | 39.820861 | 19.910430 | 0.001991 | slui.exe | C:\Windows\System32\slui.exe | "C:\windows\System32\SLUI.exe" RuleId=379cccfb... | C:\Windows\System32\SppExtComObj.Exe | 12688 | 2 | 1 | 2020-11-23 01:31:54.340000+00:00 | WinAttackSim | WORKGROUP\WinAttackSim$ |
We can also get a list of configured alert rules:
alert_rules = azs.get_alert_rules(ws.value)
alert_rules.head().drop(columns=["id", "etag", "name"])
type | kind | properties.severity | properties.query | properties.queryFrequency | properties.queryPeriod | properties.triggerOperator | properties.triggerThreshold | properties.suppressionDuration | properties.suppressionEnabled | ... | properties.description | properties.tactics | properties.alertRuleTemplateName | properties.lastModifiedUtc | properties.customFields.Filename | properties.customFields.Reason | properties.productFilter | properties.severitiesFilter | properties.displayNamesFilter | properties.displayNamesExcludeFilter | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Microsoft.SecurityInsights/alertRules | Scheduled | Medium | let failureCountThreshold = 5;\nlet successCou... | P1D | P1D | GreaterThan | 0.0 | PT5H | False | ... | Identifies evidence of brute force activity ag... | [CredentialAccess] | 28b42356-45af-40a6-a0b4-a554cdfd5d8a | 2020-11-17T08:15:49.636781Z | NaN | NaN | NaN | NaN | NaN | NaN |
1 | Microsoft.SecurityInsights/alertRules | Scheduled | Medium | let timeframe = 1d;\n//Set a threshold of fail... | P1D | P1D | GreaterThan | 0.0 | PT5H | False | ... | This query creates a list of IP addresses with... | [InitialAccess, CredentialAccess] | ba144bf8-75b8-406f-9420-ed74397f9479 | 2020-11-11T05:16:14.5036485Z | FileName | Reason | NaN | NaN | NaN | NaN |
2 | Microsoft.SecurityInsights/alertRules | Scheduled | Medium | let timeframe = 1d;\nSecurityEvent\n| where Ti... | P1D | P1D | GreaterThan | 0.0 | PT5H | False | ... | Checks for event id 1102 which indicates the s... | [DefenseEvasion] | 80da0a8f-cfe1-4cd0-a895-8bc1771a720e | 2020-11-11T01:46:53.4905768Z | NaN | NaN | NaN | NaN | NaN | NaN |
3 | Microsoft.SecurityInsights/alertRules | Scheduled | Medium | AzureActivity\n| take 1\n| extend IPCustomEnti... | P1D | P14D | GreaterThan | 0.0 | PT5H | False | ... | This analytic matches Azure Activity logs to k... | [Impact] | None | 2020-11-04T22:43:33.9845152Z | NaN | NaN | NaN | NaN | NaN | NaN |
4 | Microsoft.SecurityInsights/alertRules | Scheduled | Medium | let timeframe = 1d;\nSecurityEvent\n| where Ti... | P1D | P1D | GreaterThan | 0.0 | PT5H | False | ... | Checks for event id 1102 which indicates the s... | [DefenseEvasion] | 80da0a8f-cfe1-4cd0-a895-8bc1771a720e | 2020-11-11T07:19:24.7658031Z | NaN | NaN | NaN | NaN | NaN | NaN |
5 rows × 29 columns
We can also get a list of saved bookmarks. To see the events these bookmarks relate to you can pass the query value to a QueryProvider.
bkmarks = azs.get_bookmarks(ws.value)
bkmarks.head().drop(columns=["id", "etag", "name"])
type | properties.displayName | properties.created | properties.updated | properties.createdBy.objectId | properties.createdBy.email | properties.createdBy.name | properties.updatedBy.objectId | properties.updatedBy.email | properties.updatedBy.name | ... | properties.labels | properties.query | properties.queryResult | properties.queryStartTime | properties.queryEndTime | properties.incidentInfo.incidentId | properties.incidentInfo.title | properties.incidentInfo.relationName | properties.incidentInfo.severity | properties.notes | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | Microsoft.SecurityInsights/Bookmarks | mercury IP | 2020-11-18T09:26:54.1605891+00:00 | 2020-11-18T09:26:54.1605891+00:00 | e0139aae-7811-40ca-abc6-3fcb79140a6b | Tim.Burrell@microsoft.com | Tim Burrell (MSTIC) | e0139aae-7811-40ca-abc6-3fcb79140a6b | Tim.Burrell@microsoft.com | Tim Burrell (MSTIC) | ... | [] | print "192.168.15.6" \n | {"print_0":"192.168.15.6","__entityMapping":{"... | 2020-11-17T09:26:33.557+00:00 | 2020-11-18T09:26:33.557+00:00 | None | None | None | None | NaN |
1 | Microsoft.SecurityInsights/Bookmarks | test 1 | 2020-11-18T15:25:01.1843361+00:00 | 2020-11-18T15:25:01.1843361+00:00 | b3a76793-1a0d-4bfe-95f6-96919d4b9acf | bnick@microsoft.com | Ben Nick | b3a76793-1a0d-4bfe-95f6-96919d4b9acf | bnick@microsoft.com | Ben Nick | ... | [fluffyDogCampaign] | let auditLookback = 14d;\n// Setting threshold... | {"InitiatedBy":"seb@seccxp.ninja","IpAddress":... | NaN | NaN | None | None | None | None | this looks suspicious |
2 | Microsoft.SecurityInsights/Bookmarks | failed logons - decb171c8160 (1) | 2020-11-19T11:26:31.3053573+00:00 | 2020-11-19T11:26:31.3053573+00:00 | 518a3ca6-44f0-4ac7-8179-97d18e48d65c | pascals@microsoft.com | Pascal Sauliere | 518a3ca6-44f0-4ac7-8179-97d18e48d65c | pascals@microsoft.com | Pascal Sauliere | ... | [] | // Event: An account failed to log on\nSecurit... | {"TenantId":"8ecf8077-cf51-4820-aadd-14040956f... | NaN | NaN | None | None | None | None | NaN |
3 | Microsoft.SecurityInsights/Bookmarks | Rare Audit activity initiated by App - cbade9... | 2020-11-11T18:39:16.6537628+00:00 | 2020-11-11T18:39:16.6537628+00:00 | f6b78447-93dc-4041-a22a-6eb1c34265e2 | Umesh.Nagdev@microsoft.com | Umesh Nagdev | f6b78447-93dc-4041-a22a-6eb1c34265e2 | Umesh.Nagdev@microsoft.com | Umesh Nagdev | ... | [] | let current = 1d;\nlet auditLookback = 14d;\nl... | {"InitiatedByApp":"Microsoft Azure AD Group-Ba... | 2020-11-10T18:39:01.061+00:00 | 2020-11-11T18:39:01.061+00:00 | None | None | None | None | NaN |
4 | Microsoft.SecurityInsights/Bookmarks | ThreatIntelligenceIndicator - 4193cb45b90a (2) | 2020-11-11T16:08:45.6964987+00:00 | 2020-11-11T16:08:45.6964987+00:00 | 525c09b5-61ef-4e10-8150-b44c97ead3a1 | Andrew.Blumhardt@microsoft.com | Andrew Blumhardt | 525c09b5-61ef-4e10-8150-b44c97ead3a1 | Andrew.Blumhardt@microsoft.com | Andrew Blumhardt | ... | [] | ThreatIntelligenceIndicator | {"TenantId":"8ecf8077-cf51-4820-aadd-14040956f... | 2020-11-10T16:08:26.089+00:00 | 2020-11-11T16:08:26.089+00:00 | None | None | None | None | My Bookmark |
5 rows × 21 columns
We can also interact with Incidents via the API to get a set of all incidents, or a single incident:
incidents = azs.get_incidents(res_id=ws.value)
display(incidents.head())
id | name | etag | type | properties.title | properties.severity | properties.status | properties.owner.objectId | properties.owner.email | properties.owner.assignedTo | ... | properties.additionalData.commentsCount | properties.additionalData.alertProductNames | properties.additionalData.tactics | properties.firstActivityTimeGenerated | properties.lastActivityTimeGenerated | properties.relatedAnalyticRuleIds | properties.incidentUrl | properties.description | properties.firstActivityTimeUtc | properties.lastActivityTimeUtc | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de... | aabf6bcd-4134-b07b-1152-040aa0cdf069 | "0402f99f-0000-0100-0000-5fbd43d50000" | Microsoft.SecurityInsights/Incidents | Time series anomaly detection for total volume... | High | New | None | None | None | ... | 2 | [Azure Sentinel] | [Exfiltration] | 2020-11-24T17:10:35.7652885Z | 2020-11-24T17:10:35.7652885Z | [/subscriptions/d1d8779d-38d7-4f06-91db-9cbc8d... | https://portal.azure.com/#asset/Microsoft_Azur... | NaN | NaN | NaN |
1 | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de... | e917efd0-331d-48b7-81d7-6205cee787f5 | "0302de84-0000-0100-0000-5fbd23f30000" | Microsoft.SecurityInsights/Incidents | XASE SENSITIVITY TEST | Medium | New | None | None | None | ... | 0 | [Azure Sentinel] | [] | 2020-11-24T15:17:06.8646498Z | 2020-11-24T15:17:06.8646498Z | [/subscriptions/d1d8779d-38d7-4f06-91db-9cbc8d... | https://portal.azure.com/#asset/Microsoft_Azur... | LOWER CASE | 2020-11-24T14:55:03.95Z | 2020-11-24T14:55:03.95Z |
2 | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de... | 81d59f6e-988f-4758-a2d2-90886befccab | "03029c83-0000-0100-0000-5fbd23d10000" | Microsoft.SecurityInsights/Incidents | Case Sensitivity test UPPER | Medium | New | None | None | None | ... | 0 | [Azure Sentinel] | [] | 2020-11-24T15:16:33.5131821Z | 2020-11-24T15:16:33.5131821Z | [/subscriptions/d1d8779d-38d7-4f06-91db-9cbc8d... | https://portal.azure.com/#asset/Microsoft_Azur... | sdff | 2020-11-24T14:57:49.43Z | 2020-11-24T14:57:49.43Z |
3 | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de... | b68151e7-890f-48aa-befb-3de2bc987557 | "03022274-0000-0100-0000-5fbd222f0000" | Microsoft.SecurityInsights/Incidents | Potential Password Spray | Medium | New | None | None | None | ... | 0 | [Azure Sentinel] | [Persistence] | 2020-11-24T15:09:35.0020779Z | 2020-11-24T15:09:35.0020779Z | [/subscriptions/d1d8779d-38d7-4f06-91db-9cbc8d... | https://portal.azure.com/#asset/Microsoft_Azur... | Description with a link | 2020-11-24T10:04:32.5297051Z | 2020-11-24T15:04:32.5297051Z |
4 | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de... | 49f91f51-30ce-4028-9117-96ca3debbe14 | "0302f05f-0000-0100-0000-5fbd204e0000" | Microsoft.SecurityInsights/Incidents | Case Sensitivity test UPPER | Medium | New | None | None | None | ... | 0 | [Azure Sentinel] | [] | 2020-11-24T15:01:33.9949456Z | 2020-11-24T15:01:33.9949456Z | [/subscriptions/d1d8779d-38d7-4f06-91db-9cbc8d... | https://portal.azure.com/#asset/Microsoft_Azur... | sdff | 2020-11-24T14:41:32.13Z | 2020-11-24T14:41:32.13Z |
5 rows × 27 columns
incident = azs.get_incident(incident_id = incidents.iloc[0]['name'] , res_id=ws.value)
display(incident)
id | name | etag | type | properties.title | properties.severity | properties.status | properties.owner.objectId | properties.owner.email | properties.owner.assignedTo | ... | properties.incidentNumber | properties.additionalData.alertsCount | properties.additionalData.bookmarksCount | properties.additionalData.commentsCount | properties.additionalData.alertProductNames | properties.additionalData.tactics | properties.firstActivityTimeGenerated | properties.lastActivityTimeGenerated | properties.relatedAnalyticRuleIds | properties.incidentUrl | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | /subscriptions/d1d8779d-38d7-4f06-91db-9cbc8de... | aabf6bcd-4134-b07b-1152-040aa0cdf069 | "0402f99f-0000-0100-0000-5fbd43d50000" | Microsoft.SecurityInsights/Incidents | Time series anomaly detection for total volume... | High | New | None | None | None | ... | 4601 | 1 | 0 | 2 | [Azure Sentinel] | [Exfiltration] | 2020-11-24T17:10:35.7652885Z | 2020-11-24T17:10:35.7652885Z | [/subscriptions/d1d8779d-38d7-4f06-91db-9cbc8d... | https://portal.azure.com/#asset/Microsoft_Azur... |
1 rows × 24 columns
You can also interact with an incident - adding comments or changing properties such as severity or status:
azs.post_comment(incident_id = incident.iloc[0]['name'], comment="This is a test comment", res_id=ws.value)
Comment posted.
azs.update_incident(incident_id = incident.iloc[0]['name'], update_items={"severity":"High"}, res_id=ws.value)
Incident updated.