This notebook provides a guided example of using the Mordor data provider and browser included with MSTICpy.
For more information on the Mordor data sets see the Open Threat Research Forge Mordor GitHub repo
You must have msticpy installed to run this notebook:
%pip install --upgrade msticpy
MSTICpy versions >= 0.8.5
Using the data provider you can download and render event data as a pandas DataFrame.
Note - Mordor includes both host event data and network capture data.
Although Capture files can be downloaded and unpacked
they currently cannot be populated into a pandas DataFrame. This is the case for mostnetwork
datasets.
Host
event data is retrieved and populated into DataFrames.
from msticpy.data import QueryProvider
CACHE_FOLDER = "~/.msticpy/mordor"
mdr_data = QueryProvider("Mordor", save_folder=CACHE_FOLDER)
mdr_data.connect()
Retrieving Mitre data... Retrieving Mordor data...
Note: Many Mordor data entries have multiple data sets, so we see more queries than Mordor entries.
(Only first 15 shown)
mdr_data.list_queries()[:15]
['atomic.aws.collection.ec2_proxy_s3_exfiltration', 'atomic.linux.defense_evasion.host.sh_binary_padding_dd', 'atomic.linux.discovery.host.sh_arp_cache', 'atomic.linux.initial_access.network.log4jshell_reversheshell_netcat', 'atomic.windows.collection.host.msf_record_mic', 'atomic.windows.credential_access.host.cmd_lsass_memory_dumpert_syscalls', 'atomic.windows.credential_access.host.cmd_psexec_lsa_secrets_dump', 'atomic.windows.credential_access.host.cmd_sam_copy_esentutl', 'atomic.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges', 'atomic.windows.credential_access.host.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges', 'atomic.windows.credential_access.host.empire_mimikatz_backupkeys_dcerpc_smb_lsarpc', 'atomic.windows.credential_access.host.empire_mimikatz_extract_keys', 'atomic.windows.credential_access.host.empire_mimikatz_logonpasswords', 'atomic.windows.credential_access.host.empire_mimikatz_lsadump_patch', 'atomic.windows.credential_access.host.empire_mimikatz_sam_access']
mdr_data.atomic.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges().head(3)
https://raw.githubusercontent.com/OTRF/Security-Datasets/master/datasets/atomic/windows/credential_access/host/covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges.zip Extracting covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges_2020-08-05020926.json
@version | Keywords | ThreadID | Version | DestAddress | host | LayerRTID | Message | SourceModuleName | SourceName | ... | Properties | OperationType | QueryName | QueryResults | QueryStatus | PipeName | DisabledPrivilegeList | EnabledPrivilegeList | ShareLocalPath | RelativeTargetName | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1 | -9214364837600034816 | 4888 | 1 | 239.255.255.250 | wec.internal.cloudapp.net | 44.0 | The Windows Filtering Platform has permitted a... | eventlog | Microsoft-Windows-Security-Auditing | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
1 | 1 | -9223372036854775808 | 4452 | 2 | NaN | wec.internal.cloudapp.net | NaN | File created:\r\nRuleName: -\r\nUtcTime: 2020-... | eventlog | Microsoft-Windows-Sysmon | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
2 | 1 | -9223372036854775808 | 4452 | 2 | NaN | wec.internal.cloudapp.net | NaN | RawAccessRead detected:\r\nRuleName: -\r\nUtcT... | eventlog | Microsoft-Windows-Sysmon | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
3 rows × 145 columns
The data provider and the query functions support some parameters to control aspects of the query operation.
If you specify these when you initialize the data provider, the settings will apply to all queries.
mdr_data = QueryProvider("Mordor", save_folder="./mordor")
mdr_data.connect()
Retrieving Mitre data... Retrieving Mordor data...
Using these parameters in the query will override the provider settings and defaults for that query.
mdr_data.atomic.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges(silent=True, save_folder="./mordor").head(2)
@version | Keywords | ThreadID | Version | DestAddress | host | LayerRTID | Message | SourceModuleName | SourceName | ... | Properties | OperationType | QueryName | QueryResults | QueryStatus | PipeName | DisabledPrivilegeList | EnabledPrivilegeList | ShareLocalPath | RelativeTargetName | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1 | -9214364837600034816 | 4888 | 1 | 239.255.255.250 | wec.internal.cloudapp.net | 44.0 | The Windows Filtering Platform has permitted a... | eventlog | Microsoft-Windows-Security-Auditing | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
1 | 1 | -9223372036854775808 | 4452 | 2 | NaN | wec.internal.cloudapp.net | NaN | File created:\r\nRuleName: -\r\nUtcTime: 2020-... | eventlog | Microsoft-Windows-Sysmon | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
2 rows × 145 columns
!dir mordor
Volume in drive E has no label. Volume Serial Number is 7E50-19F7 Directory of e:\src\microsoft\msticpy\docs\notebooks\mordor 01/06/2022 05:15 PM <DIR> . 01/06/2022 05:15 PM <DIR> .. 01/06/2022 05:15 PM 76,924 atomic-windows-credential_access-host-covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges.zip 01/06/2022 05:15 PM 1,651,230 covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges_2020-08-05020926.json 2 File(s) 1,728,154 bytes 2 Dir(s) 227,291,512,832 bytes free
Call the query function with a single "?" parameter.
mdr_data.atomic.windows.credential_access.host.covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges("?")
Query: covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges Data source: Mordor Covenant DCSync Notes ----- Mordor ID: SDWIN-200805020926 This dataset represents adversaries abusing Active Directory Replication services to retrieve secret domain data (i.e. NTLM hashes) from domain accounts. Mitre Techniques: T1003: OS Credential Dumping Mitre Tactics: TA0006: Credential Access Parameters ---------- Query: https://raw.githubusercontent.com/OTRF/Security-Datasets/master/datasets/atomic/windows/credential_access/host/covenant_dcsync_dcerpc_drsuapi_DsGetNCChanges.zip
Search queries for matching attributes.
search : str Search string.
Substrings separated by commas will be treated as OR terms - e.g. "a, b" == "a" or "b".
Substrings separated by "+" will be treated as AND terms - e.g. "a + b" == "a" and "b"
List of matching query names.
mdr_data.search_queries("AWS")
['atomic.aws.collection.ec2_proxy_s3_exfiltration (AWS Cloud Bank Breach S3)']
mdr_data.search_queries("Empire + T1222")
['atomic.windows.defense_evasion.host.empire_powerview_ldap_ntsecuritydescriptor (Empire Powerview Add-DomainObjectAcl)', 'atomic.windows.defense_evasion.network.empire_powerview_ldap_ntsecuritydescriptor (Empire Powerview Add-DomainObjectAcl)']
mdr_data.search_queries("Empire + Credential")
['atomic.windows.credential_access.host.empire_mimikatz_sam_access (Empire Mimikatz SAM Extract Hashes)', 'atomic.windows.defense_evasion.host.empire_wdigest_downgrade.tar (Empire WDigest Downgrade)', 'atomic.windows.credential_access.host.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges (Empire DCSync)', 'atomic.windows.credential_access.network.empire_dcsync_dcerpc_drsuapi_DsGetNCChanges (Empire DCSync)', 'atomic.windows.credential_access.host.empire_mimikatz_lsadump_patch (Empire Mimikatz Lsadump LSA Patch)', 'atomic.windows.credential_access.host.empire_mimikatz_logonpasswords (Empire Mimikatz LogonPasswords)']
We've also built a more specialized browser for Mordor data. This uses the metadata in the repository to let you view full details of the dataset.
You can also preview the dataset (if it is convertible to a DataFrame).
For details of the data shown please see the Mordor GitHub repo
and the Threat Hunter Playbook
from msticpy.vis.mordor_browser import MordorBrowser
mdr_browser = MordorBrowser()
Retrieving Mitre data... Retrieving Mordor data...
VBox(children=(VBox(children=(HTML(value='<h2>Mordor dataset browser</h2>'), Select(description='Data sets', l…
The top scrollable list is a list of the Mordor datasets. Selecting one of these updates the data in the lower half of the browser.
To narrow your search you can filter using a text search or filter by Mitre Attack Techniques or Tactics.
search_queries()
function.This section allows you to select, download and (in most cases) display the event data relating to the attack.
Select a file and click on the Download button.
The zipped file is downloaded and extracted. If it is event data, this is converted to a pandas DataFrame and displayed below the rest of the data.
The current dataset is available as an attribute of the browser:
mdr_browser.current_dataset
Datasets that you've downloaded and displayed in this session are also cached in the browser and available in the
mdr_browser.datasets
attribute.
By default files are downloaded and extracted to the current folder. You can change this with the
save_folder
parameter when creating the MordorBrowser
object.
You can also specify the use_cached
parameter. By default, this is True
, which causes downloaded files not
to be deleted after extraction. These local copies are used if you try to view the same data set again.
This also works across sessions.
If use_cache
is set to False, files are deleted immediately after downloading, extracting and populating the
DataFrame.
Note - In the
Example
section, ignore the examples of parameters
passed to the query - these are not needed and ignored.
mdr_data.browse_queries()
VBox(children=(Text(value='', description='Filter:', style=DescriptionStyle(description_width='initial')), Sel…
Parameters
Query
https://raw.githubusercontent.com/OTRF/Security-Datasets/master/datase ts/atomic/aws/collection/ec2_proxy_s3_exfiltration.zip
Example
{QueryProvider}[.QueryPath].QueryName(params...)
qry_prov.atomic.aws.collection.ec2_proxy_s3_exfiltration(start=start, end=end, hostname=host)