This notebook demonstrates the use of the process tree data and visualization modules. These modules can be used with either Windows process creation events (ID 4688) or Linux auditd logs.
You must have msticpy installed to run this notebook:
!pip install --upgrade msticpy
There are two main components:
Note The expected schema for the Linux audit data is as produced by the auditdextract.py
module in msticpy
. This module combines related process exec messages into a single combined message that emulates the Windows 4688 event. This retains the audit schema apart from the following additions:
cmdline
: this is a concatenation of the a0
, a1
, etc argument fieldsEventType
: this is the audit message type (SYSCALL
, EXECVE
, CWD
, etc.) - the combined SYSCALL_EXECVE
created by auditextract
is the only type currently supported.Support for other formats such as Sysmon, Microsoft Defender is planned but not yet included.
The input can be either Windows 4688 events or Linux audit events (with the above caveats).
Import libraries and read in test data. Then call build_process_tree
to extract the parent-child relationships between processes.
from IPython.display import display
import pandas as pd
from msticpy.nbtools import *
from msticpy.sectools import *
win_procs = pd.read_pickle("./data/win_proc_test.pkl")
display(win_procs.head())
TenantId | Account | EventID | TimeGenerated | Computer | SubjectUserSid | SubjectUserName | SubjectDomainName | SubjectLogonId | NewProcessId | NewProcessName | TokenElevationType | ProcessId | CommandLine | ParentProcessName | TargetLogonId | SourceComputerId | TimeCreatedUtc | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 15:21:06.890 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0xd78 | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | %%1936 | 0x1994 | "CollectGuestLogs.exe" -Mode:ga -FileName:C:\W... | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | 0x0 | 263a788b-6526-4cdc-8ed9-d79402fe4aa0 | 2019-02-10 15:21:06.890 |
1 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 15:21:06.907 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x221c | C:\Windows\System32\conhost.exe | %%1936 | 0xd78 | \??\C:\Windows\system32\conhost.exe 0xffffffff... | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | 0x0 | 263a788b-6526-4cdc-8ed9-d79402fe4aa0 | 2019-02-10 15:21:06.907 |
2 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 14:15:36.253 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x638 | C:\Program Files\Microsoft Monitoring Agent\Ag... | %%1936 | 0xe24 | "C:\Program Files\Microsoft Monitoring Agent\A... | C:\Program Files\Microsoft Monitoring Agent\Ag... | 0x0 | 263a788b-6526-4cdc-8ed9-d79402fe4aa0 | 2019-02-10 14:15:36.253 |
3 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 14:15:36.270 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x115c | C:\Windows\System32\conhost.exe | %%1936 | 0x638 | \??\C:\Windows\system32\conhost.exe 0xffffffff... | C:\Program Files\Microsoft Monitoring Agent\Ag... | 0x0 | 263a788b-6526-4cdc-8ed9-d79402fe4aa0 | 2019-02-10 14:15:36.270 |
4 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 15:48:18.437 | MSTICAlertsWin1 | S-1-5-20 | MSTICAlertsWin1$ | WORKGROUP | 0x3e4 | 0x2364 | C:\Windows\System32\reg.exe | %%1936 | 0x1c24 | reg "C:\diagnostics\WinBenignActivity.cmd" -2... | C:\Windows\System32\cmd.exe | 0x0 | 263a788b-6526-4cdc-8ed9-d79402fe4aa0 | 2019-02-10 15:48:18.437 |
p_tree_win = ptree.build_process_tree(win_procs, show_progress=True)
HBox(children=(IntProgress(value=0, bar_style='info', description='Progress:'), Label(value='0%')))
{'Processes': 1010, 'RootProcesses': 10, 'LeafProcesses': 815, 'BranchProcesses': 185, 'IsolatedProcesses': 0, 'LargestTreeDepth': 7}
The module is imported as follows:
from msticpy.sectools import *
or explicitly
from msticpy.sectools import process_tree_utils as ptree
The module contains functions for building the parent-child relations as well as a number of utility functions for manipulating and extracting the trees. Most of these are described in the later section Process Tree utility functions.
proc_tree = ptree.get_descendents(p_tree_win, ptree.get_roots(p_tree_win).iloc[2])
nbdisplay.plot_process_tree(data=proc_tree, legend_col="SubjectUserName", show_table=True)
nbdisplay.plot_process_tree( data, schema=None, output_var=None, legend_colNone, show_table=False, )
build_process_tree
described above.process_name
) on to a data-specific name (e.g. exe
in the case of Linux audit data). This is usually not required since the function will try to infer the schema from fields in the input DataFrame.my_results
will be populated with a list of keys (index items) of the corresponding rows in the input DataFrame.These will normally be handled well by the Bokeh plot (up to multiple tens of thousands or more) but it will make navigation of the tree difficult. In particular, the range tool (on the right of the main plot) will be difficult to manipulate. Split the input data into smaller chunks before plotting.
The font size does not scale based on how much data is shown. If you use the range tool to select too large a subset of the data in the main plot, the font will become unreadable. If this happens, use the reset
tool to set the plot back to its defaults.
Note This assumes that the Linux audit log has been read from a file using msticpy.sectools.auditdextract.read_from_file() or read from Azure Sentinel/Log Analytics using the LinuxAudit.auditd_all query and processed using [msticpy.sectools.auditdextract.extract_events_to_df()`](https://msticpy.readthedocs.io/en/latest/msticpy.sectools.html#msticpy.sectools.auditdextract.extract_events_to_df) function.
Using either of these, the process exec events related to a single process start are merged into a single row.
linux_proc = pd.read_pickle("data/linux_proc_test.pkl")
p_tree_lx = ptree.build_process_tree(linux_proc, show_progress=True, debug=True)
HBox(children=(IntProgress(value=0, bar_style='info', description='Progress:'), Label(value='0%')))
Original # procs 1000 Merged # procs 1000 Merged # procs - dropna 295 Unique merged_procs index in merge 1000 These two should add up to top line Rows with dups 0 Rows with no dups 1000 0 + 1000 = 1000 original: 1000 inferred_parents 29 combined 1029 has parent time 1000 effectivelogonId in subjectlogonId 1029 parent_proc_lc in procs 1000 ProcessId in ParentProcessId 532 Parent_key in proc_key 1000 Parent_key not in proc_key 29 Parent_key is NA 29 {'Processes': 1029, 'RootProcesses': 29, 'LeafProcesses': 497, 'BranchProcesses': 503, 'IsolatedProcesses': 0, 'LargestTreeDepth': 5}
# Take one of those roots and get the full tree beneath it
t_root = ptree.get_roots(p_tree_lx).iloc[2]
full_tree = ptree.get_descendents(p_tree_lx, t_root)
print("Full tree size:", len(full_tree))
full_tree.head()
Full tree size: 64
EventID | TenantId | Computer | mssg_id | TimeGenerated | a0 | a1 | a2 | argc | auid | ... | TimeGenerated_orig_par | NewProcessId_par | ParentProcessName | parent_proc_lc | parent_key | IsRoot | IsLeaf | IsBranch | path | parent_index | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
proc_key | |||||||||||||||||||||
unknown13501970-01-01 00:00:00.000000 | SYSCALL_EXECVE | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | MSTICAlertsLxVM2 | NaN | 1970-01-01 00:00:00.000 | NaN | NaN | NaN | NaN | NaN | ... | NaT | NaN | NaN | NaN | NaN | True | False | False | 1002 | NaN |
/usr/bin/sudo263732019-02-17 22:01:26.357000 | SYSCALL_EXECVE | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | MSTICAlertsLxVM2 | 1550440886.357:8972854 | 2019-02-17 22:01:26.357 | sudo | /opt/microsoft/omsagent/ruby/bin/ruby | /opt/microsoft/omsagent/plugin/tailfilereader.rb | 6 | -1 | ... | 1970-01-01 00:00:00.000 | 1350 | unknown | unknown | unknown13501970-01-01 00:00:00.000000 | False | False | True | 1002/159 | 1002 |
/opt/microsoft/omsagent/ruby/bin/ruby263752019-02-17 22:01:26.369000 | SYSCALL_EXECVE | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | MSTICAlertsLxVM2 | 1550440886.369:8972858 | 2019-02-17 22:01:26.369 | /opt/microsoft/omsagent/ruby/bin/ruby | /opt/microsoft/omsagent/plugin/tailfilereader.rb | /var/log/audit/audit.log | 5 | -1 | ... | 2019-02-17 22:01:26.357 | NaN | /usr/bin/sudo | /usr/bin/sudo | /usr/bin/sudo263732019-02-17 22:01:26.357000 | False | False | True | 1002/159/160 | 159 |
/usr/bin/sudo263772019-02-17 22:01:26.449000 | SYSCALL_EXECVE | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | MSTICAlertsLxVM2 | 1550440886.449:8972859 | 2019-02-17 22:01:26.449 | sudo | test | -f | 4 | -1 | ... | 2019-02-17 22:01:26.369 | NaN | /opt/microsoft/omsagent/ruby/bin/ruby | /opt/microsoft/omsagent/ruby/bin/ruby | /opt/microsoft/omsagent/ruby/bin/ruby263752019... | False | False | True | 1002/159/160/161 | 160 |
/usr/bin/test263782019-02-17 22:01:26.465000 | SYSCALL_EXECVE | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | MSTICAlertsLxVM2 | 1550440886.465:8972863 | 2019-02-17 22:01:26.465 | test | -f | /var/log/audit/audit.log | 3 | -1 | ... | 2019-02-17 22:01:26.449 | NaN | /usr/bin/sudo | /usr/bin/sudo | /usr/bin/sudo263772019-02-17 22:01:26.449000 | False | True | False | 1002/159/160/161/162 | 161 |
5 rows × 41 columns
nbdisplay.plot_process_tree(data=full_tree[:1000], legend_col="cwd")
proc_rarity = pd.read_pickle("./data/procs_with_cluster.pkl")
proc_rarity_tree = ptree.build_process_tree(proc_rarity, show_progress=True)
HBox(children=(IntProgress(value=0, bar_style='info', description='Progress:'), Label(value='0%')))
{'Processes': 22992, 'RootProcesses': 31, 'LeafProcesses': 15587, 'BranchProcesses': 7374, 'IsolatedProcesses': 0, 'LargestTreeDepth': 839}
prar_roots = ptree.get_roots(proc_rarity_tree)
# Find the tree with the highest Rarity Score
tree_rarity = []
for row_num, (ix, row) in enumerate(prar_roots.iterrows()):
rarity_tree = ptree.get_descendents(proc_rarity_tree, row)
tree_rarity.append({
"Row": row_num,
"RootProcess": prar_roots.loc[ix].NewProcessName,
"TreeSize:": len(rarity_tree),
"AverageRarity": rarity_tree["Rarity"].mean()
})
pd.DataFrame(tree_rarity).sort_values("AverageRarity", ascending=False)
Row | RootProcess | TreeSize: | AverageRarity | |
---|---|---|---|---|
27 | 27 | C:\Windows\System32\svchost.exe | 4 | 0.666726 |
23 | 23 | C:\Windows\System32\svchost.exe | 2 | 0.500000 |
22 | 22 | C:\Windows\System32\smss.exe | 30 | 0.398288 |
20 | 20 | C:\Windows\SoftwareDistribution\Download\Insta... | 2 | 0.333333 |
9 | 9 | C:\Windows\System32\smss.exe | 7 | 0.250000 |
7 | 7 | C:\ProgramData\Microsoft\Windows Defender\plat... | 46 | 0.190123 |
10 | 10 | C:\Windows\System32\winlogon.exe | 2 | 0.166667 |
5 | 5 | C:\Windows\System32\svchost.exe | 21 | 0.146667 |
18 | 18 | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | 3 | 0.125089 |
13 | 13 | C:\Windows\Microsoft.NET\Framework64\v4.0.3031... | 2 | 0.125000 |
29 | 29 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 4 | 0.095297 |
25 | 25 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 4 | 0.095297 |
21 | 21 | C:\ProgramData\Microsoft\Windows Defender\plat... | 21 | 0.059754 |
16 | 16 | C:\Windows\System32\services.exe | 320 | 0.056118 |
15 | 15 | C:\Windows\System32\svchost.exe | 1381 | 0.038497 |
2 | 2 | C:\Windows\System32\svchost.exe | 2192 | 0.025462 |
4 | 4 | C:\Windows\System32\services.exe | 105 | 0.025112 |
3 | 3 | C:\Windows\System32\svchost.exe | 689 | 0.019825 |
0 | 0 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 13 | 0.012703 |
11 | 11 | unknown | 8936 | 0.011424 |
1 | 1 | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | 259 | 0.007365 |
19 | 19 | C:\Windows\System32\svchost.exe | 4949 | 0.007174 |
17 | 17 | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | 11 | 0.007138 |
14 | 14 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 274 | 0.003676 |
8 | 8 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 3703 | 0.000194 |
12 | 12 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 2 | 0.000178 |
24 | 24 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 2 | 0.000178 |
26 | 26 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 2 | 0.000178 |
28 | 28 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 2 | 0.000178 |
6 | 6 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 2 | 0.000178 |
30 | 30 | C:\Program Files\Microsoft Monitoring Agent\Ag... | 2 | 0.000178 |
svcs_tree = ptree.get_descendents(proc_rarity_tree, prar_roots.iloc[22])
nbdisplay.plot_process_tree(svcs_tree, legend_col="Rarity", show_table=True)
The process_tree_utils
module has a number of functions that may be useful in extracting or manipulating process trees or tree relationships.
Get summary information.
ptree.get_summary_info(p_tree_win)
{'Processes': 1010, 'RootProcesses': 10, 'LeafProcesses': 815, 'BranchProcesses': 185, 'IsolatedProcesses': 0, 'LargestTreeDepth': 7}
Get roots of all trees in the data set.
# Get roots of all trees in the set
ptree.get_roots(p_tree_win).head()
TenantId | Account | EventID | TimeGenerated | Computer | SubjectUserSid | SubjectUserName | SubjectDomainName | SubjectLogonId | NewProcessId | ... | source_index_par | ProcessId_par | NewProcessName_par | TimeGenerated_orig_par | parent_key | IsRoot | IsLeaf | IsBranch | path | parent_index | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
proc_key | |||||||||||||||||||||
c:\windows\system32\svchost.exe0x2701970-01-01 00:00:00.000000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | NaN | 4688 | 1970-01-01 | MSTICAlertsWin1 | NaN | NaN | NaN | 0x3e7 | 0x270 | ... | NaN | NaN | NaN | NaT | NaN | True | False | False | 1000 | NaN |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x7981970-01-01 00:00:00.000000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | NaN | 4688 | 1970-01-01 | MSTICAlertsWin1 | NaN | NaN | NaN | 0x3e7 | 0x798 | ... | NaN | NaN | NaN | NaT | NaN | True | False | False | 1001 | NaN |
c:\windows\system32\svchost.exe0xb3c1970-01-01 00:00:00.000000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | NaN | 4688 | 1970-01-01 | MSTICAlertsWin1 | NaN | NaN | NaN | 0x3e7 | 0xb3c | ... | NaN | NaN | NaN | NaT | NaN | True | False | False | 1002 | NaN |
c:\windows\system32\services.exe0x2201970-01-01 00:00:00.000000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | NaN | 4688 | 1970-01-01 | MSTICAlertsWin1 | NaN | NaN | NaN | 0x3e7 | 0x220 | ... | NaN | NaN | NaN | NaT | NaN | True | False | False | 1003 | NaN |
c:\program files\microsoft monitoring agent\agent\monitoringhost.exe0x8641970-01-01 00:00:00.000000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | NaN | 4688 | 1970-01-01 | MSTICAlertsWin1 | NaN | NaN | NaN | 0x3e7 | 0x864 | ... | NaN | NaN | NaN | NaT | NaN | True | False | False | 1004 | NaN |
5 rows × 35 columns
Get the full tree beneath a process.
# Take one of those roots and get the full tree beneath it
t_root = ptree.get_roots(p_tree_win).loc["c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\waappagent.exe0x19941970-01-01 00:00:00.000000"]
full_tree = ptree.get_descendents(p_tree_win, t_root)
full_tree.head()
TenantId | Account | EventID | TimeGenerated | Computer | SubjectUserSid | SubjectUserName | SubjectDomainName | SubjectLogonId | NewProcessId | ... | source_index_par | ProcessId_par | NewProcessName_par | TimeGenerated_orig_par | parent_key | IsRoot | IsLeaf | IsBranch | path | parent_index | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
proc_key | |||||||||||||||||||||
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\waappagent.exe0x19941970-01-01 00:00:00.000000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | NaN | 4688 | 1970-01-01 00:00:00.000 | MSTICAlertsWin1 | NaN | NaN | NaN | 0x3e7 | 0x1994 | ... | NaN | NaN | NaN | NaT | NaN | True | False | False | 1007 | NaN |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x123c2019-02-09 23:16:28.153000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:16:28.153 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x123c | ... | NaN | NaN | NaN | 1970-01-01 00:00:00.000 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/103 | 1007 |
c:\windows\system32\conhost.exe0x20b02019-02-09 23:16:28.163000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:16:28.163 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x20b0 | ... | 986.0 | 0x1994 | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | 2019-02-09 23:16:28.153 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | True | False | 1007/103/104 | 103 |
c:\windows\system32\cmd.exe0xccc2019-02-09 23:20:15.547000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:20:15.547 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0xccc | ... | 986.0 | 0x1994 | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | 2019-02-09 23:16:28.153 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/103/105 | 103 |
c:\windows\system32\conhost.exe0x14ec2019-02-09 23:20:15.560000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:20:15.560 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x14ec | ... | 988.0 | 0x123c | C:\Windows\System32\cmd.exe | 2019-02-09 23:20:15.547 | c:\windows\system32\cmd.exe0xccc2019-02-09 23:... | False | True | False | 1007/103/105/106 | 105 |
5 rows × 35 columns
Get the immediate children of a process
# Just get the immediate children of the root process
children = ptree.get_children(p_tree_win, t_root)
children.head()
TenantId | Account | EventID | TimeGenerated | Computer | SubjectUserSid | SubjectUserName | SubjectDomainName | SubjectLogonId | NewProcessId | ... | source_index_par | ProcessId_par | NewProcessName_par | TimeGenerated_orig_par | parent_key | IsRoot | IsLeaf | IsBranch | path | parent_index | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
proc_key | |||||||||||||||||||||
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x1c542019-02-09 22:11:59.877000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 22:11:59.877 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x1c54 | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/69 | 1007 |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x123c2019-02-09 23:16:28.153000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:16:28.153 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x123c | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/103 | 1007 |
c:\windowsazure\secagent\wasecagentprov.exe0xda82019-02-09 23:55:46.057000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:55:46.057 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0xda8 | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/123 | 1007 |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x7fc2019-02-10 00:22:33.813000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 00:22:33.813 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x7fc | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/134 | 1007 |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x172c2019-02-10 01:27:25.173000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 01:27:25.173 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x172c | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/343 | 1007 |
5 rows × 35 columns
Get the depth of a tree.
# Get the depth of the full tree
depth = ptree.get_tree_depth(full_tree)
print(f"depth of tree is {depth}")
depth of tree is 4
Get the parent process or all ancestors.
# Get Ancestors
# Get a child process that's at the bottom of the tree
btm_descnt = full_tree[full_tree["path"].str.count("/") == depth - 1].iloc[0]
print("parent")
display(ptree.get_parent(p_tree_win, btm_descnt)[:20])
print("ancestors")
ptree.get_ancestors(p_tree_win, btm_descnt).head()
parent
TenantId 52b1ab41-869e-4138-9e40-2a4457f09bf0 Account WORKGROUP\MSTICAlertsWin1$ EventID 4688 TimeGenerated 2019-02-09 23:20:15.547000 Computer MSTICAlertsWin1 SubjectUserSid S-1-5-18 SubjectUserName MSTICAlertsWin1$ SubjectDomainName WORKGROUP SubjectLogonId 0x3e7 NewProcessId 0xccc NewProcessName C:\Windows\System32\cmd.exe TokenElevationType %%1936 ProcessId 0x123c CommandLine "cmd" ParentProcessName C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... TargetLogonId 0x0 SourceComputerId 263a788b-6526-4cdc-8ed9-d79402fe4aa0 TimeCreatedUtc 2019-02-09 23:20:15.547000 EffectiveLogonId 0x3e7 new_process_lc c:\windows\system32\cmd.exe Name: c:\windows\system32\cmd.exe0xccc2019-02-09 23:20:15.547000, dtype: object
ancestors
TenantId | Account | EventID | TimeGenerated | Computer | SubjectUserSid | SubjectUserName | SubjectDomainName | SubjectLogonId | NewProcessId | ... | source_index_par | ProcessId_par | NewProcessName_par | TimeGenerated_orig_par | parent_key | IsRoot | IsLeaf | IsBranch | path | parent_index | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
proc_key | |||||||||||||||||||||
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\waappagent.exe0x19941970-01-01 00:00:00.000000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | NaN | 4688 | 1970-01-01 00:00:00.000 | MSTICAlertsWin1 | NaN | NaN | NaN | 0x3e7 | 0x1994 | ... | NaN | NaN | NaN | NaT | NaN | True | False | False | 1007 | NaN |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x123c2019-02-09 23:16:28.153000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:16:28.153 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x123c | ... | NaN | NaN | NaN | 1970-01-01 00:00:00.000 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/103 | 1007 |
c:\windows\system32\cmd.exe0xccc2019-02-09 23:20:15.547000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:20:15.547 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0xccc | ... | 986.0 | 0x1994 | C:\WindowsAzure\GuestAgent_2.7.41491.901_2019-... | 2019-02-09 23:16:28.153 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/103/105 | 103 |
c:\windows\system32\conhost.exe0x14ec2019-02-09 23:20:15.560000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:20:15.560 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x14ec | ... | 988.0 | 0x123c | C:\Windows\System32\cmd.exe | 2019-02-09 23:20:15.547 | c:\windows\system32\cmd.exe0xccc2019-02-09 23:... | False | True | False | 1007/103/105/106 | 105 |
4 rows × 35 columns
Get a process record by its key. Build a key from a process object (pandas Series).
proc_key = btm_descnt.name
print(proc_key)
ptree.get_process(p_tree_win, proc_key)
process2 = full_tree[full_tree["path"].str.count("/") == depth - 1].iloc[-1]
ptree.build_process_key(process2)
c:\windows\system32\conhost.exe0x14ec2019-02-09 23:20:15.560000
'c:\\windows\\system32\\conhost.exe0x15842019-02-10 15:24:56.050000'
Get the siblings of a process.
Some functions take an include_source
parameter. Setting this to True returns the source process with the result set.
src_proc = ptree.get_children(p_tree_win, t_root, include_source=False).iloc[0]
ptree.get_siblings(p_tree_win, src_proc, include_source=True).head()
TenantId | Account | EventID | TimeGenerated | Computer | SubjectUserSid | SubjectUserName | SubjectDomainName | SubjectLogonId | NewProcessId | ... | source_index_par | ProcessId_par | NewProcessName_par | TimeGenerated_orig_par | parent_key | IsRoot | IsLeaf | IsBranch | path | parent_index | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
proc_key | |||||||||||||||||||||
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x1c542019-02-09 22:11:59.877000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 22:11:59.877 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x1c54 | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/69 | 1007 |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x123c2019-02-09 23:16:28.153000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:16:28.153 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x123c | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/103 | 1007 |
c:\windowsazure\secagent\wasecagentprov.exe0xda82019-02-09 23:55:46.057000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-09 23:55:46.057 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0xda8 | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/123 | 1007 |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x7fc2019-02-10 00:22:33.813000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 00:22:33.813 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x7fc | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/134 | 1007 |
c:\windowsazure\guestagent_2.7.41491.901_2019-01-14_202614\collectguestlogs.exe0x172c2019-02-10 01:27:25.173000 | 52b1ab41-869e-4138-9e40-2a4457f09bf0 | WORKGROUP\MSTICAlertsWin1$ | 4688 | 2019-02-10 01:27:25.173 | MSTICAlertsWin1 | S-1-5-18 | MSTICAlertsWin1$ | WORKGROUP | 0x3e7 | 0x172c | ... | NaN | NaN | NaN | 1970-01-01 | c:\windowsazure\guestagent_2.7.41491.901_2019-... | False | False | True | 1007/343 | 1007 |
5 rows × 35 columns
import networkx as nx
import matplotlib.pyplot as plt
p_graph = nx.DiGraph()
p_graph = nx.from_pandas_edgelist(
df=full_tree.reset_index(),
source="parent_key",
target="proc_key",
edge_attr=["TimeGenerated", "NewProcessName", "NewProcessId"],
create_using=nx.DiGraph,
)
plt.gcf().set_size_inches((20,20))
pos = nx.circular_layout(p_graph)
nx.draw_networkx(p_graph, pos=pos, with_labels=False, node_size=50)
# Get the root binary name to plot labels (change the split param for Linux)
labels = full_tree.apply(lambda x: x.NewProcessName.split("\\")[-1], axis=1).to_dict()
nx.draw_networkx_labels(p_graph, pos, labels=labels, font_size=10, font_color='k', font_family='sans-serif', font_weight='normal', alpha=1.0)
plt.show()