This notebook takes you through setting up your MSTICPy configuration for the first time. Some sections are specific to using MSTICPy with Azure Sentinel.
You must have msticpy installed to run this notebook:
%pip install --upgrade msticpy
MSTICpy versions >= 1.0.0
from msticpy.config import MpConfigFile, MpConfigEdit
You can use MpConfigFile from the command line or as an interactive UI to do this.
mpconfig = MpConfigFile()
mpconfig.load_default()
mpconfig.view_settings()
VBox(children=(Textarea(value="{'Azure': {'auth_methods': ['cli', 'interactive'], 'cloud': 'global'},\n 'Azure…
...but a pair of curly braces in the settings view above it means that you should set up a msticpyconfig.yaml
You can search for this file using MpConfigFile. Click on Load file.
Once you've done that go to the Setting the path to your msticpyconfig.yaml
Follow these steps:
mpconfig
VBox(children=(HTML(value='<h3>MSTICPy settings</h3>'), HBox(children=(VBox(children=(Text(value='C:\\Users\\I…
mpconfig = MpConfigFile()
mpconfig.load_from_file("E:\\src\\asi-jupyter\\notebooks\\config.json")
mpconfig.map_json_to_mp_ws()
mpconfig.settings
If you loaded a config.json file you should see your workspace displayed. If not, you can add one or more workspaces here. The Name, WorkspaceId and TenantId are mandatory. The other fields are helpful but not essential.
Use the Help drop-down panel to find more information about adding workspaces and finding the correct values for your workspace.
If this the workspace that you use frequently or all of the time, you may want to set this as the default. This creates a duplicate entry named "Default" and this is used when you connect to AzureSentinel as the default workspace to connect to (you can override this by specifying a workspace name at connect time).
When you've finished, type a file name (usually "msticpyconfig.yaml") into the Conf File text box and click Save File,
You can also try the Validate Settings button. This should show that you have a few missing sections (we'll fill these in later) but should show nothing under the the "Type Validation Results".
mpedit = MpConfigEdit(settings=mpconfig)
mpedit
Label(value='Loading. Please wait.')
VBox(children=(Tab(children=(VBox(children=(Label(value='Microsoft Sentinel workspace settings'), HBox(childre…
This is a good point to set up an environment variable so that you can keep a single configuration file in a known location and always load the same settings. (Of course, you're free to use multiple configs if you need to use different settings for each notebook folder)
msticpyconfig.yaml
- this could be in "~/.msticpyconfig.yaml" or "%userprofile%/msticpyconfig.yaml"msticpyconfig.yaml
file that you just created to this location.MSTICPYCONFIG
environment variable to point to that location:Open Settings and navigate to System->About and click on Advanced System Settings
In your .bashrc (or somewhere else convenient) add:
export MSTICPYCONFIG=~/.msticpyconfig.yaml
You should see the settings editor loaded with the settings you created earlier
Note the environment variable set here is not persistent
To make this persistent you need to add it to a startup file like .bashrc or your Windows registry.
%env MSTICPYCONFIG=./msticpyconfig2.yaml
mpedit = MpConfigEdit()
mpedit
env: MSTICPYCONFIG=./msticpyconfig2.yaml
Label(value='Loading. Please wait.')
e:\src\msticpy\msticpy\config\mp_config_file.py:121: UserWarning: MSTICPYCONFIG env variable is pointing to invalid path../msticpyconfig2.yaml warnings.warn(
VBox(children=(Tab(children=(VBox(children=(Label(value='Microsoft Sentinel workspace settings'), HBox(childre…
You will likely want to do lookups of IP Addresses, URLs and other items to check for any Threat Intelligence reports. To do that you need to add the providers that you want to use. Most TI providers require that you have an account with them and supply an API key or other authentication items when you connect.
Most providers have a free use tier (or in cases like AlienVault OTX) are entirely free. Free tiers for paid providers usually impose a certain number of requests that you can make in a given time period.
For account creation, each provider does this slightly differently. Use the help links in the editor help to find where to go set each of these up.
Assuming that you have done this, we can configure a provider. Be sure to store any authentication keys somewhere safe (and memorable).
We are going to use VirusTotal (VT) as an example TI Provider.
For this you will need a VirusTotal API key from the
VirusTotal website.
We also support a range of other threat intelligence providers - you can read about this here MSTICPy TIProviders
Taking VirusTotal as our example.
This should show you the values that you need to provide:
You can paste the key into the Value field and click the Save button.
You can opt to store the VT AuthKey as an environment variable. This is a bit more secure than having it laying around in configuration files. Assuming that you have set you VT key as an environment variable
set VT_KEY=VGhpcyBzaG91bGQgc2hvdyB5b3UgdGhlIHZhbHVlcyB (Windows)
export VT_KEY=VGhpcyBzaG91bGQgc2hvdyB5b3UgdGhlIHZhbHVlcyB (Linux/MAC)
Flip the Storage radio button to EnvironmentVar and type the name of the
variable (VT_KEY
in our example) into the value box.
You can also use Azure Key Vault to store secrets like these but we will need to set up the Key Vault settings before this will work.
Click the Save File button to save your changes.
mpedit.set_tab("TI Providers")
mpedit
VBox(children=(Tab(children=(VBox(children=(Label(value='Microsoft Sentinel workspace settings'), HBox(childre…
This assumes that you have set an enviroment variable pointed at your msticpyconfig.yaml. Use the MpConfigFile tool to force MSTICPy to reload settings from the disk.
mpconfig.refresh_mp_config()
# import the TI module
from msticpy.context import TILookup
result = TILookup().lookup_ioc('ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa')
TILookup.result_to_df(result)
MSTICPy supports two Geo IP providers - Maxmind GeoIPLite and IP Stack. The main difference between the two is that Maxmind downloads and uses a local database, while IPStack is a purely online solution.
For either you need API keys to either download the free database from MaxMind or access the IPStack online lookup
We'll use GeoIPLite as our example. You can sign up for a free account and API key at https://www.maxmind.com/en/geolite2/signup. You'll need the API for the following steps.
Set the maxmind data folder:
%USERPROFILE%/.msticpy
..msticpy
in your home folder.Note: as with the TI providers you can opt to store your key as an environment variable or keep it in Key Vault.
mpedit.set_tab("GeoIP Providers")
mpedit
VBox(children=(Tab(children=(VBox(children=(Label(value='Microsoft Sentinel workspace settings'), HBox(childre…
mpconfig.refresh_mp_config()
from msticpy.sectools import IPStackLookup
geoip = IPStackLookup()
geoip.lookup_ip("52.96.165.18")[1][0]
mpconfig.refresh_mp_config()
from msticpy.sectools import GeoLiteLookup
geoip = GeoLiteLookup()
geoip.lookup_ip("52.96.165.18")[1][0]
To access Azure APIs (such as the Sentinel APIs or Azure resource APIs) you need to be able to use Azure Authentication. The setting is named "AzureCLI" for historical reasons - don't let that confuse you. We currently support two ways of authenticating:
The former can try up to four methods of authentication:
To use chained authentication methods select the methods to want to use and leave the clientId/tenantiId/clientSecret fields empty.
The Splunk provider has many options. Typically you need only:
(You can also supply some or all of these values at startup)
This is a data provider that reads from local CSVs or Pickled pandas DataFrames
You can set the default data paths that it looks in for data files. This can have multiple values - put each on a new line.
The mordor provider has two options:
mpedit.set_tab("Data Providers")
mpedit
You only need this if you want to store secrets in Azure Key Vault.
You need to create the Key Vault first - do that at your Azure portal. Here is the link for the global KeyVault management portal
The required settings are all values that you get from the Vault properties (albeit a couple of them have different names)
Note: cCurrently we've only implemented and tested "global" fully but if
you are using a regional or national cloud and this isn't working please
let us know msticpy@microsoft.com and we'll get it fixed.
Click Save and then Save File when you are done.
mpedit.set_tab("Key Vault")
mpedit
See if you can connect and view any secrets. Of course nothing will show up if you haven't entered a secret. Add a test secret to the vault to show here.
WARNING *don't leave this output in your saved notebook.
If there are real secrets in the output, use the notebook Clear output* before saving the notebook.
Also delete cached copies of this notebook. Look in the.ipynb_checkpoints
sub-folder of this folder and delete copies of this notebook.
mpconfig.refresh_mp_config()
mpconfig.show_kv_secrets()
This section controls which, if any query providers you want to load automatically
when you run nbinit.init_notebook
.
This can save a lot of time if you are frequently authoring new notebooks. It also allows the right providers to be loaded before other components that might use them such as
(more about these in the next section)
There are two types of provider support:
Available Azure Sentinel workspaces are taken from the items you configured in the Azure Sentinel tab. Other providers are taken from the list of available provider types in MSTICPy.
There are two options for each of these:
Note if you lose track of which providers have been loaded by this mechanism they are added to the
current_providers
attribute ofmsticpy
import msticpy
msticpy.current_providers
mpedit.set_tab("Autoload QueryProvs")
mpedit
This section controls which, if other components you want to load automatically
when you run nbinit.init_notebook()
.
This includes
These are loaded in this order, since the Pivot component needs query and other providers loaded in order to find the pivot functions that it will attach to entities. For more information see pivot functions
Some components do not require any parameters (e.g. TILookup and Pivot). Others do support or require additional settings:
GeoIpLookup
You must type the name of the GeoIP provider that you want to use - either "GeoLiteLookup" or "IPStack"
AzureData and AzureSentinelAPI
Notebooklets
This has a single parameter block AzureSentinel. At minumum you should specify the workspace name. This needs to be in the following format:
workspace:WORKSPACENAME
WORKSPACENAME must be one of the workspaces defined in the Azure Sentinel tab.
You can also add addition parameters to send to the notebooklets init function: Specify these as addition key:value pairs, separated by newlines.
workspace:WORKSPACENAME
providers=["LocalData","geolitelookup"]
See the
msticnb init
documentation
for more details
mpedit.set_tab("Autoload Components")
mpedit
MpConfigFile
to check and manage your msticpyconfig.yaml¶This tool lets you:
mpconfig.load_default()
method or just create mpconfig = MpConfigFile()
with no parametersmpconfig.load_from_file(file_path)
mpconfig.save_to_file(file_path)
mpconfig.validate_settings()
mpconfig.browse_for_file()
mpconfig.view_settings()
There are also a couple of utility methods:
mpconfig.show_kv_secrets()
- to view secrets in your Key Vault (more on this below)mpconfig.map_json_to_mp_ws()
- you can load an AzureSentinel config.json
file and map
its settings into a format compatible with msticpyconfig.yaml
mpconfig.refresh_mp_config()
- reloads the settings for MSTICPy based on the saved default
config file (this is either the file pointed to by MSTICPYCONFIG env variable or
a file "msticpyconfig.yaml" in the current directory.You can get more information on these methods by entering:
help(MpConfigFile)