The data provider module of msticpy provides functions to allow for the defining of data sources, connectors to them and queries for them as well as the ability to return query result from the defined data sources.
For more information on Data Propviders, check documentation
In this notebooks we will demonstrate Sumologic data connector feature of msticpy. This feature is built on-top of the [Sumologic SDK for Python] (https://github.com/SumoLogic/sumologic-python-sdk) with some customizations and enhancements.
# Only run first time to install/upgrade msticpy to latest version
# %pip install --upgrade msticpy
Authentication for the Sumologic data provider is handled by specifying credentials (accessid and accesskey) directly in the connect call or specifying the credentials in msticpy config file.
For more information on how to create credentials, follow Sumologic Docs Access Keys and Users and Roles. The user should have permission to at least run its own searches or more depending upon the actions to be performed by user.
Once you created user account with the appropriate roles, you will require the following details to specify while connecting
Once you have details, you can specify it in msticpyconfig.yaml
as shown in below example
DataProviders:
Sumologic:
Args:
connection_str: "{Sumologic url endpoint}"
accessid: "{accessid with search permissions to connect}"
accesskey: "{accesskey of the user specified}"
#Check we are running Python 3.6
import sys
MIN_REQ_PYTHON = (3,6)
if sys.version_info < MIN_REQ_PYTHON:
print('Check the Kernel->Change Kernel menu and ensure that Python 3.6')
print('or later is selected as the active kernel.')
sys.exit("Python %s.%s or later is required.\n" % MIN_REQ_PYTHON)
#imports
import pandas as pd
from datetime import datetime,timedelta
#data library imports
from msticpy.data.data_providers import QueryProvider
print('Imports Complete')
Imports Complete
You can instantiate a data provider for Sumologic by specifying the credentials in connect or in msticpy config file.
If the details are correct and authentication is successful, it will show connected.
URL endpoints are referenced on Sumo Logic Endpoints and Firewall Security
sumologic_prov = QueryProvider('Sumologic')
#sumologic_prov.connect(connection_str=<url>, accessid=<accessid>, accesskey=<accesskey>)
sumologic_prov.connect()
connected
You can define your own sumologic query and run it via sumologic provider via QUERY_PROVIDER.exec_query(<queryname>)
For more information, check documentation Running and Ad-hoc Query
sumologic_query = '''
*
| formatDate(_messageTime,"yyyy/dd/MM HH:mm:ss") as date
| first(date), last(date) by _sourceCategory
| count _sourceCategory,_first,_last
| sort -_count
'''
df = sumologic_prov.exec_query(sumologic_query, days=0.0005, verbosity=3)
df.head()
sumologic_query = '''_index=WINDOWS | count _sourceCategory,hostname'''
df = sumologic_prov.exec_query(sumologic_query, start_time=datetime.now() - timedelta(days=6.001), end_time=datetime.now() - timedelta(days=6))
df.head()