#!/usr/bin/env python
# coding: utf-8

# # MSTICpy - Data Uploaders
# 
# ### Description
# This notebook provides a guided example of using the Log Analytics and Splunk Data Uploader included with MSTICpy.<br><br>
# Contents:
# - How to instanciate Uploaders
# - Uploading DataFrames
# - Uploading Files
# - Uploading Folders
# 
# You must have msticpy installed with the Azure components to run this notebook:
# ```
# %pip install --upgrade msticpy[azure]
# ```
# 

# In[1]:


#Setup
from msticpy.init import nbinit
extra_imports = ["msticpy.data.uploaders.splunk_uploader, SplunkUploader", 
                "msticpy.data.uploaders.loganalytics_uploader, LAUploader"]

nbinit.init_notebook(
    namespace=globals(),
    extra_imports=extra_imports,
);

WIDGET_DEFAULTS = {
    "layout": widgets.Layout(width="95%"),
    "style": {"description_width": "initial"},
}


# In[2]:


# Load some sample data
df = pd.read_csv('https://raw.githubusercontent.com/microsoft/msticpy/master/tests/testdata/az_net_flows.csv', parse_dates=['TimeGenerated'])
df.head(2)


# ## LogAnalytics Data Uploader
# Below we collect some details required for our uploader, instanciate our LogAnalytics data uploader and pass our DataFrame loaded above to be uploaded.
# We are setting the debug flag on our uploader so we can get some additional details on our upload progress.

# In[3]:


la_ws_id = widgets.Text(description='Workspace ID:')
la_ws_key = widgets.Password(description='Workspace Key:')
display(la_ws_id)
display(la_ws_key)


# In[3]:


# Instanciate our Uploader
la_up = LAUploader(workspace=la_ws_id.value, workspace_secret=la_ws_key.value, debug=True)
# Upload our DataFrame
la_up.upload_df(data=df, table_name='upload_demo')


# ### Upload a file
# We can now upload a file to our Workspace using the same Uploader. We simply pass the path to the file we want to upload, and we can also pass a table name for the data to be uploaded to.

# In[5]:


la_up.upload_file(file_path='data/alertlist.csv', table_name='upload_demo')


# ### Upload a folder
# We can now upload a file to our Workspace using the same Uploader. We simply pass the the path to the folder we want to upload file from. In this case we aren't going to pass a table name, in which case the name will be generated automatically for each file from the file's name. With a folder we get a progress bar showing the progress uploading each file.

# In[6]:


la_up.upload_folder(folder_path='data/')


# ## Splunk Data Uploader
# The Splunk Uploader functions in the same manner as the LogAnalytics one. <br>
# Below we collect some details required for our uploader, instanciate our Splunk data uploader and pass our DataFrame loaded above to be uploaded.
# We are setting the debug flag on our uploader so we can get some additional details on our upload progress.<br>
# When uploading our DataFrame the only difference is that as well as providing a table name (which is represneted as sourcetype in Splunk), we also need to pass a Splunk index that we want to data uploaded to. Also as Splunk uploads data a line at a time we get a progress bar for the file as it uploads.

# In[7]:


sp_host = widgets.Text(description='Splunk host')
sp_user = widgets.Text(description='Username')
sp_pwrd = widgets.Password(description='Password')
display(sp_host)
display(sp_user)
display(sp_pwrd)


# In[3]:


# Instanciate our Uploader
spup = SplunkUploader(username=sp_user.value, host=sp_host.value, password=sp_pwrd.value, debug=True)
# Upload our DataFrame
spup.upload_df(data=df, table_name='upload_test', index_name='upload_test')


# ### Upload a file
# We can now upload a file to our Workspace using the same Uploader. We simply pass the path to the file we want to upload along with the index name, and we can also pass a table name for the data to be uploaded to.

# In[4]:


spup.upload_file(file_path='data/alertlist.csv', index_name='upload_demo', table_name='upload_demo')


# ### Upload a folder
# We can now upload a file to our Workspace using the same Uploader. We simply pass the the path to the folder we want to upload file from. In this case we aren't going to pass a table name, in which case the name will be generated automatically for each file from the file's name however we still need to pass and index name.

# In[7]:


spup.upload_folder(folder_path='data/', index_name='upload_demo')    


# In[ ]: