This notebook describes the use of the Threat Intelligence lookup class in msticpy. The class allows lookup of individual or multiple IoCs from one or more TI providers.
TILookup is also extensible - you can subclass TIProvider to implement your own custom lookups. You can also subclass the HTTPProvider or KqlProvider classes, which provide support for querying a REST endpoint or Log Analytics table respectively.
# Imports
import sys
import warnings
from msticpy.common.utility import check_py_version
MIN_REQ_PYTHON = (3,6)
check_py_version(MIN_REQ_PYTHON)
from IPython import get_ipython
from IPython.display import display, HTML, Markdown
import pandas as pd
pd.set_option('display.max_rows', 100)
pd.set_option('display.max_columns', 50)
pd.set_option('display.max_colwidth', 50)
from msticpy.data import QueryProvider
from msticpy.nbtools import *
from msticpy.sectools import *
Input can be a single IoC observable or a pandas DataFrame containing multiple observables. Processing may require a an API key and processing performance may be limited to a specific number of requests per minute for the account type that you have.
# TILookup class
display(Markdown("### Constructor\n"))
print(TILookup.__init__.__doc__)
display(Markdown("### Attributes\n"))
for name in [att for att in dir(TILookup) if not att.startswith("_")]:
display(Markdown(f"#### _{name}()_"))
print(getattr(TILookup, name).__doc__)
print()
Initialize TILookup instance. Parameters ---------- primary_providers : Optional[List[TIProvider]], optional Primary TI Providers, by default None secondary_providers : Optional[List[TIProvider]], optional Secondary TI Providers, by default None
Add a TI provider to the current collection. Parameters ---------- provider : TIProvider Provider instance name : str, optional The name to use for the provider (overrides the class name of `provider`) primary : bool, optional "primary" or "secondary" if False, by default "primary"
Return a list of builtin providers. Returns ------- List[str] List of TI Provider classes.
Print a list of builtin providers with optional usage. Parameters ---------- show_query_types : bool, optional Show query types supported by providers, by default False
Return dictionary of loaded providers. Returns ------- Dict[str, TIProvider] [description]
Lookup single IoC in active providers. Parameters ---------- observable : str IoC observable (`ioc` is also an alias for observable) ioc_type : str, optional One of IoCExtract.IoCType, by default None If none, the IoC type will be inferred ioc_query_type: str, optional The ioc query type (e.g. rep, info, malware) providers: List[str] Explicit list of providers to use prov_scope : str, optional Use "primary", "secondary" or "all" providers, by default "primary" kwargs : Additional arguments passed to the underlying provider(s) Returns ------- Tuple[bool, List[Tuple[str, LookupResult]]] The result returned as a tuple(bool, list): bool indicates whether a TI record was found in any provider list has an entry for each provider result
Lookup a collection of IoCs. Parameters ---------- data : Union[pd.DataFrame, Mapping[str, str], Iterable[str]] Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in `obs_col` parameter) 2. Mapping (e.g. a dict) of [observable, IoCType] 3. Iterable of observables - IoCTypes will be inferred obs_col : str, optional DataFrame column to use for observables, by default None ioc_type_col : str, optional DataFrame column to use for IoCTypes, by default None ioc_query_type: str, optional The ioc query type (e.g. rep, info, malware) providers: List[str] Explicit list of providers to use prov_scope : str, optional Use "primary", "secondary" or "all" providers, by default "primary" kwargs : Additional arguments passed to the underlying provider(s) Returns ------- pd.DataFrame DataFrame of results
Return loaded provider status. Returns ------- Iterable[str] List of providers and descriptions.
Print usage of loaded providers.
Reload provider settings from config.
Reload providers based on currret settings in config.
Return DataFrame representation of IoC Lookup response. Parameters ---------- ioc_lookup : Tuple[bool, List[Tuple[str, LookupResult]]] Output from `lookup_ioc` Returns ------- pd.DataFrame The response as a DataFrame with a row for each provider response.
The msticpy TI Provider library can lookup IoCs in multiple providers.
The list below shows the current set of providers.
ti_lookup = TILookup()
# List available providers
ti_lookup.available_providers
['AzSTI', 'OTX', 'VirusTotal', 'XForce']
You can configure primary and secondary providers. Primary providers are used by default.
You may need to supply an authorization (API) key and in some cases a user ID for each provider.
For LogAnalytics/Azure Sentinel providers, you will need the workspace ID and tenant ID and will need to authenticate in order to access the data (although if you have an existing authenticated connection with the same workspace/tenant, this connection will be re-used).
The configuration file is read from the current directory.
Alternatively, you can specify a location for this file in an environment variable MSTICPYCONFIG
.
If you need to create a config file, uncomment the lines in the following cell.
Warning - this will overwrite a file of the same name in the current directory
Delete any provider entries that you do not want to use and add the missing parameters for your providers.
# %%writefile msticpyconfig.yaml
# QueryDefinitions:
# TIProviders:
# OTX:
# Args:
# AuthKey: "your-otx-key"
# Primary: True
# Provider: "OTX" # Explicitly name provider to override
# VirusTotal:
# Args:
# AuthKey: "your-vt-key"
# Primary: True
# Provider: "VirusTotal"
# XForce:
# Args:
# ApiID: "your-xforce-id"
# AuthKey: "your-xforce-key"
# Primary: True
# Provider: "XForce"
# AzureSentinel:
# Args:
# WorkspaceID: "your-azure-sentinel-workspace-id"
# TenantID: "your-azure-sentinel-tenant-id"
# Primary: True
# Provider: "AzSTI"
Reload providers to pick up new settings
ti_lookup.reload_providers()
ti_lookup.provider_status
['OTX - AlientVault OTX Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'AzSTI - Azure Sentinel TI provider class. (primary)', 'VirusTotal - VirusTotal Lookup. (secondary)']
To lookup a single IoC.
ti_lookup.lookup_ioc(
observable: str = None,
ioc_type: str = None,
ioc_query_type: str = None,
providers: List[str] = None,
prov_scope: str = 'primary',
**kwargs,
) -> Tuple[bool, List[Tuple[str, msticpy.sectools.tiproviders.ti_provider_base.LookupResult]]]
Lookup single IoC in active providers.
Parameters
----------
observable : str
IoC observable
(`ioc` is also an alias for observable)
ioc_type : str, optional
One of IoCExtract.IoCType, by default None
If none, the IoC type will be inferred
ioc_query_type: str, optional
The ioc query type (e.g. rep, info, malware)
providers: List[str]
Explicit list of providers to use
prov_scope : str, optional
Use primary, secondary or all providers, by default "primary"
kwargs :
Additional arguments passed to the underlying provider(s)
Returns
-------
Tuple[bool, List[Tuple[str, LookupResult]]]
The result returned as a tuple(bool, list):
bool indicates whether a TI record was found in any provider
list has an entry for each provider result
# Uncomment this and run to see the document string
# ti_lookup.lookup_ioc?
And show the output
result = ti_lookup.lookup_ioc(observable="52.183.120.194", providers=["AzSTI", "XForce"])
ti_lookup.result_to_df(result)
Ioc | IocType | QuerySubtype | Result | Severity | Details | RawResult | Reference | Status | |
---|---|---|---|---|---|---|---|---|---|
XForce | 52.183.120.194 | ipv4 | None | True | 1 | {'score': 1, 'cats': {}, 'categoryDescriptions... | {'ip': '52.183.120.194', 'history': [{'created... | https://api.xforce.ibmcloud.com/ipr/52.183.120... | 200 |
AzSTI | 52.183.120.194 | ipv4 | None | True | 2 | {'Action': ['alert'], 'ThreatType': ['Malware'... | Indic... | ThreatIntelligenceIndicator | where TimeGene... | 0 |
result = ti_lookup.lookup_ioc(observable="52.183.120.194")
ti_lookup.result_to_df(result).T
OTX | XForce | AzSTI | |
---|---|---|---|
Ioc | 52.183.120.194 | 52.183.120.194 | 52.183.120.194 |
IocType | ipv4 | ipv4 | ipv4 |
QuerySubtype | None | None | None |
Result | True | True | True |
Severity | 0 | 1 | 2 |
Details | {'pulse_count': 0, 'sections_available': ['gen... | {'score': 1, 'cats': {}, 'categoryDescriptions... | {'Action': ['alert'], 'ThreatType': ['Malware'... |
RawResult | {'sections': ['general', 'geo', 'reputation', ... | {'ip': '52.183.120.194', 'history': [{'created... | Indic... |
Reference | https://otx.alienvault.com/api/v1/indicators/I... | https://api.xforce.ibmcloud.com/ipr/52.183.120... | ThreatIntelligenceIndicator | where TimeGene... |
Status | 200 | 200 | 0 |
import pprint
pp = pprint.PrettyPrinter(indent=2)
result, details = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])
# the details is a list (since there could be multiple responses for an IoC)
for provider, detail in details:
print(provider)
detail.summary
# Un-comment to view raw response
# print("\nRaw Results")
# pp.pprint(detail.raw_result)
OTX ioc: 38.75.137.9 ( ipv4 ) result: True severity: 1 warning { 'names': ['Underminer EK'], 'pulse_count': 1, 'references': [ [ 'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/']], 'tags': [[]]} reference: https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general
result = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])
ti_lookup.result_to_df(result).T
OTX | |
---|---|
Ioc | 38.75.137.9 |
IocType | ipv4 |
QuerySubtype | None |
Result | True |
Severity | 1 |
Details | {'pulse_count': 1, 'names': ['Underminer EK'],... |
RawResult | {'sections': ['general', 'geo', 'reputation', ... |
Reference | https://otx.alienvault.com/api/v1/indicators/I... |
Status | 200 |
# Extract a single field (RawResult) from the dataframe (.iloc[0] is to select the row)
ti_lookup.result_to_df(result)["RawResult"].iloc[0]
{'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list', 'http_scans'], 'city': 'Los Angeles', 'area_code': 0, 'pulse_info': {'count': 1, 'references': ['https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'], 'pulses': [{'indicator_type_counts': {'URL': 16, 'FileHash-MD5': 5, 'IPv4': 3}, 'pulse_source': 'web', 'TLP': 'white', 'description': '', 'subscriber_count': 11, 'tags': [], 'export_count': 0, 'malware_families': [], 'is_modified': False, 'upvotes_count': 0, 'modified_text': '55 days ago ', 'is_subscribing': None, 'references': ['https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'], 'targeted_countries': [], 'groups': [{'name': 'DCT Security Team', 'id': 614}], 'vote': 0, 'validator_count': 0, 'threat_hunter_scannable': True, 'is_author': False, 'adversary': '', 'id': '5d41d77901a2f8c6e9b650e9', 'industries': [], 'locked': 0, 'name': 'Underminer EK', 'created': '2019-07-31T18:01:29.744000', 'cloned_from': None, 'downvotes_count': 0, 'modified': '2019-07-31T18:01:29.744000', 'comment_count': 0, 'indicator_count': 24, 'attack_ids': [], 'in_group': True, 'follower_count': 0, 'votes_count': 0, 'author': {'username': 'mattvittitoe', 'is_subscribed': False, 'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png', 'is_following': False, 'id': '79520'}, 'public': 1}]}, 'continent_code': 'NA', 'country_name': 'United States', 'postal_code': '90017', 'dma_code': 803, 'country_code': 'US', 'flag_url': '/assets/images/flags/us.png', 'asn': 'AS63023 GTHost', 'city_data': True, 'indicator': '38.75.137.9', 'whois': 'http://whois.domaintools.com/38.75.137.9', 'type_title': 'IPv4', 'region': 'CA', 'charset': 0, 'longitude': -118.278, 'country_code3': 'USA', 'reputation': 0, 'base_indicator': {'indicator': '38.75.137.9', 'description': '', 'title': '', 'access_reason': '', 'access_type': 'public', 'content': '', 'type': 'IPv4', 'id': 2127020821}, 'latitude': 34.0584, 'type': 'IPv4', 'flag_title': 'United States'}
result = ti_lookup.lookup_ioc(observable="188.127.231.124")
ti_lookup.result_to_df(result)
Ioc | IocType | QuerySubtype | Result | Severity | Details | RawResult | Reference | Status | |
---|---|---|---|---|---|---|---|---|---|
OTX | 188.127.231.124 | ipv4 | None | True | 2 | {'pulse_count': 5, 'names': ['Locky Ransomware... | {'sections': ['general', 'geo', 'reputation', ... | https://otx.alienvault.com/api/v1/indicators/I... | 200 |
XForce | 188.127.231.124 | ipv4 | None | True | 1 | {'score': 1, 'cats': {}, 'categoryDescriptions... | {'ip': '188.127.231.124', 'history': [{'create... | https://api.xforce.ibmcloud.com/ipr/188.127.23... | 200 |
AzSTI | 188.127.231.124 | ipv4 | None | False | 0 | 0 rows returned. | None | None | -1 |
This shows the supported IoC Types.
In some cases an IoC type will also support special types of sub-query such as geo-ip and passive-dns
display(ti_lookup.provider_status)
ti_lookup.loaded_providers["AzSTI"].usage()
['OTX - AlientVault OTX Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'AzSTI - Azure Sentinel TI provider class. (primary)', 'VirusTotal - VirusTotal Lookup. (secondary)']
Azure Sentinel TI provider class. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv6 ioc_type=linux_path ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=windows_path
ti_lookup.provider_usage()
Primary providers ----------------- Provider class: OTX AlientVault OTX Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=geo ioc_type=dns, ioc_query_type=passivedns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=geo ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=geo ioc_type=ipv6, ioc_query_type=passivedns ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url Provider class: XForce IBM XForce Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=malware ioc_type=dns, ioc_query_type=passivedns ioc_type=dns, ioc_query_type=whois ioc_type=file_hash ioc_type=hostname, ioc_query_type=whois ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=malware ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv4, ioc_query_type=rep ioc_type=ipv4, ioc_query_type=whois ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=malware ioc_type=ipv6, ioc_query_type=passivedns ioc_type=ipv6, ioc_query_type=rep ioc_type=ipv6, ioc_query_type=whois ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=url, ioc_query_type=malware Provider class: AzSTI Azure Sentinel TI provider class. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv6 ioc_type=linux_path ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=windows_path Secondary providers ------------------- Provider class: VirusTotal VirusTotal Lookup. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=ipv4 ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url
result = ti_lookup.lookup_ioc(observable="38.75.137.9", ico_type="ipv4", ioc_query_type="passivedns", providers=["XForce"])
print(result)
print("\nProvider result:")
result[1][0][1].raw_result
(True, [('XForce', LookupResult(ioc='38.75.137.9', ioc_type='ipv4', query_subtype='passivedns', result=True, severity=0, details={'records': 1}, raw_result={'Passive': {'query': '0x00000000000000000000ffff264b8909', 'records': []}, 'RDNS': ['9-137-75-38.clients.gthost.com'], 'total_rows': 1}, reference='https://api.xforce.ibmcloud.com/resolve/38.75.137.9', status=200))]) Provider result:
{'Passive': {'query': '0x00000000000000000000ffff264b8909', 'records': []}, 'RDNS': ['9-137-75-38.clients.gthost.com'], 'total_rows': 1}
result = ti_lookup.lookup_ioc(observable="38.75.137.9", ico_type="ipv4", ioc_query_type="geo", providers=["OTX"])
print(result)
print("\nProvider result:")
result[1][0][1].raw_result
(True, [('OTX', LookupResult(ioc='38.75.137.9', ioc_type='ipv4', query_subtype='geo', result=True, severity=0, details={}, raw_result={'flag_url': '/assets/images/flags/us.png', 'city_data': True, 'city': 'Los Angeles', 'region': 'CA', 'charset': 0, 'area_code': 0, 'continent_code': 'NA', 'country_code3': 'USA', 'latitude': 34.0584, 'postal_code': '90017', 'longitude': -118.278, 'country_code': 'US', 'country_name': 'United States', 'asn': 'AS63023 GTHost', 'dma_code': 803, 'flag_title': 'United States'}, reference='https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/geo', status=200))]) Provider result:
{'flag_url': '/assets/images/flags/us.png', 'city_data': True, 'city': 'Los Angeles', 'region': 'CA', 'charset': 0, 'area_code': 0, 'continent_code': 'NA', 'country_code3': 'USA', 'latitude': 34.0584, 'postal_code': '90017', 'longitude': -118.278, 'country_code': 'US', 'country_name': 'United States', 'asn': 'AS63023 GTHost', 'dma_code': 803, 'flag_title': 'United States'}
If you do a lookup without specifying a type, TILookup will try to infer the type by matching regexes. There are patterns for all supported types but there are some caveats:
If you know the type that you want to look up, it is always better to explicitly include it.
ioc_type
parameter.{ioc_observable: ioc_type}
Signature:
ti_lookup.lookup_iocs(
data: Union[pandas.core.frame.DataFrame, Mapping[str, str], Iterable[str]],
obs_col: str = None,
ioc_type_col: str = None,
ioc_query_type: str = None,
providers: List[str] = None,
prov_scope: str = 'primary',
**kwargs,
) -> pandas.core.frame.DataFrame
Lookup a collection of IoCs.
Parameters
----------
data : Union[pd.DataFrame, Mapping[str, str], Iterable[str]]
Data input in one of three formats:
1. Pandas dataframe (you must supply the column name in
`obs_col` parameter)
2. Mapping (e.g. a dict) of [observable, IoCType]
3. Iterable of observables - IoCTypes will be inferred
obs_col : str, optional
DataFrame column to use for observables, by default None
ioc_type_col : str, optional
DataFrame column to use for IoCTypes, by default None
ioc_query_type: str, optional
The ioc query type (e.g. rep, info, malware)
providers: List[str]
Explicit list of providers to use
prov_scope : str, optional
Use primary, secondary or all providers, by default "primary"
kwargs :
Additional arguments passed to the underlying provider(s)
Returns
-------
pd.DataFrame
DataFrame of results
# Uncomment this and run to see the document string
# ti_lookup.lookup_iocs?
ioc_ips = [
"51.75.29.61",
"33.44.55.66"
"52.183.120.194",
"13.91.229.209",
"1.2.3.4",
"52.167.223.49",
"1.2.3.5",
]
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI")
Ioc | IocType | QuerySubtype | Reference | Result | Status | Severity | Details | RawResult | Provider | |
---|---|---|---|---|---|---|---|---|---|---|
0 | 1.2.3.4 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | True | 0 | 2 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': 'BF59F0493B17FFDCDA7B8D2969342... | AzSTI |
1 | 13.91.229.209 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | True | 0 | 2 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': '0F1ED19DB6F3E209EB7B3C70F3DA5... | AzSTI |
2 | 52.167.223.49 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | True | 0 | 2 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': 'A6AF4F01F06D5977DE741A41DD1BC... | AzSTI |
3 | 51.75.29.61 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | True | 0 | 2 | {'Action': 'alert', 'ThreatType': 'WatchList',... | {'IndicatorId': '745AC38B70FF24CC7DCA13BB4467D... | AzSTI |
4 | 1.2.3.5 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | True | 0 | 2 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': 'AE428189DCD303DA2A79AF8F85030... | AzSTI |
Output sorted by IoC
Note that these URLs were picked randomly from the TI databases of the three providers used. In most cases the IoC is found by only that provider, which
ioc_urls = [
"http://cheapshirts.us/zVnMrG.php",
"http://chinasymbolic.com/i9jnrc",
"https://hotel-bristol.lu/dlry/MAnJIPnY/",
"http://businesstobuy.net",
"http://append.pl/srh9xsz",
"http://104.248.196.145/apache2",
"http://ajaraheritage.ge/g7cberv",
"http://cic-integration.com/hjy93JNBasdas",
"https://google.com", # benign
"https://microsoft.com", # benign
"https://python.org", # benign
]
results = ti_lookup.lookup_iocs(data=ioc_urls)
results.sort_values("Ioc")
Ioc | IocType | QuerySubtype | Result | Severity | Details | RawResult | Reference | Status | Provider | |
---|---|---|---|---|---|---|---|---|---|---|
5 | http://104.248.196.145/apache2 | url | None | False | 0.0 | Not found. | <Response [404]> | https://api.xforce.ibmcloud.com/url/http://104... | 404.0 | XForce |
5 | http://104.248.196.145/apache2 | url | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'indicator': 'http://104.248.196.145/apache2'... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
2 | http://104.248.196.145/apache2 | url | None | True | 2.0 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': '415EE92D312FAE7ACAAC8329C2EE4... | ThreatIntelligenceIndicator | where TimeGene... | 0.0 | AzSTI |
6 | http://ajaraheritage.ge/g7cberv | url | None | True | 2.0 | {'pulse_count': 2, 'names': ['Locky Ransomware... | {'indicator': 'http://ajaraheritage.ge/g7cberv... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
6 | http://ajaraheritage.ge/g7cberv | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'ajaraheritage.ge', 'cats':... | https://api.xforce.ibmcloud.com/url/http://aja... | 200.0 | XForce |
9 | http://ajaraheritage.ge/g7cberv | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
4 | http://append.pl/srh9xsz | url | None | True | 1.0 | {'pulse_count': 1, 'names': ['Locky Ransomware... | {'indicator': 'http://append.pl/srh9xsz', 'ale... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
0 | http://append.pl/srh9xsz | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
4 | http://append.pl/srh9xsz | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'append.pl', 'cats': {'Soft... | https://api.xforce.ibmcloud.com/url/http://app... | 200.0 | XForce |
3 | http://businesstobuy.net | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
3 | http://businesstobuy.net | url | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'indicator': 'http://businesstobuy.net', 'ale... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
3 | http://businesstobuy.net | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'businesstobuy.net', 'cats'... | https://api.xforce.ibmcloud.com/url/http://bus... | 200.0 | XForce |
4 | http://cheapshirts.us/zVnMrG.php | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
0 | http://cheapshirts.us/zVnMrG.php | url | None | True | 2.0 | {'pulse_count': 7, 'names': ['CryptoWall Ranso... | {'indicator': 'http://cheapshirts.us/zVnMrG.ph... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
0 | http://cheapshirts.us/zVnMrG.php | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'cheapshirts.us/zvnmrg.php'... | https://api.xforce.ibmcloud.com/url/http://che... | 200.0 | XForce |
1 | http://chinasymbolic.com/i9jnrc | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'chinasymbolic.com', 'cats'... | https://api.xforce.ibmcloud.com/url/http://chi... | 200.0 | XForce |
5 | http://chinasymbolic.com/i9jnrc | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
1 | http://chinasymbolic.com/i9jnrc | url | None | True | 2.0 | {'pulse_count': 2, 'names': ['Locky Ransomware... | {'indicator': 'http://chinasymbolic.com/i9jnrc... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
6 | http://cic-integration.com/hjy93JNBasdas | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
7 | http://cic-integration.com/hjy93JNBasdas | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'cic-integration.com', 'cat... | https://api.xforce.ibmcloud.com/url/http://cic... | 200.0 | XForce |
7 | http://cic-integration.com/hjy93JNBasdas | url | None | True | 1.0 | {'pulse_count': 1, 'names': ['Locky Ransomware... | {'indicator': 'http://cic-integration.com/hjy9... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
7 | https://google.com | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
8 | https://google.com | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'https://google.com', 'cats... | https://api.xforce.ibmcloud.com/url/https://go... | 200.0 | XForce |
8 | https://google.com | url | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'indicator': 'https://google.com', 'alexa': '... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
2 | https://hotel-bristol.lu/dlry/MAnJIPnY/ | url | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'indicator': 'https://hotel-bristol.lu/dlry/M... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
10 | https://hotel-bristol.lu/dlry/MAnJIPnY/ | url | None | True | 2.0 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': '53B98EB318CD0B0388240598D0FF9... | ThreatIntelligenceIndicator | where TimeGene... | 0.0 | AzSTI |
2 | https://hotel-bristol.lu/dlry/MAnJIPnY/ | url | None | False | 0.0 | Not found. | <Response [404]> | https://api.xforce.ibmcloud.com/url/https://ho... | 404.0 | XForce |
9 | https://microsoft.com | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'microsoft.com', 'cats': {'... | https://api.xforce.ibmcloud.com/url/https://mi... | 200.0 | XForce |
9 | https://microsoft.com | url | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'indicator': 'https://microsoft.com', 'alexa'... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
8 | https://microsoft.com | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
10 | https://python.org | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'python.org', 'cats': {'Sof... | https://api.xforce.ibmcloud.com/url/https://py... | 200.0 | XForce |
10 | https://python.org | url | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'indicator': 'https://python.org', 'alexa': '... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
1 | https://python.org | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
ioc_mixed = [
"http://104.248.196.145/apache2",
"http://ajaraheritage.ge/g7cberv",
"http://cic-integration.com/hjy93JNBasdas",
"51.75.29.61",
"33.44.55.66",
"52.183.120.194",
"f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092",
"cc2db822f652ca67038ba7cca8a8bde3",
"ajaraheritage.ge",
]
results = ti_lookup.lookup_iocs(data=ioc_mixed)
results
Ioc | IocType | QuerySubtype | Result | Severity | Details | RawResult | Reference | Status | Provider | |
---|---|---|---|---|---|---|---|---|---|---|
0 | http://104.248.196.145/apache2 | url | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'indicator': 'http://104.248.196.145/apache2'... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
1 | http://ajaraheritage.ge/g7cberv | url | None | True | 2.0 | {'pulse_count': 2, 'names': ['Locky Ransomware... | {'indicator': 'http://ajaraheritage.ge/g7cberv... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
2 | http://cic-integration.com/hjy93JNBasdas | url | None | True | 1.0 | {'pulse_count': 1, 'names': ['Locky Ransomware... | {'indicator': 'http://cic-integration.com/hjy9... | https://otx.alienvault.com/api/v1/indicators/u... | 200.0 | OTX |
3 | 51.75.29.61 | ipv4 | None | True | 2.0 | {'pulse_count': 28, 'names': ['SSH honeypot lo... | {'sections': ['general', 'geo', 'reputation', ... | https://otx.alienvault.com/api/v1/indicators/I... | 200.0 | OTX |
4 | 33.44.55.66 | ipv4 | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'sections': ['general', 'geo', 'reputation', ... | https://otx.alienvault.com/api/v1/indicators/I... | 200.0 | OTX |
5 | 52.183.120.194 | ipv4 | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'sections': ['general', 'geo', 'reputation', ... | https://otx.alienvault.com/api/v1/indicators/I... | 200.0 | OTX |
6 | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5... | sha256_hash | None | True | 2.0 | {'pulse_count': 3, 'names': ['Emotet IOCs 2/4/... | {'indicator': 'f8a7135496fd6168df5f0ea21c745db... | https://otx.alienvault.com/api/v1/indicators/f... | 200.0 | OTX |
7 | cc2db822f652ca67038ba7cca8a8bde3 | md5_hash | None | True | 0.0 | {'pulse_count': 0, 'sections_available': ['gen... | {'indicator': 'cc2db822f652ca67038ba7cca8a8bde... | https://otx.alienvault.com/api/v1/indicators/f... | 200.0 | OTX |
8 | ajaraheritage.ge | dns | None | True | 1.0 | {'pulse_count': 1, 'names': ['Ransomware - Loc... | {'indicator': 'ajaraheritage.ge', 'alexa': 'ht... | https://otx.alienvault.com/api/v1/indicators/d... | 200.0 | OTX |
0 | http://104.248.196.145/apache2 | url | None | False | 0.0 | Not found. | <Response [404]> | https://api.xforce.ibmcloud.com/url/http://104... | 404.0 | XForce |
1 | http://ajaraheritage.ge/g7cberv | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'ajaraheritage.ge', 'cats':... | https://api.xforce.ibmcloud.com/url/http://aja... | 200.0 | XForce |
2 | http://cic-integration.com/hjy93JNBasdas | url | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'cic-integration.com', 'cat... | https://api.xforce.ibmcloud.com/url/http://cic... | 200.0 | XForce |
3 | 51.75.29.61 | ipv4 | None | True | 1.0 | {'score': 1, 'cats': {}, 'categoryDescriptions... | {'ip': '51.75.29.61', 'history': [{'created': ... | https://api.xforce.ibmcloud.com/ipr/51.75.29.61 | 200.0 | XForce |
4 | 33.44.55.66 | ipv4 | None | True | 1.0 | {'score': 1, 'cats': {}, 'categoryDescriptions... | {'ip': '33.44.55.66', 'history': [{'created': ... | https://api.xforce.ibmcloud.com/ipr/33.44.55.66 | 200.0 | XForce |
5 | 52.183.120.194 | ipv4 | None | True | 1.0 | {'score': 1, 'cats': {}, 'categoryDescriptions... | {'ip': '52.183.120.194', 'history': [{'created... | https://api.xforce.ibmcloud.com/ipr/52.183.120... | 200.0 | XForce |
6 | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5... | sha256_hash | None | True | 2.0 | {'risk': 'high', 'family': None, 'reasonDescri... | {'malware': {'origins': {'external': {'source'... | https://api.xforce.ibmcloud.com/malware/f8a713... | 200.0 | XForce |
7 | cc2db822f652ca67038ba7cca8a8bde3 | md5_hash | None | True | 2.0 | {'risk': 'high', 'family': None, 'reasonDescri... | {'malware': {'origins': {'external': {'source'... | https://api.xforce.ibmcloud.com/malware/cc2db8... | 200.0 | XForce |
8 | ajaraheritage.ge | dns | None | True | 0.0 | {'score': 0, 'cats': None, 'categoryDescriptio... | {'result': {'url': 'ajaraheritage.ge', 'cats':... | https://api.xforce.ibmcloud.com/url/ajaraherit... | 200.0 | XForce |
0 | http://cic-integration.com/hjy93JNBasdas | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
1 | http://ajaraheritage.ge/g7cberv | url | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
2 | http://104.248.196.145/apache2 | url | None | True | 2.0 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': '415EE92D312FAE7ACAAC8329C2EE4... | ThreatIntelligenceIndicator | where TimeGene... | 0.0 | AzSTI |
3 | 33.44.55.66 | ipv4 | None | True | 2.0 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': '988978A20393BAFE37D639C36922E... | ThreatIntelligenceIndicator | where TimeGene... | 0.0 | AzSTI |
4 | 52.183.120.194 | ipv4 | None | True | 2.0 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': 'E277A374F1848BE5E04AC08D422B4... | ThreatIntelligenceIndicator | where TimeGene... | 0.0 | AzSTI |
5 | 51.75.29.61 | ipv4 | None | True | 2.0 | {'Action': 'alert', 'ThreatType': 'WatchList',... | {'IndicatorId': '745AC38B70FF24CC7DCA13BB4467D... | ThreatIntelligenceIndicator | where TimeGene... | 0.0 | AzSTI |
6 | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5... | sha256_hash | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
7 | cc2db822f652ca67038ba7cca8a8bde3 | md5_hash | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
8 | ajaraheritage.ge | dns | None | False | 0.0 | 0 rows returned. | NaN | ThreatIntelligenceIndicator | where TimeGene... | -1.0 | AzSTI |
Some providers (currently only AzSTI) support time ranges so that you can specify specific periods to search for.
If a provider does not support time ranges, the parameters will be ignored
from datetime import datetime
search_origin = datetime(2019, 8, 5)
q_times = nbwidgets.QueryTime(units="hour", auto_display=True, origin_time=search_origin, max_after=24, max_before=24)
HTML(value='<h4>Set query time boundaries</h4>')
HBox(children=(DatePicker(value=datetime.date(2019, 8, 5), description='Origin Date'), Text(value='00:00:00', …
VBox(children=(IntRangeSlider(value=(-24, 10), description='Time Range (hour):', layout=Layout(width='80%'), m…
# Using this data range returned no results
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI", start=q_times.start, end=q_times.end).head()
Ioc | IocType | QuerySubtype | Reference | Result | Details | Status | Severity | Provider | |
---|---|---|---|---|---|---|---|---|---|
0 | 1.2.3.4 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | 0 rows returned. | -1 | 0 | AzSTI |
1 | 13.91.229.209 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | 0 rows returned. | -1 | 0 | AzSTI |
2 | 52.167.223.49 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | 0 rows returned. | -1 | 0 | AzSTI |
3 | 51.75.29.61 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | 0 rows returned. | -1 | 0 | AzSTI |
4 | 1.2.3.5 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | 0 rows returned. | -1 | 0 | AzSTI |
from datetime import datetime
search_origin = datetime(2019, 8, 5)
q_times = nbwidgets.QueryTime(units="day", auto_display=True, origin_time=search_origin, max_after=24, max_before=24)
HTML(value='<h4>Set query time boundaries</h4>')
HBox(children=(DatePicker(value=datetime.date(2019, 8, 5), description='Origin Date'), Text(value='00:00:00', …
VBox(children=(IntRangeSlider(value=(-24, 10), description='Time Range (day):', layout=Layout(width='80%'), ma…
# Using a wider ranges produces results
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI", start=q_times.start, end=q_times.end)
Ioc | IocType | QuerySubtype | Reference | Result | Status | Severity | Details | RawResult | Provider | |
---|---|---|---|---|---|---|---|---|---|---|
0 | 1.2.3.4 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | True | 0.0 | 2.0 | {'Action': 'alert', 'ThreatType': 'Malware', '... | {'IndicatorId': 'BF59F0493B17FFDCDA7B8D2969342... | AzSTI |
1 | 13.91.229.209 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | -1.0 | 0.0 | 0 rows returned. | NaN | AzSTI |
2 | 52.167.223.49 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | -1.0 | 0.0 | 0 rows returned. | NaN | AzSTI |
3 | 51.75.29.61 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | -1.0 | 0.0 | 0 rows returned. | NaN | AzSTI |
4 | 1.2.3.5 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGene... | False | -1.0 | 0.0 | 0 rows returned. | NaN | AzSTI |