This notebook describes the use of the Threat Intelligence lookup class in msticpy. The class allows lookup of individual or multiple IoCs from one or more TI providers.
TILookup is also extensible - you can subclass TIProvider to implement your own custom lookups. You can also subclass the HTTPProvider or KqlProvider classes, which provide support for querying a REST endpoint or Log Analytics table respectively.
# Imports
import sys
import warnings
from msticpy.common.utility import check_py_version
MIN_REQ_PYTHON = (3,6)
check_py_version(MIN_REQ_PYTHON)
from msticpy import init_notebook
init_notebook(namespace=globals());
Processing imports.... Checking configuration.... The following configuration errors were found: ----------------------------------------------- TIProviders/AzureSentinel: Missing or invalid WorkspaceID. TIProviders/AzureSentinel: Missing or invalid TenantID. The following configuration warnings were found: ------------------------------------------------- No AzureCLI section in settings. Setting options....
Input can be a single IoC observable or a pandas DataFrame containing multiple observables. Processing may require a an API key and processing performance may be limited to a specific number of requests per minute for the account type that you have.
# TILookup class
display(Markdown("### Constructor\n"))
print(TILookup.__init__.__doc__)
display(Markdown("### Attributes\n"))
for name in [att for att in dir(TILookup) if not att.startswith("_")]:
display(Markdown(f"#### _{name}()_"))
print(getattr(TILookup, name).__doc__)
print()
Initialize TILookup instance. Parameters ---------- primary_providers : Optional[List[TIProvider]], optional Primary TI Providers, by default None secondary_providers : Optional[List[TIProvider]], optional Secondary TI Providers, by default None providers: Optional[List[str]], optional List of provider names to load, by default all available providers are loaded. To see the list of available providers call `TILookup.list_available_providers()`. Note: if primary_provides or secondary_providers is specified This will override the providers list.
Add a TI provider to the current collection. Parameters ---------- provider : TIProvider Provider instance name : str, optional The name to use for the provider (overrides the class name of `provider`) primary : bool, optional "primary" or "secondary" if False, by default "primary"
Return a list of builtin providers. Returns ------- List[str] List of TI Provider classes.
Return a list of avaliable providers that have configuration details present. Returns ------- List[str] List of TI Provider classes.
Print a list of builtin providers with optional usage. Parameters ---------- show_query_types : bool, optional Show query types supported by providers, by default False as_list : bool, optional Return list of providers instead of printing to stdout. Note: if you specify `show_query_types` this will be printed irrespective of this parameter setting. Returns ------- Optional[List[str]] A list of provider names (if `return_list=True`)
Return dictionary of loaded providers. Returns ------- Dict[str, TIProvider] [description]
Lookup single IoC in active providers. Parameters ---------- observable : str IoC observable (`ioc` is also an alias for observable) ioc_type : str, optional One of IoCExtract.IoCType, by default None If none, the IoC type will be inferred ioc_query_type: str, optional The ioc query type (e.g. rep, info, malware) providers: List[str] Explicit list of providers to use prov_scope : str, optional Use "primary", "secondary" or "all" providers, by default "primary" kwargs : Additional arguments passed to the underlying provider(s) Returns ------- Tuple[bool, List[Tuple[str, LookupResult]]] The result returned as a tuple(bool, list): bool indicates whether a TI record was found in any provider list has an entry for each provider result
Lookup a collection of IoCs. Parameters ---------- data : Union[pd.DataFrame, Mapping[str, str], Iterable[str]] Data input in one of three formats: 1. Pandas dataframe (you must supply the column name in `obs_col` parameter) 2. Mapping (e.g. a dict) of [observable, IoCType] 3. Iterable of observables - IoCTypes will be inferred obs_col : str, optional DataFrame column to use for observables, by default None ioc_type_col : str, optional DataFrame column to use for IoCTypes, by default None ioc_query_type: str, optional The ioc query type (e.g. rep, info, malware) providers: List[str] Explicit list of providers to use prov_scope : str, optional Use "primary", "secondary" or "all" providers, by default "primary" kwargs : Additional arguments passed to the underlying provider(s) Returns ------- pd.DataFrame DataFrame of results
Return loaded provider status. Returns ------- Iterable[str] List of providers and descriptions.
Print usage of loaded providers.
Reload provider settings from config.
Reload providers based on currrent settings in config. Parameters ---------- clear_keyring : bool, optional Clears any secrets cached in keyring, by default False
Return DataFrame representation of IoC Lookup response. Parameters ---------- ioc_lookup : Tuple[bool, List[Tuple[str, LookupResult]]] Output from `lookup_ioc` Returns ------- pd.DataFrame The response as a DataFrame with a row for each provider response.
The msticpy TI Provider library can lookup IoCs in multiple providers.
The list below shows the current set of providers.
TILookup.list_available_providers()
AzSTI OPR OTX Tor VirusTotal XForce
You can view the list of supported query types for each provider with the show_query_types=True
parameter
TILookup.list_available_providers(show_query_types=True)
AzSTI Azure Sentinel TI provider class. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv6 ioc_type=linux_path ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=windows_path OPR Open PageRank Lookup. Supported query types: ioc_type=dns OTX AlientVault OTX Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=geo ioc_type=dns, ioc_query_type=passivedns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=geo ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=geo ioc_type=ipv6, ioc_query_type=passivedns ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url Tor Tor Exit Nodes Lookup. Supported query types: ioc_type=ipv4 VirusTotal VirusTotal Lookup. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=ipv4 ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url XForce IBM XForce Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=malware ioc_type=dns, ioc_query_type=passivedns ioc_type=dns, ioc_query_type=whois ioc_type=file_hash ioc_type=hostname, ioc_query_type=whois ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=malware ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv4, ioc_query_type=rep ioc_type=ipv4, ioc_query_type=whois ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=malware ioc_type=ipv6, ioc_query_type=passivedns ioc_type=ipv6, ioc_query_type=rep ioc_type=ipv6, ioc_query_type=whois ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=url, ioc_query_type=malware
Calling TILookup with no parameters will load all of the available providers
that have a configuration entry in msticpyconfig.yaml
(see next section)
# load all configured providers
ti_lookup = TILookup()
ti_lookup.provider_status
# Restricting which providers get loaded
#ti_lookup = TILookup(providers=["VirusTotal", "XForce"])
Please wait. Loading Kqlmagic extension...
Using Open PageRank. See https://www.domcop.com/openpagerank/what-is-openpagerank
['OTX - AlientVault OTX Lookup. (primary)', 'OPR - Open PageRank Lookup. (primary)', 'Tor - Tor Exit Nodes Lookup. (primary)', 'VirusTotal - VirusTotal Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'AzSTI - Azure Sentinel TI provider class. (secondary)']
You can configure primary and secondary providers. Primary providers are used by default.
You may need to supply an authorization (API) key and in some cases a user ID for each provider.
For LogAnalytics/Azure Sentinel providers, you will need the workspace ID and tenant ID and will need to authenticate in order to access the data (although if you have an existing authenticated connection with the same workspace/tenant, this connection will be re-used).
The configuration file is read from the current directory.
Alternatively, you can specify a location for this file in an environment variable MSTICPYCONFIG
.
If you need to create a config file, uncomment the lines in the following cell.
Delete any provider entries that you do not want to use and add the missing parameters for your providers.
# %%writefile msticpyconfig.yaml
# QueryDefinitions:
# TIProviders:
# OTX:
# Args:
# AuthKey: "your-otx-key"
# Primary: True
# Provider: "OTX" # Explicitly name provider to override
# VirusTotal:
# Args:
# AuthKey: "your-vt-key"
# Primary: True
# Provider: "VirusTotal"
# XForce:
# Args:
# ApiID: "your-xforce-id"
# AuthKey: "your-xforce-key"
# Primary: True
# Provider: "XForce"
# AzureSentinel:
# # Note if you do not specify any settings in the Args key for the AzureSentinel
# # provider, it will default to using your default Azure Sentinel workspace.
# Args:
# WorkspaceID: "your-azure-sentinel-workspace-id"
# TenantID: "your-azure-sentinel-tenant-id"
# Primary: True
# Provider: "AzSTI"
Reload providers to pick up new settings
ti_lookup.reload_providers()
ti_lookup.provider_status
Settings reloaded. Use reload_providers to update settings for loaded providers.
Using Open PageRank. See https://www.domcop.com/openpagerank/what-is-openpagerank
['OTX - AlientVault OTX Lookup. (primary)', 'OPR - Open PageRank Lookup. (primary)', 'Tor - Tor Exit Nodes Lookup. (primary)', 'VirusTotal - VirusTotal Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'AzSTI - Azure Sentinel TI provider class. (secondary)']
To lookup a single IoC.
ti_lookup.lookup_ioc(
observable: str = None,
ioc_type: str = None,
ioc_query_type: str = None,
providers: List[str] = None,
prov_scope: str = 'primary',
**kwargs,
) -> Tuple[bool, List[Tuple[str, msticpy.sectools.tiproviders.ti_provider_base.LookupResult]]]
Lookup single IoC in active providers.
Parameters
----------
observable : str
IoC observable
(`ioc` is also an alias for observable)
ioc_type : str, optional
One of IoCExtract.IoCType, by default None
If none, the IoC type will be inferred
ioc_query_type: str, optional
The ioc query type (e.g. rep, info, malware)
providers: List[str]
Explicit list of providers to use
prov_scope : str, optional
Use primary, secondary or all providers, by default "primary"
kwargs :
Additional arguments passed to the underlying provider(s)
Returns
-------
Tuple[bool, List[Tuple[str, LookupResult]]]
The result returned as a tuple(bool, list):
bool indicates whether a TI record was found in any provider
list has an entry for each provider result
# Uncomment this and run to see the document string
# ti_lookup.lookup_ioc?
And show the output
result = ti_lookup.lookup_ioc(observable="52.183.120.194", providers=["AzSTI", "XForce"])
ti_lookup.result_to_df(result)
Ioc | IocType | QuerySubtype | Provider | Result | Severity | Details | RawResult | Reference | Status | |
---|---|---|---|---|---|---|---|---|---|---|
XForce | 52.183.120.194 | ipv4 | None | XForce | True | warning | {'score': 1, 'cats': {}, 'categoryDescriptions': {}, 'reason': 'Regional Internet Registry', 're... | {'ip': '52.183.120.194', 'history': [{'created': '2012-03-22T07:26:00.000Z', 'reason': 'Regional... | https://api.xforce.ibmcloud.com/ipr/52.183.120.194 | 0 |
AzSTI | 52.183.120.194 | ipv4 | None | AzSTI | False | information | Not found. | None | None | 0 |
result = ti_lookup.lookup_ioc(observable="52.183.120.194")
ti_lookup.result_to_df(result).T
OTX | OPR | Tor | VirusTotal | XForce | |
---|---|---|---|---|---|
Ioc | 52.183.120.194 | 52.183.120.194 | 52.183.120.194 | 52.183.120.194 | 52.183.120.194 |
IocType | ipv4 | ipv4 | ipv4 | ipv4 | ipv4 |
QuerySubtype | None | None | None | None | None |
Provider | OTX | OPR | Tor | VirusTotal | XForce |
Result | True | False | True | True | True |
Severity | information | information | information | information | warning |
Details | {'pulse_count': 0, 'sections_available': ['general', 'geo', 'reputation', 'url_list', 'passive_d... | IoC type ipv4 not supported. | Not found. | {'verbose_msg': 'IP address in dataset', 'response_code': 1, 'detected_urls': [], 'positives': 0} | {'score': 1, 'cats': {}, 'categoryDescriptions': {}, 'reason': 'Regional Internet Registry', 're... |
RawResult | {'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list',... | None | None | {'https_certificate_date': 1569589456, 'whois': 'NetRange: 52.145.0.0 - 52.191.255.255 CIDR: 52.... | {'ip': '52.183.120.194', 'history': [{'created': '2012-03-22T07:26:00.000Z', 'reason': 'Regional... |
Reference | https://otx.alienvault.com/api/v1/indicators/IPv4/52.183.120.194/general | None | https://check.torproject.org/exit-addresses | https://www.virustotal.com/vtapi/v2/ip-address/report | https://api.xforce.ibmcloud.com/ipr/52.183.120.194 |
Status | 0 | 1 | 0 | 0 | 0 |
import pprint
pp = pprint.PrettyPrinter(indent=2)
result, details = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])
# the details is a list (since there could be multiple responses for an IoC)
for provider, detail in details:
print(provider)
detail.summary
# Un-comment to view raw response
# print("\nRaw Results")
# pp.pprint(detail.raw_result)
OTX ioc: 38.75.137.9 ( ipv4 ) result: True { 'names': ['Underminer.EK - Exploit Kit IOC Feed', 'Underminer EK'], 'pulse_count': 2, 'references': [ [], [ 'https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/']], 'tags': [['Underminer.EK'], []]} reference: https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general
result = ti_lookup.lookup_ioc(observable="38.75.137.9", providers=["OTX"])
ti_lookup.result_to_df(result).T
OTX | |
---|---|
Ioc | 38.75.137.9 |
IocType | ipv4 |
QuerySubtype | None |
Provider | OTX |
Result | True |
Severity | high |
Details | {'pulse_count': 2, 'names': ['Underminer.EK - Exploit Kit IOC Feed', 'Underminer EK'], 'tags': [... |
RawResult | {'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list',... |
Reference | https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/general |
Status | 0 |
# Extract a single field (RawResult) from the dataframe (.iloc[0] is to select the row)
ti_lookup.result_to_df(result)["RawResult"].iloc[0]
{'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list', 'http_scans'], 'city': 'Los Angeles', 'area_code': 0, 'accuracy_radius': 1000, 'pulse_info': {'count': 2, 'references': ['https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'], 'pulses': [{'indicator_type_counts': {'domain': 1, 'hostname': 1, 'IPv4': 7}, 'pulse_source': 'api', 'TLP': 'white', 'description': 'IPs and hostnames for the Exploit Kit known as Underminer.EK.', 'subscriber_count': 257, 'tags': ['Underminer.EK'], 'export_count': 1, 'malware_families': [], 'is_modified': True, 'upvotes_count': 0, 'modified_text': '248 days ago ', 'is_subscribing': None, 'references': [], 'targeted_countries': [], 'groups': [], 'vote': 0, 'validator_count': 0, 'threat_hunter_scannable': True, 'is_author': False, 'adversary': '', 'id': '5db816cba3e59aeced1fad16', 'industries': [], 'locked': False, 'name': 'Underminer.EK - Exploit Kit IOC Feed', 'created': '2019-10-29T10:39:07.558000', 'related_indicator_is_active': 1, 'threat_hunter_has_agents': 1, 'cloned_from': None, 'downvotes_count': 0, 'modified': '2019-11-04T13:21:54.514000', 'comment_count': 0, 'indicator_count': 9, 'attack_ids': [], 'in_group': False, 'follower_count': 0, 'votes_count': 0, 'author': {'username': 'otxrobottwo', 'is_subscribed': False, 'avatar_url': 'https://otx20-web-media.s3.amazonaws.com/media/avatars/user_78495/resized/80/avatar_ba5a8acdbd.png', 'is_following': False, 'id': '78495'}, 'related_indicator_type': 'IPv4', 'public': 1}, {'indicator_type_counts': {'URL': 16, 'FileHash-MD5': 5, 'IPv4': 3}, 'pulse_source': 'web', 'TLP': 'white', 'description': '', 'subscriber_count': 26, 'tags': [], 'export_count': 0, 'malware_families': [], 'is_modified': False, 'upvotes_count': 0, 'modified_text': '344 days ago ', 'is_subscribing': None, 'references': ['https://blog.malwarebytes.com/threat-analysis/2019/07/exploit-kits-summer-2019-review/'], 'targeted_countries': [], 'groups': [], 'vote': 0, 'validator_count': 0, 'threat_hunter_scannable': True, 'is_author': False, 'adversary': '', 'id': '5d41d77901a2f8c6e9b650e9', 'industries': [], 'locked': False, 'name': 'Underminer EK', 'created': '2019-07-31T18:01:29.744000', 'related_indicator_is_active': 1, 'threat_hunter_has_agents': 1, 'cloned_from': None, 'downvotes_count': 0, 'modified': '2019-07-31T18:01:29.744000', 'comment_count': 0, 'indicator_count': 24, 'attack_ids': [], 'in_group': False, 'follower_count': 0, 'votes_count': 0, 'author': {'username': 'mattvittitoe', 'is_subscribed': False, 'avatar_url': 'https://otx.alienvault.com/assets/images/default-avatar.png', 'is_following': False, 'id': '79520'}, 'related_indicator_type': 'IPv4', 'public': 1}]}, 'continent_code': 'NA', 'latitude': 34.0584, 'postal_code': '90017', 'longitude': -118.278, 'country_code': 'US', 'flag_url': '/assets/images/flags/us.png', 'asn': 'AS63023 AS-GLOBALTELEHOST', 'city_data': True, 'indicator': '38.75.137.9', 'subdivision': 'CA', 'whois': 'http://whois.domaintools.com/38.75.137.9', 'type_title': 'IPv4', 'region': 'CA', 'charset': 0, 'dma_code': 803, 'country_code3': 'USA', 'country_code2': 'US', 'reputation': 0, 'base_indicator': {'indicator': '38.75.137.9', 'description': '', 'title': '', 'access_reason': '', 'access_type': 'public', 'content': '', 'type': 'IPv4', 'id': 2127020821}, 'country_name': 'United States of America', 'type': 'IPv4', 'flag_title': 'United States of America'}
result = ti_lookup.lookup_ioc(observable="188.127.231.124")
ti_lookup.result_to_df(result)
Ioc | IocType | QuerySubtype | Provider | Result | Severity | Details | RawResult | Reference | Status | |
---|---|---|---|---|---|---|---|---|---|---|
OTX | 188.127.231.124 | ipv4 | None | OTX | True | high | {'pulse_count': 4, 'names': ['Locky Ransomware C2 IP blocklist (LY_C2_IPBL)', 'Malicious IP', 'F... | {'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list',... | https://otx.alienvault.com/api/v1/indicators/IPv4/188.127.231.124/general | 0 |
OPR | 188.127.231.124 | ipv4 | None | OPR | False | information | IoC type ipv4 not supported. | None | None | 1 |
Tor | 188.127.231.124 | ipv4 | None | Tor | True | information | Not found. | None | https://check.torproject.org/exit-addresses | 0 |
VirusTotal | 188.127.231.124 | ipv4 | None | VirusTotal | True | information | {'verbose_msg': 'IP address in dataset', 'response_code': 1, 'detected_urls': [], 'positives': 0... | {'undetected_downloaded_samples': [{'date': '2018-01-09 20:05:03', 'positives': 0, 'total': 71, ... | https://www.virustotal.com/vtapi/v2/ip-address/report | 0 |
XForce | 188.127.231.124 | ipv4 | None | XForce | True | warning | {'score': 1, 'cats': {}, 'categoryDescriptions': {}, 'reason': 'Regional Internet Registry', 're... | {'ip': '188.127.231.124', 'history': [{'created': '2012-03-22T07:26:00.000Z', 'reason': 'Regiona... | https://api.xforce.ibmcloud.com/ipr/188.127.231.124 | 0 |
This shows the supported IoC Types.
In some cases an IoC type will also support special types of sub-query such as geo-ip and passive-dns
display(ti_lookup.provider_status)
ti_lookup.loaded_providers["AzSTI"].usage()
['OTX - AlientVault OTX Lookup. (primary)', 'OPR - Open PageRank Lookup. (primary)', 'Tor - Tor Exit Nodes Lookup. (primary)', 'VirusTotal - VirusTotal Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'AzSTI - Azure Sentinel TI provider class. (secondary)']
Azure Sentinel TI provider class. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv6 ioc_type=linux_path ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=windows_path
ti_lookup.provider_usage()
Primary providers ----------------- Provider class: OTX AlientVault OTX Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=geo ioc_type=dns, ioc_query_type=passivedns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=geo ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=geo ioc_type=ipv6, ioc_query_type=passivedns ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url Provider class: OPR Open PageRank Lookup. Supported query types: ioc_type=dns Provider class: Tor Tor Exit Nodes Lookup. Supported query types: ioc_type=ipv4 Provider class: VirusTotal VirusTotal Lookup. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=ipv4 ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url Provider class: XForce IBM XForce Lookup. Supported query types: ioc_type=dns ioc_type=dns, ioc_query_type=malware ioc_type=dns, ioc_query_type=passivedns ioc_type=dns, ioc_query_type=whois ioc_type=file_hash ioc_type=hostname, ioc_query_type=whois ioc_type=ipv4 ioc_type=ipv4, ioc_query_type=malware ioc_type=ipv4, ioc_query_type=passivedns ioc_type=ipv4, ioc_query_type=rep ioc_type=ipv4, ioc_query_type=whois ioc_type=ipv6 ioc_type=ipv6, ioc_query_type=malware ioc_type=ipv6, ioc_query_type=passivedns ioc_type=ipv6, ioc_query_type=rep ioc_type=ipv6, ioc_query_type=whois ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=url, ioc_query_type=malware Secondary providers ------------------- Provider class: AzSTI Azure Sentinel TI provider class. Supported query types: ioc_type=dns ioc_type=file_hash ioc_type=hostname ioc_type=ipv4 ioc_type=ipv6 ioc_type=linux_path ioc_type=md5_hash ioc_type=sha1_hash ioc_type=sha256_hash ioc_type=url ioc_type=windows_path
result = ti_lookup.lookup_ioc(observable="38.75.137.9", ico_type="ipv4", ioc_query_type="passivedns", providers=["XForce"])
print(result)
print("\nProvider result:")
result[1][0][1].raw_result
(True, [('XForce', LookupResult(ioc='38.75.137.9', ioc_type='ipv4', safe_ioc='38.75.137.9', query_subtype='passivedns', provider='XForce', result=True, severity='information', details={'records': 1}, raw_result={'Passive': {'query': '0x00000000000000000000ffff264b8909', 'records': []}, 'RDNS': ['9-137-75-38.clients.gthost.com'], 'total_rows': 1}, reference='https://api.xforce.ibmcloud.com/resolve/38.75.137.9', status=0))]) Provider result:
{'Passive': {'query': '0x00000000000000000000ffff264b8909', 'records': []}, 'RDNS': ['9-137-75-38.clients.gthost.com'], 'total_rows': 1}
result = ti_lookup.lookup_ioc(observable="38.75.137.9", ico_type="ipv4", ioc_query_type="geo", providers=["OTX"])
print(result)
print("\nProvider result:")
result[1][0][1].raw_result
(True, [('OTX', LookupResult(ioc='38.75.137.9', ioc_type='ipv4', safe_ioc='38.75.137.9', query_subtype='geo', provider='OTX', result=True, severity='information', details={}, raw_result={'city_data': True, 'accuracy_radius': 1000, 'area_code': 0, 'continent_code': 'NA', 'country_name': 'United States of America', 'postal_code': '90017', 'dma_code': 803, 'country_code': 'US', 'flag_url': '/assets/images/flags/us.png', 'asn': 'AS63023 AS-GLOBALTELEHOST', 'city': 'Los Angeles', 'subdivision': 'CA', 'region': 'CA', 'charset': 0, 'longitude': -118.278, 'country_code3': 'USA', 'country_code2': 'US', 'latitude': 34.0584, 'flag_title': 'United States of America'}, reference='https://otx.alienvault.com/api/v1/indicators/IPv4/38.75.137.9/geo', status=0))]) Provider result:
{'city_data': True, 'accuracy_radius': 1000, 'area_code': 0, 'continent_code': 'NA', 'country_name': 'United States of America', 'postal_code': '90017', 'dma_code': 803, 'country_code': 'US', 'flag_url': '/assets/images/flags/us.png', 'asn': 'AS63023 AS-GLOBALTELEHOST', 'city': 'Los Angeles', 'subdivision': 'CA', 'region': 'CA', 'charset': 0, 'longitude': -118.278, 'country_code3': 'USA', 'country_code2': 'US', 'latitude': 34.0584, 'flag_title': 'United States of America'}
If you do a lookup without specifying a type, TILookup will try to infer the type by matching regexes. There are patterns for all supported types but there are some caveats:
If you know the type that you want to look up, it is always better to explicitly include it.
ioc_type
parameter.{ioc_observable: ioc_type}
Signature:
ti_lookup.lookup_iocs(
data: Union[pandas.core.frame.DataFrame, Mapping[str, str], Iterable[str]],
obs_col: str = None,
ioc_type_col: str = None,
ioc_query_type: str = None,
providers: List[str] = None,
prov_scope: str = 'primary',
**kwargs,
) -> pandas.core.frame.DataFrame
Lookup a collection of IoCs.
Parameters
----------
data : Union[pd.DataFrame, Mapping[str, str], Iterable[str]]
Data input in one of three formats:
1. Pandas dataframe (you must supply the column name in
`obs_col` parameter)
2. Mapping (e.g. a dict) of [observable, IoCType]
3. Iterable of observables - IoCTypes will be inferred
obs_col : str, optional
DataFrame column to use for observables, by default None
ioc_type_col : str, optional
DataFrame column to use for IoCTypes, by default None
ioc_query_type: str, optional
The ioc query type (e.g. rep, info, malware)
providers: List[str]
Explicit list of providers to use
prov_scope : str, optional
Use primary, secondary or all providers, by default "primary"
kwargs :
Additional arguments passed to the underlying provider(s)
Returns
-------
pd.DataFrame
DataFrame of results
# Uncomment this and run to see the document string
# ti_lookup.lookup_iocs?
ioc_ips = [
"51.75.29.61",
"33.44.55.66"
"52.183.120.194",
"13.91.229.209",
"1.2.3.4",
"52.167.223.49",
"1.2.3.5",
]
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI")
Ioc | IocType | QuerySubtype | Reference | Result | Details | Status | Severity | Provider | |
---|---|---|---|---|---|---|---|---|---|
0 | 51.75.29.61 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2020-06-09T22:21:31.328494Z) | w... | False | Not found. | 0 | 0 | AzSTI |
1 | 13.91.229.209 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2020-06-09T22:21:31.328494Z) | w... | False | Not found. | 0 | 0 | AzSTI |
2 | 52.167.223.49 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2020-06-09T22:21:31.328494Z) | w... | False | Not found. | 0 | 0 | AzSTI |
3 | 1.2.3.4 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2020-06-09T22:21:31.328494Z) | w... | False | Not found. | 0 | 0 | AzSTI |
4 | 1.2.3.5 | ipv4 | None | ThreatIntelligenceIndicator | where TimeGenerated >= datetime(2020-06-09T22:21:31.328494Z) | w... | False | Not found. | 0 | 0 | AzSTI |
Output sorted by IoC
Note that these URLs were picked randomly from the TI databases of the three providers used. In most cases the IoC is found by only that provider, which
ioc_urls = [
"http://cheapshirts.us/zVnMrG.php",
"http://chinasymbolic.com/i9jnrc",
"https://hotel-bristol.lu/dlry/MAnJIPnY/",
"http://businesstobuy.net",
"http://append.pl/srh9xsz",
"http://104.248.196.145/apache2",
"http://ajaraheritage.ge/g7cberv",
"http://cic-integration.com/hjy93JNBasdas",
"https://google.com", # benign
"https://microsoft.com", # benign
"https://python.org", # benign
]
results = ti_lookup.lookup_iocs(data=ioc_urls)
results.sort_values("Ioc")
Ioc | IocType | SafeIoc | QuerySubtype | Provider | Result | Severity | Details | RawResult | Reference | Status | |
---|---|---|---|---|---|---|---|---|---|---|---|
5 | http://104.248.196.145/apache2 | url | http://104.248.196.145/apache2 | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
5 | http://104.248.196.145/apache2 | url | http://104.248.196.145/apache2 | None | XForce | False | information | Not found. | <Response [404]> | https://api.xforce.ibmcloud.com/url/http://104.248.196.145/apache2 | 404 |
5 | http://104.248.196.145/apache2 | url | http%3A%2F%2F104.248.196.145%2Fapache2 | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'url_list', 'http_scans', 'screenshot']} | {'indicator': 'http://104.248.196.145/apache2', 'alexa': '', 'whois': '', 'sections': ['general'... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2F104.248.196.145%2Fapache2/general | 0 |
6 | http://ajaraheritage.ge/g7cberv | url | http://ajaraheritage.ge/g7cberv | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'ajaraheritage.ge', 'cats': {}, 'score': None, 'categoryDescriptions': {}}, '... | https://api.xforce.ibmcloud.com/url/http://ajaraheritage.ge/g7cberv | 0 |
6 | http://ajaraheritage.ge/g7cberv | url | http%3A%2F%2Fajaraheritage.ge%2Fg7cberv | None | OTX | True | high | {'pulse_count': 2, 'names': ['Locky Ransomware Distribution Sites URL blocklist (LY_DS_URLBL)', ... | {'indicator': 'http://ajaraheritage.ge/g7cberv', 'alexa': 'http://www.alexa.com/siteinfo/ajarahe... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2Fajaraheritage.ge%2Fg7cberv/general | 0 |
6 | http://ajaraheritage.ge/g7cberv | url | http://ajaraheritage.ge/g7cberv | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
4 | http://append.pl/srh9xsz | url | http%3A%2F%2Fappend.pl%2Fsrh9xsz | None | OTX | True | warning | {'pulse_count': 1, 'names': ['Locky Ransomware Distribution Sites URL blocklist (LY_DS_URLBL)'],... | {'indicator': 'http://append.pl/srh9xsz', 'alexa': 'http://www.alexa.com/siteinfo/append.pl', 'w... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2Fappend.pl%2Fsrh9xsz/general | 0 |
4 | http://append.pl/srh9xsz | url | http://append.pl/srh9xsz | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'append.pl', 'cats': {'Software / Hardware': True}, 'score': 1, 'categoryDesc... | https://api.xforce.ibmcloud.com/url/http://append.pl/srh9xsz | 0 |
4 | http://append.pl/srh9xsz | url | http://append.pl/srh9xsz | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
3 | http://businesstobuy.net | url | http%3A%2F%2Fbusinesstobuy.net | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'url_list', 'http_scans', 'screenshot']} | {'indicator': 'http://businesstobuy.net', 'alexa': 'http://www.alexa.com/siteinfo/businesstobuy.... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2Fbusinesstobuy.net/general | 0 |
3 | http://businesstobuy.net | url | http://businesstobuy.net | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'businesstobuy.net', 'cats': {'Phishing URLs': True}, 'score': 10, 'categoryD... | https://api.xforce.ibmcloud.com/url/http://businesstobuy.net | 0 |
3 | http://businesstobuy.net | url | http://businesstobuy.net | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
0 | http://cheapshirts.us/zVnMrG.php | url | http://cheapshirts.us/zVnMrG.php | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'cheapshirts.us', 'cats': {'Shopping': True, 'Auctions / Classified Ads': Tru... | https://api.xforce.ibmcloud.com/url/http://cheapshirts.us/zVnMrG.php | 0 |
0 | http://cheapshirts.us/zVnMrG.php | url | http%3A%2F%2Fcheapshirts.us%2FzVnMrG.php | None | OTX | True | high | {'pulse_count': 7, 'names': ['CryptoWall Ransomware C2 URL blocklist (CW_C2_URLBL)', 'CryptoWall... | {'indicator': 'http://cheapshirts.us/zVnMrG.php', 'alexa': 'http://www.alexa.com/siteinfo/cheaps... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2Fcheapshirts.us%2FzVnMrG.php/general | 0 |
0 | http://cheapshirts.us/zVnMrG.php | url | http://cheapshirts.us/zVnMrG.php | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
1 | http://chinasymbolic.com/i9jnrc | url | http://chinasymbolic.com/i9jnrc | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
1 | http://chinasymbolic.com/i9jnrc | url | http%3A%2F%2Fchinasymbolic.com%2Fi9jnrc | None | OTX | True | high | {'pulse_count': 2, 'names': ['Locky Ransomware Distribution Sites URL blocklist (LY_DS_URLBL)', ... | {'indicator': 'http://chinasymbolic.com/i9jnrc', 'alexa': 'http://www.alexa.com/siteinfo/chinasy... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2Fchinasymbolic.com%2Fi9jnrc/general | 0 |
1 | http://chinasymbolic.com/i9jnrc | url | http://chinasymbolic.com/i9jnrc | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'chinasymbolic.com', 'cats': {}, 'score': None, 'categoryDescriptions': {}}, ... | https://api.xforce.ibmcloud.com/url/http://chinasymbolic.com/i9jnrc | 0 |
7 | http://cic-integration.com/hjy93JNBasdas | url | http://cic-integration.com/hjy93JNBasdas | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
7 | http://cic-integration.com/hjy93JNBasdas | url | http%3A%2F%2Fcic-integration.com%2Fhjy93JNBasdas | None | OTX | True | warning | {'pulse_count': 1, 'names': ['Locky Ransomware Distribution Sites URL blocklist (LY_DS_URLBL)'],... | {'indicator': 'http://cic-integration.com/hjy93JNBasdas', 'alexa': 'http://www.alexa.com/siteinf... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2Fcic-integration.com%2Fhjy93JNBasda... | 0 |
7 | http://cic-integration.com/hjy93JNBasdas | url | http://cic-integration.com/hjy93JNBasdas | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'cic-integration.com', 'cats': {}, 'score': None, 'categoryDescriptions': {}}... | https://api.xforce.ibmcloud.com/url/http://cic-integration.com/hjy93JNBasdas | 0 |
8 | https://google.com | url | https://google.com | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
8 | https://google.com | url | https%3A%2F%2Fgoogle.com | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'url_list', 'http_scans', 'screenshot']} | {'indicator': 'https://google.com', 'alexa': 'http://www.alexa.com/siteinfo/google.com', 'whois'... | https://otx.alienvault.com/api/v1/indicators/url/https%3A%2F%2Fgoogle.com/general | 0 |
8 | https://google.com | url | https://google.com | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'https://google.com', 'cats': {'Search Engines / Web Catalogues / Portals': T... | https://api.xforce.ibmcloud.com/url/https://google.com | 0 |
2 | https://hotel-bristol.lu/dlry/MAnJIPnY/ | url | https://hotel-bristol.lu/dlry/MAnJIPnY/ | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
2 | https://hotel-bristol.lu/dlry/MAnJIPnY/ | url | https://hotel-bristol.lu/dlry/MAnJIPnY/ | None | XForce | False | information | Not found. | <Response [404]> | https://api.xforce.ibmcloud.com/url/https://hotel-bristol.lu/dlry/MAnJIPnY/ | 404 |
2 | https://hotel-bristol.lu/dlry/MAnJIPnY/ | url | https%3A%2F%2Fhotel-bristol.lu%2Fdlry%2FMAnJIPnY%2F | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'url_list', 'http_scans', 'screenshot']} | {'indicator': 'https://hotel-bristol.lu/dlry/MAnJIPnY', 'alexa': 'http://www.alexa.com/siteinfo/... | https://otx.alienvault.com/api/v1/indicators/url/https%3A%2F%2Fhotel-bristol.lu%2Fdlry%2FMAnJIPn... | 0 |
9 | https://microsoft.com | url | https://microsoft.com | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
9 | https://microsoft.com | url | https%3A%2F%2Fmicrosoft.com | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'url_list', 'http_scans', 'screenshot']} | {'indicator': 'https://microsoft.com', 'alexa': 'http://www.alexa.com/siteinfo/microsoft.com', '... | https://otx.alienvault.com/api/v1/indicators/url/https%3A%2F%2Fmicrosoft.com/general | 0 |
9 | https://microsoft.com | url | https://microsoft.com | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'microsoft.com', 'cats': {'Software / Hardware': True, 'General Business': Tr... | https://api.xforce.ibmcloud.com/url/https://microsoft.com | 0 |
10 | https://python.org | url | https://python.org | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
10 | https://python.org | url | https%3A%2F%2Fpython.org | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'url_list', 'http_scans', 'screenshot']} | {'indicator': 'https://python.org', 'alexa': 'http://www.alexa.com/siteinfo/python.org', 'whois'... | https://otx.alienvault.com/api/v1/indicators/url/https%3A%2F%2Fpython.org/general | 0 |
10 | https://python.org | url | https://python.org | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'python.org', 'cats': {'Software / Hardware': True}, 'score': 1, 'application... | https://api.xforce.ibmcloud.com/url/https://python.org | 0 |
ioc_mixed = [
"http://104.248.196.145/apache2",
"http://ajaraheritage.ge/g7cberv",
"http://cic-integration.com/hjy93JNBasdas",
"51.75.29.61",
"33.44.55.66",
"52.183.120.194",
"f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092",
"cc2db822f652ca67038ba7cca8a8bde3",
"ajaraheritage.ge",
]
results = ti_lookup.lookup_iocs(data=ioc_mixed)
results
Ioc | IocType | SafeIoc | QuerySubtype | Provider | Result | Severity | Details | RawResult | Reference | Status | |
---|---|---|---|---|---|---|---|---|---|---|---|
0 | http://104.248.196.145/apache2 | url | http%3A%2F%2F104.248.196.145%2Fapache2 | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'url_list', 'http_scans', 'screenshot']} | {'indicator': 'http://104.248.196.145/apache2', 'alexa': '', 'whois': '', 'sections': ['general'... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2F104.248.196.145%2Fapache2/general | 0 |
1 | http://ajaraheritage.ge/g7cberv | url | http%3A%2F%2Fajaraheritage.ge%2Fg7cberv | None | OTX | True | high | {'pulse_count': 2, 'names': ['Locky Ransomware Distribution Sites URL blocklist (LY_DS_URLBL)', ... | {'indicator': 'http://ajaraheritage.ge/g7cberv', 'alexa': 'http://www.alexa.com/siteinfo/ajarahe... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2Fajaraheritage.ge%2Fg7cberv/general | 0 |
2 | http://cic-integration.com/hjy93JNBasdas | url | http%3A%2F%2Fcic-integration.com%2Fhjy93JNBasdas | None | OTX | True | warning | {'pulse_count': 1, 'names': ['Locky Ransomware Distribution Sites URL blocklist (LY_DS_URLBL)'],... | {'indicator': 'http://cic-integration.com/hjy93JNBasdas', 'alexa': 'http://www.alexa.com/siteinf... | https://otx.alienvault.com/api/v1/indicators/url/http%3A%2F%2Fcic-integration.com%2Fhjy93JNBasda... | 0 |
3 | 51.75.29.61 | ipv4 | 51.75.29.61 | None | OTX | True | high | {'pulse_count': 49, 'names': ['CYBSEC-TIA Bad IPs', '2020-07-02 Fail2ban b3478ecb-279e-4ad8-864b... | {'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list',... | https://otx.alienvault.com/api/v1/indicators/IPv4/51.75.29.61/general | 0 |
4 | 33.44.55.66 | ipv4 | 33.44.55.66 | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'geo', 'reputation', 'url_list', 'passive_d... | {'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list',... | https://otx.alienvault.com/api/v1/indicators/IPv4/33.44.55.66/general | 0 |
5 | 52.183.120.194 | ipv4 | 52.183.120.194 | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'geo', 'reputation', 'url_list', 'passive_d... | {'sections': ['general', 'geo', 'reputation', 'url_list', 'passive_dns', 'malware', 'nids_list',... | https://otx.alienvault.com/api/v1/indicators/IPv4/52.183.120.194/general | 0 |
6 | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092 | sha256_hash | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092 | None | OTX | True | high | {'pulse_count': 3, 'names': ['Emotet IOCs 2/4/2019', 'Emotet IOCs 2/1/2019', 'Emotet IOCs 1/31/2... | {'indicator': 'f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092', 'sections': ['... | https://otx.alienvault.com/api/v1/indicators/file/f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5... | 0 |
7 | cc2db822f652ca67038ba7cca8a8bde3 | md5_hash | cc2db822f652ca67038ba7cca8a8bde3 | None | OTX | True | information | {'pulse_count': 0, 'sections_available': ['general', 'analysis']} | {'indicator': 'cc2db822f652ca67038ba7cca8a8bde3', 'sections': ['general', 'analysis'], 'pulse_in... | https://otx.alienvault.com/api/v1/indicators/file/cc2db822f652ca67038ba7cca8a8bde3/general | 0 |
8 | ajaraheritage.ge | dns | ajaraheritage.ge | None | OTX | True | high | {'pulse_count': 22, 'names': ['Blah', 'Blah', 'Blah', 'IOCs - 2020272054 - Huge Upload', 'IOCs -... | {'indicator': 'ajaraheritage.ge', 'alexa': 'http://www.alexa.com/siteinfo/ajaraheritage.ge', 'wh... | https://otx.alienvault.com/api/v1/indicators/domain/ajaraheritage.ge/general | 0 |
0 | ajaraheritage.ge | dns | None | OPR | True | information | {'rank': '4421759', 'page_rank': 3.18, 'error': ''} | {'status_code': 200, 'error': '', 'page_rank_integer': 3, 'page_rank_decimal': 3.18, 'rank': '44... | https://openpagerank.com/api/v1.0/getPageRank?domains[0]=ajaraheritage.ge | 0 | |
3 | 51.75.29.61 | ipv4 | 51.75.29.61 | None | Tor | True | information | Not found. | None | https://check.torproject.org/exit-addresses | 0 |
4 | 33.44.55.66 | ipv4 | 33.44.55.66 | None | Tor | True | information | Not found. | None | https://check.torproject.org/exit-addresses | 0 |
5 | 52.183.120.194 | ipv4 | 52.183.120.194 | None | Tor | True | information | Not found. | None | https://check.torproject.org/exit-addresses | 0 |
0 | http://104.248.196.145/apache2 | url | http://104.248.196.145/apache2 | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
1 | http://ajaraheritage.ge/g7cberv | url | http://ajaraheritage.ge/g7cberv | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
2 | http://cic-integration.com/hjy93JNBasdas | url | http://cic-integration.com/hjy93JNBasdas | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/url/report | 403 |
3 | 51.75.29.61 | ipv4 | 51.75.29.61 | None | VirusTotal | True | information | {'verbose_msg': 'IP address in dataset', 'response_code': 1, 'detected_urls': ['http://51.75.29.... | {'https_certificate_date': 1593601749, 'undetected_downloaded_samples': [{'date': '2020-07-02 20... | https://www.virustotal.com/vtapi/v2/ip-address/report | 0 |
4 | 33.44.55.66 | ipv4 | 33.44.55.66 | None | VirusTotal | True | information | {'verbose_msg': 'IP address in dataset', 'response_code': 1, 'detected_urls': [], 'positives': 0... | {'undetected_urls': [['http://33.44.55.66/77/77.66.66-55-55-44', '27bf8e5c90a431fb1c078f16949a83... | https://www.virustotal.com/vtapi/v2/ip-address/report | 0 |
5 | 52.183.120.194 | ipv4 | 52.183.120.194 | None | VirusTotal | True | information | {'verbose_msg': 'IP address in dataset', 'response_code': 1, 'detected_urls': [], 'positives': 0} | {'https_certificate_date': 1569589456, 'whois': 'NetRange: 52.145.0.0 - 52.191.255.255 CIDR: 52.... | https://www.virustotal.com/vtapi/v2/ip-address/report | 0 |
6 | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092 | sha256_hash | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092 | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/file/report | 403 |
7 | cc2db822f652ca67038ba7cca8a8bde3 | md5_hash | cc2db822f652ca67038ba7cca8a8bde3 | None | VirusTotal | False | information | Request forbidden. Allowed query rate may have been exceeded. | <Response [403]> | https://www.virustotal.com/vtapi/v2/file/report | 403 |
8 | ajaraheritage.ge | dns | ajaraheritage.ge | None | VirusTotal | False | information | No Content | <Response [204]> | https://www.virustotal.com/vtapi/v2/domain/report | 204 |
0 | http://104.248.196.145/apache2 | url | http://104.248.196.145/apache2 | None | XForce | False | information | Not found. | <Response [404]> | https://api.xforce.ibmcloud.com/url/http://104.248.196.145/apache2 | 404 |
1 | http://ajaraheritage.ge/g7cberv | url | http://ajaraheritage.ge/g7cberv | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'ajaraheritage.ge', 'cats': {}, 'score': None, 'categoryDescriptions': {}}, '... | https://api.xforce.ibmcloud.com/url/http://ajaraheritage.ge/g7cberv | 0 |
2 | http://cic-integration.com/hjy93JNBasdas | url | http://cic-integration.com/hjy93JNBasdas | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'cic-integration.com', 'cats': {}, 'score': None, 'categoryDescriptions': {}}... | https://api.xforce.ibmcloud.com/url/http://cic-integration.com/hjy93JNBasdas | 0 |
3 | 51.75.29.61 | ipv4 | 51.75.29.61 | None | XForce | True | warning | {'score': 1, 'cats': {}, 'categoryDescriptions': {}, 'reason': 'Regional Internet Registry', 're... | {'ip': '51.75.29.61', 'history': [{'created': '2012-06-20T07:03:00.000Z', 'reason': 'Regional In... | https://api.xforce.ibmcloud.com/ipr/51.75.29.61 | 0 |
4 | 33.44.55.66 | ipv4 | 33.44.55.66 | None | XForce | True | warning | {'score': 1, 'cats': {}, 'categoryDescriptions': {}, 'reason': 'Regional Internet Registry', 're... | {'ip': '33.44.55.66', 'history': [{'created': '2012-03-22T07:26:00.000Z', 'reason': 'Regional In... | https://api.xforce.ibmcloud.com/ipr/33.44.55.66 | 0 |
5 | 52.183.120.194 | ipv4 | 52.183.120.194 | None | XForce | True | warning | {'score': 1, 'cats': {}, 'categoryDescriptions': {}, 'reason': 'Regional Internet Registry', 're... | {'ip': '52.183.120.194', 'history': [{'created': '2012-03-22T07:26:00.000Z', 'reason': 'Regional... | https://api.xforce.ibmcloud.com/ipr/52.183.120.194 | 0 |
6 | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092 | sha256_hash | f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf3145865092 | None | XForce | True | high | {'risk': 'high', 'family': None, 'reasonDescription': 0} | {'malware': {'origins': {'external': {'source': 'reversingLabs', 'firstSeen': '2019-01-31T17:30:... | https://api.xforce.ibmcloud.com/malware/f8a7135496fd6168df5f0ea21c745db89ecea9accc29c5cf281cdf31... | 0 |
7 | cc2db822f652ca67038ba7cca8a8bde3 | md5_hash | cc2db822f652ca67038ba7cca8a8bde3 | None | XForce | True | high | {'risk': 'high', 'family': None, 'reasonDescription': 0} | {'malware': {'origins': {'external': {'source': 'reversingLabs', 'firstSeen': '2019-01-22T11:37:... | https://api.xforce.ibmcloud.com/malware/cc2db822f652ca67038ba7cca8a8bde3 | 0 |
8 | ajaraheritage.ge | dns | ajaraheritage.ge | None | XForce | True | information | {'score': 0, 'cats': None, 'categoryDescriptions': None, 'reason': None, 'reasonDescription': 0,... | {'result': {'url': 'ajaraheritage.ge', 'cats': {}, 'score': None, 'categoryDescriptions': {}}, '... | https://api.xforce.ibmcloud.com/url/ajaraheritage.ge | 0 |
To make it easier to walk through the returned results there is a browser. This shows you results aggregated by the IoC value (e.g. an individual IP Address or URL) for all providers.
For each provider that returns a result for an IoC, the summarized details
will be shown in a table below the browse list.
Click on Raw results from provider...
to see all returned data.
Note: the reference URL may not work if you have not authenticated to the service
The value of the selected IoC entry is available as ti_selector.value
You can match this back to the original results DataFrame as follows:
results[results["Ioc"] == ti_selector.value[0]]
from msticpy.nbtools.ti_browser import browse_results
ti_selector = browse_results(data=results, height="200px")
ti_selector
VBox(children=(Text(value='', description='Filter:', style=DescriptionStyle(description_width='initial')), Sel…
XForce | |
score | 1 |
cats | |
categoryDescriptions | |
reason | Regional Internet Registry |
reasonDescription | One of the five RIRs announced a (new) location mapping of the IP. |
tags | [] |
{'categoryDescriptions': {},
'cats': {},
'geo': {'country': 'United States', 'countrycode': 'US'},
'history': [{'categoryDescriptions': {},
'cats': {},
'created': '2012-03-22T07:26:00.000Z',
'geo': {'country': 'United States', 'countrycode': 'US'},
'ip': '33.0.0.0/8',
'reason': 'Regional Internet Registry',
'reasonDescription': 'One of the five RIRs announced a (new) '
'location mapping of the IP.',
'score': 1}],
'ip': '33.44.55.66',
'reason': 'Regional Internet Registry',
'reasonDescription': 'One of the five RIRs announced a (new) location mapping '
'of the IP.',
'score': 1,
'subnets': [{'categoryDescriptions': {},
'cats': {},
'created': '2012-03-22T07:26:00.000Z',
'geo': {'country': 'United States', 'countrycode': 'US'},
'ip': '33.0.0.0',
'reason': 'Regional Internet Registry',
'reasonDescription': 'One of the five RIRs announced a (new) '
'location mapping of the IP.',
'score': 1,
'subnet': '33.0.0.0/8'}],
'tags': []}
Some providers (currently only AzSTI) support time ranges so that you can specify specific periods to search for.
If a provider does not support time ranges, the parameters will be ignored
from datetime import datetime
search_origin = datetime(2019, 8, 5)
q_times = nbwidgets.QueryTime(units="hour", auto_display=True, origin_time=search_origin, max_after=24, max_before=24)
# Using this data range returned no results
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI", start=q_times.start, end=q_times.end).head()
from datetime import datetime
search_origin = datetime(2019, 8, 5)
q_times = nbwidgets.QueryTime(units="day", auto_display=True, origin_time=search_origin, max_after=24, max_before=24)
# Using a wider ranges produces results
ti_lookup.lookup_iocs(data=ioc_ips, providers="AzSTI", start=q_times.start, end=q_times.end)