Friday, January 26, 2024

Msticpy - Microsoft Threat Intelligence Security Tools

Microsoft Threat Intelligence Python Security Tools.

msticpy is a library for InfoSec investigation and hunting in Jupyter Notebooks. It includes functionality to:

  • query log data from multiple sources
  • enrich the data with Threat Intelligence, geolocations and Azure resource data
  • extract Indicators of Activity (IoA) from logs and unpack encoded data
  • perform sophisticated analysis such as anomalous session detection and time series decomposition
  • visualize data using interactive timelines, process trees and multi-dimensional Morph Charts

It also includes some time-saving notebook tools such as widgets to set query time boundaries, select and display items from lists, and configure the notebook environment.



The msticpy package was initially developed to support Jupyter Notebooks authoring for Azure Sentinel. While Azure Sentinel is still a big focus of our work, we are extending the data query/acquisition components to pull log data from other sources (currently Splunk, Microsoft Defender for Endpoint and Microsoft Graph are supported but we are actively working on support for data from other SIEM platforms). Most of the components can also be used with data from any source. Pandas DataFrames are used as the ubiquitous input and output format of almost all components. There is also a data provider to make it easy to and process data from local CSV files and pickled DataFrames.

The package addresses three central needs for security investigators and hunters:

  • Acquiring and enriching data
  • Analyzing data
  • Visualizing data

We welcome feedback, bug reports, suggestions for new features and contributions.


Installing

For core install:

pip install msticpy

If you are using MSTICPy with Azure Sentinel you should install with the "azsentinel" extra package:

pip install msticpy[azsentinel]

or for the latest dev build

pip install git+https://github.com/microsoft/msticpy


Documentation

Full documentation is at ReadTheDocs

Sample notebooks for many of the modules are in the docs/notebooks folder and accompanying notebooks.

You can also browse through the sample notebooks referenced at the end of this document to see some of the functionality used in context. You can play with some of the package functions in this interactive demo on mybinder.org.


Log Data Acquisition

QueryProvider is an extensible query library targeting Azure Sentinel/Log Analytics, Splunk, OData and other log data sources. It also has special support for Mordor data sets and using local data.

Built-in parameterized queries allow complex queries to be run from a single function call. Add your own queries using a simple YAML schema.

Data Queries Notebook


Data Enrichment

Threat Intelligence providers

The TILookup class can lookup IoCs across multiple TI providers. built-in providers include AlienVault OTX, IBM XForce, VirusTotal and Azure Sentinel.

The input can be a single IoC observable or a pandas DataFrame containing multiple observables. Depending on the provider, you may require an account and an API key. Some providers also enforce throttling (especially for free tiers), which might affect performing bulk lookups.

TIProviders and TILookup Usage Notebook


GeoLocation Data

The GeoIP lookup classes allow you to match the geo-locations of IP addresses using either:

GeoIP Lookup and GeoIP Notebook


Azure Resource Data, Storage and Azure Sentinel API

The AzureData module contains functionality for enriching data regarding Azure host details with additional host details exposed via the Azure API. The AzureSentinel module allows you to query incidents, retrieve detector and hunting queries. AzureBlogStorage lets you read and write data from blob storage.

Azure Resource APIs, Azure Sentinel APIs, Azure Storage


Security Analysis

This subpackage contains several modules helpful for working on security investigations and hunting:


Anomalous Sequence Detection

Detect unusual sequences of events in your Office, Active Directory or other log data. You can extract sessions (e.g. activity initiated by the same account) and identify and visualize unusual sequences of activity. For example, detecting an attacker setting a mail forwarding rule on someone's mailbox.

Anomalous Sessions and Anomalous Sequence Notebook


Time Series Analysis

Time series analysis allows you to identify unusual patterns in your log data taking into account normal seasonal variations (e.g. the regular ebb and flow of events over hours of the day, days of the week, etc.). Using both analysis and visualization highlights unusual traffic flows or event activity for any data set.


Time Series


Visualization

Event Timelines

Display any log events on an interactive timeline. Using the Bokeh Visualization Library the timeline control enables you to visualize one or more event streams, interactively zoom into specific time slots and view event details for plotted events.


Timeline and Timeline Notebook


Process Trees

The process tree functionality has two main components:

  • Process Tree creation - taking a process creation log from a host and building the parent-child relationships between processes in the data set.
  • Process Tree visualization - this takes the processed output displays an interactive process tree using Bokeh plots.

There are a set of utility functions to extract individual and partial trees from the processed data set.


Process Tree and Process Tree Notebook


Data Manipulation and Utility functions

Pivot Functions

Lets you use MSTICPy functionality in an "entity-centric" way. All functions, queries and lookups that relate to a particular entity type (e.g. Host, IpAddress, Url) are collected together as methods of that entity class. So, if you want to do things with an IP address, just load the IpAddress entity and browse its methods.

Pivot Functions and Pivot Functions Notebook


base64unpack

Base64 and archive (gz, zip, tar) extractor. It will try to identify any base64 encoded strings and try decode them. If the result looks like one of the supported archive types it will unpack the contents. The results of each decode/unpack are rechecked for further base64 content and up to a specified depth.

Base64 Decoding and Base64Unpack Notebook


iocextract

Uses regular expressions to look for Indicator of Compromise (IoC) patterns - IP Addresses, URLs, DNS domains, Hashes, file paths. Input can be a single string or a pandas dataframe.

IoC Extraction and IoCExtract Notebook


eventcluster (experimental)

This module is intended to be used to summarize large numbers of events into clusters of different patterns. High volume repeating events can often make it difficult to see unique and interesting items.



This is an unsupervised learning module implemented using SciKit Learn DBScan.

Event Clustering and Event Clustering Notebook


auditdextract

Module to load and decode Linux audit logs. It collapses messages sharing the same message ID into single events, decodes hex-encoded data fields and performs some event-specific formatting and normalization (e.g. for process start events it will re-assemble the process command line arguments into a single string).


syslog_utils

Module to support an investigation of a Linux host with only syslog logging enabled. This includes functions for collating host data, clustering logon events and detecting user sessions containing suspicious activity.


cmd_line

A module to support he detection of known malicious command line activity or suspicious patterns of command line activity.


domain_utils

A module to support investigation of domain names and URLs with functions to validate a domain name and screenshot a URL.


Notebook widgets

These are built from the Jupyter ipywidgets collection and group common functionality useful in InfoSec tasks such as list pickers, query time boundary settings and event display into an easy-to-use format.


 



More Notebooks on Azure Sentinel Notebooks GitHub

Azure Sentinel Notebooks

Example notebooks:

View directly on GitHub or copy and paste the link into nbviewer.org


Notebook examples with saved data

See the following notebooks for more examples of the use of this package in practice:


Supported Platforms and Packages

Contributing

For (brief) developer guidelines, see this wiki article Contributor Guidelines

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.



Read more


  1. Usb Pentest Tools
  2. Pentest Tools Review
  3. Pentest Tools Open Source
  4. Hack Tool Apk No Root
  5. Pentest Tools Website
  6. Pentest Tools Online
  7. Hacker Tools
  8. Hacking Tools Software
  9. Hack Website Online Tool
  10. Pentest Tools Download
  11. Hacking Tools Usb
  12. Pentest Tools
  13. Hacker Tools Apk
  14. Hacker Tools 2019
  15. Hacker Tools Hardware
  16. Hacker
  17. Hacking Tools Free Download
  18. Wifi Hacker Tools For Windows
  19. Hack Tools For Ubuntu
  20. Pentest Tools Nmap
  21. World No 1 Hacker Software
  22. Hack Tools Github
  23. Pentest Tools Github
  24. Hacker Security Tools
  25. Pentest Tools
  26. Hack Tools Pc
  27. Tools 4 Hack
  28. Hacker Tools For Mac
  29. Hacker Tools Github
  30. Hacking Tools 2020
  31. Hacker Techniques Tools And Incident Handling
  32. Pentest Tools Download
  33. Hacking Tools Usb
  34. Pentest Tools Apk
  35. Pentest Tools Online
  36. Pentest Tools Framework
  37. Hacker Tools 2019
  38. Pentest Tools Framework
  39. Hacking Tools Windows 10
  40. Hacker Techniques Tools And Incident Handling
  41. Hack Rom Tools
  42. Termux Hacking Tools 2019
  43. Tools 4 Hack
  44. Hacker Tools Software
  45. Pentest Tools Apk
  46. Hacker Hardware Tools
  47. Hacker Hardware Tools
  48. Hacker Tools Online
  49. Hacker Tools Linux
  50. Growth Hacker Tools
  51. Hacker Tools Windows
  52. Hack Tool Apk No Root
  53. Hacker Tools For Pc
  54. Pentest Tools For Windows
  55. Hack Tools For Windows
  56. Hacker Tools For Ios
  57. Pentest Reporting Tools
  58. Pentest Tools Nmap
  59. Hacks And Tools
  60. Pentest Tools For Windows
  61. Hacking Tools For Windows
  62. Pentest Tools Alternative
  63. Nsa Hacker Tools
  64. Pentest Tools Url Fuzzer
  65. Pentest Tools Open Source
  66. Hacking Tools Download
  67. Kik Hack Tools
  68. Pentest Tools Alternative
  69. Hacking Tools Github
  70. Pentest Recon Tools
  71. Pentest Tools
  72. Hacking Tools 2020
  73. Hacker Tool Kit
  74. Hacker Techniques Tools And Incident Handling
  75. Nsa Hack Tools
  76. Bluetooth Hacking Tools Kali
  77. Hack Tools For Pc
  78. Pentest Tools Framework
  79. Best Hacking Tools 2019
  80. Pentest Tools
  81. Pentest Tools Online
  82. Wifi Hacker Tools For Windows
  83. Hacker Tool Kit
  84. Hacking Tools Hardware
  85. Hacker Tools For Pc
  86. Pentest Box Tools Download
  87. Pentest Tools Windows
  88. Hack Tools Mac
  89. Hacker Tools
  90. Pentest Tools Nmap
  91. Hacker Tools Apk Download
  92. Hacker Tools Hardware
  93. Hack Tools For Games
  94. Hacker Tools Github
  95. Hack Tools For Windows
  96. Tools Used For Hacking
  97. Hacking Tools Windows
  98. Pentest Reporting Tools
  99. How To Make Hacking Tools
  100. Hacking Tools Name
  101. Hacking Tools For Kali Linux
  102. Hacking Tools Windows
  103. Hack And Tools
  104. Hacking Tools And Software
  105. Hackrf Tools
  106. Pentest Tools Linux
  107. Ethical Hacker Tools
  108. Hacker Tools For Pc
  109. Pentest Tools Free
  110. Hacker Search Tools
  111. Pentest Tools Website Vulnerability
  112. Pentest Box Tools Download
  113. Hacking Tools For Windows Free Download
  114. Pentest Tools Apk
  115. New Hack Tools
  116. Top Pentest Tools
  117. Hacker Tools Hardware
  118. How To Make Hacking Tools
  119. Hacker Tools Software
  120. Hacking Tools For Kali Linux
  121. Black Hat Hacker Tools
  122. Pentest Tools Tcp Port Scanner
  123. Hacker Tools Software
  124. Hacks And Tools
  125. Install Pentest Tools Ubuntu
  126. Pentest Box Tools Download
  127. Hacking Tools 2019
  128. Hacker Tools Apk
  129. Hacking Tools Hardware
  130. Nsa Hack Tools
  131. Pentest Tools Android
  132. Hacker Tools Free Download
  133. Beginner Hacker Tools
  134. Pentest Tools Online
  135. Hacking Tools Free Download

No comments:

Post a Comment