spiderfoot

SpiderFoot is an open source intelligence (OSINT) automation tool. It integrates with just about every data source available and utilises a range of methods for data analysis, making that data easy to navigate.

SpiderFoot has an embedded web-server for providing a clean and intuitive web-based interface but can also be used completely via the command-line. It's written in Python 3 and MIT-licensed.

FEATURES

Web based UI or CLI
Over 200 modules (see below)
Python 3.7+
YAML-configurable correlation engine with 37 pre-defined rules
CSV/JSON/GEXF export
API key export/import
SQLite back-end for custom querying
Highly configurable
Fully documented
Visualisations
TOR integration for dark web searching
Dockerfile for Docker-based deployments
Can call other tools like DNSTwist, Whatweb, Nmap and CMSeeK
Actively developed since 2012!

WANT MORE?

Need more from SpiderFoot? Check out SpiderFoot HX for:

100% Cloud-based and managed for you
Attack Surface Monitoring with change notifications by email, REST and Slack
Multiple targets per scan
Multi-user collaboration
Authenticated and 2FA
Investigations
Customer support
Third party tools pre-installed & configured
Drive it with a fully RESTful API
TOR integration built-in
Screenshotting
Bring your own Python SpiderFoot modules
Feed scan data to Splunk, ElasticSearch and REST endpoints

See the full set of differences between SpiderFoot HX and the open source version here.

USES

SpiderFoot can be used offensively (e.g. in a red team exercise or penetration test) for reconnaissance of your target or defensively to gather information about what you or your organisation might have exposed over the Internet.

You can target the following entities in a SpiderFoot scan:

IP address
Domain/sub-domain name
Hostname
Network subnet (CIDR)
ASN
E-mail address
Phone number
Username
Person's name
Bitcoin address

SpiderFoot's 200+ modules feed each other in a publisher/subscriber model to ensure maximum data extraction to do things like:

Host/sub-domain/TLD enumeration/extraction
Email address, phone number and human name extraction
Bitcoin and Ethereum address extraction
Check for susceptibility to sub-domain hijacking
DNS zone transfers
Threat intelligence and Blacklist queries
API integration with SHODAN, HaveIBeenPwned, GreyNoise, AlienVault, SecurityTrails, etc.
Social media account enumeration
S3/Azure/Digitalocean bucket enumeration/scraping
IP geo-location
Web scraping, web content analysis
Image, document and binary file meta data analysis
Dark web searches
Port scanning and banner grabbing
Data breach searches
So much more...

INSTALLING & RUNNING

To install and run SpiderFoot, you need at least Python 3.7 and a number of Python libraries which you can install with pip. We recommend you install a packaged release since master will often have bleeding edge features and modules that aren't fully tested.

Stable build (packaged release):

 wget https://github.com/smicallef/spiderfoot/archive/v4.0.tar.gz
 tar zxvf v4.0.tar.gz
 cd spiderfoot-4.0
 pip3 install -r requirements.txt
 python3 ./sf.py -l 127.0.0.1:5001

Development build (cloning git master branch):

 git clone https://github.com/smicallef/spiderfoot.git
 cd spiderfoot
 pip3 install -r requirements.txt
 python3 ./sf.py -l 127.0.0.1:5001

Check out the documentation and our asciinema videos for more tutorials.

COMMUNITY

Whether you're a contributor, user or just curious about SpiderFoot and OSINT in general, we'd love to have you join our community! SpiderFoot now has a Discord server for seeking help from the community, requesting features or just general OSINT chit-chat.

WRITING CORRELATION RULES

We have a comprehensive write-up and reference of the correlation rule-set introduced in SpiderFoot 4.0 here.

Also take a look at the template.yaml file for a walk through. The existing 37 rules are also quite readable and good as starting points for additional rules.

MODULES / INTEGRATIONS

SpiderFoot has over 200 modules, most of which don't require API keys, and many of those that do require API keys have a free tier.

Name	Description	Type
AbstractAPI	Look up domain, phone and IP address information from AbstractAPI.	Tiered API
abuse.ch	Check if a host/domain, IP address or netblock is malicious according to Abuse.ch.	Free API
AbuseIPDB	Check if an IP address is malicious according to AbuseIPDB.com blacklist.	Tiered API
Abusix Mail Intelligence	Check if a netblock or IP address is in the Abusix Mail Intelligence blacklist.	Tiered API
Account Finder	Look for possible associated accounts on over 500 social and other websites such as Instagram, Reddit, etc.	Internal
AdBlock Check	Check if linked pages would be blocked by AdBlock Plus.	Tiered API
AdGuard DNS	Check if a host would be blocked by AdGuard DNS.	Free API
Ahmia	Search Tor 'Ahmia' search engine for mentions of the target.	Free API
AlienVault IP Reputation	Check if an IP or netblock is malicious according to the AlienVault IP Reputation database.	Free API
AlienVault OTX	Obtain information from AlienVault Open Threat Exchange (OTX)	Tiered API
Amazon S3 Bucket Finder	Search for potential Amazon S3 buckets associated with the target and attempt to list their contents.	Free API
Apple iTunes	Search Apple iTunes for mobile apps.	Free API
Archive.org	Identifies historic versions of interesting files/pages from the Wayback Machine.	Free API
ARIN	Queries ARIN registry for contact information.	Free API
Azure Blob Finder	Search for potential Azure blobs associated with the target and attempt to list their contents.	Free API
Base64 Decoder	Identify Base64-encoded strings in URLs, often revealing interesting hidden information.	Internal
BGPView	Obtain network information from BGPView API.	Free API
Binary String Extractor	Attempt to identify strings in binary content.	Internal
BinaryEdge	Obtain information from BinaryEdge.io Internet scanning systems, including breaches, vulnerabilities, torrents and passive DNS.	Tiered API
Bing (Shared IPs)	Search Bing for hosts sharing the same IP.	Tiered API
Bing	Obtain information from bing to identify sub-domains and links.	Tiered API
Bitcoin Finder	Identify bitcoin addresses in scraped webpages.	Internal
Bitcoin Who's Who	Check for Bitcoin addresses against the Bitcoin Who's Who database of suspect/malicious addresses.	Tiered API
BitcoinAbuse	Check Bitcoin addresses against the bitcoinabuse.com database of suspect/malicious addresses.	Free API
Blockchain	Queries blockchain.info to find the balance of identified bitcoin wallet addresses.	Free API
blocklist.de	Check if a netblock or IP is malicious according to blocklist.de.	Free API
BotScout	Searches BotScout.com's database of spam-bot IP addresses and e-mail addresses.	Tiered API
botvrij.eu	Check if a domain is malicious according to botvrij.eu.	Free API
BuiltWith	Query BuiltWith.com's Domain API for information about your target's web technology stack, e-mail addresses and more.	Tiered API
C99	Queries the C99 API which offers various data (geo location, proxy detection, phone lookup, etc).	Commercial API
CallerName	Lookup US phone number location and reputation information.	Free API
Censys	Obtain host information from Censys.io.	Tiered API
Certificate Transparency	Gather hostnames from historical certificates in crt.sh.	Free API
CertSpotter	Gather information about SSL certificates from SSLMate CertSpotter API.	Tiered API
CINS Army List	Check if a netblock or IP address is malicious according to Collective Intelligence Network Security (CINS) Army list.	Free API
CIRCL.LU	Obtain information from CIRCL.LU's Passive DNS and Passive SSL databases.	Free API
CleanBrowsing.org	Check if a host would be blocked by CleanBrowsing.org DNS content filters.	Free API
CleanTalk Spam List	Check if a netblock or IP address is on CleanTalk.org's spam IP list.	Free API
Clearbit	Check for names, addresses, domains and more based on lookups of e-mail addresses on clearbit.com.	Tiered API
CloudFlare DNS	Check if a host would be blocked by CloudFlare DNS.	Free API
CoinBlocker Lists	Check if a domain appears on CoinBlocker lists.	Free API
CommonCrawl	Searches for URLs found through CommonCrawl.org.	Free API
Comodo Secure DNS	Check if a host would be blocked by Comodo Secure DNS.	Tiered API
Company Name Extractor	Identify company names in any obtained data.	Internal
Cookie Extractor	Extract Cookies from HTTP headers.	Internal
Country Name Extractor	Identify country names in any obtained data.	Internal
Credit Card Number Extractor	Identify Credit Card Numbers in any data	Internal
Crobat API	Search Crobat API for subdomains.	Free API
Cross-Referencer	Identify whether other domains are associated ('Affiliates') of the target by looking for links back to the target site(s).	Internal
CRXcavator	Search CRXcavator for Chrome extensions.	Free API
Custom Threat Feed	Check if a host/domain, netblock, ASN or IP is malicious according to your custom feed.	Internal
CyberCrime-Tracker.net	Check if a host/domain or IP address is malicious according to CyberCrime-Tracker.net.	Free API
Debounce	Check whether an email is disposable	Free API
Dehashed	Gather breach data from Dehashed API.	Commercial API
Digital Ocean Space Finder	Search for potential Digital Ocean Spaces associated with the target and attempt to list their contents.	Free API
DNS Brute-forcer	Attempts to identify hostnames through brute-forcing common names and iterations.	Internal
DNS Common SRV	Attempts to identify hostnames through brute-forcing common DNS SRV records.	Internal
DNS for Family	Check if a host would be blocked by DNS for Family.	Free API
DNS Look-aside	Attempt to reverse-resolve the IP addresses next to your target to see if they are related.	Internal
DNS Raw Records	Retrieves raw DNS records such as MX, TXT and others.	Internal
DNS Resolver	Resolves hosts and IP addresses identified, also extracted from raw content.	Internal
DNS Zone Transfer	Attempts to perform a full DNS zone transfer.	Internal
DNSDB	Query FarSight's DNSDB for historical and passive DNS data.	Tiered API
DNSDumpster	Passive subdomain enumeration using HackerTarget's DNSDumpster	Free API
DNSGrep	Obtain Passive DNS information from Rapid7 Sonar Project using DNSGrep API.	Free API
DroneBL	Query the DroneBL database for open relays, open proxies, vulnerable servers, etc.	Free API
DuckDuckGo	Query DuckDuckGo's API for descriptive information about your target.	Free API
E-Mail Address Extractor	Identify e-mail addresses in any obtained data.	Internal
EmailCrawlr	Search EmailCrawlr for email addresses and phone numbers associated with a domain.	Tiered API
EmailFormat	Look up e-mail addresses on email-format.com.	Free API
EmailRep	Search EmailRep.io for email address reputation.	Tiered API
Emerging Threats	Check if a netblock or IP address is malicious according to EmergingThreats.net.	Free API
Error String Extractor	Identify common error messages in content like SQL errors, etc.	Internal
Ethereum Address Extractor	Identify ethereum addresses in scraped webpages.	Internal
Etherscan	Queries etherscan.io to find the balance of identified ethereum wallet addresses.	Free API
File Metadata Extractor	Extracts meta data from documents and images.	Internal
Flickr	Search Flickr for domains, URLs and emails related to the specified domain.	Free API
Focsec	Look up IP address information from Focsec.	Tiered API
FortiGuard Antispam	Check if an IP address is malicious according to FortiGuard Antispam.	Free API
Fraudguard	Obtain threat information from Fraudguard.io	Tiered API
F-Secure Riddler.io	Obtain network information from F-Secure Riddler.io API.	Commercial API
FullContact	Gather