toolsmith: SpiderFoot

Prerequisites/dependencies

Python 2.7 if running on *nix as well as M2Crypto, CherryPy, netaddr, dnspython, and Mako modules

Windows version comes as a pre-packaged executable, no dependencies

Introduction

All good penetration tests and threat assessments should be initiated with what you’ve seen referred to in toolsmith as OSINT, or open source intelligence gathering. This practice contributes greatly to collecting a useful list of targets of opportunity. One key element to remember though, the bad guys are conducting this same activity against you and your Internet-facing assets too. It’s probably best then that you develop your own OSINT practice so you can find the information you may not wish to, or even know, you are exposing. Steve Micallef’s SpiderFoot is another tool in the arsenal specific to this cause. You may already be aware that the four phases of a web application security assessment, as defined using the SamuraiWTF distribution, are recon, mapping, discovery, and exploitation. The SANS GIAC Certified Web Application Penetration Tester (GWAPT) curriculum follows suit given that Secure Idea’s Kevin Johnson contributed heavily (developed) to both. SpiderFoot nicely blends both recon and mapping as part of its feature set. As we consider legal, privacy, and ethics issues for the March ISSA Journal, OSINT and reconnaissance become interesting and related topics. I have, on more than one occasion, discovered very damaging data via OSINT tactics that, if in the wrong hands, could have been very damaging. When you consider findings of this nature with regard to ethics and the legality you may find yourself in an immediate quandary. Are you obligated to report findings that you know could cause harm to the target if left unmitigated? What if during your analysis you come into possession of classified or proprietary information that having in your possession could create legal challenges for you? Imagine findings of this caliber and it becomes easy to recognize why you should always conduct intelligence gathering and footprinting on your own interests before the wrong people do it for you. SpiderFoot, as a tool for just such purposes, allows you to understand “as much as possible about a given target in order to perform a more complete security penetration test.” For large networks, this can be a daunting task, and SpiderFoot automates this process significantly, allowing penetration testers to focus their efforts on security testing itself.

Steve provided us with some SpiderFoot history as well as insight on what he finds useful and interesting. He originally wrote SpiderFoot as a C# .NET application in 2005, purely as an exercise to learn C#, having been inspired by BiDiBLAH’s developers from Sensepost (who went on to create Maltego), thinking he could make a lighter open source version. For seven years that was Steve’s first and only release until he decided to resume development again in 2012. His work on next generation versions have led SpiderFoot to be cross platform (Python), far more extensible, functional, with a much nicer user interface (UI).

Steve’s current challenge with SpiderFoot is deciding what cool functionality to implement next, his to-do list is ever growing and there are a numerous features he’d love to extend it to include. He typically balances his time between UI/analysis functionality versus new checks to identify more items to aid the penetration tester. The aforementioned OSINT (Open Source Intelligence) community also continues to produce new sources which in turn inspire Steve to build new SpiderFoot checks.

He finds it interesting testing out a new module, and actually finding insightful items out there on the Internet simply during the development process. Steve’s favorite functionality at the moment is identifying owned netblocks, and co-hosted sites. Owned Netblocks indicates entire IP ranges that an organization owns, which enables penetration testers to more completely scan the perimeter of a target. Co-hosted Sites shows you any websites on the same server as the target, which can also be revealing. If your target is hosted on the same server as sites identified as being malicious by the malicious site checker, or the blacklist checker plug-in it could potentially indicate that your target is hosted on a compromised server.

As you read this it’s likely that the following planned enhancements are available in SpiderFoot or will be soon:

·         2.1.2 (early March)

o   SOCKS proxy support

o   Real-time scan progress viewer

o   Identify scan quality impacting issue

o   Autoshun (www.autoshun.org) lookup as part of malicious checks

o   SANS (isc.sans.edu) lookup as part of malicious checks (queue the Austin Powers voice: “Yeah, baby!”)

o   Update GeoIP checker

·         2.1.3 (mid April)

o   VirusTotal, SHODAN, Facebook, Xing, Pastebin and GitHub plug-ins

Note that when you pull SpiderFoot from GitHub, you are downloading a beta version of the next release, as Steve commits new functionality there periodically in preparation for the next version. For instance, SOCKS functionality is in the GitHub repository right now but not in the packaged released version (2.1.1.).

SpiderFoot is a great project with a strong development roadmap, so let’s get down to business and explore.

Quick installation notes

Windows installation is an absolute no brainer; download the package, unpack it, execute sf.exe, and browse to http://127.0.0.1:5001. All dependencies are met including a standalone Python interpreter, so you may find this option optimal.

Linux (I installed it on SamuraiWTF) users need to settle a few dependencies easily solved with the following few steps that assume pip is already installed:

sudo apt-get install swig

sudo pip install mako cherrypy netaddr M2Crypto dnspython

git clone https://github.com/smicallef/spiderfoot.git

cd spiderfoot/

sudo python ./sf.py 0.0.0.0:9999

The last line indicates that you’d like SpiderFoot to bind to all addresses (including localhost) and listen on port 9999. You can define your preferred port or just accept default if undefined (5001). Steve reminds us on his installation page to be cautious regarding exposing SpiderFoot to hostile networks (Intranet, security conference wireless) given that there is currently no authentication scheme.

SpiderFoot unleashed

The SpiderFoot UI is, how shall I say, incredibly simple, intuitive, and obvious even. To start a scan…wait for it…select New Scan. Figure 1 represents a scan being kicked off on my domain (don’t do it) as defined by the By Module view.

FIGURE 1: Kicking off a new scan with SpiderFoot

If you wish to more granularly define your scans, select the By Required Data view (default) then pick and choose your preferred data points including elements such as malicious affiliations, IP data, URL analysis, SSL certificate information, affiliate details, and many other record. You should then be treated to a success message. Scans results are stored in a SQLite DB so over time you’ll likely build up a collection if you don’t purge. Under the Scans tab as seen in Figure 2 you can click the scan in the Name column of the table view and review results. You’ll also note status here and can also halt the scan if need be. I imagine the real-time scan progress viewer will show itself here in the near future as well.

FIGURE 2: SpiderFoot Scans view

If need be (default settings work quite well), you can tune the actual scan configuration as well via Settings, with attention to how you’d like to tune storage, search engines, port scanning, spidering, TLD searches (see Figure 3), amongst others.

FIGURE 3: SpiderFoot Settings view

When my scan completed, with default settings and all checks enabled, the results included 11360 elements. For you data miners, metrics minions, and hosting harvesters, you can export the results to CSV (see Figure 4) and filter by findings type and module, or your preferred data pivot.

FIGURE 4: SpiderFoot results and export functionality

As I navigated all the results, I was intrigued to find a hit for URL (Uses Flash) simply because I didn’t recall any Flash features on my site. I immediately chuckled when I reviewed the result as it was specific to a Flash video I’d created for the 2008 ISSA Northwest Regional Conference wherein I ripped on the now defunct Hacker Safe trustmark for indicating that their customer’s sites were “hacker safe” when, in fact, they were not. Oh, the good old days.

Want to visualize your results? No problem, you can choose from a bubble view of data elements or the discovery path. Figure 5 represents the discovery path for Social Media Presence findings. Hover over each entity for details specific to initial target type, the source module, and the related result.

FIGURE 5: SpiderFoot visualizes a discovery path

SpiderFoot will absolutely uncover nuggets you may have long forgotten about and may want to remove as they are potentially vulnerable (outdated plugins, modules, etc.) or unnecessarily/unintentionally exposed. I found an old dashboard I’d built by hand eons ago with long dead extenal JavaScript calls that had no business still being available. “Be gone!”, I said. That is what SpiderFoot is all about. Add it to the tool collection for penetration tests and OSINT expeditions; you won’t be disappointed.

In Conclusion

Steve Micallef’s SpiderFoot is functionally simple but feature rich and getting better all the time as it is well built and maintained. Follow @binarypool on Twitter and keep an eye out for timely and regular releases.

Ping me via email if you have questions or suggestions for topic via russ at holisticinfosec dot org or hit me on Twitter @holisticinfosec.

Cheers…until next month.

Acknowledgements

Steve Micallef (@binarypool), Spiderfoot author