Thursday, May 01, 2014

toolsmith: Microsoft Threat Modeling Tool 2014 - Identify & Mitigate




Prerequisites/dependencies
Windows operating system

Introduction
I’ve long been deeply invested in the performance of threat modeling with particular attention to doing so in operational environments rather than limiting the practice to simply software. I wrote the ITInfrastructure Threat Modeling Guide for Microsoft in 2009 with the hope of stimulating this activity. In recent months two events have taken place that contribute significantly to the threat modeling community. In February Adam Shostack published his book, Threat Modeling: Designing for Security and I can say, without hesitation, that it is a gem. I was privileged to serve as the technical proof reader for this book and found that its direct applicability to threat modeling across the full spectrum of target opportunities is inherent throughout. I strongly recommend you add this book to your library as it is, in and of itself, a tool for threat modelers and those who wish to reduce risk, apply mitigations, and improve security posture. This was followed in mid-April by the release of the Microsoft Threat Modeling Tool 2014. The tool had become a bit stale and the 2014 release is a refreshing update that includes a number of feature improvements that we’ll discuss shortly. We’ll also use the tool to conduct a threat model that envisions the ISSA Journal’s focus for the month of May: Healthcare Threats and Controls.
First, I sought out Adam to provide us with insight regarding his perspective on operational threat modeling. As expected, he indicated that whether you're a system administrator, system architect, site reliability engineer, or IT professional, threat modeling is important and applicable to your job. Adam often asks four related questions:
1)      What are you building?
He describes that building an operational system is more likely to be building additional components on top of an existing system and that it's therefore important to model both what you have and how it's changing.    
2)      What can go wrong? 
Adam reminds us that you can use any of the threat enumeration techniques, but that, in particular, STRIDE relates closely to the “CIA” set of properties that are desirable for an operational system. I’ll add OWASP Risk Rating Methodology to the tool’s KB for good measure, given its direct integration of CIA.  
3)      What are you going to do about it? 
Several frameworks can be used here, such as prevent, detect, and respond as well as available technologies. 
4)      Did you do a good job at 1-3? 
Adam points out that assurance activities (which can include compliance) can help you.  More importantly, you can also use approaches such as penetration testing and red teaming to help you determine if you did a good job. I am a strong proponent for this approach. My team at Microsoft includes both threat engineers for threat modeling and assessment as well as penetration testers for discovery and validation of mitigations.
To supplement the commitment to operational threat modeling, I asked Steve Lipner, one of the founding fathers of Microsoft’s Security Development Lifecycle and the Security Response Center (MSRC), for his perspective, which he eloquently provided as follows:
“While threat modeling originated as an approach to evaluating the security of software components, we have found the techniques of security threat modeling to have wide applicability.  Like software components, operational services are targets of attack and can exhibit vulnerabilities.  Threat modeling and STRIDE have proven to be effective for identifying and mitigating vulnerabilities in operational services as well as software products and components.”
With clear alignment around the premise of operational threat modeling let’s take a look at what it means to apply it. 

Identifying Threats and Mitigations with TMT 2014

Emil Karafezov, who is responsible for the Threat Modeling component of the Security Development Lifecycle (SDL) at Microsoft, wrote a useful introduction to the Microsoft Threat Modeling Tool 2014 (TMT). Emil let me know that there are additional details and pointers in the Getting Started Guide and the User Guide which are part of the Threat ModelingTool 2014 Principles SDK. You should definitely read the introduction as well as the guides before proceeding here as I will not be revisiting the basic usage information for the TMT tool or how to threat model (read the book) and will instead focus more in depth on some key new capabilities. I will do so in the context of a threat model for the operational environment of a fictional medical services company called MEDSRV.
Figure 1 includes a view of the MEDSRV operational environment for its web application and databases implementation.

FIGURE 1: A MEDSRV threat model with TMT 2014
Emil offered some additional pointers not shared in his blog post that we’ll explore further with the MEDSRV threat model specific to data extraction and search capabilities.

Data extraction:
From a workflow perspective, the ability to extract information from the tool for record keeping or bug filing is quite useful. The previous version of the TMT included Product Studio and Visual Studio plugins for bug filing but Emil describes them as rather rigid templates that were problematic for users syncing with their server. With TMT 2014 there is a simple right-click Copy Threats for each entry that can be pasted into any text editor or bug tracking system. For bulk threat entry manipulation there is another feature ‘Copy Custom Threat Table’ which lets you dump results conveniently into Excel, which in turn can be imported into workflow management systems via automation. When in Analysis View with focus set in the Threat Information list use the known Ctrl+A shortcut to select all threat entries and with right-click you can edit the constants in the Custom Threat Table as seen in Figure 2.

FIGURE 2: TMT 2014’s Copy Custom Threat Table feature
Search for Threat Information:
Emil also pointed out that TMT 2014’s Search for Threat Information area, while seemingly a standard-to-have option, is new and worth mentioning. This feature is really important if you have a massive threat model with a plethora of threats; the threat list filter is not always the most efficient way to narrow down your criteria. I have found this to be absolute truth during threat modeling sessions of online services at Microsoft where a large model may include hundreds or thousands of threats. To find threats that contained keywords specific to a particular implementation of your mitigations as an example, using Search is the way to go. You might be focusing on data store accessibility as seen in Figure 3.

FIGURE 3: Search for threat information
I also asked Ralph Hood, Microsoft Trustworthy Computing’s Group Program Manager for Secure Development Policies & Tools (the group that oversees the TMT) what stood out for him with this version of the tool. He offered two items in particular:
1)      Migration capability of models from the old version of the tool
2)      The ability to customize threats
Ralph indicated that the TMT tool has not historically supported any kind of migration to newer versions; the ability to migrate models from earlier versions to the 4.1 version is therefore a powerful feature for users who have already conducted numerous threat models with older versions. Threat models should always be considered dynamic (never static) as systems always change and you’ll likely update a model at a later date.
The ability to customize threats is also very important, particularly in the operations space. The ability to change the threat elements and information (mitigation suggestions, threat categories, etc.) for specific environments is of significant importance. Ralph points out as an example that if a specific service or product owner knows that certain threats are assessed differently because of specific characteristics of the service or platform, they can change the related threat information. Threat modelers can do so using a Knowledge Base (KB) created for all related models so any user going forward can utilize the modified KB rather than having to always change threat attributes for each threat manually. According to Ralph, this is important functionality in the operations space where certain service dependencies and platform benefits and/or downfalls may consistently alter threat information. He’s absolutely right so I’ll take the opportunity to tweak the imaginary MEDSRV KB here for your consideration using Appendix II of the User Guide (read it).  The KB is installed by default in C:\Program Files (x86)\Microsoft Threat Modeling Tool 2014\KnowledgeBase. Do not tweak the original, create a copy and modify that. I called my copy KnowledgeBaseMEDSRV and saved it in C:\tmp. I focused exclusively on ThreatCategories.xml and ThreatTypes.xml. Using the OWASP Risk Rating Methodology I added Technical Impact Factors to ThreatCategories.xml and ThreatTypes.xml. Direct from the OWASP site, “technical impact can be broken down into factors aligned with the traditional security areas of concern: confidentiality, integrity, availability, and accountability. The goal is to estimate the magnitude of the impact on the system if the vulnerability were to be exploited.”
·         Loss of confidentiality
o   How much data could be disclosed and how sensitive is it? Minimal non-sensitive data disclosed (2), minimal critical data disclosed (6), extensive non-sensitive data disclosed (6), extensive critical data disclosed (7), all data disclosed (9)
·         Loss of integrity
o   How much data could be corrupted and how damaged is it? Minimal slightly corrupt data (1), minimal seriously corrupt data (3), extensive slightly corrupt data (5), extensive seriously corrupt data (7), all data totally corrupt (9)
·         Loss of availability
o   How much service could be lost and how vital is it? Minimal secondary services interrupted (1), minimal primary services interrupted (5), extensive secondary services interrupted (5), extensive primary services interrupted (7), all services completely lost (9)
·         Loss of accountability
o   Are the threat agents' actions traceable to an individual? Fully traceable (1), possibly traceable (7), completely anonymous (9)
Note: I renamed the original KnowledgeBase to KnowledgeBase.bak then copied KnowledgeBaseMEDSRV back to the original destination directory and renamed it KnowledgeBase. This prevents corruption of your original files and eliminates the need to re-install TMT. If you’d like my changes to ThreatCategories.xml and ThreatTypes.xml hit me over email or Twitter and I’ll send them to you. That said, following are snippets (Figures 4 & 5) of the changes I made.

FIGURE 4: Additions to ThreatCategories.xml
FIGURE 5: Additions to ThreatTypes.xml
Take notice of a few key elements in the modified XML. I set OTI1 for OWASP Technical Impact and O to O for OWASP. J Remember that each subsequent needs to be unique. I declared source is 'GE.P' and (target is 'GE.P' or target is 'GE.DS') and flow crosses 'GE.TB' because GE.P defines a generic process, GE.DS defines a generic data store and GE.TB defines a generic trust boundary. Therefore, per my modification, data subject to technical impact factors flows across trust boundaries between processes and data stores. Make sense? I used the resulting TMT KB update to provide a threat model of zones defined for MEDSRV as seen in Figure 6.

FIGURE 6: A threat model of MEDSRV zones using Technical Impact Factors
I’m hopeful these slightly more in depth investigations of TMT 2014 features entices you to utilize the tool and to engage in the practice of threat modeling. No time like the present to get started.

In Conclusion

We’ll learned enough here to conclude that you have two immediate actions. First, purchase Threat Modeling: Designing For Security and begin to read it. Follow this by downloading the Microsoft Threat Modeling Tool 2014 and practice threat modeling scenarios with the tool while you read the book. Conducting these in concert will familiarize you with both the practice of threat modeling as well as the use of TMT 2014.  
Remember that July’s ISSA Journal will be entirely focused on the Practical Use of InfoSec Tools. Send articles or abstracts to editor at issa dot org.
Ping me via email if you have questions or suggestions for topic via russ at holisticinfosec dot org or hit me on Twitter @holisticinfosec.
Cheers…until next month.

Acknowledgements
Microsoft’s:
Adam Shostack, author, Threat Modeling: Designing for Security & Principal Security PM, TwC Secure Ops
Emil Karafezov, Security PM II, TwC Secure Development Tools and Policies
Ralph Hood, Principal Security GPM, TwC Secure Development Tools and Policies
Steve Lipner, Partner Director, TwC Software Security

Thursday, April 03, 2014

Browse this: & Oryon C Portable & WhiteHat Aviator


Please take a moment as you read this toolsmith to honor those lost in the Oso, WA landslide disaster and those who have lost loved ones, friends, and homes. Pro Civitas et Patria.

Prerequisites/dependencies
Windows for Oryon C Portable
Mac OS X or Windows for WhiteHat Aviator

Introduction
Spring is upon us and with April comes a focus on Security and Cloud Computing in the ISSA Journal and as such a focus on security-centric Chromium-based web browsers in toolsmith. It also freaks me out just a bit to say this but with April also comes the 90th consecutive toolsmith. I sure hope you enjoy reading it as much as I do writing it; it’s been a fabulous seven year plus journey so far.
Those of you who enjoy the benefits of rich web content, fast load times, and flexible browser extensibility have likely tried or use the Chrome browser. What you may not be aware of is that there are other Chromium-based browsers that are built with a bit more attention to privacy than might be expected from Chrome proper.
Full disclosure right up front: as a reminder, I work for Microsoft, and the one thing this article won’t be is any kind of a knock on Google Chrome privacy posture or a browser comparison beyond these two Chromium variants. There are plenty of other battles to fight than one in the Browser Wars. We will however have a usability and features-based discussion on Oryon C Portable, an OSINT-focused browser built on the SRWare Iron version 31.0.1700.0 of Chromium, and WhiteHat Aviator, also Chromium based. Note that Chromium, no matter the variant, includes sandboxing which has obvious security advantages.
Oryon C Portable is a web browser designed to assist researchers in conducting Open Source Intelligence (OSINT) investigations, with more than 70 pre-installed tools, while WhiteHat Aviator describes itself the “best and easiest way to bank, shop, browse, and use social networks while stopping viruses, advertisers, hackers, and cyber-crooks.”
According to Marcin Meller of OSINT Insight, the next version of Oryon C will be named Oryon C OSINT Framework and will be based on their own build of Chromium. They’ve made some changes to the tool sets and information sources. While there will be a few new interesting solutions, they also managed to reduce features that proved to be unnecessary. The browser will be lighter, clearer, and more effective, and the new version will offer a cross-platform support including Windows, Linux, and Mac OS X along with a special edition of Oryon F based on the Mozilla source code, specifically for Firefox enthusiasts. These new releases should appear online sometime this summer at the latest. Marcin says that thanks to great feedback from users, including some excellent OSINT specialists, they are highly motivated to make Oryon an even more solid and powerful tool. The active users are the strength of this project, thus, Marcin invites everyone to share their experiences and support Oryon.   
When I pinged Jeremiah Grossman, now WhiteHat’s CEO, he reminded me that Robert ‘RSnake’ Hansen, VP of WhiteHat labs, leads the Aviator project. Ah, the fond memories of April Fools’ Day past (5 years ago now) and the birth of the Application Security Specialist (ASS) certification. Jeremiah is the master of April Fools’ mayhem. It’s not often that you get the opportunity for a photo opp with both Jeremiah and RSnake, but if you’re wearing your ASS shirt at the BlueHat conference, you just might.

FIGURE 1: Robert, Russ, and Jeremiah: certified
Robert filled me in in the Aviator project: “WhiteHat Aviator started off being a more private and secure browsing option for our own internal users. It has morphed into being a consumer product (Mac and Windows) that has additional and originally unforeseen merits.  For instance, it is significantly faster due to having no ads, and by virtue of making Flash and Java a "click-to-play" option.  Users on GoGo inflight wireless love it, because it makes the web usable over latent connections, not to mention it uses less power on your laptop.  We are giving the browser away for free for now, and all users who download it will be grandfathered in, but in the future we will charge for the browser to ensure that our interests are aligned with the user and to help pay for development without requiring us to steal personal information from our users. ;-)  We will quite possibly be the first browser with tech-support!”

Both of the browsers offer the added benefit of enhanced privacy but serve rather different purposes, so let’s explore each for their strengths.

Oryon C Portable

OSINT fans rejoice, there’s a browser dedicated to your cause! Oryon includes more than 70 pre-installed tools, more than 600 links to specialized sources of information and online investigative tools, additional privacy protection features, and a ready-to-use OPML file containing a sorted collection of information sources specific to OSINT, Intelligence, InfoSec, defense, and more. Oryon C Portable is also quite literally…portable. You can run it from all sorts of USB and optical media. I’ll pause for a second so you can take in all the glorious OSINT power at your fingertips as seen in Figure 2.

FIGURE 2: Revel in the OSINT majesty
 You can manage the Oryon C tools from the, yep, you guessed it, the Oryon C tool button. As you do so you’ll see the related button appear on the toolbar and a popup notice that the extension has been enabled. From the same tools button as seen in Figure 3 you can open the full tools menu to create extensions groups and search/sort your extensions.

FIGURE 3: Enable Oryon tool families
There are so many tools to explore with it’s hard to discuss them all but I’ll mention a few of my favorites. Do keep in mind that you may find part of the feature set using Polish as Oryon C is developed by Mediaquest in Poland. The IP Geolocator uses Google Maps and MaxMind to zoom in on the location of IP addresses you enter in the form field. Fresh Start is a cross browser session manager that allows you to save a session and reimport it or recover if it’s crashed. I love Split Screen as it lets you conduct two sessions side by side for comparison. Wappalyzer is a browser extension that uncovers the technology used on websites including content management systems, eCommerce platforms, web servers, JavaScript frameworks, analytics tools and many more. Want to spoof your user-agent? Rhetorical question; yes you do. Make use of the Chrome UA Spoofer. Don’t hesitate to dive into the hyperlinks folders as that represents an entire other level of exploration. The All in one Web Searcher aggregates results from a plethora of search results in one UI as seen in Figure 4.

FIGURE 4: All in one Web Searcher results
Oryon C = playtime for OSINT nerds, and I proudly count myself as one. I literally spent hours experimenting with Oryon and am certain to spend many more similarly. At least I can count it as time towards work. ;-)

WhiteHat Aviator

For Aviator I thought I’d conduct an interesting study, albeit not following optimal scientific standards.
On a Windows 7 virtual machine, I conducted default installations of Aviator and Chrome and made no setting changes. With no other applications running, and no processes generating any network traffic, I executed the following:
Step 1
1)      Started Wireshark
2)      Initiated a capture on the active interface
3)      Started Aviator
4)      Browsed to http://holisticinfosec.blogspot.com
5)      Terminated Aviator
6)      Stopped Wireshark with 5250 frames captured
Step 2
1)      Started Wireshark
2)      Initiated a capture on the active interface
3)      Started Chrome
4)      Browsed to http://holisticinfosec.blogspot.com
5)      Terminated Chrome
6)      Stopped Wireshark with 5250 frames captured
Step 3
1)      Open aviator.pcap in NetworkMiner 1.5 and sorted by Hostname
2)      Open chrome.pcap in NetworkMiner 1.5 and sorted by Hostname 
3)      Compared results

The results were revealing to be sure.

I’m glad to share the captures for your own comparisons; just ping me via email or Twitter if you’d like copies. Notice in Figure 5 the significant differences between counts specific to hosts, files, images, credentials, sessions, DNS, and Parameters.

FIGURE 5: Comparing the differences between Aviator and Chrome browser session network traffic
Aviator is significantly less chatty than Chrome.
Supporting statistics as derived from results seen in Figure 5:
120% less host contact in Aviator capture vs. Chrome capture
69% less file interaction (download of certs, gifs, etc.) in Aviator capture vs. Chrome capture
86% fewer images presented (ads) in Aviator capture vs. Chrome capture
63% fewer total sessions in the Aviator capture vs. the Chrome capture
69% fewer DNS lookups in the Aviator capture vs. the Chrome capture
Hopefully you get the point. :-)

These differences between default configurations of Aviator and Chrome are achieved as follows:

  • Aviator's privacy and security safeguards are preconfigured, active and enabled by default
  • Aviator eliminates hidden tracking and uses the Disconnect extension to block privacy-destroying tracking from advertisers and social media companies
  • WhiteHat is not partnering with advertisers or selling click data
  • Unwanted access is prevented as Aviator blocks internal address space to prevent malicious Web pages from hitting your websites, routers, and firewalls

It’s reasonable to ascertain that those with an affinity for strong default privacy settings will favor WhiteHat Aviator given the data noted in Figure 5 and settings provided out of the gate.

In Conclusion

These are a couple of fabulous browsers for your OSINT and privacy/security pleasure. They’re so easy to install and use (I didn’t even include an installation section, no need) that I strongly recommend that you do so immediately.
Take note, readers! July’s ISSA Journal will be entirely focused on the Practical Use of InfoSec Tools. Rather than put up what is usually just me going on about infosec tools, you should too! Send articles or abstracts to editor at issa dot org.
Ping me via email if you have questions or suggestions for topic via russ at holisticinfosec dot org or hit me on Twitter @holisticinfosec.
Cheers…until next month.

Acknowledgements

Marcin Meller, OSINT Insight
Robert ‘RSnake’ Hansen, VP WhiteHat Labs, Advanced Technology Group

Sunday, March 02, 2014

toolsmith: SpiderFoot



Prerequisites/dependencies
Python 2.7 if running on *nix as well as M2Crypto, CherryPy, netaddr, dnspython, and Mako modules
Windows version comes as a pre-packaged executable, no dependencies

Introduction
All good penetration tests and threat assessments should be initiated with what you’ve seen referred to in toolsmith as OSINT, or open source intelligence gathering. This practice contributes greatly to collecting a useful list of targets of opportunity. One key element to remember though, the bad guys are conducting this same activity against you and your Internet-facing assets too. It’s probably best then that you develop your own OSINT practice so you can find the information you may not wish to, or even know, you are exposing. Steve Micallef’s SpiderFoot is another tool in the arsenal specific to this cause. You may already be aware that the four phases of a web application security assessment, as defined using the SamuraiWTF distribution, are recon, mapping, discovery, and exploitation. The SANS GIAC Certified Web Application Penetration Tester (GWAPT) curriculum follows suit given that Secure Idea’s Kevin Johnson contributed heavily (developed) to both. SpiderFoot nicely blends both recon and mapping as part of its feature set. As we consider legal, privacy, and ethics issues for the March ISSA Journal, OSINT and reconnaissance become interesting and related topics. I have, on more than one occasion, discovered very damaging data via OSINT tactics that, if in the wrong hands, could have been very damaging. When you consider findings of this nature with regard to ethics and the legality you may find yourself in an immediate quandary. Are you obligated to report findings that you know could cause harm to the target if left unmitigated? What if during your analysis you come into possession of classified or proprietary information that having in your possession could create legal challenges for you? Imagine findings of this caliber and it becomes easy to recognize why you should always conduct intelligence gathering and footprinting on your own interests before the wrong people do it for you. SpiderFoot, as a tool for just such purposes, allows you to understand “as much as possible about a given target in order to perform a more complete security penetration test.” For large networks, this can be a daunting task, and SpiderFoot automates this process significantly, allowing penetration testers to focus their efforts on security testing itself.
Steve provided us with some SpiderFoot history as well as insight on what he finds useful and interesting. He originally wrote SpiderFoot as a C# .NET application in 2005, purely as an exercise to learn C#, having been inspired by BiDiBLAH’s developers from Sensepost (who went on to create Maltego), thinking he could make a lighter open source version. For seven years that was Steve’s first and only release until he decided to resume development again in 2012. His work on next generation versions have led SpiderFoot to be cross platform (Python), far more extensible, functional, with a much nicer user interface (UI).
Steve’s current challenge with SpiderFoot is deciding what cool functionality to implement next, his to-do list is ever growing and there are a numerous features he’d love to extend it to include. He typically balances his time between UI/analysis functionality versus new checks to identify more items to aid the penetration tester. The aforementioned OSINT (Open Source Intelligence) community also continues to produce new sources which in turn inspire Steve to build new SpiderFoot checks.
He finds it interesting testing out a new module, and actually finding insightful items out there on the Internet simply during the development process. Steve’s favorite functionality at the moment is identifying owned netblocks, and co-hosted sites. Owned Netblocks indicates entire IP ranges that an organization owns, which enables penetration testers to more completely scan the perimeter of a target. Co-hosted Sites shows you any websites on the same server as the target, which can also be revealing. If your target is hosted on the same server as sites identified as being malicious by the malicious site checker, or the blacklist checker plug-in it could potentially indicate that your target is hosted on a compromised server.
As you read this it’s likely that the following planned enhancements are available in SpiderFoot or will be soon:
·         2.1.2 (early March)
o   SOCKS proxy support
o   Real-time scan progress viewer
o   Identify scan quality impacting issue
o   Autoshun (www.autoshun.org) lookup as part of malicious checks
o   SANS (isc.sans.edu) lookup as part of malicious checks (queue the Austin Powers voice: “Yeah, baby!”)
o   Update GeoIP checker
·         2.1.3 (mid April)
o   VirusTotal, SHODAN, Facebook, Xing, Pastebin and GitHub plug-ins
Note that when you pull SpiderFoot from GitHub, you are downloading a beta version of the next release, as Steve commits new functionality there periodically in preparation for the next version. For instance, SOCKS functionality is in the GitHub repository right now but not in the packaged released version (2.1.1.).
SpiderFoot is a great project with a strong development roadmap, so let’s get down to business and explore.

Quick installation notes

Windows installation is an absolute no brainer; download the package, unpack it, execute sf.exe, and browse to http://127.0.0.1:5001. All dependencies are met including a standalone Python interpreter, so you may find this option optimal.
Linux (I installed it on SamuraiWTF) users need to settle a few dependencies easily solved with the following few steps that assume pip is already installed:
sudo apt-get install swig
sudo pip install mako cherrypy netaddr M2Crypto dnspython
git clone https://github.com/smicallef/spiderfoot.git
cd spiderfoot/
sudo python ./sf.py 0.0.0.0:9999
The last line indicates that you’d like SpiderFoot to bind to all addresses (including localhost) and listen on port 9999. You can define your preferred port or just accept default if undefined (5001). Steve reminds us on his installation page to be cautious regarding exposing SpiderFoot to hostile networks (Intranet, security conference wireless) given that there is currently no authentication scheme.

SpiderFoot unleashed

The SpiderFoot UI is, how shall I say, incredibly simple, intuitive, and obvious even. To start a scan…wait for it…select New Scan. Figure 1 represents a scan being kicked off on my domain (don’t do it) as defined by the By Module view.

FIGURE 1: Kicking off a new scan with SpiderFoot
If you wish to more granularly define your scans, select the By Required Data view (default) then pick and choose your preferred data points including elements such as malicious affiliations, IP data, URL analysis, SSL certificate information, affiliate details, and many other record. You should then be treated to a success message. Scans results are stored in a SQLite DB so over time you’ll likely build up a collection if you don’t purge. Under the Scans tab as seen in Figure 2 you can click the scan in the Name column of the table view and review results. You’ll also note status here and can also halt the scan if need be. I imagine the real-time scan progress viewer will show itself here in the near future as well.

FIGURE 2: SpiderFoot Scans view
If need be (default settings work quite well), you can tune the actual scan configuration as well via Settings, with attention to how you’d like to tune storage, search engines, port scanning, spidering, TLD searches (see Figure 3), amongst others.

FIGURE 3: SpiderFoot Settings view
When my scan completed, with default settings and all checks enabled, the results included 11360 elements. For you data miners, metrics minions, and hosting harvesters, you can export the results to CSV (see Figure 4) and filter by findings type and module, or your preferred data pivot.

FIGURE 4: SpiderFoot results and export functionality
As I navigated all the results, I was intrigued to find a hit for URL (Uses Flash) simply because I didn’t recall any Flash features on my site. I immediately chuckled when I reviewed the result as it was specific to a Flash video I’d created for the 2008 ISSA Northwest Regional Conference wherein I ripped on the now defunct Hacker Safe trustmark for indicating that their customer’s sites were “hacker safe” when, in fact, they were not. Oh, the good old days.
Want to visualize your results? No problem, you can choose from a bubble view of data elements or the discovery path. Figure 5 represents the discovery path for Social Media Presence findings. Hover over each entity for details specific to initial target type, the source module, and the related result.

FIGURE 5: SpiderFoot visualizes a discovery path
SpiderFoot will absolutely uncover nuggets you may have long forgotten about and may want to remove as they are potentially vulnerable (outdated plugins, modules, etc.) or unnecessarily/unintentionally exposed. I found an old dashboard I’d built by hand eons ago with long dead extenal JavaScript calls that had no business still being available. “Be gone!”, I said. That is what SpiderFoot is all about. Add it to the tool collection for penetration tests and OSINT expeditions; you won’t be disappointed.

In Conclusion

Steve Micallef’s SpiderFoot is functionally simple but feature rich and getting better all the time as it is well built and maintained. Follow @binarypool on Twitter and keep an eye out for timely and regular releases.
Ping me via email if you have questions or suggestions for topic via russ at holisticinfosec dot org or hit me on Twitter @holisticinfosec.
Cheers…until next month.

Acknowledgements

Steve Micallef (@binarypool), Spiderfoot author

Moving blog to HolisticInfoSec.io

toolsmith and HolisticInfoSec have moved. I've decided to consolidate all content on one platform, namely an R markdown blogdown sit...