toolsmith: Implementing Redmine for Secure Project Management

Prerequisites/dependencies

VMWare for this methodology or a dedicated installation platform if installed from ISO

Introduction

From Redline for March’s toolsmith to Redmine for April’s, we’ll change pace from hacker space to the realm of secure project management. Following is a shortened version of a much longer Redmine study written for the SANS Reading Room as part of graduate school requirements and released jointly with ISSA.

Security and collaborative project management should not be exclusive. Software designed to support secure project management and security-oriented projects can be both feature rich and hardened against attacks. Web applications such as Redmine offer just such a solution and can embrace the needs of project managers and security practitioners alike. Redmine is project management and bug tracking software built on Ruby on Rails with a focus on collaboration, functionality, and when enhanced with specific plugins, can be configured securely to facilitate security-oriented projects. As a productivity platform, Redmine allows convenient team workflow while embracing the needs of virtual or mobile project members with a focus on socially oriented processes. We’ll explore the secure implementation and configuration of a Redmine server, and then transition into step-by-step details for managing a real world web application penetration testing project using Redmine. This will include the distribution of a virtual machine ready-built for real world use during such projects, pre-configured with a project template based on workflow in the SANS 542 Web Application Penetration Testing course.

From the TurnKey Redmine Web page: “Redmine is a Rails web application that provides integrated project management features, issue tracking, and support for multiple version control programs. It includes calendar and Gantt charts to aid visual representation of projects and their deadlines. It also features multi-project support, role based access control, a per-project wiki, and project forums”. Additionally, a tool such as Redmine allows the convergence of software and security testing. As a software configuration management (SCM) tool, Redmine is ideally suited to projects related to software development. That said, the security expertise required to security test software needs equal consideration and project management. “Sometimes security, or pen-testers for short - work on the same test team as functionality testers; other times, pen-testers work as security consultants and are hired by the software development company to perform security tests”[1]. Regardless of who solicits the use of pen-testers, the related pen-test is a project, and Redmine is the ideal application to provide the agile, flexible platform pen-testers need to coordinate their efforts with the help of a PM or team lead.

Installation

Redmine installation and configuration using a TurnKey Linux Redmine appliance built on a Debian-based Linux distribution, is reasonably straightforward. Your ability to install a Linux operating system from an ISO file on a dedicated machine, or configuring a VMware virtual machine is assumed. It is also assumed you have control of or access to an Active Directory domain for LDAP authentication to Redmine, as it allows for more robust user management. As referenced later, the IP address of the Redmine instance was 192.168.248.16 and 192.168.248.248 for the domain controller. The stable version of the TurnKey virtual Redmine appliance (version 12) running on a lean instance of Debian Squeeze (CLI only, no X11 GUI) via VMWare Workstation 9 was utilized for this research. Note, readers will find running the shell via Putty or a client where you can cut and paste installation strings easier as the VMWare tools aren’t effective without the GUI. This TurnKey Redmine appliance relies on Passenger, a module for Apache that hosts Ruby on Rails applications, and supports the use of SSL/TLS (configured by default) and ModSecurity for better security.

As of this writing the current version of Redmine was 2.2.1 and will be described herein. The installed version of Redmine on the TurnKey appliance is 1.4.4; this guidance will include its upgrade to Redmine 2.2.1.

First, download the Turnkey Linux VM appliance and open it in VMWare.  The first boot routine will ask you to create passwords for the root account, the MySQL root user, and the Redmine admin. When the routine completes you should be presented the TurnKey Linux Configuration Console as seen in Figure 1.

FIGURE 1: TurnKey Linux Configuration Console

In the Hardening section, the process of disabling the services you don’t intend to use will be discussed. Take a snapshot of the virtual machine at this point and name the snapshot Base Install.

Update the Redmine version from a command prompt on the Redmine server as follows:

1.                   apt-get update

2.                   apt-get upgrade

3.                   apt-get install locate

4.                   updatedb

5.                   cd /var/www

6.                   mv redmine redmine -old

7.                   hg clone --updaterev 2.0-stable https://bitbucket.org/redmine/redmine-all redmine

8.                   cp redmine -old/config/database.yml redmine/config/database.yml

9.                   cp -r redmine-old/files/ redmine/files/

10.               chown -R root:www-data /var/www/ redmine

11.               cd redmine

12.               gem install bundler

13.               gem install test-unit

14.               bundle install --without development test rmagick

15.               mkdir public/plugin_assets

16.               rake generate_secret_token

17.               rake db:migrate RAILS_ENV=production

18.               chown -R www-data:www-data files log tmp public/plugin_assets

19.               rake redmine:plugins:migrate RAILS_ENV=production

20.               chmod -R 755 files log/ tmp/ public/plugin_assets

21.               rake tmp:cache:clear

22.               rake tmp:sessions:clear

Run the script /var/www/redmine/script/about to confirm the version upgrade.

LDAP authentication is inherent to Redmine but requires a bit of setup. The example Active Directory domain name utilized via a virtual Windows Server 2008 domain controller was REDMINE. The user redminer was established as the service-like account utilized by Redmine to access the directory. Do not user a domain administrator account for this user. Should your Redmine instance be compromised so too then would be your domain. Via your browser, as the Redmine admin user, navigate to Administration then LDAP Authentication. Refer to the Redmine LDAP Authentication page via the Redmine WIKI but refer to the following example configuration as successfully utilized for this research seen in Figure 2.

FIGURE 2: LDAP Configuration

Select Save then, assuming a correct configuration, you should receive indication of a successful connection when you click Test on the resulting Authentication Modes page.

Refer to the SANS Reading Room version regarding additional installation and hardening steps and don’t skip! These are important:

·         Installation

o   Pixel Cookers theme for a streamlined, tech-centric look as well as the

o   Email settings

o   Plugin installation (Ldap Sync, Scrum2B, Screenshot, Monitoring & Controlling)

·         Hardening

o   Disable unnecessary services

o   Tighten down SSH

o   Restrict Redmine web access to HTTPS only

o   Implement UFW (the uncomplicated firewall)

Engaging With Redmine

Following is a step by step description of a penetration testing engagement where Redmine is utilized to provide project support for a team of three.

The first and most important steps to undertake are the elimination of all unwanted permissions for the Non member and Anonymous roles. Login to Redmine as the admin user and select Administration | Roles and permissions | Non member | Uncheck all | Save. Repeat this process for the Anonyomous roles. These steps will ensure that you don’t inadvertently expose project data to those who don’t have explicit permission to view it. Next, to add users for this project, select Administration | Groups to add a group called PenTesters.  From Administration | Users add three users with appropriately defined login names pentester1 (Fred), pentester2 (Wilma), pentester3 (Barney), and pentestpm (BamBam) and add them to the PenTesters group. Remember these users need to also have been created in the domain you’re utilizing for LDAP authentication. Via the Administration menu, under Projects, create a project called Web Application Pentest. The activities related to this project are drawn directly from tasks outlined in the SANS 542: Web App Penetration Testing and Ethical Hacking course as well as the Samurai Web Testing Framework. Select all Modules and Trackers for the project. You’ll note that Monitoring and Controlling by Project and Scrum2b are available as implemented during the installation phase described earlier. These plugins will be described in more detail as their use is inherent to agile project management for projects such as penetration testing.

Redmine allows the creation of subprojects as well; the Web Application Pentest project should be divided into four subprojects named as follows: 1-Recon, 2-Mapping, 3-Discovery, and 4-Exploitation. Add each of them from Redmine Web Application Pentest project page and remember to enable all Modules and Trackers.

Add the user accounts for the three penetration testers and the project PM user as project and subproject members via the Members tab as seen in Figure 3.

FIGURE 3: Pen-test Project Members

Return to the project overview, select 1-Recon under subprojects, and add a new issue. File a bug for each recon phase task you’d like completed, with the applicable start and due dates. You can upload related files, screenshots (thanks to the plugin installed earlier), and designate an assignee, as well as watchers.

Under Settings for each project or subproject you define you can establish issue categories. This is an ideal method by which to establish penetration testing activities for each subproject. As an example, the recon phase of a web application penetration test includes general recon along with DNS and Whois lookups, search engine analysis, social network analysis, and location analysis. Establishing each of these as issues categories will then allow bugs (tasks) to be filed specific to each category. Each bug can in turn be assigned a pen-tester with start and end dates, along with files that might be useful to complete the task. Location analysis could include gleaning location data from victim Tweets as described in Violent Python[2]. Twitter provides an API to developers which allows information gathering about individuals (potential penetration test targets). A script from Violent Python to help in this information gathering can be uploaded into the Redmine bug, Location data from Tweets as seen in Figure 4.

FIGURE 4: Bug (task) assigned to Fred, with helper code

As bugs are added, assigned, and/or updated, if configured to communicate verbosely, Redmine will email notices to the appropriate parties. The email as seen in Figure 5 was received as a function of filing the bug in Figure 4.

FIGURE 5: Email notice for bug (task) filed

This allows real-time communication among penetration testers or any project participants defined in your Redmine deployment. As pen-testers generate findings, they can be uploaded to the associated bug, and if versioning is required, managed via the Mercurial SCM offering as described during installation.

Bug status can be tracked as New, In Progress, Resolved, Feedback, and Closed or Rejected, and each bug can be assigned a priority and estimated time. As completed, actual time spent on each bug can be tracked too. Overall project time allotments as defined in the bug then track quite nicely via the Redmine Gantt functionality as seen in Figure 6.

FIGURE 6: Redmine Gantt functionality

Scrum2b

The concept of agile software development has, over time, been applied directly to project management. Consider the use of Scrum methodology as part of agile project management. According to Agile Project Management with Scrum, “the heart of Scrum lies in the iteration. The team takes a look at the requirements, considers the available technology, and evaluates its own skills and capabilities. It then collectively determines how to build the functionality, modifying its approach daily as it encounters new complexities, difficulties, and surprises. The team figures out what needs to be done and selects the best way to do it. This creative process is the heart of the Scrum’s productivity”[3]. These creative processes, assessment of capabilities, and changing complexities and surprises are also inherent to any penetration test and as such, the agile project management framework is an ideal way to coordinate pen-test projects. The Scrum2b plugin for Redmine is well suited to answer this calling. If each phase of the pen-test is considered a sprint as defined by the Scrum process, the planning and awareness necessary to support the sprint is essential. The Scrum2b interface is a virtual Scrum Board that allows project participants to track activities by bug and members while editing the bug on the fly with the appropriate permission. The pentestpm user, as project manager, could adjust task’s percentage of completion right from Scrum2b using the time slider.

FIGURE 7: Scrum2b Scrum Board for pen-testers

If the assignee needs to jump right to the bug, the plugin is fully hyperlink enabled. The Scrum Board allows filtering the view by members and issues. New issues can also be added right from the Scrum Board.

Monitoring & Controlling

All projects require the right balance of monitoring and controlling, and penetration tests are no exception. The Monitoring and Controlling Project Work process includes “gathering, recording, and documenting project information that provides project status, measurements of progress, and forecasting to update cost and schedule information that is reported to stakeholders, project team members, management, and others”[4]. The Monitoring & Controlling plugin for Redmine shines in this capacity. Established as a convenient left-pane menu item with the Pixel Cookers theme, this plugin creates a dashboard for project data organized by Tasks Management, Time Management, and Human Resource Management. Tasks Management tracks Tasks by Status, Tasks by Category, and Task Management (manageability). Applied again to the context of a pen-test project, Figure 8 represents the Recon phase of a pen-test.

FIGURE 8: Monitoring & Controlling Tasks Management

Refer again to the SANS Reading Room version, page 17, for more regarding Time & Human Resources Management with the Redmine Monitoring & Controlling plugin.

In Conclusion

Project management includes a certain amount of tedium, but Redmine configured with the aforementioned plugins allows for a refreshing, dynamic approach to the overall secure project management lifecycle. While no system is ever absolutely secure (a serious Ruby on Rails SQL injection flaw was disclosed as this paper was written), the appropriate hardening steps can help ensure enhanced protection. Steady maintenance and diligence will also serve you well. The convenience of an implementation such as TurnKey Redmine makes keeping the entire system up to date quite easy.

A version of a TurnKey Redmine virtual machine as discussed here will be made available to readers via the HolisticInfoSec Skydrive. This instance will include a web application project template, with predefined subprojects, issue categories and bugs, again as defined in the SANS 542 course. Readers will need only create users, assign dates and members, and establish access to an LDAP service.

Ping me via email if you have questions or suggestions for topic via russ at holisticinfosec dot org or hit me on Twitter @holisticinfosec.

Cheers…until next month.

References

---

[1] Gallagher, Jeffries, and Landauer (2006). Hunting Security Bugs, Redmond, WA: Microsoft Press.

[2] O'Connor, T. (2013). Violent python. (p. 229). Walthm, MA: Syngress.

[3] Schwaber, K. (2004). Agile project management with scrum.

[4] Heldman, K. (2009). Pmp: Project management professional exam study guide. (Fifth ed.). Indianapolis, IN: Sybex.

toolsmith: Violent Python - A Book Review Applied to Security Analytics

Prerequisites/dependencies

Python interpreter

BackTrack 5 R3 is ideally suited to make immediate use of Violent Python scripts

Introduction

Happy New Year and congratulations on surviving the end of the world as we know it (nyah, nyah Mayan calendar). Hard to imagine we’re starting yet another year already; 2012 simply screamed by. Be sure to visit the HolisticInfoSec blog post for the 2012 Toolsmith Tool of the Year and vote for your favorite tool of 2012.

I thought I’d start off 2013 with a bit of a departure from the norm. Herein is the first treatment of a book as a tool where the content and associated code can be utilized to perform duties specific to the information security practitioner. I can think of no better book with which to initiate this approach than TJ O’Connor’s Violent Python, A Cookbook for Hackers, Forensic Analysts, Penetration Testers, and Security Engineers. Yes, this implies that you should buy the book; trust me, it’s worth every dime of the $34. Better still, TJ has donated all his proceeds to the Wounded Warrior Project. That said, I’ll post TJ’s three scripts we’ll discuss here so as to whet your appetite. I’ve had the distinct pleasure of working with TJ as part of the SANS Technical Institute’s graduate program where we, along with Beth Binde, wrote AssessingOutbound Traffic to Uncover Advanced Persistent Threat. I’ve known some extremely bright capable information security experts in my day and I can comfortably say TJ is hands down amongst the very best of that small group. As part of his service as an officer in the U.S. Army (hooah) TJ has served as the course director for both computer exploitation and digital forensics at the US Military Academy and as an communications officer supporting tactical communications. His book maps nicely to a philosophy I embrace and incorporate in the workplace. Security monitoring, incident response (and forensics), and attack and penetration testing are the three pillars of security analytics, each feeding and contributing the others in close cooperation. As an example, capable security monitoring inevitably leads to a need for incident response, and after mitigation and remediation have ensued, penetration testing is key to validating that corrective measures were successful, which in turn helps the monitoring team assess and tune detection and alerting logic. Security analytics: the information security circle of life J.

How does a book such as TJ’s Violent Python reverberate with this philosophy? How about entire chapters dedicated to each of the above mentioned pillars, including Python scripts for network traffic analysis (monitoring), forensic investigations (IR), as well as web recon and penetration testing. We’ll explore one script from each discipline shortly, but not before hearing directly from the author:

“In a lot of ways writing a book is a cathartic experience where you capture a lot of things you have done. All too often I'm writing scripts to achieve an immediate effect and then I throw away the script. For me the book was an opportunity to capture a lot of those small projects I've done and simplify the learning curve for others. My favorite example was the UAV takeover in the book. We show how to take over any really Ad-Hoc WiFi toys in under 70 lines of code. A few friends joked that I couldn't write a script in under 100 lines to crash a UAV. This was my chance to provide them a working concept and it worked! Unfortunately it left my daughter with a toy UAV cracked into several pieces as I refined the code. From a defensive standpoint, understanding a scripting language is absolutely essential in my opinion. The ability to parse data such as DNS traffic or geo-locate IP traffic (both shown in the book) can give a great deal of visibility. Forensics tools are great but the ability to build your own are even better. We show how to write tools to parse out iPhone backups for data and scrape for specific objects. The initial feedback from the book has been overwhelming and I've really enjoyed hearing positive feedback. No future plans right now but a good friend of mine has mentioned writing "Violent Powershell" so we'll see where that goes.”    

Violent Python provides readers the basis for scripts to attack network services, analyze digital artifacts, investigate network traffic for malicious activity, and data-mine social media, not to mention numerous other activities. This is a must-read book that includes a companion site with all the code discussed. Let’s take a closer look at three of these efficient and useful Python scripts.

Making Use of Violent Python

As noted above, I’ve posted the three scripts discussed in this section, along with the PCAP and PDF (malicious) discussed on my website. Email or Tweet for the zip passwords.

TJ suggests utilizing a BackTrack distribution given that many of the dependencies and libraries required to use the scripts in this book are inherent to BackTrack. We’ll follow suit on a BackTrack 5 R3 virtual machine. Before beginning, we’ll need to set up a few prerequisites. Execute easy_install pyPDF python-nmap pygeoip mechanize BeautifulSoup4 at the BT5R3 root prompt. This will install pygeoip as needed for our first exercise. I’m going to conduct these exercises a bit out of chapter sequence in order to follow the security analytics lifecycle starting with monitoring. This drops us first into Chapter 4 where we’ll utilize MaxMind’s GeoLiteCity to map IP addresses to cities. In order to do so, you’ll need to set up GeoLiteCity on BackTrack or your preferred system with the following steps:

1.  mkdir /opt/GeoIP

2.  cd /opt/GeoIP/

3.  wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

4.  gunzip GeoLiteCity.dat.gz

You’ll then need to edit line 7 of geoPrint.py to read gi = pygeoip.GeoIP('/opt/GeoIP/GeoLiteCity.dat') or download the updated copy of the script I’ve posted for you.

I’ve created a partially arbitrary scenario for you with which to walk through the security analytics lifecycle using Violent Python. To do so I’ll refer to what was, in 2009 an actual malicious domain, used to host shellcode for PDF-based malware attacks. I grabbed a malicious PDF sample from Contagio, an excellent sample resource. The IP address I associate with this domain is where I am taking creative liberties as the domain we’ll discuss, ax19.cn, no longer exists, and there is no record of what its IP address was when it was in use. The PCAP we’ll use here is one I edited with bittwiste to arbitrarily introduce a suspect Chinese IP address to what was originally a packet capture from a machine compromised by Win32.Banload.MC. I’ve shared this PCAP and the PDF as mentioned above so you can try the Python scripts with them for yourself.  

In this scenario, your analysis machine is Linux only. Just you, a Python interpreter, and a shell; no fuss, no muss.

As we’re starting in the monitoring phase, imagine you have a network for which the traffic baseline is well understood. You can assert, from one particular high value VLAN, that at no time should you ever see traffic bound for China.  Your netflow monitoring for that VLAN is showing far more egress traffic bound for IP space that is not on your approved list established from learned baselines. You initiate a real-time packet capture to confirm. Capture (suspect.pcap) in hand, you’d like to validate that the host is indeed conversing with an IP address in China. Violent Python’s geoPrint.py script is a great place to start as it leverages the above-mentioned GeoLiteCity data from MaxMind along with the PyGeoIP library from Jennifer Ennis and dpkt. Execute python geoPrint.py -p suspect.pcap and you’ll see results as noted in Figure 1.

Figure 1: geoPrint.py confirms Chinese takeout

Your internal host (RFC 1918, and thus unregistered) with IP address 192.168.248.114 is clearly conversing with 116.254.188.24 in Beijing. Uh-oh.

Your team now moves into incident response mode and seizes the host in question. You interview the system’s user who indicates they received an email what the user thought was a legitimate help desk notification to read a new policy. The email had an attached PDF file which the user downloaded and opened. Your suspicions are heightened, as such you grab a copy of the PDF and head back to your analysis workstation. You’re interested to see if there is any interesting metadata in the PDF that might help further your investigation. You refer to Chapter 3 of Violent Python which discusses Forensic Investigations with Python. The pdfRead.py script incorporates the PyPDF library which allows you to extract PDF document information (metadata) in addition to other capabilities. Execute python pdfRead.py -F suspect.pdf and dump the metadata as seen in Figure 2.

Figure 2: pdfRead.py dumps suspect PDF metadata

The author reference is a standout for you; from a workstation with a browser you search “Zeon Technical Publications” and find reference to it on VirusTotal and JSunpack; these results along with a quick MD5sum hash match indicate that this PDF is clearly malicious. The JSunpack reference indicates that shellcode phones home to www.ax19.cn (see Figure 3), a domain for which you’d now like to learn more.

Figure 3: JSunpack confirms an evil PDF

You could have sought anonymity to conduct the above mentioned search, which lead us to the third pillar of our security analytics lifecycle. This third phase here includes web recon as discussed in Chapter 6 of Violent Python, a common step in the attack and penetration testing discipline, to see what more we can learn about this malicious domain. As we often seek anonymity during the recon phase, Violent Python allows you maintain a bit of stealth by leveraging the deprecated Google API against which a few queries a day can still be executed. The newer API requires a developer’s key which one can easily argue is not anonymous. Executing python anonGoogle.py -k 'www.ax19.cn' will return yet another validating result as seen in Figure 4.

Figure 4: anonGoogle matches ax19.cn to malicious activity

With seven rich chapters of Python goodness, TJ’s Violent Python represents a golden opportunity to expanding your security analytics horizons. There is so much to learn from here while accentuating your use of Python in your information security practice.

In Conclusion

I’m hopeful this slightly different approach to toolsmith was useful for you this month. I’m looking to shake things up a bit here in 2013 and am certainly open to suggestions you may have regarding ideas and approaches to doing so. Violent Python was a great read for me and a pleasure to put to use for both this article as well as in my personal tool box. I’m certain you’ll find this book equally useful.

Ping me via email if you have questions (russ at holisticinfosec dot org).

Cheers…until next month.

Acknowledgements

TJ O’Connor, Violent Python author

Mila Parkour, Contagio

toolsmith: Memory Analysis with DumpIt and Volatility

Sept. 11, 2001: “To honor those whose lives were lost, their families, and all who sacrifice that we may live in freedom. We will never forget.“



Reprinted with permission for the author only from the September 2011 ISSA Journal

Prerequisites

SIFT 2.1 if you’d like a forensics-focused virtual machine with Volatility ready to go
Python version 2.6 or higher on Window, Linux, or Mac OS X
Some plugins require third party libraries

Introduction

Two recent releases give cause for celebration and discussion in toolsmith. First, in July, Matthieu Suiche of MoonSols released DumpIt for general consumption, a “fusion of win32dd and win64dd in one executable.” Running DumpIt on the target system generates a copy of the physical memory in the current directory. That good news was followed by Ken Pryor’s post on the SANS Computer Forensics Blog (I’m a regular reader, you should be too) mentioning the fact that Volatility 2.0 had been released in time for the Open Memory Forensics Workshop, and that SIFT 2.1 was also available. Coincidence? I think not; Volatility 2.0 is available on SIFT 2.1. Thus, the perfect storm formed creating the ideal opportunity to discuss the complete life-cycle of memory acquisition and analysis for forensics and incident response. In May 2010, we discussed SIFT 2.0 and mentioned how useful Volatility is, but didn’t give its due. Always time to make up for our shortcomings, right?
If you aren't already aware of Volatility, “the Volatility Framework is a completely open collection of tools, implemented in Python under the GPL, for the extraction of digital artifacts from volatile memory (RAM) samples.”
One thing I’ve always loved about writing toolsmith is meeting people (virtually or in person) who share the same passion for and dedication to our discipline. Such is the case with the Volatility community.
As always, I reached out to project leads/contributors and benefited from very personal feedback regarding Volatility. Mike Auty and Michael Hale Ligh (MHL) each offered valuable insight you may not glean from the impressive technical documentation available to Volatility users.
Regarding the Volatility roadmap, Mike Auty indicated that the team has an ambitious goal for their next release (which they want to release in 6 months, a big change from their last release). They're hoping to add Linux support (as written by Andrew Case), as well as 64-bit support for Windows (still being written), and a general tidy up for the code base without breaking the API.
MHL offered the following:
“At the Open Memory Forensics Workshop (OMFW) in late July, many of the developers sat on a panel and described what got them involved in the project. Some of us are experts in disk forensics, wanting to extend those skills to memory analysis. Some are experts in forensics for platforms other than Windows (such as Linux, Android, etc.) who were looking for a common platform to integrate code. I personally was looking for new tools that could help me understand the Windows kernel better and make my training course on rootkits more interesting to people already familiar with running live tools such as GMER, IceSword, Rootkit Unhooker, etc. I think the open source nature of the project is inviting to new-comers, and I often refer to the source code as a Python version of the Windows Internals book, since you can really learn a lot about Windows by just looking at how Volatility enumerates evidence.”
Man, does that say it all! Stay with this thinking and consider this additional nugget of Volatility majesty from MHL. In his blog post specific to using Volatility to detect Stuxnet, Stuxnet's Footprint in Memory with Volatility 2.0, he discusses Sysinternals tools side-by-side with artifacts identified with Volatility. MHL is dead on right when he says this may be of “interest your readers, especially those who have never heard of Volatility before, because it builds on something they do know - Sysinternals tools.”
This was an incredibly timely post for me as I read it right on the heels of hosting the venerable Mark Russinovich at the ISSA Puget Sound July chapter meeting where he presented Zero Day Malware Cleaning with the Sysinternals Tools, including live analysis of the infamous Stuxnet virus.
See how this all comes together so nicely?
Read Mark’s three posts on Technet followed immediately by MHL’s post on his MNIN Security Blog, then explore Volatility for yourself; I’ll offer you some SpyEye analysis examples below.
NOTE: MHL was one of the authors of Malware Analyst's Cookbook and DVD: Tools and Techniques for Fighting Malicious Code; I’ll let the reviews speak for themselves (there are ten reviews on Amazon and all are 5 stars). I share Harlan’s take on the book and simply recommend that you buy it if this topic interests you.
Some final thoughts from AAron Walters, the principal developer and lead for Volatility:
“We have a hard working development team and it’s appreciated when people recognize the work that is being done. The goal was to build a modular and extendable framework that would allow researchers and practitioners come together and collaborate. As a result, shortening the amount of time it takes to get cutting edge research into the hands of practitioners. We also wanted to encourage and push the technical advancement of the digital forensics field which had frequently lagged behind the offensive community. It's amazing to see how far the project has come since I dropped the initial public release more than 4 years ago. With the great community now supporting the project, there are lot more exciting enhancements in the pipe line...”

DumpIt

Before you can conduct victim system analysis you need to capture memory. Some form of dd, including MoonSols win32dd and win64dd were/are de facto standards but the recently released MoonSols DumpIt makes the process incredibly simple.
On a victim system (local or via psexec) running DumpIt is as easy as executing DumpIt.exe from the command-line or Windows Explorer. The raw memory dump will be generated and written to the same directory you’re running DumpIt from; answer yes or no when asked if you wish to continue and that’s all there is to it. A .raw memory image named for the hostname, date, and UTC time will result. DumpIt is ideal for your incident response jump kit; deploy the executable on a USB key or your preferred response media.


Figure 1: Run DumpIt

Painless and simple, yes? I ran DumpIt on a Windows XP SP3 virtual machine that had been freshly compromised with SpyEye (md5: 00B77D6087F00620508303ACD3FD846A), an exercise that resulted in my being swiftly shunted by my DSL provider. Their consumer protection program was kind enough to let me know that “malicious traffic was originating from my account." Duh, thanks for that, I didn’t know. ;-)
Clearly, it’s time to VPN that traffic out through a cloud node, but I digress.
SpyEye has been in the news again lately with USA Today Tech describing a probable surge in SpyEye attacks due to increased availability and reduced cost from what used to be as much as $10,000 for all the bells and whistles, down to as little as $95 for the latest version. Sounds like a good time for a little SpyEye analysis, yes?
I copied the DumpIt-spawned .raw image from the pwned VM to my shiny new SIFT 2.1 VM and got to work.

Volatility 2.0

So much excellent documentation exists for Volatility; on the Wiki I suggest you immediately read the FAQ, Basic Usage, Command Reference, and Features By Plugin.
As discussed in May 2010’s toolsmith on SIFT 2.0, you can make use of Volatility via PTK, but given that we’ve discussed that methodology already and the fact that there are constraints imposed by the UI, we’re going to drive Volatility from the command line for this effort. My memory image was named HIOMALVM02-20110811-165458.raw by DumpIt; I shortened it to HIOMALVM02.raw for ease of documentation and word space.

I executed vol.py imageinfo –f HIOMALVM02.raw to confirm just that, image information. This plugin provided PAE (physical address extension) status as well as hex offsets for DTB (Directory Table Base), KDBG (short for _KDDEBUGGER_DATA64), KPCR (Kernel Processor Control Region), time stamps and processor counts.


Figure 2: imageinfo plugin results

Windows XP SP3, check.
Runtime analysis of my SpyEye sample gave me a few queryable entities to throw at Volatility for good measure, but we’ll operate here as if the only information we have only suspicion of system compromise.
It’s always good to see what network connections may have been made.

vol.py --profile=WinXPSP3x86 connscan -f HIOMALVM02.raw

The connscan plugin scans physical memory for connection objects.
Results included:


Interesting, both IPs are in Germany, my VMs don’t make known good connections to Germany so let’s build from here.
The PID associated with the second connection to 188.40.138.148 over port 80 is 1512.
The pslist plugin prints active processes by walking the PsActiveProcessHead linked list.

vol.py --profile=WinXPSP3x86 pslist -P -f HIOMALVM02.raw

Use –P to acquire the physical offset for a process, rather virtual which is default.
Results included a number of PPID (parent process IDs) that matched the 1512 PID from connscan:


I highlighted the process that jumped out at me given the anomalous time stamp, a 0 thread count and no handles.
Let’s check for additional references to cleansweep.
The pstree plugin prints the process list as a tree so you can visualize the parent/child relationships.

vol.py --profile=WinXPSP3x86 pstree -f HIOMALVM02.raw

Results included the PPID of 1512, and the Pid for cleansweep.


Ah, the victim most likely downloaded cleansweep.exe and executed it via Windows Explorer.
But can we extract actual binaries for analysis via the like of Virus Total? Of course.
This is where the malware plugins are very helpful. I already know I’m not going to have much luck exploring PID 3328 as it has no threads or open handles. MHL points out that a process such as cleansweep.exe typically can't remain active with 0 threads as a process is simply a container for threads, and it will terminate when the final thread exits. Cleansweep.exe is still in the process list probably because another component of the malware (likely the one that started cleansweep.exe in the first place) never called CloseHandle to properly "clean up." That said, the PPID of 1512 has clearly spawned PID 3328 so let’s explore the PPID with the malfind plugin, which extracts injected DLLs, injected code, unpacker stubs, and API hook trampolines. The malware (malfind) plugins
don't come packaged with volatility, but are in fact a part of the above mentioned Malware Analyst's Cookbook; the latest version can also be downloaded.

vol.py --profile=WinXPSP3x86 -f HIOMALVM02.raw malfind -p 1512 -D output/ yielded PE32 gold as seen in Figure 3.


Figure 3: malfind plugin results

Malfind dropped each of the suspicious PE files it discovered to my output directory as .dmp files. I submitted each to Virus Total, and bingo, all three were malicious and identified as SpyEye variants as seen in Figure 4.


Figure 4: PE results from Virus Total

In essence, we’ve done for ourselves via memory analysis what online services such as Threat Expert will do via runtime analysis. Compare this discussion to the Threat Expert results for the SpyEye sample I used.
There is so much more I could have discussed here, but space is limited and we’ve pinned the VU meter in the red, so go read the Malware Cookbook as well as all the online Volatility resources, and push Volatility to the boundaries of your skill set and imagination. In my case the only limiting factors were constraints on my time and my lack of knowledge. There are few limits imposed on you by Volatility; 64bit and Linux analysis support are pending. Get to it!

In Conclusion

I’ve said it before and I’ll say it again. I love Volatility. Volatility 2.0 makes me squeal with delight and clap my hands like a little kid at the state fair. Oh the indignity of it all, a grown man cackling and clapping when he finds the resident evil via a quick memory image and the glorious volatile memory analysis framework that is Volatility.
An earlier comment from MHL bears repeating here. Volatility source code can be likened to “a Python version of the Windows Internals book, since you can really learn a lot about Windows by just looking at how Volatility enumerates evidence.” Yeah, what he said.
Do you really need any more motivation to explore and use Volatility for yourself?
There’s a great list of samples to grab and play with. Do so and enjoy! As it has for me, this process will likely become inherent to your IR and forensic efforts, perhaps even surpassing other tactics and methods as your preferred, go-to approach.
Ping me via email if you have questions (russ at holisticinfosec dot org).
Cheers…until next month.

Acknowledgements

Mike Auty & Michael Hale Ligh of the Volatility project.
AAron Walters – Volatility lead