It's about time...

As we mentioned in a previous blog post and in our presentations, we have recently been focusing our attention on the Reconstruction Phase of the digital investigation process. During the Reconstruction Phase, a digital investigator will attempt to organize the analysis results to help develop a theory about what happened during an incident. One method investigators have traditionally used to organize file system analysis is to elucidate the temporal relationships between digital artifacts. This technique is referred to as temporal reconstruction.

Dan Farmer demonstrated the usefulness of temporal reconstruction of filesystem events with the 'mactime' program. In fact, he called mactime "the most potentially valuable forensic tool in your digital detective toolkit" (Farmer, 2000). Rob Lee eventually extended this work with the 'mac_daddy' program and finally these tools were combined by Brian Carrier into the SleuthKit's versions of 'mactime' and 'mac-robber'. Recently, Florian Buchholz has also done a lot of interesting research exploring the characteristics of these temporal relationships and demonstrating the value of being able to combine disparate data sources, Zeitline.

In this blog entry, we will demonstrate how digital artifacts extracted from volatile memory analysis can be combined with artifacts from file system analysis to help reconstruct a more complete understanding of the digital crime scene. In fact, volatile memory analysis often provides the context necessary to link seemingly disparate events and their related artifacts, in ways that are not possible with typical live response tools. Using these temporal relationships, we have also been able to develop "temporal incident patterns" allowing us to quickly discern tools and techniques that may have been involved in an incident based on their "temporal footprints". We have also found that the ability to visualize these temporal relationships is invaluable for both presentation and knowledge discovery.

The following images will help demonstrate how a digital investigator can use both volatile memory analysis and visualization to improve temporal reconstructions of the digital crime scene. The file system events used to populate the time line in the images were generated using the Sleuthkit's 'mactime' program. These instantaneous events are represented in the image with blue dots and relate to the time attributes (LastWriteTime, LastAccessTime, CreationTime, etc) associated with files and directories in a file system, MACtimes. The following image is a visual time line representation of a filtered set of file system events.



In the next image, we augment the time line with events extracted using live response techniques, one type of run time state analysis. Live response allows us to extract events about objects that were active on the system when acquisition was performed. This could be extracted with your typical live response toolkit (RAPIER, WFT,etc). The red dots in this image are used to denote when a process was created. Unlike the file system events, this is a duration event since it has both a start time and an end time. In this image, the end time relates to when the live response was performed, represented by the gray dot. The green dot, another instantaneous event, represent when a process binds a specific port address to its socket. This augmented time line can be seen in the following image.






The final image demonstrates how using volatile memory (RAM) analysis to perform run time state analysis can be used to further augment our temporal reconstruction of the digital crime scene. In this case, the temporal events were extracted from volatile memory using Volatility. In contrast to the previous image, we are not only able to augment the time line with those objects that where active when live response was performed but also with objects that may have been relinquished by the operating system. The blue dots in the image once again represent file system events. The red dots represent process creation events, except this time a process duration event ends with memory acquisition or when a process exited. The green dots relate to binding sockets and the grey dot relates when memory acquisition was performed. The final augmented time line can be seen in the following image.





The purpose of this blog entry was to demonstrate the usefulness of being able to augment temporal reconstruction with both visualization and volatile memory analysis. In the final image, we can easily see how including volatile memory analysis and visualization allow us exploit temporal locality and volatile context to develop theories about the incident. We have found this to be invaluable during the reconstruction phase of the digital investigations process.


Despite the usefulness of these techniques, it is important to keep in mind that timestamps can be manipulated by a determined adversary and recently tools, such as timestomp, have been created to frustrate temporal reconstructions of filesystems. Recent research has also discussed important considerations for the digital investigator as they work with temporal data. Temporal reconstruction is not the panacea but a digital investigator should combine many types of analysis techniques during their digital investigations.

More details to follow ...

Commercial Support for Volatility!

While at DoD Cyber Crime last week, numerous members of the Volatility community made me aware of a company attempting to spread misinformation about Volatility. It was broadly suggested that there was no support being offered for Volatility. The goal behind the open development of Volatility was to bring together systems researchers who believed in bettering the state of the digital forensics community. One way that we have been able to continue this open development is by offering customizations and support.

Volatile Systems, LLC has been providing commercial support and maintenance for Volatility (and our other products) for the past 8 months. In fact, one of the main reasons Volatile Systems, LLC formed was to support the forensics needs of our users who required commercial support contracts. The added benefit of our commercial support contracts is that you are not only getting guaranteed support and access to our team of unparalleled memory analysts, but you are also actively contributing back to the volatile memory analysis community by allowing us to continue the open development of Volatility.

At this point, we also decided to extend a new offer to those who may be considering spending the thousands of dollars to purchase one of those other commercial products, as they become available. If you are considering investing in one of those products because you think it provides extraction functionality not currently supported in Volatility, contact us and let us know! In most cases, we would be more than willing to use those funds to build you custom modules providing the same capabilities you desire but tailored to your exact needs. In addition, we would provide you access to the source code, training on how to use the modules, and share information on how they were developed. As we have learned from our experience performing volatile memory analysis, the most valuable thing is often not the tool but the experience and training of the analyst. Knowledge is power!

On a tangential note, it was encouraging to get all the positive feedback about Volatility at the conference. We are committed to this growing open community of volatile memory analysts and we are highly appreciative of their support. I also wanted to extend a special thanks to the Volatility community for keeping me updated on this evolving issue. Little do they know, the Order of Volatility is everywhere!

They are playing you for a fool!

I have previously talked about this issue before, but based on a number of conversations I had last week at Cyber Crime, I felt it was worth bringing up again. Every time this issue comes up, it reminds me of one of my favorite blog posts, which talks about the ethical conflict in the rootkit community. I also recently came across this blog post from my former advisor, Spaf, which I found relevant as well.

One of the main reason why I dedicated myself to researching volatile memory analysis was the fact that the offensive communities and projects were flourishing. As a result, the sophistication of methods and accessibility to knowledge was continuing to grow unabated in the offensive community. At the time, I felt we drastically needed to have a similar revolution in the defensive community. A way of bringing together strong systems researchers who were interested in securing our infrastructure.

Based on the research we were doing at the time, I knew that volatile memory analysis would be an important component of securing those systems and had the potential to disrupt much of the offensive research being performed. As a result, members of our project have spent a great deal of time over the last couple of years writing research papers, giving talks, educating, and developing an open source architecture, in order to inspire research and increase the communal knowledge of the investigative community. In the process, we have had over 20 different contributors from multiple countries across the world. This includes contributions from numerous law enforcement and forensic agencies. In fact, I have been contacted by many universities that are now, or soon will be, using Volatility in their digital forensic courses.

It seems that the work being done in the live memory analysis community has also been successful at getting the attention of the offensive community (esp. rootkit). In fact, they have attempted many times in the last couple of years to disrupt the communal aspects of these projects. They began by trying to convince people that volatile memory analysis wouldn't work and was ineffective. Their methods changed last year, when they began trying to deceptively patent techniques that members of the volatile memory analysis community had already presented at conferences. Recently, I have learned that they are now trying to use their companies as real life Trojan horses to undermine and divide the open nature of the volatile memory analysis community. They are now trying to sell the techniques they had previously argued were ineffective. Once again, trying to capitalize on the problem they created.

Let's consider the following analogy:

Sadly, your child has been struggling with drug addiction for a number of years. He was recently busted by the police and mandated by the court to attend drug rehabilitation. Your child's drug dealer was a notorious individual by the name of B.S. Hary. B.S. Hary has never hidden the fact that he sells drugs and, in fact, even wrote a book and teaches classes about advanced drug dealing techniques. Often flaunting his drug dealing in the face of local law enforcement, who are overburdened dealing with the myriad of drug dealing pupils B.S has released on the streets. As a result, B.S. Hary's drugs and drug dealing techniques account for the majority of the drug problem currently faced by your community.

Recently, B.S. became concerned about the popularity of drug rehabilitation in pop culture. On the one hand, he realized that rehabilitation could be bad for business, but he also figured there was a lot money to be made in rehabilitation. As a result, he decided that he could not sit idly by and watch his drug business be swept out from under him, so he formulated a plan. He decided to capitalize on the rehabilitation market while undermining its effectiveness by starting his own rehabilitation company called Addiction Responder. B.S. Hary even had the brazenness to open Addiction Responder right next door to his crack house.

B.S. Hary is hoping to play the community for a fool!

As a parent, would you be willing to send your child to the Addiction Responder clinic? Knowing that Addiction Responder is run by a notorious drug dealer, do you think the court would be willing to trust a report that acknowledges your child's successful completion of its drug rehabilitation program? Knowing that the owner of Addiction Responder has a crack house right next door to the clinic, do you think the court would have faith in the fidelity of Addiction Responder's rehabilitation capabilities? Knowing that B.S. sells manufactured drugs out of the crack house right next door, would you be willing to ingest his magic rehabilitation pills? Knowing that the money you give to Addiction Responder for rehabilitation will be used to further his drug cartel, will you be willing to help fund the problem that is tormenting both your family and your community?

Your child's current drug dealer wants to perform his rehab.
You said, no, no, no!

On that note, it seems utterly absurd to me that anyone would consider buying volatile memory (RAM) forensics tools from an organization that freely admits to having armed and which continues to arm the enemy with "technology being used to evade forensics and response today." As a taxpayer, I'm not happy to see that all the government funding they have received for research and development has contributed to the majority of the rootkits currently found on the Internet today. As a person involved in forensic investigations, I would not want to be the person responsible for presenting those tools or results in court.

Defense Attorney: Is it true that developers of this "investigation" tool are responsible for the techniques found in the majority of rootkits found on the Internet today?
Forensic Examiner: Yes.
Defense Attorney: Is it true that the makers of this tool also sell "undetectable" software agents that allows people to secretly spy on a person/companies computers (similar to malware or spyware)?
Forensic Examiner: Yes.
Defense Attorney: Do the developers of this software also develop tools to exploit software, cheat at online games, and build rootkits?
Forensic Examiner: Yes.


One of the most important things that I have learned from the forensics and digital investigation communities is that the integrity and trust that can be placed in the collected evidence is often the most important standard. I have been confronted with many situations where we have had to forgo certain types of evidence, because it had the potential to compromise the integrity of investigation and/or case. How would you like to walk into court knowing that the evidence you collected and analyzed will immediately be called into question and, as a result, ruin the case? What happens when the malware you are investigating, as part of an incident, was written by the same people who wrote your forensic tool? Can you trust that they weren't involved?

The question is, are you willing to listen to B.S. and be played the fool?

And you wonder why I'm angry.....

PyFlag Using the Volatility Framework!

It was only a matter of time....

In case you might have missed it during the holidays, the latest version of PyFlag now leverages the Volatility Framework to add volatile memory analysis (RAM Forensics) to it's outstanding list of capabilities. As a result, making PyFlag the first and only tool publically available that allows the digital investigator to correlate disk images, log files, network traffic, and RAM captures all within an intuitive interface. While the current functionality is still preliminary, just imagine the possibilities!

Since PyFlag loads memory images through its standard IO source interface, it is also now possible to store your memory images using the EWF format, commonly used in commercial tools. Once the memory image is uploaded to PyFlag, information can either be accessed through a browseable /proc interface or through the Stats view. Michael Cohen and his team have provided a tutorial and image to get you started.






As I mentioned in a previous post, a special thanks to Europol for bringing our teams together through the High Tech Crime Expert Meeting. I also want to thank Michael Cohen for the great work he has done with PyFlag and his contributions to Volatility! Stay tuned for further exciting collaborations and future Volatility releases in 2008!

The Hague

I've just returned from a wonderful trip to the Netherlands. I want to thank the extremely nice people from Europol who invited me to brief their high tech crime experts about the latest advancements in volatile memory analysis. It's encouraging to see that the major police organizations of the European Union appreciate both the importance of and potential for volatile memory analysis. They also understand the role it will play in the future of digital investigations.

During this trip, I also had the pleasure of meeting a number of people whom I've exchanged emails with over the years. For many of them, it was great to finally be able to associate a face with a name. In fact, this was definitely one of the major highlights of my trip. In particular, I look forward to future collaborations with both Jon Evans and Michael Cohen, who both gave outstanding presentations.

While in the Netherlands, I also had the opportunity to meet up with Robert Jan Mora, Bas Kloet, and Joachim Metz from Hoffmann Investigations. Hoffmann Investigations is lucky to have such a bright group of forensics researchers. Not to mention, they are also really good at reading a menu for those not brave enough for the mystery meal! Hopefully, they will make it back to the US for DFRWS 2008.

2008 Cyber Crime Conference

The agenda for the 2008 DoD Cyber Crime Conference has been posted. I'll be giving a talk during the Research and Development Track at 0830 January 16, 2008. In this talk I will be discussing the latest advancements in the area of Volatile Memory Analysis and how they affect the way we perform digital investigations.

Title:
Advanced Volatile Memory Analysis

Abstract:
This session will focus on advanced techniques being used in volatile memory analysis (VMA) and our experiences while performing VMA. We will also discuss a number of open source tools and resources we have made available to the digital investigation community. The session will also explore how we are using VMA to perform automated malware analysis. Finally, we will demonstrate how we are combining VMA with file system analysis to help reconstruct and visualize the digital crime scene.

DFRWS 2007

Let me begin by saying that DFRWS is one of my favorite conferences of the year. I highly recommend the conference to anyone interested in the latest research being done in the forensics community. DFRWS has established itself as the main venue for research being done in volatile memory analysis and is able to draw researchers working in this area from across the world. The organizing committee does an excellent job and the conference keeps getting better every year. In particular, Matthew Geiger and rest of the CERT team did a wonderful job in hosting the conference this year.

The keynote address was "Digital Forensics, Covert Monitoring, and Active Methods" given by Greg Hoglund. Greg discussed what "bad guys" are currently doing and the importance of digital forensics for dealing with these sophisticated attackers. In particular, he emphasized the importance of volatile memory analysis for detecting these attackers. His presentation also discussed his company's "surveillance" technology that is capable of defeating all conventional anti-virus. During questions Simson asked, how this "surveillance" technology is different than what the "bad guys" are trying to build? He also asked, what differentiates them from the "bad guys"? Finally, he wanted to know if it was merely related to the fact that they worked for a different "mob"!

I thought the keynote was entertaining and Greg has a unique perspective, but I would have preferred a keynote from someone who is actually involved in the forensic research community or at least a practitioner in the area. It was too bad that Karl Levitt had to cancel!! On the other hand, it is interesting that this is the third talk this summer by people who are arming the malware community, discussing the importance of volatile memory analysis. At SyScan, in the talk "The IPO of the 0 day" Justine Aitel of Immunity mentioned "think memory dump analysis" when performing incident response. At Black Hat, Jamie Butler of Mandiant also discussed the importance of volatile memory analysis and demonstrated his desire to stunt research and collaboration by filing patents on other peoples work (I hope those bonuses are worth it!). Now that the offensive community is beginning to focus their attention on volatile memory analysis, hopefully more investigators will realize the importance of integrating volatile memory analysis into the digital investigation process.

The first talk to discuss volatile memory analysis was "Forensic Memory Analysis: From Stack and Code to Execution History", which was presented by Mohamed Saleh. Personally, I liked the fact that the talk focused more on the analysis being done than on how data structures can be extracted from memory. In the talk they discuss a formal approach to analyzing stack memory of process threads to infer execution history. I also like the approach of combining volatile memory analysis with static analysis. We have been leveraging a similar approach with the FATKit research and found it very powerful. I wish the student who worked on the research had been there to present it. I would have enjoyed discussing some of the issues we faced when analyzing larger and more complex programs.

Another intriguing talk was given by Brendan F Dolan-Gavitt called "The VAD Tree: A Process-Eye View of Physical Memory". Brendan is one of the main people that Jamie ripped off in his Black Hat talk, so it was good to finally see him present his own work. His talk discussed how an investigator can use the Virtual Address Descriptor (VAD) trees found in memory to improve volatile memory analysis. He discussed how to parse the tree and a number of different antiforensic techniques that an investigator needs to be aware of. He also mentioned how the VAD can be used to augment the techniques we presented in "FATKit: Detecting Malicious Library Injection and Upping the 'Anti'" to detect advanced DLL injection attacks. Brendan did some great work on this research and is a really nice guy. Not to mention, he also helped integrate these tools into Volatility. I can't wait to see what he decides to work on next.

The final talk relevant to volatile memory analysis was presented by Bradley Schatz, who presented "BodySnatcher: Towards reliable volatile memory acquisition by software". Bradley began by presenting an abstract model for comparing volatile memory acquisition techniques. Then Bradley presented a new method for memory acquisition that attempts to snatch control of the host hardware from the running OS. A host specific kernel driver is used to load a minimized acquisition focused operating system into memory, it then halts the running kernel and switches control to the acquisition operating system. The acquisition operating system then preserves the host memory, initializes an output device, and copies the memory image to the output device. Finally he compares his software based acquisition method with using Garner's dd. Surprisingly, the paper concludes that using BodySnatcher is less obtrusive than running Garner's dd. I am astounded by the fact that loading an entirely new OS would be less obtrusive than running a single userland application. There is also another difference which must be mentioned: Once BodySnatcher runs, it freezes the state of the system and you will not be able to unfreeze the system. It will be exciting to see if Bradley will be able to overcome the limitations that he mentions in the paper and address the comments made during the presentation. I also want to wish Bradley luck with his new company Evimetry.

During the Work in Progress session, I gave a quick talk on Volatility and I discussed some of the things we are currently working on. The first being a collaboration with Doug White of NIST to make some new resources available to the volatile memory analysis community. The second being work that we are doing on malware analysis. Finally, I showed a screen shot of a visualization tool we have built for combining filesystem and volatile memory analysis to visualize temporal relationships during investigations. I will discuss this tool in more detail in an upcoming post.

Golden Richard and Vassil Roussev also gave an very intriguing WIP on the persistence of volatile evidence. During this talk they discussed research they had performed with their students to determine how long artifacts exist in memory once power has been removed from the machine. They presented some interesting results related to different types of machines and how long digital artifacts can be recovered once power is removed. I have heard numerous people speculate about this, so it is nice to see people actually taking time to investigate it.

If you are interested, the DFRWS 2007 papers have been posted. Hope to see you next year at DFRWS 2008 in Baltimore!!!