ArticlePDF Available

Validation of Forensic Tools and Software: A Quick Guide for the Digital Forensic Examiner

Authors:

Abstract

Real world laboratory use, controlled internal tests utilizing scientific principles, and peer review should all be leveraged in a validation test plan. Sharing unique results with the digital forensics community at-large helps investigators, examiners, and even software and tool vendors ensure that current best practices are followed. As the field of digital forensics continues to grow and evolve as a science the importance of proper scientific validation will be more important than ever.
Tools and software for digital forensic analysis
should be validated quarterly.
Tools and software for digital forensic analysis should be
validated quarterly.
DEEPER INSIGHTS
The Importance of
Mobile Forensics for
Law Enforcement
Validation of Forensic Tools and
Software: A Quick Guide for the
Digital Forensic Examiner
With the field of digital forensics
growing at an almost warp-like
speed, there are many issues out
there that can disrupt and discredit
even the most experienced forensic
examiner. One of the issues that
continues to be of utmost
importance is the validation of the
technology and software associated
with performing a digital forensic
examination. The science of digital
forensics is founded on the
principles of repeatable processes and quality evidence.
Knowing how to design and properly maintain a good
validation process is a key requirement for any digital forensic
examiner. This article will attempt to outline the issues faced
when drafting tool and software validations, the legal
standards that should be followed when drafting validations,
and a quick overview of what should be included in every
validation.
Setting the Standard: Standards and Legal Baselines for
Software/Tool Validation
According to the National Institute of Standards and
Technology (NIST), test results must be repeatable and
reproducible to be considered admissible as electronic
evidence. Digital forensics test results are repeatable when the
same results are obtained using the same methods in the
same testing environment. Digital forensics test results are
reproducible when the same test results are obtained using the
same method in a different testing environment (different
mobile phone, hard drive, and so on). NIST specifically defines
these terms as follows:
Repeatability refers to obtaining the same results when using
the same method on identical test items in the same
laboratory by the same operator using the same equipment
within short intervals of time.
Reproducibility refers to obtaining the same results being
obtained when using the same method on identical test items
in different laboratories with different operators utilizing
different equipment.
In the legal community, the Daubert Standard can be used for
guidance when drafting software/tool validations. The Daubert
Standard allows novel tests to be admitted in court, as long as
certain criteria are met. According to the ruling in Daubert v.
Merrell Dow Pharmaceuticals Inc. the following criteria were
identified to determine the reliability of a particular scientific
technique:
1. Has the method in question undergone empirical
testing?
2. Has the method been subjected to peer review?
3. Does the method have any known or potential error
rate?
4. Do standards exist for the control of the technique's
operation?
5. Has the method received general acceptance in the
relevant scientific community?
The Daubert Standard requires an independent judicial
assessment of the reliability of the scientific test or method.
This reliability assessment, however, does not require, nor
does it permit, explicit identification of a relevant scientific
community and an express determination of a particular
degree of acceptance within that community. Additionally, the
Daubert Standard was quick to point out that the fact that a
theory or technique has not been subjected to peer review or
has not been published does not automatically render the
tool/software inadmissible. The ruling recognizes that scientific
principles must be flexible and must be the product of reliable
principles and methods. Although the Daubert Standard was in
no way directed toward digital forensics validations, the
scientific baselines and methods it suggests are a good starting
point for drafting validation reports that will hold up in a court
of law and the digital forensics community.
The Scientific Method and Software/Tool Validations: A
Perfect Fit
In the Daubert ruling, The Court defined scientific methodology
as “the process of formulating hypotheses and then conducting
experiments to prove or falsify the hypothesis.” The Scientific
Method refers to a body of techniques for investigating
phenomena, acquiring new knowledge, or correcting and
integrating previous knowledge. To be termed scientific, the
method must be based on gathering, observing, or
investigating, and showing measurable and repeatable results.
Most of the time, the scientific process starts with a simple
question that leads to a hypothesis, which then leads to
experimentation, and an ultimate conclusion. To exemplify, if
you are validating a particular hardware write blocking device
you may want to start with the simple question “Does this tool
successfully allow normal write-block operation to occur to
source media?” Since it is assumed that the write-blocking
device supports various types of media (SATA, IDE, and so on)
you may be required to list the various requirements of the
tool. Because if this, it is good practice for an examiner to use
the scientific method as a baseline for formulating digital
forensic validations. It is recommended that forensic
examiners follow these four basic steps as a starting point for
an internal validation program:
1) Develop the Plan
Developing the scope of the plan may involve background and
defining what the software or tool should do in a detailed
fashion. Developing the scope of the plan also involves
creating a protocol for testing by outlining the steps, tools, and
requirements of such tools to be used during the test. This
may include evaluation of multiple test scenarios for the same
software or tool. To illustrate, if validating a particular forensic
software imaging tool, that tool could be tested to determine
whether or not it successfully creates, hashes, and verifies a
particular baseline image that has been previously setup.
There are several publically available resources and guides
that can be useful in establishing what a tool should do such as
those available from NIST’s Computer Forensic Tool Testing
Project (CFTT) available from http://www.cftt.nist.gov. The
CFTT also publishes detailed validation reports on various
types of forensic hardware and software ranging from mobile
phones to disk imaging tools. In addition to CFTT, Marshall
University has published various software and tool validation
reports that are publically available for download from
http://forensics.marshall.edu/Digital/Digital-Publications.html.
These detailed reports can be used to get a feel for how your
own internal protocol should be drafted. The scope of the plan
may also include items such as: tool version, testing
manufacturer, and how often the tests will be done. These
factors should be established based on your organization
standards. Typically, technology within a lab setting is re-
validated quarterly or biannually at the very least.
2) Develop a Controlled Data Set
This area may be the longest and most difficult part of the
validation process as it is the most involved. This is because it
involves setting-up specific devices and baseline images and
then adding data to the specific areas of the media or device.
Acquisitions would then need to be performed and documented
after each addition to validate the primary baseline. This
baseline may include a dummy mobile phone, USB thumb
drive, or hard drive depending on the software or hardware
tool you are testing. In addition to building your own baseline
images, Brian Carrier has posted several publically available
disk images designed to test specific tool capabilities, such as
the ability to recover deleted files, find keywords, and process
images. These data sets are documented and are available at
http://dftt.sourceforge.net. Once baseline images are created,
tested, and validated it is a good idea to document what is
contained within these images. This will not only assist in
future validations, but may also be handy for internal
competency and proficiency examinations for digital
examiners.
3) Conduct the Tests in a Controlled Environment
Outside all the recommendations and standards set forth by
NIST and the legal community, it only makes sense that a
digital forensics examiner would perform an internal validation
of the software and tools being used in the laboratory. In some
cases these validations are arbitrary and can occur either in a
controlled or uncontrolled environment. Since examiners are
continuously bearing enormous caseloads and work
responsibilities, consistent and proper validations sometimes
fall through the cracks and are validated in a somewhat
uncontrolled “on-the-fly” manner. It’s also a common practice
in digital forensics for examiners to “borrow” validations from
other laboratories and fail to validate their own software and
tools. Be very careful with letting this happen. Keep in mind
that in order for digital forensics to be practicing true scientific
principles, the processes used must be proven to be repeatable
and reproducible. In order for this to occur, the validation
should occur within a controlled environment within your
laboratory with the tools that you will be using. If the examiner
uses a process, software, or even a tool that is haphazard or
too varied from one examination to the next, the science then
becomes more of an arbitrary art. Simply put, validations not
only protect the integrity of the evidence, they may also
protect your credibility. As stated previously, using a
repeatable, consistent, scientific method in drafting these
validations is always recommended.
4) Validate the Test Results against Known and Expected
Results
At this point, testing is conducted against the requirements set
forth for the software or tool in the previous steps. Keep in
mind that results generated through the experimentation and
validation stage must be repeatable. Validation should go
beyond a simple surface scan when it comes to the use of
those technologies in a scientific process. With that said, it is
recommended that each requirement be tested at least three
times. If there are any variables that may affect the outcome
of the validation (e.g. failure to write-block, software bugs)
they should be determined after three test runs. There may be
cases, however, where more or fewer test runs may be
required to generate valid results.
It’s also important to realize that you are probably not the first
to use and validate a particular software or tool, so chances
are that if you are experiencing inconsistent results, the
community may be experiencing the same results as well.
Utilizing peer review may be a valuable asset when performing
these validations. Organizations such as the High Technology
Crime Investigation Association (HTCIA) and the International
Association of Computer Investigative Specialists (IACIS)
maintain active member e-mail lists for members that can be
leveraged for peer review. There are also various lists and
message boards pertaining to mobile phone forensics that can
be quite helpful when validating a new mobile technology. In
addition, most forensic software vendors maintain message
boards for software, which can be used to research bugs or
inconsistencies arising during validation testing.
Conclusion
Real world laboratory use, controlled internal tests utilizing
scientific principles, and peer review should all be leveraged in
a validation test plan. Sharing unique results with the digital
forensics community at-large helps investigators, examiners,
and even software and tool vendors ensure that current best
practices are followed. As the field of digital forensics continues
to grow and evolve as a science the importance of proper
scientific validation will be more important than ever.
References
1. Brown, C. "Computer Evidence: Collection &
Preservation." Hingham: Thomson/Delmar. 2006.
2. Carrier, B. “Digital Forensics Tool Testing Images.”
Accessed 06 Feb 2011. http://dftt.sourceforge.net/.
3. Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579
(1993).
4. High Technology Crime Investigation Association,
Accessed 06 Feb 2011. www.htcia.org.
5. International Association of Computer Investigative
Specialists, Accessed 06 Feb 2011. www.iacis.org.
6. Maras, MH. Computer Forensics: Cybercriminals, Laws,
and Evidence. Sudbury: Jones & Bartlett. 2011.
7. 7. Marshall University Forensic Science Center-Digital
Publications, Accessed 06 Feb 2011.
http://forensics.marshall.edu/Digital/Digital-
Publications.html.
8. NIST Computer Forensic Tool Testing Project. Accessed
06 Feb 2011. www.cftt.nist.gov.
9. Shroader, A. "How to Validate Your Forensic Tools."
Orem: Paraben Corp. 2010.
Josh Brunty currently manages the digital forensics graduate
program and the digital forensics research and casework
laboratories at the Marshall University Forensic Science Center
in Huntington, WV. Josh holds numerous certifications within
the digital forensics discipline including: AccessData Certified
Examiner (ACE), Computer Hacking Forensic Examiner (CHFI),
Seized Computer Evidence Recovery Specialist (SCERS), and is
certified in Information Assessment Methodology (NSA-IAM).
He has developed a variety of digital forensics training
curriculum; including past recertification scenarios/exams for
the International Association of Computer Investigative
Specialists (IACIS). Josh is an active member of the Mid-
Atlantic Association of the High Technology Crime Investigation
Association (HTCIA) and the Digital-Multimedia Sciences
section of the American Academy of Forensic Sciences (AAFS).
He can be reached at josh.brunty@marshall.edu.
RELATED READS
Virtual Case
Notes:
Steganalysis
Database Will
Help Find Hidden Files in ‘Innocent’ Images
Virtual Case
Notes: You Have
the Suspect’s
Phone. It’s Soaked
in Drain Cleaner. Now What?
Houston CSU: City’s Forensic Science Center and Police at
Odds Over Scene Response
Banks Use
Cellebrite Phone
Cracking
Technology in
Internal Investigations
... Moreover, in order for the evidence to be admissible in court, tools, techniques, and procedures used must be reliable and valid [22]. ...
Article
Full-text available
ISO 17025:2017 is defined for laboratories performing tests, samplings, and calibrations, while listing minimum Quality Management System (QMS) requirements when compared to ISO 9001:2015 that is solely composed for the QMS in order to ensure the quality of work performed. This paper reviews these two accreditations in the context of their applicability to the digital forensics field. Given the updated versions of both accreditations, their similarities, and limitations, are compared along with their context in relation to the QMS. Results achieved suggest a sufficient management system for digital forensics (DF) laboratories. Analysis showed the relation of both accreditations to the activities performed in the DF laboratories, as well as a successful QMS. The paper demonstrates that it is possible to determine whether the 9001:2015 or 17025:2017 accreditation is more closely related to the tasks performed and the management systems in DF laboratories.
... This is done by identifying the data storage locations such as removable, fixed and flash memory cards, as well as identifying open communication ports for further traffic interception. This requires non-destructive methods, to protect the original data, using commercial or non-commercial forensic tools [336], or using a destructive extraction method. • Reporting and Analysis Phase: it is based on an initial review of the extracted data since the first stored images are the suspect's own images including initial take off/landing spot, available personnel, surrounding location, area coordinates, etc.. Thus, it is important to know how the recording function works to intercept the data and translate it into a human readable form. ...
Article
Recently, the world witnessed a significant increase in the number of used drones, with a global and continuous rise in the demand for their multi-purpose applications. The pervasive aspect of these drones is due to their ability to answer people’s needs. Drones are providing users with a bird’s eye that can be activated and used almost anywhere and at any time. However, recently, the malicious use of drones began to emerge among criminals and cyber-criminals alike. The probability and frequency of these attacks are both high and their impact can be very dangerous with devastating effects. Therefore, the need for detective, protective and preventive counter-measures is highly required. The aim of this survey is to investigate the emerging threats of using drones in cyber-attacks, along the countermeasures to thwart these attacks. The different uses of drones for malicious purposes are also reviewed, along the possible detection methods. As such, this paper analyzes the exploitation of drones vulnerabilities within communication links, as well as smart devices and hardware, including smart-phones and tablets. Moreover, this paper presents a detailed review on the drone/Unmanned Aerial Vehicle (UAV) usage in multiple domains (i.e civilian, military, terrorism, etc.) and for different purposes. A realistic attack scenario is also presented, which details how the authors performed a simulated attack on a given drone following the hacking cycle. This review would greatly help ethical hackers to understand the existing vulnerabilities of UAVs in both military and civilian domains. Moreover, it allows them to adopt and come up with new techniques and technologies for enhanced UAV attack detection and protection. As a result, various civilian and military anti-drones/UAVs (detective and preventive) countermeasures will be reviewed.
... When considering potentially non-validated open source tools, validation of results will prove necessary [23]. ...
Article
Drones (also known as Unmanned Aerial Vehicles, UAVs) is a potential source of evidence in a digital investigation, partly due to their increasing popularity in our society. However, existing UAV/drone forensics generally rely on conventional digital forensic investigation guidelines such as those of ACPO and NIST, which may not be entirely fit_for_purpose. In this paper, we identify the challenges associated with UAV/drone forensics. We then explore and evaluate existing forensic guidelines, in terms of their effectiveness for UAV/drone forensic investigations. Next, we present our set of guidelines for UAV/drone investigations. Finally, we demonstrate how the proposed guidelines can be used to guide a drone forensic investigation using the DJI Phantom 3 drone as a case study.
... Digital evidence must be authentic and accurate, meaning that it is a precise representation of the originally created data (Cohen, 2012). The forensic process must be repeatable and reproducible (Brunty, 2016). The National Institute of Standards and Technology conducts research and publishes standards under which digital evidence can be deemed reliable for use in legal proceedings (Goodison, 2015). ...
Research
Full-text available
Using single board computers for digital forensic image acquisition.
... This difficulty is further complicated by the nonavailability and lack of court-validated digital forensic tools that are available for the collection and analysis of social media evidence. The use of freeware and/ or shareware capturing utilities or tools for capturing evidence from social media sites, absent consideration of the legal implications and ramifications that may arise, must be resisted in the light of recent case law indicating that courts are applying increasingly stringent requirements for admissibility where social media evidence is involved (Brunty, 2011). With that said, in order to provide social media evidence that can withstand intense legal scrutiny, we must look at the relevant case law and legal baselines that currently exist. ...
Book
Social media is becoming an increasingly important—and controversial—investigative source for law enforcement. Social Media Investigation for Law Enforcement provides an overview of the current state of digital forensic investigation of Facebook and other social media networks and the state of the law, touches on hacktivism, and discusses the implications for privacy and other controversial areas. The authors also point to future trends.
... The article Validation of Forensic Tools and Software: A Quick Guide for the Digital Forensic Examiner by Josh Brunty[8] summarises the Daubert Standard quite well: ...
Book
Implementing Digital Forensic Readiness: From Reactive to Proactive Process shows information security and digital forensic professionals how to increase operational efficiencies by implementing a pro-active approach to digital forensics throughout their organization. It demonstrates how digital forensics aligns strategically within an organization's business operations and information security's program. This book illustrates how the proper collection, preservation, and presentation of digital evidence is essential for reducing potential business impact as a result of digital crimes, disputes, and incidents. It also explains how every stage in the digital evidence lifecycle impacts the integrity of data, and how to properly manage digital evidence throughout the entire investigation. Using a digital forensic readiness approach and preparedness as a business goal, the administrative, technical, and physical elements included throughout this book will enhance the relevance and credibility of digital evidence. Learn how to document the available systems and logs as potential digital evidence sources, how gap analysis can be used where digital evidence is not sufficient, and the importance of monitoring data sources in a timely manner. This book offers standard operating procedures to document how an evidence-based presentation should be made, featuring legal resources for reviewing digital evidence. Explores the training needed to ensure competent performance of the handling, collecting, and preservation of digital evidence Discusses the importance of how long term data storage must take into consideration confidentiality, integrity, and availability of digital evidence Emphasizes how incidents identified through proactive monitoring can be reviewed in terms of business risk Includes learning aids such as chapter introductions, objectives, summaries, and definitions
Computer Evidence: Collection & Preservation
  • C Brown
Brown, C. "Computer Evidence: Collection & Preservation." Hingham: Thomson/Delmar. 2006.
Digital Forensics Tool Testing Images
  • B Carrier
Carrier, B. "Digital Forensics Tool Testing Images." Accessed 06 Feb 2011. http://dftt.sourceforge.net/.
How to Validate Your Forensic Tools
  • A Shroader
Shroader, A. "How to Validate Your Forensic Tools." Orem: Paraben Corp. 2010.