## File: README.md [](https://github.com/mandiant/flare-floss/releases) [](https://github.com/mandiant/flare-floss/actions/workflows/tests.yml) [](https://github.com/mandiant/flare-floss/releases) [](LICENSE.txt) # FLARE Obfuscated String Solver The FLARE Obfuscated String Solver (FLOSS, formerly FireEye Labs Obfuscated String Solver) uses advanced static analysis techniques to automatically extract and deobfuscate all strings from malware binaries. You can use it just like `strings.exe` to enhance the basic static analysis of unknown binaries. ### Obfuscated Strings Rather than heavily protecting backdoors with hardcore packers, many malware authors evade heuristic detections by obfuscating only key portions of an executable. Often, these portions are strings and resources used to configure domains, files, and other artifacts of an infection. These key features will not show up as plaintext in the output of the `strings.exe` utility that we commonly use during basic static analysis. FLOSS extracts all the following string types: 1. static strings: "regular" ASCII and UTF-16LE strings 2. stack strings: strings constructed on the stack at run-time 3. tight strings: a special form of stack strings, decoded on the stack 4. decoded strings: strings decoded in a function Please review the theory behind FLOSS [here](doc/theory.md). Our [blog post](https://www.mandiant.com/resources/automatically-extracting-obfuscated-strings) talks more about the motivation behind FLOSS and details how the tool works. FLOSS version 2.0 updates are detailed in this [blog post](https://www.mandiant.com/resources/floss-version-2). ### Language-specific Strings Not all compilers use string formats that the classic `strings.exe` algorithm supports. For example, if strings are UTF-8 encoded or stored without a NULL-terminator. FLOSS can identify and extract strings from programs compiled from the following languages: 1. Go 2. Rust The strings FLOSS extracts specific to a compiler are much easier to inspect by humans. Please consult the documentation to learn more about the [language-specific string extraction](doc/language_specific_strings.md). ## Installation To use FLOSS, download a standalone executable file from the releases page: https://github.com/mandiant/flare-floss/releases See the [installation documentation](doc/installation.md) for a detailed description of all methods to install FLOSS. ## Usage Examples Extract obfuscated strings from a malware binary: $ floss malware.exe Only extract stack and tight strings: $ floss --only stack tight -- suspicious.exe Do not extract static strings: $ floss --no static -- backdoor.exe Display the help/usage screens: $ floss -h # show core arguments $ floss -H # show all supported arguments For a detailed description of using FLOSS, review the documentation [here](doc/usage.md). ## Scripts FLOSS also contains additional Python scripts in the [scripts](scripts) directory which can be used to load its output into other tools such as Binary Ninja or IDA Pro. For detailed description of these scripts review the documentation [here](scripts/README.md). ## Mailing List Subscribe to the FLARE mailing list for community announcements by sending an email with the subject "subscribe" to [flare-external@google.com](mailto:flare-external@google.com?subject=subscribe&body=subscribe). --- ## File: doc/installation.md # FLARE Obfuscated String Solver ## Installation You can install FLOSS in a few different ways. First, if you simply want to use FLOSS to extract strings, just download the [standalone binaries](https://github.com/mandiant/flare-floss/releases/latest). However, if you want to use FLOSS as a Python library, you can install the package directly from GitHub using `pip`. Finally, if you'd like to contribute patches or features to FLOSS, you'll need to work with a local copy of the source code. ## Method 1: Using FLOSS standalone If you simply want to use FLOSS to extract strings, use the standalone binaries we host on GitHub: https://github.com/mandiant/flare-floss/releases. These binary executable files contain all the source code, Python interpreter, and associated resources needed to make FLOSS run. This means you can run it without any installation! Just invoke the file using your terminal shell to see the help documentation. We use PyInstaller to create these packages. ### MacOS Standalone installation By default, on macOS Catalina or greater, Gatekeeper will block execution of the standalone binary. To resolve this, simply try to execute it once on the command-line and then go to `System Preferences` / `Security & Privacy` / `General` and approve the application. ## Method 2: Using FLOSS as a Python library If you'd like to use FLOSS as part of an automated analysis system, you might want to invoke it as a Python library. We designed FLOSS to be as easy to use from a client program as from the command line. :warning: **FLOSS requires Python >= 3.10.** ### Step 1: Install FLOSS module Use `pip` (Python >= 3.10) to install the `flare-floss` module to your local Python environment. This fetches the library code to your computer, but does not keep editable source files around for you to hack on. If you'd like to edit the source files, see Method 3. - Install FLOSS: `$ pip install flare-floss` ### Step 2: Use FLOSS from a Python script You can now import the `floss` module from a Python script: #!/usr/env/python import floss print(dir(floss)) ## Method 3: Inspecting the FLOSS source code If you'd like to review and modify the FLOSS source code, you'll need to check it out from GitHub and install it locally. By following these instructions, you'll maintain a local directory of source code that you can modify and run easily. ### Step 1: Check out source code - Clone the FLOSS git repository: `$ git clone https://github.com/mandiant/flare-floss /local/path/to/src` ### Step 2: Install the local source code Next, use `pip` to install the source code in "editable" mode. This means that Python will load the FLOSS module from this local directory rather than copying it to `site-packages` or `dist-packages`. This is good, because it is easy for us to modify files and see the effects reflected immediately. But be careful not to remove this directory unless uninstalling FLOSS! If you encounter the error `ERROR: Project has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook.`, please ensure that you have upgraded to the latest versions of pip and setuptools. - Install FLOSS: `$ pip install -e /local/path/to/src` You'll find that the `floss.exe` (Windows) or `floss` (Linux, macOS) executables in your path now invoke the FLOSS binary from this directory. ### Step 3: Install development and testing dependencies When developing FLOSS, please use the pinned dependencies found in `requirements.txt`. This ensures that everyone has the exact same, reproducible environment. Please install these dependencies before install FLOSS (from source or from PyPI): `$ pip install -r requirements.txt` To install all testing and development dependencies, run: `$ pip install -e /local/path/to/src[dev]` We use a git submodule to separate [code](https://github.com/mandiant/flare-floss) and [test data](https://github.com/mandiant/flare-floss-testfiles). To clone everything use the `--recurse-submodules` option: - `$ git clone --recurse-submodules https://github.com/mandiant/flare-floss.git /local/path/to/src` (HTTPS) - `$ git clone --recurse-submodules git@github.com:mandiant/flare-floss.git /local/path/to/src` (SSH) Or use the manual option: - clone repository - `$ git clone https://github.com/mandiant/flare-floss.git /local/path/to/src` (HTTPS) - `$ git clone git@github.com:mandiant/flare-floss.git /local/path/to/src` (SSH) - `$ cd /local/path/to/src` - `$ git submodule update --init tests/data` We use the following tools to ensure consistent code style and formatting: - [black](https://github.com/psf/black) code formatter - [isort](https://pypi.org/project/isort/) code formatter - [mypy](https://mypy-lang.org/) type checking We use [pre-commit](https://pre-commit.com/) so that its trivial to run the same linters & configuration locally as in CI. Run all linters liks: ``` ❯ pre-commit run --all-files isort....................................................................Passed black....................................................................Passed mypy.....................................................................Passed ``` Or run a single linter like: ``` ❯ pre-commit run --all-files isort isort....................................................................Passed ``` Importantly, you can configure pre-commit to run automatically before every commit by running: ``` ❯ pre-commit install --hook-type pre-commit pre-commit installed at .git/hooks/pre-commit ❯ pre-commit install --hook-type pre-push pre-commit installed at .git/hooks/pre-push ``` This way you can ensure that you don't commit code style or formatting offenses. You can always temporarily skip the checks by using the `-n`/`--no-verify` git option. ### Step 4: Building standalone executables Once you're happy with your contribution to FLOSS, you can package and distribute a standalone executable for your friends using PyInstaller. This combines the source code, Python interpreter, and required resources into a single file that can be run without installation. - Install pyinstaller: `$ pip install pyinstaller` - Build standalone executable: `$ pyinstaller .github/pyinstaller/floss.spec` - Distribute standalone executable: `$ cp ./dist/floss.exe /the/internet` --- ## File: doc/language_specific_strings.md ## Go String Extraction Programs compiled by the Go compiler use a string representation that is difficult to interpret by humans. Although they are UTF-8 encoded, and therefore show up in the output of `strings.exe`, program strings are not NULL-terminated. This means separate strings within the binary may appear as a large chunk of indistinguishable string data. FLOSS implements an algorithm to handle the unusual characteristics of strings in Go binaries. This approach analyzes instances of the `struct String` type to identify candidate strings and reasons about the length-sorted order to avoid false positives. Crucially, FLOSS automatically handles the complexities of Go strings and displays strings as written in the program's source code. It's important to mention that there are other types of strings, such as runtime strings, which are not derived from the program strings. ### Algorithm: 1. Analyze the string instances within the binary. - In Go, strings are encoded as structs (see source code links below) containing two fields: a pointer to the string's underlying data and the length of the string. - By examining these instances, we can identify the strings and their locations within the binary. 2. Identify the longest continuous sequence of monotonically increasing string lengths to find the string blob. 3. Use the byte sequence `00 00 00 00` as a delimiter to accurately mark the boundaries of the string blob. 4. Extract the string blob located between the identified boundaries. 5. Split the identified string blob, based on the cross-references available in the binary to separate the individual strings. Please note that while FLOSS handles many scenarios effectively, there are certain optimizations, such as inlined constants, that may not be fully supported yet. For more information on Go strings, you can refer to the Go project's documentation and the source code of the struct String layout. Learn more: Go Project: [Go Project](https://github.com/golang/go) Blog post: [Unveiling Go Strings: A Google Summer of Code Journey](https://medium.com/p/92f6d9fee97c) Source code: - https://github.com/golang/go/blob/36ea4f9680f8296f1c7d0cf7dbb1b3a9d572754a/src/builtin/builtin.go#L70-L73 - https://github.com/golang/go/blob/38e2376f35907ebbb98419f1f4b8f28125bf6aaf/src/go/types/builtins.go#L824-L825 ## Rust String Extraction Similar to Go, Rust binaries may contain strings that are not NULL terminated. Separate strings within the binary may appear as larger chunks of indistinguishable string data. FLOSS analyzes the data and code in Rust binaries to identify individual candidate strings. ### Algorithm: 1. Extract all UTF-8 encoded strings 2. Analyze data and code references to identify substring boundaries 3. Split strings from step 1 into individual parts as found in step 2 For more information on Rust strings, you can refer to the Rust project's documentation and the source code of the Rust String layout. Learn more: Rust Project: [Rust Project](https://github.com/rust-lang/rust) Source code: - https://github.com/rust-lang/rust/blob/3911a63b7777e19dad4043542f908018e70c0bdd/library/alloc/src/string.rs --- ## File: doc/theory.md # FLARE Obfuscated String Solver ## Theory Malware authors pack their software to resist reverse engineering and enable their operations to survive longer. However, many features of packing are easy to automatically identify during static or dynamic analysis. Therefore, some authors obfuscate only the most sensitive resources used by malware in an attempt to blend in. We call this "string obfuscation". String obfuscation maintains some difficulty around extracting host or network based signatures (such as filenames, registry keys, or domain names), while structuring the executable file like legitimate programs. This is a technique that balances moderate anti-reverse engineering tricks with a moderate level of stealth. As a reverse engineer, it takes significant effort to extract obfuscated strings from a malware sample. This is because there are a huge number of possible encoding functions, configurations, and control flows. For example, some malware uses a single-byte XOR operation with a static key for all obfuscated strings, while other malware uses RC4 encryption with a unique key per string. Its often difficult to figure out how encoded data is protected without opening IDA Pro or reviewing a debugger trace. Manual extraction of obfuscated strings commonly involves thoroughly studying a decryption routine and reimplementing it in a scripting language. This is a tedious and error-prone process that is fun at first, and mind-numbing after a few iterations. Alternatively, an analyst may instrument a debugger to hop around hundreds of locations in hopes of forcing the malware to decode itself. This is also complex, tedious, and error-prone. FLOSS combines and automates the best manual reverse engineering techniques for string decoding. First, it uses heuristics to identify decoding routines in a sample. Then FLOSS extracts cross references and arguments to decoders using control flow analysis. Next FLOSS emulates decoder functions using extracted arguments. Finally, FLOSS diffs the emulator memory states from before and after decoder emulation and extracts human readable strings. ### Algorithm 1. Analyze control flow of malware to identify functions, basic blocks, etc. 2. Use heuristics to find potential decoding routines 3. Brute force emulate all code paths among basic blocks and functions 4. Snapshot emulator state (registers, memory) at appropriate points 5. Extract arguments to decoder functions from emulator snapshots 6. Emulate decoder functions using extracted arguments and emulator state 7. Diff memory state from before and after decoder emulation 8. Extract human-readable strings from memory state difference --- ## File: doc/usage.md # FLARE Obfuscated String Solver ## Usage You can use FLOSS just like you'd use `strings.exe` to extract human-readable strings from binary data. The enhancement that FLOSS provides is that it statically analyzes executable files and decodes obfuscated strings. These include: * strings encrypted in global memory or deobfuscated onto the heap * strings manually created on the stack (stackstrings) * strings created on the stack and then further modified (tight strings) Since FLOSS also extracts static strings (like `strings.exe`), you should consider replacing `strings.exe` with FLOSS within your analysis workflow. Here's a summary of the command line flags and options you can provide to FLOSS to modify its behavior. See `floss -h` for all supported arguments and usage examples. This displays the most used arguments only. To see all supported arguments run `floss -H`. ### Extract static, obfuscated, and stack strings (default mode) floss.exe malware.exe The default mode for FLOSS is to extract the following string types from an executable file: - static ASCII and UTF-16LE strings - stack strings - tight strings - obfuscated strings See the section on [Shellcode analysis](#shellcode) below on how to analyze raw binary files containing shellcode. By default, FLOSS uses a minimum string length of four (4). ### Language-specific strings FLOSS can identify programs compiled from selected programming languages and extract strings that are easier to inspect by humans. By default, this process is automatic. However, you can use the `--language` argument to manually select or disable this feature. ### Disable string type extraction (`--no {static,decoded,stack,tight}`) When FLOSS searches for static strings, it looks for human-readable ASCII and UTF-16 strings across the entire binary contents of the file. This means you may be able to replace `strings.exe` with FLOSS in your analysis workflow. However, you may disable the extraction of static strings via the `--no static` switch. floss.exe --no static -- malware.exe Since `--no` supports multiple arguments, end the command options with a double dash `--`. Analogous, you can disable the extraction of obfuscated strings, stackstrings or any combination. floss.exe --no decoded -- malware.exe floss.exe --no stack tight -- malware.exe ### Enable string type extraction (`--only {static,decoded,stack,tight}`) Sometimes it's easier to specify only the string type(s) you want to extract. Use the `--only` option for that. floss.exe --only decoded -- malware.exe Please note that `--no` and `--only` cannot be used at the same time. ### Write output as JSON (`-j/--json`) Write FLOSS results to `stdout` structured in JSON to make it easy to ingest by a script. floss.exe -j malware.exe > malware_strings.json ### Load FLOSS results (`-l/--load`) Load a FLOSS results JSON document. This allows to explore FLOSS results without re-running the analysis. floss.exe -l malware_floss_results.json ### Verbose results (`-v`) Enable verbose results output, e.g. including function offsets and string encoding. This does not affect the JSON output. floss.exe -v malware.exe ### Quiet mode (`-q/--quiet`) You can suppress the formatting of FLOSS output by providing the flags `-q` or `--quiet`. These flags are appropriate if you will pipe the results of FLOSS into a filtering or searching program such as grep, and want to avoid matches on the section headers. In quiet mode, each recovered string is printed on its own line. The "type" of the string (static, decoded, stackstring, tightstring) is not included. floss.exe -q malware.exe ### Minimum string length (`-n/--minimum-length`) By default, FLOSS searches for human-readable strings with a length of at least four characters. You can use the `-n` or `--minimum-length` options to specific a different minimum length. Supplying a larger minimum length reduces the chances of identifying random data that appears to be ASCII; however, FLOSS may then pass over short legitimate human-readable strings floss.exe -n 10 malware.exe ### Decoding function specification (`--functions`) You can instruct FLOSS to decode the strings provided to specific functions by using the `--functions` option. By default, FLOSS uses heuristics to identify decoding routines in malware. This mode circumvents the identification phase and skips directly to the decoding phase. If you've previously done analysis on an executable program and manually identified the decoding routines, use this mode. This can improve performance as FLOSS by perhaps one-third (on the order of seconds, so it is usually _not_ worth it to always manually identify decoding routines). Specify functions by using their hex-encoded virtual address. floss.exe --functions 0x401000 0x402000 malware.exe ### Install/Uninstall right click menu option for Windows (`--install-right-click-menu/--uninstall-right-click-menu`) You can use the `--install-right-click-menu` and `--uninstall-right-click-menu` options to install/remove the `Open with FLOSS` option from the right-click menu of the Windows file explorer. After this option is installed, you can right-click on any file and select `Open with FLOSS` to quickly open the target file with FLOSS for analysis. ## Shellcode analysis options Malicious shellcode often times contains obfuscated strings or stackstrings. FLOSS can analyze raw binary files containing shellcode via the `-f/--format` switch. All options mentioned above can also be applied when analyzing shellcode. floss.exe -f sc32 malware.raw32 floss.exe -f sc64 malware.raw64 --- ## File: floss/sigs/README.md # FLOSS signatures This directory contains FLIRT signatures that FLOSS uses to identify library functions. Typically, FLOSS will ignore library functions, which reduces false positives and improves runtime. These FLIRT signatures were generated by Mandiant using the Hex-Rays FLAIR tools such as `pcf` and `sigmake`. Mandiant generated the signatures from source data that they collected; these signatures are not derived from the FLIRT signatures distributed with IDA Pro. The signatures in this directory have the same license as FLOSS: Apache 2.0. --- ## File: scripts/README.md # FLOSS Scripts FLOSS supports converting its output into scripts for various tools. Please see the render scripts in this directory. Additionally, there is another [plugin for IDA](idaplugin.py) to allow FLOSS to automatically extract obfuscated strings and apply them to the currently loaded module in IDA. `idaplugin.py` is a IDAPython script you can directly run within IDA Pro (File - Script File... [ALT + F7]). # Installation These scripts can be downloaded from the FLOSS [GitHub](https://github.com/mandiant/flare-floss) repository alongside the source, which is required for the scripts to run. To install FLOSS as source, see the documentation [here](../doc/installation.md). # Usage ## Convert FLOSS output for use by other tools - Run FLOSS on the desired executable with the `-j` or `--json` argument to emit a JSON result and redirect it to a JSON file. `$ floss -j suspicious.exe > floss_results.json` For Binary Ninja, IDA Pro, Ghidra or Radare2: - Run the script for your tool of choice by passing the result json file as an argument and redirect the output to a Python (.py) file. Ghidra Example: `$ python render-ghidra-import-script.py floss_results.json > apply_floss.py` - Run the Python script `apply_floss.py` using the desired tool. For x64dbg: - Instead of a Python file, redirect the output to a .json file. `$ python render-x64dbg-database.py floss-results.json > database.json` - Open the JSON file `database.json` in x64dbg.