Today we are happy to announce the release of version 0.9.8. As we are now approaching the 1.0 release, we have focused our efforts on freezing parts of the API, creating documentation (including the long-awaited file parser doc!) and making various quality of life improvements for this release. Let us dive into some of the most impactful changes:

Scripting

API changes for carved/virtual files

We have now somewhat unified the interface to deal with virtual and carved files. Both file types now have an open method, which returns a malcat.File instance:

• malcat.VirtualFile.open()
• malcat.CarvedFile.open()

The returned file object may be used directly with the malcat.analyse() function for further analysis. This should make some workflows a lot easier. For instance, we have added a new -r option to the utility script malcat.report.py responsible for printing textual reports from the command line (which I'm pretty sure you did not know existed :):

if options.recursive:
    for sf in a.carved:
        a = malcat.analyse(sf.open(), options=analysis_options) # sf.open() always succeeds
        a.raise_if_failed()
        result = a.template(template_path)
        print(result)

    for vf in a.vfiles:
        try:
            f = vf.open()
        except Exception as e:
            print(f"Cannot open virtual file {vf.path}: {e}")
            continue
        a = malcat.analyse(f, options=analysis_options)
        a.raise_if_failed()
        result = a.template(template_path)
        print(result)

Note that the malcat.analyse() method is only available in headless mode

More API

We have also added a couple of new API functions. They mostly target code and reference objects and should be useful to write code analysis scripts:

• Reworked and simplified the python interface for cross references
• Added new inrefs and outrefs attributes to ease reference gathering for code objects: Function.inrefs, Function.outrefs, BasicBlock.inrefs, BasicBlock.outrefs, Instruction.inrefs and Instruction.outrefs
• Added new Function.callers and Function.callees attribute to ease naviguation into the call graph
• Added new attributes Instruction.bb and Instruction.function to the Instruction object
• Added new attributes BasicBlock.function to the BasicBlock object
• Added new attributes Function.callers and Function.callees to the Function object
• Added new methods Instruction.disasm, BasicBlock.disasm and Function.disasm to easily disassemble code in these objects
• Added new methods BasicBlock.hex, Function.hex and Analysis.hex to easily get the hexadecimalbytes of an object/data range with magic-masking. This should be useful to automatically output Yara patterns for instance.

While all these functionalities where already available, the addition of the new API methods should make scripts a bit more concise. For instance, let's say I want to list all the callers of all the functions referencing the string "cmd.exe". Using the new methods, it is just a couple of python lines:

import malcat 

s = analysis.strings["cmd.exe"]
for ref in analysis.xref[s.address]:
    if ref.is_code:
        instr = analysis.asm[ref.address]
        print(f"Found code reference at {analysis.ppa(instr.address)} ({instr.mnemonic}). Callers: {instr.function.callers}")

>>> Found code reference at 0x0040a3f0 (getDrives+64) (ldstr). Callers: []
>>> Found code reference at 0x0040a3d7 (getDrives+4b) (ldstr). Callers: []
>>> Found code reference at 0x00408fb1 (nYekikJU+231) (ldstr). Callers: [Stub.Form1.data]
>>> Found code reference at 0x00408e55 (nYekikJU+d5) (ldstr). Callers: [Stub.Form1.data]
>>> Found code reference at 0x00408f41 (nYekikJU+1c1) (ldstr). Callers: [Stub.Form1.data]
>>> Found code reference at 0x00408ecb (nYekikJU+14b) (ldstr). Callers: [Stub.Form1.data]
>>> Found code reference at 0x0040908a (QRhvlBW+7a) (ldstr). Callers: [Stub.Form1.data, Stub.Func.AddHome]
>>> Found code reference at 0x0040a475 (getFolders+31) (ldstr). Callers: []

New scripts

We have added a bunch of new utility scripts that you can run using Malcat's script launcher: Analysis > Run script or faster: Ctrl+U. The list is given below:

• A script to remove null bytes in last section / overlay of PE files. Note that it will just handle the most simple cases. For more complex bloating, use the excellent debloat
• A script to fix section addresses for badly dumped PE files
• A script to extract BMP pixels data in the right order for malware that like their steganography
• A simple configuration extractor for LummaStealer
• A simple configuration extractor for DarkComet
• A simple configuration extractor for Remcos

Additionally, we have added a new command-line tool in bin/malcat.keakode.py that will perform a Kesakode lookup from the command line (for full/pro versions):

If you pass the -v option, you will also be able to see the details of all functions and strings that did match.

User interface

Transform dialog improvements

We've built upon the latest improvements made to the transform dialog to make it even better. We did get some complaints that the small 256 bytes preview in the dialog was not enough to assess the validity of your transform. So we have replaced this preview with the same hexadecimal / text /disassembly views that you have in Malcat's main view. You can now switch between the different views (your display preferences will be used) and scroll through the complete output of your transform.

Additionally, we've also added the ability to save your transform chain to a template that you may load later on. We've also added button to add your own transform using the built-in python editor. While you could do it already before using an external editor, it was indeed annoying to have to locate your user data directory first.

Note that Malcat will automatically add a new python transform to the file located at <user data dir>/transform/my_transforms.py, meaning you need to have defined your user data directory first.

As a final note, you can now put your transformed data directly into the clipboard, instead of being forced to open it in a new tab. You can do it either:

• By clicking on the To clipboard button
• By selecting what you want to copy in the output preview and pressing Ctrl+C (and closing the dialog)

This should save a few clicks when you are decrypting small temporary data such as decryption keys.

Hex & data editor improvements

People reversing firmware are sometimes facing data that needs to be visualised as a sequence of dwords or sequence of words. To make the process a bit easier on the eye, we have added to options to the hexadecimal view:

• You can now change the column size to whatever size you want (the default staying at 8)
• You can group (i.e. remove spaces) bytes inside a column

Additionally, we have improved the data preview window:

• New data types have been added (different date formats and GUID)
• most fields are can now be modified, like in the structure preview
• you can now switch between little and big endian interpretation
• you can now switch between decimal and hexadecimal display

This should make data exploration easier for unknown structures / file types hopefully.

Parsers & analysis

Parser API freeze and documentation

Malcat supports more than 50 file formats. Some of the parsers were written at the very beginning of Malcat's development (5 years ago now!). So there was some technical debt in this department. In this release, we have cleaned up a few of the parsers, freezed the file parser API and, most important, written a short documentation on how to write your own parser. So starting from now, and if you're brave enough, you can start adding your parsers if you want.

Feel free to look at the doc, look at the existing parsers in data/filetypes, and if you have more questions just ask in our discord server.

Improved Golang string detection

Retrieving Golang strings is by nature tricky and requires the use of heuristics. We have added a new heuristic to support the main one in covering corner cases. This new heuristic scans the data sections of Golang binaries and tries to infer strings size by looking at incoming cross-references. Any ascii-looking string enclosed by 2 cross references will now be considered as a candidate. This heuristic will also be used for Rust programs.

Bat2exe support

Bat2Exe is an utility used to pack batch files inside PE executables. This tool was very beloved by malware authors a few years ago, and you can still see a few threat actors using it here and there. The batch scripts are stored compressed and encrypted inside RCDATA resources. It is very easy to come by the scripts suing a sandbox (they re dropper to temp), but it's often overkill to fire up a VM for these files.

In order to save some time, we have added bat2exe decryption/decompression capabilities into the PE parser, with support for most bat2exe stubs generated after 2016. Extracted script will be listed in the virtual file system tab under the #Bat2Exe directory.

Threat intelligence

New provider: InQuest

InQuest provides a high-throughput Deep File Inspection for threat prevention. Their DFI scan engine is focused toward documents and scripts, and they crawl the internet and various CTI sources constantly looking for new documents to analyse. We have added a new threat intelligence provider that queries their portal. If you are analysing a document, that is a good place to look!

You will need an API key to query their database. But good new, creating an account is free!

New download providers

In version 0.9.5 we have added the possibility to download samples by their hash, leveraging the numerous threat intelligence providers present in Malcat. That's a rather convenient way to download samples from public sources: they won't be written to disk (thus no AV trigger and you can discard it when you're done), and Malcat will do all the unzipping (with the usual password) for you.

We have added two sources to download samples by hash:

• The Triage sandbox
• The FileScanIO platform

Note that you will need API keys for both these services.

Other improvements

There have been several smaller quality of life improvements and bug fixes made to Malcat in this release. If you want the complete list, have a look at the changelog:

● Intelligence:
    - Added InQuest as threat intel source
    - FileScanIO can now be used as download source
    - Triage can now be used as download source
    - Added API key support to MalwareBazaar (will soon be mandatory)
● Strings:
    - Added an additional Xref-based heuristic to recover tricky Golang strings
    - Improved Rust strings detection
● Decompiler:
    - Better parameters recovery for Delphi programs
● Scripting:
    - Added a script to remove null bytes in last section / overlay
    - Added a script to fix section addresses for badly dumped PE files
    - Added a script to extract BMP pixels data in the right order
    - Added a simple config extractor for LummaStealer
    - Added a simple config extractor for DarkComet
    - Added a simple config extractor for Remcos
    - Added malcat.kesakode.py to bin/ folder (query kesakode on a file from the command line, only for full/pro versions)
    - Added analysis.selection.as_hex method
    - Reworked the interface to analysis.xrefs (cf. doc)
    - Added File.read_until function (cf. doc)
    - Added .bb, .function, .inrefs and .outrefs attributes to malcat.Instruction (cf. doc)
    - Added .function, .inrefs and .outrefs attributes to malcat.BasicBlock (cf. doc)
    - Added .callers, .callees, .inrefs and .outrefs attributes to malcat.Function (cf. doc)
    - Added disasm() function to malcat.Instruction, malcat.BasicBlock and malcat.Function objects (with formatting options, cf. doc)
    - Added hex() function to malcat.Analysis, malcat.BasicBlock and malcat.Function objects (with magic masking options)
    - malcat.CarvedFile.file is now malcat.CarvedFile.open() and returns a malcat.File instance
    - malcat.VirtualFile.unpack() is now malcat.VirtualFile.open() and returns a malcat.File instance
    - Added helper function Analysis.open_vfile(path)
    - Added a method malcat.setup() to make malcat's data/bindings/user dirs accessible from scripts in headless mode
    - Added an option in Preferences>General to display an empty script in the script editor by default (instead of the tutorial script)
● Analysis:
    - .NET static arrays are now cross-referenced by instructions accessing their correspondig field 
    - "Force imagebase" menu action is now working as intended
    - User types are now stored in a "usertypes" directory instead of "types"
    - User types directory can now also contain python files containing Malcat's Struct type definitions
● Parsers:
    - Added an heuristic to retrieve obfuscated/trashed PCLn header in Golang 1.18+ programs
    - Proper symbol parsing for relocatable (.o) ELF files
    - Reimplemented zipfile's CRC32 decryption in c++ for faster non-AES encrypted ZIP decryption
    - Added documentation on how to write your own parser
    - Bat2Exe batch files now appear in the virtual file system tab (post-2016 bat2exe only, does not support multi-bat setups yet) 
● Transforms:
    - Improved data preview: hex, text and disasm mode, scrolling support, copy / paste
    - Added a button to add a new transform directly into your user data directory using Malcat's python editor
    - You can now save transform chains into templates for future use
    - You can now store the transform result directly into the clipboard
    - You can now also transform dynamic strings
    - Transform dialog now selects the result only if the "In place" option has been chosen
    - Improved "reverse" and "skip" transforms
    - Added "flip2d" and "keep" transforms
    - Added "nrv2b decompress" transform
    - Added "zlb1 decompress" transform
    - Added "insert" transform
    - Added "salsa20" transform
● Data preview:
    - Added more data types (timestamp, filetime, dostime and GUID)
    - You can now modify data in the same way as in the structure editor
    - Added the possibility to switch between little endian and big endian interpretation
    - Added the possibility to switch between decimal/hexadecimal display
● User interface:
    - Find dialog: your search query is now kept when switching the search mode
    - Find dialog: find pattern now defaults to selected text in the source view
    - The "find in current file" context menu action now pops up the find dialog (gives you a chance to modify the pattern)
    - Hex view: you can now set the width of the colored blocks/columns in the Options dialog
    - Hex view: you can now groups bytes (i.e. no space) belonging to the same colored block/column (also in the Options dialog)
    - Added a context menu in strings view when multiple strings are selected
    - Added a context menu in the files tab when multiple carved/virtual files are selected
    - You can now copy module names to the clipboard in the symbol view
    - Kesakode view: the accordion labels now display the number of elements currently displayed (before, was ALL elements)
    - Improved charset conversion performances
    - Structure editor now uses OS's locale for date fields
    - Disassembly view: clicking on a hexa byte now displays the data preview
    - Dates are now displayed in ISO format in the structure view and the quickview
● Bug fixing:
    - Fixed a parameter parsing issue in install_api.py script
    - Fixed a slowdown in the Yara scanner module when handling very large .yar files
    - Fixed a bug when modifying huge (u)int64_t value in the structure editor
    - Fixed a bug when modifying a DOS imestamp using the structure editor, the result would be in the local timezone instead of UTC
    - Fixed a regression where function discovery was not performed for x86 code embedded inside p-code VB programs
    - Fixed a parsing bug in InnoSetup language section: license text is always utf8, even in unicode versions
    - Fixed a bug in the .NET disassembler for very large (>256 targets) switch opcodes
    - Fixed kesakode http error message being hidden