Doorgaan naar hoofdcontent

Walkthrough: Intel HEX format

One format that you normally would not encounter very often unless working with certain microcontrollers is the Intel HEX format. This format is a text format to transfer binary information in a text representation. The Wikipedia article about the format is very informative and lists almost everything that needs to be known about the format (but not everyting, as I will show later).

Most scanners would say that these files are text files, but they are actually binary files in disguise! This is why I try to recognize them and process them.

Unless you are working a lot with microcontrollers then the most likely place where you will find these files is in the Linux kernel, where many firmware files (for chips) are included in Intel HEX format.

Creating an unpacker for this file format is quite easy, but you could also use the the SRecord package, which also is able to extract/convert files in different, but similar file formats, such as SREC and others.

For example to convert the "ls" binary from a regular raw file to Intel HEX you would use the srec_cat utility from SRecord:

$ srec_cat ls -binary -Output output-file -intel

which takes the ls binary and spits out the file "output-file" which is an ASCII file:

$ file output-file 
output-file: ASCII text

Converting back is easy as well:

$ srec_cat output-file -intel -Output ls.new -raw

which will spit it out in binary format:

$ file ls.new
ls.new: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=8f8149dbcfdd68a9e7d0e8d29115d05c390522d0, stripped

and which is identical to the original file:

$ md5sum ls ls.new
82646b653d15523d9d54789132e23434  ls
82646b653d15523d9d54789132e23434  ls.new

In the Linux kernel there are many Intel HEX files that cannot be unpacked by srec_cat for one simple reason: srec_cat assumes that the whole file is Intel HEX code and there are no lines with for example comments. The solution is simple: write your own parser and don't read past the "End Of File" line.
Labels:

Reacties

Een reactie posten

Populaire posts van deze blog

Fuzzy hash matching

Fuzzy hash matching, or proximity hashing, is a powerful method to find files that are close to the scanned file. But: it is not a silver bullet. In this blogpost I want to look a bit into proximity matching, when it works and especially when it does not work. Cryptographic hashes Most programmers are familiar with cryptographic hashes such as MD5, SHA256, and so on. These hashes are very useful when needing to uniquely identify files (except in the case of hash collisions, but those are extremely rare). These algorithms work by taking an input (the contents of a file) and then computing a very long number. A slight change in the input will lead to a drastically different number. This is why these cryptographic hashes are great for uniquely identifying files as the same input will lead to the same hash, but useless for comparing files, as different inputs will lead to a very different hash and a comparison of hashes is completely useless. Locality sensitive hashes A different ...
Een reactie posten
Meer lezen

Introducing BANG

Binary Analysis Next Generation (short: BANG) is a framework for unpacking files (like firmware) recursively and running checks on the unpacked files. Its intended use is to classify/label files and making them available for further analysis such as provenance research, license analysis and security analysis. There are quite a few open source licensed tools out there for analyzing  firmware files like binwalk, Hachoir or Sleuthkit. Most of these focus on either forensics, or on unpacking firmware, but none of them focus specifically on where open source, firmware reverse engineering and security meet. Experience creating earlier tools shows that the sometimes simplistic and naive approaches from other tools (assuming correct files instead of broken data, reliance on magic headers) is not realistic. This is why I created BANG, which tries to take these into account. Focus in BANG is on correctness, but also on speed. Currently around 150 different file formats can be unpacked or l...
Een reactie posten
Meer lezen

Walkthrough: ZIP file format

A very popular format for compressing and archiving files is the ZIP format. Its history is well documented at the PKZIP page on Wikipedia . While on Unix systems (such as Linux) other compression tools (tar for archiving, combined with gzip for compression) have historically been more popular it is the dominant archiving and compression format in the rest of the world. Other systems, like Java and Android are also using ZIP extensively (in JAR and APK files respectively) and other formats, such as Open Document Format (ODF) are also based on ZIP. The specifications for ZIP have been open for years and can be freely (re)implemented and every decent programming language has good support for working with ZIP files, although not every implementation fully implements all functionality. For example, Python 3 has is a fairly complete implementation with support for LZMA and bzip2 compression and large files (ZIP64), but not for multi-file ZIP. The references in the rest of this blog post...
Een reactie posten
Meer lezen