Doorgaan naar hoofdcontent

In the beginning...

This blog is a collection of articles about analysis of binary and source code files, and anything related to scanning for license compliance, security or code provenance. I started this blog because I think that the software scanning industry (especially for binary files) is not in a good shape and I feel that it is time to change that, as we collectively deserve better.

One of the observations that I made in the last few years is that people and companies are flocking to expensive tools and solutions when there is no need to because better working (and cheaper) solutions exist. The reasons are bad understanding of the problem needing to be solved, plus effective marketing.

The companies then use the wrong tools to try to get things done and shoehorn these tools into their process, leading to suboptimal results, frustration, time wasted and money lost. The reason: someone paid a lot of money for the license for the tool, so they'd better make use for it!

What I have also seen is that for most of the problems that people are trying to solve there are perfectly fine commodity solutions available at zero or very little cost. Usually it just means taking a few steps back, understanding the problem, using common sense and then using the right tool for the right job.

In this blog I want to describe best practices of scanning (when, what and how), open source alternatives to commercial scanning tools, but also to deepdive into specifics of file formats, open source tools and analysis techniques.

Specifically I will be talking about the following:
  1. open source license compliance
  2. security
  3. software archaeology/provenance
You might find example code here and there as well. This code will mostly be in Python3, plus the odd bash script here and there. Let's start!
Labels:

Reacties

Een reactie posten

Populaire posts van deze blog

Walkthrough: WebP file format

A graphics file format that I am encountering a bit more often during my work is Google's WebP file format. Even though it is fairly recent (or the history it is best to read the Wikipedia page about WebP ) it builds on some quite old foundations. One reason for Google to come up with a new graphics file format was file size: Google indexes and stores and sends many graphics files. By reducing the size of files they could significantly save on bandwidth and storage space. Shaving off some bytes here and there really starts to add up when you are doing it by the billions. Everyting counts in large amounts - Depeche Mode WebP file format The WebP format uses the Resource Interchange File Format (RIFF) as its container. This format is also used by other formats such as WAV and very easy to process automatically. A WebP file consists of a header, and then a number of chunks. The data in the header applies to the entire file, while data in the chunks only apply to the individu...
Een reactie posten
Meer lezen

Fuzzy hash matching

Fuzzy hash matching, or proximity hashing, is a powerful method to find files that are close to the scanned file. But: it is not a silver bullet. In this blogpost I want to look a bit into proximity matching, when it works and especially when it does not work. Cryptographic hashes Most programmers are familiar with cryptographic hashes such as MD5, SHA256, and so on. These hashes are very useful when needing to uniquely identify files (except in the case of hash collisions, but those are extremely rare). These algorithms work by taking an input (the contents of a file) and then computing a very long number. A slight change in the input will lead to a drastically different number. This is why these cryptographic hashes are great for uniquely identifying files as the same input will lead to the same hash, but useless for comparing files, as different inputs will lead to a very different hash and a comparison of hashes is completely useless. Locality sensitive hashes A different ...
Een reactie posten
Meer lezen

Walkthrough: PNG file format

A relatively straightforward file format that is used a lot in firmware files that I see is the Portable Network Graphics file format, or simply PNG. To give an example of how widespread it is: in a regular Android firmware with a few applications installed you can easily find over 50,000 PNG files, with quite a few duplicates as well. What baffles me is that quite a few of the license scanning tools out there (including some open source tools) also try to do a license scan of a PNG file. This makes no sense to me at all. While possibly interesting from a copyright perspective (which is about what is in the picture or possibly in the metadata ) the files themselves are not interesting when scanning software: valid PNG files do not contain executable code (maliciously crafted PNG files that exploit errors in PNG parsers are of course a different story). PNG files cannot be combined with other files to create "derivative" software: software cannot be linked with a PNG fil...
Een reactie posten
Meer lezen