Thursday, 28 April 2011

Concatenating binary raw data files

At my company, Breivoll Inspection Technologies, we perform inline ultrasound scans of water pipelines, and the resulting raw data is stored in a number of binary files for postprocessing. I was looking for a simple way to combine these files, and found out that the familiar cat command (on Linux) does the trick:

cat *.dat > mergedFile.dat

This concatenates all the .dat files in the current directory and outputs them into mergedFile.dat. Note that although cat is often used for text files, it works just as well for binary files.

Another note: In some cases, a raw data file may contain headers or footers that should not be included when the files are concatenated. I found a simple solution in this forum post. For binary files, a number of bytes can easily be stripped away from the beginning and the end of the file by using the dd command, for example:

dd if=input.dat of=output.dat bs=1 skip=X count=Y

This skips the X first bytes of the input file, and writes the following Y bytes to the output file. See the help section on the dd command for more options.


Post a Comment