6

I have a large (2GB) file that looks like this:

^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^
<binary data>
^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^ 
<binary data>
^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^
<binary data>
...

The ^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^ lines are separators. The binary segments are large. There are about fifty of them in the file.

I am trying to extract the binary parts of this file. Each binary segment needs to go into its own file.

I tried using csplit,

csplit --digits=2 --prefix=out stu.ear '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/'

but received the following output and two out?? files,

1
2097951144

Is there a tool for this job (a csplit implementation that works with binary files, perhaps?)

Improve this question
2
  • 1
    did you try awk '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/{n++}{print >"out" n ".ear" }' stu.ear, This should work but I am not sure about the awk data input limits Commented Jul 1, 2013 at 1:32
  • To only split big files into smaller ones is not sufficient. I asked a similar question but with simultaneous progress information here stackoverflow.com/q/31021731/54964 I think the main tool here is with dclfdd. Commented Jun 24, 2015 at 8:41

3 Answers 3

Reset to default
6

I wrote a little python tool to do this. https://github.com/mypalmike/csplitb

csplitb.py --prefix X --suffix Y --number Z XXXXXXXX input-file.extension
X = Name at beginning of output filename
Y = Desired output file extension
Z = Number of digits used to differentiate output files
XXXXXXXX = Starting hex of each binary file to be split out of the input file
input-file.extension = The file being split

Example:
csplitb.py --prefix photo --suffix .png --number 4 89504e47 block-file.raw

Output:

photo0000.png
photo0001.png
photo0002.png
.............
Improve this answer
3
  • 1
    Welcome to Unix & Linux Stack Exchange! Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. Commented Nov 15, 2013 at 1:47
  • 1
    You should at the least show an example of how to use the tool you are recommending. Commented Nov 15, 2013 at 3:57
  • Thanks, but it's not huge-files-friendly. Traceback (most recent call last): File "./csplitb/csplitb.py", line 64, in <module> main() File "./csplitb/csplitb.py", line 61, in main return csplitb.run() File "./csplitb/csplitb.py", line 30, in run self.finish() File "./csplitb/csplitb.py", line 42, in finish self.write(self.mm[self.last_idx:]) MemoryError Commented Nov 5, 2016 at 8:54
3

The following will work:

      awk '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/{n++}{print >"out" n ".ear" }
Improve this answer
1
  • Have you tried dcfldd command? Awk solution may become problematic in some situations. Commented Jun 24, 2015 at 8:19
1

You told csplit to split the file at one location, at the first occurrence of the text ^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%^. So you naturally got two pieces: one that contains the first byte of the file (is there a newline or blank before the first separator?) and one that contains everything from the first separator onwards.

If you want to split into separate files, you'll have to repeat the pattern as many times as you want pieces minus one. Portably, you need to count the pieces.

csplit --digits=2 --prefix=out stu.ear '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/'"{$(grep -c '\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^' stu.ear)}"

GNU csplit has an extension that lets you repeat a pattern an indefinite number of times:

csplit --digits=2 --prefix=out stu.ear '/\^%%-=-=-=-=-=-=-=-=-=-=-=-=-=-%%\^/ {*}'

But this doesn't do what you want, because the separator is included in the output. You can remove it from the files afterward; it would be a little easier if you arrange to have the separators at the end of the files, using % rather than / as the pattern delimiter (so csplit … '%\^\%\%-=-=-=-=-=-=-=-=-=-=-=-=-=-\%\%\^% {*}'). But you might as well resign yourself to the fact that csplit, while cute, has a very narrow use case, and yours doesn't fit. Use a better-suited tool such as awk.

Improve this answer

You must log in to answer this question.