Concatenate multiple log files into one single file to automate debug package generation

Question

Here are the files in my folder:

messages        
messages.1      
messages.2      
messages.3      
messages.4      
messages.5      
messages.6      
messages.7

I want to merge this files in one single file(e.g. log.txt) so that it is in this way:

messages.7 -> messages.6 -> messages.5 -> messages.4 -> messages.3 -> messages.2 -> messages.1 -> messages

I was manually executing cat commands i.e.

cat messages.7 >> log.txt
cat messages.6 >> log.txt
cat messages.5 >> log.txt
cat messages.4 >> log.txt
cat messages.3 >> log.txt
cat messages.2 >> log.txt
cat messages.1 >> log.txt
cat messages >> log.txt

I want to write a bash script which can handle any number of files.

FYI: messages is file with latest logs and messages.7 is the oldest.

Also, I do not want to reverse the content of the file. Number of files are dynamic and follows the same numbering patterns.

Actually, to do this manually, you can do it in a single statement: cat messages{.{7..1},} >total_messages. — user1934428
– user1934428, Commented Jun 6, 2023 at 5:46

user3159253 · Accepted Answer · 2023-06-06 01:03:33Z

2

The key point of the answer is to use a properly crafted sort command for file names. For the given filename pattern messages[.N] the command would be:

sort -t . -k 2 -n -r

-t . -k 2 allows to sort by the numeric suffix, -n actually instructs sort to make 2 < 10, -r reverses the order of the output.

So you can use something like this to check the order of files:

ls -1 messages* | sort -t . -k 2 -n -r

... and to actually get the concatenated output in 'total_messages':

ls -1 messages* | sort -t . -k 2 -n -r | xargs cat > total_messages

answered Jun 6, 2023 at 1:03

user3159253

17.6k3 gold badges36 silver badges61 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user1934428 Over a year ago

Perhaps it should be noted that this would also include a, say, file messages.zip. I guess you can restrict the glob pattern to include only those files with numeric extension, by using extended globbing.

Renaud Pacalet · Accepted Answer · 2023-06-06 07:02:17Z

There are 3 main difficulties:

List only the files you want, no more: use the extglob and nullglob bash options.
List them in a portable way (ls is not recommended for scripting): use regular pathname expansion.
Sort them in reverse numeric order of extension: use sort.

You can try:

#!/usr/bin/env bash

shopt -s extglob nullglob
m=( messages?(.+([0-9])) )
printf '%s\n' "${m[@]}" | sort -t. -rnk2 | xargs cat >> log.txt

With the extglob option of bash messages?(.+([0-9])) expands as messages (if the file exists) and all existing messages.n files where n is a string of one or more digits. We store the file names in array m, print its content and pipe to sort with the -t. (use . as field separator) and -rnk2 (reverse, numeric, second field) options.

The nullglob bash option takes care of the case where none of the files exist (without nullglob the pathname expansion produces the messages?(.+([0-9])) pattern itself, which causes an error because no such file exists).

xpusostomos · Accepted Answer · 2023-06-06 18:58:30Z

1

Strange that 7 is the oldest, but anyway....

cat `ls -r messages*` >total_messages

or I might suggest using timestamps...

cat `ls -t messages*` >total_messages

answered Jun 6, 2023 at 18:58

xpusostomos

1,69616 silver badges18 bronze badges

Collectives™ on Stack Overflow

Concatenate multiple log files into one single file to automate debug package generation

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related