Merging log files with different timestamp formats

Question

I have two types of log files

Standard Syslog format

The timestamp looks as below

5:2015 Dec 21 07:35:06:ABC:foo1:1559: common.c:946:Enabling filter
6:2015 Dec 21 07:35:08:ABC:bar1:1461: api.c:124:Trigger activated
6:2015 Dec 21 07:35:16:BMC:kernel:-:<6>drivers/usb1_1.c:598:Error processing request on endpoint 0

Custom Log format

The date time is given in first line of the file. following lines are "relative" to the first line. example as below:

Timestamp H:M:S 15:4:1 D:M:Y 16:1:2015
Firmware Version: 121020150140
[04:01]------------[ Logs ]------------
[04:03]Device Data: -> Supported Attributes -> 0x8033B
[04:01]Device Cleanup

[04:19]SendClearMsg ...
[04:23]Param:GetData failed
[04:51]Current Update Count:7
[05:01]MODECHK:Normal mode

a timestamp of 4:01 is 4 minutes, 1 second since 15:4:1 which should be translated as 15:08:2.

Any suggestions to achieve this ?

translate relative times into absolute time
Merge with syslog into one big file, sorted by time

What is the cost of changing the way the second timestamp is displayed ? — netmonk
– netmonk, Commented Dec 21, 2015 at 9:16
@netmonk okay..i got what you are asking. We don't have the source for second log, it is from a different module from third party. — rajaganesh87
– rajaganesh87, Commented Dec 21, 2015 at 9:52
@netmonk i am exploring with "date "-d +5 min 3 sec",it works but i am having trouble replacing it back in same line number in log — rajaganesh87
– rajaganesh87, Commented Dec 21, 2015 at 10:32
What are those leading 5:, 6:, do you want to keep them? Before the timestamp? Do you want to keep the Timestamp and Firmware Version line? Do you mind the final format for the timestamp? Is 2015-12-21 13:12:02 OK? — Stéphane Chazelas
– Stéphane Chazelas, Commented Dec 21, 2015 at 12:29

meuh · Accepted Answer · 2015-12-21 11:44:47Z

This solution uses an awk script to convert the dates in the first file to the number of seconds since the epoch and preprends this number to the output. We use date +%s --date to do the hard work, and capture the output of the command into awk variable secs by calling awk's getline function. (The awk syntax is: command | getline variable).

awk <log1 >log1.new '
{ y = substr($0,3,4); m = $2; d = $3; hms = substr($0,15,8)
  "date \"+%s\" --date \"" d " " m " " y " " hms "\"" | getline secs
  print secs " " $0
}'

A second awk script does the same for the second file, but only for the first line with the Timestamp, which is saved in awk variable base. On the other lines we simply add the minutes and seconds offset to this base and use date to convert the seconds since the epoch into a real date, in the format of the first file.

awk <log2 >log2.new '
/^Timestamp/{ split($5,x,":"); dmy = sprintf("%04d/%02d/%02d",x[3],x[2],x[1])
              split($3,x,":"); hms = sprintf("%02d:%02d:%02d",x[1],x[2],x[3])
              "date \"+%s\" --date \"" dmy " " hms "\"" | getline base
}
/^\[/ { mins = substr($0,2,2); secs = substr($0,5,2);
        tot = base + mins*60+secs
        "date \"+%Y %b %d %H:%M:%S\" --date @" tot | getline date
        print tot " -:" date " " substr($0,8)
}'

The two files are then merged by a sort on the number field, and finally the number is removed by a sed.

sort -m -n -k1,1 log1.new log2.new |
sed 's/^[^ ]* //'

the_velour_fog · Accepted Answer · 2015-12-21 12:25:29Z

This adds the relative numbers produced by your custom log file to the original base number and updates all lines

#!/usr/bin/env bash

f=$(head -n 1 custom_log_format.log)
base=$(sed 's/.*H:M:S \(.*\) D:M:Y.*/\1/' <<<$f)

OLDIFS=$IFS
IFS=$'n' 
readarray lines < custom_log_format.log
IFS=$OLDIFS
for i in ${!lines[@]}
do
    b="${lines[$i]}"
    if [[ $b == "["* ]]
    then
        rel_time=$(sed 's/^\[\(.*[^ ]\)\].*/\1/' <<<$b)
        time=$(echo $rel_time | awk -F: '{ print ($1 * 60) + $2 * 60 }')
        # convert base to seconds
        base_seconds=$(date -d"$base" +"%s")
        new_time_seconds=$(( base_seconds + time ))

        new_time=$(date -d"@$new_time_seconds" +"%H:%M:%S")
    fi
    echo ${b/$rel_time/$new_time}
done

output

Timestamp H:M:S 15:4:1 D:M:Y 16:1:2015
Firmware Version: 121020150140
[15:09:01]------------[ Logs ]------------
[15:11:01]Device Data: -> Supported Attributes -> 0x8033B
[15:09:01]Device Cleanup

[15:27:01]SendClearMsg ...
[15:31:01]Param:GetData failed
[15:59:01]Current Update Count:7
[15:10:01]MODECHK:Normal mode

JamesThomasMoon · Accepted Answer · 2022-10-11 06:28:00Z

0

Try rust-based tool Super Speedy Syslog Searcher

(assuming you have rust installed)

cargo install super_speedy_syslog_searcher

then

s4 log1 log2

Super Speedy Syslog Searcher will sort varying log message datetime formats by the interpreted datetime.

answered Oct 11, 2022 at 6:28

JamesThomasMoon

98810 silver badges25 bronze badges

Add a comment |

Stack Exchange Network

Merging log files with different timestamp formats

3 Answers 3

You must log in to answer this question.

Hot Network Questions

Merging log files with different timestamp formats

3 Answers 3

You must log in to answer this question.

Related

Hot Network Questions