how to change date-format in a log file using bash, avoiding while loop

Question

This is not a new question here and here, but the details make it differ.

My input log file looks like:

TEMP MON -=- Sat Aug 15 02:20:24 EEST 2020 -=- 48.6
TEMP MON -=- Sat Aug 15 02:20:50 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:13 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:44 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:45 EEST 2020 -=- 48.6
TEMP MON -=- Sat Aug 15 02:21:52 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:53 EEST 2020 -=- 48.6
TEMP MON -=- Sat Aug 15 02:21:54 EEST 2020 -=- 49.6
TEMP MON -=- Sat Aug 15 02:21:56 EEST 2020 -=- 49.1
TEMP MON -=- Sat Aug 15 02:21:57 EEST 2020 -=- 49.1

and the output should look like:

TEMP MON -=- 2020-08-15_02:20:24 EEST -=- 48.6
...

So it is simple enough to change the format of a date in bash using

date -d ${date_in_current_format} "+DATE_IN_NEW_FORMAT"

It is also possible (albeit inefficient) to iterate over the log file using a while loop and change the dates line by line (see the 1st link again).

However, I am looking for a bash solution that uses sed or perl (or awk or anything else for that matter) to carry out the same task.

The tip of what I have tried but still does not work are the following search and replace functions:

perl -pe "s/(.*) -=- (.*) -=- (.*)/\1 -=- $( date \2 "+%Z %Y-%m-%d_%H:%M:%S" ) -=- \3/" <file>

and with sed something similar:

sed "s:\(.*\) -=- \(.*\) -=- \(.*\):\1 -=- $( date -d \2 "+%Z %Y-%m-%d_%H:%M:%S" ) -=- \3:" <file>

In both cases the problem is that I cannot get the search and replace substitution "\2" to be expanded within the bash date command execution.

This might help: Formatting date strings in a file with linux bash shell — Cyrus
– Cyrus, Commented Sep 2, 2020 at 14:22
@Cyrus: Reason why I asked because I kept trying TZ=EEST in my date command and it wasn't working. After your answer I figured in terms of a Unix command it is TZ=EET instead. — anubhava
– anubhava, Commented Sep 2, 2020 at 14:57
In Perl, $( will be interpreted as a variable. And \1 is better written as $1. — TLP
– TLP, Commented Sep 2, 2020 at 16:28
@nass if you care for efficiency and have large log files to convert, calling date for every line would make any script extremely slow. — thanasisp
– thanasisp, Commented Sep 5, 2020 at 2:43

thanasisp · Accepted Answer · 2020-09-02 18:59:46Z

4

With awk using only string functions, you can avoid calling the GNU awk datetime functions or the external command date, as we want to modify only the month and re-order the data.

> cat tst.awk
BEGIN { OFS=FS="-=-" }
{
    split($2, arr, " ")
    m=(index("JanFebMarAprMayJunJulAugSepOctNovDec", arr[2])+2)/3
    $2=sprintf(" %04d-%02d-%02d_%s %s ", arr[6], m, arr[3], arr[4], arr[5])
    print
}

Usage:

> awk -f tst.awk file
TEMP MON -=- 2020-08-15_02:20:24 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:20:50 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:13 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:44 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:45 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:21:52 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:53 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:21:54 EEST -=- 49.6
TEMP MON -=- 2020-08-15_02:21:56 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:57 EEST -=- 49.1

edited Sep 2, 2020 at 18:59

answered Sep 2, 2020 at 15:09

thanasisp

5,9953 gold badges17 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

anubhava · Accepted Answer · 2020-09-02 14:54:18Z

3

You may use this awk solution:

awk 'BEGIN {
   FS=OFS=" -=- "
}
{
   cmd = sprintf("TZ=EET date -d \"%s\" +\"%Y-%m-%%d_%T %Z\"", $2);
   if ((cmd | getline output) > 0)
      $2 = output
   close(cmd)
} 1' file

TEMP MON -=- 2020-08-15_02:20:24 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:20:50 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:13 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:44 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:45 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:21:52 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:53 EEST -=- 48.6
TEMP MON -=- 2020-08-15_02:21:54 EEST -=- 49.6
TEMP MON -=- 2020-08-15_02:21:56 EEST -=- 49.1
TEMP MON -=- 2020-08-15_02:21:57 EEST -=- 49.1

answered Sep 2, 2020 at 14:54

anubhava

790k67 gold badges603 silver badges671 bronze badges

3 Comments

tripleee Over a year ago

The OP is clearly on Linux, so this is more for future visitors; this is the least portable of these answers, as it requires GNU date.

anubhava Over a year ago

date -d command is mentioned in OP's question itself.

tripleee Over a year ago

Yes yes, I just feel that it needed to be pointed out for people who are not necessarily on Linux.

Håkon Hægland · Accepted Answer · 2020-09-02 17:58:22Z

2

Using core module Time::Piece in Perl:

perl -MTime::Piece -pe 's/-=-\s+\K(.*)(?=\s+-=-)/convert($1)/e;
  sub convert {
    $s = $_[0];
    $s =~ s/\s+EEST\s+/ /;
    $t = Time::Piece->strptime($s, "%a %b %d %T %Y");
    $res = $t->strftime("%Y-%m-%d_%H:%M:%S");
    "$res EEST"
 }' file

answered Sep 2, 2020 at 17:58

Håkon Hægland

40.9k22 gold badges96 silver badges209 bronze badges

Collectives™ on Stack Overflow

how to change date-format in a log file using bash, avoiding while loop

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related