bash how to insert multiple empty lines into a file according to another file

Question

I am meeting some difficulties when using bash to insert multiple empty lines into a file (called file1) according to an index file(called file2). (these files can be treated as passed variables) the index file (file2) looks like:

-
-
-
M
H
A
-
N
X
X
M
-
-
-
F
G
A
...

the file1 looks like this:

M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0
...

the output should be looks like this:

-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0
...

if file2 deletes the '-', the content as well as the order will always be the same as the first column in file1.

I tried dataframe in Python to deal with it, but it's too slow. So I was wondering how to use bash to figure this out. Thanks!

would the letters in the index file / file2 always be in the same order as in the data file / file1? I.e. that there's no need think about reordering any lines? I guess so, since the letters in the data file aren't unique, so the rules for reordering would need to be complicated — ilkkachu
– ilkkachu, Commented Apr 19, 2022 at 13:14
Do you really mean bash or do you just mean use the command line? This is a very bad problem for shell programming, are you OK with solutions using awk or perl or any other tool? — terdon
– terdon ♦, Commented Apr 19, 2022 at 13:30
yes, It's always in the same order @ilkkachu sorry, I thought bash contains awk, sed, grep, etc.@terdon — Jiao
– Jiao, Commented Apr 19, 2022 at 13:39
@Jiao, Bash is really just the shell, the tool that implements the command-line where you start other commands. Awk and sed etc. are distinct tools but standard. Don't do text processing with just the shell, that's a bad idea :) — ilkkachu
– ilkkachu, Commented Apr 19, 2022 at 13:58
I see. maybe I can just call them bash or shell tools or specify which tool — Jiao
– Jiao, Commented Apr 19, 2022 at 14:10

ilkkachu · Accepted Answer · 2022-04-19 13:28:44Z

6

Assuming the letters in the index file are always the right ones in the right order (so we can ignore which letter we see), and that the empty lines actually contain the dashes and aren't totally empty, maybe this should work:

$ awk -v datafile=data.txt '$1 == "-" { print "-"; next} { getline < datafile; print }' < index.txt 
-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0
...

It reads the index file one line at a time; if the first field there is exactly -, prints that; and otherwise reads and prints a line from the other file. (Which means that if a totally empty line comes up in the index file, it will also go to the next line from the data file.)

answered Apr 19, 2022 at 13:28

ilkkachu

148k16 gold badges268 silver badges441 bronze badges

1

a stupid question: if I use datafile="$data" it will not work. So, how to pass that variable?

Jiao
– Jiao

2022-04-19 14:13:06 +00:00
Commented Apr 19, 2022 at 14:13
1

@Jiao, hmm, I wonder why not? How exactly are you doing it? awk -v var=something sets the awk variable var to something. So if you have e.g. foo=123; awk -v bar="$foo" 'BEGIN {print bar}', you have the shell variable foo, that gets assigned to the awk variable bar that the awk script then prints. (Note the -v.)

ilkkachu
– ilkkachu

2022-04-19 14:15:34 +00:00
Commented Apr 19, 2022 at 14:15
1

data=`cat data.txt ` index=`cat index.txt` awk -v datafile="$data" '$1 == "-" { print "-"; next} { getline < datafile; print }' <(echo "$index")| less the output was the content of $index

Jiao
– Jiao

2022-04-19 14:18:31 +00:00
Commented Apr 19, 2022 at 14:18
1

@Jiao, ah, you're loading the whole contents of the files into the $data and $index variables in the shell. Hmm, I guess you can do that, but one usually doesn't. That makes it harder for the awk script to access the other file. You'd need to do essentially the same as the shell does with the process substitution (<(...)). But... do you have to use the variables in between?

ilkkachu
– ilkkachu

2022-04-19 14:37:26 +00:00
Commented Apr 19, 2022 at 14:37
1

the reality is that I have thousands of files that need to deal with. maybe I can generate them into a tmp folder. Thanks

Jiao
– Jiao

2022-04-19 14:44:14 +00:00
Commented Apr 19, 2022 at 14:44

| Show 4 more comments

steeldriver · Accepted Answer · 2022-04-19 17:29:20Z

3

Using GNU sed, also assuming that the order of the initial letters is strictly the same in the two file ignoring the - lines, read and insert one line from file1 for each "letter" line of file2:

$ sed -e '/^[A-Z]/{R file1' -e 'd;}' file2
-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0

answered Apr 19, 2022 at 17:29

steeldriver

83.9k12 gold badges124 silver badges175 bronze badges

Add a comment |

guest_7 · Accepted Answer · 2022-04-20 01:26:34Z

With file2 being the index file and file1 as the data file, using POSIX sed , using the index file we first generate a sed code that will then be applied to the data file to get the final desired output.

sed '
  1{
    x;s:.*:H;s/.*/-/;x:;x
    i\
x;s/.*/-/;x
    :loop
      /^-/!d;g;n
    b loop
  }
  /^-/s/.*/G/;t
  c\
n
' file2 |sed -f - file1

Output:-

-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0

guest_7 · Accepted Answer · 2022-04-20 01:40:17Z

2

[1] Perl Supply data file on the stdin of Perl program, and take the input from stdin unless the current line begins with a dash.

perl -pe '
  $_ = <STDIN> if !/^-/;
' file2 < file1

[2] GNU sed

sed '
  /^-/!Rfile1
  $!N;//P;D
' file2

[3] Python

python3 -c 'import sys
data_file,index_file = sys.argv[1:]

with open(data_file) as f, open(index_file) as g:
  for l in g:
    var = l if l.startswith("-") else next(f)
    print(var,end="")

' file1 file2

answered Apr 20, 2022 at 1:40

guest_7

5,7881 gold badge8 silver badges13 bronze badges

Add a comment |

Praveen Kumar BS · Accepted Answer · 2022-04-20 06:50:05Z

1

for j in $(awk '/-/{print NR}' file2); do sed -i ''$j'i -' file1; done

output

cat file1
-
-
-
M   x1 y1 z1 m1 n1
H   x2 y2 z2 m2 n2
A   x3 y3 z3 m3 n3
-
N   x4 y4 z4 m4 n4
X   x5 y5 z5 m5 n5
X   x6 y6 z6 m6 n6
M   x7 y7 z7 m7 n7
-
-
-
F   x8 y8 z8 m8 n8
G   x9 y9 z9 m9 n9
A   x0 y0 z0 m0 n0

answered Apr 20, 2022 at 6:50

Praveen Kumar BS

5,3212 gold badges12 silver badges16 bronze badges

1

The problem here is that you're running one instance of sed for each empty line added. That's not as bad as reading input line by line and running five instances of cut for every line, but it's still uselessly slow with all the processes being launched (and probably slower than whatever they did with Python dataframe). You could mitigate that by stacking multiple insert commands to the sed script, though I'm not sure how well that would scale. (Also, the quoting in ''$j'i -' seems off, why not just double-quote the whole thing, "${j}i -"?)

ilkkachu
– ilkkachu

2022-04-20 10:17:27 +00:00
Commented Apr 20, 2022 at 10:17

Add a comment |

Stack Exchange Network

bash how to insert multiple empty lines into a file according to another file

5 Answers 5

You must log in to answer this question.

Hot Network Questions

bash how to insert multiple empty lines into a file according to another file

5 Answers 5

You must log in to answer this question.

Related

Hot Network Questions