extracting specified line numbers from file using shell script

Question

I have a file with a list of address it looks like this (ADDRESS_FILE)

0xf012134  
0xf932193  
.  
.  
0fx12923a

I have another file with a list of numbers it looks like this (NUMBERS_FILE)

I want to cut the first 20 lines from ADDRESS_FILE and put that into a new file
then cut the next 40 lines from ADDRESS_FILE so on ...

I know that a series of sed commands like the one given below does the job

sed -n 1,20p ADDRESSS_FILE > temp_file_1
sed -n 20,60p ADDRESSS_FILE > temp_file_2
.  
.
sed -n somenumber,endofilep.  ADDRESS_FILE > temp_file_n

But I want to does this automatically using shell scripting which will change the numbers of lines to cut on each sed execution.

How to do this ???

Also on a general note, which are the text processing commands in linux which are very useful in such cases?

Wrt text processing commands, a lot of it can be done directly by your shell so get to know it intimately. If you're on a GNU based system I would recommend reading coreutils.info and sed.info. For more complicated tasks awk and perl are good. — Thor
– Thor, Commented Aug 2, 2012 at 20:03

Brian Agnew · Accepted Answer · 2012-08-02 15:43:47Z

2

Assuming your line numbers are in a file called lines, sorted etc., try:

#!/bin/sh

j=0
count=1
while read -r i; do
  sed -n $j,$i > filename.$count  # etc... details of sed/redirection elided
  j=$i
  count=$(($count+1))
done < lines

Note. The above doesn't assume a consistent number of lines to split on for each iteration.

Since you've additionally asked for a general utility, try split. However this splits on a consistent number of lines, and is perhaps of limited use here.

edited Aug 2, 2012 at 15:43

answered Aug 2, 2012 at 14:08

Brian Agnew

273k38 gold badges342 silver badges443 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

jordanm Over a year ago

You shouldn't use a for loop to read lines in a file. It happens to work in this case, only because there is one word per line. It is overall bad practice. Use while read -r line; do ...; done < file.

Brian Agnew Over a year ago

It works in this case because he's specified it as such. See the question.

jordanm Over a year ago

This is not justification to do it the wrong way. Don't Read Lines With For.

Brian Agnew Over a year ago

I don't disagree that there are better ways and it's worth pointing it out. To downvote the complete solution (including detailing alternative options) seems particularly petty, however

Deepthought Over a year ago

Well this worked but I dont understand how it worked :) Dont just give fish to a hungry man , teach him to fish :) ...... What about the tools one needs to learn for text manipulation ? Please answer so that I too can give the same help to someone as I have got from you people.

|

Thor · Accepted Answer · 2012-08-02 14:51:08Z

1

Here's an alternative that reads directly from the NUMBERS_FILE:

n=0; i=1
while read; do 
  sed -n ${i},+$(( REPLY - 1 ))p ADDRESS_FILE > temp_file_$(( n++ ))
  (( i += REPLY ))
done < NUMBERS_FILE

answered Aug 2, 2012 at 14:51

Thor

47.7k12 gold badges125 silver badges140 bronze badges

Comments

Igor Chubin · Accepted Answer · 2012-08-02 20:18:11Z

1

size=$(wc -l ADDRESSS_FILE)
i=1
n=1
while [ $n -lt $size ]
do
  sed -n $n,$((n+19))p ADDRESSS_FILE > temp_file_$i
  i=$((i+1))
  n=$((n+20))
done

or just

split -l20 ADDRESSS_FILE temp_file_

(thanks Brian Agnew for the idea).

edited Aug 2, 2012 at 20:18

answered Aug 2, 2012 at 14:11

Igor Chubin

65.1k14 gold badges132 silver badges148 bronze badges

2 Comments

Deepthought Over a year ago

I think $ cant be used in sed.I got errors. Instead use this a=2 b=4 sed '/$a/,/$b/p' input_file > new_file

Igor Chubin Over a year ago

@Deepthought: sorry, it must be $(()) not $() in many cases in my example; I fixed that. $n and $((n+20)) can be used in sed of course because that is no in sed actually, that is in shell; and the expands it

Hasturkun · Accepted Answer · 2012-08-05 12:29:49Z

1

An ugly solution which works with a single sed invocation, can probably be made less horrible.

This generates a tiny sed script to split the file

#!/bin/bash
sum=0
count=0
sed -n -f <(while read -r n ; do
    echo $((sum+1),$((sum += n)) "w temp_file_$((count++))" ;
done < NUMBERS_FILE) ADDRESS_FILE

edited Aug 5, 2012 at 12:29

answered Aug 2, 2012 at 15:21

Hasturkun

36.7k6 gold badges78 silver badges109 bronze badges

2 Comments

Deepthought Over a year ago

Well this worked but I dont understand how it worked :) Dont just give fish to a hungry man , teach him to fish :) ...... What about the tools one needs to learn for text manipulation ? Please answer so that I too can give the same help to someone as I have got from you people

Hasturkun Over a year ago

Sorry, here's what this does: It goes over the NUMBERS_FILE, generating a sed write command for each set of lines, eg, 1,20 w temp_file_0, 21,60 w temp_file_1 and so on. sed processes each address range in turn, writing the output into the relevant file

Collectives™ on Stack Overflow

extracting specified line numbers from file using shell script

4 Answers 4

6 Comments

Comments

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

6 Comments

Comments

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related