Bash script to delete line in CSV file

Question

I have a CSV file with million records for different users, there are multiple records for each user. I am doing some processing on the file and managed to get the record for each user which I want to delete.

I am using the below commands, but I can't delete the line I want to

get_date=$(grep -n "[email protected]"  francis_test.csv| awk -F, '{print $22}' | sed 's/"//g' | awk '{print $1" "$2}'  |  sort -k 1.7n -k 1.4,1.5n -k 1.1,1.2n | tail -n1)

record=$(grep -n "[email protected]"  francis_test.csv| grep "$get_date")

To delete the record I use the below command

sed '/"$record"/d'

I feel like with that many records, a proper database (Sqlite3 would be the simplest to use) might be better than CSV for pretty much anything you want to do with it. — Shawn
– Shawn, Commented Mar 22, 2021 at 11:26
millions of records you say? you might want to switch to at the very lest SQLlite. However you are using 3 tools here where you can use one. all you need is awk or sed. I think things are getting list in the piping. However you are using sed wrong here. you want sed -i "///" formate not sed with the d operator — Josh Beauregard
– Josh Beauregard, Commented Mar 22, 2021 at 11:30
Welcome to StackOverflow! For us to be able to help please provide your efforts as shown in Minimal, Reproducible Example. — Thomas Hansen
– Thomas Hansen, Commented Mar 22, 2021 at 11:32

P.P · Accepted Answer · 2021-03-22 11:31:28Z

1

Single quote prevents the variable record from being expanded. Quote it like:

sed '/'"$record"/'d'

sed also has an option -i to in-place modification (if you want to actually delete the lines from the file).

sed -i.bak '/'"$record"/'d'

Providing a suffix to -i makes a backup of the original file.

answered Mar 22, 2021 at 11:31

P.P

122k20 gold badges193 silver badges251 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

francis fernandes · Accepted Answer · 2021-03-22 13:09:24Z

0

thanks but now i am facing issue with the the below command

get_date=$(grep -n "[email protected]" francis_test.csv| awk -F, '{print $22}' | sed 's/"//g' | awk '{print $1" "$2}' | sort -k 1.7n -k 1.4,1.5n -k 1.1,1.2n | tail -n1)

cant get the output all of a sudden it fails

answered Mar 22, 2021 at 13:09

francis fernandes

415 bronze badges

Collectives™ on Stack Overflow

Bash script to delete line in CSV file

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related