how to Redis-cli command to mass insert CSV data into Redis persistence system?

Question

My CSV data looks something like this ->

    Header1   Header2
    Key100    Value100
    Key200    Value200

It has only two columns with Header as first row. The columns are keys and respective values. Now, my requirement is -

I want to store key & value as String data type
Need to exclude the Header(the first row) while storing
Need to exclude null or " " keys
it is a ',' separated data
The word "Data:" should be appended to each key while inserting.

For example - for the above provided csv data, redis should store as -

Data:Key100
Data:Key200

Result:

When I do get Data:Key100 my output should be : "Value100"

Have tried with awk command but it is keep on giving error - invalid stream id specified

awk -F ',' 'FNR > 1 {print "XADD myStream \""($1=="" ? "NA" : $1)" column_1 "($2=="" ? "NA" : $2)" column_2 ... \n"}' data.csv | redis-cli --pipe

Please help me to get the command. Thank you

Thanks for sharing your efforts, could you please post samples of input and expected output more clearly in your question, to make your question more clear, thank you. — RavinderSingh13
– RavinderSingh13, Commented Jul 30, 2021 at 5:26
Hi, I have edited my question by adding the expected sample. Hope this clarifies. Thank you. — Vedant
– Vedant, Commented Jul 30, 2021 at 5:40
By using XADD, are you intentionally using a redis stream data type? A redis hash structure using HMSET would suit your data best. — bigtony
– bigtony, Commented Jul 30, 2021 at 6:39
Mass insertion (see redis.io/topics/mass-insert) gives speed benefits by using a raw redis protocol. You probably don’t know yet if you need high speed, so your question is a little confusing because it says you want to mass insert and you have used the —pipe parameter. — bigtony
– bigtony, Commented Jul 30, 2021 at 6:44
It might be best to work in the redis-cli interpreter to learn the commands and test your data structure before you automate the batch import with awk. — bigtony
– bigtony, Commented Jul 30, 2021 at 6:48

Mark Setchell · Accepted Answer · 2021-07-30 11:35:25Z

1

I think you mean this, which sets an individual key to a value for each line of your file, but I am not sure why you seem to be trying to use a stream:

awk -F ',' 'FNR>1 { if(!$1){$1="NA"}; if(!$2){$2="NA"}; printf("SET %s %s\n",$1,$2)}'  data.csv | redis-cli --pipe

For this file (which has the key missing on line 3 and the value missing on line 5):

Header1,Header2
Key100,Value100
,fred
Key200,Value200
bill,

That produces this code:

SET Key100 Value100
SET NA fred
SET Key200 Value200
SET bill NA

Which then enables you to do this:

redis-cli get Key100
"Value100"

Also, you say you want to exclude null keys, but your code replaces them with "NA" and sets them anyway, so you may prefer to ignore them altogether with:

awk -F ',' 'FNR > 1 && $1 && $2 {printf("SET %s %s\n",$1,$2)}'  data.csv

which produces:

SET Key100 Value100
SET Key200 Value200

edited Jul 30, 2021 at 11:35

answered Jul 30, 2021 at 10:58

Mark Setchell

210k32 gold badges309 silver badges503 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Vedant Over a year ago

Hi, thank you so much for the needful, but I forgot to add my another requirement. Can you please try to me help me on that part ? i.e., the provided word to be appended to all the keys while insert, for example -> my csv key is 1) Key100 then in redis it should store as Data:Key100 2) my csv key is Key200 then my redis should store as Data:Key200. Eventually, get Data:Key100 gives me the value.

Mark Setchell Over a year ago

just change the printf to SET Data:%s %s\n

Vedant Over a year ago

On the same one, what if I dont have my headers row and data start from first row?

Mark Setchell Over a year ago

Just remove FNR > 1 && so that it looks like this awk -F ',' '$1 && $2 {...}'

Mark Setchell Over a year ago

FNR>1 just tests if the record number in the file (i.e. the line number) is greater than 1, so it doesn't print the first line.

|

Collectives™ on Stack Overflow

how to Redis-cli command to mass insert CSV data into Redis persistence system?

Result:

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Result:

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related