Write data to Redis from PySpark

Question

In Scala, we would write an RDD to Redis like this:

datardd.foreachPartition(iter => {
      val r = new RedisClient("hosturl", 6379)
      iter.foreach(i => {
        val (str, it) = i
        val map = it.toMap
        r.hmset(str, map)
      })
    })

I tried doing this in PySpark like this: datardd.foreachPartition(storeToRedis), where function storeToRedis is defined as:

def storeToRedis(x):
    r = redis.StrictRedis(host = 'hosturl', port = 6379)
    for i in x:
        r.set(i[0], dict(i[1]))

It gives me this:

ImportError: ('No module named redis', function subimport at 0x47879b0, ('redis',))

Of course, I have imported redis.

@zero323 Is that the way to do it? Install redis on every worker. — kamalbanga
– kamalbanga, Commented Aug 29, 2015 at 7:33
python modules to be used in the workers must be on all the workers.... so he means the python redis module, not a redis db installation. — Paul
– Paul, Commented Aug 29, 2015 at 10:50
@Paul: I understood what he meant, and that's what I am asking. Do I have to install the python redis module on all the workers manually? There should be an easier and shortcut way, like Scala API's addJars method. — kamalbanga
– kamalbanga, Commented Aug 29, 2015 at 11:26
@kamalbanga I'm unaware of a good way. Of course you could try to use spark to make the workers run pip or easy_install but unless you can limit workers to one per machine, it might not behave very well. — Paul
– Paul, Commented Aug 29, 2015 at 11:37

Community · Accepted Answer · 2017-05-23 12:17:11Z

7

PySpark's SparkContext has a addPyFile method specifically for this thing. Make the redis module a zip file (like this) and just call this method:

sc = SparkContext(appName = "analyze")
sc.addPyFile("/path/to/redis.zip")

edited May 23, 2017 at 12:17

CommunityBot

11 silver badge

answered Aug 30, 2015 at 7:36

kamalbanga

2,0515 gold badges30 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Write data to Redis from PySpark

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related