Apache Spark -Unable to load native-hadoop library for your platform... using builtin-java classes where applicable" and terminate the execution

Question

I am using Apache Spark on Windows 10 64 bit machine. I have installed Java, Python 3.6 ,spark-2.3.1-bin-hadoop2.7. I am using VSCode editor for PySpark codeing.

When I'm executing the Python spark code in VSCode using spark-submit, it is showing

Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

and is terminating the execution.

Relevant code:

from pyspark import SparkContext, SparkConf 
if name == "main": 
    conf = SparkConf().setAppName("word count").setMaster("local[2]") 
    sc = SparkContext(conf=conf) 
    lines = sc.textFile("in/word_count.text") 
    words = lines.flatMap(lambda line: line.split(" ")) 
    wordcounts = words.countByValue() 
    for word, count in wordcounts.items(): 
        print("{} : {}".format(word,count))

Spark Execution Error:

Please add your code and exception as text instead of an image. Also, that's just a warning and would not cause termination of the program. It'd be helpful if you can add the entire code (assuming it's not too big), a minimal reproducible example. — philantrovert
– philantrovert, Commented Sep 6, 2018 at 9:50
Hi, I am getting the same waring message when executing the pyspark command from windows command prompt. — user3364545
– user3364545, Commented Sep 6, 2018 at 12:52

Community · Accepted Answer · 2020-06-20 09:12:55Z

3

You can safely ignore the warning as it is not the reason behind your shutdown call. According to documentation:

The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

The native hadoop library is mainly used on the GNU/Linus platform and has been tested on these distributions:

RHEL4/Fedora Ubuntu Gentoo On all the above distributions a 32/64 bit native hadoop library will work with a respective 32/64 bit jvm.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Feb 8, 2019 at 13:21

Nauman Naeem

4083 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Lawrence Over a year ago

Thanks, this is reassuring. Does that apply to Windows as well?

Collectives™ on Stack Overflow

Apache Spark -Unable to load native-hadoop library for your platform... using builtin-java classes where applicable" and terminate the execution

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related