Could not load HiveConf in Sqoop workflow in Oozie

Question

I created a Sqoop workflow in Oozie to sqoop data from MySQL into Hive table.

If I run Sqoop job in terminal, it run fine and data were successfully inserted into Hive table, but if I put the job inside Oozie and run it, it gives me an error:

Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.

Encountered IOException running import job: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf

Can anyone help me with this?

I'm using HDP 2.5 here.

Below is my properties file for Oozie:

nameNode=hdfs://master.nodes:8020
jobTracker=master.nodes:8050
queueName=default
examplesRoot=jas-oozie

oozie.use.system.libpath=true
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.action.sharelib.for.pig=hive,pig,hcatalog
oozie.action.sharelib.for.hive=pig,hcatalog,atlas,hive

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/jas-oozie-workflow.xml
outputDir=jas

and below is the xml file for Oozie:

<?xml version="1.0" encoding="UTF-8"?>

<workflow-app xmlns="uri:oozie:workflow:0.2" name="jas-import-wf">
    <start to="sqoop-import-air-quality-node"/>

    <action name="sqoop-import-air-quality-node">
        <sqoop xmlns="uri:oozie:sqoop-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data"/>
                <!-- <mkdir path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data"/> -->
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <arg>import</arg>
            <arg>--connect</arg>
            <arg>jdbc:mysql://xx.xx.xx.xx:3306/xxxx?dontTrackOpenResources=true&amp;defaultFetchSize=1000&amp;useCursorFetch=true&amp;zeroDateTimeBehavior=convertToNull</arg>
            <arg>--driver</arg>
            <arg>com.mysql.jdbc.Driver</arg>
            <arg>--username</arg>
            <arg>xx</arg>
            <arg>--password</arg>
            <arg>xx</arg>
            <arg>--query</arg>
            <arg>
                select <fields> from <table> where $CONDITIONS
            </arg>
            <arg>--hive-import</arg>
            <arg>--hive-drop-import-delims</arg>
            <arg>--hive-overwrite</arg>
            <arg>--hive-table</arg>
            <arg>table</arg>
            <arg>--target-dir</arg>
            <arg>/user/${wf:user()}/${examplesRoot}/output-data/sqoop-import</arg>
            <arg>-m</arg>
            <arg>1</arg>
        </sqoop>
        <ok to="end"/>
        <error to="import-air-quality-fail"/>
    </action>

    <kill name="import-air-quality-fail">
        <message>Sqoop from ICP failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>

    <end name="end"/>
</workflow-app>

Thank you.

You need to configure oozie share lib--docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/… — Rahul Sharma
– Rahul Sharma, Commented Oct 30, 2016 at 19:25
There is already a share/lib folder in my HDFS /user/oozie folder. In fact if I run this command: oozie admin -shareliblist I will get all the lib list — Xonos
– Xonos, Commented Oct 31, 2016 at 3:33

Nirmal Ram · Accepted Answer · 2016-10-31 14:38:41Z

0

1) create a directory lib in path ${nameNode}/user/${user.name}/${examplesRoot}/

2) add hive-exec jar in ${nameNode}/user/${user.name}/${examplesRoot}/lib/ path and try again

answered Oct 31, 2016 at 14:38

Nirmal Ram

1,2202 gold badges10 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Could not load HiveConf in Sqoop workflow in Oozie

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related