Java Program imports data using sqoop

Question

I have created a web application(JSP) to extract data from mysql database to HDFS.In my Java code, I have used sqoop import command to achieve my requirement. The program executed successfully but the extracted data written to normal unix file system instead of hdfs.

Can anyone let me know how to provide hdfs file system path in sqoop import command?

    package com.archival.da;
    import java.sql.*;

    public class DataImportSetup {

static int status=0;

public static int importsetup(String policy_id){


    Connection con=GetCon.getCon();
    PreparedStatement ps;
    try {
        ps = con.prepareStatement("SELECT  
                   CON.SERVER,CON.PORT,CON.DB,CON.USER,
                   CON.PWD,POLICY.SQL_TEXT FROM POLICY 
            JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID WHERE POLICY.ID=?");
        ps.setString(1,policy_id);
        ResultSet rs=ps.executeQuery();
        rs.next();
        String ServerNm =   
                    "jdbc:mysql://"+rs.getString(1)+":
                    "+rs.getString(2)+"/"
                    +rs.getString(3);
        String ConUser=rs.getString(4);
        String ConPass=rs.getString(5);
        String SqlText=rs.getString(6);

        String[] str={"import","--connect",ServerNm,"--hadoop-mapred- home","/ms/hadoop-1.2.0", "--query", SqlText , "--target-dir", "/user/root/city","--username", ConUser, "--password", ConPass,"--split-by","id"};
       status=Sqoop.runTool(str);
        System.out.println(status);

    } catch (SQLException e) {

        e.printStackTrace();
    }
    return status;

}

   }

Community · Accepted Answer · 2017-05-23 12:28:58Z

2

It's writing to the local file system instead of HDFS because the default file system is local unless otherwise configured. You can configure this to be HDFS using SqoopOptions - see this question / answer for an example:

How can I execute Sqoop in Java?

Specifically you need to locate and pass the location of your clusters core-site and hdfs-site xml files:

Configuration config = new Configuration(); 
config.addResource(new Path("/usr/local/hadoop/conf/core-site.xml"));
config.addResource(new Path("/usr/local/hadoop/conf/hdfs-site.xml"));

edited May 23, 2017 at 12:28

CommunityBot

11 silver badge

answered Jun 8, 2013 at 8:09

Chris White

30.1k4 gold badges75 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Vijay Over a year ago

Hello Chris, I used sqoop options and getting error on new ImportTool().run(options); The method run(SqoopOptions) in the type ImportTool is not applicable for the arguments (SqoopOptions) My configurations Hadoop 1.2.0 Sqoop 1.4.3 Can you please help on this issue?

Chris White Over a year ago

Check the package of your SqoopOptions - the ImportTool expects a type of com.cloudera.sqoop.SqoopOptions

Vijay Over a year ago

Thanks Chris, I have imported from com.cloudear.sqoop.SqoopOptions and It is working now.I am able to load data to hdfs file system now. I have one more question. Can I give the target file name(not directory) to load the data?

Chris White Over a year ago

I'm not sure about that - best to post another question

Collectives™ on Stack Overflow

Java Program imports data using sqoop

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related