Need help in creating dynamic elasticsearch index using Java high level rest client bulk API

Question

I am creating an Elasticsearch cluster that will integrate with our Java codebase. I want to create an Elasticsearch index and insert SQL Query data into it from multiple databases. The query results from all the databases should be inserted into the same index. For that purpose I am using Java High level Rest Client. But I am not quite sure how to do this because a lot of methods from the old APIs are deprecated. I also am not quite so sure what to do with the createIndexResponse instance too. Can anyone help me in this?

        public static void method_1(Connection con) throws Exception {

            Statement statement = con.createStatement();
            try {
                ResultSet result = statement.executeQuery("SELECT Field_1, Field_2, Field_3 from Table_1");
                int counter = 1;
                CreateIndexRequest createIndexRequest = new CreateIndexRequest("index_name");
                createIndexRequest.settings(new Settings.Builder()
                        .put("cluster.name", "my_cluster")
                        .put("http.enabled", true)
                        .put("node.data", true)
                        .put("index.number_of_shards", 3)
                        .put("index.number_of_replicas", 1)
                        .build());
                CreateIndexResponse createIndexResponse = ElasticSearch.eclient.indices().create(createIndexRequest, RequestOptions.DEFAULT);
                BulkRequest bulkRequest = new BulkRequest();
                while (result.next()) {
                    String field_1 = result.getString("Field_1");
                    int field_2 = result.getInt("Field_2");
                    String field_3 = result.getString("Field_3");
                    XContentBuilder builder = XContentFactory.jsonBuilder()
                            .startObject()
                            .field("Field 1", field_1)
                            .field("Field 2", field_2)
                            .field("Field 3", field_3)
                            .endObject();
                    UpdateRequest updateRequest = new UpdateRequest("index_name", "_doc", Integer.toString(counter));
                    updateRequest.doc(builder);
                    bulkRequest.add(updateRequest);
                }
                BulkResponse response = ElasticSearch.eclient.bulk(bulkRequest, RequestOptions.DEFAULT);
                if (response.hasFailures()) {
                    for (BulkItemResponse item : response.getItems()) {
                        System.out.println(item.getFailureMessage());
                    }
                }
                counter++;
                statement.close();
            } catch (Exception ex) {
                ex.printStackTrace();
            }
        }

The title is misleading IMO. You should fix it.

dadoonet
– dadoonet

2019-05-09 14:06:32 +00:00
Commented May 9, 2019 at 14:06 — dadoonet
– dadoonet, Commented May 9, 2019 at 14:06

dadoonet · Accepted Answer · 2019-05-09 17:00:10Z

3

You did not give which version you are using. Anyway, I'm considering that you are using 7.0.0.

Here is an example which should be ok I guess:

try (RestHighLevelClient client = new RestHighLevelClient(
        RestClient.builder(HttpHost.create("http://localhost:9200")))) {
    CreateIndexRequest createIndexRequest = new CreateIndexRequest("index_name");
    createIndexRequest.settings(Settings.builder()
            .put("index.number_of_shards", 3)
            .put("index.number_of_replicas", 1)
            .build());
    client.indices().create(createIndexRequest, RequestOptions.DEFAULT);
    BulkRequest bulkRequest = new BulkRequest();
    XContentBuilder builder = XContentFactory.jsonBuilder()
            .startObject()
                .field("foo", "bar")
            .endObject();
    IndexRequest indexRequest = new IndexRequest("index_name");
    indexRequest.source(builder);
    bulkRequest.add(indexRequest);
    BulkResponse response = client.bulk(bulkRequest, RequestOptions.DEFAULT);
    if (response.hasFailures()) {
        for (BulkItemResponse item : response.getItems()) {
            System.out.println(item.getFailureMessage());
        }
    }
} catch (IOException e) {
    e.printStackTrace();
}

I'd recommend using the BulkProcessor class which is much easier to deal with IMO. I have a full demo repository which I'm updating on every new release. It might help you as well.

answered May 9, 2019 at 17:00

dadoonet

14.6k3 gold badges46 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Soumya C Over a year ago

Thank you for the helpful insights. So I guess for inserting documents in a loop, I don't have to use UpdateRequest, IndexRequest itself is enough.

dadoonet Over a year ago

Update API is useful to update only a small part of a very big document (like 10mb document where you want to update a single field). In general, just use the Index Document API for inserting and updating documents. That's the preferred way.

Soumya C Over a year ago

I have also noticed another thing. In the updated IntelliJ version, the method "create(createIndexRequest, RequestOptions.DEFAULT)" is showing as deprecated (striked out). Is there any workaround for this?

Collectives™ on Stack Overflow

Need help in creating dynamic elasticsearch index using Java high level rest client bulk API

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related