0

I need to load the data to the elasticsearch index. I am using BULK API of elasticsearch to load the JSONs to index.

private String FOLDER_PATH = "src/main/resources/allJsons";
    private String index = "test1";
    private static final String TYPE = "test_type";

 @Autowired
    private RestHighLevelClient restHighLevelClient;

 public String loadBulkData() throws IOException {

        BulkRequest bulkRequest = new BulkRequest();
        AtomicInteger counter = new AtomicInteger();
        try (Stream<Path> filePathStream = Files.walk(Paths.get(FOLDER_PATH))) {
            filePathStream.forEach(filePath -> {
                if (Files.isRegularFile(filePath)) {
                    counter.getAndIncrement();
                    try {
                        String content = Files.readString(filePath);
                        JSONObject jsonObject1 = new JSONObject(content);
                        HashMap yourHashMap1 = new Gson().fromJson(jsonObject1.toString(), HashMap.class);
                        IndexRequest indexRequest = new IndexRequest(index, TYPE).source(yourHashMap1);
                        bulkRequest.add(indexRequest);

                    } catch (IOException e) {
                        e.printStackTrace();
                    }


                }
            });
        }
        try {
            restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
        } catch (IOException e) {
            e.printStackTrace();
        }
        return "Bulk data loaded to index " + index + "";
    }
}

I have multiple JSONs based on the following format

[
 {
  "Nutrient" : "Calories",
  "Amount" : " 289.00",
  "Unit" : " kcal"
}, {
  "Nutrient" : "Fat",
  "Amount" : " 17.35",
  "Unit" : " g"
}
]

While running the code it gives me error ,

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.json.JSONException: A JSONObject text must begin with '{' at 1 [character 2 line 1]

I think the data is in JSONArray and for the code, we need JSONObject. Anyone could please guide to how to do this

6
  • from where u are reading this data? Commented May 4, 2020 at 13:24
  • Intitially I have CSV files. I have converted the CSV to JSON files. and now need to load these JSONs Commented May 4, 2020 at 13:26
  • error message is pretty clear, ` JSONObject text must begin with '{' at 1 [character 2 line 1],, it expects to start the JSON with {` ie curly braces but its starting with [ . can you change this and try? Commented May 4, 2020 at 13:27
  • Also it would be better if you can share your JSON file, we can also test it locally real quick and fix it Commented May 4, 2020 at 13:27
  • @OpsterElasticsearchNinja I have shared the JSON sample in question. please consider that Commented May 4, 2020 at 13:29

1 Answer 1

1

You can do bulk insertion by passing hashmap of your json objects to Elasticsearch Bulk API. You can create Hashmap by parsing your JSON file through JSONParser.

Here is the code for the same :

Code :

          Integer id= 1;

          //You need to call this method for inserting bulk documents which 
          // internally calls `createBulkRequest` and `parseObjectList` methods.
          //This method uses JSONParser to parse your file and convert into JSONArray.
           public String insertBulkDocuments() throws Exception {
                Object obj = new JSONParser().parse(new FileReader(<path-of-file>)); 
                JSONArray objList= (JSONArray) obj;       
                BulkRequest request = createBulkRequest(objList);
                BulkResponse bulkresp=restHighLevelClient.bulk(request, RequestOptions.DEFAULT);
                return bulkresp.status().toString();
            }

            // Each JSONArray element that was obtained through first method 
           //is parsed individually through Gson and converted into you defined Object. 
           //This object is then converted to Map and passed to IndexRequest object.
            private BulkRequest createBulkRequest(JSONArray objList) {
                BulkRequest request = new BulkRequest();
                objList.forEach( obj -> parseObjectList((JSONObject) obj, request,id++));
                return request;
            }

            private void parseObjectList(JSONObject obj, BulkRequest request, int id) {
                Gson gson = new GsonBuilder().create();
                NutrientDocument doc = gson.fromJson(obj.toJSONString(), NutrientDocument .class);

                Map<String, Object> documentMapper = objectMapper.convertValue(doc, Map.class);

                IndexRequest indexRequest = new IndexRequest(<your-index-name>).id(Integer.toString(id)).source(documentMapper);
                request.add(indexRequest);
            }

You need to create Custom object which has same feilds as your json . I have created NutrientDocument for testing which has same fields as your JSON and this I am using in parseObjectList method.

public class NutrientDocument {
    private String Nutrient;
    private Float Amount;
    private String Unit;
    public String getNutrient() {
        return Nutrient;
    }
    public void setNutrient(String nutrient) {
        Nutrient = nutrient;
    }
    public Float getAmount() {
        return Amount;
    }
    public void setAmount(Float amount) {
        Amount = amount;
    }
    public String getUnit() {
        return Unit;
    }
    public void setUnit(String unit) {
        Unit = unit;
    }



}

NOTE :

For each document elasticserach generates unique id .

For creating our own id value instead of Elasticsearch autogenerated value, we are using id variable. But, if you want to go with Elasticsearch autogenerated number , you can create IndexRequest object as below in parseObjectList method and remove id variable wherever we are passing.

IndexRequest indexRequest = new IndexRequest().source(documentMapper);

Sign up to request clarification or add additional context in comments.

7 Comments

Sir what is the id in createBulkRequest method?
id is your autogenerate integer field if you want to give your custom id to each document you can declare at the top of your class Integer id= 1;and increate this id for each createindex object. Otherwise you can let Elasticsearch create autogenerate id by using IndexRequest indexRequest = new IndexRequest(<your-index-name>).source(documentMapper);. I will update answer as well
Also type field which you are using in your code is removed from ES 7.x onwards
This line is asking for a parameter to pass in JSONParser() Object obj = new JSONParser().parse(new FileReader());
You need to use like this : Object obj = new JSONParser().parse(new FileReader("src/main/resources/allJsons")); You had not paased file name to FileReader() object.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.