Bulk Insert object in Elasticsearch

Question

I am trying create an index and then do a bulk insert using RestHighLevelClient to my ES (the code is in Kotlin).

The bulk insert code is :

private fun insertEntity(entityList: List<Person>, indexName: String) {
    var count = 0
    val bulkRequest = BulkRequest()

    entityList.forEach {
        bulkRequest.add(IndexRequest(indexName).source(it,XContentType.JSON))
        count++

        if (count == batchSize) {
            performBulkInsert(bulkRequest)
        }
    }
}

When executing this, I am getting an exception saying : Limit of 1000 fields is crossed.

On analysing my code, I feel the implementation is wrong, because :

bulkRequest.add(IndexRequest(indexName).source(it,XContentType.JSON))

source takes a String type but I am passing the Person (it)object itself. So I believe that is causing some issue related to 1000 fields based on my mapping or something.

Not sure if my assumption is correct. If yes, how can I achieve the bulk insert then ?

EDIT

Index creation:

private fun createIndex(indexName: String) {
    val request = CreateIndexRequest(indexName)

    val settings = FileUtils.readFileToString(
        ResourceUtils.getFile(
            ResourceUtils.CLASSPATH_URL_PREFIX + "settings/settings.json"), "UTF-8")

    val mappings = FileUtils.readFileToString(
        ResourceUtils.getFile(
            ResourceUtils.CLASSPATH_URL_PREFIX + "mappings/personMapping.json"), "UTF-8")

    request.settings(Settings
        .builder()
        .loadFromSource(settings, XContentType.JSON))
        .source(mappings, XContentType.JSON)
    restHighLevelClient.indices().create(request, RequestOptions.DEFAULT)
    
}

Mapping.json Please note original has 16 fields.

{
  "properties": {
    "accessible": {
      "type": "boolean"
    },
    "person_id": {
      "type": "long"
    },
    "person_name": {
      "type": "string",
      "analyzer": "lower_keyword"
    }
}
}

Thanks.

I already commented that your answer unfortunately did not resolve the original issue. Hence, I did not accept. — newLearner
– newLearner, Commented Aug 5, 2020 at 8:40

Amit · Accepted Answer · 2020-07-29 09:58:40Z

1

Looks like you are using the dynamic mapping and due to some mistake when you index a document it ends up creating new fields in your index which crossed the 1000 fields limit.

Please see if you can use the static mapping or debug the code which prepares the document and compare it with your mapping to see if its creating new fields.

Please refer this SO answer to increase the limit if its legitimate or use static mapping or debug the code to figure out why you are adding new fields to elasticsearch index.

answered Jul 29, 2020 at 9:58

Amit

32.5k7 gold badges68 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

22 Comments

newLearner Over a year ago

Ok. So 2 questions. 1. Passing the person Object to the source() is correct ? That is not a problem ? 2. I have my mapping defined in a json file. And while creating an index, I am just reading that json file as a string passing it. Is that an issue ? Please check I have updated my code with the index creation part as well.

newLearner Over a year ago

Also, I am fairly new to this so not sure if this way of mapping is dynamic or static (looks static as I am just reading it from a file). Would really appreciate help. Struggling with this since yesterday.

Amit Over a year ago

@newLearner yeah by looking at your code, its a static mapping but in file you mentioned carmapping while you are passing person object? is this typo?

newLearner Over a year ago

Hey, I got the issue but not sure why this is happening. So when I am doing bulkRequest.add(IndexRequest(indexName).source(it,XContentType.JSON)), in the kibana I can see that my mapping is completely wrong.

{   "myindexneww" : {     "mappings" : {       "properties" : { and inside this properties instead of fields there are Person Objects with all the data inside them

Any idea what can be the issue for this to happen ?

newLearner Over a year ago

Basically the way I am trying to create index with mapping is wrong. The index is getting created but mappings are not present. { "adindex" : { "mappings" : { } } }

|

Collectives™ on Stack Overflow

Bulk Insert object in Elasticsearch

1 Answer 1

22 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

22 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related