How to bulk insert Json using NEST elasticsearch?

Question

I'am trying to insert multiple records into my database using Nest. Inserting using IndexMany class does work however I also need to insert objects by json string.

I did look on github, and found some examples how to use the RAWclient. Below a code example I insert my json.

    > var twitter = _jsonData;          
    > var result = client.Raw.BulkPost(
    >               new { twitter }
    >               , qs => qs
    >                   //.Replication(ReplicationOptions.Async)
    >                   .Refresh(true)          );

some additional info:

jsondata:

tweet tweet1 = new tweet { id = "104", name = "test104", lastname = "test107" }; //ect....
List<tweet> data; //multiple tweet objects are added
string json = Newtonsoft.Json.JsonConvert.SerializeObject(data);

var twitter:

{
      "twitter": "[{'name':'test104','lastname':'test107','id':'104'},{'name':'test105','lastname':'test108','id':'105'},{'name':'test106','lastname':'test109','id':'106'}]"
}

result i receive from the database:

{"error":"Unexpected end-of-input: expected close marker for OBJECT (from [Source: [B@10893e4; line: 1, column: 0])\n at [Source: [B@10893e4; line: 2, column: 3]"}

Does anyone know what the issue might be? or what I'am missing in my json/code snipped?

v.shashenko · Accepted Answer · 2015-11-12 10:55:17Z

5

Your json is not correct for elasticsearch bulk operation. See the documentation.

In a bulk request every data object should be preceded by a command because a single bulk request can contain inserts, updates or deletes, not just inserts. So your json should look like

  { "index" : { "_index" : "twitter", "_type" : "tweets" } }\n
  {'name':'test104','lastname':'test107','id':'104'}\n
  { "index" : { "_index" : "twitter", "_type" : "tweets" } }\n
  {'name':'test105','lastname':'test108','id':'105'}\n
  { "index" : { "_index" : "twitter", "_type" : "tweets" } }\n
  {'name':'test106','lastname':'test109','id':'106'}\n

To reduce overhead from repetitive commands you can move some arguments to the request uri. Then the json can be shorter:

  { "index" : { } }\n
  {'name':'test104','lastname':'test107','id':'104'}\n

In IRawElasticClient that means moving them to BulkPost arguments.

  var result = client.Raw.BulkPost(new { twitter }, "twitter", "tweets");

edited Nov 12, 2015 at 10:55

answered Feb 25, 2014 at 15:09

v.shashenko

7367 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Greezer Over a year ago

thanks, your last edit did work. Seems I missed the doc on the json format for bulking.

Greezer Over a year ago

It seems like a lot of overhead in a bulk, is there a another way of bulking? with only 1 operator for example, the data will be the same format anyway.

v.shashenko Over a year ago

I don't think there's another way of bulking. But it's possible to use shorter commands. See the edit.

Calle Over a year ago

Now a year later, seems that client.Raw.BulkPost() is unavailable in NEST. How would one make the bulk call with the current version?

Paul Lambert Over a year ago

It is now client.LowLevel.Bulk().

|

Collectives™ on Stack Overflow

How to bulk insert Json using NEST elasticsearch?

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related