Mapping an array field to a C# string list with Elasticsearch

Question

As shown in the picture I have a groups field with an array of string values. However, I am getting an exception when trying to map it to a List<string> property.

Something like Error converting value \"[134706634,134706635]\" to type System.Collections.Generic.IList[System.String]'

I tried using some various attributes that elasticsearch provides but none worked. A json converter attribute works fine but need to write a lot of code to make it work the way I want.

Is there a cleaner and more native way to do this with NEST ?

C# Code:

 var groupQuery = await
     elastic.SearchAsync<CorrelationGroup>(s => s
             .Query(q => q
             .Bool(b => b
             .Must(
                  m => m.ConstantScore(c => c
                       .Filter(f => f
                       .Term(x => x.Type, counterType))
                   ),
                  m => m.ConstantScore(c => c.
                        Filter(f => f
                       .Term(x => x.TypeValue, counterTypeValue))))))
             .Index("correlation-groups").AllTypes());

 public class CorrelationGroup
 {
     [Text(Name = "type")]
     public string Type { get; set; }

     [Text(Name = "type_value")]
     public string TypeValue { get; set; }

     public List<string> Groups { get; set; }
 }

Source json file:

[ { "type": "APN", "type_value": "internet", "groups": [150994936,150994940] }, { "type": "APN", "type_value": "internet", "groups": [150984921,150984922] }, { "type": "APN", "type_value": "internet", "groups": [150984917,150984918,150984921,150984922] } ]

My template is:

{
    "template": "correlation-groups",
    "settings" : {
        "number_of_shards" : 2,
        "number_of_replicas" : 0,
        "index" : {
            "store" : { "compress" : { "stored" : true, "tv": true } }
        }
    },
    "dynamic_templates": [
    {
        "string_template" : { 
            "match" : "*",
            "mapping": { "type": "string", "index": "not_analyzed" },
            "match_mapping_type" : "string"
         } 
     }
    ],
    "mappings": {
        "_default_": {
            "properties": {
                "type": { "type": "string", "index": "not_analyzed" },
                "type_value": { "type": "string", "index": "not_analyzed" },
                "groups": { "type": "string"}
            }
         }
     }
 }

It seems your array is actually a string. See the escaped doubles quotes around the array \"[134706634,134706635]\" ? It's very likely that your groups array has been stringified at some point and stored as a string. — Val
– Val, Commented Aug 10, 2016 at 14:24
Would be great to see how do you create mapping for index and how do you index data. — Rob
– Rob, Commented Aug 10, 2016 at 14:25
@Val I specified type string in the template (thats what I understood from docs) should I use something else instead ? — Cemre Mengü
– Cemre Mengü, Commented Aug 10, 2016 at 14:25
That's correct, however, the issue seem to lie upstream in the code that indexes your document. Very likely that your client code is stringifying that array somehow and neither sending a list nor an array. — Val
– Val, Commented Aug 10, 2016 at 14:26

Russ Cam · Accepted Answer · 2016-08-11 00:59:00Z

2

The issue is related to your template; Within the __default__ mapping, because you have specified groups to be of type string, the incoming property is being stringified and persisted in Elasticsearch as a string. The Logstash codec will be correctly sending the groups property from your source json as a number array but will be persisted as a string because of the default mapping you have.

To rectify, change the __default__ mapping to

"mappings": {
    "_default_": {
        "properties": {
            "type": { "type": "string", "index": "not_analyzed" },
            "type_value": { "type": "string", "index": "not_analyzed" },
            "groups": { "type": "integer" }
        }
     }
 }

If you only have one type within the index, you may want to explicitly define a mapping for that type instead.

Once you have done this, change your C# POCO to

public class CorrelationGroup
{
    [Keyword]
    public string Type { get; set; }

    [Keyword(Name = "type_value")]
    public string TypeValue { get; set; }

    public List<int> Groups { get; set; }
}

If you're using Elasticsearch 5.x, you should use the keyword type that maps to the older not_analyzed string type. Additionally, you only need to specify the name for TypeValue as it uses snake casing (NEST client understands camel case naming coming from Elasticsearch and will map to pascal case POCO property names). Finally, I've changed the Groups property to be an int list.

answered Aug 11, 2016 at 0:59

Russ Cam

126k34 gold badges211 silver badges275 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Cemre Mengü Over a year ago

Thanks so much for your detailed answer. I tried your suggestion but now logstash is complaining that it cannot convert string to int.

Russ Cam Over a year ago

You'll need to delete that index and create it again (I'm guessing this is OK for you to do at this point). Logstash should be sending through the correct json; can you also output to stdout to check?

Cemre Mengü Over a year ago

Alright so by looking at the stdout seems like it works. One thing that I had to do was make the json file single line. However, I dont see the groups column in GUI (head plugin). Any idea why that is ?

Cemre Mengü Over a year ago

Alright code works as well. I think this head plugin also deceived me that it wasnt working. It does not show my groups column but it is there :) Cheers for the help.

Russ Cam Over a year ago

haven't looked at HEAD plugin for a little while. I'd recommend using curl or Kibana's Sense to check: elastic.co/guide/en/sense/current/installing.html

Collectives™ on Stack Overflow

Mapping an array field to a C# string list with Elasticsearch

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related