Update a string parameter in Elasticsearch _mapping

Question

I have such a _mapping in Elasticsearch 6.8:

{
  "grch38_test__wes__grch38__variants__20210222" : {
    "mappings" : {
      "variant" : {
        "_meta" : {
          "gencodeVersion" : "25",
          "hail_version" : "0.2.20",
          "genomeVersion" : "38",
          "sampleType" : "WES",
          "sourceFilePath" : "s3://my_folder/my_vcf.vcf"
        },
    ...

My goal is to issue a query in Kibana to modify variant._meta.sourceFilePath. Following thread:

Elastic search mapping for nested json objects

I was able to come up with the query:

PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant
{
  "properties": {
    "variant": {
      "type": "nested",
      "properties": {
        "_meta": {
          "type": "nested",
          "properties": {
            "type": "text",
            "sourceFilePath": "s3://my_folder/my_vcf.vcf"
          }
        }
      }
    }
  }
}

But its giving me an error:

elasticsearch mapping Expected map for property [fields] on field [name] but got a class java.lang.String

Full error message:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Expected map for property [fields] on field [type] but got a class java.lang.String"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Expected map for property [fields] on field [type] but got a class java.lang.String"
  },
  "status": 400
}

I have also tried:

PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant
{
  "properties": {
    "variant": {
      "type": "nested",
      "properties": {
        "_meta": {
          "type": "nested",
          "properties": {
            "sourceFilePath": {
              "type": "text",
              "value":"s3://my_folder/my_vcf.vcf"
            }
          }
        }
      }
    }
  }
}

But its telling me that value is unsupported:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "Mapping definition for [sourceFilePath] has unsupported parameters:  [value : s3://seqr-dp-data--prod/vcf/dev/grch38_test_contracted.vcf]"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "Mapping definition for [sourceFilePath] has unsupported parameters:  [value : s3://seqr-dp-data--prod/vcf/dev/grch38_test_contracted.vcf]"
  },
  "status": 400
}

What am I doing wrong? How to modify the field?

Jozef - Spatialized.io · Accepted Answer · 2021-02-24 14:24:26Z

_meta is a reserved field for storing application-specific metadata. It's not meant to be searchable and can be only retrieved through the GET Mapping API.

This means that if your _meta content was intended to be consistent with what the _meta field is designed for, you cannot apply any mappings to it. It's a "final" hashmap of concrete values and would need to be defined at the top level of your update-mapping payload:

PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant
{
  "_meta": {
    "variant": {            <-- shared index-level metadata
      "gencodeVersion": "25",
      "hail_version": "0.2.20",
      "genomeVersion": "38",
      "sampleType": "WES",
      "sourceFilePath": "s3://my_folder/my_vcf.vcf"
    }
  },
  "properties": {
    "some_text_field": {    <-- actual document properties
      "type": "text" 
    }
  }
}

If, on the other hand, your _meta field is an unfortunate naming coincidence, you can declare the mappings for it like so:

PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant
{
  "properties": {
    "_meta": {
      "properties": {
        "variant": {
          "properties": {
            "gencodeVersion": {
              "type": "text"
            },
            "genomeVersion": {
              "type": "text"
            },
            "hail_version": {
              "type": "text"
            },
            "sampleType": {
              "type": "text"
            },
            "sourceFilePath": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}

and ingest documents of the form:

POST grch38_test__wes__grch38__variants__20210222/variant/_doc
{
  "_meta": {
    "variant": {
      "gencodeVersion": "25",
      "hail_version": "0.2.20",
      "genomeVersion": "38",
      "sampleType": "WES",
      "sourceFilePath": "s3://my_folder/my_vcf.vcf"
    }
  }
}

But again, the _meta content would be document-specific, not index-wide!

And BTW, the nested mapping only makes sense if you're dealing with arrays of objects, not objects of objects. But if you insist on wanting it, here's how you'd do it:

PUT /grch38_test__wes__grch38__variants__20210222/_mapping/variant?include_type_name
{
  "properties": {
    "_meta": {
      "type": "nested",            <---
      "properties": {
        "variant": {
          "type": "nested",        <---
          "properties": {
            "gencodeVersion": {
              "type": "text"
            },
            "genomeVersion": {
              "type": "text"
            },
            "hail_version": {
              "type": "text"
            },
            "sampleType": {
              "type": "text"
            },
            "sourceFilePath": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}

So, does it mean (first case, _meta corresponds to what it should be) that sourceFilePath was defined at the moment of index creation and it is simply not allowed to be modified and there is no way to do that?
No -- you can modify the shared _meta attributes with my 1st snippet -- just make sure to remove the properties part.

Collectives™ on Stack Overflow

Update a string parameter in Elasticsearch _mapping

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related