filter empty array fields in elasticsearch

Question

My document structure is something like:

{
    title: string,
    description: string,
    privacy_mode: string,
    hidden: boolean,
    added_by: string,
    topics: array
}

I am trying to query elasticsearch. However I dont want any document with empty topics array field.

Below is a function which builds the query object:

function getQueryObject(data) {
    var orList = [{ "term": {"privacy_mode": "public", "hidden": false} }]
    if (data.user) {
        orList.push({ "term": {"added_by": data.user} });
    }

    var queryObj = {
        "fields": ["title", "topics", "added_by", "img_url", "url", "type"],
        "query": {
            "filtered" : {
                "query" : {
                    "multi_match" : {
                        "query" : data.query + '*',
                        "fields" : ["title^4", "topics", "description^3", "tags^2", "body^2", "keywords",
                                "entities", "_id"]
                    }
                },
                "filter" : {
                    "or": orList
                },
                "filter" : {
                    "limit" : {"value" : 15}
                },
                "filter": {
                   "script": {
                        "script": "doc['topics'].values.length > 0"
                   }
               }
            }
        }
    }
    return queryObj;
};

This still gives me elements with empty topics array. wondering whats wrong!

Thank for the help

Qy Zuo · Accepted Answer · 2017-11-01 07:27:43Z

16

The keyword missing is remove since ES5.0 and it suggests using exists(see here):

curl -XGET 'localhost:9200/_search?pretty' -H 'Content-Type: 
application/json' -d'
{
    "query": {
        "bool": {
            "must_not": {
                   "exists": {
                       "field": "topics"
                   }
            }
        }
    }
}'

answered Nov 1, 2017 at 7:27

Qy Zuo

2,75228 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Janek Bogucki Over a year ago

"must" "exists"?

Alex Brasetvik · Accepted Answer · 2013-12-27 16:51:22Z

15

You probably want the missing-filter. Your script approach will load all the values of topics into memory, which will be very wasteful if you are not also e.g. faceting on them.

Also, the structure of your filter is wrong. You cannot have repeated values for filter, but should wrap them with a bool-filter. (Here is why you usually want to use bool and not and|or|not: http://www.elasticsearch.org/blog/all-about-elasticsearch-filter-bitsets/

Lastly, you probably want to specify the size on the search object, instead of using the limit-filter.

I made a runnable example you can play with: https://www.found.no/play/gist/aa59b987269a24feb763

#!/bin/bash

export ELASTICSEARCH_ENDPOINT="http://localhost:9200"

# Index documents
curl -XPOST "$ELASTICSEARCH_ENDPOINT/_bulk?refresh=true" -d '
{"index":{"_index":"play","_type":"type"}}
{"privacy_mode":"public","topics":["foo","bar"]}
{"index":{"_index":"play","_type":"type"}}
{"privacy_mode":"private","topics":[]}
'

# Do searches

curl -XPOST "$ELASTICSEARCH_ENDPOINT/_search?pretty" -d '
{
    "query": {
        "filtered": {
            "filter": {
                "bool": {
                    "must": [
                        {
                            "term": {
                                "privacy_mode": "public"
                            }
                        }
                    ],
                    "must_not": [
                        {
                            "missing": {
                                "field": "topics"
                            }
                        }
                    ]
                }
            }
        }
    }
}
'

answered Dec 27, 2013 at 16:51

Alex Brasetvik

11.8k2 gold badges40 silver badges38 bronze badges

7 Comments

Chirag Jain Over a year ago

Thanks alex. I think you are right in most part of it. I will make the changes accordingly. As for missing filter, I dont know if it will work. 'topics: []' is present the document. its empty not missing.

Chirag Jain Over a year ago

Hi alex, Missing worked very well. However using bool will break my logic. What i am trying to do is select document which are either public or added by the logged in user. Hence i used or filter. What do you think about this case?

Alex Brasetvik Over a year ago

bool's should corresponds to or. Note that you can nest bools as well.

Alex Brasetvik Over a year ago

@FelikZ, see this answer regarding nested and bools: stackoverflow.com/questions/21691882/…

ZedTuX Over a year ago

The missing-filter is dead. Here is the one for ES 2.4: elastic.co/guide/en/elasticsearch/reference/2.4/…. This link says that you have now to use the Missing Query which is deprecated since ES 2.2.0 and now you have to use the Exists Query

|

Collectives™ on Stack Overflow

filter empty array fields in elasticsearch

2 Answers 2

1 Comment

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related