How do I filter query results using elasticsearch

Question

I am trying to create a component that provides typeahead autocomplete suggestions for users and groups. I'm using elasticsearch 6.5.3. I created an index that contains the fields I want searched and 3 additional fields to filter by (isGroup,isUser,organizationId). In some cases I want to use this component to search all users and group, sometimes just users or just groups or just users belonging to a specific organization. I was planning to supply a filter along with the search terms depending on what the specific use case was. I'm using nest to do the search, but I can't figure out how to do that. Is it possible to do this and if so how? Am I going down the wrong path for this? I mainly followed this guide to create analyzer and stuff. I can post my index if that will help but it's kind of long.

Here is a search with two of the items returned.

 return client.Search<UserGroupDocument>(s => s
            .Query(q=>q
                .QueryString(qs=>qs.Query("adm"))                    
            )
        );

    {
    "_index": "users_and_groups_autocomplete_index",
    "_type": "usergroupdocument",
    "_id": "c54956ab-c50e-481c-b093-f9855cc74480",
    "_score": 2.2962174,
    "_source": {
        "id": "c54956ab-c50e-481c-b093-f9855cc74480",
        "isUser": true,
        "isGroup": false,
        "name": "admin",
        "email": "[email protected]",
        "organizationId": 2
    }
},
    {
        "_index": "users_and_groups_autocomplete_index",
        "_type": "usergroupdocument",
        "_id": "80f98d24-39e3-475d-9cb6-8f16ca472525",
        "_score": 0.8630463,
        "_source": {
        "id": "80f98d24-39e3-475d-9cb6-8f16ca472525",
        "isUser": false,
        "isGroup": true,
        "name": "new Group",
        "users": [
            {
                "name": "Test User 1",
                "email": "[email protected]"
            },
            {
                "name": "admin",
                "email": "[email protected]"
            }
        ],
        "organizationId": 0
    }
},

So depending on where I use this component I may want all of these return, just the user, just the group, or just the user in organization 2.

Here is my UserGroupDocument class

public class UserGroupDocument
{
    public string Id { get; set; }
    public bool IsUser { get; set; }
    public bool IsGroup { get; set; }
    public string Name { get; set; }
    public string Email { get; set; }
    public List<User> Users { get; set; }
    public long OrganizationId { get; set; }

}

And the User class

public class User
{
    public string Name { get; set; }
    public string Email { get; set; }
}

Based on Russ Cam's answer below, I altered the Must statement as shown below. This gives me the filtering I wanted, but not the typeahead functionality. I still have to type and entire word before I start getting matches.

.Must(mu => mu
    .QueryString(mmp => mmp
        .Query(searchTerms)
        .Fields(f => f
            .Field(ff => ff.Name)
            .Field(ff => ff.Users.Suffix("name"))
        )
    )
)

Here is the index I'm using.

{
"users_and_groups_autocomplete_index": {
    "aliases": {},
    "mappings": {
        "usergroupdocument": {
            "properties": {
                "email": {
                    "type": "text",
                    "fields": {
                        "autocomplete": {
                            "type": "text",
                            "analyzer": "autocomplete"
                        }
                    }
                },
                "id": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "isGroup": {
                    "type": "boolean"   
                },
                "isUser": {
                    "type": "boolean"
                },
                "name": {
                    "type": "text",
                    "fields": {
                        "autocomplete": {
                            "type": "text",
                            "analyzer": "autocomplete"
                        }
                    }
                },
                "organizationId": {
                    "type": "long"
                },
                "users": {
                    "properties": {
                        "email": {
                            "type": "text",
                            "fields": {
                                "autocomplete": {
                                    "type": "text",
                                    "analyzer": "autocomplete"
                                }
                            }
                        },
                        "name": {
                            "type": "text",
                            "fields": {
                                "autocomplete": {
                                    "type": "text",
                                    "analyzer": "autocomplete"
                                }
                            }
                        }
                    }
                }
            }
        }
    },
    "settings": {
        "index": {
            "number_of_shards": "5",
            "provided_name": "users_and_groups_autocomplete_index",
            "creation_date": "1548363729311",
            "analysis": {
                "analyzer": {
                    "autocomplete": {
                        "filter": [
                            "lowercase"
                        ],
                        "type": "custom",
                        "tokenizer": "autocomplete"
                    }
                },
                "tokenizer": {
                    "autocomplete": {
                        "token_chars": [
                            "digit",
                            "letter"
                        ],
                        "min_gram": "1",
                        "type": "edge_ngram",
                        "max_gram": "20"
                    }
                }
            },
            "number_of_replicas": "1",
            "uuid": "Vxv-y58qQTG8Uh76Doi_dA",
            "version": {
                "created": "6050399"
            }
        }
    }
}
}

Please add query you tried, expected result and sample data you have. It will help to understand the problem. — Sathishkumar
– Sathishkumar, Commented Jan 24, 2019 at 14:38
Does that help or do you need more info? I'm happy to add anything, I just don't want to clutter up the question with a bunch of unhelpful json — scott
– scott, Commented Jan 24, 2019 at 14:54

Russ Cam · Accepted Answer · 2019-01-25 07:09:10Z

What you're looking for is a way to combine multiple queries together:

query on search terms
query on isGroup
query on isUser
query on organizationId

and execute a search with some combination of these. This is where a compound query like the bool query comes in. Given the following POCO

public class UserGroupDocument 
{
    public string Name { get; set; }
    public bool IsGroup { get; set; }
    public bool IsUser { get; set; }
    public string OrganizationId { get; set; }
}

We can start with

private static void Main()
{
    var defaultIndex = "default-index";
    var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); 
    var settings = new ConnectionSettings(pool)
        .DefaultIndex(defaultIndex);

    var client = new ElasticClient(settings);

    var isUser = true;
    var isGroup = true;
    var organizationId = "organizationId";

    var searchResponse = client.Search<UserGroupDocument>(x => x
        .Index(defaultIndex)
        .Query(q => q
            .Bool(b => b
                .Must(mu => mu
                    .QueryString(mmp => mmp
                        .Query("some admin")
                        .Fields(f => f
                            .Field(ff => ff.Name)
                        )
                    )
                )
                .Filter(fi => 
                    {
                        if (isUser)
                        {
                            return fi
                                .Term(f => f.IsUser, true);
                        }

                        return null;
                    }, fi =>
                    {
                        if (isGroup)
                        {
                            return fi
                                .Term(f => f.IsGroup, true);
                        }

                        return null;
                    }, fi => fi
                    .Term(f => f.OrganizationId, organizationId)
                )
            )
        )
    );
}

This will yield the following query

POST http://localhost:9200/default-index/usergroupdocument/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "isUser": {
              "value": true
            }
          }
        },
        {
          "term": {
            "isGroup": {
              "value": true
            }
          }
        },
        {
          "term": {
            "organizationId": {
              "value": "organizationId"
            }
          }
        }
      ],
      "must": [
        {
          "query_string": {
            "fields": [
              "name"
            ],
            "query": "some admin"
          }
        }
      ]
    }
  }
}

If

isUser is false, the term query filter on isUser field will not be included in the search query
isGroup is false, the term query filter on isGroup field will not be included in the search query
organizationId is null or an empty string, the term query filter on organizationId will not be included in the search query.

Now, we could go further and make isGroup and isUser nullable booleans (bool?). Then, when the value of either is null, the respective term query filter will not be included in the search query sent to Elasticsearch. This takes advantage of a feature known as conditionless queries in Nest, which is intended to make writing more complex queries easier. In addition, we can use operator overloading on queries to make it easier to write bool queries. This all means that we can refine the query down to

bool? isUser = true;
bool? isGroup = true;
var organizationId = "organizationId";

var searchResponse = client.Search<UserGroupDocument>(x => x
    .Index(defaultIndex)
    .Query(q => q
        .QueryString(mmp => mmp
            .Query("some admin")
            .Fields(f => f
                .Field(ff => ff.Name)
            )
        ) && +q
        .Term(f => f.IsUser, isUser) && +q
        .Term(f => f.IsGroup, isGroup) && +q
        .Term(f => f.OrganizationId, organizationId)
    )
);

The issues with this is that I don't get any results until I type an entire word. So I would expect to see the admin user show up as a result after typing "ad" but it currently doesn't show up until I type "admin"
this worked once I removed the fields from the querystring function
@scott how text is analysed at index and search time governs which terms are generated and compared, for full text search. That's a large topic in itself, but start with: elastic.co/guide/en/elasticsearch/reference/current/…
thanks for the answer @RussCam. How to add weights to different fields in your query?

Collectives™ on Stack Overflow

How do I filter query results using elasticsearch

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related