Spring Data Elasticsearch query by JSON structure

Question

I'm using spring data elasticsearch, and it is a lot easier to associate the code to the actual JSON elasticsearch query when I use the @Query annotation, as in the examples in this linked reference:

https://www.programcreek.com/java-api-examples/index.php?api=org.springframework.data.elasticsearch.annotations.Query

I was wondering if there is a way to make a query by the full JSON body via the elasticsearch java library without an annotation. I.E. within a method implementation or something. This will help me parse highlighting in the response, etc..

Thanks for any information.

Clarification from comments: I'm using spring-data-elasticsearch 3.0.10.RELEASE with Elasticsearch 6. As spring-data-elasticsearch does not seem to support the RestHighLevelClient yet, I'm using the TransportClient client = new PreBuiltTransportClient(elasticsearchSettings); approach when creating the ElasticsearchTemplate: return new ElasticsearchTemplate(client());

There are multiple ways you can access Elasticsearch from Java, but one way is to use a standard HTTP client with the Jest library. It also depends on what version of Elasticsearch you're referring to. github.com/searchbox-io/Jest — tom
– tom, Commented Sep 17, 2018 at 0:44
@tom I've updated the question to include technology versions & clarification. A better answer would show how to do it with the client & included libs that spring-data-elasticsearch has, but any examples with other libs like Jest would be welcome too! — TheJeff
– TheJeff, Commented Sep 17, 2018 at 17:32

tom · Accepted Answer · 2018-09-18 00:15:27Z

I figured out one way to do it but it requires you to make a script that lives on the Elastic node. See File-based scripts. It's not extremely flexible but give it a shot. Here's what to do.

Create a file named template_doctype.mustache and copy it to $ELASTIC_HOME/config/scripts. This is the script you could tailor as needed. Restart Elastic or wait 60 seconds for it to reload.

{
    "query" : {
        "match" : {
            "type" : "{{param_type}}"
        }
    }
}

My pom.xml dependencies:

<dependencies>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-data-elasticsearch</artifactId>
        <version>3.0.10.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>transport</artifactId>
        <version>5.5.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
        <version>2.8.2</version>
        <scope>compile</scope>
    </dependency>
</dependencies>

(FYI, I discovered using mvn dependency:tree that your version of spring-data-elasticsearch implicitly uses the 5.5 version of the ElasticSearch library, even though you're using ElasticSearch 6.)

Create a dummy index:

curl -X PUT http://localhost:9200/myindex

Create a couple of documents that can be used to match to ensure the code works:

curl -X POST http://localhost:9200/myindex/mydoc -d '{"title":"foobar", "type":"book"}'
curl -X POST http://localhost:9200/myindex/mydoc -d '{"title":"fun", "type":"magazine"}'

Try running a query. This code should return a single document:

String clusterName = "my-application";
Settings elasticsearchSettings = Settings.builder().put("cluster.name", clusterName).build();
TransportClient client = new PreBuiltTransportClient(elasticsearchSettings)
        .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("localhost"),9300));
Map<String, Object> template_params = new HashMap<>();

// Here is where you put parameters to your script.
template_params.put("param_type", "book");
SearchResponse sr = new SearchTemplateRequestBuilder(client)
        .setScript("template_doctype")  // this is where you specify what template to use
        .setScriptType(ScriptType.FILE)
        .setScriptParams(template_params)
        .setRequest(new SearchRequest())
        .get()
        .getResponse();

SearchHit[] results = sr.getHits().getHits();
for(SearchHit hit : results){

    String sourceAsString = hit.getSourceAsString();
    if (sourceAsString != null) {
        Gson gson = new GsonBuilder().setPrettyPrinting()
                .create();
        Map map = gson.fromJson(sourceAsString, Map.class);
        System.out.println( gson.toJson(map));
    }
}

Output:

{
  "title": "foobar",
  "type": "book"
}

Wow. This is actually really cool. I wouldn't be sure how to deploy that to a cluster for release builds. I found better docs for this here: elastic.co/guide/en/elasticsearch/client/java-api/5.0/…, and it appears I can set the script inline too, or set a script template from code using their examples.
Oh cool. I wonder if for that, you'd need special privileges on the cluster since that is executing an admin API to store the script. Might not be a problem for you.

tom · Accepted Answer · 2018-09-18 22:58:37Z

1

Here's another way to do it, but not using the transport client.

Add these dependencies to your pom.xml:

<dependency>
    <groupId>com.sun.jersey</groupId>
    <artifactId>jersey-client</artifactId>
    <version>1.19</version>
    <scope>compile</scope>
</dependency>

Then do this:

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;

Client client = new Client();
final WebResource r = client.resource("http://localhost:9200").path("/myindex/_search");
String requestJson = "{\"query\" : {\"match\" : {\"type\" : \"book\"} }}";
ClientResponse response = r.post(ClientResponse.class, requestJson);
String json = response.getEntity(String.class);

Gson gson = new GsonBuilder().setPrettyPrinting()
        .create();
Map map = gson.fromJson(json, Map.class);
System.out.println(gson.toJson(map));

// to convert to SearchResponse:
JsonXContentParser xContentParser = new JsonXContentParser(NamedXContentRegistry.EMPTY,
            new JsonFactory().createParser(json));
SearchResponse searchResponse = SearchResponse.fromXContent(xContentParser);

Example output:

{
    "took": 9.0,
    "timed_out": false,
    "_shards": {
        "total": 5.0,
        "successful": 5.0,
        "failed": 0.0
    },
    "hits": {
        "total": 1.0,
        "max_score": 0.2876821,
        "hits": [
        {
            "_index": "myindex",
            "_type": "mydoc",
            "_id": "AWXp8gZjXyu6lA_2Kpi2",
            "_score": 0.2876821,
            "_source": {
                "title": "foobar",
                "type": "book"
            }
        }
        ]
    }
}

edited Sep 18, 2018 at 22:58

answered Sep 18, 2018 at 0:31

tom

1,5031 gold badge18 silver badges31 bronze badges

3 Comments

TheJeff Over a year ago

Neat solution! Thanks for posting. Seems that I probably can't use ElasticSearch's SearchResponse entity with the response? One thing that might work with this is serializing the response into an entity somehow...

tom Over a year ago

I just added an example of how to convert the response string into a SearchResponse.

tom Over a year ago

If you like either of these answers please mark them as such, thanks!

Collectives™ on Stack Overflow

Spring Data Elasticsearch query by JSON structure

2 Answers 2

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related