Msearch + prefrence issue

Siren Federate (Elasticsearch Plugin)
1

I have encountered an issue with this query. (version: 38.5)

I have 40 shards and I want to run two queries like this (this is an example): the first query fetches data, and the second query counts documents using the same filter.

The problem is that it returns different results if I run it multiple times. Without Siren, it works fine.

POST siren/person/_msearch

{"index":["foo"],"preference":"test-key"}

{"query":{"bool":{"must":[{"constant_score":{"boost":0.0,"filter":{"bool":{"must":[{"query_string":{"query":"saved:true"}}],"must_not":[{"query_string":{"query":"cached:true"}}]}}}}]}},"from":0,"min_score":0.0,"size":25,"sort":[{"_score":{"order":"desc"}}],"_source":{"includes":["id"]},"track_total_hits":false}

{"index":["foo"],"preference":"test-key"}

{"query":{"bool":{"must":[{"constant_score":{"boost":0.0,"filter":{"bool":{"must":[{"query_string":{"query":"saved.premium:true"}}],"must_not":[{"query_string":{"query":"cached:true"}}]}}}}]}},"min_score":0.0,"size":1,"sort":[{"_score":{"order":"desc"}}],"_source":{"includes":["id"]},"track_total_hits":true}
2

Hello Sayad,

  • What version of Elasticsearch are you running ?
  • Do you use any Federate-specific cluster setting ?
  • Does this happen only in case the preference is used ?

The problem is that it returns different results if I run it multiple times

  • Is it the 1st query fetching data that returns different results, or that the count of the 2nd query changes ?
  • Do the results you get back match the filter you set ?
  • Do you get a different set of documents on each run ?
3

  1. Elasticsearch version is 8.19.82. Siren Federate settings are used (no specific custom settings).

  2. Configuration:

{
  "io": {
    "pipeline": {
      "batch_size": "65536",
      "hash": { "partitions_per_node": "19" },
      "max_packet_size": "1048576",
      "number_of_nodes": "-1"
    }
  },
  "consolidator": {
    "max_merge_size": "50",
    "max_cache_size": "10000"
  },
  "memory": {
    "job": { "limit": "16G" },
    "root": { "limit": "16G" },
    "task": { "limit": "16G" }
  }
}

  1. Without a preference, I get different results in almost every 3–5 requests. When I used preference with normal queries (without Siren), the results were stable. I tried the same with Siren, but it seems that the preference is somehow ignored.

  2. The total count does not change; only the order of the results changes in the first query.

  3. Yes, the documents are the same (only the order changes). I believe this is due to shard routing, which is why I am using preference.

  4. In every 5 requests, I get about 1–2 different result orders.

For example, when I run the same query 10 times, about 3 of the results are identical, and the other 7 are also identical to each other, so I basically have only 2 variations of the data, not more

4

Hello Sayad,

If you order by id instead of _score you are getting also different results ?

Cheers

5

hi Issac

not actually, but as you know it’s slow and I don’t want to use it

6

Hi Sayad,
Yes true, sorry, I just want to understand the issue, so you are not getting different results, but only the order is different, am I right ? So always the same 25 hits ?

I see you are ordering by _score and at the same time using the constant score query. There is a way you use another field to sort and check if you see the same behaviour ?

Btw, the preference parameter should makes the request to use the same nodes and shards, and I’m not sure it should impact the order of the hits this request, as does not contain any join, it is simply proxied to ES.

Another question, you are always using the same coordinator node ? or do you have a load balancer in front ?

Cheers
Issac

7

Another last question,
If the same query is executed in a single search ie /siren/_search, you see the same behaviour with the preference parameter ?

8

No, in a single query, because I send the preference parameter in the URL, it is not ignored:
POST siren/person/_search?request_cache=false&preference=test-key

Note: As far as I know, this happens because of the bouncing results issue in Elasticsearch, so I had to use preference. I think that in _msearch it is ignored.
https://www.elastic.co/docs/solutions/search/full-text/search-relevance/consistent-scoring

9

i even tried "search_type":"dfs_query_then_fetch" but made no difference

10

hi again,

@Issac_Garcia any update on this?

11

Hi Sayad,

We were not able to reproduce it with a query similar as yours without a join,
Are you able to provide us a way to reproduce it ?

If the query does not contains a join, as said before, the request is bypassing Federate and executed directly by ES.

A test we can do to validate the preference is well handled is to do the search by passing a single shard like: preference=_shards:0

In this case, it should route the request to a single shard, and as the sort is done only on documents of a single shard, the order should be consistent

In any case the ES documentation in the link you provided says:

This ensures that all queries of a given user are always going to hit the same shards, so scores remain more consistent across queries.

As far as I understand, “more consistent” means less variation, and not necessary deterministic.
Order can vary in your case because:

  • segments changes in the same shard
  • merging shard results to build final result

Also, as there is no tie breaker and all docs has a constant score of zero, the order will be determined by internal doc id for every shard (which can change if there are some segment changes).

After the shards are completed, they are merged again to build the top N final results, but here again, there is no tie breaker neither, so all will depends on the order of the shards responses, which is not deterministic (ie the order on which every shard finish can vary from run to run).

So, having a constant score of zero is not helping to have deterministic result order.
Can you also test if you remove the constant score, and left ES calculate the score based on stats & the query, also if you use the preference + the dfs_query_then_fetch maybe can help to have a “more consistent” result order.

Regards,
Issac

12

Hi Issac,

Thanks for your detailed explanation.

  1. To reproduce the issue, the index should typically have more than 7–8 shards and contain a large volume of data (like tens of millions of documents).

  2. I tested with preference=_shards:0. The issue still exists. In any case, I do not want to route requests to a single shard only.

  3. Regarding “more consistent”: since there are no updates on the index, segments are not changing at all. Therefore, the data should be deterministic.

  4. About the constant score being zero and sorting by internal doc ID: I already bypassed that logic on my side. Again, since there are no updates and the preference routes to the same shards, the results should always be identical.

  5. I also tested without constant score (letting ES compute the score normally). The results are still random.

  6. The key point is that everything works correctly with _msearch directly in Elasticsearch, but when running the same query through Siren, the order becomes inconsistent.

So the behavior seems specific to Siren rather than Elasticsearch itself.

13

hi @Issac_Garcia I tested it more carefully and you are right, in the sample I sent before, there is no issue, and I apologize for that (:face_with_peeking_eye: :melting_face:) and for any inconvenience caused by the wrong query.
The issue appears when I have a join.
If you look at the following query, the third part includes a join, but the first two items don’t.
Without the third query, it works fine; as soon as I add the third one, it returns random data

POST siren/_msearch

{"index":["index1"],"preference":"test","search_type":"dfs_query_then_fetch"}

{"query":{"bool":{"must":[{"query_string":{"query":"saved:true"}}],"must_not":[{"query_string":{"query":"cached:true"}}]}},"from":0,"min_score":0.0,"size":25,"sort":[{"_score":{"order":"desc"}}],"_source":{"includes":["sid"]},"track_total_hits":false}

{"index":["index1"],"preference":"test","search_type":"dfs_query_then_fetch"}

{"query":{"bool":{"must":[{"query_string":{"query":"saved:true"}}],"must_not":[{"query_string":{"query":"cached:true"}}]}},"from":0,"min_score":0.0,"size":0,"sort":[{"_score":{"order":"desc"}}],"_source":{"includes":["sid"]},"track_total_hits":true}

{"index":["index1"],"preference":"test","search_type":"dfs_query_then_fetch"}

{"query":{"bool":{"must":[{"join":{"indices":["index2"],"on":["id","id"],"request":{"query":{"bool":{"must":[{"term":{"workspace_id":{"value":"84ec7d14-407b-46ed-8286-c213bc2c39d6","case_insensitive":true}}}]}}}}},{"query_string":{"query":"saved:true"}}],"must_not":[{"query_string":{"query":"cached:true"}}]}},"min_score":0.0,"size":0,"sort":[{"_score":{"order":"desc"}}],"_source":{"includes":["id"]},"track_total_hits":true}
14

Hi @Sayad_Aazami
Thanks for the clarification. We’ll investigate with this new information and come to you once we have more information.
Regards,
Issac

1 Like
15

Hello @Sayad_Aazami ,

We have found the bug, we will ship the fix in the next release.

Thanks!

16

hi again dear @Stephane_Campinas thank you for the response and great support
I appreciate it and look forward to the next release.

17

Hello,

The fix is now available in the 39.1 release of Siren Federate.

2 Likes
18

hi @Stephane_Campinas
thanks, It helped me a lot.
I hope your team continues to shine :folded_hands: :folded_hands:

1 Like