Siren Automatic Relational filter: unable to allocate buffer

Siren Platform/Investigate Tech Support Community
1

Hey guys,

I’m trying to enable the Relational Navigator inside a Dashboard that is linked to a big index (200 millions of documents).
But I get the following message:

Siren Automatic Relational filter: { "root_cause": [ { "type": "c", "reason": "m: unable to allocate buffer of size 2097152. Current allocation: root(786432000/786432000), job(786432000/786432000), task(2097152/786432000)." }, { "type": "c", "reason": "m: unable to allocate buffer of size 4194304. Current allocation: root(784334848/786432000), job(784334848/786432000), task(2097152/786432000)." }, { "type": "c", "reason": "m: unable to allocate buffer of size 8388608. Current allocation: root(786432000/786432000), job(786432000/786432000), task(18874368/786432000)." }, { "type": "c", "reason": "m: unable to allocate buffer of size 4194304. Current allocation: root(786432000/786432000), job(786432000/786432000), task(4194304/786432000)." }, { "type": "c", "reason": "m: unable to allocate buffer of size 4194304. Current allocation: root(784334848/786432000), job(784334848/786432000), task(8388608/786432000)." } ], "type": "i", "reason": "Failures present in task node response", "suppressed": [ { "type": "c", "reason": "m: unable to allocate buffer of size 2097152. Current allocation: root(786432000/786432000), job(786432000/786432000), task(2097152/786432000).", "caused_by": { "type": "out_of_memory_exception", "reason": "Unable to allocate buffer of size 2097152 due to memory limit. Current allocation: 2097152" } }, { "type": "c", "reason": "m: unable to allocate buffer of size 4194304. Current allocation: root(784334848/786432000), job(784334848/786432000), task(2097152/786432000).", "caused_by": { "type": "out_of_memory_exception", "reason": "Unable to allocate buffer of size 4194304 due to memory limit. Current allocation: 2097152" } }, { "type": "c", "reason": "m: unable to allocate buffer of size 8388608. Current allocation: root(786432000/786432000), job(786432000/786432000), task(18874368/786432000).", "caused_by": { "type": "out_of_memory_exception", "reason": "Unable to allocate buffer of size 8388608 due to memory limit. Current allocation: 18874368" } }, { "type": "c", "reason": "m: unable to allocate buffer of size 4194304. Current allocation: root(786432000/786432000), job(786432000/786432000), task(4194304/786432000).", "caused_by": { "type": "out_of_memory_exception", "reason": "Unable to allocate buffer of size 4194304 due to memory limit. Current allocation: 4194304" } }, { "type": "c", "reason": "m: unable to allocate buffer of size 4194304. Current allocation: root(784334848/786432000), job(784334848/786432000), task(8388608/786432000).", "caused_by": { "type": "out_of_memory_exception", "reason": "Unable to allocate buffer of size 4194304 due to memory limit. Current allocation: 8388608" } } ] }

I already have increased the size of total heap space to 30gb.

Is it about this setting? https://docs.support.siren.io/10.3.2/platform/en/siren-federate/federate-modules/memory.html

I cant find the config file where I should change it.

Thanks!

2

Figured out that the number in my error is the siren.memory.root.limit …
I dont know how to fix it… Its not using the free memory …

3

Hi,

Yes this is about the setting described in https://docs.support.siren.io/10.3.2/platform/en/siren-federate/federate-modules/memory.html.
This is to configure the maximum limit of “off-heap” memory allocated. Federate is using off-heap memory instead of heap memory for in-memory computation.
The error message is telling you that your query is generating more data than what it is allowed, which is normal if you are joining an index with 200M documents. The default memory limit is quite conservative and works for small indices. For large indices you’ll have to increase it. I would suggest increasing it by steps. You should first try to set a limit of 2gb and see if it is enough.
You can use the cluster memory stats api (GET /_siren/nodes/stats/memory) to retrieve some stats about off-heap memory allocation using. This is described in https://docs.support.siren.io/10.3.2/platform/en/siren-federate/cluster-apis.html.

1 Like
4

Thank u! 7gb was enough.