Is it possible to run Siren ER to find hidden relationships between documents that are already reflected in your elasticsearch?

I dont want to insert new data, just want to reanalyze my indexs with Siren ER.
Is it possible?

Hi,

This should be possible using the Elastic reindex API to run the ER pipeline instead of at ingest time, and then use a drop processor to ensure no new index is actually created.

  1. Take the content of ingest2SirenWithER.json and add a drop processor to the end of the list of processors:
    ...
              ctx.remove('er_id');
              ctx.remove('er_dataSource');
              ctx.remove('er_output');
              '''
          }
        },
        {
          "drop": {}
        }
      ]
    }
  1. In DevTools, create a named pipeline definition:
PUT _ingest/pipeline/siren-to-er
{
  "processors": [
    {
      "script": {
	  ...
  1. Perform the reindexing using the new pipeline:
POST _reindex
{
  "source": { 
    "index": "myindex"
  },
  "dest": {
    "index": "myindex-processed",
    "pipeline": "siren-to-er"
  }
}

For a large index, DevTools may timeout before the _reindex is complete and you may see:

 Failed to connect to Console's backend.
 Please check the Investigate server is up and running

In that case you can do the _reindex outside DevTools in a curl command.

Please let us know if it works.

1 Like

We are going to support this use case explicitly (job to process data that is already indexdx ) in some of the new version likely 10.5 or 11