Investigate: "dashboard.groups not found"

Upgraded, now broke, investigate – thoughts?

It’d be nice to get this fixed … but it also might be a nice reason to start fresh, recreate all the configs, entities, etc. from scratch.

History:
Was on 7.9.3 (ES) and we just moved to 7.17.6
Updated federate plugin to newer/newest
Updated investigate to newest

  • did the sudo -u siren bin/investigate backup, then upgrade, then systemd start siren

Red bar across top, “dashboard.groups not found”
Twice now, trying to fix it:

  • close .siren index
  • restore last week (of 4) weekly snapshot’s .siren index
  • restarted siren
  • repeat this for the other 3 snapshots

additionally

  • upgrade output didn’t suggest any issue(s)
  • logs show migrations, etc and serving the site - just broken dashboards, etc.

Biggest killer is this – I can’t look at the previous config mgmt pane to recreate the configuration
!!

Versions

  • Federate 7.17.6-28.1 on ES* 7.17.6
  • so … ES is 7.17.6
  • Investigate 12.1.4

Hi Judge

Would it be possible to see Investigate logs from the upgrade operation?
Basically the terminal logs when you run

./bin/investigate upgrade 

That might help to see what migrations run and if there was any error/warning message there

If this do not help the next step would be to provide the content of the backup folder you’ve done before upgrading
Basically the content of the folder /here/path/backup_dir obtained by running the following command
before the upgrade

./bin/investigate backup --backup-dir=/here/path/backup_dir

Before sharing here remember that this folder might contain sensitive information about your setup as it contains the configurations of all objects like dashboards, scripts, searches etc

What might also help would be the Investigate version you were running before the upgrade

Cheers
Simon

Looking through the multiple times I ran upgrade (1 each for the 4 backups of .siren that I had), there seem to be 0 failures for any “migrations” tags – but nearly everyone was “0 objects to upgrade” was the text.

The text of 1 of those sequence of 49 upgrades is too large to paste here. But, they are just a repeated set of this example subset:

{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“status”,“plugin:migrations@12.0.0”,“info”],“pid”:17221,“state”:“yellow”,“message”:“Status changed from uninitialized to yellow - Checking for out of date objects.”,“prevState”:“uninitialized”,“prevMsg”:“uninitialized”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“------------------------------------------------------------ --------------------”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“Migration 1 out of 49: Upgrade Sentinl Watcher filters, remo ve type:phrase”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“0 objects to upgrade detected”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“0.01s execution time”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“------------------------------------------------------------ --------------------”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“Migration 2 out of 49: Change custom watchers’ reference to watcher templates from title to ID”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“0 objects to upgrade detected”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“0.01s execution time”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“------------------------------------------------------------ --------------------”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“Migration 3 out of 49: Move the lenses state from the dashbo ard object to the graph browser vizualization object.”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“0 objects to upgrade detected”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“0.06s execution time”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“------------------------------------------------------------ --------------------”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“Migration 4 out of 49: Upgrade Siren Graph Browser lenses co nfiguration with new EID ids”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“0 objects to upgrade detected”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“0.05s execution time”}
{“type”:“log”,“@timestamp”:“2022-10-18T15:25:12Z”,“tags”:[“info”,“migrations”],“pid”:17221,“message”:“------------------------------------------------------------ --------------------”}

in re: the backups
The old siren is saved in /usr/share/siren_old and I’d never looked at that backup directory. There are 4 files (2 acl and 2 config json files). A quick look at the files (jq -r . xxxx.json) shows ownership with an account that no longer exists.

How can I share these with you?

If you do not mind you can share the full logs to my email
simon at siren.io

Cheers
Simon

send … 20k for the second email w/ the backup files, sorry for size

no problem
Just last missing bit, what was the version of Investigate you used before the upgrade to 12.1.4 ?

Cheers
Simon

We came from 7.9.3 elasticsearch and Investigate 12.0.0

I’m sure we had (earlier) been on Investigate 11.0.0 (tar.gz file) then updated to 12.0.0 (deb file) months ago.

The data indices are still 6.8.xxx, needing reindexing … so we can jump ES and Kibana up to 8.4.whatever soon.

edit no stress on this. We’re starting to rebuild the configuration, entities, etc from scratch so we have a documented, purposefully created design vs. what we inherited from the previous, undocumented config :wink:

Hi Judge
Big thanks for provided data
We were able to reproduce the issue, (bug) and we are working on the fix which will be available in the upcoming 12.1.5 release

Cheers
Simon

Thanks - let me know if there is something I can do to leverage my backup.
e.g., manual transform or something so that it will work, etc.

cheers -

I am afraid that at this point there is no easy workaround
The good news is that we should have the new release in a week
Moving to it you should be able to open the saved objects and data model pages pages

Cheers
Simon

Ah good - thanks Simon. Cheers!

Hi Judge,

Siren 12.1.5 got released today which has a fix for this issue.

You can download it from here

Regards
Manu Agarwal