Hi i am trying to create and see some dashboards in siren, but siren just found some of the fields of the index (105/3622). In addition, most of the fields it finds appear the following text: “Field not stored in documents” and cannot be used. I would like to upload other photo but i cant because im new user.
Thanks in advance.
Hi Adrian,
It seems the values in the fields are not constant means the value in XYZ field are not constant across the table like the format of the value.
Also do you see any error while ingesting the data when you did it at this page:
Regards
Manu Agarwal
the data was ingested with logstash to elasticsearh, and it seems to be constant, also i can’t ingest the data on that page because i ingest it from a .pcap
PD: i can see all data in kibana
Hi Adrian,
Can you share the mappings for the index?
Also is it possible for you to share the sample document?
Regards
Manu Agarwal
the documents size is 7.4GB so i can give you the page where i dowloaded:
https://www.unb.ca/cic/datasets/ids.html
Hi Adrian,
It will be worth if you can send me the single document from this as it will be more worth to test on single document rather than download all from it.
Also, Siren will also not ingest the fields if the field data is empty, and as elastiflow is another tool I can not assure whether the elastiflow template will work in Siren.
Regards
Manu
Hello Adrian22,
So, you have ingested some data into Elasticsearch using Logstash but not all fields seem to be detected in Data Model of Investigate whereas Kibana is showing them all.
Is Kibana and Investigate using the same Elasticsearch?
Can you:-
GET elasti*
i can’t attach files here so i only can give you this screenshot and tell you there are 145.000 lines of fields.
Hello @Adrian22 ,
the reason why in Investigate you do not see the full list of possible Elastiflow fields is because Elastiflow provides a pre-defined index patterns that is loaded during the setup (https://github.com/robcowart/elastiflow/blob/master/INSTALL.md#kibana-65x-and-later).
Without a pre-built index pattern, both Investigate and Kibana will be able to see only the fields that are effectively present in the mappings when the index pattern is created or refreshed, as returned by the field_caps API. This API cannot see the fields declared in dynamic templates if there are no documents that contain them.
In order to see all the possible Elastiflow fields in Investigate, it would be necessary to convert the files in https://github.com/robcowart/elastiflow/tree/master/kibana into a format suitable for Investigate, which is something we could consider in the future.
Meanwhile, you could still create visualizations from scratch on the fields that are available in your indices, but reusing the standard elastiflow patterns / dashboards / visualizations it won’t be possible without conversion.
