Understanding ElasticSearch Pipeline Geopoint

Siren Platform/Investigate Tech Support Community
1

I used the following code to combine two columns into a geopoint however the end result is type unknown. When I go to investigate the columns some of the fields are “” empty and/or null. What is the expect behavior of this code when a lat and/or lon is null (null or “'”)?
Also, when I run the transform/ingest how does siren know to make this type geo_point?

{
 "description": "Create geo point field",
 "processors": [
   {
     "set": {
       "field": "geo_location",
       "value": {
           "lat": "{{latitude_field}}",
           "lon": "{{longitude_field}}"
       }
     }
   }
 ]
}
2

Hi Jeff,

If the mapping does not already exists and the first instance of ‘geo_location’ field that Elasticsearch encounters is a valid lat and lon, then ES will auto-map the field to geo_point. To ensure that in a spreadsheet with missing column values, skip setting ‘geo_location’ field where the columns have invalid data:

{
 "description": "Create geo point field, skipping null column values.",
 "processors": [
   {
     "set": {
       "if": "ctx.latitude_field != null && ctx.longitude_field != null",
       "field": "geo_location",
       "value": {
           "lat": "{{latitude_field}}",
           "lon": "{{longitude_field}}"
       }
     }
   }
 ]
}

This will skip the processor in case the value is not set, please keep this is mind: null is not equal to string literal ‘null’ which can be set in certain spreadsheets. The if condition can be modified as desired.

Regards
Manu

1 Like
3

Update. I tried the method above. It still left the geo_location Field of unknown Type. I tried it with setting the mapping first. That seemed to work but I had to refresh the fields on the data source. Not as intuitive as I’d like. It works now.

4

Hi Jeff,

If you set the field type to be geo_point in the mappings before you insert the data, it should work. In the following link, the very first PUT command specifies the mapping for it, as an example.

Regards,
Jeferson

5

Hi Jeferson, Thanks for the response. Can the mapping be added during time of of ingest in an ingest pipeline? If I add it after the ingest pipeline I get errors (i.e., already exists).

6

Hi Jeff,

The mappings are usually applied before the ingestion, as it defines the type of the field.

Regards,
Jeferson