Mon Aug 15, 2022 10:22 am
Login Register Lost Password? Contact Us

Please Note: The HPCC Systems forums are moving to Stack Overflow. We invite you to post your questions on Stack Overflow utilizing the tag hpcc-ecl (https://stackoverflow.com/search?tab=newest&q=hpcc-ecl). This legacy forum will be active and monitored during our transition to Stack Overflow but will become read only beginning September 1, 2022.



elastic4hpcclogs fields on Kibana question

Questions or comments related to Cloud Computing and the HPCC Systems Instant Cloud for AWS

Mon Jan 31, 2022 6:28 pm Change Time Zone

I have Deployed elastic4hpcclogs 1.2.0 on an Azure AKS cluster, and found that the hpcc.log.* fields created don't seem to be acknowledged as searchable/filterable by Kibana, is there any way to fix this?
g-pan
 
Posts: 6
Joined: Tue Apr 08, 2014 6:35 pm

Mon Jan 31, 2022 6:44 pm Change Time Zone

We've noticed this once but we're unable to recreate.
However, one possible solution is to explicitly provide field types for the generated hpcc.log.* fields in the hpccpipeline.

In Kibana, under Stack Management | Ingest Pipelines, find the pre-configured "hpccpipeline" and choose to edit it. Change the Grok to include field types.
In this example we declare all fields as "strings":
Code: Select all
{{[
{ "grok":
{ "field": "message", "patterns": [ "%\{BASE16NUM:hpcc.log.sequence:string}

\\s+%{HPCC_LOG_AUDIENCE:hpcc.log.audience:string}\\s+%{HPCC_LOG_CLASS:hpcc.log.class:string}\\s+%{TIMESTAMP_ISO8601:hpcc.log.timestamp:string}\\s+%{POSINT:hpcc.log.procid:string}\\s+%{POSINT:hpcc.log.threadid:string}\\s+%{HPCC_LOG_WUID:hpcc.log.jobid:string}
s+%{QUOTEDSTRING:hpcc.log.message:string}" ], "pattern_definitions":
{ "HPCC_LOG_WUID": "([A-Z][0-9]\{8}

-[0-9]{6})|(UNK)", "HPCC_LOG_CLASS": "DIS|ERR|WRN|INF|PRO|MET|UNK", "HPCC_LOG_AUDIENCE": "OPR|USR|PRG|AUD|UNK" }
}
}
]}}


Since the pipeline creates the hpcc.log fields on the target Elastic index(es), preexisting indexes will not update the field types. The field types will take affect on new indexes, pre-existing indexes will need to be removed.

If the issue continues, please let us know on the HPCC bug tracking system:
https://track.hpccsystems.com/browse/HPCC-27084
rodrigo.pastrana
 
Posts: 29
Joined: Tue Jun 10, 2014 2:19 pm


Return to Cloud

Who is online

Users browsing this forum: No registered users and 1 guest

cron