【已解决】Splunk 8.2.X 升级ES 后红色报警

1： 背景：

  由于splunk ES 占有很大的computing resource, 所以，Splunk ES 升级到7.1.1 后，有红色的alert.

2: 解决方法：

  降低iowait 的 threshold:

Investigation

The default threshold setting for IOWait is pre-set to a low value and may not be relevant to the specific environment/performance.  For new installations or upgrades, the IOWait threshold will need to be increased to accommodate the server environment.

Splunk's IOWait Calculations:

Every 10 seconds, this feature takes 3 measurements from the introspection log:

avg_cpu__max_perc_last_3m:

index=_introspection source=*resource_usage.log* component=IOStats data.cpu_pct=* | timechart max("data.cpu_pct") by host span=1m

single_cpu__max_perc_last_3m:

index=_introspection source=*resource_usage.log* component=IOSt