Elasticsearch：使用 ILM 示例运行降采样 (downsampling)

如果你对降采样还不是很熟的话，请阅读之前的文章 “Elasticsearch：对时间序列数据流进行降采样（downsampling)”。这是一个简化的示例，可让你快速了解降采样如何作为 ILM 策略的一部分来减少一组采样指标的存储大小。 该示例使用典型的 Kubernetes 集群监控数据。 要使用 ILM 测试降采样，请执行以下步骤：

  • 检查先决条件。
  • 创建索引生命周期策略。
  • 创建索引模板。
  • 摄取时间序列数据。
  • 查看结果。

以下展示是使用最新的 Elastic Stack 8.11 来进行的。

先决条件

  • 请参阅时间序列数据流先决条件。
    • 集群权限：manage_ilm 和 manage_index_templates。
    • 索引权限：你创建或转换的任何 TSDS 的 create_doc 和 create_index。 要滚动 TSDS，你必须具有 manage 权限。
  • 在运行此示例之前，你可能需要尝试手动运行降采样示例。

创建索引生命周期策略

为你的时间序列数据创建 ILM 策略。 虽然不是必需的，但建议使用 ILM 策略来自动管理时间序列数据流索引。

要启用降采样，请添加 Downsample 操作并将 fixed_interval 设置为要聚合原始时间序列数据的降采样间隔。

本例中为热阶段配置了 ILM 策略。 降采样发生在索引滚动并且索引时间序列结束时间（index.time_series.end_time）已过之后，因为在此之前源索引仍有望接收主要写入。 在索引结束时间过去之前，索引生命周期管理不会继续执行任何期望索引不再接收写入的操作。 在继续等待结束时间之前的索引生命周期管理操作包括： - 删除 (Delete) - 降采样 (Downsample) - 强制合并 (Force merge) - 只读 (Read only) - 可搜索快照 (Searchable snapshot) - 收缩 (Shrink)

PUT _ilm/policy/datastream_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_age": "5m"
          },
          "downsample": {
  	        "fixed_interval": "1h"
  	      }
        }
      }
    }
  }
}

创建索引模板

这将为基本数据流创建索引模板。 设置时间序列数据流中详细描述了索引模板的可用参数。

为了简单起见，在时间序列映射中，所有 time_series_metric 参数都设置为 gauge 类型，但也可以使 counter 指标类型。 time_series_metric 值确定降采样期间使用的统计表示的类型。

索引模板包含一组静态时间序列维度：主机 (host)、命名空间 (namespace)、节点 (node) 和 Pod。 时间序列维度不会因降采样过程而改变。

PUT _index_template/datastream_template
{
  "index_patterns": [
    "datastream*"
  ],
  "data_stream": {},
  "template": {
    "settings": {
      "index": {
        "mode": "time_series",
        "number_of_replicas": 0,
        "number_of_shards": 2
      },
      "index.lifecycle.name": "datastream_policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "kubernetes": {
          "properties": {
            "container": {
              "properties": {
                "cpu": {
                  "properties": {
                    "usage": {
                      "properties": {
                        "core": {
                          "properties": {
                            "ns": {
                              "type": "long"
                            }
                          }
                        },
                        "limit": {
                          "properties": {
                            "pct": {
                              "type": "float"
                            }
                          }
                        },
                        "nanocores": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        },
                        "node": {
                          "properties": {
                            "pct": {
                              "type": "float"
                            }
                          }
                        }
                      }
                    }
                  }
                },
                "memory": {
                  "properties": {
                    "available": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        }
                      }
                    },
                    "majorpagefaults": {
                      "type": "long"
                    },
                    "pagefaults": {
                      "type": "long",
                      "time_series_metric": "gauge"
                    },
                    "rss": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        }
                      }
                    },
                    "usage": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        },
                        "limit": {
                          "properties": {
                            "pct": {
                              "type": "float"
                            }
                          }
                        },
                        "node": {
                          "properties": {
                            "pct": {
                              "type": "float"
                            }
                          }
                        }
                      }
                    },
                    "workingset": {
                      "properties": {
                        "bytes": {
                          "type": "long",
                          "time_series_metric": "gauge"
                        }
                      }
                    }
                  }
                },
                "name": {
                  "type": "keyword"
                },
                "start_time": {
                  "type": "date"
                }
              }
            },
            "host": {
              "type": "keyword",
              "time_series_dimension": true
            },
            "namespace": {
              "type": "keyword",
              "time_series_dimension": true
            },
            "node": {
              "type": "keyword",
              "time_series_dimension": true
            },
            "pod": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        }
      }
    }
  }
}

摄取时间序列数据

使用 bulk API 请求自动创建 TSDS 并为一组 10 个文档编制索引。

重要提示：在运行此批量请求之前，你需要将时间戳更新为当前时间后三到五个小时内。 也就是说，搜索 2022-06-21T15 并替换为你当前的日期，并将小时调整为你当前的时间加三个小时。

PUT /datastream/_bulk?refresh
{"create": {}}
{"@timestamp":"2022-06-21T15:49:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":91153,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":463314616},"usage":{"bytes":307007078,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":585236},"rss":{"bytes":102728},"pagefaults":120901,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:45:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":124501,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":982546514},"usage":{"bytes":360035574,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1339884},"rss":{"bytes":381174},"pagefaults":178473,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":38907,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":862723768},"usage":{"bytes":379572388,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":431227},"rss":{"bytes":386580},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":86706,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":103266017,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1724908},"rss":{"bytes":105431},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:44:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":150069,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":639054643},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1786511},"rss":{"bytes":189235},"pagefaults":138172,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:42:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":82260,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":854735585},"usage":{"bytes":309798052,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":924058},"rss":{"bytes":110838},"pagefaults":259073,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:42:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":153404,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":279586406},"usage":{"bytes":214904955,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1047265},"rss":{"bytes":91914},"pagefaults":302252,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:40:20Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":125613,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":822782853},"usage":{"bytes":100475044,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2109932},"rss":{"bytes":278446},"pagefaults":74843,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:40:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":100046,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":362826547,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1986724},"rss":{"bytes":402801},"pagefaults":296495,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2022-06-21T15:38:30Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":40018,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":1062428344},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2294743},"rss":{"bytes":340623},"pagefaults":224530,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}

针对我的情况：

PUT /datastream/_bulk?refresh
{"create": {}}
{"@timestamp":"2023-11-30T06:49:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":91153,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":463314616},"usage":{"bytes":307007078,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":585236},"rss":{"bytes":102728},"pagefaults":120901,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:45:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":124501,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":982546514},"usage":{"bytes":360035574,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1339884},"rss":{"bytes":381174},"pagefaults":178473,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:50Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":38907,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":862723768},"usage":{"bytes":379572388,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":431227},"rss":{"bytes":386580},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":86706,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":103266017,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1724908},"rss":{"bytes":105431},"pagefaults":233166,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:44:00Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":150069,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":639054643},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1786511},"rss":{"bytes":189235},"pagefaults":138172,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:42:40Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":82260,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":854735585},"usage":{"bytes":309798052,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":924058},"rss":{"bytes":110838},"pagefaults":259073,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:42:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":153404,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":279586406},"usage":{"bytes":214904955,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1047265},"rss":{"bytes":91914},"pagefaults":302252,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:40:20Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":125613,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":822782853},"usage":{"bytes":100475044,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2109932},"rss":{"bytes":278446},"pagefaults":74843,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:40:10Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":100046,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":567160996},"usage":{"bytes":362826547,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":1986724},"rss":{"bytes":402801},"pagefaults":296495,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}
{"create": {}}
{"@timestamp":"2023-11-30T06:38:30Z","kubernetes":{"host":"gke-apps-0","node":"gke-apps-0-0","pod":"gke-apps-0-0-0","container":{"cpu":{"usage":{"nanocores":40018,"core":{"ns":12828317850},"node":{"pct":2.77905e-05},"limit":{"pct":2.77905e-05}}},"memory":{"available":{"bytes":1062428344},"usage":{"bytes":265142477,"node":{"pct":0.01770037710617187},"limit":{"pct":9.923134671484496e-05}},"workingset":{"bytes":2294743},"rss":{"bytes":340623},"pagefaults":224530,"majorpagefaults":0},"start_time":"2021-03-30T07:59:06Z","name":"container-name-44"},"namespace":"namespace26"}}

查看结果

现在你已创建文档并将其添加到数据流中，请检查以确认新索引的当前状态。

GET _data_stream

如果尚未应用 ILM 策略，你的结果将如下所示。 请注意原始索引名称：.ds-datastream-<timestamp>-000001。

{
  "data_streams": [
    {
      "name": "datastream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-datastream-2023.11.30-000001",
          "index_uuid": "muM9y4_ORDK1wLY-t_UtnA",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 1,
      "status": "GREEN",
      "template": "datastream_template",
      "ilm_policy": "datastream_policy",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T03:21:48.000Z",
            "end": "2023-11-30T07:27:32.000Z"
          }
        ]
      }
    },
    {
      "name": "kibana_sample_data_logs",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-kibana_sample_data_logs-2023.11.21-000001",
          "index_uuid": "JkbH5-8uTvyeZL6icuHN1A",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        }
      ],
      "generation": 1,
      "status": "YELLOW",
      "template": "kibana_sample_data_logs",
      "next_generation_managed_by": "Unmanaged",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false
    },
    {
      "name": "my-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-my-data-stream-2023.11.30-000001-downsample",
          "index_uuid": "KXtWAQCBRlW0sWYCVUu7Fw",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        },
        {
          "index_name": ".ds-my-data-stream-2023.11.30-000002",
          "index_uuid": "yBDLixmpRDahmS4G5_uxQw",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        }
      ],
      "generation": 4,
      "status": "GREEN",
      "template": "my-data-stream-template",
      "next_generation_managed_by": "Unmanaged",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T02:24:20.000Z",
            "end": "2023-11-30T08:32:32.000Z"
          }
        ]
      }
    }
  ]
}

接下来，运行搜索查询：

GET datastream/_search

经过一段时间后，它会变成如下的结果：

{
  "data_streams": [
    {
      "name": "datastream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": "downsample-1h-.ds-datastream-2023.11.30-000001",
          "index_uuid": "KQMxcaJATf24XAesWs6Xbw",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-datastream-2023.11.30-000002",
          "index_uuid": "yKLevDQkQuS-8yqJQb8gWw",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "status": "GREEN",
      "template": "datastream_template",
      "ilm_policy": "datastream_policy",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T03:21:48.000Z",
            "end": "2023-11-30T10:45:44.000Z"
          }
        ]
      }
    },
    {
      "name": "kibana_sample_data_logs",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-kibana_sample_data_logs-2023.11.21-000001",
          "index_uuid": "JkbH5-8uTvyeZL6icuHN1A",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        }
      ],
      "generation": 1,
      "status": "YELLOW",
      "template": "kibana_sample_data_logs",
      "next_generation_managed_by": "Unmanaged",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false
    },
    {
      "name": "my-data-stream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-my-data-stream-2023.11.30-000001-downsample",
          "index_uuid": "KXtWAQCBRlW0sWYCVUu7Fw",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        },
        {
          "index_name": ".ds-my-data-stream-2023.11.30-000002",
          "index_uuid": "yBDLixmpRDahmS4G5_uxQw",
          "prefer_ilm": true,
          "managed_by": "Unmanaged"
        }
      ],
      "generation": 4,
      "status": "GREEN",
      "template": "my-data-stream-template",
      "next_generation_managed_by": "Unmanaged",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T02:24:20.000Z",
            "end": "2023-11-30T10:45:44.000Z"
          }
        ]
      }
    }
  ]
}

默认情况下，索引生命周期管理每十分钟检查一次是否符合策略标准。 等待大约十分钟（也许可以冲泡一杯咖啡或茶☕），然后重新运行 GET _data_stream 请求。

GET _data_stream

ILM 策略生效后，原始 .ds-datastream-2023.11.30-000001 索引将替换为新的降采样索引，在本例中为 downsample-1h-.ds-datastream-2023.11.30-000001。

{
  "data_streams": [
    {
      "name": "datastream",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": "downsample-1h-.ds-datastream-2023.11.30-000001",
          "index_uuid": "KQMxcaJATf24XAesWs6Xbw",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        },
        {
          "index_name": ".ds-datastream-2023.11.30-000002",
          "index_uuid": "yKLevDQkQuS-8yqJQb8gWw",
          "prefer_ilm": true,
          "ilm_policy": "datastream_policy",
          "managed_by": "Index Lifecycle Management"
        }
      ],
      "generation": 3,
      "status": "GREEN",
      "template": "datastream_template",
      "ilm_policy": "datastream_policy",
      "next_generation_managed_by": "Index Lifecycle Management",
      "prefer_ilm": true,
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false,
      "time_series": {
        "temporal_ranges": [
          {
            "start": "2023-11-30T03:21:48.000Z",
            "end": "2023-11-30T10:45:44.000Z"
          }
        ]
      }
    },

在数据流上运行搜索查询（请注意，在查询降采样索引时，需要注意一些细微差别）。

GET datastream/_search

新的降采样索引仅包含一个文档，其中包括基于原始采样指标的 min、max、sum 和 value_count 统计信息。

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 4,
    "successful": 4,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "downsample-1h-.ds-datastream-2023.11.30-000001",
        "_id": "0eL0wC_4-45SnTNFAAABjB7QCwA",
        "_score": 1,
        "_source": {
          "@timestamp": "2023-11-30T06:00:00.000Z",
          "_doc_count": 10,
          "kubernetes": {
            "container": {
              "cpu": {
                "usage": {
                  "core": {
                    "ns": 12828317850
                  },
                  "limit": {
                    "pct": 0.0000277905
                  },
                  "nanocores": {
                    "min": 38907,
                    "max": 153404,
                    "sum": 992677,
                    "value_count": 10
                  },
                  "node": {
                    "pct": 0.0000277905
                  }
                }
              },
              "memory": {
                "available": {
                  "bytes": {
                    "min": 279586406,
                    "max": 1062428344,
                    "sum": 7101494721,
                    "value_count": 10
                  }
                },
                "majorpagefaults": 0,
                "pagefaults": {
                  "min": 74843,
                  "max": 302252,
                  "sum": 2061071,
                  "value_count": 10
                },
                "rss": {
                  "bytes": {
                    "min": 91914,
                    "max": 402801,
                    "sum": 2389770,
                    "value_count": 10
                  }
                },
                "usage": {
                  "bytes": {
                    "min": 100475044,
                    "max": 379572388,
                    "sum": 2668170609,
                    "value_count": 10
                  },
                  "limit": {
                    "pct": 0.00009923134
                  },
                  "node": {
                    "pct": 0.017700378
                  }
                },
                "workingset": {
                  "bytes": {
                    "min": 431227,
                    "max": 2294743,
                    "sum": 14230488,
                    "value_count": 10
                  }
                }
              },
              "name": "container-name-44",
              "start_time": "2021-03-30T07:59:06.000Z"
            },
            "host": "gke-apps-0",
            "namespace": "namespace26",
            "node": "gke-apps-0-0",
            "pod": "gke-apps-0-0-0"
          }
        }
      }
    ]
  }
}

使用数据流统计 API 获取数据流的统计信息，包括存储大小。

GET /_data_stream/datastream/_stats?human=true

此示例演示了降采样如何作为 ILM 策略的一部分来工作，以减少指标数据的存储大小，因为它变得不那么最新且查询频率较低。

你还可以尝试我们的手动运行降采样示例，了解降采样如何在 ILM 策略之外工作。

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：/a/205756.html

如若内容造成侵权/违法违规/事实不符，请联系我们进行投诉反馈qq邮箱809451989@qq.com，一经查实，立即删除！

相关文章

如何充分了解客户需求

如何充分了解客户需求

如何充分了解客户需求 如何充分了解客户需求&#xff0c;以提供贴心服务&#xff1f; 想要提供超出客户期望的优质服务&#xff0c;首先需要了解他们的需求。通过多种方式收集客户反馈、深入挖掘数据、建立紧密的客户关系&#xff0c;我们可以更好地理解客户需求&#xff0c;…
阅读更多...
Flutter使用flutter_gen管理资源文件

Flutter使用flutter_gen管理资源文件

pub地址&#xff1a; https://pub.dev/packages/flutter_gen 1.添加依赖 在你的pubspec.yaml文件中添加flutter_gen作为开发依赖 dependencies:build_runner:flutter_gen_runner: 2.配置pubspec.yaml 在pubspec.yaml文件中&#xff0c;配置flutter_gen的参数。指定输出路…
阅读更多...
6.1 Windows驱动开发：内核枚举SSDT表基址

6.1 Windows驱动开发：内核枚举SSDT表基址

SSDT表&#xff08;System Service Descriptor Table&#xff09;是Windows操作系统内核中的关键组成部分&#xff0c;负责存储系统服务调用的相关信息。具体而言&#xff0c;SSDT表包含了系统调用的函数地址以及其他与系统服务相关的信息。每个系统调用对应SSDT表中的一个表项…
阅读更多...
Netty Review - 探索Channel和Pipeline的内部机制

Netty Review - 探索Channel和Pipeline的内部机制

文章目录 概念Channel Pipeline实现原理分析详解 Inbound事件和Outbound事件演示Code 概念 Netty中的Channel和Pipeline是其核心概念&#xff0c;它们在构建高性能网络应用程序时起着重要作用。 Channel&#xff1a; 在Netty中&#xff0c;Channel表示一个开放的连接&#xff…
阅读更多...
Spring整合web环境

Spring整合web环境

目录 Javaweb三大组件及环境特点 Spring整合web环境的思路及实现 Spring的web开发组件spring-web MVC框架思想及其设计思路 Javaweb三大组件及环境特点 Spring整合web环境的思路及实现 package com.xfy.listener;import com.xfy.config.SpringConfig; import org.springfra…
阅读更多...
Scrapy框架中间件(一篇文章齐全)

Scrapy框架中间件(一篇文章齐全)

1、Scrapy框架初识&#xff08;点击前往查阅&#xff09; 2、Scrapy框架持久化存储&#xff08;点击前往查阅&#xff09; 3、Scrapy框架内置管道&#xff08;点击前往查阅&#xff09; 4、Scrapy框架中间件 Scrapy 是一个开源的、基于Python的爬虫框架&#xff0c;它提供了…
阅读更多...
业务连续性的实施步骤及价值

业务连续性的实施步骤及价值

业务连续性计划不仅仅是面对灾难时的应急措施&#xff0c;更是一种战略性的投资。通过综合的风险评估、计划制定和实施&#xff0c;企业可以提高对各种风险的适应能力&#xff0c;确保业务在任何情况下都能够持续运营。这种全面的准备不仅有助于应对突发事件&#xff0c;还能为…
阅读更多...
【社会网络分析第6期】Ucient实操

【社会网络分析第6期】Ucient实操

一、导入数据处理二、核心——边缘分析三、聚类分析四、网络密度 一、导入数据处理 将数据导入Ucinet首先需要对数据进行处理。 承接上一期的数据格式&#xff1a;【社会网络分析第5期】gephi使用指南 原先得到的数据格式如下&#xff1a; 接下来打开ucinet&#xff1a; 之后…
阅读更多...
latex中算法的几种模板

latex中算法的几种模板

latex中算法的几种模板_latex算法模板-CSDN博客文章浏览阅读6.2k次&#xff0c;点赞3次&#xff0c;收藏45次。latex中几种算法模板_latex算法模板https://blog.csdn.net/weixin_50514171/article/details/125136121?spm1001.2014.3001.5506 latex排版原理 常用算法排版伪代码…
阅读更多...
「Linux」使用C语言制作简易Shell

「Linux」使用C语言制作简易Shell

&#x1f4bb;文章目录 &#x1f4c4;前言简易shell实现shell的概念系统环境变量shell的结构定义内建命令完整代码 &#x1f4d3;总结 &#x1f4c4;前言 对于很多学习后端的同学来讲&#xff0c;学习了C语言&#xff0c;发现除了能写出那个经典的“hello world”以外&#xff…
阅读更多...
43.0BaseDao抽取dao公共父类

43.0BaseDao抽取dao公共父类

43.1. 回顾 1. 把数据库表中查询的结果封装到一个实体类中。 命名规则:类名和表名一致 类中属性和表的字段对应。 表中的一条记录对应实体的一个对象 多条记录→集合 43.2. 正文 目录 43.1. 回顾 43.2. 正文 43.3. 抽取dao公共父类。 43.4. 引入数据源 43.3. 抽取dao公共…
阅读更多...
文件夹重命名技巧：用关键词替换文件夹名称指定内容的右侧文字

文件夹重命名技巧：用关键词替换文件夹名称指定内容的右侧文字

在日常生活中&#xff0c;经常要管理大量的文件夹&#xff0c;这时候掌握一些文件夹重命名的技巧就非常实用。例如文件夹重命名时&#xff0c;经常要将一些通用的文字替换成其他关键词&#xff0c;以便更好地标识和分类文件夹。而用关键词替换文件夹名称指定内容的右侧文字&…
阅读更多...
【论文阅读】1 SkyChain：一个深度强化学习的动态区块链分片系统

【论文阅读】1 SkyChain：一个深度强化学习的动态区块链分片系统

SkyChain 一、文献简介二、引言及重要信息2.1 研究背景2.2 研究目的和意义2.3 文献的创新点 三、研究内容3.1模型3.2自适应分类账协议3.2.1状态块创建3.2.2合并过程3.2.3拆分过程 3.3评价框架3.3.1性能3.3.1.1共识延迟3.3.1.2重新分片延迟3.3.1.3处理事务数3.3.1.4 约束 3.3.2 …
阅读更多...
使用RobotFramework编写BDD代码

使用RobotFramework编写BDD代码

背景 行为驱动开发&#xff08;Behavior Driven Development&#xff09;即BDD&#xff0c;是一种敏捷开发方法&#xff0c;通常应用在自动化测试中&#xff0c;或者也可称为行为驱动测试。通过使用自然描述语言确定自动化脚本&#xff0c;通过这种方式&#xff0c;能够大大促…
阅读更多...
EI级 | Matlab实现TCN-BiLSTM-Multihead-Attention多头注意力机制多变量时间序列预测

EI级 | Matlab实现TCN-BiLSTM-Multihead-Attention多头注意力机制多变量时间序列预测

EI级 | Matlab实现TCN-BiLSTM-Multihead-Attention多头注意力机制多变量时间序列预测 目录 EI级 | Matlab实现TCN-BiLSTM-Multihead-Attention多头注意力机制多变量时间序列预测预测效果基本介绍程序设计参考资料 预测效果 基本介绍 1.【EI级】Matlab实现TCN-BiLSTM-Multihead-…
阅读更多...
Openai通用特定领域的智能语音小助手

Openai通用特定领域的智能语音小助手

无穷尽的Q&A 钉钉...钉钉... 双双同学刚到工位,报销答疑群的消息就万马纷沓而来。她只能咧嘴无奈的摇摇头。水都还没有喝一口就开始“人工智能”的去回复。原本很阳光心情开始蒙上一层薄薄阴影。在这无休无止的Q&A中&#xff0c;就算你对工作有磐石一般强硬&#xff0…
阅读更多...
XXL-Job详解（一）：组件架构

XXL-Job详解（一）：组件架构

目录 XXL-Job特性系统组成架构图调度模块剖析任务 “运行模式” 剖析执行器 XXL-Job XXL-JOB是一个分布式任务调度平台&#xff0c;其核心设计目标是开发迅速、学习简单、轻量级、易扩展。现已开放源代码并接入多家公司线上产品线&#xff0c;开箱即用。 特性 1、简单&#…
阅读更多...
探究Kafka原理-6.CAP理论实践

探究Kafka原理-6.CAP理论实践

&#x1f44f;作者简介&#xff1a;大家好&#xff0c;我是爱吃芝士的土豆倪&#xff0c;24届校招生Java选手&#xff0c;很高兴认识大家&#x1f4d5;系列专栏&#xff1a;Spring源码、JUC源码、Kafka原理&#x1f525;如果感觉博主的文章还不错的话&#xff0c;请&#x1f44…
阅读更多...
解决keil右键Go To Definition跳转不过去的问题

解决keil右键Go To Definition跳转不过去的问题

解决&#xff1a; 在魔法棒中如图所示打上√
阅读更多...
Pycharm新手开发指南

Pycharm新手开发指南

文章目录 前言一、常用功能介绍二、常用高效pycharm使用方法关于Python技术储备一、Python所有方向的学习路线二、Python基础学习视频三、精品Python学习书籍四、Python工具包项目源码合集①Python工具包②Python实战案例③Python小游戏源码五、面试资料六、Python兼职渠道 前言…
阅读更多...
最新文章

Copyright @ 2022~2023