Kubelet监控指标说明

Probe路径

• 含义：kubelet以及kubelet监控的Pod的存活性
• 请求路径：“/metrics/probes”
• 指标

名称	类型	含义
up	Gauge	服务是否存活
prober_probe_total	Counter	按结果计算容器的活动探测、就绪探测或启动探测的累计数目以及存活性结果

Metrics路径

• 含义：kubelet自身指标统计
• 请求路径：“/metrics”
• 指标

名称	类型	含义
kubelet_running_pods	Gauge	kubelet上运行pod的数目
kubelet_started_containers_errors_total	Counter	启动容器时累计错误数

cadvisor路径

• 含义：容器和硬件级别监控
• 请求路径：“/metrics/cadvisor”

容器指标

名称	类型	含义	单位	组件	additional build flag
container_accelerator_duty_cycle	Gauge	Percent of time over the past sample period during which the accelerator was actively processing	percentage	accelerator
container_accelerator_memory_total_bytes	Gauge	Total accelerator memory	bytes	accelerator
container_accelerator_memory_used_bytes	Gauge	Total accelerator memory allocated	bytes	accelerator
container_blkio_device_usage_total	Counter	Blkio device bytes usage	bytes	diskIO
container_cpu_cfs_periods_total	Counter	Number of elapsed enforcement period intervals		cpu
container_cpu_cfs_throttled_periods_total	Counter	Number of throttled period intervals		cpu
container_cpu_cfs_throttled_seconds_total	Counter	Total time duration the container has been throttled	seconds	cpu
container_cpu_load_average_10s	Gauge	Value of container cpu load average over the last 10 seconds		cpuLoad
container_cpu_schedstat_run_periods_total	Counter	Number of times processes of the cgroup have run on the cpu		sched
container_cpu_schedstat_runqueue_seconds_total	Counter	Time duration processes of the container have been waiting on a runqueue	seconds	sched
container_cpu_schedstat_run_seconds_total	Counter	Time duration the processes of the container have run on the CPU	seconds	sched
container_cpu_system_seconds_total	Counter	Cumulative system cpu time consumed	seconds	cpu
container_cpu_usage_seconds_total	Counter	Cumulative cpu time consumed	seconds	cpu
container_cpu_user_seconds_total	Counter	Cumulative user cpu time consumed	seconds	cpu
container_file_descriptors	Gauge	Number of open file descriptors for the container		process
container_fs_inodes_free	Gauge	Number of available Inodes		disk
container_fs_inodes_total	Gauge	Total number of Inodes		disk
container_fs_io_current	Gauge	Number of I/Os currently in progress		diskIO
container_fs_io_time_seconds_total	Counter	Cumulative count of seconds spent doing I/Os	seconds	diskIO
container_fs_io_time_weighted_seconds_total	Counter	Cumulative weighted I/O time	seconds	diskIO
container_fs_limit_bytes	Gauge	Number of bytes that can be consumed by the container on this filesystem	bytes	disk
container_fs_reads_bytes_total	Counter	Cumulative count of bytes read	bytes	diskIO
container_fs_read_seconds_total	Counter	Cumulative count of seconds spent reading		diskIO
container_fs_reads_merged_total	Counter	Cumulative count of reads merged		diskIO
container_fs_reads_total	Counter	Cumulative count of reads completed		diskIO
container_fs_sector_reads_total	Counter	Cumulative count of sector reads completed		diskIO
container_fs_sector_writes_total	Counter	Cumulative count of sector writes completed		diskIO
container_fs_usage_bytes	Gauge	Number of bytes that are consumed by the container on this filesystem	bytes	disk
container_fs_writes_bytes_total	Counter	Cumulative count of bytes written	bytes	diskIO
container_fs_write_seconds_total	Counter	Cumulative count of seconds spent writing	seconds	diskIO
container_fs_writes_merged_total	Counter	Cumulative count of writes merged		diskIO
container_fs_writes_total	Counter	Cumulative count of writes completed		diskIO
container_hugetlb_failcnt	Counter	Number of hugepage usage hits limits		hugetlb
container_hugetlb_max_usage_bytes	Gauge	Maximum hugepage usages recorded	bytes	hugetlb
container_hugetlb_usage_bytes	Gauge	Current hugepage usage	bytes	hugetlb
container_last_seen	Gauge	Last time a container was seen by the exporter	timestamp	-
container_llc_occupancy_bytes	Gauge	Last level cache usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM).	bytes	resctrl
container_memory_bandwidth_bytes	Gauge	Total memory bandwidth usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM).	bytes	resctrl
container_memory_bandwidth_local_bytes	Gauge	Local memory bandwidth usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM).	bytes	resctrl
container_memory_cache	Gauge	Total page cache memory	bytes	memory
container_memory_failcnt	Counter	Number of memory usage hits limits		memory
container_memory_failures_total	Counter	Cumulative count of memory allocation failures		memory
container_memory_mapped_file	Gauge	Size of memory mapped files	bytes	memory
container_memory_max_usage_bytes	Gauge	Maximum memory usage recorded	bytes	memory
container_memory_migrate	Gauge	Memory migrate status		cpuset
container_memory_numa_pages	Gauge	Number of used pages per NUMA node		memory_numa
container_memory_rss	Gauge	Size of RSS	bytes	memory
container_memory_swap	Gauge	Container swap usage	bytes	memory
container_memory_usage_bytes	Gauge	Current memory usage, including all memory regardless of when it was accessed	bytes	memory
container_memory_working_set_bytes	Gauge	Current working set	bytes	memory
container_network_advance_tcp_stats_total	Gauge	advanced tcp connections statistic for container		advtcp
container_network_receive_bytes_total	Counter	Cumulative count of bytes received	bytes	network
container_network_receive_errors_total	Counter	Cumulative count of errors encountered while receiving		network
container_network_receive_packets_dropped_total	Counter	Cumulative count of packets dropped while receiving		network
container_network_receive_packets_total	Counter	Cumulative count of packets received		network
container_network_tcp6_usage_total	Gauge	tcp6 connection usage statistic for container		tcp
container_network_tcp_usage_total	Gauge	tcp connection usage statistic for container		tcp
container_network_transmit_bytes_total	Counter	Cumulative count of bytes transmitted	bytes	network
container_network_transmit_errors_total	Counter	Cumulative count of errors encountered while transmitting		network
container_network_transmit_packets_dropped_total	Counter	Cumulative count of packets dropped while transmitting		network
container_network_transmit_packets_total	Counter	Cumulative count of packets transmitted		network
container_network_udp6_usage_total	Gauge	udp6 connection usage statistic for container		udp
container_network_udp_usage_total	Gauge	udp connection usage statistic for container		udp
container_oom_events_total	Counter	Count of out of memory events observed for the container		oom_event
container_perf_events_scaling_ratio	Gauge	Scaling ratio for perf event counter (event can be identified by event label and cpu indicates the core for which event was measured). See perf event configuration.		perf_event	libpfm
container_perf_events_total	Counter	Scaled counter of perf core event (event can be identified by event label and cpu indicates the core for which event was measured). See perf event configuration.		perf_event	libpfm
container_perf_uncore_events_scaling_ratio	Gauge	Scaling ratio for perf uncore event counter (event can be identified by event label, pmu and socket lables indicate the PMU and the CPU socket for which event was measured). See perf event configuration. Metric exists only for main cgroup (id=“/”).		perf_event	libpfm
container_perf_uncore_events_total	Counter	Scaled counter of perf uncore event (event can be identified by event label, pmu and socket lables indicate the PMU and the CPU socket for which event was measured). See perf event configuration). Metric exists only for main cgroup (id=“/”).		perf_event	libpfm
container_processes	Gauge	Number of processes running inside the container		process
container_referenced_bytes	Gauge	Container referenced bytes during last measurements cycle based on Referenced field in /proc/smaps file, with /proc/PIDs/clear_refs set to 1 after defined number of cycles configured through referenced_reset_interval cAdvisor parameter.Warning: this is intrusive collection because can influence kernel page reclaim policy and add latency. Refer to https://github.com/brendangregg/wss#wsspl-referenced-page-flag for more details.	bytes	referenced_memory
container_sockets	Gauge	Number of open sockets for the container		process
container_spec_cpu_period	Gauge	CPU period of the container		-
container_spec_cpu_quota	Gauge	CPU quota of the container		-
container_spec_cpu_shares	Gauge	CPU share of the container		-
container_spec_memory_limit_bytes	Gauge	Memory limit for the container	bytes	-
container_spec_memory_reservation_limit_bytes	Gauge	Memory reservation limit for the container	bytes
container_spec_memory_swap_limit_bytes	Gauge	Memory swap limit for the container	bytes
container_start_time_seconds	Gauge	Start time of the container since unix epoch	seconds
container_tasks_state	Gauge	Number of tasks in given state (sleeping, running, stopped, uninterruptible, or ioawaiting)		cpuLoad
container_threads	Gauge	Number of threads running inside the container		process
container_threads_max	Gauge	Maximum number of threads allowed inside the container		process
container_ulimits_soft	Gauge	Soft ulimit values for the container root process. Unlimited if -1, except priority and nice		process

硬件指标

名称	类型	含义	单位	组件	additional build flag
machine_cpu_cache_capacity_bytes	Gauge	Cache size in bytes assigned to NUMA node and CPU core	bytes	cpu_topology
machine_cpu_cores	Gauge	Number of logical CPU cores
machine_cpu_physical_cores	Gauge	Number of physical CPU cores
machine_cpu_sockets	Gauge	Number of CPU sockets
machine_dimm_capacity_bytes	Gauge	Total RAM DIMM capacity (all types memory modules) value labeled by dimm type,
information is retrieved from sysfs edac per-DIMM API (/sys/devices/system/edac/mc/) introduced in kernel 3.6	bytes
machine_dimm_count	Gauge	Number of RAM DIMM (all types memory modules) value labeled by dimm type,
information is retrieved from sysfs edac per-DIMM API (/sys/devices/system/edac/mc/) introduced in kernel 3.6
machine_memory_bytes	Gauge	Amount of memory installed on the machine	bytes
machine_node_hugepages_count	Gauge	Numer of hugepages assigned to NUMA node		cpu_topology
machine_node_memory_capacity_bytes	Gauge	Amount of memory assigned to NUMA node	bytes	cpu_topology
machine_nvm_avg_power_budget_watts	Gauge	NVM power budget	watts		libipmctl
machine_nvm_capacity	Gauge	NVM capacity value labeled by NVM mode (memory mode or app direct mode)	bytes		libipmctl
machine_thread_siblings_count	Gauge	Number of CPU thread siblings		cpu_topology