当前位置: 首页 > news >正文

flink源码分析-获取JVM最大堆内存

flink版本: flink-1.11.2

代码位置: org.apache.flink.runtime.util.EnvironmentInformation#getMaxJvmHeapMemory

如果设置了-Xmx参数,就返回这个参数,如果没设置就返回机器物理内存的1/4.  这里主要看各个机器内存的获取方法。

	/*** The maximum JVM heap size, in bytes.** <p>This method uses the <i>-Xmx</i> value of the JVM, if set. If not set, it returns (as* a heuristic) 1/4th of the physical memory size.** @return The maximum JVM heap size, in bytes.*/public static long getMaxJvmHeapMemory() {final long maxMemory = Runtime.getRuntime().maxMemory();if(maxMemory != Long.MAX_VALUE) {// we have the proper max memoryreturn maxMemory;} else {// max JVM heap size is not set - use the heuristic to use 1/4th of the physical memoryfinal long physicalMemory = Hardware.getSizeOfPhysicalMemory();if(physicalMemory != -1) {// got proper value for physical memoryreturn physicalMemory / 4;} else {throw new RuntimeException("Could not determine the amount of free memory.\n" + "Please set the maximum memory for the JVM, e.g. -Xmx512M for 512 megabytes.");}}}

进入getSizeOfPhysicalMemory()方法,里面有获取各种操作系统物理内存的方法:

/** Licensed to the Apache Software Foundation (ASF) under one* or more contributor license agreements.  See the NOTICE file* distributed with this work for additional information* regarding copyright ownership.  The ASF licenses this file* to you under the Apache License, Version 2.0 (the* "License"); you may not use this file except in compliance* with the License.  You may obtain a copy of the License at**     http://www.apache.org/licenses/LICENSE-2.0** Unless required by applicable law or agreed to in writing, software* distributed under the License is distributed on an "AS IS" BASIS,* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.* See the License for the specific language governing permissions and* limitations under the License.*/package org.apache.flink.runtime.util;import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.lang.management.ManagementFactory;
import java.lang.management.OperatingSystemMXBean;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.regex.Matcher;
import java.util.regex.Pattern;import org.apache.flink.util.OperatingSystem;import org.slf4j.Logger;
import org.slf4j.LoggerFactory;/*** Convenience class to extract hardware specifics of the computer executing the running JVM.*/
public class Hardware {private static final Logger LOG = LoggerFactory.getLogger(Hardware.class);private static final String LINUX_MEMORY_INFO_PATH = "/proc/meminfo";private static final Pattern LINUX_MEMORY_REGEX = Pattern.compile("^MemTotal:\\s*(\\d+)\\s+kB$");// ------------------------------------------------------------------------/*** Gets the number of CPU cores (hardware contexts) that the JVM has access to.* * @return The number of CPU cores.*/public static int getNumberCPUCores() {// TODO_MA 注释: 获取 Cpu Coresreturn Runtime.getRuntime().availableProcessors();}/*** Returns the size of the physical memory in bytes.* * @return the size of the physical memory in bytes or {@code -1}, if*         the size could not be determined.*/public static long getSizeOfPhysicalMemory() {// first try if the JVM can directly tell us what the system memory is// this works only on Oracle JVMstry {Class<?> clazz = Class.forName("com.sun.management.OperatingSystemMXBean");Method method = clazz.getMethod("getTotalPhysicalMemorySize");OperatingSystemMXBean operatingSystemMXBean = ManagementFactory.getOperatingSystemMXBean();// someone may install different beans, so we need to check whether the bean// is in fact the sun management beanif (clazz.isInstance(operatingSystemMXBean)) {return (Long) method.invoke(operatingSystemMXBean);}}catch (ClassNotFoundException e) {// this happens on non-Oracle JVMs, do nothing and use the alternative code paths}catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {LOG.warn("Access to physical memory size: " +"com.sun.management.OperatingSystemMXBean incompatibly changed.", e);}// we now try the OS specific access pathsswitch (OperatingSystem.getCurrentOperatingSystem()) {case LINUX:return getSizeOfPhysicalMemoryForLinux();case WINDOWS:return getSizeOfPhysicalMemoryForWindows();case MAC_OS:return getSizeOfPhysicalMemoryForMac();case FREE_BSD:return getSizeOfPhysicalMemoryForFreeBSD();case UNKNOWN:LOG.error("Cannot determine size of physical memory for unknown operating system");return -1;default:LOG.error("Unrecognized OS: " + OperatingSystem.getCurrentOperatingSystem());return -1;}}/*** Returns the size of the physical memory in bytes on a Linux-based* operating system.* * @return the size of the physical memory in bytes or {@code -1}, if*         the size could not be determined*/private static long getSizeOfPhysicalMemoryForLinux() {try (BufferedReader lineReader = new BufferedReader(new FileReader(LINUX_MEMORY_INFO_PATH))) {String line;while ((line = lineReader.readLine()) != null) {Matcher matcher = LINUX_MEMORY_REGEX.matcher(line);if (matcher.matches()) {String totalMemory = matcher.group(1);return Long.parseLong(totalMemory) * 1024L; // Convert from kilobyte to byte}}// expected line did not comeLOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo'). " +"Unexpected format.");return -1;}catch (NumberFormatException e) {LOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo'). " +"Unexpected format.");return -1;}catch (Throwable t) {LOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo') ", t);return -1;}}/*** Returns the size of the physical memory in bytes on a Mac OS-based* operating system* * @return the size of the physical memory in bytes or {@code -1}, if*         the size could not be determined*/private static long getSizeOfPhysicalMemoryForMac() {BufferedReader bi = null;try {Process proc = Runtime.getRuntime().exec("sysctl hw.memsize");bi = new BufferedReader(new InputStreamReader(proc.getInputStream()));String line;while ((line = bi.readLine()) != null) {if (line.startsWith("hw.memsize")) {long memsize = Long.parseLong(line.split(":")[1].trim());bi.close();proc.destroy();return memsize;}}} catch (Throwable t) {LOG.error("Cannot determine physical memory of machine for MacOS host", t);return -1;} finally {if (bi != null) {try {bi.close();} catch (IOException ignored) {}}}return -1;}/*** Returns the size of the physical memory in bytes on FreeBSD.* * @return the size of the physical memory in bytes or {@code -1}, if*         the size could not be determined*/private static long getSizeOfPhysicalMemoryForFreeBSD() {BufferedReader bi = null;try {Process proc = Runtime.getRuntime().exec("sysctl hw.physmem");bi = new BufferedReader(new InputStreamReader(proc.getInputStream()));String line;while ((line = bi.readLine()) != null) {if (line.startsWith("hw.physmem")) {long memsize = Long.parseLong(line.split(":")[1].trim());bi.close();proc.destroy();return memsize;}}LOG.error("Cannot determine the size of the physical memory for FreeBSD host " +"(using 'sysctl hw.physmem').");return -1;}catch (Throwable t) {LOG.error("Cannot determine the size of the physical memory for FreeBSD host " +"(using 'sysctl hw.physmem')", t);return -1;}finally {if (bi != null) {try {bi.close();} catch (IOException ignored) {}}}}/*** Returns the size of the physical memory in bytes on Windows.* * @return the size of the physical memory in bytes or {@code -1}, if*         the size could not be determined*/private static long getSizeOfPhysicalMemoryForWindows() {BufferedReader bi = null;try {Process proc = Runtime.getRuntime().exec("wmic memorychip get capacity");bi = new BufferedReader(new InputStreamReader(proc.getInputStream()));String line = bi.readLine();if (line == null) {return -1L;}if (!line.startsWith("Capacity")) {return -1L;}long sizeOfPhyiscalMemory = 0L;while ((line = bi.readLine()) != null) {if (line.isEmpty()) {continue;}line = line.replaceAll(" ", "");sizeOfPhyiscalMemory += Long.parseLong(line);}return sizeOfPhyiscalMemory;}catch (Throwable t) {LOG.error("Cannot determine the size of the physical memory for Windows host " +"(using 'wmic memorychip')", t);return -1L;}finally {if (bi != null) {try {bi.close();} catch (Throwable ignored) {}}}}// --------------------------------------------------------------------------------------------private Hardware() {}
}

另外注意try catch的这种用法:

		try (BufferedReader lineReader = new BufferedReader(new FileReader(LINUX_MEMORY_INFO_PATH))) {String line;while ((line = lineReader.readLine()) != null) {Matcher matcher = LINUX_MEMORY_REGEX.matcher(line);if (matcher.matches()) {String totalMemory = matcher.group(1);return Long.parseLong(totalMemory) * 1024L; // Convert from kilobyte to byte}}// expected line did not comeLOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo'). " +"Unexpected format.");return -1;}catch (NumberFormatException e) {LOG.error("Cannot determine the size of the physical memory for Linux host (using '/proc/meminfo'). " +"Unexpected format.");return -1;}

相关文章:

flink源码分析-获取JVM最大堆内存

flink版本: flink-1.11.2 代码位置: org.apache.flink.runtime.util.EnvironmentInformation#getMaxJvmHeapMemory 如果设置了-Xmx参数&#xff0c;就返回这个参数&#xff0c;如果没设置就返回机器物理内存的1/4. 这里主要看各个机器内存的获取方法。 /*** The maximum JVM…...

第17节 R语言分析:生物统计数据集 R 编码分析和绘图

生物统计数据集 R 编码分析和绘图 生物统计学,用于对给定文件 data.csv 中的医疗数据应用 R 编码,该文件是患者人口统计数据集,包含有关来自各种祖先谱系的个体的标准信息。 数据集特征解释 脚本 output= file("Output.txt") # File name of output log sink(o…...

一文了解什么是Selenium自动化测试?

目录 一、Selenium是什么&#xff1f; 二、Selenium History 三、Selenium原理 四、Selenium工作过程总结&#xff1a; 五、remote server端的这些功能是如何实现的呢&#xff1f; 六、附&#xff1a; 一、Selenium是什么&#xff1f; 用官网的一句话来讲&#xff1a;Sel…...

java接口实现

文章目录 java接口实现接口中成员组成默认方法静态方法私有接口&#xff08;保证自己的JDK版本大于等于9版本&#xff09;类和接口的关系抽象类与接口之间的区别 java接口实现 1.接口关键字 interface2.接口不能实例化3.类与接口之间的关系是实现关系&#xff0c;通过 impleme…...

数据结构入门指南:链表(新手避坑指南)

目录 前言 1.链表 1.1链表的概念 1.2链表的分类 1.2.1单向或双向 1.2.2.带头或者不带头 1.2.33. 循环或者非循环 1.3链表的实现 定义链表 总结 前言 前边我们学习了顺序表&#xff0c;顺序表是数据结构中最简单的一种线性数据结构&#xff0c;今天我们来学习链表&#x…...

SpringBoot第24讲:SpringBoot集成MySQL - MyBatis XML方式

SpringBoot第24讲&#xff1a;SpringBoot集成MySQL - MyBatis XML方式 上文介绍了用JPA方式的集成MySQL数据库&#xff0c;JPA方式在中国以外地区开发而言基本是标配&#xff0c;在国内MyBatis及其延伸框架较为主流。本文是SpringBoot第24讲&#xff0c;主要介绍MyBatis技栈的演…...

linux 查看网卡,网络情况

1&#xff0c;使用nload命令查看 #yum -y install nload 2&#xff0c; 查看eth0网卡网络情况 #nload eth0 Incoming也就是进入网卡的流量&#xff0c;Outgoing&#xff0c;也就是从这块网卡出去的流量&#xff0c;每一部分都有下面几个。 – Curr&#xff1a;当前流量 – Avg…...

在Mac上搭建Gradle环境

在Mac上搭建Gradle环境&#xff1a; 步骤1&#xff1a;下载并安装Java开发工具包&#xff08;JDK&#xff09; Gradle运行需要Java开发工具包&#xff08;JDK&#xff09;。您可以从Oracle官网下载适合您的操作系统版本的JDK。请按照以下步骤进行操作&#xff1a; 打开浏览器…...

Docker网络与Docker Compose服务编排

docker网络 docker是以镜像一层一层构建的&#xff0c;而基础镜像是linux内核&#xff0c;因此docker之间也需要通讯&#xff0c;那么就需要有自己的网络。就像windows都有自己的内网地址一样&#xff0c;每个docker容器也是有自己的私有地址的。 docker inspect [docker_ID]…...

opencv+ffmpeg环境(ubuntu)搭建全面详解

一.先讲讲opencv和ffmpeg之间的关系 1.1它们之间的联系 我们知道opencv主要是用来做图像处理的&#xff0c;但也包含视频解码的功能&#xff0c;而在视频解码部分的功能opencv是使用了ffmpeg。所以它们都是可以处理图像和视频的编解码&#xff0c;我个人感觉两个的侧重点不一…...

开发基于 LoRaWAN 的设备须知--最大兼容性

最大兼容性配置简介 LoRaWAN开放协议的建立前提是每个制造的设备都可以被唯一且安全地识别。配置是创建唯一标识和相应秘密的过程。虽然配置过程是常规的,但存在一些可能并不明显的陷阱。本章尝试描述配置基于 LoRa 的设备的一些最佳实践。 配置概念 基于 LoRa 的设备配置与银…...

一、SpringBoot基础[日志]

一、日志 解释&#xff1a;SpringBoot使用logback作为默认的日志框架&#xff0c;其中还可以导入log4j2等优秀的日志框架 1.修改日志内容 修改整个日志格式&#xff1a;logging.pattern.console%d{yyyy-MM-dd HH:mm:ss} %-5level [%thread] %logger{15} 你好 %msg%n %d{yyy…...

libuv库学习笔记-networking

Networking 在 libuv 中&#xff0c;网络编程与直接使用 BSD socket 区别不大&#xff0c;有些地方还更简单&#xff0c;概念保持不变的同时&#xff0c;libuv 上所有接口都是非阻塞的。它还提供了很多工具函数&#xff0c;抽象了恼人、啰嗦的底层任务&#xff0c;如使用 BSD …...

C++多线程编程(第三章 案例1,使用互斥锁+ list模拟线程通信)

主线程和子线程进行list通信&#xff0c;要用到互斥锁&#xff0c;避免同时操作 1、封装线程基类XThread控制线程启动和停止&#xff1b; 2、模拟消息服务器线程&#xff0c;接收字符串消息&#xff0c;并模拟处理&#xff1b; 3、通过Unique_lock和mutex互斥方位list 消息队列…...

IOS UICollectionView 设置cell大小不生效问题

代码设置flowLayout.itemSize 单元格并没有改变布局大小&#xff0c; 解决办法如下图&#xff1a;把View flow layout 的estimate size 设置为None&#xff0c;上面设置的itemSize 生效了。...

浅谈3D隐式表示(SDF,Occupancy field,NeRF)

本篇文章介绍了符号距离函数Signed Distance Funciton(SDF)&#xff0c;占用场Occupancy Field&#xff0c;神经辐射场Neural Radiance Field&#xff08;NeRF&#xff09;的概念、联系与区别。 显式表示与隐式表示 三维空间的表示形式可以分为显式和隐式。 比较常用的显式表…...

软件测试技能大赛任务二单元测试试题

任务二 单元测试 执行代码测试 本部分按照要求&#xff0c;执行单元测试&#xff0c;编写java应用程序&#xff0c;按照要求的覆盖方法设计测试数据&#xff0c;使用JUnit框架编写测试类对程序代码进行测试&#xff0c;对测试执行结果进行截图&#xff0c;将相关代码和相关截…...

MybatisPlus拓展篇

文章目录 逻辑删除通用枚举字段类型处理器自动填充功能防全表更新与删除插件MybatisX快速开发插件插件安装逆向工程常见需求代码生成 乐观锁问题引入乐观锁的使用效果测试 代码生成器执行SQL分析打印多数据源 逻辑删除 逻辑删除的操作就是增加一个字段表示这个数据的状态&…...

设置k8s中节点node的ROLES值,K8S集群怎么修改node1的集群ROLES

设置k8s中节点node的ROLES值 1.查看集群 [rootk8s-master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready control-plane,master 54d v1.23.8 k8s-node1 Ready <none> 54d v1.…...

【*1900 图论】CF1328 E

Problem - E - Codeforces 题意&#xff1a; 思路&#xff1a; 注意到题目的性质&#xff1a;满足条件的路径个数是极少的&#xff0c;因为每个点离路径的距离<1 先考虑一条链&#xff0c;那么直接就选最深那个点作为端点即可 为什么&#xff0c;因为我们需要遍历所有点…...

stm32G473的flash模式是单bank还是双bank?

今天突然有人stm32G473的flash模式是单bank还是双bank&#xff1f;由于时间太久&#xff0c;我真忘记了。搜搜发现&#xff0c;还真有人和我一样。见下面的链接&#xff1a;https://shequ.stmicroelectronics.cn/forum.php?modviewthread&tid644563 根据STM32G4系列参考手…...

FFmpeg 低延迟同屏方案

引言 在实时互动需求激增的当下&#xff0c;无论是在线教育中的师生同屏演示、远程办公的屏幕共享协作&#xff0c;还是游戏直播的画面实时传输&#xff0c;低延迟同屏已成为保障用户体验的核心指标。FFmpeg 作为一款功能强大的多媒体框架&#xff0c;凭借其灵活的编解码、数据…...

java调用dll出现unsatisfiedLinkError以及JNA和JNI的区别

UnsatisfiedLinkError 在对接硬件设备中&#xff0c;我们会遇到使用 java 调用 dll文件 的情况&#xff0c;此时大概率出现UnsatisfiedLinkError链接错误&#xff0c;原因可能有如下几种 类名错误包名错误方法名参数错误使用 JNI 协议调用&#xff0c;结果 dll 未实现 JNI 协…...

2021-03-15 iview一些问题

1.iview 在使用tree组件时&#xff0c;发现没有set类的方法&#xff0c;只有get&#xff0c;那么要改变tree值&#xff0c;只能遍历treeData&#xff0c;递归修改treeData的checked&#xff0c;发现无法更改&#xff0c;原因在于check模式下&#xff0c;子元素的勾选状态跟父节…...

C# 求圆面积的程序(Program to find area of a circle)

给定半径r&#xff0c;求圆的面积。圆的面积应精确到小数点后5位。 例子&#xff1a; 输入&#xff1a;r 5 输出&#xff1a;78.53982 解释&#xff1a;由于面积 PI * r * r 3.14159265358979323846 * 5 * 5 78.53982&#xff0c;因为我们只保留小数点后 5 位数字。 输…...

使用LangGraph和LangSmith构建多智能体人工智能系统

现在&#xff0c;通过组合几个较小的子智能体来创建一个强大的人工智能智能体正成为一种趋势。但这也带来了一些挑战&#xff0c;比如减少幻觉、管理对话流程、在测试期间留意智能体的工作方式、允许人工介入以及评估其性能。你需要进行大量的反复试验。 在这篇博客〔原作者&a…...

Kafka入门-生产者

生产者 生产者发送流程&#xff1a; 延迟时间为0ms时&#xff0c;也就意味着每当有数据就会直接发送 异步发送API 异步发送和同步发送的不同在于&#xff1a;异步发送不需要等待结果&#xff0c;同步发送必须等待结果才能进行下一步发送。 普通异步发送 首先导入所需的k…...

作为测试我们应该关注redis哪些方面

1、功能测试 数据结构操作&#xff1a;验证字符串、列表、哈希、集合和有序的基本操作是否正确 持久化&#xff1a;测试aof和aof持久化机制&#xff0c;确保数据在开启后正确恢复。 事务&#xff1a;检查事务的原子性和回滚机制。 发布订阅&#xff1a;确保消息正确传递。 2、性…...

【学习笔记】erase 删除顺序迭代器后迭代器失效的解决方案

目录 使用 erase 返回值继续迭代使用索引进行遍历 我们知道类似 vector 的顺序迭代器被删除后&#xff0c;迭代器会失效&#xff0c;因为顺序迭代器在内存中是连续存储的&#xff0c;元素删除后&#xff0c;后续元素会前移。 但一些场景中&#xff0c;我们又需要在执行删除操作…...

git: early EOF

macOS报错&#xff1a; Initialized empty Git repository in /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/ remote: Enumerating objects: 2691797, done. remote: Counting objects: 100% (1760/1760), done. remote: Compressing objects: 100% (636/636…...