【HDFS联邦（2）】HDFS Router-based Federation官网解读：HDFSRouterFederation的架构、各组件基本原理

本文主要参考官网：HDFSRouterFederation 对HDFSRouterFederation进行了解：

1. viewfs会带来的问题，HDFSRouterFederation是如何解决的
2. HDFSRouterFederation的架构、各组件基本原理说明

一. 介绍

NameNodes have scalability limits because of the metadata overhead comprised of inodes (files and directories) and file blocks, the number of Datanode heartbeats, and the number of HDFS RPC client requests. The common solution is to split the filesystem into smaller subclusters HDFS Federation and provide a federated view ViewFs. The problem is how to maintain the split of the subclusters (e.g., namespace partition), which forces users to connect to multiple subclusters and manage the allocation of folders/files to them.

因为元数据（文件和目录、文件块）的开销、datanode心跳管理以及HDFS RPC请求，namenode有对于datanode的拓展有限制。我们将文件系统分为几个子HDFS联邦系统，然后提供一个联邦的 ViewFs 。但会出现维护子集群的分裂(例如，namespace分区)的问题，这强制用户连接到多个子集群并管理文件夹/文件的分配。

 

二、HDFS Router-based Federation 架构

我们可以对联邦分区扩展可以添加一个管理namespace联邦的软件层。

架构特性

• 这个额外层允许用户透明的访问任何子系统，让子集群独立地管理自己的块池，并且将支持子集群之间的数据rebalancing。
• Router-based Federation下的子集群不需要是独立的HDFS集群，也可以是普通的federation集群(包含多个块池)，或者是federation和独立集群的混合集群。
• 为了实现这些目标，联邦层需要将块访问引导到适当的子集群，维护名称空间的状态，并提供数据再平衡机制。所以这一层必须具有可伸缩性、高可用性和容错性。

 

架构组成

This federation layer comprises multiple components. The Router component that has the same interface as a NameNode, and forwards the client requests to the correct subcluster, based on ground-truth information from a State Store. The State Store combines a remote Mount Table (in the flavor of ViewFs, but shared between clients) and utilization (load/capacity) information about the subclusters. This approach has the same architecture as YARN federation.

该联邦层由多个组件组成。

• Router组件具有与NameNode相同的接口，并根据来自State Store的真实信息将客户端请求转发到正确的子集群。
• State Store包含了一个远程挂载表(类似于视图，但在客户端之间共享)和关于子集群的利用率(负载/容量)信息。这种方法具有与YARN联合相同的体系结构。

 

1. 示例说明

The simplest configuration deploys a Router on each NameNode machine. The Router monitors the local NameNode and its state and heartbeats to the State Store. The Router monitors the local NameNode and heartbeats the state to the State Store. When a regular DFS client contacts any of the Routers to access a file in the federated filesystem, the Router checks the Mount Table in the State Store (i.e., the local cache) to find out which subcluster contains the file. Then it checks the Membership table in the State Store (i.e., the local cache) for the NameNode responsible for the subcluster. After it has identified the correct NameNode, the Router proxies the request. The client accesses Datanodes directly. ing

client通过Router方式请求文件过程

• nn状态定期维护：Router监控本地namenode的状态、并定期（心跳）给State Store。
• 找到子集群：当一个普通的DFS客户机通过任意的Router访问 federated filesystem 中的文件时，Router检查State Store (即本地缓存)中的Mount Table，来找出哪个子集群包含该文件。
• 找到负责的nn：然后，它检查State Store 中的Membership表，查找负责子集群的NameNode。在它识别了正确的NameNode之后，Router代理请求。客户端直接访问datanode。(参考client向namenode读写文件过程)

 

2. Router

There can be multiple Routers in the system with soft state. Each Router has two roles:

• Federated interface: expose a single, global NameNode interface to the clients and forward the requests to the active NameNode in the correct subcluster
• NameNode heartbeat: maintain the information about a NameNode in the State Store

系统中可以有多个（soft state的）Router，每个Router都有两个作用；

• 联邦接口：暴露单个、全局的namenode接口给客户端，将请求转发到正确子集群的active NameNode
• NameNode heartbeat：维护namenode的状态信息在 State Store。

 

2.1. Federated interface

The Router receives a client request, checks the State Store for the correct subcluster, and forwards the request to the active NameNode of that subcluster. The reply from the NameNode then flows in the opposite direction. The Routers are stateless and can be behind a load balancer. For health checking, you can use /isActive endpoint as a health probe (e.g. http://ROUTER_HOSTNAME:ROUTER_PORT/isActive). For performance, the Router also caches remote mount table entries and the state of the subclusters. To make sure that changes have been propagated to all Routers, each Router heartbeats its state to the State Store.

The communications between the Routers and the State Store are cached (with timed expiration for freshness). This improves the performance of the system.

Router接收到client的请求，确认正确子集群的State Store，然后将请求发送给指定子集群的namenode，然后接收namenode的response。路由器是无状态的，可以在负载均衡器后面。

• 对于健康检查，你可以使用/isActive 作为一个健康探针。
• 对于性能方面，Router缓存远程挂载表入口和子集群的状态。为了保证改变能同步到所有的Routers，每个Routers给State Store层汇报其心跳。

Routers和State Store之间的通信被缓存(有一个刷新时间)。这样可以提高系统的性能。

 

2.2. Router heartbeat

Router周期性的汇报心跳给State Store。

 

2.3. NameNode heartbeat

For this role, the Router periodically checks the state of a NameNode (usually on the same server) and reports their high availability (HA) state and load/space status to the State Store. Note that this is an optional role, as a Router can be independent of any subcluster. For performance with NameNode HA, the Router uses the high availability state information in the State Store to forward the request to the NameNode that is most likely to be active. Note that this service can be embedded into the NameNode itself to simplify the operation.

Router周期性的检查namenode的状态，并将高可用状态、空间、负载等汇报给State Store。

注意：此功能不是必须的，Router可以独立于任何的子集群。

考虑到NameNode HA的使用性能，Router使用State Store中的高可用性状态信息将请求转发到最有可能处于活动状态的NameNode。注意，可以将此服务嵌入到NameNode本身以简化操作。

 

2.4. Availability and fault tolerance

The Router operates with failures at multiple levels.

• Federated interface HA: The Routers are stateless and metadata operations are atomic at the NameNodes. If a Router becomes unavailable, any Router can take over for it. The clients configure their DFS HA client (e.g., ConfiguredFailoverProvider or RequestHedgingProxyProvider) with all the Routers in the federation as endpoints.

• Unavailable State Store: If a Router cannot contact the State Store, it will enter into a Safe Mode state which disallows it from serving requests. Clients will treat Routers in Safe Mode as it was an Standby NameNode and try another Router. There is a manual way to manage the Safe Mode for the Router.
• Expired NameNodes: If a NameNode heartbeat has not been recorded in the State Store for a multiple of the heartbeat interval, the monitoring Router will record that the NameNode has expired and no Routers will attempt to access it. If an updated heartbeat is subsequently recorded for the NameNode, the monitoring Router will restore the NameNode from the expired state.

Router失败的几个级别

1. 联邦接口高可用：router是无状态的，元数据操作在namenode操作时原子性的。如果一个router不可用，其他router将会接管。客户端（flink sql等）可以配置（比如ConfiguredFailoverProvider 或 RequestHedgingProxyProvider）联邦中所有Router作为端点来实现HA客户端。
2. State Store不可用：如果Router不能与State Store通讯，router将会进入安全模式状态，即不允许处理请求。客户端将会将处于安全模式的Routers 看成 Standby NameNode，router将会尝试其他Router。这里有手工的方式管理Router的安全模式。
3. 过期的NameNode：如果State Store 连续几次心跳间隔后没有接收到namenode的心跳，Router的监控将会记录NameNode已经过期，将不会有 Routers 访问它。

 

Interfaces

Router暴露了几个接口，以便与用户、管理员进行交互：

• RPC：Router RPC实现了客户端与HDFS交互最常见的接口
• Admin：管理员可以通过RPC添加或移除挂载表。
• Web UI ：Router暴露了一个联邦状态的可视化界面。
• WebHDFS：除了 RPC 接口之外，路由器还提供 HDFS REST 接口 (WebHDFS)。
• JMX：通过JMX模仿namenode来暴露指标，Web UI 使用它来获取集群状态。

Router不支持以下操作，当遇见以下操作时将会抛出异常：

• 重命名、复制文件/文件夹在两个不同的namespace
• 写入正在重新平衡的文件/文件夹

 

3. Quota management

在挂载表级别，联邦支持全局quota（份额）的控制。考虑到性能方面，Router缓存quota的使用情况，并定时更新。当调用RouterRPCSever中的写RPC请求时，会进行quota使用情况的检查。

详细的见：See HDFS Quotas Guide for the quota detail.
 

4. State Store

State Store维护的内容：

• 维护子集群中块加载、磁盘可用率、HA情况等状态。
• 维护远程挂载表：子集群与文件夹/文件之间的映射。

State Store后端是可插拔的。以下是State Store主要的存储信息，以及其实现：

• Membership: 成员信息对namenodes的状态进行了编码，包括：子集群的存储容量、node的数量等。Router定期获取一个或多个namenode的信息。鉴于多个Routers可以监控同一个Namenode。当从State Store查询信息时，Router会从数据层面来仲裁，具体的：Router将会淘汰超过某个阈值的条目(例如，10个Router的心跳周期)。
• Mount Table: 包含了文件夹和子集群之间的映射。类似于ViewFS的映射表：指定联邦文件夹、目标子集群和该文件夹中的路径。

 

三、部署 ing

涉及到Router启动、挂载表管理、废除nameservice、Router定时刷新、客户端配置、namenode、Router配置与指标配置等。

 
参考官网：[[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Deployment]]