模拟实现消息队列项目(系列4) -- 服务器模块(内存管理)

目录

前言

1. 创建MemoryDataCenter

2. 封装Exchange 和 Queue方法

3. 封装Binding操作

4. 封装Message操作

4.1 封装消息中心集合messageMap

4.2 封装消息与队列的关系集合queueMessageMap的操作

5. 封装未确认消息集合waitMessage的操作

6. 从硬盘中恢复数据到内存中

7. MemoryDataCenter单元测试

结语

---

前言

        上一节我们总结了服务器模块的硬盘管理,将交换机,队列,绑定存书到Sqlite数据库中,将消息按照队进行创建文件存储在本地硬盘中.并且封装了对于数据库和文件的各种操作.实现了持久化的效果,但是实际的消息存储/转发,主要靠内存的结构.对于消息队列来说,内存部分是更关键的,内存速度更快,可以达到更高的并发.本节就对内存管理进行封装.本项目全部代码已上传Gitee,链接放在文章末尾,欢迎大家访问!

---

1. 创建MemoryDataCenter

路径:mqserver.datacenter.MemoryDataCenter

考虑到多线程的原因,我们将HashMap替换成ConcurrentHashMap (对每个哈希桶进行加锁,相对来说是线程安全的)

@Data
public class MemoryDataCenter {// 1. 交换机  多线程环境下使用,使用ConcurrentHashMap会相对线程安全//         key:ExchangeName,value:Exchange对象private ConcurrentHashMap<String, Exchange> exchangeMap = new ConcurrentHashMap<>();// 2. 队列  key:QueueName,value:MSQueue对象private ConcurrentHashMap<String, MSQueue> queueMap = new ConcurrentHashMap<>();// 3. 绑定  key:ExchangeName,value:HashMap(key:QueueName,value:MSQueue对象)private ConcurrentHashMap<String,ConcurrentHashMap<String, Binding>> bindingsMap = new ConcurrentHashMap<>();// 4. 消息  key:MessageID,value:Message对象private ConcurrentHashMap<String, Message> messageMap = new ConcurrentHashMap<>();// 5. 消息和队列的映射关系 HashMap: key:QueueName,value:LinkedList(Message对象)private ConcurrentHashMap<String, LinkedList<Message>> queueMessageMap = new ConcurrentHashMap<>();// 6. 未确认的消息  HashMap: key:QueueName,value:HashMap(key:MessageID,value:Message对象)private ConcurrentHashMap<String,ConcurrentHashMap<String, Message>> queueMessageWaitAckMap = new ConcurrentHashMap<>();
}

2. 封装Exchange 和 Queue方法

主要就是插入和获取数据以及删除

   /*** 1. 针对内存中的交换机,队列设置操作*/public void insertExchange(Exchange exchange) {exchangeMap.put(exchange.getName(), exchange);System.out.println("[MemoryDataCenter] 新交换机添加成功! exchangeName=" + exchange.getName());}public Exchange getExchange(String exchangeName) {return exchangeMap.get(exchangeName);}public void deleteExchange(String exchangeName) {exchangeMap.remove(exchangeName);System.out.println("[MemoryDataCenter] 交换机删除成功! exchangeName=" + exchangeName);}public void insertQueue(MSQueue queue) {queueMap.put(queue.getName(), queue);System.out.println("[MemoryDataCenter] 新队列添加成功! queueName=" + queue.getName());}public MSQueue getQueue(String queueName) {return queueMap.get(queueName);}public void deleteQueue(String queueName) {queueMap.remove(queueName);System.out.println("[MemoryDataCenter] 队列删除成功! queueName=" + queueName);}

3. 封装Binding操作

这里呢之所以将绑定的操作单独列举出来,是因为存储绑定信息的数据结构是相对比较复杂的,是嵌套的HashMap.

对于插入绑定信息:

1, 首先按照交换机的名字进行查找,如果查找不到就进行创建一个HashMap的数据结构存储到含有绑定信息的HashMap中,如果存在的话在按照队列名字进行查找绑定信息,如果查找到了,说明改绑定信息已经插入过就不要进行插入了,如果没找到就进行插入操作.

2. 在上述查找和插入的操作比并不是原子的,所以我们要给是上述操作,按照bindingMap进行加锁.以保证我们的线程操作是安全的.

下述是相关对于绑定的操作的代码:

    /*** 2. 针对绑定进行操作*//*** 2.1插入绑定信息* @param binding* @throws MqException*/public void insertBinding(Binding binding) throws MqException {
//        ConcurrentHashMap<String, Binding> bindingMap = bindingsMap.get(binding.getExchangeName());
//        if (bindingMap == null) {
//            bindingMap = new ConcurrentHashMap<>();
//            bindingsMap.put(binding.getExchangeName(), bindingMap);
//        }// 先使用 exchangeName 查一下, 对应的哈希表是否存在. 不存在就创建一个.ConcurrentHashMap<String, Binding> bindingMap = bindingsMap.computeIfAbsent(binding.getExchangeName(),k -> new ConcurrentHashMap<>());synchronized (bindingMap) {// 再根据 queueName 查一下目前的绑定的交换机绑定的是否是当前传入的队列. 如果已经存在(存在相同的绑定关系了,就不需要进行传入), 就抛出异常. 不存在才能插入.if (bindingMap.get(binding.getQueueName()) != null) {throw new MqException("[MemoryDataCenter] 绑定已经存在! exchangeName=" + binding.getExchangeName() +", queueName=" + binding.getQueueName());}// 最后将绑定关系传入到bingMap中bindingMap.put(binding.getQueueName(), binding);}System.out.println("[MemoryDataCenter] 新绑定添加成功! exchangeName=" + binding.getExchangeName()+ ", queueName=" + binding.getQueueName());}/*** 2.2 获取绑定1: 根据exchangeName, queueName 获取唯一的绑定* @param exchangeName* @param queueName*/public Binding getBinding(String exchangeName, String queueName){ConcurrentHashMap<String, Binding> bindingMap = bindingsMap.get(exchangeName);if (bindingMap == null){return null;}synchronized (bindingMap){// 防止当别的操作删除了这个队列的绑定信息,而导致的线程错误return bindingMap.get(queueName);}}/*** 2.3 获取绑定2: 根据exchangeName 查询所有绑定* @param exchangeName* @return*/public ConcurrentHashMap<String, Binding> getBindings(String exchangeName) throws MqException {if (bindingsMap.get(exchangeName) == null){return null;}return bindingsMap.get(exchangeName);}/*** 2.4 删除绑定关系(单个) 一个交换机对应的单个队列的绑定关系* @param binding* @throws MqException*/public void deleteBinding(Binding binding) throws MqException {ConcurrentHashMap<String, Binding> bindingMap = bindingsMap.get(binding.getExchangeName());if (bindingMap == null) {// 该交换机没有绑定任何队列. 报错.throw new MqException("[MemoryDataCenter] 绑定不存在! exchangeName=" + binding.getExchangeName()+ ", queueName=" + binding.getQueueName());}bindingMap.remove(binding.getQueueName());System.out.println("[MemoryDataCenter] 绑定删除成功! exchangeName=" + binding.getExchangeName()+ ", queueName=" + binding.getQueueName());}/*** 2.5 删除绑定关系(多个) 1个交换机对应的多个队列的绑定关系.*/public void deleteBinding(String exchangeName){bindingsMap.remove(exchangeName);}

4. 封装Message操作

4.1 封装消息中心集合messageMap

• 1. 添加消息到消息中心
• 2. 根据消息ID查询消息
• 3. 根据消息ID删除消息

 /*** 3. 针对消息进行操作*//*** 3.1 添加消息* @param message*/public void addMessage(Message message) {messageMap.put(message.getMessageID(), message);System.out.println("[MemoryDataCenter] 新消息添加成功! messageId=" + message.getMessageID());}/*** 3.2 根据 id 查询消息* @param messageId* @return*/public Message getMessage(String messageId) {return messageMap.get(messageId);}/*** 3.3 根据 id 删除消息* @param messageId*/public void removeMessage(String messageId) {messageMap.remove(messageId);System.out.println("[MemoryDataCenter] 消息被移除! messageId=" + messageId);}

4.2 封装消息与队列的关系集合queueMessageMap的操作

• 1. 发送消息到指定队列名字的队列
• 2. 从指定队列中获取消息集合
• 3. 获取指定队列名字队列中消息的个数

 /*** 4 针对消息和队列的关系进行操作*//*** 4.1 发送消息到指定队列* @param queue* @param message*/public void sendMessage(MSQueue queue, Message message) {// 先根据队列的名字, 找到该队列对应的消息链表.// 先根据队列的名字进行查询,查不到就进行创建该队列对应的链表  // computeIfAbsent线程安全的LinkedList<Message> messages = queueMessageMap.computeIfAbsent(queue.getName(),k-> new LinkedList<>());// 再把数据加到 messages 里面synchronized (messages) {// 对该队列进行添加的时候需要进行加锁messages.add(message);}// 在这里把该消息也往消息中心中插入一下. 假设如果 message 已经在消息中心存在, 重复插入也没关系.// 主要就是相同 messageId, 对应的 message 的内容一定是一样的. (服务器代码不会对 Message 内容做修改 basicProperties 和 body)addMessage(message);System.out.println("[MemoryDataCenter] 消息被添加到队列中! messageId=" + message.getMessageID());}/*** 4.2 从指定队列名字中进行提取信息* @param queueName* @return*/public Message pollMessage(String queueName){LinkedList<Message> messages = queueMessageMap.get(queueName);// 队列中没有信息if (messages == null){System.out.println("[MemoryDataCenter] 该队列中没有信息! queueName=" + queueName);return null;}// 将队列进行头删除(提取信息)synchronized (messages){if (messages.size() == 0){System.out.println("[MemoryDataCenter] 该队列中没有信息! queueName=" + queueName);return null;}Message currentMessage = messages.remove(0); System.out.println("[MemoryDataCenter] 消息已经从队列中取出! queueName=" + queueName + ", MessageID=" + currentMessage.getMessageID() );return currentMessage;}}/*** 4.3 获取指定队列名字中消息的个数* @param queueName* @return*/public int getMessageCount(String queueName){LinkedList<Message> messages = queueMessageMap.get(queueName);// 队列中没有信息if (messages == null){System.out.println("[MemoryDataCenter] 该队列中没有信息! queueName=" + queueName);return 0;}// 将队列进行头删除(提取信息)synchronized (messages){if (messages.size() == 0){System.out.println("[MemoryDataCenter] 该队列中没有信息! queueName=" + queueName);return 0;}return messages.size();}}

5. 封装未确认消息集合waitMessage的操作

• 1. 添加消息到等待确认队列
• 2. 从指定未确认队列中删除消息
• 3. 根据指定的消息ID与未确认队列名字获取消息内容

/*** 5. 未确认消息Map的操作*//*** 5.1 添加消息到指定等待确认队列* @param queueName* @param message*/public void addMessageWaitAck(String queueName, Message message){ConcurrentHashMap<String,Message> waitMessage = queueMessageWaitAckMap.computeIfAbsent(queueName, k-> new ConcurrentHashMap<>());waitMessage.put(message.getMessageID(),message);System.out.println("[MemoryDataCenter] 消息进入等待确认队列! messageID=" + message.getMessageID());}/*** 5.2 从指定的未确认消息队列中进行删除消息* @param queueName* @param messageId*/public void removeMessageWaitAck(String queueName, String messageId){ConcurrentHashMap<String,Message> waitMessage = queueMessageWaitAckMap.get(queueName);if (waitMessage == null){System.out.println("[MemoryDataCenter] 该队列为空! queueName=" + queueName);return;}waitMessage.remove(messageId);System.out.println("[MemoryDataCenter] 消息已经从等待确认队列中移除! messageId=" + messageId);}/*** 5.3 根据指定消息ID从队列中进行获取信息* @param queueName* @param messageId* @return*/public Message geMessageWaitAck(String queueName, String messageId){ConcurrentHashMap<String,Message> waitMessage = queueMessageWaitAckMap.get(queueName);if (waitMessage == null){System.out.println("[MemoryDataCenter] 该队列为空! queueName=" + queueName);return null;}return waitMessage.get(messageId);}

6. 从硬盘中恢复数据到内存中

使用之前封装过的diskDataCenter进行恢复数据.

1. 清空当前内存数据结构中的数据

2. 恢复所有的交换机,队列,绑定,消息数据,恢复消息数据的时候,要将消息中心和消息与队列的映射进行恢复.

/*** 6. 从硬盘中恢复数据到内存中 (使用之前封装好的管理硬盘的类进行实现)*/public void recovery(DiskDataCenter diskDataCenter) throws IOException, MqException, ClassNotFoundException {// 1. 清空内存中各种数据信息queueMap.clear();exchangeMap.clear();bindingsMap.clear();messageMap.clear();queueMessageMap.clear();// 2. 恢复所有的交换机信息List<Exchange> exchanges = diskDataCenter.selectAllExchange();for (Exchange exchange :exchanges) {exchangeMap.put(exchange.getName(),exchange);}// 3. 恢复所有的队列信息List<MSQueue> queues = diskDataCenter.selectAllMSQueue();for (MSQueue msQueue :queues) {queueMap.put(msQueue.getName(),msQueue);}// 4. 恢复所有的绑定数据List<Binding> bindings = diskDataCenter.selectAllBinding();for (Binding binding: bindings){ConcurrentHashMap<String,Binding> bindingMap = bindingsMap.computeIfAbsent(binding.getExchangeName(), k-> new ConcurrentHashMap<>());bindingMap.put(binding.getQueueName(),binding);}// 4. 恢复所有的消息数据// 4.1 遍历所有的队列// List<MSQueue> queues = diskDataCenter.selectAllMSQueue();for (MSQueue msQueue:queues) {LinkedList<Message> messages = diskDataCenter.loadAllMessageFromQueue(msQueue.getName());// 4.2 将获取的消息进行进行加入到队列queueMessageMap.put(msQueue.getName(),messages);// 4.3 将消息添加上到消息中心for (Message message : messages) {messageMap.put(message.getMessageID(),message);}}

7. MemoryDataCenter单元测试

package com.example.demo.mqserver.datacenter;import com.example.demo.DemoApplication;
import com.example.demo.common.MqException;
import com.example.demo.mqserver.core.*;
import org.apache.tomcat.util.http.fileupload.FileUtils;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.test.context.SpringBootTest;import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;import static org.junit.jupiter.api.Assertions.*;/*** Created with IntelliJ IDEA.* Description:* User: YAO* Date: 2023-07-31* Time: 10:30*/
@SpringBootTest
class MemoryDataCenterTest {MemoryDataCenter memoryDataCenter = null;@BeforeEachvoid setUp() {memoryDataCenter = new MemoryDataCenter();}@AfterEachvoid tearDown() {memoryDataCenter = null;}// 创建一个测试交换机private Exchange createTestExchange(String exchangeName) {Exchange exchange = new Exchange();exchange.setName(exchangeName);exchange.setType(ExchangeType.DIRECT);exchange.setAutoDelete(false);exchange.setDurable(true);return exchange;}// 创建一个测试队列private MSQueue createTestQueue(String queueName) {MSQueue queue = new MSQueue();queue.setName(queueName);queue.setDurable(true);queue.setExclusive(false);queue.setAutoDelete(false);return queue;}/*** 1. 针对交换机进行操作*/@Testpublic void testExchange(){// 1. 创建交换机进行插入Exchange expectExchange = createTestExchange("testExchange");memoryDataCenter.insertExchange(expectExchange);// 2. 查询交换机Exchange actualExchange = memoryDataCenter.getExchange("testExchange");// 比较内存中的引用是否是同一个引用Assertions.assertEquals(expectExchange,actualExchange);// 3. 删除交换机memoryDataCenter.deleteExchange("testExchange");// 4. 查询交换机,比较结果actualExchange = memoryDataCenter.getExchange("testExchange");Assertions.assertNull(actualExchange);}/*** 2. 针对队列进行操作*/@Testpublic void testQueue(){// 1. 创建交换机进行插入MSQueue expectQueue = createTestQueue("testQueue");memoryDataCenter.insertQueue(expectQueue);// 2. 查询交换机MSQueue actualQueue = memoryDataCenter.getQueue("testQueue");// 比较内存中的引用是否是同一个引用Assertions.assertEquals(expectQueue,actualQueue);// 3. 删除交换机memoryDataCenter.deleteQueue("testQueue");// 4. 查询交换机,比较结果actualQueue = memoryDataCenter.getQueue("testQueue");Assertions.assertNull(actualQueue);}/*** 3. 针对绑定进行测试*/@Testpublic void testBinding() throws MqException {// 1.创建绑定并加入到集合中Binding expectedBinding = new Binding();expectedBinding.setExchangeName("testExchange");expectedBinding.setQueueName("testQueue");memoryDataCenter.insertBinding(expectedBinding);// 2. 查询绑定(单个)Binding actualBinding = memoryDataCenter.getBinding("testExchange","testQueue");Assertions.assertEquals(expectedBinding,actualBinding);// 2.1 查询所有的绑定ConcurrentHashMap<String, Binding> bindingMap = memoryDataCenter.getBindings("testExchange");Assertions.assertEquals(1, bindingMap.size());Assertions.assertEquals(expectedBinding, bindingMap.get("testQueue"));// 3. 删除绑定memoryDataCenter.deleteBinding("testExchange");actualBinding = memoryDataCenter.getBinding("testExchange","testQueue");Assertions.assertNull(actualBinding);bindingMap = memoryDataCenter.getBindings("testExchange");Assertions.assertNull(bindingMap);}private Message createTestMessage(String content) {Message message = Message.createMessageWithId("testRoutingKey", null, content.getBytes());return message;}/*** 4. 针对消息进行测试*/@Testpublic void testMessage(){// 1. 创建消息并插入Message expectedMessage = createTestMessage("testMessage");memoryDataCenter.addMessage(expectedMessage);// 2. 查询消息并比较Message actualMessage = memoryDataCenter.getMessage(expectedMessage.getMessageID());Assertions.assertEquals(expectedMessage, actualMessage);// 4. 删除消息memoryDataCenter.removeMessage(expectedMessage.getMessageID());// 5. 查询消息并比较actualMessage = memoryDataCenter.getMessage(expectedMessage.getMessageID());Assertions.assertNull(actualMessage);}/*** 5. 测试将消息发送到对列中*/@Testpublic void sendMessage(){// 1. 创建一个队列. 创建10条消息,进行插入到队列MSQueue expectQueue = createTestQueue("testQueue");List<Message> expectMessage = new ArrayList<>();for (int i = 0; i < 10; i++) {Message message = createTestMessage("testMessage" + i);memoryDataCenter.sendMessage(expectQueue,message);expectMessage.add(message);}// 2.从队列进行取出消息List<Message> actualMessage = new ArrayList<>();while (true){Message message = memoryDataCenter.pollMessage("testQueue");if (message == null){break;}actualMessage.add(message);}// 3. 比较消息前后是否一致Assertions.assertEquals(expectMessage.size(),actualMessage.size());for (int i = 0; i < expectMessage.size(); i++) {Assertions.assertEquals(expectMessage.get(i),actualMessage.get(i));}}/*** 6. 测试未被确认的消息*/@Testpublic void testMessageWaitAck(){// 1. 创建消息,插入到未被确认的队列中Message expectedMessage = createTestMessage("expectedMessage");memoryDataCenter.addMessageWaitAck("testQueue", expectedMessage);// 2. 获取消息从未被确认的队列中Message actualMessage = memoryDataCenter.geMessageWaitAck("testQueue", expectedMessage.getMessageID());Assertions.assertEquals(expectedMessage, actualMessage);// 3. 从未被确认的队列中进行删除消息memoryDataCenter.removeMessageWaitAck("testQueue", expectedMessage.getMessageID());// 4. 比较删除之后的队列是否还有消息actualMessage = memoryDataCenter.geMessageWaitAck("testQueue", expectedMessage.getMessageID());Assertions.assertNull(actualMessage);}/*** 7. 测试从硬盘中恢复数据到内存*/@Testpublic void testRecovery() throws IOException, MqException, ClassNotFoundException {// 由于后续需要进行数据库操作, 依赖 MyBatis. 就需要先启动 SpringApplication, 这样才能进行后续的数据库操作.DemoApplication.context = SpringApplication.run(DemoApplication.class);// 1. 在硬盘上构造好数据DiskDataCenter diskDataCenter = new DiskDataCenter();diskDataCenter.init();// 构造交换机Exchange expectedExchange = createTestExchange("testExchange");diskDataCenter.insertExchange(expectedExchange);// 构造队列MSQueue expectedQueue = createTestQueue("testQueue");diskDataCenter.insertQueue(expectedQueue);// 构造绑定Binding expectedBinding = new Binding();expectedBinding.setExchangeName("testExchange");expectedBinding.setQueueName("testQueue");expectedBinding.setBindingKey("testBindingKey");diskDataCenter.insertBinding(expectedBinding);// 构造消息Message expectedMessage = createTestMessage("testContent");diskDataCenter.sendMessage(expectedQueue, expectedMessage);// 2. 执行恢复操作memoryDataCenter.recovery(diskDataCenter);// 3. 对比结果Exchange actualExchange = memoryDataCenter.getExchange("testExchange");Assertions.assertEquals(expectedExchange.getName(), actualExchange.getName());Assertions.assertEquals(expectedExchange.getType(), actualExchange.getType());Assertions.assertEquals(expectedExchange.isDurable(), actualExchange.isDurable());Assertions.assertEquals(expectedExchange.isAutoDelete(), actualExchange.isAutoDelete());MSQueue actualQueue = memoryDataCenter.getQueue("testQueue");Assertions.assertEquals(expectedQueue.getName(), actualQueue.getName());Assertions.assertEquals(expectedQueue.isDurable(), actualQueue.isDurable());Assertions.assertEquals(expectedQueue.isAutoDelete(), actualQueue.isAutoDelete());Assertions.assertEquals(expectedQueue.isExclusive(), actualQueue.isExclusive());Binding actualBinding = memoryDataCenter.getBinding("testExchange", "testQueue");Assertions.assertEquals(expectedBinding.getExchangeName(), actualBinding.getExchangeName());Assertions.assertEquals(expectedBinding.getQueueName(), actualBinding.getQueueName());Assertions.assertEquals(expectedBinding.getBindingKey(), actualBinding.getBindingKey());Message actualMessage = memoryDataCenter.pollMessage("testQueue");Assertions.assertEquals(expectedMessage.getMessageID(), actualMessage.getMessageID());Assertions.assertEquals(expectedMessage.getRoutingKey(), actualMessage.getRoutingKey());Assertions.assertEquals(expectedMessage.getDeliverMode(), actualMessage.getDeliverMode());Assertions.assertArrayEquals(expectedMessage.getBody(), actualMessage.getBody());// 4. 清理硬盘的数据, 把整个 data 目录里的内容都删掉(包含了 meta.db 和 队列的目录).DemoApplication.context.close();File dataDir = new File("./data");FileUtils.deleteDirectory(dataDir);}
}

---

结语

        以上内容就是针对内存管理的封装,主要是设计了6中数据机构进行存储交换机 队列 绑定 消息 消息和队列的映射 未确认信息.后续对数据进行操作的时候会更加具有效率.这样我们虚拟主机中两大核心部分:硬盘管理和内存管理都总结完成,下一节会对上述两种操作进一步封装到(VirtualHost)中,然后正式的提出消息队列服务器BrokerServer这个概念,对其进行完善和功能封装.请持续关注,谢谢!!!

完整的项目代码已上传Gitee,欢迎大家访问.👇👇👇

模拟实现消息队列https://gitee.com/yao-fa/advanced-java-ee/tree/master/My-mq