多级缓存架构(三)OpenResty Lua缓存

通过本文章，可以完成多级缓存架构中的Lua缓存。

一、nginx服务

在docker/docker-compose.yml中添加nginx服务块。

  nginx:container_name: nginximage: nginx:stablevolumes:- ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf- ./nginx/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf- ./nginx/dist:/usr/share/nginx/distports:- "8080:8080"networks:multi-cache:ipv4_address: 172.30.3.3

删除原来docker里的multiCache项目并停止springboot应用。

nginx部分配置如下，监听端口为8080，并且将请求反向代理至172.30.3.11，下一小节，将openresty固定在172.30.3.11。

upstream nginx-cluster {server 172.30.3.11;
}server {listen       8080;listen  [::]:8080;server_name  localhost;location /api {proxy_pass http://nginx-cluster;}
}

重新启动multiCache看看nginx前端网页效果。

docker-compose -p multi-cache up -d

访问http://localhost:8080/item.html?id=10001查询id=10001商品页

这里是假数据，前端页面会向/api/item/10001发送数据请求。

二、OpenResty服务

1. 服务块定义

在docker/docker-compose.yml中添加openresty1服务块。

  openresty1:container_name: openresty1image: openresty/openresty:1.21.4.3-3-jammy-amd64volumes:- ./openresty1/conf/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf- ./openresty1/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf- ./openresty1/lua:/usr/local/openresty/nginx/lua- ./openresty1/lualib/common.lua:/usr/local/openresty/lualib/common.luanetworks:multi-cache:ipv4_address: 172.30.3.11

2. 配置修改

前端向后端发送/api/item/10001请求关于id=10001商品信息。

根据nginx的配置内容，这个请求首先被nginx拦截，反向代理到172.30.3.11 （即openresty1）。

upstream nginx-cluster {server 172.30.3.11;
}server {location /api {proxy_pass http://nginx-cluster;}
}

openresty1收到的也是/api/item/10001，同时，openresty将/api/item/(\d+)请求代理到指定lua程序，在lua程序中完成数据缓存。

因此，openresty的conf/conf.d/default.conf如下

upstream tomcat-cluster {hash $request_uri;server 172.30.3.4:8081;
#     server 172.30.3.5:8081;
}server {listen       80;listen  [::]:80;server_name  localhost;# intercept /item and join lualocation ~ /api/item/(\d+) {default_type application/json;content_by_lua_file lua/item.lua;}# intercept lua and redirect to back-endlocation /path/ {rewrite ^/path/(.*)$ /$1 break;proxy_pass http://tomcat-cluster;}error_page   500 502 503 504  /50x.html;location = /50x.html {root   /usr/share/nginx/dist;}
}

conf/nginx.conf在http块最后添加3行，引入依赖。

    #lua 模块lua_package_path "/usr/local/openresty/lualib/?.lua;;";#c模块lua_package_cpath "/usr/local/openresty/lualib/?.so;;";#本地缓存lua_shared_dict item_cache 150m;

3. Lua程序编写

common.lua被挂载到lualib，表示可以被其他lua当做库使用，内容如下

-- 创建一个本地缓存对象item_cache
local item_cache = ngx.shared.item_cache;-- 函数，向openresty本身发送类似/path/item/10001请求，根据conf配置，将被删除/path前缀并代理至tomcat程序
local function read_get(path, params)local rsp = ngx.location.capture('/path'..path,{method = ngx.HTTP_GET,args = params,})if not rsp thenngx.log(ngx.ERR, "http not found, path: ", path, ", args: ", params);ngx.exit(404)endreturn rsp.body
end-- 函数，如果本地有缓存，使用缓存，如果没有代理到tomcat然后将数据存入缓存
local function read_data(key, expire, path, params)-- query local cachelocal rsp = item_cache:get(key)-- query tomcatif not rsp thenngx.log(ngx.ERR, "redis cache miss, try tomcat, key: ", key)rsp = read_get(path, params)end-- write into local cacheitem_cache:set(key, rsp, expire)return rsp
endlocal _M = {read_data = read_data
}return _M

item.lua是处理来自形如/api/item/10001请求的程序，内容如下

-- include
local commonUtils = require('common')
local cjson = require("cjson")-- get url params 10001
local id = ngx.var[1]
-- redirect item, 缓存过期时间1800s, 适合长时间不改变的数据
local itemJson = commonUtils.read_data("item:id:"..id, 1800,"/item/"..id,nil)
-- redirect item/stock, 缓存过期时间4s, 适合经常改变的数据
local stockJson = commonUtils.read_data("item:stock:id:"..id, 4 ,"/item/stock/"..id, nil)
-- json2table
local item = cjson.decode(itemJson)
local stock = cjson.decode(stockJson)
-- combine item and stock
item.stock = stock.stock
item.sold = stock.sold
-- return result
ngx.say(cjson.encode(item))

4. 总结

1. 这里lua对item(tb_item表)和stock(tb_stock表)两个信息都有缓存，并使用cjson库将两者合并后返回到前端。
2. 关于expire时效性的问题，如果后台改变了数据，但是openresty关于此数据的缓存未过期，前端得到的是旧数据。
3. 大致来说openresty = nginx + lua，不仅具有nginx反向代理的能力，还能介入lua程序进行扩展。

三、运行

到此为止，docker-compose.yml应该如下

version: '3.8'networks:multi-cache:driver: bridgeipam:driver: defaultconfig:- subnet: 172.30.3.0/24services:mysql:container_name: mysqlimage: mysql:8volumes:- ./mysql/conf/my.cnf:/etc/mysql/conf.d/my.cnf- ./mysql/data:/var/lib/mysql- ./mysql/logs:/logsports:- "3306:3306"environment:- MYSQL_ROOT_PASSWORD=1009networks:multi-cache:ipv4_address: 172.30.3.2nginx:container_name: nginximage: nginx:stablevolumes:- ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf- ./nginx/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf- ./nginx/dist:/usr/share/nginx/distports:- "8080:8080"networks:multi-cache:ipv4_address: 172.30.3.3openresty1:container_name: openresty1image: openresty/openresty:1.21.4.3-3-jammy-amd64volumes:- ./openresty1/conf/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf- ./openresty1/conf/conf.d/default.conf:/etc/nginx/conf.d/default.conf- ./openresty1/lua:/usr/local/openresty/nginx/lua- ./openresty1/lualib/common.lua:/usr/local/openresty/lualib/common.luanetworks:multi-cache:ipv4_address: 172.30.3.11

启动各项服务

docker-compose -p multi-cache up -d

启动springboot程序。

四、测试

清空openresty容器日志。
访问http://localhost:8080/item.html?id=10001
查看openresty容器日志，可以看到两次commonUtils.read_data都没有缓存，于是代理到tomcat，可以看到springboot日志出现查询相关记录。

2024-01-12 11:45:53 2024/01/12 03:45:53 [error] 7#7: *1 [lua] common.lua:99: read_data(): redis cache miss, try tomcat, key: item:id:10001, client: 172.30.3.3, server: localhost, request: "GET /api/item/10001 HTTP/1.0", host: "nginx-cluster", referrer: "http://localhost:8080/item.html?id=10001"
2024-01-12 11:45:53 2024/01/12 03:45:53 [error] 7#7: *1 [lua] common.lua:99: read_data(): redis cache miss, try tomcat, key: item:stock:id:10001 while sending to client, client: 172.30.3.3, server: localhost, request: "GET /api/item/10001 HTTP/1.0", host: "nginx-cluster", referrer: "http://localhost:8080/item.html?id=10001"
2024-01-12 11:45:53 172.30.3.3 - - [12/Jan/2024:03:45:53 +0000] "GET /api/item/10001 HTTP/1.0" 200 486 "http://localhost:8080/item.html?id=10001" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0"

再次访问此网址，强制刷新+禁用浏览器缓存+更换浏览器
间隔超过4s但小于1800s时，日志如下，只出现一次miss。

2024-01-12 11:48:04 2024/01/12 03:48:04 [error] 7#7: *4 [lua] common.lua:99: read_data(): redis cache miss, try tomcat, key: item:stock:id:10001, client: 172.30.3.3, server: localhost, request: "GET /api/item/10001 HTTP/1.0", host: "nginx-cluster", referrer: "http://localhost:8080/item.html?id=10001"
2024-01-12 11:48:04 172.30.3.3 - - [12/Jan/2024:03:48:04 +0000] "GET /api/item/10001 HTTP/1.0" 200 486 "http://localhost:8080/item.html?id=10001" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0"

再次访问此网址，强制刷新+禁用浏览器缓存+更换浏览器
间隔小于4s，日志如下，未出现miss。

2024-01-12 11:49:16 172.30.3.3 - - [12/Jan/2024:03:49:16 +0000] "GET /api/item/10001 HTTP/1.0" 200 486 "http://localhost:8080/item.html?id=10001" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0"

五、高可用集群

1. openresty

对于openresty高可用，可以部署多个openresty docker实例，并在nginx的docker/nginx/conf/conf.d/default.conf的upstream nginx-cluster将多个openresty地址添加进去即可。比如

upstream nginx-cluster {hash $request_uri;# hash $request_uri consistent;server 172.30.3.11;server 172.30.3.12;server 172.30.3.13;
}

多个openresty 无论是conf还是lua都保持一致即可。
并且使用hash $request_uri负载均衡作为反向代理策略，防止同一请求被多个实例缓存数据。

2. tomcat

对于springboot程序高可用，也是类似。可以部署多个springboot docker实例，并在openresty 的docker/openresty1/conf/conf.d/default.conf的upstream nginx-cluster将多个springboot地址添加进去即可。比如

upstream tomcat-cluster {hash $request_uri;server 172.30.3.4:8081;server 172.30.3.5:8081;
}