kubesphere中间件部署

微服务部署前中间件部署

一、MySQL部署

1.1 使用Docker实现MySQL主从复制

docker run -p 3307:3306 --name mysql-master \
-v /mydata/mysql/master/log:/var/log/mysql \
-v /mydata/mysql/master/data:/var/lib/mysql \
-v /mydata/mysql/master/conf:/etc/mysql \
-e MySQL_ROOT_PASSWORD=root \
-d mysql:5.7 

vim /mydata/mysql/master/conf/my.cnf[client]
default-character-set=utf8[mysql]
default-character-set=utf8[mysqld]
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake
skip-name-resolve

skip-name-resolve一定要加，不然连接mysql 会慢

server_id=1
log_bin=mysql-bin
read-only=0
binlog-do-db=mall_oms
binlog-do-db=mall_pms
binlog-do-db=mall_sms
binlog-do-db=mall_ums
binlog-do-db=mall_wmsreplicate-ignore-db=mysql
replicate-ignore-db=sys
replicate-ignore-db=information_schema
replicate-ignore-db=performance_schema

docker run -p 3317:3306 --name mysql-slaver-01 \
-v /mydata/mysql/slaver/log:/var/log/mysql \
-v /mydata/mysql/slaver/data:/var/lib/mysql \
-v /mydata/mysql/slaver/conf:/etc/mysql \
-e MYSQL_ROOT_PASSWORD=root \
-d mysql:5.7

vim /mydata/mysql/slaver/conf/my.cnf[client]
default-character-set=utf8[mysql]
default-character-set=utf8[mysqld]
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake
skip-name-resolve

server_id=2
log_bin=mysql-bin
read-only=1
binlog-do-db=mall_oms
binlog-do-db=mall_pms
binlog-do-db=mall_sms
binlog-do-db=mall_ums
binlog-do-db=mall_wmsreplicate-ignore-db=mysql
replicate-ignore-db=sys
replicate-ignore-db=information_schema
replicate-ignore-db=performance_schema

docker restart mysql-master mysql-slaver-01

为master授权用户实现数据同步

进入master容器
docker exec -it mysql /bin/bash访问mysql数据库
mysql -uroot -proot授权root可以远程访问，为了方便连接使用
grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;
flush privileges;添加用于同步用户
grant replication slave on *.* to 'backup'@'%' identified by '123456';查看master状态
show master status\G;

配置slaver同步master数据

进入slaver容器
docker exec -it mysql-slaver-01 /bin/bash访问mysql数据库
mysql -uroot -proot授权root可以远程访问，为了方便连接使用
grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;
flush privileges;设置主库连接
changer master to
master_host='192.168.56.10'
master_user='backup'
master_password='123456'
master_log_file='mysql-bin.000001'
master_log_pos=0
master_port=3307启动从库同步
start slave;查看从库的状态
show slave status\G;

可以使用kubesphere快速完成MySQL部署

• 有状态服务抽取配置为ConfigMap
• 有状态服务必须使用PVC持久化存储数据
• 服务集群内访问使用DNS提供稳定的域名

1.2 通过KubeSphere实现MySQL主从复制部署

1.2.1 持久存储准备

1.2.1.1 master节点存储

1.2.1.2 slave节点存储

1.2.2 配置文件准备

1.2.2.1 master节点配置文件

[client]
default-character-set=utf8[mysql]
default-character-set=utf8[mysqld]
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake
skip-name-resolveserver_id=1
log_bin=mysql-bin
read-only=0
binlog-do-db=mall_oms
binlog-do-db=mall_pms
binlog-do-db=mall_sms
binlog-do-db=mall_ums
binlog-do-db=mall_wmsreplicate-ignore-db=mysql
replicate-ignore-db=sys
replicate-ignore-db=information_schema
replicate-ignore-db=performance_schema

1.2.2.2 slave节点配置文件

[client]
default-character-set=utf8[mysql]
default-character-set=utf8[mysqld]
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake
skip-name-resolveserver_id=2
log_bin=mysql-bin
read-only=1
binlog-do-db=mall_oms
binlog-do-db=mall_pms
binlog-do-db=mall_sms
binlog-do-db=mall_ums
binlog-do-db=mall_wmsreplicate-ignore-db=mysql
replicate-ignore-db=sys
replicate-ignore-db=information_schema
replicate-ignore-db=performance_schema

1.2.3 mysql管理员root用户密码

1.2.4 master部署

# dig -t a mysql-master.sangomall.svc.cluster.local. @10.96.0.10

1.2.5 slaver部署

# dig -t a mysql-slave.sangomall.svc.cluster.local. @10.96.0.10

1.2.6 MySQL主从复制

1.2.6.1 MySQL Master节点授权

mysql>grant replication slave on *.* to 'backup'@'%' identified by '123456';

mysql>show master status\G;

1.2.6.2 MySQL Slave配置

mysql>change master to
master_host='mysql-master.sangomall.svc.cluster.local.',
master_user='backup',
master_password='123456',
master_log_file='mysql-bin.000003',
master_log_pos=439,
master_port=3306;

mysql>start slave;

mysql> show slave status\G;
*************************** 1. row ***************************Slave_IO_State: Waiting for master to send eventMaster_Host: mysql-master.sangomall.svc.cluster.local.Master_User: backupMaster_Port: 3306Connect_Retry: 60Master_Log_File: mysql-bin.000003Read_Master_Log_Pos: 439Relay_Log_File: mysql-slave-v1-0-relay-bin.000002Relay_Log_Pos: 652Relay_Master_Log_File: mysql-bin.000003Slave_IO_Running: Yes  一定要为YesSlave_SQL_Running: Yes  一定要为YesReplicate_Do_DB:

1.2.6.3 验证

二、Redis部署

2.1 准备配置PVC

2.2 准备配置文件

2.3 部署Redis

# dig -t a redis.sangomall.svc.cluster.local. @10.96.0.10

三、ES&Kibana部署

3.1 elasticsearch pvc准备

3.2 elasticsearch 配置文件准备

http.host: 0.0.0.0
discovery.type: single-node
ES_JAVA_OPTS: -Xms64m -Xmx512m

3.3 elasticsearch 部署

3.4 kibana部署

ELASTICSEARCH_HOSTS=http://elasticsearch.sangomall.svc.cluster.local.:9200

3.5 kibana访问

lb.kubesphere.io/v1alpha1: openelb
protocol.openelb.kubesphere.io/v1alpha1: layer2
eip.openelb.kubesphere.io/v1alpha2: layer2-eip

[root@dnsserver ~]# cat /var/named/msb.com.zone
$TTL 1D
@       IN SOA  msb.com admin.msb.com. (0       ; serial1D      ; refresh1H      ; retry1W      ; expire3H )    ; minimum
@       NS      ns.msb.com.
ns      A       192.168.10.145
harbor  A       192.168.10.146
reg-test        A       192.168.10.72
kibana  A       192.168.10.73

[root@dnsserver ~]# systemctl restart named

四、RabbitMQ部署

4.1 RabbitMQ持久存储准备 PVC

4.2 RabbitMQ部署

4.3 RabbitMQ访问

五、Nacos部署

5.1 Nacos Server数据持久存储 PVC

5.2 Nacos Server部署

MODE: standalone

# dig -t a nacos-server.sangomall.svc.cluster.local. @10.96.0.10

5.3 Nacos Server访问

[root@dnsserver ~]# cat /var/named/msb.com.zone
$TTL 1D
@       IN SOA  msb.com admin.msb.com. (0       ; serial1D      ; refresh1H      ; retry1W      ; expire3H )    ; minimum
@       NS      ns.msb.com.
ns      A       192.168.10.145
harbor  A       192.168.10.146
nacos-server    A       192.168.10.72 添加主机记录

[root@dnsserver ~]# systemctl restart named

六、链路跟踪服务 zipkin

6.1 依赖服务检查

# dig -t a elasticsearch.sangomall.svc.cluster.local. @10.96.0.10

6.2 zipkin部署

STORAGE_TYPE: elasticsearch
ES_HOSTS: elasticsearch.sangomall.svc.cluster.local.:9200

# dig -t a zipkin-server.sangomall.svc.cluster.local. @10.96.0.10

6.3 zipkin访问

[root@dnsserver ~]# cat /var/named/msb.com.zone
$TTL 1D
@       IN SOA  msb.com admin.msb.com. (0       ; serial1D      ; refresh1H      ; retry1W      ; expire3H )    ; minimum
@       NS      ns.msb.com.
ns      A       192.168.10.145
harbor  A       192.168.10.146zipkin-server    A       192.168.10.72 添加主机记录

[root@dnsserver ~]# systemctl restart named

七、sentinel 流量卫兵

7.1 获取容器镜像

7.2 sentinel部署

bladex/sentinel-dashboard:latest

#  dig -t a sentinel-server.sangomall.svc.cluster.local. @10.96.0.10

7.3 sentinel访问

[root@dnsserver ~]# cat /var/named/msb.com.zone
$TTL 1D
@       IN SOA  msb.com admin.msb.com. (0       ; serial1D      ; refresh1H      ; retry1W      ; expire3H )    ; minimum
@       NS      ns.msb.com.
ns      A       192.168.10.145
harbor  A       192.168.10.146sentinel-server    A       192.168.10.72 添加主机记录

[root@dnsserver ~]# systemctl restart named

八、Skywalking部署

8.1 获取容器镜像方法及ES服务确认

8.1.1 获取Skywalking oap server及Skywalking ui容器镜像

8.1.2 elasticsearch服务确认

elasticsearch.sangomall.svc.cluster.local.

8.2 Skywalking oap server部署

CPU及内存限制要注意：CPU 500m至1000m，内存 100M至2000M

SW_STORAGE  elasticsearchSW_STORAGE_ES_CLUSTER_NODES    elasticsearch.sangomall.svc.cluster.local.:9200

# dig -t a skywalking-oap-server.sangomall.svc.cluster.local. @10.96.0.10

8.3 Skywalking ui部署

SW_OAP_ADDRESS: http://skywalking-oap-server.sangomall.svc.cluster.local.:12800

# dig -t a skywalking-ui.sangomall.svc.cluster.local. @10.96.0.10

8.4 Skywalking ui访问

[root@dnsserver ~]# cat /var/named/msb.com.zone
$TTL 1D
@       IN SOA  msb.com admin.msb.com. (0       ; serial1D      ; refresh1H      ; retry1W      ; expire3H )    ; minimum
@       NS      ns.msb.com.
ns      A       192.168.10.145
harbor  A       192.168.10.146
reg-test        A       192.168.10.72
kibana          A       192.168.10.72
rabbitmq        A       192.168.10.72
nacos-server    A       192.168.10.72
zipkin-server   A       192.168.10.72
sentinel-server A       192.168.10.72
skywalking-ui   A       192.168.10.72

[root@dnsserver ~]# systemctl restart named

九、RocketMQ部署

9.1 rocketmq namesrv存储准备

9.2 rocketmq namesrv部署

启动命令:/bin/bash参数:mqnamesrv

环境变量
JAVA_OPT_EXT: -Xms512M -Xmx512M -Xmn128m

dig -t a  rocketmq-namesrv.sangomall.svc.cluster.local. @10.96.0.10

9.3 rocketmq broker存储准备

9.4 rocketmq broker部署

启动命令:
/bin/bash参数:
mqbroker,-n,rocketmq-namesrv.sangomall.svc.cluster.local.:9876

JAVA_OPT_EXT: -server -Xms128m -Xmx128m -Xmn128m
NAMESRV_ADDR: rocketmq-namesrv.sangomall.svc.cluster.local.:9876

9.5 rocketmq dashboard部署

JAVA_OPTS: -Drocketmq.namesrv.addr=rocketmq-namesrv.sangomall.svc.cluster.local.:9876

9.6 rocketmq dashboard创建应用路由

9.7 rocketmq dashboard访问

[root@dnsserver ~]# cat /var/named/msb.com.zone
$TTL 1D
@       IN SOA  msb.com admin.msb.com. (0       ; serial1D      ; refresh1H      ; retry1W      ; expire3H )    ; minimum
@       NS      ns.msb.com.
ns      A       192.168.10.145
harbor  A       192.168.10.146
reg-test        A       192.168.10.72
kibana          A       192.168.10.72
rabbitmq        A       192.168.10.72
nacos-server    A       192.168.10.72
zipkin-server   A       192.168.10.72
sentinel-server A       192.168.10.72
skywalking-ui   A       192.168.10.72
rocketmq-dashboard      A       192.168.10.72

[root@dnsserver ~]# systemctl restart named