美团CPS分销系统中Java接口高并发下的性能瓶颈排查与优化技巧

美团CPS分销系统中Java接口高并发下的性能瓶颈排查与优化技巧在美团CPS分销系统中订单回调、佣金计算、分佣发放等核心接口在午晚高峰面临每秒数千QPS的压力。若未提前识别和优化性能瓶颈极易出现RT飙升、线程阻塞甚至服务雪崩。本文结合Arthas、Prometheus、SkyWalking等工具从代码、JVM、数据库三个层面提供可落地的排查与优化方案。1. 使用Arthas实时诊断热点方法通过trace命令定位慢方法# 启动Arthasjava-jararthas-boot.jar# 跟踪分销计算入口trace baodanbao.com.cn.cps.service.CommissionService calculateCommissionDistribution若发现calculateRule()耗时占比80%则重点优化该逻辑。2. 避免同步锁竞争无锁化设计原始代码使用synchronized导致线程排队// ❌ 反例全局锁publicsynchronizedvoidupdateBalance(LonguserId,BigDecimalamount){UserAccountaccountaccountMapper.selectById(userId);account.setBalance(account.getBalance().add(amount));accountMapper.updateById(account);}改为CAS重试或数据库乐观锁// ✅ 优化数据库版本控制publicvoidupdateBalanceOptimistic(LonguserId,BigDecimalamount){intretries0;while(retries3){UserAccountaccountaccountMapper.selectForUpdate(userId);// SELECT ... FOR UPDATEBigDecimalnewBalanceaccount.getBalance().add(amount);intupdatedaccountMapper.updateBalanceWithVersion(userId,newBalance,account.getVersion());if(updated0)break;// 成功try{Thread.sleep(10);}catch(InterruptedExceptione){/* ignore */}}}Mapper XMLupdateidupdateBalanceWithVersionUPDATE user_account SET balance #{newBalance}, version version 1 WHERE user_id #{userId} AND version #{version}/update3. 异步化非核心链路将日志、通知、埋点等操作异步化ServicepublicclassOrderCallbackService{AutowiredprivateExecutorServiceasyncTaskExecutor;publicvoidhandleOrderCallback(OrderCallbackDTOdto){// 1. 核心校验并计算佣金CommissionResultresultcommissionCalculator.calculate(dto);// 2. 异步记录审计日志、推送消息asyncTaskExecutor.execute(()-{baodanbao.com.cn.cps.audit.AuditLogger.log(result);baodanbao.com.cn.cps.notify.MessageSender.send(result.getUserId(),佣金到账);});}}4. 数据库连接池与SQL优化检查HikariCP指标# application.ymlspring:datasource:hikari:maximum-pool-size:40metric-registry:io.micrometer.core.instrument.Metrics.globalRegistry通过/actuator/metrics/hikaricp.connections.active监控活跃连接数。优化慢SQL为order_id、user_id、status添加联合索引ALTERTABLEcps_commission_recordADDINDEXidx_user_status_time(user_id,status,create_time);避免SELECT *只查必要字段Select(SELECT order_id, amount, status FROM cps_commission_record WHERE user_id #{userId})ListSimpleCommissionselectSimpleByUserId(Param(userId)LonguserId);5. 缓存击穿防护与本地缓存对高频访问的分销规则使用CaffeineRedis二级缓存ComponentpublicclassDistributeRuleCache{privatefinalCacheLong,DistributeRulelocalCacheCaffeine.newBuilder().maximumSize(1000).expireAfterWrite(10,TimeUnit.MINUTES).build();publicDistributeRulegetRule(LongmerchantId){DistributeRulerulelocalCache.getIfPresent(merchantId);if(rule!null)returnrule;StringredisKeydistribute:rule:merchantId;ObjectobjredisTemplate.opsForValue().get(redisKey);if(objinstanceofDistributeRule){localCache.put(merchantId,(DistributeRule)obj);return(DistributeRule)obj;}// 回源DB带Redis分布式锁防击穿ruleloadFromDbWithLock(merchantId);if(rule!null){redisTemplate.opsForValue().set(redisKey,rule,30,TimeUnit.MINUTES);localCache.put(merchantId,rule);}returnrule;}}6. JVM GC调优与内存分析观察GC日志若Young GC频繁且耗时长说明对象生命周期过短或Eden区过小# 启动参数-Xms8g-Xmx8g-XX:UseG1GC-XX:MaxGCPauseMillis200-Xlog:gc*:filegc.log:time使用MAT分析堆转储查找大对象或内存泄漏// 避免在循环中创建大集合publicListOrderStatprocessOrders(ListOrderorders){// ❌ 反例每次循环new ArrayList// ✅ 正确预分配容量ListOrderStatstatsnewArrayList(orders.size());for(Ordero:orders){stats.add(convert(o));}returnstats;}7. 接口限流与熔断使用Sentinel保护核心接口PostConstructpublicvoidinitFlowRules(){ListFlowRulerulesnewArrayList();FlowRulerulenewFlowRule(commission_callback_api).setGrade(RuleConstant.FLOW_GRADE_QPS).setCount(2000);// QPS上限2000rules.add(rule);FlowRuleManager.loadRules(rules);}SentinelResource(valuecommission_callback_api,blockHandlerhandleBlocked)publicResponseEntity?handleCallback(RequestBodyCallbackDTOdto){// 业务逻辑}publicResponseEntity?handleBlocked(BlockExceptionex){returnResponseEntity.status(429).body(Too many requests);}本文著作权归 俱美开放平台 转载请注明出处