如何给ELK日志加上索引

问题记录

1、遇到长流程的时候，日志记录是非常重要的。如何排查日志，可以在MDC中去put对应的值，这样就等于对你关心的关键字段加上了索引，在elk中可以通过该索引就能 容易排查到问题

logback的设置 可以参照：

【总体日志】

<?xml version="1.0" encoding="UTF-8"?>
<configuration><property name="CHARSET" value="UTF-8" /><!-- 使用SkyWalking提供的traceID --><conversionRule conversionWord="tid" converterClass="org.apache.skywalking.apm.toolkit.log.logback.v1.x.LogbackPatternConverter"/><!-- 使用application.yml配置 --><springProperty scope="context" name="LOG_HOME" source="logging.file.path" /><property name="LOG_PATTERN" value="%d{yyyy-MM-dd HH:mm:ss:SSS} %highlight([%tid] [%X{X-B3-TraceId}] [%p])  %boldYellow(%c{36}) - %m%n" /><include resource="logback/logback-project.xml"/>
<!--    <include resource="logback/logback-request.xml"/>--><include resource="logback/logback-error.xml"/><!-- 总日志输出到控制台 --><appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"><encoder><pattern>${LOG_PATTERN}</pattern><charset>${CHARSET}</charset></encoder></appender><root level="INFO"><appender-ref ref="STDOUT" /><appender-ref ref="PROJECT_APPENDER" /><appender-ref ref="ERROR_APPENDER" /></root>
</configuration>

【业务日志】

<!-- 这里配置项目业务日志-->
<included><!-- 项目业务日志Appender --><appender name="PROJECT_APPENDER" class="ch.qos.logback.core.rolling.RollingFileAppender"><file>${LOG_HOME}/project.log</file><rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"><fileNamePattern>${LOG_HOME}/project-%d{yyyyMMdd}.%i.log</fileNamePattern><timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"><maxFileSize>500MB</maxFileSize></timeBasedFileNamingAndTriggeringPolicy></rollingPolicy><filter class="ch.qos.logback.classic.filter.LevelFilter"><level>ERROR</level><onMatch>DENY</onMatch><onMismatch>ACCEPT</onMismatch></filter><append>true</append><encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"><providers><pattern><pattern>{"logtime":"%date{yyyy-MM-dd HH:mm:ss.SSS}","traceId":"%tid","level":"%level","seq":"%X{seq}","offset":"%X{offset}","partition":"%X{partition}","threadName":"%thread","className":"%class","content":"%message","stackTrace":"%exception{10}"}</pattern></pattern></providers></encoder></appender><logger name="com.myhexin.zixun.cmp.review.server" level="DEBUG"><appender-ref ref="PROJECT_APPENDER" /></logger>
</included>

【aop切面类】

import cn.hutool.json.JSONUtil;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang.StringUtils;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.slf4j.MDC;
import org.springframework.stereotype.Component;import java.util.UUID;/*** @description:* @author: [e-mail]* @date: 2021/6/3  10:58**/@Aspect
@Component
@Slf4j
public class NewsOfflineAspect {private final String POINT_CUT = "execution(* com.myhexin.zixun.server.content.operate.deploy.domain.news.service.impl.OfflineOrOnlineServiceImpl.offlineOrOnlineNews(..))";@Pointcut(POINT_CUT)public void pointCut() {}@Around(value = "pointCut()")public Object doAround(ProceedingJoinPoint joinPoint) throws Throwable {Object[] args = joinPoint.getArgs();String s = String.valueOf(args[0]);String seq = StringUtils.substringBetween(s, "seq=", ")");if (StringUtils.isNotEmpty(seq)) {MDC.put("seq", seq);} else {MDC.put("seq", "Random-" + UUID.randomUUID().toString());}try {return joinPoint.proceed();} catch (Throwable throwable) {log.error(">>>>入参 {} <<<<<", JSONUtil.toJsonStr(args), throwable);throw throwable;} finally {MDC.remove("seq");}}
}

业务日志里的log patter放了seq 字段，那就可以在进入aop的时候 打上seq作为日志的字段索引，然后在处理流程结束的时候，拿出seq字段索引，这样排查的时候，就可以根据这个字段来排查了。
同样的，如果是kafka的接受类，你可以把 offset和partition作为字段索引，也是及其好用的。