Elasticsearch文件存储

分析Elasticsearch Index文件是如何存储的？
主要是想看一下FST文件是以什么粒度创建的？

首先通过kibana找一个索引的shard，此处咱们就以logstash-2023.05.30索引为例

查看下shard分布情况

GET /_cat/shards/logstash-2023.05.30?vindex               shard prirep state      docs   store ip             node
logstash-2023.05.30 3     p      STARTED 1520736 408.1mb 10.138.40.73  10.138.40.73-node1
logstash-2023.05.30 5     p      STARTED 1520888 409.9mb 10.138.40.74  10.138.40.74-node1
logstash-2023.05.30 6     p      STARTED 1518331 408.2mb 10.138.40.221 10.138.40.221-node1
logstash-2023.05.30 4     p      STARTED 1518186 409.3mb 10.138.204.194 10.138.204.194-node1
logstash-2023.05.30 1     p      STARTED 1519231 408.8mb 10.138.40.220 10.138.40.220-node1
logstash-2023.05.30 2     p      STARTED 1519970 409.9mb 10.138.204.195 10.138.204.195-node1
logstash-2023.05.30 0     p      STARTED 1520024 410.6mb 10.138.204.193 10.138.204.193-node1

这里以位于10.138.204.193上的shard 0为例分析。

要找到存储目录先要找到index的id

GET /logstash-2023.05.30/_settings{"logstash-2023.05.30" : {"settings" : {"index" : {"codec" : "best_compression","routing" : {"allocation" : {"include" : {"_tier_preference" : "data_content"}}},"refresh_interval" : "60s","number_of_shards" : "7","provided_name" : "logstash-2023.05.30","creation_date" : "1685376005206","number_of_replicas" : "0","uuid" : "FYWtFGTIS2CLB8yJhFXG9g",//这里就是索引的id"version" : {"created" : "7130499"}}}}
}

登录机器，找到存储索引文件的对应目录

/data3/10.138.204.193-node1/nodes/0/indices/FYWtFGTIS2CLB8yJhFXG9g

展开一下该目录下的文件

root@prd-paas-es-01:/data3/10.138.204.193-node1/nodes/0/indices/FYWtFGTIS2CLB8yJhFXG9g# tree -C -s
.
├── [       4096]  0
│   ├── [      20480]  index
│   │   ├── [        158]  _17f.fdm
│   │   ├── [   25578562]  _17f.fdt
│   │   ├── [       1939]  _17f.fdx
│   │   ├── [       4636]  _17f.fnm
│   │   ├── [    7981735]  _17f.kdd
│   │   ├── [      20898]  _17f.kdi
│   │   ├── [        716]  _17f.kdm
│   │   ├── [    7945983]  _17f_Lucene80_0.dvd
│   │   ├── [       3916]  _17f_Lucene80_0.dvm
│   │   ├── [    6230127]  _17f_Lucene84_0.doc
│   │   ├── [    3875001]  _17f_Lucene84_0.pos
│   │   ├── [    7448815]  _17f_Lucene84_0.tim
│   │   ├── [     108786]  _17f_Lucene84_0.tip
│   │   ├── [       1637]  _17f_Lucene84_0.tmd
│   │   ├── [        593]  _17f.si
│   │   ├── [        158]  _3uv.fdm
│   │   ├── [   33652243]  _3uv.fdt
│   │   ├── [       2555]  _3uv.fdx
│   │   ├── [       4636]  _3uv.fnm
│   │   ├── [   10520395]  _3uv.kdd
│   │   ├── [      27689]  _3uv.kdi
│   │   ├── [        716]  _3uv.kdm
│   │   ├── [   10573208]  _3uv_Lucene80_0.dvd
│   │   ├── [       3916]  _3uv_Lucene80_0.dvm
│   │   ├── [    8298061]  _3uv_Lucene84_0.doc
│   │   ├── [    5154427]  _3uv_Lucene84_0.pos
│   │   ├── [    9716222]  _3uv_Lucene84_0.tim
│   │   ├── [     142063]  _3uv_Lucene84_0.tip
│   │   ├── [       1620]  _3uv_Lucene84_0.tmd
│   │   ├── [        593]  _3uv.si
│   │   ├── [        158]  _5bg.fdm
│   │   ├── [   16433011]  _5bg.fdt
│   │   ├── [       1259]  _5bg.fdx
│   │   ├── [       4636]  _5bg.fnm
│   │   ├── [    5158094]  _5bg.kdd
│   │   ├── [      13396]  _5bg.kdi
│   │   ├── [        716]  _5bg.kdm
│   │   ├── [    5140762]  _5bg_Lucene80_0.dvd
│   │   ├── [       3916]  _5bg_Lucene80_0.dvm
│   │   ├── [    4005897]  _5bg_Lucene84_0.doc
│   │   ├── [    2583880]  _5bg_Lucene84_0.pos
│   │   ├── [    4873082]  _5bg_Lucene84_0.tim
│   │   ├── [      70979]  _5bg_Lucene84_0.tip
│   │   ├── [       1593]  _5bg_Lucene84_0.tmd
│   │   ├── [        593]  _5bg.si
│   │   ├── [        158]  _60h.fdm
│   │   ├── [   24664753]  _60h.fdt
│   │   ├── [       1886]  _60h.fdx
│   │   ├── [       4636]  _60h.fnm
│   │   ├── [    7640438]  _60h.kdd
│   │   ├── [      19996]  _60h.kdi
│   │   ├── [        716]  _60h.kdm
│   │   ├── [    7754954]  _60h_Lucene80_0.dvd
│   │   ├── [       3916]  _60h_Lucene80_0.dvm
│   │   ├── [    6147241]  _60h_Lucene84_0.doc
│   │   ├── [    3998559]  _60h_Lucene84_0.pos
│   │   ├── [    7254035]  _60h_Lucene84_0.tim
│   │   ├── [     105673]  _60h_Lucene84_0.tip
│   │   ├── [       1719]  _60h_Lucene84_0.tmd
│   │   ├── [        593]  _60h.si
│   │   ├── [        200]  _7jq.fdm
│   │   ├── [   63208093]  _7jq.fdt
│   │   ├── [       4692]  _7jq.fdx
│   │   ├── [       4636]  _7jq.fnm
│   │   ├── [   19306117]  _7jq.kdd
│   │   ├── [      51562]  _7jq.kdi
│   │   ├── [        716]  _7jq.kdm
│   │   ├── [   20228561]  _7jq_Lucene80_0.dvd
│   │   ├── [       3916]  _7jq_Lucene80_0.dvm
│   │   ├── [   15606568]  _7jq_Lucene84_0.doc
│   │   ├── [    9581341]  _7jq_Lucene84_0.pos
│   │   ├── [   17383473]  _7jq_Lucene84_0.tim
│   │   ├── [     272615]  _7jq_Lucene84_0.tip
│   │   ├── [       1592]  _7jq_Lucene84_0.tmd
│   │   ├── [        593]  _7jq.si
│   │   ├── [        437]  _82w.cfe
│   │   ├── [    4489379]  _82w.cfs
│   │   ├── [        408]  _82w.si
│   │   ├── [        437]  _87w.cfe
│   │   ├── [    4932636]  _87w.cfs
│   │   ├── [        408]  _87w.si
│   │   ├── [        437]  _8ao.cfe
│   │   ├── [   13905317]  _8ao.cfs
│   │   ├── [        408]  _8ao.si
│   │   ├── [        437]  _8ls.cfe
│   │   ├── [   20181047]  _8ls.cfs
│   │   ├── [        408]  _8ls.si
│   │   ├── [        437]  _8nq.cfe
│   │   ├── [    1234712]  _8nq.cfs
│   │   ├── [        408]  _8nq.si
│   │   ├── [        437]  _8oa.cfe
│   │   ├── [     872798]  _8oa.cfs
│   │   ├── [        408]  _8oa.si
│   │   ├── [        437]  _8pp.cfe
│   │   ├── [    1593677]  _8pp.cfs
│   │   ├── [        408]  _8pp.si
│   │   ├── [        437]  _8r5.cfe
│   │   ├── [     914008]  _8r5.cfs
│   │   ├── [        408]  _8r5.si
│   │   ├── [        437]  _8rf.cfe
│   │   ├── [     940473]  _8rf.cfs
│   │   ├── [        408]  _8rf.si
│   │   ├── [        437]  _8rz.cfe
│   │   ├── [    1315312]  _8rz.cfs
│   │   ├── [        408]  _8rz.si
│   │   ├── [        437]  _8s9.cfe
│   │   ├── [    1121692]  _8s9.cfs
│   │   ├── [        408]  _8s9.si
│   │   ├── [        437]  _8sk.cfe
│   │   ├── [     243476]  _8sk.cfs
│   │   ├── [        408]  _8sk.si
│   │   ├── [       1678]  segments_6
│   │   └── [          0]  write.lock
│   ├── [       4096]  _state
│   │   ├── [        186]  retention-leases-2865.st
│   │   └── [        125]  state-0.st
│   └── [       4096]  translog
│       ├── [         55]  translog-29.tlog
│       └── [         88]  translog.ckp
└── [       4096]  _state└── [       1230]  state-2.st5 directories, 118 files

有了文件信息，我们再来看下，segment信息

GET /logstash-2023.05.30/_segments// 这里为了直观 只展示shard 0对应的segment
{"_shards": {"total": 7,"successful": 7,"failed": 0},"indices": {"logstash-2023.05.30": {"shards": {"0": [{"routing": {"state": "STARTED","primary": true,"node": "4hEWcF8hRFWTEkQxlKQmqg"},"num_committed_segments": 17,"num_search_segments": 17,"segments": {"_17f": {"generation": 1563,"num_docs": 210331,"deleted_docs": 0,"size_in_bytes": 59203502,"memory_in_bytes": 5140,"committed": true,"search": true,"version": "8.8.2","compound": false,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_3uv": {"generation": 4999,"num_docs": 278411,"deleted_docs": 0,"size_in_bytes": 78098502,"memory_in_bytes": 5140,"committed": true,"search": true,"version": "8.8.2","compound": false,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_5bg": {"generation": 6892,"num_docs": 132645,"deleted_docs": 0,"size_in_bytes": 38291972,"memory_in_bytes": 5140,"committed": true,"search": true,"version": "8.8.2","compound": false,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_60h": {"generation": 7793,"num_docs": 199809,"deleted_docs": 0,"size_in_bytes": 57599273,"memory_in_bytes": 5140,"committed": true,"search": true,"version": "8.8.2","compound": false,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_7jq": {"generation": 9782,"num_docs": 520420,"deleted_docs": 0,"size_in_bytes": 145654675,"memory_in_bytes": 5204,"committed": true,"search": true,"version": "8.8.2","compound": false,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_82w": {"generation": 10472,"num_docs": 15416,"deleted_docs": 0,"size_in_bytes": 4490224,"memory_in_bytes": 5140,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_87w": {"generation": 10652,"num_docs": 16837,"deleted_docs": 0,"size_in_bytes": 4933481,"memory_in_bytes": 5140,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8ao": {"generation": 10752,"num_docs": 48855,"deleted_docs": 0,"size_in_bytes": 13906162,"memory_in_bytes": 5140,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8ls": {"generation": 11152,"num_docs": 70903,"deleted_docs": 0,"size_in_bytes": 20181892,"memory_in_bytes": 5140,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8nq": {"generation": 11222,"num_docs": 3954,"deleted_docs": 0,"size_in_bytes": 1235557,"memory_in_bytes": 6924,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8oa": {"generation": 11242,"num_docs": 2785,"deleted_docs": 0,"size_in_bytes": 873643,"memory_in_bytes": 6820,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8pp": {"generation": 11293,"num_docs": 5194,"deleted_docs": 0,"size_in_bytes": 1594522,"memory_in_bytes": 7060,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8r5": {"generation": 11345,"num_docs": 2936,"deleted_docs": 0,"size_in_bytes": 914853,"memory_in_bytes": 6748,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8rf": {"generation": 11355,"num_docs": 2920,"deleted_docs": 0,"size_in_bytes": 941318,"memory_in_bytes": 6836,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8rz": {"generation": 11375,"num_docs": 4304,"deleted_docs": 0,"size_in_bytes": 1316157,"memory_in_bytes": 6820,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8s9": {"generation": 11385,"num_docs": 3647,"deleted_docs": 0,"size_in_bytes": 1122537,"memory_in_bytes": 6892,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}},"_8sk": {"generation": 11396,"num_docs": 657,"deleted_docs": 0,"size_in_bytes": 244321,"memory_in_bytes": 7620,"committed": true,"search": true,"version": "8.8.2","compound": true,"attributes": {"Lucene87StoredFieldsFormat.mode": "BEST_COMPRESSION"}}}}]}}}
}

对比segment与shard目录中文件可以看出，两者是一一对应的。

看下es及对应lucene的版本

GET /{"name" : "10.138.204.193-node1","cluster_name" : "elasticsearch","cluster_uuid" : "XWDyVuo6TgK4yUp2XWD3lw","version" : {"number" : "7.13.4","build_flavor" : "default","build_type" : "docker","build_hash" : "c5f60e894ca0c61cdbae4f5a686d9f08bcefc942","build_date" : "2021-07-14T18:33:36.673943207Z","build_snapshot" : false,"lucene_version" : "8.8.2","minimum_wire_compatibility_version" : "6.8.0","minimum_index_compatibility_version" : "6.0.0-beta1"},"tagline" : "You Know, for Search"
}

那么shard目录中各种后缀的文件具体是什么含义呢？下面来看下

截图出处:
https://lucene.apache.org/core/8_8_2/core/org/apache/lucene/codecs/lucene87/package-summary.html#package.description

从表格中可以看出与FST相关的文件后缀有:tip、tim,从这里就可以看出FST文件是以segment维度来创建的。