ceph rados对象存储索引残留问题排查与处理

问题现象

对象存储存储桶无法删除，检查发现生命周期过期后存储桶中有文件残留，未完全删除，但实际访问文件时为404，通过s3cmd无法删除对象，且无报错。

问题定位

检查bucket当前状态，发现桶内有大量object残留：

# radosgw-admin bucket stats --bucket=xxx
{"bucket": "xxx","zonegroup": "97cde842-9840-46d2-98f5-2f2ff2b356b9","placement_rule": "default-placement","explicit_placement": {"data_pool": "","data_extra_pool": "","index_pool": ""},"id": "a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242","marker": "a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242","index_type": "Normal","owner": "e638db8fbfbd4eeba23f4e7f386e2234","ver": "0#115872937,1#124262992,2#124096868,3#113933319,4#113752484,5#115978810,6#119804274,7#122088764,8#117238420,9#114292015,10#122281621,11#112069481,12#117514195,13#113249706,14#114992906,15#119836492,16#109344437,17#115584159,18#113904518,19#118475333,20#118442717,21#119161849,22#119636967,23#115195682,24#114825666,25#116687890,26#110362647,27#114200698,28#122991031,29#111118759,30#122337033,31#114870449,32#115295855,33#117904918,34#121221126,35#116571732,36#124130503,37#118485487,38#110489698,39#113517117,40#119301165,41#115423515,42#118595461,43#115347045,44#120019831,45#113194194,46#114868275,47#123410209,48#119010861,49#114866634,50#111604486,51#117146989,52#113854552,53#111169375,54#120663269,55#116393793,56#114043425,57#117807012,58#122349615,59#113390481,60#124332341,61#114732340,62#114933373,63#114692587,64#118841860,65#118823413,66#115203887,67#122762813,68#109180753,69#115918036,70#116907631,71#115166297,72#126285010,73#117308131,74#118488176,75#117960296,76#115163648,77#116434667,78#114904720,79#128437168,80#119323730,81#114563504,82#117850906,83#119937587,84#121641776,85#111226459,86#115483044,87#119067683,88#115415015,89#112365103,90#119033043,91#111300726,92#117803502,93#116570158,94#110132992,95#117143542,96#118852347,97#121028481,98#111041704,99#109977398,100#116800374,101#121455617,102#119020462,103#116583479,104#117083862,105#119048007,106#118643510,107#115722285,108#114712589,109#119526889,110#122204433,111#118918806,112#119785343,113#123881328,114#114785575,115#116062597,116#120837772,117#118009592,118#112875590,119#119865516,120#111902759,121#112887532,122#112571750,123#119188027,124#115919787,125#118978645,126#114970199,127#116637842","master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0","mtime": "2022-09-19 23:05:51.793737","max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#","usage": {"rgw.main": {"size": 57217593456,"size_actual": 57233543168,"size_utilized": 57217593456,"size_kb": 55876557,"size_kb_actual": 55892132,"size_kb_utilized": 55876557,"num_objects": 7896,"": {"size": 515899392,"size_actual": 515899392,"size_utilized": 515899392,"size_kb": 503808,"size_kb_actual": 503808,"size_kb_utilized": 503808,"num_objects": 123},"STANDARD": {"size": 56701694064,"size_actual": 56717643776,"size_utilized": 56701694064,"size_kb": 55372749,"size_kb_actual": 55388324,"size_kb_utilized": 55372749,"num_objects": 7773}},"rgw.multimeta": {"size": 0,"size_actual": 0,"size_utilized": 216,"size_kb": 0,"size_kb_actual": 0,"size_kb_utilized": 1,"num_objects": 6,"STANDARD": {"size": 0,"size_actual": 0,"size_utilized": 216,"size_kb": 0,"size_kb_actual": 0,"size_kb_utilized": 1,"num_objects": 6}}},"bucket_quota": {"enabled": false,"check_on_raw": false,"max_size": -1,"max_size_kb": -1,"max_objects": -1}
}

拉取残余文件列表，首先获取pool、bucket id和num_shards：

# 获取pool，ningbo1-zone1.rgw.buckets.index
rados lspools | grep "rgw.buckets.index"
# 获取bucket id，a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242
radosgw-admin bucket stats --bucket {bucket name} | egrep "id"
# 获取shard数量，128
radosgw-admin metadata get bucket.instance:{bucket name}:{bucket id} | grep num_shards

然后拉取文件列表：

for i in {0..127}
dorados -p ningbo1-zone1.rgw.buckets.index listomapkeys .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$i >> objects.txt
done

检查object状态，发现相应数据底层均已删除：

for object in `cat objects.txt`
doradosgw-admin object stat --bucket=xxx --object=$object
done

均报错ERROR: failed to stat object, returned error: (2) No such file or directory。
列出bucket索引，发现所有索引"exists": "true"，判断为索引残留问题，及索引存在但数据已删除。

问题处置

可通过rados rmomapkey删除无效索引：

for i in {0..127}
dofor object in `rados -p ningbo1-zone1.rgw.buckets.index listomapkeys .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$i`dorados -c /etc/ceph/ceph.conf -p ningbo1-zone1.rgw.buckets.index rmomapkey .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$i $objectdone
done

重新扫描修复桶状态（可多次执行）：

# radosgw-admin bucket check --fix --check-objects --bucket=xxx
[]
{}
{"existing_header": {"usage": {}},"calculated_header": {"usage": {}}
}

再次检查，发现桶已正常清空：

# radosgw-admin bucket stats --bucket=xxx
{"bucket": "xxx","zonegroup": "97cde842-9840-46d2-98f5-2f2ff2b356b9","placement_rule": "default-placement","explicit_placement": {"data_pool": "","data_extra_pool": "","index_pool": ""},"id": "a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242","marker": "a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242","index_type": "Normal","owner": "e638db8fbfbd4eeba23f4e7f386e2234","ver": "0#115872943,1#124262998,2#124096874,3#113933325,4#113752490,5#115978816,6#119804280,7#122088770,8#117238426,9#114292021,10#122281627,11#112069487,12#117514201,13#113249712,14#114992912,15#119836498,16#109344443,17#115584165,18#113904524,19#118475339,20#118442723,21#119161855,22#119636973,23#115195688,24#114825672,25#116687896,26#110362653,27#114200704,28#122991037,29#111118765,30#122337039,31#114870455,32#115295861,33#117904924,34#121221132,35#116571738,36#124130509,37#118485493,38#110489704,39#113517123,40#119301171,41#115423521,42#118595467,43#115347051,44#120019837,45#113194200,46#114868281,47#123410215,48#119010867,49#114866640,50#111604492,51#117146995,52#113854558,53#111169381,54#120663275,55#116393799,56#114043431,57#117807018,58#122349621,59#113390487,60#124332347,61#114732346,62#114933379,63#114692593,64#118841866,65#118823419,66#115203893,67#122762819,68#109180759,69#115918042,70#116907637,71#115166303,72#126285016,73#117308137,74#118488182,75#117960302,76#115163654,77#116434673,78#114904726,79#128437174,80#119323736,81#114563510,82#117850912,83#119937593,84#121641782,85#111226465,86#115483050,87#119067689,88#115415021,89#112365109,90#119033049,91#111300732,92#117803508,93#116570164,94#110132998,95#117143548,96#118852353,97#121028487,98#111041710,99#109977404,100#116800380,101#121455623,102#119020468,103#116583485,104#117083868,105#119048013,106#118643516,107#115722291,108#114712595,109#119526895,110#122204439,111#118918812,112#119785349,113#123881334,114#114785581,115#116062603,116#120837778,117#118009598,118#112875596,119#119865522,120#111902765,121#112887538,122#112571756,123#119188033,124#115919793,125#118978651,126#114970205,127#116637848","master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0","mtime": "2022-09-19 23:05:51.793737","max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#","usage": {},"bucket_quota": {"enabled": false,"check_on_raw": false,"max_size": -1,"max_size_kb": -1,"max_objects": -1}
}

处理完毕。