cowa新的数据筛选代码

cowa新的数据筛选代码

代码地址： https://git.cowarobot.com/lhb/data_extracting

一阶段筛选

修改配置文件

config/common_stage.yaml

version: '3'
services:de:image: harbor.cowarobot.cn/lhb/data:crpilot2.5-torch2.2environment:- CRPILOT_INSTALL_VERSION=x86_v2.5-v2.5.669.6 #超过669.6会用gpu解码图像,慎重设置parallel-tasks,太大炸显存- NVIDIA_VISIBLE_DEVICES=all- NVIDIA_DRIVER_CAPABILITIES=all- GDK_SCALE- DISPLAY=unix$DISPLAY- GDK_DPI_SCALE- DAY_RANGE- EXIST_OK=1 #0则跳过已经清洗的包,1则不跳过,当为1时,不要使用多台机器同时清洗- USE_POSE_IN_BAG=0 #0使用localization.yaml生成pose,1则使用包中pose,包中没有pose则跳过- FILTER_DISTANCE=-1 #-1时不使用pose,可以清洗不包含gps和pose的包- FILTER_TIME=0.1
#      - MONGO_DB=ai-cowa-2d-extract-test
#      - MINIO_BUCKET=ai-cowa-2d-test- MONGO_DB=cowa-2D-extract- MINIO_BUCKET=ai-cowa-2d- DONE_KEY=_id #用于多机清洗时判断是否已经清洗过,清洗新的包设置为_id,重复清洗之前的包可以设置为任意不重复的值,比如done_240407- TL_PRIOR=/cowarobot/data_extracting/de/utils/tl_prior.csv
#    deploy:
#      resources:
#        reservations:
#          devices:
#            - driver: "nvidia"
#              count: "all"
#              capabilities: [ "gpu" ]# network_mode: hostvolumes:- /data:/data- /disk:/disk- /disk1:/disk1- /tmp/.X11-unix:/tmp/.X11-unixtty: truecommand: "bash -c 'source /cowarobot/start.sh && cd /cowarobot/data_extracting/ && unset HTTP_PROXY https_proxy HTTPS_PROXY http_proxy && git checkout main && git pull && \cd /cowarobot/data_extracting/ && \python3 scripts/run_common_stage1.py--type all--id all--year 2024--month all--day all--uuid 5c799adf-49dd-45e1-a475-eb2178d444c0--data-pathes /disk1/AIStudiodata/dcenter/TruthSystemData/ \/disk1/AIStudiodata/dcenter/EventData/ \/disk1/AIStudiodata/dcenter/PerceptionData/ \/disk1/AIStudiodata/dcenter/DebugData/--parallel-tasks 64--topics /camera/surround/front/image_raw \/camera/surround/back/image_raw \/camera/surround/left/image_raw \/camera/surround/right/image_raw \/camera/panorama/3/image_raw \/camera/traffic_light/image_raw \/camera/traffic_light/front/image_raw \/camera/traffic_light/back/image_raw \/camera/stereo/front/1/image_raw \/camera/stereo/front/2/image_raw \/camera/panorama/1/image_raw \/camera/panorama/2/image_raw \/camera/panorama/4/image_raw \/camera/work/right/image_raw \/camera/work/left/image_raw'"

运行

进入服务器

ssh 242

cd /disk/deepdata/diyun_workspace/work/python_project/new_data_extracting/data_extracting

然后运行

docker-compose -f config/common_stage1.yaml up

二阶段筛选

不进入docker即可运行

python de/tasks/traffic_light/stage2_traffic_light_filter_data.py -f config/traffic_light_filter.py