Triton推理服务器部署YOLOv8（onnxruntime后端和TensorRT后端）

一、Trition推理服务器基础知识

1）推理服务器设计概述

• 设计思想和特点
  1、支持多种机器学习框架
  
  2、支持多种部署场景

3、高性能推理

4、灵活的模型管理

5、可扩展性

6、强大的客户端支持

2）Trition推理服务器quickstart

（1）创建模型仓库（Create a model Repository）

（2）启动Triton （launching triton）并验证是否正常运行

cpu运行

$ docker run --rm -p8000:8000 -p8001:8001 -p8002:8002 -v/full/path/to/docs/examples/model_repository:/models nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver --model-repository=/models

GPU运行

命令

$ docker run --gpus=1 --rm -p8000:8000 -p8001:8001 -p8002:8002 -v/full/path/to/docs/examples/model_repository:/models nvcr.io/nvidia/tritonserver:<xx.yy>-py3 tritonserver --model-repository=/models

输出

+----------------------+---------+--------+
| Model                | Version | Status |
+----------------------+---------+--------+
| <model_name>         | <v>     | READY  |
| ..                   | .       | ..     |
| ..                   | .       | ..     |
+----------------------+---------+--------+
...
...
...
I1002 21:58:57.891440 62 grpc_server.cc:3914] Started GRPCInferenceService at 0.0.0.0:8001
I1002 21:58:57.893177 62 http_server.cc:2717] Started HTTPService at 0.0.0.0:8000
I1002 21:58:57.935518 62 http_server.cc:2736] Started Metrics Service at 0.0.0.0:8002

• 验证是否正常运行
  

$ curl -v localhost:8000/v2/health/ready
...
< HTTP/1.1 200 OK
< Content-Length: 0
< Content-Type: text/plain

（3）发送推理请求（send a inference request）

①从NGC拉去客户端库和示例
Use docker pull to get the client libraries and examples image from NGC.

$ docker pull nvcr.io/nvidia/tritonserver:<xx.yy>-py3-sdk

②拉取特定版本镜像的客户端
Where <xx.yy> is the version that you want to pull. Run the client image.

$ docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:<xx.yy>-py3-sdk

③开始请求模型获取前三个分类

$ /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg
Request 0, batch size 1
Image '/workspace/images/mug.jpg':15.346230 (504) = COFFEE MUG13.224326 (968) = CUP10.422965 (