SpringBoot对接火山引擎大模型api实现图片识别与分析

一、前言

Spring AI实战初体验——实现可切换模型AI聊天助手-CSDN博客

如上，在上一篇博客，我们已经实现了spring ai对接本地大模型实现了聊天机器人，但是目前有个新需求：

• 上传某场所的图片，通过AI进行分析，描述图片里的内容以及存在的安全隐患
• 进一步通过AI分析场所的安全隐患如何治理，需要依据法律法规（联网）分析

最终效果如下所示：

由于目前了解到的本地大模型都无法实现上述的需求，于是这次借助了火山引擎平台来实现

https://console.volcengine.com/ark/

火山引擎目前新用户会赠送每个模型50万token的体验量，对于学习、测试用还是足够的

如下所示，本次对接的模型有 doubao-vision-pro（图片识别）和 deepseek-v3（联网分析）

整体的逻辑：

1. 先传入图片到doubao模型，分析图片里的场所和存在的隐患
2. 然后将1分析的文字结果传到deepseek-v3模型联网结合法律法规分析隐患的整改措施

二、创建应用

https://console.volcengine.com/ark/

如下所示，创建2个零代码应用

1. 图片识别

   

2. 联网分析

   

三、后端

1.SDK集成

如下图所示，火山引擎里部分模型像deepseek-v3是可以直接集成SDK来对接的

代码示例

@RestController
@RequestMapping("/huoShan")
public class HuoShanController {private final ArkService service;private final String imageAnalyzeBotId;public HuoShanController(@Value("${ai.ark.apiKey}") String apiKey, @Value("${ai.ark.base-url}") String baseUrl, @Value("${ai.ark.image-analyze-botId}") String imageAnalyzeBotId) {this.imageAnalyzeBotId = imageAnalyzeBotId;this.service = ArkService.builder().dispatcher(new Dispatcher()).connectionPool(new ConnectionPool(5, 1, TimeUnit.SECONDS)).baseUrl(baseUrl).apiKey(apiKey).build();}@PostMapping("/image/chat")public ResponseEntity<String> imageChat(@RequestBody String userMessage) {List<ChatMessage> messages = new ArrayList<>();messages.add(ChatMessage.builder().role(ChatMessageRole.SYSTEM).content("你是一个对中国法律法规有深入理解的专家").build());messages.add(ChatMessage.builder().role(ChatMessageRole.USER).content(userMessage).build());BotChatCompletionRequest chatCompletionRequest = BotChatCompletionRequest.builder().botId(imageAnalyzeBotId).messages(messages).build();BotChatCompletionResult chatCompletionResult = service.createBotChatCompletion(chatCompletionRequest);StringBuilder result = new StringBuilder();chatCompletionResult.getChoices().forEach(choice -> result.append(choice.getMessage().getContent()));return ResponseEntity.ok(result.toString());}
}

相关的apiKey、base-url、botId都可以从火山的API调用指南获取，获取完我们配置在application.yml里就可以从上面的代码获取

2.调用Rest API

有些模型例如doubao-pro-vision就没提供java SDK，所以需要采用直接调用rest api的方式来对接

代码示例

    @PostMapping("/notStream")public Mono<String> imageAnalysis(MultipartFile file) {return imageAiService.imageAnalysisNotStream(file);}

@Service
@Slf4j
public class ImageAiService {@Value("${ai.ark.image-botId}")private String MODEL;private final WebClient webClient;public ImageAiService(@Value("${ai.ark.apiKey}") String apiKey, @Value("${ai.ark.base-url}") String baseUrl) {this.webClient = WebClient.builder().clientConnector(new ReactorClientHttpConnector(HttpClient.create())).baseUrl(baseUrl).defaultHeader("Authorization", "Bearer "+ apiKey).build();public Mono<String> imageAnalysisNotStream(MultipartFile file) {// 压缩图片并转成 Base64 格式String base64 = ImageCompressor.compressImageFileToBase64UnderSize(file, 400, 400, 100);if (base64 == null || base64.isEmpty()) {log.error("图片压缩失败");return Mono.just("图片压缩失败");}// 构造请求体Map<String, Object> body = new HashMap<>();body.put("model", MODEL);// 非流式返回body.put("stream", false);body.put("stream_options", Map.of("include_usage", true));Map<String, Object> imageContent = Map.of("type", "image_url","image_url", Map.of("url", base64));Map<String, Object> message = Map.of("role", "user","content", List.of(imageContent));body.put("messages", List.of(message));// 调用非流式接口，直接返回拼接后的完整结果字符串return webClient.post().uri("/bots/chat/completions").contentType(MediaType.APPLICATION_JSON).bodyValue(body)// 此时接口返回的是 JSON 数据，所以指定 JSON 类型.accept(MediaType.APPLICATION_JSON).retrieve().bodyToMono(String.class).map(responseStr -> {try {// 解析返回结果，取出 assistant 返回的内容JsonNode jsonNode = new ObjectMapper().readTree(responseStr);// 此处根据实际返回结构调整解析逻辑JsonNode contentNode = jsonNode.path("choices").get(0).path("message").path("content");return contentNode.asText();} catch (Exception e) {log.error("解析返回结果异常", e);return "解析返回结果异常";}});}    }

注意：

上述调用火山引擎api都是非流式的，如果流式输出就把stream设置成true，再使用Flux类或SseEmitter类去接收返回就行，但是由于我流式输出得到的结果前端进行格式处理时候总是有问题，所以改用了非流式，等完整答案出来后再一次性处理格式化

/*** 压缩到不超过100KB的Base64编码*/
public static String compressImageFileToBase64UnderSize(MultipartFile file, int maxWidth, int maxHeight, int maxSizeKB) {try {// 读取 MultipartFile 图片BufferedImage originalImage = ImageIO.read(file.getInputStream());// 按比例缩放图片Image scaledImage = originalImage.getScaledInstance(maxWidth, maxHeight, Image.SCALE_SMOOTH);BufferedImage resizedImage = new BufferedImage(maxWidth, maxHeight, BufferedImage.TYPE_INT_RGB);Graphics2D g2d = resizedImage.createGraphics();g2d.drawImage(scaledImage, 0, 0, null);g2d.dispose();// 获取JPEG写入器Iterator<ImageWriter> writers = ImageIO.getImageWritersByFormatName("jpg");if (!writers.hasNext()) throw new IllegalStateException("No writers found for jpg");ImageWriter writer = writers.next();ByteArrayOutputStream baos = new ByteArrayOutputStream();MemoryCacheImageOutputStream output = new MemoryCacheImageOutputStream(baos);writer.setOutput(output);// 设置初始压缩质量float quality = 1.0f;byte[] imageBytes;do {baos.reset();ImageWriteParam param = writer.getDefaultWriteParam();param.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);param.setCompressionQuality(quality);writer.write(null, new IIOImage(resizedImage, null, null), param);output.flush();imageBytes = baos.toByteArray();quality -= 0.05f; // 每次降低压缩质量} while (imageBytes.length > maxSizeKB * 1024 && quality > 0.05f);writer.dispose();output.close();//System.out.println("Final image size: " + (imageBytes.length / 1024) + " KB, final quality: " + quality);return "data:image/jpeg;base64," + Base64.getEncoder().encodeToString(imageBytes);} catch (IOException e) {e.printStackTrace();return null;}
}

注意：

Map<String, Object> imageContent = Map.of("type", "image_url","image_url", Map.of("url", base64)
);

这里的图片可以传递http/https网络地址或者图片的base64编码，由于我想是用电脑本地的文件来测试，所以采用图片转base64编码的方式来传递

四、前端

基于上次的html，新增了图片上传，图片回显，调用新接口等处理

直接放完整代码：

<!DOCTYPE html>
<html lang="zh">
<head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><title>AI 聊天</title><style>html, body {height: 100%;width: 100%;margin: 0;background-color: #f9f9f9;display: flex;align-items: center;justify-content: center;}.container {display: flex;flex-direction: column;height: 90vh;max-width: 800px;width: 100%;margin: auto;}.chat-container {flex: 1;display: flex;flex-direction: column;background: white;border-radius: 10px;padding: 20px;box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);overflow-y: auto; /* 确保内容超出时显示滚动条 */min-height: 0; /* 防止 flex 容器压缩子元素 */}.chat-container::-webkit-scrollbar {width: 8px;}.chat-container::-webkit-scrollbar-track {background: #f1f1f1;border-radius: 4px;}.chat-container::-webkit-scrollbar-thumb {background: #888;border-radius: 4px;}.chat-container::-webkit-scrollbar-thumb:hover {background: #555;}.ai-message h3 {font-size: 1.2em;margin-top: 1em;}.ai-message ul {padding-left: 1.5em;}.ai-message li {margin-bottom: 0.5em;}.ai-message {white-space: pre-wrap;}.loading-spinner {border: 4px solid #f3f3f3;border-top: 4px solid #007bff;border-radius: 50%;width: 30px;height: 30px;animation: spin 1s linear infinite;margin: 10px auto;}@keyframes spin {0% { transform: rotate(0deg); }100% { transform: rotate(360deg); }}.message {padding: 10px 15px;border-radius: 15px;margin: 5px 0;max-width: 80%;word-wrap: break-word;}.user-message {background-color: #007bff;color: white;align-self: flex-end;}.ai-message {background-color: #e5e5e5;color: black;align-self: flex-start;}.think-message {background-color: #add8e6;color: black;border-radius: 10px;padding: 10px;margin: 5px 0;max-width: 80%;align-self: flex-start;font-style: italic;}.think-content {flex: 1; /* 允许内容自由扩展 */overflow-y: auto; /* 内容过多时显示滚动条 */padding: 5px;}.think-title {font-weight: bold;margin-bottom: 5px;display: flex;align-items: center;}.toggle-button {padding: 5px 10px;background-color: #007bff;color: white;border: none;border-radius: 5px;cursor: pointer;margin-right: 10px;}.toggle-button:hover {background-color: #0056b3;}.input-container {display: flex;flex-direction: column;padding: 10px;background: white;box-shadow: 0 -2px 5px rgba(0, 0, 0, 0.1);}.model-container {display: flex;align-items: center;margin-bottom: 5px;}.model-label {margin-right: 10px;font-weight: bold;}.model-select {padding: 5px;border-radius: 5px;border: 1px solid #ccc;}.input-box-container {display: flex;align-items: center;}.input-box {flex: 1;padding: 10px;border: 1px solid #ccc;border-radius: 5px;}.send-button, .clear-button, .stop-button {padding: 10px 20px;margin-left: 10px;color: white;border: none;border-radius: 5px;cursor: pointer;}.send-button { background-color: #007bff; }.send-button:hover { background-color: #0056b3; }.send-button:disabled { background-color: #a0c4ff; cursor: not-allowed; }.clear-button { background-color: #dc3545; }.clear-button:hover { background-color: #a71d2a; }.clear-button:disabled { background-color: #f5a6a6; cursor: not-allowed; }.stop-button { background-color: #ff9800; }.stop-button:hover { background-color: #e68900; }.stop-button:disabled { background-color: #ffb74d; cursor: not-allowed; }</style><script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
</head>
<body>
<div class="container"><div class="chat-container" id="chatContainer"><div class="message ai-message">👋 你好，我是你的 AI 助手！</div></div><div class="input-container"><div class="model-container"><span class="model-label">选择模型:</span><select id="modelSelect" class="model-select" onchange="changeModel()"><option value="deepseek-r1:latest">DeepSeek-R1（推理）</option><option value="qwen:7b">Qwen</option><option value="image-analysis">火山引擎-Doubao（场所图片分析，无记忆）</option></select></div><div class="input-box-container"><input id="userInput" class="input-box" placeholder="请输入消息..."><input id="imageUpload" type="file" accept="image/jpeg, image/png" style="display: none;" onchange="validateFile()" /><button id="sendButton" class="send-button" onclick="handleSend()">发送</button><button id="clearButton" class="clear-button" onclick="clearMemory()">清除上下文</button><button id="stopButton" class="stop-button" onclick="stopAIResponse()">停止回答</button></div></div>
</div>
<script>const chatContainer = document.getElementById('chatContainer');const userInput = document.getElementById('userInput');const modelSelect = document.getElementById('modelSelect');const sendButton = document.getElementById('sendButton');const clearButton = document.getElementById('clearButton');const stopButton = document.getElementById('stopButton');let userId = '1';let currentModel = modelSelect.value;let eventSource = null;function validateFile() {const fileInput = document.getElementById('imageUpload');const file = fileInput.files[0];if (file) {const fileSize = file.size / 1024; // 文件大小，单位为KBconst fileType = file.type.toLowerCase();// 判断文件大小是否小于200KB，格式是否为JPG或PNGif (fileSize > 200) {alert('文件大小必须小于200KB。');fileInput.value = ''; // 清空选择框return;}if (fileType !== 'image/jpeg' && fileType !== 'image/png') {alert('仅允许上传JPG和PNG格式的图片。');fileInput.value = ''; // 清空选择框return;}}}function handleSend() {if (currentModel === 'image-analysis') {const fileInput = document.getElementById('imageUpload');const file = fileInput.files[0];if (!file) {alert("请选择图片文件");return;}uploadAndAnalyzeImage(file);} else {sendMessage();}}function uploadAndAnalyzeImage(file) {const formData = new FormData();formData.append('file', file);// 回显图片const reader = new FileReader();reader.onload = function(e) {const imgElement = document.createElement('img');imgElement.src = e.target.result;imgElement.style.maxWidth = '200px';imgElement.style.borderRadius = '10px';imgElement.style.margin = '10px 0';const userImgMessage = document.createElement('div');userImgMessage.classList.add('message', 'user-message');userImgMessage.appendChild(imgElement);chatContainer.appendChild(userImgMessage);toggleAllButtons(false);           // 禁用按钮showLoadingSpinner();          // 显示加载动画chatContainer.scrollTop = chatContainer.scrollHeight;};reader.readAsDataURL(file);fetch('http://192.168.100.72:8081/image/notStream', {method: 'POST',body: formData}).then(response => response.text()).then(text => {const fixedText = text.replace(/\\n/g, '\n');const html = marked.parse(fixedText);const aiMessage = document.createElement('div');aiMessage.classList.add('message', 'ai-message');aiMessage.innerHTML = html;chatContainer.appendChild(aiMessage);// 添加进一步分析提示与按钮const followUp = document.createElement('div');followUp.classList.add('message', 'ai-message');followUp.innerHTML = `<div style="display: flex; align-items: center;"><span style="margin-right: 10px;">是否进一步分析风险隐患及对应整改措施？</span><button class="send-button" onclick="startRiskAnalysis(\`${text.replace(/`/g, '\\`')}\`)">是</button></div>
`;chatContainer.appendChild(followUp);}).catch(err => {console.error("图片分析请求失败:", err);appendMessage("❌ 图片分析失败", "ai-message");}).finally(() => {hideLoadingSpinner();     // 移除加载动画toggleAllButtons(true);      // 启用按钮chatContainer.scrollTop = chatContainer.scrollHeight;});}function startRiskAnalysis(content) {toggleAllButtons(false); // 禁用按钮// 显示转圈动画showLoadingSpinner();fetch('http://192.168.100.72:8081/huoShan/image/chat', {method: 'POST',headers: {'Content-Type': 'text/plain'},body: content}).then(response => response.text())  // 获取文本响应.then(result => {// 隐藏转圈动画hideLoadingSpinner()// 格式化结果并解析为 HTMLconst fixedText = result.replace(/\\n/g, '\n');  // 解除转义const html = marked.parse(fixedText);// 创建新的 AI 消息并添加到界面const aiMessage = document.createElement('div');aiMessage.classList.add('message', 'ai-message');aiMessage.innerHTML = html;chatContainer.appendChild(aiMessage);chatContainer.scrollTop = chatContainer.scrollHeight;  // 滚动到底部toggleAllButtons(true); // 启用按钮}).catch(err => {console.error("风险分析请求失败:", err);// 隐藏转圈动画并显示失败消息hideLoadingSpinner()appendMessage("❌ 风险分析请求失败", "ai-message");toggleAllButtons(true); // 启用按钮});}function showLoadingSpinner() {const spinner = document.createElement('div');spinner.id = 'loadingSpinner';spinner.className = 'loading-spinner';chatContainer.appendChild(spinner);chatContainer.scrollTop = chatContainer.scrollHeight;}function hideLoadingSpinner() {const spinner = document.getElementById('loadingSpinner');if (spinner) spinner.remove();}function sendMessage() {let message = userInput.value.trim();if (!message) return;appendMessage(message, 'user-message');streamAIResponse(userId, message);userInput.value = '';}function appendMessage(text, type) {const messageElement = document.createElement('div');messageElement.classList.add('message', type);messageElement.textContent = text;chatContainer.appendChild(messageElement);chatContainer.scrollTop = chatContainer.scrollHeight;}function streamAIResponse(userId, message) {// 先终止可能存在的旧 eventSourceif (eventSource) {eventSource.close();}eventSource = new EventSource(`http://192.168.100.72:8081/ai/chatStreamWithMemory?userId=${encodeURIComponent(userId)}&message=${encodeURIComponent(message)}&model=${encodeURIComponent(currentModel)}`);let aiMessage = null;let thinkMode = false;let thinkMessage = null;eventSource.onmessage = event => {let response = event.data;if (response.includes('<think>') && currentModel === 'deepseek-r1:latest') {thinkMode = true;response = response.replace('<think>', '');// 创建思考过程气泡thinkMessage = document.createElement('div');thinkMessage.classList.add('think-message');thinkMessage.innerHTML = `<div class="think-title"><button class="toggle-button" onclick="toggleThinkMessage(this)">折叠</button><span class="think-title-text">思考过程：</span></div><div class="think-content" style="display: block;"></div>`;chatContainer.appendChild(thinkMessage);}if (thinkMode) {const thinkContent = thinkMessage.querySelector('.think-content');if (response.includes('</think>')) {response = response.replace('</think>', '');thinkMode = false;aiMessage = document.createElement('div');aiMessage.classList.add('message', 'ai-message');chatContainer.appendChild(aiMessage);}thinkContent.innerHTML += response;} else {if (!aiMessage) {aiMessage = document.createElement('div');aiMessage.classList.add('message', 'ai-message');chatContainer.appendChild(aiMessage);}aiMessage.textContent += response;}chatContainer.scrollTop = chatContainer.scrollHeight;};eventSource.onerror = () => {eventSource.close();toggleButtons(true);};eventSource.onopen = () => {toggleButtons(false);};eventSource.addEventListener("close", () => {toggleButtons(true);});}function toggleAllButtons(enabled) {sendButton.disabled = !enabled;clearButton.disabled = !enabled;stopButton.disabled = !enabled;}function toggleButtons(enabled) {sendButton.disabled = !enabled;clearButton.disabled = !enabled;}function toggleThinkMessage(button) {const thinkMessage = button.closest('.think-message');const thinkContent = thinkMessage.querySelector('.think-content');if (thinkContent.style.display === 'none') {thinkContent.style.display = 'block';button.textContent = '折叠';} else {thinkContent.style.display = 'none';button.textContent = '展开';}}function stopAIResponse() {if (eventSource) {eventSource.close();eventSource = null;}fetch(`http://192.168.100.72:8081/ai/stopChat?userId=${userId}`, { method: 'GET' }).then(() => appendMessage('AI 回答已停止。', 'ai-message')).catch(error => console.error(error));}function clearMemory() {fetch(`http://192.168.100.72:8081/ai/clearMemory?userId=${userId}`, { method: 'GET' }).then(() => appendMessage('上下文已清除。', 'ai-message')).catch(error => console.error(error));}function changeModel() {currentModel = modelSelect.value;let currentModelName = modelSelect.options[modelSelect.selectedIndex].text;appendMessage(`已切换模型为 ${currentModelName}`, 'ai-message');if (currentModel === 'image-analysis') {userInput.disabled = true;userInput.placeholder = '请选择图片进行分析...';sendButton.textContent = '分析图片';document.getElementById('imageUpload').style.display = 'block';} else {userInput.disabled = false;userInput.placeholder = '请输入消息...';sendButton.textContent = '发送';document.getElementById('imageUpload').style.display = 'none';}}userInput.addEventListener('keypress', event => {if (event.key === 'Enter') {sendMessage();}});
</script>
</body>
</html>