《RuoYi基于SpringBoot+Vue前后端分离的Java快速开发框架学习》系列博客_Part4_三模态融合

系列博客目录

---

---

目标

 实现三模态融合，将文本、图片、音频从前端上传，后端返回数据库数据。主要包含两种类别，电脑和弓。而且电脑和弓的数据在数据库中不同的表中，表的列名存在不同(仅最后一列不同)。

Step1:之前工作形成子组件

 之前已经实现了图文识别、音文识别、音图识别的前端页面，而当时的要求，比如图文识别就是用户上传图片，得到相应文字信息(来自数据库)。所以图文识别，就已经写好了上传图片的vue。同理上传音频的vue也写好了。这次直接把之前的组件当作子组件。

Step2:弥补缺失的文本子组件，同时举例如何子组件向父组件传数据

 通过下面代码可以看出，当点击按钮时，将文本框中内容传递给父组件。
文本子组件代码如下：

<template><div class="app-container"><label for="textInput">请输入文本内容：</label><textareaid="textInput"v-model="textContent"rows="5"placeholder="在此输入文本内容..."style="width: 100%; margin-bottom: 10px;"></textarea><el-buttontype="primary"@click="submitTextContent":disabled="!textContent.trim()">点击此按钮上传文本</el-button></div>
</template><script>
export default {name: "TextInput",data() {return {textContent: "", // 文本输入框内容};},methods: {submitTextContent() {// 通过事件将文本内容传递给父组件this.$emit("fileUploaded", this.textContent.trim());//trim()去除文本两端的空白字符this.$message.success("文本提交成功！");},},
};
</script>

当上面子组件点击按钮后，调用父组件fileUploaded指定的函数。 <PictureToText v-if="showChild" @fileUploaded="handleImageUpload" />
父组件中的表格的表头是根据后端返回的resultType 动态变化的。
setTimeout()函数是在延迟后，执行里面的代码。
父组件核心代码如下：

<template><div><div class="container"><!-- 模态选择区域 目前不需要，因为要直接提供三个模态的上传 -->
<!--    <el-radio-group v-model="radio2">-->
<!--      <el-radio :label="3">图片</el-radio>-->
<!--      <el-radio :label="6">音频</el-radio>-->
<!--      <el-radio :label="9">文本</el-radio>-->
<!--    </el-radio-group>--><!-- 图片上传区域 --><!-- 动态显示子组件 --><div class="child">
<!--      <PictureToText v-if="radio2 === 3 && showChild" @updateResult="handleResult"/>--><PictureToText v-if="showChild" @fileUploaded="handleImageUpload" /></div><!-- 音频上传区域 --><div class="child">
<!--      <AudioToText v-if="radio2 === 6"/>--><AudioToText v-if="showChild" @fileUploaded="handleAudioUpload" /></div><!-- 文本上传区域 --><div class="child"><TextUploader v-if="showChild" @fileUploaded="handleTextUpload"/></div><!-- 添加提交按钮 -->
<!--    <button v-if="showChild" @click="submitData">提交数据</button>--><!-- 提交按钮，独占一行 --><div class="button-container"><el-buttonv-if="showChild"type="primary"class="action-button"@click="submitData">提交数据进行分析</el-button></div></div><div><el-table v-if="tableOfResultVisible" v-loading="loading"  element-loading-text="拼命分析中,预计加载结果需要等待大约10秒钟...." :data="jmwlList" >
<!--      <el-table-column type="selection" width="55" align="center" />--><el-table-column label="交易id" align="center" prop="交易id" show-overflow-tooltip/><el-table-column label="品牌" align="center" prop="品牌" show-overflow-tooltip/><el-table-column label="商品名称" align="center" prop="商品名称" show-overflow-tooltip /><el-table-column label="评论用户名" align="center" prop="评论用户名" show-overflow-tooltip/><el-table-column label="评论内容" align="center" prop="评论内容" show-overflow-tooltip/><el-table-column label="赞数" align="center" prop="赞数" show-overflow-tooltip/><el-table-column label="商品交易客体图片" align="center" ><template slot-scope="scope1"><img v-if="scope1.row.商品交易主体图片" :src="scope1.row.商品交易主体图片" prop="商品交易客体图片" style="width:100px;height:100px;cursor:pointer" @click="showImage(scope1.row.商品交易主体图片)"/><img v-else src="./default.jpg" style="width:100px;height:100px"/><el-dialog :visible="dialogVisible" @close="dialogVisible=false"><img :src="dialogImageUrl" style="width:300px;height:300px"/></el-dialog></template></el-table-column><el-table-column label="商品交易客体图片1" align="center" ><template slot-scope="scope1"><img v-if="scope1.row.商品交易客体图片1" :src="scope1.row.商品交易客体图片1" prop="商品交易客体图片1" style="width:100px;height:100px;cursor:pointer" @click="showImage(scope1.row.商品交易客体图片1)"/><img v-else src="./default.jpg" style="width:100px;height:100px"/><el-dialog :visible="dialogVisible" @close="dialogVisible=false"><img :src="dialogImageUrl" style="width:300px;height:300px"/></el-dialog></template></el-table-column><el-table-column label="产品质量" align="center" prop="产品质量" show-overflow-tooltip/><el-table-column label="产品价格" align="center" prop="产品价格" show-overflow-tooltip/><el-table-column label="客户评分" align="center" prop="客户评分" show-overflow-tooltip/><el-table-column label="店铺总分" align="center" prop="店铺总分" show-overflow-tooltip/><el-table-column label="店铺名称" align="center" prop="店铺名称" show-overflow-tooltip/><el-table-column label="店铺链接" align="center" prop="店铺链接" show-overflow-tooltip/><el-table-column label="评论类别" align="center" prop="评论类别" show-overflow-tooltip/><el-table-column label="交易平台" align="center" prop="交易平台" show-overflow-tooltip/>
<!--      <el-table-column label="疑似假冒伪劣原因" align="center" prop="疑似假冒伪劣原因" show-overflow-tooltip/>--><el-table-column v-if="resultType === 0" label="疑似假冒伪劣原因" align="center" prop="疑似假冒伪劣原因" show-overflow-tooltip/><el-table-column v-else-if="resultType === 1" label="疑似禁限售原因" align="center" prop="疑似禁限售原因" show-overflow-tooltip/></el-table><!-- 条件显示，当数据总数大于 0 时显示分页组件 --><!-- 数据总数，决定分页的总页数 --><!-- 当前页码，使用.sync实现双向绑定 --><!-- 每页显示的数据条数，使用.sync实现双向绑定 --><!-- 当分页发生变化时触发的事件，调用getList函数 --><paginationv-show="tableOfResultVisible":total="total":page.sync="queryParams.pageNum":limit.sync="queryParams.pageSize"@pagination="getList"/></div></div>
</template><script>
import PictureToText from "../../pictureToText/pictureToText/index.vue";
import AudioToText from "../../audioToText/audioToText/index.vue"
import TextUploader from "@/views/3MFusion/3MFusion/TextUploader.vue";
import {uploadAudioToText, uploadPictureToText, uploadTextToText} from "@/api/pictureToText/pictureToText";export default {components: {TextUploader, PictureToText,AudioToText },name: "ThreeMFusion",data() {return {//用来判断哪一类文件已经上传textFile: null,imageFile: null,audioFile: null,showChild: true, // 控制子组件是否显示tableOfResultVisible: false,//控制结果表格是否显示// 遮罩层loading: true,// 总条数total: 0,// 假冒伪劣产品研究表格数据也可以显示禁限售表格数据，通过后端传过来的resultType动态参数决定jmwlList: [],// 弹出层标题title: "",// 表单参数form: {},// 表单校验rules: {}};},methods: {//显示哪一模态的文件已经上传handleTextUpload(file) {this.textFile = file;},handleImageUpload(file) {this.imageFile = file;},handleAudioUpload(file) {this.audioFile = file;},submitData() {this.loading = true;if (this.textFile) {this.tableOfResultVisible = true;this.showChild = false;console.log('提交文本数据:', this.textFile);uploadTextToText({ content: this.textFile }).then((response) => {if (response.code === 200) {setTimeout(() => {this.loading = false;this.jmwlList = response.rows;this.total = response.total;this.resultType = response.resultType;}, 5000);}});} else if (this.imageFile) {this.showChild = false;this.tableOfResultVisible = true;console.log('提交图片数据到3MFusion');uploadPictureToText(this.imageFile).then(response => {if (response.code === 200) {this.imgUrl=response.pictureUrl;console.log(this.imgUrl);setTimeout(() => {this.loading = false;this.jmwlList = response.rows;this.total = response.total;this.resultType = response.resultType;}, 5000);}});} else if (this.audioFile) {this.showChild = false;this.tableOfResultVisible = true;console.log('提交音频数据到3MFusion');uploadAudioToText(this.audioFile).then(response => {if (response.code === 200) {console.log("uploadAudio()已完成");this.audioUrl = response.audioUrlconsole.log(response.audioUrl);setTimeout(() => {this.loading = false;this.jmwlList = response.rows;this.total = response.total;this.resultType = response.resultType;}, 5000);}});} else {this.$message.error("没有可提交的数据！");console.warn('没有可提交的数据');}},}
};
</script>

Step3:后端代码需要根据上传的文件传给python服务器

    @PostMapping("/uploadAudioToText")public TableDataInfo insertUploadOfAudio(@RequestParam("audioName") MultipartFile file) throws Exception{List<Integer> idsList = new ArrayList<>();int resulttype = -1;if (!file.isEmpty()){LoginUser loginUser = getLoginUser();String audioPath = FileUploadUtils.upload(RuoYiConfig.getAudioPath(), file, MimeTypeUtils.MEDIA_EXTENSION);Upload upload=new Upload();upload.setUsername(loginUser.getUsername());upload.setAudioPath(audioPath);String pictureName=audioPath.substring(audioPath.lastIndexOf("/")+1,audioPath.length());upload.setPictureName(pictureName);if (iUploadService.insertUpload(upload)!=0){try {// 定义 URLURL url = new URL("http://127.0.0.1:6000/api");HttpURLConnection connection = (HttpURLConnection) url.openConnection();connection.setRequestMethod("POST");connection.setRequestProperty("Content-Type", "application/json; utf-8");connection.setRequestProperty("Accept", "application/json");connection.setDoOutput(true);String number = String.valueOf(6);String jsonInputString = "{\"audioPath\": \"" + audioPath + "\", \"number\": " + number + "}";try (OutputStream os = connection.getOutputStream()) {byte[] input = jsonInputString.getBytes("utf-8");os.write(input, 0, input.length);}// 检查响应码int responseCode = connection.getResponseCode();if (responseCode == 200) {// 读取响应内容BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));StringBuilder response = new StringBuilder();String line;while ((line = reader.readLine()) != null) {response.append(line);}reader.close();// 打印完整的响应内容System.out.println("Python 服务返回的数据: " + response);// 将响应字符串解析为 JSON 数组try {
//                            JSONArray jsonResponse = new JSONArray(response.toString());JSONArray jsonResponse = JSONArray.parseArray(response.toString());// 查看 jsonResponse 的长度int length = jsonResponse.size();  // 获取 JSONArray 的长度System.out.println("JSON 数组的长度: " + length);// 获取第一个元素，应该是数字（0 或 1）resulttype = jsonResponse.getIntValue(0);  // 这里获取 0 或 1JSONArray idsArray = jsonResponse.getJSONArray(1);  // 获取 IDs 数组// 打印 type 和 idsSystem.out.println("type: " + resulttype);System.out.println("ids: " + idsArray);// 如果你需要将 ids 转换为 Java List，可以使用以下方法for (int i = 0; i < idsArray.size(); i++) {idsList.add(Integer.valueOf(idsArray.getString(i)));  // 添加每个 ID}System.out.println("IDs as List: " + idsList);} catch (Exception e) {e.printStackTrace();System.err.println("解析 JSON 响应时发生错误.");}} else {System.err.println("HTTP 请求失败，响应码: " + responseCode);}connection.disconnect();} catch (Exception e) {e.printStackTrace();}}}if(resulttype==0){List<Jmwl> list = jmwlService.selectJmwlListByIdList(idsList);return getDataTable(list,resulttype);}else {List<Jxs> list = jxsService.selectJxsListByIdList(idsList);System.out.println("resulttype = " + resulttype);return getDataTable(list,resulttype);}}

Step4:python服务器进行分析

from database_server import get_ids_by_keyword  # 导入函数
from flask import Flask, jsonify, request
from detectKeyword import detect_keyword
from audio import getKeywordOfAudio
import sys
import os# 将项目根目录添加到 sys.path
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))from predict import get_keywordOfImage  # 导入 predict.py 中的函数app = Flask(__name__)@app.route('/api', methods=['POST'])
def api():data = request.json # 假设数据通过 JSON 格式传递file_type = data.get('number')print("file_type = ", file_type)# 根据类型处理不同逻辑if file_type == 3:# 图像处理逻辑image_path = data.get('imagePath')print("image_path+file_type: " + str(image_path) + str(file_type))keyword = get_keywordOfImage(image_path)elif file_type == 6:# 音频处理逻辑audioPath = data.get('audioPath')print("audioPath =" + str(audioPath))keyword = getKeywordOfAudio(audioPath)keyword = detect_keyword(keyword)else:# 处理不支持的类型contextOfText = data.get('context')keyword = detect_keyword(contextOfText)ids = get_ids_by_keyword(keyword)  # 调用函数查询print(f"与关键词 '{keyword}' 相关的 ID 列表: {ids}")  # 打印结果if keyword == "电脑":response = [0, ids]elif keyword == "弓":response = [1, ids]else:response = [None, ids]return jsonify(response)if __name__ == "__main__":app.run(port=6000)

python服务器输出示例

file_type = 9
keyword = 弓
与关键词 ‘弓’ 相关的 ID 列表: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71]

返回给java后端的数据示例如下，第一个0，代表电脑。

[0,[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71]]