文章目录
1.1 项目背景1.2 项目目标2.1 业务流程2.2 系统用例2.3 系统总体功能3.1 技术选型4.1 文件上传模块4.2 文件下载模块4.3 数据库设计5.1 大文件上传实现5.2 大文件下载实现
源码:
https://gitee.com/gaode-8/big-file-upload演示视频
https://www.bilibili.com/video/BV1CA411f7np/?vd_source=1fe29350b37642fa583f709b9ae44b351.1 项目背景
对于超大文件上传我们可能遇到以下问题
• 大文件直接上传,占用过多内存,可能导致内存溢出甚至系统崩溃
• 受网络环境影响,可能导致传输中断,只能重新传输
• 传输时间长,用户无法知道传输进度,用户体验不佳
1.2 项目目标
对于上述问题,我们需要对文件做分片传输。分片传输就是把文件分割成许多较小的文件,然后分多次上传,最后再完成合并。
受网络环境影响,我们还要实现断点续传,以节省传输时间和资源。断点续传就是已经上传或者下载过的文件分片不再传输。
对于已经上传过的文件,可以不再上传,实现秒传。秒传就是根据文件的唯一标识,确认是否需要上传。
实现多任务上传或下载。多任务就是同时多个文件上传或下载。
2.1 业务流程
用户上传文件的流程图如图1所示,用户首先选择要上传的文件,上传过程中可以选择暂停或继续上传。
用户上传文件的流程图如图2所示,用户首先可以浏览可以下载的文件列表,然后点击下载,下载过程中可以选择暂停或继续下载
2.2 系统用例
系统用例图如图3所示,用户可以上传文件,在文件上传过程中可以查看文件的上传进度和速度,也可以暂停或开始上传;用户可以查看已经上传过的,也就是可以下载的文件列表;用户可以下载文件,在下载过程中可以查看文件下载的速度和进度,用户可以暂停或开始下载。
2.3 系统总体功能
系统总体功能图如图4所示,分为上传和下载。上传包括秒传,分片上传,断点续传,多任务。下载包括分片下载,断点续传,多任务。
3.1 技术选型
后端:
• 语言:Java8
• 框架:SpringBoot2.6
• 开发工具:Idea 2021
前端:
• 语言:Html5、css3、JavaScript
• 框架:Vue3
• 开发工具:Vscode、Edge
数据库:
• mysql8
4.1 文件上传模块
文件上传模块的流程图如图6所示,顺序图如图7所示
首先前端读取文件生成文件的唯一标识MD5,这里采用常用的MD5生成框架:spark-md5.js。对于大文件一次性读取比较慢,而且容易造成浏览器崩溃,因此这里采用分片读取的方式计算MD5。
然后向服务器发送请求,查看该文件时候已经上传,如果已经上传,就提示用户已经秒传。
如果数据库中没有记录该文件,就表示该文件没有上传或没有上传完成,那么服务器就查询并返回记录的chunk分片列表。
async 和 await配可以实现等待异步函数计算完成
//计算文件的md5值function computeMd5(file, uploadFile) { return new Promise((resolve, reject) => { //分片读取并计算md5 const chunkTotal = 100; //分片数 const chunkSize = Math.ceil(file.size / chunkTotal); const fileReader = new FileReader(); const md5 = new SparkMD5(); let index = 0; const loadFile = (uploadFile) => { uploadFile.parsePercentage.value = parseInt((index / file.size) * 100); const slice = file.slice(index, index + chunkSize); fileReader.readAsBinaryString(slice); }; loadFile(uploadFile); fileReader.onload = (e) => { md5.appendBinary(e.target.result); if (index < file.size) { index += chunkSize; loadFile(uploadFile); } else { // md5.end() 就是文件md5码 resolve(md5.end()); } }; });}//检查文件是否存在function checkFile(md5) { return request({ url: "/check", method: "get", params: { md5: md5, }, });}//文件上传之前,el-upload自动触发async function beforeUpload(file) { console.log("2.上传文件之前"); var uploadFile = {}; uploadFile.name = file.name; uploadFile.size = file.size; uploadFile.parsePercentage = ref(0); uploadFile.uploadPercentage = ref(0); uploadFile.uploadSpeed = "0 M/s"; uploadFile.chunkList = null; uploadFile.file = file; uploadFile.uploadingStop = false; uploadFileList.value.push(uploadFile); var md5 = await computeMd5(file, uploadFile);//async 和 await配可以实现等待异步函数计算完成 uploadFile.md5 = md5; var res = await checkFile(md5); //上传服务器检查,以确认是否秒传 var data = res.data.data; if (!data.isUploaded) { uploadFile.chunkList = data.chunkList; uploadFile.needUpload = true; } else { uploadFile.needUpload = false; uploadFile.uploadPercentage.value = 100; console.log("文件已秒传"); ElMessage({ showClose: true, message: "文件已秒传", type: "warning", }); }}
前端分片请求文件,如果分片编号被包含在分片列表内,就标识该分片已经上传,跳过;反之,表示还未上传,那么前端通过file的slice方法分割文件,向服务端传递。同时在页面上显示上传进度和速度。
服务端,收到前端的分片文件后,通过Java的RandomAccess类(随机读写类),从文件的指定位置,写入指定字节,并记录chunk到数据库,如果是最后一个分片再记录file到数据库。
图6 文件上传流程图
图7 文件上传顺序图
前端代码
<template> <div class="main"> <!-- 文件上传按钮 --> <el-upload action="#" :http-request="upload" :before-upload="beforeUpload" :show-file-list="false" > <el-button type="primary">选择上传文件</el-button> </el-upload> <el-divider content-position="left">上传列表</el-divider> <!-- 正在上传的文件列表 --> <div class="uploading" v-for="uploadFile in uploadFileList"> <span class="fileName">{{ uploadFile.name }}</span> <span class="fileSize">{{ formatSize(uploadFile.size) }}</span> <div class="parse"> <span>解析进度: </span> <el-progress :text-inside="true" :stroke-width="16" :percentage="uploadFile.parsePercentage" > </el-progress> </div> <div class="progress"> <span>上传进度:</span> <el-progress :text-inside="true" :stroke-width="16" :percentage="uploadFile.uploadPercentage" > </el-progress> <span v-if=" (uploadFile.uploadPercentage > 0) & (uploadFile.uploadPercentage < 100) " > <span class="uploadSpeed">{{ uploadFile.uploadSpeed }}</span> <el-button circle link @click="changeUploadingStop(uploadFile)"> <el-icon size="20" v-if="uploadFile.uploadingStop == false" ><VideoPause /></el-icon> <el-icon size="20" v-else><VideoPlay /></el-icon> </el-button> </span> </div> </div> </div></template><script setup>import emitter from "../utils/eventBus.js";import { ElMessage } from "element-plus";import SparkMD5 from "spark-md5";import { VideoPause, VideoPlay } from "@element-plus/icons-vue";import { ref, reactive, getCurrentInstance, nextTick } from "vue";const { appContext } = getCurrentInstance();const request = appContext.config.globalProperties.request;var uploadFileList = ref([]);//换算文件的大小单位function formatSize(size) { //size的单位大小k var unit; var units = [" B", " K", " M", " G"]; var pointLength = 2; while ((unit = units.shift()) && size > 1024) { size = size / 1024; } return ( (unit === "B" ? size : size.toFixed(pointLength === undefined ? 2 : pointLength)) + unit );}//计算文件的md5值function computeMd5(file, uploadFile) { return new Promise((resolve, reject) => { //分片读取并计算md5 const chunkTotal = 100; //分片数 const chunkSize = Math.ceil(file.size / chunkTotal); const fileReader = new FileReader(); const md5 = new SparkMD5(); let index = 0; const loadFile = (uploadFile) => { uploadFile.parsePercentage.value = parseInt((index / file.size) * 100); const slice = file.slice(index, index + chunkSize); fileReader.readAsBinaryString(slice); }; loadFile(uploadFile); fileReader.onload = (e) => { md5.appendBinary(e.target.result); if (index < file.size) { index += chunkSize; loadFile(uploadFile); } else { // md5.end() 就是文件md5码 resolve(md5.end()); } }; });}//检查文件是否存在function checkFile(md5) { return request({ url: "/check", method: "get", params: { md5: md5, }, });}//文件上传之前,el-upload自动触发async function beforeUpload(file) { console.log("2.上传文件之前"); var uploadFile = {}; uploadFile.name = file.name; uploadFile.size = file.size; uploadFile.parsePercentage = ref(0); uploadFile.uploadPercentage = ref(0); uploadFile.uploadSpeed = "0 M/s"; uploadFile.chunkList = null; uploadFile.file = file; uploadFile.uploadingStop = false; uploadFileList.value.push(uploadFile); var md5 = await computeMd5(file, uploadFile);//async 和 await配可以实现等待异步函数计算完成 uploadFile.md5 = md5; var res = await checkFile(md5); //上传服务器检查,以确认是否秒传 var data = res.data.data; if (!data.isUploaded) { uploadFile.chunkList = data.chunkList; uploadFile.needUpload = true; } else { uploadFile.needUpload = false; uploadFile.uploadPercentage.value = 100; console.log("文件已秒传"); ElMessage({ showClose: true, message: "文件已秒传", type: "warning", }); }}//点击暂停或开始上传function changeUploadingStop(uploadFile) { uploadFile.uploadingStop = !uploadFile.uploadingStop; if (!uploadFile.uploadingStop) { uploadChunk(uploadFile.file, 1, uploadFile); }}//上传文件,替换el-upload的actionfunction upload(xhrData) { var uploadFile = null; for (var i = 0; i < uploadFileList.value.length; i++) { if ( (xhrData.file.name == uploadFileList.value[i].name) & (xhrData.file.size == uploadFileList.value[i].size) ) { uploadFile = uploadFileList.value[i]; break; } } if (uploadFile.needUpload) { console.log("3.上传文件"); // 分片上传文件 // 确定分片的大小 uploadChunk(xhrData.file, 1, uploadFile); }}//上传文件分片function uploadChunk(file, index, uploadFile) { var chunkSize = 1024 * 1024 * 10; //10mb var chunkTotal = Math.ceil(file.size / chunkSize); if (index <= chunkTotal) { // 根据是否暂停,确定是否继续上传 // console.log("4.上传分片"); var startTime = new Date().valueOf(); var exit = uploadFile.chunkList.includes(index); // console.log("是否存在",exit); if (!exit) { // console.log("3.3上传文件",uploadingStop); if (!uploadFile.uploadingStop) { // 分片上传,同时计算进度条和上传速度 // 已经上传的不在上传、 // 上传完成后提示,上传成功 // console.log("上传分片1",index); var form = new FormData(); var start = (index - 1) * chunkSize; let end = index * chunkSize >= file.size ? file.size : index * chunkSize; let chunk = file.slice(start, end); // downloadBlob(chunk,file) // console.log("chunk",chunk); form.append("chunk", chunk); form.append("index", index); form.append("chunkTotal", chunkTotal); form.append("chunkSize", chunkSize); form.append("md5", uploadFile.md5); form.append("fileSize", file.size); form.append("fileName", file.name); // console.log("上传分片", index); request({ url: "/upload/chunk", method: "post", data: form, }).then((res) => { var endTime = new Date().valueOf(); var timeDif = (endTime - startTime) / 1000; // console.log("上传文件大小",formatSize(chunkSize)); // console.log("耗时",timeDif); // console.log("then",index); // uploadSpeed = (chunkSize/(1024*1024)) / timeDif +" M / s" uploadFile.uploadSpeed = (10 / timeDif).toFixed(1) + " M/s"; // console.log(res.data.data); // console.log("f2",uploadFile); uploadFile.chunkList.push(index); // console.log("f3",uploadFile); uploadFile.uploadPercentage = parseInt( (uploadFile.chunkList.length / chunkTotal) * 100 ); // console.log("上传进度",uploadFile.uploadPercentage); if (index == chunkTotal) { emitter.emit("reloadFileList"); } uploadChunk(file, index + 1, uploadFile); }); } } else { uploadFile.uploadPercentage = parseInt( (uploadFile.chunkList.length / chunkTotal) * 100 ); uploadChunk(file, index + 1, uploadFile); } // } }}</script><style scoped>.main { margin-top: 40px; margin-bottom: 40px;}.uploading { padding-top: 27px;}.progress { /* width: 700px; */ display: flex;}.uploading .parse { display: flex;}.parse .el-progress { /* font-size: 18px; */ width: 590px;}.progress .el-progress { /* font-size: 18px; */ width: 590px;}.uploading .fileName { font-size: 17px; margin-right: 40px; margin-left: 80px; /* width: 80px; */}.uploading .fileSize { font-size: 17px; /* width: 80px; */}.progress .uploadSpeed { font-size: 17px; margin-left: 5px; padding-left: 5px; padding-right: 10px;}</style>
后端代码
package com.cugb.bigfileupload.controller;import com.cugb.bigfileupload.bean.FilePO;import com.cugb.bigfileupload.bean.Result;import com.cugb.bigfileupload.servie.ChunkService;import com.cugb.bigfileupload.servie.FileService;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.beans.factory.annotation.Value;import org.springframework.web.bind.annotation.*;import org.springframework.web.multipart.MultipartFile;import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.Objects;@RestController@CrossOriginpublic class FileController { Logger logger = LoggerFactory.getLogger(getClass()); @Value("${file.path}") private String filePath; @Autowired private FileService fileService; @Autowired private ChunkService chunkService; @GetMapping("/check") public Result checkFile(@RequestParam("md5") String md5){ logger.info("检查MD5:"+md5); //首先检查是否有完整的文件 Boolean isUploaded = fileService.selectFileByMd5(md5); Map<String, Object> data = new HashMap<>(); data.put("isUploaded",isUploaded); //如果有,就返回秒传 if(isUploaded){ return new Result(201,"文件已经秒传",data); } //如果没有,就查找分片信息,并返回给前端 List<Integer> chunkList = chunkService.selectChunkListByMd5(md5); data.put("chunkList",chunkList); return new Result(201,"",data); } @PostMapping("/upload/chunk") public Result uploadChunk(@RequestParam("chunk") MultipartFile chunk, @RequestParam("md5") String md5, @RequestParam("index") Integer index, @RequestParam("chunkTotal")Integer chunkTotal, @RequestParam("fileSize")Long fileSize, @RequestParam("fileName")String fileName, @RequestParam("chunkSize")Long chunkSize ){ String[] splits = fileName.split("\\."); String type = splits[splits.length-1]; String resultFileName = filePath+md5+"."+type; chunkService.saveChunk(chunk,md5,index,chunkSize,resultFileName); logger.info("上传分片:"+index +" ,"+chunkTotal+","+fileName+","+resultFileName); if(Objects.equals(index, chunkTotal)){ FilePO filePO = new FilePO(fileName, md5, fileSize); fileService.addFile(filePO); chunkService.deleteChunkByMd5(md5); return new Result(200,"文件上传成功",index); }else{ return new Result(201,"分片上传成功",index); } } @GetMapping("/fileList") public Result getFileList(){ logger.info("查询文件列表"); List<FilePO> fileList = fileService.selectFileList(); return new Result(201,"文件列表查询成功",fileList); }}
4.2 文件下载模块
文件下载的流程图如图8所示,顺序图如图9所示
文件下载是首先,前端向后端发送分片下载的请求,请求的responseType设为blob(Binary large Object) ,然后后端通过RandomAccess类读取指定字节的内容,再写入到响应的文件流中。
浏览器前端的请求的分片数据,会暂时保存在“C:\Users\用户名\AppData\Local\Microsoft\Edge\User Data\Default\blob_storage\”中,(请确保c盘有足够的空间),当所有分片下载完成,会合并成一个大文件(很快),分片不是放在内存中,所以不用担心文件太大是不是不行。
,
刷新浏览器,也会删除已经下载好的分片
当前端请求了所有的文件分片之后,再把所有的blob合并成一个blob
if (index == chunkTotal) { var resBlob = new Blob(file.blobList, { type: "application/octet-stream", }); // console.log("resb", resBlob); let url = window.URL.createObjectURL(resBlob); // 将获取的文件转化为blob格式 let a = document.createElement("a"); // 此处向下是打开一个储存位置 a.style.display = "none"; a.href = url; // 下面两行是自己项目需要的处理,总之就是得到下载的文件名(加后缀)即可 var fileName = file.name; a.setAttribute("download", fileName); document.body.appendChild(a); a.click(); //点击下载 document.body.removeChild(a); // 下载完成移除元素 window.URL.revokeObjectURL(url); // 释放掉blob对象 }
图9文件上传顺序图
前端代码
<template> <div class="main"> <div class="fileList"> <div class="title"> 文件列表 <!-- <hr> --> </div> <el-table :data="fileList" border style="width: 360px"> <el-table-column prop="name" label="文件名" width="150"> </el-table-column> <el-table-column prop="size" label="文件大小" width="110"> <template #default="scope"> {{ formatSize(scope.row) }} </template> </el-table-column> <el-table-column prop="" label="操作" width="100"> <template #default="scope"> <el-button size="small" type="primary" @click="downloadFile(scope.row)" >下载</el-button > </template> </el-table-column> </el-table> </div> <div class="downloadList"> <el-divider content-position="left">下载列表</el-divider> <div v-for="file in downloadingFileList"> <div class="downloading"> <span class="fileName">{{ file.name }}</span> <span class="fileSize">{{ formatSize(file) }}</span> <span class="downloadSpeed">{{ file.downloadSpeed }}</span> <div class="progress"> <span>下载进度:</span> <el-progress :text-inside="true" :stroke-width="16" :percentage="file.downloadPersentage" > </el-progress> <el-button circle link @click="changeDownloadStop(file)"> <el-icon size="20" v-if="file.downloadingStop == false" ><VideoPause /></el-icon> <el-icon size="20" v-else><VideoPlay /></el-icon> </el-button> </div> </div> </div> </div> </div></template><script setup>import axios from "axios";import { ref, reactive, getCurrentInstance } from "vue";import emitter from "../utils/eventBus.js";import { VideoPause, VideoPlay } from "@element-plus/icons-vue";const { appContext } = getCurrentInstance();const request = appContext.config.globalProperties.request;var fileList = reactive([]);var downloadingFileList = ref([]);//上传文件之后,重新加载文件列表emitter.on("reloadFileList", () => { load();});function load() { fileList.length = 0; request({ url: "/fileList", method: "get", }).then((res) => { // console.log("res", res.data.data); fileList.push(...res.data.data); });}load();//换算文件的大小单位function formatSize(file) { //console.log("size",file.size); var size = file.size; var unit; var units = [" B", " K", " M", " G"]; var pointLength = 2; while ((unit = units.shift()) && size > 1024) { size = size / 1024; } return ( (unit === "B" ? size : size.toFixed(pointLength === undefined ? 2 : pointLength)) + unit );}//点击暂停下载function changeDownloadStop(file) { file.downloadingStop = !file.downloadingStop; if (!file.downloadingStop) { console.log("开始。。"); downloadChunk(1, file); }}//点击下载文件function downloadFile(file) { // console.log("下载", file); file.downloadingStop = false; file.downloadSpeed = "0 M/s"; file.downloadPersentage = 0; file.blobList = []; file.chunkList = []; downloadingFileList.value.push(file); downloadChunk(1, file);}//点击下载文件分片function downloadChunk(index, file) { var chunkSize = 1024 * 1024 * 5; var chunkTotal = Math.ceil(file.size / chunkSize); if (index <= chunkTotal) { // console.log("下载进度",index); var exit = file.chunkList.includes(index); console.log("存在", exit); if (!exit) { if (!file.downloadingStop) { var formData = new FormData(); formData.append("fileName", file.name); formData.append("md5", file.md5); formData.append("chunkSize", chunkSize); formData.append("index", index); formData.append("chunkTotal", chunkTotal); if (index * chunkSize >= file.size) { chunkSize = file.size - (index - 1) * chunkSize; formData.set("chunkSize", chunkSize); } var startTime = new Date().valueOf(); axios({ url: "http://localhost:9001/download", method: "post", data: formData, responseType: "blob", timeout: 50000, }).then((res) => { file.chunkList.push(index); var endTime = new Date().valueOf(); var timeDif = (endTime - startTime) / 1000; file.downloadSpeed = (5 / timeDif).toFixed(1) + " M/s"; //todo file.downloadPersentage = parseInt((index / chunkTotal) * 100); // var chunk = res.data.data.chunk // const blob = new Blob([res.data]); const blob = res.data; file.blobList.push(blob); // console.log("res", blobList); if (index == chunkTotal) { var resBlob = new Blob(file.blobList, { type: "application/octet-stream", }); // console.log("resb", resBlob); let url = window.URL.createObjectURL(resBlob); // 将获取的文件转化为blob格式 let a = document.createElement("a"); // 此处向下是打开一个储存位置 a.style.display = "none"; a.href = url; // 下面两行是自己项目需要的处理,总之就是得到下载的文件名(加后缀)即可 var fileName = file.name; a.setAttribute("download", fileName); document.body.appendChild(a); a.click(); //点击下载 document.body.removeChild(a); // 下载完成移除元素 window.URL.revokeObjectURL(url); // 释放掉blob对象 } downloadChunk(index + 1, file); }); } } else { file.downloadPersentage = parseInt((index / chunkTotal) * 100); downloadChunk(index + 1, file); } }}</script><style scoped>.main { display: flex;}.fileList { width: 400px;}.downloadList { width: 450px;}.title { margin-top: 5px; margin-bottom: 5px;}.downloading { margin-top: 10px;}.downloading .fileName { margin-left: 76px; margin-right: 30px;}.downloading .fileSize { /* margin-left: 70px; */ margin-right: 30px;}.downloading .progress { display: flex;}.progress .el-progress { /* font-size: 18px; */ width: 310px;}</style>
后端代码
package com.cugb.bigfileupload.controller;import com.cugb.bigfileupload.servie.ChunkService;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.beans.factory.annotation.Value;import org.springframework.stereotype.Controller;import org.springframework.web.bind.annotation.*;import javax.servlet.ServletOutputStream;import javax.servlet.http.HttpServletResponse;import java.io.File;import java.io.IOException;import java.util.Objects;@Controller@CrossOriginpublic class DownLoadController { Logger logger = LoggerFactory.getLogger(getClass()); @Value("${file.path}") private String filePath; @Autowired private ChunkService chunkService; @PostMapping("/download") public void download(@RequestParam("md5") String md5, @RequestParam("fileName") String fileName, @RequestParam("chunkSize") Integer chunkSize, @RequestParam("chunkTotal") Integer chunkTotal, @RequestParam("index")Integer index, HttpServletResponse response) { String[] splits = fileName.split("\\."); String type = splits[splits.length - 1]; String resultFileName = filePath + md5 + "." + type; File resultFile = new File(resultFileName); long offset = (long) chunkSize * (index - 1); if(Objects.equals(index, chunkTotal)){ offset = resultFile.length() -chunkSize; } byte[] chunk = chunkService.getChunk(index, chunkSize, resultFileName,offset); logger.info("下载文件分片" + resultFileName + "," + index + "," + chunkSize + "," + chunk.length+","+offset);// response.addHeader("Access-Control-Allow-Origin","Content-Disposition"); response.addHeader("Content-Disposition", "attachment;filename=" + fileName); response.addHeader("Content-Length", "" + (chunk.length)); response.setHeader("filename", fileName); response.setContentType("application/octet-stream"); ServletOutputStream out = null; try { out = response.getOutputStream(); out.write(chunk); out.flush(); out.close(); } catch (IOException e) { e.printStackTrace(); } }}
4.3 数据库设计
4.3.1 概念结构设计
数据库设计只有俩个表,一个file表来记录已经完整上传的文件信息,一个chunk表用来记录还未上传完成的分片信息
5.1 大文件上传实现
上传页面如图13所示,有一个“选择上传文件”的按钮,下面是显示正在上传文件的列表
图13 上传页面首页
我们选择要上传的文件,确认上传,首先会显示解析进度,当解析完成后,就会开始上传,并显示上传进度和速度;同时,我们可以选择多个文件一同上传;在上传的同时我们还可以暂停上传。如图14所示
图14 上传文件中
当文件上传成功之后,就会弹窗提示文件上传成功。如图15所示
图15 文件上传成功
5.2 大文件下载实现
文件下载页面如图16所示,左边是可以下载文件的列表,右边是下载中的文件
当所有的分片下载完成后,前端会将所有的分片合并成一个文件。如图18所示