springboot整合vosk实现简单的语音识别功能

vosk开源语音识别

Vosk是开源的语音识别工具包。Vosk支持的事情包括：

支持十九种语言 - 中文，英语，印度英语，德语，法语，西班牙语，葡萄牙语，俄语，土耳其语，越南语，意大利语，荷兰人，加泰罗尼亚语，阿拉伯, 希腊语, 波斯语, 菲律宾语，乌克兰语, 哈萨克语。
移动设备上脱机工作-Raspberry Pi，Android，iOS。
使用简单的 pip3 install vosk 安装。
每种语言的手提式模型只有是50Mb, 但还有更大的服务器模型可用。
提供流媒体API，以提供最佳用户体验（与流行的语音识别python包不同）。
还有用于不同编程语言的包装器-java / csharp / javascript等。
可以快速重新配置词汇以实现最佳准确性。
支持说话人识别。

vosk-api

离线语音识别API，适用于Android，iOS，Raspberry Pi和具有Python，Java，C＃等

链接: vosk-api github地址

有各语言的使用的示例

vosk-server

基于Vosk和Kaldi库的WebSocket，gRPC和WebRTC语音识别服务器

链接: vosk-server github地址

有各语言的使用的示例

vosk-api - java - springboot中的使用

导入依赖包

  <!-- 语音识别 --><dependency><groupId>net.java.dev.jna</groupId><artifactId>jna</artifactId><version>5.13.0</version></dependency><dependency><groupId>com.alphacephei</groupId><artifactId>vosk</artifactId><version>0.3.45</version></dependency><!-- JAVE2（Java音频视频编码器）库是ffmpeg项目上的Java包装器。 --><dependency><groupId>ws.schild</groupId><artifactId>jave-core</artifactId><version>3.1.1</version></dependency><!-- 在windows上开发 开发机可实现压缩效果 window64位 --><dependency><groupId>ws.schild</groupId><artifactId>jave-nativebin-win32</artifactId><version>3.1.1</version></dependency><dependency><groupId>ws.schild</groupId><artifactId>jave-nativebin-win64</artifactId><version>3.1.1</version></dependency>

VoskResult

public class VoskResult {private String text;public String getText() {return text;}public void setText(String text) {this.text = text;}
}

vosk模型加载

package com.fjdci.vosk;import org.vosk.LibVosk;
import org.vosk.LogLevel;
import org.vosk.Model;import java.io.IOException;/*** vosk模型加载* @author zhou*/
public class VoskModel {/*** 3. 使用 volatile 保证线程安全* 禁止指令重排* 保证可见性* 不保证原子性*/private static volatile VoskModel instance;private Model voskModel;public Model getVoskModel() {return voskModel;}/*** 1.私有构造函数*/private VoskModel() {System.out.println("SingleLazyPattern实例化了");//String modelStr = "D:\\work\\project\\fjdci-vosk\\src\\main\\resources\\vosk-model-small-cn-0.22";String modelStr = "D:\\work\\fjdci\\docker\\vosk\\vosk-model-cn-0.22";try {voskModel = new Model(modelStr);LibVosk.setLogLevel(LogLevel.INFO);} catch (IOException e) {e.printStackTrace();}}/*** 2.通过静态方法获取一个唯一实例* DCL 双重检查锁定 （Double-CheckedLocking）* 在多线程情况下保持⾼性能*/public static VoskModel getInstance() {if (instance == null) {synchronized (VoskModel.class) {if (instance == null) {// 1. 分配内存空间 2、执行构造方法，初始化对象 3、把这个对象指向这个空间instance = new VoskModel();}}}return instance;}/*** 多线程测试加载* @param args*/public static void main(String[] args) {for (int i = 0; i < 5; i++) {new Thread(() -> {VoskModel.getInstance();}).start();}}}

语言识别工具类

package com.fjdci.vosk;import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import org.vosk.Model;
import org.vosk.Recognizer;
import ws.schild.jave.EncoderException;
import ws.schild.jave.MultimediaObject;
import ws.schild.jave.info.AudioInfo;
import ws.schild.jave.info.MultimediaInfo;import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.UUID;@Slf4j
@Component
public class VoiceUtil {public static void main(String[] args) throws EncoderException {String wavFilePath = "D:\\fjFile\\annex\\xwbl\\tem_2.wav";// 秒long cutDuration = 20;String waveForm = acceptWaveForm(wavFilePath, cutDuration);System.out.println(waveForm);}/*** 对Wav格式音频文件进行语音识别翻译** @param wavFilePath* @param cutDuration* @return* @throws EncoderException*/private static String acceptWaveForm(String wavFilePath, long cutDuration) throws EncoderException {// 判断视频的长度long startTime = System.currentTimeMillis();MultimediaObject multimediaObject = new MultimediaObject(new File(wavFilePath));MultimediaInfo info = multimediaObject.getInfo();// 时长/毫秒long duration = info.getDuration();AudioInfo audio = info.getAudio();// 通道数int channels = audio.getChannels();// 秒long offset = 0;long forNum = (duration / 1000) / cutDuration;if (duration % (cutDuration * 1000) > 0) {forNum = forNum + 1;}// 进行切块处理List<String> strings = cutWavFile(wavFilePath, cutDuration, offset, forNum);// 循环进行翻译StringBuilder result = new StringBuilder();for (String string : strings) {File f = new File(string);result.append(VoiceUtil.getRecognizerResult(f, channels));}long endTime = System.currentTimeMillis();String msg = "耗时：" + (endTime - startTime) + "ms";System.out.println(msg);return result.toString();}/*** 对wav进行切块处理** @param wavFilePath 处理的wav文件路径* @param cutDuration 切割的固定长度/秒* @param offset      设置起始偏移量(秒)* @param forNum      切块的次数* @return* @throws EncoderException*/private static List<String> cutWavFile(String wavFilePath, long cutDuration, long offset, long forNum) throws EncoderException {UUID uuid = UUID.randomUUID();// 大文件切割为固定时长的小文件List<String> strings = new ArrayList<>();for (int i = 0; i < forNum; i++) {String target = "D:\\fjFile\\annex\\xwbl\\" + uuid + "\\" + i + ".wav";Float offsetF = Float.valueOf(String.valueOf(offset));Float cutDurationF = Float.valueOf(String.valueOf(cutDuration));Jave2Util.cut(wavFilePath, target, offsetF, cutDurationF);offset = offset + cutDuration;strings.add(target);}return strings;}/*** 进行翻译** @param f* @param channels*/public static String getRecognizerResult(File f, int channels) {StringBuilder result = new StringBuilder();Model voskModel = VoskModel.getInstance().getVoskModel();// 采样率为音频采样率的声道倍数log.info("====加载完成，开始分析====");try (Recognizer recognizer = new Recognizer(voskModel, 16000 * channels);InputStream ais = new FileInputStream(f)) {int nbytes;byte[] b = new byte[4096];while ((nbytes = ais.read(b)) >= 0) {if (recognizer.acceptWaveForm(b, nbytes)) {// 返回语音识别结果result.append(getResult(recognizer.getResult()));}}// 返回语音识别结果。和结果一样，但不要等待沉默。你通常在流的最后调用它来获得音频的最后部分。它刷新功能管道，以便处理所有剩余的音频块。result.append(getResult(recognizer.getFinalResult()));log.info("识别结果：{}", result.toString());} catch (Exception e) {e.printStackTrace();}return result.toString();}/*** 获取返回结果** @param result* @return*/private static String getResult(String result) {VoskResult voskResult = JacksonMapperUtils.json2pojo(result, VoskResult.class);return Optional.ofNullable(voskResult).map(VoskResult::getText).orElse("");}}

jave2 音频处理工具类

package com.fjdci.vosk;import ws.schild.jave.Encoder;
import ws.schild.jave.EncoderException;
import ws.schild.jave.InputFormatException;
import ws.schild.jave.MultimediaObject;
import ws.schild.jave.encode.AudioAttributes;
import ws.schild.jave.encode.EncodingAttributes;
import ws.schild.jave.info.AudioInfo;
import ws.schild.jave.info.MultimediaInfo;import java.io.File;public class Jave2Util {/*** @param src      来源文件路径* @param target   目标文件路径* @param offset   设置起始偏移量(秒)* @param duration 设置切片的音频长度(秒)* @throws EncoderException*/public static void cut(String src, String target, Float offset, Float duration) throws EncoderException {File targetFile = new File(target);if (targetFile.exists()) {targetFile.delete();}File srcFile = new File(src);MultimediaObject srcMultiObj = new MultimediaObject(srcFile);MultimediaInfo srcMediaInfo = srcMultiObj.getInfo();Encoder encoder = new Encoder();EncodingAttributes encodingAttributes = new EncodingAttributes();//设置起始偏移量(秒)encodingAttributes.setOffset(offset);//设置切片的音频长度(秒)encodingAttributes.setDuration(duration);// 输入格式encodingAttributes.setInputFormat("wav");//设置音频属性AudioAttributes audio = new AudioAttributes();audio.setBitRate(srcMediaInfo.getAudio().getBitRate());//audio.setSamplingRate(srcMediaInfo.getAudio().getSamplingRate());// 转换为16KHZ 满足vosk识别的标准audio.setSamplingRate(16000);audio.setChannels(srcMediaInfo.getAudio().getChannels());//如果截取的时候，希望同步调整编码，可以设置不同的编码
//        audio.setCodec("pcm_u8");//audio.setCodec(srcMediaInfo.getAudio().getDecoder().split(" ")[0]);encodingAttributes.setAudioAttributes(audio);//写文件encoder.encode(srcMultiObj, new File(target), encodingAttributes);}/*** 转化音频格式** @param oldFormatPath : 原音乐路径* @param newFormatPath : 目标音乐路径* @return*/public static boolean transforMusicFormat(String oldFormatPath, String newFormatPath) {File source = new File(oldFormatPath);File target = new File(newFormatPath);// 音频转换格式类Encoder encoder = new Encoder();// 设置音频属性AudioAttributes audio = new AudioAttributes();audio.setCodec(null);// 设置转码属性EncodingAttributes attrs = new EncodingAttributes();attrs.setInputFormat("wav");attrs.setAudioAttributes(audio);try {encoder.encode(new MultimediaObject(source), target, attrs);System.out.println("传唤已完成...");return true;} catch (IllegalArgumentException e) {e.printStackTrace();} catch (InputFormatException e) {e.printStackTrace();} catch (EncoderException e) {e.printStackTrace();}return false;}public static void main(String[] args) throws EncoderException {String src = "D:\\fjFile\\annex\\xwbl\\ly8603f22f24e0409fa9747d50a78ff7e5.wav";String target = "D:\\fjFile\\annex\\xwbl\\tem_2.wav";Jave2Util.cut(src, target, 0.0F, 60.0F);String inputFormatPath = "D:\\fjFile\\annex\\xwbl\\ly8603f22f24e0409fa9747d50a78ff7e5.m4a";String outputFormatPath = "D:\\fjFile\\annex\\xwbl\\ly8603f22f24e0409fa9747d50a78ff7e5.wav";info(inputFormatPath);// audioEncode(inputFormatPath, outputFormatPath);}/*** 获取音频文件的编码信息** @param filePath* @throws EncoderException*/private static void info(String filePath) throws EncoderException {File file = new File(filePath);MultimediaObject multimediaObject = new MultimediaObject(file);MultimediaInfo info = multimediaObject.getInfo();// 时长long duration = info.getDuration();String format = info.getFormat();// format:movSystem.out.println("format:" + format);AudioInfo audio = info.getAudio();// 它设置将在重新编码的音频流中使用的音频通道数（1 =单声道，2 =立体声）。如果未设置任何通道值，则编码器将选择默认值。int channels = audio.getChannels();// 它为新的重新编码的音频流设置比特率值。如果未设置比特率值，则编码器将选择默认值。// 该值应以每秒位数表示。例如，如果您想要128 kb / s的比特率，则应调用setBitRate（new Integer（128000））。int bitRate = audio.getBitRate();// 它为新的重新编码的音频流设置采样率。如果未设置采样率值，则编码器将选择默认值。该值应以赫兹表示。例如，如果您想要类似CD// 采样率、音频采样级别 16000 = 16KHzint samplingRate = audio.getSamplingRate();// 设置音频音量// 可以调用此方法来更改音频流的音量。值为256表示音量不变。因此，小于256的值表示音量减小，而大于256的值将增大音频流的音量。// setVolume(Integer volume)String decoder = audio.getDecoder();System.out.println("声音时长:毫秒" + duration);System.out.println("声道:" + channels);System.out.println("bitRate:" + bitRate);System.out.println("samplingRate 采样率、音频采样级别 16000 = 16KHz:" + samplingRate);// aac (LC) (mp4a / 0x6134706D)System.out.println("decoder:" + decoder);}/*** 音频格式转换* @param inputFormatPath* @param outputFormatPath* @return*/public static boolean audioEncode(String inputFormatPath, String outputFormatPath) {String outputFormat = getSuffix(outputFormatPath);String inputFormat = getSuffix(inputFormatPath);File source = new File(inputFormatPath);File target = new File(outputFormatPath);try {MultimediaObject multimediaObject = new MultimediaObject(source);// 获取音频文件的编码信息MultimediaInfo info = multimediaObject.getInfo();AudioInfo audioInfo = info.getAudio();//设置音频属性AudioAttributes audio = new AudioAttributes();audio.setBitRate(audioInfo.getBitRate());audio.setSamplingRate(audioInfo.getSamplingRate());audio.setChannels(audioInfo.getChannels());// 设置转码属性EncodingAttributes attrs = new EncodingAttributes();attrs.setInputFormat(inputFormat);attrs.setOutputFormat(outputFormat);attrs.setAudioAttributes(audio);// 音频转换格式类Encoder encoder = new Encoder();// 进行转换encoder.encode(new MultimediaObject(source), target, attrs);return true;} catch (IllegalArgumentException | EncoderException e) {e.printStackTrace();}return false;}/*** 获取文件路径的.后缀* @param outputFormatPath* @return*/private static String getSuffix(String outputFormatPath) {return outputFormatPath.substring(outputFormatPath.lastIndexOf(".") + 1);}}

import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.DeserializationFeature;
import com.fasterxml.jackson.databind.JavaType;
import com.fasterxml.jackson.databind.ObjectMapper;import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;public class JacksonMapperUtils {private final static ObjectMapper objectMapper = new ObjectMapper();public static ObjectMapper getInstance() {return objectMapper;}/*** 转换为 JSON 字符串** @param obj* @return* @throws Exception*/public static String obj2json(Object obj) throws Exception {ObjectMapper mapper = new ObjectMapper();mapper.setSerializationInclusion(JsonInclude.Include.ALWAYS);return mapper.writeValueAsString(obj);}/*** 转换为 JSON 字符串，忽略空值** @param obj* @return* @throws Exception*/public static String obj2jsonIgnoreNull(Object obj) throws Exception {ObjectMapper mapper = new ObjectMapper();mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);return mapper.writeValueAsString(obj);}/*** 转换为 JavaBean** @param jsonString* @param clazz* @return* @throws Exception*/public static <T> T json2pojo(String jsonString, Class<T> clazz){try {objectMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);return objectMapper.readValue(jsonString, clazz);} catch (JsonProcessingException e) {e.printStackTrace();}return null;}/*** 字符串转换为 Map<String, Object>** @param jsonString* @return* @throws Exception*/public static <T> Map<String, Object> json2map(String jsonString) throws Exception {ObjectMapper mapper = new ObjectMapper();mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);return mapper.readValue(jsonString, Map.class);}/*** 字符串转换为 Map<String, T>*/public static <T> Map<String, T> json2map(String jsonString, Class<T> clazz) throws Exception {Map<String, Map<String, Object>> map = (Map<String, Map<String, Object>>) objectMapper.readValue(jsonString, new TypeReference<Map<String, T>>() {});Map<String, T> result = new HashMap<String, T>();for (Map.Entry<String, Map<String, Object>> entry : map.entrySet()) {result.put(entry.getKey(), map2pojo(entry.getValue(), clazz));}return result;}/*** 深度转换 JSON 成 Map** @param json* @return*/public static Map<String, Object> json2mapDeeply(String json) throws Exception {return json2MapRecursion(json, objectMapper);}/*** 把 JSON 解析成 List，如果 List 内部的元素存在 jsonString，继续解析** @param json* @param mapper 解析工具* @return* @throws Exception*/private static List<Object> json2ListRecursion(String json, ObjectMapper mapper) throws Exception {if (json == null) {return null;}List<Object> list = mapper.readValue(json, List.class);for (Object obj : list) {if (obj != null && obj instanceof String) {String str = (String) obj;if (str.startsWith("[")) {obj = json2ListRecursion(str, mapper);} else if (obj.toString().startsWith("{")) {obj = json2MapRecursion(str, mapper);}}}return list;}/*** 把 JSON 解析成 Map，如果 Map 内部的 Value 存在 jsonString，继续解析** @param json* @param mapper* @return* @throws Exception*/private static Map<String, Object> json2MapRecursion(String json, ObjectMapper mapper) throws Exception {if (json == null) {return null;}Map<String, Object> map = mapper.readValue(json, Map.class);for (Map.Entry<String, Object> entry : map.entrySet()) {Object obj = entry.getValue();if (obj != null && obj instanceof String) {String str = ((String) obj);if (str.startsWith("[")) {List<?> list = json2ListRecursion(str, mapper);map.put(entry.getKey(), list);} else if (str.startsWith("{")) {Map<String, Object> mapRecursion = json2MapRecursion(str, mapper);map.put(entry.getKey(), mapRecursion);}}}return map;}/*** 将 JSON 数组转换为集合** @param jsonArrayStr* @param clazz* @return* @throws Exception*/public static <T> List<T> json2list(String jsonArrayStr, Class<T> clazz) throws Exception {JavaType javaType = getCollectionType(ArrayList.class, clazz);List<T> list = (List<T>) objectMapper.readValue(jsonArrayStr, javaType);return list;}/*** 获取泛型的 Collection Type** @param collectionClass 泛型的Collection* @param elementClasses  元素类* @return JavaType Java类型* @since 1.0*/public static JavaType getCollectionType(Class<?> collectionClass, Class<?>... elementClasses) {return objectMapper.getTypeFactory().constructParametricType(collectionClass, elementClasses);}/*** 将 Map 转换为 JavaBean** @param map* @param clazz* @return*/public static <T> T map2pojo(Map map, Class<T> clazz) {return objectMapper.convertValue(map, clazz);}/*** 将 Map 转换为 JSON** @param map* @return*/public static String mapToJson(Map map) {try {return objectMapper.writeValueAsString(map);} catch (Exception e) {e.printStackTrace();}return "";}/*** 将 JSON 对象转换为 JavaBean** @param obj* @param clazz* @return*/public static <T> T obj2pojo(Object obj, Class<T> clazz) {return objectMapper.convertValue(obj, clazz);}}

语音模型下载地址翻墙

https://alphacephei.com/vosk/models

链接: 模型下载地址

踩坑记录

java.io.IOException: Failed to create a modelat org.vosk.Model.<init>(Model.java:14)

java.lang.Error: Invalid memory accessat org.vosk.LibVosk.vosk_recognizer_new(Native Method)at org.vosk.Recognizer.<init>(Recognizer.java:19)

链接: Exception: Failed to create a model 原因

在这里插入图片描述

vosk模型加载需要服务器有足够的内存

项目启动未加载模型时使用了500M
在这里插入图片描述

项目加载模型时使用了4G多内存
在这里插入图片描述

参考链接

链接: vosk开源语音识别

链接: 基于Whisper的音频转录服务汇总

链接: 几款免费的语音转文字工具推荐

链接: java 离线中文语音文字识别

链接: Asr - python使用vosk进行中文语音识别

链接: NeMo非常强大，覆盖了ASR, NLP, TTS，提供了预训练模型及完整的训练模块。其商业版本为RIVA。

链接: ASRT语音识别文档
ASRT是一个基于深度学习的语音识别工具，可以用于开发最先进的语音识别系统，是由AI柠檬博主（西安电子科技大学 · 西安市大数据与视觉智能重点实验室）从2016年起做的开源语音识别项目，基线为85%识别准确率，在某些条件下可做到95%左右的识别准确率。ASRT包含了语音识别算法服务端(用于训练或部署API服务)和多种平台及编程语言的客户端SDK，支持一句话识别和实时流式识别，相关的代码已经开源在GitHub和Gitee上。
ASRT语音识别系统的API已经为AI柠檬站内搜索引擎提供了语音识别服务，用于该站语音搜索功能的实现。