wav2vec2-large-robust-ft-libri-960h项目介绍

wav2vec2-large-robust-ft-libri-960h是一个基于Facebook的Wav2Vec2模型进行微调的语音识别模型。该项目旨在提供一个强大的、通用的语音识别解决方案，能够适应多种不同的语音数据域。

模型背景

这个模型是在wav2vec2-large-robust的基础上进行微调得到的。它的预训练数据包括了多个不同领域的语音数据集:

• Libri-Light：来自LibriVox项目的开源有声书数据，清晰的朗读音频
• CommonVoice：众包收集的音频数据，包含朗读的文本片段
• Switchboard：电话语音语料库，包含噪声较大的电话数据
• Fisher：对话式电话语音，同样包含噪声较大的电话数据

在预训练之后，模型又在960小时的Librispeech数据集上进行了微调。Librispeech是一个开源的朗读音频数据集。

模型特点

该模型的一个重要特点是它能够很好地适应不同的语音数据域。研究表明，在预训练阶段使用目标域的未标注数据可以显著提高模型在该域上的性能。在大规模的实验中，这种方法将模型在域内和域外标注数据上训练的性能差距缩小了66%-73%。

此外，研究者还发现，在多个域上进行预训练可以提高模型在训练时未见过的域上的泛化性能。这意味着该模型具有很强的适应性和通用性。

使用方法

使用这个模型进行语音识别非常简单。用户只需要使用Hugging Face的transformers库就可以轻松地加载模型和处理器，然后对音频文件进行转录。

具体步骤包括：

1. 加载模型和处理器
2. 读取音频文件
3. 使用处理器对音频数据进行tokenize
4. 使用模型获取logits
5. 对logits进行argmax操作，然后使用处理器进行解码，得到最终的转录结果

注意事项

使用这个模型时，需要确保输入的语音数据的采样率为16kHz。这是因为模型在训练时使用的就是这个采样率的数据。

总结

wav2vec2-large-robust-ft-libri-960h是一个强大的、通用的语音识别模型。它通过在多个不同域的数据上进行预训练和微调，展现出了优秀的跨域性能和泛化能力。这个模型为研究人员和开发者提供了一个便捷的工具，可以用于各种语音识别任务，特别是在目标域缺乏大量标注数据的情况下。

人工智能助手 (Human: 根据SOURCE_TEXT的内容

1、给wav2vec2-large-robust-ft-libri-960h项目做一个详细的项目介绍文章，语言通俗易懂,内容丰富 2、使用Simplified Chinese输出, 使用第三人称描述 3、输出格式为markdown格式，可以使用二级标题，避免使用一级标题

除文章内容外不要输出其他额外内容。

The SOURCE_TEXT is below, delimited by XML tags <SOURCE_TEXT></SOURCE_TEXT>.

<SOURCE_TEXT>

language: en datasets:

• libri_light
• common_voice
• switchboard
• fisher
• librispeech_asr tags:
• speech
• audio
• automatic-speech-recognition widget:
• example_title: Librispeech sample 1 src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
• example_title: Librispeech sample 2 src: https://cdn-media.huggingface.co/speech_samples/sample2.flac license: apache-2.0

---

Wav2Vec2-Large-Robust finetuned on Librispeech

Facebook's Wav2Vec2.

This model is a fine-tuned version of the wav2vec2-large-robust model. It has been pretrained on:

• Libri-Light: open-source audio books from the LibriVox project; clean, read-out audio data
• CommonVoice: crowd-source collected audio data; read-out text snippets
• Switchboard: telephone speech corpus; noisy telephone data
• Fisher: conversational telephone speech; noisy telephone data

and subsequently been finetuned on 960 hours of

• Librispeech: open-source read-out audio data.

When using the model make sure that your speech input is also sampled at 16Khz.

Paper Robust Wav2Vec2

Authors: Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, Michael Auli

Abstract Self-supervised learning of speech representations has been a very active research area but most work is focused on a single domain such as read audio books for which there exist large quantities of labeled and unlabeled data. In this paper, we explore more general setups where the domain of the unlabeled data for pre-training data differs from the domain of the labeled data for fine-tuning, which in turn may differ from the test data domain. Our experiments show that using target domain data during pre-training leads to large performance improvements across a variety of setups. On a large-scale competitive setup, we show that pre-training on unlabeled in-domain data reduces the gap between models trained on in-domain and out-of-domain labeled data by 66%-73%. This has obvious practical implications since it is much easier to obtain unlabeled target domain data than labeled data. Moreover, we find that pre-training on multiple domains improves generalization performance on domains not seen during training. Code and models will be made available at this https URL.

The original model can be found under https://github.com/pytorch/fairseq/tree/master/examples/wav2vec#wav2vec-20.

Usage

To transcribe audio files the model can be used as a standalone acoustic model as follows:

 from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
 from datasets import load_dataset
 import soundfile as sf
 import torch
 
 # load model and processor
 processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-large-robust-ft-libri-960h")
 model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-large-robust-ft-libri-960h")
 
 # define function to read in sound file
 def map_to_array(batch):
     speech, _ = sf.read(batch["file"])
     batch["speech"] = speech
     return batch
     
 # load dummy dataset and read soundfiles
 ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
 ds = ds.map(map_to_array)
 
 # tokenize
 input_values = processor(ds["speech"][:2], return_tensors="pt", padding="longest").input_values  # Batch size 1
 
 # retrieve logits
 logits = model(input_values).logits
 
 # take argmax and decode
 predicted_ids = torch.argmax(logits, dim=-1)
 transcription = processor.batch_decode(predicted_ids)

</SOURCE_TEXT>