Stabilizing Timestamps for Whisper

This library modifies Whisper to produce more reliable timestamps and extends its functionality.

https://github.com/jianfch/stable-ts/assets/28970749/7adf0540-3620-4b2b-b2d4-e316906d6dfa

Setup
Usage
Quick 1.X → 2.X Guide

Setup

<details> <summary>Prerequisites: FFmpeg & PyTorch</summary> <details> <summary>FFmpeg</summary>

Requires FFmpeg in PATH

# on Ubuntu or Debian
sudo apt update && sudo apt install ffmpeg

# on Arch Linux
sudo pacman -S ffmpeg

# on MacOS using Homebrew (https://brew.sh/)
brew install ffmpeg

# on Windows using Chocolatey (https://chocolatey.org/)
choco install ffmpeg

# on Windows using Scoop (https://scoop.sh/)
scoop install ffmpeg

</details> <details> <summary>PyTorch</summary>

If PyTorch is not installed when installing Stable-ts, the default version will be installed which may not have GPU support. To avoid this issue, install your preferred version with instructions at https://pytorch.org/get-started/locally/.

</details> </details>

pip install -U stable-ts

To install the latest commit:

pip install -U git+https://github.com/jianfch/stable-ts.git

<details> <summary>Whisperless Version</summary>

To install Stable-ts without Whisper as a dependency:

pip install -U stable-ts-whisperless

To install the latest Whisperless commit:

pip install -U git+https://github.com/jianfch/stable-ts.git@whisperless

</details>

Usage

Transcribe

import stable_whisper
model = stable_whisper.load_model('base')
result = model.transcribe('audio.mp3')
result.to_srt_vtt('audio.srt')

stable-ts audio.mp3 -o audio.srt

</details>

Docstrings:

<details> <summary>load_model()</summary>

Load an instance if :class:`whisper.model.Whisper`.

Parameters
----------
name : {'tiny', 'tiny.en', 'base', 'base.en', 'small', 'small.en', 'medium', 'medium.en', 'large-v1',
    'large-v2', 'large-v3', or 'large'}
    One of the official model names listed by :func:`whisper.available_models`, or
    path to a model checkpoint containing the model dimensions and the model state_dict.
device : str or torch.device, optional
    PyTorch device to put the model into.
download_root : str, optional
    Path to download the model files; by default, it uses "~/.cache/whisper".
in_memory : bool, default False
    Whether to preload the model weights into host memory.
cpu_preload : bool, default True
    Load model into CPU memory first then move model to specified device
    to reduce GPU memory usage when loading model
dq : bool, default False
    Whether to apply Dynamic Quantization to model to reduced memory usage and increase inference speed
    but at the cost of a slight decrease in accuracy. Only for CPU.
engine : str, optional
    Engine for Dynamic Quantization.

Returns
-------
model : "Whisper"
    The Whisper ASR model instance.

Notes
-----
The overhead from ``dq = True`` might make inference slower for models smaller than 'large'.

</details> <details> <summary>transcribe()</summary>

Transcribe audio using Whisper.

This is a modified version of :func:`whisper.transcribe.transcribe` with slightly different decoding logic while
allowing additional preprocessing and postprocessing. The preprocessing performed on the audio includes:
voice isolation / noise removal and low/high-pass filter. The postprocessing performed on the transcription
result includes: adjusting timestamps with VAD and custom regrouping segments based punctuation and speech gaps.

Parameters
----------
model : whisper.model.Whisper
    An instance of Whisper ASR model.
audio : str or numpy.ndarray or torch.Tensor or bytes or AudioLoader
    Path/URL to the audio file, the audio waveform, or bytes of audio file or
    instance of :class:`stable_whisper.audio.AudioLoader`.
    If audio is :class:`numpy.ndarray` or :class:`torch.Tensor`, the audio must be already at sampled to 16kHz.
verbose : bool or None, default False
    Whether to display the text being decoded to the console.
    Displays all the details if ``True``. Displays progressbar if ``False``. Display nothing if ``None``.
temperature : float or iterable of float, default (0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
    Temperature for sampling. It can be a tuple of temperatures, which will be successfully used
    upon failures according to either ``compression_ratio_threshold`` or ``logprob_threshold``.
compression_ratio_threshold : float, default 2.4
    If the gzip compression ratio is above this value, treat as failed.
logprob_threshold : float, default -1
    If the average log probability over sampled tokens is below this value, treat as failed
no_speech_threshold : float, default 0.6
    If the no_speech probability is higher than this value AND the average log probability
    over sampled tokens is below ``logprob_threshold``, consider the segment as silent
condition_on_previous_text : bool, default True
    If ``True``, the previous output of the model is provided as a prompt for the next window;
    disabling may make the text inconsistent across windows, but the model becomes less prone to
    getting stuck in a failure loop, such as repetition looping or timestamps going out of sync.
initial_prompt : str, optional
    Text to provide as a prompt for the first window. This can be used to provide, or
    "prompt-engineer" a context for transcription, e.g. custom vocabularies or proper nouns
    to make it more likely to predict those word correctly.
word_timestamps : bool, default True
    Extract word-level timestamps using the cross-attention pattern and dynamic time warping,
    and include the timestamps for each word in each segment.
    Disabling this will prevent segments from splitting/merging properly.
regroup : bool or str, default True, meaning the default regroup algorithm
    String for customizing the regrouping algorithm. False disables regrouping.
    Ignored if ``word_timestamps = False``.
suppress_silence : bool, default True
    Whether to enable timestamps adjustments based on the detected silence.
suppress_word_ts : bool, default True
    Whether to adjust word timestamps based on the detected silence. Only enabled if ``suppress_silence = True``.
use_word_position : bool, default True
    Whether to use position of the word in its segment to determine whether to keep end or start timestamps if
    adjustments are required. If it is the first word, keep end. Else if it is the last word, keep the start.
q_levels : int, default 20
    Quantization levels for generating timestamp suppression mask; ignored if ``vad = true``.
    Acts as a threshold to marking sound as silent.
    Fewer levels will increase the threshold of volume at which to mark a sound as silent.
k_size : int, default 5
    Kernel size for avg-pooling waveform to generate timestamp suppression mask; ignored if ``vad = true``.
    Recommend 5 or 3; higher sizes will reduce detection of silence.
denoiser : str, optional
    String of the denoiser to use for preprocessing ``audio``.
    See ``stable_whisper.audio.SUPPORTED_DENOISERS`` for supported denoisers.
denoiser_options : dict, optional
    Options to use for ``denoiser``.
vad : bool or dict, default False
    Whether to use Silero VAD to generate timestamp suppression mask.
    Instead of ``True``, using a dict of keyword arguments will load the VAD with the arguments.
    Silero VAD requires PyTorch 1.12.0+. Official repo, https://github.com/snakers4/silero-vad.
vad_threshold : float, default 0.35
    Threshold for detecting speech with Silero VAD. Low threshold reduces false positives for silence detection.
min_word_dur : float or None, default None meaning use ``stable_whisper.default.DEFAULT_VALUES``
    Shortest duration each word is allowed to reach for silence suppression.
min_silence_dur : float, optional
    Shortest duration of silence allowed for silence suppression.
nonspeech_error : float, default 0.1
    Relative error of non-speech sections that appear in between a word for silence suppression.
only_voice_freq : bool, default False
    Whether to only use sound between 200 - 5000 Hz, where majority of human speech are.
prepend_punctuations : str or None, default None meaning use ``stable_whisper.default.DEFAULT_VALUES``
    Punctuations to prepend to next word.
append_punctuations : str or None, default None meaning use ``stable_whisper.default.DEFAULT_VALUES``
    Punctuations to append to previous word.
stream : bool or None, default None
    Whether to loading ``audio`` in chunks of 30 seconds until the end of file/stream.
    If ``None`` and ``audio`` is a string then set to ``True`` else ``False``.
mel_first : bool, optional
    Process entire audio track into log-Mel spectrogram first instead in chunks.
    Used if odd behavior seen in stable-ts but not in whisper, but use significantly more memory for long audio.
split_callback : Callable, optional
    Custom callback for grouping tokens up with their corresponding words.
    The callback must take two arguments, list of tokens and tokenizer.
    The callback returns a tuple with a list of words and a corresponding nested list of tokens.
suppress_ts_tokens : bool, default False
    Whether to suppress timestamp tokens during inference for timestamps are detected at silent.
    Reduces hallucinations in some cases, but also prone to ignore disfluencies and repetitions.
    This option is ignored if ``suppress_silence = False``.
gap_padding : str, default ' ...'
    Padding prepend to each segments for word timing alignment.
    Used to reduce the probability of model predicting timestamps earlier than the first utterance.
only_ffmpeg : bool, default False
    Whether to use only FFmpeg (instead of not yt-dlp) for URls
max_instant_words : float, default 0.5
    If percentage of instantaneous words in a segment exceed this amount, the segment is removed.
avg_prob_threshold: float or None, default None
    Transcribe the gap after the previous word and if the average word proababiliy of a segment falls below this
    value, discard the segment. If ``None``, skip transcribing the gap to reduce chance of timestamps starting
    before the next utterance.
progress_callback : Callable, optional
    A function that will be called when transcription progress is updated.
    The callback need two parameters.
    The first parameter is a float for seconds of the audio that has been transcribed.
    The second parameter is a float for total duration of audio in seconds.
ignore_compatibility : bool, default False
    Whether to ignore warnings for compatibility issues with the detected Whisper version.
extra_models : list of whisper.model.Whisper, optional
    List of additional Whisper model instances to use for computing word-timestamps along with ``model``.
decode_options
    Keyword arguments to construct class:`whisper.decode.DecodingOptions` instances.

Returns
-------
stable_whisper.result.WhisperResult
    All timestamps, words, probabilities, and other data from the transcription of ``audio``.

See Also
--------
stable_whisper.non_whisper.transcribe_any : Return :class:`stable_whisper.result.WhisperResult` containing all the
    data from transcribing audio with unmodified :func:`whisper.transcribe.transcribe` with preprocessing and
    postprocessing.
stable_whisper.whisper_word_level.faster_whisper.faster_transcribe : Return
    :class:`stable_whisper.result.WhisperResult` containing all the data from transcribing audio with
    :meth:`faster_whisper.WhisperModel.transcribe` with preprocessing and postprocessing.

Examples
--------
>>> import stable_whisper
>>> model = stable_whisper.load_model('base')
>>> result = model.transcribe('audio.mp3', vad=True)
>>> result.to_srt_vtt('audio.srt')
Saved: audio.srt

</details> <details> <summary>transcribe_minimal()</summary>

Transcribe audio using Whisper.

This is uses the original whisper transcribe function, :func:`whisper.transcribe.transcribe`, while still allowing
additional preprocessing and postprocessing. The preprocessing performed on the audio includes: voice isolation /
noise removal and low/high-pass filter. The postprocessing performed on the transcription result includes:
adjusting timestamps with VAD and custom regrouping segments based punctuation and speech gaps.

Parameters
----------
model : whisper.model.Whisper
    An instance of Whisper ASR model.
audio : str or numpy.ndarray or torch.Tensor or bytes
    Path/URL to the audio file, the audio waveform, or bytes of audio file.
    If audio is ``numpy.ndarray`` or ``torch.Tensor``, the audio must be already at sampled to 16kHz.
verbose : bool or None, default False
    Whether to display the text being decoded to the console.
    Displays all the details if ``True``. Displays progressbar if ``False``. Display nothing if ``None``.
word_timestamps : bool, default True
    Extract word-level timestamps using the cross-attention pattern and dynamic time warping,
    and include the timestamps for each word in each segment.
    Disabling this will prevent segments from splitting/merging properly.
regroup : bool or str, default True, meaning the default regroup algorithm
    String for customizing the regrouping algorithm. False disables regrouping.
    Ignored if ``word_timestamps = False``.
suppress_silence : bool, default True
    Whether to enable timestamps adjustments based on the detected silence.
suppress_word_ts : bool, default True
    Whether to adjust word timestamps based on the detected silence. Only enabled if ``suppress_silence = True``.
use_word_position : bool, default True
    Whether to use position of the word in its segment to determine whether to keep end or start timestamps if
    adjustments are required. If it is the first word, keep end. Else if it is the last word, keep the start.
q_levels : int, default 20
    Quantization levels for generating timestamp suppression mask; ignored if ``vad = true``.
    Acts as a threshold to marking sound as silent.
    Fewer levels will increase the threshold of volume at which to mark a sound as silent.
k_size : int, default 5
    Kernel size for avg-pooling waveform to generate timestamp suppression mask; ignored if ``vad = true``.
    Recommend 5 or 3; higher sizes will reduce detection of silence.
denoiser : str, optional
    String of the denoiser to use for preprocessing ``audio``.
    See ``stable_whisper.audio.SUPPORTED_DENOISERS`` for supported denoisers.
denoiser_options : dict, optional
    Options to use for ``denoiser``.
vad : bool or dict, default False
    Whether to use Silero VAD to generate timestamp suppression mask.
    Instead of ``True``, using a dict of keyword arguments will load the VAD with the arguments.
    Silero VAD requires PyTorch 1.12.0+. Official repo, https://github.com/snakers4/silero-vad.
vad_threshold : float, default 0.35
    Threshold for detecting speech with Silero VAD. Low threshold reduces false positives for silence detection.
min_word_dur : float, default 0.1
    Shortest duration each word is allowed to reach for silence suppression.
min_silence_dur : float, optional
    Shortest duration of silence allowed for silence suppression.
nonspeech_error : float, default 0.1
    Relative error of non-speech sections that appear in between a word for silence suppression.
only_voice_freq : bool, default False
    Whether to only use sound between 200 - 5000 Hz, where majority of human speech are.
only_ffmpeg : bool, default False
    Whether to use only FFmpeg (instead of not yt-dlp) for URls
options
    Additional options used for :func:`whisper.transcribe.transcribe` and
    :func:`stable_whisper.non_whisper.transcribe_any`.
Returns
-------
stable_whisper.result.WhisperResult
    All timestamps, words, probabilities, and other data from the transcription of ``audio``.

Examples
--------
>>> import stable_whisper
>>> model = stable_whisper.load_model('base')
>>> result = model.transcribe_minimal('audio.mp3', vad=True)
>>> result.to_srt_vtt('audio.srt')
Saved: audio.srt

</details> <br> <details> <summary>faster-whisper</summary>

Use with faster-whisper: