livestream_video

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

livestream_video

★ 17 Open GitHub ↗

playlist4whisper manages media streams playlists for livestream_video.sh, plays media, and transcribes audio via AI with configurable timeshift, multi-instance/user support, translation, and TTS. Compatible with Linux, Windows (WSL2), macOS.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# playlist4whisper / livestream_video.sh

## About the Project

`playlist4whisper` manages media stream playlists, plays media, creates `.srt` subtitle files from local media files, and locally transcribes live video/audio streams with **[whisper.cpp](https://github.com/ggerganov/whisper.cpp)** (a highly efficient implementation of OpenAI's Whisper). It includes configurable time-shift, multi-instance/multi-user support, translation, and text-to-speech functionality.

`playlist4whisper` is a graphical application that allows you to easily manage and launch `livestream_video.sh`. It provides a user-friendly interface to configure all transcription and translation options, manage TV channel playlists, and store settings.

> [!NOTE]
> *This new version of the Python application and the Bash script supports the latest versions of **whisper.cpp 1.8.x**, which introduces an internal **VAD (Voice Activity Detection)**. The code has also been updated to support changes in the whisper.cpp toolchain, such as the new `whisper-quantize` executable name.*

### Some Notable Features:

-   **Online Translation:** Translate transcriptions in real-time using either Google Translate or the high-quality Google Gemini API.
-   **Context Control for AI:** Fine-tune Gemini translations with context levels (0-3) to balance between literal accuracy and creative, context-aware fluency. This can also be used to improve the transcription.
-   **Text-to-Speech (TTS):** Read translated text aloud for a more immersive experience.
-   **Save Session Transcripts:** Save the complete transcription and translation logs from the current session with a single click.
-   **Subtitle Generation:** Automatically create `.srt` subtitle files from local media files.
-   **Subtitle Video Editor:** Integrated tool to cut and merge video segments, ideal for creating multilingual subtitles or refining timing.
-   **Timeshift:** A fully configurable timeshift feature, exclusive to the VLC player.
-   **Timeshift Recording:** Save and merge selected video segments directly from the timeshift buffer using the integrated video saver tool.
-   **Broad Service Support:** Access a wide range of video services through `streamlink` or `yt-dlp`.
-   **VAD Support:** Integrated Voice Activity Detection based on the powerful, license-free Silero model (auto-downloaded on first use) to improve transcription segmentation by cutting audio during silences.
-   **Multi-Platform:** Compatible with Linux, macOS, and Windows (via WSL2).
-   **Advanced Database Features**: Support for multiple playlists in the same window or in separate instances with different whisper.cpp versions and configurations, including customizable names and tab border colors. Playlists support wildcard searches, as well as loading, saving, and appending other playlists. Playlist elements support editing, moving, and drag-and-drop (files and URLs, as links or plain text).

> [!WARNING]
> **Important Note on Subtitle Generation:** Sometimes models make too many errors or get stuck repeating words or phrases; even the most powerful model, such as the larger-v3 model, can be especially problematic. You can try using a different model or splitting the audio/video file into smaller chunks with the integrated `Subtitle Video Editor`.
>

> [!WARNING]
> To avoid generational quality loss when merging video parts, the video segments are concatenated without a full re-encode. This approach is efficient and preserves the original media data, but it may cause timestamp discontinuities or synchronization drift in some players and processing tools.
>
> If the final video is made up of many segments, this drift can accumulate over time and may reach several seconds in some cases. This is especially important when reprocessing the merged file with tools that depend on precise timing information, such as `whisper.cpp` for subtitle generation.
>
> If this happens, you may need to merge the parts again yourself while re-encoding the final output file to rebuild clean timestamps.
---

## Online Translation with Google Gemini API

The latest version introduces high-quality online translation using Google's Gemini AI models. This feature serves as a powerful alternative to the standard translation provided by `translate-shell`.

**Key Features:**
*   **Superior Quality:** Gemini models often provide more accurate and context-aware translations than traditional services.
*   **Model Selection:** You can choose from several available Gemini models, such as `gemini-3-flash-preview` (default) or the most powerful `gemini-3.1-pro-preview`, directly from the application's UI.
*   **API Key Integration:** The system securely manages your Google Gemini API key.

### Context Level Control

To further enhance translation quality, you can now control the amount of context the Gemini AI uses. This is crucial for achieving translations that are not only accurate but also fluent and coherent.

*   **Level 0 (Literal):** No context is used. Translates each segment independently. Best for simple, disconnected phrases.
*   **Level 1 (Minimal):** Uses the immediately preceding and succeeding segments for context. Good for general use.
*   **Level 2 (Standard - Default):** Uses a wider "window" of several surrounding segments. Excellent for understanding the flow of conversation.
*   **Level 3 (Creative):** Uses the same wide context as Level 2 but gives the AI permission to intelligently fix or complete fragmented sentences, which is ideal for live streams where words might be cut off.

You can select your preferred context level from the new dropdown menu in the "Online translation" section of the UI. If the API key is not found, it will automatically fall back to the standard `translate-shell` engine. In Live Stream mode, it will immediately fall back to translate-shell for a failed block to maintain real-time flow, indicated by a `(*)` prefix.

> [!NOTE]
> Using the Gemini API is subject to Google’s pricing and usage policies. Please consult the [Google AI Platform pricing page](https://ai.google.dev/pricing) for details.
> 
> The Gemini API **free tier** is available with lower rate limits for testing purposes. Google AI Studio usage is completely free in all supported countries. The Gemini API **paid tier** provides higher rate limits, additional features, and different data handling.

**Author:** Antonio R. | **Version:** 5.40 | **License:** GPL 3.0

---

## Related Project: Audio Transcription Browser Extension

If you are looking to transcribe and translate audio directly from your web browser (Chrome/Chromium/Microsoft Edge), check out my standalone extension project: **[Audio-Transcription](https://github.com/antor44/Audio-Transcription)**.

It is completely independent of `playlist4whisper` and `livestream_video.sh`, based on a local WhisperLive server. It provides real-time transcription, live translation (via Google Translate & Gemini), and Text-to-Speech (TTS) capabilities natively in your browser.

👉 **[Click here to visit the extension repository for more details and installation instructions.](https://github.com/antor44/Audio-Transcription)**

![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV12.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV5.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV13.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV10.jpg)

---

## playlist4whisper Quick Start

`playlist4whisper` and `livestream_video.sh` is based on `whisper.cpp` and also supports OpenAI's Whisper. It depends on other executables and libraries. Please ensure that `mpv`, `smplayer`, `translate-shell`, `ffmpeg`, `vlc`, `python3-tk`, `python3-pip`, `imageio`, `imageio-ffmpeg`, `Pillow`, `tkinterdnd2`, `jq`, `curl`, `bc`, and `xterm` are installed.

To install `whisper.cpp`, choose one of these options (you can install all and choose any, or Whisper executables are prioritized, with `./build/bin/whisper-cli` executable being the first choice):

```bash
pip3 install pywhispercpp
```

For macOS and Linux with brew repository:
```bash
brew install whisper-cpp
```

Or for OpenAI's Whisper (not fully supported, except for generating subtitles for local audio/video files):
```bash
pip3 install openai-whisper
```

For the latest version of `whisper.cpp` or to compile an accelerated version, follow the instructions provided at [whisper.cpp GitHub](https://github.com/ggerganov/whisper.cpp).

Finally, launch the app by entering the following command in the terminal. Make sure you are inside the `whisper.cpp` directory (`cd ~/whisper.cpp`) and that your Python environment is activated (`source ~/python-environments/whisper/bin/activate`), if required.

```bash
python3 playlist4whisper.py
```

> [!TIP]
> **VLC Configuration:** Before using VLC go to `Tools -> Preferences` and Show setting: `All`. Then `Interface -> Main Interfaces -> Qt -> Uncheck: 'Resize interface to the native video size'`, and `'When to raise the interface' -> Never`. Ensure that VLC is configured to repeat the playlist infinitely, not just the current file, and save the configurations.

> [!NOTE]
> **Model Files:** The required model files must be stored in the subdirectory `./models`. They can be automatically downloaded using `playlist4whisper` (this feature is only supported for the compiled version of whisper.cpp), or you can follow the instructions provided [here](https://github.com/ggerganov/whisper.cpp/blob/master/models/README.md).
>
> *Please note that the model installed by `playlist4whisper` may not be optimized for an accelerated version of Whisper.cpp.*
>
> *OpenAI's Whisper automatically downloads the required model if it does not exist. Keep in mind that its format is different from the whisper.cpp model.*

---

## Detailed Installation for Linux

### 1. Download and build whisper.cpp
Download and build `whisper.cpp` to a new directory following the instructions provided in the [documentation](https://github.com/ggerganov/whisper.cpp).

Download the source code of `whisper.cpp` from your home directory:
```bash
git clone https://github.com/ggerganov/whisper.cpp.git
```
Change the default directory to the `whisper.cpp` directory:
```bash
cd whisper.cpp
```
Compile `whisper.cpp` for CPU mode:
```bash
# build the project
cmake -B build
cmake --build build -j --config Release
# transcribe an audio file
./build/bin/whisper-cli -f samples/jfk.wav
```

### 2. Download playlist4whisper
Download `playlist4whisper` and `livestream_video.sh`:
```bash
git clone https://github.com/antor44/livestream_video.git
```
Stay where you executed the command, move all to `whisper.cpp` directory:
```bash
mv livestream_video/* ~/whisper.cpp
```
`playlist4whisper.py`, `livestream_video.sh`, and the default `playlist_xxx.m3u` files must be located in the same directory as `whisper.cpp`.

### 3. Install Dependencies
This program depends on other Linux programs and their libraries. For example, Ubuntu Linux users can install the following packages:
```bash
sudo apt-get install mpv smplayer translate-shell vlc ffmpeg make cmake python3-tk python3-pip jq curl bc xterm
```

You need to install some libraries via pip. One option is to install a python virtualenv:
```bash
sudo apt install virtualenv
```
Then:
```bash
mkdir ~/python-environments
virtualenv ~/python-environments/whisper
source ~/python-environments/whisper/bin/activate
```
And finally:
```bash
pip3 install imageio imageio-ffmpeg Pillow tkinterdnd2

```
*(Remember, in every session you need to run the command `source ~/python-environments/whisper/bin/activate` before executing playlist4whisper.py).*

For YouTube `yt-dlp` is required, for Twitch and Others `streamlink` is required. The easy way to install them:
```bash
pip3 install yt-dlp
pip3 install streamlink
```
Or to create a Python virtual environment alongside Python applications, install pipx:
```bash
sudo apt-get install pipx
pipx install yt-dlp
pipx install streamlink
```
And to upgrade them:
```bash
pip3 install --upgrade yt-dlp
pip3 install --upgrade streamlink
```

### 4. Launch the App
Make sure you are inside the `whisper.cpp` directory and that your Python environment is activated:
```bash
python3 playlist4whisper.py
```

> [!TIP]
> - **VLC Users:** Go to `Tools -> Preferences`, Show setting: `All`. Then `Interface -> Main Interfaces -> Qt -> Uncheck: 'Resize interface to the native video size'`, and `'When to raise the interface' -> Never`. Ensure VLC is configured to repeat the playlist infinitely.
> - **SMPlayer Multi-instances:** Go to `Preferences -> Interface -> Instances`, and turn off the option to use only one instance.

---

## Detailed Installation for macOS

You can run `playlist4whisper.py` on macOS by following these steps:

### 1. Install Homebrew
Visit [brew.sh](https://brew.sh/) and follow the installation instructions.

### 2. Install Dependencies
Once Homebrew is installed, open a terminal and run the following commands:
```bash
brew install make
brew install cmake
brew install python@3.12
brew install python-tk@3.12
brew install ffmpeg
brew install xquartz
brew install xterm
brew install vlc
brew install mpv
brew install smplayer
brew install translate-shell
brew install yt-dlp
brew install streamlink
brew install jq
```
Reboot, and then install a Python virtualenv:
```bash
brew install virtualenv
```
Then:
```bash
mkdir ~/python-environments
virtualenv -p python3.12 ~/python-environments/whisper
source ~/python-environments/whisper/bin/activate
```
Finally:
```bash
pip3 install imageio imageio-ffmpeg Pillow tkinterdnd2

```

### 3. macOS Version Compatibility Notes
`playlist4whisper` has been successfully tested on macOS versions for Mx or ARM without issues, on macOS Ventura (Intel), and also an older version of playlist4whisper/livestream.sh (successful version 2.60 + whisper.cpp v. 1.42) on very old versions like Big Sur (although it requires some extra adjustments).

> [!WARNING]
> For older versions of macOS, Homebrew has introduced significant changes in its recent versions. Depending on how Python was installed or updated, you may need to adjust your system settings to detect the new version of Python.
> 
> Also, for older macOS versions like Big Sur, you may encounter issues when installing Homebrew applications and compiling whisper.cpp. Alternatively, you can install older versions of applications (FFmpeg+FFplay+FFprobe, VLC, SMPlayer, MPV) from other sources. If you trust the source, copy the .app file to the Applications folder and link the path (in lowercase):
> ```bash
> ln -s /Applications/[executable].app/Contents/MacOS/[executable] /usr/local/bin/[executable_lowercase]
> ```
> For example, for VLC:
> ```bash
> ln -s /Applications/VLC.app/Contents/MacOS/VLC.app /usr/local/bin/vlc
> ```

### 4. Terminal Configuration (`xterm`)

To display text in a specific size and color, create a file named `.Xresources` in your home directory (/Users/[user]). Inside the file, add:
```text
.xterm*background: black
.xterm*foreground: yellow
.xterm*font: 10x20
.xterm*vt100*geometry: 80x10
.xterm*saveLines: 10000
.xterm*locale: true 
```
In this example, the settings modify the background color to black, the foreground color to yellow, the font size to 10x20, the terminal's geometry to 80x10, and the number of lines to save to 10,000. After saving these changes in the `.Xresources` file, you need to relaunch XQuartz for the new settings to take effect. Once you launch an xterm terminal, you will see the desired customization of the transcription text.

The final option `.xterm*locale: true` will enable the same language settings in xterm as those in your macOS's default terminal. Although you may need to make changes and/or install additional components to display characters in other languages.

### 5. Compile whisper.cpp
Compile `whisper.cpp` following the instructions in their documentation, or for CPU mode:
```bash
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
# build the project
cmake -B build
cmake --build build -j --config Release
# transcribe an audio file
./build/bin/whisper-cli -f samples/jfk.wav
```

Download `playlist4whisper` and move it:
```bash
git clone https://github.com/antor44/livestream_video.git
mv livestream_video/* ~/whisper.cpp
```

### 6. Launch the App
```bash
python3 playlist4whisper.py
```
> [!NOTE]
> On macOS, only the `xterm` terminal and the `mpv`/`vlc` video player are supported. Additionally, the `xterm` terminal automatically closes its window when `Control+C` is used.

---

## Detailed Installation for Windows 10/11

`playlist4whisper` can run on Windows Subsystem for Linux (WSL2), which is the default virtual system on Windows 10/11 for running native Linux software. Keep in mind that WSL2 may not provide the same level of stability and smooth performance as Linux or macOS.

Open PowerShell or Windows Command Prompt in administrator mode:
```bash
wsl --install
```
This command will enable the features necessary to run WSL and install the Ubuntu distribution. Once this is done, you will need to restart.

If Ubuntu (or whatever recent Linux distribution among available ones) didn't install, then:
```bash
wsl.exe --install Ubuntu
```

Open the **Linux terminal** (not Windows), update the packages, and install dependencies:
```bash
sudo apt update
sudo apt upgrade
sudo apt-get install mpv smplayer translate-shell vlc ffmpeg make cmake python3-tk python3-pip jq curl bc gnome-terminal xterm
```

Set up your Python virtual environment:
```bash
sudo apt install virtualenv
mkdir ~/python-environments
virtualenv ~/python-environments/whisper
source ~/python-environments/whisper/bin/activate
pip3 install imageio imageio-ffmpeg Pillow tkinterdnd2
```

Install `yt-dlp` and `streamlink`:
```bash
pip3 install yt-dlp streamlink
```

Download and compile `whisper.cpp`:
```bash
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
# build the project
cmake -B build
cmake --build build -j --config Release
# transcribe an audio file
./build/bin/whisper-cli -f samples/jfk.wav
```

Download `playlist4whisper`:
```bash
git clone https://github.com/antor44/livestream_video.git
mv livestream_video/* ~/whisper.cpp
```

Launch the app:
```bash
python3 playlist4whisper.py
```
> [!NOTE]
> MPV player is not working on Windows, potentially due to the need for additional configuration and/or installation of packages.

---

## Usage

> [!WARNING]
> **To use the Timeshift feature, ensure that VLC is configured to repeat the playlist infinitely, not just the current file.**

Make sure that you are in the same directory as `whisper.cpp`, `playlist4whisper.py`, and `livestream_video.sh`, and also ensure that the Python environment for `playlist4whisper` is active. Then:
```bash
python3 playlist4whisper.py
```

`playlist4whisper` accepts optional command-line arguments to add any number of tabs:
- `--tabs`: Accepts a list of tab names. If a tab name contains spaces, enclose it in double quotes.
- `--colors`: Accepts a list of tab colors (e.g., "red" or "#ff0000").

**Example:**
```bash
python playlist4whisper.py --tabs Tab1 Tab2 Tab3 --colors red green "#ff7e00"
```

*Check out the section on `livestream_video.sh` for help with the available options.

Local audio/video files must be referenced with the full file path. Alternatively, if the file is in the same directory, it can be referenced with `./` preceding the file name.
The program will load the default playlists `playlist_iptv.m3u`, `playlist_youtube.m3u`, etc., and will store options in `config_xxx.json`.

---

## livestream_video.sh

This is a command-line program that includes the same transcription functions as `playlist4whisper` and can be run independently of the GUI. Options are set via parameters in a terminal, or you can create desktop shortcuts for each TV channel.

`livestream_video.sh` is a Linux script that transcribes video livestreams by feeding `ffmpeg` output to `whisper.cpp` at regular intervals. It is based on `livestream.sh` from the `whisper.cpp` project.

> [!NOTE]
> *The Text-to-Speech (TTS) feature is performed online via the **Translate Shell** app, which utilizes a free Google service. The availability of this service is not guaranteed. Additionally, the TTS feature is designed for short audio segments and is limited to certain languages.*

### Features Overview:
-   **Online Translation:** Translate transcriptions in real-time using either Google Translate or the high-quality Google Gemini API.
-   **Context Control for AI:** Fine-tune Gemini translations with context levels (0-3).
-   **VAD Support:** Integrated Voice Activity Detection.
-   **Text-to-Speech (TTS):** Read translated text aloud using Translate Shell.
-   **Subtitle Generation:** Automatically create `.srt` subtitle files.
-   **Timeshift:** A fully configurable timeshift feature (VLC Player only).
-   **Broad Service Support:** IPTV, YouTube, Twitch, and anything supported by `streamlink` or `yt-dlp`.
-   **Multi-Instance Execution:** Run multiple instances of the script simultaneously.
-   **Whisper AI Options:** Full control over Whisper parameters, autodetection, direct translation (`--translate`), and quantized models.
-   **Multi-Platform:** Compatible with Linux, macOS, and Windows (via WSL2).
-   **Audio Inputs:** Transcribe from any audio input, including loopback devices to capture desktop audio (`pulse:index` / `avfoundation:index`).

### Usage Options

**Syntax:**
```bash
./livestream_video.sh stream_url[or /path/media_file or pulse:index or avfoundation:index] [--step step_s] [--model model] [--language language] [--executable exe_path] [--translate] [--vad] [--subtitles] [--timeshift] [--segments segments (2<n<99)][--segment_time minutes (1<minutes<99)][--sync seconds (0 <= seconds <= (Step - 3))] [--trans trans_language output_text speak] [--gemini-trans [gemini_model]][--gemini-level [0-3]] [player player_options]
```

**Example:**
```bash
./livestream_video.sh https://cbsn-mia.cbsnstream.cbsnews.com/out/v1/ac174b7938264d24ae27e56f6584bca0/master.m3u8 --lower --step 8 --model base --language auto --translate --vad --timeshift --segments 4 --segment_time 10 --trans es both speak --gemini-trans --gemini-level 3
```

>[!TIP]
> **Only for the bash script and only for local audio/video:** Files must be enclosed in double quotation marks, with the full path. If the file is in the same directory, it should be preceded with `./`.
> 
> **pulse:index or avfoundation:index:** Live transcription from the selected device index. Pulse for PulseAudio for Linux and Windows WSL2, AVFoundation for macOS. The quality of the transcription depends on your computer's capabilities, the chosen model, volume, and ambient noise. Please note that this is a preliminary feature. There is a several-second delay between live sound and transcriptions, with no possibility for synchronization.

### Argument Options

**Stream Handling**
- `--streamlink`: Forces the URL to be processed by Streamlink.
- `--yt-dlp`: Forces the URL to be processed by yt-dlp.
- `--[raw, upper, lower]`: Video quality options. Affects timeshift for IPTV.

**Player**
- `--player`: Specify player executable and options (`smplayer`, `mpv`, `mplayer`, `vlc`). Use `'none'` or `'true'` for no player.

**Whisper AI Configuration**
- `--step`: Size of the sound parts into which audio is divided for AI inference (seconds).
- `--model`: Base Models (`tiny`, `base`, `small`, `medium`, `large-v1`, `v2`, `v3`, `turbo`) or Quantized Suffixes (`-q4_0`, `-q8_0`, etc.).
- `--executable`: Specify the whisper executable to use.
- `--language`: Transcription language (`es`, `fr`, `de`, `auto`, etc.).
- `--translate`: Automatic English translation using Whisper AI.
- `--subtitles`: Generate subtitles (`.srt`) from a local audio/video file.
- `--vad`: Enable VAD (Voice Activity Detection) to find silences near the step boundary. Uses whisper.cpp's built-in Silero VAD model.

**Online Translation**
- `--trans`: Enables online translation. Must be followed by a language code (`es`, `fr`), output text (`original`, `translation`, `both`, `none`), and optionally `speak` for TTS.
- `--gemini-trans`: Use the Google Gemini API for higher quality translation.
- `--gemini-level`: Set the context level for Gemini translation (0-3).

**Timeshift Configuration (VLC Player Only)**
- `--timeshift`: Enables the timeshift feature.
- `--sync`: Transcription/video synchronization time in seconds.
- `--segments`: Number of segment files for timeshift.
- `--segment_time`: Time for each segment file.

---

## 🌐 Online Translation with Google Gemini API

This application introduces high-quality online translation and grammar correction using Google's Gemini AI models. This feature serves as a powerful alternative to the standard translation provided by `translate-shell`.

### 1. ⚠️ Privacy Warning: Free Tier vs. Paid Tier
Google AI Studio offers a generous Free Tier, but it comes with a critical privacy trade-off:
*   **Free Tier:** By using the free API, you agree to Google's terms, which allow them to collect and use your translated text (anonymously) to train and improve their AI models. 
*   **Paid Tier (Pay-As-You-Go):** When you set up a billing account, **your data is strictly private**. Google explicitly states that Paid API data is *not* used to train their models.

**For processing private, copyrighted, or sensitive media files, enabling the Paid Tier is highly recommended.** The Paid API operates strictly on a pay-per-use basis: if you don't use it, you pay nothing. Under normal usage, translating or correcting subtitles costs just fractions of a cent per video.

### 2. Model Recommendations & Rate Limits
Different tasks require different models. Based on current Google AI Studio limits, here is how you should configure the app:

*   **For Subtitle Generation (Local Files):**
    *   *Recommended Model:* **`gemini-3-flash-preview`** (or `gemini-3.1-flash-lite`).
    *   *Why:* Subtitle generation processes the entire file in rapid bursts. The Flash models offer a very high burst limit (up to 1,000 Requests Per Minute), preventing the API from timing out while processing your video.
*   **For Prolonged Live Streams (IPTV / Twitch / YouTube):**
    *   *Recommended Model:* The **`gemma-4`** family (e.g., `gemma-4-26b-a4b-it`).
    *   *Why:* Live streams run for hours, doing slow, steady requests. Gemma-4 models have a massive daily quota (over 14,000 Requests Per Day) on the Free Tier, making them perfect for long sessions. *Tip: Use Context Level 0 or 1 to avoid hitting their low per-minute rate limit (30 RPM).*
*   **For High-Quality / Minority Languages:**
    *   *Recommended Model:* **`gemini-3.1-pro`**.
    *   *Why:* Pro models offer the absolute best translation quality and context awareness, especially when either the source or destination language is not widely spoken. A Paid Tier account is generally recommended to avoid rate limits with Pro models.

### 3. How to Obtain and Set Your API Key
To use these features, you need a Google Gemini API Key.

**Step 1: Obtain the Key**
1. Go to **[Google AI Studio](https://aistudio.google.com/)**.
2. Sign in with your Google account.
3. Click on **"Get API key"** and then **"Create API key"**.
4. Copy the generated key immediately.

**Step 2: Provide the Key to the Application**
You must make the API key available to the script. There are three methods:

*   **Method A: System-Wide Environment Variable (Recommended for Desktop Users)**
    Open `~/.profile` with a text editor and add the following line at the end:
    ```bash
    export GEMINI_API_KEY="YOUR_API_KEY_HERE"
    ```
    *(Log out and log back in for changes to take effect).*

*   **Method B: Shell-Specific (Recommended for Terminal Users)**
    Open your `~/.bashrc` (or `~/.zshrc`) and add the same line:
    ```bash
    export GEMINI_API_KEY="YOUR_API_KEY_HERE"
    ```
    *(Run `source ~/.bashrc` to apply immediately).*

*   **Method C: Set in the GUI (Easiest)**
    In the `playlist4whisper` GUI, click the **"API Key"** button located in the "Online translation" section. Paste your key and click "OK". The GUI will automatically pass the key to the bash script.

### 4. Using Gemini Translation via Command Line
If you are running the `livestream_video.sh` script directly from the terminal, use these flags:
* Add `--trans <language_code>` (e.g., `--trans en`) to enable translation.
* Add `--gemini-trans` to force the use of the Gemini engine instead of `translate-shell`.
* *(Optional)* Specify a model: `--gemini-trans gemini-3-flash-preview`.
* *(Optional)* Specify a context level: `--gemini-level 2`.

*Note: If the Gemini API fails, times out, or the key is missing, the script is designed to automatically fallback to the standard `translate-shell` engine to ensure your live stream translation is never interrupted.*

---

## More Screenshots

**playlist4whisper Screenshots**
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV2.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV8.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV1.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV7.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV9.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV11.jpg)

**livestream_video.sh Screenshots:**
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV3.jpg)
 
![Screenshot](https://github.com/antor44/livestream_video/blob/main/whisper_TV4.jpg)

---

## To-Do List

- Advanced GUI as a standalone application.
- Support for different AI engines.
- Sound filters.
- ...

---

## FAQ

### **Q: What quality of transcription can I expect when using only a low-level processor?**

**A:** This program is based on `whisper.cpp`, which is a highly optimized implementation of OpenAI's Whisper AI. The performance of the transcription largely depends on this software. For English language, you can expect very good transcriptions of video streams or media files even on low-end or old PCs, even those that are at least 10 years old. You can easily configure the application with models such as `small.en` or `base.en`, which offer excellent transcriptions for the English language. Even the `tiny.en` model, despite its small size, provides great results. However, transcriptions of other major languages are not as good with small models, and minority languages do not perform well at all. For these, you will need a better CPU or a supported GPU.

Another option is to fine-tune a model to improve transcription for a non-English language.

### **Q: How much data is needed to fine-tune a model?**

**A:** Fine-tuning a Whisper model can be harder and more expensive than expected, and the amount of data required depends on the target language and use case. Some users get good results with a relatively small dataset, while others see little or no improvement. Results depend on how well Whisper already supports the language, how similar it is to other languages it knows, and the quality of the training data. In general, more high-quality data is better. Ideally, this means thousands of hours of short audio clips with accurate transcriptions.

You may be able to fine-tune with free datasets such as Common Voice. You can find the latest version of the Common Voice dataset on the [Mozilla Foundation](https://commonvoice.mozilla.org) page. Keep in mind that OpenAI already used Common Voice data for validation during training, so some languages or datasets may not improve Whisper as much as expected. Even so, languages such as Catalan, Esperanto, and Basque have a significant amount of data in Common Voice, while some major languages such as Spanish have a surprisingly weak dataset.

For specialized use cases, the effort required can vary substantially. For example, when improving technical vocabulary, slang, or a regional accent, you may need less effort to collect enough data if the language is already reasonably well supported.

A good example is Hindi: it was not especially well supported by Whisper at first, but after sufficient fine-tuning it now comes much closer to English in accuracy and efficiency. In practice, that means much better transcription even with the smallest models on low-powered processors or mobile devices. For more information, see the Collabora article: [https://www.collabora.com/news-and-blog/news-and-events/breaking-language-barriers-20-moving-closer-production-ready-hindi-asr.html](https://www.collabora.com/news-and-blog/news-and-events/breaking-language-barriers-20-moving-closer-production-ready-hindi-asr.html)

The Hindi fine-tuned model was trained using a relatively accessible consumer GPU, specifically an Nvidia RTX 4090. That suggests the most important factors are the training data and how it is processed, rather than raw training power alone.

### **Q: Why isn't there a precompiled and packaged distribution for the program "playlist4whisper"?**
**A:** First, this is a Linux application, and macOS and Windows are not fully supported. For Linux systems, compiling and distributing packages is the responsibility of each distribution’s maintainers, not the programmer.

Linux prioritizes security, which is best ensured through precompiled packages stored in official repositories—especially in major Linux distributions, on which many others depend.

'whisper.cpp' itself is not available in the repositories of major Linux distributions, despite its reputation for being significantly more efficient than OpenAI's original implementation. It can only be installed via PyPI—a repository for Python software—or Homebrew, a non-traditional package manager.

Additionally, Linux distributions are very different, even Python environments, and it is not easy to support every one, especially with this project that depends on numerous other libraries and programs.

The absence of precompiled ‘playlist4whisper’ packages on the project’s webpage is due to frequent updates and optional hardware-specific optimizations in the underlying ‘whisper.cpp’ program. By providing the source code, users can adapt to these ongoing changes and tailor performance optimizations to their hardware and preferences.

Keep in mind that compiling `whisper.cpp` with certain hardware accelerations—such as CUDA on any NVIDIA RTX graphics card—can result in a significant performance boost, compared to the default or CPU-optimized builds of `whisper.cpp`, including those running on powerful modern CPUs.

However, compiling whisper.cpp with CUDA is not always a trivial task: the success of the build or executable will depend on factors such as the CUDA version, NVIDIA driver, compiler version, and other libraries installed on the user’s Linux operating system.

Another question is that "playlist4whisper" relies on the included bash script "livestream_video.sh". This script can be executed independently, supporting accessibility technologies. It can also run on Linux terminals without a desktop environment and potentially be used as a server application in multi-instance and multi-user scenarios, making it versatile for various use cases. By providing the source code, advanced users can review or customize the programs to suit their specific requirements and environments.

### **Q: In a low-power processor, is it possible to improve transcription in languages other than English?**
**A:** Yes, several advanced methods can significantly boost performance:

*   **GPU/NPU Acceleration:** Instead of running the AI engine on the CPU, you can compile `whisper.cpp` with hardware acceleration. This can increase execution speed by 2x or more, allowing you to use larger, more accurate models. Supported technologies include:
    *   **NVIDIA GPU:** via [cuBLAS](https://github.com/ggerganov/whisper.cpp#nvidia-gpu-support-via-cublas).
    *   **Cross-Vendor GPU (NVIDIA, AMD, Intel):** via [Vulkan](https://github.com/ggerganov/whisper.cpp#vulkan-gpu-support) or [OpenCL (CLBlast)](https://github.com/ggerganov/whisper.cpp/pull/1037).
    *   **Intel CPU/GPU:** via [OpenVINO](https://github.com/ggerganov/whisper.cpp#openvino-support).
    *   **Apple Silicon (M1/M2/M3/M4):** via [Core ML](https://github.com/ggerganov/whisper.cpp#core-ml-support) to use the Apple Neural Engine (ANE), offering speed-ups of over 3x.
    *   **Ascend NPU:** via [CANN](https://github.com/ggerganov/whisper.cpp#ascend-npu-support).
    *   **Moore Threads GPU:** via [MUSA SDK](https://github.com/ggerganov/whisper.cpp#moore-threads-gpu-support).
*   **CPU Acceleration:** For CPU-only systems, you can still get a boost by compiling with [OpenBLAS](https://github.com/ggerganov/whisper.cpp#blas-cpu-support-via-openblas) or [POWER VSX](https://github.com/ggerganov/whisper.cpp#power-vsx-intrinsics).
*   **Quantized Models:** You can try using the quantized models option, which can improve execution speed on certain processors with minimal loss in accuracy.
*   **Fine-Tuning:** If you have AI programming experience, you can fine-tune a default model by retraining it with a dataset of voices and transcriptions in a specific language. This can also improve recognition of specific accents, slang, or dialects. You can find instructions for converting models to the required `ggml` format on the [whisper.cpp repository](https://github.com/ggerganov/whisper.cpp/blob/master/models/README.md).

Keep in mind that compiling `whisper.cpp` with certain hardware accelerations—such as CUDA on any NVIDIA RTX graphics card—can result in a significant performance boost, compared to the default or CPU-optimized builds of whisper.cpp, including those running on powerful modern CPUs.

The CPU and CPU-accelerated builds of `whisper.cpp` are limited by default to 4 threads, as specified in whispercpp’s default configuration. This setting helps preserve system resources, allowing other applications to run smoothly or enabling multi-instance and multi-user setups. However, modern CPUs can often achieve significantly better performance with more threads. To adjust this, edit the `livestream_video.sh` script and modify the two lines that define the default arguments for `WHISPER_EXECUTABLE`. Replace `-t 4` with a higher value, such as `-t 16`, to allocate 16 threads per instance. For other variants of the whisper executable, also update the arguments `--n_threads 4` and `--threads 4` as needed.

*The accelerated versions of `whisper.cpp` may require specific model versions to achieve better performance.*

### **Q: How many simultaneous instances can be run on a PC?**
**A:** On a PC or any other computer, it depends on its hardware, the models chosen for each of the executed instances, as well as the power and memory, both in CPU and GPU. There is also the possibility of hybrid execution, with several instances running on the CPU and system RAM, and others on the GPU and graphics card VRAM. It also differs if it involves real-time transcriptions, while subtitle generation for long pre-recorded files could be executed without the need for them to finish in an urgent time frame.

For older PCs such as Intel i7/Xeon processors from the Haswell series, for real-time transcriptions with CPU without acceleration, models up to `base` or `small` can be run at most, perhaps 2-3 instances at a time. On modern systems, the possibilities are much greater; even with a mid-range graphics card like any NVIDIA RTX, it is possible to run many instances with larger models like `large-v2`.

It is important to clarify that `playlist4whisper.py` and `livestream_video.sh` are currently based on `whisper.cpp`, which requires each process to load its own full copy of the model into VRAM, or into system RAM when using CPU without GPU acceleration. This differs from architectures like `TensorRT` and `faster-whisper`, which can serve multiple concurrent clients from a single loaded model—a feature that may be supported in future versions.

**Critical Distinction: Short Chunks vs. Continuous Processing**
Real-world testing on an **NVIDIA RTX 16GB** with the `large-v2` model reveals two dramatically different scenarios:

*   **Short Audio Chunks (8-10 seconds) - High Concurrency:**
    - Up to 10+ simultaneous instances achievable without perceivable errors
    - Each instance continuously processes short audio chunks (8-10 seconds), with VRAM being allocated per chunk and released upon completion
    - Fast chunk processing prevents memory saturation
    - Ideal for video streaming, live transcription
*   **Continuous Processing (Subtitles) - Limited Concurrency:**
    - Only 3-4 instances before Segmentation Faults and crashes
    - Memory accumulates without release, causing exhaustion
    - Processing time per audio minute degrades from 5-7s (1-2 instances) to 12s (3-4 instances) as GPU resources become saturated
    - Batch processing fails beyond 3-4 instances
    - However, for subtitle generation workflows, longer processing times or delayed instance launches are acceptable since real-time performance is not required
*   **Mixed Workload:**
    - 5 short-chunk base instances + only 1 long-file instance before failure

> [!NOTE]
> Mid-range NVIDIA RTX GPUs can likely achieve 20-30+ concurrent instances for short-chunk processing, as these GPUs process up to 1 minute of audio in just a few seconds. However, neither `playlist4whisper.py` nor `livestream_video.sh` includes concurrency control at the code level (meaning external load balancing cannot manage this), making unpredictable errors occur when cumulative VRAM requirements exceed physical GPU memory. Use quantized models (e.g., `Q8_0`) to drastically reduce VRAM consumption and eliminate paging risks.

### **Q: Do quantized models run faster on NVIDIA RTX GPUs with playlist4whisper.py and livestream_video.sh, or do they only reduce VRAM usage? Does whisper.cpp leverage RTX 5000 series optimizations for 4-bit quantized models?**
**A:** Important context first: Modern GPUs and CPUs are more than capable of running large models like `large-v2` or `large-v3` for live transcription without breaking a sweat. The main benefit of quantization is enabling more concurrent instances on the same hardware, although in many cases, performance gains can also be achieved. In scenarios with multi-instance or multi-user executions, even small gains accumulate geometrically. This is particularly evident in Whisper applications.

These improvements are especially valuable in business or public administration environments, translating into cost savings for hardware or cloud usage. This applies to many AI applications, not just Whisper.

For `playlist4whisper.py` and `livestream_video.sh`, currently only `whisper.cpp` can be used as a backend. It's a stable, lightweight implementation with few dependencies and broad architecture support. I've never encountered stability issues in long-running or multi-instance executions. It allows using fine-tuned models for accents, slang, or technical terms—useful in diverse linguistic contexts like mixing languages or regional variations.

If `TensorRT`or `faster-whisper` (based on CTranslate2) could be integrated, it would excel in raw concurrency (16 or more instances with one large model in memory), but `whisper.cpp` remains reliable for now.

With `whisper.cpp` (current backend) on NVIDIA RTX GPUs:
Quantized models reduce VRAM consumption AND often improve processing speed thanks to Tensor Core support for INT8/INT4.

**VRAM Reduction - Guaranteed Benefit:**
*   **Q8_0 (INT8):** ~50% less VRAM than FP16
*   **Q5_0/Q5_1 (INT5):** ~65% less
*   **Q4_0/Q4_1 (INT4):** ~75% less

This enables more concurrent instances (e.g., 6-8 instead of 3-4 with `large-v2` on 16GB VRAM).

**Speed Performance - Verified Results:**
Based on user benchmarks in the `whisper.cpp` repository (`ggml_mul_mat` and encode timings via CUDA), quantization typically provides gains on RTX GPUs with Tensor Cores (2000 series+). Q8_0 is often the fastest, with notable GFLOPS improvements in large matrix operations, though varying by GPU, threads, and workload.

Real user examples (from benchmark threads):
*   On modern RTX GPUs (including 50xx series entries), Q8_0 achieves significantly higher GFLOPS than FP16 for large matrices.
*   On RTX 4090, transcription times are very similar between FP16 and Q8_0, but with half the memory.

**Key takeaway:** On RTX with Tensor Cores, Q8_0 is usually faster than FP16 with much less VRAM—ideal for most cases. Q4_0 is great for max concurrency when memory-limited (still faster than FP16 on many RTX, but below Q8_0). On older GPUs without Tensor Cores (e.g., GTX 10xx), speed may be similar or slightly worse, but VRAM savings remain.

**Deduced Performance for Older GPUs like GTX 1080 Ti:**
*   **GTX 1080 Ti** (older GPU without Tensor Cores): Q8_0 ~28 GFLOPS vs FP16 ~29 GFLOPS (slightly slower, deduced from similar older GPU patterns and user reports).

The primary benefit on such hardware is VRAM reduction, with neutral or minor speed impact from quantization.

**Quality:**
All quantization levels maintain high transcription accuracy, with no significant degradation in real tests.

**Understanding Whisper.cpp Quantization and GPU Acceleration**
`whisper.cpp` uses integer quantization (INT4/INT5/INT8), not floating-point. It leverages Tensor Cores on RTX 2000+ for INT8 and INT4. The RTX 5000 series (Blackwell) adds FP4 support, but `whisper.cpp` benefits from existing INT support since Turing.

*   All RTX from 2000 support INT4/INT8 via Tensor Cores. Gains on 50xx for `whisper.cpp` come from the same INT support, not new FP4.

> [!TIP]
> **In simple terms:** Use **Q8_0** models (faster + half memory). Switch to **Q4_0** only for maximum concurrent streams when out of memory. Avoid Q5_0 unless specific needs.

### **Q: How do I configure whisper.cpp for hardware accelerations (e.g., CUDA, Core ML, OpenVINO) and generate the specific models needed? Why don't the models downloaded by `playlist4whisper.py` work with all accelerations?**
**A:** Whisper.cpp supports various hardware accelerations (CPU, CUDA, Core ML, OpenVINO, Vulkan, BLAS, CANN, MUSA), each requiring specific compilation flags and, for some, custom model formats. The models downloaded by `playlist4whisper.py` are in the standard `ggml` format (`.bin`) and work only with CPU, CUDA, Vulkan, BLAS, CANN, and MUSA backends. **These models do not work with Core ML (Apple M1/M2/M3/M4 NPU) or OpenVINO (Intel CPU/GPU) without conversion to their specific formats.** Core ML or OpenVINO require converted models (`.mlmodelc` for Core ML, `.xml`/`.bin` IR for OpenVINO). Attempting to use a standard `.bin` with these backends will fail, as whisper.cpp expects the optimized format in the `models/` directory. Below are the steps to compile whisper.cpp for each acceleration, generate the required models, and test them with `livestream_video.sh`.

**Commands for Compilation and Model Generation**  
Download the source code of `whisper.cpp` from your home directory:
```bash
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp
```

#### 1. CPU (Standard, No Acceleration)
- **Compilation**:  
```bash
cmake -B build
cmake --build build -j --config Release
```
- **Model**: Download a standard model:  
```bash
make base.en
```
  - Optional: Quantize for reduced memory/faster processing:  
```bash
./build/bin/whisper-quantize models/ggml-base.en.bin models/ggml-base.en-q8_0.bin q8_0
```
- **Run**:  
```bash
./livestream_video.sh ./samples/jfk.wav --model base.en --subtitles
```
*(Models from `playlist4whisper.py` work directly).*

#### 2. CUDA (NVIDIA GPU, e.g., RTX 4060 Ti)
- **Requirements**: Install CUDA toolkit (https://developer.nvidia.com/cuda-downloads).  
- **Compilation**:  
```bash
cmake -B build -DGGML_CUDA=1
cmake --build build -j --config Release
```
  - For newer GPUs (e.g., RTX 5000 series):  
```bash
cmake -B build -DGGML_CUDA=1 -DCMAKE_CUDA_ARCHITECTURES="86"
cmake --build build -j --config Release
```
- **Model**: Use the same `.bin` as for CPU:  
```bash
make base.en
```
  - Optional: Quantize:  
```bash
./build/bin/whisper-quantize models/ggml-base.en.bin models/ggml-base.en-q8_0.bin q8_0
```
- **Run**:  
```bash
./livestream_video.sh ./samples/jfk.wav --model base.en --subtitles
```
*(Models from `playlist4whisper.py` work directly).*

#### 3. Core ML (Apple Silicon M1/M2/M3/M4 NPU)
This option enables hardware-accelerated transcription using the Neural Engine on Apple Silicon Macs.
- **Requirements**: macOS Sonoma (14 or later) and Xcode command-line tools (`xcode-select --install`).
- **Python Environment (Choose one option)**:
  - **Option A (Recommended): Using `virtualenv`**
    ```bash
    brew install virtualenv
    virtualenv -p python3.12 whisper-env
    source whisper-env/bin/activate
    pip install ane_transformers openai-whisper coremltools
    ```
  - **Option B (Alternative): Using `venv`**
    ```bash
    python3.12 -m venv whisper-env
    source whisper-env/bin/activate
    pip install ane_transformers openai-whisper coremltools
    ```
  - **Option C (Alternative): Using `Miniconda` or `Conda`**
    ```bash
    conda create -n py312-whisper python=3.12 -y
    conda activate py312-whisper
    pip install ane_transformers openai-whisper coremltools
    ```
- **Compilation**:
  ```bash
  cmake -B build -DWHISPER_COREML=1
  cmake --build build -j --config Release
  ```
- **Generate Model**:
  ```bash
  sh ./models/generate-coreml-model.sh base.en
  ```
  *(This creates the `models/ggml-base.en-encoder.mlmodelc` file).*
- **Run**:
  ```bash
  ./livestream_video.sh ./samples/jfk.wav --model base.en --subtitles
  ```
*(Models downloaded via `make` or used directly by `playlist4whisper.py` **will not work** for Core ML acceleration. You must generate the `.mlmodelc` model).*

#### 4. OpenVINO (Intel CPU/GPU)
- **Requirements**:
  - Python Environment:
    ```bash
    cd models
    python3 -m venv openvino_conv_env
    source openvino_conv_env/bin/activate
    python -m pip install --upgrade pip
    pip install -r requirements-openvino.txt
    ```
  - OpenVINO Toolkit: Install from [Intel](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit-download.html).
  - Set up Environment: `source /path/to/openvino/setupvars.sh`
- **Compilation**:
  ```bash
  cmake -B build -DWHISPER_OPENVINO=1
  cmake --build build -j --config Release
  ```
- **Generate Model**:
  ```bash
  python convert-whisper-to-openvino.py --model base.en
  ```
- **Run**:
  ```bash
  ./livestream_video.sh ./samples/jfk.wav --model base.en --subtitles
  ```
*(Models from `playlist4whisper.py` **do not work**. You must generate the OpenVINO IR model).*

#### 5. Vulkan (AMD/Intel/NVIDIA GPUs)
- **Requirements**: Install drivers with Vulkan API support.  
- **Compilation**:  
```bash
cmake -B build -DGGML_VULKAN=1
cmake --build build -j --config Release
```
- **Model / Run**: Same as CPU.

#### 6. BLAS (CPU with OpenBLAS)
- **Requirements**: Install OpenBLAS (https://www.openblas.net/).  
- **Compilation**:  
```bash
cmake -B build -DGGML_BLAS=1
cmake --build build -j --config Release
```
- **Model / Run**: Same as CPU.

#### 7. Ascend NPU (Huawei Atlas 300T A2)
- **Requirements**: Install CANN toolkit.  
- **Compilation**:  
```bash
cmake -B build -DGGML_CANN=1
cmake --build build -j --config Release
```
- **Model / Run**: Same as CPU.

#### 8. MUSA (Moore Threads GPU)
- **Requirements**: Install MUSA SDK rc4.2.0.  
- **Compilation**:  
```bash
cmake -B build -DGGML_MUSA=1
cmake --build build -j --config Release
```
- **Model / Run**: Same as CPU.

### **Q: Why do I sometimes get errors or poor-quality translations with Gemini AI?**
**A:** Online translation issues with the Gemini API can stem from several factors, ranging from API availability to the inherent behavior of AI models. Here are the most common causes:
*   **API Unavailability:** The service may be temporarily unavailable or experiencing high traffic. The script handles this differently depending on the mode:
    *   In **Subtitle Generation**, it will retry a few times before falling back to `translate-shell`.
    *   In **Live Stream** mode, it will immediately fall back to `translate-shell` for a failed block to maintain real-time flow, indicated by a `(*)` prefix.
*   **API Rate Limits:** The primary cause of failures is exceeding the usage limits imposed by Google, which are **particularly strict on the free tier**. While paid tiers have much higher limits, the following applies to free accounts:
    *   **Gemma 4 models** have a very high daily limit on the free tier (**RPD**) but a very low per-minute limit (**TPM**). This makes them excellent for **prolonged, low-intensity use** (such as long live streams), but they may fail on high-intensity tasks (such as subtitle generation) that exceed the TPM limit.
    *   **Gemini 3 Flash models** on the free tier have the opposite profile: a low daily limit but a high per-minute limit. This makes them ideal for **short, high-intensity tasks** like generating subtitles, but their daily quota can be exhausted during long live streams.
*   **Inherent AI Translation Errors:** Even with a stable connection and within rate limits, all Gemini API models can occasionally produce translation errors. Users should be aware that issues such as **confusing languages** (especially with multilingual source text), processing previous sentences and **modifying timestamps** in subtitle files, or **occasionally repeating previous phrases** may occur.

**Model Recommendations & Strategies (for Free Tier Users):**
*   **For Subtitle Generation:** Use a **Gemini 3 Flash model** (`gemini-3-flash-preview` or `gemini-3-flash-lite`). Their high TPM can handle the processing burst required for an entire file.
*   **For Prolonged Live Streams (Hours):** Use a **Gemma 4 model**. Its large daily request quota is ideal for long-running sessions. To avoid hitting the low TPM limit during dense dialogue, it is highly recommended to use a lower context level. You can do this by selecting "Level 0" or "Level 1" from the "Gemini Level" menu in the `playlist4whisper` application, or, if using the `livestream_video.sh` script independently, by adding `--gemini-level 0` or `--gemini-level 1` to your command.
*   **For Moderate Live Streams (Casual Use):** The **`gemini-3-flash-lite`** model (or the recent **`gemini-3-flash-preview`** model) is the best all-around choice, offering a strong balance of quality, speed, and a reasonable daily quota when using the default context level.

**Model Recommendations for High-Quality Subtitles (Long Media):**
*   **Paid API key recommended:** The Gemma 4 model on the Free Tier can work when both source and destination are major languages, but for reliable, high-quality subtitles on large media files (films, long audio, or videos longer than a few minutes), use **Level 2** — or the more creative **Level 3** — with at least the **Gemini-3-Flash** model. It provides good results for major-language translations and for improving same-language transcription. Use **Gemini 3.1 Pro** when either the source or destination language is non-major.

### **Q: Why do subtitles translated via the Gemini API sometimes differ in quality from the web version in Google AI Studio?**
**A:** While the quality from the API is very high, you may notice that pasting an entire SRT file into a web chat like Google AI Studio can sometimes yield superior results. The primary reason for this is **global context**. When you paste a full SRT file into the web interface, it generally uses a powerful model like Gemini 2.5 Pro and can process the entire document as a single piece of context. This allows the AI to understand overarching themes and the relationships between distant parts of the dialogue, resulting in excellent quality translations in a wide range of languages. However, this manual method is less practical and can fail with very long texts, and occasionally generates other errors as well.

The application, on the other hand, processes the subtitle file in **segmented parts** to handle very large files robustly without failing. To maintain coherence, it provides the AI with a "sliding window" of context, including several phrases before and after the current segment being translated. For most content, the quality difference is barely noticeable. The most significant drop in quality usually occurs due to external factors like API server overload or rate limit errors. You may occasionally see minor errors, such as the repetition of words.

Despite this, the script's approach is highly effective. Even with smaller models like `gemini-2.5-flash-lite`, it does an excellent job with common languages, often correcting misspelled words and accurately identifying well-known entities such as places, acronyms, companies, organizations, political parties, and the names of famous people. However, it's important to note a universal limitation: when the name of an unknown person is transcribed incorrectly (e.g., “Jon” or “Jan” instead of “John”, or “Sara” instead of “Sarah”), the AI has no way of determining the correct spelling — a challenge that even the web version of the AI cannot overcome.

### **Q: What's the use of the loopback ports? Could I see my videos from the internet?**
**A:** Loopback ports are needed for features like timeshift, and for upper and lower video quality options. The application uses one port to communicate with VLC for information about the currently playing video, or is used by ffmpeg to stream video to mpv or smplayer. The loopback interface is a virtual network interface per user that by default is not accessible outside your computer, although you can configure your firewall and network interfaces to control VLC and transmit video outside to the internet using VLC's streaming option. However, this would not include live transcriptions, only subtitles. Nevertheless, there are some solutions to stream your entire desktop with decent image quality over the internet, NoMachine (freeware) or Moonlight (open source license) are both great options.

### **Q: Some streams don't work, especially with upper and lower qualities, and sometimes timeshift doesn't work?**
**A:** Streams can often experience interruptions or temporary cuts, and their proper functioning in playlist4whisper can also depend on the ads they insert, and in other cases on whether yt-dlp and streamlink support the frequent changes made by various online video providers.

In general, processing online videos is very difficult due to the nature of different video codecs and network protocols used on servers and their various implementations, or due to incorrect timestamps and metadata information, or because of the difficulty of recording and cutting videos with ffmpeg. To add the somewhat random nature of the artificial intelligence algorithm for transcription.

### **Q: How does timeshift work and where are the files stored? Can I save these videos?**
**A:** Timeshift functions similarly to other applications, but in playlist4whisper, it involves a playlist repeated indefinitely with temporary videos or buffers. You can configure the number of these buffers and their size in minutes. For example, from just a few minutes up to a maximum of 99 videos, each lasting 99 minutes. This allows for a maximum of 163 hours of live streaming for timeshift, although at the cost of requiring a very large amount of disk space. When the chosen maximum size is reached, the oldest video file is deleted, and this process continues indefinitely until the application is stopped or an error occurs.

The videos can be navigated in VLC as if it were a playlist with multiple videos, allowing you to switch between videos or rewind and fast forward within a single video. The transcription will automatically switch to the chosen point. However, you should be cautious not to approach the current live recording moment too closely, as the player may jump to another video. Occasionally, video stream errors and transcription errors could cause the player to jump to another video in this timeshift playlist.

The temporary video buffer files can be saved in another directory, they will have names similar to `whisper-live_131263_3.avi`, or for the last one being recorded, `whisper-live_131263_buf033.avi`. The files are stored in the `/tmp` directory in both Linux and macOS, which typically serves as the temporary directory and is usually cleared upon restarting the computer.

In the `playlist4whisper` Python application, there is a dedicated **'Save Videos'** button. This opens a video saver window with a built-in player to preview the temporary files. You can select individual video buffers or choose a range of consecutive buffers to **merge** them into a single file and save it to a directory of your choice.

### **Q: What can Playlist4Whisper do in terms of transcribing or translating sound card or sound device inputs? Are USB devices supported in Linux?**
**A:** The quality of the transcription depends on your computer's capabilities, the chosen model, volume, and sound configuration on your operating system, as well as the ambient noise. Please note that this is a preliminary feature. There is a several-second delay between live sound and transcriptions, with no possibility for synchronization.

The capabilities to transcribe audio inputs or 'what you hear' on the desktop exist in numerous scenarios depending on the capabilities of sound cards, webcams, microphones, and USB-connected headphones. In Linux, USB sound cards and sound integrated into webcams, microphones, and headphones might be problematic, especially for devices that are too old or too new. Sometimes you may need to install device controllers or modules for Linux. Most USB drivers are usually available in one of three ways: within the kernel, as a compilable standalone module, or available as a pre-compiled (packaged) binary driver from your Linux distribution. However, sometimes these devices or their modules are deactivated in the Linux kernel. Although this is a rare situation, it may occur if Windows WSL2 uses a Linux kernel with deactivated device controllers, or requires some configurations. Anyway, you may encounter issues when using these devices in a Linux virtualized environment on WSL2.

Apart from device drivers or Linux modules, operating systems could use different libraries or sound systems. Playlist4Whisper depends on PulseAudio for Linux and Windows WSL2, and AVFoundation for macOS. If you use other libraries, your operating system might require software wrappers to ensure compatibility between different software sound systems.

Sound cards or devices in duplex mode support recording and listening simultaneously; you may need to configure your sound card for duplex mode. Some models allow different mixes or combinations for recording multiple inputs and simultaneous playback, while there is also software available, both free and paid, that enables this functionality. For Linux, devices suffixed with 'monitor' are loopback devices that allow you to record all sounds 'what you hear' on your desktop. Some sound cards or Linux distributions need to activate this feature with PulseAudio commands, or you could add links between inputs and outputs of your sound cards by editing your Linux sound card configuration files. These devices, along with applications, can be configured individually using PulseAudio Volume Control. For macOS, you can only use loopback devices through a virtual device with an Audio Loopback Driver like Blackhole (free license), Loopback (commercial license) or VB-Cable (donationware).

Windows supports loopback sound for native applications through a virtual driver, such as VB-Cable (donationware), which is an additional piece of software similar to those used in macOS. However, Playlist4Whisper is executed in the Linux virtual environment WSL2. Sound capacities depend on each Linux application and on PulseAudio and Microsoft support for virtualized sound in WSL2. Currently, the default virtualized system in WSL2 or WSLg (Windows Subsystem for Linux GUI) can utilize PulseAudio, albeit through an RDP (Remote Desktop Protocol) loopback network connection, which is not the optimal solution.

### **Q: Is it possible to achieve speech translation from a microphone without delay?**
**A:** There are few possibilities to achieve something similar to real-time or live transcriptions or speech translations. Different approaches exist for Speech-to-Speech translation systems: one involves a step-by-step process utilizing various specialized AI models, ranging from two to four steps. Another approach is a multimodal or all-in-one AI that handles all steps using a single model, like the SeamlessM4T free AI from Meta, which is currently in its early development stage. Neither approach can achieve real-time or live translations, nor anything resembling "near-live," due to the necessity of processing the input speech by dividing the recorded sound into small chunks. This is mainly due to contextual challenges or the differing nature of languages, which require the AI to understand the context of phrases or even anticipate what the speaker is going to say, akin to a wizard's alchemy. While it may be possible to introduce some shortcuts to shorten the steps, these tricks may not be suitable for a scientific solution or for accurate translation, as they could lead to reductionism or reinterpretations of what the speaker is saying.

Some applications use a trick to achieve a near-live result during the first transcription step, even with a Whisper AI and low-power computers, which playlist4whisper will support in the near future. This trick involves rapidly transcribing live speech, with one of the smallest models being something similar to a transcript without context or with a lot of errors, and immediately reintroducing the same sound data continuously to correct any possible errors in the last seconds transcribed. With this solution, the last words may frequently change, but it's not a significant issue for the user. However, this reintroduction trick cannot be applied to the speech translation step, as it is not serious to hear a translation of a sentence, and then the AI might repeatedly change the sentence because the initial version was incorrect.

![Live Transcription](https://github.com/antor44/livestream_video/blob/main/live-transcription.jpg)

### **Q: Is it possible to process audio recordings from multiple streams and trigger an alarm when a specific topic is discussed?**
**A:** Yes, an AI or an AI agent can process transcribed files searching for words or subjects, even within a specific context. However, the success and accuracy of this task will heavily depend on the size and quality of the language model (LLM) used to process the texts. On the other hand, if the goal is simply to search for isolated words, a set of terms, or to count the number of times they appear within a given time interval, it is possible to process the texts much faster and more efficiently using more "classical" programming code (such as regular expressions or keyword matching).

For example, an AI agent (such as OpenClaw) could be used to analyze transcribed files from various video, television, or radio streams. Notably, playlist4whisper.py—specifically its underlying livestream_video.sh script—is optimal for this initial step of continuously capturing and transcribing multiple live sources. If the agent detects that a specific soccer team or public figure is being discussed, it can trigger an alert by sending a near real-time text message.

Alternatively, classical programming could be used to rapidly analyze numerous transcriptions for keywords. For instance—the following "Big Brother" examples are for theoretical purposes only—a script could track colloquial or derogatory Spanish terms used to refer to the police, such as "bofia," "pasma," "poli," "madero," "picoleto," or "cana" (widely used in South America). Another script could monitor criminal topics by tracking sets of related words; for example, terms linked to specific crimes such as sexual offenses, pedophilia, or corruption, or to terrorism by searching for geopolitical discussions containing sensitive terms such as Palestine, Jews, Zionism, Israel, or Netanyahu.

However, whisper.cpp, the backend that actually powers playlist4whisper.py/livestream_video.sh, is not optimized for such tasks, primarily due to high RAM and VRAM usage when processing a large number of streams simultaneously. Furthermore, in real-world security scenarios, malicious actors often employ code words or cryptic language, or rarely discuss relevant topics at all. This is where the contextual understanding of an advanced LLM can outperform classical exact-match programming, albeit at a higher cost. A hybrid approach—preprocessing terms with classical code first—could also be considered.

### **Q: Why is the program not working?**
**A:** There could be various reasons why the script/program is not functioning correctly. It relies on other Linux programs and their libraries, such as whisper.cpp and mpv. To compile the `./build/bin/whisper-cli` executable, your operating system must have the necessary tools and development libraries installed, including those related to the chosen acceleration options. If you are not familiar with compilation, errors can often occur due to missing development libraries in your operating system. You will need to install only the necessary development libraries and the specific or compatible versions used by whisper.cpp. The `./build/bin/whisper-cli` executable from `whisper.cpp` must be placed in that exact subdirectory within the directory containing `playlist4whisper.py` and `livestream_video.sh`. Before lauch playlist4whisper.py, make sure that you are in the same directory as whisper.cpp, playlist4whisper.py, and livestream_video.sh, and eventually too the python environment for playlist4whisper is active. Additionally, it is crucial to have the Whisper model files in the "models" directory, following the correct format and name used by Whisper.cpp.

The whisper.cpp installation can be accomplished using terminal commands:
First clone the repository:
```bash
git clone https://github.com/ggerganov/whisper.cpp.git
```
Now change directory to the whisper-ccp folder and build the `./build/bin/whisper-cli` example to transcribe an audio file using the following command.
```bash
cmake -B build
cmake --build build -j --config Release
```
Transcribe an audio file (for model base.en):
```bash
./build/bin/whisper-cli -f samples/jfk.wav
```
For YouTube `yt-dlp` is required. For Twitch `streamlink` is required. The easy way to install/upgrade them:
```bash
pip3 install --upgrade yt-dlp
pip3 install --upgrade streamlink
```

### **Q: When I run the script, I always encounter an error related to the "whisper-cli" or "main" executable.**
**A:** There could be various reasons for this error. The "whisper-cli" executable, originally named "main" executable, is one of the examples provided in whisper.cpp, a high-level implementation of OpenAI's Whisper AI. The executable's name should remain as the default "whisper-cli" and reside in the subdirectory `./build/bin/` of the directory where playlist4whisper.py and the bash script livestream_video.sh were copied. Ensure that both the whisper-cli executable and livestream_video.sh have executable permissions, at least for the current user. Additionally, the models you use should be stored in the "models" subdirectory, which are created by running terminal commands like `make base.en` or `make tiny.en`. It's important to note that quantized models and large-v3 may not be compatible with older versions of whisper.cpp, which could lead to a "whisper-cli error". After building the whisper-cli example using the terminal command "make", you can test an audio file example using the command (for model base.en): `./build/bin/whisper-cli -f samples/jfk.wav`

### **Q: Why does the script sometimes close or show an error? Is it not reliable or stable?**
**A:** There could be several reasons for this. I have tested it extensively in many different scenarios, including running multiple instances simultaneously, and I have never encountered errors other than those caused by poor network connections or issues with the source streams themselves.

However, this program depends on several external programs and libraries, and it is impossible to guarantee stable operation across all possible system configurations. Also, keep in mind that macOS and Windows are not as fully supported as Linux, and I have not been able to test those operating systems as extensively.

There are also system-level issues that may affect the program, such as hardware instability caused by aggressive overclocking configurations or unstable sound card drivers. Additionally, some Linux systems enable aggressive audio power-saving features by default (commonly on HDA/ALSA integrated audio devices). In practice, on some hardware and driver combinations, this wake/suspend behavior may contribute to instability in multimedia applications like VLC, which can cause the script to fail.

These are system environment or configuration issues and are not necessarily related to a bug in playlist4whisper, livestream_video.sh, or the AI models themselves. Issues of this kind may also manifest as system hangs or as audio pops, clicks, and glitches in other multimedia applications on the same system.

### **Q: Can I run playlist4whisper without using the terminal, from a desktop shortcut on Linux?**
**A:** Yes, you can run it with the command "python" followed by the full path of playlist4whisper.py. In this case, before launching playlist4whisper.py, make sure that the Python environment for playlist4whisper is active. You can alternatively launch a bash script that first includes a command like `source ~/python-environments/whisper/bin/activate`. If you are using an Anaconda environment, use the command `/home/[user]/anaconda3/bin/conda run python /home/[user]/[app directory]/playlist4whisper.py`. In both cases, provide the working directory where the program is located.

### **Q: How can I change the size of the transcribed text snippets?**
**A:** You can change the size of the text snippets in seconds with the "step_s" option, which determines the duration of each part into which the videos are divided for transcription.

### **Q: How can I change the size and colors of the transcription text?**
**A:** You can change the size and colors of the transcription text in the options of the terminal program you are using.

### **Q: How can I position the terminal window for transcriptions in a fixed position?**
**A:** You can use a program for placing windows in Linux, such as `devilspie`, and configure a name for the terminal window and another for the main program terminal, such as Konsole or Xterm, to avoid name conflicts. For example, a configuration for the Gnome terminal in devilspie would be:
```text
; generated_rule Terminal
( if
( and
( matches (window_name) "Terminal" )
)
( begin
( geometry "+644+831" )
( println "match" )
)
)
```

### **Q: How can I permanently change the size and colors of the transcription text on macOS? How to display the correct local characters?**
**A:** To achieve this, you can create a file named `.Xresources` in your user's home directory (`/Users/[user]`). Inside the file, you can define specific settings to customize the appearance of the transcription text. For example:
```text
.xterm*background: black
.xterm*foreground: yellow
.xterm*font: 10x20
.xterm*vt100*geometry: 80x10
.xterm*saveLines: 10000
.xterm*locale: true 
```
In this example, the settings modify the background color to black, the foreground color to yellow, the font size to 10x20, the terminal's geometry to 80x10, and the number of lines to save to 10,000. After saving these changes in the `.Xresources` file, you need to relaunch XQuartz for the new settings to take effect. Once you launch an xterm terminal, you will see the desired customization of the transcription text.
The final option `.xterm*locale: true` will enable the same language settings in xterm as those in your macOS's default terminal. Although you may need to make changes and/or install additional components to display characters in other languages.

### **Q: How can I copy the transcribed text on macOS after xterm has been closed?**
**A:** The xterm window closes when the bash script stops. The behavior of this terminal differs in playlist4whisper due to macOS's distinct handling of closed windows. However, you can still find text files containing all the transcriptions and translations in your `/tmp` directory. Recent versions of playlist4whisper now allow you to easily save these files by clicking the 'Save Texts' button. Please note that the operating system clears the `/tmp` directory upon computer boot.

### **Q: How can I synchronize the video and the transcription?**
**A:** Without the Timeshift option, you can manually synchronize the video and transcription to your desired timing by using the pause and forward/backward buttons of the video player. When Timeshift is activated, the transcription is synchronized with the sync option, which is an automatic but fixed synchronization that cannot be changed during play.

### **Q: Why does the video and transcription get desynchronized?**
**A:** There could be several potential causes. Online video streams may not always provide reliable video synchronization or timestamps. Some versions of the programs used by playlist4whisper, like VLC player or ffmpeg, may not accurately address the timestamps of the video being played. Additionally, video and transcription applications work independently, each with its own stream of video or audio. Over time, desynchronization can fluctuate, and choosing a model that is too large for the processor's capabilities can also impact synchronization.
The timeshift feature alongside with an automatic video/transcription synchronization option may help address the issue, although this may lead to some issues with phrases at the beginning of each segmented video chunk.

### **Q: Why does the beginning and end of the transcription often get lost? Is this an AI issue?**
**A:** This issue often occurs because standard audio segmentation can cut words in half. To mitigate this **in part**, version 5.10 introduces a **Voice Activity Detection (VAD)** option (`--vad`), based on the powerful Silero model. This feature attempts to cut the audio during moments of silence near the chunk boundary. However, it is not a perfect solution; word truncations can still happen due to inaccurate timestamps from `ffmpeg`, the inherent nature of how Whisper processes audio, and other stream-related problems.

We are continuously working on advanced audio processing methods. Alternative solutions include the [Audio Transcription Chrome extension](https://github.com/antor44/Audio-Transcription) based on WhisperLive (which re-transcribes recent audio for better context) or potential future implementations of sliding window algorithms. However, the current VAD implementation provides an excellent balance of performance and accuracy without the complexity of re-transcribing audio.

It's worth noting that while playlist4whisper and livestream_video.sh currently process audio in isolated chunks —which can lead to word truncation and loss of broader sentence context— this approach also inherently minimizes the impact of hallucinations and other errors common to all Whisper AI model versions and model sizes. Any such errors are typically confined to a single audio/video fragment and occur infrequently, rather than propagating extensively throughout the transcription. The proposed additions to correct these word truncation issues are essentially classic coding workarounds, akin to applying a patch to address these noticeable flaws in OpenAI's Whisper model. Despite this development effort, guaranteed results are not assured. Furthermore, it's highly probable that OpenAI will implement similar or more sophisticated solutions directly within future Whisper model versions, or any other company in another transcription AI; it's almost inevitable they will address such evident and undesirable known errors sooner rather than later. Keep in mind that a very efficient and low power consumption, multilingual transcription AI is more than necessary for a lot of electronic devices and for all robots. This is the most fundamental part of the software to translate voice commands to robot's actions. Consequently, despite the significant programming work, and probably more inefficiently, to try and mitigate these issues externally, it might ultimately be time spent on a temporary fix.

Regarding the low-level development of the AI utilized by playlist4whisper.py or livestream_video.sh, OpenAI states: "The Whisper architecture is a simple end-to-end approach, implemented as an encoder-decoder Transformer. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and English speech translation."

In essence, Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. The latest and largest model, large-v3, has been trained with over one million hours of labeled audio and over four million hours of audio. The sound is converted into spectrogram images, specifically into log-Mel spectrogram images, similar to other AIs that process input sound directly. These same spectrogram images are used to convert new sound files for inference or speech processing.

Additionally, Whisper utilizes a GPT-2 model, which incorporates a tokenizer responsible for breaking down input text into individual units called tokens. In Whisper’s case, the tokenizer employs the same set of Byte-Pair Encoding (BPE) tokens as the GPT-2 model for English-only models, with additional tokens included for multilingual models. GPT-2 is a Transformer-based language model trained to predict the next word fragment without explicit supervision. Its purpose is to fill in gaps and resolve audio inconsistencies to accurately transcribe coherent words within context, processing up to the last 224 tokens (approximately 168 words).

However, by default, when Whisper AI generates text during inference, the influence of these GPT-2 tokens for English, or the custom tokens introduced for other languages, is minimal. Whisper does not produce context-based predictions or make speculative guesses for incomprehensible words unless a prompt is provided, which may have a slight influence. This effect only becomes more pronounced with specialized fine-tuning or when an additional Large Language Model (LLM), such as GPT-4 or DeepSeek-R1, is applied to the output text.

Despite these limitations, Whisper excels at processing speech even when inaudible or unintelligible. For example, if someone speaks while covering their mouth, Whisper can often transcribe full sentences by supplementing missing information as needed.

Choosing a larger Whisper model improves its accuracy. However, it's important to note that Playlist4Whisper doesn't enhance transcription accuracy by adding tokens from previous chunks processed. This is because smaller models don't see much benefit from the added complexity, and we often encounter the well-known problem of hallucinations and other behaviors. If accuracy is your top priority and your computer has sufficient power, using a larger model and longer step time would be optimal, resulting in fewer audio splits. Nonetheless, hallucinations and other errors could occur with large chunks.

Alternatively, you can generate subtitles for local audio/video files. This feature supports any model size with any processor; the only limitation is the processing time. When generating subtitles, the AI takes into account the maximum tokens supported by this implementation of OpenAI's Whisper AI. It's important to note that while this feature is powerful, users might still occasionally experience minor text inconsistencies or, more rarely, a phenomenon where the transcription appears to loop on a repeated phrase. This latter behavior, along with other types of 'hallucinations' (generating text not present in the audio), are recognized limitations of current AI transcription models like Whisper.

If you wish, you can try the [optional Chrome extension](https://github.com/antor44/Audio-Transcription) for transcribing audio and video solely from web pages, and also local audio/video files recorded with codecs supported by the browser. Otherwise, with the exception of subtitle generation for locally stored files, the playlist4whisper and livestream_video.sh applications should be primarily viewed as helpful tools for understanding video and audio content, capable of transcribing or translating a significant portion, though not always with complete coverage. This is still a significant capability, considering Whisper AI's extensive language support and accuracy in a variety of scenarios. However, at least with present versions, users should not expect perfectly exact results; if high precision is critical, exploring alternative solutions might be more appropriate.

### **Q: Sometimes the transcriptions are wrong or not appears, what could be the issue?**
**A:** The AI model is designed to transcribe audio from various situations, but certain factors can affect its accuracy. For instance, challenging accents or voice tones, or changes in voices during conversations can pose difficulties. In the future, the addition of sound filters options aims to minimize these issues. Additionally, the model may occasionally produce incorrect transcriptions due to gaps in its neural network connections. This can happen more frequently with smaller or quantized models, as well as with languages that have not been extensively trained.

### **Q: Sometimes when I generate a subtitle, the text is ruined and gets stuck with a repeated phrase.**
**A:** This is one of the well-known limitations of the current version of OpenAI's Whisper AI, known as hallucinations. This issue persists even with larger models and is even more problematic in the latest and more accurate larger-v3 model. You could try using a different model or attempt to divide the audio/video file into various chunks. The optional OpenAI Whisper executable could help reduce hallucinations when generating subtitles, you could run different executables simultaneously, each with its own 'playlist4whisper' copied into a separate directory, along with its own playlist files. However, many parameters are shared among users, and as of yet, no one has discovered "the magic formula".

### **Q: The transcriptions I get are not accurate?**
**A:** The quality of the transcriptions depends on several factors, especially the size of the model chosen. Larger models generally yield better results, but they also require more processing power. The English models tend to perform better than models for other languages. For languages other than English, you may need to use a larger model. If you choose the option auto for language autodetection or translate for simultaneous translation to English, it may also significantly increase processor consumption.

### **Q: smplayer does not work with online TV?**
**A:** First, check if you have passed a player option that is not compatible with smplayer. smplayer depens of mpv or mplayer, the installed version of mplayer may not support online video streams, depending on how it was compiled or its configurations, or there may be conflicts with video encoding libraries used by mplayer. In general, mpv is a better option, or if you prefer smplayer, make sure it is configured to use mpv.

### **Q: Is this program legal to watch TV channels?**
**A:** Yes, many of the URLs for channels found on the internet are made freely accessible by companies, governments, or public institutions for personal use only. Channels that may be restricted to a specific country or region for copyright reasons cannot be viewed, as the broadcasting company typically blocks viewers with IPs from other countries. However, playlists found on the internet may contain some channels with legality that is not entirely clear. Even if their URLs are publicly known, direct distribution or commercialization of the television channel is likely not allowed.

In the case of YouTube and Twitch, watching channels through standalone applications such as `yt-dlp` or `streamlink` poses a conflict with the companies, as they cannot control the advertisements you see. In other words, they are not interested in this approach and may make it difficult or prevent you from accessing their video streams.

Additionally, `playlist4whisper/livestream_video.sh` does not support configuring a username and password in `yt-dlp` or `streamlink` to watch ad-free videos or live streams from paid accounts or subscribed channels. Therefore, you will be limited to watching videos and live streams that do not include advertisements that could interrupt the transmission.

---

## Licensing and Attribution

This project is distributed under the **GNU General Public License v3.0 or later** (GPL-3.0-or-later). See the `LICENSE` file for details.

### Third-Party Code
The script `livestream_video.sh` was originally derived from an example script provided in the [whisper.cpp](https://github.com/ggml-org/whisper.cpp) repository (based on an idea by @semiformal-net).

The upstream `whisper.cpp` project and its examples are distributed under the **MIT License**. In compliance with its terms, the required copyright and permission notices for the derived script are preserved in the script header and detailed in the `THIRD_PARTY_NOTICES.md` file included in this repository.

The Python application `playlist4whisper.py` is original software distributed under the GPL-3.0-or-later license.

livestream_video

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README