discord-ext-listening

KO | EN

gitlite — search

#javascript #react #ai #golang #go #android #deep-learning #nodejs #windows #awesome #awesome-list #pytorch #mcp #open-source

discord-ext-listening

(★ 38)

Voice receive extension for discord.py built on multiprocessing and designed to be flexible.

discord-ext-listening 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

This is a [discord.py](https://github.com/rapptz/discord.py/) extension with functionality for listening in voice 
channels. It uses Python's standard multiprocessing library for audio processing, striving for efficiency and a 
flexible design.

# How to use
`pip install git+https://github.com/Sheppsu/discord-ext-listening`

Check examples folder to see how to use.
I might get some actual documentation up eventually.
Read section below if you want a full explanation.

# How it works
### VoiceClient
The listening functionality is tied to `VoiceClient` just as the playing functionality is.
The `VoiceClient` can be instantiated like normally, but specifying to use this extension's `VoiceClient` class.
All the listening methods of `VoiceClient` call `AudioReceiver` methods.
- `VoiceClient.listen` -> `AudioReceiver.start_listening`
- `VoiceClient.stop_listening` -> `AudioReceiver.stop_listening`
- `VoiceClient.pause_listening` -> `AudioReceiver.pause`
- `VoiceClient.resume_listening` -> `AudioReceiver.resume`
- `VoiceClient.wait_for_listen_ready` -> `AudioReceiver.wait_for_standby` & `AudioReceiver.wait_for_clean`

### AudioReceiver
**Functionality:** `AudioReceiver` facilitates the flow of unprocessed audio frames to processing areas and exposes methods
for `VoiceClient` to interact with that flow.

**Design:** `AudioReceiver` inherits from `threading.Thread` and will constantly receive audio while 
running and the voice client being connected. If there's not a listening session in progress or the 
listening session is paused, the audio data is simply discarded after being received. Otherwise,
the receiver calls `AudioProcessPool.submit`, adding a callback to the returned `Future`. The callback
handles the processed audio frame or any errors, sending audio frames to and `AudioSink` object.
Stopping the listening session calls `AudioSink.cleanup` and any callback function passed to `VoiceClient.listen`.

### AudioProcessPool
**Functionality:** Manages child processes and facilitates sending and receiving audio frames to those processes
for processing.

**Design:** The design is based off `ProcessPoolExecutor`, with a `submit` function for sending audio frames to
a child process. The maximum number of child processes and other options are specified upon creation, but it doesn't
spawn a process until it's needed. The processes are spawned with the multiprocessing library, and it uses a duplex
pipe to send and receive audio frames. When audio is sent across the pipe, a `Future` object is created, which is
returned by `submit`, and the future along with a process index are put on a queue that another thread will poll.
When this separate thread pulls an item off the queue, it knows to wait for an audio frame from that process
(specified by the process index) and set the result of the future.

### AudioUnpacker
**Functionality:** Processes audio frames, decrypting and optionally decoding them. This is the class which receives
audio frames from `AudioProcessPool`.

**Design:** Inheriting from `multiprocessing.Process`, this process receives, processes, and send back audio frames
for the entirety of its runtime. If its pipe breaks or closes or another error occurs, it terminates. In the case
of an error not related to its pipe, the error is returned.

### AudioSink
**Functionality:** Abstract class that outlines the functionality of any class which inherits from this one.

**Design:** Defines `on_audio`, `on_rtcp`, and `cleanup` methods. `on_audio` receives all processed audio frames. 
`on_rtcp` receives all RTCP packets. `cleanup` is called when a listening session ends.

### AudioHandlingSink
**Functionality:** Abstract class that implements functionality for dealing with out-of-order packets and delays.

**Design:** Audio is received from `on_audio` and after being validated, is passed to `on_valid_audio`
(an abstract method). When an audio frame is initially received, it's put on a queue where another thread will
handle validation. When this separate thread pulls the audio frame off the queue, it goes through a validation 
procedure:
1. If the last recorded sequence number is >= 65000 and this audio frame sequence number is <= 1000, the sequence number
has likely looped back around, so we'll subtract 65536 from the last recorded sequence number to prevent issues.
2. If this audio frame sequence number is lower than the last recorded audio sequence number, drop it. The last
recorded audio sequence marks the sequence number of the last validated audio frame, so this audio frame is already
too late.
3. If this is the first audio frame being received, or it's the next audio frame in the sequence, it's valid. 
Since it's valid, the last recorded sequence number is updated, the audio frame is passed to on_valid_audio, and
the buffer is emptied (more on that below).
4. Otherwise, this audio frame is added to a buffer.

Emptying the buffer: The buffer is emptied when a valid frame is found, a specific amount of time has passed, or
when the thread handling validation is finishing. When the buffer is emptied, all the audio frames are sorted in
ascending order by their sequence number and put back in the queue. The last recorded sequence is then set to the first
audio frame's sequence minus one. Whenever the buffer is emptied, if the validation loop is finishing, a new one is
created.

### AudioFileSink
**Functionality:** Inherits from `AudioHandlingSink`, implementing functionality for managing multiple SSRC's and
associated audio files.

**Design:** Each SSRC (user in the vc) has its own associated `AudioFile` object for handling audio frame writing.
When `on_valid_audio` is called, it passes the audio to its respective `AudioFile.on_audio` handler, creating a new one
if needed. `AudioFileSink.cleanup` calls `AudioFile.cleanup` on all the managed `AudioFile` objects.

### AudioFile
**Functionality:** Abstract class for managing an audio file of a specific format.

**Design:** `on_audio` writes audio frames to as raw pcm to a file. `cleanup` closes the files handles and marks the
`AudioFile` as `done`. The class has an abstract method `convert` meant to convert the pcm file to a specific format.
`convert` is meant to call `_convert_cleanup`, which will fix some attributes of `AudioFile` after the convert.

### convert_with_ffmpeg
Function for using ffmpeg to convert a pcm file to another format. Uses `asyncio.create_subprocess_exec` with the
arguments `ffmpeg -f s16le -ar {sampling_rate} -ac {channels} -y -i {path} {new_path}`. It then asynchronously waits
for the process to finish.

### WaveAudioFile
**Functionality:** Converts pcm file to a wave file.

**Design:** Calls `convert_with_ffmpeg` with the new path ending in `.wav`.

### MP3AudioFile
**Functionality:** Converts pcm file to an mp3 file.

**Design:** Calls `convert_with_ffmpeg` with the new path ending in `.mp3`.

discord-ext-listening

discord-ext-listening 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

discord-ext-listening

discord-ext-listening 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README