Neural-Enhanced-Live-Streaming

KO | EN

gitlite — search

#javascript #react #ai #typescript #llm #android #linux #docker #nodejs #awesome #cli #kubernetes #security #ios

Neural-Enhanced-Live-Streaming

(★ 15)

A real-time streaming framework that utilizes neural network-based super-resolution to enhance live video quality

Neural-Enhanced-Live-Streaming 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Neural-Enhanced Super-Resolution for Real-Time Video Streaming

## Introduction

The proposed framework, Epsilon, is a real-time video streaming system 
that utilizes server's and viewer's computational resources to enhance live video quality.
Epsilon concerns two communicating peers,
video sender and video receiver, which are interconnected by a streaming server.
The framework is built on top of `aiortc`.

## System Design

In our scenario, a streamer streams a low-resolution (LR) video to the server.
At the same time, the streamer will sample training patches, which are sent to the server along with the video.
Upon receiving the LR video sent from streamer, the server relays the video to the viewer. 
The server also maintains a super-resolution (SR) model for the video, and uses patches to train it.
After each training epoch, the trained model is delivered to the viewer.
Once the viewer receives an SR model, 
it applies it to the ongoing LR video in a per-frame super-resolution manner, generating a high-resolution (HR) video.

## Files

### Core

```
.
├── dataset.py      dataset objects used for neural training
├── media.py        a copy of aiortc.contrib.media with modifications on some classes and functions
├── misc.py         utilities
├── model.py        the Super-Resolution neural network and its components
├── receiver.py     implementation of video viewer
├── sender.py       implementation of video streamer
└── server.py       live-streaming server augmented with online neural training
```

### Aside

```
.
├── analysis        scripts for analysis and visualization
├── assets          images in README
├── data
│   └── video       contains a sample video  
├── demo            scripts used in the university demo
├── quickstart      scripts with minimal settings to run the system [USE THIS FOR QUICKSTART]
├── ice.json        configuration for STUN and TURN servers (not a necessity)
├── offline.py      an offline training example
└── result          result generated at runtime
    ├── logs        runtime log files
    └── records     recorded raw and super-resolved video
```

## Dependencies

The system runs on Python 3.8 with following third-party libraries:
```
aiortc==1.2.0
numpy==1.19.2
opencv==4.0.1
pytorch==1.7.1
torchvision==0.8.2
```
We also use `matplotlib` and `pandas` in analysis part, but they are not essential for the system.

## Quickstart (LAN)

[Here](https://github.com/ROBINADC/Neural-Enhanced-Live-Streaming/tree/master/quickstart)
are some scripts for you to run the program with minimal configuration changes.

## Usage (LAN)

- Server - `server.py`
- Streamer - `sender.py`
- Viewer - `receiver.py`

Whenever possible, run three programs in three seperate computers.
When only two computers are available, run streamer and viewer on the same device.

### Step 1 - start the server

To start the server, you need to specify WebRTC's signaling server,
and assign two ports for connecting sender and receiver.

**Frequently-used arguments**
```
--model-num-blocks [NUM] : the number of residual block of the SR model
--model-num-features [NUM] : the number of channels for convolutional filters
--use-gpu
--signaling-host [ADDRESS_TO_SIGNALING_SERVER] : signaling server address
--signaling-port-sender [PORT]
--signaling-port-receiver [PORT] 
--debug : detailed logging
```

### Step 2 - start the sender

The streamer can use local file or webcam as media source.

**Frequently-used arguments**
```
--play-from [VIDEO_FILE] : play from local file 
--framerate-degradation [LEVEL] : reduce framerate to accommodate weak machines
--apply-psnr-filter : use PSNR filter
--signaling-host [ADDRESS_TO_SIGNALING_SERVER] : signaling server address
--signaling-port [PORT]
--debug : detailed logging
```

### Step 3 - start the receiver

To run the receiver, make sure the framerate is correctly specified.

**Frequently-used arguments**
```
--record-sr-fn [FILE] : the file name of the super-resolved video
--record-raw-fn [FILE] : the file name of the raw video
--fps [FPS] : recording framerate (must identical to the sending framerate)
--model-num-blocks [NUM] : the number of residual block of the SR model
--model-num-features [NUM] : the number of channels for convolutional filters
--use-gpu
--signaling-host [ADDRESS_TO_SIGNALING_SERVER] : signaling server address
--signaling-port [PORT]
--debug : detailed logging
```

### Termination

Use keyboard interrupt to stop any program.

## WAN and ICE

A straight-up connection across WAN between two peers might not work for many reasons.

- Intermediate firewalls may block such connections.
- Peers might not have public IP address to distinguish themselves.
- Routers might not allow direct connection to certain peers.

Interactive Connectivity Establishment (ICE) is a framework for establishing peer connections despite the above limitations.
ICE uses STUN and/or TURN internally.
  
In the CLI, specify relevant arguments to explicitly use ICE in the system.
A sample ice configuration file is provided in `ice.json`.
```
--ice-config [ICE_CONFIGURATION_FILE] : JSON file for ICE server configurations
--ice-provider [KEY] : a top-level key in the JSON file (such as "google" or "xirsys")
```

## Reference

**LiveNAS**.
Kim, J., Jung, Y., Yeo, H., Ye, J., & Han, D. (2020). 
Neural-enhanced live streaming: Improving live video ingest via online learning. 
Proceedings of the Annual Conference of the ACM Special Interest Group on Data Communication 
on the Applications, Technologies, Architectures, and Protocols for Computer Communication, 107–125. 
https://doi.org/10.1145/3387514.3405856

**NAS**.
Yeo, H., Jung, Y., Kim, J., Shin, J., & Han, D. (2018). 
Neural adaptive content-aware internet video delivery. 
13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), 645–661. 
https://www.usenix.org/conference/osdi18/presentation/yeo

**MDSR**.
Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017). 
Enhanced deep residual networks for single image super-resolution. 
2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1132–1140. 
https://doi.org/10.1109/CVPRW.2017.151

**ICE**.
https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Protocols

**Provided video**.
https://www.youtube.com/watch?v=4AtOU0dDXv8

Neural-Enhanced-Live-Streaming

Neural-Enhanced-Live-Streaming 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Neural-Enhanced-Live-Streaming

Neural-Enhanced-Live-Streaming 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README