KO
|
EN
gitlite — search
Search
#typescript
#ai-agents
#ai
#dsh-plugin
#deepseek-harness
#open-source
#cli
#claude-code
#codex
#developer-tools
#react
#windows
PassionSR
★ 59
Open GitHub ↗
No description available.
Download README (.md)
Explore Similar Repositories
mcp-duckdb-memory-server
:
MCP Memory Server with DuckDB backend
netbeep
:
Chrome and Firefox extension to play sounds on network requests, depending on request type and file size.
sourcedialog
:
Easily create TUI forms with pure Bash (no external tools)
code-2nd
:
🔢 《编码》第二版官方参考资源（勘误表，书籍插图、交互式网站和英文PDF）以及个人笔记✍️，大师Charles Petzold对《编码》这一力作的再版，理解计算机底层原理的极佳入门读物📔
voice-ai-primer-web
:
No description available.
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
PassionSR
?
Download (.md)
<div align="center"> <p align="center"> <img src="asserts/figs/Passion_title_compressed.png" width="200px"> </p> </div> # 🚀 PassionSR: Low-Bit Quantized Super-Resolution [LiBo Zhu](https://github.com/LiBoZhu030073), [Jianze Li](https://github.com/JianzeLi-114), [Haotong Qin](https://htqin.github.io/), [Wenbo Li](https://fenglinglwb.github.io/), [Yulun Zhang](http://yulunzhang.com/), [Yong Guo](https://www.guoyongcs.com/) and [Xiaokang Yang](https://scholar.google.com/citations?user=yDEavdMAAAAJ) **"PassionSR: Post-Training Quantization with Adaptive Scale in One-Step Diffusion based Image Super-Resolution", CVPR 2025** [](https://labor555.github.io/PassionSR_Web/) [](https://arxiv.org/abs/2411.17106) [](https://github.com/libozhu03/PassionSR/releases/download/v.0.0.1/supp.pdf) [](https://github.com/libozhu03/PassionSR/releases) [](https://github.com/libozhu03/PassionSR) [](https://github.com/libozhu03/PassionSR) --- ## 📚 Table of Contents - [🔥 News](#-news) - [📘 Abstract](#-abstract) - [📝 Structure Overview](#-structure-overview) - [⚙️ Installation](#️-installation) - [📥 Download Pretrained Models and Datasets](#-download-pretrained-models-and-datasets) - [📈 Training](#-training) - [🧪 Inference](#-inference) - [📦 Measure](#-measure) - [🔎 Results](#-results) - [📝 Acknowledgements](#-acknowledgements) - [📌 Citation](#-citation) --- ## 🔥 News - 🎉 **[2025-06-09]** Release code. - 🚩 **[2025-03-10]** The 2/4-bit version [QArtSR](https://github.com/libozhu03/QArtSR) is released. - 🏆 **[2025-02-27]** Congratulations, PassionSR has been accepted to CVPR 2025. - **[2024-11-25]** Create repository. ### ⭐⭐⭐ If PassionSR is helpful to your projects, please help star this repo. Thanks! --- ## 📘 Abstract > Diffusion-based image super-resolution (SR) models have shown superior performance at the cost of multiple denoising steps. However, even though the denoising step has been reduced to one, they require high computational costs and storage requirements, making it difficult for deployment on hardware devices. To address these issues, we propose a novel post-training quantization approach with adaptive scale in one-step diffusion (OSD) image SR, PassionSR. First, we simplify OSD model to two core components, UNet and Variational Autoencoder (VAE) by removing the CLIPEncoder. Secondly, we propose Learnable Boundary Quantizer (LBQ) and Learnable Equivalent Transformation (LET) to optimize the quantization process and manipulate activation distributions for better quantization. Finally, we design a Distributed Quantization Calibration (DQC) strategy that stabilizes the training of quantized parameters for rapid convergence. Comprehensive experiments demonstrate that PassionSR with 8-bit and 6-bit obtains comparable visual results with full-precision model. Moreover, our PassionSR achieves significant advantages over recent leading low-bit quantization methods for image SR. --- ## 📝 Structure Overview <p align="center"> <img src="asserts/figs/overview_compress_v2.png" width="800px"> </p> | HR | LR | OSEDiff(32-bit) | EfficientDM(8-bit) | PassionSR(8-bit) | | :------------------------------------------: | :------------------------------------------: | :---------------------------------------------: | :---------------------------------------------: | :---------------------------------------------: | | <img src="asserts/figs/Nikon_049_HRUV_U_W8A8_V_W8A8/HR_org.png" height=110> | <img src="asserts/figs/Nikon_049_HRUV_U_W8A8_V_W8A8/lr_Image.png" height=110> | <img src="asserts/figs/Nikon_049_HRUV_U_W8A8_V_W8A8/fp context Image.png" height=110> | <img src="asserts/figs/Nikon_049_HRUV_U_W8A8_V_W8A8/Qalora Image.png" height=110> | <img src="asserts/figs/Nikon_049_HRUV_U_W8A8_V_W8A8/PassionSR Image.png" height=110> | | <img src="asserts/figs/Canon_032_HRUV_U_W8A8_V_W8A8/HR_org.png" height=110> | <img src="asserts/figs/Canon_032_HRUV_U_W8A8_V_W8A8/lr_Image.png" height=110> | <img src="asserts/figs/Canon_032_HRUV_U_W8A8_V_W8A8/fp context Image.png" height=110> | <img src="asserts/figs/Canon_032_HRUV_U_W8A8_V_W8A8/Qalora Image.png" height=110>| <img src="asserts/figs/Canon_032_HRUV_U_W8A8_V_W8A8/PassionSR Image.png" height=110> | </details> --- ## ⚙️ Installation To set up the environment, clone the repository and create a new Conda environment using the provided dependencies. ```bash git clone https://github.com/libozhu03/PassionSR.git cd PassionSR conda create -n passionsr python=3.10 conda activate passionsr conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple ``` Tested with: * Python 3.10 * PyTorch 2.0.1 * CUDA 11.8 --- ## 📥 Download Pretrained Models and Datasets We provide pretrained weights for PassionSR under different settings. | Model | Information | Link | | --------- | --------------- | ----------------------------------- | | PassionSR | The calibrated model weights under different settings | [Google Drive](https://drive.google.com/drive/folders/1F-1DDPcBIp29gvxCxgNL-l3FqphGVLY4?usp=drive_link) | | SD2.1 | Official model weights of stable diffusion 2.1 | [Huggingface](https://huggingface.co/stabilityai/stable-diffusion-2-1-base) | Place PassionSR's weights in `./weights` and SD2.1 in `./hf-models`. Used training and testing sets can be downloaded as follows: |Training Set | Testing Set | Visual results | | --------- | --------------- | --------------- | | 500 training images [[Google Drive](https://drive.google.com/drive/folders/1WO804NdqJxNyZpYx7_HfWiZdv17jOz_K?usp=drive_link)] | RealSR + DRealSR + DIV2K_val [[Google Drive](https://drive.google.com/drive/folders/1bMPM9eS5UgYf8Pk6Na1MUOyspivmmaVW?usp=drive_link)] | [Google Drive](https://drive.google.com/drive/folders/121TetRqUdup1MueGJFZJF8jeXGU3RM3_?usp=drive_link) | Download training and testing datasets and put them into the corresponding folders of `./data`. --- ## 📈 Training Run the command below to perform Post-Training Quantization (PTQ) using your desired configuration file. The script loads pretrained Stable Diffusion and OSEDiff weights, and applies quantization to selected components (e.g., UNet and/or VAE). ```bash # Train the W8A8 models of Table 2 in the main paper. CUDA_VISIBLE_DEVICES="0" python ptq_quantize_single.py --config_file scripts/PTQ/config/UV/saw_sep/saw_U_W8A8_V_W8A8.yaml # Train the W6A6 models of Table 2 in the main paper. CUDA_VISIBLE_DEVICES="0" python ptq_quantize_single.py --config_file scripts/PTQ/config/UV/saw_sep/saw_U_W6A6_V_W6A6.yaml # Train the W8A8 models of Table 1 in the supplementary material. CUDA_VISIBLE_DEVICES="0" python ptq_quantize_single.py --config_file scripts/PTQ/config/U/saw_sep/saw_W8A8.yaml # Train the W6A6 models of Table 1 in the supplementary material. CUDA_VISIBLE_DEVICES="0" python ptq_quantize_single.py --config_file scripts/PTQ/config/U/saw_sep/saw_W6A6.yaml ``` <details> <summary> 🔧 Training Configuration Example:</summary> The example YAML config demonstrates typical usage and can be adapted for different settings. ```yaml # device setting device: "cuda:0" cali_img_path: "data/cali_dataset" # path of calibration dataset basic_config: # basic config for OSEDiff inference process seed: 42 precision: "autocast" # "full", "autocast" upscale: 4 process_size: 512 scale: 9.0 lora_weights_path: preset/models/osediff.ckpt # OSEDiff ckpt path pretrained_model_name_or_path: hf-models/sd21 # stable diffusion path config: hf-models/ldm_Config/stable-diffusion/intel/v2-inference-v-fp32.yaml ckpt: hf-models/sd21/v2-1_512-ema-pruned.ckpt # stable diffusion ckpt path context_embedding_path: preset/models/empty_context_embedding.pt # empty text embedding path align_method: "nofix" # 'wavelet', 'adain', 'nofix' merge_lora: True # merge lora into weight quantize_config: quantize: True # quantize or not only_Unet: True # only quantize Unet or quantize both Unet and Vae Unet: # quantize setting for U-net quantype: PTQ # don't change method: saw_sep # name of method only_weight: False # weight only quantization method weight_quant_bits: 8 weight_sym: False # weight quantization asymmetrical or not weight_sign: False # weight quantiztion sign or not act_quant_bits: 8 act_sign: False # act quantiztion sign or not act_sym: False # act quantization asymmetrical or not split: True # half split for activation layer_type: 2Dquant # two quantizer types (2Dquant and normal_quant) s_alpha: 0.3 # scale factor intialization exponents Vae: quantype: PTQ method: saw only_weight: False weight_quant_bits: 8 weight_sym: False weight_sign: False act_quant_bits: 8 act_sign: False act_sym: False split: True layer_type: 2Dquant output_modelpath: results/quantize/saw_sep/UV/W8A8 # output path # calibration settings cali_batch_size: 4 cali_learning_rate: 1e-5 cali_epochs: 2 loss_function: mse scheduler: milestones: [1] gamma: 0.1 save_interval: 2 ``` </details> --- ## 🧪 Inference Use the following command to run inference with quantized models. The pipeline supports various datasets (e.g., DIV2K_val, RealSR, DRealSR) and includes options for tiling, LoRA merging. ```bash # Reprodue the W8A8 results of Table 2 in the main paper. CUDA_VISIBLE_DEVICES="0" python inference_single.py --config scripts/inference/config/saw_sep/UV/saw_U_W8A8_V_W8A8.yaml # Reprodue the W6A6 results of Table 2 in the main paper. CUDA_VISIBLE_DEVICES="0" python inference_single.py --config scripts/inference/config/saw_sep/UV/saw_U_W6A6_V_W6A6.yaml # Reprodue the W8A8 results of Table 1 in the supplementary material. CUDA_VISIBLE_DEVICES="0" python inference_single.py --config scripts/inference/config/saw_sep/U/saw_W8A8.yaml # Reprodue the W6A6 results of Table 1 in the supplementary material. CUDA_VISIBLE_DEVICES="0" python inference_single.py --config scripts/inference/config/saw_sep/U/saw_W8A8.yaml ``` <details> <summary> 🔧 Inference Configuration Example:</summary> The example YAML config demonstrates typical usage and can be adapted for different settings. ```yaml # device setting device: cuda:0 out_dir: results/quantize/saw_sep/U/W8A8 # output path # dataset to inference, set detailed dataset path in preset/data_construct.py dataset: DIV2K_val # ["DIV2K_val", "RealSR", "DRealSR"] basic_config: seed: 42 precision: "autocast" # ["full", "autocast"] process_size: 512 config: hf-models/ldm_Config/stable-diffusion/intel/v2-inference-v-fp32.yaml ckpt: hf-models/sd21/v2-1_512-ema-pruned.ckpt lora_weights_path: preset/models/osediff.ckpt pretrained_model_name_or_path: hf-models/sd21 context_embedding_path: preset/models/empty_context_embedding.pt upscale: 4 align_method: adain # ['wavelet', 'adain', 'nofix'] merge_lora: True # scale: 9.0 # tile setting tile_config: vae_decoder_tiled_size: 224 vae_encoder_tiled_size: 1024 latent_tiled_size: 64 latent_tiled_overlap: 32 # quantize config quantize_config: quantize: True only_Unet: True Unet: # keep same with quantize config quant_ckpt: weights/U_W8A8/PTQ/unet_ckpt_merge_saw_sep.pth # Unet quantize ckpt path quantype: PTQ method: saw only_weight: False weight_quant_bits: 8 weight_sym: False weight_sign: False act_quant_bits: 8 act_sign: False act_sym: False split: True layer_type: 2Dquant s_alpha: 0.3 ``` </details> ## 📦 Measure Evaluate model performance by comparing super-resolution outputs against high-resolution ground truth images: ```bash CUDA_VISIBLE_DEVICES="0" python measure.py -i YOUR_IMAGE_PATH -r HR_IMAGE_PATH ``` This script computes the image quality metrics presented in paper to assess the effectiveness of quantized inference. --- ## <a name="-results"></a> 🔎 Results PassionSR significantly out-performs previous methods at the setting of W8A8 and W6A6. Detailed results can be downloaded at [Google Drive](https://drive.google.com/drive/folders/121TetRqUdup1MueGJFZJF8jeXGU3RM3_?usp=drive_link). <details> <summary> 📊 Quantitative comparisons in Table 2 of the main paper (click to expand)</summary> <p align="center"> <img width="900" src="asserts/figs/results_UNet_Vae.png"> </p> </details> <details> <summary> 🖼 Visual comparison in Figure 6 of the main paper (click to expand) </summary> <p align="center"> <img width="900" src="asserts/figs/visual_UNet_Vae.png"> </p> </details> --- ## 📝 Acknowledgements We would like to thank the developers and maintainers of [Stable Diffusion](https://github.com/Stability-AI/StableDiffusion), [Diffusers](https://github.com/huggingface/diffusers), and [OSEDiff](https://github.com/cswry/OSEDiff) for their open-source contributions, which have greatly facilitated our research and development. This project is supported in part by the Shanghai Jiao Tong University Artificial Intelligence Institute. We also thank our collaborators and contributors for their valuable feedback and technical discussions. ## 📌 Citation ```bibtex @inproceedings{zhu2025passionsr, title={{PassionSR}: Post-Training Quantization with Adaptive Scale in One-Step Diffusion based Image Super-Resolution}, author={Zhu, Libo and Li, Jianze and Qin, Haotong and Zhang, Yulun and Guo, Yong and Yang, Xiaokang}, booktitle={CVPR}, year={2025} } ```
