KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #codex #typescript #ai-agents #ai-agent #cli #claude-code #open-source #deepseek-harness #dsh-plugin #python #mcp

semantic_based_painting

★ 43 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## :smiley::heart_eyes::fire::fire: Segmentation-Based Parametric Painting :fire::fire::heart_eyes::smiley:

This repository contains a semantic-based painting optimization algorithm which aims to transform a given input image into a painting. The algorithm takes advantage of modern computer vision techniques, segmentation networks, and a differentiable renderer to generate results.

[Project Website](https://manuelladron.github.io/semantic_based_painting/)

![Reference Image 1](https://raw.githubusercontent.com/manuelladron/semantic_based_painting/main/docs/static/images/website_teaser.png)

![Reference Image 2](https://raw.githubusercontent.com/manuelladron/semantic_based_painting/main/media_readme/teaser_img.png)

![Reference Image 3](https://raw.githubusercontent.com/manuelladron/semantic_based_painting/main/media_readme/giraffe_1.gif)

## What It Does

![Reference Image 4](https://raw.githubusercontent.com/manuelladron/semantic_based_painting/main/media_readme/thesis_chp_teaser.png)

The algorithm aims to efficiently optimize a collection of stroke parameters to create a painting from a given image input. The method starts with an image and optimizes a parameter vector of strokes. When rendered on a canvas, it recreates the input image in the form of a painting. The approach is designed to efficiently manage any input size or aspect ratio. It divides the canvas into semantic areas using a segmentation network. This provides a higher control over the painting compared to previous optimization and neural methods.

- **Layered Painting & Patch-Based Approach**: 
  - Implements a coarse-to-fine progression.
  - Uses a layered approach, starting with a rough first coarse painting pass and progressively refines.
  - Uses patches of `128x128` size and batch-optimizes all stroke parameters in each patch.

- **Semantic Segmentation**:
  - Provides precision over the granularity of each semantic zone in the artwork.
  - Ensures strokes remain within the designated semantic segment, improving the painting's accuracy.

- **Visual Working Memory**:
  - Uses a dynamic attention maps system to focus on areas that need more attention.
  - Yields an organic feel to the painting.

- **Stroke Initialization, Renderer, and Blending**:
  - Uses the stroke parameter representation and differentiable renderer.
  - Strokes are parameterized by a 13-dimensional tuple encoding various properties like start, middle, end points, radii, transparency, and RGB color.
  - Strokes are composited into patches with soft blending.

- **Optimization & Loss Functions**:
  - Optimizes all stroke parameters in batch for efficiency.
  - Uses both pixel loss and perceptual loss to ensure accurate recreation of the input image.

## How To Use

### 1. Requirements:

- Python 3.9 or later
- Install the required packages from `requirements.txt` using the following command (if using Ubuntu):

```bash
pip install -r requirements.txt
```

- **For Mac OS with Apple Silicon (M1/M2/M3) - OPTIMIZED VERSION:**

```bash
pip install -r requirements_os.txt
```

**Note:** The requirements_os.txt has been updated with all necessary dependencies and is fully optimized for Apple Silicon Macs with MPS (Metal Performance Shaders) support.

- **macOS Setup (Automatic):** The code now automatically detects and configures MPS device support. No manual changes needed!

- Download the renderer and perceptual network [here](https://drive.google.com/drive/folders/1f1dMbU5Yj9T-lGq0ZTc1MPPPJ-R7v0YX?usp=sharing) and store them in a folder under the main directory. *Update: also provided in folder model_checkpoints.

### 🚀 **NEW PERFORMANCE OPTIMIZATIONS (2024)**

This version includes significant performance and memory optimizations:

- **Mixed Precision Training**: Reduces memory usage by ~40%
- **Early Termination**: Stops optimization when loss plateaus (20-50% speedup)
- **Aggressive Patch Filtering**: Only processes areas that need refinement
- **Memory Management**: Automatic cleanup prevents memory accumulation
- **MPS Optimization**: Full Apple Silicon support with Metal Performance Shaders
- **Image Resizing**: Built-in support for processing smaller images

### 2. Command:

```bash
python main.py [options]
```

### 3. Key Arguments:

**Core Settings:**
- `--image_path`: Path to input image (default: 'images/paris2.jpeg')
- `--style`: Painting style - 'realistic', 'painterly', 'abstract', 'expressionist' (default: 'expressionist')
- `--canvas_size`: Canvas size for patches (default: 128, use 16-32 for faster processing)
- `--save_dir`: Directory to save results (default: './results')

**🆕 New Optimization Arguments:**
- `--use_mixed_precision`: Enable mixed precision training (default: True)
- `--image_resize_factor`: Resize input image by factor (e.g., 0.33 for 3x smaller, 0.5 for 2x smaller)
- `--upsample`: Enable upsampling of small images (default: True, set to False to keep resized images small)
- `--min_image_size`: Minimum image size for upsampling (default: 1800, set lower to avoid upsampling after resize)
- `--texturize`: Use texturize feature (default: True, set to False for memory savings)
- `--save_animation`: Save animation frames (default: True, set to False for memory savings)

**Performance Settings:**
- `--lr`: Learning rate (default: 0.004, try 0.008 for faster convergence)
- `--aspect_ratio_downsample`: Automatic downsampling factor for very large images (default: 3)
- `--global_loss`: Global loss strategy (default: False)
- `--exp_name`: Experiment name (default: 'exp_320')

**Advanced Options:**
- `--brush_type`: Brush type - 'straight' or 'curved' (default: 'curved')
- `--canvas_color`: Canvas background - 'black' or 'white' (default: 'black')
- `--use_transparency`: Enable stroke transparency (default: False)
- `--stroke_init_mode`: Stroke initialization - 'random' or 'grid' (default: 'grid')
- `--renderer_ckpt_path`: Path to renderer checkpoint (default: './model_checkpoints/renderer.pkl')

**Style Transfer Options:**
- `--add_style`: Enable CLIP-based style transfer (default: False)
- `--style_prompt`: Text prompt for style (default: 'Starry Night by Vincent Van Gogh')
- `--style_transfer`: Enable neural style transfer (default: False)
- `--style_img_path`: Path to style reference image (default: None)

### 4. Examples:

**Basic usage:**
```bash
python main.py --image_path images/paris2.jpeg --style painterly
```

**Optimized for memory/speed (recommended for Apple Silicon):**
```bash
python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 16 \
  --image_resize_factor 0.33 \
  --use_mixed_precision True \
  --texturize False \
  --save_animation False
```

**For maximum memory efficiency:**
```bash
export PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0
python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 8 \
  --image_resize_factor 0.5 \
  --lr 0.008
```

**Ultra-fast testing (10x smaller image):**
```bash
python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 8 \
  --image_resize_factor 0.1 \
  --upsample False \
  --lr 0.008
```

**⚠️ Upsampling Fix:** When using `--image_resize_factor`, the algorithm now automatically skips upsampling to preserve your intended image size. Use `--upsample False` to completely disable upsampling, or adjust `--min_image_size` to control the upsampling threshold.

## 🛠️ Troubleshooting

**Memory Issues:**
- Use smaller `--canvas_size` (8, 16, 32)
- Enable `--image_resize_factor 0.5` or smaller
- Set `--texturize False` and `--save_animation False`
- Use `export PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0` on macOS

**Performance Tips:**
- Higher `--lr` (0.008) for faster convergence
- Early termination automatically stops when loss plateaus
- Patch filtering reduces unnecessary computations
- Mixed precision is enabled by default for optimal performance

**Common Issues:**
- **"MPS out of memory"**: Reduce image size or canvas size
- **"Module not found"**: Install missing packages with `pip install <package>`
- **Slow performance**: Use optimized arguments from examples above

## 📊 Performance Improvements

### Before vs After Optimizations:

| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| Memory Usage | 18.13 GB (failed) | ~12 GB (successful) | ~35% reduction |
| Level 0 Time | Failed | ~3.4 minutes | ✅ Completion |
| Early Termination | None | 20-50% speedup | New feature |
| Patch Processing | All patches | Error-filtered only | ~30-60% reduction |
| Apple Silicon | Manual setup | Automatic MPS | Plug-and-play |

### Optimization Features:

- **🧠 Smart Early Termination**: Automatically stops when loss plateaus
- **🎯 Patch Filtering**: Only processes areas that need improvement
- **⚡ Mixed Precision**: Reduces memory usage without quality loss
- **🍎 Apple Silicon Optimized**: Native MPS support for M1/M2/M3 chips
- **🔧 Memory Management**: Automatic cleanup prevents memory leaks
- **📏 Image Resizing**: Process smaller images for faster results

## Method Overview

The method uses various techniques and algorithms to produce a painting from an input image. Key components include:

- **Semantic Segmentation**: Dividing the canvas into areas of interest.
- **Layered Painting**: A coarse-to-fine progression.
- **Visual Working Memory**: A dynamic attention maps system that focus on areas that need more attention.
- **Optimization & Loss Functions**: Ensuring the painting closely resembles the input image.

### Implementation Details:

- **Optimizer**: Adam with a learning rate of 0.0002.
- All painting layers are optimized for 300 iterations.
- **Canvas Background**: Black.
- **Segmentation Network**: Uses the DETR model with a CNN (ResNet) backbone followed by an encoder-decoder Transformer.

## More Results:

![Reference Image 5](https://raw.githubusercontent.com/manuelladron/semantic_based_painting/main/media_readme/motorcycle_0.gif)

![Reference Image 6](https://raw.githubusercontent.com/manuelladron/semantic_based_painting/main/media_readme/onemushroom_texture_lvl_0.jpg)

![Reference Image 7](https://raw.githubusercontent.com/manuelladron/semantic_based_painting/main/media_readme/painterly_vs_realistic_middle.png)

## References

- [Huang, et al. 2019. Learning to Paint]
- [Liu et al. 2021. Paint Transformer]
- [Zou et al. 2020. Stylized Neural Painting]
- [DETR: Carion, et al. 2020. End-to-End Object Detection with Transformers]

## Citation
```bash
@misc{deguevara2023segmentationbased,
      title={Segmentation-Based Parametric Painting}, 
      author={Manuel Ladron de Guevara and Matthew Fisher and Aaron Hertzmann},
      year={2023},
      eprint={2311.14271},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      }
```
## Feedback

For any queries or feedback related to the algorithm, please open an issue on GitHub or contact the authors directly.

semantic_based_painting

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README