clifs - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #ai #golang #android #linux #docker #macos #windows #awesome #awesome-list #pytorch #ios #open-source

clifs

(★ 484)

Contrastive Language-Image Forensic Search allows free text searching through videos using OpenAI's machine learning model CLIP

File Explorer

Download Latest Version (.zip)

- bicyclist.jpg
- bmw.jpg
- jcn.jpg
- odwalla.jpg
- smart.jpg
- - clifs.py
  - requirements.txt
  - server.py
- .DS_Store
- Dockerfile
- - style.css
- - __init__.py
  - asgi.py
  - manage.py
  - settings.py
  - urls.py
  - wsgi.py
- - style.css
- Dockerfile
- entrypoint.sh
- requirements.txt
.gitignore
docker-compose-gpu.yml
docker-compose.yml
LICENSE
README.md
setup.sh

# Installation Guide

1. Get the code

git clone https://github.com/johanmodin/clifs

Downloads the entire project code from GitHub to your computer.

cd clifs

Moves into the project folder you just downloaded.

2. Docker

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker-compose build && docker-compose up

Runs the command against the services defined in the compose file.

docker-compose build && docker-compose -f docker-compose-gpu.yml up

Runs the command against the services defined in the compose file.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

Pulled directly from this repo's README.

3. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r search/app/requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Contrastive Language-Image Forensic Search

## Overview

CLIFS is a proof-of-concept for free text searching through videos for video frames with matching contents.
This is done using [OpenAI's CLIP](https://openai.com/blog/clip/) model, which is trained to match images with the corresponding captions and vice versa.
The searching is done by first extracting features from video frames using the CLIP image encoder and then
getting the features for the search query through the CLIP text encoder. The features are then matched by similarity
and the top results are returned, if above a set threshold.

To allow easy use of the CLIFS backend, a simple web server running django is used to provide an interface to the search engine.

## Examples
To give an idea of the capability of this model, a few examples are shown below, with the search query in bold and the result below.
These search queries are done against the 2 minute Sherbrooke video from the [UrbanTracker Dataset](https://www.jpjodoin.com/urbantracker/dataset.html).
Only the top image result for each query is shown. Note that the model is in fact quite capable of OCR.

#### A truck with the text "odwalla"
![alt text](https://raw.githubusercontent.com/johanmodin/clifs/master/media/odwalla.jpg)
======

#### A white BMW car
![alt text](https://raw.githubusercontent.com/johanmodin/clifs/master/media/bmw.jpg)
======

#### A truck with the text "JCN"
![alt text](https://raw.githubusercontent.com/johanmodin/clifs/master/media/jcn.jpg)
======

#### A bicyclist with a blue shirt
![alt text](https://raw.githubusercontent.com/johanmodin/clifs/master/media/bicyclist.jpg)
======

#### A blue SMART car
![alt text](https://raw.githubusercontent.com/johanmodin/clifs/master/media/smart.jpg)
======

## Setup
1. Run the setup.sh script to setup the folders and optionally download a video file for testing:
```sh
./setup.sh
```

2. Put your own video files that you want to index in the `data/input` directory

3. Build and start the search engine and web server containers through docker-compose:
```sh
docker-compose build && docker-compose up
```

Optionally, a docker-compose file with GPU support can be used if the host environment has a NVIDIA GPU and is setup for docker GPU support:

```sh
docker-compose build && docker-compose -f docker-compose-gpu.yml up
```
 
4. Once the features for the files in the `data/input` directory have been encoded, as shown in the log, navigate to [127.0.0.1:8000](http://127.0.0.1:8000) and search away.