nightly

KO | EN

gitlite — search

#javascript #python #react #ai #typescript #android #machine-learning #rust #nodejs #macos #awesome #kubernetes #vue #mcp

(★ 11)

Feather-light scheduler, monitor and storage for your research tasks

nightly 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Nightly
=======

![CI](https://github.com/izgzhen/nightly/workflows/CI/badge.svg)

## Introduction

Nightly is a system for scheduling, monitoring and collecting outputs from configuration-defined
command-line tasks. It is developed to support my daily research work since I can't find an existing
solution that is lightweight (depends on nothing more than python, mysql and ssh) and
customizable (written in less than a few hundreds lines of Python in total). 
I hope it might be useful to you as well.

**Features**:

- Define compute/storage resources declaratively
- Define job declaratively (similar to CI)
- Run job periodically, or as a daemon, or just once
    - For "just once", you can also run an ad-hoc command line job without defining it
- Automatically transfer job log and output to storage
- Monitor the progress via web panel
- Receive notification in job success/fail/...

Data objects:

- Job
- Resource

Components:

- Log DB: database containing job log (status) entries
- Scheduler: master node, launch jobs and monitor process status
- Runner: launch job on runner node through SSH
- Storage: persisting job output and selected files
- Panel: web panel daemon
- Notif: notification (only email for now)

## Dependencies

```
poetry shell # get poetry: https://python-poetry.org/
poetry install
# symblink the virtual env created by poetry to `.venv` under project root as well
```

## Config

There should be two files inside `config` folder, which is ignored by git.
See `example_config` for examples and comments.

Users can use different `config` folders in different environment (production or development).

See https://app.sendgrid.com/guide/integrate/langs/python for email notification configuration keys
`sendgrid_api_key`, `notif_sender_email` and `notif_receiver_email`.

## Runbook

Initialize or update database:

```
make upgrade-db
```

Truncate and backup log table:

```
make truncate-all-log
```

Run the master daemon for scheduling jobs (on a server within tmux):

```
make run # it will tee the log to main.log as well
```

Run web panel daemon:

```
make panel # debug mode
export BASIC_AUTH_USERNAME=...; export BASIC_AUTH_PASSWORD=...; make panel-prod
```

Currently, panel server and master daemon must be on the same node.

Launch an ad-hoc job from command line:

```bash
scripts/launch-nightly sleep 10 # launch an ad-hoc one on the master node
scripts/launch-nightly --job ... # launch a defined job. Note that it must has "once" schedule
```

## Demo

Homepage:

![](https://raw.githubusercontent.com/izgzhen/nightly/master/home.png)

Output page:

![](https://raw.githubusercontent.com/izgzhen/nightly/master/output.png)

Email notification:

![](https://raw.githubusercontent.com/izgzhen/nightly/master/email-notif.png)

nightly

nightly 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

nightly

nightly 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README