mel

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 47 Open GitHub ↗

A scalable asynchronous event-driven jobs engine

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Mel

**Mel** is an asychronous event-driven jobs processing engine designed to scale. *Mel* simplifies jobs management by abstracting away the nuances of scheduling and running jobs.

In *Mel*, a scheduled job is called a *task*. A single job may be scheduled in multiple ways, yielding multiple tasks from the same job.

Mel schedules all tasks in the chosen storage backend as a set of task `id`s sorted by their times of next run. For recurring tasks, the next run is scheduled right after the current run completes.

This makes the storage backend the *source of truth* for schedules, allowing to easily scale out *Mel* to multiple instances (called *workers*), or replace or stop workers without losing schedules.

*Mel* supports *bulk scheduling* of jobs as a single atomic unit. There's also support for *sequential scheduling* to track a series of jobs and perform some action after they are all complete.

### Types of tasks

1. **Instant Tasks:** These are tasks that run only once after they are scheduled, either immediately or at some specified time in the future.

1. **Periodic Tasks:** These are tasks that run regularly at a specified interval. They may run forever, or till some specified time in the future.

1. **Cron Tasks:** These are tasks that run according to a specified schedule in *Unix Cron* format. They may run forever, or till some specified time in the future.

## Installation

1. Add the dependency to your `shard.yml`:

```yaml
   dependencies:
     mel:
       github: GrottoPress/mel
     #redis: # Uncomment if using the Redis backend
     #  github: jgaskins/redis
     #pg: # Uncomment if using the Posgres backend
     #  github: will/crystal-pg
   ```

1. Run `shards update`

1. Require and configure *Mel* in your app (we'll configure workers later):

```crystal
   # ->>> src/app/config.cr

# ...

require "mel"

require "../jobs/**"

Mel.configure do |settings|
     settings.error_handler = ->(error : Exception) { puts error.message }
     settings.timezone = Time::Location.load("Africa/Accra")
   end

Log.setup(Mel.log.source, :info, Log::IOBackend.new)
   # Redis::Log.level = :error # Uncomment if using the Redis backend
   # DB::Log.level = :error # Uncomment if using the Postgres backend

# ...
   ```

- Using the Redis backend

```crystal
     # ->>> src/app/config.cr

# ...

require "mel/redis"

Mel.configure do |settings|
       # ...
       settings.store = Mel::Redis.new(ENV["REDIS_URL"], namespace: "mel")
       # ...
     end

# ...
     ```

- Using the Postgres backend

```crystal
     # ->>> src/app/config.cr

# ...

require "mel/postgres"

Mel.configure do |settings|
       # ...
       settings.store = Mel::Postgres.new(ENV["DATABASE_URL"], namespace: "mel")
       # OR pass an existing `DB::Database` instance
       #settings.store = Mel::Postgres.new(db, namespace: "mel")
       # ...
     end

# You may use this in your app's migrations to migrate
     Mel.postgres.migrate_database

# You may use this in your app's migrations to roll back
     #Mel.postgres.rollback_database

# ...
     ```

- Using the Memory backend (Not for production use)

```crystal
     # ->>> src/app/config.cr

# ...

require "mel"

Mel.configure do |settings|
       # ...
       settings.store = Mel::Memory.new
       # ...
     end

# ...
     ```

- Skip storage

You may disable storage altogether by setting `Mel.settings.store` to `nil` (This is the default).

## Usage

1. Define job:

```crystal
   # ->>> src/jobs/do_some_work.cr

struct DoSomeWork
     include Mel::Job # <= Required

def initialize(@arg_1 : Int32, @arg_2 : String)
     end
     # <= Instance vars must be JSON-serializable

# (Required)
     #
     # Main operation to be performed.
     # Called in a new fiber.
     def run
       # << Do work here >>
     end

# Called in the main fiber, before spawning the fiber
     # that calls the `#run` method above.
     def before_run
       # ...
     end

# Called in the same fiber that calls `#run`.
     # `success` is `true` only if the run succeeded.
     def after_run(success)
       if success
         # ...
       else
         # ...
       end
     end

# Called in the main fiber before enqueueing the task in
     # the store.
     def before_enqueue
       # ...
     end

# Called in the main fiber after enqueueing the task in
     # the store. `success` is `true` only if the enqueue succeeded.
     def after_enqueue(success)
       if success
         # ...
       else
         # ...
       end
     end
   end
   ```

1. Schedule job:

- Run job now: