job-iteration - GitLite

KO | EN

gitlite — search

#hacktoberfest #react #ai #typescript #golang #rust #windows #awesome #kubernetes #security #pytorch #mcp #openai #open-source

job-iteration

(★ 1,313)

Makes your background jobs interruptible and resumable by design.

파일 탐색기

최종 버전 다운로드 (.zip)

- - ci.yml
  - cla.yml
- CODEOWNERS
- dependabot.yml
- setup
- test
- rails_gems.gemfile
- argument-semantics.md
- best-practices.md
- custom-enumerator.md
- iteration-how-it-works.md
- parallel-iteration.md
- throttling.md
- - active_record_batch_enumerator.rb
  - active_record_cursor.rb
  - active_record_enumerator.rb
  - csv_enumerator.rb
  - enumerator_builder.rb
  - interruption_adapters.rb
  - iteration.rb
  - log_subscriber.rb
  - nested_enumerator.rb
  - parallel_enumerator.rb
  - railtie.rb
  - test_helper.rb
  - throttle_enumerator.rb
  - version.rb
- job-iteration.rb
- - integration_behaviour.rb
  - interruption_adapters_test.rb
  - resque_test.rb
  - sidekiq_test.rb
- - jobs.rb
  - sample_csv_with_headers.csv
- - active_job_iteration_test.rb
  - active_record_batch_enumerator_test.rb
  - active_record_enumerator_test.rb
  - csv_enumerator_test.rb
  - enumerator_builder_test.rb
  - iteration_test.rb
  - nested_enumerator_test.rb
  - parallel_enumerator_test.rb
  - test_helper_test.rb
  - throttle_enumerator_test.rb
- ci_matrix_test.rb
- test_helper.rb
.gitignore
.rubocop.yml
.ruby-version
.yardopts
CHANGELOG.md
CODE_OF_CONDUCT.md
dev.yml
Gemfile
Gemfile.lock
job-iteration.gemspec
LICENSE.txt
Rakefile
README.md

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/Shopify/job-iteration

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd job-iteration

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Ruby

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Ruby bundle 명령어를 쓰려면 Ruby가 필요합니다.

$ bundle

이 명령어를 터미널에 그대로 입력해 실행하세요.

✅ bundle install이 에러 없이 끝나면 README에 안내된 실행 명령어(예: rails server)로 마저 실행해보세요.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Job Iteration API

[![CI](https://github.com/Shopify/job-iteration/actions/workflows/ci.yml/badge.svg)](https://github.com/Shopify/job-iteration/actions/workflows/ci.yml)

Meet Iteration, an extension for [ActiveJob](https://github.com/rails/rails/tree/main/activejob) that makes your jobs interruptible and resumable, saving all progress that the job has made (aka checkpoint for jobs).

## Background

Imagine the following job:

```ruby
class SimpleJob < ApplicationJob
  def perform
    User.find_each do |user|
      user.notify_about_something
    end
  end
end
```

The job would run fairly quickly when you only have a hundred `User` records. But as the number of records grows, it will take longer for a job to iterate over all Users. Eventually, there will be millions of records to iterate and the job will end up taking hours or even days.

With frequent deploys and worker restarts, it would mean that a job will be either lost or restarted from the beginning. Some records (especially those in the beginning of the relation) will be processed more than once.

Cloud environments are also unpredictable, and there's no way to guarantee that a single job will have reserved hardware to run for hours and days. What if AWS diagnosed the instance as unhealthy and will restart it in 5 minutes? What if a Kubernetes pod is getting [evicted](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)? Again, all job progress will be lost. At Shopify, we also use it to interrupt workloads safely when moving tenants between shards and move shards between regions.

Software that is designed for high availability [must be resilient](https://12factor.net/disposability) to interruptions that come from the infrastructure. That's exactly what Iteration brings to ActiveJob. It's been developed at Shopify to safely process long-running jobs, in Cloud, and has been working in production since May 2017.

We recommend that you watch one of our [conference talks](https://www.youtube.com/watch?v=XvnWjsmAl60) about the ideas and history behind Iteration API.

## Getting started

Add this line to your application's Gemfile:

```ruby
gem 'job-iteration'
```

And then execute:

$ bundle

In the job, include `JobIteration::Iteration` module and start describing the job with two methods (`build_enumerator` and `each_iteration`) instead of `perform`:

```ruby
class NotifyUsersJob < ApplicationJob
  include JobIteration::Iteration

def build_enumerator(cursor:)
    enumerator_builder.active_record_on_records(
      User.all,
      cursor: cursor,
    )
  end

def each_iteration(user)
    user.notify_about_something
  end
end
```

`each_iteration` will be called for each `User` model in `User.all` relation. The relation will be ordered by primary key, exactly like `find_each` does.

Check out more examples of Iterations:

```ruby
class BatchesJob < ApplicationJob
  include JobIteration::Iteration

def build_enumerator(product_id, cursor:)
    enumerator_builder.active_record_on_batches(
      Comment.where(product_id: product_id).select(:id),
      cursor: cursor,
      batch_size: 100,
    )
  end

def each_iteration(batch_of_comments, product_id)
    comment_ids = batch_of_comments.map(&:id)
    CommentService.call(comment_ids: comment_ids)
  end
end
```

```ruby
class BatchesAsRelationJob < ApplicationJob
  include JobIteration::Iteration

def build_enumerator(product_id, cursor:)
    enumerator_builder.active_record_on_batch_relations(
      Product.find(product_id).comments,
      cursor: cursor,
      batch_size: 100,
    )
  end

def each_iteration(batch_of_comments, product_id)
    # batch_of_comments will be a Comment::ActiveRecord_Relation
    batch_of_comments.update_all(deleted: true)
  end
end
```

```ruby
class ParallelIterationJob < ApplicationJob
  include JobIteration::Iteration
  
  def build_enumerator(cursor:)
    enumerator_builder.parallel_active_record_on_records(User.all, instances: 5, cursor: cursor)
  end

# Runs in 5 separate jobs, with each job processing a subset of the users
  def each_iteration(user)
    user.notify_about_something
  end
end
```

```ruby
class ArrayJob < ApplicationJob
  include JobIteration::Iteration

def build_enumerator(cursor:)
    enumerator_builder.array(['build', 'enumerator', 'from', 'any', 'array'], cursor: cursor)
  end

def each_iteration(array_element)
    # use array_element
  end
end
```

```ruby
class CsvJob < ApplicationJob
  include JobIteration::Iteration

def build_enumerator(import_id, cursor:)
    import = Import.find(import_id)
    enumerator_builder.csv(import.csv, cursor: cursor)
  end

def each_iteration(csv_row, import_id)
    # insert csv_row to database
  end
end
```

```ruby
class NestedIterationJob < ApplicationJob
  include JobIteration::Iteration

def build_enumerator(cursor:)
    enumerator_builder.nested(
      [
        ->(cursor) { enumerator_builder.active_record_on_records(Shop.all, cursor: cursor) },
        ->(shop, cursor) { enumerator_builder.active_record_on_records(shop.products, cursor: cursor) },
        ->(_shop, product, cursor) { enumerator_builder.active_record_on_batch_relations(product.product_variants, cursor: cursor) }
      ],
      cursor: cursor
    )
  end

def each_iteration(product_variants_relation)
    # do something
  end
end
```

Iteration hooks into most popular queue adapters out of the box to support graceful interruption. No extra configuration is required.

## Supported dependencies

Job-iteration currently supports the following queue adapters (in order of implementation):

- [Resque](https://github.com/resque/resque)
- [Sidekiq](https://github.com/sidekiq/sidekiq/)
- [GoodJob](https://github.com/bensheldon/good_job)
- [Solid Queue](https://github.com/rails/solid_queue)
- [Amazon SQS](https://github.com/aws/aws-activejob-sqs-ruby)
- [Delayed::Job](https://github.com/collectiveidea/delayed_job)

It supports the following platforms:

- Ruby 3.1 and later
- Rails 7.1 and later

Support for older platforms that have reached end of life may occasionally be dropped if maintaining backwards compatibility is cumbersome.

## Guides

* [Iteration: how it works](guides/iteration-how-it-works.md)
* [Job argument semantics](guides/argument-semantics.md)
* [Best practices](guides/best-practices.md)
* [Writing custom enumerator](guides/custom-enumerator.md)
* [Throttling](guides/throttling.md)
* [Parallel iteration](guides/parallel-iteration.md)

For more detailed documentation, see [rubydoc](https://www.rubydoc.info/gems/job-iteration).

## Requirements

Active Job is the primary requirement for Iteration. For iteration without Active Job in Sidekiq, see [Sidekiq Iteration](https://github.com/sidekiq/sidekiq/wiki/Iteration).

### API

Iteration job must respond to `build_enumerator` and `each_iteration` methods. `build_enumerator` must return an [Enumerator](http://ruby-doc.org/core-2.5.1/Enumerator.html) object that respects the `cursor` value.

### Sidekiq adapter

Running iterating jobs on Sidekiq should work with the default configuration. The most important setting is Sidekiq's [timeout](https://github.com/mperham/sidekiq/wiki/Deployment#overview) option, which defaults to 25 seconds. That allows the last `each_iteration` to complete and gracefully shutdown.

### Resque adapter

There a few configuration assumptions that are required for Iteration to work with Resque. `GRACEFUL_TERM` must be enabled (giving the job ability to gracefully interrupt), and `FORK_PER_JOB` is recommended to be disabled (set to `false`).

## FAQ

**Why can't I just iterate in `#perform` method and do whatever I want?** You can, but then your job has to comply with a long list of requirements, such as the ones above. This creates leaky abstractions more easily, when instead we can expose a more powerful abstraction for developers--without exposing the underlying infrastructure.

**What happens when my job is interrupted?** A checkpoint will be persisted to Redis after the current `each_iteration`, and the job will be re-enqueued. Once it's popped off the queue, the worker will work off from the next iteration.

**What happens with retries?** An interruption of a job does not count as a retry. If an exception occurs, the job will retry or be discarded as normal using Active Job configuration for the job. If the job retries, it re-processes the iteration that originally failed and progress will continue from there on if successful.

**What happens if my iteration takes a long time?** We recommend that a single `each_iteration` should take no longer than 30 seconds. In the future, this may raise an exception.

**Why is it important that `each_iteration` runs quickly?** When the job worker is scheduled for restart or shutdown, it gets a notice to finish remaining unit of work. To guarantee that no progress is lost we need to make sure that `each_iteration` completes within a reasonable amount of time. The exact timeout depends on your queue adapter configuration.

**Why do I use have to use this ugly helper in `build_enumerator`? Why can't you automatically infer it?** This is how the first version of the API worked. We checked the type of object returned by `build_enumerable`, and whether it was ActiveRecord Relation or an Array, we used the matching adapter. This caused opaque type branching in Iteration internals and it didn’t allow developers to craft their own Enumerators and control the cursor value. We made a decision to _always_ return Enumerator instance from `build_enumerator`. Now we provide explicit helpers to convert ActiveRecord Relation or an Array to Enumerator, and for more complex iteration flows developers can build their own `Enumerator` objects.

**What is the difference between Enumerable and Enumerator?** We recomend [this post](http://blog.arkency.com/2014/01/ruby-to-enum-for-enumerator/) to learn more about Enumerators in Ruby.

**My job has a complex flow. How do I write my own Enumerator?** Iteration API takes care of persisting the cursor (that you may use to calculate an offset) and controlling the job state. The power of Enumerator object is that you can use the cursor in any way you want. One example is a cursorless job that pops records from a datastore until the job is interrupted:

```ruby
class MyJob < ApplicationJob
  include JobIteration::Iteration

def build_enumerator(cursor:)
    Enumerator.new do
      Redis.lpop("mylist") # or: Kafka.poll(timeout: 10.seconds)
    end
  end

def each_iteration(element_from_redis)
    # ...
  end
end
```

## Credits

This project would not be possible without these individuals (in alphabetical order):

* Daniella Niyonkuru
* Emil Stolarsky
* Florian Weingarten
* Guillaume Malette
* Hormoz Kheradmand
* Mohamed-Adam Chaieb
* Simon Eskildsen

## Development

After checking out the repo, run `bin/setup` to install dependencies and create mysql database. Then, run `bundle exec rake test` to run the tests.

To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).

## Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/Shopify/job-iteration. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.

## License

The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

## Code of Conduct

Everyone interacting in the Job::Iteration project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/Shopify/job-iteration/blob/main/CODE_OF_CONDUCT.md).