How to pass variables between serial workers?

[homanchou]

Some serial jobs may depend on the data from workers that ran before them. What do you suggest is a good mechanism to pass variables along, that are not known in the beginning?

Superworker.create(:get_products_from_amazon, :user_id) do
ReportRequest :user_id #request_id is ready
GetReportStatus :user_id, :request_id #keep doing this until we get generated_report_id
GetReport :user_id, :generated_report_id
end

[tombenner]

Passing data between subjobs isn't currently supported, but if you see a good way to implement it, definitely feel free to submit a PR.

Another way to approach this might be to track the states in the database. This may not be ideal for your case, but as an example:

1. ReportRequest creates a Request
2. GetReportStatus finds Requests for the User that don't have a Report yet and creates a Report for each
3. GetReport finds Reports for the User that haven't been processed yet and processes them

[felipeclopes]

I'm not very satisfied with my solution, but I came up with something like this:

module PersistWorkflowSteps
  def persist_step(<your args>)
    if block_given?
      result = yield(<your args>)
      $redis.hset <key>, "#{self.class}", result
    end
  end
end

class MySuperworkerStep
  include Sidekiq::Worker
  include PersistWorkflowSteps

  def perform(<your args>)
      persist_step args do 
        execute(<your args>)
      end
  end

  def execute(<your args>)
    throw new Error("Not implemented!!!");
  end
end

[timuckun]

Any progress on this front?

I would like to submit a job and have each job down the change modify the data and pass it on to the next worker.