Table of contents

Common Mistakes When Using Sidekiq

Ruby on Rails

Sidekiq is a popular background job processing library for Ruby that allows you to offload time-consuming tasks from your main application thread. While Sidekiq can significantly improve the performance and responsiveness of your application, it's essential to use it correctly to avoid common pitfalls. In this blog post, we'll explore some of the most common mistakes made when using Sidekiq and provide tips on how to avoid them.
Ref: https://github.com/sidekiq/sidekiq/wiki/Best-Practices

1. Keep Job Parameters Small and Simple

Sidekiq persists the arguments to perform_async to Redis as JSON. So the arguments you pass to perform_async must be composed of simple JSON datatypes: string, integer, float, boolean, null(nil), array, and hash. This means you must not use symbol, ActiveRecord object, Marshall object, ...

If you need to pass a hash, use methods like Hash#stringify_keys to convert symbol keys. Remember that when the worker runs this job, the hash now has string keys:

// language: ruby
data = {count: 3}
MyJob.perform_async(data)

// language: ruby
class MyJob  
  include Sidekiq::Worker  
  def perform(data)     
    puts data # {"count"=>3}  
  end
end

2. Ensure Job Idempotency and Transactionality

Make your jobs idempotent so they can safely execute multiple times. Sidekiq might re-execute a job in case of errors or retries. Consider using a database transaction to roll back data changes in case of an error, or write your code to be resilient to errors. Remember that Sidekiq does not guarantee exactly once execution, even for completed jobs.

// language: ruby
def perform(charge_id)  
  charge = Charge.find!(charge_id)  
  ActiveRecord::Base.transaction do    
    charge.void!    
    Mailer.charge_refunded(charge).deliver_now  
  end
end

If Mailer.charge_refunded(charge).deliver_now fails, it will trigger a rollback of charge.void!, ensuring that your job remains in a safe, re-executable state.

An alternative approach is to ensure that charge.void_transaction won't modify anything if the charge has already been voided. This way, even if the job is re-executed, it won't have unintended consequences if the operation has already been completed.

// language: ruby
def perform(charge_id)  
  charge = Charge.find(charge_id)  
  unless charge.void?    
    charge.void!  
  end  
  Mailer.charge_refunded(charge).deliver_now
end

3. Concurrency Limits

Sidekiq is designed for parallel execution, so design your jobs with concurrency in mind. Take advantage of Sidekiq's basic features for tuning concurrency, and embrace parallelism in your system architecture. However, setting the concurrency level too high can lead to resource exhaustion and decreased performance. Conversely, setting it too low might not fully utilize your system resources.

Solution: Monitor your system's resource usage and adjust the Sidekiq concurrency settings accordingly. A good practice is to start with a moderate concurrency level and then fine-tune based on performance testing.
If you're using Heroku, you should take a look at their documentation https://github.com/sidekiq/sidekiq/wiki/Heroku

4. Neglecting Error Handling

Background jobs are often executed asynchronously, making it easy to overlook error handling. Ignoring errors in your background jobs can lead to silent failures and unexpected behavior.

Solution: Always implement proper error handling in your Sidekiq jobs. Use rescue blocks to catch exceptions and log error messages. Additionally, consider implementing retry mechanisms for transient errors to ensure robust job processing.

5. Lack of Job Prioritization

Not all jobs are equal, some may be more critical than others. Ignoring job prioritization can result in important tasks being stuck behind less critical ones.

Solution: Leverage Sidekiq's support for job prioritization. Assign appropriate priority levels to your jobs based on their importance. This ensures that high-priority jobs are processed promptly, even in the presence of a large job queue.
Please note that if you define too many queues, it will decrease the performance (https://github.com/sidekiq/sidekiq/wiki/Advanced-Options)

// language: yaml
:queues:
  - high_priority
  - default
  - low_priority

Created at 2023-12-30 18:34:02 +0700

Related blogs

Integer and String Enum in Rails

TL;DRFor General Use: I would lean towards using strings for enums due to their readability and maintainability benefits.For Performance-Critical Appl...

Ruby on Rails

2024-09-05 23:53:58 +0700

How to customize YJIT in the Rails app

In this post, we'll dive into what YJIT is, how to enable it in your Rails app, and how to monitor and configure it for optimal performance.What is YJ...

Ruby on Rails

2024-12-04 16:29:15 +0700