Per-queue receive_message parameter specification

[cjlarose]

I'm interested in being able to specify different options for the calls to receive_message, but on a per-queue basis. Right now, it seems possible to define those options globally, but it would be very useful for me to be able to, for example, set the visibility_timeout on the receive_message calls to a different value depending on the queue.

Currently, you can set the wait_time_seconds for all receive_message calls to the same value:

aws:
  access_key_id:      ...       # or <%= ENV['AWS_ACCESS_KEY_ID'] %>
  secret_access_key:  ...       # or <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
  region:             us-east-1 # or <%= ENV['AWS_REGION'] %>
  receive_message:
    wait_time_seconds: 90
queues:
  - [high_priority, 6]
  - [default, 2]
  - [low_priority, 1]

Here's an example of what per-queue receive_message configuration might look like. Adding anything additional to the queues arrays might look awkward, so a new queue configuration format using hashes might be more appropriate:

aws:
  access_key_id:      ...       # or <%= ENV['AWS_ACCESS_KEY_ID'] %>
  secret_access_key:  ...       # or <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
  region:             us-east-1 # or <%= ENV['AWS_REGION'] %>
  receive_message:
    wait_time_seconds: 90
queues:
  high_priority:
    weight: 6
    receive_message:
      wait_time_seconds: 30
  default:
    weight: 2
    receive_message:
      visibility_timeout: 10
  low_priority:
    weight: 1
    receive_message:
      visibility_timeout: 45

I think these changes can be made in a backward-compatible way and that the behavior of specifying global receive_message options which can be overridden on a per-queue basis is natural. Would a pull request with these changes be welcome?

[phstc]

@mariokostelac is working in a PR #236 to decouple the fetcher, which might allow a specific fetcher to support your needs. In the current code, it isn't supported.

Why do you need long polling wait_time_seconds? With Shoryuken you shouldn't need it. Do you have a very specific need?

[cjlarose]

I just used wait_time_seconds as an example. In practice, what we'd really like to do is set visibility_timeout on a per-queue basis because we don't like having to rely on the default visibility timeout associated with the queue. It's just easier for us to ensure that we're using the same visibility timeout value in all of our deployment environments if we can configure it in the code itself.

[phstc]

@cjlarose if you want to set visibility timeout per queue, you need to set directly into your queues, Shoryuken does not create or change queues visibility timeout.

The maximum what Shoryuken does is to extend the visibility timeout per message, when it's configured to.

Basically, what you configure in shoryuken.yml are the queues Shoryuken is going to read, not to create or configure.

[cjlarose]

I'm not sure you understand what I'm trying to do. I'm not suggesting Shoryuken create queues or modify queue attributes. I understand that these features are outside of the scope of Shoryuken.

The SQS ReceiveMessage call accepts a parameter called VisibilityTimeout. If left unspecified, SQS will use the VisibilityTimeout attribute associated with the queue. Currently, Shoryuken::Fetcher#receive_message passes along values for max_number_of_messages, message_attribute_names, attribute_names, and anything specified in Shoryuken::AwsConfig.options[:receive_message]. To my knowledge, there is currently no mechanism to specify a value for visibility_timeout on a per-queue basis. The only way to specify the value of visibility_timeout on a ReceiveMessage call now is to set it globally with Shoryuken::AwsConfig.options[:receive_message][:visibility_timeout].

[phstc]

Hi @cjlarose

I see your point. That might be possible with the custom fetcher in #236, not with the current implementation. With the current implementation, the maximum you can do, is to write a middleware similar to this one to change the visibility on a message basis, or change it in your Worker#perform.

To my knowledge, there is currently no mechanism to specify a value for visibility_timeout on a per-queue basis.

You can change a queue visibility using set_queue_attributes, or set one while creating a queue.