Emit a whole batch of messages

[sngnd]

Is your feature request related to a problem? Please describe.
I have an integration with 3d-party API that has restrictions on having more than one concurrent request from the same customer. I use SQS to trigger API call, so sometimes it happens that the requests are processed and the same time. Thus, I really need to somehow filter messages from SQS based on the content of a message so some of the messages from the batch that have a specific combination of parameters will be released/deleted/etc, therefore not processed on my end at the same time. Overall this is maybe not the straight-forward issue, but I'm really frustrated with this API restrictions I cannot manage.

Describe the solution you'd like
In my opinion the simplest way to solve the problem I'm facing is to have a new event that emits an array of messages. Then I will be able to process every message on my end.

Additional context
Version used: 4.4.1
Node version: 16.5.0

[regevbr]

I think you need to use sqs FIFO queues and restrict the concurrent number of messages handled by this library to 1 using the maxInFlight parameter.
Emmiting a bunch of messages at once is not an option here as the library is focused on high throughput and once a message is handled, a new one is fetched immediately. If you want you can create your own logic that waits for X messages and then handles them in batch, thought I would not recommend it at all, at least not with this library.

[sngnd]

@regevbr, thanks for your suggestions, but I already considered using FIFO queues and unfortunately came to the conclusion that it is not an option here. As concurrency should be restricted only if a batch of messages includes 2 or more messages that trigger the request and have the same parameter inside message. Otherwise even if messages trigger the request but do not have the same parameter they can be processed concurrently. I don't want any message to be handled one by one, only specific ones.

I don't need to wait for X messages, sqs.receiveMessage returns an array and the library processes it inside of the forEach loop, and that's the array I need. I don't want to break any logic this library already has, just add a new event.

[regevbr]

Seems like you are trying to enforce something that does not fit sqs at all. I would suggest to use a separate queue for those unique messages and use concurrency of 1 message.
More over you don't seem to take under account cases where you have more than 1 consumer of messages which will break your logic entirely.

If the queue is working in full throttle, you will always be notified of a single element array, as it does not operate like you think it does.

I don't see a valid case yet to add that event as it might cause confusion.

If you can implement it, run it, verify is solves your case, and present in a more clear and concise way your use case I will be more than happy to merge your change.

Currently I don't fully understand your use case and how it will be solved in multi consumer, fully throttled situation.

I think Kafka will br a much better fit for you and will get you going out of the box iregardless of number of consumers and throttling