I am continuously receiving data feed from third party and want to process around 100k messages in less than a minute. So thinking to implement a messaging queue through which I can offload the processing part and will push the message to a queue, where one worker out of 100 (or whatever number) picks the job and process it.
I've read about JMS and Redis based messaging, but I am not sure how to run multiple listeners. The single listener is already configured.
Spring JMS allows you to specify concurrency limits via a "lower-upper" String, e.g. "5-10", or a simple upper limit String, e.g. "10" (the lower limit will be 1 in this case).
The listener container will always hold on to the minimum number of consumers (setConcurrentConsumers(int)) and will slowly scale up to the maximum number of consumers
See:
An example in a Spring boot configuration bean:
@Configuration
@EnableJms
public class AppConfig {
@Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory
= new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setConcurrency("3-10");
return factory;
}
}