新到一家公司，由于对业务还不太熟悉。正好又有一个需求，由于我们公司是第三方支付公司需要把订单状态的更新(比如成功或者失败等状态)发送给调用接口的商户。在设计这个系统的时候，由于公司的业务都是在亚马逊云上。而且之前使用的 MQ 就是使用的 Amazon Simple Queue Service也就是 SQS，所以这次技术选型也是 SQS 。主要用于当发送商户失败时候，会有一个重试策略。把发送失败的消息发送到 SQS 里面通过它的延时消费来做发送重度。还有就是接口限流，当单位时候内调用接口次数达到阈值就把消息发送到 SQS 里面。主要的作用就是削峰填谷。
由于之前公司使用过 RabbitMQ ，而且我也看了一下 Spring RabbitMQ 的实现。为了让消息系统的扩展性更高一点所以我决定模仿Spring RabbitMQ 的实现。写一个基于 Amazon SQS 的实现。
在 Spring RabbitMQ 的实现中它是对于每一个消息队列都是对应一个MessageListenerContainer来处理的。同样的在实现 Amazon SQS 也定义一个这样的接口。
public interface MessageListenerContainer extends Lifecycle {
void setupMessageListener(MessageListener messageListener);
}
添加一个 MessageListener 把真正的队列监听处理添加到这个容器当中。
@FunctionalInterface
public interface MessageListener {
void onMessage(NotifyMessageContent message) ;
}
@EnableAmazonSQS 注解激活标注了 @AmazonSQSListener 注解的方法，把它生成 MessageListener,它是消息消息的真正的处理逻辑。MessageListenerContainer 中用来处理 Amazon SQS 队列的处理。
@EnableAmazonSQS 激活 @AmazonSQSListener
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Import(AmazonSQSBootstrapConfiguration.class)
public @interface EnableAmazonSQS {
}
配置在spring bean 的方法上面的 @AmazonSQSListener，标记这个方法是用来处理 Amazon SQS 指定队列的。
@AmazonSQSListener
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface AmazonSQSListener {
/**
* 队列名称
* @return
*/
String queue() default "";
/**
* Amazon SQS 并发消费者(值必须大于 1 且小于 CPU 核心的 2 倍)
* @return
*/
int consumers() default 0;
}
标注可以使用 @AmazonSQSListener 注解，通过 Import 配置类 AmazonSQSBootstrapConfiguration 来实现。
@Configuration
@EnableConfigurationProperties(AmazonSQSQueueProperties.class)
public class AmazonSQSBootstrapConfiguration {
@Bean(name = "cn.carlzone.amazon.sqs.messageing.annotation.innerAmazonSQSListenerAnnotationBeanPostProcessor")
@Role(BeanDefinition.ROLE_INFRASTRUCTURE)
public AmazonSQSListenerAnnotationBeanPostProcessor amazonSQSListenerAnnotationProcessor() {
return new AmazonSQSListenerAnnotationBeanPostProcessor();
}
}
AmazonSQSListenerAnnotationBeanPostProcessor 通过实现 BeanPostProcessor 增强 spring bean。当检测到 spring bean 方法上标注了 @AmazonSQSListener 注解的 Bean 然后创建 MessageListenerContainer 用于处理这个方法上需要处理的队列。
public class AmazonSQSListenerAnnotationBeanPostProcessor implements BeanPostProcessor, BeanFactoryAware {
private static final int MAX_CONCURRENT_CONSUMERS_LIMIT = Runtime.getRuntime().availableProcessors() * 2;
private BeanFactory beanFactory;
@Override
public void setBeanFactory(BeanFactory beanFactory) {
this.beanFactory = beanFactory;
}
@Override
public Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException {
return bean;
}
@Override
public Object postProcessAfterInitialization(final Object bean, final String beanName) throws BeansException {
Class<?> targetClass = AopUtils.getTargetClass(bean);
ListenerMethod methodAnnotation = getMethodAnnotation(targetClass);
if(methodAnnotation != null) {
processAmazonSQSListener(methodAnnotation.annotations, bean, methodAnnotation.method);
}
return bean;
}
private void processAmazonSQSListener(AmazonSQSListener annotation, Object bean, Method method) {
SimpleAmazonSQSListenerContainerFactory containerFactory = beanFactory.getBean("amazonSQSListenerContainerFactory", SimpleAmazonSQSListenerContainerFactory.class);
Method useMethod = checkProxy(method, bean);
String queue = annotation.queue();
Assert.hasText(queue, "amazon simple queue service name can not be null");
SimpleMessageListenerContainer listenerContainer = containerFactory.createListenerContainer(queue, message -> useMethod.invoke(bean, message));
int consumers = annotation.consumers();
if(consumers > 1 && consumers <= MAX_CONCURRENT_CONSUMERS_LIMIT) {
listenerContainer.setMaxConcurrentConsumers(consumers);
}
if (listenerContainer instanceof InitializingBean) {
try {
((InitializingBean) listenerContainer).afterPropertiesSet();
}
catch (Exception ex) {
throw new BeanInitializationException("Failed to initialize message listener container", ex);
}
}
listenerContainer.start();
}
private ListenerMethod getMethodAnnotation(Class<?> targetClass) {
final List<ListenerMethod> methodAnnotations = new ArrayList<>();
ReflectionUtils.doWithMethods(targetClass, method -> {
ListenerMethod listenerAnnotations = findListenerAnnotation(method);
if(listenerAnnotations != null) {
methodAnnotations.add(listenerAnnotations);
}
}, ReflectionUtils.USER_DECLARED_METHODS);
if (methodAnnotations.isEmpty() && methodAnnotations.isEmpty()) {
return null;
}
return methodAnnotations.get(0);
}
private ListenerMethod findListenerAnnotation(Method method) {
AmazonSQSListener ann = AnnotationUtils.findAnnotation(method, AmazonSQSListener.class);
if(ann == null) {
return null;
}
return new ListenerMethod(method, ann);
}
private Method checkProxy(Method methodArg, Object bean) {
Method method = methodArg;
if (AopUtils.isJdkDynamicProxy(bean)) {
try {
// Found a @RabbitListener method on the target class for this JDK proxy ->
// is it also present on the proxy itself?
method = bean.getClass().getMethod(method.getName(), method.getParameterTypes());
Class<?>[] proxiedInterfaces = ((Advised) bean).getProxiedInterfaces();
for (Class<?> iface : proxiedInterfaces) {
try {
method = iface.getMethod(method.getName(), method.getParameterTypes());
break;
}
catch (NoSuchMethodException noMethod) {
}
}
}
catch (SecurityException ex) {
ReflectionUtils.handleReflectionException(ex);
}
catch (NoSuchMethodException ex) {
throw new IllegalStateException(String.format(
"@RabbitListener method '%s' found on bean target class '%s', " +
"but not found in any interface(s) for a bean JDK proxy. Either " +
"pull the method up to an interface or switch to subclass (CGLIB) " +
"proxies by setting proxy-target-class/proxyTargetClass " +
"attribute to 'true'", method.getName(), method.getDeclaringClass().getSimpleName()), ex);
}
}
return method;
}
private static class ListenerMethod {
final Method method; // NOSONAR
final AmazonSQSListener annotations; // NOSONAR
ListenerMethod(Method method, AmazonSQSListener annotations) { // NOSONAR
this.method = method;
this.annotations = annotations;
}
}
}
下面是定义创建 MessageListenerContainer 实例的 AmazonSQSListenerContainerFactory 的工厂，在上面的AmazonSQSListenerAnnotationBeanPostProcessor检测到 spring bean 方法上标注了 @AmazonSQSListener 就会创建一个 MessageListenerContainer 实例。
public interface AmazonSQSListenerContainerFactory<C extends MessageListenerContainer> {
C createListenerContainer(String queueName, MessageListener messageListener);
}
因为 RabbitMQ 是推模式，所以只需要定义一个回调函数就可以消费消息队列了。而 Amazon SQS 需要拉模式所以，对于队列里面的消息需要自己主动去拉取并且每次拉取消息的条数只能是 10 条。所以在 @AmazonSQSListener 注解里面定义了一个 consumers() 方法用于定义 MessageListenerContainer 创建的消费该队列的并发数。而且 Amazon SQS 在消息拉取的时候并不保证线程安全，也就是如果线程 1 拉取了消息 A，如果没有删除线程 2 也会拉到了消息 B。所以在消费消息的时候需要保证幂等。以下就是定义消息拉取的逻辑。
@Slf4j
public class BlockingQueueConsumer {
private final BlockingQueue<Message> queue;
private final AmazonSQS amazonSQS;
private final String queueName;
private final Executor taskExecutor;
volatile Thread thread;
volatile boolean declaring;
public BlockingQueueConsumer(AmazonSQS amazonSQS, Executor taskExecutor, String queueName, int prefetchCount) {
this.amazonSQS = amazonSQS;
this.taskExecutor = taskExecutor;
this.queueName = queueName;
this.queue = new LinkedBlockingQueue<>(prefetchCount);
}
public Executor getTaskExecutor() {
return taskExecutor;
}
public void start() {
log.info("Starting consumer to consume queue : " + queueName );
this.thread = Thread.currentThread();
this.declaring = true;
consumeFromQueue(queueName);
}
public synchronized void stop() {
this.queue.clear(); // in case we still have a client thread blocked
}
private void consumeFromQueue(String queueName) {
InternalConsumer consumer = new InternalConsumer(this.amazonSQS, queueName);
getTaskExecutor().execute(consumer);
}
public MessageContent nextMessage() throws InterruptedException {
return handle(this.queue.take());
}
public MessageContent nextMessage(long timeout) throws InterruptedException {
MessageContent message = handle(this.queue.poll(timeout, TimeUnit.MILLISECONDS));
return message;
}
protected boolean hasDelivery() {
return !this.queue.isEmpty();
}
private MessageContent handle(Message message) {
if(message == null) {
return null;
}
if(StringUtils.isEmpty(message.getBody())) {
return null;
}
MessageContent notifyMessageContent = JSON.parseObject(message.getBody(), MessageContent.class);
return notifyMessageContent;
}
private final class InternalConsumer implements Runnable {
private final AmazonSQS amazonSQS;
private final String queueName;
public InternalConsumer(AmazonSQS amazonSQS, String queueName) {
this.amazonSQS = amazonSQS;
this.queueName = queueName;
}
@Override
public void run() {
while(true){
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(queueName);
receiveMessageRequest.withMaxNumberOfMessages(10);
//WaitTimeSeconds 使用长轮询
receiveMessageRequest.withWaitTimeSeconds(15);
//设定消息不可见时间，若消息未处理或者宕机，消息将在10分钟后被其他通知服务消费
receiveMessageRequest.setVisibilityTimeout(600);
List<Message> messages = amazonSQS.receiveMessage(receiveMessageRequest).getMessages();
if(CollectionUtils.isNotEmpty(messages)) {
for (Message message : messages) {
try {
log.info(Thread.currentThread().getName() + "receive message : " + message.getBody());
BlockingQueueConsumer.this.queue.put(message);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
String receiptHandle = message.getReceiptHandle();
DeleteMessageRequest deleteMessageRequest = new DeleteMessageRequest(queueName, receiptHandle);
amazonSQS.deleteMessage(deleteMessageRequest);
}
}
}
}
}
}
}
在拉取消息的时候会把拉取的消息添加到 BlockingQueue 队列当中。MessageListenerContainer 的具体实现类会调用 BlockingQueueConsumer#nextMessage 方法，判断是否拉取到了 Amazon SQS 对应队列的消息。如果拉取到了就调用 MessageListener 进行消息消费。
下面我们来看一下 MessageListenerContainer 的的具体实现，下面是 AbstractMessageListenerContainer 它抽象了一些公有方法：
@Slf4j
public abstract class AbstractMessageListenerContainer extends AmazonSQSAccessor implements MessageListenerContainer, DisposableBean {
public static final int DEFAULT_PREFETCH_COUNT = 250;
private Executor taskExecutor = new SimpleAsyncTaskExecutor();
private final Object lifecycleMonitor = new Object();
private volatile MessageListener messageListener;
private String queueName;
private volatile int prefetchCount = DEFAULT_PREFETCH_COUNT;
private volatile boolean active = false;
private volatile boolean running = false;
private volatile boolean initialized;
private boolean forceCloseChannel = true;
public Executor getTaskExecutor() {
return taskExecutor;
}
public void setTaskExecutor(Executor taskExecutor) {
this.taskExecutor = taskExecutor;
}
public boolean isForceCloseChannel() {
return forceCloseChannel;
}
public void setForceCloseChannel(boolean forceCloseChannel) {
this.forceCloseChannel = forceCloseChannel;
}
public MessageListener getMessageListener() {
return messageListener;
}
public void setMessageListener(MessageListener messageListener) {
this.messageListener = messageListener;
}
public int getPrefetchCount() {
return prefetchCount;
}
public void setPrefetchCount(int prefetchCount) {
this.prefetchCount = prefetchCount;
}
public String getQueueName() {
return queueName;
}
public void setQueueName(String queueName) {
this.queueName = queueName;
decorateTaskExecutorName(queueName);
}
@Override
public void start() {
if (isRunning()) {
return;
}
if (!this.initialized) {
synchronized (this.lifecycleMonitor) {
if (!this.initialized) {
afterPropertiesSet();
}
}
}
try {
if (log.isDebugEnabled()) {
log.debug("Starting Rabbit listener container.");
}
doStart();
} catch (Exception e) {
throw convertAmazonSQSAccessorException(e);
}
}
public final boolean isActive() {
synchronized (this.lifecycleMonitor) {
return this.active;
}
}
protected void doStart() throws Exception {
synchronized (this.lifecycleMonitor) {
this.active = true;
this.running = true;
this.lifecycleMonitor.notifyAll();
}
}
@Override
public void stop() {
try {
doStop();
}
catch (Exception ex) {
throw convertAmazonSQSAccessorException(ex);
}
finally {
synchronized (this.lifecycleMonitor) {
this.running = false;
this.lifecycleMonitor.notifyAll();
}
}
}
protected void invokeListener(MessageContent message) throws Exception {
MessageListener listener = getMessageListener();
try {
listener.onMessage(message);
}
catch (Exception e) {
throw wrapToListenerExecutionFailedExceptionIfNeeded(e, message);
}
}
protected Exception wrapToListenerExecutionFailedExceptionIfNeeded(Exception e, MessageContent message) {
if (!(e instanceof ListenerExecutionFailedException)) {
// Wrap exception to ListenerExecutionFailedException.
return new ListenerExecutionFailedException("Listener threw exception", e, message);
}
return e;
}
protected void doStop() {
shutdown();
}
@Override
public final boolean isRunning() {
synchronized (this.lifecycleMonitor) {
return (this.running);
}
}
@Override
public final void afterPropertiesSet() {
super.afterPropertiesSet();
initialize();
}
@Override
public void setupMessageListener(MessageListener messageListener) {
setMessageListener(messageListener);
}
@Override
public void destroy() {
shutdown();
}
private void initialize() {
try {
doInitialize();
this.initialized = true;
} catch (Exception e) {
throw convertAmazonSQSAccessorException(e);
}
}
protected abstract void doInitialize() throws Exception;
public void shutdown() {
synchronized (this.lifecycleMonitor) {
if (!isActive()) {
log.info("Shutdown ignored - container is not active already");
return;
}
this.active = false;
this.lifecycleMonitor.notifyAll();
}
log.debug("Shutting down Rabbit listener container");
// Shut down the invokers.
try {
doShutdown();
}
catch (Exception ex) {
throw convertAmazonSQSAccessorException(ex);
}
finally {
synchronized (this.lifecycleMonitor) {
this.running = false;
this.lifecycleMonitor.notifyAll();
}
}
}
protected abstract void doShutdown();
private void decorateTaskExecutorName(String queueName) {
if(this.taskExecutor == null) {
return;
}
if(!(taskExecutor instanceof SimpleAsyncTaskExecutor)){
return;
}
SimpleAsyncTaskExecutor asyncTaskExecutor = SimpleAsyncTaskExecutor.class.cast(taskExecutor);
if(queueName.indexOf("/") != -1){
String taskExecutorName = queueName.substring(queueName.lastIndexOf("/") + 1);
asyncTaskExecutor.setThreadNamePrefix(taskExecutorName);
} else {
asyncTaskExecutor.setThreadNamePrefix(queueName);
}
}
}
下面MessageListenerContainer的实现类 SimpleMessageListenerContainer：
@Slf4j
public class SimpleMessageListenerContainer extends AbstractMessageListenerContainer {
private static final long DEFAULT_START_CONSUMER_MIN_INTERVAL = 10000;
private static final long DEFAULT_STOP_CONSUMER_MIN_INTERVAL = 60000;
private static final long DEFAULT_CONSUMER_START_TIMEOUT = 60000L;
private static final int DEFAULT_CONSECUTIVE_ACTIVE_TRIGGER = 10;
private static final int DEFAULT_CONSECUTIVE_IDLE_TRIGGER = 10;
public static final long DEFAULT_RECEIVE_TIMEOUT = 1000;
protected final Object consumersMonitor = new Object(); //NOSONAR
private volatile int txSize = 1;
private volatile int consecutiveActiveTrigger = DEFAULT_CONSECUTIVE_ACTIVE_TRIGGER;
private volatile int consecutiveIdleTrigger = DEFAULT_CONSECUTIVE_IDLE_TRIGGER;
private volatile Integer maxConcurrentConsumers;
private final AtomicReference<Thread> containerStoppingForAbort = new AtomicReference<>();
private volatile long lastConsumerStarted;
private volatile long lastConsumerStopped;
private volatile long startConsumerMinInterval = DEFAULT_START_CONSUMER_MIN_INTERVAL;
private volatile long stopConsumerMinInterval = DEFAULT_STOP_CONSUMER_MIN_INTERVAL;
private volatile int concurrentConsumers = 1;
private long receiveTimeout = DEFAULT_RECEIVE_TIMEOUT;
private Set<BlockingQueueConsumer> consumers;
private long consumerStartTimeout = DEFAULT_CONSUMER_START_TIMEOUT;
public SimpleMessageListenerContainer() {}
public SimpleMessageListenerContainer(AmazonSQS amazonSQS){
super.setAmazonSQS(amazonSQS);
}
public final void setStartConsumerMinInterval(long startConsumerMinInterval) {
Assert.isTrue(startConsumerMinInterval > 0, "'startConsumerMinInterval' must be > 0");
this.startConsumerMinInterval = startConsumerMinInterval;
}
public final void setStopConsumerMinInterval(long stopConsumerMinInterval) {
Assert.isTrue(stopConsumerMinInterval > 0, "'stopConsumerMinInterval' must be > 0");
this.stopConsumerMinInterval = stopConsumerMinInterval;
}
public final void setConsecutiveActiveTrigger(int consecutiveActiveTrigger) {
Assert.isTrue(consecutiveActiveTrigger > 0, "'consecutiveActiveTrigger' must be > 0");
this.consecutiveActiveTrigger = consecutiveActiveTrigger;
}
public final void setConsecutiveIdleTrigger(int consecutiveIdleTrigger) {
Assert.isTrue(consecutiveIdleTrigger > 0, "'consecutiveIdleTrigger' must be > 0");
this.consecutiveIdleTrigger = consecutiveIdleTrigger;
}
public void setReceiveTimeout(long receiveTimeout) {
this.receiveTimeout = receiveTimeout;
}
public void setTxSize(int txSize) {
Assert.isTrue(txSize > 0, "'txSize' must be > 0");
this.txSize = txSize;
}
public void setConcurrentConsumers(final int concurrentConsumers) {
Assert.isTrue(concurrentConsumers > 0, "'concurrentConsumers' value must be at least 1 (one)");
if (this.maxConcurrentConsumers != null) {
Assert.isTrue(concurrentConsumers <= this.maxConcurrentConsumers,
"'concurrentConsumers' cannot be more than 'maxConcurrentConsumers'");
}
synchronized (this.consumersMonitor) {
if (log.isDebugEnabled()) {
log.debug("Changing consumers from " + this.concurrentConsumers + " to " + concurrentConsumers);
}
int delta = this.concurrentConsumers - concurrentConsumers;
this.concurrentConsumers = concurrentConsumers;
if (isActive()) {
adjustConsumers(delta);
}
}
}
public void setMaxConcurrentConsumers(int maxConcurrentConsumers) {
Assert.isTrue(maxConcurrentConsumers >= this.concurrentConsumers,
"'maxConcurrentConsumers' value must be at least 'concurrentConsumers'");
this.maxConcurrentConsumers = maxConcurrentConsumers;
}
@Override
protected void doStart() throws Exception {
super.doStart();
synchronized (this.consumersMonitor) {
if (this.consumers != null) {
throw new IllegalStateException("A stopped container should not have consumers");
}
int newConsumers = initializeConsumers();
if (this.consumers == null) {
log.info("Consumers were initialized and then cleared (presumably the container was stopped concurrently)");
return;
}
if (newConsumers <= 0) {
if (log.isInfoEnabled()) {
log.info("Consumers are already running");
}
return;
}
Set<AsyncMessageProcessingConsumer> processors = new HashSet<>();
for (BlockingQueueConsumer consumer : this.consumers) {
AsyncMessageProcessingConsumer processor = new AsyncMessageProcessingConsumer(consumer);
processors.add(processor);
getTaskExecutor().execute(processor);
}
for (AsyncMessageProcessingConsumer processor : processors) {
AmazonSQSStartupException startupException = processor.getStartupException();
if (startupException != null) {
throw new AmazonSQSIllegalStateException("Fatal exception on listener startup", startupException);
}
}
}
}
@Override
protected void doShutdown() {
Thread thread = this.containerStoppingForAbort.get();
if (thread != null && !thread.equals(Thread.currentThread())) {
log.info("Shutdown ignored - container is stopping due to an aborted consumer");
return;
}
try {
List<BlockingQueueConsumer> canceledConsumers = new ArrayList<>();
synchronized (this.consumersMonitor) {
if (this.consumers != null) {
Iterator<BlockingQueueConsumer> consumerIterator = this.consumers.iterator();
while (consumerIterator.hasNext()) {
BlockingQueueConsumer consumer = consumerIterator.next();
canceledConsumers.add(consumer);
consumerIterator.remove();
if (consumer.declaring) {
consumer.thread.interrupt();
}
}
}
else {
log.info("Shutdown ignored - container is already stopped");
return;
}
}
log.info("Workers not finished.");
if (isForceCloseChannel()) {
canceledConsumers.forEach(consumer -> {
if (log.isWarnEnabled()) {
log.warn("Closing channel for unresponsive consumer: " + consumer);
}
consumer.stop();
});
}
} catch (Exception e) {
Thread.currentThread().interrupt();
log.warn("Interrupted waiting for workers. Continuing with shutdown.");
}
synchronized (this.consumersMonitor) {
this.consumers = null;
}
}
protected int initializeConsumers() {
int count = 0;
synchronized (this.consumersMonitor) {
if (this.consumers == null) {
this.consumers = new HashSet<>(this.concurrentConsumers);
for (int i = 0; i < this.concurrentConsumers; i++) {
BlockingQueueConsumer consumer = createBlockingQueueConsumer();
this.consumers.add(consumer);
count++;
}
}
}
return count;
}
private BlockingQueueConsumer createBlockingQueueConsumer() {
String queueName = getQueueName();
int actualPrefetchCount = getPrefetchCount() > this.txSize ? getPrefetchCount() : this.txSize;
BlockingQueueConsumer consumer = new BlockingQueueConsumer(getAmazonSQS(), getTaskExecutor(), queueName, actualPrefetchCount);
return consumer;
}
@Override
protected void doInitialize() throws Exception {
// do nothing
}
private boolean isActive(BlockingQueueConsumer consumer) {
boolean consumerActive;
synchronized (this.consumersMonitor) {
consumerActive = this.consumers != null && this.consumers.contains(consumer);
}
return consumerActive && this.isActive();
}
private final class AsyncMessageProcessingConsumer implements Runnable {
private final BlockingQueueConsumer consumer;
private final CountDownLatch start;
private volatile AmazonSQSStartupException startupException;
private AmazonSQSStartupException getStartupException() throws TimeoutException, InterruptedException {
if (!this.start.await(
SimpleMessageListenerContainer.this.consumerStartTimeout, TimeUnit.MILLISECONDS)) {
log.error("Consumer failed to start in "
+ SimpleMessageListenerContainer.this.consumerStartTimeout
+ " milliseconds; does the task executor have enough threads to sqs the container "
+ "concurrency?");
}
return this.startupException;
}
AsyncMessageProcessingConsumer(BlockingQueueConsumer consumer) {
this.consumer = consumer;
this.start = new CountDownLatch(1);
}
@Override
public void run() {
if (!isActive()) {
return;
}
int consecutiveIdles = 0;
int consecutiveMessages = 0;
try {
try {
this.consumer.start();
this.start.countDown();
} catch (Throwable t) {
this.start.countDown();
throw t;
}
while (isActive(this.consumer) || this.consumer.hasDelivery()) {
boolean receivedOk = receiveAndExecute(this.consumer);
if (receivedOk) {
if (isActive(this.consumer)) {
if (consecutiveMessages++ > SimpleMessageListenerContainer.this.consecutiveActiveTrigger) {
considerAddingAConsumer();
consecutiveMessages = 0;
}
}
} else {
consecutiveMessages = 0;
if (consecutiveIdles++ > SimpleMessageListenerContainer.this.consecutiveIdleTrigger) {
considerStoppingAConsumer(this.consumer);
consecutiveIdles = 0;
}
}
}
} catch (InterruptedException e) {
log.debug("Consumer thread interrupted, processing stopped.");
Thread.currentThread().interrupt();
} catch (Error e){
log.error("Consumer thread error, thread abort.", e);
} catch (Throwable t) {
log.debug("Consumer raised exception, processing can restart if the connection factory supports it", t);
} // In all cases count down to allow container to progress beyond startup
this.start.countDown();
if (!isActive(this.consumer)){
log.debug("Cancelling " + this.consumer);
this.consumer.stop();
}
}
}
private boolean receiveAndExecute(final BlockingQueueConsumer consumer) throws Throwable {
return doReceiveAndExecute(consumer);
}
private boolean doReceiveAndExecute(BlockingQueueConsumer consumer) throws Throwable { //NOSONAR
for (int i = 0; i < this.txSize; i++) {
log.trace("Waiting for message from consumer.");
MessageContent message = consumer.nextMessage(this.receiveTimeout);
if (message == null) {
break;
}
try {
invokeListener(message);
} catch (Throwable e) { //NOSONAR
if (log.isDebugEnabled()) {
log.debug("process message fail the reason is : " + e.getMessage() + "the message is " + message);
}
break;
}
}
return true;
}
protected void adjustConsumers(int delta) {
synchronized (this.consumersMonitor) {
if (isActive() && this.consumers != null) {
if (delta > 0) {
Iterator<BlockingQueueConsumer> consumerIterator = this.consumers.iterator();
while (consumerIterator.hasNext() && delta > 0
&& (this.maxConcurrentConsumers == null
|| this.consumers.size() > this.maxConcurrentConsumers)) {
BlockingQueueConsumer consumer = consumerIterator.next();
consumerIterator.remove();
delta--;
}
}
else {
addAndStartConsumers(-delta);
}
}
}
}
private void considerAddingAConsumer() {
synchronized (this.consumersMonitor) {
if (this.consumers != null
&& this.maxConcurrentConsumers != null && this.consumers.size() < this.maxConcurrentConsumers) {
long now = System.currentTimeMillis();
if (this.lastConsumerStarted + this.startConsumerMinInterval < now) {
this.addAndStartConsumers(1);
this.lastConsumerStarted = now;
}
}
}
}
private void considerStoppingAConsumer(BlockingQueueConsumer consumer) {
synchronized (this.consumersMonitor) {
if (this.consumers != null && this.consumers.size() > this.concurrentConsumers) {
long now = System.currentTimeMillis();
if (this.lastConsumerStopped + this.stopConsumerMinInterval < now) {
this.consumers.remove(consumer);
if (log.isDebugEnabled()) {
log.debug("Idle consumer terminating: " + consumer);
}
this.lastConsumerStopped = now;
}
}
}
}
protected void addAndStartConsumers(int delta) {
synchronized (this.consumersMonitor) {
if (this.consumers != null) {
for (int i = 0; i < delta; i++) {
if (this.maxConcurrentConsumers != null
&& this.consumers.size() >= this.maxConcurrentConsumers) {
break;
}
BlockingQueueConsumer consumer = createBlockingQueueConsumer();
this.consumers.add(consumer);
AsyncMessageProcessingConsumer processor = new AsyncMessageProcessingConsumer(consumer);
if (log.isDebugEnabled()) {
log.debug("Starting a new consumer: " + consumer);
}
getTaskExecutor().execute(processor);
try {
AmazonSQSStartupException startupException = processor.getStartupException();
if (startupException != null) {
this.consumers.remove(consumer);
throw new AmazonSQSIllegalStateException("Fatal exception on listener startup", startupException);
}
}
catch (InterruptedException ie) {
Thread.currentThread().interrupt();
}
catch (Exception e) {
consumer.stop();
log.error("Error starting new consumer", e);
this.consumers.remove(consumer);
}
}
}
}
}
}
它的作用就是创建线程来拉取 Amazon SQS 队列里面的消息，并且通过 AmazonSQSListenerAnnotationBeanPostProcessor 创建的 MessageListener 来消费消息。主要是通过他的内部类 AsyncMessageProcessingConsumer 来创建一个我们上面提到的 BlockingQueueConsumer 拉取队列里面的消息,然后死循环调用 SimpleMessageListenerContainer#receiveAndExecute 它会调用 BlockingQueueConsumer#nextMessage 方法判断是否拉到到消息。如果有消息的话，就会调用实现 BeanPostProcess 的 AmazonSQSListenerAnnotationBeanPostProcessor 传入的 MessageListener 也就是在 spring bean 标注 @AmazonSQSListener 的方法。上面就是消息处理的整个逻辑了。
由于 Amazon SQS 必须在 Amazon 云上才能使用，本地无法调试。所以我通过在 local 环境就 mock 了一下它。
public class MockAmazonSQS extends AbstractAmazonSQSAsync {
@Getter
private static final Map<String, Message> messages = new ConcurrentHashMap<>();
static {
mockMessage();
}
@Override
public ReceiveMessageResult receiveMessage(ReceiveMessageRequest request) {
ReceiveMessageResult result = new ReceiveMessageResult();
String queueUrl = request.getQueueUrl();
Message message = messages.get(queueUrl);
if(message != null) {
result.setMessages(Lists.newArrayList(message));
}
return result;
}
@Override
public SendMessageResult sendMessage(SendMessageRequest request){
String queueUrl = request.getQueueUrl();
Message message = new Message();
message.setBody(request.getMessageBody());
messages.put(queueUrl, message);
return new SendMessageResult();
}
private static void mockMessage() {
messages.put("amazon.local.mock", genMessage("mock message"));
}
@Override
public DeleteMessageResult deleteMessage(DeleteMessageRequest deleteMessageRequest){
String queueUrl = deleteMessageRequest.getQueueUrl();
messages.remove(queueUrl);
return new DeleteMessageResult();
}
private static Message genMessage(String message) {
Message localMessage = new Message();
MessageContent localMessageContent = new MessageContent();
localMessageContent.setMessage(message);
localMessage.setBody(JSON.toJSONString(localMessageContent));
return localMessage;
}
}
并且定义了一下添加消息的方法。可以在 Controller 里面添加消息，模拟队列里面有消息需要消费。最后就可以直接定义一个监听者用于消费 SQS 里面的消息了。
@Component
public class LocalMessageListener {
@AmazonSQSListener(queue = "amazon.local.mock")
public void onMessage(MessageContent messageContent){
System.out.println(messageContent);
}
}
由于我在 MockAmazonSQS 实例初始化的时候添加了一条消息。所以在项目启动的时候就会消费这条消息。所以在控制台打印出了这条消息：
然后我通过 Controller 向 MockAmazonSQS 添加消息。这条消息同样也可以消费，以下是 Postman 发送消息：
下面是控制台打印消费消息的日志：
这写这篇 blog 的时候，由于需要创建 demo。发现 spring 提供了 amazon simple queue server 的自动依赖，它的功能肯定比我的强大。发现自己是多此一举，但是为了把自己的思路分享给大家还是决定写这篇 blog。
源码地址：sqs-demo
