Skip to main content

Application Integration Services

In previous lessons, you've learned how to build resilient, highly available applications on AWS. Now we'll explore how to make these applications communicate effectively. Application Integration Services are the glue that connects distributed components, enabling them to work together seamlessly while remaining loosely coupled.

Learning Goals

  • Understand the purpose and benefits of loose coupling
  • Learn core AWS integration services: SQS, SNS, and EventBridge
  • Implement message queues, pub/sub patterns, and event-driven architectures
  • Choose the right service for different integration scenarios

Why Application Integration Matters​

Modern applications are built as collections of distributed services. Without proper integration patterns, these services can become tightly coupled, leading to:

  • Cascading failures when one service goes down
  • Difficulty scaling individual components
  • Complex error handling and retry logic
  • Challenges implementing new features

AWS Application Integration Services solve these problems by providing managed messaging and event routing capabilities.

Core Integration Services​

Amazon Simple Queue Service (SQS)​

SQS provides fully managed message queues that enable asynchronous communication between services. Messages are stored redundantly across multiple availability zones.

producer.py
import boto3
import json

# Create SQS client
sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

# Send message to queue
response = sqs.send_message(
    QueueUrl=queue_url,
    MessageBody=json.dumps({
        'order_id': '12345',
        'customer_email': 'user@example.com',
        'total_amount': 99.99
    }),
    MessageAttributes={
        'MessageType': {
            'DataType': 'String',
            'StringValue': 'OrderCreated'
        }
    }
)

print(f"Message ID: {response['MessageId']}")
consumer.py
import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

# Receive messages from queue
response = sqs.receive_message(
    QueueUrl=queue_url,
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20,  # Long polling
    MessageAttributeNames=['All']
)

if 'Messages' in response:
    for message in response['Messages']:
        body = json.loads(message['Body'])
        print(f"Processing order: {body['order_id']}")
        
        # Delete message after processing
        sqs.delete_message(
            QueueUrl=queue_url,
            ReceiptHandle=message['ReceiptHandle']
        )
tip

Use long polling (WaitTimeSeconds=20) instead of short polling to reduce costs and latency. Long polling waits for messages to arrive rather than making frequent empty requests.

Amazon Simple Notification Service (SNS)​

SNS is a pub/sub service for sending messages to multiple subscribers. It supports multiple protocols including HTTP/S, email, SMS, and SQS.

sns_publisher.py
import boto3
import json

sns = boto3.client('sns')
topic_arn = 'arn:aws:sns:us-east-1:123456789012:order-notifications'

# Publish message to topic
response = sns.publish(
    TopicArn=topic_arn,
    Message=json.dumps({
        'default': json.dumps({
            'order_id': '12345',
            'status': 'shipped',
            'tracking_number': '1Z999AA10123456784'
        })
    }),
    MessageStructure='json',
    MessageAttributes={
        'Priority': {
            'DataType': 'String',
            'StringValue': 'High'
        }
    }
)

print(f"Message ID: {response['MessageId']}")

Amazon EventBridge​

EventBridge is a serverless event bus that makes it easy to connect applications using data from your own apps, SaaS, and AWS services.

eventbridge_rule.py
import boto3
import json

events = boto3.client('events')

# Put custom event
response = events.put_events(
    Entries=[
        {
            'Source': 'my.application',
            'DetailType': 'OrderCompleted',
            'Detail': json.dumps({
                'order_id': '12345',
                'completion_time': '2024-01-15T10:30:00Z'
            }),
            'EventBusName': 'default'
        }
    ]
)

print(f"Entries: {response['Entries']}")

Integration Patterns in Action​

SQS + SNS: Fan-out Pattern​

Combine SNS and SQS to deliver messages to multiple queues simultaneously:

fanout_setup.py
import boto3

sns = boto3.client('sns')
sqs = boto3.client('sqs')

# Create SNS topic
topic_response = sns.create_topic(Name='order-events')
topic_arn = topic_response['TopicArn']

# Create multiple SQS queues
queues = ['inventory-updates', 'shipping-notifications', 'analytics-events']
for queue_name in queues:
    queue_response = sqs.create_queue(QueueName=queue_name)
    queue_arn = f"arn:aws:sqs:us-east-1:123456789012:{queue_name}"
    
    # Subscribe queue to topic
    sns.subscribe(
        TopicArn=topic_arn,
        Protocol='sqs',
        Endpoint=queue_arn
    )

Event-Driven Architecture with EventBridge​

Create decoupled workflows using EventBridge rules:

eventbridge-rule.yaml
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  OrderProcessingRule:
    Type: AWS::Events::Rule
    Properties:
      EventBusName: default
      EventPattern:
        source:
          - "my.application"
        detail-type:
          - "OrderCreated"
      Targets:
        - Arn: !GetArg OrderProcessorFunction.Arn
          Id: "OrderProcessorTarget"
note

EventBridge uses JSON-based event patterns for routing. You can match on any field in the event, including nested attributes, making it extremely flexible for complex routing logic.

Choosing the Right Service​

Use SQS when:

  • Need guaranteed message delivery with exactly-once processing
  • Implementing task queues for background processing
  • Buffering between producers and consumers with different throughput

Use SNS when:

  • Broadcasting messages to multiple subscribers
  • Need multiple delivery protocols (email, SMS, HTTP)
  • Implementing event notifications

Use EventBridge when:

  • Building event-driven architectures
  • Integrating with third-party SaaS applications
  • Need complex event routing and transformation
  • Working with scheduled events

Common Pitfalls​

  • Ignoring visibility timeouts: Messages can be processed multiple times if visibility timeout is shorter than processing time
  • Overlooking dead-letter queues: Always configure DLQ for handling failed messages
  • Not using message attributes: Missing opportunities for filtering and routing
  • Forgetting error handling: Assume network issues and service throttling will occur
  • Poor message size management: SQS/SNS have 256KB message limits; use S3 for larger payloads

Summary​

Application Integration Services are essential for building scalable, resilient cloud applications. SQS provides reliable message queuing, SNS enables pub/sub messaging, and EventBridge facilitates complex event-driven architectures. By leveraging these services, you can create loosely coupled systems that are easier to maintain, scale, and extend.

Quiz​

AWS Messaging & Integration Fundamentals

Which service would you choose to ensure a message is processed exactly once by a single consumer?

Question 1/4