Amazon SQS (Simple Queue Service) 📨 made simple
Step by step guide using AWS CLI and SDK for C#
What is Messaging?
Messaging is a method of communication between software components or systems, enabling them to exchange data asynchronously without tight coupling. This is crucial in distributed architectures, where components need to operate independently for scalability, reliability, and fault tolerance. Messaging is widely used in event-driven systems, microservices, and real-time data processing pipelines. For example, RabbitMQ and Amazon SQS facilitate message queuing for tasks like order processing or notification services, ensuring no data is lost if a component goes down. Apache Kafka excels in handling high-throughput, real-time event streams, such as logging or analytics. Redis Pub/Sub and Azure Service Bus are often used for lightweight, real-time notifications in applications like chat or live updates. By decoupling components, messaging ensures seamless integration, improved system responsiveness, and easier maintenance.
What is Amazon SQS?
Amazon Simple Queue Service (SQS) is a fully managed message queuing service provided by AWS. It enables the decoupling and scaling of microservices, distributed systems, and serverless applications by allowing components to communicate asynchronously through messages.
SQS provides a way to send, store, and receive messages between different services or components of an application. It ensures message delivery without requiring the sender and receiver to interact directly.
Amazon SQS Queues come in two types; Standard Queues and FIFO Queues (First In, First Out).
Standard Queues
Key Characteristics:
- Nearly unlimited throughput: Processes a virtually unlimited number of transactions per second.
- At-least-once delivery: A message may be delivered more than once (duplicates possible).
- Best-effort ordering: Messages are not guaranteed to be delivered in the order they were sent.
Use Case Example is Order Processing System: A high-traffic e-commerce platform where customer orders are processed independently. Even if a duplicate message appears, the system can handle it by deduplication mechanisms in the application.
FIFO Queues (First In, First Out)
Key Characteristics:
- Limited throughput: Supports up to 300 transactions per second with batching or 3,000 transactions per second with message groups.
- Exactly-once delivery: Messages are delivered exactly once, eliminating duplicates.
- Strict ordering: Messages are delivered in the order they were sent within a message group.
Use Case Example is Financial Transactions System: For a banking application where the order of transactions (like debits and credits) is critical to maintaining account integrity.
Which one is the right choice?
You may consider Standard Queue if:
- High throughput and scalability are your primary concerns.
- Your system can tolerate occasional duplicate messages and out-of-order delivery.
Example: A real-time log processing system where message ordering and duplication don’t matter as long as all messages are processed.
You may consider FIFO Queue if:
- Order and exact message delivery are critical to your application.
- You can operate within the throughput limits of FIFO queues.
Example: A ticket reservation system where issuing tickets in the exact order of requests is essential to avoid overbooking.
More detailed example, Imagine you are building a Food Delivery Application:
Use Standard Queue for distributing restaurant promotions or push notifications to users. Duplication or order isn’t critical, and high throughput is required for a large user base.
Use FIFO Queue for placing and confirming customer orders where maintaining the sequence of order requests and ensuring no duplicates are crucial. For instance, if a customer adds items to a cart and proceeds to checkout, those actions must follow a strict sequence.
When in doubt, assess your application’s need for strict ordering and exact delivery guarantees. Start with a Standard Queue for general use cases, and switch to FIFO only when the requirements clearly demand it. Balancing performance and cost-effectiveness is key.
Key Configuration Fields in Amazon SQS
Amazon SQS allows you to configure several fields to customize its behaviour for your application needs. These fields dictate how messages are handled, delivered, and retained.
1. Queue Type
Determines whether the queue is Standard or FIFO.
Configuration Options:
- Standard Queue: Offers high throughput with at-least-once delivery and best-effort ordering.
- FIFO Queue: Ensures strict ordering and exactly-once delivery but supports limited throughput.
Example: Choose FIFO for a payment processing system where transaction order is critical.
2. Message Retention Period
Defines how long a message is stored in the queue if it is not consumed.
Configuration Options:
- Range: 1 minute to 14 days.
- Default: 4 days.
Example: For an analytics application, set retention to the maximum (14 days) to ensure no event data is lost during system downtime.
3. Visibility Timeout
The time during which a message is hidden from other consumers after it is retrieved by one.
Configuration Options:
- Range: 0 to 12 hours.
- Default: 30 seconds.
Example: For a batch processing system, set the visibility timeout longer than the maximum processing time to prevent duplicate processing.
4. Maximum Message Size
Specifies the largest payload size (in bytes) allowed for a single message.
Configuration Options:
- Range: 1 KB to 256 KB.
- Default: 256 KB.
Example: Use the maximum size when transmitting large JSON payloads in a system that processes detailed user logs.
5. Delivery Delay
The time delay before a message becomes visible to consumers.
Configuration Options:
- Range: 0 to 15 minutes.
- Default: 0 seconds.
Example: For email reminders, set a delay to send messages closer to event times, like 15 minutes before a meeting.
6. Receive Message Wait Time
Configures long polling to reduce empty responses when retrieving messages.
Configuration Options:
- Range: 0 to 20 seconds.
- Default: 0 seconds (short polling).
Example: Use a wait time of 20 seconds in a cost-sensitive system to reduce API calls when the queue has infrequent messages.
7. Message Deduplication ID (FIFO Only)
Ensures message deduplication by assigning a unique identifier.
Configuration Options:
- Application-generated or auto-assigned.
- Deduplication window: 5 minutes.
Example: For an e-commerce system, use the order ID as the deduplication ID to prevent duplicate order processing.
8. Message Group ID (FIFO Only)
Groups messages for ordered processing within FIFO queues.
Configuration Options:
- Application-generated for each logical message group.
Example: In a stock trading system, use the user account ID as the group ID to ensure transactions for a single user are processed sequentially.
9. Dead-Letter Queue (DLQ)
Captures messages that failed to process after a maximum number of retries.
Configuration Options:
- Set a target queue as the DLQ.
- Configure Maximum Receives: Number of times a message can be retried before moving to the DLQ.
Example: For a customer feedback processing system, use a DLQ to store problematic feedback for manual review.
10. Encryption Settings
Protects the contents of messages using server-side encryption (SSE).
Configuration Options:
- Use AWS Key Management Service (KMS) keys.
Example: For a healthcare application, enable SSE to comply with data privacy regulations like HIPAA.
11. Access Control (Permissions)
Defines who can send and receive messages from the queue.
Configuration Options:
- Configure Amazon IAM (Identity and Access Management) policies or use the SQS Access Policy JSON.
Example: Allow only specific microservices in your architecture to send or receive messages to ensure secure communication.
12. Redrive Policy
Moves messages to a DLQ (Dead Letter Queue) after exceeding the retry limit.
Configuration Options:
- Specify the target DLQ and the Maximum Receives threshold.
Example: For a subscription notification system, configure a DLQ for messages that fail delivery after 5 retries.
13. Tags
Add metadata to organize and manage SQS resources.
Configuration Options:
- Key-value pairs for categorizing queues.
Example: Tag a queue as Environment:Production and Application:OrderService for easy tracking in billing and monitoring.
Dealing with SQS Queues (AWS CLI and C#):
Assuming the LocalStack has been created using the steps in a previous post Installing LocalStack (To Simulate AWS) using Docker Desktop for Windows and the AWS CLI has been installed.
Using AWS CLI Commands
Create a Standard Type Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs create-queue --queue-name MyQueueCreate a FIFO Type Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs create-queue --queue-name MyQueue.fifo --attributes FifoQueue=trueList the Queues
aws --profile localstack --endpoint-url=http://localhost:4566 sqs list-queuesResponse
{
"QueueUrls": [
"http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/MyQueue",
"http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/MyQueue.fifo"
]
}Get Queue Attributes
aws --profile localstack --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url http://localhost:4566/000000000000/MyQueue --attribute-names AllResponse
{
"Attributes": {
"ApproximateNumberOfMessages": "0",
"ApproximateNumberOfMessagesNotVisible": "0",
"ApproximateNumberOfMessagesDelayed": "0",
"CreatedTimestamp": "1732272815",
"DelaySeconds": "0",
"LastModifiedTimestamp": "1732272815",
"MaximumMessageSize": "262144",
"MessageRetentionPeriod": "345600",
"QueueArn": "arn:aws:sqs:us-east-1:000000000000:MyQueue",
"ReceiveMessageWaitTimeSeconds": "0",
"VisibilityTimeout": "30",
"SqsManagedSseEnabled": "true"
}
}Set Queue Attributes (for example visibility timeout to 60 seconds)
aws --profile localstack --endpoint-url=http://localhost:4566 sqs set-queue-attributes --queue-url http://localhost:4566/000000000000/MyQueue --attributes '{"VisibilityTimeout":"60"}'Get a Queue Attribute (for example visibility timeout)
aws --profile localstack --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url http://localhost:4566/000000000000/MyQueue --attribute-names VisibilityTimeoutResponse
{
"Attributes": {
"VisibilityTimeout": "60"
}
}Send a Message
aws --profile localstack --endpoint-url=http://localhost:4566 sqs send-message --queue-url http://localhost:4566/000000000000/MyQueue --message-body "Hello, World!"Response
{
"MD5OfMessageBody": "65a8e27d8879283831b664bd8b7f0ad4",
"MessageId": "06109823-13f1-465c-b241-7b3359e0edbb"
}Receive Messages
aws --profile localstack --endpoint-url=http://localhost:4566 sqs receive-message --queue-url http://localhost:4566/000000000000/MyQueueResponse
{
"Messages": [
{
"MessageId": "06109823-13f1-465c-b241-7b3359e0edbb",
"ReceiptHandle": "NjEwNGNlOWYtODNmYi00Yjk5LWJlNjctOTM3MTM3OTFmMDk2IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6TXlRdWV1ZSAwNjEwOTgyMy0xM2YxLTQ2NWMtYjI0MS03YjMzNTllMGVkYmIgMTczMjI3MzkxNC42MjU3OTgy",
"MD5OfBody": "65a8e27d8879283831b664bd8b7f0ad4",
"Body": "Hello, World!"
}
]
}Delete Message
aws --profile localstack --endpoint-url=http://localhost:4566 sqs delete-message --queue-url http://localhost:4566/000000000000/MyQueue --receipt-handle "NjEwNGNlOWYtODNmYi00Yjk5LWJlNjctOTM3MTM3OTFmMDk2IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6TXlRdWV1ZSAwNjEwOTgyMy0xM2YxLTQ2NWMtYjI0MS03YjMzNTllMGVkYmIgMTczMjI3MzkxNC42MjU3OTgy"Purge Queue (Delete all messages)
aws --profile localstack --endpoint-url=http://localhost:4566 sqs purge-queue --queue-url http://localhost:4566/000000000000/MyQueueTag a Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs tag-queue --queue-url http://localhost:4566/000000000000/MyQueue --tags Key=Environment,Value=DevList Queue Tags
aws --profile localstack --endpoint-url=http://localhost:4566 sqs list-queue-tags --queue-url http://localhost:4566/000000000000/MyQueueResponse
{
"Tags": {
"Key": "Environment",
"Value": "Dev"
}
}Untag a Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs untag-queue --queue-url http://localhost:4566/000000000000/MyQueue --tag-keys "Environment"Add Permissions to a Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs add-permission --queue-url http://localhost:4566/000000000000/MyQueue --label AllowSend --aws-account-ids 123456789012 --actions SendMessageList Permissions of a Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url http://localhost:4566/000000000000/MyQueue --attribute-names PolicyRemove Permissions from a Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs remove-permission --queue-url http://localhost:4566/000000000000/MyQueue --label AllowSendReceive Message with Long Polling
aws --profile localstack --endpoint-url=http://localhost:4566 sqs receive-message --queue-url http://localhost:4566/000000000000/MyQueue --wait-time-seconds 10Configure Dead-Letter Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs set-queue-attributes --queue-url http://localhost:4566/000000000000/MyQueue --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:000000000000:DLQ\",\"maxReceiveCount\":\"5\"}"}'Get Redrive Policy
aws --profile localstack --endpoint-url=http://localhost:4566 sqs get-queue-attributes --queue-url http://localhost:4566/000000000000/MyQueue --attribute-names RedrivePolicyResponse
{
"Attributes": {
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:000000000000:DLQ\",\"maxReceiveCount\":\"5\"}"
}
}Delete a Queue
aws --profile localstack --endpoint-url=http://localhost:4566 sqs delete-queue --queue-url http://localhost:4566/000000000000/MyQueueUsing AWS SDK for .NET (C# Examples):
Create an SQS Client
using Amazon.SQS;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
await Console.Out.WriteLineAsync("SQS Client has been created");
}
static AmazonSQSClient CreateAmazonSQSClient()
{
var awsAccessKeyId = "test";
var awsSecretAccessKey = "test";
var serviceURL = "http://localhost:4566";
var region = "us-east-1";
var sqsConfig = new AmazonSQSConfig { ServiceURL = serviceURL, AuthenticationRegion = region };
return new AmazonSQSClient(awsAccessKeyId, awsSecretAccessKey, sqsConfig);
}
}Create a Standard Queue
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
await CreateStandardQueueAsync(standardQueueName, sqsClient);
await Console.Out.WriteLineAsync($"{standardQueueName} has been created");
}
static async Task CreateStandardQueueAsync(string queueName, AmazonSQSClient sqsClient)
{
var createQueueRequest = new CreateQueueRequest { QueueName = queueName };
await sqsClient.CreateQueueAsync(createQueueRequest);
}
}Create a FIFO Queue
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string fifoQueueName = "my-queue.fifo";
await CreateFIFOQueueAsync(fifoQueueName, sqsClient);
await Console.Out.WriteLineAsync($"{fifoQueueName} has been created");
}
static async Task CreateFIFOQueueAsync(string queueName, AmazonSQSClient sqsClient)
{
var createQueueRequest = new CreateQueueRequest
{
QueueName = queueName,
Attributes = new Dictionary<string, string> { { "FifoQueue", "true" }, { "ContentBasedDeduplication", "true" } }
};
await sqsClient.CreateQueueAsync(createQueueRequest);
}
}List Queues
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
await ListAllQueuesAsync(sqsClient);
}
static async Task ListAllQueuesAsync(AmazonSQSClient sqsClient)
{
var listQueuesResponse = await sqsClient.ListQueuesAsync(new ListQueuesRequest());
foreach (var url in listQueuesResponse.QueueUrls)
Console.WriteLine(url);
}
}Get the Queue URL
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await Console.Out.WriteLineAsync($"the queue URL of {standardQueueName} is {queueUrl}");
}
static async Task<string> GetQueueUrlAsync(string queueName, AmazonSQSClient sqsClient)
{
var getQueueUrlRequest = new GetQueueUrlRequest { QueueName = queueName };
var getQueueUrlResponse = await sqsClient.GetQueueUrlAsync(getQueueUrlRequest);
return getQueueUrlResponse.QueueUrl;
}
}List all Attributes
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await ListQueueAttributesAsync(queueUrl, sqsClient);
}
static async Task ListQueueAttributesAsync(string queueUrl, AmazonSQSClient sqsClient)
{
var getQueueAttributesRequest = new GetQueueAttributesRequest
{
QueueUrl = queueUrl,
AttributeNames = new List<string> { "All" }
};
var attributes = await sqsClient.GetQueueAttributesAsync(getQueueAttributesRequest);
foreach (var attribute in attributes.Attributes)
Console.WriteLine($"{attribute.Key}: {attribute.Value}");
}
}Set Queue Attribute
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await SetQueueAttributesAsync(queueUrl, attribute: "VisibilityTimeout", value: "60", sqsClient);
}
static async Task SetQueueAttributesAsync(string queueUrl, string attribute, string value, AmazonSQSClient sqsClient)
{
var setAttributesRequest = new SetQueueAttributesRequest
{
QueueUrl = queueUrl,
Attributes = new Dictionary<string, string> { { attribute, value } }
};
await sqsClient.SetQueueAttributesAsync(setAttributesRequest);
}
}Retrieve Queue Attribute
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await PrintQueueAttributeAsync(queueUrl, attributeName: "VisibilityTimeout", sqsClient);
}
static async Task PrintQueueAttributeAsync(string queueUrl, string attributeName, AmazonSQSClient sqsClient)
{
var getAttributesRequest = new GetQueueAttributesRequest
{
QueueUrl = queueUrl,
AttributeNames = new List<string> { attributeName }
};
var attributes = await sqsClient.GetQueueAttributesAsync(getAttributesRequest);
if (attributes.Attributes.TryGetValue(attributeName, out var value))
Console.WriteLine($"{attributeName}: {value}");
}
}Send Message
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await SendMessageAsync(queueUrl, messageBody: "Hello, SQS!", sqsClient);
}
static async Task SendMessageAsync(string queueUrl, string messageBody, AmazonSQSClient sqsClient)
{
var sendMessageRequest = new SendMessageRequest { QueueUrl = queueUrl, MessageBody = messageBody };
await sqsClient.SendMessageAsync(sendMessageRequest);
}
}Retrieve Messages
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await ReceiveMessagesAsync(queueUrl, sqsClient);
}
static async Task ReceiveMessagesAsync(string queueUrl, AmazonSQSClient sqsClient)
{
var receiveMessagesRequest = new ReceiveMessageRequest { QueueUrl = queueUrl, MaxNumberOfMessages = 10 };
var messages = await sqsClient.ReceiveMessageAsync(receiveMessagesRequest);
foreach (var message in messages.Messages)
Console.WriteLine($"{message.MessageId}: {message.Body}");
}
}Delete Message (usually after retrieving it)
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await ReceiveAndMessagesAsync(queueUrl, sqsClient);
}
static async Task ReceiveAndMessagesAsync(string queueUrl, AmazonSQSClient sqsClient)
{
var receiveMessagesRequest = new ReceiveMessageRequest { QueueUrl = queueUrl, MaxNumberOfMessages = 10 };
var messages = await sqsClient.ReceiveMessageAsync(receiveMessagesRequest);
foreach (var message in messages.Messages)
{
var messageBody = message.Body;
// do something with messageBody
var receiptHandle = message.ReceiptHandle;
await DeleteMessageAsync(queueUrl, receiptHandle, sqsClient);
}
}
static async Task DeleteMessageAsync(string queueUrl, string receiptHandle, AmazonSQSClient sqsClient)
{
var deleteMessageRequest = new DeleteMessageRequest { QueueUrl = queueUrl, ReceiptHandle = receiptHandle };
await sqsClient.DeleteMessageAsync(deleteMessageRequest);
}
}Purge Queue
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await PurgeQueueAsync(queueUrl, sqsClient);
}
static async Task PurgeQueueAsync(string queueUrl, AmazonSQSClient sqsClient)
{
var purgeRequest = new PurgeQueueRequest { QueueUrl = queueUrl };
await sqsClient.PurgeQueueAsync(purgeRequest);
}
}Tag a Queue
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await TagQueueAsync(queueUrl, new Dictionary<string, string> { { "Environment", "Dev" } }, sqsClient);
}
static async Task TagQueueAsync(string queueUrl, Dictionary<string, string> tags, AmazonSQSClient sqsClient)
{
var tagQueueRequest = new TagQueueRequest { QueueUrl = queueUrl, Tags = tags };
await sqsClient.TagQueueAsync(tagQueueRequest);
}
}List Queue Tags
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await ListQueueTagsAsync(queueUrl, sqsClient);
}
static async Task ListQueueTagsAsync(string queueUrl, AmazonSQSClient sqsClient)
{
var tags = await sqsClient.ListQueueTagsAsync(new ListQueueTagsRequest { QueueUrl = queueUrl });
foreach (var tag in tags.Tags)
Console.WriteLine($"{tag.Key}: {tag.Value}");
}
}Untag a Queue
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await UntagQueueAsync(queueUrl, new List<string> { "Environment" }, sqsClient);
}
static async Task UntagQueueAsync(string queueUrl, List<string> tagKeys, AmazonSQSClient sqsClient)
{
var untagRequest = new UntagQueueRequest { QueueUrl = queueUrl, TagKeys = tagKeys };
await sqsClient.UntagQueueAsync(untagRequest);
}
}Add Permission
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await AddQueuePermissionsAsync(queueUrl, "PermissionLabel", new List<string> { "SendMessage" }, awsAccount: "123456789012", sqsClient);
}
static async Task AddQueuePermissionsAsync(string queueUrl, string label, List<string> actions, string awsAccount, AmazonSQSClient sqsClient)
{
var addPermissionRequest = new AddPermissionRequest
{
QueueUrl = queueUrl,
Label = label,
AWSAccountIds = new List<string> { awsAccount },
Actions = actions
};
await sqsClient.AddPermissionAsync(addPermissionRequest);
}
}Remove Permission
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await RemoveQueuePermissionAsync(queueUrl, "PermissionLabel", sqsClient);
}
static async Task RemoveQueuePermissionAsync(string queueUrl, string label, AmazonSQSClient sqsClient)
{
var removePermissionRequest = new RemovePermissionRequest { QueueUrl = queueUrl, Label = label };
await sqsClient.RemovePermissionAsync(removePermissionRequest);
}
}Configure Dead Letter Queue
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
string standardQueueDlqName = "standard-queue-dlq";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
var dlqUrl = await GetQueueUrlAsync(standardQueueDlqName, sqsClient);
await ConfigureDeadLetterQueueAsync(queueUrl, dlqUrl, maxReceiveCount: 5, sqsClient);
}
static async Task ConfigureDeadLetterQueueAsync(string sourceQueueUrl, string dlqUrl, int maxReceiveCount, AmazonSQSClient sqsClient)
{
var attributes = new Dictionary<string, string>
{
{ "RedrivePolicy", $"{{\"deadLetterTargetArn\":\"{dlqUrl}\",\"maxReceiveCount\":\"{maxReceiveCount}\"}}" }
};
var setAttributesRequest = new SetQueueAttributesRequest { QueueUrl = sourceQueueUrl, Attributes = attributes };
await sqsClient.SetQueueAttributesAsync(setAttributesRequest);
}
}Retrieve Redrive Policy of Queue
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await PrintRedrivePolicyAsync(queueUrl, sqsClient);
}
static async Task PrintRedrivePolicyAsync(string queueUrl, AmazonSQSClient sqsClient)
{
var attributes = await sqsClient.GetQueueAttributesAsync(new GetQueueAttributesRequest
{
QueueUrl = queueUrl,
AttributeNames = new List<string> { "RedrivePolicy" }
});
if (attributes.Attributes.TryGetValue("RedrivePolicy", out var redrivePolicy))
Console.WriteLine($"RedrivePolicy: {redrivePolicy}");
}
}Delete a Queue
using Amazon.SQS;
using Amazon.SQS.Model;
static partial class Program
{
static async Task Main(string[] args)
{
AmazonSQSClient sqsClient = CreateAmazonSQSClient();
string standardQueueName = "standard-queue";
var queueUrl = await GetQueueUrlAsync(standardQueueName, sqsClient);
await DeleteQueueAsync(queueUrl, sqsClient);
}
static async Task DeleteQueueAsync(string queueUrl, AmazonSQSClient sqsClient)
{
var deleteQueueRequest = new DeleteQueueRequest { QueueUrl = queueUrl };
await sqsClient.DeleteQueueAsync(deleteQueueRequest);
}
}Best Practices for Using AWS SDK for .NET with Amazon SQS
When working with Amazon SQS (Simple Queue Service) using the AWS SDK for .NET, adhering to best practices ensures reliability, scalability, and performance in your application. Here’s a breakdown with examples:
1. Use Dependency Injection for the SQS Client
Integrate the IAmazonSQS interface into your application using dependency injection for testability and modularity.
Example:
services.AddAWSService<IAmazonSQS>();Usage in the application:
public class SqsService
{
private readonly IAmazonSQS _sqsClient;
public SqsService(IAmazonSQS sqsClient)
{
_sqsClient = sqsClient;
}
public async Task SendMessageAsync(string queueUrl, string messageBody)
{
var request = new SendMessageRequest
{
QueueUrl = queueUrl,
MessageBody = messageBody
};
await _sqsClient.SendMessageAsync(request);
}
}2. Avoid Hardcoding Queue URLs and Names
Use environment variables, AWS Secrets Manager, or AWS Systems Manager Parameter Store to store queue information.
Example:
var queueUrl = Environment.GetEnvironmentVariable("SQS_QUEUE_URL");3. Optimize Polling with Long Polling
Enable long polling to reduce the number of empty responses and lower API costs.
Example:
var receiveMessageRequest = new ReceiveMessageRequest
{
QueueUrl = queueUrl,
MaxNumberOfMessages = 10,
WaitTimeSeconds = 20 // Enables long polling
};
var response = await _sqsClient.ReceiveMessageAsync(receiveMessageRequest);4. Use Batch Operations Where Possible
Use batch APIs for sending or deleting multiple messages to improve performance and reduce costs.
Example:
var sendMessageBatchRequest = new SendMessageBatchRequest
{
QueueUrl = queueUrl,
Entries = new List<SendMessageBatchRequestEntry>
{
new SendMessageBatchRequestEntry("msg1", "Message Body 1"),
new SendMessageBatchRequestEntry("msg2", "Message Body 2")
}
};
var response = await _sqsClient.SendMessageBatchAsync(sendMessageBatchRequest);5. Implement Error Handling and Retry Logic
Handle exceptions like OverLimitException and implement retries with exponential backoff.
Example:
try
{
var response = await _sqsClient.SendMessageAsync(request);
}
catch (OverLimitException ex)
{
Console.WriteLine($"Error: {ex.Message}");
// Implement retry logic
}6. Ensure Message Idempotency
Use MessageDeduplicationId to ensure messages are not processed multiple times in FIFO queues.
Example:
var request = new SendMessageRequest
{
QueueUrl = queueUrl,
MessageBody = messageBody,
MessageDeduplicationId = Guid.NewGuid().ToString(), // Ensure uniqueness
MessageGroupId = "Group1" // Required for FIFO queues
};
await _sqsClient.SendMessageAsync(request);7. Delete Messages After Processing
Manually delete messages after successfully processing them to prevent reprocessing.
Example:
var deleteMessageRequest = new DeleteMessageRequest
{
QueueUrl = queueUrl,
ReceiptHandle = message.ReceiptHandle
};
await _sqsClient.DeleteMessageAsync(deleteMessageRequest);8. Use Dead Letter Queues (DLQ)
Configure DLQs to handle failed message processing. This ensures that problematic messages are not retried indefinitely.
Example Configuration in AWS Console:
- Set DLQ for the SQS queue.
- Specify maximum receives before a message is sent to the DLQ.
9. Monitor Queue Metrics
Use Amazon CloudWatch to monitor SQS metrics like NumberOfMessagesSent, ApproximateNumberOfMessagesDelayed, and NumberOfMessagesDeleted.
Example with AWS SDK:
var cloudWatchClient = new AmazonCloudWatchClient();
var metricRequest = new GetMetricDataRequest
{
MetricDataQueries = new List<MetricDataQuery>
{
new MetricDataQuery
{
MetricStat = new MetricStat
{
Metric = new Metric
{
Namespace = "AWS/SQS",
MetricName = "NumberOfMessagesSent",
Dimensions = new List<Dimension>
{
new Dimension { Name = "QueueName", Value = "my-queue" }
}
},
Period = 300,
Stat = "Sum"
},
Id = "messagesSent"
}
},
StartTimeUtc = DateTime.UtcNow.AddMinutes(-5),
EndTimeUtc = DateTime.UtcNow
};
var response = await cloudWatchClient.GetMetricDataAsync(metricRequest);10. Encrypt Sensitive Data
Enable server-side encryption (SSE) for queues or encrypt message bodies in your application before sending.
Example:
var request = new SendMessageRequest
{
QueueUrl = queueUrl,
MessageBody = EncryptMessage(messageBody) // Custom encryption method
};As a summary, Messaging enables asynchronous communication between software components, promoting scalability and fault tolerance in distributed systems. Amazon SQS, AWS’s fully managed messaging service, supports decoupling and scaling of microservices via Standard and FIFO queues. Standard Queues handle high throughput with at-least-once delivery, suitable for tasks like logging, while FIFO Queues ensure strict ordering and exactly-once delivery, ideal for financial transactions. Key features include configurable retention periods, visibility timeouts, and dead-letter queues for error handling. SQS also supports encryption, access control, and tagging for security and organization, making it a versatile tool for diverse application needs.
💡 Please let me know in the comments if you agree with me or if I missed something; if you don’t agree, please let me know your opinion.
Thank you for your time 😊
