Integrate large payload support for SQS #2116
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hello, Do you have any roadmap or due date to integrate this feature to support large message which more than 256kb(in SQS Message Broker) |
This adds support for handling large payloads in SQS. The 'sqs_extended_client' is imported and utilized for fetching file from S3 as payload when necessary. As Kombu asynchronously fetches new messages from the queue, not using the standard boto3 APIs, we have to manually fetch the s3 file, rather than rely on the sqs_extended_client to perform that action Relates to: celery#279
The try/except block was triggering when sqs_extended_client isn't installed, which results in boto being overwritten with None
Amwam
force-pushed
the
add-sqs-large-payload-support
branch
from
ca4eb4b to
2d1f691
Compare
Amwam
force-pushed
the
add-sqs-large-payload-support
branch
from
f9a0f71 to
2fa40f4
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2116 +/- ##
==========================================
- Coverage 81.50% 79.34% -2.17%
==========================================
Files 77 77
Lines 9524 9555 +31
Branches 1152 1155 +3
==========================================
- Hits 7763 7581 -182
- Misses 1569 1806 +237
+ Partials 192 168 -24 ☔ View full report in Codecov by Sentry. |
|
Coverage needs to be improved |
Introduce two tests to verify S3 client creation behavior: one for insecure connections and another for custom endpoint usage. This ensures proper configuration of boto3 client initialization in these scenarios.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds support for handling large payloads in SQS. The 'sqs_extended_client' is imported and utilized for fetching file from S3 as payload when necessary.
As Kombu asynchronously fetches new messages from the queue, not using the standard boto3 APIs, we have to manually fetch the s3 file, rather than rely on the sqs_extended_client to perform that action
Relates to: #279