Example of Batch Processing Job for Celery Usage

[zczarnecki]

Description

Description:
Implement a batch processing system using Celery to handle large file uploads and processing in the background.

Problem Statement:

1. Users frequently upload large files, which can cause slowdowns and performance issues if processed synchronously.
2. Without an asynchronous processing system, users may experience UI freezes or timeouts.
3. The system should be able to efficiently handle large file processing while providing real-time status updates to users.

Use Cases:

1. A user uploads a CSV file containing bulk data, and the system should process it asynchronously without affecting performance.
2. Users should be able to check the status of their file processing task.
3. Completed tasks should allow users to view/download processed data.

Describe the solution you'd like

Describe the Solution:

1. Use Celery workers to offload processing of large files asynchronously.
2. Implement a status tracking mechanism where users can query the state of their processing job.
3. Once processing is complete, provide an option to download or view the processed output.

Acceptance Criteria:

1. Given a user uploads a large file, when the upload is complete, then the file should be processed asynchronously in the background.
2. Given an ongoing file processing task, when a user queries the task status, then the system should return the current state.
3. Given a completed processing task, when the user checks the results, then they should be able to download/view the processed data.

TDD Test Cases:

1. Test if the Celery worker correctly picks up the task.
2. Test for handling large file uploads efficiently.
3. Test failure scenarios and retries.

Describe alternatives you've considered

No response

Additional context

No response

Validations

• Follow our Code of Conduct.
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.