Back to challenges
Challenge

Handling large file uploads

The application needed to handle potentially large files without creating unnecessary impact on the server, enabling asynchronous validation and staged upload strategies.

JavaSpring BootAWS S3Redis

Approach 1: Direct Upload with Async Validation

The client uploads directly to object storage via a signed URL generated by the backend. Once the user confirms the upload, the backend enqueues validation jobs that run asynchronously. Validations include existence verification, integrity checks, antivirus scanning, format verification, and other business rules. The validation status is exposed via an API that the frontend can poll or subscribe to via WebSocket, transitioning through states: pending → validating → valid/invalid → completed. This approach is suitable when validations can run in the background and the user can continue working.

Flow

  1. Backend issues signed PUT URL for target bucket
  2. Client uploads file directly to storage
  3. User confirms upload via API call
  4. Backend enqueues validation jobs
  5. Validation workers process checks in parallel
  6. Status updated in real-time store
  7. Frontend polls/subscribes for completion
Approach 1: Direct Upload + Async Validation Client signed PUT URL Permanent Bucket (final destination) 1. upload file 2. confirm Backend 3. enqueue validations Validation Pipeline (async) Existence Integrity Antivirus Format Business Rules Quotas & Limits Status API

Approach 2: Staged Upload with Synchronous Validation

For operations where the file is required to complete a business transaction (e.g., creating an item that requires an image), a staged approach with synchronous validation is used.

Phase 1 - Pre-upload: The client uploads the file to a temporary bucket using a signed URL. The object path follows the pattern /userid/operationid/, where userid ensures the user can only access their own files, and operationid distinguishes between multiple upload attempts for the same operation.

Phase 2 - Confirm & Validate: The client sends the business operation request (e.g., POST /items with item data) including the operationid instead of the file itself. The backend retrieves the file from the temp bucket using the path, runs all validations synchronously (existence, integrity, antivirus, format, business rules), and only if all pass, moves the file to the permanent bucket and completes the business operation. If any validation fails, the operation is rejected and the temp file remains for retry or cleanup via TTL.

This approach guarantees that the business operation never completes with an invalid file, while keeping large file transfers out of the application server.

Flow

  1. Backend issues signed PUT URL for temp bucket at /userid/operationid/
  2. Client uploads file directly to temp location
  3. Client sends business operation request with operationid (no file)
  4. Backend retrieves file from temp bucket using path
  5. Validation pipeline runs synchronously
  6. If all validations pass: move file to permanent bucket, complete operation
  7. If any validation fails: reject operation, return error, temp file expires via TTL
  8. On success: temp file cleaned up, permanent file linked to created entity
Approach 2: Staged Upload + Sync Validation Client signed PUT URL Temp Bucket /userid/operationid/ (short TTL · user-scoped · attempt-scoped) 1. pre-upload file 2. POST { operationid } Backend 3. GET /userid/opid/ Validation Pipeline (sync - blocks) Existence Integrity Antivirus Format Business Rules Quotas & Limits Move to Permanent Bucket ✓ pass → move ✗ fail → reject Operation Complete

Context

The application needed to handle potentially large files without creating unnecessary impact on the server infrastructure, while supporting multiple validation workflows.

Problem

Direct file uploads through the backend were consuming significant server resources and creating bottlenecks, especially for large files. Additionally, the system required different validation workflows depending on the use case: some files needed asynchronous validation after upload confirmation, while others required synchronous validation as part of a business operation (e.g., creating an entity that requires an image).

Solution

Common infrastructure:

  • Signed URL generation for direct-to-storage uploads
  • Temporary bucket with TTL-based cleanup
  • Path-based ownership enforcement (/userid/operationid/)
  • Validation pipeline (reusable across both approaches)

Validation pipeline (generic, reusable):

  • Existence: verify object exists and is accessible
  • Integrity: checksum/hash verification
  • Security: malware/virus scanning
  • Format: structure and MIME type validation
  • Business rules: size limits, type restrictions, quotas

Result

The dual-strategy approach allowed handling files of any size efficiently by offloading transfer to object storage. The backend only manages metadata, coordination, and validation orchestration. The synchronous staged approach guarantees data integrity for critical business operations while the asynchronous approach provides better UX for non-blocking workflows. The path-based ownership model (/userid/operationid/) simplified access control without additional authorization checks.