S3

Part of the MediaBridge series. Overview MediaBridge has two deployable components and two AWS-side components: Backend: Node.js process running as a systemd service behind nginx Frontend: Static React build deployed to S3, served via CloudFront Thumbnail Lambda: AWS Lambda handling S3 object events Archive restore pipeline: CloudTrail, EventBridge, and two Lambda functions Prerequisites A Linux server with Node.js 18+, PostgreSQL, and nginx An AWS account with at least one S3 bucket and IAM credentials for it A domain with DNS pointing to your server (for TLS) Backend Clone and install: ...

Part of the MediaBridge series. Why Thumbnails in Lambda Thumbnail generation on the backend server would mean every file upload triggers a download from S3, a resize operation, and an upload back to S3 - all inline with the upload flow. That adds latency to every upload, burns bandwidth on the server, and blocks the upload response until the thumbnail is ready. Lambda is a better fit. S3 fires an event for every object creation. The Lambda processes it asynchronously, after the upload has already completed and the user has their presigned URL. The backend is not involved. ...

Part of the MediaBridge series. The Problem S3 Glacier and Glacier Deep Archive are cheap cold storage tiers. Files in Glacier take 3-5 hours to restore. Files in Deep Archive take up to 12 hours. When a user tries to access an archived file, S3 returns a 403 (or the object is simply missing from the listing depending on how the bucket is configured). The user gets no feedback about what happened or when the file will be available. ...

Part of the MediaBridge series. Per-Bucket Search vs Global Search The single-bucket search described in the previous post runs one WebSocket and searches one bucket from a given prefix. Global search is different: it searches every bucket the user has access to simultaneously, merging results from all of them into a single view. A user assigned to 10 buckets gets 10 WebSocket connections opened in parallel. Results from all 10 stream in at the same time. Each bucket runs its own DFS traversal independently. ...

Part of the MediaBridge series. Why Not a Normal HTTP Search S3 has no search API. To find a file, you have to list folders. A bucket with a deep directory tree could require hundreds of ListObjectsV2 calls to walk completely, each taking 100-500ms. A synchronous HTTP endpoint would either time out or make the user wait minutes for a response. The alternative is to stream results as they arrive. Start the traversal, send each matching file to the browser the moment it is found, let the user see results accumulating in real time. WebSocket is the natural transport: a single persistent connection that the server writes to continuously. ...