AWS HLS pipeline share

A video file is too large to stream raw, so engineers first turn it into smaller chunks and multiple quality levels. Apple’s HTTP Live Streaming, or HLS, serves those chunks over ordinary web servers and content delivery networks. (developer.apple.com) Engineer Rohit recently shared one way to automate that job on Amazon Web Services: store uploads in Amazon Simple Storage Service, queue work in Amazon Simple Queue Service, trigger logic with AWS Lambda, and run the heavy encoding inside Amazon Elastic Container Service on AWS Fargate. (x.com) (docs.aws.amazon.com) Amazon S3 can emit an event when a new object lands in a bucket, and Amazon SQS can hold that work until a consumer is ready. AWS describes SQS as a managed queue for decoupling software components, which lets upload spikes pile up in the queue instead of crashing the encoder. (docs.aws.amazon.com 1) (docs.aws.amazon.com 2) Lambda fits the orchestration step because it reacts to events quickly, but AWS caps a single Lambda invocation at 15 minutes. Video transcoding often runs longer, so Rohit’s design hands the actual conversion to container tasks on Fargate instead of trying to finish the whole job inside a function. (docs.aws.amazon.com 1) (docs.aws.amazon.com 2) Fargate runs containers without requiring the customer to provision or scale virtual machine clusters first. In practice, that means a media team can package FFmpeg or another encoder in a container image, set CPU and memory, and launch tasks only when the queue fills. (docs.aws.amazon.com) AWS already sells a managed transcoder, AWS Elemental MediaConvert, for offline video formatting and compression, including HLS output. Rohit’s post shows a different path: teams can assemble lower-level AWS building blocks themselves when they want more control over the workflow, container image, or scaling logic. (docs.aws.amazon.com) (x.com) AWS has published adjacent sample architectures before, including an archived example that used S3 notifications, SQS, Lambda, and Fargate to autoscale batch processing from 0 to as many as 100 concurrent Fargate tasks. That sample was built for image preprocessing, but the same queue-and-container pattern maps neatly onto video pipelines that need burst capacity. (github.com) The appeal of HLS is compatibility as much as scale. Apple says HLS sends audio and video over HTTP to iPhone, iPad, Apple TV, Mac, and desktop browsers, using the same web infrastructure companies already use to deliver other files. (developer.apple.com) That makes Rohit’s diagram less a product launch than a field note from the way cloud video systems are often built now: storage catches the upload, a queue absorbs the burst, lightweight code dispatches the job, and containers do the slow work. (x.com) (docs.aws.amazon.com)