Etsy Details Modular Recommendation Serving Platform

- Etsy's platform utilizes a two-pass architecture, common in large-scale systems, consisting of candidate selection to narrow down from over 100 million items, followed by a more computationally intensive ranking pass to personalize the final recommendations. - This modular approach contrasts with Netflix's recent shift towards a single, unified foundation model for personalization, which aims to centralize learning from user interaction histories and reduce the maintenance cost of having multiple specialized models. - While Etsy's system is designed for real-time personalization by incorporating session data, Spotify's architecture explicitly separates its low-latency, high-availability personalization pipelines from its experimentation systems to ensure that testing new models doesn't risk production stability. - Pinterest's recommendation engine, by contrast, is heavily optimized for visual discovery and employs a graph convolutional network called PinSage, which was trained on 18 terabytes of data, to learn embeddings and connections between billions of visually and thematically related "Pins". - The move to a centralized platform allows Etsy's product teams to chain together different machine learning "building blocks," such as retrieval models and ranking models, to create tailored user experiences like recommending items similar to those recently viewed. - Other platforms are increasingly integrating generative AI; for instance, Pinterest has developed "PinRec," a transformer-based generative retrieval system, to move beyond traditional two-tower models and directly generate recommendation candidates. - Etsy's migration to the cloud, specifically Google Cloud Platform, was a key enabler for advancing its ML capabilities, allowing the company to leverage managed services like Vertex AI for model training and reduce the time from idea to live experiment by an estimated 50%.