GPU Cost Optimization for ML

- YouTube's recommendation system employs a two-stage process to handle its massive scale, first using deep learning models for candidate generation from millions of videos, and then a separate ranking model to select the top recommendations for each user. This ranking model balances objectives like predicted watch time and user satisfaction, which is measured through surveys and metrics like rewatch rates. - Netflix is developing a centralized foundation model for recommendations to move away from maintaining numerous specialized models. This new architecture aims to learn from a user's entire interaction history, overcoming the limitations of previous models that were restricted to shorter timeframes due to latency and cost. - Spotify has leveraged Google Cloud's auto-scaling features to manage the computational resources for its data processing, which has led to significant cost reductions. The company developed an internal tool called "Cost Insights" to provide engineers with visibility into cloud spending, encouraging optimizations that improve performance and reliability. - Pinterest utilizes Graph Convolutional Networks (GCNs) in its PinSage framework to power its recommendation engine. This approach allows the system to analyze the relationships between billions of pins and boards, helping to differentiate between pins that are visually similar but semantically different. - Meta has developed a "Self-Taught Evaluator" that uses a "chain of thought" process to assess the responses of other AI models, reducing the need for human feedback and potentially lowering costs. Additionally, Meta's "Layer Skip" technique optimizes large language models by selectively executing layers, which improves energy efficiency and reduces computational expenses. - Google Research is focused on enhancing ML efficiency through the development of more efficient model architectures and training methods. One area of their research is "subset selection," which aims to identify the most critical features to retain while pruning redundant ones within complex models. - MLOps best practices emphasize the importance of resource utilization monitoring to control the costs associated with experiments and model deployment. It is estimated that around 85% of machine learning models do not make it into production, often due to a lack of automation and robust pipelines. - Companies are increasingly adopting model compression techniques like pruning and quantization to reduce the computational and memory requirements of their models. Pruning involves removing less important neurons to decrease model size, while quantization lowers memory usage by using lower-precision numerical formats for model weights.