Understand this story
Amazon Science and Booking.com combined knowledge distillation with Medusa technology to optimize Booking.com's travel large language model. The approach compresses a 70B-parameter Llama-3.1 teacher model into a 1.1B TinyLlama student model, maintaining the teacher model's quality while increasing inference speed by 10 to 20 times and reducing costs to less than one-third. The solution has been applied to three features—AI Trip Planner, Smart Filters, and accommodation description generation—and won the Best Paper Award in the Industry Track at ACL.
AI-generated from public reports
Latest development · Report
Amazon Science and Booking.com use knowledge distillation plus Medusa to speed up LLM inference by up to 20xDevelopments
1 confirmed developments
Amazon Science and Booking.com use KD plus Medusa to accelerate travel LLMs
Source:Amazon Science
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 30
Amazon ScienceFirst-party
Amazon Science and Booking.com use knowledge distillation plus Medusa to speed up LLM inference by up to 20x
Amazon Science and Booking.com combined knowledge distillation with Medusa to compress a 70B-parameter Llama-3.1 teacher model into a 1.1B TinyLlama student model, increasing inference speed by 10 to 20 times and reducing cost to less than one-third while maintaining the teacher model's quality. The approach has been used in three features—AI Trip Planner, Smart Filters, and accommodation description generation—and won the Best Paper Award in the Industry Track at ACL.