Understand this story
Property Finder selected Llama 4 Maverick 17B Instruct on Amazon Bedrock. An article published by AWS Machine Learning revealed that Property Finder found 96.8% of its evaluation labels came directly from predictions of the existing vision-language model rather than manual verification, causing any replacement model to be structurally disadvantaged in comparisons. The team re-annotated 356 UAE images by hand and switched to multi-label classification, then iterated and optimized prompts from v0 to v27 to select the model on Amazon Bedrock, ultimately achieving improved accuracy and an 86% reduction in cost.
AI-generated from public reports
Latest development · Report
Fixing evaluation bias to select a better model on Amazon BedrockDevelopments
1 confirmed developments
- Source:AWS Machine Learning
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 28
AWS Machine LearningFirst-party
Fixing evaluation bias to select a better model on Amazon Bedrock
Property Finder found that 96.8% of its evaluation labels came directly from predictions of an existing vision-language model rather than human verification, causing any alternative model to be structurally disadvantaged in comparisons. The team re-annotated 356 UAE images by hand and switched to multi-label classification, then iterated and optimized prompts from v0 to v27 to select a model on Amazon Bedrock, ultimately achieving higher accuracy and an 86% cost reduction.