NAll news

All news

Wednesday4 items

Tuesday3 items

Monday1 items

Thursday1 items

Tuesday1 items

Monday1 items

Saturday2 items

Friday3 items

Thursday3 items

  1. realmadhuguru
    AI score64

    English translation is being prepared.

    Quoted postMadhu Guru · @realmadhuguru

    How to build great evals — Part 9 The Eval Roadmap Problem Most evals fail because teams treat them as static artifacts while their users expectations and behaviors have evolved. Your evals need a roadmap that evolves with your product and actual usage patterns. Take a financial research agent. Here is how use cases will evolve over time. Early user: Summarize this 5-page earnings report. 3 weeks later: review the last 5 earnings reports and explain their growth story. 2 months later: Here are 15 filings, earnings transcripts and research reports. Build an investment thesis. Eventually: Monitor my stock portfolio and tell send me alerts when something materially changes my thesis. Each stage requires different capabilities and hence different evals. Your evals should change with usage patterns: short-context -> long-context single-turn QA -> multi-turn passage citations -> doc and line citations Simple QA -> complex synthesis reactive chat -> proactive agent If your evals are stuck in week 1 while users are in week 3, it will show in your product and churn metrics. Here’s a practical way to build the roadmap: 1/ Map the dimensions along which usage will evolve (eg # turns, document size, tool use, autonomy, journey coverage). 2/ Prioritize the use cases and dimensions that matter most for your product. 3/ Talk to your users, mine production traces to look for shifts. 4/ Build P0 evals for the next stage of usage. 5/ Run the evals, find failure modes (remember the taxonomy from Part 3), and hill-climb (from part 6). The goal is to stay ahead of usage patterns. PM 101. Drop your questions and I will address in future posts. Share this with your teammates who think your evals are perfect.

    #Tutorials / practice#Agents#English text is being prepared.

  2. Chips and CheeseAnalysis
    AI score52

    Benchmarking Windows 11 Prism Binary Translation: Performance Loss of x86-64 Programs on Arm

    Chips and Cheese author Chester Lam used Geekbench 7 to benchmark Windows 11's Prism binary translation, finding that x86-64 programs execute roughly twice as many aarch64 instructions after translation as they would natively, with performance losses of several generations across all cores. Testing was conducted on a Snapdragon X2 Elite Extreme X2E-96-100 laptop and on Neoverse N1 and N2 instances on Azure; even after bearing the translation losses, the Snapdragon's E-Cores and P-Cores still respectively outperformed the Neoverse N1 and N2. The author also examined the translated code in C:\Windows\XtaCache, noting that Prism performs unnecessary register spills and splits 256-bit AVX operations into multiple 128-bit NEON instructions.

    #Evaluations / benchmarks#English text is being prepared.#English text is being prepared.

Wednesday4 items