AI News

TypeSafe AI shares Jev's best practices for speculative questions

TypeSafe AIUpdated Nov 14 06:32

AI summary

TypeSafe AI shares Allie the Icon's "speculative questions" pattern regarding Jev: in most cases, batching questions together into a single call is faster and more cost-effective, even when you are not yet sure whether you will need those answers; this is because a single call charges the state fee only once per token, and questions within the same call do not affect each other's accuracy.

Quoted post

Allie the Icon

@allietheicon

One of the most important Jev patterns to learn is what we sometimes call "speculative questions". In the majority of cases, it's faster and even often cheaper to batch questions together in to a single call, *even when you don't yet know if you'll need the answers!* Why? Many reasons: Speed! Oftentimes you can accomplish the entire task in a single call, instead of multiple sequential calls in different logic paths. Cost! You only pay for tokens in state once per call, so batching questions also avoids paying for the state multiple times. When your state is large, batching questions often saves cost compared to sequential calls that duplicate state. Questions never affect the output of other questions in the same call, so accuracy will not suffer.

View original

One of the most important Jev patterns to learn is what we sometimes call "speculative questions". In the majority of cases, it's faster and even often cheaper to batch questions together in to a single call, *even when you don't yet know if you'll need the answers!* Why? Many reasons: Speed! Oftentimes you can accomplish the entire task in a single call, instead of multiple sequential calls in different logic paths. Cost! You only pay for tokens in state once per call, so batching questions also avoids paying for the state multiple times. When your state is large, batching questions often saves cost compared to sequential calls that duplicate state. Questions never affect the output of other questions in the same call, so accuracy will not suffer.

View original post

Body

can’t get answers to questions you don’t ask! Jev first, figure it out later.

fresh best practices drop from @allietheicon

Source

TypeSafe AI