TypeSafe AI discusses Jev's state billing and question strategy
TypeSafe AI explains how to use Jev: the `state` portion of a request is billed only once, no matter how many questions you ask based on it; therefore it is usually more cost-effective to first ask all the questions you might need, then use code to filter and organize the results. The cited Allie the Icon points out that Jev bills input tokens as (state + (sum of all questions)); asking questions in parallel against the same state charges the state fee only once, but within a single call two questions still cost more than one; when cost-sensitive, you can run low-frequency follow-up questions only when actually needed, while when speed is the priority, batch questioning is usually better.
View original postThe answer has nuance! The way Jev charges for input tokens is: (state+(sum of all questions)) The important bit here is that you only get charged once for your state tokens no matter how many questions you run in parallel. Within a single call, two questions do cost more than one. But if you ask those two questions as separate requests with the same state, you pay more overall, because you pay for the state again in the second request. In cases where you are more cost-sensitive than speed-sensitive, it makes sense to do analysis on how often the follow-up questions are used in your use-case. If some follow-ups are needed very rarely, then you can save on cost by running them only when actually needed. But in cases where speed is more important than cost, then batching pretty much always wins (although there's some nuance there as well, such as a single super-huge follow-up question slowing down the call, so shunting it to its own call could make sense, depending on the frequency that that answer is needed!)
View original
The answer has nuance! The way Jev charges for input tokens is: (state+(sum of all questions)) The important bit here is that you only get charged once for your state tokens no matter how many questions you run in parallel. Within a single call, two questions do cost more than one. But if you ask those two questions as separate requests with the same state, you pay more overall, because you pay for the state again in the second request. In cases where you are more cost-sensitive than speed-sensitive, it makes sense to do analysis on how often the follow-up questions are used in your use-case. If some follow-ups are needed very rarely, then you can save on cost by running them only when actually needed. But in cases where speed is more important than cost, then batching pretty much always wins (although there's some nuance there as well, such as a single super-huge follow-up question slowing down the call, so shunting it to its own call could make sense, depending on the frequency that that answer is needed!)
Body
how to Jev
• you pay for the `state` part of your request once, no matter how many questions you ask about it • so it's usually cheaper to ask every question you might want answered, even before you know if you need the answer • then use code to sort it out
take in all the information you might need, and filter down to what's important! reliable automation means covering all your bases