从 NER 到智能体：自动化提示词工程 MIPRO 能否扩展到复杂任务？

Summary

Automated prompt engineering with MIPRO can significantly improve performance in simpler tasks, but the benefits start to diminish as task complexity grows.

Larger models seem to benefit more from MIPRO optimization in complex settings. We hypothesize this difference is due to a better ability to handle long multi-turn demonstrations; Unsurprisingly, the quality of the feedback materially affects the quality of the MIPRO optimization process. But at the same time, we still see meaningful improvements from noisy feedback, including AI-generated feedback

Crafting good system prompts is one of the trickiest parts of working with language models — especially when you’re dealing with complex, multi-step tasks. Doing it manually doesn’t scale, and the results are often hit-or-miss. That’s where automated prompt engineering methods like MIPRO come in: they promise to optimize system prompts while handling the trial and error for you.

In this post, we put MIPRO to the test across a range of tasks — from simple named entity recognition (CoNLL++), to multi-hop retrieval (HoVer), to text-based game navigation (BabyAI), to customer support with agentic tool use (τ-bench).

We wanted to know… how well does automated prompt engineering hold up as task complexity increases?

What is MIPRO?

MIPRO (Multi-prompt Instruction PRoposal Optimizer) is a method for automatically improving system instructions and few-shot demonstrations in LLM applications — including ones with multiple LLM functions or calls.

MIPRO can optimize prompts across an entire LLM pipeline without needing fine-grained labels or gradients. Instead, it uses a Bayesian optimizer to figure out which instructions and demonstrations actually improve end-to-end performance. By combining application-aware prompt proposals and stochastic mini-batch evaluations, MIPRO can improve downstream task performance compared to traditional prompt engineering approaches.

MIPROv2 is the particular variant of this algorithm recommended by the authors of the DSPy library, but you don’t need DSPy to try it out. We’ve reimplemented a standalone version (around 200 lines of code).

Optimization Search Space

MIPRO searches over two key parts of a system prompt:

Instructions: what the system message actually tells the model to do; Demonstrations: a number of example inputs/outputs (i.e. few-shot examples)

To get instruction candidates, we prompt an LLM to rewrite the initial system prompt in ten different ways, aiming for diverse and semantically equivalent variants. For this post, we used OpenAI’s o1 model.

To get a set of candidate demonstrations, we:

Run the app with an initial prompt to collect behavior traces (observations and actions); Filter those traces based on task feedback (e.g. did the task succeed?); Randomly sample multiple sets of demonstrations from the good traces

Optimization Procedure

The optimization loop is pretty straightforward:

Pick a candidate instruction and set of demonstrations (randomly, but guided by a Bayesian optimizer); Format a system prompt using those choices; Run multiple episodes using the new prompt and collect feedback (e.g. accuracy, success rate); Score the batch based on feedback; Use the score to update the optimizer, improving its guesses for the next round

After a fixed number of rounds, MIPRO returns the best-performing instructions and demonstration set for each function. Under the hood, it uses a Tree-structured Parzen Estimator — the same optimization approach used in the DSPy implementation.

Experiments

We explore two types of interactive environments to evaluate the MIPRO optimizer: single-turn and multi-turn environments. Either type of environment simulates specific tasks, offering distinct interaction models and feedback mechanisms for assessing the performance of MIPRO-optimized system prompts.

For each environment, we compare the performance of the MIPRO optimized system prompt to two baselines: a Zero-shot baseline using the example system prompt without demonstrations and a baseline with randomly sampled demonstrations (Random Demos).

Named Entity Recognition (CoNLL++)

Named entity recognition (NER) is the task of identifying and classifying entities (e.g. people, organizations, locations) in free-form text. We use the CoNLL++ dataset to set up a simple single-turn, single-function environment for automated prompt engineering. Environment feedback for this task includes Jaccard similarity and exact match scores computed against ground-truth references.

We observe that MIPRO optimization results in significant 140.1% and 43.4% relative increases over the Zero-shot and Random Demos baselines, respectively.

Multi-hop Retrieval (HoVer)

HoVer is a two-function environment providing a claim (i.e. a statement to investigate), a search engine, and feedback on how well an LLM agent can find information to verify the claim based on the titles of Wikipedia articles it retrieves.

In contrast to NER, the agent must implement two functions (policies) to complete the task. The first function (Generate Query) generates a search query given the claim and an evolving set of notes. The second function (Append Notes) generates summary notes for the Generate Query function and extracts the most relevant titles from the search results of the last query.

The environment feedback is a top-5 recall score of the set of titles proposed by the Append Notes function compared to a ground-truth set of titles.

We do not observe a significant difference in the performance of MIPRO when comparing to either of the baselines.

Text-Based Game Navigation (BabyAI)

BabyAI is a multi-turn environment. In a multi-turn environment, the state changes after each agent-environment interaction. For example, the user can ask new questions after the LLM agent responds in a conversation, or a tool call can provide previously unknown information as a conversation proceeds. Moreover, feedback (task success) may be delayed until the end of a sequence of such interactions (an episode).

At each turn in BabyAI, the environment provides a text description of what is currently visible to the agent and a history of environment-agent interactions as observation. Tasks are grouped into categories like GoTo and PickUp. The agent must then act by providing a valid move (e.g. go forward, turn left, or pickup). The agent receives feedback after each interaction as a value indicating whether the agent has completed the task. An episode terminates when the agent completes the task, or the episode times out by reaching the maximum number of allowed turns.

The BabyAI GoTo environment tasks an agent to navigate to a specified object.

We observe that MIPRO optimization results in significant 55.8% and 17.0% relative increases over the Zero-shot and Random Demos baselines, respectively.

The BabyAI PickUp environment tasks an agent to navigate to a specified object and pick it up.

We observe that MIPRO optimization results in significant 35.9% and 42.9% relative increases over the Zero-shot and Random Demos baselines, respectively.

The BabyAI PickUpSeqGoTo environment tasks an agent to (1) navigate to an object, (2) pick it up, and (3) go to another object.

We observe that MIPRO optimization results in a 68.1% increase over the Random Demos baseline but do not observe a statistically significant improvement over the Zero-shot baseline.

At each turn in τ-bench Retail, the environment provides either a customer response or the result of a tool call and the conversation history as observation. The agent must act by responding to the customer or making a tool call. Feedback is given as a boolean indicating successful task completion. An episode terminates when the agent completes the user request, transfers the customer to a human, or the episode times out.

We do not observe a significant difference in performance between MIPRO and the baselines using GPT-4o Mini. Moving to the larger GPT-4o model, we see evidence that MIPRO optimization leads to more effective system prompts; however, we need more experiments to verify this observation.

Discussion

Task Complexity

Our above results suggest that the performance improvement of MIPRO-optimized system prompts diminishes as task complexity increases.

We can see that the system prompts for more complex tasks (HoVer, BabyAI GoToSeqPickUp, τ-bench Retail) are much longer than for less complex tasks (NER, BabyAI GoTo, BabyAI PickUp) by comparing the example optimized system prompts above. This is reflected in the input token count histograms for the MIPRO optimized functions below. The histogram bars for each environment are stacked. The proportion for each environment at a given bin corresponds to the height of its colored segment.

The increase in prompt length is primarily due to the demonstrations: either because the average number of turns in a successful episode increases (BabyAI PickUpSeqGoTo) or because environment observations and agent actions are more verbose (HoVer and τ-bench Retail).

This observation motivates our experimentation with a larger model (GPT-4o instead of GPT-4o Mini) with larger context capacity for the τ-bench Retail environment. The results suggests that larger models benefit more from MIPRO optimization in complex settings. Further experimentation is required to verify this claim.

Filtering Demonstrations with Feedback

Developers of real-world applications may not have access to ground-truth demonstrations and instead need to rely on semi-synthetic demonstrations using model predictions instead of ground-truth labels.

Unsurprisingly, the quality of the feedback materially affects the quality of the MIPRO optimization process. But at the same time, we still see meaningful improvements from noisy feedback, including AI-generated feedback.

We run our experiments assuming access to high-quality feedback metrics, like Jaccard similarity or exact match scores for NER and task success in BabyAI or τ-bench. We use these metrics to filter the semi-synthetic demonstrations from which we sample candidate demonstration sets. In the NER task, we observe that the quality of the semi-synthetic examples can significantly affect the quality of solutions found by the MIPRO algorithm.

This chart shows that filtering the set of demonstrations based on the Exact Match indicator or a stringent value of the Jaccard Similarity score leads to the best overall performance. However, including lower-quality demonstrations by lowering the Jaccard Similarity threshold to 0.5 significantly reduces the optimized system prompt performance.

LLM as a Judge

Assessing the quality of semi-synthetic demonstrations is hard in general. Using an LLM as a Judge is an increasingly popular paradigm to address this challenge. We quickly experiment with a general-purpose LLM Judge to assess the quality of semi-synthetic demonstrations.

Expanding the following block shows how the LLM Judge receives a specific demonstration (with the system instruction).

We observe that the results are competitive with using ground-truth metrics when using GPT-4o Mini as the judge model and thresholding its score prediction at 0.9. However, it is unclear whether these results reliably generalize as we did not observe promising initial results in the multi-turn setting. Moreover, in the following chart reveals a counterintuitive result: more powerful judge models (GPT-4o or o3-mini) result in significantly worse NER performance.

We are asking a lot of the LLM Judge by prompting it to score a demonstration without reference to other demonstrations. Prompting it to rank sets of demonstrations or adding proxy performance metrics to the context are more principled approaches. We leave these exploring these ideas in these settings to future work.

Recent work from Anthropic demonstrates that thoughtful, creative modifications to system instructions can yield substantial performance gains for complex, multi-turn τ-bench environments. Our results align with these findings, though they highlight differences between automated and (presumably) human-crafted system prompt optimization.

While automated prompt engineering via MIPRO provides clear performance benefits in simpler environments (e.g. NER and BabyAI GoTo tasks), it did not achieve comparable improvements in the more complex setting (e.g. τ-bench Retail).

Anthropic’s optimized system prompt leveraged sophisticated instructions that creatively utilized available agent tools, resulting in substantial performance improvements. In contrast, even though our τ-bench Retail agent had explicit access to the “think” tool (a standard capability provided by τ-bench), the instructions generated by OpenAI’s o1 model lacked the creativity observed in Anthropic’s system prompt. This observation suggests two insights: First, significant room for improvement remains in how automated systems generate diverse and creatively structured prompts. Second, explicitly informing the instruction-generation model of available agent tools and their capabilities may enhance the quality and innovativeness of automatically generated instructions.

An interesting experiment to explore this hypothesis would see the candidate instruction-generation model receive a list and descriptions of all available tools. By constraining optimization solely to instructions (excluding demonstrations) and leveraging richer context about the LLM agent’s affordances, we could investigate if the automated system can independently discover creative, high-performance prompts comparable to those identified by Anthropic. Such an experiment could help clarify whether automated approaches could match—or potentially exceed—the creativity and effectiveness of expert-crafted instructions.

Conclusion

Our results show that MIPRO can significantly improve performance in simpler tasks — like NER or single-goal navigation. But as task complexity grows, the benefits start to diminish. In some cases, scaling up the model (e.g. GPT-4o vs GPT-4o Mini) helps, but is unclear if scaling-up is all we need (or even desired).

We also found that demonstration quality and prompt creativity matter a lot. More complex tasks demand better filtering, richer context, and perhaps more structured ways of informing models about available tools and task affordances. There’s clear potential for automated prompting to match or even surpass human-crafted instructions — but there is a lot of work left to be done.

Code to replicate these experiments is available through LLM Gym.

Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs Krista Opsahl-Ong, Michael J. Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. (2024). arXiv preprint; Algorithms for Hyper-parameter Optimization James Bergstra, Rémi Bardenet, and Yoshua Bengio, and Balázs Kégl. (2011). In Advances in Neural Information Processing Systems (NeurIPS), vol. 24. Conference Paper; CrossWeigh: Training Named Entity Tagger from Imperfect Annotations Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. (2019). In Proceedings of EMNLP-IJCNLP 2019, vol. 1, pp. 5157–5166. Conference Paper; HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. (2020). In Proceedings of EMNLP 2020. Conference Paper; BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. (2019). In International Conference on Learning Representations (ICLR). Conference Paper; Balrog: Benchmarking Agentic LLM and VLM Reasoning on Games Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, et al. (2024). arXiv preprint; τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. (2024). arXiv preprint; Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. (2023). In Advances in Neural Information Processing Systems (NeurIPS), vol. 36. Conference Paper; Optuna: A Next-Generation Hyperparameter Optimization Framework Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. (2019) In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Conference Paper; GNU Parallel 20250222 (‘Grete Tange’) Ole Tange. (2025). GNU Parallel: A general parallelizer for command-line programs. Zenodo. doi:10.5281/zenodo.14911163

This exploration was enabled by 11.5KTensorZero, our open-source stack for building industrial-grade LLM applications.

TensorZero creates a feedback loop for optimizing LLM applications — turning production data into smarter, faster, and cheaper models. Under the hood, TensorZero enables a learning flywheel by unifying inference, observability, optimization (prompts, models, inference strategies — including MIPRO), evaluations, and experimentation.

With TensorZero, you can try this method (and many others) on the problems, data, and metrics that matter to you!