Introducing Agent Evals: Turn Feedback into Better Instructions
Evaluate agent output with your team, customize evaluation questions, and review suggested prompt updates in Grep.
Day 4 of our launch week: native Evals in Grep. Your team can evaluate agent output, provide feedback, and review suggested prompt changes in the same place you build and run the agent.
Keep evaluations with the agent
We built this after seeing a customer maintain a separate evaluation spreadsheet for every agent. Their evaluations and agents lived in different places.
Grep brings that work together. When you create an agent, Grep automatically creates a set of evaluation questions. You can edit and update those questions to reflect what matters in your workflow.
The questions give your team a starting point for review. You decide what the agent needs to get right and adapt the evaluation as your requirements change.
Give feedback as the agent runs
Your team can evaluate the agent while it is running and provide feedback on its accuracy. The review happens alongside the agent, so you can move from inspecting its work to recording what needs attention.
Keep feedback specific to the output: which requirement did it meet, and where did it miss something?
Review suggested prompt changes
Feedback goes back into the agent harness, and Grep suggests prompt updates. You can review and apply those changes in a process similar to a GitHub pull request.
You remain in control of the prompt. Review the proposed changes, apply the ones you want, and evaluate subsequent work against your criteria. This creates a practical cycle of running the agent, reviewing its output, and deciding how to improve its instructions.
Get started
Create an agent and review its automatically generated evaluation questions.
Edit the questions to match your workflow and requirements.
Evaluate the agent's work and provide feedback on its accuracy.
Review Grep's suggested prompt updates and apply the changes you choose.
Set up Evals for your agent in Grep.