A lead or outreach agent works well in the first weeks, then starts breaking rules or producing weaker output. Nobody notices because the human review stopped once the agent looked trustworthy.
Vercel's COO said their qualification prompt grew to about 1,000 lines and the model did not always follow rules inside it, as Tomasz Tunguz reports. Inputs also change over time, so a prompt tuned on early leads meets new cases.
Put hard rules (region, size, existing owner) in code or CRM lookups, and keep the prompt for judgement. Start in shadow mode, then review every output, then sample a fixed share each week and log the pass rate. Consider a second check that only looks for rule breaches.
Vercel moved from full human review to sampling, saw the model miss rules in a long prompt, moved qualification into 14 code rules and added a second agent to watch for breaches.
TechCrunch quotes a VC who found an AI SDR product stopped generating effective leads after about a month, and customers who described hallucinations and manual correction.
A vendor lists silent scoring drift, stale data and headcount-only scoring among the failure modes of a Claude lead routine.