--- name: prompt-engineering-cove description: Production-grade prompt engineering with Chain-of-Verification (CoVe) for factual accuracy. Combines structured outputs, few-shot learning, chain-of-thought, and verification loops to maximize LLM reliability. Use when building LLM applications that need verifiable outputs, reducing hallucinations, or implementing self-correcting prompt chains. ---
Prompt Engineering with Chain-of-Verification
Master production prompt patterns with built-in verification for factual accuracy and self-correction.
When to Use This Skill
- Building LLM apps requiring factual accuracy
- Reducing hallucinations in production systems
- Implementing self-correcting prompt chains
- Designing multi-step reasoning with verification
- Creating reliable structured outputs
- Building RAG systems with answer validation
Core Architecture: Chain-of-Verification (CoVe)
CoVe reduces hallucinations through a 4-stage pipeline:
┌─────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐
│ Baseline │───▶│ Plan │───▶│ Execute │───▶│ Refined │
│ Response │ │ Verification │ │ Verification │ │ Response │
└─────────────┘ └─────────────────┘ └─────────────────┘ └──────────────┘
Draft Generate Answer each Cross-check
answer verification question and refine
questions independently
Stage 1: Generate Baseline Response
BASELINE_PROMPT = """Answer the question concisely.
Question: {question}
Answer:"""
Stage 2: Plan Verification Questions
Generate questions that probe each factual claim:
VERIFICATION_PLANNING_PROMPT = """Given this question and answer, generate verification questions to check factual accuracy.
Question: {question}
Answer: {baseline_response}
For each factual claim in the answer, create a verification question.
Example:
- If answer mentions "X was born in Y" → "Was X born in Y?"
- If answer mentions "X invented Y in Z" → "Did X invent Y?" and "Was Y invented in Z?"
Verification Questions:"""
Stage 3: Execute Verification
Answer each verification question independently (optionally with tools):
EXECUTE_VERIFICATION_PROMPT = """Answer this verification question accurately.
Question: {verification_question}
Answer (yes/no with brief explanation):"""
Stage 4: Refine Based on Verification
FINAL_REFINEMENT_PROMPT = """Refine the original answer based on verification results.
Original Question: {question}
Baseline Answer: {baseline_response}
Verification Results:
{verification_qa_pairs}
Instructions:
- Keep claims that passed verification
- Remove or correct claims that failed
- Do not add new unverified information
Refined Answer:"""
---
Complete CoVe Implementation
from anthropic import Anthropic
from typing import List, Dict
import json
class ChainOfVerification:
def __init__(self, model="claude-sonnet-4-5"):
self.client = Anthropic()
self.model = model
def generate(self, question: str, use_tools: bool = False) -> Dict:
# Stage 1: Baseline
baseline = self._get_baseline(question)
# Stage 2: Plan verification
v_questions = self._plan_verification(question, baseline)
# Stage 3: Execute verification
v_answers = self._execute_verification(v_questions, use_tools)
# Stage 4: Refine
final = self._refine(question, baseline, v_questions, v_answers)
return {
"baseline": baseline,
"verification_questions": v_questions,
"verification_answers": v_answers,
"final_answer": final
}
def _get_baseline(self, question: str) -> str:
response = self.client.messages.create(
model=self.model,
max_tokens=500,
messages=[{
"role": "user",
"content": f"Answer concisely:\n\n{question}"
}]
)
return response.content[0].text
def _plan_verification(self, question: str, baseline: str) -> List[str]:
prompt = f"""Generate verification questions for each factual claim.
Question: {question}
Answer: {baseline}
Output as JSON array of strings:
["question1", "question2", ...]"""
response = self.client.messages.create(
model=self.model,
max_tokens=500,
messages=[{"role": "user", "content": prompt}]
)
return json.loads(response.content[0].text)
def _execute_verification(
self,
questions: List[str],
use_tools: bool
) -> List[Dict]:
results = []
for q in questions:
if use_tools:
answer = self._verify_with_search(q)
else:
answer = self._verify_self(q)
results.append({"question": q, "answer": answer})
return results
def _verify_self(self, question: str) -> str:
response = self.client.messages.create(
model=self.model,
max_tokens=200,
messages=[{
"role": "user",
"content": f"Answer yes or no with brief explanation:\n{question}"
}]
)
return response.content[0].text
def _refine(
self,
question: str,
baseline: str,
v_questions: List[str],
v_answers: List[Dict]
) -> str:
v_pairs = "\n".join(
f"Q: {a['question']}\nA: {a['answer']}"
for a in v_answers
)
prompt = f"""Refine the answer based on verification.
Original Question: {question}
Baseline Answer: {baseline}
Verification Results:
{v_pairs}
Keep verified claims. Remove/correct failed claims.
Refined Answer:"""
response = self.client.messages.create(
model=self.model,
max_tokens=500,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
---
CoVe Question Type Chains
Different question types need different verification strategies:
Wiki/List Questions
Questions asking for lists of entities (e.g., "Name scientists who won Nobel Prize in 1970")
# Generate verification template first
WIKI_TEMPLATE_PROMPT = """Create a verification question template.
Example Question: Who are movie actors born in Boston?
Example Template: Was [actor name] born in Boston?
Actual Question: {question}
Template:"""
# Then instantiate for each entity in baseline
WIKI_VERIFY_PROMPT = """Using this template and baseline, generate verification questions.
Question: {question}
Baseline: {baseline}
Template: {template}
Verification Questions (one per entity):"""
Multi-Span Questions
Questions with multiple independent answers (e.g., "Who invented the printing press and when?")
MULTI_VERIFY_PROMPT = """Generate verification questions for each independent claim.
Question: {question}
Baseline: {baseline}
Each claim should have its own verification question.
Example:
Question: Who invented the telephone and when?
Baseline: Alexander Graham Bell invented it in 1876.
Verification Questions:
1. Did Alexander Graham Bell invent the telephone?
2. Was the telephone invented in 1876?
Verification Questions:"""
Long-Form Questions
Questions requiring detailed answers
LONG_VERIFY_PROMPT = """Generate verification questions for key factual claims.
Question: {question}
Baseline: {baseline}
Focus on verifiable facts (names, dates, numbers, relationships).
Skip opinions and general statements.
Verification Questions:"""
---
Routing to the Right Chain
ROUTER_PROMPT = """Classify this question type.
Categories:
- WIKI: Asks for a list of entities (people, places, things)
- MULTI: Contains multiple independent sub-questions
- LONG: Requires detailed explanation
Question: {question}
Output JSON: {{"category": "WIKI|MULTI|LONG"}}"""
class CoVERouter:
def route(self, question: str) -> str:
response = self.client.messages.create(
model=self.model,
max_tokens=50,
messages=[{
"role": "user",
"content": ROUTER_PROMPT.format(question=question)
}]
)
result = json.loads(response.content[0].text)
return result["category"]
---
Integration Patterns
CoVe + RAG
class CoVERAG:
def __init__(self, retriever, llm):
self.retriever = retriever
self.llm = llm
def answer(self, question: str) -> str:
# Retrieve context
docs = self.retriever.get_relevant_documents(question)
context = "\n".join(d.page_content for d in docs)
# Baseline with context
baseline = self._generate_baseline(question, context)
# Verify against retrieved docs
v_questions = self._plan_verification(question, baseline)
# Execute verification using RAG
v_answers = []
for vq in v_questions:
# Re-retrieve for each verification question
vq_docs = self.retriever.get_relevant_documents(vq)
vq_context = "\n".join(d.page_content for d in vq_docs)
answer = self._verify_with_context(vq, vq_context)
v_answers.append({"question": vq, "answer": answer})
# Refine
return self._refine(question, baseline, v_answers)
CoVe + Tool Use
TOOL_VERIFICATION_PROMPT = """Answer this verification question using the search results.
Search Results:
{search_results}
Question: {verification_question}
Answer (cite sources):"""
class CoVEWithTools:
def __init__(self, search_tool):
self.search = search_tool
def verify_with_search(self, question: str) -> str:
results = self.search.run(question)
prompt = TOOL_VERIFICATION_PROMPT.format(
search_results=results,
verification_question=question
)
return self.llm.invoke(prompt)
CoVe + Structured Output
from pydantic import BaseModel, Field
from typing import List, Literal
class VerificationResult(BaseModel):
question: str
answer: Literal["verified", "failed", "uncertain"]
evidence: str
confidence: float = Field(ge=0, le=1)
class CoVEOutput(BaseModel):
baseline: str
verifications: List[VerificationResult]
final_answer: str
overall_confidence: float
def structured_cove(question: str) -> CoVEOutput:
# ... implementation with structured output enforcement
pass
---
Chain-of-Thought + CoVe
Combine reasoning with verification:
COT_COVE_PROMPT = """Solve this problem step by step, then verify each step.
Problem: {problem}
REASONING:
Step 1: [reasoning]
Verification: [check this step]
Step 2: [reasoning]
Verification: [check this step]
...
FINAL ANSWER: [answer]
CONFIDENCE: [high/medium/low based on verification results]"""
---
Few-Shot Learning
Dynamic Example Selection
from langchain_core.example_selectors import SemanticSimilarityExampleSelector
from langchain_voyageai import VoyageAIEmbeddings
examples = [
{"input": "reset password", "output": "Settings > Security > Reset Password"},
{"input": "order history", "output": "Account > Orders"},
{"input": "contact support", "output": "Help > Contact Us"},
]
selector = SemanticSimilarityExampleSelector.from_examples(
examples=examples,
embeddings=VoyageAIEmbeddings(model="voyage-3-large"),
vectorstore_cls=Chroma,
k=2
)
async def few_shot_prompt(query: str) -> str:
selected = await selector.aselect_examples({"input": query})
examples_text = "\n".join(
f"User: {ex['input']}\nResponse: {ex['output']}"
for ex in selected
)
return f"Examples:\n{examples_text}\n\nUser: {query}\nResponse:"
---
Structured Outputs with Pydantic
from pydantic import BaseModel, Field
from typing import Literal, List
class Analysis(BaseModel):
sentiment: Literal["positive", "negative", "neutral"]
confidence: float = Field(ge=0, le=1)
key_points: List[str]
reasoning: str
# With Anthropic
client = Anthropic()
def analyze(text: str) -> Analysis:
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=500,
messages=[{
"role": "user",
"content": f"""Analyze this text.
Text: {text}
Respond with JSON:
{{
"sentiment": "positive|negative|neutral",
"confidence": 0.0-1.0,
"key_points": ["point1", "point2"],
"reasoning": "explanation"
}}"""
}]
)
return Analysis(**json.loads(response.content[0].text))
---
System Prompt Patterns
Expert Role
EXPERT_SYSTEM = """You are a senior {domain} expert with 15+ years experience.
Responsibilities:
- Provide accurate, well-reasoned analysis
- Cite sources when making factual claims
- Acknowledge uncertainty when present
- Recommend next steps
Constraints:
- Do not speculate beyond your knowledge
- Ask clarifying questions when needed
- Flag potential risks or concerns"""
Verification-First Role
VERIFICATION_SYSTEM = """You are a fact-checking assistant.
For every claim you make:
1. State the claim
2. Assess confidence (high/medium/low)
3. Note if verification is needed
If you're uncertain, say so. Never present speculation as fact."""
---
Performance Optimization
Prompt Caching
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=1000,
system=[{
"type": "text",
"text": LONG_SYSTEM_PROMPT,
"cache_control": {"type": "ephemeral"}
}],
messages=[{"role": "user", "content": query}]
)
Parallel Verification
import asyncio
async def parallel_verify(questions: List[str]) -> List[Dict]:
tasks = [verify_single(q) for q in questions]
return await asyncio.gather(*tasks)
Token Efficiency
# Verbose (150+ tokens)
verbose = """I would like you to take the text and provide a comprehensive summary..."""
# Efficient (30 tokens)
efficient = """Summarize key points:\n\n{text}\n\nSummary:"""
---
Evaluation Metrics
Track these for production:
| Metric | Description | Target | |--------|-------------|--------| | Factual Accuracy | % verified claims | >95% | | Hallucination Rate | % unverified claims | <5% | | Verification Coverage | % claims with verification | >90% | | Latency | P95 response time | <3s | | Token Efficiency | Tokens per verified answer | Minimize |
---
Best Practices
1. Always verify factual claims — Use CoVe for any question with verifiable facts 2. Use tools when available — External verification beats self-verification 3. Route by question type — Different questions need different chains 4. Structure outputs — Pydantic enforcement reduces parsing errors 5. Cache aggressively — System prompts, examples, verification patterns 6. Monitor verification rates — Track what fails and why 7. Fail gracefully — When verification fails, say so
Common Pitfalls
- Skipping verification — "It's probably right" leads to hallucinations
- Self-verification only — Model can confidently verify its own mistakes
- Over-verification — Not every sentence needs a verification question
- Ignoring routing — Using wrong chain for question type
- No structured output — Parsing failures in production
Resources
references/cove-chains.md— Full chain implementationsreferences/prompts.md— Complete prompt libraryscripts/cove-runner.py— CLI tool for testing
Want your agent to fetch skills itself? The Academy MCP serves this whole library.
Connect via MCP