AEP-004: LoopAgent for Self-Healing Remediation¶
Update (2026-06-07): The
LoopAgentimplementation described here has been migrated to a native ADK 2.0 Graph Workflow. See ADR-003 for details on the newverify_routefunction node that replacesLoopAgent.
| Field | Value |
|---|---|
| Status | completed |
| Priority | P1 |
| Effort | Medium (3-4 days) |
| Impact | High |
| Dependencies | None |
Gap Analysis¶
Current Implementation¶
The project uses three ADK workflow agent types: - SequentialAgent: For ordered pipelines (e.g., incident_triage_agent) - ParallelAgent: For concurrent health checks - LLM Agent + AgentTool: For dynamic routing
However, there is no iterative workflow. The incident triage agent runs once: 1. Parallel health checks 2. Summarize findings 3. Write to journal
If a remediation action fails or the system doesn't recover, there is no retry loop.
What ADK Provides¶
ADK has LoopAgent (since v0.1.0) for iterative workflows: - Executes sub-agents in a loop until a condition is met or max_iterations is reached - Sub-agents can signal termination via tool_context.actions.escalate = True - Supports the pattern: Act -> Check -> Decide (continue or stop)
Gap¶
DevOps automation requires closed-loop remediation: - Restart a pod -> check if it's healthy -> if not, try scaling -> check again - Create a topic -> verify it exists -> if not, retry with different config - Silence an alert -> verify the silence took effect -> if not, escalate
The current system is open-loop: it takes actions but never verifies outcomes.
Proposed Solution¶
Step 1: Create a Remediation Loop Pattern¶
from google.adk.agents import LoopAgent, SequentialAgent, LlmAgent
# Step 1: The actor - takes remediation action
remediation_actor = LlmAgent(
name="RemediationActor",
model=resolve_model(),
instruction="""
You are a DevOps remediation agent. Based on the diagnosis in {diagnosis},
take the most appropriate remediation action using your tools.
Record what you did in your output.
""",
tools=[restart_deployment, scale_deployment, ...],
output_key="remediation_action",
)
# Step 2: The verifier - checks if remediation worked
remediation_verifier = LlmAgent(
name="RemediationVerifier",
model=resolve_model(),
instruction="""
Verify whether the remediation action "{remediation_action}" was successful.
Check the current state of the affected system.
If the issue is resolved, call the exit_loop tool.
If not, describe what's still wrong so the actor can try a different approach.
""",
tools=[get_deployment_status, get_pod_logs, exit_loop],
output_key="verification_result",
)
# Step 3: Combine into a loop
remediation_loop = LoopAgent(
name="RemediationLoop",
sub_agents=[remediation_actor, remediation_verifier],
max_iterations=3, # Safety limit
)
Step 2: Integrate into Incident Triage Pipeline¶
Extend the existing incident_triage_agent:
incident_triage_agent = SequentialAgent(
name="incident_triage_agent",
sub_agents=[
health_check_agent, # Parallel health checks (existing)
triage_summarizer, # Summarize findings (existing)
remediation_loop, # NEW: attempt auto-remediation
journal_writer, # Record results (existing)
],
)
Step 3: Create Exit Loop Tool¶
from google.adk.tools.tool_context import ToolContext
async def exit_loop(tool_context: ToolContext) -> dict:
"""Signal that remediation is complete and the loop should stop."""
tool_context.actions.escalate = True
tool_context.actions.skip_summarization = True
return {"status": "remediation_complete"}
Step 4: Add Remediation-Specific Scenarios¶
| Scenario | Loop Steps | Max Iterations |
|---|---|---|
| Pod crash loop | Restart -> Check -> Scale -> Check | 3 |
| High consumer lag | Reset offsets -> Check lag -> Scale consumers | 3 |
| Alertmanager noise | Silence -> Verify silence -> Adjust duration | 2 |
| Deployment rollback | Rollback -> Check health -> Verify pods | 2 |
Affected Files¶
| File | Change |
|---|---|
core/orrery_core/base.py | create_loop_agent() factory function |
core/orrery_core/__init__.py | Export create_loop_agent |
agents/k8s-health/k8s_health_agent/tools.py | rollback_deployment tool (@destructive) |
agents/k8s-health/k8s_health_agent/agent.py | Register rollback_deployment |
agents/orrery-assistant/orrery_assistant/remediation.py | New: exit_loop, actor, verifier, loop, summarizer, pipeline |
agents/orrery-assistant/orrery_assistant/agent.py | Expose remediation_pipeline as AgentTool on root agent |
core/tests/test_loop_agent.py | 5 tests for create_loop_agent factory |
agents/k8s-health/tests/test_k8s_tools.py | 4 tests for rollback_deployment |
agents/orrery-assistant/tests/test_devops_remediation.py | 11 tests for exit_loop + agent wiring |
Acceptance Criteria¶
- [x]
create_loop_agent()factory function added to core - [x] Remediation loop integrated into orrery-assistant (as AgentTool)
- [x] Loop exits on successful remediation (via
exit_looptool) - [x] Loop respects
max_iterationssafety limit (default: 3) - [x] Remediation actions are logged to ops journal (
log_operationtool) - [x] Remediation scenarios: pod restart, scale, rollback (actor tools)
- [x] RBAC enforced:
@destructive/@confirmdecorators on all remediation tools
Notes¶
- Safety is critical: the remediation loop must respect RBAC and confirmation guards. A viewer should never trigger auto-remediation.
- Consider adding a "dry-run" mode where the loop plans actions but doesn't execute them, useful for review before committing to changes.
- The
max_iterationslimit prevents runaway loops but should also be paired with a timeout to prevent long-running remediations.