Qodo Best Practices

Rules, Reviews, and Governance at Scale

— Sri Rang, Solutions Architect @ Qodo

8 Practices

  • 1. Rules — Enforce coding standards per-repo
  • 2. PR Findings Sync — Persist findings for analytics
  • 3. Incremental vs Full Re-Review — Review only what changed
  • 4. Draft PRs — Suppress noise while WIP
  • 5. Compliance at Merge — Silent audit on every merge
  • 6. Verifiable Work Items — Good acceptance criteria = good review
  • 7. Auto-Generate Descriptions — Structure every PR automatically
  • 8. Chat Agent — Follow-up without leaving ADO

1. Rules

Enforce Team Standards on Every PR

Rules — Two Sources

Repo Rules
.qodo/rules.yaml

Per-repo, version-controlled
alongside the code

Platform Rules
Qodo Console

Org-wide governance
overrides repo rules

Repo Rules — The YAML File

- title: "Work item must be linked in the PR description"
  objective: "Ensure every PR is traceable to a work item"
  success_criteria: "PR description contains AB#1234"
  failure_criteria: "No AB# reference found in the PR description"
  compliance_label: true
  category: "traceability"
  external_id: "ORG-POLICY-001"

- title: "No secrets or connection strings in committed code"
  compliance_label: true
  category: "security"
  severity: "critical"
  external_id: "ORG-POLICY-002"

Repo Rules — Wire It Up

.pr_agent.toml:

[config]
add_repo_metadata = true
add_repo_metadata_file_list = [".qodo/rules.yaml"]

On the next PR, Qodo:

  1. Reads .qodo/rules.yaml — no LLM step
  2. Evaluates each rule against the diff
  3. Reports violations in the review comment

Repo Rules — Key Constraints

Constraint Detail
File must end in .yaml or .yml .json, .toml, .md not picked up
YAML list at the top level Dict at root → skipped with warning
title is the only required field All other fields default
compliance_label: true adds a blocking label Defaults to false
enable_rules_platform: true overrides file rules Platform rules replace repo rules entirely

Platform Rules — Org Wide Governance

flowchart LR subgraph Org["Engineering Org"] Devs["Developers\n(all)"] A1(["Engineering Leads"]) A2(["Security + Compliance"]) end Console[("Qodo Console\nPlatform Rules")] subgraph Repos["All Repositories"] R1["Repo A"] R2["Repo B"] R3["Repo C"] end A1 -->|"define & manage"| Console A2 -->|"define & manage"| Console Console -->|"enforced on every PR"| R1 Console -->|"enforced on every PR"| R2 Console -->|"enforced on every PR"| R3 classDef admin fill:#6b21a8,stroke:#4c1d95,color:#fff classDef dev fill:#e2e8f0,stroke:#94a3b8,color:#334155 classDef console fill:#7c3aed,stroke:#5b21b6,color:#fff classDef repo fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f class A1,A2 admin class Devs dev class Console console class R1,R2,R3 repo

Platform Rules — Rules via Platform

Screenshot 1 Screenshot 2 Screenshot 3 Screenshot 4 Screenshot 5
×

Platform Rules — Metrics: Rules & Findings

Screenshot 1 Screenshot 2 Screenshot 3
×

Platform Rules — Metrics: Usage Analytics

Screenshot 1 Screenshot 2 Screenshot 3 Screenshot 4 Screenshot 5 Screenshot 6 Screenshot 7 Screenshot 8
×

Rules — Usage

Screenshot 1 Screenshot 2
×

2. PR Findings Sync

Persist Findings — Unlock Analytics

PR Findings Sync — Why Enable It

Without sync With sync
Re-reports the same findings on every push Tracks new vs. still open across pushes
Pre-PR findings lost when PR opens Surfaced in the PR review automatically
No record of accepted/dismissed suggestions Tracked per finding; feeds impact metrics
Compliance statistics unavailable Captured on merge
No workspace analytics Aggregate trends across repos and teams

PR Findings Sync — Configuration

Required

[qodo_platform]
enable_findings_sync = true

[azure_devops_server]
pr_commands = ["/agentic_review"]
push_commands = ["/agentic_review"]

Optional — control which findings appear inline:

[review_agent]
inline_comments_severity_threshold = 2  # 1=informational 2=remediation_recommended 3=action_required
comments_location_policy = "both"       # "inline" | "summary" | "both"

PR Findings Sync — What Gets Stored

Per finding
  • Title and description
  • Category (Security, Performance…)
  • Action level / severity
  • Diff reference — file, line range
  • Fix suggestion
Per finding (cont.)
  • Evidence (context used)
  • Git metadata — SHA, branch, repo, org
  • PR URL and author
  • Attribution status


Attribution status: pending → accepted / dismissed / deferred
This is the foundation for impact metrics.

PR Findings Sync — In Practice

Findings Dashboard

Browse, filter, and triage all findings across repos — by severity, category, state, and PR.

Workspace Analytics

Aggregate trends over time — fix rates, top violation categories, per-team and per-repo breakdowns.


See Metrics: Rules & Findings section for live screenshots of the Findings dashboard.

3. Incremental vs. Full Re-Review

Review Only What Changed

Option A — Full Re-Review on Every Push

[azure_devops_server]
push_commands = ["/agentic_review"]

[review_agent]
enable_incremental_review = false

Pros - Always shows the complete picture of open findings - No dependency on previous review state

Cons - Re-reports findings the developer already saw - Higher LLM cost per push - Noisy on PRs with many small commits

Best for: one-off audits on significant PRs, not daily pushes.

Option B — Incremental Review (Recommended)

[azure_devops_server]
push_commands = ["/agentic_review"]

[review_agent]
enable_incremental_review = true
persistent_comment = true

[qodo_platform]
enable_findings_sync = true

Pros - Focuses only on what changed since the last review - Developers see new findings, not repeats - Lower LLM cost per push

Requires findings persistence — if a prior review failed, the incremental focus may be incomplete.

Incremental vs. Full — Recommendation

Daily development
Option B

Incremental — less noise,
lower cost, faster feedback loop

Audit / significant PR
Option A

Full re-review — complete
picture, no state dependency

4. Draft PRs

Suppress Noise While Work is in Progress

Draft PRs — Configuration & When to Use

[azure_devops_server]
feedback_on_draft_pr = false

When false, Qodo skips automatic review on draft PRs entirely. Manual /agentic_review is still available on demand.


Enable suppression when your team uses drafts as genuine WIP — code that isn't ready for feedback.

Keep it enabled when your team uses drafts for early collaboration and expects feedback during drafting.

Draft PRs — Pros & Cons

Pro Con
No premature feedback on WIP Developers lose early signal
Saves LLM cost Draft → never-converted = never reviewed
Cleaner PR thread Relies on developer discipline for manual /agentic_review

Draft PRs — Recommendation

Suppress (false)

Team treats drafts as genuine WIP.
Review only fires when PR is marked ready.

Keep enabled (true)

Team uses drafts for early collaboration.
Pair with incremental review to reduce noise.

5. Compliance at Merge Time

Silent Audit on Every Merge

Compliance at Merge — How It Works

flowchart TD W["git.pullrequest.merged webhook fires"] C["Qodo runs compliance-only pass (issues agent disabled — lower cost)"] S["Nothing posted to the PR"] E["pr_compliance_statistics emitted to analytics pipeline"] W --> C --> S --> E classDef step fill:#1e1b4b,stroke:#6d28d9,color:#e0d7ff class W,C,S,E step

Runs automatically if pr_commands includes /agentic_review:

[azure_devops_server]
pr_commands = ["/agentic_review"]

Compliance at Merge — What Gets Captured

Per-PR Statistics
  • Rules checked
  • Rules passed / failed
  • Per-rule outcomes
Questions It Answers
  • Which teams have lowest compliance?
  • Are scores improving over sprints?
  • Which rules are most frequently violated?

Compliance at Merge — Requirements

Compliance rules defined

YAML rules file or platform rules must be configured — without rules, the statistics are empty.

Analytics pipeline

Statistics are log events, not a dashboard. Your team must consume the Qodo analytics pipeline or export to a BI tool.


Without both, the feature runs but produces no actionable output.

Compliance at Merge — Recommendation

Enable when
  • Compliance rules are defined
  • You have a way to consume analytics output
  • Leadership wants trend data across teams
Skip for now if
  • No rules configured yet
  • No analytics pipeline in place
  • Feature runs silently — zero visible effect

6. Verifiable Work Items

Good Acceptance Criteria = Good Review

Work Items — How Qodo Discovers Them

Qodo finds linked work items from three sources:

  1. Work items linked via the ADO Boards API
  2. References in the PR description — AB#123, full URLs, #123
  3. Branch name prefix — 1234-fix-auth-flow → work item 1234
Highest-signal input
Acceptance Criteria field

Read separately from the description — if you only fill the description, you're leaving signal on the table.

Work Items — Write Verifiable Criteria

Weak Strong
"The API should be fast" "Must respond within 200ms under normal load"
"Handle errors properly" "Must return HTTP 400 with a structured error body on missing fields"
"Make it secure" "JWT signatures must be validated before granting access"

Work Items — One Condition Per Bullet

Good:
- Return HTTP 404 when the resource does not exist
- Include an error_code field on all failures
- Use parameterized statements for all DB queries

Avoid:
- Handle errors correctly, return the right status codes,
  and ensure no SQL injection is possible

Work Items — Configuration

Feature is on by default. To verify explicitly:

[pr_compliance]
require_ticket_analysis_review = true

[review_agent]
requirements_gap_enabled = true

No work item linked or referenced → ticket compliance pass is skipped entirely.

Work Items — Pitfalls

Pitfall Impact
No acceptance criteria + vague description Zero compliance rules generated
No linked or referenced work item Ticket compliance pass skipped entirely
Description longer than 10,000 characters Truncated — put key constraints in Acceptance Criteria
Ambiguous or compound criteria Inconsistent rule extraction

Work Items — Recommendation

  • Fill in the Acceptance Criteria field — not just the description
  • Write one verifiable condition per bullet
  • Use technical constraints, not business requirements
  • Link work items explicitly via ADO PR interface or branch naming


The quality of the AI review is a direct function of the quality of the work item.

7. Auto-Generate PR Descriptions

Structure Every PR Automatically

/agentic_describe — What It Generates

PR Title
Concise, meaningful

Replaces vague titles
like "fix stuff"

PR Body
Structured summary

What changed and why

File Walkthrough
Per-file explanation

Role of each changed file

Persistent comment
Updated in-place

No comment spam on push

/agentic_describe — Configuration

[azure_devops_server]
pr_commands = [
  "/agentic_describe",
  "/agentic_review",
]

Keep publish_as_comment = true (default) under [pr_agentic_description] on ADO. ADO does not support collapsible sections — writing to the PR body makes large PRs very long.

/agentic_describe — Reviewer Impact

  • Faster onboarding — reviewers understand scope and intent before opening the diff
  • Consistent format — every PR gets a structured title and summary regardless of author habits
  • Less back-and-forth — context that would need author clarification is auto-generated
  • Zero dev overhead — authors don't need to write detailed descriptions manually

/agentic_describe — Pitfall

Pitfall: push_commands includes /agentic_describe by default — description updates on every push. Large PRs can produce a long persistent comment.

Mitigation: remove from push_commands if per-push updates are noisy:

[azure_devops_server]
push_commands = ["/agentic_review"]

Keep publish_as_comment = true under [pr_agentic_description] on ADO — no collapsible sections means writing to the PR body makes large PRs very long.

8. Chat Agent

Developer Follow-Up Without Leaving ADO

Chat Agent — Two Interaction Modes

Inline Finding Replies

Developer replies to a Qodo finding comment.
Agent answers in context — knows exactly what was flagged and why.

General PR Chat

Developer posts /qodo <question>.
Agent answers questions about the diff, code, or findings.

Access: full PR diff + published findings. No access to external systems (Jira, ADO Boards, etc.).

Chat Agent — Configuration

[chat_agent]
enabled = true

That's it. No additional setup needed for Azure DevOps.

Chat Agent — Benefits

  • Stays in ADO — no need to switch to a chat tool to ask about a finding
  • Finding-aware answers — full context of what was flagged and why
  • Accelerates learning — junior devs can ask "why is this a problem?" and get a code-level explanation
  • Reduces false positive friction — developers can justify a finding before dismissing it

Chat Agent — Pitfalls

Pitfall Mitigation
Off by default — easy to miss Explicitly add chat_agent.enabled = true
Triggered by any reply to bot comments Agent may respond to human discussion threads unexpectedly
Adds LLM calls per reply Monitor usage if cost is a concern

Chat Agent — Recommendation

Workflow
  1. Enable: chat_agent.enabled = true
  2. Run /agentic_review on the PR
  3. Developers reply to findings in-thread
  4. Agent responds with reasoning + code references
Best for
  • Teams with junior developers
  • High finding volume — reduce dismissal friction
  • Remote teams — async discussion in ADO

Recommended Baseline Config

Baseline .pr_agent.toml

[config]
add_repo_metadata = true
add_repo_metadata_file_list = [".qodo/rules.yaml"]

[azure_devops_server]
pr_commands = ["/agentic_describe", "/agentic_review"]
push_commands = ["/agentic_review"]
feedback_on_draft_pr = false

[review_agent]
enable_incremental_review = true
persistent_comment = true
inline_comments_severity_threshold = 2

[qodo_platform]
enable_findings_sync = true

[pr_compliance]
require_ticket_analysis_review = true

[chat_agent]
enabled = true