[delight] User Experience Analysis Report - 2026-08-06

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (docs/src/content/docs/reference/outcomes.md, docs/src/content/docs/examples/maintaining-repos.md)
• 2 workflow message configurations (smoke-create-cross-repo-pr.md, architecture-guardian.md)
• 1 validation file (pkg/workflow/model_costs_pricing_validation_test.go → traced to pkg/workflow/model_costs_pricing_validation.go)

Overall Quality: Professional overall, with one clear inconsistency in error message construction.

Key Finding: validateDefaultAiCreditsPricing uses two different error-reporting mechanisms within the same function — a structured NewValidationError (field/value/reason/suggestion) for the AWF-version check, but plain fmt.Errorf strings for the positivity checks — producing inconsistent error shape and missing actionable suggestions for the most commonly hit errors (zero/negative pricing values).

Quality Highlights ✅

Example 1: Outcomes Reference Documentation

• File: docs/src/content/docs/reference/outcomes.md
• What works well: Clear definitions table for outcome states, a comprehensive per-safe-output "accepted" lookup table, and explicit cross-links to related docs (Cost Management, OpenTelemetry, Safe Outputs). The tone is precise and enterprise-appropriate throughout.
• Quote/Reference: "Outcome data is based on repository state, not on the workflow's self-assessment." — precise, trust-building framing.

Example 2: Architecture Guardian Workflow Messages

• File: .github/workflows/architecture-guardian.md
• What works well: messages block uses consistent, sparing emoji (🏛️/✅/❌), clear run-state framing, and includes workflow name + run URL links for traceability.
• Quote/Reference: "run-failure": "❌ Architecture scan failed. [{workflow_name}]({run_url}) {status}. Scan did not complete." — concise and actionable.

Improvement Opportunities 💡

High Priority

Opportunity 1: Inconsistent Error Message Construction - Single File Improvement

• File: pkg/workflow/model_costs_pricing_validation.go
• Current State: Lines 32–37 use the structured NewValidationError(field, value, reason, suggestion) helper (which produces a consistently formatted, severity-classified error with an explicit remediation suggestion) for the AWF-version compatibility check. Lines 39–50, covering input, output, cache_read, and cache_write positivity checks, instead use raw fmt.Errorf(...) strings with no Field/Suggestion structure.
• Issue: Within a single validation function, users hitting different sub-checks get differently-shaped errors — one with a clear "Suggestion:" remediation block, the others as plain sentences. This breaks the "Consistent experience across touchpoints" and "Clear error messages with actionable solutions" principles, and likely affects users more often, since a zero/negative pricing value is a much more common typo (e.g., input: 0) than hitting an outdated AWF pin.
• User Impact: Users configuring models.default-ai-credits-pricing who make a simple typo (entering 0 for input) get a terser, less structured message than users who hit the AWF version issue, even though both are validation failures in the same feature area.
• Suggested Change: Convert the four fmt.Errorf calls (lines 40, 43, 46, 49) to use NewValidationError("models.default-ai-credits-pricing", value, reason, suggestion) for consistency, adding a concrete suggestion (e.g., "Set input to a positive rate, such as 0.000001 for effectively-free self-hosted models.").
• Design Principle: Trust and Reliability — consistent error shape across all sub-checks in the same validation path.

Before (lines 39-44):

if p.Input <= 0 {
    return fmt.Errorf("models.default-ai-credits-pricing: input must be a positive value (got %g); use a small positive rate such as 0.000001 for effectively-free self-hosted models", p.Input)
}
if p.Output <= 0 {
    return fmt.Errorf("models.default-ai-credits-pricing: output must be a positive value (got %g); use a small positive rate such as 0.000001 for effectively-free self-hosted models", p.Output)
}

After:

if p.Input <= 0 {
    return NewValidationError(
        "models.default-ai-credits-pricing.input",
        fmt.Sprintf("%g", p.Input),
        "input must be a positive value",
        "Use a small positive rate such as 0.000001 for effectively-free self-hosted models.",
    )
}
if p.Output <= 0 {
    return NewValidationError(
        "models.default-ai-credits-pricing.output",
        fmt.Sprintf("%g", p.Output),
        "output must be a positive value",
        "Use a small positive rate such as 0.000001 for effectively-free self-hosted models.",
    )
}

Files Reviewed

Documentation

• docs/src/content/docs/reference/outcomes.md - Rating: ✅
• docs/src/content/docs/examples/maintaining-repos.md - Rating: ✅

Workflow Messages

• .github/workflows/smoke-create-cross-repo-pr.md - Rating: ✅
• .github/workflows/architecture-guardian.md - Rating: ✅

Validation Code

• pkg/workflow/model_costs_pricing_validation.go - Rating: ⚠️ Needs Minor Work

Metrics

• Files Analyzed: 5
• Quality Distribution:
  • ✅ Professional: 4
  • ⚠️ Needs Minor Work: 1
  • ❌ Needs Significant Work: 0

🎯 Actionable Tasks

Task 1: Standardize error construction in model_costs_pricing_validation.go

File to Modify: pkg/workflow/model_costs_pricing_validation.go

Current Experience

The function validateDefaultAiCreditsPricing (lines 23–53) mixes two error styles: the AWF-version check (lines 32–37) returns a *WorkflowValidationError via NewValidationError, with a structured Field, Value, Reason, and Suggestion. The four positivity checks (lines 39–50) return plain fmt.Errorf strings that embed the same information ad hoc in a single sentence, with no separate suggestion field.

Quality Issue

Design Principle: Trust and Reliability — "Consistent experience across touchpoints" and "Clear error messages with actionable solutions."

Because the positivity checks are far more likely to be triggered by ordinary user typos (e.g., accidentally setting input: 0), inconsistent error shape here has an outsized impact on the everyday validation experience, even though the less-common AWF-version error already has the more polished treatment.

Proposed Improvement

Convert all four fmt.Errorf calls to use NewValidationError, matching the field-naming and suggestion conventions already established by the AWF-version check in the same function.

Before:

if p.CachedInput != nil && *p.CachedInput <= 0 {
    return fmt.Errorf("models.default-ai-credits-pricing: cache_read must be a positive value when set (got %g)", *p.CachedInput)
}
if p.CacheWrite != nil && *p.CacheWrite <= 0 {
    return fmt.Errorf("models.default-ai-credits-pricing: cache_write must be a positive value when set (got %g)", *p.CacheWrite)
}

After:

if p.CachedInput != nil && *p.CachedInput <= 0 {
    return NewValidationError(
        "models.default-ai-credits-pricing.cache_read",
        fmt.Sprintf("%g", *p.CachedInput),
        "cache_read must be a positive value when set",
        "Remove cache_read to use the default, or set it to a small positive rate such as 0.000001.",
    )
}
if p.CacheWrite != nil && *p.CacheWrite <= 0 {
    return NewValidationError(
        "models.default-ai-credits-pricing.cache_write",
        fmt.Sprintf("%g", *p.CacheWrite),
        "cache_write must be a positive value when set",
        "Remove cache_write to use the default, or set it to a small positive rate such as 0.000001.",
    )
}

Why This Matters

• User Impact: Every user configuring self-hosted model pricing now gets consistent, actionable error output regardless of which validation rule they trip.
• Quality Factor: Trust and Reliability, Clarity and Precision.
• Frequency: This is the most common failure mode for this feature (typo/zero pricing), more frequent than hitting the AWF version gate.

Success Criteria

• Changes made to pkg/workflow/model_costs_pricing_validation.go only
• All four positivity-check errors return *WorkflowValidationError via NewValidationError with matching field naming and existing test assertions (assert.Contains(err.Error(), "input"), etc.) still pass
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/model_costs_pricing_validation.go
• No changes to other files required (existing tests in model_costs_pricing_validation_test.go should continue to pass unmodified since assertions only check for substring presence)
• Can be completed independently

📊 User experience analysis by Delight · auto · 28.9 AIC · ⌖ 4.24 AIC · ⊞ 12.1K · ◷

• expires on Aug 9, 2026, 7:31 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #51118.