How to Build Your Own Accessibility Agent: A Step-by-Step Guide

Overview

Accessibility is a critical aspect of modern web development, but catching every barrier manually is time-consuming and error-prone. Inspired by GitHub's experimental general-purpose accessibility agent—which reviewed over 3,500 pull requests with a 68% resolution rate—this guide shows you how to create your own agent. The agent uses large language models (LLMs) and workflow automation to provide just-in-time accessibility advice and automatically fix common issues before they reach production. By the end, you'll understand how to design, implement, and iterate on an accessibility agent that augments your team's efforts.

Prerequisites

Before diving in, ensure you have:

• Basic understanding of LLMs and AI agents — Familiarity with concepts like prompt engineering, context windows, and agent orchestration. If needed, review resources such as A guide to deciding what AI model to use in GitHub Copilot and GitHub for Beginners: How to get LLMs to do what you want.
• Experience with front-end code review — You'll work with HTML, ARIA attributes, and accessibility testing tools.
• Access to a code hosting platform — The agent integrates with pull request workflows, so GitHub or a similar service is required.
• An LLM API key (e.g., OpenAI, Anthropic) — Used for generating suggestions and automated remediation.

Step-by-Step Instructions

1. Define Your Agent's Goals

GitHub's agent had two primary goals: providing engineers with reliable, just-in-time answers to accessibility questions, and catching and automatically fixing simple, objective issues. Start by clarifying your own objectives. For example:

• Automate the detection of missing alt text on images.
• Ensure interactive controls have clear, unique names.
• Flag unclear focus order in dynamic content.

Write down three to five concrete, measurable goals. This scope helps you avoid attempting to solve every accessibility problem—a mistake that can delay your launch.

2. Set Up the Trigger: Pull Request Evaluation

Your agent should evaluate changes that modify front-end code. Configure a webhook or CI pipeline to trigger on pull requests that include HTML, JSX, or template files. In GitHub Actions, you can create a workflow file:

.github/workflows/accessibility-agent.yml
name: Accessibility Agent
on:
  pull_request:
    paths:
      - '**/*.html'
      - '**/*.jsx'
      - '**/*.tsx'
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run Agent Script
        run: python agent_review.py --pr-number ${{ github.event.pull_request.number }}

The script (agent_review.py) will fetch the diff, analyze it using an LLM, and post results as a comment or review.

3. Build the Core Agent Logic

Your agent needs two modes: answering questions and fixing issues. For simplicity, focus on the fix mode first. Use an LLM to classify each changed line or block. For example, prompt the model with:

You are an accessibility expert. Review the following code changes and identify issues from this list:
1. Missing alt text on images
2. Interactive controls without accessible names
3. Non-semantic headings
4. Focus order problems

For each issue, provide:
- The line number and code snippet
- A suggestion to fix it (with code)
- Confidence level (high/medium/low)

Code diff: [insert diff]

Parse the LLM's response and, for high-confidence issues, apply the fix automatically. Use a tool like ast or beautifulsoup to make precise edits.

4. Prioritize Common Issue Types

GitHub's agent found that the top five issues were: structure/relationships, clear interactive names, announcements, text alternatives, and logical keyboard focus. Your agent should focus on these first. Create a priority matrix:

• High priority (auto-fix): Missing alt attributes, missing labels for input fields.
• Medium priority (suggest only): Color contrast ratios, focus order improvements.
• Low priority (log for manual review): Complex ARIA practices, custom widgets.

This tiered approach matches GitHub's philosophy: the agent is an augmentation, not a silver bullet.

5. Implement Feedback Loop

Once the agent posts a review, engineers can dismiss or adjust its suggestions. Track these interactions to improve the model's prompts. For instance, if many developers override the agent's alt text suggestion, update your prompt to consider context (e.g., decorative images should have alt="" rather than a description).

Collect metrics: number of pull requests evaluated, issues flagged, issues resolved automatically, and user satisfaction. Share this dashboard with your team to build trust.

Common Mistakes

Over‑relying on the Agent

The agent cannot catch everything. Expecting it to replace human review leads to missed barriers—especially those requiring subjective judgment (e.g., proper reading order in complex layouts). Treat the agent as a first pass that frees up time for deep testing.

Setting Too Broad a Scope

Attempting to solve every WCAG criterion at once overwhelms the LLM and confuses engineers. GitHub's agent succeeded because it started with simple, objective checks. Start small, then expand as you learn.

Ignoring False Positives

If the agent flags code that is actually correct (e.g., a hidden ARIA label being redundant), developers lose confidence. Log all false positives and adjust your prompt or add exclusion rules. For example, skip elements that already have aria-label when checking for accessible names.

Neglecting the Social Model of Disability

Remember that barriers arise from how environments are built, not from users themselves. The agent should be framed as a tool to remove those barriers, not to “fix” users. Use language that aligns with the social model, as GitHub does.

Summary

Building an accessibility agent is a practical way to automate the low-hanging fruit of inclusive design. By defining clear goals, setting up pull request triggers, and focusing on common issue types, you can reduce friction for both engineers and users. Avoid common pitfalls like over-reach and ignoring false positives. With continuous feedback, your agent will become a trusted partner in your accessibility journey—just as GitHub's has for over 3,500 pull requests.