​

GitHub Integration

Install the 2027-evals GitHub App to automatically run evaluations whenever a PR deploys a preview.

​

How it works

1. You open a PR that changes your docs
2. Your deployment platform (Vercel, Netlify, Mintlify) creates a preview deployment
3. 2027-evals detects the preview URL from the deployment status
4. An eval runs against the preview deployment
5. Results are posted as a PR comment with grade, score, and comparison to the previous eval

​

Privacy

The GitHub App requires zero code access. It only needs:

• Deployment status — to detect preview URLs
• Commit statuses — to detect deployments from platforms like Mintlify
• Pull requests — to post eval result comments

Your source code is never read or accessed.

​

Setup

​

1. Install the GitHub App

Go to your org’s Settings page on 2027.dev/evals and click Connect GitHub. This installs the 2027-evals GitHub App.

​

2. Select repositories

Choose which repositories should trigger evals. Typically this is your docs repo.

​

3. Link repos to prompts

In your prompt settings, add the GitHub repository under GitHub Repos. This tells the system which eval to run when that repo gets a PR.

​

4. That’s it

Next time you open a PR, you’ll see:

1. A “Waiting for preview deployment…” comment when the PR is opened
2. A “Running eval…” comment when the preview deploys
3. Final results with grade, score, and comparison when the eval completes

​

Supported platforms