Show HN: Semcheck – AI Tool for checking implementation follows spec

原始链接: https://github.com/rejot-dev/semcheck

Semcheck is a tool that leverages large language models to verify code implementations against specifications. It operates by defining semantic rules that link code files to specification documents, enabling the LLM to identify discrepancies. Semcheck supports various LLMs, including OpenAI, Anthropic, Gemini, and local options like Ollama and Cerebras. The setup involves creating a `semcheck.yaml` configuration file, defining rules that specify file inclusions, exclusions, specification paths (local or remote), and LLM-related settings. Semcheck runs once per rule, allowing for granular checks focused on specific areas of code. It can be integrated into pre-commit hooks or CI/CD pipelines (GitHub Actions) for automated verification. Semcheck can be executed on specific files, staged changes, or the entire project. Using the `-pre-commit` flag or setting the `fail-on-issues` flag to 'true' in the configuration file will cause the check to fail when it detects issues. It aims to be non-intrusive, requiring no modification to existing code or specification files.

The Hacker News post introduces Semcheck, a CLI tool using LLMs to verify code implementation against specifications, aiming to bridge the gap in "natural language programming." Inspired by using Claude to implement a GeoJSON data structure, the author found discrepancies arose when the LLM lost context with the RFC. Semcheck addresses this by formally defining checks runnable in CI/CD pipelines. The author experimented with "spec-driven-development," using Claude Code to generate specifications and implementations, then iteratively refining them. Semcheck's own self-testing revealed implementation issues and helped refine specifications. The tool also checks documentation consistency. The author noted that while LLMs excel at finding discrepancies, they can generate false positives due to their extensive world knowledge (e.g., outdated software versions). They also experimented with confidence scores and reasoning prompts to mitigate this, finding that while reasoning helps, "thinking" models don't significantly improve performance. Claude 4 and GPT-4 performed best. The author seeks feedback on potential usefulness and desired features.

原文

Semcheck is a tool that uses large language models to verify that your implementation matches your specification. Define semantic rules to describe how your code should align with your specification, then let Semcheck handle the comparison. Use it as a final check before committing or merging code.

Non-intrusive: no changes required to existing code or specification files
Bring Your Own Model: supports OpenAI, Anthropic, Gemini, Cerebras and Ollama (local)
Supports remote specification files (e.g., https://www.rfc-editor.org/rfc/rfc7946.txt)
Easy setup with semcheck -init

go install github.com/rejot-dev/semcheck@latest

Semcheck requires a configuration file. Generate one interactively using the -init flag:

This command creates a semcheck.yaml file. Edit this file to suit your project.

Example configuration:

version: "1.0"
provider: openai  # Options: openai, anthropic, gemini, ollama, cerebras
model: gpt-4.1
api_key: ${OPENAI_API_KEY}
timeout: 30
fail_on_issues: true

rules:
  - name: function-spec-compliance
    description: Check if functions match their specifications
    enabled: true
    files:
      include:
        - "**/*.go"
      exclude:
        - "*_test.go"
    specs:
      - path: "docs/api.md"
      # Remote specs are supported
      - path: "https://example.com/spec.md"
    fail_on: "error"

# Create a config file
semcheck -init

# Check all spec/implementation rules
semcheck

# Pass specific files that need checking,
# semcheck uses the rules to determine which comparisons need to be made
semcheck spec.md

# Both implementation and specification files can be passed
semcheck impl.go

# Run on staged files (pre-commit)
semcheck -pre-commit

# Use a custom config file
semcheck -config my-config.yaml file1.go

# You can also use double dash syntax for flags
semcheck --config my-config.yaml

# Show help
semcheck -help

Define rules that link specification files to implementation files. Semcheck runs the LLM once per rule, and in pre-commit mode, only for rules with modified files. For best results, try to keep the number of files per rule small, LLMs perform best with focused context.

Example rules:

rules:
  - name: "config-spec"
    enabled: true
    files:
      include:
        - "./internal/config/*.go"
      exclude:
        - "*_test.go"
    specs:
      - path: "config-spec.md"

  - name: "geojson"
    description: "Ensure GeoJSON implementation matches RFC 7946"
    enabled: true
    files:
      include:
        - "packages/geojson/src/*.ts"
      exclude:
        - "*.test.ts"
    specs:
      - path: "https://www.rfc-editor.org/rfc/rfc7946.txt"
    prompt: |
      Our GeoJSON implementation is incomplete; only check implemented features.

A Justfile is included for common development tasks.

# List available commands
just

just test
just test-coverage

Checking Semcheck with Semcheck

Semcheck uses itself to check that it has correctly implemented it's own specification. Run Semcheck without arguments to check that is has been correctly implemented.

Semcheck can be used in a GitHub Actions workflow to check for semantic issues in your code.

Available configuration options:

config-file: Path to the semcheck configuration file
fail-on-issues: Whether to fail the action if issues are found
semcheck-version: Version of Semcheck to use
go-version: Version of Go to use

on:
  pull_request:
    branches:
      - main

jobs:
  semcheck:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Run semcheck
        uses: rejot-dev/semcheck@main
        with:
          config-file: semcheck.yaml
          fail-on-issues: false
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} # or other provider API key, configurable in semcheck.yaml