Repo CI Strategy

cverna commented

2026-07-09 18:18:38 +00:00

Member

I have opened a PR proof of concept for running CI on this skills library.

The tool I used is https://github.com/stbenjam/skillsaw (there might be other), which offers a large set of rules we can use.

I would like to start the conversation on which rules we want to enforce, currently the default are:

AgentSkills:

agentskill-valid — SKILL.md must have valid frontmatter
agentskill-name — skill name must be lowercase/hyphens, match directory
agentskill-rename-refs — fix stale name refs after rename
agentskill-description — description must be meaningful and within length limits
agentskill-evals — validate evals/evals.json if present
agentskill-unreferenced-files — all bundled files must be referenced from SKILL.md

Commands / Agents / Hooks

command-frontmatter — command files need valid frontmatter
command-naming — command files must be kebab-case
agent-frontmatter — agent files need valid frontmatter
skill-frontmatter — SKILL.md needs frontmatter
hooks-json-valid — hooks.json must be valid JSON
hooks-dangerous — flag dangerous hook commands (curl|sh, eval, base64, etc.)

MCP

mcp-valid-json — MCP config must be valid JSON

Instruction Files

instruction-file-valid — AGENTS.md/CLAUDE.md/GEMINI.md must be valid and non-empty
instruction-imports-valid — @path imports must point to existing files

Content Intelligence (all auto, most warning)

content-embedded-secrets — no API keys/tokens (error)
content-weak-language — no hedging/vague language
content-tautological — no instructions the model follows by default
content-contradiction — no contradictory instructions
content-negative-only — prohibitions need a positive alternative
content-placeholder-text — no TODO/unfilled template text
content-banned-references — no deprecated model names/APIs
content-broken-internal-reference — no broken markdown links
content-critical-position — no critical instructions buried in the middle
content-redundant-with-tooling — no instructions duplicating ESLint/Prettier/etc.
content-instruction-budget — stay under ~150 instructions per file
content-section-length — sections under ~500 tokens (info)
content-hook-candidate — flag prose that should be a hook (info)
content-actionability-score — score verb density/commands (info)
content-cognitive-chunks — use headings to organize content (info)
content-inconsistent-terminology — consistent use of terms (info)
content-unlinked-internal-reference — wrap bare paths in markdown links (info)

Other

context-budget — instruction/config files within token limits
rules-valid — .claude/rules/ files must be valid markdown
coderabbit-yaml-valid — .coderabbit.yaml must be valid YAML
promptfoo-valid — promptfoo eval YAML must be valid
settings-dangerous — flag dangerous settings keys/env vars
apm-yaml-valid — apm.yml must be valid with required fields
apm-structure-valid — .apm/ must have valid structure
openclaw-metadata — validate metadata.openclaw fields
plugin-* rules — plugin structure, naming, readme (all auto)
marketplace-* rules — marketplace JSON validity and registration (all auto)

Rules are detailed in https://skillsaw.org/rules/#all-rules

I have opened a [PR proof of concept](https://forge.fedoraproject.org/ai-ml/skills-library/pulls/10) for running CI on this skills library. The tool I used is https://github.com/stbenjam/skillsaw (there might be other), which offers a large set of rules we can use. I would like to start the conversation on which rules we want to enforce, currently the default are: AgentSkills: - agentskill-valid — SKILL.md must have valid frontmatter - agentskill-name — skill name must be lowercase/hyphens, match directory - agentskill-rename-refs — fix stale name refs after rename - agentskill-description — description must be meaningful and within length limits - agentskill-evals — validate evals/evals.json if present - agentskill-unreferenced-files — all bundled files must be referenced from SKILL.md Commands / Agents / Hooks - command-frontmatter — command files need valid frontmatter - command-naming — command files must be kebab-case - agent-frontmatter — agent files need valid frontmatter - skill-frontmatter — SKILL.md needs frontmatter - hooks-json-valid — hooks.json must be valid JSON - hooks-dangerous — flag dangerous hook commands (curl|sh, eval, base64, etc.) MCP - mcp-valid-json — MCP config must be valid JSON Instruction Files - instruction-file-valid — AGENTS.md/CLAUDE.md/GEMINI.md must be valid and non-empty - instruction-imports-valid — @path imports must point to existing files Content Intelligence (all auto, most warning) - content-embedded-secrets — no API keys/tokens (error) - content-weak-language — no hedging/vague language - content-tautological — no instructions the model follows by default - content-contradiction — no contradictory instructions - content-negative-only — prohibitions need a positive alternative - content-placeholder-text — no TODO/unfilled template text - content-banned-references — no deprecated model names/APIs - content-broken-internal-reference — no broken markdown links - content-critical-position — no critical instructions buried in the middle - content-redundant-with-tooling — no instructions duplicating ESLint/Prettier/etc. - content-instruction-budget — stay under ~150 instructions per file - content-section-length — sections under ~500 tokens (info) - content-hook-candidate — flag prose that should be a hook (info) - content-actionability-score — score verb density/commands (info) - content-cognitive-chunks — use headings to organize content (info) - content-inconsistent-terminology — consistent use of terms (info) - content-unlinked-internal-reference — wrap bare paths in markdown links (info) Other - context-budget — instruction/config files within token limits - rules-valid — .claude/rules/ files must be valid markdown - coderabbit-yaml-valid — .coderabbit.yaml must be valid YAML - promptfoo-valid — promptfoo eval YAML must be valid - settings-dangerous — flag dangerous settings keys/env vars - apm-yaml-valid — apm.yml must be valid with required fields - apm-structure-valid — .apm/ must have valid structure - openclaw-metadata — validate metadata.openclaw fields - plugin-* rules — plugin structure, naming, readme (all auto) - marketplace-* rules — marketplace JSON validity and registration (all auto) Rules are detailed in https://skillsaw.org/rules/#all-rules