feat: scaffold tolkien v0 — local-LLM classifier wrapper + CLI + eval
Initial skeleton for the deterministic compound-AI pipeline that turns `tolkien deploy <name>` into a valinor scaffold. v0 wires one classifier (`db_kind`) end-to-end against the Ollama-on-radagast backend so we can iterate prompts with confidence before adding the rest. - pyproject.toml: uv project, Python 3.13, httpx/pydantic/typer/pyyaml. - src/tolkien/llm.py: single-turn /api/chat wrapper. JSON-mode + temp 0 + 30m keep-alive. Pydantic schema validation on parse. - src/tolkien/cli.py: `tolkien classify db-kind --doc <file>` runs the classifier and prints JSON. `tolkien deploy <name>` is stubbed. - src/tolkien/prompts/db_kind.md: tight system prompt with 1-shot example and explicit "return ONLY JSON" guard. - src/tolkien/schemas.py: DbKindResult pydantic model. - eval/cases/mealie.yml + run_eval.py: regression harness. Currently one case (mealie); failures print field-level diffs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
36
eval/cases/mealie.yml
Normal file
36
eval/cases/mealie.yml
Normal file
@@ -0,0 +1,36 @@
|
||||
# Eval case for the `db_kind` classifier.
|
||||
# The pipeline's correctness on real apps is gated on this kind of fixture —
|
||||
# re-run `eval/run_eval.py` after every prompt change.
|
||||
|
||||
name: mealie
|
||||
input: |
|
||||
App: mealie (recipe manager). Excerpt from official docker-compose docs:
|
||||
|
||||
environment:
|
||||
DB_ENGINE: postgres
|
||||
POSTGRES_USER: mealie
|
||||
POSTGRES_PASSWORD: change-me
|
||||
POSTGRES_SERVER: postgres
|
||||
POSTGRES_PORT: 5432
|
||||
POSTGRES_DB: mealie
|
||||
ALLOW_SIGNUP: "true"
|
||||
BASE_URL: https://mealie.example.com
|
||||
OPENAI_API_KEY: sk-... # optional, enables recipe parsing via OpenAI
|
||||
TZ: Europe/Amsterdam
|
||||
|
||||
expected:
|
||||
needs_postgres: true
|
||||
db_env_vars:
|
||||
- DB_ENGINE
|
||||
- POSTGRES_USER
|
||||
- POSTGRES_PASSWORD
|
||||
- POSTGRES_SERVER
|
||||
- POSTGRES_PORT
|
||||
- POSTGRES_DB
|
||||
secrets:
|
||||
- POSTGRES_PASSWORD
|
||||
- OPENAI_API_KEY
|
||||
config:
|
||||
- ALLOW_SIGNUP
|
||||
- BASE_URL
|
||||
- TZ
|
||||
Reference in New Issue
Block a user