RAG Infra-Docs Chat

the problem

My infrastructure documentation was accurate but scattered — the answer usually existed somewhere, and finding it meant remembering where. I wanted to ask a question and get an answer grounded in the actual docs.

architecture

Documents are chunked, embedded, and stored in Postgres with the pgvector extension, indexed with HNSW for fast approximate nearest-neighbor search. Retrieval is hybrid — it combines keyword matching with vector similarity so exact terms and semantic matches both surface. promptfoo drives an evaluation harness that the pipeline is measured against as it changes.

engineering decisions

• → pgvector instead of a dedicated vector database

  The corpus lives comfortably in Postgres. Keeping vectors in the same database avoids a second system to run, back up, and keep consistent.

• → Hybrid retrieval, not vector-only

  Infrastructure docs are full of exact identifiers — hostnames, flags, error strings. Pure semantic search fumbles those, so keyword retrieval runs alongside vector similarity.

• → promptfoo as a regression gate

  Retrieval quality is easy to break silently. An eval harness turns 'it feels worse' into a measurable, repeatable check before a change ships.

reliability & security

Because answers are only as trustworthy as their grounding, the promptfoo suite is the safety net: it catches retrieval regressions before they reach the point of me trusting a wrong answer.