## Why you are getting this [phpbotscout](https://gitlab.com/phpboyscout/phpbotscout) answers support questions in Discord from this project's published documentation, citing what it used. It is being calibrated against a set of real support questions, and the calibration says the corpus is not yet strong enough to depend on. That is a documentation finding, not a bot finding. Retrieval is working — it returns the best thing available. For a lot of questions the best thing available does not answer them. **This is not a request to write answers to the calibration questions.** Those questions are *indicators*: they show the shape of what people ask and where the docs run out. Writing thirty targeted answers would score well and help nobody. What is wanted is documentation that would have answered them as a side effect of being complete. ## What we measured here | | | |---|---| | Documentation pages indexed | **38** | | Retrievable chunks | **141** | | Tutorials / How-to / Reference / Explanation | **0 / 7 / 0 / 23** | **No pages** under `reference/`, `tutorials/`. 8 pages sit outside the four quadrants (about, branding, legal and similar) — fine in themselves, but they are not documentation. Chunk density 3.7 per page against an estate median of 6.4 — low, which usually means long sections under few headings; those retrieve poorly. **Specific to this project.** 23 explanation pages and **zero reference**. The eleven `rust/*` modules that decompose this toolkit each have their own ticket; nine of them have no pages in any Diátaxis quadrant at all, carrying only branding and about pages. This project is where a reader starts, so its reference tier and its pointers into the modules both matter. ## What the estate looks like Across the 52 indexed sources — 829 doc pages, 5,303 chunks: | Tier | Pages | Any at all | 10+ pages | |------|------:|-----------:|----------:| | How-to | 245 | 38 of 52 | **4** | | Explanation | 167 | 35 of 52 | **4** | | Reference | 150 | 14 of 52 | **3** | How-to is the healthiest tier. **Reference is the weakest and matters most** — it is what answers "what is this key, what does this flag do, what happens if I set it wrong", which is the bulk of what people ask a support bot. Tutorials are excluded from this table because the policy just changed and the old numbers measure the old policy — see below. ## The four gaps we actually observed **1. The "why" is deferred to specs, and specs are leaving.** Pages routinely justify a decision by citing a spec — "off by default (`0001` §5)", "see spec `0004`" — rather than stating the reasoning. Specs are being removed from repositories in favour of forge wikis, because a spec is a point-in-time design record rather than documentation. phpbotscout deliberately does not index them, and would not cite one at a user if it could. Anything currently explained only by a spec reference is, as far as the docs are concerned, not explained. The reasoning needs to live in an explanation page in its own words. **2. Constraints and limitations are not written down.** Documentation describes what the software does. People ask what it will not do. A real example: one project can only use a single AI provider for both chat and image generation, and asking "can I use Gemini for images and OpenAI for chat" has no documented answer — the correct answer is "no, that is not supported", and it appears nowhere. The bot handles this well *if the constraint is written down*: it is instructed that contradicting a question's premise is a legitimate answer. It cannot infer a limitation from silence. **3. Retrieval happens at heading level, not page level.** Pages are split into chunks at H2/H3 — a median of 6.4 chunks per page across the estate — and each chunk is retrieved and handed to the model **on its own**. Three consequences: - **Headings are the retrieval surface.** A heading that names the thing being asked about gets found. "Notes" and "Details" do not. - **A section must stand alone.** "As described above" or "see the previous section" is useless in a chunk that arrives without the page around it. - **A rationale buried mid-section under an unrelated heading is unreachable**, however well written. **4. Compound questions have no home.** Real questions span two things at once — "what happens when I enable X, and why is it opt-in?" — where the first half is a how-to and the second an explanation. Neither page answers both, and neither links to the other. ## What would make this project dependable Judged by whether a stranger's question could be answered from the docs alone: - **A reference tier** covering every configuration key, flag and command — what it is, what it defaults to, what happens when it is wrong. This is the single biggest gap estate-wide. - **An explanation tier that explains**, in prose, rather than pointing at a spec number. - **Limitations stated explicitly** — what is not supported, what combinations do not work, what is deliberately absent. A "Limitations" or "What this does not do" section is worth more to a support bot than another feature description. - **Headings that name the question**, not the document's internal structure. - **Self-contained sections** that survive being read in isolation. - **A tutorials tier, in the docs.** Newly mandated, and a reversal: tutorials previously lived only as blog posts with the repo carrying a link stub. They are being backported into the docs, which become canonical, with the blog posts kept and given a `rel=canonical` pointing at the docs page. Five projects already carry tutorial content in-repo — they were ahead, not out of line. ## Standardise the directory names while you are in there Coverage cannot be reasoned about across the estate while projects file the same tier under different names. Four non-canonical directories are in use: | Current | Should be | Where | |---------|-----------|-------| | `commands/` | `reference/` | keryx (68 pages), krites (19) | | `components/` | `reference/` or `explanation/`, by content | rust-tool-base (17), keryx (2) | | `concepts/` | `explanation/` | keryx (6), rust-tool-base (6) | | `tutorial/` | `tutorials/` | go/output (1) | The canonical four are `tutorials/`, `how-to/`, `reference/`, `explanation/`. Non-Diátaxis sections that are not part of the framework — `about/`, `legal/`, `contributing/` — are fine as they are. **Redirects matter here.** These are published pages with links pointing at them, including from other projects' docs and from blog posts. A rename without a redirect turns every one of those into a 404, and the bot will cite the new URL while older references rot. ## How to use the calibration questions They will be supplied as indicators. For each one, the useful question is not "how do I answer this" but **"why could a reader not have worked this out, and what is missing that would have let them?"** A question the docs genuinely should not answer — about an unreleased feature, another project, or something deliberately unsupported — is a valid outcome. Say so; that is a real finding and the bot is built to decline gracefully. ## Not in scope Do not add content solely to satisfy the bot. If a section would not help a human reading the docs, it will not help the bot either — it will just be one more thing to maintain and one more chunk competing in retrieval.