# Module 5 — Answer Density

AIVE can measure how structurally extractable a published item's answers are:
how much of its heading and `details` structure reads as a question, and how
much of that is answered immediately rather than buried further down. The
result is a deterministic, local, 0-100 score with a transparent breakdown.

This document states what the score measures, how it is computed, what
triggers a scan, and what the feature does **not** promise. Read the last
section before reading the score as more than it is.

## What the score is, and is not

The score is a structural heuristic. It says nothing about whether an answer
is factually correct, complete or well written, does not predict how any AI
system will treat the page, and is not a ranking or citation guarantee. Two
pages with identical structure score identically regardless of whether one is
accurate and the other is not.

## Eligibility

A post is scanned only when:

- Answer Density is enabled (`enable_answer_density`, default on);
- its status is `publish` — draft, pending, scheduled, private, trashed and
  auto-draft are never scanned or shown as scored;
- it is not a revision or an autosave;
- its post type is in the configured, currently-allowed list
  (`answer_density_post_types`, default `post` and `page`).

A publicly queryable type may be configured. `post` and `page` are always
offered, matching core's own inconsistency about marking `page` as
`publicly_queryable`; `attachment`, `revision`, `nav_menu_item` and `wp_block`
may never be configured. A password-protected published item **is** eligible:
scoring happens in administration and discloses nothing publicly.

Removing a type from the configured list does not delete its posts' stored
rows. It only stops new scans and hides those rows from the admin screen and
its summary until the type is added back; nothing is lost in between.

## Source and parsing bounds

Analysis reads the stored `post_content` only — never `the_content`, a
shortcode or a block render — parsed locally with a bounded, no-network
`DOMDocument` (`LIBXML_NONET`). Nothing is rendered, executed or sent anywhere.

Two bounds apply before classification, and both fail the same way — an
over-budget item saves an **error** result with no score at all, retryable
once the content changes:

- at most 256 KiB of the stored content is read;
- at most 2,000 elements are inspected.

Content that parses to nothing — empty, whitespace, or markup the parser
cannot recover a document from — saves a **successful** zero-question result
instead: a real page with no headings looks exactly like this too, and
neither case is a failure worth surfacing as one.

## Question detection

A candidate question is every non-empty `h2`-`h6` heading and every
`details > summary`, in document order. After normalisation (whitespace
collapsed, non-breaking space folded to a space), a candidate qualifies if it
ends in `?` or `？`, or opens with a recognised interrogative word in English
or German, or contains one in Chinese or Japanese — the same four-language
heuristic Module 3's FAQ detection and the Unit suite already exercise via
`Helpers::is_question_text()`. Repeated questions are **not** deduplicated:
each is a separate answer location.

`total_headings` counts every non-empty `h2`-`h6`, question or not; a
`summary` never adds to it.

## Answer boundaries and classification

A heading question's section runs from just after the heading to the next
heading at the same or a higher level. The classification looks at the first
non-empty node in that section:

- **direct** — a `ul`, `ol`, `table` or `dl`; a short paragraph (at most 200
  Unicode characters); a paragraph beginning with `strong` or `b`; or text
  opening with a bounded definition phrase ("X is", "X means", "X 是", "X
  とは", "X ist", where X is at most six tokens);
- **buried** — non-empty content that matches none of the above;
- **unanswered** — nothing eligible before the boundary, including when a
  *nested* heading is reached first: its subtree belongs to that nested
  question, not to this one.

A `details > summary` question is independent of any heading section. Its
answer is the `details` element's own text, excluding the `summary` and any
nested `details` subtree (to avoid double-counting an inner question). Any
non-empty answer is **direct**; an empty one is **unanswered** — a `details`
question is never **buried**. It counts toward the question total but not
toward `total_headings`.

## Score

```
coverage_raw     = question_count / max(total_headings, 1)
coverage_points  = min(coverage_raw, 1) * 40
quality_points   = direct_answers / max(question_count, 1) * 40
structure_points = 20 if direct_answers >= 1, else 0
score            = round(coverage_points + quality_points + structure_points)   // half-up, clamped 0-100
```

`question_count` includes both heading and `details` questions. Zero-question
content scores 0. Ties in the admin table sort by ascending title under the
site locale (via PHP's `Collator`, when the `intl` extension is available),
then ascending post id. The admin table and the single-scan AJAX response both
also show the raw coverage ratio (two decimal places), each point
contribution (one decimal place) and the algorithm version alongside the
score, so the displayed breakdown is never a number the stored score cannot
be checked against.

## What triggers a scan

- **Save.** `save_post` scans the item that was just saved, unless its stored
  content hash and algorithm version both already match — then nothing is
  rewritten and `scanned_at` is left as it was.
- **Cron.** The existing six-hour `aive_scan_answer_density` event scans up to
  50 eligible stale-or-never-scanned posts, oldest successful scan first,
  never-scanned first, then post id. There is no separate cursor: every
  attempt — success or failure — writes a fresh `scanned_at`, which is what
  moves an attempted row to the back of the same ordering for the next run, so
  repeated runs sweep the whole eligible set in turn.
- **On demand.** A single scan (`aive_scan_density`) or a bulk "Scan All
  Published Content" run (`aive_bulk_scan`, at most 10 posts per request,
  resuming from an opaque server-side job token that expires after 15
  minutes) from the Answer Density screen. Both are AJAX requests requiring
  `manage_options` and a dedicated verified nonce.

A post's row is deleted the instant it leaves `publish`, and when the post
itself is deleted. Disabling the module keeps every stored row but stops all
scanning and hides score claims.

## Storage

One row per post in `aive_answer_density` (`UNIQUE KEY post_id`). The numeric
columns (`total_headings`, `question_headings`, `direct_answers`,
`buried_answers`, `derivative_answers`, `score`) carry the legacy names the
Module 0 table shipped with; `question_headings` is the total qualifying
question count (headings **and** `details`) despite its name, and
`derivative_answers` is the unanswered count. `state` (`scanned` or `error`)
and `algorithm_version` are Module 5's own columns, added to the existing
table so a bounded cron sweep can tell the three outcomes apart in SQL without
decoding JSON per row. `details` is a JSON record with the algorithm version,
a SHA-256 hash of the raw input, the result state, diagnostics, counts and the
ordered per-question breakdown — never raw HTML.

## Admin screen

**SummitCite → Answer Density** shows the average score and scanned count
over successfully-scanned, currently in-scope rows only; a per-type breakdown;
a "Scan All Published Content" control with an accessible `<progress>` element
and a live status region; a type filter; and a paginated table, worst-first by
default — unscanned and errored rows sort ahead of every scored row, because
not having a trustworthy score at all is a stronger reason to look than a low
but known one. A scan error is shown as the fixed word "Error"; no stored
content, parse diagnostic or PHP message ever reaches the page. The same
aggregates are published through the existing `aive_dashboard_stats` filter as
`answer_density_average_score`, `answer_density_scanned_count` and
`answer_density_type_breakdown`.

## Known limitations

- The admin table's title tie-break is locale-aware only when the server's
  PHP has the `intl` extension loaded (`Collator` class present). Without it,
  ties fall back to the database's own column collation, exactly as before
  this correction.
- Making the tie-break locale-aware required fetching every eligible row for
  the current filter on each page view, instead of only the requested page,
  so that a tied pair of titles cannot land on the wrong side of a page
  boundary. This is still bounded by the number of eligible published items —
  the same count the "total" figure and the COUNT query it replaced already
  computed — but it is a larger single query than the previous
  `LIMIT`/`OFFSET` page fetch on a very large site.
- The bulk snapshot holds every eligible post id in server-side job state for
  the duration of the job; a very large site's snapshot is correspondingly
  large.
- Scoring is English/German/Chinese/Japanese-only. Content in another
  language is scored only through terminal question punctuation (`?`, `？`)
  and explicit `details`/`summary` markup; it has no reduced accuracy beyond
  that — it simply is not read for interrogative openings the heuristic does
  not know.