RAGMap prototype review
Date: 2026-06-01
Script: scripts/ragmap_answer_prototype.py
Sample outputs: ragmap_answers/
1. What the prototype does
Runs a fixed question mode over regenerated data/graph_exports/ CSVs plus structured YAML and local Markdown. It:
- Applies the same default graph safety filters as topical-edge-audit.md (no topical/admin/affiliation for technical ranking; UNI-001 excluded from hubs/bridges).
- Computes graph answers (hubs, bridges, or per-resource evidence).
- Retrieves evidence from
evidence_text,evidence_file, repo Markdown paths, and YAML notes—without fetching URLs or calling an LLM. - Writes Markdown reports under
docs/reports/ragmap_answers/.
2. Supported questions
| Mode | CLI | Output |
|---|---|---|
| Technical hubs | python3 scripts/ragmap_answer_prototype.py technical_hubs |
ragmap_answers/technical_hubs.md |
| Bridge people | python3 scripts/ragmap_answer_prototype.py bridge_people |
ragmap_answers/bridge_people.md |
| Evidence for resource | python3 scripts/ragmap_answer_prototype.py evidence_for_resource CDM-016 |
ragmap_answers/evidence_for_CDM-016.md |
| Proposal support (keywords) | python3 scripts/ragmap_answer_prototype.py proposal_support "imaging machine learning" |
ragmap_answers/proposal_support_<slug>.md |
3. Data used
| Source | Role |
|---|---|
data/graph_exports/*.csv |
Nodes, edges, include_in_graph, evidence_level |
data/person_resource_links.yaml (and other link YAML) |
Full evidence_text (not in CSV) |
data/course_resource_links.yaml |
Documented course ↔ facility |
data/resource_equipment.yaml |
Equipment categories per resource |
data/search_synonyms.yaml |
Transparent synonym groups for proposal_support |
resources/**/*.md |
Snippets when evidence_file points to a repo path |
4. Evidence rules
- Order: edge
evidence_text→evidence_filepath → local Markdown excerpt → YAMLnotes/source_note. - No invention: if a field is empty, the report says so.
- URLs: listed as references only (not fetched).
- Default graph:
include_in_graph=true; exclude topical, administrative, excluded, affiliation from technical counts; person–resource hubs/bridges use direct only. - Not facility use: person–grant, person–publication, person–output alone (output–resource required for facility claim).
5. vs static website (MkDocs)
| Capability | Static site | Prototype |
|---|---|---|
| Browse one resource | Yes | Yes (evidence mode) |
| Rank hubs across units | No | Yes (technical_hubs) |
| Cross-prefix bridge people | No | Yes (bridge_people) |
| Aggregate evidence types per resource | Manual | Yes (evidence_for_resource) |
| Enforce graph policy consistently | Human reader | Automated filters |
6. vs Neo4j
| Neo4j | Prototype | |
|---|---|---|
| Runtime | Database server | Python + pandas |
| Queries | Cypher | Pandas + predefined modes |
| Multi-hop | Flexible | Fixed patterns only |
| Evidence | Properties + external RAG | YAML/Markdown retrieval in-script |
The prototype validates policy and evidence packaging before investing in Neo4j import (neo4j-import-plan.md).
7. vs full LLM RAG
| LLM RAG | Prototype | |
|---|---|---|
| Natural language | Arbitrary questions | Three modes only |
| Synthesis | Model-generated prose | Template + retrieved snippets |
| Hallucination risk | Requires guardrails | Low (no generation) |
| Citations | Must be enforced | Every section tied to link IDs / files |
Next step for LLM RAG: use these reports as gold structure and require the model to cite link_id / paths from retrieved chunks only.
Keyword proposal-support mode
What it does
proposal_support runs a transparent keyword search over resources, people, grants, publications, outputs, equipment text, course rationales, and resource Markdown. It scores matches with documented weights, optionally expands along graph-included edges, and writes docs/reports/ragmap_answers/proposal_support_<slug>.md.
python3 scripts/ragmap_answer_prototype.py proposal_support "environmental sensing"
python3 scripts/ragmap_answer_prototype.py proposal_support "imaging machine learning" --limit 10
python3 scripts/ragmap_answer_prototype.py proposal_support "student electronics fabrication" --direct-only
Flags: --include-topical, --direct-only, --limit N, --skip-export, --no-synonyms, --show-expanded-query.
Why it is useful for ORS and faculty
Supports proposal planning and resource discovery without browsing every unit page: surfaces facilities, PIs, grants, and outputs that already mention a theme, with why matched and evidence snippets.
Why it is not yet a full LLM RAG system
No embedding model, no generative synthesis. Queries use keyword overlap plus optional transparent synonym expansion (see below). Answers are structured reports, not conversational prose.
Graph filters and evidence
Default expansion uses include_in_graph=true, excludes topical/administrative/excluded edges, and applies +2 / +1 bonuses for direct vs affiliation neighbors. UNI-001 is deprioritized in resource rankings.
Limitations of keyword matching
- Cannot rank conceptual similarity (synonyms are hand-listed terms, not embeddings).
- Over-broad synonym groups can surface weak or noisy hits.
- Sparse grant abstracts in the seed layer limit recall.
Transparent synonym expansion
Why it was added
Proposal themes often use different words than the catalog (microscopy vs imaging, VR vs virtual reality). A fixed synonym file helps proposal_support find terms already present in YAML and Markdown without pretending to do semantic AI search.
How it differs from LLM semantic search
data/search_synonyms.yaml |
LLM / embedding RAG | |
|---|---|---|
| Mechanism | Literal substring match on listed terms | Vectors or model-inferred meaning |
| Editable | Human YAML groups | Opaque model weights |
| Evidence | Each hit shows query vs synonym terms and weights | Requires citation guardrails |
| Graph | Never creates edges | Must not be assumed safe |
Activation: if any original query token (length > 2) or the full phrase matches a term in a group, that concept activates and its other terms are searched—except terms already in the query.
Why synonym matches score lower
Exact phrase and query-keyword hits use the highest weights (+10 phrase in title, +5 query keyword in title/topics). Synonym hits use roughly half (+5/+3 phrase/keyword in title, +2/+3 in body). That keeps strong exact evidence above keyword-adjacent matches unless scores clearly justify ranking.
Graph bonuses remain small (+2 direct neighbor, +1 affiliation) and are labeled separately from synonym hits.
Editing data/search_synonyms.yaml
Add or trim groups with concept and terms lists. Re-run:
python3 scripts/ragmap_answer_prototype.py proposal_support "your phrase" --show-expanded-query
Use --no-synonyms to compare behavior without expansion.
Limitations and risks
- False positives: shared terms (e.g.
sensors,production,visualization) appear in multiple groups. - No new facts: synonyms only find text already in the repo; they do not infer expertise or facility use.
- Not collaboration: synonym co-occurrence must not be read as partnership or resource use.
- Maintenance: groups need curator review as the catalog grows.
Example reports generated
| Query | Report |
|---|---|
| environmental sensing | proposal_support_environmental_sensing.md |
| imaging machine learning | proposal_support_imaging_machine_learning.md |
| student electronics fabrication | proposal_support_student_electronics_fabrication.md |
| film production technology | proposal_support_film_production_technology.md |
8. What to add next
- Align synonym groups with
equipment_taxonomy.yamlwhere helpful (keep human-editable). - Export evidence_text in CSV to reduce YAML coupling.
- Batch mode writing all ragmap-question-examples.md prompts.
- Internal CLI or MkDocs plugin (still no public graph pages until policy review).
Regenerate answers:
python3 scripts/ragmap_answer_prototype.py technical_hubs
python3 scripts/ragmap_answer_prototype.py bridge_people
python3 scripts/ragmap_answer_prototype.py evidence_for_resource CDM-016
python3 scripts/ragmap_answer_prototype.py proposal_support "your keywords here"