Methodology

1. The Rule Engine

Our quality analysis is a coded implementation of Haladyna, Downing & Rodriguez (2002), AERA/APA/NCME Standards (2014), and ITC/ATP Guidelines (2022). 57 deterministic rules run first — no LLM, no hallucination. Each rule maps to a specific citation and produces a finding with severity, span location, and suggested fix.

2. AI-Assisted Analysis

For subjective assessments — Bloom's taxonomy classification, nuanced DIF analysis, rewrite suggestions — we use Claude (Anthropic) as a second-pass reviewer. The AI operates within a structured system prompt requiring JSON output, confidence intervals, and evidence-based reasoning. The AI never overrides a deterministic rule.

3. Difficulty Prediction

Text-only item difficulty prediction was validated as a BEA 2024 NAACL shared task using USMLE items (Tack et al., 2024). We position our predictions as pre-field triage signals with 80% confidence intervals — not as substitutes for field-test calibration. Current v0 baseline: RMSE ≤ 0.32 on held-out data.

4. The Evidence Packet

For batch engagements, we generate a 15–25 page branded PDF report covering: executive summary, methodology citations, per-item findings with prediction cards, aggregate statistics, and an appendix of cited works.

5. What's Orthodox vs. Frontier

6. Cited Works

• Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333.
• Haladyna, T. M. (2013). Developing and Validating Multiple-Choice Test Items (3rd ed.). Routledge.
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing. AERA.
• International Test Commission & Association of Test Publishers. (2022). ITC/ATP Guidelines for Technology-Based Assessment.
• Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Longman.
• Tack, A., et al. (2024). The BEA 2024 Shared Task on the Automated Prediction of Item Difficulty and Item Response Time. Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA), NAACL 2024.
• Holland, P. W., & Wainer, H. (Eds.). (1993). Differential Item Functioning. Lawrence Erlbaum Associates.
• Zumbo, B. D. (1999). A Handbook on the Theory and Methods of Differential Item Functioning (DIF). National Defense Headquarters, Ottawa.