Coach education / Exam quality
Coaching Certification Exam IRT Service
For coach-education teams that run a written certification exam and want evidence on how well its questions work. A bounded review of one exam form using the response data you already have.
You keep the content expertise and the pass/fail decisions. I check which items behave as intended, which need review, and how consistently the exam measures.
Who this is for
An exam that matters, reviewed by the numbers.
Coaching certification exams are often written by experienced coaches and educators, then reused for years. Without routine item analysis it is hard to know whether a question is miskeyed, too easy to tell candidates apart, or measuring something other than the course content.
This pilot is for the person responsible for that exam: a coach-education manager, certification coordinator or assessment lead at a governing body, association or coaching-course provider.
Pilot scope
One exam form. One written report.
Deliverables
- Classical item statistics: difficulty, corrected item-total correlation and distractor use where option-level data exists.
- A Rasch (one-parameter IRT) calibration with item fit, and a test information summary across the score range.
- Score reliability, with its uncertainty explained in plain language.
- A prioritized list of items to review, each with the evidence behind it.
- A findings memo and a 45-minute walkthrough with your exam team.
Sample evidence / Simulated data
A check against known answers.
Before offering this, I tested the analysis on a fictional 40-item exam taken by 400 simulated candidates. Because the data was simulated, the true item difficulties were known, and two problem items were planted on purpose: one scored against the wrong key, one answered at random.
Entirely simulated. Figures are from one run; the same pass criteria held across 11 random seeds. This demonstrates the process on clean, model-consistent data; it is not a client case study and does not predict results on a real exam.
Process
Describe it. Check it. Review it.
- Describe the examNumber of items, candidates per window, item format and how scores are used. No data files at this stage.
- Feasibility and scopeWe confirm data format, de-identification, deliverables, fee and timing in writing before any data is shared.
- Report and walkthroughYou receive the memo and item list, then we review them together. Decisions about items stay with your team.
Limitations
What this pilot does not do.
- It does not set or change pass marks. Standard setting is a separate, panel-based process.
- It does not certify accreditation compliance or replace an independent psychometric audit.
- It does not write or rewrite exam items.
- It does not cover multiple-form equating, adaptive testing or bias (DIF) analysis unless separately scoped.
- It does not need, and should not receive, identifiable candidate data.
- It does not guarantee any outcome for candidates, courses or certification programs.
FAQ
Common questions.
How many candidates do we need?
It depends on the number of items and how varied candidate scores are. Classical item statistics are useful with fewer candidates than a Rasch calibration. The feasibility check answers this for your exam before you commit.
Why Rasch rather than a more complex IRT model?
For a single certification form and modest candidate numbers, the Rasch model is stable and easy to explain. If your data supports a richer model and it would change a decision, that can be discussed during scoping.
What data format works?
A spreadsheet or CSV with one row per candidate and one column per item, holding either the selected option or a 0/1 score, plus the answer key. Candidate identifiers should be removed or replaced with codes before sharing.
Who will do the work?
Nedim Yel, PhD — Measurement and Statistics. This is a new service; the simulated check above is the only example so far, and there are no previous coaching-exam clients to cite.
Request a scoped pilot.
Tell me about the exam: how many items, how many candidates per window, how scores are used and what prompted the question. No data files yet.
An inquiry doesn't book a pilot or create a payment obligation. Please don't send candidate data, exam content or passwords in a first message.