NULL
 
Null
Home Computer Based Assessment Register for Testing Our Team Clients Contact us
NULL
Computer Based Assessment  
EPEC staff have been involved in computer-based testing since 1986 and have developed and used systems for various purposes such as
             High stakes licensure examination
             Selection and placement testing
             Scholarship assessments
             Workplace assessments
             Diagnostic testing

EPEC has three systems that support computer-based assessment:
NULL
EPEC - Offline
        o       A server-based system
        o       Includes comprehensive item banking, test compilation and test delivery both traditionally and adaptive
        o       A secure system managed by EPEC for clients
        o       Can be used as a seamless, end-to-end solution from the authoring of items to the delivery of tests and reporting
        o       If the same tests are to be delivered at different venues, the applications are securely ‘beamed’ and installed on the servers

EPEC - Prognoser
        o       A cloud-based online system
        o       Can be used to administer tests traditionally, adaptive and within Option Probability Theory (see below)
        o       Tests are pre-constructed and are delivered directly over the internet at the convenience of the test taker
        o       After a username and password is assigned, test takers log into the system and the tests are delivered
        o       Tests can be timed, set to activate at a certain time on a specific day or be untimed
        o       Tests are typically delivered as practice tests to candidates before exams or to pilot new questions
        o       Tests can be timed, set to activate at a certain time on a specific day or be untimed
        o       Tests can also be used for secure assessments

EPEC - Complete
        o       A cloud-based online system
        o       Includes all the functionality of EPEC - Offline, and also;
                    ▪       allows access to users to author, review and approve questions remotely over the internet
                    ▪       support online marking
                    ▪       web-based test delivery; traditionally, linear-on-the-fly or adaptive
        o       Numerous question types can be accommodated, e.g. multiple-choice, multiple-response, short-answer, essay and Likert through the powerful editor which allows for any symbol, character, image, table or video.
        o       Detailed information is stored with each question such as keywords, description, author, objectives, rubrics and a variety of statistics
        o       Changes to questions are tracked in detail by author, date and version with labels such as new, reviewed or approved
        o       Using powerful IRT methodology, tests with specific psychometric properties, e.g. maximum information around the cut-score, can be compiled according to a specific blueprint. This allows for parallel tests which can, for example, be administered to different cohorts in different time zones for security.
        o       Master copies of tests can be provided for paper-and-pencil delivery or the tests can be delivered online.
        o       Various types of reports can be provided, for example, score reports, real-time summary reports or detailed psychometric reports.


Traditional Testing

Traditional (also termed conventional) testing has been in use for a long time. In this type of testing questions are:
             Administered sequentially to test takers in the same or in randomized order
             Different formats of questions can be used, for example, multiple-choice questions (items) in which the test taker has to select an answer from several possible answers, short-answer questions or essay-type
             Multiple-choice questions are typically scored as correct or incorrect, but partial credit can also be allowed in the scoring regime
             Short-answer and essay-type questions are usually marked according to a marking guide and scored on a scale
Whether the tests are delivered as paper-and-pencil tests or online and test takers are seated in such a way that they can see each-others answer booklets or monitors, the questions in the same test can be randomized so that test takers are administered the same questions in different order. Alternatively “parallel” tests can be constructed so that test takers are administered different but “equivalent” tests. It should be noted that parallel is a strong requirement which does not simply mean that two or more tests should have the same blueprint and mean (average), but also that score points should be pairwise comparable. Such tests can be constructed by using information functions of Item Response Theory (IRT) – see our Scoring and Data Analysis Services page. Parallel tests are useful, for example, in situations where test takers are in different time zones and security is an issue.


Computer Adaptive Testing (CAT)

Computer Adaptive Tests (CATs) have been described as the most efficient mode of assessment, especially in achievement testing. In a CAT;
             An item is selected from a bank of calibrated items (questions with statistical information) and administered to the test taker
             If the item is answered correctly, the computer administers a more difficult item. If this item is answered incorrectly, the computer administers an easier item
This process is applied throughout the testing session, constantly adjusting to ascertain the test takers ability level. Irrespective of the test taker’s ability, the intelligent item selection algorithm ensures that items are increasingly targeted to the appropriate ability level resulting in precise measures for all test takers with a fraction of the number of items administered in conventional testing.
As the testing progresses the test taker’s ability estimate converges to a certain value and the precision of the measure increases (the measurement error becomes increasingly smaller). The figure below shows the pathway of a typical CAT. The circles represent the ability estimate and the vertical lines indicate the error margin. Note that items that were too difficult were initially administered (start from the left) and the test taker mostly responded incorrectly (red) resulting in a decreasing ability estimate. After a number of items were administered the ability estimates stabilised whilst the error range also became increasingly smaller.

NULL
EPEC staff has extensive knowledge and experience in simulation studies, construction and delivery of CATs dating back to the 1980s. EPEC’s Executive Director is the current President of the International Association for Computerized Adaptive Testing (IACAT; see the site here) and a Consulting Editor of the Journal of Computerized Adaptive Testing (JCAT).
EPEC can deliver different types of CATs through all three its systems mentioned above. For example, if different domains (sub-specialties or content areas) are to be examined, sequential CATs can be administered to ensure precise measures for each separate domain as well as overall.
Alternatively CATs within CATs can be administered in an integrated exam. Item exposure and usage can be monitored and other constraints such as content representation, termination rules and so on can be accommodated.


Option Probability Theory (OPT)
 

For decades there has been concerns about guessing in multiple-choice questions (MCQ) and people often claim that scores can be inflated as a result of guessing. The fact is that if a test taker has answered an item correctly, one cannot tell with certainty whether the test taker knew the answer or guessed.
Option Probability Theory (OPT) is a unique measurement paradigm, developed by the Executive Director since the late 1990s, that directly addresses this issue.
Instead of selecting a single alternative in a MCQ the test taker is asked to assign 100% to one or more possible answers. If the test taker is sure that say answer A is correct, 100% can be assigned to this answer and by default 0% will be assigned to all the other possible answers. If, however, the test taker is not perfectly sure about answer A, then say 80% can be assigned to it. In this case there is thus a 20% uncertainty which can be interpreted as guessing or even faking in psychological testing. The remaining 20% can be assigned to any one or more of the remaining alternatives in any way. If answer A is correct, the test taker has underestimated their knowledge somewhat, but if answer A is incorrect, it could indicate a serious misconception. Depending on the percentages assigned, the diagnostic feedback highlights strengths and weaknesses.
In addition to a performance score, the test taker also gets a “realism” score which basically captures the amount of guessing or uncertainty in the testing session. The test taker’s score and realism index is interpreted in a customized report.
OPT is operationalized through EPEC - Prognoser (see above) in which test takers use sliders to indicate the percentages assigned to different alternatives. Equal percentages are initially assigned to each answer. In the figure below the test taker was 70% sure that answer ‘A’ was correct and 20% sure that option ‘B’ could possibly be correct and 10% sure that option ‘D’ could be correct.

Null
This theory has already been used very successfully for diagnostic purposes and also for some high stakes testing. For example, if a borderline candidate sits an exam and assigns a high percentage to an incorrect answer, it indicates over confidence and a misconception/fallacy which may result in unsafe practice in their profession.
For a FREE demonstration, please Contact Us
.