Setting up Scaled Scoring

If you are already using scaled scoring on an external platform where you launch labs, or if you want to move into scaled scoring, this set up will help you to streamline the scoring and reporting of the labs using scaled scoring.

What is Scaled Scoring?

Scaled scoring, a concept in standardized testing, ensures accurate comparisons and fairness across different test versions. This method is essential for high-stakes exams, where maintaining consistent standards is paramount.

Scaled scoring adjusts raw scores to a common scale, often ranging from 0 to 1000 points. This adjustment ensures that scores are comparable across different versions of a test or assessment, even if the difficulty or content varies. For example, if one version of a test is slightly harder than another, scaled scoring helps to balance the scores so that they can be fairly compared.

Benefits of Scaled Scoring

1. Consistency: Scaled scoring maintains the “bar” at the same place, ensuring that the difficulty level remains consistent across different test versions. This consistency is crucial for high-stakes exams where maintaining a uniform standard is essential.

2. Fairness: By adjusting for variations in test difficulty, scaled scores provide a fair comparison of candidates’ performance, regardless of which version of the test they took. This ensures that no candidate is disadvantaged or advantaged by the specific version of the test they encounter.

3. Accuracy: Scaled scoring accounts for slight differences in test difficulty, leading to more accurate assessments of candidates’ abilities. This precision helps in making more informed decisions based on the test results.

4. Standardization: It enables standardized reporting of scores, which is crucial for institutions and organizations that rely on these scores for decision-making. Standardized scores facilitate comparisons across different groups and time periods.

Understanding Scaled Scoring Terms

Scaled scoring is set up at the Organization level and defines the scaled range used for every lab within the organization. Here are the key settings:

1. Minimum Score: This is the lowest scaled score of the range.
2. Maximum Score: This is the highest scaled score of the range.
3. Passing Score: This designates the mark within the scaled score distribution that equates to a passing grade.

The above settings are configured at the organization level, but not all labs within that organization may need to leverage scaled scoring. For that reason, each instruction set requires configuration standardize it within the Scaled Scoring distribution range. Within the Instruction Set, a Cut Score is defined.

• Cut Score: Defined in the score options of the Manage Instruction Set or within the Activities manager. It determines the mark that equates to the passing scaled score. For example, if the scaled scoring range is between 0 and 1000 with a passing score of 700, and a lab has 10 activities all with a weight of 1 and a cut score of 6, then users who score 6 would equate to a passing scaled score of 700.

Within the Instruction Set > Activities panel, each activity's weight should be set.

• Weight: The weight of an activity, as compared with other activities.

Configuring Scaled Scoring Settings Quick Overview

Setting Up Scaled Scoring

Organization Setup

Users with Scaled Scoring Management permissions can configure Scaled Scoring for any organization they manage.

1. Access Organization Settings:
In the Admin screen, locate the Organization tile, and then Find the Organization you want to set up with scaled scoring

• Once in that screen, scroll down to find the Scaled Scoring section
• Click on +Add Scaled Score Configuration

2. Configure Scaled Scoring:

• Specify the desired scale for your scores (e.g., 0-1000).

• Configure the passing score
  

• Click on OK to save the configuration

• This will enable scaled scoring for your organization and any child organization.

Lab Setup

After navigating to the Instructions Editor by clicking on Edit Instructions on any Lab Profile.

1. Manage Activities

• Navigate to the Activities tab
• Here you may review the weight of each Activity. Again, it is recommended that each Activity has an equal weight for Scaled Scoring use cases.
• At the top of the screen you can configure the Cut Score for the instruction set you are actively editing.
  

2. Manage Instruction Sets
   Since the cut score may defer from different versions of the same test, we equipped the Instruction Set's manager to manage all the cut scores and being able to preview the scaled score. To do this:
   • Go to the Manage Instruction Set section.
   • Here click on the Columns button
     
   • Make sure that the Scaled Scoring related columns are showing, these are: Cut Score, Max Score, and Scaled Scoring
   • Define the Cut Score for each Instruction Set or within the Activities manager.
   • You may preview the scaled score by hovering on the eye icon.

• Click OK: To save the settings to ensure they are applied correctly within the lab.

A note on scoring types

Scaled Scoring works with both Activity Scoring and Activity Group scoring.

• For Activity Scoring, the cut score and maximum score are determined by the total weights of all of the activities within an instruction set.

• For Activity Group Scoring, the cut score and maximum score are determined by the total points awarded for passing for all activity groups within that instruction set.

By following these steps, you can effectively implement scaled scoring in your labs and high-stakes exams, ensuring fair, reliable, and valid assessments for your users.