Setting Up Scaled Scoring

If you're already using scaled scoring on an external lab platform --or planning to start-- this article will help you streamline lab scoring and reporting for scaled scoring.

Overview

Scaled scoring, a concept used in standardized testing, ensures accurate comparisons and fairness across different versions of a test. It's especially critical for high-stakes exams, where maintaining consistent standards is essential.

Scaled scoring adjusts raw scores to a common scale, often ranging from 0 to 1000 points. This adjustment ensures scores are comparable even when test difficulty or content varies. For example, if one version of a test is slightly more challenging than another, scaled scoring helps balance the results to enable fair comparisons across test versions.

Scaled scoring works with both activity scoring and Activity Group scoring.

For activity scoring, the cut score and maximum score are based on the total weights of all activities within an Instruction Set.

For Activity Group scoring, the cut score and maximum score are based on the total points awarded for passing for all Activity Groups within that Instruction Set.

Benefits of Scaled Scoring

1. Consistency: Scaled scoring maintains a consistent “bar”, ensuring the difficulty level remains the same across different test versions. This is particularly important for high-stakes exams, where maintaining a uniform standard is essential.

2. Fairness: Scaled scoring adjusts for variations in test difficulty, providing a fair comparison of candidates’ performance regardless of which test version they took. This prevents candidates from being unfairly advantaged or disadvantaged by a particular test version.

3. Accuracy: By accounting for variations in test difficulty, scaled scoring provides a more precise measure of a candidate's abilities, which supports better decision-making best on test results.

4. Standardization: Scaled scoring standardizes score reporting, which is crucial for institutions and organizations that rely on test results for decision-making. It also enables meaningful comparisons across different groups and time periods.

Scaled Scoring Terms

Scaled scoring is configured at the organization level and defines the scaled range applied to all labs within the organization. Key settings include:

• Minimum Score: The lowest value in the scaled score range.
• Maximum Score: The highest value in the scaled score range.
• Passing Score: The mark within the scaled score range that represents a passing grade.

Although these settings are defined at the organization level, not all labs within an organization will require scaled scoring. For that reason, each lab Instruction Set must be configured to align with the scaled scoring distribution, defining a cut score and a weight.

The cut score represents the raw score that corresponds to the passing scaled score. The weight of an activity defines its relative value in calculating the overall score, compared to other activities. For example:

• Scaled score range: 0-1000
• Passing score: 700
• Lab activity weight: 10 equally weighted activities
• Cut score: 6

In the example above, learners who score a 6 would receive a passing scaled score of 700.

Configure Scaled Scoring on an Organization

Users with scaled scoring management permissions can configure scaled scoring for any organization they manage.

1. Access the organization settings:
   • From Skillable Studio, select the Admin tab.
   • In the Organizations tile, select Find Organizations.
   • Use the search features to find the organization you want to configure scaled scoring on.
   • From the organization details page, navigate to the Scaled Scoring section and click + Add Scaled Score Configuration.
     
2. Fill in the scaled scoring details:
   • Minimum Score: In the text field, enter the minimum score for your scaled score range (e.g., 0).
   • Maximum Score: In the text field, enter the maximum score for your scaled score range (e.g., 1000).
   • Passing Score: In the text field, enter the passing score for your scaled score range (e.g., 700).
     
3. After the scaled scoring details are filled in, click OK to save the configuration.
   This enables scaled scoring for the organization and any child organizations.

Configure Scaled Scoring on a Lab

1. From Skillable Studio, navigate to your lab profile.
2. Along the top of the lab profile, click Edit Instructions to open the Instructions Editor.
3. Select the Activities tab and review the weight of each activity. Again, we recommend assigning the same weight to each activity when using scaled scoring.
4. Along the top of the tab, use the text field to set the Cut Score for your instruction set.
   

Configure Scaled Scoring on an Instruction Set

Since the cut score may vary between different versions of a test, the Instruction Set manager is equipped to manage all cut scores and preview the resulting scaled score.

To manage cut scores through an Instruction Set:

1. From the Instructions Editor, use the Instructions Set dropdown to select Manage Instructions Sets.
2. In the Instructions Sets dialog, select Columns.
   
3. Ensure these columns related to scaled scoring are selected: Cut Score, Max Score, and Scaled Scoring.
4. Define the Cut Score for each Instruction Set or within the activities manager.
5. Hover over the eye icon to preview the scaled score, and then click OK to save your settings.
   

Conclusion

By following these steps, you can implement scaled scoring effectively in your labs and high-stakes exams—ensuring fair, reliable, and valid assessments for your users.