Setting up Scaled Scoring

If you are already using scaled scoring on an external platform where you launch labs, or if you want to move into scaled scoring, this set up will help you to streamline the scoring and reporting of the labs using scaled scoring.

What is Scaled Scoring?

Scaled scoring is a crucial concept in standardized testing, ensuring accurate comparisons and fairness. This method is essential for high-stakes exams, where maintaining consistent standards across different test versions is paramount.

Scaled scoring adjusts raw scores to a common scale, often ranging from 0 to 1000 points. This adjustment ensures that scores are comparable across different versions of a test or assessment, even if the difficulty or content varies. For example, if one version of a test is slightly harder than another, scaled scoring helps to balance the scores so that they can be fairly compared.

Benefits of Scaled Scoring

1. Consistency: Scaled scoring maintains the “bar” at the same place, ensuring that the difficulty level remains consistent across different test versions. This consistency is crucial for high-stakes exams where maintaining a uniform standard is essential.

2. Fairness: By adjusting for variations in test difficulty, scaled scores provide a fair comparison of candidates’ performance, regardless of which version of the test they took. This ensures that no candidate is disadvantaged or advantaged by the specific version of the test they encounter.

3. Accuracy: Scaled scoring accounts for slight differences in test difficulty, leading to more accurate assessments of candidates’ abilities. This precision helps in making more informed decisions based on the test results.

4. Standardization: It enables standardized reporting of scores, which is crucial for institutions and organizations that rely on these scores for decision-making. Standardized scores facilitate comparisons across different groups and time periods.

Importance of Scaled Scoring for Labs and High-Stakes Exams

1. Equity: Scaled scoring ensures that all candidates are evaluated on a level playing field, regardless of which version of the test they take. This is particularly important in high-stakes exams where outcomes can significantly impact candidates' futures.

2. Dependability: By reducing the impact of random fluctuations in test difficulty, scaled scoring improves the reliability of scores. Reliable scores are essential for making consistent and dependable decisions based on test results.

3. Meaningfulness: Scaled scoring enhances the validity of scores by making them more meaningful and interpretable. Valid scores accurately reflect the abilities and knowledge of the test-takers, ensuring that the test measures what it is intended to measure.

Understanding Scaled Scoring Terms

Scaled scoring is set up at the Organization level and defines the scaled range used for every lab within the organization. Here are the key settings:

1. Minimum Score: This is the lowest scaled score of the range.
2. Maximum Score: This is the highest scaled score of the range.
3. Passing Score: This designates the mark within the scaled score distribution that equates to a passing grade.

Each lab needs to be configured to properly standardize it within the Scaled Scoring distribution range. These settings are managed within each Instruction Set and at each Activity referenced within that Instruction Set.

• Cut Score: Defined in the score options of the Manage Instruction Set or within the Activities manager. It determines the mark that equates to the passing scaled score. For example, if the scaled scoring range is between 0 and 1000 with a passing score of 700, and a lab has 10 activities all with a weight of 1 and a cut score of 6, then users who score 6 would equate to a passing scaled score of 700.
• Weight: Defined at each activity, it assigns the weight of this particular activity against all other activities.

Configuring Scaled Scoring Settings Quick Overview

Setting Up Scaled Scoring for Your Organization and within each Lab

Enabling Scaled Scoring requires a specific permission. If you or someone within your organization needs to set up Scaled Scoring, please reach out to your account executive.

Organization Setup

1. Access Organization Settings:

• In the Admin screen, locate the Organization tile, and then Find the Organization you want to set up with scaled scoring
• Once in that screen, scroll down to find the Scaled Scoring section
• Click on +Add Scaled Score Configuration

2. Configure Scaled Scoring:

• Specify the desired scale for your scores (e.g., 0-1000).

• Configure the passing score
  

• Click on OK to save the configuration

• This will enable scaled scoring for your organization and any child organization.

Lab Setup

After navigating to the Instructions Editor by clicking on Edit Instructions on any Lab Profile.

1. Manage Activities

• Navigate to the Activities tab
• Here you may review the weight of each Activity. Again, it is recommended that each Activity has an equal weight for Scaled Scoring use cases.
• At the top of the screen you can configure the Cut Score for the instruction set you are actively editing.
  

2. Manage Instruction Sets
   Since the cut score may defer from different versions of the same test, we equipped the Instruction Set's manager to manage all the cut scores and being able to preview the scaled score. To do this:
   • Go to the Manage Instruction Set section.
   • Here click on the Columns button
     
   • Make sure that the Scaled Scoring related columns are showing, these are: Cut Score, Max Score, and Scaled Scoring
   • Define the Cut Score for each Instruction Set or within the Activities manager.
   • You may preview the scaled score by hovering on the eye icon.

• Click OK: To save the settings to ensure they are applied correctly within the lab.

By following these steps, you can effectively implement scaled scoring in your labs and high-stakes exams, ensuring fair, reliable, and valid assessments for your users.