This feature is currently in an open beta. To request access for your organization, please reach out to your account executive. If you are unsure who your Account Executive is, please contact Skillable Support.
If you are already using scaled scoring on an external platform where you launch labs, or if you want to move into scaled scoring, this set up will help you to streamline the scoring and reporting of the labs using scaled scoring.
What is Scaled Scoring?
Scaled scoring is a crucial concept in standardized testing, ensuring accurate comparisons and fairness. This method is essential for high-stakes exams, where maintaining consistent standards across different test versions is paramount.
Scaled scoring adjusts raw scores to a common scale, often ranging from 0 to 1000 points. This adjustment ensures that scores are comparable across different versions of a test or assessment, even if the difficulty or content varies. For example, if one version of a test is slightly harder than another, scaled scoring helps to balance the scores so that they can be fairly compared.
Benefits of Scaled Scoring
-
Consistency: Scaled scoring maintains the “bar” at the same place, ensuring that the difficulty level remains consistent across different test versions. This consistency is crucial for high-stakes exams where maintaining a uniform standard is essential.
-
Fairness: By adjusting for variations in test difficulty, scaled scores provide a fair comparison of candidates’ performance, regardless of which version of the test they took. This ensures that no candidate is disadvantaged or advantaged by the specific version of the test they encounter.
-
Accuracy: Scaled scoring accounts for slight differences in test difficulty, leading to more accurate assessments of candidates’ abilities. This precision helps in making more informed decisions based on the test results.
-
Standardization: It enables standardized reporting of scores, which is crucial for institutions and organizations that rely on these scores for decision-making. Standardized scores facilitate comparisons across different groups and time periods.
Importance of Scaled Scoring for Labs and High-Stakes Exams
-
Equity: Scaled scoring ensures that all candidates are evaluated on a level playing field, regardless of which version of the test they take. This is particularly important in high-stakes exams where outcomes can significantly impact candidates' futures.
-
Dependability: By reducing the impact of random fluctuations in test difficulty, scaled scoring improves the reliability of scores. Reliable scores are essential for making consistent and dependable decisions based on test results.
-
Meaningfulness: Scaled scoring enhances the validity of scores by making them more meaningful and interpretable. Valid scores accurately reflect the abilities and knowledge of the test-takers, ensuring that the test measures what it is intended to measure.
Understanding Scaled Scoring Terms
Scaled scoring is set up at the Organization level and defines the scaled range used for every lab within the organization. Here are the key settings:
- Minimum Score: This is the lowest scaled score of the range.
- Maximum Score: This is the highest scaled score of the range.
- Passing Score: This designates the mark within the scaled score distribution that equates to a passing grade.
Scaled Scoring settings defined at the Organization level will propagate to all subsequent child organizations under the current Organization. This means it is easy to manage Scaled Scoring for all child organizations with a single configuration at the top Parent Organization.
Each lab needs to be configured to properly standardize it within the Scaled Scoring distribution range. These settings are managed within each Instruction Set and at each Activity referenced within that Instruction Set.
- Cut Score: Defined in the score options of the Manage Instruction Set or within the Activities manager. It determines the mark that equates to the passing scaled score. For example, if the scaled scoring range is between 0 and 1000 with a passing score of 700, and a lab has 10 activities all with a weight of 1 and a cut score of 6, then users who score 6 would equate to a passing scaled score of 700.
- Weight: Defined at each activity, it assigns the weight of this particular activity against all other activities. This means some activities can be designated with a higher weight if they are considered more valuable than others within the Instruction Set.
If Scaled Scoring is configured at the Organization level, the activity Score field is renamed to Weight. To set up a simple linear distribution where all activities are of equal weight, set each activity Weight to the same value, e.g., a value of 1.
Configuring Scaled Scoring Settings Quick Overview
Setting Up Scaled Scoring for Your Organization and within each Lab
Enabling Scaled Scoring requires a specific permission. If you or someone within your organization needs to set up Scaled Scoring, please reach out to your account executive.
Organization Setup
1. Access Organization Settings:
- In the Admin screen, locate the Organization tile, and then Find the Organization you want to set up with scaled scoring
- Once in that screen, scroll down to find the Scaled Scoring section
- Click on +Add Scaled Score Configuration
2. Configure Scaled Scoring:
-
Specify the desired scale for your scores (e.g., 0-1000).
-
Configure the passing score
-
Click on OK to save the configuration
-
This will enable scaled scoring for your organization and any child organization.
Any child organization can override the scaled scoring configuration if needed
Lab Setup
After navigating to the Instructions Editor by clicking on Edit Instructions on any Lab Profile.
- Manage Activities
- Navigate to the Activities tab
- Here you may review the weight of each Activity. Following up with the suggestion above, make sure that each activity has a weight of 1.0.
- At the top of the screen you can configure the Cut Score for the instruction set you are actively editing.
It is important to consider the Activities that are actually being used, because the maximum sum of weights (raw score) is used to calculate the scaled score.
- Manage Instruction Sets
Since the cut score may defer from different versions of the same test, we equipped the Instruction Set's manager to manage all the cut scores and being able to preview the scaled score. To do this:- Go to the Manage Instruction Set section.
- Here click on the Columns button
- Make sure that the Scaled Scoring related columns are showing, these are: Cut Score, Max Score, and Scaled Scoring
- Define the Cut Score for each Instruction Set or within the Activities manager.
- You may preview the scaled score by hovering on the eye icon.
- Click OK: To save the settings to ensure they are applied correctly within the lab.
Note that if you have set up scaled scoring, and you have decided to show a score report to learners and candidates, they will now see the report with the scaled score.
By following these steps, you can effectively implement scaled scoring in your labs and high-stakes exams, ensuring fair, reliable, and valid assessments for your users.