Creating AI Vision Activities

What is AI Vision?

AI Vision, a new activity type in Skillable Studio, helps lab builders create high-quality, performance-based activities. Lab builders can leverage AI-powered computer vision technology to create natural language prompts that verify, based on a learner’s screen, whether the learner successfully completed an activity or set of activities within the lab.

Purpose

• Reduces technical skills required to create a scored activity. (No need to write scripts.)
• Enables builders to check for the outcomes that matter. (No more false negatives due to misspelled cell headers.)
• Supports more environments and software types. (Measure what's on a learner's screen, without worrying about whether the software has an API to call.)

Scenarios

• Lab builders who want to evaluate activities in practice, learning, or validation labs with visual components
• Lab builders who want to evaluate practice, learning, or validation labs on software products with limited API endpoints
• Lab building teams who want to reduce the time required to build quality labs
• Lab building teams without the technical skills required to write script-based activities

Key Elements

1. Prompt: Allows the lab author to create a prompt using natural language. This prompt determines what the computer vision technology will check on the learner's screen.
2. Test prompt: Enables the lab author to test a prompt using a 1-5 star rating and AI-generated feedback to help improve prompt strength and clarity.
3. (Optional) Reference image: Adds a reference image of the correct answer to further guide the computer vision technology as it checks a learner's screen.

4. Evaluation/Scoring: Uses computer vision to check a learner's screen when they click Check/Score in the lab, evaluating whether it matches the prompt and, if provided, the reference image.

Safeguards

• Ensure the AI prompt is clear and precise to avoid any ambiguity in evaluation.
• Validate the prompt's strength using the Test Prompt button.
• Use the feedback provided to refine the prompt and improve its effectiveness.

Create an AI Vision Activity

1. Navigate to your lab profile (which opens automatically after you create a lab) and click Edit Instructions.
2. Select the Activities tab and then click + New AI Vision Activity. This option is only available if your lab profile contains a virtualized environment.
3. Configure the prompt and then click Next.
4. Configure the activity settings.
5. Click Save or Save and Insert to finalize and create the activity.

Configure the Prompt

AI Vision activities work by comparing an image of a learner's screen (taken at the time of evaluation) to the contents of your prompt.

1. Choose the environment: Use the Choose Environment dropdown to select which VM the AI should evaluate.

2. (Optional) Provide a reference image: Upload a reference image for the AI to use when evaluating a learner's screen. Providing a screenshot of the correct response further informs the AI model and its evaluation.
3. Write the prompt: In the Write Prompt text box, write a clear prompt using natural language that describes what the computer vision AI should look for on the learner's screen. Include specific outcomes that indicate success in the activity.
4. Test the prompt: Click Test prompt to verify the strength of your prompt. If the AI-generated feedback shows that your prompt strength is weak (three stars or below), update it based on the feedback, then test the prompt again.

Configure the Activity Settings

1. Name: In the text field, specify the activity name. This is hidden from learners but visible to lab builders in the Activities tab and the Activities left pane.
2. Replacement Token Alias: In the text field, assign a recognizable name to the token for easier identification when reviewing a lab's Markdown code. A default alias is automatically generated.
3. (Optional) Skills: Tag the activity with relevant skills. This setting is hidden if your organization is not configured to use a Skills Framework.
4. (Optional) Instructions: Add instructions to display in the Instructions panel above Check/Score.
5. Evaluation: Use the checkboxes to set the activity's evaluation conditions:
   • (Required) On-demand evaluation: This option is checked by default because all AI Vision activities are currently only capable of on-demand evaluation. This allows learners to click a button in the lab instructions and receive feedback on their activity progress in real time.
   • (Optional) Custom evaluation button text: Specify the text of the evaluation button. If left unchecked, the buttons read Check for a non-scored activity and Score for a scored activity.
   • (Optional) Allow retries: Allow learners to retry and rescore the activity if they initially receive an Incorrect response. Use the corresponding Maximum attempts checkbox to limit the number of activity evaluation attempts.
6. Scoring: Use the Scored checkbox to determine whether the outcome of this activity will contribute to the learner's lab score. If checked, use the corresponding Score Value text field to set the point value the activity generates with a correct response.

7. Settings: Use the checkboxes to configure additional activity settings:
   • Show results in reports: Check to include the results of this activity in the end-of-lab detailed report.
   • Required for submission: Check to require learners to evaluate the activity before they can end the lab.
   • Blocks page navigation until answered: Check to disable the Next button.
8. Feedback for User: Use this section to define the feedback learners see:
   • Correct Answer Feedback: In the text box, specify the text a learner sees when a correct response is generated. If left blank, the text simply reads Correct.
   • Incorrect Answer Feedback: In the text box, specify the text a learner sees when an incorrect response is generated. If left blank, the text simply reads Incorrect.
   • Show AI Feedback for Incorrect Answers: This checkbox is checked by default so incorrect responses display AI feedback to learners. Uncheck this option if you do not want to provide AI feedback.
9. Outcomes: Click Add Outcome to configure adaptive learning experiences based on the result of the activity. See more on outcomes in our Activity Outcomes article.

AI Vision Activity Frequently Asked Questions

How do I manage or edit AI Vision activities?

Just as you'd do with all other activity types, navigate to the activity via Lab Profile > Edit Instructions > Activities. Find the activity you'd like to edit, and then click Edit. (Always remember to save!)

Lab builders can also move, clone, or delete activities from this view.

How do I use prompt feedback to improve my prompts?

To improve your prompts using the system-provided feedback:

• Review feedback: Carefully read the feedback and check for conciseness, specificity, clarity, and whether the prompt matches the image. This will give you a clear understanding of areas that need improvement.
• Analyze ratings: Look at the prompt strength rating out of 5 and the summary of possible corrections. This will help you identify the specific aspects of your prompt that need enhancement.
• Implement corrections: Make the necessary adjustments to your prompt based on the feedback. Focus on making your prompt more concise, specific, clear, and well-matched with the image.
• Mitigate errors: Address any error messages related to primary failures such as timeouts and content errors. Ensure that your prompt is correctly parsed by the AI.
• Structure output: If applicable, specify a structured output format, such as JSON schema, to ensure the response is returned in a consistent format.

Happy prompting!