Evaluate Test Cases task

The evaluate test cases task grades your LLM task output during automated tests. 

For example, if you created an automated test to improve your rewrite utterance task prompt, the evaluate test case task would help you quickly evaluate how well each utterance was rewritten using pass and fail criteria defined in the automated test. Each automated test defines its own pass and fail criteria.

The evaluate test cases task is in the automated tests usage. You need an administrator or publisher role in your team to create a usage.

By default, new usages you create are set as the primary usage (the usage the chatbot will use). You can create a usage without setting it as primary if you need to.

When editing tasks and usages, check the top of the usage page to see if it is the primary usage and whether your changes must be published to take effect. If there is no message indicating your changes must be published, then the changes are applied immediately and will not appear in versions.

You can test your task by running an automated test.

To create an automated tests usage, you must have:

• Generative AI configurations and automated tests enabled for your chatbot by inGenious AI support.
• A connector to power the task. 
  You can see the list of connectors that support the task when you create the usage.

To create an automated test usage and configure the evaluate test cases task:

1. Click Manage in the left navigation, then click LLM Usages.
2. Click the filter icon next to + LLM Usage and make sure Show Platform Usages is selected.
   
3. Click your primary automated tests usage.
   If you don't have an automated tests usage, create one:
   1. Click + LLM Usage.
   2. In the Type field, select Automated Tests.
   3. Optionally, edit the Name for the usage.
   4. If you don't want the chatbot to use this task yet, clear the Set As Primary checkbox.
   5. Select the Connector to use for the task or select Create compatible Connector to create a new connector.
      Only compatible connectors will be available.
      You must have an administrator role in your team to create connectors.
   6. Click Create. 
4. By default, the inGenious AI platform waits a maximum of 60 seconds for the model to generate a result. If it doesn't receive a result within that time, it notifies your content creator that the request timed out. 
   If you want to change this period:
   1. Expand the Timeout section.
   2. Update the number of seconds the inGenious AI platform will wait.
5. If you want to change the number of test cases the automated test uses when running a sampled test:
   1. Expand the Options section.
   2. Update the Number of Rows.
6. Click Save.