Create an automated test for LLM Matching

Create an automated test for the LLM matching task to quickly test and improve your prompt.

You need an administrator or publisher role in your team to create automated tests.

Each automated test is created for a specific LLM task. When you create the automated test, the prompt and configurations of the task are copied into the automated test as default values. The test prompt and settings are independent of your task's prompt and settings: any changes you make in the automated test won't impact your task or usage, and any changes you make to the task or usage won't impact the automated test.

Automated tests require a CSV of initial test cases to generate the LLM task output. Once you've prepared your test cases, you can create the automated test.

Prepare your test cases

You''ll need a CSV of test cases to upload. Each row in the CSV must contain:

• A column with the utterance the chatbot user sent.
• A column with the chat history leading up to that utterance.
  If you don't have the chat history, create this column in the CSV but leave it blank for each row.

You can also upload result sets from other automated tests or include additional fields. You'll select which columns in your CSV contain the data when you upload, so make sure the first row is your column headings. The CSV can have a maximum of 1000 rows, and a maximum size of 50Mb.

To make the most of your testing, try to include a wide variety of utterances in your test cases and use real utterances and chat histories from your chatbot users if you can. You can download transcripts to collect real examples to test.

Create the automated test

To create an automated test for LLM matching:

1. Click Improve in the left navigation, then click Automated Tests.
   
2. Click + Automated Test.
3. Select the TrueIntent Usage and LLM Matching Task.
   You cannot change the task once the test has been created.
4. Type a Name for the automated test.
5. Click Create.
6. In the automated test page, click Upload CSV.
7. Select a CSV to upload.
8. Select the CSV columns for the chat history and user utterance.
9. If your CSV includes matched passages:
   1. Select I have Step 1 results 
   2. Select the column with the matched passage.
      You can choose to generate new matched passages instead when you run a test.
10. If your CSV includes additional metadata you'd like to use during evaluation:
    1. Select Include extra columns for evaluation criteria.
    2. Select each additional column you'd like to include. See Additional fields.
       You can select multiple columns. You'll tell the LLM model how to use these columns later.
11. If your CSV includes evaluations:
    1. Select I have Step 2 results.
    2. Select the CSV column with the 'PASS' and 'FAIL' evaluations.
       You can choose to generate new evaluations instead when you run a test.
12. Click Upload & Start at Step 1.
    If you've uploaded matched passages or evaluations, the test will start at a later step. You can still run the test from an earlier step if you want to.
13. Click Save.

You can now run automated tests for your LLM Matching task.

Additional fields

You can also add columns for:

• Matched passages generated by Step 1 in another automated test.
  You can skip Step 1 and evaluate these matched passages instead. Make sure the passages correspond to the original test cases.
• Additional metadata you want to use when evaluating matched passages in Step 2.
• Evaluations generated by Step 2 in another automated test.
  You can skip Steps 1 and 2 and generate insights using these evaluations instead. Make sure all the matched passages are marked as either "PASS" or "FAIL".