Relevancy check task

The relevancy check task checks whether a passage matched by your NLP questions is relevant to the chatbot user's question. The relevancy check response is never visible to your chatbot users.

You can configure the task to only check for relevancy on passages from specific conversations or only when the user's question has been rewritten by the rewrite utterance task, and specify fallback passages to start when the matched passage isn't relevant.

Relevancy checks can also make use of additional context from annotations in your conversations and passages.

The relevancy check task is in the TrueIntent usage. You need an administrator or publisher role in your team to create a usage.

By default, new usages you create are set as the primary usage (the usage the chatbot will use). You can create a usage without setting it as primary if you need to.

When you create the TrueIntent usage, the relevancy check task is disabled. Disabled tasks won't appear in your live chatbot, but you can still configure and test them using automated tests. When you're ready to use the relevancy check task in your live chatbot, enable the task and publish the change.

When editing tasks and usages, check the top of the usage page to see if it is the primary usage and whether your changes must be published to take effect. If there is no message indicating your changes must be published, then the changes are applied immediately and will not appear in versions.

The relevancy check response will not be visible to your chatbot users but may impact their experience when the chatbot is matching their questions, so you should test your changes thoroughly before you publish. You can create an automated test to test your prompt against a variety of utterances and generate improvements.

You can also test the task by asking questions in the TestBot once the task is enabled. Relevancy check responses are shown in the test bot logs as LLM Action: This answer was rated as Helpful by LLM, or LLM Action: This answer was rated as Unhelpful by LLM. The relevancy check task is not used during regression tests.

To create usages and configure tasks, you must have:

• Generative AI configurations enabled for your chatbot by inGenious AI support.
• A connector to power the task. 
  You can see the list of connectors that support the task when you create the usage.

To create a TrueIntent usage and configure the relevancy check task:

1. Click Manage in the left navigation, then click LLM Usages. 
   
2. Click your primary TrueIntent usage.
   If you don't have a TrueIntent usage, create one:
   1. Click + LLM Usage.
   2. In the Type field, select TrueIntent.
   3. Optionally, edit the Name for the usage.
   4. If you don't want the chatbot to use this task yet, clear the Set As Primary checkbox.
   5. Select the Connector to use for the task or select Create compatible Connector to create a new connector.
      Only compatible connectors will be available.
      You must have an administrator role in your team to create connectors.
   6. Click Create.
3. Click the Relevancy Check tab.
4. Leave the Enable Relevancy Check field unselected for now.
   You can configure the task and test it in automated tests without enabling it.
   You'll need to enable the task if you want to test it in TestBot. 
   The task won't run in your live chatbot until you've enabled it and published that change.
5. By default, the inGenious AI platform waits a maximum of 5 seconds for the model to generate a result. If it doesn't receive a result within that time, it starts the passage that was matched by your NLP questions. 
   If you want to change this period:
   1. Expand the Timeout section.
   2. Update the number of seconds the inGenious AI platform will wait.
6. If you want to edit the default prompt used to generate the summary, type any edits into the Prompt area.
   Make sure you don't delete the prompt variables: ${userQuestion}, ${chatHistory}, ${botMessage}, ${trainingPhrases}, ${conversationAnnotation}, and ${passageAnnotation}.
7. If you want to include the transcript of the chat for context, select Populate ${chatHistory}.
   If this field is not selected, the ${chatHistory} prompt variable is blank.
8. Select whether to check for relevancy either:
   1. After every NLP match
   2. Only if the user's question was rewritten by the rewrite utterance task. 
9. If you want to check relevancy only when passages from specific conversations are matched:
   1. Select When the passage returned is part of the following Conversations.
   2. Click + Conversation, then select the conversation to include.
   3. If you want to specify a fallback passage, select Custom Fallback, then select the passage to start.
      If Custom Fallback is unselected, the chatbot will use the appropriate fallback or intellimem passage from your chatbot's settings.
10. Select the information to use when checking the passage's relevancy:
    • Passage Content.
    • Training Phrases in the passage questions.
    • Annotations you have added to passages and conversations.
      See Control when relevancy is checked.
11. Click Save.

Enable relevancy checks

The relevancy check task is only available to your live chatbot once you've enabled the task and published the change. To enable the task: 

1. Click Manage in the left navigation, then click LLM Usages.
2. Click the primary TrueIntent usage.
   Only the task from the primary usage will be used.
3. Click the Relevancy Check tab.
4. Click Enable Relevancy Checks.
5. Click Save.

You must publish the change for it to take effect.