The relevancy check task checks whether the passage matched by your NLP questions directly answers the user's utterance. You can limit the check to be performed only when the matched passage belongs to conversations you specify. The relevancy check response is never visible to your chatbot users.
You need an administrator or publisher role in your team to change a task prompt.
The task contains one prompt:
-
Relevancy Check returns true if the bot message directly answers the user's question, or false if it doesn't.
Prompt variables: ${userQuestion}, ${chatHistory}, ${botMessage}, ${trainingPhrases}, ${conversationAnnotation}, ${passageAnnotation}.
Check the top of the usage page to see if this is the primary usage and whether your changes must be published to take effect. If there is no note indicating your changes must be published, then the changes are applied immediately and will not appear in versions.
You can use automated tests to test a prompt or model against a set of test cases and generate suggested prompt improvements.
Changes to this prompt may impact your users' experience when the chatbot is matching their questions, so you should test your changes thoroughly before you publish. In addition to automated tests, you can test the prompt by asking questions in TestBot. Relevancy check responses are shown in the TestBot logs as LLM Action: This answer was rated as Helpful by LLM, or LLM Action: This answer was rated as Unhelpful by LLM.
Prompt variables and requirements
The prompt uses the following prompt variables:
- ${userQuestion}: the original user utterance to rephrase.
-
${chatHistory}: the transcript of the chat so far, which can be included to add more context to the chatbot user's utterance.
You can control whether the chat history is included using the Populate ${chatHistory} field. - ${botMessage}: the bot message content of the matched passage.
- ${trainingPhrases}: the NLP training phrases of the matched passage's inbound questions.
- ${conversationAnnotation}: annotations added to the conversation the matched passage belongs to.
- ${passageAnnotation}: annotations added to the matched passage.
You don't need to provide a value for these variables: they're automatically supplied when the chatbot uses the task to check a matched passage. Make sure you don't change or delete any of the prompt variables.
Suggested customisations
The default prompt instructs the model to return only 'true' or 'false'. You should keep this instruction as it is to ensure the relevancy check feature functions properly, but you may like to adjust the evaluation instructions to be more specific to your needs. Consider your passage content and the types of changes that would best help the LLM model determine whether a passage answers a user's question:
- Add your organisation's context, such as a banking organisation or city council.
- Request that irrelevant details such as dates, email addresses, or phone numbers are ignored when considering the relevancy of the response.
- Provide specific instructions if the utterance contains a specific term or the intent of the utterance matches that term.
This technique can help provide a more accurate relevancy check for terms relating to regulatory compliance or other highly sensitive issues, where the chatbot should respond with a passage that does not directly address the question. You can also limit the relevancy check to specific conversations.