Automation Action: Embedded Knowledge Store

Update, Get, Search and Delete from a knowledge base store using embedded document database. This action enables you to store any number of knowledge base articles against a Collection Name. Articles can later be retrieved by their title, or you can search and retrieve the top x most relevant articles for any search text.

This action is useful when used with the ChatGPT action. When a user asks a question, you can search the knowledge base to provide context to the asked question. ChatGPT will then be able to answer organization specific questions based on private data.

You could also use this action to lookup knowledge base articles in response to incoming emails when sending automated responses etc.

Collection Name

Articles are contained within a Collection. Multiple collections can be used. Collection names can contain letters or numbers only. Article collections are global to the ThinkAutomation instance (IE: The same collection can be used on all Solutions/Automations). A collection can contain any number of articles - however it is recommended to keep articles per collection below 5000.

Operations:

Update

Add or update a knowledge base article within the Collection. Specify the Article Title, Tag and Article Text (these can contain %variable% replacements). If the title already exists then it's article text will be updated, otherwise a new article will be created. The Tag is optional.

The Assign To variable will receive the number of articles updated (or blank if an error occurred). You can use the Update Now button to manually add/update articles.

Update: Importing Document Files

You can optionally specify a Import Document File for the Update operation. This is a path to a document file (Word, PDF, HTML, Markdown, RTF, Email EML,Outlook MSG, Text or Excel files). If a file is specified the contents will be read, converted to plain text and added to the article text. When importing a document, the file name will be used as the Article Title if no title is specified.

Note: Imported documents will be split into pages and a separate article will be added for each page.

Update: Special Handling Of Markdown Files

When importing Markdown files (.md extension), the Markdown text will be split into articles based on # Headings (up to 4 levels) and the title of each article will be the full heading for all levels (eg: Main Heading, Heading Level 2, Heading Level 3).

Note: This operation may take several minutes when importing large documents with many pages, or large Markdown files. You can use the Knowledge Store Browser option to manually import documents into your Knowledge Store.

Update: Add Embeddings

This option is used to include embeddings with the article record. Embeddings are a list of numbers (a 'vector') that are used to measure the relatedness of text strings. This enables a much more accurate list of relevant articles when doing a search. If you are using the Embedded Knowledge Store action in conjunction with ChatGPT then you should enable this option. Enter your Open AI API Key (or specify a global OpenAI API Key in the Server Settings - Integrations section).

Adding Embeddings is not expensive (you could add embeddings for 1000 articles for less than $0.25).

If you do not add embeddings then the Search operation will only do a keyword match based search rather than a semantic .

Delete

Delete an existing article. Specify the Article Title to delete. The Assign To variable will receive the title value - or blank if the title does not exist. You can use the Delete Now button to manually delete articles.

Search

Search the Knowledge Base for relevant articles based on the Search For text. You can return the Top x most relevant articles - in relevance order. The Relevancy Threshold setting controls the relevancy level. Articles below the relevancy % will not be included. This value defaults to 75%.

Enable the Add Embeddings option if you added embeddings when the article was created. When this option is enabled then embeddings are obtained for the Search For text and then used to perform a semantic search of stored articles. This will provide a much more accurate list of relevant articles.

When using with ChatGPT you can specify the Max Tokens and Max Characters that should be returned in the search. This will prevent your requests from going over the token limit.

In the Return As list select either Text or Json. Specify Json if you are searching for articles to add as context for the ChatGPT action. The Assign To variable will be set to the returned articles.

You can optionally assign the top (most relevant) title and tag to variables, select from the Assign Top Title and Assign Top Tag to lists.

You can also perform a Knowledge Base search on the ChatGPT action itself when adding context to a conversation.

Get

Retrieve an existing article. Specify the Article Title. The Assign To variable will receive the article text - or be set to blank if article does not exist. You can use this option to lookup a specific article. In the Return As list select either Text or Json. Specify Json if you are looking up an article to add as context for the ChatGPT action. The Assign To variable will be set to the returned article.

List

Get a list of all article titles in the specified Collection Name. The Assign To variable will receive the returned titles (one per line). You can use the returned data in a For Each.. Line In loop if you need to perform actions on each value returned.

Drop

Removes all articles from the specified Collection Name. The Assign To variable will receive the collection name - or blank if the collection does not exist. You can use the Drop Now button to manually drop the collection (all articles in the collection will be deleted).

Manual Import And Editing

Click the Browse Knowledge Store button to open the Knowledge Store Browser. Here you can view articles for each collection. You can edit, add & delete articles. You can also test Searches and import multiple files & documents. You can also access the Knowledge Store Browser from the Studio - File menu.

You can also use the Web Spider action if you want to import an entire web site into your Knowledge Store.

Tips For Using With ChatGPT

Ensure the Add Embeddings option is enabled when adding articles to the Knowledge Store.

If you are building a Knowledge Store in order to provide context to a ChatGPT conversation you should ensure each article text is less than approximately 10000 characters. It is better to have lots of specific knowledge store articles than a few long ones (when importing files & documents into your Knowledge Store, these will be split into pages automatically - ensuring no single article text is > 10000 characters - or thereabouts).

If you have thousands of articles it is better to split them in to separate Collections based on some category. This will provide better performance during searches. For example, suppose you want to have a Chatbot that provides product information. You could ask for the product name at the start of the chat (using the Subject field). Then use this to lookup from the relevant Knowledge Base collection during the chat.

If you are adding product support type articles, or frequently asked questions, each article should be a question and answer, for example:

I have forgotten my password, what can I do?
Answer:
Click the forgot password link in the control panel 'Account' menu. This will ask for an email address. Provided the email address is registered on our system, a reset password email will be sent. Click the link in the email to reset your password.              

There may be occasions where you need to ensure a specific article is added to the context instead of relying on a search. For, example: If a specific keyword or combination of keywords exist in the incoming message. You can use the Get operation to lookup a specific article via it's title, and add this as context after adding context using a search. The context wont be added twice if the article was already included in the search.

If you want to add data related context for ChatGPT (for example, customer accounts/order information) - and the data is available via a Database/CRM etc, then you don't need to add this to the Knowledge Store. Instead, lookup the customer information via their email address from your database and add the context directly.

Using The Tag Value

The Tag is an optional text string you can add to articles. You can return the top tag when performing a Knowledge Store search. This can be useful if you want to perform specific actions within your Automation based on the tag value.

For example: Suppose you have several articles relating to 'service status'. You could add the tag 'service status' to each of these. When a search is performed, if the top tag returned is 'service status' your Automation could lookup your service status and add this as context.

General Tips For Building A Knowledge Store

You can use separate Automations to update the Knowledge Store. For example: You could use an Email message source to read emails from a specific 'knowledgebase' mailbox. Any emails coming into this mailbox could update the Knowledge Store - using the %Msg_Subject% as the title and the %Msg_Body% as the article text. You can then simply send/forward emails to that email address to add/update the Knowledge Store. Or you could use the File Pickup message source to monitor a folder - and simply add files/documents to the folder that you want adding to the knowledge store.