# External Contents
As a Dydu bot manager, you have the ability to centralize and organize your external content sources directly from an intuitive interface in the BMS, allowing you to generate instant responses based on these sources and thereby improve the quality of responses provided to end users. Through the BMS navigation menu, you can access the External Content page: Content > External Content.
## **Create a collection**
By clicking on the "Your Collections" modal, you need to choose the name of the collection you want to create, and then click on "Create."
A page for your collection is displayed as shown below:
## **Feed the collection**
### Import files
It is possible to import multiple documents of the following types: PDF, DOCX, PPTX, TXT.
### Register Sharepoints
* The Dydu SharePoint reader enables the indexing of pages and files.
* Please refer to [this documentation](https://learn.microsoft.com/en-us/azure/healthcare-apis/register-application).
{% hint style="info" %}
You need to register a new application in your tenant that has read permissions.\
The tutorial below explains the process:\
\
When you reach the "API permissions" step, the two necessary permissions are:
* Microsoft Graph -> Application Permissions -> Files.ReadAll (Grant Admin Consent)
* Microsoft Graph -> Application Permissions -> BrowserSiteLists.Read.All (Grant Admin Consent)\*
{% endhint %}
* Set the permissions Files.ReadAll and BrowserSiteLists.Read.All for the Dydu application.
* The required elements for the configuration are:
a. clientId
b. client Secret (the value)
c. tenant Id
d. SharePoint site ID
#### Details of the necessary steps on how to retrieve the required values from Azure for the Dydu LLM configuration:
Go to the [Azure](https://portal.azure.com/#home) portal:
1. Click on App registrations
1. Click on New registration
3. Give a name and click on "Register"
4. The application ID is the client\_id
5. Click on Certificates & secrets. Then, in the "Client secrets" tab, click on New client secret.
6. Click on Certificates & secrets
7. Copy the generated secret value (client\_secret)
8. Click on API permissions. Then click on Add a permission.
9. **Click on Microsoft Graph**
10. Then click on "Application permissions". Then add the Sites.Selected and Files.Read.All permissions.
11. Click on Grant admin consent for XXXX
12. To find the tenant ID:
Go to the website:
Click on "Overview":
The client ID corresponds to the tenant ID.
13. To find the SharePoint ID:
Compose the following URL: https\://\.sharepoint.com/sites/\/\_api/site/id
The SharePoint ID can be found in the result
```xml
67a90b63-3384-495d-9456-66141cf4ac28
```
* **Features:**
\
1\. Indexing pages and/or files from an entire SharePoint site:
* Standard RAG
* Displaying the original SharePoint URL in the result provided by the RAG
2. Possible integration with SAML authentication:
* A user must be authenticated via SAML
* We retrieve their group memberships
* Document permission filtering is possible: access to a subset depending on the access rights.
{% hint style="info" %}
**Not indexed:**
* "Embedded" files in pages
* Videos, and some other types (Excel, WMF, ...)
Currently, the process of retrieving documents and indexing takes time (several minutes), and the most frequent refresh is once a day.
{% endhint %}
### Register Smart Tribune
To configure a Smart Tribune source, the following informations are required :
* Name: URL of the Smart Tribune API to use
* API Key
* API Secret
* Knowledge base ID list : refers to the IDs of the knowledge bases to retrieve (the same API key / API secret combination can grant access to multiple knowledge bases.)
Using these informations, all documents contained in the specified knowledge bases are retrieved, regardless of their original channel (FAQ).
### Register Websites
Type of Websites:
* Domain
* Sitemap
* Specific URLs
The informations about adding your source to your collection are displayed as follows:
* Name: the name of your source that you added
* Added by: the bot manager's identifier
* Created at: the date when you added your source
* Status: the status of your source
There are three states for the status:
1. "Waiting for action": status when no action has been taken.
2. "Completed": status when the operation (indexation or suggestion) is successful.
3. "Completed with errors": status when the operation (indexation or suggestion) was completed, but there are errors.
4. "Processing": status when the operation is ongoing.
* Action: the actions you can perform on your added source > delete, edit.
### **Suggestion and Indexation**
* Suggest knowledge from the collection
* Indexation: index the content of the collection
### **D**etails of items in the collection with the status "Completed with errors"
After indexing or suggestion has been performed, it is possible to obtain a status of **"Completed with errors"**.
By clicking on the status, a report is displayed with error details.
* **Details of Errors from Websites:**
In the report details, a **success and error percentage** is shown.
A **breakdown of HTTP error codes** is displayed.
The errors can be classified into different categories, such as **server-side issues or others**.
* **Details of Errors from SharePoints:**
In the report details, a **success and error percentage** is also displayed.
The report provides **comprehensive information about all pages that could not be retrieved**, as well as the affected folders.
It also specifies, for each folder, the **specific files that could not be retrieved**, making it easier to identify missing elements.
## **Collection configuration**
### Customizing responses
**Configuring the indexing parameters of a collection** allows you to precisely adapt the bot’s behavior to your business needs and the desired user experience. Each collection has a **dedicated card** where you can adjust several options to optimize the **relevance**, **length**, and **style** of generated answers, as well as the **selection of information sources**.
* **Temperature** defines the style of the bot’s answers: the higher the temperature, the more creative the answers can be; conversely, a low temperature favors strictly factual answers. This setting is especially useful to ensure that the tone and level of creativity of the bot match your usage context.
* **Number of output tokens** refers to the length of generated answers. You can choose between short, medium, or detailed answers depending on the complexity of the topics covered or your users’ preferences. Adjusting this parameter helps deliver more concise or, on the contrary, more in-depth information.
* **Minimum score required for answer sources** lets you filter the documents used by the bot: only sources with a score equal to or higher than the defined value will be considered in generating answers and displaying cited sources. This setting ensures that only sources deemed sufficiently relevant or reliable are used to build the answer.
* **Additional prompt** gives you the possibility to add specific context or an instruction that will always be considered when generating answers for the relevant collection. This free-text field allows you, for example, to impose a tone, specify a business instruction, or guide the bot on a sensitive topic.
* The **flexible management of the additional prompt** feature provides better control over the final prompt sent to the model. It allows users to view the complete final prompt and choose the precise placement of their additional prompt: at the beginning, in the middle, or at the end.
{% hint style="info" %}
You cannot modify the content of the final prompt itself; you can only insert the additional prompt and define its position to optimize the model's response.
{% endhint %}
### Dynamic variables
**Dynamic variables** can be used in the additional prompt of each collection. For example, **`${capture.user_name}`** is automatically replaced by the actual value retrieved during the conversation or from a web service.
If a variable is not available, it is ignored or replaced by an empty string.\
This makes it possible to **personalize the instructions** sent to the RAG engine, resulting in answers tailored to each user’s context.
In order for the capture variables to be correctly replaced in the prompt, they need to be added to the parameters of the Web service: Dydu\_RAG.\
Here is an example with the capture variable `user_name`:
{% hint style="warning" %}
When the prompt is sent to the **RAG engine**, it no longer contains the variable in the form **`${capture.XXXX}`**, but directly its value.
For example, a prompt like:
* "*Give me the value contained in ${capture.city}*"
Will be sent to the engine as:
* "*Give me the value contained in Paris*"
If the variable **`${capture.city}`** contains "Paris".
{% endhint %}
### Contextualizing RAG with metadata
It is possible to precisely target the documents used by the RAG to generate a response. To do this, you can filter content using the metadata associated with each document (such as a URL or a category). All metadata can be used for this filtering, with the exception of the score.
Example of metadata:
{% hint style="info" %}
It is possible to view the metadata of your documents by clicking the button  of an indexed collection.
{% endhint %}
The configuration is done directly in the webservice named Dydu\_RAG. Simply add a new parameter titled metadataFilters. The value of this parameter must be entered in this format: \[{"key": "key", "operator": "operator", "value": "value"}].
Three operators are available to define your filter:
* EQUALS: keeps only the content exactly matching the value.
* NOT\_EQUAL: excludes content exactly matching the value.
* SUB\_STRING: keeps content that includes all or part of the value.
For example, to limit the bot exclusively to Dydu product pages, use the SUB\_STRING operator on the URL as follows: \[{"key": "url", "operator": "SUB\_STRING", "value": ""}].
### Displaying the RAG Score in Responses
To display the **metadata score** attributed by the RAG (Retrieval-Augmented Generation) model for each response provided, it is necessary to modify the `Dydu_RAG` **webservice** and integrate the information into the **response format**.
To retrieve the **RAG score**, the variable must be extracted from the webservice's return JSON.
Add the following line **inside the JSON** structure to **extract the score value**:
```
var score = returnedJson.metadata[i]["score"];
```
After extracting the value, you must define the **display variable** and format how the score will be presented.
This code checks for the existence of the score and formats it for display, for example, by adding a line break and the label:
```
var scoreDisplay = "";
// Formatting the score for display (e.g.: "Score : [value]")
if (score != undefined && score != null) {
scoreDisplay = " Score : " + score;
}
```
This completes the configuration; the response score will now be displayed with every answer generated by the RAG.
***
**Properly configuring these parameters** allows you to obtain **relevant, reliable, and tailored answers**, while maintaining control over how the bot interacts with your users for each indexed data collection.
## **Content management**
### **Automatic reindexing**
You can configure the **reindexing frequency** of collections using four modes: **none**, **daily**, **weekly**, or **monthly**.
* **None**: no reindexing is scheduled, data remains unchanged.
* **Daily**: reindexing is performed automatically every day at midnight.
* **Weekly**: reindexing takes place every Monday at midnight.
* **Monthly**: reindexing is performed on the first Monday of each month at midnight.
The day and time of reindexing are predefined and cannot be changed.\
This configuration allows you to adjust the **data update frequency** to your needs, while keeping the process simple and automatic.
### Content access optimization
This option provides fast responses while maintaining high accuracy. It is essential when processing a large volume of data.
However, it may be less effective if your knowledge base is small. It is therefore recommended to test it first to verify its effectiveness on your content using the "Test the RAG" feature.
{% hint style="warning" %}
This option is not enabled by default.
{% endhint %}
