## File: README.md ## Introduction This project serves as a functional RAG UI for both end users who want to do QA on their documents and developers who want to build their own RAG pipeline. ```yml +----------------------------------------------------------------------------+ | End users: Those who use apps built with `kotaemon`. | | (You use an app like the one in the demo above) | | +----------------------------------------------------------------+ | | | Developers: Those who built with `kotaemon`. | | | | (You have `import kotaemon` somewhere in your project) | | | | +----------------------------------------------------+ | | | | | Contributors: Those who make `kotaemon` better. | | | | | | (You make PR to this repo) | | | | | +----------------------------------------------------+ | | | +----------------------------------------------------------------+ | +----------------------------------------------------------------------------+ ``` ### For end users - **Clean & Minimalistic UI**: A user-friendly interface for RAG-based QA. - **Support for Various LLMs**: Compatible with LLM API providers (OpenAI, AzureOpenAI, Cohere, etc.) and local LLMs (via `ollama` and `llama-cpp-python`). - **Easy Installation**: Simple scripts to get you started quickly. ### For developers - **Framework for RAG Pipelines**: Tools to build your own RAG-based document QA pipeline. - **Customizable UI**: See your RAG pipeline in action with the provided UI, built with [Gradio ](https://github.com/gradio-app/gradio). - **Gradio Theme**: If you use Gradio for development, check out our theme here: [kotaemon-gradio-theme](https://github.com/lone17/kotaemon-gradio-theme). ## Key Features - **Host your own document QA (RAG) web-UI**: Support multi-user login, organize your files in private/public collections, collaborate and share your favorite chat with others. - **Organize your LLM & Embedding models**: Support both local LLMs & popular API providers (OpenAI, Azure, Ollama, Groq). - **Hybrid RAG pipeline**: Sane default RAG pipeline with hybrid (full-text & vector) retriever and re-ranking to ensure best retrieval quality. - **Multi-modal QA support**: Perform Question Answering on multiple documents with figures and tables support. Support multi-modal document parsing (selectable options on UI). - **Advanced citations with document preview**: By default the system will provide detailed citations to ensure the correctness of LLM answers. View your citations (incl. relevant score) directly in the _in-browser PDF viewer_ with highlights. Warning when retrieval pipeline return low relevant articles. - **Support complex reasoning methods**: Use question decomposition to answer your complex/multi-hop question. Support agent-based reasoning with `ReAct`, `ReWOO` and other agents. - **Configurable settings UI**: You can adjust most important aspects of retrieval & generation process on the UI (incl. prompts). - **Extensible**: Being built on Gradio, you are free to customize or add any UI elements as you like. Also, we aim to support multiple strategies for document indexing & retrieval. `GraphRAG` indexing pipeline is provided as an example. ## Installation > If you are not a developer and just want to use the app, please check out our easy-to-follow [User Guide](https://cinnamon.github.io/kotaemon/). Download the `.zip` file from the [latest release](https://github.com/Cinnamon/kotaemon/releases/latest) to get all the newest features and bug fixes. ### System requirements 1. [Python](https://www.python.org/downloads/) >= 3.10 2. [Docker](https://www.docker.com/): optional, if you [install with Docker](#with-docker-recommended) 3. [Unstructured](https://docs.unstructured.io/open-source/installation/full-installation#full-installation) if you want to process files other than `.pdf`, `.html`, `.mhtml`, and `.xlsx` documents. Installation steps differ depending on your operating system. Please visit the link and follow the specific instructions provided there. ### With Docker (recommended) 1. We support both `lite` & `full` version of Docker images. With `full` version, the extra packages of `unstructured` will be installed, which can support additional file types (`.doc`, `.docx`, ...) but the cost is larger docker image size. For most users, the `lite` image should work well in most cases. - To use the `full` version. ```shell docker run \ -e GRADIO_SERVER_NAME=0.0.0.0 \ -e GRADIO_SERVER_PORT=7860 \ -v ./ktem_app_data:/app/ktem_app_data \ -p 7860:7860 -it --rm \ ghcr.io/cinnamon/kotaemon:main-full ``` - To use the `full` version with bundled **Ollama** for _local / private RAG_. ```shell # change image name to docker run <...> ghcr.io/cinnamon/kotaemon:main-ollama ``` - To use the `lite` version. ```shell # change image name to docker run <...> ghcr.io/cinnamon/kotaemon:main-lite ``` 2. We currently support and test two platforms: `linux/amd64` and `linux/arm64` (for newer Mac). You can specify the platform by passing `--platform` in the `docker run` command. For example: ```shell # To run docker with platform linux/arm64 docker run \ -e GRADIO_SERVER_NAME=0.0.0.0 \ -e GRADIO_SERVER_PORT=7860 \ -v ./ktem_app_data:/app/ktem_app_data \ -p 7860:7860 -it --rm \ --platform linux/arm64 \ ghcr.io/cinnamon/kotaemon:main-lite ``` 3. Once everything is set up correctly, you can go to `http://localhost:7860/` to access the WebUI. 4. We use [GHCR](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) to store docker images, all images can be found [here.](https://github.com/Cinnamon/kotaemon/pkgs/container/kotaemon) ### Without Docker 1. Clone the repository: ```shell git clone https://github.com/Cinnamon/kotaemon cd kotaemon ``` 2. Setup the environment: - **Option 1: Using [uv](https://docs.astral.sh/uv/getting-started/installation/) (recommended)** ```shell uv sync --python 3.10 source .venv/bin/activate ``` - **Option 2: Using conda** ```shell conda create -n kotaemon python=3.10 conda activate kotaemon pip install -e "libs/kotaemon[all]" pip install -e "libs/ktem" ``` 3. Create a `.env` file in the root of this project. Use `.env.example` as a template. The `.env` file is there to serve use cases where users want to pre-config the models before starting up the app (e.g. deploy the app on HF hub). The file will only be used to populate the db once upon the first run, it will no longer be used in consequent runs. 4. (Optional) To enable in-browser `PDF_JS` viewer, download [PDF_JS_DIST](https://github.com/mozilla/pdf.js/releases/download/v4.0.379/pdfjs-4.0.379-dist.zip) then extract it to `libs/ktem/ktem/assets/prebuilt`. 5. Start the web server: ```shell python app.py ``` - The app will be automatically launched in your browser. - Default username and password are both `admin`. You can set up additional users directly through the UI. 6. Check the `Resources` tab and `LLMs and Embeddings` and ensure that your `api_key` value is set correctly from your `.env` file. If it is not set, you can set it there. ### Setup GraphRAG > [!NOTE] > Official MS GraphRAG indexing only works with OpenAI or Ollama API. > We recommend most users to use NanoGraphRAG implementation for straightforward integration with Kotaemon. Setup Nano GRAPHRAG - Install nano-GraphRAG: `pip install nano-graphrag` - `nano-graphrag` install might introduce version conflicts, see [this issue](https://github.com/Cinnamon/kotaemon/issues/440) - To quickly fix: `pip uninstall hnswlib chroma-hnswlib && pip install chroma-hnswlib` - Launch Kotaemon with `USE_NANO_GRAPHRAG=true` environment variable. - Set your default LLM & Embedding models in Resources setting and it will be recognized automatically from NanoGraphRAG. Setup LIGHTRAG - Install LightRAG: `pip install git+https://github.com/HKUDS/LightRAG.git` - `LightRAG` install might introduce version conflicts, see [this issue](https://github.com/Cinnamon/kotaemon/issues/440) - To quickly fix: `pip uninstall hnswlib chroma-hnswlib && pip install chroma-hnswlib` - Launch Kotaemon with `USE_LIGHTRAG=true` environment variable. - Set your default LLM & Embedding models in Resources setting and it will be recognized automatically from LightRAG. Setup MS GRAPHRAG - **Non-Docker Installation**: If you are not using Docker, install GraphRAG with the following command: ```shell pip install "graphrag<=0.3.6" future ``` - **Setting Up API KEY**: To use the GraphRAG retriever feature, ensure you set the `GRAPHRAG_API_KEY` environment variable. You can do this directly in your environment or by adding it to a `.env` file. - **Using Local Models and Custom Settings**: If you want to use GraphRAG with local models (like `Ollama`) or customize the default LLM and other configurations, set the `USE_CUSTOMIZED_GRAPHRAG_SETTING` environment variable to true. Then, adjust your settings in the `settings.yaml.example` file. ### Setup Local Models (for local/private RAG) See [Local model setup](docs/local_model.md). ### Setup multimodal document parsing (OCR, table parsing, figure extraction) These options are available: - [Azure Document Intelligence (API)](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence) - [Adobe PDF Extract (API)](https://developer.adobe.com/document-services/docs/overview/pdf-extract-api/) - [Docling (local, open-source)](https://github.com/DS4SD/docling) – see [integrations/docling.md](./docs/integrations/docling.md) for Kotaemon-specific setup. - [PaddleOCR (local, open-source)](https://github.com/PADDLEPADDLE/PADDLEOCR) – see [integrations/paddle_ocr.md](./docs/integrations/paddle_ocr.md) for Kotaemon-specific setup. Select corresponding loaders in `Settings -> Retrieval Settings -> File loader` ### Customize your application - By default, all application data is stored in the `./ktem_app_data` folder. You can back up or copy this folder to transfer your installation to a new machine. - For advanced users or specific use cases, you can customize these files: - `flowsettings.py` - `.env` #### `flowsettings.py` This file contains the configuration of your application. You can use the example [here](flowsettings.py) as the starting point. Notable settings ```python # setup your preferred document store (with full-text search capabilities) KH_DOCSTORE=(Elasticsearch | LanceDB | SimpleFileDocumentStore) # setup your preferred vectorstore (for vector-based search) KH_VECTORSTORE=(ChromaDB | LanceDB | InMemory | Milvus | Qdrant) # Enable / disable multimodal QA KH_REASONINGS_USE_MULTIMODAL=True # Setup your new reasoning pipeline or modify existing one. KH_REASONINGS = [ "ktem.reasoning.simple.FullQAPipeline", "ktem.reasoning.simple.FullDecomposeQAPipeline", "ktem.reasoning.react.ReactAgentPipeline", "ktem.reasoning.rewoo.RewooAgentPipeline", ] ``` #### `.env` This file provides another way to configure your models and credentials. Configure model via the .env file - Alternatively, you can configure the models via the `.env` file with the information needed to connect to the LLMs. This file is located in the folder of the application. If you don't see it, you can create one. - Currently, the following providers are supported: - **OpenAI** In the `.env` file, set the `OPENAI_API_KEY` variable with your OpenAI API key in order to enable access to OpenAI's models. There are other variables that can be modified, please feel free to edit them to fit your case. Otherwise, the default parameter should work for most people. ```shell OPENAI_API_BASE=https://api.openai.com/v1 OPENAI_API_KEY= OPENAI_CHAT_MODEL=gpt-3.5-turbo OPENAI_EMBEDDINGS_MODEL=text-embedding-ada-002 ``` - **Azure OpenAI** For OpenAI models via Azure platform, you need to provide your Azure endpoint and API key. You might also need to provide your developments' name for the chat model and the embedding model depending on how you set up Azure development. ```shell AZURE_OPENAI_ENDPOINT= AZURE_OPENAI_API_KEY= OPENAI_API_VERSION=2024-02-15-preview AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-35-turbo AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT=text-embedding-ada-002 ``` - **Local Models** - Using `ollama` OpenAI compatible server: - Install [ollama](https://github.com/ollama/ollama) and start the application. - Pull your model, for example: ```shell ollama pull llama3.1:8b ollama pull nomic-embed-text ``` - Set the model names on web UI and make it as default: - Using `GGUF` with `llama-cpp-python` You can search and download a LLM to be ran locally from the [Hugging Face Hub](https://huggingface.co/models). Currently, these model formats are supported: - GGUF You should choose a model whose size is less than your device's memory and should leave about 2 GB. For example, if you have 16 GB of RAM in total, of which 12 GB is available, then you should choose a model that takes up at most 10 GB of RAM. Bigger models tend to give better generation but also take more processing time. Here are some recommendations and their size in memory: - [Qwen1.5-1.8B-Chat-GGUF](https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat-GGUF/resolve/main/qwen1_5-1_8b-chat-q8_0.gguf?download=true): around 2 GB Add a new LlamaCpp model with the provided model name on the web UI. ### Adding your own RAG pipeline #### Custom Reasoning Pipeline 1. Check the default pipeline implementation in [here](libs/ktem/ktem/reasoning/simple.py). You can make quick adjustment to how the default QA pipeline work. 2. Add new `.py` implementation in `libs/ktem/ktem/reasoning/` and later include it in `flowssettings` to enable it on the UI. #### Custom Indexing Pipeline - Check sample implementation in `libs/ktem/ktem/index/file/graph` > (more instruction WIP). ## Citation Please cite this project as ```BibTeX @misc{kotaemon2024, title = {Kotaemon - An open-source RAG-based tool for chatting with any content.}, author = {The Kotaemon Team}, year = {2024}, howpublished = {\url{https://github.com/Cinnamon/kotaemon}}, } ``` ## Star History ## Contribution Since our project is actively being developed, we greatly value your feedback and contributions. Please see our [Contributing Guide](https://github.com/Cinnamon/kotaemon/blob/main/CONTRIBUTING.md) to get started. Thank you to all our contributors! --- ## File: docs/development/contributing.md # Contributing ## Setting up - Clone the repo ```shell git clone git@github.com:Cinnamon/kotaemon.git cd kotaemon ``` - Install the environment - Create a conda environment (python >= 3.10 is recommended) ```shell conda create -n kotaemon python=3.10 conda activate kotaemon # install dependencies cd libs/kotaemon pip install -e ".[all]" ``` - Or run the installer (one of the `scripts/run_*` scripts depends on your OS), then you will have all the dependencies installed as a conda environment at `install_dir/env`. ```shell conda activate install_dir/env ``` - Pre-commit ```shell pre-commit install ``` - Test ```shell pytest tests ``` ## Package overview `kotaemon` library focuses on the AI building blocks to implement a RAG-based QA application. It consists of base interfaces, core components and a list of utilities: - Base interfaces: `kotaemon` defines the base interface of a component in a pipeline. A pipeline is also a component. By clearly define this interface, a pipeline of steps can be easily constructed and orchestrated. - Core components: `kotaemon` implements (or wraps 3rd-party libraries like Langchain, llama-index,... when possible) commonly used components in kotaemon use cases. Some of these components are: LLM, vector store, document store, retriever... For a detailed list and description of these components, please refer to the [API Reference](../reference/Summary.md) section. - List of utilities: `kotaemon` provides utilities and tools that are usually needed in client project. For example, it provides a prompt engineering UI for AI developers in a project to quickly create a prompt engineering tool for DMs and QALs. It also provides a command to quickly spin up a project code base. For a full list and description of these utilities, please refer to the [Utilities](utilities.md) section. ```mermaid mindmap root((kotaemon)) Base Interfaces Document LLMInterface RetrievedDocument BaseEmbeddings BaseChat BaseCompletion ... Core Components LLMs AzureOpenAI OpenAI Embeddings AzureOpenAI OpenAI HuggingFaceEmbedding VectorStore InMemoryVectorstore ChromaVectorstore Agent Tool DocumentStore ... Utilities Scaffold project PromptUI Documentation Support ``` ## Common conventions - PR title: One-line description (example: Feat: Declare BaseComponent and decide LLM call interface). - [Encouraged] Provide a quick description in the PR, so that: - Reviewers can quickly understand the direction of the PR. - It will be included in the commit message when the PR is merged. ## Environment caching on PR - To speed up CI, environments are cached based on the version specified in `__init__.py`. - Since dependencies versions in `setup.py` are not pinned, you need to pump the version in order to use a new environment. That environment will then be cached and used by your subsequence commits within the PR, until you pump the version again - The new environment created during your PR is cached and will be available to others once the PR is merged. - If you are experimenting with new dependencies and want a fresh environment every time, add `[ignore cache]` in your commit message. The CI will create a fresh environment to run your commit and then discard it. - If your PR include updated dependencies, the recommended workflow would be: - Doing development as usual. - When you want to run the CI, push a commit with the message containing `[ignore cache]`. - Once the PR is final, pump the version in `__init__.py` and push a final commit not containing `[ignore cache]`. ## Merge PR guideline - Use squash and merge option - 1st line message is the PR title. - The text area is the PR description. --- ## File: docs/development/create-a-component.md # Creating a component A fundamental concept in kotaemon is "component". Anything that isn't data or data structure is a "component". A component can be thought of as a step within a pipeline. It takes in some input, processes it, and returns an output, just the same as a Python function! The output will then become an input for the next component in a pipeline. In fact, a pipeline is just a component. More appropriately, a nested component: a component that makes use of one or more other components in the processing step. So in reality, there isn't a difference between a pipeline and a component! Because of that, in kotaemon, we will consider them the same as "component". To define a component, you will: 1. Create a class that subclasses from `kotaemon.base.BaseComponent` 2. Declare init params with type annotation 3. Declare nodes (nodes are just other components!) with type annotation 4. Implement the processing logic in `run`. The syntax of a component is as follow: ```python from kotaemon.base import BaseComponent from kotaemon.llms import LCAzureChatOpenAI from kotaemon.parsers import RegexExtractor class FancyPipeline(BaseComponent): param1: str = "This is param1" param2: int = 10 param3: float node1: BaseComponent # this is a node because of BaseComponent type annotation node2: LCAzureChatOpenAI # this is also a node because LCAzureChatOpenAI subclasses BaseComponent node3: RegexExtractor # this is also a node bceause RegexExtractor subclasses BaseComponent def run(self, some_text: str): prompt = (self.param1 + some_text) * int(self.param2 + self.param3) llm_pred = self.node2(prompt).text matches = self.node3(llm_pred) return matches ``` Then this component can be used as follow: ```python llm = LCAzureChatOpenAI(endpoint="some-endpont") extractor = RegexExtractor(pattern=["yes", "Yes"]) component = FancyPipeline( param1="Hello" param3=1.5 node1=llm, node2=llm, node3=extractor ) component("goodbye") ``` This way, we can define each operation as a reusable component, and use them to compose larger reusable components! ## Benefits of component By defining a component as above, we formally encapsulate all the necessary information inside a single class. This introduces several benefits: 1. Allow tools like promptui to inspect the inner working of a component in order to automatically generate the promptui. 2. Allow visualizing a pipeline for debugging purpose. --- ## File: docs/development/data-components.md # Data & Data Structure Components The data & data structure components include: - The `Document` class. - The document store. - The vector store. ## Data Loader - PdfLoader - Layout-aware with table parsing PdfLoader - MathPixLoader: To use this loader, you need MathPix API key, refer to [mathpix docs](https://docs.mathpix.com/#introduction) for more information - OCRLoader: This loader uses lib-table and Flax pipeline to perform OCR and read table structure from PDF file (TODO: add more info about deployment of this module). - Output: - Document: text + metadata to identify whether it is table or not ``` - "source": source file name - "type": "table" or "text" - "table_origin": original table in markdown format (to be feed to LLM or visualize using external tools) - "page_label": page number in the original PDF document ``` ## Document Store - InMemoryDocumentStore ## Vector Store - ChromaVectorStore - InMemoryVectorStore --- ## File: docs/development/index.md {% include-markdown "../../README.md" start="" end="" %} --- ## File: docs/development/utilities.md # Utilities ## Prompt engineering UI **_Important:_** despite the name prompt engineering UI, this tool allows testers to test any kind of parameters that are exposed by developers. Prompt is one kind of param. There can be other type of params that testers can tweak (e.g. top_k, temperature...). In the development process, developers typically build the pipeline. However, for use cases requiring expertise in prompt creation, non-technical members (testers, domain experts) can be more effective. To facilitate this, `kotaemon` offers a user-friendly prompt engineering UI that developers integrate into their pipelines. This enables non-technical members to adjust prompts and parameters, run experiments, and export results for optimization. As of Sept 2023, there are 2 kinds of prompt engineering UI: - Simple pipeline: run one-way from start to finish. - Chat pipeline: interactive back-and-forth. ### Simple pipeline For simple pipeline, the supported client project workflow looks as follow: 1. [tech] Build pipeline 2. [tech] Export pipeline to config: `$ kotaemon promptui export --output ` 3. [tech] Customize the config 4. [tech] Spin up prompt engineering UI: `$ kotaemon promptui run ` 5. [non-tech] Change params, run inference 6. [non-tech] Export to Excel 7. [non-tech] Select the set of params that achieve the best output The prompt engineering UI prominently involves from step 2 to step 7 (step 1 is normally done by the developers, while step 7 happens exclusively in Excel file). #### Step 2 - Export pipeline to config Command: ```shell $ kotaemon promptui export --output ``` where: - `` is a dot-separated path to the pipeline. For example, if your pipeline can be accessed with `from projectA.pipelines import AnsweringPipeline`, then this value is `projectA.pipelines.AnswerPipeline`. - `` is the target file path that the config will be exported to. If the config file already exists, and contains information of other pipelines, the config of current pipeline will additionally be added. If it contains information of the current pipeline (in the past), the old information will be replaced. By default, all params in a pipeline (including nested params) will be export to the configuration file. For params that you do not wish to expose to the UI, you can directly remove them from the config YAML file. You can also annotate those param with `ignore_ui=True`, and they will be ignored in the config generation process. Example: ```python class Pipeline(BaseComponent): param1: str = Param(default="hello") param2: str = Param(default="goodbye", ignore_ui=True) ``` Declared as above, and `param1` will show up in the config YAML file, while `param2` will not. #### Step 3 - Customize the config developers can further edit the config file in this step to get the most suitable UI (step 4) with their tasks. The exported config will have this overall schema: ```yml : params: ... (Detail param information to initiate a pipeline. This corresponds to the pipeline init parameters.) inputs: ... (Detail the input of the pipeline e.g. a text prompt. This corresponds to the params of `run(...)` method.) outputs: ... (Detail the output of the pipeline e.g. prediction, accuracy... This is the output information we wish to see in the UI.) logs: ... (Detail what information should show up in the log.) ``` ##### Input and params The inputs section have the overall schema as follow: ```yml inputs: : component: params: # this section is optional) value: : ... # similar to above params: : ... # similar to those in the inputs ``` The list of supported prompt UI and their corresponding gradio UI components: ```python COMPONENTS_CLASS = { "text": gr.components.Textbox, "checkbox": gr.components.CheckboxGroup, "dropdown": gr.components.Dropdown, "file": gr.components.File, "image": gr.components.Image, "number": gr.components.Number, "radio": gr.components.Radio, "slider": gr.components.Slider, } ``` ##### Outputs The outputs are a list of variables that we wish to show in the UI. Since in Python, the function output doesn't have variable name, so output declaration is a little bit different than input and param declaration: ```yml outputs: - component: step: item: - ... # similar to above ``` where: - component: the same text string and corresponding Gradio UI as in inputs & params - step: the pipeline step that we wish to look fetch and show output on the UI - item: the jsonpath mechanism to get the targeted variable from the step above ##### Logs The logs show a list of sheetname and how to retrieve the desired information. ```yml logs: : inputs: - name: step: variable: - ... outputs: - name: step: item: ``` #### Step 4 + 5 - Spin up prompt engineering UI + Perform prompt engineering Command: ```shell $ kotaemon promptui run ``` This will generate an UI as follow: where: - The tabs at the top of the UI corresponds to the pipeline to do prompt engineering. - The inputs and params tabs allow users to edit (these corresponds to the inputs and params in the config file). - The outputs panel holds the UI elements to show the outputs defined in config file. - The Run button: will execute pipeline with the supplied inputs and params, and render result in the outputs panel. - The Export button: will export the logs of all the run to an Excel files users to inspect for best set of params. #### Step 6 - Export to Excel Upon clicking export, the users can download Excel file. ### Chat pipeline Chat pipeline workflow is different from simple pipeline workflow. In simple pipeline, each Run creates a set of output, input and params for users to compare. In chat pipeline, each Run is not a one-off run, but a long interactive session. Hence, the workflow is as follow: 1. Set the desired parameters. 2. Click "New chat" to start a chat session with the supplied parameters. This set of parameters will persist until the end of the chat session. During an ongoing chat session, changing the parameters will not take any effect. 3. Chat and interact with the chat bot on the right panel. You can add any additional input (if any), and they will be supplied to the chatbot. 4. During chat, the log of the chat will show up in the "Output" tabs. This is empty by default, so if you want to show the log here, tell the AI developers to configure the UI settings. 5. When finishing chat, select your preference in the radio box. Click "End chat". This will save the chat log and the preference to disk. 6. To compare the result of different run, click "Export" to get an Excel spreadsheet summary of different run. --- ## File: docs/pages/app/ext/user-management.md `ktem` provides user management as an extension. To enable user management, in your `flowsettings.py`, set the following variables: - `KH_FEATURE_USER_MANAGEMENT`: True to enable. - `KH_FEATURE_USER_MANAGEMENT_ADMIN`: the admin username. This user will be created when the app 1st start. - `KH_FEATURE_USER_MANAGEMENT_PASSWORD`: the admin password. This value accompanies the admin username. Once enabled, you have access to the following features: - User login/logout (located in Settings Tab) - User changing password (located in Settings Tab) - Create / List / Edit / Delete user (located in Resources > Users Tab) --- ## File: docs/pages/app/index/file.md The file index stores files in a local folder and index them for retrieval. This file index provides the following infrastructure to support the indexing: - SQL table Source: store the list of files that are indexed by the system - Vector store: contain the embedding of segments of the files - Document store: contain the text of segments of the files. Each text stored in this document store is associated with a vector in the vector store. - SQL table Index: store the relationship between (1) the source and the docstore, and (2) the source and the vector store. The indexing and retrieval pipelines are encouraged to use the above software infrastructure. ## Indexing pipeline The ktem has default indexing pipeline: `ktem.index.file.pipelines.IndexDocumentPipeline`. This default pipeline works as follow: - **Input**: list of file paths - **Output**: list of nodes that are indexed into database - **Process**: - Read files into texts. Different file types has different ways to read texts. - Split text files into smaller segments - Run each segments into embeddings. - Store the embeddings into vector store. Store the texts of each segment into docstore. Store the list of files in Source. Store the linking between Sources and docstore + vectorstore in Index table. You can customize this default pipeline if your indexing process is close to the default pipeline. You can create your own indexing pipeline if there are too much different logic. ### Customize the default pipeline The default pipeline provides the contact points in `flowsettings.py`. 1. `FILE_INDEX_PIPELINE_FILE_EXTRACTORS`. Supply overriding file extractor, based on file extension. Example: `{".pdf": "path.to.PDFReader", ".xlsx": "path.to.ExcelReader"}` 2. `FILE_INDEX_PIPELINE_SPLITTER_CHUNK_SIZE`. The expected number of characters of each text segment. Example: 1024. 3. `FILE_INDEX_PIPELINE_SPLITTER_CHUNK_OVERLAP`. The expected number of characters that consecutive text segments should overlap with each other. Example: 256. ### Create your own indexing pipeline Your indexing pipeline will subclass `BaseFileIndexIndexing`. You should define the following methods: - `run(self, file_paths)`: run the indexing given the pipeline - `get_pipeline(cls, user_settings, index_settings)`: return the fully-initialized pipeline, ready to be used by ktem. - `user_settings`: is a dictionary contains user settings (e.g. `{"pdf_mode": True, "num_retrieval": 5}`). You can declare these settings in the `get_user_settings` classmethod. ktem will collect these settings into the app Settings page, and will supply these user settings to your `get_pipeline` method. - `index_settings`: is a dictionary. Currently it's empty for File Index. - `get_user_settings`: to declare user settings, return a dictionary. By subclassing `BaseFileIndexIndexing`, You will have access to the following resources: - `self._Source`: the source table - `self._Index`: the index table - `self._VS`: the vector store - `self._DS`: the docstore Once you have prepared your pipeline, register it in `flowsettings.py`: `FILE_INDEX_PIPELINE = ""`. ## Retrieval pipeline The ktem has default retrieval pipeline: `ktem.index.file.pipelines.DocumentRetrievalPipeline`. This pipeline works as follow: - Input: user text query & optionally a list of source file ids - Output: the output segments that match the user text query - Process: - If a list of source file ids is given, get the list of vector ids that associate with those file ids. - Embed the user text query. - Query the vector store. Provide a list of vector ids to limit query scope if the user restrict. - Return the matched text segments ### Create your own retrieval pipeline Your retrieval pipeline will subclass `BaseFileIndexRetriever`. The retriever has the same database, vectorstore and docstore accesses like the indexing pipeline. You should define the following methods: - `run(self, query, file_ids)`: retrieve relevant documents relating to the query. If `file_ids` is given, you should restrict your search within these `file_ids`. - `get_pipeline(cls, user_settings, index_settings, selected)`: return the fully-initialized pipeline, ready to be used by ktem. - `user_settings`: is a dictionary contains user settings (e.g. `{"pdf_mode": True, "num_retrieval": 5}`). You can declare these settings in the `get_user_settings` classmethod. ktem will collect these settings into the app Settings page, and will supply these user settings to your `get_pipeline` method. - `index_settings`: is a dictionary. Currently it's empty for File Index. - `selected`: a list of file ids selected by user. If user doesn't select anything, this variable will be None. - `get_user_settings`: to declare user settings, return a dictionary. Once you build the retrieval pipeline class, you can register it in `flowsettings.py`: `FILE_INDEXING_RETRIEVER_PIPELIENS = ["path.to.retrieval.pipelie"]`. Because there can be multiple parallel pipelines within an index, this variable takes a list of string rather than a string. ## Software infrastructure | Infra | Access | Schema | Ref | | ---------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | | SQL table Source | self.\_Source | - id (int): id of the source (auto)- name (str): the name of the file- path (str): the path of the file- size (int): the file size in bytes- note (dict): allow extra optional information about the file- date_created (datetime): the time the file is created (auto) | This is SQLALchemy ORM class. Can consult | | SQL table Index | self.\_Index | - id (int): id of the index entry (auto)- source_id (int): the id of a file in the Source table- target_id: the id of the segment in docstore or vector store- relation_type (str): if the link is "document" or "vector" | This is SQLAlchemy ORM class | | Vector store | self.\_VS | - self.\_VS.add: add the list of embeddings to the vector store (optionally associate metadata and ids)- self.\_VS.delete: delete vector entries based on ids- self.\_VS.query: get embeddings based on embeddings. | kotaemon > storages > vectorstores > BaseVectorStore | | Doc store | self.\_DS | - self.\_DS.add: add the segments to document stores- self.\_DS.get: get the segments based on id- self.\_DS.get_all: get all segments- self.\_DS.delete: delete segments based on id | kotaemon > storages > docstores > base > BaseDocumentStore | --- ## File: docs/pages/app/settings/overview.md # Overview There are 3 kinds of settings in `ktem`, geared towards different stakeholders for different use cases: - Developer settings. These settings are meant for very basic app customization, such as database URL, cloud config, logging config, which features to enable... You will be interested in the developer settings if you deploy `ktem` to your customers, or if you build extension for `ktem` for developers. These settings are declared inside `flowsettings.py`. - Admin settings. These settings show up in the Admin page, and are meant to allow admin-level user to customize low level features, such as which credentials to connect to data sources, which keys to use for LLM... - [User settings](/pages/app/settings/user-settings/). These settings are meant for run-time users to tweak ktem to their personal needs, such as which output languages the chatbot should generate, which reasoning type to use... --- ## File: docs/pages/app/settings/user-settings.md # User settings `ktem` allows developers to extend the index and the reasoning pipeline. In many cases, these components can have settings that should be modified by users at run-time, (e.g. `topk`, `chunksize`...). These are the user settings. `ktem` allows developers to declare such user settings in their code. Once declared, `ktem` will render them in a Settings page. There are 2 places that `ktem` looks for declared user settings. You can refer to the respective pages. - In the index. - In the reasoning pipeline. ## Syntax of a settings A collection of settings is a dictionary of type `dict[str, dict]`, where the key is a setting id, and the value is the description of the setting. ```python settings = { "topk": { "name": "Top-k chunks", "value": 10, "component": "number", }, "lang": { "name": "Languages", "value": "en", "component": "dropdown", "choices": [("en", "English"), ("cn", "Chinese")], } } ``` Each setting description must have: - name: the human-understandable name of the settings. - value: the default value of the settings. - component: the UI component to render such setting on the UI. Available: - "text": single-value - "number": single-value - "checkbox": single-value - "dropdown": choices - "radio": choices - "checkboxgroup": choices - choices: the list of choices, if the component type allows. ## Settings page structure