Skip to content

Commit

Permalink
Add information about JSON ingestion
Browse files Browse the repository at this point in the history
  • Loading branch information
thomashacker committed Feb 3, 2025
1 parent bda0565 commit 5badcfb
Showing 1 changed file with 24 additions and 0 deletions.
24 changes: 24 additions & 0 deletions TECHNICAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,27 @@ For handling large upload of files, the `BatchManager` class handles batches of
### How to control the position of context sent to the Generator to generate a response?

Every `generator` class has a `prepare_messages` method. This method is used to format the messages that are sent to the LLM. The position of the context in the messages is important because it determines where the context is placed in the conversation.

### How to upload a JSON file to Verba?

## Verba JSON Structure

A Verba Document can be created from a JSON object. The JSON object is converted to a Verba Document object and then uploaded to the vector database. Here's the general structure of a Verba Document (you can also find the implementation in the `Document.py` file):

```python
{
"title": "string", # The title of the document
"content": "string", # The content of the document
"extension": "string", # The extension of the document (Optional)
"fileSize": "number", # The size of the document in bytes (Optional)
"labels": "array", # The labels of the document (can be empty, used for filtering)
"source": "string", # The source of the document (can be an URL, optional)
"meta": "object", # The meta data of the document used internally
"metadata": "string" # Metadata information of the document, will be used in the embedding process
}
```

## Custom JSON Structure

There is currently no support for custom JSON structure. Instead the whole JSON will simply be dumped into the content field of the Verba document.
There are plans to add support for custom JSON structure in the future.

0 comments on commit 5badcfb

Please sign in to comment.