Skip to content

Commit

Permalink
Merge pull request #47 from raid-guild/improve-docs-grammar
Browse files Browse the repository at this point in the history
improve: grammar round
  • Loading branch information
santteegt authored Nov 5, 2024
2 parents f43bc05 + bd7cc73 commit 3c53590
Show file tree
Hide file tree
Showing 9 changed files with 232 additions and 378 deletions.
17 changes: 4 additions & 13 deletions docs/pages/apis.mdx
Original file line number Diff line number Diff line change
@@ -1,24 +1,15 @@
# API Examples

The repository already includes API pipeline manifest definitions for generating knowledge bases from a few REST APIs. Each aims to demonstrate how to define
a YAML manifest for extracting data from target API endpoints using different Authentication/Pagination strategies. For a more in-depth review on how to build a
manifest for creating a RAG pipeline for your own API, visit [Defining the API Pipeline Manifest](/manifest-definition)
The repository already includes API pipeline manifest definitions for generating knowledge bases from several REST APIs. Each demonstrates how to define a YAML manifest for extracting data from target API endpoints using different Authentication/Pagination strategies. For a more in-depth review of how to build a manifest for creating a RAG pipeline for your own API, visit [Defining the API Pipeline Manifest](/manifest-definition).

## Boardroom Governance API

[Boardroom](https://boardroom.io/) offers its `Boardrooms Governance API` to find comprehensive data on 350+ DAOs across chains.
It offer endpoints that fetches information about proposals, delegates, discussions and much more. You can find the complete API documentation
in this [link](https://docs.boardroom.io/docs/api/cd5e0c8aa2bc1-overview)
[Boardroom](https://boardroom.io/) offers its `Boardrooms Governance API` to provide comprehensive data on 350+ DAOs across chains. It offers endpoints that fetch information about proposals, delegates, discussions, and much more. You can find the complete API documentation at this [link](https://docs.boardroom.io/docs/api/cd5e0c8aa2bc1-overview).

Check the [Boardroom API](/apis/boardroom-api) section for details on how to extract data from the API and generate a knowledge base from the DAO of your interest.

## Optimism Agora API

The [Agora](https://www.agora.xyz/#Product) OP API provides various endpoints to interact with the OP Collective platform, focusing on RetroPGF (Retroactive Public Goods Funding) for Optimism.
A comprehensive guide to the API endpoints, categorized by functionality can be found in this [link](https://vote.optimism.io/api_v1).
The [Agora](https://www.agora.xyz/#Product) OP API provides various endpoints to interact with the OP Collective platform, focusing on RetroPGF (Retroactive Public Goods Funding) for Optimism. A comprehensive guide to the API endpoints, categorized by functionality, can be found at this [link](https://vote.optimism.io/api_v1).

Check the [Agora API](/apis/agora-api) section for details on how to extract data from the API and generate a knowledge base related to RetroPGF projects and proposals within the OP collective.

## Working with Other APIs

If you are interested in working with any other API, you can find the [API Examples](/apis/other-api-sources) section to get started.
If you are interested in working with any other API, visit the [API Examples](/apis/other-api-sources) section to get started.
47 changes: 19 additions & 28 deletions docs/pages/apis/agora-api.mdx
Original file line number Diff line number Diff line change
@@ -1,40 +1,35 @@
# Optimism Agora API

The repository includes both the [API pipeline manifest](https://github.com/raid-guild/gaianet-rag-api-pipeline/blob/main/config/agora_api_pipeline.yaml)
definition and the [OpenAPI spec](https://github.com/raid-guild/gaianet-rag-api-pipeline/blob/main/config/agora_openapi.yaml) to create a RAG pipeline to generate a
knowledge base from RetroPGF projects and proposals within the OP collective. You can usually find these files within the `config` folder.
This repository contains the [API pipeline manifest](https://github.com/raid-guild/gaianet-rag-api-pipeline/blob/main/config/agora_api_pipeline.yaml) and [OpenAPI specification](https://github.com/raid-guild/gaianet-rag-api-pipeline/blob/main/config/agora_openapi.yaml) needed to create a RAG pipeline. This pipeline generates a knowledge base from RetroPGF projects and proposals within the OP collective. These files are typically located in the `config` folder.

In order to consume data from this API, you are required to request an API key. You can do so by dong a manual request on [Agora's Discord server](https://www.agora.xyz/#Product)
Do not forget to set it under `config/secrets/api-key` or specify it directly using the `--api-key` argument in the CLI.
To access this API, you'll need an API key. You can request one through [Agora's Discord server](https://www.agora.xyz/#Product). Once obtained, store the key in `config/secrets/api-key` or provide it directly using the `--api-key` CLI argument.

## API Pipeline manifest - Overview
## API Pipeline Manifest - Overview

This API pipeline will extract data from the `/proposals` and `/projects` [endpoints](https://github.com/raid-guild/gaianet-rag-api-pipeline/blob/main/config/agora_api_pipeline.yaml#L79).
There's no need to specify `api_parameters` so this sections is [empty](https://github.com/raid-guild/gaianet-rag-api-pipeline/blob/main/config/agora_api_pipeline.yaml#L5).
The requester definition can be seen below. The API uses a BearerAuthenticator schema and gets the `api_token` from the `config` object.
The API pipeline extracts data from the `/proposals` and `/projects` [endpoints](https://github.com/raid-guild/gaianet-rag-api-pipeline/blob/main/config/agora_api_pipeline.yaml#L79). Since no `api_parameters` are required, this section remains [empty](https://github.com/raid-guild/gaianet-rag-api-pipeline/blob/main/config/agora_api_pipeline.yaml#L5).

```yaml [agora_api_pipeline.yaml]
...
Below is the requester definition. The API implements a BearerAuthenticator schema and retrieves the `api_token` from the `config` object:

```yaml
# agora_api_pipeline.yaml
definition:
requester_base:
requester_base:
type: HttpRequester
url_base: "https://vote.optimism.io/api/v1"
http_method: "GET"
authenticator: # More on https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/authentication
authenticator: # Details at https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/authentication
type: BearerAuthenticator
api_token: "{{ config['api_key'] }}"
...
```
The API handles pagination via Offset-based pagination strategy. The page_size is set to 50 while `offset` and `limit` parameter values are dynamically
injected into the URL as request parameters.
The API uses an Offset-based pagination strategy. The `page_size` is set to 50, while `offset` and `limit` parameters are dynamically inserted into the URL as request parameters:

```yaml [agora_api_pipeline.yaml]
...
```yaml
# agora_api_pipeline.yaml
definition:
paginator: # More on https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/pagination
paginator: # Details at https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/pagination
type: DefaultPaginator
pagination_strategy: # More on https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/pagination#offset-increment
pagination_strategy: # Details at https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/pagination#offset-increment
type: "OffsetIncrement"
page_size: 50
page_token_option:
Expand All @@ -45,18 +40,14 @@ definition:
type: "RequestOption"
inject_into: "request_parameter"
field_name: "limit"
...
```

## Generating a knowledge base using the `rag-api-pipeline` CLI
## Generating a Knowledge Base Using the `rag-api-pipeline` CLI

Before executing the `run-all` command, make sure you have `Ollama` running locally with the LLM embeddings model of your preference:
Before running the `run-all` command, ensure that `Ollama` is running locally with your preferred LLM embeddings model:

```bash [Terminal]
```bash
poetry run rag-api-pipeline run-all config/agora_api_pipeline.yaml --openapi-spec-file config/agora_openapi.yaml --llm-provider ollama
```

Once the command above finishes, you'll find a compressed knowledge base snapshot in
`{OUTPUT_FOLDER}/optimism_agora_api/` with name optimism_agora_api_collection-xxxxxxxxxxxxxxxx-yyyy-mm-dd-hh-mm-ss.snapshot.tar.gz`. To import it into your
gaianet node, you can find the instructions on how to select a knowledge base [here](https://docs.gaianet.ai/node-guide/customize#select-a-knowledge-base).
The recommended prompts and node config settings can be found [here](/node-deployment#recommended-gaianet-node-configuration).
After execution, you'll find a compressed knowledge base snapshot in `{OUTPUT_FOLDER}/optimism_agora_api/` named `optimism_agora_api_collection-xxxxxxxxxxxxxxxx-yyyy-mm-dd-hh-mm-ss.snapshot.tar.gz`. For instructions on importing this into your Gaianet node, refer to the documentation on [selecting a knowledge base](https://docs.gaianet.ai/node-guide/customize#select-a-knowledge-base). Find recommended prompts and node configuration settings [here](/node-deployment#recommended-gaianet-node-configuration).
Loading

0 comments on commit 3c53590

Please sign in to comment.