Lot of fixes, improvements etc...

.github/workflows/tests.yml

@@ -28,7 +28,9 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
-    - uses: actions/setup-python@v2
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.10'
     - name: Install dependencies
       run: |
         pip install poetry
@@ -47,7 +49,9 @@ jobs:
 
     steps:
     - uses: actions/checkout@v3
-    - uses: actions/setup-python@v2
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.10'
     - name: Install dependencies
       run: |
         pip install poetry
@@ -89,7 +93,9 @@ jobs:
     if: false
     steps:
     - uses: actions/checkout@v3
-    - uses: actions/setup-python@v2
+    - uses: actions/setup-python@v4
+      with:
+        python-version: '3.10'
     - name: Install dependencies
       run: |
         pip install poetry

.vscode/launch.json

@@ -11,13 +11,42 @@
             "cwd": "${workspaceFolder}",
             "module": "clairvoyance",
             "args": [
-                "-o", "/tmp/t.json",
-                "-w", "tests/data/wordlist-for-apollo-server.txt",
-                "http://localhost:4000", "--verbose"
+                "-o",
+                "/tmp/t.json",
+                "-w",
+                "tests/data/wordlist-for-apollo-server.txt",
+                "http://localhost:4000",
+                "--verbose"
+            ],
+            "console": "integratedTerminal",
+            "justMyCode": true
+        },
+        {
+            "name": "Clairvoyance on localhost:4000",
+            "type": "python",
+            "request": "launch",
+            "cwd": "${workspaceFolder}",
+            "module": "clairvoyance",
+            "args": [
+                "-o",
+                "target/t.json",
+                "http://localhost:4000",
+                "--verbose"
+            ],
+            "console": "integratedTerminal",
+            "justMyCode": true
+        },
+        {
+            "name": "Clairvoyance on rick and morty GQL application",
+            "type": "python",
+            "request": "launch",
+            "cwd": "${workspaceFolder}",
+            "module": "clairvoyance",
+            "args": [
+                "https://rickandmortyapi.com/graphql"
             ],
             "console": "integratedTerminal",
             "justMyCode": true
         }
-
     ]
 }

README.md

@@ -1,60 +1,48 @@
-# clairvoyance
+# Clairvoyance
 
-Some GraphQL APIs have disabled introspection. For example, [Apollo Server disables introspection automatically if the `NODE_ENV` environment variable is set to `production`](https://www.apollographql.com/docs/tutorial/schema/#explore-your-schema).
-
-Clairvoyance allows us to get GraphQL API schema when introspection is disabled. It produces schema in JSON format suitable for other tools like [GraphQL Voyager](https://github.com/APIs-guru/graphql-voyager), [InQL](https://github.com/doyensec/inql) or [graphql-path-enum](https://gitlab.com/dee-see/graphql-path-enum).
+Obtain GraphQL API Schema even if the introspection is disabled.
 
-## Acknowledgments
+[![PyPI](https://img.shields.io/pypi/v/clairvoyance)](https://pypi.org/project/clairvoyance/)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/clairvoyance)](https://pypi.org/project/clairvoyance/)
+[![PyPI - Downloads](https://img.shields.io/pypi/dm/clairvoyance)](https://pypi.org/project/clairvoyance/)
+[![GitHub](https://img.shields.io/github/license/nikitastupin/clairvoyance)](https://github.com/nikitastupin/clairvoyance/blob/main/LICENSE)
 
-Thanks to [Swan](https://github.com/c3b5aw) from [Escape-Technologies](https://github.com/Escape-Technologies) for 2.0 version.
+## Introduction
 
-## Usage
+Some GraphQL APIs have disabled introspection. For example, [Apollo Server disables introspection automatically if the `NODE_ENV` environment variable is set to `production`](https://www.apollographql.com/docs/tutorial/schema/#explore-your-schema).
 
-You may find more details on how the tool works in the second half of the [GraphQL APIs from bug hunter's perspective by Nikita Stupin](https://youtu.be/nPB8o0cSnvM) talk.
+Clairvoyance allows us to get GraphQL API schema when introspection is disabled. It produces schema in JSON format suitable for other tools like [GraphQL Voyager](https://github.com/APIs-guru/graphql-voyager), [InQL](https://github.com/doyensec/inql) or [graphql-path-enum](https://gitlab.com/dee-see/graphql-path-enum).
 
-### From PyPI
+## Contributors
 
-```bash
-pip install clairvoyance
-```
+Thanks to the [contributors](#contributors) for their work.
 
-### From Python interpreter
+- [nikitastupin](https://github.com/nikitastupin)
+- [Escape](https://escape.tech) team :
+  - [iCarossio](https://github.com/iCarossio)
+  - [Swan](https://github.com/c3b5aw)
+  - [QuentinN42](https://github.com/QuentinN42)
+  - [Nohehf](https://github.com/Nohehf)
+- [i-tsaturov](https://github.com/i-tsaturov)
+- [EONRaider](https://github.com/EONRaider)
+- [noraj](https://github.com/noraj)
+- [belane](https://github.com/belane)
 
-```bash
-git clone https://github.com/nikitastupin/clairvoyance.git
-cd clairvoyance
-pip install poetry
-poetry config virtualenvs.in-project true
-poetry install --no-dev
-source .venv/bin/activate
-```
+## Getting started
 
 ```bash
-python3 -m clairvoyance --help
-```
-
-```bash
-python3 -m clairvoyance -o /path/to/schema.json https://swapi-graphql.netlify.app/.netlify/functions/index
+pip install clairvoyance
+clairvoyance https://rickandmortyapi.com/graphql -o schema.json
+# should take about 2 minute
 ```
 
-### From Docker Image
+## Docker Image
 
 ```bash
 docker run --rm nikitastupin/clairvoyance --help
 ```
 
-```bash
-# Assuming the wordlist.txt file is found in $PWD
-docker run --rm -v $(pwd):/tmp/ nikitastupin/clairvoyance -vv -o /tmp/schema.json -w /tmp/wordlist.txt https://swapi-graphql.netlify.app/.netlify/functions/index
-```
-
-### From BlackArch Linux
-
-> NOTE: this distribution is supported by a third-party (i.e. not by the mainainters of clairvoyance)
-
-```bash
-pacman -S clairvoyance
-```
+## Advanced Usage
 
 ### Which wordlist should I use?
 
@@ -80,3 +68,7 @@ In case of question or issue with clairvoyance please refer to [wiki](https://gi
 ## Contributing
 
 Pull requests are welcome! For major changes, please open an issue first to discuss what you would like to change. For more information about tests, internal project structure and so on refer to [Development](https://github.com/nikitastupin/clairvoyance/wiki/Development) wiki page.
+
+## Documentation
+
+- You may find more details on how the tool works in the second half of the [GraphQL APIs from bug hunter's perspective by Nikita Stupin](https://youtu.be/nPB8o0cSnvM) talk.

clairvoyance/cli.py

@@ -1,8 +1,9 @@
 import asyncio
 import json
 import logging
+import re
 import sys
-import os
+from pathlib import Path
 from typing import Dict, List, Optional
 
 from clairvoyance import graphql, oracle
@@ -18,6 +19,10 @@ def setup_context(
     logger: logging.Logger,
     headers: Optional[Dict[str, str]] = None,
     concurrent_requests: Optional[int] = None,
+    proxy: Optional[str] = None,
+    max_retries: Optional[int] = None,
+    backoff: Optional[int] = None,
+    disable_ssl_verify: Optional[bool] = None,
 ) -> None:
     """Initialize objects and freeze them into the context."""
 
@@ -26,12 +31,16 @@ def setup_context(
         url,
         headers=headers,
         concurrent_requests=concurrent_requests,
+        proxy=proxy,
+        max_retries=max_retries,
+        backoff=backoff,
+        disable_ssl_verify=disable_ssl_verify,
     )
     logger_ctx.set(logger)
 
 
 def load_default_wordlist() -> List[str]:
-    wl = os.path.join(os.path.dirname(__file__), 'wordlist.txt')
+    wl = Path(__file__).parent / 'wordlist.txt'
     with open(wl, 'r', encoding='utf-8') as f:
         return [w.strip() for w in f.readlines() if w.strip()]
 
@@ -45,6 +54,10 @@ async def blind_introspection(
     input_document: Optional[str] = None,
     input_schema_path: Optional[str] = None,
     output_path: Optional[str] = None,
+    proxy: Optional[str] = None,
+    max_retries: Optional[int] = None,
+    backoff: Optional[int] = None,
+    disable_ssl_verify: Optional[bool] = None,
 ) -> str:
     wordlist = wordlist or load_default_wordlist()
     assert wordlist, 'No wordlist provided'
@@ -54,6 +67,10 @@ async def blind_introspection(
         logger=logger,
         headers=headers,
         concurrent_requests=concurrent_requests,
+        proxy=proxy,
+        max_retries=max_retries,
+        backoff=backoff,
+        disable_ssl_verify=disable_ssl_verify,
     )
 
     logger.info(f'Starting blind introspection on {url}...')
@@ -65,7 +82,10 @@ async def blind_introspection(
 
     input_document = input_document or 'query { FUZZ }'
     ignored = set(e.value for e in GraphQLPrimitive)
+    iterations = 1
     while True:
+        logger.info(f'Iteration {iterations}')
+        iterations += 1
         schema = await oracle.clairvoyance(
             wordlist,
             input_document=input_document,
@@ -107,6 +127,17 @@ def cli(argv: Optional[List[str]] = None) -> None:
     wordlist = []
     if args.wordlist:
         wordlist = [w.strip() for w in args.wordlist.readlines() if w.strip()]
+        # de-dupe the wordlist.
+        wordlist = list(set(wordlist))
+
+    # remove wordlist items that don't conform to graphQL regex github-issue #11
+    if args.validate:
+        wordlist_parsed = [w for w in wordlist if re.match(r'[_A-Za-z][_0-9A-Za-z]*', w)]
+        logging.info(
+            f'Removed {len(wordlist) - len(wordlist_parsed)} items from Wordlist, to conform to name regex. '
+            f'https://spec.graphql.org/June2018/#sec-Names'
+        )
+        wordlist = wordlist_parsed
 
     asyncio.run(
         blind_introspection(
@@ -118,5 +149,9 @@ def cli(argv: Optional[List[str]] = None) -> None:
             input_schema_path=args.input_schema,
             output_path=args.output,
             wordlist=wordlist,
+            proxy=args.proxy,
+            max_retries=args.max_retries,
+            backoff=args.backoff,
+            disable_ssl_verify=args.no_ssl,
         )
     )

clairvoyance/client.py

@@ -1,4 +1,5 @@
 import asyncio
+import json
 from typing import Dict, Optional
 
 import aiohttp
@@ -8,20 +9,26 @@
 
 
 class Client(IClient):
-
     def __init__(
         self,
         url: str,
         max_retries: Optional[int] = None,
         headers: Optional[Dict[str, str]] = None,
         concurrent_requests: Optional[int] = None,
+        proxy: Optional[str] = None,
+        backoff: Optional[int] = None,
+        disable_ssl_verify: Optional[bool] = None,
     ) -> None:
         self._url = url
         self._session = None
 
         self._headers = headers or {}
         self._max_retries = max_retries or 3
         self._semaphore = asyncio.Semaphore(concurrent_requests or 50)
+        self.proxy = proxy
+        self.backoff = backoff
+        self._backoff_semaphore = asyncio.Lock()
+        self.disable_ssl_verify = disable_ssl_verify or False
 
         client_ctx.set(self)
 
@@ -37,7 +44,8 @@ async def post(
 
         async with self._semaphore:
             if not self._session:
-                self._session = aiohttp.ClientSession(headers=self._headers)
+                connector = aiohttp.TCPConnector(verify_ssl=(not self.disable_ssl_verify))
+                self._session = aiohttp.ClientSession(headers=self._headers, connector=connector)
 
             # Translate an existing document into a GraphQL request.
             gql_document = {'query': document} if document else None
@@ -46,6 +54,7 @@ async def post(
                 response = await self._session.post(
                     self._url,
                     json=gql_document,
+                    proxy=self.proxy,
                 )
 
                 if response.status >= 500:
@@ -57,9 +66,17 @@ async def post(
             except (
                 aiohttp.client_exceptions.ClientConnectionError,
                 aiohttp.client_exceptions.ClientPayloadError,
+                asyncio.TimeoutError,
+                json.decoder.JSONDecodeError,
             ) as e:
                 log().warning(f'Error posting to {self._url}: {e}')
 
+            if self.backoff:
+                async with self._backoff_semaphore:
+                    delay = 0.5 * self.backoff**retries
+                    log().debug(f'Waiting for backoff {delay} seconds.')
+                    await asyncio.sleep(delay)
+
         return await self.post(document, retries + 1)
 
     async def close(self) -> None:

clairvoyance/config.py

@@ -4,8 +4,8 @@
 
 # pylint: disable=too-few-public-methods
 class Config(IConfig):
-
     def __init__(self) -> None:
+        super().__init__()
         self._bucket_size: int = 64
 
         config_ctx.set(self)

clairvoyance/entities/interfaces.py

@@ -8,7 +8,6 @@
 
 
 class IConfig(ABC):
-
     _bucket_size: int
 
     @property

clairvoyance/entities/meta.py

@@ -3,7 +3,6 @@
 
 
 class MetaEnum(EnumMeta):
-
     """Meta class for Enum."""
 
     # pylint: disable=no-value-for-parameter

clairvoyance/entities/oracle.py

@@ -0,0 +1,9 @@
+"""Oracle definitions."""
+
+from enum import Enum
+
+
+class FuzzingContext(str, Enum):
+    """Contexts."""
+    ARGUMENT = 'InputValue'
+    FIELD = 'Field'

clairvoyance/entities/primitives.py

@@ -7,7 +7,6 @@
 
 @unique
 class GraphQLPrimitive(str, Enum, metaclass=MetaEnum):
-
     """The default GraphQL Scalar primitives.
 
     ref: https://spec.graphql.org/draft/#sec-Input-Values
@@ -22,7 +21,6 @@ class GraphQLPrimitive(str, Enum, metaclass=MetaEnum):
 
 @unique
 class GraphQLKind(str, Enum, metaclass=MetaEnum):
-
     """The default GraphQL kinds.
 
     ref: https://spec.graphql.org/draft/#sec-Types