Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Documentation Generation: Transition from Mermaid to Code2Flow #27

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Refactor Documentation Generation: Transition from Mermaid to Code2Flow #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4beda10
Revert Flowchart Implementation from Mermaid to code2flow
TroyDoesAI Dec 5, 2023
5497c31
Enhanced flowchart generation with additional code2flow options
TroyDoesAI Dec 5, 2023
7a48971
Added checks for code2flow and Graphviz in check_tools_and_files func…
TroyDoesAI Dec 5, 2023
3852b54
Refactor README for Code2Flow Integration and Remove Mermaid References
TroyDoesAI Dec 5, 2023
359a501
Removed Ref to Mermaid
TroyDoesAI Dec 6, 2023
68217ec
This is the most popular AI tool today December 2023
TroyDoesAI Dec 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 144 additions & 0 deletions Colab-TextGen-GPU.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"private_outputs": true,
"provenance": [],
"gpuType": "T4",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/github/Troys-Code/AI-Automation/blob/main/Colab-TextGen-GPU.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"# oobabooga/text-generation-webui\n",
"\n",
"After running both cells, a public gradio URL will appear at the bottom in a few minutes. You can optionally generate an API link.\n",
"\n",
"* Project page: https://github.com/oobabooga/text-generation-webui\n",
"* Gradio server status: https://status.gradio.app/"
],
"metadata": {
"id": "MFQl6-FjSYtY"
}
},
{
"cell_type": "code",
"source": [
"#@title 1. Keep this tab alive to prevent Colab from disconnecting you { display-mode: \"form\" }\n",
"\n",
"#@markdown Press play on the music player that will appear below:\n",
"%%html\n",
"<audio src=\"https://oobabooga.github.io/silence.m4a\" controls>"
],
"metadata": {
"id": "f7TVVj_z4flw"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#@title 2. Launch the web UI\n",
"\n",
"#@markdown If unsure about the branch, write \"main\" or leave it blank.\n",
"\n",
"import torch\n",
"from pathlib import Path\n",
"\n",
"if Path.cwd().name != 'text-generation-webui':\n",
" print(\"Installing the webui...\")\n",
"\n",
" !git clone https://github.com/oobabooga/text-generation-webui\n",
" %cd text-generation-webui\n",
"\n",
" torver = torch.__version__\n",
" print(f\"TORCH: {torver}\")\n",
" is_cuda118 = '+cu118' in torver # 2.1.0+cu118\n",
" is_cuda117 = '+cu117' in torver # 2.0.1+cu117\n",
"\n",
" textgen_requirements = open('requirements.txt').read().splitlines()\n",
" if is_cuda117:\n",
" textgen_requirements = [req.replace('+cu121', '+cu117').replace('+cu122', '+cu117').replace('torch2.1', 'torch2.0') for req in textgen_requirements]\n",
" elif is_cuda118:\n",
" textgen_requirements = [req.replace('+cu121', '+cu118').replace('+cu122', '+cu118') for req in textgen_requirements]\n",
" with open('temp_requirements.txt', 'w') as file:\n",
" file.write('\\n'.join(textgen_requirements))\n",
"\n",
" !pip install -r extensions/openai/requirements.txt --upgrade\n",
" !pip install -r temp_requirements.txt --upgrade\n",
"\n",
" print(\"\\033[1;32;1m\\n --> If you see a warning about \\\"previously imported packages\\\", just ignore it.\\033[0;37;0m\")\n",
" print(\"\\033[1;32;1m\\n --> There is no need to restart the runtime.\\n\\033[0;37;0m\")\n",
"\n",
" try:\n",
" import flash_attn\n",
" except:\n",
" !pip uninstall -y flash_attn\n",
"\n",
"# Parameters\n",
"model_url = \"https://huggingface.co/TheBloke/MythoMax-L2-13B-GPTQ\" #@param {type:\"string\"}\n",
"branch = \"gptq-4bit-32g-actorder_True\" #@param {type:\"string\"}\n",
"command_line_flags = \"--n-gpu-layers 128 --load-in-4bit --use_double_quant\" #@param {type:\"string\"}\n",
"api = False #@param {type:\"boolean\"}\n",
"\n",
"if api:\n",
" for param in ['--api', '--public-api']:\n",
" if param not in command_line_flags:\n",
" command_line_flags += f\" {param}\"\n",
"\n",
"model_url = model_url.strip()\n",
"if model_url != \"\":\n",
" if not model_url.startswith('http'):\n",
" model_url = 'https://huggingface.co/' + model_url\n",
"\n",
" # Download the model\n",
" url_parts = model_url.strip('/').strip().split('/')\n",
" output_folder = f\"{url_parts[-2]}_{url_parts[-1]}\"\n",
" branch = branch.strip('\"\\' ')\n",
" if branch.strip() not in ['', 'main']:\n",
" output_folder += f\"_{branch}\"\n",
" !python download-model.py {model_url} --branch {branch}\n",
" else:\n",
" !python download-model.py {model_url}\n",
"else:\n",
" output_folder = \"\"\n",
"\n",
"# Start the web UI\n",
"cmd = f\"python server.py --share\"\n",
"if output_folder != \"\":\n",
" cmd += f\" --model {output_folder}\"\n",
"cmd += f\" {command_line_flags}\"\n",
"print(cmd)\n",
"!$cmd"
],
"metadata": {
"id": "LGQ8BiMuXMDG",
"cellView": "form"
},
"execution_count": null,
"outputs": []
}
]
}
111 changes: 0 additions & 111 deletions documentation/create_overview_doc/AI_Prompts/fix_mermaid_syntax_prompt.txt
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion documentation/create_overview_doc/AI_Prompts/mermaid_doc_prompt.txt
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion documentation/create_overview_doc/AI_Prompts/system_introduction_prompt.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Give a brief high level system overview and its purpose using the provided context for this program, keep it detailed, concise and confident.
Write a high level system overview and its purpose using the provided context for this program, keep it detailed, concise, and more importantly factual.
4 changes: 0 additions & 4 deletions documentation/create_overview_doc/AI_Prompts/system_overview_mermaid_update_prompt.txt
View file
Open in desktop

This file was deleted.

36 changes: 15 additions & 21 deletions documentation/create_overview_doc/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,17 @@ This powerful tool streamlines the creation of high-level documentation for soft

- **Auto-Retry Logic and Enhanced Error Handling**: Ensures reliable documentation generation even in unstable environments through sophisticated error handling and retry mechanisms.

- **Visualization with Mermaid.js Flow Maps and Enhanced Syntax Handling**: AI-generated visual flow maps to represent module interactions. Now with improved handling of Mermaid diagram syntax, minimizing manual interventions.

- **Mermaid CLI Image Conversion**: Transforms Mermaid.js diagrams into images, offering a clear visual representation of your code architecture.
- **Visualization with Code2Flow**: AI-generated visual flow maps to represent module interactions, configured with flags such as --hide-legend, --no-trimming, --verbose, and --skip-parse-errors for optimal detail and clarity.

- **Code2Flow CLI Image Conversion**: The tool uses Code2Flow for creating detailed flow maps, offering a clear visual representation of your code architecture.
- **Customizable Flowchart Generation: Code2Flow** Configured with flags such as --hide-legend, --no-trimming, --verbose, and --skip-parse-errors to optimize the detail and clarity of flowcharts.**
Comment on lines +22 to +25
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security Issue: The integration of Code2Flow in the documentation generation process introduces potential security risks if the flags and configurations are not properly validated. The use of flags such as --no-trimming and --verbose could inadvertently expose sensitive information or system details through the generated documentation if not handled securely.
Fix: Implement input validation for the flags and configurations used in Code2Flow to ensure that only safe and intended modifications are allowed. This could involve sanitizing inputs to remove potentially harmful options or validating against a list of allowed flags.
Code Suggestion:

- Visualization with Code2Flow: AI-generated visual flow maps to represent module interactions, configured with flags such as --hide-legend, --no-trimming, --verbose, and --skip-parse-errors for optimal detail and clarity.
+ Visualization with Code2Flow: AI-generated visual flow maps to represent module interactions, configured with validated flags for optimal detail and clarity.

Comment on lines +22 to +25
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scalability Issue: The introduction of Code2Flow with additional flags such as --hide-legend, --no-trimming, --verbose, and --skip-parse-errors may introduce scalability concerns due to increased complexity and potential processing overhead. These options could lead to more intensive computation and memory usage, especially with large codebases.
Fix: Consider implementing conditional logic to enable these detailed features only when necessary, or provide configuration options to adjust the level of detail based on the system's capacity and expected load.
Code Suggestion:

+ if (systemCapacity >= HIGH_CAPACITY) {
+     configureCode2Flow(--hide-legend, --no-trimming, --verbose, --skip-parse-errors);
+ } else {
+     configureCode2Flow();
+ }

Comment on lines +22 to +25
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance Issue: The integration of Code2Flow with additional flags such as --hide-legend, --no-trimming, --verbose, and --skip-parse-errors might lead to increased computational overhead and potential performance bottlenecks due to the verbose output and no trimming options. These settings can significantly increase the amount of data processed and stored, impacting the performance of the documentation generation tool.
Fix: Consider implementing conditional flags or environment-based configurations that enable detailed logging and no trimming only during development or debugging sessions. For production usage, default settings should favor performance with minimal verbosity and enabled trimming to reduce the load.
Code Suggestion:

- Visualization with Code2Flow: AI-generated visual flow maps to represent module interactions, configured with flags such as --hide-legend, --no-trimming, --verbose, and --skip-parse-errors for optimal detail and clarity.
+ Visualization with Code2Flow: AI-generated visual flow maps to represent module interactions, configured with environment-based flags to optimize detail and clarity while maintaining performance.

Comment on lines +22 to +25
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optimization Issue: The introduction of multiple flags such as --hide-legend, --no-trimming, --verbose, and --skip-parse-errors in the Code2Flow CLI might lead to increased complexity and potential performance overhead. Each flag potentially adds additional processing layers or disables optimizations, which could affect the performance of the flowchart generation, especially in larger projects.
Fix: Evaluate the necessity of each flag in the context of typical use cases. If certain flags are seldom used or significantly impact performance, consider providing a more streamlined default configuration with optional flags that can be enabled as needed. This approach balances flexibility with performance.
Code Suggestion:

- **Visualization with Code2Flow**: AI-generated visual flow maps to represent module interactions, configured with flags such as --hide-legend, --no-trimming, --verbose, and --skip-parse-errors for optimal detail and clarity.
+ **Visualization with Code2Flow**: AI-generated visual flow maps to represent module interactions. Default settings optimized for performance with optional flags for detailed configuration.

- **Skippable Files and Directories**: Customize which files and directories to ignore during documentation via the `skip_list.csv`.

- **Comprehensive Documentation**: Generates detailed markdown files for each module and compiles them into an overarching High_Level_Doc, complete with visual flow maps.

- **Documentation Metrics Logging**: Track your session duration and token usage metrics, recorded in `bito_usage_log.txt`.

- **Required Tool and File Verification**: Checks for the presence of necessary tools ("bito", "mmdc") and prompt files before starting the documentation process.
- **Required Tool and File Verification**: Checks for the presence of necessary tools ("bito", "code2flow", "dot") and prompt files before starting the documentation process.

## Supported Languages

Expand All @@ -51,21 +51,27 @@ sh
kt
```

For flow map generation with Code2Flow, the tool currently supports the following languages, each with specific dependencies:

- JavaScript: Requires Acorn for parsing. Ensure Acorn is installed and accessible in your environment.
- Ruby: Uses Parser for parsing. Ensure the Parser gem is installed and accessible.
- PHP: Employs PHP-Parser for parsing. PHP-Parser should be installed and accessible in your environment.
Comment on lines +56 to +58
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security Issue: The code introduces potential security risks by assuming that dependencies like Acorn, Parser, and PHP-Parser are installed without verifying their presence or versions. This could lead to execution with outdated or vulnerable versions of these dependencies.
Fix: Implement checks to verify that the required dependencies are installed and up-to-date before proceeding with their usage.
Code Suggestion:

+if ! npm list -g acorn; then npm install -g acorn@latest; fi
+if ! gem list -i parser; then gem install parser; fi
+if ! php -m | grep -q 'PHP-Parser'; then pecl install PHP-Parser; fi

Comment on lines +56 to +58
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scalability Issue: The addition of language-specific dependencies for Code2Flow might introduce scalability issues if not managed properly. Each language parser (Acorn for JavaScript, Parser gem for Ruby, PHP-Parser for PHP) increases the complexity and potential overhead during the documentation generation process. This could lead to increased memory usage and slower processing times, especially with large codebases or multiple concurrent documentation generation processes.
Fix: Consider implementing a lazy loading mechanism or an on-demand initialization of these parsers only when a specific language is detected in the project being documented. This approach minimizes the memory footprint and initialization overhead by loading resources only when necessary.
Code Suggestion:

+if project_languages.include? 'JavaScript'
+  require 'acorn'
+end
+if project_languages.include? 'Ruby'
+  require 'parser'
+end
+if project_languages.include? 'PHP'
+  require 'php-parser'
+end

Comment on lines +56 to +58
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance Issue: The addition of language-specific dependencies for Code2Flow might introduce performance bottlenecks if these dependencies are not optimized or if they introduce significant overhead during the flow map generation process.
Fix: Ensure that the dependencies such as Acorn for JavaScript, Parser gem for Ruby, and PHP-Parser are optimized for performance. Consider lazy loading these dependencies or using alternative lightweight libraries if performance issues are detected.
Code Suggestion:

+Ensure that the dependencies such as Acorn for JavaScript, Parser gem for Ruby, and PHP-Parser are optimized for performance. Consider lazy loading these dependencies or using alternative lightweight libraries if performance issues are detected.

- Python: No additional dependencies are required for Python projects.

If your project uses a language not supported by Code2Flow for flow map generation, the documentation generation capabilities will still be available, but without the flow map visualization for those specific languages.
Comment on lines +56 to +61
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optimization Issue: The implementation details for language support in Code2Flow are mentioned, but there is no error handling for unsupported languages which could lead to failures or unhandled exceptions during the documentation generation process.
Fix: Implement error handling to manage cases where the project uses an unsupported language. This could involve skipping the flow map generation for those languages with a clear log message or providing a fallback mechanism.
Code Suggestion:

+if [ ! supported_languages.contains($language) ]; then
+    echo "Warning: Language $language is not supported for flow map generation. Skipping..."
+    continue
+fi


Comment on lines +54 to +62
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue: The addition of language-specific dependency information for Code2Flow is useful but lacks clarity on how to install these dependencies. This could lead to confusion or errors during setup by end-users.
Fix: Provide explicit commands or steps on how to install each mentioned dependency for different operating systems or environments. This will ensure users can easily set up their environment without needing to search for installation instructions elsewhere.
Code Suggestion:

+For flow map generation with Code2Flow, the tool currently supports the following languages, each with specific dependencies:
+- JavaScript: Requires Acorn for parsing. Install it using 'npm install acorn -g'.
+- Ruby: Uses Parser for parsing. Install it using 'gem install parser'.
+- PHP: Employs PHP-Parser for parsing. Install it using 'composer require nikic/php-parser'.
+- Python: No additional dependencies are required for Python projects.

## Prerequisites

Ensure the following tools are installed:

- `bito` : https://github.com/gitbito/CLI

- `mermaidcli` : https://github.com/mermaid-js/mermaid-cli
- `Code2Flow:` : https://github.com/scottrogowski/code2flow

Also, make sure these prompt files are present in a specified prompt folder (`AI_Prompts` by default):

- `high_level_doc_prompt.txt`
- `mermaid_doc_prompt.txt`
- `fix_mermaid_syntax_prompt.txt`
- `system_introduction_prompt.txt`
- `system_overview_mermaid_update_prompt.txt`

## How to Use

Expand All @@ -87,7 +93,7 @@ The directory includes:

- Module Documentation: Individual markdown files for each module, titled `<module_name>_Doc.md`, detailing the module's purpose, functions, and interactions.

- Aggregated Documentation: A comprehensive markdown file `High_Level_Doc.md`, which consolidates the documentation from each module. This file also includes SVG format flow maps created by Mermaid.js for a visual overview of module interactions, and a final Full System Flow Map in PNG format generated by code2flow for a broader system perspective.
- Aggregated Documentation: A comprehensive markdown file `High_Level_Doc.md`, which consolidates the documentation from each module. This file also includes SVG format flow maps created by Code2Flow for a visual overview of module interactions, and a final Full System Flow Map in PNG format generated by code2flow for a broader system perspective.

## Skip List

Expand Down Expand Up @@ -123,18 +129,6 @@ To customize the Skip List to fit your project's needs, follow these steps:
5. **Re-run the Script**:
- Execute the script again to apply the new Skip List settings.

## Known Issues and Solutions

### Syntax Errors in Mermaid Diagrams
- **Issue**: Occasional syntax errors in AI-generated Mermaid diagrams, such as misplaced quotes or empty parentheses.
- **Current Solutions**:
- **Automated Fixes**: Script (`fix_mermaid_syntax`) and AI-driven (`fix_mermaid_syntax_with_bito`) methods are used for common syntax corrections.
- **Manual Editing**: For unresolved errors, manual editing can be done using the [Mermaid Live Editor](https://mermaid.live/).
- **Update Command**: Post-editing, update diagrams in your markdown documentation using:
```
mmdc -i High_Level_Doc.md -o High_Level_Doc.md
```

### Ongoing Efforts
- We're continuously improving our AI models and scripts based on user feedback.
- Future updates will focus on reducing manual intervention and enhancing the user experience.
Expand Down
Loading