Add LiteLLMAICaller class for AI model interactions, integrate settin… #20

mrT23 · 2024-05-21T18:20:38Z

User description

…gs with config loader, and update dependencies

addressing this
#15

PR Type

Enhancement, Configuration changes, Dependencies

Description

Introduced LiteLLMAICaller class for AI model interactions, integrating various API keys and settings.
Updated UnitTestGenerator to use LiteLLMAICaller.
Integrated settings retrieval in main.py and updated model selection logic.
Added Dynaconf configuration loader and get_settings function.
Updated dependencies to include litellm and dynaconf.
Added configuration template for AI models and API keys in config.toml.

Changes walkthrough 📝

Relevant files

Enhancement

LiteLLMAICaller.py Introduce `LiteLLMAICaller` class for AI model interactions. cover_agent/LiteLLMAICaller.py Added `LiteLLMAICaller` class for AI model interactions. Integrated various API keys and settings from `config_loader`. Implemented methods `call_model` and `count_tokens`.	+115/-0
UnitTestGenerator.py Update import to use `LiteLLMAICaller` in `UnitTestGenerator`. cover_agent/UnitTestGenerator.py Replaced import of `AICaller` with `LiteLLMAICaller`.	+1/-1
main.py `Integrate settings retrieval and update model selection logic.` cover_agent/main.py Integrated settings retrieval from `config_loader`. Updated model selection logic in the `main` function.	+6/-1

Configuration changes

config_loader.py Add `Dynaconf` configuration loader and `get_settings` function. settings/config_loader.py Added `Dynaconf` configuration loader. Implemented `get_settings` function to retrieve global settings.	+10/-0
config.toml `Add configuration template for AI models and API keys.` settings/config.toml Added configuration template for various AI models and API keys.	+10/-0

Dependencies

pyproject.toml Update dependencies to include `litellm` and `dynaconf`. pyproject.toml Added `litellm` and `dynaconf` to dependencies.	+2/-0

💡 PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

…gs with config loader, and update dependencies

qodo-merge-pro · 2024-05-21T18:21:12Z

PR Review 🔍

⏱️ Estimated effort to review [1-5]	3, because the PR includes multiple enhancements and configuration changes across several files, involving integration with various APIs and settings management. The complexity of the new class `LiteLLMAICaller` and its interactions with external services increases the review effort.
🧪 Relevant tests	No
⚡ Possible issues	Possible Bug: The `LiteLLMAICaller` class sets API keys directly on modules like `openai` and `litellm`, which might lead to unintended side effects or security issues if these keys are not properly managed or if they leak into logs or error messages.
	Performance Concern: The method `call_model` in `LiteLLMAICaller` could potentially block or delay due to network calls to various AI services, which might not be handled asynchronously.
	Error Handling: The `count_tokens` method in `LiteLLMAICaller` catches a broad `Exception`, which might obscure the underlying issue. More specific exception handling could be beneficial.
🔒 Security concerns	- Sensitive information exposure: The handling of API keys in the `LiteLLMAICaller` class could expose sensitive information if not properly secured. The use of environment variables or more secure storage methods is recommended to mitigate this risk.

qodo-merge-pro · 2024-05-21T18:21:37Z

CI Failure Feedback 🧐

Action: test
Failed stage: Run tests and generate reports [❌]
Failure summary: The action failed because the required test coverage of 70% was not reached. The total coverage was 63.34%. The file `cover_agent/version.py` had 91% coverage. The file `templated_tests/python_fastapi/app.py` had 60% coverage. The file `templated_tests/python_fastapi/test_app.py` had 100% coverage.
Relevant error logs: 1: ##[group]Operating System 2: Ubuntu ... 636: collected 31 items 637: templated_tests/python_fastapi/test_app.py::test_root 638: -------------------------------- live log call --------------------------------- 639: INFO httpx:_client.py:1026 HTTP Request: GET http://testserver/ "HTTP/1.1 200 OK" 640: PASSED [ 3%] 641: tests/test_AICaller.py::TestAICaller::test_initialization_with_env_var PASSED [ 6%] 642: tests/test_AICaller.py::TestAICaller::test_count_tokens PASSED [ 9%] 643: tests/test_AICaller.py::TestAICaller::test_initialization_without_env_var PASSED [ 12%] 644: tests/test_AICaller.py::TestAICaller::test_initialization_with_failed_encoding PASSED [ 16%] 645: tests/test_AICaller.py::TestAICaller::test_count_tokens_encoding_error PASSED [ 19%] 646: tests/test_CoverageProcessor.py::TestCoverageProcessor::test_parse_coverage_report_cobertura PASSED [ 22%] 647: tests/test_FilePreprocessor.py::TestFilePreprocessor::test_c_file PASSED [ 25%] 648: tests/test_FilePreprocessor.py::TestFilePreprocessor::test_py_file_with_function_only PASSED [ 29%] 649: tests/test_FilePreprocessor.py::TestFilePreprocessor::test_py_file_with_commented_class PASSED [ 32%] 650: tests/test_FilePreprocessor.py::TestFilePreprocessor::test_py_file_with_class PASSED [ 35%] 651: tests/test_PromptBuilder.py::TestPromptBuilder::test_initialization_reads_file_contents PASSED [ 38%] 652: tests/test_PromptBuilder.py::TestPromptBuilder::test_build_prompt_replaces_placeholders_correctly PASSED [ 41%] 653: tests/test_PromptBuilder.py::TestPromptBuilder::test_initialization_handles_file_read_errors PASSED [ 45%] 654: tests/test_PromptBuilder.py::TestPromptBuilder::test_empty_included_files_section_not_in_prompt PASSED [ 48%] 655: tests/test_PromptBuilder.py::TestPromptBuilder::test_non_empty_included_files_section_in_prompt PASSED [ 51%] 656: tests/test_PromptBuilder.py::TestPromptBuilder::test_empty_additional_instructions_section_not_in_prompt PASSED [ 54%] 657: tests/test_PromptBuilder.py::TestPromptBuilder::test_empty_failed_test_runs_section_not_in_prompt PASSED [ 58%] 658: tests/test_PromptBuilder.py::TestPromptBuilder::test_non_empty_additional_instructions_section_in_prompt PASSED [ 61%] 659: tests/test_PromptBuilder.py::TestPromptBuilder::test_non_empty_failed_test_runs_section_in_prompt PASSED [ 64%] 660: tests/test_ReportGenerator.py::TestReportGeneration::test_generate_report PASSED [ 67%] 661: tests/test_Runner.py::TestRunner::test_run_command_success PASSED [ 70%] 662: tests/test_Runner.py::TestRunner::test_run_command_with_cwd PASSED [ 74%] 663: tests/test_Runner.py::TestRunner::test_run_command_failure PASSED [ 77%] ... 687: tests/test_version.py::TestGetVersion::test_get_version_happy_path PASSED [ 93%] 688: tests/test_version.py::TestGetVersion::test_get_version_file_missing PASSED [ 96%] 689: tests/test_version.py::TestGetVersion::test_get_version_empty_or_whitespace_file PASSED [100%] 690: =============================== warnings summary =============================== 691: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/httpx/_client.py:680 692: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/httpx/_client.py:680: DeprecationWarning: The 'app' shortcut is now deprecated. Use the explicit style 'transport=WSGITransport(app=...)' instead. 693: warnings.warn(message, DeprecationWarning) 694: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/pydantic/_internal/_config.py:284: 25 warnings 695: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/pydantic/_internal/_config.py:284: PydanticDeprecatedSince20: Support for class-based `config` is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 696: warnings.warn(DEPRECATION_MESSAGE, DeprecationWarning) 697: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:259 698: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:259: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 699: @root_validator(pre=True) 700: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:346 701: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:346: PydanticDeprecatedSince20: `pydantic.config.Extra` is deprecated, use literal values instead (e.g. `extra='allow'`). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 702: extra = Extra.allow # Allow extra fields 703: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:349 704: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:349: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 705: @root_validator(pre=True) 706: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:378 707: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:378: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 708: @root_validator(pre=True) 709: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:425 710: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:425: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 711: @root_validator(pre=True) 712: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:494 713: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:494: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 714: @root_validator(pre=True) 715: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:514 716: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:514: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 717: @root_validator(pre=True) 718: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:527 719: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:527: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 720: @root_validator(pre=True) 721: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:572 722: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:572: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 723: @root_validator(pre=True) 724: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:609 725: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:609: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 726: @root_validator(pre=True) 727: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:927 728: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:927: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 729: @root_validator(pre=True) 730: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:954 731: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:954: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ 732: @root_validator(pre=True) 733: ../../../.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:975 734: /home/runner/.cache/pypoetry/virtualenvs/cover-agent-a0pACUfe-py3.12/lib/python3.12/site-packages/litellm/proxy/_types.py:975: PydanticDeprecatedSince20: Pydantic V1 style `@root_validator` validators are deprecated. You should migrate to Pydantic V2 style `@model_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/ ... 755: cover_agent/version.py 11 1 91% 756: templated_tests/python_fastapi/app.py 43 17 60% 757: templated_tests/python_fastapi/test_app.py 7 0 100% 758: ---------------------------------------------------------------- 759: TOTAL 461 169 63% 760: Coverage XML written to file cobertura.xml 761: FAIL Required test coverage of 70% not reached. Total coverage: 63.34% 762: ======================= 31 passed, 40 warnings in 23.39s ======================= 763: ##[error]Process completed with exit code 1.

✨ CI feedback usage guide:

The CI feedback tool (/checks) automatically triggers when a PR has a failed check.
The tool analyzes the failed checks and provides several feedbacks:

Failed stage
Failed test name
Failure summary
Relevant error logs

In addition to being automatically triggered, the tool can also be invoked manually by commenting on a PR:

/checks "https://github.com/{repo_name}/actions/runs/{run_number}/job/{job_number}"

where {repo_name} is the name of the repository, {run_number} is the run number of the failed check, and {job_number} is the job number of the failed check.

Configuration options

enable_auto_checks_feedback - if set to true, the tool will automatically provide feedback when a check is failed. Default is true.
excluded_checks_list - a list of checks to exclude from the feedback, for example: ["check1", "check2"]. Default is an empty list.
enable_help_text - if set to true, the tool will provide a help message with the feedback. Default is true.
persistent_comment - if set to true, the tool will overwrite a previous checks comment with the new feedback. Default is true.
final_update_message - if persistent_comment is true and updating a previous checks message, the tool will also create a new message: "Persistent checks updated to latest commit". Default is true.

See more information about the checks tool in the docs.

qodo-merge-pro · 2024-05-21T18:21:52Z

PR Code Suggestions ✨

Category	Suggestion	Score
Enhancement	Simplify the model selection logic by using the `get` method with a default value Simplify the model selection logic by using the `get` method with a default value. cover_agent/main.py [144-147] -if get_settings().get("config.model", None): - model = get_settings().get("config.model") -else: - model = args.openai_model +model = get_settings().get("config.model", args.openai_model) Suggestion importance[1-10]: 9 Why: This suggestion significantly simplifies the code by using the `get` method with a default value, making the code cleaner and more Pythonic. It directly addresses the logic shown in the PR code diff.	9
Performance	Store the result of `get_settings()` in a local variable to avoid multiple calls and improve performance Instead of calling `get_settings()` multiple times, store the result in a local variable to improve performance and readability. cover_agent/LiteLLMAICaller.py [15-21] -if get_settings().get("OPENAI_API_KEY", None): - openai.api_key = get_settings().OPENAI_API_KEY - litellm.openai_key = get_settings().OPENAI_API_KEY -if get_settings().get("LITELLM.DROP_PARAMS", None): - litellm.drop_params = get_settings().litellm.drop_params -if get_settings().get("OPENAI.ORG", None): - litellm.organization = get_settings().openai.org +settings = get_settings() +if settings.get("OPENAI_API_KEY", None): + openai.api_key = settings.OPENAI_API_KEY + litellm.openai_key = settings.OPENAI_API_KEY +if settings.get("LITELLM.DROP_PARAMS", None): + litellm.drop_params = settings.litellm.drop_params +if settings.get("OPENAI.ORG", None): + litellm.organization = settings.openai.org Suggestion importance[1-10]: 8 Why: This suggestion is valid and improves performance by reducing the number of function calls to `get_settings()`, which is called multiple times in the provided code. Storing the result in a local variable also enhances readability.	8
Possible issue	Ensure `boto3` is imported before using it to initialize `self.aws_bedrock_client` Add a check to ensure `boto3` is imported before using it to initialize `self.aws_bedrock_client` to avoid potential runtime errors. cover_agent/LiteLLMAICaller.py [56-59] -self.aws_bedrock_client = boto3.client( - service_name="bedrock-runtime", - region_name=get_settings().aws.bedrock_region, -) +if 'boto3' in globals(): + self.aws_bedrock_client = boto3.client( + service_name="bedrock-runtime", + region_name=get_settings().aws.bedrock_region, + ) +else: + raise ImportError("boto3 is required for AWS Bedrock integration but is not imported.") Suggestion importance[1-10]: 7 Why: This suggestion addresses a potential runtime error if `boto3` is not imported. It's a good practice to check for necessary imports before their usage, especially in a dynamic language like Python.	7
Possible issue	Add a timeout parameter to the `completion` call to prevent indefinite blocking Add a timeout parameter to the `completion` call to avoid potential indefinite blocking. cover_agent/LiteLLMAICaller.py [98] -response = completion(kwargs) +response = completion(kwargs, timeout=180) Suggestion importance[1-10]: 6 Why: Adding a timeout parameter is a good practice to avoid potential indefinite blocking. However, the existing code already includes a "force_timeout" parameter in the kwargs, which might serve a similar purpose, thus the suggestion might be slightly redundant.	6

EmbeddedDevops1 · 2024-05-21T19:39:30Z

Some things that need to be accounted for before closing this MR:

Reintroduce streaming into cover_agent/LiteLLMAICaller.py
settings/config.toml needs to be added to pyproject.toml and the pyinstaller include path
Need to test for all LiteLLM support LLMs

Add LiteLLMAICaller class for AI model interactions, integrate settin…

ceffd76

…gs with config loader, and update dependencies

mrT23 requested a review from coditamar May 21, 2024 18:20

qodo-merge-pro bot added enhancement New feature or request configuration changes dependencies Pull requests that update a dependency file labels May 21, 2024

qodo-merge-pro bot added the Review effort [1-5]: 3 label May 21, 2024

mrT23 requested a review from EmbeddedDevops1 May 21, 2024 18:21

mrT23 closed this May 22, 2024

EmbeddedDevops1 deleted the tr/litellm branch May 28, 2024 03:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add LiteLLMAICaller class for AI model interactions, integrate settin… #20

Add LiteLLMAICaller class for AI model interactions, integrate settin… #20

mrT23 commented May 21, 2024 •

edited

Loading

qodo-merge-pro bot commented May 21, 2024

qodo-merge-pro bot commented May 21, 2024

Configuration options

qodo-merge-pro bot commented May 21, 2024

EmbeddedDevops1 commented May 21, 2024 •

edited

Loading

Add LiteLLMAICaller class for AI model interactions, integrate settin… #20

Add LiteLLMAICaller class for AI model interactions, integrate settin… #20

Conversation

mrT23 commented May 21, 2024 • edited Loading

User description

PR Type

Description

Changes walkthrough 📝

qodo-merge-pro bot commented May 21, 2024

PR Review 🔍

qodo-merge-pro bot commented May 21, 2024

CI Failure Feedback 🧐

Configuration options

qodo-merge-pro bot commented May 21, 2024

PR Code Suggestions ✨

EmbeddedDevops1 commented May 21, 2024 • edited Loading

mrT23 commented May 21, 2024 •

edited

Loading

EmbeddedDevops1 commented May 21, 2024 •

edited

Loading