Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(py): Automatically tag pytest experiment runs with environment #1493

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat(py): Automatically tag pytest experiment runs with environment #1493

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b322fd1
Automatically tag created experiment runs with environment
jacoblee93 Feb 6, 2025
7e6eb3d
Bump to 0.3.7
jacoblee93 Feb 6, 2025
86f904a
Just use LANGSMITH_ENVIRONMENT
jacoblee93 Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 15 additions & 8 deletions python/langsmith/testing/_internal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from __future__ import annotations

Check notice on line 1 in python/langsmith/testing/_internal.py

View workflow job for this annotation

GitHub Actions / benchmark

Benchmark results 

...........
create_5_000_run_trees: Mean +- std dev: 675 ms +- 66 ms
...........
create_10_000_run_trees: Mean +- std dev: 1.37 sec +- 0.10 sec
...........
create_20_000_run_trees: Mean +- std dev: 2.74 sec +- 0.10 sec
...........
dumps_class_nested_py_branch_and_leaf_200x400: Mean +- std dev: 706 us +- 8 us
...........
dumps_class_nested_py_leaf_50x100: Mean +- std dev: 25.1 ms +- 0.3 ms
...........
dumps_class_nested_py_leaf_100x200: Mean +- std dev: 104 ms +- 3 ms
...........
dumps_dataclass_nested_50x100: Mean +- std dev: 25.3 ms +- 0.2 ms
...........
WARNING: the benchmark result may be unstable
* the standard deviation (17.5 ms) is 24% of the mean (72.5 ms)

Try to rerun the benchmark with more runs, values and/or loops.
Run 'python -m pyperf system tune' command to reduce the system jitter.
Use pyperf stats, pyperf dump and pyperf hist to analyze results.
Use --quiet option to hide these warnings.

dumps_pydantic_nested_50x100: Mean +- std dev: 72.5 ms +- 17.5 ms
...........
dumps_pydanticv1_nested_50x100: Mean +- std dev: 197 ms +- 5 ms

Check notice on line 1 in python/langsmith/testing/_internal.py

View workflow job for this annotation

GitHub Actions / benchmark

Comparison against main 

+-----------------------------------------------+----------+------------------------+
| Benchmark                                     | main     | changes                |
+===============================================+==========+========================+
| dumps_pydanticv1_nested_50x100                | 227 ms   | 197 ms: 1.15x faster   |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_branch_and_leaf_200x400 | 722 us   | 706 us: 1.02x faster   |
+-----------------------------------------------+----------+------------------------+
| dumps_dataclass_nested_50x100                 | 25.8 ms  | 25.3 ms: 1.02x faster  |
+-----------------------------------------------+----------+------------------------+
| create_5_000_run_trees                        | 682 ms   | 675 ms: 1.01x faster   |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_leaf_100x200            | 104 ms   | 104 ms: 1.00x faster   |
+-----------------------------------------------+----------+------------------------+
| dumps_class_nested_py_leaf_50x100             | 25.1 ms  | 25.1 ms: 1.00x faster  |
+-----------------------------------------------+----------+------------------------+
| create_20_000_run_trees                       | 2.71 sec | 2.74 sec: 1.01x slower |
+-----------------------------------------------+----------+------------------------+
| create_10_000_run_trees                       | 1.35 sec | 1.37 sec: 1.01x slower |
+-----------------------------------------------+----------+------------------------+
| dumps_pydantic_nested_50x100                  | 70.8 ms  | 72.5 ms: 1.02x slower  |
+-----------------------------------------------+----------+------------------------+
| Geometric mean                                | (ref)    | 1.02x faster           |
+-----------------------------------------------+----------+------------------------+

import atexit
import contextlib
Expand Down Expand Up @@ -408,6 +408,19 @@
return client.read_dataset(dataset_name=test_suite_name)


def _get_environment_metadata():
metadata = {
"revision_id": ls_env.get_langchain_env_var_metadata().get("revision_id"),
"__ls_runner": "pytest",
}

if langsmith_environment := ls_env.get_langchain_env_var_metadata().get(
"LANGSMITH_ENVIRONMENT"
):
metadata["LANGSMITH_ENVIRONMENT"] = langsmith_environment
return metadata


def _start_experiment(
client: ls_client.Client,
test_suite: ls_schemas.Dataset,
Expand All @@ -418,12 +431,7 @@
experiment_name,
reference_dataset_id=test_suite.id,
description="Test Suite Results.",
metadata={
"revision_id": ls_env.get_langchain_env_var_metadata().get(
"revision_id"
),
"__ls_runner": "pytest",
},
metadata=_get_environment_metadata(),
)
except ls_utils.LangSmithConflictError:
return client.read_project(project_name=experiment_name)
Expand Down Expand Up @@ -456,8 +464,7 @@
metadata={
**git_info,
"dataset_version": dataset_version,
"revision_id": ls_env.get_langchain_env_var_metadata().get("revision_id"),
"__ls_runner": "pytest",
**_get_environment_metadata(),
},
)
if dataset_version and git_info["commit"] is not None:
Expand Down
2 changes: 1 addition & 1 deletion python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "langsmith"
version = "0.3.6"
version = "0.3.7"
description = "Client library to connect to the LangSmith LLM Tracing and Evaluation Platform."
authors = ["LangChain <[email protected]>"]
license = "MIT"
Expand Down