Skip to content

Commit

Permalink
point to pytest 0.3.1 (#638)
Browse files Browse the repository at this point in the history
  • Loading branch information
baskaryan authored Jan 22, 2025
1 parent e9ec569 commit 3e5b9e2
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 7 deletions.
15 changes: 9 additions & 6 deletions docs/evaluation/how_to_guides/pytest.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,12 @@ The JS/TS SDK has an analogous [Vitest/Jest integration](./vitest_jest).

## Installation

This functionality requires Python SDK version `langsmith>=0.3`.
This functionality requires Python SDK version `langsmith>=0.3.1`.

For extra features like [rich terminal outputs](./pytest#rich-outputs) and [test caching](./pytest#caching) install with `pip install -U "langsmith[pytest]"`.
For extra features like [rich terminal outputs](./pytest#rich-outputs) and [test caching](./pytest#caching) install:
```bash
pip install -U "langsmith[pytest]"
```

## Define and run tests

Expand Down Expand Up @@ -70,11 +73,11 @@ def test_sql_generation_select_all() -> None:
user_query = "Get all users from the customers table"
t.log_inputs({"user_query": user_query}) # <-- Log example inputs, optional

sql = generate_sql(user_query)
t.log_outputs({"sql": sql}) # <-- Log run outputs, optional

expected = "SELECT * FROM customers;"
t.log_reference_outputs({"sql": expected}) # <-- Log example reference outputs, optional

sql = generate_sql(user_query)
t.log_outputs({"sql": sql}) # <-- Log run outputs, optional

t.log_feedback(key="valid_sql", score=is_valid_sql(sql)) # <-- Log feedback, optional

Expand Down Expand Up @@ -131,7 +134,7 @@ def test_foo() -> None:

Running this test will create/update an example with name "test_foo", inputs `{"a": 1, "b": 2}`, reference outputs `{"foo": "bar"}` and trace a run with outputs `{"foo": "baz"}`.

**NOTE**: If you run any of `log_inputs`, `log_outputs`, or `log_reference_outputs` twice the previous values will be overwritten.
**NOTE**: If you run `log_inputs`, `log_outputs`, or `log_reference_outputs` twice, the previous values will be overwritten.

Another way to define example inputs and reference outputs is via pytest fixtures/parametrizations.
By default any arguments to your test function will be logged as inputs on the corresponding example.
Expand Down
2 changes: 1 addition & 1 deletion docs/evaluation/tutorials/testing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ Next, install the testing framework:
value: "python",
label: "Pytest",
language: "bash",
content: `# Make sure you have langsmith>=0.3\npip install -U "langsmith[pytest]"`,
content: `# Make sure you have langsmith>=0.3.1\npip install -U "langsmith[pytest]"`,
},
{
value: "vitest",
Expand Down

0 comments on commit 3e5b9e2

Please sign in to comment.