old_code.qmd

# Polars Series

Much like Pandas, polars' fundamental building block is the series. A series is a column of data, with a name, and a dtype.

## Series-Object Housekeeping

Construct a series

```{python}
s = pl.Series("a", [1, 2, 3])
s
```

Make pandas series for comparison:

```{python}
s_pandas = pd.Series([1, 2, 3], name = "a")
```

```{python}
type(s)
```

```{python}
type(s_pandas)
```

```{python}
s.dtype
```

```{python}
s_pandas.dtype
```

Renaming a series; will be very useful when operating on dataframe columns.

```{python}
s.alias("b")
```

```{python}
s.clone()
```

```{python}
s.clone().append(pl.Series("a", [4, 5, 6]))
```

Note: `series.append` operates in-place. That is why we cloned the series first.

Flatten a list of lists using `explode()`; this will not work for more than 2 levels of nesting.

```{python}
pl.Series("a", [[1, 2], [3, 4], [9, 10]]).explode()
```

```{python}
s.extend_constant(666, n=2)
```

```{python}
s.rechunk()
```

```{python}
s.rename("b", in_place=False) # has an in_place option. Unlike .alias()
```

```{python}
s.to_dummies()
```

```{python}
s.clear() # creates an empty series, with same dtype. Previously called s.cleared()
```

Constructing a series of floats, for later use.

```{python}
f = pl.Series("a", [1., 2., 3.])
f
```

```{python}
f.dtype
```

## Memory Representation of Series

Object size in memory. Super useful for profiling:

```{python}
s.estimated_size(unit="gb")
```

```{python}
s.chunk_lengths() # what is the length of each memory chunk?
```



## Filtering and Subsetting

```{python}
s[0] # same as s.__getitem__(0)
```

Filtering with `[` and Booleans will not work:

```{python}
#| eval: false
s[[True, False, True]]
```

Filtering with a Polars Boolean series, worked in previous versions of polars (\<=15). Currently it does not.

```{python}
#| eval: false
s[pl.Series("a", [True, False, True])]
```

Filtering with a pandas (Boolean) series will not work (why should it?), nor with a numpy array.

For an easy transition to work with lazy dataframes and query planning (@sec-query-planning), you may want to prefer the `filter` method, which can actually take a polars series, or list of booleans (but not a pandas series or numpy array):

```{python}
s.filter(pl.Series("a", [True, False, True])) # works
```

```{python}
s.filter([True, False, True])
```

```{python}
s.head(2)
```

```{python}
s.limit(2)
```

```{python}
s.tail(2)
```

```{python}
s.sample(2, with_replacement=False)
```

```{python}
s.take([0, 2]) # same as s[0,2] and pandas .iloc[[0,2]]
```

```{python}
s.slice(1, 2) # same as pandas .iloc[1:2]
```

```{python}
s.take_every(2) # same as pandas .iloc[::2]
```


## Aggregations

::: {.callout-important}
The following are Polars series methods. Almost all have their expression equivalent (which should be preferred). 
:::

```{python}
s.sum()
```

```{python}
s.min()
```

```{python}
s.arg_min()
```

```{python}
s.max()
```

```{python}
s.arg_max()
```

```{python}
s.mean()
```

```{python}
s.median()
```

```{python}
s.entropy()
```

```{python}
s.describe() # Note the different output than Pandas.
```

```{python}
s.value_counts()
```




### Examples of Series Methods that do not have an Expression Equivalent

For these examples we use the `.select()` context, which will be presented in @sec-sec-dataframes.

A method that will work:
```{python}
pl.DataFrame(s).select(pl.col("a").sum())
```

Methods that will not work:
```{python}
#| eval: false
pl.DataFrame(s).select(pl.col("a").describe())
pl.DataFrame(s).select(pl.col("a").value_counts())
```



## Shape Transformations

```{python}
pl.Series("a",[1,2,3,4]).reshape((2,2))
```

```{python}
s.shift(1)
```

```{python}
s.shift(-1)
```

```{python}
s.shift_and_fill(999)
```


## Mathematical Transformations

```{python}
s.abs()
```

```{python}
s.sin()
```

```{python}
s.exp()
```

```{python}
s.hash()
```

```{python}
s.log()
```

```{python}
s.peak_max()
```

```{python}
s.sqrt()
```

## Comparisons

```{python}
s.clip_max(2)
```

```{python}
s.clip_min(1)
```

```{python}
s.clip(1,2) # AKA Winsorizing
```

You cannot round integers, but you can round floats.

```{python}
f.round(2)
```

```{python}
f.ceil()
```

```{python}
f.floor()
```


## Search

```{python}
s.is_in(pl.Series([1, 10]))
```

```{python}
s.is_in([1, 10])
```


```{python}
s.is_between(1, 10)
```



## Apply (map_elements)

Applying your own function:

```{python}
s.map_elements(lambda x: x + 1)
```

Using your own functions comes with a performance cost:

```{python}
s1 = pl.Series(np.random.randn(int(1e5)))
```

Adding 1 with apply:

```{python}
%timeit -n2 -r2 s1.map_elements(lambda x: x + 1)
```

Adding 1 without apply:

```{python}
%timeit -n2 -r2 s1+1
```


Things to note:

- Polars had an apply method until version 19.0.0, where it has been replaced with `map_elements()`. Other family members include:
  - `map` to `map_batches` 
  - `GroupBy.apply` to `map_groups` .
  - `DataFrame.apply` to `map_rows`. 
  - `Series/Expr.rolling_apply` to `rolling_map` .
  - `Series/Expr.apply` to `map_elements` .
- `map_elements()` (i.e. apply in Pandas) is usually very slow. As of version 18, Polars will tell you how to make your code more efficient(!).



## Cumulative Operations

```{python}
s.cummax()
```

```{python}
s.cumsum()
```

```{python}
s.cumprod()
```

```{python}
s.ewm_mean(com=0.5)
```

## Differentiation Operations

```{python}
s.diff()
```

```{python}
s.pct_change()
```

## Windowed Operations

```{python}
s.rolling_map(
  sum, 
  window_size=2)
```

Not all functions will work within a `rolling_apply`! Only polars' functions will.

```{python}
#| eval: false

s.rolling_map(np.sum, window_size=2) # will not work
```

Some rolling functions have been prepared for you.

```{python}
s.rolling_max(window_size=2)
```

## Logical Aggregations

```{python}
b = pl.Series("a", [True, True, False])
b.dtype
```

```{python}
b.all()
```

```{python}
b.any()
```

```{python}
~b
```



## Uniques and Duplicates

```{python}
s.is_duplicated()
```

```{python}
s.is_unique()
```

```{python}
s.unique() # Same as Pandas drop_duplicates()
```

```{python}
s.n_unique()
```

```{python}
pl.Series([1,2,3,4,1]).unique_counts()
```

The first appearance of a value in a series:

```{python}
pl.Series([1,2,3,1]).is_first()
```

```{python}
pl.Series([1,2,3,1]).first() # no series method
```

But has an expression method! Very useful within a `group_by()`.

```{python}
pl.DataFrame(pl.Series("a",[1,2,3,1])).select(pl.col('a').first())
```



## dtypes

### Testing

```{python}
s.is_numeric()
```

```{python}
s.is_float()
```

```{python}
s.is_utf8()
```

```{python}
s.is_boolean()
```

```{python}
s.is_temporal() # previously called .is_datelike()
```



### Casting

```{python}
s.cast(pl.Int32)
```

Things to note:

-   The dtypes to cast to are **polars** dtypes. Don't try `s.cast("int32")`, `s.cast(np.int32)`, or `s.cast(pd.int)`
-   `cast()` is polars' equivalent of pandas' `astype()`.
-   For a list of dtypes see the official [documentation](see%20https://pola-rs.github.io/polars/py-polars/html/reference/datatypes.html).


### Optimizing dtypes

Find the most efficient dtype for a series:

```{python}
s.shrink_dtype() # like pandas pd.to_numeric(..., downcast="...") and pandas_dtype_efficiency module

```

Also see [here](http://braaannigan.github.io/software/2022/10/31/polars-dtype-diet.html).

Shrink the memory allocation to the size of the actual data (in place).

```{python}
s.shrink_to_fit() 
```


## Ordering and Sorting

```{python}
s.sort()
```

```{python}
s.reverse()
```

```{python}
s.rank()
```

```{python}
s.arg_sort() 
```

`arg_sort()` returns the indices that would sort the series. Same as R's `order()`.

```{python}
(s.sort() == s[s.arg_sort()]).all()
```

`arg_sort()` can also be used to return the original series from the sorted one:

```{python}
(s == s[s[s.arg_sort()].arg_sort()]).all()
```

```{python}
s.shuffle(seed=1) # random permutation
```

## Missing

Pandas users will be excited to know that thanks to arrow, polars has built in missing value support for all(!) dtypes. This has been a long awaited feature in the Python data science ecosystem with implications on speed, memory, style and more. The [Polars User Guide](https://pola-rs.github.io/polars-book/user-guide/howcani/missing_data.html) has a great overview of the topic from which we collect some take-homes:

-   `np.nan` is also supported along `pl.Null`, but is not considered as a missing value by polars. This has implications on null counts, statistical aggregations, etc.
-   `pl.Null`, and `np.nan`s have their own separate functions for imputing, counting, etc.

PS - Arrow support is also expected in [Pandas 2.0](https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i).

```{python}
m = pl.Series("a", [1, 2, None, np.nan])
```

```{python}
m.is_null() # checking for None's. Like pandas .isna()
```

```{python}
m.is_nan() # checking for np.nan's
```

For comparison with pandas:

```{python}
m_pandas = pd.Series([1, 2, None, np.nan])
```

```{python}
m_pandas.isna()
```

```{python}
m_pandas.isnull() # alias for pd.isna()
```

```{python}
# Polars
m1 = pl.Series("a", [1, None, 2, ]) # python native None
m2 = pl.Series("a", [1, np.nan, 2, ]) # numpy's nan
m3 = pl.Series("a", [1, float('nan'), 2, ]) # python's nan

# Pandas
m4 = pd.Series([1, None, 2 ])
m5 = pd.Series([1, np.nan, 2, ])
m6 = pd.Series([1, float('nan'), 2, ])
```

```{python}
[m1.sum(), m2.sum(), m3.sum(), m4.sum(), m5.sum(), m6.sum()]
```

Things to note:

-   Aggregating pandas and polars series behave differently w.r.t. missing values:
    -   Both will ignore `None`; which is unsafe.
    -   Polars will not ignore `np.nan`; which is safe. Pandas is unsafe w.r.t. `np.nan`, and will ignore it.

Filling missing values; `None` and `np.nan` are treated differently:

```{python}
m1.fill_null(0)
```

```{python}
m2.fill_null(0)
```

```{python}
m2.fill_nan(0)
```

```{python}
m1.drop_nulls()
```

```{python}
m1.drop_nans()
```

```{python}
m2.drop_nulls()
```

```{python}
m1.interpolate()
```

```{python}
m2.interpolate() # np.nan is not considered missing, so why interpolate?
```

## Export To Other Python Objects

The current section deals with exports to other python objects in memory. See @sec-disk-export for exporting to disk.

```{python}
s.to_frame() 
```

```{python}
s.to_list()
```

```{python}
s.to_numpy() # useful for preparing data for learning with scikit-learn
```

```{python}
s.to_pandas()
```

```{python}
s.to_arrow() # useful for preparing data for learning with XGBoost. Maybe sklearn in the future?
```

## Strings

Like Pandas, accessed with the `.str` attribute.

```{python}
st = pl.Series("a", ["foo", "bar", "baz"])
```

```{python}
st.str.n_chars() # gets number of chars. In ASCII this is the same as lengths()
```

```{python}
st.str.lengths() # gets number of bytes in memory
```

```{python}
st.str.concat("-")
```

```{python}
st.str.contains("foo|tra|bar")
```

```{python}
st.str.count_match(pattern= 'o') # count literal metches
```

Regex is supported. The `r` prefix in `r"<regex pattern>"` is useful for emphasizing regular expressions, but not really necessary (more about it [here](https://stackoverflow.com/questions/2241600/python-regex-r-prefix)).

```{python}
st.str.count_match(pattern=r"\w") # \w is regex for alphanumeric
```

```{python}
st.str.ends_with("oo")
```

```{python}
st.str.starts_with("fo")
```

To extract the **first** appearance of a pattern, use `extract`:

```{python}
url = pl.Series("a", [
            "http://vote.com/ballon_dor?candidate=messi&ref=polars",

            "http://vote.com/ballon_dor?candidate=jorginho&ref=polars",

            "http://vote.com/ballon_dor?candidate=ronaldo&ref=polars"
            ])
```

```{python}
url.str.extract(r"=(\w+)", 1) # "=(\w+)" is read: match an equality, followed by any number of alphanumerics.
```

To extract **all** appearances of a pattern, use `extract_all`:

```{python}
url.str.extract_all("=(\w+)")
```

```{python}
st.str.ljust(8, "*")
```

```{python}
st.str.rjust(8, "*")
```

```{python}
st.str.lstrip('f')
```

```{python}
st.str.rstrip('r')
```

Replacing first appearance of a pattern:

```{python}
st.str.replace(r"o+", "ZZ")  
```

Replace all appearances of a pattern:

```{python}
st.str.replace_all("o", "ZZ")
```

String to list of strings. Number of splits inferred.

```{python}
st.str.split(by="o")
```

```{python}
st.str.split(by="a", inclusive=True)
```

String to dict of strings. Number of **splits** fixed.

```{python}
st.str.split_exact("a", 2)
```

String to dict of strings. **Length of output** fixed.

```{python}
st.str.splitn("a", 4)
```

Strip white spaces.

```{python}
pl.Series(['   ohh   ','   yeah   ']).str.strip()
```

```{python}
st.str.to_uppercase()
```

```{python}
st.str.to_lowercase()
```

```{python}
st.str.zfill(5)
```

```{python}
st.str.slice(offset=0, length=2)
```

## Date and Time

There are 4 datetime dtypes in polars:

1.  Date: A date, without hours. Generated with `pl.Date()`.
2.  Datetime: Date and hours. Generated with `pl.Datetime()`.
3.  Time: Hour of day. Generated with `pl.Time()`.
4.  Duration: As the name suggests. Similar t o `timedelta` in pandas. Generated with `pl.Duration()`.

**Warning**: Python has a sea of modules that support datetimes. A partial list includes: [datetime module](https://docs.python.org/3/library/datetime.html), extensions in [dateutil](https://dateutil.readthedocs.io/en/stable/), [numpy](https://numpy.org/doc/stable/reference/arrays.datetime.html), [pandas](https://pandas.pydata.org/pandas-docs/version/1.1/user_guide/timeseries.html), [arrow](https://arrow.readthedocs.io/en/latest/), the deprecated [scikits.timeseries](https://pytseries.sourceforge.net/) and certainly others. Be aware of the dtype you are using, and the accompanying methods.

### Time Range

```{python}
from datetime import datetime, timedelta

start = datetime(year= 2001, month=2, day=2)
stop = datetime(year=2001, month=2, day=3)

date = pl.date_range(
  low=start, 
  high=stop, 
  interval=timedelta(seconds=500*61))
date
```

Things to note:

-   How else could I have constructed this series? What other types are accepted as `low` and `high`?
-   `pl.date_range` may return a series of dtype `Date` or `Datetime`. This depens of the granularity of the inputs.

```{python}
date.dtype
```

Cast to different time unit. May be useful when joining datasets, and the time unit is different.

```{python}
date.dt.cast_time_unit(tu="ms")
```

### Extract Time Sub-Units

```{python}
date.dt.second()
```

```{python}
date.dt.minute()
```

```{python}
date.dt.hour()
```

```{python}
date.dt.day()
```

```{python}
date.dt.week()
```

```{python}
date.dt.weekday()
```

```{python}
date.dt.month()
```

```{python}
date.dt.year()
```

```{python}
date.dt.ordinal_day() # day in year
```

```{python}
date.dt.quarter()
```

### Durations

Equivalent to Pandas `period` dtype.

```{python}
diffs = date.diff()
diffs
```

```{python}
diffs.dtype
```

```{python}
diffs.dt.seconds()
```

```{python}
diffs.dt.minutes()
```

```{python}
diffs.dt.days()
```

```{python}
diffs.dt.hours()
```

### Date Aggregations

Note that aggregating dates, returns a `datetime` type object.

```{python}
date.dt.max()
```

```{python}
date.dt.min()
```

```{python}
date.dt.mean()
```

```{python}
date.dt.median()
```

### Date Transformations

Notice the syntax of `offset_by`. It is similar to R's `lubridate` package.

```{python}
date.dt.offset_by(by="1y2m20d")
```

Negative offset is also allowed.

```{python}
date.dt.offset_by(by="-1y2m20d")
```

```{python}
date.dt.round("1y")
```

```{python}
date2 = date.dt.truncate("30m") # round to period
pd.crosstab(date,date2)
```

### From Date to String

```{python}
date.dt.strftime("%Y-%m-%d")
```

### From String to Datetime

```{python}
sd = pl.Series(
    "date",
    [
        "2021-04-22",
        "2022-01-04 00:00:00",
        "01/31/22",
        "Sun Jul  8 00:34:60 2001",
    ],
)
```

Parse into `Date` type.

```{python}
sd.str.strptime(datatype= pl.Date, fmt="%F", strict=False)
```

```{python}
sd.str.strptime(pl.Date, "%D", strict=False)
```

Parse into `Datetime` type.

```{python}
sd.str.strptime(pl.Datetime, "%F %T",strict=False)
```

```{python}
sd.str.strptime(pl.Datetime, "%a %h %d %T %Y",strict=False)
```

Parse into `Time` type.

```{python}
sd.str.strptime(pl.Time, "%a %h %d %T %Y",strict=False)
```

## Comparing Series

```{python}
s.series_equal(pl.Series("a", [1, 2, 3]))
```

# DataFrames {#sec-dataframes}

General:

1.  There is no row index (like R's `data.frame`, `data.table`, and `tibble`; unlike Python's `pandas`).
2.  Will not accept duplicate column names (unlike pandas).

## DataFrame-Object Hosekeeping

A frame can be created as you would expect. From a dictionary of series, a numpy array, a pandas sdataframe, or a list of polars (or pandas) series, etc.

```{python}
#| label: make-dataframe


df = pl.DataFrame({
  "integer": [1, 2, 3], 
  "date": [
    (datetime(2022, 1, 1)), 
    (datetime(2022, 1, 2)), 
    (datetime(2022, 1, 3))], 
    "float":[4.0, 5.0, 6.0],
    "string": ["a", "b", "c"]})

df
```

```{python}
print(df)
```

```{python}
df.glimpse() #useful for wide frames
```

Things to note:

1.  The frame may be printed with Jupter's styling, or as ASCII with a `print()` statement.
2.  Shape, and dtypes, are part of the output.

```{python}
df.columns
```

```{python}
df.shape
```

```{python}
df.height # probably more useful than df.shape[0]
```

```{python}
df.width
```

```{python}
df.schema # similar to pandas info()
```

```{python}
df.with_row_count()
```

Add a single column

```{python}
df.with_columns(
    pl.Series("new", [1, 2, 3])
    ) # replaces the now-deprecated function `df.with_column()`
```

Add multiple columns

```{python}
df.with_columns(
  pl.Series("new1", [1, 2, 3]),
  pl.Series("new2", [4, 5, 6])
  )
```

```{python}
df.clone() # deep copy
```

The following commands make changes in place; I am thus creating a copy of `df`.

```{python}
df_copy = df.clone() # making a copy since 
df_copy.insert_at_idx(1, pl.Series("new", [1, 2, 3])) 
```

```{python}
df_copy.replace_at_idx(0, pl.Series("new2", [1, 2, 3]))
```

```{python}
df_copy.replace('float', pl.Series("new_float", [4.0, 5.0, 6.0])) 
```

```{python}
def foo(frame):
  return frame.with_columns(pl.Series("new", [1, 2, 3]))
df.pipe(foo)
```

```{python}
df.is_empty()
```

```{python}
df.clear() # make empty copy. replaced .cleared()
```

```{python}
df.clear().is_empty()
```

```{python}
df.rename({'integer': 'integer2'})
```

## Convert to Other Python Objects

### To Pandas

```{python}
df.to_pandas()
```

### To Numpy

```{python}
df.to_numpy()
```

### To List

```{python}
df.get_columns() # columns as list of polars series
```

```{python}
df.rows() # rows as list of tuples
```

### To Python Dict

```{python}
df.to_dict() # columns as dict of polars series
```

## Dataframe in Memory

```{python}
df.estimated_size(unit="gb")
```

```{python}
df.n_chunks() # number of ChunkedArrays in the dataframe
```

```{python}
df.rechunk() # ensure contiguous memory layout
```

```{python}
df.shrink_to_fit() # reduce memory allocation to actual size
```

## Statistical Aggregations

```{python}
df.describe()
```

Compare to pandas:

```{python}
df.to_pandas().describe()
```

Things to note:

-   Comparing to pandas:
    -   Polars will summarize all columns even if they are not numeric.
    -   The statistics returned are different.

Statistical aggregations operate column-wise (and in parallel).

```{python}
df.max()
```

```{python}
df.min()
```

```{python}
df.mean()
```

```{python}
df.median()
```

```{python}
df.sum()
```

```{python}
df.with_columns(pl.all()sum_horizontal())
```

```{python}
df.std()
```

```{python}
df.quantile(0.1)
```

## Extraction

1.  If you are used to pandas, recall there is no index. There is thus no need for `loc` vs. `iloc`, `reset_index()`, etc. See [here](https://pola-rs.github.io/polars-book/user-guide/howcani/selecting_data/selecting_data_indexing.html) for a comparison of extractors between polars and pandas.
2.  Filtering and selection is possible with the `[` operator, or the `filter()` and `select()` methods. The latter is recommended to facilitate query planning (discussed in @sec-query-planning).

Single cell extraction.

```{python}
df[0,0] # like pandas .iloc[]
```

Slicing along rows.

```{python}
df[0:1] 
```

Slicing along columns.

```{python}
df[:,0:1]
```

### Selecting Columns

Column selection by label

```{python}
df.select("integer")
# or df['integer']
# or df[:,'integer']
```

Select columns with list of labels

```{python}
df.select(["integer", "float"])
# or df[['integer', 'float']]
```

As of polars\>=15.0.0, you don't have to pass a list:

```{python}
df.select("integer", "float")
```

Column slicing by label

```{python}
df[:,"integer":"float"]
```

Note: `df.select()` does not support slicing ranges such as `df.select("integer":"float")`.

Get a column as a 1D polars frame.

```{python}
df.get_column('integer')
```

Get a column as a polars series.

```{python}
df.to_series(0)
```

```{python}
df.find_idx_by_name('float')
```

```{python}
df.drop("integer")
```

`df.drop()` not have an `inplace` argument. Use `df.drop_in_place()` instead.

### pl.col()

The `pl.col()` is **super important** for referencing columns. It will be used to select columns within a `df.select()` context, and to transform columns within a `df.with_columns()` context. It may extract a single column, a list, a particular (polars) dtype, a regex pattern, or simply all columns.

When exctracting along dtype, use polars' dtypes, not pandas' dtypes. For example, use `pl.Int64` instead of `np.int64`.

Select along dtype

```{python}
df.select(pl.col(pl.Int64))
```

```{python}
df.select(pl.col(pl.Float64))
```

```{python}
df.select(pl.col(pl.Utf8))
```

List of dtypes

```{python}
df.select(pl.col([pl.Int64, pl.Float64]))
```

Regular Expression

```{python}
df.select(pl.col("*")) # same as df.select(pl.all())
```

```{python}
df.select(pl.col("*").exclude("integer"))
```

```{python}
df.select(pl.col("*").exclude(pl.Float64))
```

::: {.callout-important}
`pl.col()` with regex is insanely powerful!
:::

```{python}
#| label: pl-col-with-regex


df.select(pl.col("^.*te.*$")) # regex matching anything with a "te"
```

Exciting! [New API](https://pola-rs.github.io/polars/py-polars/html/reference/selectors.html) for column selection. 

```{python}
#| label: pl-column-selector
import polars.selectors as cs

df.select(cs.starts_with('i'))
```

```{python}
#| label: pl-column-selector-set-opeartions

df.select(cs.starts_with('i') | cs.starts_with('d'))
```

```{python}
df.select(cs.starts_with('i') | cs.starts_with('d'))
```



### Filtering Rows

```{python}
df.head(2)
```

```{python}
df.limit(2) # same as pl.head()
```

```{python}
df.tail(1)
```

```{python}
df.take_every(2)
```

```{python}
df.slice(offset=1, length=1)
```

```{python}
df.sample(1)
```

```{python}
df.row(1) # get row as tuple
```

Row filtering by label

```{python}
df.filter(pl.col("integer") == 2)
```

Things to note:

-   The `[` operator does not support indexing with boolean such as `df[df["integer"] == 2]`.
-   The `filter()` method is recommended over `[` by the authors of polars, to facilitate lazy evaluation (discussed later).

### Selecting A Single Item

Exctracts the first element as a scalar. Useful when you output a single number as a frame object.

```{python}
pl.DataFrame([1]).item() # notice the output is not a frame, rather, a scalar.
```

## Uniques and Duplicates

```{python}
df.is_unique()
```

```{python}
df.is_duplicated()
```

```{python}
df.unique() # same as pd.drop_duplicates()
```

```{python}
df.n_unique()
```

## Missing

```{python}
df_with_nulls = df.with_columns(
    pl.Series("missing", [3, None, np.nan]),
)
```

```{python}
df_with_nulls.null_count() # same as pd.isnull().sum()
```

```{python}
df_with_nulls.drop_nulls() # same as pd.dropna()
```

**Note**: There is no `drop_nan()` method. See [here](https://stackoverflow.com/questions/75548444/polars-dataframe-drop-nans) for workarounds.

```{python}
df_with_nulls.fill_null(0) # same as pd.fillna(0)
```

But recall that `None` and `np.nan` are not the same thing.

```{python}
df_with_nulls.fill_nan(99)
```

```{python}
df_with_nulls.interpolate()
```

## Transformations

-   The general idea of colum trasformation is to wrap all transformations in a `with_columns()` method, and the select colums to operat on with `pl.col()`.
-   Previous versions of polars used `df.with_column()` and `df.with_columns()`. The `with_column()` method is now deprecated.
-   The output column will have the same name as the input, unless you use the `alias()` method to rename it.
-   The `with_columns()` is called a **polars context**.
-   The flavor of the `with_columns()` context is similar to pandas' `assign()`.
-   One can use `df.iter_rows()` to get an iterator over rows.

```{python}
df.with_columns(
    pl.col("integer") * 2,
    pl.col("integer").alias("integer2"),
    integer3 = pl.col("integer") * 3
)
```

Things to note:

-   The columns `integer` is multiplied by 2 in place, because no `alias` is used.
-   The column `integer` is copied, by renaming it to `integer2`.
-   As of polars version \>15.*.* (I think), you can use `=` to assign. That is how `integer3` is created.
-   You cannot use `[` to assign! This would not have worked `df['integer3'] = df['integer'] * 2`

If a selection returns multiple columns, all will be transformed:

```{python}
df.with_columns(
    pl.col([pl.Int64,pl.Float64])*2
)
```

```{python}
df.with_columns(
    pl.all().cast(pl.Utf8)
)
```

Apply your own lambda function.

```{python}
df.select([pl.col("integer"), pl.col("float")]).apply(lambda x: x[0] + x[1])
```

As usual, using your own functions may have a very serious toll on performance:

```{python}
df_big = pl.DataFrame(np.random.randn(1000000, 2), schema=["a", "b"]) # previous versions used columns= instead of schema=
```

```{python}
%timeit -n2 -r2 df_big.sum(axis=1)
```

```{python}
%timeit -n2 -r2 df_big.apply(lambda x: x[0] + x[1])
```

How would numpy and pandas deal with this row-wise summation?

```{python}
df.shift(1)
```

```{python}
df.shift_and_fill(1, 'WOW')
```


### Conditional Transformation (if-else)

```{python}
df.with_columns(
    pl.when(
      pl.col("integer") > 2
    )
    .then(1)
    .otherwise(pl.col("integer"))
    .alias("new_col")
)
```

Note what happens if `otherwise()` omitted:

```{python}
df.with_columns(
    pl.when(
      pl.col("integer") > 2
    )
    .then(1)
    # .otherwise(pl.col("integer"))
    .alias("new_col")
)
```



## Struct (multi-colum operations)

TODO

### Unset

https://pola-rs.github.io/polars/py-polars/html/reference/dataframe/api/polars.DataFrame.unnest.html#polars.DataFrame.unnest


## Sorting

```{python}
df.sort(by=["integer","float"])
```

```{python}
df.reverse()
```

## Joins {#sec-joins}

High level:

-   `df.hstack()` for horizontal concatenation; like pandas `pd.concat([],axis=1)` or R's `cbind`.
-   `df.vstack()` for vertical concatenation; like pandas `pd.concat([],axis=0)` or R's `rbind`.
-   `df.merge_sorted()` for vertical stacking, with sorting.
-   `pl.concat()`, which is similar to the previous two, but with memory re-chunking. `pl.concat()` also allows diagonal concatenation, if columns are not shared.
-   `df.extend()` for vertical concatenation, but with memory re-chunking. Similar to `df.vstack().rechunk()`.
-   `df.join()` for joins; like pandas `pd.merge()` or `df.join()`.

For more on the differences between these methods, see [here](https://www.rhosignal.com/posts/polars-extend-vstack/).

### hstack

```{python}
new_column = pl.Series("c", np.repeat(1, df.height))

df.hstack([new_column])
```

### vstack

```{python}
df2 = pl.DataFrame({
  "integer": [1, 2, 3], 
  "date": [
    (datetime(2022, 1, 4)), 
    (datetime(2022, 1, 5)), 
    (datetime(2022, 1, 6))], 
    "float":[7.0, 8.0, 9.0],
    "string": ["d", "d", "d"]})


df.vstack(df2)
```

### Concatenation

```{python}
pl.concat([df, df2]) 
# equivalent to:
# pl.concat([df, df2], how='vertical', rechunk=True, parallel=True) 
```

```{python}
pl.concat([df,new_column.to_frame()], how='horizontal')
```

### extend

```{python}
df.extend(df2) # like vstack, but with memory re-chunking. Similar to df.vstack().rechunk()
```

### merge_sorted

```{python}
df.merge_sorted(df2, key="integer") # vstacking with sorting.
```

**Caution**: Joining along rows is possible only if matched columns have the same dtype. Timestamps may be tricky because they may have different time units. Recall that timeunits may be cast before joining using `series.dt.cast_time_unit()`:

```{python}
#| eval: false
df.with_columns(
    pl.col(pl.Datetime("ns")).dt.cast_time_unit(tu="ms")
)            
```

If you cannot arrange schema before concatenating, use a diagonal concatenation:

```{python}
pl.concat(
    [df,new_column.to_frame()], 
    how='diagonal')
```

### join

```{python}
df.join(df2, on="integer", how="left")
```

Things to note:

-   Repeating column names have been suffixed with "\_right".
-   Unlike pandas, there are no indices. The `on`/`left_on`/`right_on` argument is always required.
-   `how=` may take the following values: 'inner', 'left', 'outer', 'semi', 'anti', 'cross'.
-   The join is super fast, as demonstrated in @sec-motivation above.

### join_asof

```{python}
df.join_asof(
    df2, 
    left_on="date", 
    right_on='date', 
    by="integer", 
    strategy="backward", 
    tolerance='1w')
```

Things to note:

-   Yes! `merge_asof()` is also available.
-   The `strategy=` argument may take the following values: 'backward', 'forward'.
-   The `tolerance=` argument may take the following values: '1w', '1d', '1h', '1m', '1s', '1ms', '1us', '1ns'.

## Reshaping

High level:

-   `df.transpose()` as the name suggests.
-   `df.melt()` for wide to long.
-   `df.pivot()` for long to wide.
-   `df.explode()` for breaking strings into rows.
-   `df.unstack()`

```{python}
df.transpose()
```

### Wide to Long

```{python}
# The following example is adapted from Pandas documentation: https://pandas.pydata.org/docs/reference/api/pandas.wide_to_long.html

np.random.seed(123)
wide = pl.DataFrame({
    'famid': ["11", "12", "13", "2", "2", "2", "3", "3", "3"],
    'birth': [1, 2, 3, 1, 2, 3, 1, 2, 3],
    'ht1': [2.8, 2.9, 2.2, 2, 1.8, 1.9, 2.2, 2.3, 2.1],
    'ht2': [3.4, 3.8, 2.9, 3.2, 2.8, 2.4, 3.3, 3.4, 2.9]})

wide.head(2)
```

```{python}
wide.melt(
  id_vars=['famid', 'birth'], 
  value_vars=['ht1', 'ht2'], 
  variable_name='treatment', 
  value_name='height').sample(5)
```

Break strings into rows.

```{python}
wide.explode(columns=['famid']).limit(5)
```

### Long to Wide

```{python}
# Example adapted from https://stackoverflow.com/questions/5890584/how-to-reshape-data-from-long-to-wide-format

long = pl.DataFrame({
    'id': [1, 1, 1, 2, 2, 2, 3, 3, 3],
    'treatment': ['A', 'A', 'B', 'A', 'A', 'B', 'A', 'A', 'B'],
    'height': [2.8, 2.9, 2.2, 2, 1.8, 1.9, 2.2, 2.3, 2.1]
    })
  
long.limit(5)
```

```{python}
long.pivot(
  index='id', # index in the wide format
  columns='treatment', # defines columns in the wide format
  values='height')
```

```{python}
long.unstack(step=2) # works like a transpose, and then wrap rows. Change the `step=` to get the feeling. 
```

## Groupby

Grouping over categories:

-   `df.partion_by()` will return a list of frames.
-   `df.groupby()` for grouping. Just like pandas, only parallelized, etc. The output will have the length of the number of groups.
-   `over()` will assign each row the aggregate in the group. Like pandas `groupby.transform`. The output will have the same length as the input.

Grouping over time:

-   `df.grouby_rolling()` for rolling window grouping, a.k.a. a sliding window. Each row will be assigned the aggregate in the window.
-   `df.groupby_dynamic()` for dynamic grouping. Each period will be assigned the agregate in the period. The output may have more rows than the input.

After grouping:

-   `df.groupby().agg()` for aggregating.
-   `df.groupby().apply()` for applying a function to each group.
-   `df.groupby().count()` for counting.
-   `df.groupby().first()` for getting the first row of each group.
-   ...

See the [API reference](https://pola-rs.github.io/polars/py-polars/html/reference/dataframe/groupby.html) for the various options. Also see the [user guide](https://pola-rs.github.io/polars-book/user-guide/howcani/timeseries/temporal_groupby.html) for more details.

```{python}
df2 = pl.DataFrame({
    "integer": [1, 1, 2, 2, 3, 3],
    "float": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0],
    "string": ["a", "b", "c", "d", "e", "f"],
    "datetime": [
        (datetime(2022, 1, 4)), 
        (datetime(2022, 1, 4)), 
        (datetime(2022, 1, 4)), 
        (datetime(2022, 1, 9)), 
        (datetime(2022, 1, 9)), 
        (datetime(2022, 1, 9))],
})
```

```{python}
df2.partition_by("integer")
```

```{python}
groupper = df2.groupby("integer")
groupper.count()
```

```{python}
groupper.sum()
```

Groupby a fixed time window with `df.groupby_dynamic()`:

```{python}
(
  df2
  .groupby_dynamic(index_column="datetime", every="1d")
  .agg(pl.col("float").sum())
)
```

If you do not want a single summary per period, rather, a window at each datapoint, use `df.groupby_rolling()`:

```{python}
(
  df2
  .groupby_rolling(index_column="datetime", period='1d')
  .agg(pl.col("float").sum())
)
```

### Over

You may be familar with pandas `groupby().transform()`, which will return a frame with the same row-count as its input. You may be familiar with Postgres SQL [window function](https://www.postgresql.org/docs/current/tutorial-window.html). You may not be familiar with either, and still want to aggregate within group, but propagate the result to all group members. Polars' `over()` is the answer.

```{python}
df.with_columns(
  pl.col("float").sum().over("string").alias("sum")
).limit(5)
```

**Careful**: `over()` should follow the aggregation. The following will not fail, but return the wrong result:

```{python}
df.with_columns(
  pl.col("float").over("string").sum().alias("sum")
).limit(5)
```

## Processing Multiple Frames Simultanously

Q: What if you want to access a column from frame `df`, when processing frame `df2`?\
A: Just join them.\
Q: What if they are not joinable?\
A: Use a diagonal join. Q: Can't I just add a search-space into the lazy query? A: Ahhh! Use `df.with_context()`.

```{python}
df3 = pl.Series("blah", [100,2,3]).to_frame()

q = (
    df.lazy()
    .with_context( # add colums of df2 to the search space
        df3.lazy()
        )
    .with_columns(
        pl.col('float').map_dict(remapping={4.0:None}, default=100).fill_null(pl.col('blah').mean()).alias('float2'),
        )
    )

q.collect()
```

Things to note:

-   `with_context()` is a lazy operation. This is great news, since it means both frames will benefit from query planning, etc.
-   `with_context()` will not copy the data, but rather, add a reference to the data.
-   Why not use `pl.col('blah').mean()` within the `map_dict()`? That is indeed more reasonable. It simply did not work.
-   Try it yourself: Can you use multiple `with_context()`?


### Imputation Example

```{python}
train_lf = pl.LazyFrame(
    {"feature_0": [-1.0, 0, 1], "feature_1": [-1.0, 0, 1]}
)
test_lf = pl.LazyFrame(
    {"feature_0": [-1.0, None, 1], "feature_1": [-1.0, 0, 1]}
)

(
  test_lf
  .with_context(
    train_lf
    .select(pl.all().suffix("_train")))
    .select(
      pl.col("feature_0")
      .fill_null(
        pl.col("feature_0_train").median()
      )
  )
).collect()

```


# Query Planning and Optimization {#sec-query-planning}

The take-home of this section, is that polar can take advantage of half-a-century's worth of research in query planning and optimization. You will not have to think about the right order of operations, or the right data structures to use. Rather, replace the polars dataframe with a polars lazy-dataframe, state all the operations you want, and just finish with a `collect()`. Polars will take care of the rest, and provide you with the tools to understand its plan.

We will not go into the details of the difference between a lazy and a non-lazy dataframe. Just assume a lazy frame allows everything a non-lazy frame can do, but it does not execute the operations until you call `collect()`. This is not entirely true, but you will get an informative error if you try to do something that is not supported.

Get your lazy dataframe:

```{python}
df_lazy = df.lazy()
```

State all your operations:

```{python}
q = (
  df_lazy
  .filter(pl.col("float") > 2.0)
  .filter(pl.col("float") > 3.0)
  .filter(pl.col("float") > 7.0)
  .select(["integer"])
  .sort("integer")
)
```

And now visualize the query.

```{python}
q # same as q.show_graph(optimized=False)
```

```{python}
q.show_graph(optimized=True)
```

Things to note:

-   You will need Graphviz installed to visualize the query plan.
-   To understand the plan, you need some terminology from [relational databases](https://www.ibm.com/docs/en/informix-servers/14.10?topic=concepts-selection-projection). Namely:
    -   A *selection* is a polars' filter, i.e. subset of rows, marked in the graph with a $\sigma$.
    -   A *projection* is polars seletion, i.e. a subset of columns, marked in the graph with a $\pi$.
-   The optimized plan removes redudancies, and orders the operations in the most efficient way.

You can now execute the plan with a `collect()`:

```{python}
q.collect()
```

```{python}
q.describe_plan()
```




## Inspecting, Profiling, and Debugging a Query

For early stopping (debugging?) you can replace `collect()` with `fetch()`:

```{python}
q.fetch(2)
```

You can inspect the data at any point in the query. `df.inspect()` will print the state of a single node in the query graph: \[TODO: replace with example with multiple nodes\]

```{python}
q = (
    pl.scan_parquet(f'{path}/*.parquet')
    .filter(
        (pl.col('passenger_count') > 0) &
        (pl.col('passenger_count') < 5) &
        (pl.col('trip_distance') > 0) &
        (pl.col('trip_distance') < 10) &
        (pl.col('fare_amount') > 0) &
        (pl.col('fare_amount') < 100) &
        (pl.col('tip_amount') > 0) &
        (pl.col('tip_amount') < 20) &
        (pl.col('total_amount') > 0) &
        (pl.col('total_amount') < 100)
    )
    .inspect() # here is the inspect
    .groupby('passenger_count')
    .agg([pl.mean('tip_amount')])
    )
q.collect()
```

You can profile the execution of a query with `df.profile()`:

```{python}
q.profile(show_plot=True)
```

## Exporting a Query

You can export your query, as a JSON file.

```{python}
q.write_json("query.json") # export
```

This is how the query will look on disk:

```{python}
import json
json.loads(open("query.json").read())# inspect
```

You can now load it and run it.

```{python}
pl.LazyFrame.read_json("query.json").collect() 
```

## SQL Flavor

If you are a hardcore SQL user, you may want to use the SQL flavor of polars. The following syntax is experimental, and may change.

```{python}
sql = pl.SQLContext() 
sql.register("lazy_frame", lazy_frame) # register the lazy frame as a table

sql.query("""
    SELECT passenger_count, AVG(tip_amount) FROM lazy_frame 
    WHERE passenger_count < 3
    GROUP BY passenger_count
    """) # query the table
```

# I/O

You will find that polars is blazing fast at reading and writing data. This is due to:

1.  Very good heuristics/rules implemented in the `read_csv` function.
2.  The use of [Apache Arrow](https://arrow.apache.org/) as an internal data structure, which maps seamlesly to the parquet file format.
3.  Parallelism, whenever possible.
4.  Lazy scans/imports, which allows the materialization only of required data; i.e., filters and projections are executed at scan time.

## Import

High level:

-   `pl.read_X()` will read a file into a non-lazy frame.
-   `pl.scan_X()` will read a file into a lazy frame.
-   You can use globs to import multiple files but:
    -   You may need to teak schema manually.
    -   Filesystme operations are handeled by [fsspec](https://filesystem-spec.readthedocs.io/en/latest/), which may open only the first file when using globs in remote filesystems (e.g. S3). This is discussed in @sec-multiple-files.

### From a Single File

Let's firs make a csv to import:

```{python}
df.write_csv("df.csv")
```

Import the csv into a non-lazy frame:

```{python}
pl.read_csv("df.csv")
```

Importing as a lazy frame:

```{python}
df_lazy = pl.scan_csv("df.csv")
```

Things become interesting when you manipulate the lazy frame before materializing it:

```{python}
q = (
  df_lazy
  .filter(pl.col("float") > 2.0)
  .filter(pl.col("float") > 3.0)
  .filter(pl.col("float") > 7.0)
  .select(["integer"])
  .sort("integer")
)

q.show_graph(optimized=True)
```

```{python}
q.collect()
```

Things to note:

-   From the graph we see that the filtering ($\sigma$) is done at scan time, and not after the materialization of the data. This is crucial for processing datasets that are larger then memory.
-   To get the actual data, we naturally need to `collect()`.

Cleary, .csv is not the only format that can be read. It is possibly the least recommended. Other file types can be found [here](https://pola-rs.github.io/polars/py-polars/html/reference/io.html) and include:

-   Excel.
-   Arrow IPC: A binary format for storing columnar data.
-   Feather (V2): Multiple IPC files with a shared schema.
-   Parquet (non-partitioned): A tabular file format (not columnar) that is optimized for long-term storage, more compressed than Feather.
-   JSON: Short for JavaScript Object Notation, a textual data-interchange format (like XML).
-   Avro: A binary row-based format. Good for streaming.

Each of the above formats has a non-lazy reader using `pl.read_*` and a lazy reader using `pl.scan_*`.

Currently unsuported formats:

-   Feather (V1).
-   [HDF5](https://github.com/pola-rs/polars/issues/3520).

### From Multiple Files in Your Filesystem

Most of today's datasets will span more than a single file on disk. Polar supports reading from multiple files in your file system (as opposed to a remote datalake such as S3), and will automatically merge them into a single dataframe. There are, however, many file formats, and each has its own way of partitioning the data. Multi-file storage supported by polars (at the time of writing):

1.  Parquet (partitioned): A collection of files with a common schema, partitioned as folders on disk.
2.  [Delta-Lake](https://www.databricks.com/wp-content/uploads/2020/08/p975-armbrust.pdf): If your data is saves as many parquet files on S3, a failed copy operation may "break" the data. Systems that protect data from such failures (failed copy is only an example) are called "transactional systems", and the garantees they provide are called ["ACID"](https://www.databricks.com/glossary/acid-transactions). A Delta-Lake, is a piece of open source software, that manages your queries to give your data-lake the ACID properties.
3.  [Arrow Dataset](https://arrow.apache.org/docs/python/dataset.html): A collection of files (csv, parquet, feather, etc) with a common schema.

TODO: https://pola-rs.github.io/polars-book/user-guide/multiple_files/intro.html



#### Arbitrary Collection of Files {#sec-multiple-files}

You can always scan from some arbitrary collection of files and concatenate the result.

```{python}
path = 'data/NYC' # Data from https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page
file_names = os.listdir(path)
file_names
```

```{python}
df_lazy_list = []
for file in file_names:
    df_lazy_list.append(
        pl.scan_parquet(f'{path}/{file}')
        )
```

With a list of lazy frames you can proceed by concatenating into a single lazy frame using `pl.concat()`, or collecting them into a list of eager frames using `pl.collect_all()`. The best option depends on your use case.

Things to note:

-   The arrow data format uses caching for string and categorical data (i.e. pl.Series). If importing multiple files, such as multiple parquet/feather files, or an arrow dataset, different files may be cached differnetly. This will cause an error when trying to concatenate the dataframes. To avoid this, you can disable string caching, or enforce [joint caching](https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.StringCache.html) of all files.
-   Dataframes may have incompatible schema, as discussed in @sec-joins above. You may need to manually adjust the schema before concatenating.

Here is an example that deals with both issues:

```{python}

with pl.StringCache(): # Enforce joint caching of all files
  df_lazy_list = []
  for file in file_names:
    lazy_frame = (
        pl.scan_parquet(f'{path}/{file}') # read a lazy frame
        .with_columns(
            pl.col(pl.Datetime('ns')).dt.cast_time_unit('ms')
            ) # ensure joinable time units
    )
    df_lazy_list.append(lazy_frame)

q= (
    pl.concat(df_lazy_list) # concat into into a single lazy frame
    .filter(pl.col('passenger_count') < 3)
    .groupby('passenger_count')
    .agg([pl.mean('tip_amount')])
)
q.collect() # execute query
```



#### Partitioned Parquet

The code snipped above (@sec- multiple_files) is fully generalizable wrt the files you import and what you do to them. Most often, you don't need such generality. For instance, when importing multiple parquet files form the local file system, the `pl.read_parquet()` function will allow you to use globs. The above may thus read:

```{python}
with pl.StringCache(): # Enforce joint caching of all files
  lazy_frame = pl.scan_parquet(f'{path}/*.parquet')
  
q= (
    lazy_frame # concat into into a single lazy frame
    .filter(pl.col('passenger_count') < 3)
    .groupby('passenger_count')
    .agg([pl.mean('tip_amount')])
)
q.collect() # execute query
```




#### Apache Arrow Dataset

TODO: The exampe below deals with partitioned parquet, and not arrow datasets. Fix.

An Apache Arrow dataset is a collection of parquet files, with an index file. It is a very efficient way to store data on disk, and to read it in parallel.

Writing an Arrow dataset:

```{python}
#| echo: false

# clean "df" if exists
import shutil
if os.path.exists("df"):
    shutil.rmtree("df")
```

```{python}
# Write df as an arrow dataset:
df.to_pandas().to_parquet(
    "df", 
    engine="pyarrow", 
    partition_cols=["integer"])

os.listdir("df") # inspect folder on disk
```

```{python}
# inspect partitions
[os.listdir(f"df/{x}/") for x in os.listdir("df")]
```

```{python}
import pyarrow.dataset as ds
dset = ds.dataset("data/df", format="parquet")  # define folder as dataset
pl.scan_pyarrow_dataset(dset).collect() # import
```

Things to note:

-   We used pandas to write the arrow dataset. It seemed easier than the [pyarrow syntax](https://arrow.apache.org/docs/python/dataset.html#dataset).
-   The `partition_cols` argument is used to partition the dataset on disk. Each partition is a parquet file (or another partition).
-   Reading from the web (not from the local filesystem) is slightly different. TODO: add reference.

#### Multiple CSVs

TODO: `pl.read_csv_batched()`

### From Multiple Files on a Remote Datalake

If you are coming from Pandas, reading from a remote datalake (say S3), and a local filesystem may feel the same. This is because the authors of pandas went to great lengths to make the API feel the same. At the time of writing, if you give polars a remote glob, it will only read the first file ([ref](https://github.com/pola-rs/polars/issues/5863)).

Your current options for reading multiple files stored remotely are:

1.  Read one file at a time, and concatenate the results, or use the `pl.scan_parquet()` as in @sec- multiple_files.
2.  Use third party functionality that can link to multiple remote files. Luckily, the pyarrow library gives you this functionality. See [here](https://pola-rs.github.io/polars-book/user-guide/howcani/io/aws.html) for an example.

### Reading from a Database

See [here](https://pola-rs.github.io/polars-book/user-guide/howcani/io/read_db.html).

### Serverless

See [here](https://www.rhosignal.com/posts/polars-aws-lambda/) for working in serverless environments such as AWS Lambda.

## Export to Disk {#sec-disk-export}

Well, there is not much to say here; just look for `pl.write_*` functions. Alternatively, export to pandas, arrow, numpy, and use their exporters.

# Plotting

To get an intuition of what you may expect in this chapter you should know the following. There are various approaches to plotting in python:

1.  The object oriented, where a dataframe has a plotting method. E.g. `df.plot()`. The method may use a single, or even multiple backends. Such is the pandas dataframe, which may use a matplotlib, plotly, or bokeh backend.
2.  The functional method, where a plotting function takes a dataframe as an argument. E.g. `plot(df)`. Such are the matplotlib, seaborn, and plotly functions, which may take pandas dataframes as inputs.

Plotting support in polars thus boils down to the folowing questions: (1) Do polars dataframes have a plotting method? With which backend? (2) Can plotting functions take polars dataframes as inputs?

The answer to the first is negative. Polars dataframes do not have a plotting method, and it seems they are not planned to have one (TODO: add reference). The answer to the second is "almost yes". Any plotting function that can take an iterable such as a list, or numpy 1D arrays, will work. Either becaus polars series are iterable, or because one can convert them (to arrow or numpy being the fastest).

Passing polars frames may cause trouble. You may expect to use a `plot(df, x='col1', y='col2')` syntax; it may work if `df` is a pandas dataframe, but not with polars. Support of this syntax does not depend on polars developers, rather, on the plotting function developpers. I suspect that the plotly and bokeh teams will eventually supprts polars. I do not know about the seaborm, or matplotlib teams.

The current state of affairs:

-   Plotly, matplotlib, and seaborn support polars series as input.
-   Matplotlib and seaborn support polars frames as input. Plotly (5.12.0) does not.

## Plotly Functions

The `iris` dataset is provided by plotly as a pandas frame. We convert it to a polars frame.

```{python}
iris = pl.DataFrame(px.data.iris())
iris.head()
```

```{python}
fig = px.scatter(
    x=iris["sepal_width"].to_list(), 
    y=iris["sepal_length"].to_list())
fig.show()
```

But wait! Maybe a polars series is "array-like" and can be used as input? Yes it can!

```{python}
fig = px.scatter(
    x=iris["sepal_width"], 
    y=iris["sepal_length"])
fig.show()
```

Can a polars frame be used as input? No it can not. The following will currently not work:

```{python}
#| eval: false
#| 
fig = px.scatter(
    data_frame=iris,
    x="sepal_width",
    y="sepal_length")
fig.show()
```

## Matplotlib Functions

The above discussion applies to matplotlib functions as well; with the exception that matplotlib functions already support polars frames as input.

Inputing polars series:

```{python}
fig, ax = plt.subplots()
ax.scatter(
    x=iris["sepal_width"], 
    y=iris["sepal_length"])
```

Inputing polars frames:

```{python}
fig, ax = plt.subplots()
ax.scatter(
    data=iris,
    x="sepal_width",
    y="sepal_length")
```

## Seborn Functions

Because Seaborn uses a matplotlib backend, the above discussion applies to seaborn functions as well.

```{python}
import seaborn as sns
sns.scatterplot(
    data=iris,
    x="sepal_width",
    y="sepal_length")

```

# Polars and ML

"How do to machine learning with polars?" is not a well defined question. ML can be done with many libraries, and the answer depends on the library you are using. One possibility is converting polars dataframes to a numpy arrays. This is very easy when dealing with numerical data. Converting `pl.Utf8` and `pl.Categorical` dtypes is a bit more involved, but still possible. For instance, by using `polars.DataFrame.to_dummies()`, `polars.get_dummies()`, or `polars.Series.to_dummies()`.

But wait! Isn't the conversion to numpy an expensive operation? Not terribly, but there is a better way. At the time of writing, ML libraries such as scikit-learn and xgboost, do not support polars dataframes as inputs. XGboost, however, does support arrow dataframes. This is great news since converting polars to arrow is just passing a pointer. See an example [here](https://www.rhosignal.com/posts/polars-arrow-xgboost/).

## Polars and Patsy

Patsy is a python library for describing statistical models (especially linear models and generalized linear models) and building design matrices.

```{python}
import patsy as pt
#make a dataframe
data_pandas = pd.DataFrame(
    np.random.randn(100, 3), 
    columns=["y", "x1", "x2"])
```

Use patsy to make a design matrix $X$, and a target vector $y$ from a pandas dataframe.

```{python}
formula = 'y ~ x1 + x2'
y, X = pt.dmatrices(formula, data_pandas)

X[:3]
```

```{python}
y[:3]
```

Does the same work with polars? Yes!

```{python}
data_polars= pl.DataFrame(data_pandas)
X, y = pt.dmatrices(formula, data_polars)
X[:3]
```

## Effect Coding and Contrasts

There are [many ways](https://stats.oarc.ucla.edu/spss/faq/coding-systems-for-categorical-variables-in-regression-analysis/) to encode categorical variables. For predictions, dummy coding is enough. If you want to discuss and infer on effect sizes, you may want to use other coding schemes.

One way to go about is to use the [category_encoders](http://contrib.scikit-learn.org/category_encoders/) library.

We start by making some categorical data.

```{python}
import string
import random
cat = pl.Series(
    name="cat",
    values=random.choices(
        population=string.ascii_letters[:5], 
        k=data_polars.height)
    ).to_frame()
data_polars = data_polars.hstack(cat)
data_polars.head()
```

The category encoders currently expects pandas dataframes as input, and does not support polars dataframes.

```{python}
import category_encoders as ce
encoder = ce.HelmertEncoder()
encoder.fit(data_polars.to_pandas())
```