19. Polars#
In addition to what’s in Anaconda, this lecture will need the following libraries:
!pip install --upgrade polars yfinance
19.1. Overview#
Polars is a fast data manipulation library for Python written in Rust.
It has gained significant popularity as a modern alternative to pandas due to its performance advantages.
Polars is designed with performance and memory efficiency in mind, leveraging:
Apache Arrow columnar format for fast data access
Lazy evaluation to optimize query execution
Parallel processing to utilize all available CPU cores
An expressive API built around column expressions
Tip
Why consider Polars over pandas?
Memory: pandas typically needs 5–10x your dataset size in RAM; Polars needs only 2–4x
Speed: Polars is 10–100x faster for many common operations
See: Polars TPC-H benchmarks for up-to-date performance comparisons
Throughout the lecture, we will assume that the following imports have taken place
import polars as pl
import numpy as np
import matplotlib.pyplot as plt
Like Pandas, Polars defines two important data types: Series and DataFrame.
You can think of a Series as a column of data, such as a collection of observations on a single variable.
A DataFrame is a two-dimensional object for storing related columns of data.
19.2. Series#
Let’s start with Series.
We begin by creating a series of four random observations
s = pl.Series(name='daily returns', values=np.random.randn(4))
s
| daily returns |
|---|
| f64 |
| -1.0887 |
| 0.051126 |
| -0.470457 |
| -1.014881 |
Note
Unlike pandas Series, Polars Series have no row index. Polars is column-centric — data access is managed through column expressions and boolean masks rather than row labels. See the Polars migration guide for pandas users for more detail.
Polars Series are built on top of Apache Arrow arrays and support many familiar operations
s * 100
| daily returns |
|---|
| f64 |
| -108.870007 |
| 5.112558 |
| -47.045728 |
| -101.488137 |
Absolute values are available as a method
s.abs()
| daily returns |
|---|
| f64 |
| 1.0887 |
| 0.051126 |
| 0.470457 |
| 1.014881 |
We can also get quick summary statistics
s.describe()
| statistic | value |
|---|---|
| str | f64 |
| "count" | 4.0 |
| "null_count" | 0.0 |
| "mean" | -0.630728 |
| "std" | 0.53164 |
| "min" | -1.0887 |
| "25%" | -1.014881 |
| "50%" | -0.470457 |
| "75%" | -0.470457 |
| "max" | 0.051126 |
Since Polars has no row index, labelled data requires a DataFrame.
For example, to associate ticker symbols with returns:
df = pl.DataFrame({
'company': ['AMZN', 'AAPL', 'MSFT', 'GOOG'],
'daily returns': np.random.randn(4)
})
df
| company | daily returns |
|---|---|
| str | f64 |
| "AMZN" | -1.852153 |
| "AAPL" | 0.008383 |
| "MSFT" | -0.482803 |
| "GOOG" | -2.283774 |
We access a value by filtering on a column expression
df.filter(
pl.col('company') == 'AMZN'
).select('daily returns').item()
-1.8521529661706666
Updates also use expressions rather than index assignment
df = df.with_columns(
pl.when(pl.col('company') == 'AMZN')
.then(0)
.otherwise(pl.col('daily returns'))
.alias('daily returns')
)
df
| company | daily returns |
|---|---|
| str | f64 |
| "AMZN" | 0.0 |
| "AAPL" | 0.008383 |
| "MSFT" | -0.482803 |
| "GOOG" | -2.283774 |
We can also check membership
'AAPL' in df['company']
True
19.3. DataFrames#
While a Series is a single column of data, a DataFrame is several columns, one for each variable.
As in Pandas, let’s work with data from the Penn World Tables.
We read this in using pl.read_csv
url = ('https://raw.githubusercontent.com/QuantEcon/'
'lecture-python-programming/main/lectures/_static/'
'lecture_specific/pandas/data/test_pwt.csv')
df = pl.read_csv(url)
df
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "Argentina" | "ARG" | 2000 | 37335.653 | 0.9995 | 295072.21869 | 75.716805 | 5.578804 |
| "Australia" | "AUS" | 2000 | 19053.186 | 1.72483 | 541804.6521 | 67.759026 | 6.720098 |
| "India" | "IND" | 2000 | 1.0063e6 | 44.9416 | 1.7281e6 | 64.575551 | 14.072206 |
| "Israel" | "ISR" | 2000 | 6114.57 | 4.07733 | 129253.89423 | 64.436451 | 10.266688 |
| "Malawi" | "MWI" | 2000 | 11801.505 | 59.543808 | 5026.221784 | 74.707624 | 11.658954 |
| "South Africa" | "ZAF" | 2000 | 45064.098 | 6.93983 | 227242.36949 | 72.71871 | 5.726546 |
| "United States" | "USA" | 2000 | 282171.957 | 1.0 | 9.8987e6 | 72.347054 | 6.032454 |
| "Uruguay" | "URY" | 2000 | 3219.793 | 12.099592 | 25255.961693 | 78.97874 | 5.108068 |
19.3.1. Selecting data#
We can select rows by slicing and columns by name
df[2:5]
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "India" | "IND" | 2000 | 1.0063e6 | 44.9416 | 1.7281e6 | 64.575551 | 14.072206 |
| "Israel" | "ISR" | 2000 | 6114.57 | 4.07733 | 129253.89423 | 64.436451 | 10.266688 |
| "Malawi" | "MWI" | 2000 | 11801.505 | 59.543808 | 5026.221784 | 74.707624 | 11.658954 |
To select specific columns, pass a list of names to select
df.select(['country', 'tcgdp'])
| country | tcgdp |
|---|---|
| str | f64 |
| "Argentina" | 295072.21869 |
| "Australia" | 541804.6521 |
| "India" | 1.7281e6 |
| "Israel" | 129253.89423 |
| "Malawi" | 5026.221784 |
| "South Africa" | 227242.36949 |
| "United States" | 9.8987e6 |
| "Uruguay" | 25255.961693 |
These can be combined
df[2:5].select(['country', 'tcgdp'])
| country | tcgdp |
|---|---|
| str | f64 |
| "India" | 1.7281e6 |
| "Israel" | 129253.89423 |
| "Malawi" | 5026.221784 |
19.3.2. Filtering by conditions#
The filter method accepts boolean expressions built from pl.col
df.filter(pl.col('POP') >= 20000)
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "Argentina" | "ARG" | 2000 | 37335.653 | 0.9995 | 295072.21869 | 75.716805 | 5.578804 |
| "India" | "IND" | 2000 | 1.0063e6 | 44.9416 | 1.7281e6 | 64.575551 | 14.072206 |
| "South Africa" | "ZAF" | 2000 | 45064.098 | 6.93983 | 227242.36949 | 72.71871 | 5.726546 |
| "United States" | "USA" | 2000 | 282171.957 | 1.0 | 9.8987e6 | 72.347054 | 6.032454 |
Multiple conditions can be combined with & (and) and | (or)
df.filter(
(pl.col('country').is_in(['Argentina', 'India', 'South Africa'])) &
(pl.col('POP') > 40000)
)
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "India" | "IND" | 2000 | 1.0063e6 | 44.9416 | 1.7281e6 | 64.575551 | 14.072206 |
| "South Africa" | "ZAF" | 2000 | 45064.098 | 6.93983 | 227242.36949 | 72.71871 | 5.726546 |
Expressions can involve arithmetic across columns
df.filter(
(pl.col('cc') + pl.col('cg') >= 80) & (pl.col('POP') <= 20000)
)
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "Malawi" | "MWI" | 2000 | 11801.505 | 59.543808 | 5026.221784 | 74.707624 | 11.658954 |
| "Uruguay" | "URY" | 2000 | 3219.793 | 12.099592 | 25255.961693 | 78.97874 | 5.108068 |
Select the country with the largest household consumption share
df.filter(pl.col('cc') == pl.col('cc').max())
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "Uruguay" | "URY" | 2000 | 3219.793 | 12.099592 | 25255.961693 | 78.97874 | 5.108068 |
19.3.3. Column expressions#
A key difference from pandas is that Polars uses column expressions for transformations rather than element-wise apply calls.
Here is an example computing the max of each numeric column
df.select(
pl.col(['year', 'POP', 'XRAT', 'tcgdp', 'cc', 'cg'])
.max()
.name.suffix('_max')
)
| year_max | POP_max | XRAT_max | tcgdp_max | cc_max | cg_max |
|---|---|---|---|---|---|
| i64 | f64 | f64 | f64 | f64 | f64 |
| 2000 | 1.0063e6 | 59.543808 | 9.8987e6 | 78.97874 | 14.072206 |
Expressions can be used inside with_columns to add or modify columns
df.with_columns(
(pl.col('XRAT') / 10).alias('XRAT_scaled'),
pl.col(pl.Float64).round(2)
)
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg | XRAT_scaled |
|---|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 | f64 |
| "Argentina" | "ARG" | 2000 | 37335.65 | 1.0 | 295072.22 | 75.72 | 5.58 | 0.09995 |
| "Australia" | "AUS" | 2000 | 19053.19 | 1.72 | 541804.65 | 67.76 | 6.72 | 0.172483 |
| "India" | "IND" | 2000 | 1006300.3 | 44.94 | 1.7281e6 | 64.58 | 14.07 | 4.49416 |
| "Israel" | "ISR" | 2000 | 6114.57 | 4.08 | 129253.89 | 64.44 | 10.27 | 0.407733 |
| "Malawi" | "MWI" | 2000 | 11801.5 | 59.54 | 5026.22 | 74.71 | 11.66 | 5.954381 |
| "South Africa" | "ZAF" | 2000 | 45064.1 | 6.94 | 227242.37 | 72.72 | 5.73 | 0.693983 |
| "United States" | "USA" | 2000 | 282171.96 | 1.0 | 9.8987e6 | 72.35 | 6.03 | 0.1 |
| "Uruguay" | "URY" | 2000 | 3219.79 | 12.1 | 25255.96 | 78.98 | 5.11 | 1.209959 |
Conditional logic uses pl.when(...).then(...).otherwise(...)
df.with_columns(
pl.when(pl.col('POP') >= 20000)
.then(pl.col('POP'))
.otherwise(None)
.alias('POP_filtered')
).select(['country', 'POP', 'POP_filtered'])
| country | POP | POP_filtered |
|---|---|---|
| str | f64 | f64 |
| "Argentina" | 37335.653 | 37335.653 |
| "Australia" | 19053.186 | null |
| "India" | 1.0063e6 | 1.0063e6 |
| "Israel" | 6114.57 | null |
| "Malawi" | 11801.505 | null |
| "South Africa" | 45064.098 | 45064.098 |
| "United States" | 282171.957 | 282171.957 |
| "Uruguay" | 3219.793 | null |
Note
Polars provides map_elements as an escape hatch for applying arbitrary
Python functions row-by-row, but it bypasses the optimized expression
engine and should be avoided when a native expression exists.
19.3.4. Missing values#
Let’s insert some null values to demonstrate imputation techniques
df_nulls = df.with_row_index().with_columns(
pl.when(pl.col('index') == 0)
.then(None).otherwise(pl.col('XRAT')).alias('XRAT'),
pl.when(pl.col('index') == 3)
.then(None).otherwise(pl.col('cc')).alias('cc'),
pl.when(pl.col('index') == 5)
.then(None).otherwise(pl.col('tcgdp')).alias('tcgdp'),
pl.when(pl.col('index') == 6)
.then(None).otherwise(pl.col('POP')).alias('POP'),
).drop('index')
df_nulls
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "Argentina" | "ARG" | 2000 | 37335.653 | null | 295072.21869 | 75.716805 | 5.578804 |
| "Australia" | "AUS" | 2000 | 19053.186 | 1.72483 | 541804.6521 | 67.759026 | 6.720098 |
| "India" | "IND" | 2000 | 1.0063e6 | 44.9416 | 1.7281e6 | 64.575551 | 14.072206 |
| "Israel" | "ISR" | 2000 | 6114.57 | 4.07733 | 129253.89423 | null | 10.266688 |
| "Malawi" | "MWI" | 2000 | 11801.505 | 59.543808 | 5026.221784 | 74.707624 | 11.658954 |
| "South Africa" | "ZAF" | 2000 | 45064.098 | 6.93983 | null | 72.71871 | 5.726546 |
| "United States" | "USA" | 2000 | null | 1.0 | 9.8987e6 | 72.347054 | 6.032454 |
| "Uruguay" | "URY" | 2000 | 3219.793 | 12.099592 | 25255.961693 | 78.97874 | 5.108068 |
Fill all nulls with zero
df_nulls.fill_null(0)
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "Argentina" | "ARG" | 2000 | 37335.653 | 0.0 | 295072.21869 | 75.716805 | 5.578804 |
| "Australia" | "AUS" | 2000 | 19053.186 | 1.72483 | 541804.6521 | 67.759026 | 6.720098 |
| "India" | "IND" | 2000 | 1.0063e6 | 44.9416 | 1.7281e6 | 64.575551 | 14.072206 |
| "Israel" | "ISR" | 2000 | 6114.57 | 4.07733 | 129253.89423 | 0.0 | 10.266688 |
| "Malawi" | "MWI" | 2000 | 11801.505 | 59.543808 | 5026.221784 | 74.707624 | 11.658954 |
| "South Africa" | "ZAF" | 2000 | 45064.098 | 6.93983 | 0.0 | 72.71871 | 5.726546 |
| "United States" | "USA" | 2000 | 0.0 | 1.0 | 9.8987e6 | 72.347054 | 6.032454 |
| "Uruguay" | "URY" | 2000 | 3219.793 | 12.099592 | 25255.961693 | 78.97874 | 5.108068 |
Or fill with column means
cols = ['cc', 'tcgdp', 'POP', 'XRAT']
df_nulls.with_columns(
pl.col(cols).fill_null(pl.col(cols).mean())
)
| country | country isocode | year | POP | XRAT | tcgdp | cc | cg |
|---|---|---|---|---|---|---|---|
| str | str | i64 | f64 | f64 | f64 | f64 | f64 |
| "Argentina" | "ARG" | 2000 | 37335.653 | 18.618141 | 295072.21869 | 75.716805 | 5.578804 |
| "Australia" | "AUS" | 2000 | 19053.186 | 1.72483 | 541804.6521 | 67.759026 | 6.720098 |
| "India" | "IND" | 2000 | 1.0063e6 | 44.9416 | 1.7281e6 | 64.575551 | 14.072206 |
| "Israel" | "ISR" | 2000 | 6114.57 | 4.07733 | 129253.89423 | 72.400502 | 10.266688 |
| "Malawi" | "MWI" | 2000 | 11801.505 | 59.543808 | 5026.221784 | 74.707624 | 11.658954 |
| "South Africa" | "ZAF" | 2000 | 45064.098 | 6.93983 | 1.8033e6 | 72.71871 | 5.726546 |
| "United States" | "USA" | 2000 | 161269.871714 | 1.0 | 9.8987e6 | 72.347054 | 6.032454 |
| "Uruguay" | "URY" | 2000 | 3219.793 | 12.099592 | 25255.961693 | 78.97874 | 5.108068 |
Polars also supports forward fill (fill_null(strategy='forward')) and interpolation.
There are more advanced imputation tools available in scikit-learn.
19.3.5. Visualization#
Let’s build a GDP per capita column and plot it
df = (df
.select(['country', 'POP', 'tcgdp'])
.rename({'POP': 'population', 'tcgdp': 'total GDP'})
.with_columns(
(pl.col('population') * 1e3).alias('population')
)
.with_columns(
(pl.col('total GDP') * 1e6 / pl.col('population'))
.alias('GDP percap')
)
.sort('GDP percap', descending=True)
)
df
| country | population | total GDP | GDP percap |
|---|---|---|---|
| str | f64 | f64 | f64 |
| "United States" | 2.82171957e8 | 9.8987e6 | 35080.381854 |
| "Australia" | 1.9053186e7 | 541804.6521 | 28436.433261 |
| "Israel" | 6.11457e6 | 129253.89423 | 21138.672749 |
| "Argentina" | 3.7335653e7 | 295072.21869 | 7903.229085 |
| "Uruguay" | 3.219793e6 | 25255.961693 | 7843.97062 |
| "South Africa" | 4.5064098e7 | 227242.36949 | 5042.647686 |
| "India" | 1.0063e9 | 1.7281e6 | 1717.324719 |
| "Malawi" | 1.1801505e7 | 5026.221784 | 425.896679 |
We can extract columns directly for matplotlib
Note
Polars also provides a built-in plotting API
based on Altair (e.g., df.plot.bar(x=..., y=...)).
We use matplotlib here for consistency with the rest of the lecture series.
fig, ax = plt.subplots()
ax.bar(df['country'].to_list(), df['GDP percap'].to_list())
ax.set_xlabel('country', fontsize=12)
ax.set_ylabel('GDP per capita', fontsize=12)
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()
19.4. Lazy evaluation#
One of Polars’ most powerful features is lazy evaluation.
Instead of executing each operation immediately, lazy mode collects the full query plan and optimizes it before running.
19.4.1. Eager vs lazy#
# Reload the dataset
url = ('https://raw.githubusercontent.com/QuantEcon/'
'lecture-python-programming/main/lectures/_static/'
'lecture_specific/pandas/data/test_pwt.csv')
df_full = pl.read_csv(url)
The eager API executes immediately (like pandas)
result_eager = (df_full
.filter(pl.col('tcgdp') > 1000)
.select(['country', 'year', 'tcgdp'])
.sort('tcgdp', descending=True)
)
result_eager.head()
| country | year | tcgdp |
|---|---|---|
| str | i64 | f64 |
| "United States" | 2000 | 9.8987e6 |
| "India" | 2000 | 1.7281e6 |
| "Australia" | 2000 | 541804.6521 |
| "Argentina" | 2000 | 295072.21869 |
| "South Africa" | 2000 | 227242.36949 |
The lazy API builds a query plan instead
lazy_query = (df_full.lazy()
.filter(pl.col('tcgdp') > 1000)
.select(['country', 'year', 'tcgdp'])
.sort('tcgdp', descending=True)
)
print(lazy_query.explain())
SORT BY [descending: [true]] [col("tcgdp")]
FILTER col("tcgdp") > 1000.0
FROM
DF ["country", "country isocode", "year", "POP", ...]; PROJECT["country", "year", "tcgdp"] 3/8 COLUMNS
Call collect to execute the plan
result_lazy = lazy_query.collect()
result_lazy.head()
| country | year | tcgdp |
|---|---|---|
| str | i64 | f64 |
| "United States" | 2000 | 9.8987e6 |
| "India" | 2000 | 1.7281e6 |
| "Australia" | 2000 | 541804.6521 |
| "Argentina" | 2000 | 295072.21869 |
| "South Africa" | 2000 | 227242.36949 |
19.4.2. Query optimization#
The lazy engine applies several optimizations automatically:
Predicate pushdown — filters are applied as early as possible
Projection pushdown — only required columns are read from the source
Common subexpression elimination — duplicate calculations are merged
Let’s see how Polars rewrites a multi-step query
optimized = (df_full.lazy()
.select(['country', 'year', 'tcgdp', 'POP'])
.filter(pl.col('tcgdp') > 500)
.with_columns(
(pl.col('tcgdp') / pl.col('POP')).alias('gdp_per_capita')
)
.filter(pl.col('gdp_per_capita') > 10)
.select(['country', 'year', 'gdp_per_capita'])
)
print("Optimized plan:")
print(optimized.explain())
Optimized plan:
FILTER col("gdp_per_capita") > 10.0
FROM
simple π 3/3 ["country", "year", ... 1 other column]
WITH_COLUMNS:
[(col("tcgdp") / col("POP")).alias("gdp_per_capita")]
FILTER col("tcgdp") > 500.0
FROM
DF ["country", "country isocode", "year", "POP", ...]; PROJECT["country", "year", "tcgdp", "POP"] 4/8 COLUMNS
Executing the plan gives us the final result
optimized.collect()
| country | year | gdp_per_capita |
|---|---|---|
| str | i64 | f64 |
| "Australia" | 2000 | 28.436433 |
| "Israel" | 2000 | 21.138673 |
| "United States" | 2000 | 35.080382 |
19.4.3. Performance comparison#
Let’s compare pandas, Polars eager, and Polars lazy on the same task.
We start with a small dataset (the Penn World Tables we used above) to show that for small data the differences are negligible
import pandas as pd
import time
# Small dataset -- Penn World Tables (~8 rows)
url = ('https://raw.githubusercontent.com/QuantEcon/'
'lecture-python-programming/main/lectures/_static/'
'lecture_specific/pandas/data/test_pwt.csv')
small_pd = pd.read_csv(url)
small_pl = pl.read_csv(url)
Now we time the same filter-select-sort operation in each library
# pandas
start = time.perf_counter()
_ = (small_pd
.query('tcgdp > 500')
[['country', 'year', 'tcgdp', 'POP']]
.assign(gdp_pc=lambda d: d['tcgdp'] / d['POP'])
.sort_values('gdp_pc', ascending=False))
pd_small = time.perf_counter() - start
# Polars eager
start = time.perf_counter()
_ = (small_pl
.filter(pl.col('tcgdp') > 500)
.select(['country', 'year', 'tcgdp', 'POP'])
.with_columns((pl.col('tcgdp') / pl.col('POP')).alias('gdp_pc'))
.sort('gdp_pc', descending=True))
pl_small = time.perf_counter() - start
print(f"Small data -- pandas: {pd_small:.4f}s | Polars eager: {pl_small:.4f}s")
Small data -- pandas: 0.0060s | Polars eager: 0.0013s
On a handful of rows the speed difference is immaterial — use whichever API you find more convenient.
Now let’s scale up to 5 million rows where the difference becomes clear.
The task is: filter rows where value > 0, compute a weighted product
value * weight, then take the mean of that product within each group —
a grouped weighted average.
n = 5_000_000
np.random.seed(42)
groups = np.random.choice(['A', 'B', 'C', 'D'], n)
values = np.random.randn(n)
weights = np.random.rand(n)
extra1 = np.random.randn(n)
extra2 = np.random.randn(n)
big_pd = pd.DataFrame({
'group': groups, 'value': values,
'weight': weights, 'extra1': extra1, 'extra2': extra2
})
big_pl = pl.DataFrame({
'group': groups, 'value': values,
'weight': weights, 'extra1': extra1, 'extra2': extra2
})
First, the pandas baseline
start = time.perf_counter()
tmp = big_pd[big_pd['value'] > 0][['group', 'value', 'weight']].copy()
tmp['weighted'] = tmp['value'] * tmp['weight']
_ = tmp.groupby('group')['weighted'].mean()
pd_time = time.perf_counter() - start
print(f"pandas: {pd_time:.4f}s")
pandas: 0.2139s
Next, Polars in eager mode
start = time.perf_counter()
_ = (big_pl
.filter(pl.col('value') > 0)
.select(['group', 'value', 'weight'])
.with_columns(
(pl.col('value') * pl.col('weight')).alias('weighted'))
.group_by('group')
.agg(pl.col('weighted').mean()))
eager_time = time.perf_counter() - start
print(f"Polars eager: {eager_time:.4f}s")
Polars eager: 0.0604s
And finally, Polars in lazy mode
start = time.perf_counter()
_ = (big_pl.lazy()
.filter(pl.col('value') > 0)
.select(['group', 'value', 'weight'])
.with_columns(
(pl.col('value') * pl.col('weight')).alias('weighted'))
.group_by('group')
.agg(pl.col('weighted').mean())
.collect())
lazy_time = time.perf_counter() - start
print(f"Polars lazy: {lazy_time:.4f}s")
Polars lazy: 0.0314s
The take-away:
For small data (thousands of rows), pandas and Polars perform similarly — choose based on API preference and ecosystem fit.
For medium to large data (hundreds of thousands of rows and above), Polars can be significantly faster thanks to its Rust engine, parallel execution, and (in lazy mode) query optimization.
The lazy API is particularly powerful when reading from disk — scan_csv returns a LazyFrame directly, so filters and projections are pushed down to the file reader.
Tip
Use pl.scan_csv(path) instead of pl.read_csv(path) when working with
large CSV files.
Only the columns and rows you actually need will be read from disk.
See the Polars I/O documentation.
19.5. On-line data sources#
As in Pandas, Python makes it straightforward to query online databases.
An important database for economists is FRED — a vast collection of time series data maintained by the St. Louis Fed.
Polars’ read_csv can fetch data from a URL directly.
We use try_parse_dates=True to parse the date column automatically
fred_url = ('https://fred.stlouisfed.org/graph/fredgraph.csv?'
'bgcolor=%23e1e9f0&chart_type=line&drp=0&'
'fo=open%20sans&graph_bgcolor=%23ffffff&'
'height=450&mode=fred&recession_bars=on&'
'txtcolor=%23444444&ts=12&tts=12&width=1318&'
'nt=0&thu=0&trc=0&show_legend=yes&'
'show_axis_titles=yes&show_tooltip=yes&'
'id=UNRATE&scale=left&cosd=1948-01-01&'
'coed=2024-06-01&line_color=%234572a7&'
'link_values=false&line_style=solid&'
'mark_type=none&mw=3&lw=2&ost=-99999&'
'oet=99999&mma=0&fml=a&fq=Monthly&fam=avg&'
'fgst=lin&fgsnd=2020-02-01&line_index=1&'
'transformation=lin&vintage_date=2024-07-29&'
'revision_date=2024-07-29&nd=1948-01-01')
data = pl.read_csv(fred_url, try_parse_dates=True)
Let’s inspect the first few rows
data.head()
| observation_date | UNRATE |
|---|---|
| date | f64 |
| 1948-01-01 | 3.4 |
| 1948-02-01 | 3.8 |
| 1948-03-01 | 4.0 |
| 1948-04-01 | 3.9 |
| 1948-05-01 | 3.5 |
And get summary statistics
data.describe()
| statistic | observation_date | UNRATE |
|---|---|---|
| str | str | f64 |
| "count" | "918" | 918.0 |
| "null_count" | "0" | 0.0 |
| "mean" | "1986-03-17 06:30:35.294117" | 5.693246 |
| "std" | null | 1.710248 |
| "min" | "1948-01-01" | 2.5 |
| "25%" | "1967-02-01" | 4.4 |
| "50%" | "1986-04-01" | 5.5 |
| "75%" | "2005-05-01" | 6.7 |
| "max" | "2024-06-01" | 14.8 |
Plot the unemployment rate from 2006 to 2012
filtered = data.filter(
(pl.col('observation_date') >= pl.date(2006, 1, 1)) &
(pl.col('observation_date') <= pl.date(2012, 12, 31))
)
fig, ax = plt.subplots()
ax.plot(filtered['observation_date'].to_list(),
filtered['UNRATE'].to_list())
ax.set_title('US Unemployment Rate')
ax.set_xlabel('year', fontsize=12)
ax.set_ylabel('%', fontsize=12)
plt.show()
Polars supports many file formats including Excel, JSON, Parquet, and direct database connections.
19.6. Exercises#
Exercise 19.1
With these imports:
import datetime as dt
import yfinance as yf
Write a program to calculate the percentage price change over 2021 for the following shares:
ticker_list = {'INTC': 'Intel',
'MSFT': 'Microsoft',
'IBM': 'IBM',
'BHP': 'BHP',
'TM': 'Toyota',
'AAPL': 'Apple',
'AMZN': 'Amazon',
'C': 'Citigroup',
'QCOM': 'Qualcomm',
'KO': 'Coca-Cola',
'GOOG': 'Google'}
Here’s a function that reads closing prices into a Polars DataFrame:
def read_data_polars(ticker_list,
start=dt.datetime(2021, 1, 1),
end=dt.datetime(2021, 12, 31)):
"""
Read closing price data from Yahoo Finance
and return a Polars DataFrame.
"""
dataframes = []
for tick in ticker_list:
stock = yf.Ticker(tick)
prices = stock.history(start=start, end=end)
df = pl.DataFrame({
'Date': list(prices.index.date),
tick: prices['Close'].values
}).with_columns(pl.col('Date').cast(pl.Date))
dataframes.append(df)
result = dataframes[0]
for df in dataframes[1:]:
result = result.join(
df, on='Date', how='full', coalesce=True
)
return result.sort('Date')
ticker = read_data_polars(ticker_list)
Note
Polars joins do not guarantee the order of the output rows — keys that match
only one side are appended rather than slotted into place.
This is the same “no index, no automatic alignment” theme from above: with no
row labels to align on, ordering is something we ask for explicitly.
Hence the sort('Date') before returning, which any later
first()/last() calculation relies on.
Complete the program to plot the result as a bar graph.
Solution
Calculate percentage changes using Polars expressions:
price_change = ticker.select([
((pl.col(tick).last() / pl.col(tick).first() - 1) * 100)
.alias(tick)
for tick in ticker_list.keys()
]).transpose(
include_header=True,
header_name='ticker',
column_names=['pct_change']
).with_columns(
pl.col('ticker')
.replace_strict(ticker_list, default=pl.col('ticker'))
.alias('company')
).sort('pct_change')
print(price_change)
shape: (11, 3)
┌────────┬────────────┬───────────┐
│ ticker ┆ pct_change ┆ company │
│ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ str │
╞════════╪════════════╪═══════════╡
│ BHP ┆ -2.249103 ┆ BHP │
│ C ┆ 3.550569 ┆ Citigroup │
│ AMZN ┆ 5.845049 ┆ Amazon │
│ INTC ┆ 6.868537 ┆ Intel │
│ KO ┆ 14.922507 ┆ Coca-Cola │
│ … ┆ … ┆ … │
│ TM ┆ 23.416755 ┆ Toyota │
│ QCOM ┆ 25.318529 ┆ Qualcomm │
│ AAPL ┆ 38.55077 ┆ Apple │
│ MSFT ┆ 57.179631 ┆ Microsoft │
│ GOOG ┆ 68.960918 ┆ Google │
└────────┴────────────┴───────────┘
Plot the results using matplotlib directly:
companies = price_change['company'].to_list()
changes = price_change['pct_change'].to_list()
colors = ['red' if x < 0 else 'blue' for x in changes]
fig, ax = plt.subplots(figsize=(10, 8))
ax.bar(companies, changes, color=colors)
ax.set_xlabel('stock', fontsize=12)
ax.set_ylabel('percentage change in price', fontsize=12)
plt.xticks(rotation=45, ha='right')
plt.tight_layout()
plt.show()
Exercise 19.2
Using read_data_polars from Exercise 19.1, obtain year-on-year percentage change for these indices:
indices_list = {'^GSPC': 'S&P 500',
'^IXIC': 'NASDAQ',
'^DJI': 'Dow Jones',
'^N225': 'Nikkei'}
Plot the result as a time series graph.
Solution
indices_data = read_data_polars(
indices_list,
start=dt.datetime(1971, 1, 1),
end=dt.datetime(2021, 12, 31)
)
indices_data = indices_data.with_columns(
pl.col('Date').dt.year().alias('year')
)
Calculate yearly returns using group-by operations:
yearly_returns = indices_data.group_by('year').agg([
*[pl.col(idx).drop_nulls().first().alias(f'{idx}_first')
for idx in indices_list],
*[pl.col(idx).drop_nulls().last().alias(f'{idx}_last')
for idx in indices_list]
])
for idx, name in indices_list.items():
yearly_returns = yearly_returns.with_columns(
((pl.col(f'{idx}_last') - pl.col(f'{idx}_first'))
/ pl.col(f'{idx}_first') * 100).alias(name)
)
yearly_returns = (yearly_returns
.select(['year', *indices_list.values()])
.sort('year')
)
print(yearly_returns)
shape: (51, 5)
┌──────┬────────────┬────────────┬───────────┬────────────┐
│ year ┆ S&P 500 ┆ NASDAQ ┆ Dow Jones ┆ Nikkei │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i32 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │
╞══════╪════════════╪════════════╪═══════════╪════════════╡
│ 1971 ┆ 12.002188 ┆ 14.120003 ┆ null ┆ 36.407234 │
│ 1972 ┆ 16.110952 ┆ 17.668274 ┆ null ┆ 92.011231 │
│ 1973 ┆ -18.094035 ┆ -31.523435 ┆ null ┆ -17.697016 │
│ 1974 ┆ -29.811633 ┆ -35.350697 ┆ null ┆ -9.914309 │
│ 1975 ┆ 28.4209 ┆ 27.874797 ┆ null ┆ 16.798024 │
│ … ┆ … ┆ … ┆ … ┆ … │
│ 2017 ┆ 18.415027 ┆ 27.155799 ┆ 24.331151 ┆ 16.182267 │
│ 2018 ┆ -7.009394 ┆ -5.303631 ┆ -6.028635 ┆ -14.853703 │
│ 2019 ┆ 28.714796 ┆ 34.603667 ┆ 22.23998 ┆ 20.931737 │
│ 2020 ┆ 15.292907 ┆ 41.751104 ┆ 6.019231 ┆ 18.269064 │
│ 2021 ┆ 29.132182 ┆ 23.964416 ┆ 20.428169 ┆ 5.625169 │
└──────┴────────────┴────────────┴───────────┴────────────┘
Summary statistics:
yearly_returns.select(list(indices_list.values())).describe()
| statistic | S&P 500 | NASDAQ | Dow Jones | Nikkei |
|---|---|---|---|---|
| str | f64 | f64 | f64 | f64 |
| "count" | 51.0 | 51.0 | 30.0 | 51.0 |
| "null_count" | 0.0 | 0.0 | 21.0 | 0.0 |
| "mean" | 9.20986 | 13.094786 | 9.10453 | 7.850346 |
| "std" | 16.398012 | 24.616625 | 14.134825 | 24.384181 |
| "min" | -37.58465 | -40.197764 | -32.716831 | -39.695649 |
| "25%" | 0.255632 | 2.470561 | 2.072082 | -6.094919 |
| "50%" | 11.677594 | 14.312575 | 9.387316 | 7.667034 |
| "75%" | 19.671602 | 27.874797 | 21.451016 | 20.931737 |
| "max" | 34.157394 | 84.294285 | 33.311106 | 92.011231 |
Plot each index in a subplot:
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
years = yearly_returns['year'].to_list()
for iter_, ax in enumerate(axes.flatten()):
name = list(indices_list.values())[iter_]
values = yearly_returns[name].to_list()
ax.plot(years, values, 'o-', linewidth=2, markersize=4)
ax.axhline(y=0, color='k', linestyle='--', alpha=0.3)
ax.set_ylabel('yearly return (%)', fontsize=12)
ax.set_xlabel('year', fontsize=12)
ax.set_title(name, fontsize=12)
plt.tight_layout()
plt.show()
