> We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience
I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
That being said Polars is one of the few Python libraries from the hundreds I use that I need to read the notes of every minor release (eg 1.44 -> 1.45), because they tend to frequently deprecate, remove or change features.
What is it about polars syntax you don't like? The fact that is very verbose? At first I wasn't a fan, but over time I've grown to really like it. That never happened to me with pandas, always felt the syntax was messy
I tend to agree. SQL may have been harder to write in the past (worse autocomplete than pandas/polars), but now that AI is writing the code, SQL is usually much easier to read. So DuckDB is another interesting alternative to pandas.
The cool thing about polars is that you can conditionally collect expressions over many layers of business logic, and then compute the result at the end. Doing this in SQL ends up in a hodgepodge of strings and trimmed ends to please the syntax. You can also pretty effortlessly write quite complex conditionals directly in polars, and bridge it easily to the surrounding python.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
The decision to default to the streaming engine is really interesting. My intuition is that this would be slower than other data frame operations that are more parallelizable with batch processing, because streaming engines necessarily process rows sequentially. Is my intuition off/am I overestimating how much auto-parallelization polars does?
Streaming here has a different meaning than perhaps what you're used to. It's not referring to online processing where you maintain aggregates/state while an endless stream of data comes in.
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
I like when large projects do that. This gives leeway for sister projects (eg wrappers) to anticipate, room for apps that use it intensively to test things out (release candidate etc), something which has really helped me in the past.
In that specific case I use a Polars wrapper in Elixir (called Explorer) all week long, and I am very happy they are giving us early hints.
I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
Can there be deprecated cruft without new features? :-D
Pandas is a mess though.
df.select(
)with
df |> select(x, y = w/z)
`df.select("x", y=pl.col.w/pl.col.z)`
ggplot vs matplotlib
dplyr vs pandas
And I loved that everything in RStudio was so easily inspectable. Have a huge dataframe? Just look at it right in your IDE.
Unfortunately, polars does not support parameterized queries, so the risk of SQL injection is extremely high.
I find that SQL is only easier to read with minimal abstraction, but as soon as the project gets bigger SQL becomes an unwieldy island of different that has served its purpose after we’re done with reading/writing the data.
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
In that specific case I use a Polars wrapper in Elixir (called Explorer) all week long, and I am very happy they are giving us early hints.