polar
PolarsIt is Rusta fast DataFramelibrary based on writing.
It has the following properties:
- multi-core utilization
- Optimize queries to reduce unnecessary work and memory allocation
- Handle datasets larger than memory
- have a consistent and predictable
API - have a strict format (the data type should be known before running the query)
PolarsHas C、C++the performance of , and has full control over the performance-critical parts of the query engine.
PolarsDo your best:
- Reduce redundant copies
- Efficiently traverse the memory cache
- Minimize Parallelism Contention
- Process data in fast form
- reuse memory allocation
PolarsIO can also be controlled, allowing it to save redundant copies and reduce projectionsand predicatesto scan level.
PolarsWritten from scratch to parallelize DataFramequeries on .
PolarsHas lazyand semi-lazyschemas.
In lazymode, we can optimize the entire query to further improve performance and memory pressure.
Pandas
PandasAlso DataFramea library for handling .
PolarsCompared Pandaswith , it has the following advantages:
- Internal implementation results in less overhead
- Default parallel processing and optimization