python-v1.1.4

What's Changed

• chore: bump version for release by @rtyler in #3633
• feat: add keep_versions parameter to vacuum command for python by @corwinjoy in #3635
• chore: remove deprecated use of kernel's Table by @rtyler in #3639
• chore: use pytest-xdist for speeding up python tests by @rtyler in #3642
• fix: aws special paths encoding by @roeap in #3656
• fix: handle checking partition filters in array/list when converting … by @smeyerre in #3657
• ci: run integration tests against next branches by @roeap in #3658
• feat: support converting parquet with non-microsecond timestamps to d… by @smeyerre in #3654
• fix: use RFC3896 percent encoding with delta protocol correctness by @ion-elgreco in #3661
• chore: bump python by @ion-elgreco in #3664

Full Changelog: python-v1.1.3...python-v1.1.4

python-v1.1.3

What's Changed

• refactor: make "cloud" feature in object_store optional by @zeevm in #3590
• fix: creating new DeltaTable with invalid table name path no longer creates empty directory by @smeyerre in #3504
• fix: fix typo to fix CI typo check by @alamb in #3604
• fix: scan time was always 0 for merge metrics by @rtyler in #3596
• chore: minor API changes after integration testing by @rtyler in #3598
• fix: switch the url schemes for Azure integration tests by @rtyler in #3614
• fix: allow non-string primitive types for partition filters when converting to pyarrow dataset by @smeyerre in #3613
• refactor: make match_partitions and new_metadata public by @zeevm in #3605
• fix: ensure new checkpoints can be written after old checkpoints by @rtyler in #3616
• docs: fix broken daft links (how daft) by @rtyler in #3617
• fix: allow writing to DeltaTable objects across Python threads by @rtyler in #3618
• fix: ensure openssl-sys doesn't creep into the dependency via the kernel default engine by @rtyler in #3619
• chore: update to DataFusion 49.0.0 by @alamb in #3603
• chore(deps): update rstest requirement from 0.25.0 to 0.26.1 by @dependabot[bot] in #3627
• fix: coerce polars.Array into a suitable Arrow list type by @rtyler in #3623
• fix: make the docs link checking more useful/less faily by @rtyler in #3630
• fix: avoid parsing generationExpressions as JSON by @rtyler in #3632
• feat: build musl wheels upon release by @rtyler in #3631

Full Changelog: python-v1.1.0...python-v1.1.3

python-v1.1.0

What's Changed

• refactor: compute stats schema with kernel types by @roeap in #3514
• chore: set java version to 21 for pyspark 4.0 by @ion-elgreco in #3524
• refactor: remove unecessary uses of datafusion subcrates by @alamb in #3521
• chore: pin aws crates by @ion-elgreco in #3532
• fix: using state provided in args in merge op by @gtrawinski in #3522
• fix: remove forced table update from python writer by @ohanf in #3515
• feat: make TableConfig accessible by @ion-elgreco in #3518
• chore: update the minor version to reflect a behavior change by @rtyler in #3542
• chore: update arrow/parquet to 55.2.0 by @alamb in #3558
• chore: clean up licenses in python project which are causing build issues by @rtyler in #3560
• fix: version binary search by @aditanase in #3549
• chore: update to DataFusion 48.0.0 / arrow to 55.2.0 by @alamb in #3520
• chore: upgrade to delta_kernel 0.12.x by @rtyler in #3561
• docs: ensure create_checkpoint() is visible in the Python API docs by @itamarst in #3564
• fix: use proper DeltaTableState for vacuum commits by @jeromegn in #3550
• chore: remove redundant words in comment by @shangchenglumetro in #3568
• refactor: move schema code to kernel module by @roeap in #3569
• feat: convert partition filters to kernel predicates by @roeap in #3570
• chore: latest clippy by @roeap in #3571
• chore: remove the deltalake-sql crate by @rtyler in #3582
• feat: write engineInfo with delta-rs version by @zachschuermann in #3584
• chore: bump patch versions for another relaese by @rtyler in #3585
• feat: vacuum with version retention by @corwinjoy in #3537
• chore: bump minor version for rust crate by @rtyler in #3586
• refactor!: use delta-kernel Protocol and Metadata actions by @roeap in #3581
• chore: generate a more recentish updated changelog by @rtyler in #3588

New Contributors

• @ohanf made their first contribution in #3515
• @itamarst made their first contribution in #3564
• @jeromegn made their first contribution in #3550
• @shangchenglumetro made their first contribution in #3568

Full Changelog: python-v1.0.2...python-v1.1.0

rust-v0.27.0

Full Changelog

Implemented enhancements:

• Feature: Vacuum with version retention #3530
• Any way to prune the delta_log or support shallow clones #3565
• Upgrade Arrow version to 55.1.0 #3540
• Add config option to suppress deltalake_core::writer::stats warnings about bytes columns #3519
• Remove pyarrow dependency (make opt-in), replace with arro3 for core components #3455
• Don't retry lakefs commit or merge on 412 response (precondition failed) #3429
• Use object_store spawnService #3427
• Alter table description #3401
• Remove put if absent options injection #3310
• v1.0 Release tracking issue #3250
• feat: add a table description and name to the Delta Table from Python #3464 (fvaleye)

Fixed bugs:

• Python building broken on main due to maturin issue #3559
• TypeError: write_deltalake() got an unexpected keyword argument 'schema' (deltalake/polars) #3546
• SchemaMismatchError on empty ArrayType field while contains_null=True #3544
• Can't open a delta-table: Unsupported reader features required: DeletionVectors #3543
• Attempting to write a transaction 3 but the underlying table has been updated to 3 #3534
• DeltaOps not recognizing abfss scheme for Azure #3523
• Query execution time difference between QueryBuilder and using DataFusion directly. #3517
• bug: timezone not preserved & raise exc on merge operation #3507
• allow_unsafe_rename option stopped working in version 1 #3493
• predicate appears to ignore partition and stats in pruning #3491
• max_rows_per_file ignored when writing with rust engine #3490
• delta-rs includes pending versions written by spark #3422

Merged pull requests:

• chore: bump minor version for rust crate #3586 (rtyler)
• refactor!: use delta-kernel Protocol and Metadata actions #3581 (roeap)
• feat: vacuum with version retention #3537 (corwinjoy)
• chore: bump patch versions for another relaese #3585 (rtyler)
• feat: write engineInfo with delta-rs version #3584 (zachschuermann)
• chore: remove the deltalake-sql crate #3582 (rtyler)
• chore: latest clippy #3571 (roeap)
• feat: convert partition filters to kernel predicates #3570 (roeap)
• refactor: move schema code to kernel module #3569 (roeap)
• chore: remove redundant words in comment #3568 (shangchenglumetro)
• docs: ensure create_checkpoint() is visible in the Python API docs #3564 (itamarst)
• chore: upgrade to delta_kernel 0.12.x #3561 (rtyler)
• chore: clean up licenses in python project which are causing build issues #3560 (rtyler)
• chore: update arrow/parquet to 55.2.0 #3558 (alamb)
• fix: use proper DeltaTableState for vacuum commits #3550 (jeromegn)
• fix: version binary search #3549 (aditanase)
• chore: update the minor version to reflect a behavior change #3542 (rtyler)
• chore: pin aws crates #3532 (ion-elgreco)
• chore: set java version to 21 for pyspark 4.0 #3524 (ion-elgreco)
• fix: using state provided in args in merge op #3522 (gtrawinski)
• refactor: remove unecessary uses of datafusion subcrates #3521 (alamb)
• chore: update to DataFusion 48.0.0 / arrow to 55.2.0 #3520 (alamb)
• feat: make TableConfig accessible #3518 (ion-elgreco)
• fix: remove forced table update from python writer #3515 (ohanf)
• refactor: compute stats schema with kernel types #3514 (roeap)
• feat: add convenience extension for kernel engine types #3510 (roeap)
• refactor: move LazyTableProvider into python crate #3509 (roeap)
• fix: setting wrong schema in table provider for merge #3508 (ion-elgreco)
• fix: constraint parsing, roundtripping #3503 (ion-elgreco)
• refactor!: have DeltaTable::version return an Option #3500 (roeap)
• chore!: remove get_earliest_version #3499 (roeap)
• chore: prepare for the next python release #3498 (rtyler)
• ci: improve coverage collection #3497 (roeap)
• chore: update runner #3494 (ion-elgreco)
• docs: update link to df #3489 (rluvaton)
• refactor!: remove and deprecate some python methods #3488 (roeap)
• fix: ensure projecting only columns that exist in new files afte sche… #3487 (alexwilcoxson-rel)
• chore: exclude Invariants from the default writer v2 feature set #3486 (rtyler)
• test: improve storage config testing #3485 (roeap)
• refactor!: get transaction versions for specific applications #3484 (roeap)
• docs: fix bullet list formatting in dagster docs #3483 (avriiil)
• fix: set casting safe param to False #3481 (ion-elgreco)
• chore: update kernel to 0.11 #3480 (roeap)
• chore: update migration docs #3479 (ion-elgreco)
• chore: remove unused stats_parsed field #3475 (roeap)
• refactor: remove protocol error #3473 (roeap)
• chore: more typos #3471 (roeap)
• chore: remove unused time_utils #3470 (roeap)
• chore: set correct markers...

python-v1.0.2

What's Changed

• fix: setting wrong schema in table provider for merge by @ion-elgreco in #3508
• refactor: move LazyTableProvider into python crate by @roeap in #3509
• feat: add convenience extension for kernel engine types by @roeap in #3510

Full Changelog: python-v1.0.1...python-v1.0.2

python-v1.0.1

Bug Fixes

• fix: constraint parsing, roundtripping by @ion-elgreco in #3503

Other Changes

• docs: update link to df by @rluvaton in #3489
• chore: update runner by @ion-elgreco in #3494
• ci: improve coverage collection by @roeap in #3497
• chore: prepare for the next python release by @rtyler in #3498
• chore!: remove get_earliest_version by @roeap in #3499
• refactor!: have DeltaTable::version return an Option by @roeap in #3500

New Contributors

• @rluvaton made their first contribution in #3489

Full Changelog: python-v1.0.0...python-v1.0.1

python-v1.0.0: Zero to One

It only took us 5 years, but we made it! You can find the upgrade guide here.

Performance improvements

• refactor: async writer + multi-part by @ion-elgreco in #3255
• perf: use lazy sync reader by @ion-elgreco in #3338

New features

• feat: remove optimize operations when building without Apache Datafusion by @rtyler in #3290
• feat(api): add rustls and native-tls features by @zeevm in #3335
• feat!: update storage configuration system by @roeap in #3383
• feat: derive macro for config implementations by @roeap in #3389
• feat: upgrade to DataFusion 47.0.0 by @alamb in #3378
• feat: introduce VacuumMode::Full for cleaning up orphaned files by @rtyler in #3368
• feat: during LakeFS file operations, skip merge when 0 changes by @smeyerre in #3346
• feat: added a check for gc code to run by @JustinRush80 in #3419
• feat: spawn io with spawn service by @ion-elgreco in #3426
• feat: optimize datafusion predicate pushdown and partition pruning by @rtyler in #3436
• feat: expose kernel Engine on LogStore by @roeap in #3446
• refactor: remove pyarrow dependency by @ion-elgreco in #3459
• feat: write checkpoints with kernel by @roeap in #3466
• feat: add a table description and name to the Delta Table from Python by @fvaleye in #3464
• refactor!: remove and deprecate some python methods by @roeap in #3488

Bug Fixes

• fix: use field physical name when resolving partition columns by @zeevm in #3349
• fix(pandas): retain pyarrow decimal datatype in to_pandas() by adding types_mapper to prevent precision loss by @Abhishek1005 in #3296
• fix: prevent panics when peek_next_commit() encounters invalid data by @rtyler in #3308
• fix: serialize empty deletionVector in add actions as absent by @rtyler in #3309
• fix: stats column binary_column has unsupported type binary by @omkar-foss in #3146
• fix: check for all known valid delta files in is_deltatable by @umartin in #3318
• fix: block_in_place to allow nested tasks by @ion-elgreco in #3324
• fix: parse snapshot by @ion-elgreco in #3355
• fix: added restored metadata as action to the next committed version by @Nordalf in #3303
• fix: parse unconventional logs by @roeap in #3373
• fix: clippy warnings by @alamb in #3390
• fix: the default target size should be 100MB by @HiromuHota in #3404
• fix: if field contains space in constraint expression, checks will fail by @Nordalf in #3374
• fix: build Unity Catalog crate without DataFusion by @linhr in #3420
• fix: drop column update by @ion-elgreco in #3416
• fix: ignore temp log entries by @corwinjoy in #3423
• fix: use more accurate log path parsing by @roeap in #3461
• fix: correct spelling errors found by CI spell checker by @fvaleye in #3465
• fix: schema conversion, add conversion test cases by @ion-elgreco in #3468
• fix: set casting safe param to False by @ion-elgreco in #3481
• fix: ensure projecting only columns that exist in new files afte sche… by @alexwilcoxson-rel in #3487

Other Changes

• refactor: drop pyarrow support, restructure python modules by @ion-elgreco in #3285
• chore: bump python version for release by @rtyler in #3291
• chore: use flags for apple arm64 by @ion-elgreco in #3213
• chore: upgrade the kernel version and bump our majorish versions too by @rtyler in #3289
• chore: upgrade to DataFusion 46.0.0 by @alamb in #3261
• refactor: add 'cloud' feature to 'core' to enable 'cloud' on 'object_store' only when needed by @zeevm in #3332
• docs: update dataFusion integration example by @riziles in #3343
• refactor(python): improve typing, linting by @ion-elgreco in #3344
• chore: remove pyarrow upper by @ion-elgreco in #3325
• chore: improve io error msg by @ion-elgreco in #3328
• docs: update merge-tables.md with "Optimizing Merge Performance" section by @ldacey in #3351
• docs: add example how to authenticate using Azure CLI for Azure ADSL integration by @DanielBertocci in #3357
• chore: remove cdf feature by @ion-elgreco in #3365
• fix: correct Python docs for incremental compaction on OPTIMIZE by @roykim98 in #3301
• chore: fix some minor build warnings by @rtyler in #3366
• refactor: move transaction module to kernel by @roeap in #3380
• chore: clippy by @roeap in #3379
• chore: move proofs into dedicated folder by @roeap in #3381
• refactor!: move storage module into logstore by @roeap in #3382
• chore: put a couple symbols behind the right feature gate by @rtyler in #3393
• chore: update delta_kernel to 0.10.0 by @zachschuermann in #3403
• refactor: make "cloud" feature in object_store optional by @zeevm in #3398
• chore: bump versions of rust crates for another release party by @rtyler in #3406
• chore: commit the contents of the 0.26.0 release by @rtyler in #3408
• chore: reduce scope of feature flags and compilation requirements for subcrates by @rtyler in #3409
• chore(deps): update sqlparser requirement from 0.53.0 to 0.56.0 by @dependabot in #3413
• chore(deps): update foyer requirement from 0.16.1 to 0.17.0 by @dependabot in #3412
• chore: bringing dat integration testing in ahead of kernel replay by @rtyler in #3411
• chore: missed a version bump for core by @rtyler in #3415
• chore: include license file in deltalake-derive crate by @ankane in #3417
• chore(deps): bump foyer to v0.17.2 to prevent from wrong result by @MrCroxx in #3428
• chore: bump crate versions which are due for release by @rtyler in #3430
• chore: rely on the testing during coverage generation to speed up tests by @rtyler in #3431
• chore: make codecov more vigorously enforced to help ensure quality by @rtyler in #3434
• chore: prepare py-1.0 release by @ion-elgreco in #3435
• chore: experiment with using sccache in GitHub Actions by @rtyler in #3437
• chore: remove unused code and deps by @roeap in #3441
• chore: minor table module refactors by @rtyler in #3442
• docs: add 1.0.0 migration guide by @ion-elgreco in #3443
• refactor: more specific factory parameter names by @roeap in #3445
• refactor: use LogStore in Snapshot / LogSegment APIs by @roeap in #3452
• test: avoid circular dependency with core/test crates by @roeap in #3450
• chore: ensuring default builds work without datafusion by @rtyler in #3453
• ci: add spellchecker to pr tests by @roeap in #3457
• chore: mark more tests which require datafusion by @rtyler in #3458
• refactor: use full paths in log processing by @roeap in #3456
• chore: set correct markers by @ion-elgreco in https://github.com/delta-io/...

python-v0.25.5

What's Changed

• chore: relax the pyarrow constraints now that 19.0.1 is out by @rtyler in #3361

Full Changelog: python-v0.25.4...python-v0.25.5

python-v0.25.4

What's Changed

• chore: more manual inline format args by @nyurik in #3251
• fix: chrono 0.4.40 causes disambiguation syntax when building arrow deps by @Nordalf in #3272
• fix(rust): on write, have a schema evolution maintain metadata table id by @liamphmurphy in #3275
• chore: remove unused dev dependency on home by @alamb in #3277
• refactor: replaced asterisk with constraint name in get_constraints by @Nordalf in #3270
• refactor: simplify expressions by @ion-elgreco in #3281
• fix: compare before after schema before creating action by @ion-elgreco in #3282
• fix: timestamp truncation in stats parsed by @ion-elgreco in #3288

Full Changelog: python-v0.25.2...python-v0.25.4

python-v0.25.2

New features

• feat: set clientVersion for python binding by @ion-elgreco in #3242

Bug Fixes

• fix: aarch64 ssl handshakes by @ion-elgreco in #3263
• fix: enable missing cloud features for uc crate by @omkar-foss in #3260
• fix: refresh snapshot after vacuuming logs by @ion-elgreco in #3252
• fix: use tableprovider scan in z-order by @ion-elgreco in #3238
• fix: commitlint config by @roeap in #3268

Other Changes

• chore: inline format args to improve readability by @nyurik in #3154
• chore: update readme for Generated columns by @ion-elgreco in #3247
• docs: update CONTRIBUTING.md and Makefile to account for switch to uv by @adamreeve in #3257
• chore: fix some compiler warnings by @alamb in #3262
• chore: fix clippy warnings on main by @alamb in #3264
• chore: use builder API to create FileScanConfig by @alamb in #3266
• chore: update tests to use Column::new and other expr_fn functions by @alamb in #3265

New Contributors

• @adamreeve made their first contribution in #3257

Full Changelog: python-v0.25.1...python-v0.25.2