fix tqdm lock deletion

[lhoestq]

related to #6066

[lhoestq]

it reverts #6067

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006573 / 0.011353 (-0.004780)	0.004014 / 0.011008 (-0.006994)	0.084999 / 0.038508 (0.046491)	0.074965 / 0.023109 (0.051855)	0.313294 / 0.275898 (0.037396)	0.349678 / 0.323480 (0.026198)	0.005401 / 0.007986 (-0.002585)	0.003401 / 0.004328 (-0.000927)	0.065363 / 0.004250 (0.061112)	0.057159 / 0.037052 (0.020107)	0.313260 / 0.258489 (0.054771)	0.354654 / 0.293841 (0.060813)	0.030895 / 0.128546 (-0.097651)	0.008605 / 0.075646 (-0.067042)	0.289190 / 0.419271 (-0.130081)	0.052474 / 0.043533 (0.008942)	0.316193 / 0.255139 (0.061054)	0.339966 / 0.283200 (0.056767)	0.024112 / 0.141683 (-0.117571)	1.515606 / 1.452155 (0.063452)	1.571428 / 1.492716 (0.078711)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.203284 / 0.018006 (0.185278)	0.452720 / 0.000490 (0.452230)	0.003891 / 0.000200 (0.003691)	0.000094 / 0.000054 (0.000040)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.028992 / 0.037411 (-0.008419)	0.083170 / 0.014526 (0.068644)	0.097739 / 0.176557 (-0.078817)	0.153401 / 0.737135 (-0.583734)	0.098628 / 0.296338 (-0.197711)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.390190 / 0.215209 (0.174981)	3.901272 / 2.077655 (1.823617)	1.887194 / 1.504120 (0.383074)	1.723696 / 1.541195 (0.182501)	1.800537 / 1.468490 (0.332047)	0.481758 / 4.584777 (-4.103019)	3.605098 / 3.745712 (-0.140614)	3.304482 / 5.269862 (-1.965380)	2.053515 / 4.565676 (-2.512161)	0.056997 / 0.424275 (-0.367278)	0.007347 / 0.007607 (-0.000260)	0.461367 / 0.226044 (0.235323)	4.606152 / 2.268929 (2.337223)	2.470048 / 55.444624 (-52.974576)	2.060019 / 6.876477 (-4.816458)	2.320507 / 2.142072 (0.178435)	0.575050 / 4.805227 (-4.230178)	0.133030 / 6.500664 (-6.367634)	0.061508 / 0.075469 (-0.013962)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.275430 / 1.841788 (-0.566357)	19.725453 / 8.074308 (11.651145)	14.396360 / 10.191392 (4.204968)	0.157980 / 0.680424 (-0.522443)	0.018516 / 0.534201 (-0.515685)	0.394717 / 0.579283 (-0.184566)	0.404948 / 0.434364 (-0.029415)	0.474001 / 0.540337 (-0.066336)	0.668463 / 1.386936 (-0.718474)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.006697 / 0.011353 (-0.004656)	0.004206 / 0.011008 (-0.006802)	0.065458 / 0.038508 (0.026950)	0.075845 / 0.023109 (0.052735)	0.365051 / 0.275898 (0.089153)	0.400919 / 0.323480 (0.077439)	0.005347 / 0.007986 (-0.002638)	0.003386 / 0.004328 (-0.000943)	0.065398 / 0.004250 (0.061148)	0.057320 / 0.037052 (0.020268)	0.379161 / 0.258489 (0.120672)	0.406892 / 0.293841 (0.113051)	0.031986 / 0.128546 (-0.096560)	0.008674 / 0.075646 (-0.066972)	0.071723 / 0.419271 (-0.347549)	0.049897 / 0.043533 (0.006364)	0.372034 / 0.255139 (0.116895)	0.394293 / 0.283200 (0.111094)	0.023681 / 0.141683 (-0.118002)	1.479793 / 1.452155 (0.027639)	1.553105 / 1.492716 (0.060389)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.233660 / 0.018006 (0.215654)	0.454412 / 0.000490 (0.453923)	0.004473 / 0.000200 (0.004273)	0.000085 / 0.000054 (0.000030)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.031115 / 0.037411 (-0.006296)	0.090541 / 0.014526 (0.076015)	0.104363 / 0.176557 (-0.072193)	0.161022 / 0.737135 (-0.576114)	0.105114 / 0.296338 (-0.191225)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.427668 / 0.215209 (0.212459)	4.263145 / 2.077655 (2.185490)	2.247043 / 1.504120 (0.742923)	2.082554 / 1.541195 (0.541360)	2.170505 / 1.468490 (0.702015)	0.491802 / 4.584777 (-4.092975)	3.587295 / 3.745712 (-0.158417)	3.344697 / 5.269862 (-1.925165)	2.060529 / 4.565676 (-2.505148)	0.057829 / 0.424275 (-0.366446)	0.007780 / 0.007607 (0.000173)	0.503374 / 0.226044 (0.277330)	5.034742 / 2.268929 (2.765814)	2.701957 / 55.444624 (-52.742667)	2.479002 / 6.876477 (-4.397474)	2.622055 / 2.142072 (0.479982)	0.591363 / 4.805227 (-4.213864)	0.133834 / 6.500664 (-6.366830)	0.062276 / 0.075469 (-0.013193)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.338788 / 1.841788 (-0.503000)	20.333599 / 8.074308 (12.259291)	14.783196 / 10.191392 (4.591804)	0.168695 / 0.680424 (-0.511729)	0.018478 / 0.534201 (-0.515723)	0.397398 / 0.579283 (-0.181885)	0.409900 / 0.434364 (-0.024464)	0.475315 / 0.540337 (-0.065023)	0.644267 / 1.386936 (-0.742669)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007315 / 0.011353 (-0.004038)	0.004294 / 0.011008 (-0.006714)	0.100300 / 0.038508 (0.061792)	0.077780 / 0.023109 (0.054670)	0.353728 / 0.275898 (0.077830)	0.400538 / 0.323480 (0.077058)	0.005807 / 0.007986 (-0.002178)	0.003649 / 0.004328 (-0.000680)	0.077548 / 0.004250 (0.073297)	0.058834 / 0.037052 (0.021781)	0.352064 / 0.258489 (0.093574)	0.399951 / 0.293841 (0.106110)	0.036472 / 0.128546 (-0.092074)	0.008653 / 0.075646 (-0.066994)	0.323089 / 0.419271 (-0.096182)	0.075127 / 0.043533 (0.031594)	0.334412 / 0.255139 (0.079273)	0.375718 / 0.283200 (0.092519)	0.027915 / 0.141683 (-0.113768)	1.698795 / 1.452155 (0.246640)	1.781447 / 1.492716 (0.288730)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.216111 / 0.018006 (0.198104)	0.507706 / 0.000490 (0.507216)	0.000851 / 0.000200 (0.000651)	0.000085 / 0.000054 (0.000030)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.030451 / 0.037411 (-0.006960)	0.087488 / 0.014526 (0.072962)	0.105094 / 0.176557 (-0.071462)	0.168130 / 0.737135 (-0.569006)	0.106791 / 0.296338 (-0.189547)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.426291 / 0.215209 (0.211082)	4.281046 / 2.077655 (2.203391)	2.162268 / 1.504120 (0.658148)	1.909503 / 1.541195 (0.368309)	1.943165 / 1.468490 (0.474675)	0.516667 / 4.584777 (-4.068110)	4.113218 / 3.745712 (0.367506)	5.931372 / 5.269862 (0.661510)	3.563521 / 4.565676 (-1.002155)	0.062415 / 0.424275 (-0.361860)	0.007577 / 0.007607 (-0.000030)	0.534588 / 0.226044 (0.308543)	5.183490 / 2.268929 (2.914561)	2.790662 / 55.444624 (-52.653962)	2.258630 / 6.876477 (-4.617846)	2.499930 / 2.142072 (0.357857)	0.606154 / 4.805227 (-4.199073)	0.136093 / 6.500664 (-6.364571)	0.061151 / 0.075469 (-0.014318)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.398392 / 1.841788 (-0.443396)	21.482150 / 8.074308 (13.407842)	15.477336 / 10.191392 (5.285944)	0.192878 / 0.680424 (-0.487546)	0.021764 / 0.534201 (-0.512437)	0.437149 / 0.579283 (-0.142134)	0.439976 / 0.434364 (0.005612)	0.514498 / 0.540337 (-0.025840)	0.762642 / 1.386936 (-0.624294)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007504 / 0.011353 (-0.003849)	0.004526 / 0.011008 (-0.006482)	0.071008 / 0.038508 (0.032500)	0.078305 / 0.023109 (0.055195)	0.436160 / 0.275898 (0.160262)	0.439048 / 0.323480 (0.115568)	0.006061 / 0.007986 (-0.001925)	0.003681 / 0.004328 (-0.000648)	0.069445 / 0.004250 (0.065195)	0.059258 / 0.037052 (0.022206)	0.437745 / 0.258489 (0.179256)	0.464247 / 0.293841 (0.170406)	0.033286 / 0.128546 (-0.095260)	0.009846 / 0.075646 (-0.065800)	0.076330 / 0.419271 (-0.342941)	0.051919 / 0.043533 (0.008386)	0.432817 / 0.255139 (0.177678)	0.426295 / 0.283200 (0.143095)	0.029818 / 0.141683 (-0.111865)	1.747640 / 1.452155 (0.295485)	1.726653 / 1.492716 (0.233937)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.251253 / 0.018006 (0.233247)	0.483394 / 0.000490 (0.482904)	0.003992 / 0.000200 (0.003793)	0.000096 / 0.000054 (0.000041)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032180 / 0.037411 (-0.005231)	0.095425 / 0.014526 (0.080900)	0.105908 / 0.176557 (-0.070648)	0.164732 / 0.737135 (-0.572403)	0.115903 / 0.296338 (-0.180435)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.469467 / 0.215209 (0.254258)	4.633239 / 2.077655 (2.555584)	2.517557 / 1.504120 (1.013437)	2.352726 / 1.541195 (0.811531)	2.314618 / 1.468490 (0.846128)	0.548446 / 4.584777 (-4.036331)	3.908797 / 3.745712 (0.163085)	3.525941 / 5.269862 (-1.743921)	2.178858 / 4.565676 (-2.386819)	0.057614 / 0.424275 (-0.366661)	0.008604 / 0.007607 (0.000997)	0.554756 / 0.226044 (0.328711)	5.325635 / 2.268929 (3.056706)	3.014266 / 55.444624 (-52.430359)	2.844165 / 6.876477 (-4.032312)	2.903019 / 2.142072 (0.760947)	0.617750 / 4.805227 (-4.187478)	0.144259 / 6.500664 (-6.356405)	0.065944 / 0.075469 (-0.009525)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.504625 / 1.841788 (-0.337163)	22.400787 / 8.074308 (14.326479)	15.223702 / 10.191392 (5.032310)	0.213357 / 0.680424 (-0.467067)	0.019310 / 0.534201 (-0.514891)	0.456596 / 0.579283 (-0.122687)	0.473811 / 0.434364 (0.039447)	0.517800 / 0.540337 (-0.022537)	0.792468 / 1.386936 (-0.594468)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007420 / 0.011353 (-0.003933)	0.004502 / 0.011008 (-0.006506)	0.097882 / 0.038508 (0.059374)	0.079084 / 0.023109 (0.055975)	0.361797 / 0.275898 (0.085899)	0.416563 / 0.323480 (0.093083)	0.006106 / 0.007986 (-0.001879)	0.003803 / 0.004328 (-0.000526)	0.074669 / 0.004250 (0.070418)	0.062168 / 0.037052 (0.025116)	0.378844 / 0.258489 (0.120355)	0.426601 / 0.293841 (0.132760)	0.035619 / 0.128546 (-0.092927)	0.009686 / 0.075646 (-0.065960)	0.336481 / 0.419271 (-0.082790)	0.065553 / 0.043533 (0.022021)	0.362501 / 0.255139 (0.107362)	0.399752 / 0.283200 (0.116552)	0.028685 / 0.141683 (-0.112998)	1.683495 / 1.452155 (0.231340)	1.786105 / 1.492716 (0.293388)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.220792 / 0.018006 (0.202786)	0.501936 / 0.000490 (0.501447)	0.000389 / 0.000200 (0.000189)	0.000057 / 0.000054 (0.000003)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032180 / 0.037411 (-0.005232)	0.093079 / 0.014526 (0.078553)	0.107967 / 0.176557 (-0.068589)	0.171747 / 0.737135 (-0.565389)	0.107920 / 0.296338 (-0.188418)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.444431 / 0.215209 (0.229222)	4.454934 / 2.077655 (2.377279)	2.140265 / 1.504120 (0.636145)	1.960126 / 1.541195 (0.418931)	2.049649 / 1.468490 (0.581158)	0.557861 / 4.584777 (-4.026916)	4.046240 / 3.745712 (0.300528)	4.513748 / 5.269862 (-0.756114)	2.593643 / 4.565676 (-1.972034)	0.066795 / 0.424275 (-0.357480)	0.008302 / 0.007607 (0.000694)	0.535643 / 0.226044 (0.309599)	5.299429 / 2.268929 (3.030500)	2.656019 / 55.444624 (-52.788606)	2.281214 / 6.876477 (-4.595263)	2.302910 / 2.142072 (0.160837)	0.661696 / 4.805227 (-4.143532)	0.149787 / 6.500664 (-6.350877)	0.069609 / 0.075469 (-0.005860)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.509842 / 1.841788 (-0.331946)	21.717504 / 8.074308 (13.643196)	15.825102 / 10.191392 (5.633710)	0.168115 / 0.680424 (-0.512309)	0.021637 / 0.534201 (-0.512564)	0.454270 / 0.579283 (-0.125013)	0.458531 / 0.434364 (0.024167)	0.523052 / 0.540337 (-0.017285)	0.711219 / 1.386936 (-0.675717)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.007189 / 0.011353 (-0.004164)	0.004437 / 0.011008 (-0.006571)	0.075111 / 0.038508 (0.036603)	0.079245 / 0.023109 (0.056136)	0.423169 / 0.275898 (0.147270)	0.455007 / 0.323480 (0.131527)	0.006076 / 0.007986 (-0.001909)	0.003819 / 0.004328 (-0.000509)	0.074976 / 0.004250 (0.070726)	0.062127 / 0.037052 (0.025075)	0.456809 / 0.258489 (0.198320)	0.474707 / 0.293841 (0.180867)	0.036221 / 0.128546 (-0.092325)	0.009428 / 0.075646 (-0.066218)	0.082842 / 0.419271 (-0.336429)	0.057086 / 0.043533 (0.013553)	0.436121 / 0.255139 (0.180982)	0.453934 / 0.283200 (0.170734)	0.026045 / 0.141683 (-0.115638)	1.789782 / 1.452155 (0.337627)	1.820934 / 1.492716 (0.328218)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.230790 / 0.018006 (0.212784)	0.497987 / 0.000490 (0.497497)	0.002775 / 0.000200 (0.002575)	0.000093 / 0.000054 (0.000038)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.034418 / 0.037411 (-0.002994)	0.105567 / 0.014526 (0.091041)	0.113134 / 0.176557 (-0.063423)	0.173742 / 0.737135 (-0.563394)	0.115936 / 0.296338 (-0.180403)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.502259 / 0.215209 (0.287050)	4.969877 / 2.077655 (2.892222)	2.684860 / 1.504120 (1.180740)	2.484386 / 1.541195 (0.943192)	2.543061 / 1.468490 (1.074571)	0.545733 / 4.584777 (-4.039044)	4.029660 / 3.745712 (0.283948)	5.927883 / 5.269862 (0.658021)	3.528372 / 4.565676 (-1.037305)	0.065957 / 0.424275 (-0.358318)	0.008933 / 0.007607 (0.001326)	0.601630 / 0.226044 (0.375585)	5.825872 / 2.268929 (3.556944)	3.230721 / 55.444624 (-52.213904)	2.891308 / 6.876477 (-3.985169)	3.054994 / 2.142072 (0.912922)	0.665480 / 4.805227 (-4.139747)	0.154815 / 6.500664 (-6.345849)	0.072997 / 0.075469 (-0.002472)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.549892 / 1.841788 (-0.291896)	22.337484 / 8.074308 (14.263176)	16.308286 / 10.191392 (6.116894)	0.189594 / 0.680424 (-0.490830)	0.021844 / 0.534201 (-0.512357)	0.456958 / 0.579283 (-0.122325)	0.459957 / 0.434364 (0.025593)	0.529014 / 0.540337 (-0.011323)	0.700359 / 1.386936 (-0.686577)

[mariosasko]

A bit surprised this hasn't been reported earlier, considering this part of tqdm.concurrent is pretty old (3 years, to be precise)

[github-actions]

Show benchmarks

PyArrow==8.0.0

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009050 / 0.011353 (-0.002303)	0.004968 / 0.011008 (-0.006040)	0.114315 / 0.038508 (0.075807)	0.084475 / 0.023109 (0.061366)	0.426325 / 0.275898 (0.150427)	0.457870 / 0.323480 (0.134390)	0.007076 / 0.007986 (-0.000910)	0.004635 / 0.004328 (0.000307)	0.082950 / 0.004250 (0.078700)	0.065414 / 0.037052 (0.028361)	0.441936 / 0.258489 (0.183447)	0.476983 / 0.293841 (0.183142)	0.048575 / 0.128546 (-0.079972)	0.013929 / 0.075646 (-0.061717)	0.377498 / 0.419271 (-0.041774)	0.081503 / 0.043533 (0.037970)	0.426706 / 0.255139 (0.171567)	0.460374 / 0.283200 (0.177175)	0.046052 / 0.141683 (-0.095631)	1.894896 / 1.452155 (0.442741)	1.998639 / 1.492716 (0.505923)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.313267 / 0.018006 (0.295261)	0.607501 / 0.000490 (0.607012)	0.003369 / 0.000200 (0.003169)	0.000102 / 0.000054 (0.000047)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.032266 / 0.037411 (-0.005145)	0.120138 / 0.014526 (0.105613)	0.115044 / 0.176557 (-0.061513)	0.181374 / 0.737135 (-0.555761)	0.114681 / 0.296338 (-0.181657)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.648039 / 0.215209 (0.432830)	6.005048 / 2.077655 (3.927394)	2.674524 / 1.504120 (1.170404)	2.284831 / 1.541195 (0.743637)	2.360150 / 1.468490 (0.891660)	0.888021 / 4.584777 (-3.696756)	5.419840 / 3.745712 (1.674128)	4.825816 / 5.269862 (-0.444046)	3.140876 / 4.565676 (-1.424801)	0.099511 / 0.424275 (-0.324764)	0.009176 / 0.007607 (0.001569)	0.735646 / 0.226044 (0.509602)	7.224026 / 2.268929 (4.955097)	3.551146 / 55.444624 (-51.893478)	2.844374 / 6.876477 (-4.032103)	3.145307 / 2.142072 (1.003235)	1.077636 / 4.805227 (-3.727591)	0.217754 / 6.500664 (-6.282910)	0.081755 / 0.075469 (0.006286)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.670956 / 1.841788 (-0.170831)	25.524961 / 8.074308 (17.450653)	23.061596 / 10.191392 (12.870204)	0.247524 / 0.680424 (-0.432899)	0.031712 / 0.534201 (-0.502489)	0.513049 / 0.579283 (-0.066234)	0.614568 / 0.434364 (0.180204)	0.574669 / 0.540337 (0.034331)	0.816621 / 1.386936 (-0.570315)

PyArrow==latest

Show updated benchmarks!

Benchmark: benchmark_array_xd.json

metric	read_batch_formatted_as_numpy after write_array2d	read_batch_formatted_as_numpy after write_flattened_sequence	read_batch_formatted_as_numpy after write_nested_sequence	read_batch_unformated after write_array2d	read_batch_unformated after write_flattened_sequence	read_batch_unformated after write_nested_sequence	read_col_formatted_as_numpy after write_array2d	read_col_formatted_as_numpy after write_flattened_sequence	read_col_formatted_as_numpy after write_nested_sequence	read_col_unformated after write_array2d	read_col_unformated after write_flattened_sequence	read_col_unformated after write_nested_sequence	read_formatted_as_numpy after write_array2d	read_formatted_as_numpy after write_flattened_sequence	read_formatted_as_numpy after write_nested_sequence	read_unformated after write_array2d	read_unformated after write_flattened_sequence	read_unformated after write_nested_sequence	write_array2d	write_flattened_sequence	write_nested_sequence
new / old (diff)	0.009384 / 0.011353 (-0.001969)	0.004959 / 0.011008 (-0.006049)	0.084782 / 0.038508 (0.046274)	0.098086 / 0.023109 (0.074977)	0.544395 / 0.275898 (0.268497)	0.585157 / 0.323480 (0.261677)	0.006507 / 0.007986 (-0.001479)	0.004151 / 0.004328 (-0.000178)	0.088596 / 0.004250 (0.084345)	0.069149 / 0.037052 (0.032097)	0.533109 / 0.258489 (0.274620)	0.604117 / 0.293841 (0.310276)	0.047685 / 0.128546 (-0.080861)	0.013651 / 0.075646 (-0.061996)	0.096566 / 0.419271 (-0.322705)	0.062022 / 0.043533 (0.018489)	0.561897 / 0.255139 (0.306758)	0.617636 / 0.283200 (0.334436)	0.034636 / 0.141683 (-0.107047)	1.854667 / 1.452155 (0.402512)	1.908923 / 1.492716 (0.416207)

Benchmark: benchmark_getitem_100B.json

metric	get_batch_of_1024_random_rows	get_batch_of_1024_rows	get_first_row	get_last_row
new / old (diff)	0.260633 / 0.018006 (0.242627)	0.622268 / 0.000490 (0.621778)	0.002116 / 0.000200 (0.001916)	0.000101 / 0.000054 (0.000047)

Benchmark: benchmark_indices_mapping.json

metric	select	shard	shuffle	sort	train_test_split
new / old (diff)	0.035161 / 0.037411 (-0.002250)	0.103707 / 0.014526 (0.089181)	0.115467 / 0.176557 (-0.061090)	0.180077 / 0.737135 (-0.557059)	0.118871 / 0.296338 (-0.177467)

Benchmark: benchmark_iterating.json

metric	read 5000	read 50000	read_batch 50000 10	read_batch 50000 100	read_batch 50000 1000	read_formatted numpy 5000	read_formatted pandas 5000	read_formatted tensorflow 5000	read_formatted torch 5000	read_formatted_batch numpy 5000 10	read_formatted_batch numpy 5000 1000	shuffled read 5000	shuffled read 50000	shuffled read_batch 50000 10	shuffled read_batch 50000 100	shuffled read_batch 50000 1000	shuffled read_formatted numpy 5000	shuffled read_formatted_batch numpy 5000 10	shuffled read_formatted_batch numpy 5000 1000
new / old (diff)	0.628481 / 0.215209 (0.413271)	6.304929 / 2.077655 (4.227275)	3.027775 / 1.504120 (1.523655)	2.753880 / 1.541195 (1.212686)	2.820442 / 1.468490 (1.351952)	0.851103 / 4.584777 (-3.733674)	5.427383 / 3.745712 (1.681670)	7.434310 / 5.269862 (2.164449)	4.418790 / 4.565676 (-0.146887)	0.101733 / 0.424275 (-0.322542)	0.009701 / 0.007607 (0.002094)	0.763033 / 0.226044 (0.536989)	7.497927 / 2.268929 (5.228998)	3.735335 / 55.444624 (-51.709290)	3.149200 / 6.876477 (-3.727277)	3.306214 / 2.142072 (1.164141)	1.085440 / 4.805227 (-3.719787)	0.207562 / 6.500664 (-6.293102)	0.078091 / 0.075469 (0.002622)

Benchmark: benchmark_map_filter.json

metric	filter	map fast-tokenizer batched	map identity	map identity batched	map no-op batched	map no-op batched numpy	map no-op batched pandas	map no-op batched pytorch	map no-op batched tensorflow
new / old (diff)	1.820097 / 1.841788 (-0.021691)	25.525539 / 8.074308 (17.451231)	21.874219 / 10.191392 (11.682827)	0.228391 / 0.680424 (-0.452033)	0.029584 / 0.534201 (-0.504617)	0.511546 / 0.579283 (-0.067737)	0.602719 / 0.434364 (0.168355)	0.581874 / 0.540337 (0.041537)	0.802861 / 1.386936 (-0.584075)