Repository navigation
Read v7.3 variables lazily on HDF5.jl 0.18 - #279
Draft
Thenewmanator15 wants to merge 2 commits into
Draft
Thenewmanator15 wants to merge 2 commits into
Thenewmanator15 wants to merge 2 commits into
Conversation
From HDF5.jl 0.18 (JuliaIO/HDF5.jl#1246) a dataset is a DiskArrays AbstractDiskArray: indexing, views, broadcasting and reductions read only the blocks they need, chunk by chunk. MAT.readlazy returns a numeric, logical or complex variable that way, with the conversions read applies: logical data, stored as uint8, as a lazy Bool array, and complex data, stored as a real/imag compound, as a lazy Complex array. Other classes, and v4, v6 and v7 files, raise an ArgumentError. On HDF5.jl 0.16 and 0.17 it raises an ArgumentError saying it needs 0.18, and its tests check only that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # test/runtests.jl
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft until HDF5.jl 0.18 is released.
From HDF5.jl 0.18 a dataset is a DiskArrays
AbstractDiskArray(JuliaIO/HDF5.jl#1246, still a draft): indexing, views, broadcasting and reductions read only the blocks they need, chunk by chunk. This addsMAT.readlazy, which returns a numeric, logical or complex v7.3 variable that way:The MAT.jl side is small. It checks that the variable is a plain array, and applies the conversions
readdoes, lazily: logical data (stored as uint8) reads asBool, and complex data (a real/imag compound) asComplex. Other classes, and v4, v6 and v7 files, raise anArgumentError, as does any HDF5.jl before 0.18. There is no new dependency, since DiskArrays comes with HDF5.jl.Without this, the only lazy route is slicing the internal
fid.plain[...]dataset, which returns logical data asUInt8, complex data as named tuples, and an empty array's stored dimensions as if they were its data.Depends on JuliaIO/HDF5.jl#1246 being released in HDF5.jl 0.18, and on #248 for the compat bound. I will mark this ready then.
Tests.
test/lazy.jlcovers a 3-D array, chunked (compressed) reads, logical, complex andInt8data, and the errors for cells, structs, text, empty arrays and v7 files. On HDF5.jl 0.17, which CI uses today, only the "needs 0.18" error runs. Against the #1246 branch with the compat bound raised, the full suite passes (841); with #267 to #278 merged in as well, it passes on 0.17 (888) and on 0.18 (900). With a file saved from MATLAB R2025a, a 10×20 block of a 1000×2000 variable reads in 0.07 ms against 31.8 ms forread, and logical and complex values matchread.🤖 Generated with Claude Code