Skip to content

Read v7.3 variables lazily on HDF5.jl 0.18 - #279

Draft
Thenewmanator15 wants to merge 2 commits into
JuliaIO:masterfrom
Thenewmanator15:readlazy-hdf5-018
Draft

Thenewmanator15 wants to merge 2 commits into
JuliaIO:masterfrom
Thenewmanator15:readlazy-hdf5-018

Conversation

@Thenewmanator15

Copy link
Copy Markdown
Contributor

Draft until HDF5.jl 0.18 is released.

From HDF5.jl 0.18 a dataset is a DiskArrays AbstractDiskArray (JuliaIO/HDF5.jl#1246, still a draft): indexing, views, broadcasting and reductions read only the blocks they need, chunk by chunk. This adds MAT.readlazy, which returns a numeric, logical or complex v7.3 variable that way:

matopen("data.mat") do fid
    A = MAT.readlazy(fid, "A")
    A[1:10, :]           # reads the first ten rows
    sum(view(A, :, 1))   # reads the first column
end

The MAT.jl side is small. It checks that the variable is a plain array, and applies the conversions read does, lazily: logical data (stored as uint8) reads as Bool, and complex data (a real/imag compound) as Complex. Other classes, and v4, v6 and v7 files, raise an ArgumentError, as does any HDF5.jl before 0.18. There is no new dependency, since DiskArrays comes with HDF5.jl.

Without this, the only lazy route is slicing the internal fid.plain[...] dataset, which returns logical data as UInt8, complex data as named tuples, and an empty array's stored dimensions as if they were its data.

Depends on JuliaIO/HDF5.jl#1246 being released in HDF5.jl 0.18, and on #248 for the compat bound. I will mark this ready then.

Tests. test/lazy.jl covers a 3-D array, chunked (compressed) reads, logical, complex and Int8 data, and the errors for cells, structs, text, empty arrays and v7 files. On HDF5.jl 0.17, which CI uses today, only the "needs 0.18" error runs. Against the #1246 branch with the compat bound raised, the full suite passes (841); with #267 to #278 merged in as well, it passes on 0.17 (888) and on 0.18 (900). With a file saved from MATLAB R2025a, a 10×20 block of a 1000×2000 variable reads in 0.07 ms against 31.8 ms for read, and logical and complex values match read.

🤖 Generated with Claude Code

Thenewmanator15 and others added 2 commits October 23, 2026 14:53
From HDF5.jl 0.18 (JuliaIO/HDF5.jl#1246) a dataset is a DiskArrays
AbstractDiskArray: indexing, views, broadcasting and reductions read
only the blocks they need, chunk by chunk. MAT.readlazy returns a
numeric, logical or complex variable that way, with the conversions
read applies: logical data, stored as uint8, as a lazy Bool array, and
complex data, stored as a real/imag compound, as a lazy Complex array.
Other classes, and v4, v6 and v7 files, raise an ArgumentError.

On HDF5.jl 0.16 and 0.17 it raises an ArgumentError saying it needs
0.18, and its tests check only that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	test/runtests.jl

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant