Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault when reading a file with a bad checksum #1184

Closed
nhz2 opened this issue Dec 28, 2024 · 7 comments
Closed

Segfault when reading a file with a bad checksum #1184

nhz2 opened this issue Dec 28, 2024 · 7 comments

Comments

@nhz2
Copy link
Member

nhz2 commented Dec 28, 2024

I was trying to learn about how HDF5 uses checksums and created the following file that crashes julia:

t7.txt

On Linux:

julia> using Pkg

julia> using HDF5

julia> Pkg.status(Pkg.PKGMODE_MANIFEST)
Status `/tmp/jl_aOHUF3/Manifest.toml`
  [34da2185] Compat v4.16.0
  [f67ccb44] HDF5 v0.17.2
  [692b3bcd] JLLWrappers v1.7.0
  [3da0fdf6] MPIPreferences v0.1.11
  [21216c6a] Preferences v1.4.3
  [ae029012] Requires v1.3.0
  [0234f1f7] HDF5_jll v1.14.3+3
  [e33a78d0] Hwloc_jll v2.11.2+2
  [7cb0a576] MPICH_jll v4.2.3+0
  [f1f71cc9] MPItrampoline_jll v5.5.1+1
  [9237b28f] MicrosoftMPI_jll v10.1.4+3
⌅ [fe0851c0] OpenMPI_jll v4.1.6+0
  [458c3c95] OpenSSL_jll v3.0.15+2
  [477f73a3] libaec_jll v1.1.2+1
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching v1.11.0
  [4af54fe1] LazyArtifacts v1.11.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [a63ad114] Mmap v1.11.0
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.11.0
  [de0858da] Printf v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.1.1+0
  [deac9b47] LibCURL_jll v8.6.0+0
  [e37daf67] LibGit2_jll v1.7.2+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.6+0
  [14a3606d] MozillaCACerts_jll v2023.12.12
  [83775a58] Zlib_jll v1.2.13+1
  [8e850ede] nghttp2_jll v1.59.0+0
  [3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`

julia> versioninfo()
Julia Version 1.11.2
Commit 5e9a32e7af2 (2024-12-01 20:02 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × 12th Gen Intel(R) Core(TM) i5-1240P
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, alderlake)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

julia> h5open("t7.txt", "r") do f
           f["test-data"][1]
       end

[640] signal 11 (1): Segmentation fault
in expression starting at REPL[7]:1
H5_checksum_fletcher32 at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5Z__filter_fletcher32 at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5Z_pipeline at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__chunk_lock.constprop.19 at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__chunk_read at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__read at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5VL__native_dataset_read at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5VL_dataset_read_direct at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__read_api_common.constprop.2 at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5Dread at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
h5d_read at /home/nathan/.julia/packages/HDF5/Z859u/src/api/functions.jl:796
_generic_read at /home/nathan/.julia/packages/HDF5/Z859u/src/readwrite.jl:180
generic_read at /home/nathan/.julia/packages/HDF5/Z859u/src/readwrite.jl:146
unknown function (ip: 0x7fd68ef7f7d5)
getindex at /home/nathan/.julia/packages/HDF5/Z859u/src/readwrite.jl:59
#1 at ./REPL[7]:2
#17 at /home/nathan/.julia/packages/HDF5/Z859u/src/file.jl:101
task_local_storage at ./task.jl:315
#h5open#16 at /home/nathan/.julia/packages/HDF5/Z859u/src/file.jl:96
h5open at /home/nathan/.julia/packages/HDF5/Z859u/src/file.jl:94
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
do_call at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_value at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:223
eval_stmt_value at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:174 [inlined]
eval_body at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:663
jl_interpret_toplevel_thunk at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:430 [inlined]
eval_user_input at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:245
repl_backend_loop at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:342
#start_repl_backend#59 at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:327
start_repl_backend at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:324
#run_repl#72 at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:483
run_repl at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:469
jfptr_run_repl_10104.1 at /home/nathan/.julia/juliaup/julia-1.11.2+0.x64.linux.gnu/share/julia/compiled/v1.11/REPL/u0gqU_4x0TT.so (unknown line)
#1150 at ./client.jl:446
jfptr_YY.1150_14803.1 at /home/nathan/.julia/juliaup/julia-1.11.2+0.x64.linux.gnu/share/julia/compiled/v1.11/REPL/u0gqU_4x0TT.so (unknown line)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_f__call_latest at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/builtins.c:875
#invokelatest#2 at ./essentials.jl:1055 [inlined]
invokelatest at ./essentials.jl:1052 [inlined]
run_main_repl at ./client.jl:430
repl_main at ./client.jl:567 [inlined]
_start at ./client.jl:541
jfptr__start_73406.1 at /home/nathan/.julia/juliaup/julia-1.11.2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
true_main at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/jlapi.c:1059
main at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/cli/loader_exe.c:58
unknown function (ip: 0x7fd6e8652d8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 11447121 (Pool: 11443847; Big: 3274); GC: 15
Segmentation fault (core dumped)
@nhz2
Copy link
Member Author

nhz2 commented Dec 29, 2024

This one only segfaults sometimes.

t20.txt

julia> h5open("t20.txt", "r") do f
           collect(f["test-data"])
       end

[1742] signal 11 (1): Segmentation fault
in expression starting at REPL[5]:1
unknown function (ip: 0x7f2772da9aca)
H5VM_memcpyvv at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__compact_readvv at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__select_io at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__select_read at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__chunk_read at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__read at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5VL__native_dataset_read at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5VL_dataset_read_direct at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5D__read_api_common.constprop.2 at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
H5Dread at /home/nathan/.julia/artifacts/3f844d84068534dcd6606936ab5f28e1120a9bb0/lib/libhdf5.so (unknown line)
h5d_read at /home/nathan/.julia/packages/HDF5/Z859u/src/api/functions.jl:796
_generic_read at /home/nathan/.julia/packages/HDF5/Z859u/src/readwrite.jl:180
generic_read! at /home/nathan/.julia/packages/HDF5/Z859u/src/readwrite.jl:141 [inlined]
copyto! at /home/nathan/.julia/packages/HDF5/Z859u/src/readwrite.jl:94
unknown function (ip: 0x7f271957939d)
_collect at ./array.jl:722
collect at ./array.jl:716 [inlined]
#7 at ./REPL[5]:2 [inlined]
#17 at /home/nathan/.julia/packages/HDF5/Z859u/src/file.jl:101
task_local_storage at ./task.jl:315
#h5open#16 at /home/nathan/.julia/packages/HDF5/Z859u/src/file.jl:96
h5open at /home/nathan/.julia/packages/HDF5/Z859u/src/file.jl:94
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
do_call at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_value at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:223
eval_stmt_value at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:174 [inlined]
eval_body at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:663
jl_interpret_toplevel_thunk at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/interpreter.c:821
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:430 [inlined]
eval_user_input at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:245
repl_backend_loop at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:342
#start_repl_backend#59 at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:327
start_repl_backend at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:324
#run_repl#72 at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:483
run_repl at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/REPL/src/REPL.jl:469
jfptr_run_repl_10104.1 at /home/nathan/.julia/juliaup/julia-1.11.2+0.x64.linux.gnu/share/julia/compiled/v1.11/REPL/u0gqU_4x0TT.so (unknown line)
#1150 at ./client.jl:446
jfptr_YY.1150_14803.1 at /home/nathan/.julia/juliaup/julia-1.11.2+0.x64.linux.gnu/share/julia/compiled/v1.11/REPL/u0gqU_4x0TT.so (unknown line)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_f__call_latest at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/builtins.c:875
#invokelatest#2 at ./essentials.jl:1055 [inlined]
invokelatest at ./essentials.jl:1052 [inlined]
run_main_repl at ./client.jl:430
repl_main at ./client.jl:567 [inlined]
_start at ./client.jl:541
jfptr__start_73406.1 at /home/nathan/.julia/juliaup/julia-1.11.2+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
true_main at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/src/jlapi.c:1059
main at /cache/build/tester-amdci5-12/julialang/julia-release-1-dot-11/cli/loader_exe.c:58
unknown function (ip: 0x7f2772c32d8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 10179476 (Pool: 10176099; Big: 3377); GC: 14

@mkitti
Copy link
Member

mkitti commented Dec 30, 2024

This seems like an upstream issue. Could you try the HDF5 command line tools such as h5ls and h5dump? Otherwise, we'll have to write a small MWE in C.

https://github.com/hdfgroup/hdf5

@nhz2
Copy link
Member Author

nhz2 commented Dec 30, 2024

Yes, h5dump crashes on the files as well.

@mkitti
Copy link
Member

mkitti commented Dec 30, 2024

Just curious, what happens when you use the Julia port of the checksum code:
https://github.com/JuliaIO/JLD2.jl/blob/b8b0f9adaff35dbf39e0e4d58c164ff9f278f54b/src/Lookup3.jl#L112

@nhz2 nhz2 mentioned this issue Dec 30, 2024
Open
@nhz2
Copy link
Member Author

nhz2 commented Dec 30, 2024 

julia> using JLD2

julia> jldopen("t7.txt") do f
       f["test-data"]
       end
┌ Warning: File likely not written by JLD2. Skipping header verification.
└ @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/file_header.jl:21
ERROR: KeyError: key 0x0003 not found
Stacktrace:
  [1] getindex(h::Dict{UInt16, Tuple{Symbol, Symbol, Symbol, String}}, key::UInt16)
    @ Base ./dict.jl:477
  [2] get_decompressor(filters::JLD2.FilterPipeline)
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/compression.jl:110
  [3] read_compressed_array!(v::Vector{…}, f::JLD2.JLDFile{…}, rr::JLD2.SameRepr{…}, data_length::Int64, filters::JLD2.FilterPipeline)
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/compression.jl:221
  [4] read_array(f::JLD2.JLDFile{…}, dataspace::JLD2.ReadDataspace, rr::JLD2.ReadRepresentation, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{…})
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/datasets.jl:237
  [5] read_data(f::JLD2.JLDFile{…}, rr::Any, read_dataspace::Tuple{…}, attributes::Vector{…})
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/datasets.jl:102
  [6] read_data(f::JLD2.JLDFile{…}, dataspace::JLD2.ReadDataspace, dt::JLD2.H5Datatype, layout::JLD2.DataLayout, filters::JLD2.FilterPipeline, header_offset::JLD2.RelOffset, attributes::Vector{…})
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/datasets.jl:84
  [7] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/datasets.jl:48
  [8] getindex(g::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, name::String)
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/groups.jl:99
  [9] getindex
    @ ~/.julia/packages/JLD2/NKGUi/src/JLD2.jl:349 [inlined]
 [10] (::var"#1#2")(f::JLD2.JLDFile{JLD2.MmapIO})
    @ Main ./REPL[4]:2
 [11] jldopen(f::Function, args::String; kws::@Kwargs{})
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/loadsave.jl:4
 [12] jldopen(f::Function, args::String)
    @ JLD2 ~/.julia/packages/JLD2/NKGUi/src/loadsave.jl:1
 [13] top-level scope
    @ REPL[4]:1
Some type information was truncated. Use `show(err)` to see complete types.

@mkitti
Copy link
Member

mkitti commented Dec 30, 2024

I'm going to close this since this can only be addressed upstream:
HDFGroup/hdf5#5193

@mkitti mkitti closed this as completed Dec 30, 2024
@simonbyrne
Copy link
Collaborator

Is the segfault only with HDF5.jl?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@simonbyrne @mkitti @nhz2