Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nc_create("file://foo.zarr#mode=nczarr,noxarray", 0, &ncid) consumes all memory #2573

Closed
opoplawski opened this issue Dec 16, 2022 · 4 comments · Fixed by #2574
Closed

nc_create("file://foo.zarr#mode=nczarr,noxarray", 0, &ncid) consumes all memory #2573

opoplawski opened this issue Dec 16, 2022 · 4 comments · Fixed by #2574

Comments

@opoplawski
Copy link
Contributor

We are seeing this with netcdf 4.9.0 and current git main head on Fedora Rawhide. The following is a simple reproducer:

#include <netcdf.h>

int main(int argc, char **argv) {
   int ncid;
   nc_create("file://foo.zarr#mode=nczarr,noxarray", 0, &ncid);
}

This is derived from a test in python-xarray - an issue was initially raised there: pydata/xarray#6854
@opoplawski
Copy link
Contributor Author

It seems to get stuck in an infinite allocation/copy loop. backtrace:

#0  _int_malloc (av=av@entry=0x7ffff7bf6c80 <main_arena>, bytes=bytes@entry=7) at malloc.c:4368
#1  0x00007ffff7abe392 in __GI___libc_malloc (bytes=bytes@entry=7) at malloc.c:3297
#2  0x00007ffff7ac451e in __GI___strdup (s=s@entry=0x45bab0 "nczarr") at strdup.c:42
#3  0x00007ffff7c4edfc in processinferences (fraglenv=0x45bba0) at ../libdispatch/../../libdispatch/dinfermodel.c:526
#4  NC_infermodel (path=0x45b680 "file://foo.zarr#mode=nczarr,noxarray", omodep=<optimized out>, iscreate=1, useparallel=0, params=0x0, model=0x7fffffffd800, newpathp=0x7fffffffd7f0)
    at ../libdispatch/../../libdispatch/dinfermodel.c:852
#5  0x00007ffff7c42c66 in NC_create (path0=<optimized out>, cmode=<optimized out>, initialsz=0, basepe=basepe@entry=0, chunksizehintp=0x0, useparallel=useparallel@entry=0, parameters=0x0, ncidp=0x7fffffffd88c)
    at ../libdispatch/../../libdispatch/dfile.c:1866
#6  0x00007ffff7c42e69 in nc__create (path=<optimized out>, cmode=<optimized out>, initialsz=<optimized out>, chunksizehintp=<optimized out>, ncidp=<optimized out>) at ../libdispatch/../../libdispatch/dfile.c:469

@opoplawski
Copy link
Contributor Author

Pointer to the python-xarray test: https://github.com/pydata/xarray/blob/main/xarray/tests/test_backends.py#L5733

@DennisHeimbigner
Copy link
Collaborator

ok, I have a fix.But it has revealed an odd HDF5 error that I need to track down.

DennisHeimbigner added a commit to DennisHeimbigner/netcdf-c that referenced this issue Dec 18, 2022
@DennisHeimbigner
Fix infinite loop in file inferencing
a03bb5e 
re: Issue Unidata#2573

The file type inferencer in libdispatch/dinference.c has a simple
forward inference mechanism so that the occurrence of certain mode
values in a URL fragment implies inclusion of additional mode values.
This kind of inference is notorious for leading to cycles if not
careful. Unfortunately, this occurred in the one in dinference.c.

This was fixed by providing a more complicated, but more reliable inference
mechanism.

## Misc. Other Changes
* Found and fixed a couple of memory leaks.
* There is a recent problem in building HDF4 support on github actions. Fixed by using the internal HDF4 xdr capability.
* Some filter-related code was not being properly ifdef'd with ENABLE_NCZARRA_FILTERS.
@DennisHeimbigner DennisHeimbigner mentioned this issue Dec 18, 2022
Merged
@DennisHeimbigner
Copy link
Collaborator

Fixed by PR #2574

@QuLogic QuLogic mentioned this issue May 8, 2023
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix infinite loop in file inferencing DennisHeimbigner/netcdf-c
2 participants
@opoplawski @DennisHeimbigner