Maximum filename length

[jokedurnez]

This code runs without problem:
(note that you can reproduce the full analysis if you change the name of the nifti in the first line to one that exists on a local machine)

copes = ['/oak/stanford/groups/russpold/data/ds000030_R1.0.3_analysis_0.4.4_group/sub-50073/stopsignal.feat/stats/cope1.nii.gz']*2
copemerge = Node(interface=fsl.Merge(dimension='t',in_files=copes),name='copemerge')
median = Node(interface=fsl.maths.MaxImage(dimension="T"),name="max")
CNPgroup = Workflow(name='cnp_group')
CNPgroup.connect([(copemerge,median,[('merged_file','in_file')])])
CNPgroup.run()

When I make the first two lines into an iterable, like the following, the problem appears.

copes = [['/oak/stanford/groups/russpold/data/ds000030_R1.0.3_analysis_0.4.4_group/sub-50073/stopsignal.feat/stats/cope1.nii.gz']*2]*2
copemerge = Node(interface=fsl.Merge(dimension='t'),name='copemerge')
copemerge.iterables = ('in_files',copes)
(next lines equal to the ones in the previous code snippet)

This is the error trace:

170601-16:26:25,882 workflow ERROR:
         ['Node copemerge.aI.a0 failed to run on host sh-5-36.local.']
170601-16:26:25,883 workflow INFO:
         Saving crash info to /home/jdurnez/crash-20170601-162625-jdurnez-copemerge.aI.a0-c7647a64-69ca-49a2-8e44-50c99755d74e.pklz
170601-16:26:25,883 workflow INFO:
         Traceback (most recent call last):
  File "/scratch/PI/russpold/crn-agave/anaconda/lib/python2.7/site-packages/nipype/pipeline/plugins/linear.py", line 39, in run
    node.run(updatehash=updatehash)
  File "/scratch/PI/russpold/crn-agave/anaconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 387, in run
    outdir = make_output_dir(outdir)
  File "/scratch/PI/russpold/crn-agave/anaconda/lib/python2.7/site-packages/nipype/pipeline/engine/utils.py", line 1076, in make_output_dir
    raise OSError('Could not create %s'%outdir)

OSError: Could not create /local-scratch/jdurnez/15385190/tmpKhcTIF/cnp_group/_in_files_..oak..stanford..groups..russpold..data..ds000030_R1.0.3_analysis_0.4.4_group..sub-50073..stopsign
al.feat..stats..cope1.nii.gz...oak..stanford..groups..russpold..data..ds000030_R1.0.3_analysis_0.4.4_group..sub-50073..stopsignal.feat..stats..cope1.nii.gz/copemerge

I think because this outputdir exceeds the maximum number of characters in a file/folder name.
See the following, where I cut off the following directory name (not full path) at 255 characters in the second try:

[jdurnez@sh-5-36 ~]$ getconf NAME_MAX /local-scratch
255
[jdurnez@sh-5-36 ~]$ mkdir /local-scratch/jdurnez/15385190/tmp8WFMe1/cnp_group/_in_files_..oak..stanford..groups..russpold..data..ds000030_R1.0.3_analysis_0.4.4_group..sub50073..stopsignal.feat..stats..cope1.nii.gz...oak..stanford..groups..russpold..data..ds000030_R1.0.3_analysis_0.4.4_group..sub50073..stopsignal.feat..stats..cope1.nii.gz
mkdir: cannot create directory `/local-scratch/jdurnez/15385190/tmp8WFMe1/cnp_group/_in_files_..oak..stanford..groups..russpold..data..ds000030_R1.0.3_analysis_0.4.4_group..sub50073..stopsignal.feat..stats..cope1.nii.gz...oak..stanford..groups..russpold..data..ds000030_R1.0.3_analysis_0.4.4_group..sub50073..stopsignal.feat..stats..cope1.nii.gz': File name too long
[jdurnez@sh-5-36 ~]$ mkdir /localscratch/jdurnez/15385190/tmp8WFMe1/cnp_group/_in_files_..oak..stanford..groups..russpold..data..ds000030_R1.0.3_analysis_0.4.4_group..sub50073..stopsignal.feat..stats..cope1.nii.gz...oak..stanford..groups..russpold..data..ds000030_R1.0.3_analysis_0.4.4_group..sub-50073..stopsignal.feat..stats..cope
[jdurnez@sh-5-36 ~]$

I'm going to solve it by manually looping instead of using the iterables, but I thought you'd want to know. I'm not going to try to solve this, looks too deep in the nipype code, but maybe it would be better to use a random string for the iter folder instead?

[satra]

@jokedurnez - do you set this config option to false by any chance?

config['execution']['parameterize_dirs']

the default value is true and by default it should have turned that long parameterization to a hash.

[mgxd]

@satra that option should be set to False in order to convert iterables > 32 characters into a hash. Right now the default is set to True, but turning this option off in the base config would be a good idea.

[satra]

@mgxd - thank you. @jokedurnez - i stand corrected. to prevent the error you are receiving you can do:

workflow_variable.config['execution']['parameterize_dirs'] = False

[jokedurnez]

Cool, thanks !

[jokedurnez]

btw this didn't work for me.

[satra]

@jokedurnez - do you have node object that crashes? if so can you please inspect it to see if the config is set correctly?

from nipype.utils.filemanip import loadpkl

node = loadpkl('_node.pklz')
node.config['execution']['parameterize_dirs']

was this run with current nipype? (although that particular feature has been around for a while).

[mgxd]

@jokedurnez is this still a problem? If so, can you try clearing your working directory?

[jokedurnez]

I changed my pipeline so that it loops over the iterables to circumvent the problem. I'll try the proposed solution later to verify.

[mgxd]

closing - feel free to reopen

[MontrealSergiy]

The section about config file and parameterize_dirs disappeared from the recent nipype documentation at https://nipype.readthedocs.io. Is it deprecated now?

[effigies]

I think there was a push to deduplicate with https://miykael.github.io/nipype_tutorial/. See: https://miykael.github.io/nipype_tutorial/notebooks/basic_execution_configuration.html

[erikglee]

Would you ever consider setting parameterize_dirs to false as a default in nipype? We are working on integrating MRIQC into CBRAIN (http://www.cbrain.ca/), and have run into an issue with file name lengths. While the solution posted above should hopefully work for this version of MRIQC, having this changed in nipype's defaults would make it so we don't have to implement this change in future containerized versions of MRIQC + limit the likelihood of this happening for other pipelines we work with. I would understand if there are negative repercussions of this change that would outweigh the potential utility. Thanks!