"HYPHYMPI" failing but "hyphy" working

[lrschaff]

Hello HyPhy team,

When using RELAX on 2.5.68, the "hyphy" (non MPI) version works perfectly fine on my dataset. However when using the MPI version, "HYPHYMPI" it seems to fail at "fitting the general descriptive model."

Running with:

$ srun HYPHYMPI RELAX --alignment ./gene_600.fa --tree ./536mammals_ML_tree.nwk --test Foreground --models All --srv Branch-site --multiple-hit "Double+Triple" --keepident true

And getting the error:

[...]
### Fitting the general descriptive (separate k per branch) model
Grid result 7 = -213986.1435143871 (best = -192163.4429078236)Error:

Master node received an error:Operation '^' is not implemented/defined for a AssociativeList
    While computing: utility.Extend(^(mpi.gJHXkMzL.arguments[2]),mpi.gJHXkMzL.result)

Function call stack
1 :  [namespace = mpi.gJHXkMzL] utility.Extend(^(arguments[2]),result);

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
2 :  [namespace = mpi.aux.EaU_Gajr] Call((queue[from])[utility.getGlobalValue("terms.mpi.callback")],from,Eval(result),(queue[from])[utility.getGlobalValue("terms.mpi.arguments")]);

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
3 :  [namespace = mpi.uWUXdajl] node=aux._handle_receieve(queue);

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
4 :  [namespace = mpi.jfvuBvnJ] mpi.QueueComplete(queue);

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
5 :  [namespace = DhYdAqBM] grid_results=mpi.ComputeOnGridSetValues(&likelihoodFunction,run_options[utility.getGlobalValue("terms.search_grid")],{0:model_objects,1:initial_values,2:run_options[utility.getGlobalValue("terms.run_options.proportional_branch_length_scaler")]},"mpi.ComputeOnGrid.SimpleEvaluatorWithValues","mpi.ComputeOnGrid.ResultHandler");

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
6 :  relax.grid_search.results=estimators.FitLF(relax.filter_names,relax.trees,relax.ge_model_map,relax.final_partitioned_mg_results,relax.model_object_map,{"retain-lf-object":TRUE,terms.run_options.apply_user_constraints:"relax.init.k",terms.run_options.proportional_branch_length_scaler:relax.nm.bl_scalers,terms.run_options.optimization_settings:{"OPTIMIZATION_METHOD":"nedler-mead","MAXIMUM_OPTIMIZATION_ITERATIONS":500,"OPTIMIZATION_PRECISION":relax.nm.precision},terms.search_grid:relax.initial_grid,terms.search_restarts:relax.N.initial_guesses,terms.run_options.optimization_log:relax.optimization_log_file(".GE-1-log.json")});

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
[...]

Any insight on how to fix this error will be greatly appreciated.

Thank you!

Lillie

[spond]

Dear @lrschaff,

Looks like a bug with something not being properly exported to compute nodes by MPI. I'll mark it as such and investigate. Thanks for reporting!

Best,
Sergei

[lrschaff]

Thank you for looking into it!

Best,

Lillie