Pre splitting meshes may cause problems for simulations requiring geometric <- algebraic <- coupling ghosting

[lindsayad]

Bug Description

See #25618 for an example. If the mesh is pre-split without any knowledge about what elements must be geometrically ghosted, then the simulation using the pre-split mesh will be hosed.

Steps to Reproduce

Pre-split a mesh and then run a simulation involving something like periodic boundaries

Impact

Surprises users with usually confusing errors.

[lindsayad]

There are a couple ways we could combat this:

• do the requisite ghosting during the mesh splitting step
• recover the required ghosts using parallel communication in the simulation step

I'll ping @roystgnr for his thoughts on which he thinks would be wiser. The latter would be much more convenient for users who often want to use different inputs for the splitting and simulation

[lindsayad]

This issue has some conceptual links to #24140

[roystgnr]

"recover the required ghosts" would obviously be better, if we could do it.

The problem is that the current GhostingFunctor interface doesn't give any way for a functor subclass to say "these are the requirements under which we'd want a hypothetical element E to be ghosted on the processor owning element F", all it can do is answer "these are the already-ghosted elements {E} that should remain ghosted on the processor owning element F". So if we're getting rid of superfluous elements, or if we've got some set of elements {F} that we're moving from processor to processor, we're fine, but there's currently no way (other than serializing the entire mesh and then removing the stuff we shouldn't have serialized) to recover erroneously remoted elements.

So if "do the requisite ghosting during the mesh splitting step" is still at all an option then that's clearly the right short-run solution.

That's not a good long-run solution, but I'm not sure what the right way is to design an API that can do this in the long run. I could figure out how to sync very specific types of ghosting, using neighbor links and recursive communication if a functor is ghosting by stencil size, or octree queries if a functor is ghosting across periodic boundary condition offsets, or using parallel intersections of bounding volumes with octrees if a functor is ghosting by physical distance... but how do we put all that into one API? We could add a GhostingFunctor::recover() pure virtual and require subclasses to implement it, but the implementation is going to either be "a ton of tricky parallel code" or "spit out a warning and hope nobody really needed a recover for this functor in particular" in each case. My best idea so far is "hope that every single subclass's recovery requirements are a subset of the requirements I've already thought of", in which case we can just farm off to a non-virtual GhostingFunctor::recover_by_foo(param_bar) in each case, but that's not a very bright idea. In transient simulations getting decent performance would mean doing as few recoveries as we can get away with, e.g. requesting a larger geometric bounding volume than we need and only re-requesting if we find that a displaced mesh has ‖displacement‖_∞ of less than half the margin between "what we asked for" and "what we really need", and that makes the API design even trickier.