catkin_make_isolated should provide rosbuild parallelity

[wjwwood]

I see no reason to not parallelize project builds of projects without dependencies between them. Code can be reused from rosmake or from rosinstall.

This might require changing the catkin_pkg API not just to return a topologically ordered list of packages, but a dependency graph (maybe without transitive dependencies)

[dirk-thomas]

This is not schedule to be implemented (assinged to milestone "untargeted"). Anyway patches implementing this feature are welcome.

[dirk-thomas]

Pass to @wjwwood.

[wjwwood]

Upgraded to a pull request.

[wjwwood]

This pull request is still a work in progress, I have chosen to rewrite the command from scratch, not reusing (by importing) anything used in the current catkin_make_isolated and catkin_make commands because of the significant restructuring of the flow control and logic. Much of the code is obviously based on or copied from the old implementations.

Some high level concepts I have been using:

• Command: single action to execute as part of a group of commands which can build a package
• Job: contains a list of Commands and meta data about how to build a package
  • extended to typed jobs (CatkinJob and CMakeJob)
• Executor: multiprocessing process which consumes Job's and executes each of the Job's Commands in order

Some things I had to address:

• Synchronization of I/O: Each executor generates events, which the main thread of execution process
• Capturing Color: The executors capture colored output from the subprocesses (on unix only for now)
• Interleaved I/O: The output has to remain readable with proper handling for multi line and carriage return output from multiple commands simultaneously
• Locking of Spaces: There is a system for commands to indicate which spaces (devel/install) it uses, so that they can be locked to prevent multiple jobs from writing the same file at the same time.
  • Currently only for the install space
  • May not be necessary

Currently I can build almost the entire hydro-desktop-full workspace on my mac, and the remaining build errors do not appear to be related to this command (they fail in catkin_make_isolated as well right now). I already found three instances of missing calls to include_directories in various packages using this command. I have mostly been doing unmerged devel space and no install.

This is what remains to be done from my point of view:

• Test install
• Test merged devel space
• Test isolated install space
• Add support for printing output of commands as contiguous blocks
• Fixup verbose/quiet behavior
• Test with DESTDIR
• Code review
• Finish API Documentation
• Refactor CLI interface
• Tutorials
• Colorize output and optimize whitespace for visibility

These are some additional TODO's, which would be nice to have, but not strictly necessary:

• Add checks for reusing build/devel spaces with different command options, i.e. warn when using the same devel space for merged and nonmerged devel on seprarte runs of the command
• Automated tests for all of the corner cases

[wjwwood]

@jonbinney d7b2b26

[wjwwood]

Ok, 86a89b2 seems to work for me too.

[wjwwood]

@dirk-thomas can you look at this commit 0548d50?

Does that make sense as a better default?

[dirk-thomas]

Yeah, that looks much more straight forward.

[wjwwood]

Ok, I address the comments and I removed the last os.remove as the file should not exist after os.rename anyways.

[wjwwood]

I am going to close this pull request in favor of the implementation in the new catkin_tools package as catkin build:

https://github.com/catkin/catkin_tools

I'll leave the branch around for people who might be using it.

[jbohren]

Thanks for the update, Will. First glance at the docs and this is looking pretty awesome!

[wjwwood]

Sure, I'm hoping to get this into a good state for Indigo.

[jbohren]

I just downloaded and installed it from source. I ran catkin build and said "woah" out loud.

[wjwwood]

Colors man, colors.

[tkruse]

well done