Add CI jobs

[cottsay]

TODO list for this feature:

• Workspace bringup
• Colcon build and test
• Docker invalidation based on
  • Changes in the repos file
  • Changes in the rosdep index
  • Changes in the base OS' package versions
• Output artifact for use as underlay for subsequent job
• ccache
• Jenkins parameters
• Lots of other ros_buildfarm_config values (priority, etc)
• Scheduling
• Overlay job
• Documentation
  • Build file / Configuration - configuration_options.rst
  • Usage - ci_jobs.rst
• Catkin support
• Package selection
• Non-Jenkins / Script support
• Travis CI entry invoking the script version of the job
• Solve the installed package enumeration issue in Colcon (remove the hacky workaround)
  This may not be as complicated as I thought - at least some of the issues I was trying to work around are actually package bugs that can be fixed.
• Support for overlay job on the script version of the job (and subsequent Travis entry)
• Solve or work around the locals vs globals issue in template expansion.
  This is needed to support referencing other variables in generator expressions.
• Explore concurrency challenges on workers
  After discussing this, I think there will be some work to do here.

[dirk-thomas]

As mentioned in #575 (comment) CI builds should work with colcon as well as catkin_make_isolated (depending on a configuration flag) - similar to the devel jobs. As a consequence several names (files, directories, functions) shouldn't contain colcon in. E.g. the workspace should be named just ws as it is for all the other job types. Also why is colcon_build.py necessary? What does it do beyond the existing build_and_install / build_and_test scripts? Are there any other parts where the existing parts for devel jobs can be reused instead of creating new ones for CI builds?

Please also create a .rst file for the new CI build file documenting the options. Looking at the code build_ignore seems to be a relative path. Using just the package name might be easier.

Please also create the "script" versions of the script in order to enable running the same build locally. This should then also be used in the Travis file to test do some CI builds ensuring it works.

[cottsay]

Thanks @dirk-thomas - I updated the TODO list above per your feedback. Please adjust as necessary.

[cottsay]

I believe this is ready for more eyes now. Some things I know aren't working yet:

• More than 2 workspaces (more than 1 underlay)
• Installing ROS 1 prerequisites for building the ros1_bridge
• RTI Connext - I don't yet have the special sauce wired in for leveraging the environment scripts

[nuclearsandwich]

Only got as far as the documentation so far. But I figured I'd do this in chunks given the size.

[nuclearsandwich]

Make sure this commit doesn't get merged.

[nuclearsandwich]

I'm marking this as an approval because I think the next step for these changes requires integration with the rest of the farm and doing that with a long-running branch deploy is just going to be a headache (I'm still recovering from the long running deploy of #587).

I've raised a few questions, left a few comments about iterative workflow for future changes. But in general I think this is feature is ready-for-use.

Building off of @cottsay's show & tell a couple weeks back it would be great to get a little doc together that enumerates situations where the CI jobs can be used for quick iteration before cross-platform CI on Crystal-targeted stuff right now but we should also spin up Dashing or some evergreen master distro codename to get the benefits here for mainline dev.

[nuclearsandwich]

What makes this necessary where it wasn't before? Alternatively, what separate issue does this fix for all dev/pr/ci jobs?

[cottsay]

I hit this pretty early on, but it must boil down to the order in which things are installed. IIRC, I was consistently missing compiler symlinks when certain subsets of packages were built.

Add this specific re-linking resolved the issue, but we could drop it and see what breaks. If we get weird missing compiler issues, we know what the fix might look like...

[cottsay]

Reminder to myself to change this after ros2/ros_buildfarm_config#17 is merged.

[cottsay]

Thanks @nuclearsandwich and @dirk-thomas for your feedback thus far. I've incorporated it and merged a series of spin-off PRs, and this PR is now ready to be reviewed once again. It's still pretty bug, but is much more focused. I would appreciate further feedback when you have some time.

[dirk-thomas]

Beside a few inline comments my main feedback is about how the CI jobs are being configured. Atm a single CI build file results in a pair of jobs - a "nightly" one (actually depending on the configured schedule - could be hourly) and a "overlay" one which is only triggered manually. The fact that both jobs are coupled with each other through a single build file makes it less reusable.

Just to be clear this doesn't need to be addressed in order to merge this PR but I think the following should be done in a follow up PR.

Instead of coupling these two job types I would rather keep them separate - once CI build file should only generate one job. The configuration should allow to specify the differences of these two jobs:

• for the nightly e.g. specify a schedule
• for the overlay e.g. specify which artifacts should be used for the underlay(s)

That will allow to configure CI jobs in a more flexible fashion. Just a couple of examples:

• a "nightly" job building a set of packages - but no corresponding "overlay" job (the equivalent of the nightly and packaging jobs on ci.ros2.org)
• multiple "overlay jobs with different setting using the artifact from the same "nightly" job

[cottsay]

I moved all of the arguments used more than once to the common location. The ci task generation now differs from devel task generation in terms of argument format, but it matches the release task generation, which was the only job type that used positional arguments for task generations prior to this change.

I really like the idea of designing a system that can generate any number of CI jobs for any purpose, which can run on any schedule, and I think it is totally doable. Let's follow-up with a change that makes that happen - it should be possible to merge that work without affecting the "experience" presented in this PR.

Changed

[dirk-thomas]

Let's follow-up with a change that makes that happen - it should be possible to merge that work without affecting the "experience" presented in this PR.

Once CI is green (and this PR only adding new code but not really changing any existing code) I am fine to merge this (after reverting the custom branch names mentioned above - which will fail CI).

After that please follow up with a PR replacing the current fixed tuple of underlay / overlay jobs with the more flexible individual configuration.

[cottsay]

I rebased and squashed the WIP commits, and removed the branch name and config override. CI will fail in this PR (and in master once merged) until ros2/ros_buildfarm_config#17 is merged.

[nuclearsandwich]

Congrats on a long haul merge. 🎆