Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract empty group dependencies for topological ordering. #766

Closed
wants to merge 1 commit into from

Conversation

nuclearsandwich
Copy link
Contributor

This is a fix for an issue introduced by #758. Now that conditional
dependencies are being evaluated and checked, an assert is triggering
during release jobs since group memberships are not being extracted.

However, if we were to extract true group memberships we could create a
discrepancy between the topological order according to ros_buildfarm and
the dependency tree used by bloom as bloom does not currently support
group dependencies.

Drel_reconfigure-jobs#762 is an example of the failure this is meant to mitigate.
Drel_reconfigure-jobs#761 used a modified config targeting the previous commit which is how I know #758 introduced the issue.

This is a fix for an issue introduced by #758. Now that conditional
dependencies are being evaluated and checked, an assert is triggering
during release jobs since group memberships are not being extracted.

However, if we were to extract true group memberships we could create a
discrepancy between the topological order according to ros_buildfarm and
the dependency tree used by bloom as bloom does not currently support
group dependencies.
@cottsay
Copy link
Member

cottsay commented Mar 13, 2020

Too bad CI didn't catch this :( sorry about that.

I've been experimenting with what it will take to "do groups right," and part of that is making sure that the topological ordering respects the groups. So since the groups can only make the topological order more strict, I'd rather see us just enumerate the dependencies correctly.

Here's an alternative I committed earlier, but hadn't opened a PR for: #767

@nuclearsandwich
Copy link
Contributor Author

My preference is for parity between bloom and release jobs because anything else will be a source of confusion. A scenario I'm thinking of is a package maintainer who forgot to patch an explicit dependency over their group depend, seeing the dependency in the buildfarm topology but not having the package installed.

With that being said I'll yield on that point if the other PR is preferred but I think it's trading smell for strife. I'd rather graduate to the approach in #767 when Bloom supports it.

@cottsay
Copy link
Member

cottsay commented Mar 13, 2020

I'm not proposing that we add group dependencies to the Jenkins job dependencies. I am proposing we enumerate the group dependencies properly and use them in topological ordering.

If/when I have a solution for groups that doesn't break existing workflows, we can add the groups as job dependencies.

Here is where the job dependencies are enumerated:

def get_direct_dependencies(pkg_name, cached_pkgs, pkg_names):
if pkg_name not in cached_pkgs:
return None
pkg = cached_pkgs[pkg_name]
# test dependencies are treated as build dependencies by bloom
# so we need them here to ensure that all dependencies are available
# before starting a build
depends = set([
d.name for d in (
pkg.buildtool_depends +
pkg.build_depends +
pkg.buildtool_export_depends +
pkg.build_export_depends +
pkg.exec_depends +
pkg.test_depends)
if d.name in pkg_names and
d.evaluated_condition is not False])
return depends

@nuclearsandwich
Copy link
Contributor Author

I'm not proposing that we add group dependencies to the Jenkins job dependencies.I am proposing we enumerate the group dependencies properly and use them in topological ordering.

Ah, I got it. In that case yes I'll take #767. Thanks for clarifying.

@nuclearsandwich
Copy link
Contributor Author

Closing in favor of #767.

@nuclearsandwich nuclearsandwich deleted the extract-empty-group-dependencies branch March 14, 2020 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants