Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to provide their own dependency to imports mapping #260

Closed
Tracked by #256
Nour-Mws opened this issue Mar 22, 2023 · 5 comments · Fixed by #306
Closed
Tracked by #256

Allow users to provide their own dependency to imports mapping #260

Nour-Mws opened this issue Mar 22, 2023 · 5 comments · Fixed by #306
Assignees
Labels
P2 major: an upcoming release

Comments

@Nour-Mws
Copy link
Collaborator

Check #256 to see how this fits into the overall mapping strategy.

@Nour-Mws Nour-Mws added this to the Extend mapping strategy milestone Mar 22, 2023
@Nour-Mws Nour-Mws added the P2 major: an upcoming release label Mar 23, 2023
@jherland
Copy link
Member

Remember to have a look at other tools that provide a way for users to define their own package -> import mapping, e.g. Pants.

@mknorps
Copy link
Collaborator

mknorps commented Mar 27, 2023

How Pants handles that:

  1. Default mapping is the identity mapping with normalized name (- -> _)
  2. For dependencies of type stub, they are stripped of types and stubs prefixes and suffixes, so "types-requests gives type stubs for the module requests"
  3. User may provide mapping in the build file, like:
    python_requirements(
        name="reqs",
        module_mapping={"my_distribution": ["custom_module"]},
    )
  4. There is a static list of standard mappings which is checked in the dependencies inference.

Pigar allows to link a local mapping database, but it is more a form of a cache than what a user may conveniently provide.

@mknorps
Copy link
Collaborator

mknorps commented Mar 28, 2023

Creosote does not have an option for user-defined mapping.

Deptry also does not have this option and relies heavily on top-level.txt mapping.

@mknorps
Copy link
Collaborator

mknorps commented Mar 28, 2023

We may use TOML format for mappings.toml. It gives a user option to comment is readable and easily editable:

  apache-airflow = ["airflow"]
  atlassian-python-api = ["atlassian"]
  attrs = ["attr", "attrs"]

@jherland
Copy link
Member

I agree that TOML is a good format for this mapping, but I wonder if it has to be a different file than pyproject.toml? By keeping it in pyproject.toml, we:

  • don't have to parse any more files than we already do
  • don't have to document this new file to the user, instead their mapping can be defined right next to their other FD configuration in a new [tool.fawltydeps.custom_mapping] section.
  • can rely on the existing deserialization provided by pydantic to support passing the same mapping via the environment for free, that is, the pyproject.toml section:
[tool.fawltydeps.custom_mapping]
apache-airflow = ["airflow"]
atlassian-python-api = ["atlassian"]
attrs = ["attr", "attrs"]

can also be expressed as a similar JSON-formatted environment variable:

fawltydeps_custom_mapping='{ \
  "apache-airflow": ["airflow"], \
  "atlassian-python-api": ["atlassian"], \
  "attrs": ["attr", "attrs"] \
}'

and both can end up in our Settings object as a dict:

settings.custom_mapping: Dict[str, Set[str]] = {
    "apache-airflow": {"airflow"},
    "atlassian-python-api": {"atlassian"},
    "attrs": {"attr", "attrs"},
}

This can also be achieved with the mapping stored in a separate file, but I don't yet see the advantages of keeping it separate?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 major: an upcoming release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants