Support namespace packages

[mknorps]

Python namespace packages allow splitting the subpackages and modules within a single package across multiple separate distribution packages. See these references:

• Packaging namespace packages - from the main Python packaging guide
• PEP 420 - Implicit Namespace Packages
• What's a Python Namespace Package, and What's It For - Real Python

Currently, when we want to check a sub-package with FawltyDeps, we need to run the script in a sub-package directory.

Also, when FawltyDeps runs in a project that imports one or more namespace packages, it always reports that the imported package is unused. This means that effectively, you must exclude any imported package that provides an implicit namespace package, until FawltyDeps gains better support for namespace packages.

Proposed Solution

We would like to discover namespace packages automatically. Another todo here is to check namespace packages relation to "extras".

Background

This issue is related to:

• Different reporting behavior for "core" and "extra" dependencies #44 (loosely related)
• Roadmap, short- to medium-term #195
• Analyze PyPI packages with FawltyDeps #212
• boto3-stubs not detected as (naively) expected #437
• potentially other namespaceissues

History

Early history

This issue is a result of early design docs discussion between @tbagrel, @Nour-Mws and @jherland:

[Nour] We are only checking for first level packages (so no modules).
Distinguishing modules from packages when multiple packages exist in the namespace is a possible source of headache and deserves more discussion. See https://packaging.python.org/en/latest/guides/packaging-namespace-packages/

Checking that imported modules actually exist in the declared package is out of our scope but is a useful extension. And it seems to be possible with minimal extension to the current solution design but we haven't thought this through, so don't quote me on this :)

[Johan]
Even though FawltyDeps only targets the package level (not individual modules), I still think it would be useful to mention modules in this section and how they relate to packages. AFAIU, from FawltyDeps' POV, both packages and modules are "import names", but since FawltyDeps is only concerned with the package level, I guess it would have to either ignore module imports, or resolve them to their parent package?

2025 revival

Furthermore, @jherland and @obscurerichard discussed this in January 2025 with respect to https://github.com/freezingsaddles/freezing-web - which imports a namespace package freezing-model from https://github.com/freezingsaddles/freezing-model that exports 'freezing.model.*` implicit namespace packages.

@jherland wrote in more depth:

FawltyDeps only looks at the top-level name in any import statement.

The top-level name is AFAICS also the only thing that is reported by importlib_metadata , so I don't think FD is able to find/discern namespaces packages when walking the Python environment.

Thus, any/all undeclared imports will be reported using only the top-level part of their name.

Conversely, A package that provides only one part of the namespace will be assumed to "provide" the top-level name, and will thus appear to cover any import that refer to a different (sub)module within the namespace. To illustrate: since "freezing-model" appears to provide freezing, then as far as FD is concerned, if you import freezing.something_else , FawltyDeps will happily match this import against "freezing-model".

From FawltyDep's point of view this is no different than the case where two unrelated packages provide the same import name, even if their actual APIs are completely different or disjoint. FawltyDeps is merely happy to match the import name from the code to a provided import name in the package metadata.

For namespace packages, one could argue that we should try to drill deeper into the package, and match more than just the top-level name. I have the feeling that this has the potential to get really complex, really quickly...

@obscurerichard replied:

This is spot-on, but clearly this is not the norm for Python packages. This project does indeed use PEP 420 implicit namespace packages, thank you for giving me the right background on these to identify them properly.

But even when configured like this:

 [tool.fawltydeps.custom_mapping]
 # FIXME: What should this be? freezing-model brings in freezing.model.* classes... but this does not work:
 freezing-model = ["freezing"]
FawltyDeps still yields:
These dependencies appear to be unused (i.e. not imported):
- 'freezing-model'

See this GitHub Actions run for the failure related to commit 0c95ccc

@jherland replied:

Yeah, that unused report looks obviously invalid at first glance.

[jherland]

https://github.com/tweag/chainsail/blob/3e6af8048a2aa0278a59e7c401fa70523f7951fa/app/controller/pyproject.toml#L17 is another example of (mono)repos installing package dependencies by local path.

[mknorps]

We will go back to that once we have an example of a namespace package and the expected behavior.