Coverage 6.0 reports less than 100% coverage for Tomli

[hukkin]

Describe the bug
Hello, Tomli maintainer here!

Coverage version 6.0 seems to report less than 100% coverage for tomli. Coverage versions 5.x correctly report 100% coverage. Note that version 6.0 is the first one that depends on Tomli. I expect this may be relevant.

I definitely do not expect you to debug this for me 😄 but given how well you know coverage, perhaps you can tell what's wrong or what I'm doing wrong off the top of your head.

To Reproduce

1. What version of Python are you using?
   • Python 3.8.10
2. What version of coverage.py are you using? The output of coverage debug sys is helpful.
   • coverage-6.0
3. What versions of what packages do you have installed? The output of pip freeze is helpful.

   • argcomplete==1.12.3
     attrs==19.3.0
     backcall==0.1.0
     backports.entry-points-selectable==1.1.0
     beautifulsoup4==4.8.2
     blinker==1.4
     Brlapi==0.7.0
     cached-property==1.5.1
     certifi==2019.11.28
     cfgv==3.3.0
     chardet==3.0.4
     chrome-gnome-shell==0.0.0
     Click==7.0
     colorama==0.4.3
     command-not-found==0.3
     construct==2.8.16
     coverage==6.0
     cryptography==2.8
     cupshelpers==1.0
     dbus-python==1.2.16
     decorator==4.4.2
     defer==1.0.6
     distlib==0.3.2
     distro==1.4.0
     distro-info===0.23ubuntu1
     docker==4.1.0
     docker-compose==1.25.0
     dockerpty==0.4.1
     docopt==0.6.2
     ecdsa==0.15
     entrypoints==0.3
     filelock==3.0.12
     gpg===1.13.1-unknown
     html5lib==1.0.1
     httpie==1.0.3
     httplib2==0.14.0
     identify==2.2.11
     idna==2.8
     importlib-metadata==1.5.0
     iniconfig==1.1.1
     ipython==7.13.0
     ipython-genutils==0.2.0
     jedi==0.15.2
     jsonschema==3.2.0
     keyring==18.0.1
     language-selector==0.1
     launchpadlib==1.10.13
     lazr.restfulclient==0.14.2
     lazr.uri==1.0.3
     libusb1==1.7
     louis==3.12.0
     lxml==4.5.0
     macaroonbakery==1.3.1
     Mako==1.1.0
     MarkupSafe==1.1.0
     meld==3.20.2
     mnemonic==0.18
     more-itertools==4.2.0
     netifaces==0.10.4
     nodeenv==1.6.0
     oauthlib==3.1.0
     olefile==0.46
     packaging==21.0
     parso==0.5.2
     pbkdf2==1.3
     pdfarranger==1.4.2
     pexpect==4.6.0
     pickleshare==0.7.5
     pikepdf==1.10.3+dfsg
     Pillow==7.0.0
     pipx==0.12.3.1
     platformdirs==2.0.2
     pluggy==0.13.1
     pre-commit==2.13.0
     prompt-toolkit==2.0.10
     protobuf==3.6.1
     psutil==5.5.1
     py==1.10.0
     pycairo==1.16.2
     pycups==1.9.73
     Pygments==2.3.1
     PyGObject==3.36.0
     pyinotify==0.9.6
     PyJWT==1.7.1
     pymacaroons==0.13.0
     PyNaCl==1.3.0
     pyparsing==2.4.7
     pyRFC3339==1.1
     pyrsistent==0.15.5
     pytest==6.2.5
     pytest-cov==3.0.0
     python-apt==2.0.0+ubuntu0.20.4.6
     python-dateutil==2.8.2
     python-debian===0.1.36ubuntu1
     python-pam==1.8.4
     python-xapp==1.8.1
     python-xlib==0.23
     pytz==2019.3
     pyxdg==0.26
     PyYAML==5.3.1
     requests==2.22.0
     requests-unixsocket==0.2.0
     SecretStorage==2.3.1
     setproctitle==1.1.10
     simplejson==3.16.0
     six==1.14.0
     soupsieve==1.9.5
     speedtest-cli==2.1.2
     systemd-python==234
     texttable==1.6.2
     tinycss==0.4
     toml==0.10.2
     tomli==1.2.1
     traitlets==4.3.3
     trezor==0.11.6
     typing-extensions==3.7.4.1
     ubuntu-advantage-tools==27.2
     ufw==0.36
     unattended-upgrades==0.1
     urllib3==1.25.8
     usb-creator==0.3.7
     userpath==1.7.0
     vboxapi==1.0
     virtualenv==20.6.0
     wadllib==1.3.3
     wcwidth==0.1.8
     webencodings==0.5.1
     websocket-client==0.53.0
     zipp==1.0.0
     

4. What code are you running? Give us a specific commit of a specific repo that we can check out.
   • https://github.com/hukkin/tomli/tree/c5045f47782341190517ae62307be2bd14e13d2d
5. What commands did you run?

   • foo@bar:~$ cd /tmp/
     foo@bar:/tmp$ git clone https://github.com/hukkin/tomli.git
     Cloning into 'tomli'...
     remote: Enumerating objects: 1885, done.
     remote: Counting objects: 100% (510/510), done.
     remote: Compressing objects: 100% (260/260), done.
     remote: Total 1885 (delta 313), reused 335 (delta 229), pack-reused 1375
     Receiving objects: 100% (1885/1885), 310.30 KiB | 1.85 MiB/s, done.
     Resolving deltas: 100% (775/775), done.
     foo@bar:/tmp$ cd tomli/
     foo@bar:/tmp/tomli$ python3 -m venv venv
     foo@bar:/tmp/tomli$ . venv/bin/activate
     (venv) foo@bar:/tmp/tomli$ pip install . pytest-cov python-dateutil
     Processing /tmp/tomli
       Installing build dependencies ... done
       Getting requirements to build wheel ... done
         Preparing wheel metadata ... done
     Collecting pytest-cov
       Using cached pytest_cov-3.0.0-py3-none-any.whl (20 kB)
     Collecting python-dateutil
       Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     Collecting pytest>=4.6
       Using cached pytest-6.2.5-py3-none-any.whl (280 kB)
     Collecting coverage[toml]>=5.2.1
       Using cached coverage-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (255 kB)
     Collecting six>=1.5
       Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
     Collecting py>=1.8.2
       Using cached py-1.10.0-py2.py3-none-any.whl (97 kB)
     Collecting iniconfig
       Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
     Collecting attrs>=19.2.0
       Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
     Collecting packaging
       Using cached packaging-21.0-py3-none-any.whl (40 kB)
     Collecting toml
       Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
     Collecting pluggy<2.0,>=0.12
       Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
     Collecting pyparsing>=2.0.2
       Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     Building wheels for collected packages: tomli
       Building wheel for tomli (PEP 517) ... done
       Created wheel for tomli: filename=tomli-1.2.1-py3-none-any.whl size=11869 sha256=9771a834d2d82bfb701afd93a640bbd093c9ac5d7896425e2b03ac0a6d8dcbc6
       Stored in directory: /tmp/pip-ephem-wheel-cache-4_qivjx_/wheels/2a/be/a2/c2e2f410105e5a607018317cab46427d50645c558623d032be
     Successfully built tomli
     Installing collected packages: py, iniconfig, attrs, pyparsing, packaging, toml, pluggy, pytest, tomli, coverage, pytest-cov, six, python-dateutil
     Successfully installed attrs-21.2.0 coverage-6.0 iniconfig-1.1.1 packaging-21.0 pluggy-1.0.0 py-1.10.0 pyparsing-2.4.7 pytest-6.2.5 pytest-cov-3.0.0 python-dateutil-2.8.2 six-1.16.0 toml-0.10.2 tomli-1.2.1
     (venv) foo@bar:/tmp/tomli$ python -m pytest --cov --cov-fail-under=100
     ========================================== test session starts ==========================================
     platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
     rootdir: /tmp/tomli, configfile: pyproject.toml
     plugins: cov-3.0.0
     collected 466 items                                                                                     
     
     tests/test_error.py ...                                                                           [  0%]
     tests/test_extras.py ............................................................................ [ 16%]
     ................................................................................................. [ 37%]
     ...............................................................................................   [ 58%]
     tests/test_flags.py .                                                                             [ 58%]
     tests/test_for_profiler.py .                                                                      [ 58%]
     tests/test_misc.py ....                                                                           [ 59%]
     tests/test_toml_compliance.py .............................................................x.x... [ 73%]
     ................................................................................................. [ 94%]
     .........................                                                                         [100%]
     
     ---------- coverage: platform linux, python 3.8.10-final-0 -----------
     Name               Stmts   Miss Branch BrPart  Cover
     ----------------------------------------------------
     tomli/_parser.py     452     60    178      0    90%
     tomli/_re.py          33     13      8      0    68%
     ----------------------------------------------------
     TOTAL                485     73    186      0    89%
     
     FAIL Required test coverage of 100% not reached. Total coverage: 88.82%
     
     ==================================== 464 passed, 2 xfailed in 1.34s =====================================

Expected behavior
Expected 100% coverage (as reported by coverage==5.*) but got 88%.

[nedbat]

Worlds collide! This is definitely because coverage is using tomli now. The missing lines are all of the top-level statements in _parser.py and _re.py, presumably because they are imported by coverage.py before coverage measurement has started. I'll have to think about what we can do about that...

[hukkin]

Thanks for the quick response!

I'll have to think about what we can do about that...

Awesome! But don't worry/think too much about it. If this is something that affects exactly one project and one maintainer on the entire planet, then I think it's perfectly reasonable to decide that coverage does not support measuring coverage of its own dependencies.

[nedbat]

Well, full disclosure, when I said "what we can do about that", there was a strong possibility that the answer would be something you would do in the tomli repo :)

[nedbat]

This could be the fix: #1230

[hukkin]

Thank you! Can confirm that I do get 100% coverage after installing your branch with

pip install git+https://github.com/nedbat/coveragepy.git@nedbat/fix-tomli-tests-1228

[nedbat]

Great! I've made the change for the next release. You can use sha 613446c in the meantime.

[nedbat]

This is now released as part of coverage 6.0.1.