Running npm react-scripts test ignores the --coverage option

[zjullion]

Describe the bug

As per #7164, running npm react-scripts test ignores the --coverage option for jest - that is, jest will pass even if coverage thresholds are not reached. Note that this is almost certainly NOT an issue with jest - it is an issue with create-react-app's configuration of jest.

Did you try recovering your dependencies?

I've tried this with both yarn version 1.16.0 and 1.17.3 (thanks for reminding me to update yarn) - and I also removed node_modules and yarn.lock and then re-installed, which didn't change anything.

Which terms did you search for in User Guide?

I didn't search the user guide at all, I searched open + closed issues in this repo, and found #7164.

Environment

Environment Info:

System:
OS: macOS High Sierra 10.13.6
CPU: (8) x64 Intel(R) Core(TM) i7-4771 CPU @ 3.50GHz
Binaries:
Node: 8.11.0 - ~/.nvm/versions/node/v8.11.0/bin/node
Yarn: 1.17.3 - /usr/local/bin/yarn
npm: 5.6.0 - ~/.nvm/versions/node/v8.11.0/bin/npm
Browsers:
Chrome: 75.0.3770.142
Firefox: Not Found
Safari: 12.1.1
npmPackages:
react: ^16.8.6 => 16.8.6
react-dom: ^16.8.6 => 16.8.6
react-scripts: 3.0.1 => 3.0.1
npmGlobalPackages:
create-react-app: Not Found

Steps to reproduce

See $7164 for a trivial reproduction of the issue.

Expected behavior

The --coverage flag should fail tests when the coverage threshold is not met.

Actual behavior

The --coverage flag does not fail tests when the coverage threshold is not met.

More detailed explanation

Our repo is somewhat complex - we're using react on the front-end (tested with jest) and node on our back-end (tested with the same jest binary). Here are the two commands we run in package.json:

"test-frontend": "react-scripts test --coverage --watchAll=false"
"test-backend": "jest --config=amplify/jest.config.js --coverage --watchAll=false"

Here's some relevant parts of package.json:

"jest": {
    "snapshotSerializers": [
      "enzyme-to-json/serializer"
    ],
    "collectCoverageFrom": [
      // Snipped
    ],
    "coverageThreshold": {
      "global": {
        "statements": 98.38,
        "branches": 98.38,
        "functions": 96.36,
        "lines": 98.58
      }
    }
  },

And here's jest.config.js:

module.exports = {
  collectCoverageFrom: [
    // Snipped
  ],
  coverageThreshold: {
    global: {
      statements: 100,
      branches: 91.67,
      functions: 100,
      lines: 100
    }
  },
  roots: [
    'backend/function',
    'backend/api/reactkart/customTransformers'
  ],
  testEnvironment: 'node'
}

For the test-backend command, jest will fail if the coverage thresholds are not met. However, jest still passes for the test-frontend command if thresholds are not met. I'm fairly confident this means the issues lies with create-react-app and not jest, as both commands use the same jest binary and have fairly minimal configuration.

[bugzpodder]

does CI=true or --bail help?

[zjullion]

CI=true does help, but --bail does not:

However, using CI=true removes all the coloured highlighting (not the end of the world, but unfortunate), and I don't believe it's documented anywhere that you must set CI=true in order to get coverage to work...

Furthermore, I've found something even worse - if you don't set CI=true or --bail, and have failing tests, jest still returns a 0 exit code:

Again, this is the same jest binary that runs our back-end tests, and those work correctly. This is a very serious bug, and if people don't know to set CI=true specifically (that is, even using the --ci option doesn't work) this will mean CI builds will succeed with failing tests.

[bugzpodder]

Yes, the exit code issue you mentioned here is definitely a bug. There is already a handful of duplicate issues, however I don't think anyone has looked into what went wrong.

[rxgx]

The CI param doesn't work. In dev, we have to include the CI=true to get our CI command to work properly.

npm t -- --bail --ci --coverage --coverageDirectory=coverage \
  --no-cache --reporters=default --reporters=jest-junit --runInBand

[bugzpodder]

correct, CI=true is what I meant.

[JoeDevGeeeee]

CI=true does let Jest fail if threshold is not met. However, as @zjullion mentioned, it strips the hightlight of failed and passed errors. Any one found a solution?

[bugzpodder]

We have a fix and it will be released in 3.1: #7433

[timoteialbu]

@bugzpodder Thanks for fixing it! When is 3.1 coming out?

EDIT: I just saw this: https://github.com/facebook/create-react-app/projects/8. Am I guessing its out when all those items are done.

[bugzpodder]

Thanks for the patience. 3.1 is expected to be out this week. Unfinished items from the list will be moved to 3.2.

[zjullion]

Can confirm this is fixed in 3.1.0 - great work!