cabal bench suggests to add tests: True instead of benchmarks: True to cabal.project.local file

[Lev135]

Describe the bug
Running cabal bench for project without benchmarks: True provides the following hint:

cabal-3.6.2.0.exe: Cannot benchmark the package space-consumers-0.1.0.0
because none of the components are available to build: the benchmark
'bench-foo' is not available because the solver did not find a plan that
included the benchmarks. Force the solver to enable this for all packages by
adding the line 'tests: True' to the 'cabal.project.local' file.

however, only adding benchmarks: True, but not tests: True solves this problem

To Reproduce
Steps to reproduce the behavior:

• create a new project without bechmarks: True in cabal.project file
• add benchmark
• run cabal bench

Expected behavior
Hint should contain correct suggestion (to add benchmarks: True)

System information

• Operating system: Windows 10
• cabal-install version 3.6.2.0

[jneira]

oof, good catch, thanks
pr fixing it totally welcomed, it should not be too difficult but wuld be good to include a regression test for it

[erinclemmer]

@Lev135 I'm having trouble reproducing this because it's a solver issue and not a configuration issue. I assume the project you are working on is called space-consumers based on the output and it looks like it is having trouble finding a benchmark. Can you share your .cabal file so that I can replicate it for the regression test?

[ulysses4ever]

@cbclemmer very good call, thanks! I guess even without a reproducer, it should still be possible to grep through the source for this message, and see how it's used. It should be more or less clear where it's misused.

[erinclemmer]

@ulysses4ever It's used in this error handler. Whenever you explicitly set benchmarks: False in the config file or --disable-benchmarks as a command line arg it uses this case and that's why I can't reproduce it. I think the solver has to run into some kind of problem before it gives that message. I'm just not sure what problem it is exactly.

[ulysses4ever]

@cbclemmer makes sense, thank you for checking. Over to @Lev135 for a reproducer.

[Lev135]

Unfortunatly, I haven't preserved the repository at that stage. Now, I also fail to reproduce a bug. I'm sure I didn't use --disable-benchmarks or benchmarks: False in the config, but cabal failed to build my benchmark with that strange error. I tried to clean up project, but this didn't lead to any result. That's all I can say about it

[erinclemmer]

@Lev135 No worries, the error came from some kind of configuration error in your .cabal file. Let us know if it happens again.

[eyeinsky]

I've just ran into this issue. Though I unfortunately can't share the repo then cabal-install version 3.10.3.0. The package has a library, an executable and a test component. Running cabal clean didn't help.

[ulysses4ever]

The confusing line was added in 0ce6ba9 by @erikd:

cabal/cabal-install/src/Distribution/Client/CmdErrorMessages.hs

Lines 405 to 406 in eda072d

	++ "packages, by adding the line 'tests: True' to the "
	++ "'cabal.project.local' file."

I guess the assumption was that renderTargetProblemNoneEnabled with TargetDisabledBySolver can only be used when the failing target is a test target, which is, clearly, not true. In fact, a comment near the single place producing TargetDisabledBySolver says "testsuite or benchmark".

Possible ways forward:

• revert 0ce6ba9

• change the tests part of the message with something more appropriate; one thing that could work is renderOptionalStanza Plural stanza which will produce tests or benchmarks accordingly. This feels a little hacky though.

Dear all, please, submit a PR fixing the issue as you like! It'd be great to have a regression test for this too but it may be less trivial than the actual fix, so this shouldn't be a requirement, I think.

[eyeinsky]

Would it be against some broader principle for cabal bench to just run the benchmark suite? I.e, as I just typed that command then I'd just like to run the benchmark suite then and there.

[ulysses4ever]

@eyeinsky that's a much bigger issue than what this ticket is about. See #7883 (it talks about tests but all the same applies to benches). There is a path to a solution there but it needs a careful implementation, as we had a big fallout from doing similar thing with cabal haddock #8725

[Bodigrim]

This is a duplicate of #7524