Skip to content

timers "depth" benchmark usefulness #9493

Closed
@Trott

Description

@Trott
Member
  • Version: v8.0.0-pre
  • Platform: all
  • Subsystem: benchmark, timers

The depth benchmark for timers sets a timer that sets a timer that sets a timer that... 500K of them.

Since each timer has to wait for the next tick of the event loop:

  • This benchmark takes a very long time to run compared to the breadth test that is already in the file.
  • This may be more of an event loop benchmark than a timer benchmark.

I wonder if it makes sense to do any of the following or something similar:

  • Reduce the number of iterations for the depth test as it's really just running the iterations in sequence, not in parallel. And even on an infinitely fast machine, it would take over 8 minutes to run because each tick of the event loop would have to wait 1ms before firing the timer.

  • Move and/or rename the depth benchmark as it is unlikely to be something significantly impacted by changes in the Node.js timers code.

I know I can send command line arguments to skip the depth test or change the value of N. I just suspect that the default behavior right now isn't ideal. Every time I touch timers code and run a benchmark, this is an annoyance.

Activity

added
timersIssues and PRs related to the timers subsystem / setImmediate, setInterval, setTimeout.
benchmarkIssues and PRs related to the benchmark subsystem.
on Nov 6, 2016
Trott

Trott commented on Nov 6, 2016

@Trott
MemberAuthor

@mscdex @misterdjules @Fishrock123 @AndreasMadsen

Oh, I'll also note that it was moved from misc to timers earlier this year without apparent significant disruption, so presumably another move or split would not be disruptive either.

Fishrock123

Fishrock123 commented on Nov 6, 2016

@Fishrock123
Contributor

We could probably remove it, a test assertion asserting that timers are indeed pooled in this case mentioning benchmarks should be good enough combined with other benchmarks.

Fishrock123

Fishrock123 commented on Nov 6, 2016

@Fishrock123
Contributor

Hmmm, at a second look I definitely think there are not enough benchmarks to replace it yet.

self-assigned this
on Nov 6, 2016
AndreasMadsen

AndreasMadsen commented on Nov 6, 2016

@AndreasMadsen
Member

There are definitely benchmarks that uses many more iterations than what is required. I guess this is one of them.

I'm don't like moving the benchmark to a diffrent category, I think timer benchmarks should be in the timer category. @Fishrock123 suggestion sounds reasonable!

If you are interested in reducing the number of iterations, a rough estimate on the appropriate number of iterations could be found by tuning the coefficient of variation std(x)/mean(x) (use the unbiased estimate). See #8139 (comment) for the practical meaning of this.

mscdex

mscdex commented on Nov 6, 2016

@mscdex
Contributor

I think in general there are lots of cases like this in many of the different benchmarks where some configurations take longer than others but the same iteration count is used for all of them. I don't know of a good way to solve this, since implicitly altering the iteration count for certain configurations could be seen as unexpected (even if the new iteration count is reported in the output).

Trott

Trott commented on Nov 6, 2016

@Trott
MemberAuthor

Maybe it makes sense to split this benchmark into two files and but leave them both in the timers directory? This way they can have separate N values but both will still be categorzied as timer benchmarks.

Trott

Trott commented on Nov 7, 2016

@Trott
MemberAuthor
added a commit that references this issue on Nov 22, 2016

18 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

benchmarkIssues and PRs related to the benchmark subsystem.timersIssues and PRs related to the timers subsystem / setImmediate, setInterval, setTimeout.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @mscdex@AndreasMadsen@Trott@Fishrock123

      Issue actions

        timers "depth" benchmark usefulness · Issue #9493 · nodejs/node