Benchmark alpha 0 tip selection performance

[jakubcech]

Description

We want to benchmark tipselection with alpha set to 0 to set some good baseline to reflect upon.

Reuse JMeter tests. https://github.com/iotaledger/iri-regression-tests/tree/master/Nightly-Tests/Jmeter-Tests
We should also use the profiler.

Motivation

Have a [naive] baseline for tipsel improvements.

Requirements

• If alpha 0, we ignore CW calculation, so needs to be run on a changed version of IRI. (Bonus quest:) Can be a PR in IRI that integrates it already. Just setting the alpha 0 will make the node not avoid the CW calculation.
• Have a node that uses an existing DB in our S3 bucket. Note it down in the comment here so that we can always perform the same test. This will allow us to redo the test later on whenever.
• Use an isolated machine, make sure you use the same for Benchmark uniform random tipselection performance #1568
• Run both the Jmeter tests (can be run manually). https://github.com/iotaledger/iri-regression-tests/tree/master/Nightly-Tests/Jmeter-Tests
• Run the yourkit profiler.
• Give us results and write a beautiful summary.

[GalRogozinski]

When changing in IRI do the following changes:
In TipSelectorImpl, if alpha==0 return EmptyRatingsCalculator that simply returns Collections.EMPTY_MAP

In WalkerAlpha reimplement select by adding the following if clause:

if (Collections.EMPTY_MAP.equals(ratings)) {
            int randApprover = random.nextInt(approversSet.size() + 1);
....

One needs of course to convert the approver set to array, or iterate the set randApprover times.
Also should check for edge conditions

[DyrellC]

Test setup

DB: https://dbfiles.iota.org/mainnet/1.7.0/db-1072888.tar
VM: 8GB RAM, 160GB HDD, Ubuntu16.04.4x64

At Rest/Pre Test Operational Requirements

CPU: ~3%
Mem: ~780MB

[DyrellC]

Test run using 1 thread looping 1000 times with gtta calls

Standard Performance

API response speed: Averages half a second per response when sequentially sent
Memory usage: Memory usage continued to climb from ~780MB to 1GB over the span of the test, and did not drop after completion
CPU usage: Usage hovered at ~23-25% on average, less than with the 1000 calls at once due to the need to only process one request at a time.

With alpha == 0

API response speed: Moderately small reduction in individual call time (485 vs 416)
Memory usage: Took longer to get to 1GB but still got there after ~500 calls, same as in the instance of 1000 calls at once.
CPU usage: Marginally less usage than when alpha != 0

With IRI Modification and alpha == 0

API response speed: Average call response reduced to a third of standard performance
Memory usage: No increase during call processing, in fact there are moments where usage drops lower than it was before the calls were made.
CPU usage: Usage only exceeds 20% for a brief moment then stabilizes at ~8%

Notes: As seen here, the results speak for themselves. Significant reduction in call response time, CPU usage during call processing, and memory usage.

[DyrellC]

Test run using 1000 thread looping 1 times with gtta calls

Standard Performance

API response speed: In order to return all 1000 calls, the node takes ~279 seconds.
Memory usage: Memory usage goes from ~780MB to 1GB within the first 200 response cycles.
CPU usage: Averages around ~30% during processing.

With alpha == 0

API response speed: Longest call returned in ~263 seconds showing a marginally improved calculation time due to the introduced alpha 0.
Memory usage: Memory still reaches 1GB but it takes longer (~500 calls in) to get there
CPU usage: Usage does not show any reduction or increase from standard.

With IRI Modification and alpha == 0

API response speed: Significant improvement in response time. Out of the 1000 calls, the longest wait time was 32 seconds.
Memory usage: Memory usage increase is negligible for these calls
CPU usage: CPU usage is ~30% as it was with the other implementation

Notes: This modification has resulted in a significant reduction in memory usage and response time for GTTA calls made to the node.

[GalRogozinski]

Can you please:

1. Estimate what is your network latency? Currently IRI still returns duration which is the time it took to execute the the call w/o the latency. You should be able to see those values on Jmeter.
2. Put a link to the modified branch?

[GalRogozinski]

3. When cpu spikes, can you tell what causes it? You should be able to see it on YourKit

[DyrellC]

@GalRogozinski #1596

[GalRogozinski]

@DyrellC
Thanks, what about questions 1 & 3?

[DyrellC]

1. Average latency is approximately 350-360 milliseconds
2. When running the tests over and over I haven't seen these spikes happen again