addr2line taking an exorbitant amount of time

[Licenser]

Hi,
I've recently been using flamegraph on a Linux system and once the recording is done it takes an extremely long time. It seems to be invoking addr2line over and over again making the whole process quite slow.

The output is:

[ perf record: Woken up 682 times to write data ]
[ perf record: Captured and wrote 170,633 MB perf.data (21198 samples) ]

21k samples don't sound that much but if it's invoking a program for every sample it that seems to become very expensive.

[bjorn3]

This is perf trying to compute which functions are inlined at every stack frame for every sample. If you don't want this, you need to pass --no-inline to perf. The invocation can be found at

flamegraph/src/lib.rs

Lines 80 to 86 in 0b8d12d

	pub fn output() -> Vec<u8> {
	Command::new("perf")
	.arg("script")
	.output()
	.expect("unable to call perf script")
	.stdout
	}

[tonyg]

Hi, I wrote a patch for perf which uses a long-running addr2line process instead of one subprocess per address-to-look-up. It dramatically improves performance of perf, making flamegraphs on large samples with inlining a possibility again. See: https://eighty-twenty.org/2021/09/09/perf-addr2line-speed-improvement

[djc]

@tonyg very cool! Did you submit your patch to perf upstream? It seems like the kind of thing they might consider merging.

[bjorn3]

The blog post links to https://lore.kernel.org/linux-perf-users/20210909112202.1947499-1-tonyg@leastfixedpoint.com/ which was submitted about an hour ago.

[tonyg]

Thanks @djc! As @bjorn3 said, I've sent it upstream, but it's too soon to say if it'll be acceptable or not. Another possible audience is the Debian maintainers, if the kernel folks won't take it. I'm not sure if it's just Debian that has the slowdown; certainly, other distributions are linking perf against libbfd which doesn't suffer from the problem.

[Licenser]

I can cofirm in that ubnutu (admittedly based on Debian) also suffers the perf problem with addr2line so it is not just (vanilla) debian

[Geal]

I am trying the solution of @tonyg and it's indeed a lot faster, but it generates flamegraphs that are missing a lot of information, like I can get a graph that only has kernel level traces, or has the application's call stack but does not show function names (debug info is properly generated). I'm on ubuntu, linux 5.13

Any advice on how I could debug this?

[tonyg]

I am trying the solution of @tonyg and it's indeed a lot faster, but it generates flamegraphs that are missing a lot of information, like I can get a graph that only has kernel level traces, or has the application's call stack but does not show function names (debug info is properly generated). I'm on ubuntu, linux 5.13

That's interesting! Does the same binary, run with the unpatched perf, yield better flamegraphs? (My advice, though I'm sure you've already done this, would be to double-check your Cargo.toml settings for profile.bench and profile.release, ensuring that debug = true and strip = false...)

[Geal]

I was not adding strip = false but debug=true was there for the release profile, and I verified that the generated binary had symbols using strings.
The unpatched perf yielded better flamegraphs yes

[Geal]

ok, so that was a stupid mistake on my part: libdw and others were not installed, so perf was built without the ability to read the symbols. That was written plainly right at the beginning of make's output 😑

[osa1]

Could anyone update us about the perf patch mentioned above please? Is it merged to upstream? In the linked thread I don't see an email that announces that it's merged so I think it's not?

Currently on my application a recording of 30 seconds takes about an hour to render.

[djc]

It does appear in the current Linux tree which is easy to check, here is the commit on GitHub.

Looks like it's available as of 5.16.