Test with BOLT

[rui314]

Facebook/Meta's BOLT is becoming popular. I don't know if BOLT works on mold's output. We need to test it and fix issues if any.

[ptr1337]

I will test tomorrow to build a clang toolchain with mold as linker, and then instrument and optimize it.
I will report it here.

I just tried to BOLT mold itself, compiled it with --emit-relocs and -fno-reorder-blocks-and-partition.
When trying to create the instrumented binary, I get a Segmentation fault.

❯ ./bolt-anything.bash
Instrument binary with llvm-bolt
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 39f01240e76678fa6385d36e7a96670677a26d65
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x1000000, offset 0x1000000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling -align-macro-fusion=all since no profile was specified
BOLT-INFO: enabling lite mode
/home/ptr1337/llvm-bolt-scripts/bolt-anything.bash: line 41: 37593 Segmentation fault      ${BOLTPATH}/llvm-bolt --instrument --instrumentation-file-append-pid --instrumentation-file=${FDATA}/${BINARY}.fdata ${BINARYPATH}/${BINARY} -o ${BOLTBIN}/${BINARY}
Could not create instrumented binary

[gcflymoto]

@rui314 we are also eagerly trying to combine MOLD + CLANG + PGO + LTO + BOLT... it's certainly a journey to get there for big apps.

[rui314]

@ptr1337 Thanks! I'd start with a "hello world" program though.

[rui314]

Here is a brain dump of things that might affect the compatibility with BOLT.

• Our .eh_frame section does not have a relocation section even if --emit-relocs is given. We synthesize .eh_frame, and we probably need to synthesize .rela.eh_frame too
• Relocation sections emitted when --emit-relocs is given is always in the RELA format even for a target with REL type. This is not a bug but a feature, as it should make binary post-processing easier. I don't think it's causing an issue, as x86-64 and aarch64 are both RELA, though.
• Our .plt and .plt.got machine instructions are different from the ones in the psABIs.

[rui314]

@gcflymoto Is there anything we can help?

[ptr1337]

@rui314

I think this could help you, I have just tested to bolt clang.

Instrument clang with llvm-bolt
BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 39f01240e76678fa6385d36e7a96670677a26d65
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x4e00000, offset 0x4e00000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling -align-macro-fusion=all since no profile was specified
BOLT-INFO: enabling lite mode
BOLT-WARNING: _etext/1 (0x490955c) does not have any section
BOLT-WARNING: etext/1 (0x490955c) does not have any section
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/home/ptr1337/llvm-bolt-scripts/build_stage3-bolt-without-sampling.bash: line 19: 69152 Aborted                 ${BOLTPATH}/llvm-bolt --instrument --instrumentation-file-append-pid --instrumentation-file=${TOPLEV}/stage3-without-sampling/intrumentdata

[rui314]

@ishitatsuyuki Did you want to take this?

[ishitatsuyuki]

@rui314 Yes, I'm tentatively trying to get a Rust CI build to work, which uses BOLT to optimize LLVM (rust-lang/rust#94381). I'll report if it succeeds or not.

[ishitatsuyuki]

OK, it failed:

BOLT-INFO: shared object or position-independent executable detected
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: <unknown>
BOLT-INFO: first alloc address is 0x0
BOLT-INFO: creating new program header table at address 0x5000000, offset 0x5000000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling -align-macro-fusion=all since no profile was specified
BOLT-INFO: enabling lite mode
BOLT-WARNING: _etext/1 (0x4a4f843) does not have any section
BOLT-WARNING: etext/1 (0x4a4f843) does not have any section
BOLT-WARNING: split function detected on input : _ZNSi6ignoreEl.cold/1. The support is limited in relocation mode.
 #0 0x0000000000958a1f PrintStackTraceSignalHandler(void*) Signals.cpp:0:0
 #1 0x0000000000956814 SignalHandler(int) Signals.cpp:0:0
 #2 0x00007fef06d83db0 __restore_rt (/lib64/libc.so.6+0x59db0)
 #3 0x00007fef06e0ffc0 __memmove_avx_unaligned_erms (/lib64/libc.so.6+0xe5fc0)
 #4 0x000000000099ded9 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (.constprop.0) RewriteInstance.cpp:0:0
 #5 0x00000000009d959b llvm::bolt::RewriteInstance::analyzeRelocation(llvm::object::RelocationRef const&, unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&, bool&, unsigned long&, long&, unsigned long&, bool&) const (/rustroot/bin/llvm-bolt+0x9d959b)
 #6 0x00000000009d9e6b llvm::bolt::RewriteInstance::readRelocations(llvm::object::SectionRef const&) (/rustroot/bin/llvm-bolt+0x9d9e6b)
 #7 0x00000000009dbdcd llvm::bolt::RewriteInstance::processRelocations() (.part.0) RewriteInstance.cpp:0:0
 #8 0x00000000009fa56a llvm::bolt::RewriteInstance::discoverFileObjects() (/rustroot/bin/llvm-bolt+0x9fa56a)
 #9 0x00000000009fbf49 llvm::bolt::RewriteInstance::run() (/rustroot/bin/llvm-bolt+0x9fbf49)
#10 0x0000000000419bab main (/rustroot/bin/llvm-bolt+0x419bab)
#11 0x00007fef06d6ee50 __libc_start_call_main (/lib64/libc.so.6+0x44e50)
#12 0x00007fef06d6eefc __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x44efc)
#13 0x000000000047baa5 _start (/rustroot/bin/llvm-bolt+0x47baa5)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: llvm-bolt -instrument /checkout/obj/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVM-15-rust-1.66.0-nightly.so --instrumentation-file-append-pid -o /tmp/instrumented.so

I'll try to gather more notes for reproduction later, but let me know if you need something in particular.

[ishitatsuyuki]

Instrument runs fine with llvm main so it's some fun bug that got fixed between the 15.0 and 16.0 cycle...

[rui314]

If we use LLVM 16, can we use BOLT with mold without doing anything special?

[ishitatsuyuki]

I still haven't tested the optimization pass (the part that is not instrumentation) yet.

[ishitatsuyuki]

I think I spoke too soon. Bisection resulted in an all-pass so this might be an environment dependent problem (build flags? silent memory errors?).

[ishitatsuyuki]

This line is failing, even in main: https://github.com/llvm/llvm-project/blob/3ee58e2f355f8fdb8e0fe29dc366c8833fafa7d3/bolt/lib/Rewrite/RewriteInstance.cpp#L1885

Lesson to me is to always turn on LLVM assertions when debugging. Without assertions this turned into a heisenbug.

I'll try to see why.

[rui314]

I usually build LLVM (and mold) with -DCMAKE_RELEASE_TYPE=Debug unless I'm doing performance optimization.