1.9.0-beta2: Unable to find compatible target in system image, when loading a package #48111
Comments
|
Can you describe a bit more what your setup was? We should have rejected the cache file before we loaded it. Did you compile on a more modern machine and then used the cache on a older node? What are the architectures involved? |
|
Also can you rerun Julia with: |
This is a single Linux machine with the processor I detailed above. Something seems heterogeneous about the cores though. If I compile Julia from source without specifying
No . It's just the processor I reported above. |
|
|
Now with $ JULIA_CPU_TARGET="generic; native" JULIA_DEBUG=loading julia-1.9.0-beta2/bin/julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.9.0-beta2 (2022-12-29)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.9) pkg> activate --temp
Activating new project at `/tmp/jl_6AE6hc`
(jl_6AE6hc) pkg> add Interpolations
Updating registry at `~/.julia/registries/General.toml`
Resolving package versions...
Updating `/tmp/jl_6AE6hc/Project.toml`
[a98d9a8b] + Interpolations v0.14.7
Updating `/tmp/jl_6AE6hc/Manifest.toml`
[79e6a3ab] + Adapt v3.4.0
[13072b0f] + AxisAlgorithms v1.0.1
[d360d2e6] + ChainRulesCore v1.15.6
[34da2185] + Compat v4.5.0
[a98d9a8b] + Interpolations v0.14.7
[6fe1bfb0] + OffsetArrays v1.12.8
[c84ed2f1] + Ratios v0.4.3
[ae029012] + Requires v1.3.0
[90137ffa] + StaticArrays v1.5.12
[1e83bf80] + StaticArraysCore v1.4.0
[efce3f68] + WoodburyMatrices v0.5.5
[0dad84c5] + ArgTools v1.1.1
[56f22d72] + Artifacts
[2a0f44e3] + Base64
[ade2ca70] + Dates
[8ba89e20] + Distributed
[f43a241f] + Downloads v1.6.0
[7b1f6079] + FileWatching
[b77e0a4c] + InteractiveUtils
[b27032c2] + LibCURL v0.6.3
[76f85450] + LibGit2
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[a63ad114] + Mmap
[ca575930] + NetworkOptions v1.2.0
[44cfe95a] + Pkg v1.8.0
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA v0.7.0
[9e88b42a] + Serialization
[1a1011a3] + SharedArrays
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics v1.9.0
[fa267f1f] + TOML v1.0.3
[a4e569a6] + Tar v1.10.0
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
[e66e0078] + CompilerSupportLibraries_jll v1.0.2+0
[deac9b47] + LibCURL_jll v7.84.0+0
[29816b5a] + LibSSH2_jll v1.10.2+0
[c8ffd9c3] + MbedTLS_jll v2.28.0+0
[14a3606d] + MozillaCACerts_jll v2022.10.11
[4536629a] + OpenBLAS_jll v0.3.21+0
[bea87d4a] + SuiteSparse_jll v5.10.1+0
[83775a58] + Zlib_jll v1.2.13+0
[8e850b90] + libblastrampoline_jll v5.2.0+0
[8e850ede] + nghttp2_jll v1.48.0+0
[3f19e933] + p7zip_jll v17.4.0+0
┌ Debug: Skipping mtime check for file /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /home/mkitti/src/julia-1.9.0-beta2/share/julia/compiled/v1.9/Statistics/ERcPL_2A8RB.ji, since it is a stdlib
└ @ Base loading.jl:2627
┌ Debug: Rejecting cache file /home/mkitti/.julia/compiled/v1.9/Adapt/rUIgN_QorxP.ji for [top-level] since pkgimage can't be loaded on this target
└ @ Base loading.jl:2538
┌ Debug: Rejecting cache file /home/mkitti/.julia/compiled/v1.9/Compat/GSFWK_QorxP.ji for [top-level] since pkgimage can't be loaded on this target
└ @ Base loading.jl:2538
┌ Debug: Skipping mtime check for file /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Zlib_jll/src/Zlib_jll.jl used by /home/mkitti/src/julia-1.9.0-beta2/share/julia/compiled/v1.9/Zlib_jll/xjq3Q_2A8RB.ji, since it is a stdlib
└ @ Base loading.jl:2627
┌ Debug: Skipping mtime check for file /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl used by /home/mkitti/src/julia-1.9.0-beta2/share/julia/compiled/v1.9/MbedTLS_jll/u5NEn_2A8RB.ji, since it is a stdlib
└ @ Base loading.jl:2627
┌ Debug: Skipping mtime check for file /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/SuiteSparse_jll/src/SuiteSparse_jll.jl used by /home/mkitti/src/julia-1.9.0-beta2/share/julia/compiled/v1.9/SuiteSparse_jll/ME9At_2A8RB.ji, since it is a stdlib
└ @ Base loading.jl:2627
┌ Debug: Rejecting cache file /home/mkitti/.julia/compiled/v1.9/Adapt/rUIgN_W9fcW.ji due to it containing an invalid cache header
└ @ Base loading.jl:2518
┌ Debug: Rejecting cache file /home/mkitti/.julia/compiled/v1.9/Compat/GSFWK_W9fcW.ji due to it containing an invalid cache header
└ @ Base loading.jl:2518
┌ Debug: Skipping mtime check for file /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/LibSSH2_jll/src/LibSSH2_jll.jl used by /home/mkitti/src/julia-1.9.0-beta2/share/julia/compiled/v1.9/LibSSH2_jll/K6mup_2A8RB.ji, since it is a stdlib
└ @ Base loading.jl:2627
Precompiling environment...
5 dependencies successfully precompiled in 20 seconds. 11 already precompiled.
julia> using Interpolations
┌ Debug: Skipping mtime check for file /cache/build/default-amdci5-1/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /home/mkitti/src/julia-1.9.0-beta2/share/julia/compiled/v1.9/Statistics/ERcPL_2A8RB.ji, since it is a stdlib
└ @ Base loading.jl:2627
┌ Debug: Loading object cache file /home/mkitti/src/julia-1.9.0-beta2/share/julia/compiled/v1.9/Statistics/ERcPL_2A8RB.so for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/StaticArraysCore/Tzw28_Cd0mR.so for StaticArraysCore [1e83bf80-4336-4d27-bf5d-d5a4f845583c]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/StaticArrays/yY9vm_Cd0mR.so for StaticArrays [90137ffa-7385-5640-81b9-e52037218182]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/WoodburyMatrices/LdtKf_Cd0mR.so for WoodburyMatrices [efce3f68-66dc-5838-9240-27a6d6f5f9b6]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/Requires/IyxeS_Cd0mR.so for Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/Ratios/zQEsd_Cd0mR.so for Ratios [c84ed2f1-dad5-54f0-aa8e-dbefe2724439]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/AxisAlgorithms/lW7vi_Cd0mR.so for AxisAlgorithms [13072b0f-2c55-5437-9ae7-d433b7a33950]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/Adapt/rUIgN_Cd0mR.so for Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/OffsetArrays/UDEDl_Cd0mR.so for OffsetArrays [6fe1bfb0-de20-5000-8ca7-80f57d26f881]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/Compat/GSFWK_Cd0mR.so for Compat [34da2185-b29b-5c13-b0c7-acf172513d20]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/ChainRulesCore/G6ax7_Cd0mR.so for ChainRulesCore [d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4]
└ @ Base loading.jl:948
┌ Debug: Loading object cache file /home/mkitti/.julia/compiled/v1.9/Interpolations/VpKVx_Cd0mR.so for Interpolations [a98d9a8b-a2ab-59e6-89dd-64a1c18fca59]
└ @ Base loading.jl:948 |
|
Two suggestions:
|
|
I found the difference between my cores. Only half of them have "LWP" or Lightweight Profiling Specification using CpuId
f = [Symbol[] for i in 1:Threads.nthreads()];
Threads.@threads for i in 1:Threads.nthreads()
f[Threads.threadid()] = cpufeatures()
end
all_features = union(f...);
julia> setdiff.((all_features,), f)
8-element Vector{Vector{Symbol}}:
[]
[]
[:LWP]
[:LWP]
[]
[]
[:LWP]
[:LWP]
flags = split.(Iterators.filter(startswith("flag"), eachline("/proc/cpuinfo"))," ");
all_flags = union(flags...);
julia> setdiff.((all_flags,), flags)
8-element Vector{Vector{SubString{String}}}:
["lwp"]
[]
["lwp"]
[]
["lwp"]
[]
["lwp"]
[]
Part of this is because the Bulldozer architecture is actually 4 cores masquerading as 8 cores: |
|
Related: |
|
In my case, I think I want to use either I got the processor name from julia> Sys.CPU_NAME
"bdver1" |
|
I don't think we can do much about this, apart from documenting this more explicitly somewhere. But heterogeneous systems are always problematic like that, and this isn't limited to Julia as the GCC bug report shows. What changed is that pkgimages cache the native code in advance, like statically compiled languages typically do, instead of compiling on-the-fly. |
|
Oh this is devious... I think we can on our side define bulldozer as |
https://patchwork.kernel.org/project/xen-devel/patch/[email protected]/ To summarize the above, AMD Bulldozer, Piledriver, Steamroller and Excavator processors have a featured called LWP, or "Lightweight Profiling". This feature was later disabled by a microcode update. However, the Linux kernel only applied the microcode update to half the cores due to a bug. A Linux kernel patch has been developed to deploy the microcode update to all cores, disabling lwp everywhere. While we could just wait for the Linux kernel to be patched, an alternative would be just to drop detection of that feature in Julia since AMD no longer supports it. |
We can special case some CPUs (is it worth it since it's so old? Also, if anything should it be done at llvm-level rather than in Julia?), but again, there can be other heterogeneous systems where you can't do much. |
|
@mkitti can you try if #48131 solves the issue for you? @giordano Yeah the particular devious case is that the scheduler migrated use between cores with different feature flags and so we verified that we could load the cache on one core, and then loaded it on another core. I think everyone else with heterogeneous cores is currently using the same feature set among them (since otherwise migrating a process wouldn't work). |
|
I'm building it now. |
Fix: diff --git a/src/processor_x86.cpp b/src/processor_x86.cpp
index 6f064ddd47..c61712ada7 100644
--- a/src/processor_x86.cpp
+++ b/src/processor_x86.cpp
@@ -219,7 +219,7 @@ constexpr auto btver2 = btver1 | get_feature_masks(sse41, sse42, avx, aes, pclmu
movbe, xsave, xsaveopt);
constexpr auto bdver1 = amdfam10 | get_feature_masks(xop, fma4, avx, ssse3, sse41, sse42, aes,
- prfchw, pclmul, xsave, lwp);
+ prfchw, pclmul, xsave);
constexpr auto bdver2 = bdver1 | get_feature_masks(f16c, bmi, tbm, fma);
constexpr auto bdver3 = bdver2 | get_feature_masks(xsaveopt, fsgsbase);
constexpr auto bdver4 = bdver3 | get_feature_masks(avx2, bmi2, mwaitx, movbe, rdrnd); |
|
With my patch, it seems to be working well so far! |
|
Thanks for the fix! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I encountered "ERROR: Unable to find compatible target in system image." when loading a package I was developing while using Julia 1.9.0-beta2.
I suspect this is a new variant of an old problem. The old problem occurs when compiling a system image when Julia from source. Now with native code precompilation, the problem can occurring when precompiling a package. This is likely to affect more users.
For users encountering this problem, setting the environment variable
JULIA_CPU_TARGETto"generic; native"will resolve the issue. This can be done within Julia with the following command.This command can be inserted into the
.julia/config/startup.jlfile. Additionally, since I already have .so files, I had to remove some of the .so files located in.julia/compiled/v1.9for the affected packages.Previously I would say that the above would suffice, but I think a much larger set of users will encounter this issue. Perhaps we should attempt to detect older processor architectures and automatically select or include
genericas a target architecture.cat /proc/cpuinfo
The text was updated successfully, but these errors were encountered: