[Bug] dubbo3.3.0 + zookeeper consumer local service list cache refresh failure?

[skotty-open]

Pre-check

• I am sure that all the content I provide is in English.

Search before asking

• I had searched in the issues and found no similar issues.

Apache Dubbo Component

JDK version: openjdk17
springboot version: 3.3.1

Dubbo Version

dubbo-spring-boot-starter 3.3.0
dubbo-zookeeper-curator5-spring-boot-starter 3.3.0

Steps to reproduce this issue

1. Service A (consumer) calls service B (service provider)
2. Service B restarts (k8s pod)
3. Service A calls the old service B address (Pod IP)

comsumer log：
{"dateTime":"2025-01-23T20:41:30.030+0800","level":"WARN","traceId":"TID: N/A", "serviceName":"-user-center", "thread":"Dubbo-framework-SD-address-refresh-0-thread-1", "logger":"org.apache.dubbo.registry.client.ServiceDiscoveryRegistryDirectory", "message":" [DUBBO] Received url with EMPTY protocol from registry ServiceDiscoveryRegistryDirectory(registry: localhost:2181, subscribed key: [-platform-account])-Directory(invokers: 1[10.3.137.225:8080], validInvokers: 1[10.3.137.225:8080], invokersToReconnect: 0[]), will clear all available addresses., dubbo version: 3.3.0, current host: 10.3.139.21, error code: 4-1. This may be caused by , go to https://dubbo.apache.org/faq/4/1 to find instructions. "}
{"dateTime":"2025-01-23T20:41:30.030+0800","level":"INFO","traceId":"TID: N/A", "serviceName":"-user-center", "thread":"Dubbo-framework-SD-address-refresh-0-thread-1", "logger":"org.apache.dubbo.registry.client.migration.DefaultMigrationAddressComparator", "message":" [DUBBO] No instance address available, stop compare., dubbo version: 3.3.0, current host: 10.3.139.21 "}
{"dateTime":"2025-01-23T20:41:30.030+0800","level":"INFO","traceId":"TID: N/A", "serviceName":"-user-center", "thread":"Dubbo-framework-SD-address-refresh-0-thread-1", "logger":"org.apache.dubbo.registry.client.ServiceDiscoveryRegistryDirectory", "message":" [DUBBO] Received invokers changed event from registry. Registry type: instance. Service Key: com.*.atlas.module.account.api.PortfolioServiceApi. Urls Size : 1. Invokers Size : 0. Available Size: 0. Available Invokers : empty, dubbo version: 3.3.0, current host: 10.3.139.21 "}
{"dateTime":"2025-01-23T20:41:30.030+0800","level":"DEBUG","traceId":"TID: N/A", "serviceName":"-user-center", "thread":"Apollo-RemoteConfigLongPollService-1", "logger":"sun.net.www.protocol.http.HttpURLConnection", "message":"sun.net.www.MessageHeader@5e9ea39b10 pairs: {null: HTTP/1.1 304}{Date: Thu, 23 Jan 2025 12:41:30 GMT}{Connection: keep-alive}{X-Content-Type-Options: nosniff}{X-XSS-Protection: 1; mode=block}{Cache-Control: no-cache, no-store, max-age=0, must-revalidate}{Pragma: no-cache}{Expires: 0}{X-Frame-Options: DENY}{Request-Id: a372318842cc72901902dba4982fe1ef} "}
{"dateTime":"2025-01-23T20:41:30.030+0800","level":"DEBUG","traceId":"TID: N/A", "serviceName":"-user-center", "thread":"Apollo-RemoteConfigLongPollService-1", "logger":"com.ctrip.framework.apollo.internals.RemoteConfigLongPollService", "message":"Long polling response: 304, url: http://apollo-configservice.system-service.common-test.**.com/notifications/v2?cluster=test-3&appId=**-user-center&ip=10.3.139.21&notifications=%5B%7B%22namespaceName%22%3A%22RD1.portfolio%22%2C%22notificationId%22%3A263609%7D%2C%7B%22namespaceName%22%3A%22application%22%2C%22notificationId%22%3A2234%7D%2C%7B%22namespaceName%22%3A%22application.yaml%22%2C%22notificationId%22%3A263611%7D%5D "}

From the log, we can see that the A service consumer receives the B service provider status changes normally (including the old service and interface offline and the new service and interface online), but still requests the old pod IP address of the B service.

What you expected to happen

1. Service A (consumer) calls service B (service provider)
2. Service B restarts (k8s pod)
3. Service A calls the new service B address (Pod IP)

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wcy666103]

Please re-submit this pr in English only.

[AlbumenJ]

What is your metadata-center

[skotty-open]

What is your metadata-center
zookeeper is both a registration center and a metadata center

[skotty-open]

server:
port: 8080

dubbo:
application:
logger: slf4j
name: **-user-center
registry:
protocol: zookeeper
address: zoo1:2181, zoo2:2181, zoo3:2181
group: ${apollo.cluster}
parameters:
rootPath: /exchange/${apollo.cluster}/services
username: # 账号
password: # 密码
protocol:
name: tri
port: 8080

This issue appears only when protocol is configured as tri.

[AlbumenJ]

Have you ever tried with 3.3.3?

[skotty-open]

Have you ever tried with 3.3.3?

1. After using 3.3.3 to configure the tri protocol, the issue has been resolved. Can you explain what caused it and where the bug fix code is located
2. After using 3.3.3, I still cannot freely switch to the Dubbo protocol and there are still exceptions

---

Startup exception:

{"dateTime":"2025-02-43T12:11:21.021+0800","level":"ERROR","traceId":"http-nio-8080-exec-1raceId", "serviceName":"atlas-platform-user-center", "thread":"http-nio-8080-exec-1", "logger":"org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[dispatcherServlet]", "message":"Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception \njava.lang.ClassCastException: class org.springframework.http.server.observation.ServerRequestObservationContext cannot be cast to class org.apache.dubbo.tracing.context.DubboServerContext (org.springframework.http.server.observation.ServerRequestObservationContext and org.apache.dubbo.tracing.context.DubboServerContext are in unnamed module of loader org.springframework.boot.loader.launch.LaunchedClassLoader @c6da8bb)\n\tat org.apache.dubbo.tracing.handler.DubboServerTracingObservationHandler.onScopeOpened(DubboServerTracingObservationHandler.java:28) ~[dubbo-3.3.3.jar!/:3.3.3]\n\tat io.micrometer.observation.SimpleObservation.notifyOnScopeOpened(SimpleObservation.java:235) ~[micrometer-observation-1.13.1.jar!/:1.13.1]\n\tat io.micrometer.observation.SimpleObservation.openScope(SimpleObservation.java:194) ~[micrometer-observation-1.13.1.jar!/:1.13.1]\n\tat org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:106) ~[spring-web-6.1.10.jar!/:6.1.10]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.10.jar!/:6.1.10]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.25.jar!/:?]\n\tat com.*.atlas.framework.tracer.core.filter.TraceFilter.doFilterInternal(TraceFilter.java:29) ~[atlas-spring-boot-starter-monitor-2.12.149-snapshot.jar!/:?]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.10.jar!/:6.1.10]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.1.10.jar!/:6.1.10]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.10.jar!/:6.1.10]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.StandardHostValve.$sw$original$invoke$005sj03(StandardHostValve.java:115) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.StandardHostValve.$sw$original$invoke$005sj03$accessor$$sw$p8ebm33(StandardHostValve.java) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.StandardHostValve$$sw$auxiliary$m6nk332.call(Unknown Source) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86) [skywalking-agent.jar:9.3.0]\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:731) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:389) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:904) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-10.1.25.jar!/:?]\n\tat

---

Call exception:

request path: /user-app/user/getValidCountries

{"time":"2025-02-12 12:20:05.570", "level":"DEBUG", "traceID":"reactor-http-epoll-1raceId", "threadID":"reactor-http-epoll-1", "className":"io.netty.resolver.dns.DnsNameResolverBuilder", "message":"resolveCache and TTLs are mutually exclusive. TTLs are ignored. "}
{"time":"2025-02-12 12:20:05.570", "level":"DEBUG", "traceID":"reactor-http-epoll-1raceId", "threadID":"reactor-http-epoll-1", "className":"io.netty.resolver.dns.DnsNameResolverBuilder", "message":"cnameCache and TTLs are mutually exclusive. TTLs are ignored. "}
{"time":"2025-02-12 12:20:05.570", "level":"DEBUG", "traceID":"reactor-http-epoll-1raceId", "threadID":"reactor-http-epoll-1", "className":"io.netty.resolver.dns.DnsNameResolverBuilder", "message":"authoritativeDnsServerCache and TTLs are mutually exclusive. TTLs are ignored. "}
{"time":"2025-02-12 12:20:05.592", "level":"WARN ", "traceID":"reactor-http-epoll-1raceId", "threadID":"reactor-http-epoll-1", "className":"reactor.netty.http.client.HttpClientConnect", "message":"[ace3a860-1, L:/10.3.136.182:55992 - R:10.3.136.41/10.3.136.41:20880] The connection observed an error
java.lang.IllegalArgumentException: invalid version format: UNSUPPORTED
at io.netty.handler.codec.http.HttpVersion.(HttpVersion.java:135) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:89) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.http.HttpResponseDecoder.createMessage(HttpResponseDecoder.java:195) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:354) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:320) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) [netty-transport-classes-epoll-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501) [netty-transport-classes-epoll-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399) [netty-transport-classes-epoll-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.113.Final.jar!/:4.1.113.Final]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

"}
{"time":"2025-02-12 12:20:05.593", "level":"WARN ", "traceID":"reactor-http-epoll-1raceId", "threadID":"reactor-http-epoll-1", "className":"com.*.atlas.gateway.handler.GlobalErrorReactiveHandler", "message":"GlobalErrorReactiveHandler={}
java.lang.IllegalArgumentException: invalid version format: UNSUPPORTED
at io.netty.handler.codec.http.HttpVersion.(HttpVersion.java:135) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ com.alibaba.csp.sentinel.adapter.spring.webflux.SentinelWebFluxFilter [DefaultWebFilterChain]
*__checkpoint ⇢ HTTP GET "/user-app/user/getValidCountries" [ExceptionHandlingWebHandler]
Original Stack Trace:
at io.netty.handler.codec.http.HttpVersion.(HttpVersion.java:135) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:89) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.http.HttpResponseDecoder.createMessage(HttpResponseDecoder.java:195) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:354) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:320) ~[netty-codec-http-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[netty-codec-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) [netty-codec-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) [netty-transport-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) [netty-transport-classes-epoll-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501) [netty-transport-classes-epoll-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399) [netty-transport-classes-epoll-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.113.Final.jar!/:4.1.113.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.113.Final.jar!/:4.1.113.Final]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

[AlbumenJ]

Please close Dubbo tracing first, it might not work well with Skywalking

[skotty-open]

After using 3.3.3 to configure the tri protocol, the issue has been resolved. Can you explain what caused it and where the bug fix code is located