如果您跟我一樣,目前正在使用Spring Cloud Config做為配置中心的話,本篇將來(lái)要描述的問(wèn)題,強(qiáng)烈推薦了解和關(guān)注!因?yàn)檫@個(gè)問(wèn)題目前存在于所有的版本中,還沒(méi)有完全修復(fù)。問(wèn)題現(xiàn)象原文鏈接:http://blog.didispace.com/Spr...
為了說(shuō)明下面的內(nèi)容,我們可以先嘗試重現(xiàn)一下問(wèn)題:在一個(gè)測(cè)試環(huán)境中,將Spring Cloud Config的配置中心遷移到另外一個(gè)節(jié)點(diǎn)上,即配置中心的IP地址發(fā)生了變化。在完成遷移之后,我們會(huì)發(fā)現(xiàn)該環(huán)境下各個(gè)微服務(wù)應(yīng)用的健康狀態(tài)會(huì)變得時(shí)好時(shí)壞,并且在日志中會(huì)出現(xiàn)類似下面的報(bào)錯(cuò):
2018-05-13 17:01:28,569 WARN [http-nio-9920-exec-1] org.springframework.cloud.config.client.ConfigServerHealthIndicator - Health check failed java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:132) at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$FastClassBySpringCGLIB$$fa44b2a.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91) at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287) at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164) at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118) at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:153) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$EnhancerBySpringCGLIB$$3a43a1f4.locate( ) at org.springframework.cloud.config.client.ConfigServerHealthIndicator.getPropertySource(ConfigServerHealthIndicator.java:54) at org.springframework.cloud.config.client.ConfigServerHealthIndicator.doHealthCheck(ConfigServerHealthIndicator.java:35) at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:43) at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:85) at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getCurrentHealth(HealthMvcEndpoint.java:177) at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:166) at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:143) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.yonghui.feign.filter.RequestOriginFilter.doFilter(RequestOriginFilter.java:41) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at com.yonghui.rpc.feature.web.boot.RpcHolder4BootFilter.doFilterInternal(RpcHolder4BootFilter.java:29) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at com.yonghui.rpc.feature.web.boot.FeatureSupport4BootFilter.doFilterInternal(FeatureSupport4BootFilter.java:24) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://192.168.5.103:9010/config-server/test": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:539) at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.getRemoteEnvironment(ConfigServicePropertySourceLocator.java:172) at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:93) ... 95 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) at sun.net.www.http.HttpClient. (HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339) at sun.net.www.http.HttpClient.New(HttpClient.java:357) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78) at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660) ... 99 more
可以看到類似上面的健康檢查失敗錯(cuò)誤,但是并不是一直這樣,這個(gè)環(huán)境下的微服務(wù)會(huì)出現(xiàn)時(shí)好時(shí)壞的情況,那么為什么會(huì)出現(xiàn)這種現(xiàn)象呢?
原因分析從錯(cuò)誤日志中我們可以發(fā)現(xiàn)一個(gè)非常關(guān)鍵的信息: I/O error on GET request for "http://192.168.5.103:9010/config-server/test"。
報(bào)錯(cuò)說(shuō)明了微服務(wù)檢查配置中心獲取配置的連接是否暢通的時(shí)候出現(xiàn)了連接不上的情況,但是這個(gè)鏈接信息其實(shí)并不是當(dāng)前配置中心的地址,而是我們遷移之前的配置中心的地址。
從健康檢查的實(shí)現(xiàn)源碼ConfigServerHealthIndicator中為入口去分析和調(diào)試,我們可以解答上面現(xiàn)象的兩個(gè)疑問(wèn):
@Override protected void doHealthCheck(Builder builder) throws Exception { PropertySource> propertySource = getPropertySource(); builder.up(); if (propertySource instanceof CompositePropertySource) { List為什么會(huì)健康檢查訪問(wèn)的還是老的配置中心地址?sources = new ArrayList<>(); for (PropertySource> ps : ((CompositePropertySource) propertySource).getPropertySources()) { sources.add(ps.getName()); } builder.withDetail("propertySources", sources); } else if (propertySource!=null) { builder.withDetail("propertySources", propertySource.toString()); } else { builder.unknown().withDetail("error", "no property sources located"); } } private PropertySource> getPropertySource() { long accessTime = System.currentTimeMillis(); if (isCacheStale(accessTime)) { this.lastAccess = accessTime; this.cached = locator.locate(this.environment); } return this.cached; }
真正導(dǎo)致健康檢查失敗的語(yǔ)句是getPropertySource中的 this.cached = locator.locate(this.environment);而這里的具體實(shí)現(xiàn)在org.springframework.cloud.config.client.ConfigServicePropertySourceLocator類中,具體實(shí)現(xiàn)如下:
@Override @Retryable(interceptor = "configServerRetryInterceptor") public org.springframework.core.env.PropertySource> locate( org.springframework.core.env.Environment environment) { ConfigClientProperties properties = this.defaultProperties.override(environment); CompositePropertySource composite = new CompositePropertySource("configService"); RestTemplate restTemplate = this.restTemplate == null ? getSecureRestTemplate(properties) : this.restTemplate; Exception error = null; String errorBody = null; logger.info("Fetching config from server at: " + properties.getRawUri()); ... }
可以看到,真正去訪問(wèn)的地址是直接從properties.getRawUri()獲取的,它已經(jīng)是一個(gè)固化的值,而不是通過(guò)服務(wù)發(fā)現(xiàn)機(jī)制來(lái)動(dòng)態(tài)獲取的。這就導(dǎo)致了當(dāng)我們把配置中心做了遷移,或者直接部署在容器中出現(xiàn)重啟的時(shí)候,IP發(fā)生變化,而所有的微服務(wù)還以為訪問(wèn)的是原來(lái)的配置中心地址,就會(huì)出現(xiàn)健康檢查失敗的問(wèn)題,導(dǎo)致服務(wù)不可用的現(xiàn)象。
為什么健康檢查時(shí)好時(shí)壞?上面的問(wèn)題會(huì)導(dǎo)致健康檢查失敗,但是這個(gè)服務(wù)并不是一直都不好,而是間斷性的出現(xiàn)不健康。這主要還是健康檢查時(shí)間中的機(jī)制導(dǎo)致,這里可以具體看ConfigServerHealthIndicator的getPropertySource函數(shù),該方法執(zhí)行的時(shí)候中間并不是每一次檢查都會(huì)去訪問(wèn)配置中心(執(zhí)行locator.locate(this.environment)方法),因此客戶端的健康檢查并不會(huì)每次都健康檢查失敗,從而出現(xiàn)了微服務(wù)健康檢查時(shí)好時(shí)壞的情況。
如何解決該問(wèn)題目前也在官方的issue中被提出,還處于open狀態(tài),具體可見(jiàn):https://github.com/spring-clo...
由于該問(wèn)題目前并沒(méi)有得到解決,雖然提交了一個(gè)PR,但是還有待完善以及提供一些測(cè)試,原本想完全處理好之后再寫(xiě)一篇文章,但是發(fā)現(xiàn)最近不少問(wèn)過(guò)類似問(wèn)題,所以索性先寫(xiě)一篇文章,提醒一下用戶以及給一些相關(guān)的建議。
當(dāng)前版本上不太容易通過(guò)擴(kuò)展的方式去解決的這個(gè)問(wèn)題,所以大家可以變通的去避免這個(gè)問(wèn)題:
部署在虛擬機(jī)上而不是容器上,避免IP的變動(dòng)
可以考慮關(guān)閉微服務(wù)隊(duì)config客戶端的健康檢查,增加參數(shù)management.health.config.enabled=false;但是這個(gè)操作有一個(gè)弊端,雖然遷移不會(huì)引發(fā)服務(wù)時(shí)好時(shí)壞的問(wèn)題了,但是如果有動(dòng)態(tài)配置刷新需求,如果遷移了配置中心,刷新配置操作也是會(huì)失敗的。
補(bǔ)充一個(gè)網(wǎng)友給的思路,通過(guò)內(nèi)部域名來(lái)綁定配置中心,而不是ip地址
文章版權(quán)歸作者所有,未經(jīng)允許請(qǐng)勿轉(zhuǎn)載,若此文章存在違規(guī)行為,您可以聯(lián)系管理員刪除。
轉(zhuǎn)載請(qǐng)注明本文地址:http://www.ezyhdfw.cn/yun/69498.html
摘要:服務(wù)網(wǎng)關(guān)服務(wù)網(wǎng)關(guān)涵蓋的功能包括路由,鑒權(quán),限流,熔斷,降級(jí)等對(duì)入站請(qǐng)求的統(tǒng)一攔截處理。具體可以進(jìn)一步劃分為外部網(wǎng)關(guān)面向互聯(lián)網(wǎng)和內(nèi)部網(wǎng)關(guān)面向服務(wù)內(nèi)部管理。應(yīng)用服務(wù)應(yīng)用服務(wù)是企業(yè)業(yè)務(wù)核心。到此實(shí)際上已經(jīng)完成服務(wù)遷移工作。 導(dǎo)讀 Spring Cloud基于Spring Boot開(kāi)發(fā),提供一套完整的微服務(wù)解決方案,具體包括服務(wù)注冊(cè)與發(fā)現(xiàn),配置中心,全鏈路監(jiān)控,API...
摘要:而微服務(wù)架構(gòu)能否成功實(shí)踐,利用各種工具解決潛在問(wèn)題是關(guān)鍵。因此,微服務(wù)本身可以通過(guò)庫(kù)和運(yùn)行時(shí)代理解決客戶端服務(wù)發(fā)現(xiàn)負(fù)載均衡配置更新統(tǒng)計(jì)跟蹤等。與相比,解決了更廣的微服務(wù)架構(gòu)問(wèn)題。和處理了不同范圍的微服務(wù)架構(gòu)技術(shù)點(diǎn),而且是用了不同的方法。 Spring Cloud vs. Kubernetes,誰(shuí)才是部署微服務(wù)的最佳拍檔? Spring Cloud和Kubernetes都聲稱自己是開(kāi)發(fā)和...
摘要:授權(quán)框架使第三方應(yīng)用程序來(lái)獲取對(duì)服務(wù)的有限訪問(wèn)機(jī)會(huì)。無(wú)論是通過(guò)編排資源所有者和服務(wù)之間的交互批準(zhǔn)的資源所有者,或通過(guò)允許第三方應(yīng)用程序來(lái)獲取自己的訪問(wèn)權(quán)限。 SpringCloud打造微服務(wù)平臺(tái)--概覽 簡(jiǎn)述 SpringCloud是什么 Spring Boot和SpringCloud是什么關(guān)系 Spring Boot是Spring的一套快速WEB開(kāi)發(fā)的腳手架,可建立獨(dú)立的Sprin...
閱讀 3371·2021-11-18 10:02
閱讀 3521·2021-10-11 10:58
閱讀 3448·2021-09-24 09:47
閱讀 1218·2021-09-22 15:21
閱讀 4185·2021-09-10 11:10
閱讀 3324·2021-09-03 10:28
閱讀 1805·2019-08-30 15:45
閱讀 2223·2019-08-30 14:22