1. Background
Skywalking7.0 is completed and other functions are normal.
Try to create a profile task task, the response data is empty, and the creation fails.
Check the skywalking log and found that it was the failure caused by the es write error.vim skywalking-oap-server.log
2. Error analysis:
The following is the key information:
Exception while fetching data (/createTask) :
Elasticsearch exception [
type=illegal_argument_exception,
reason=no write index is defined for alias [skywalking_profile_task]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index
]
org.elasticsearch.ElasticsearchStatusException:
Elasticsearch exception [
type=illegal_argument_exception,
reason=no write index is defined for alias [skywalking_profile_task]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index
]
According to the error report, there are two possible reasons:
- is_write_index=false, the index is disabled for writing and the writing fails
- The alias points to multiple indexes, none of which can be written
Check the es data and find that it is the second type.
3. Solution
Clear all logs and
curl -XDELETE http://localhost:9200/skywalking*
restart skywalking
./bin/startup.sh
4. Detailed log
2020-06-16 20:14:09,050 - graphql.execution.SimpleDataFetcherExceptionHandler - 23 [qtp888378046-47] WARN [] - Exception while fetching data (/createTask) : Elasticsearch exception [type=illegal_argument_exception, reason=no write index is defined for alias [skywalking_profile_task]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=illegal_argument_exception, reason=no write index is defined for alias [skywalking_profile_task]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1706) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1683) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1446) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:821) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.forceInsert(ElasticSearch7Client.java:207) ~[storage-elasticsearch7-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.NoneStreamEsDAO.insert(NoneStreamEsDAO.java:45) ~[storage-elasticsearch-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.analysis.worker.NoneStreamPersistentWorker.in(NoneStreamPersistentWorker.java:49) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.analysis.worker.NoneStreamingProcessor.in(NoneStreamingProcessor.java:56) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.profile.ProfileTaskMutationService.createTask(ProfileTaskMutationService.java:100) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.query.graphql.resolver.ProfileMutation.createProfileTask(ProfileMutation.java:51) ~[query-graphql-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.query.graphql.resolver.ProfileMutationMethodAccess.invoke(Unknown Source) ~[reflectasm-1.11.7.jar:7.0.0]
at com.coxautodev.graphql.tools.MethodFieldResolverDataFetcher.get(MethodFieldResolver.kt:147) ~[graphql-java-tools-5.2.3.jar:?]
at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:227) ~[graphql-java-8.0.jar:?]
at graphql.execution.ExecutionStrategy.resolveField(ExecutionStrategy.java:170) ~[graphql-java-8.0.jar:?]
at graphql.execution.AsyncSerialExecutionStrategy.lambda$execute$1(AsyncSerialExecutionStrategy.java:42) ~[graphql-java-8.0.jar:?]
at graphql.execution.Async.eachSequentiallyImpl(Async.java:72) [graphql-java-8.0.jar:?]
at graphql.execution.Async.eachSequentially(Async.java:62) [graphql-java-8.0.jar:?]
at graphql.execution.AsyncSerialExecutionStrategy.execute(AsyncSerialExecutionStrategy.java:37) [graphql-java-8.0.jar:?]
at graphql.execution.Execution.executeOperation(Execution.java:158) [graphql-java-8.0.jar:?]
at graphql.execution.Execution.execute(Execution.java:100) [graphql-java-8.0.jar:?]
at graphql.GraphQL.execute(GraphQL.java:558) [graphql-java-8.0.jar:?]
at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:500) [graphql-java-8.0.jar:?]
at graphql.GraphQL.executeAsync(GraphQL.java:470) [graphql-java-8.0.jar:?]
at graphql.GraphQL.execute(GraphQL.java:401) [graphql-java-8.0.jar:?]
at org.apache.skywalking.oap.query.graphql.GraphQLQueryHandler.execute(GraphQLQueryHandler.java:93) [query-graphql-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.query.graphql.GraphQLQueryHandler.doPost(GraphQLQueryHandler.java:83) [query-graphql-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.library.server.jetty.JettyJsonHandler.doPost(JettyJsonHandler.java:59) [library-server-7.0.0.jar:7.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.apache.skywalking.oap.server.library.server.jetty.JettyJsonHandler.service(JettyJsonHandler.java:107) [library-server-7.0.0.jar:7.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.apache.skywalking.oap.server.library.server.jetty.JettyJsonHandler.service(JettyJsonHandler.java:112) [library-server-7.0.0.jar:7.0.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) [jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:543) [jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481) [jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.Server.handle(Server.java:564) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://localhost:9200], URI [/skywalking_profile_task/_doc/1592309649047_2?refresh=true&timeout=1m], status line [HTTP/1.1 400 Bad Request]
{
"error":{
"root_cause":[{
"type":"illegal_argument_exception","reason":"no write index is defined for alias [skywalking_profile_task]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index"}],"type":"illegal_argument_exception","reason":"no write index is defined for alias [skywalking_profile_task]. The write index may be explicitly disabled using is_write_index=false or the alias points to multiple indices without one being designated as a write index"},"status":400}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:260) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:238) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[elasticsearch-rest-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1433) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:821) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.forceInsert(ElasticSearch7Client.java:207) ~[storage-elasticsearch7-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.NoneStreamEsDAO.insert(NoneStreamEsDAO.java:45) ~[storage-elasticsearch-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.analysis.worker.NoneStreamPersistentWorker.in(NoneStreamPersistentWorker.java:49) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.analysis.worker.NoneStreamingProcessor.in(NoneStreamingProcessor.java:56) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.core.profile.ProfileTaskMutationService.createTask(ProfileTaskMutationService.java:100) ~[server-core-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.query.graphql.resolver.ProfileMutation.createProfileTask(ProfileMutation.java:51) ~[query-graphql-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.query.graphql.resolver.ProfileMutationMethodAccess.invoke(Unknown Source) ~[reflectasm-1.11.7.jar:7.0.0]
at com.coxautodev.graphql.tools.MethodFieldResolverDataFetcher.get(MethodFieldResolver.kt:147) ~[graphql-java-tools-5.2.3.jar:?]
at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:227) ~[graphql-java-8.0.jar:?]
at graphql.execution.ExecutionStrategy.resolveField(ExecutionStrategy.java:170) ~[graphql-java-8.0.jar:?]
at graphql.execution.AsyncSerialExecutionStrategy.lambda$execute$1(AsyncSerialExecutionStrategy.java:42) ~[graphql-java-8.0.jar:?]
at graphql.execution.Async.eachSequentiallyImpl(Async.java:72) [graphql-java-8.0.jar:?]
at graphql.execution.Async.eachSequentially(Async.java:62) [graphql-java-8.0.jar:?]
at graphql.execution.AsyncSerialExecutionStrategy.execute(AsyncSerialExecutionStrategy.java:37) [graphql-java-8.0.jar:?]
at graphql.execution.Execution.executeOperation(Execution.java:158) [graphql-java-8.0.jar:?]
at graphql.execution.Execution.execute(Execution.java:100) [graphql-java-8.0.jar:?]
at graphql.GraphQL.execute(GraphQL.java:558) [graphql-java-8.0.jar:?]
at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:500) [graphql-java-8.0.jar:?]
at graphql.GraphQL.executeAsync(GraphQL.java:470) [graphql-java-8.0.jar:?]
at graphql.GraphQL.execute(GraphQL.java:401) [graphql-java-8.0.jar:?]
at org.apache.skywalking.oap.query.graphql.GraphQLQueryHandler.execute(GraphQLQueryHandler.java:93) [query-graphql-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.query.graphql.GraphQLQueryHandler.doPost(GraphQLQueryHandler.java:83) [query-graphql-plugin-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.library.server.jetty.JettyJsonHandler.doPost(JettyJsonHandler.java:59) [library-server-7.0.0.jar:7.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.apache.skywalking.oap.server.library.server.jetty.JettyJsonHandler.service(JettyJsonHandler.java:107) [library-server-7.0.0.jar:7.0.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.apache.skywalking.oap.server.library.server.jetty.JettyJsonHandler.service(JettyJsonHandler.java:112) [library-server-7.0.0.jar:7.0.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) [jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:543) [jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481) [jetty-servlet-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.Server.handle(Server.java:564) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) [jetty-util-9.4.2.v20170220.jar:9.4.2.v20170220]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Reference link: https://www.cnblogs.com/Neeo/articles/10897280.html