The root cause of elasticsearch's 403 problem is insufficient disk space
I don’t have much experience in using es for the project, except for a few tables to be synchronized, and all logs are written in it. (The average index is about 500M)
Finally one day, an online synchronization error 403 was reported. I checked Google and said that the disk space was insufficient. I saw that there was still 3g, and deleted the older log index. According to the read_only_allow_delete setting from Google , the problem was temporarily solved. .
Two months later, a 403 error occurred again. At first glance, there was only 3g left. I got a lazy illness that day, so I still delete the log setting read_only_allow_delete.
It is another 403 after almost two months. As more and more places are used for es, the scope of the impact of this situation is more serious, and I want to cure this problem.
We later added disks for online services, so I wanted to make good use of this disk for capacity expansion.
1. I googled Ubuntu extend disk space, and I found out what lvextend and vextend various expansion solutions were.
2. Then do the migration. Check the data path in the es configuration, there can be more than one, configure multiple paths in the test server (keep the original path, add the path of another disk), and then restart, the unexpected is reported The following exception:
java.lang.ExceptionInInitializerError: null
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_212]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_212]
at org.elasticsearch.painless.lookup.PainlessLookupBuilder.addPainlessClass(PainlessLookupBuilder.java:170) ~[?:?]
at org.elasticsearch.painless.lookup.PainlessLookupBuilder.build(PainlessLookupBuilder.java:686) ~[?:?]
at org.elasticsearch.painless.PainlessScriptEngine.<init>(PainlessScriptEngine.java:109) ~[?:?]
at org.elasticsearch.painless.PainlessPlugin.getScriptEngine(PainlessPlugin.java:89) ~[?:?]
at org.elasticsearch.script.ScriptModule.<init>(ScriptModule.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.node.Node.<init>(Node.java:339) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.1.jar:6.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.lang.ArrayIndexOutOfBoundsException: 4
at java.time.chrono.JapaneseEra.<clinit>(JapaneseEra.java:179) ~[?:1.8.0_212]
... 19 moreOnly someone raised this issue on the official forum, and it seems to be caused by Japan’s change of currency last month. To update the version. I thought of gg in case the restart fails on the line, and I am in a second.
3. Expansion of the disk is not possible. Modifying the data path is also dangerous. Is it nowhere to go? Suddenly, I had an inspiration and created an 8G swapfile before adding the hard disk . Can I just kill it ?
Simple and fool-proof! I immediately created a 16G swapfile on the new disk and enabled it. The 8Gswapfile before swapoff...
......
......
After a long wait (more than half an hour)
Finally there is a response! Then free -h, make sure to unbind, carefully rm off the 8G swapfile
Hit df again, and see that there is 11G of disk space left. It's successful!
These spaces can store more indexes, which is great!