The disk of a Cassandra node is broken. There are two cases. One is that the node can still start normally. Another is that the node cannot be started.
Case 1: The node can still start normally
1. Replace the bad disk. If you don't have a new disk to replace, you can directly comment out the bad disk in cassandra.yaml
2. Start cassandra. If an error is reported during startup, saying that the keyspace cannot be found, then you should use the solution in the second case.
3. Use nodetool repair to repair the lost data of the node.
The second case: the node cannot be started normally
1. Replace the bad disk. If you don't have a new disk to replace, you can directly comment out the bad disk in cassandra.yaml
2. Execute on a normal node:
$ nodetool ring | grep ip_address_of_node | awk '{print $NF ","}' | xargs
To get the tokens of the broken node, configure them (separated by commas) into the initial_token option of cassandra.yaml.
3. Set configuration items in cassandra.yaml:
auto_bootstrap: false
The official documentation for this step is missing, refer to
https://issues.apache.org/jira/browse/CASSANDRA-11365
4. Delete the system directory under all your data disks.
rm -fr /mnt1/cassandra/data/system
rm -fr /mnt2/cassandra/data/system
。。。。
5. Start cassandra. If an error is reported during the startup process, it is normal to say that the schema does not exist, and the system library will be automatically rebuilt. As long as the node can join the cluster normally, it is normal.
6. Also use nodetool repair to repair the lost data of the node.