The official explanation of nodetool cleanup is: When Cassandra expands a node to a new cluster, some data will be copied to the new node, and then the data belonging to the new node still exists on those old nodes. It will not actively delete itself. So after each expansion, you need to perform a cleanup operation on each old node.
According to practice:
the cleanup operation may generate some temporary files during the execution process, and cause a slight increase in IO utilization and a slight increase in machine load.
Cleanup is a single-threaded operation, which has little overall impact and does not need to be closed.
Cleanup is a stand-alone behavior and does not need to limit node streaming.
As long as the business is not particularly busy, you can safely execute it node by node.
After performing cleanup, the node data is significantly reduced.
cleanup can be executed at any time after the expansion. You can also specify to cleanup only a certain keyspace or columnFamily
nodetool -h 192.168.1.1 [keyspace [cfnames]]