Decommission a Data-Center In Cassandra

Share us

In this post, we will discuss how to decommission a Data center in Cassandra.

Prerequisites to Decommission a Data Center:-

  • Make sure no clients are still writing to any nodes in the data center.
  • All traffic should move to other data center.
  • Comment all cron-job/script which restart Cassandra and repair ahead of time.
  • Check whether repair is running or not, if running then stop.

Step 1) Run repair in the datacenter with nodetool repair –dc [dc_name]

This ensures that all data is propagated from the data-center being decommissioned. And with this, we don’t need to run repair on each node. It restrict the repair to the local data-center, use the -dc option followed by the name of the data-center. Issue the command from a node in the data-center you specify in the command. If you issue the command from different data-center, Cassandra returns an error. Do not use-dc and -pr together to repair only a local data-center.

nohup nodetool repair -dc NY &

[2014-07-24 21:59:55,326] Nothing to repair for keyspace ‘system’
[2014-07-24 21:59:55,617] Starting repair command #2, repairing 490 ranges
for keyspace system_traces (seq=true, full=true)
[2014-07-24 22:23:14,299] Repair session 323b9490-137e-11e4-88e3-c972e09793ca
for range (820981369067266915,822627736366088177] finished
[2014-07-24 22:23:14,320] Repair session 38496a61-137e-11e4-88e3-c972e09793ca
for range (2506042417712465541,2515941262699962473] finished

Below is the output from nohup.out.

[2020-04-23 03:50:08,152] Replication factor is 1. No repair is needed for keyspace ‘cycling’
[2020-04-23 03:50:08,168] Replication factor is 1. No repair is needed for keyspace ‘system_auth’
[2020-04-23 03:50:08,200] Starting repair command #2 (6f1e44f0-853f-11ea-8de8-5954b2c6b259), repairing keyspace system_traces with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [NY], hosts: [], # of ranges: 512, pull repair: false)
[2020-04-23 03:50:10,147] Repair session 6f4c59d0-853f-11ea-8de8-5954b2c6b259 for range
[2020-04-23 03:50:10,184] Repair completed successfully
[2020-04-23 03:50:10,185] Repair command #2 finished in 1 second

We can also check the system.log from the nodes where we ran repair command, it will show repair is taking place only on IP addresses which are in NY DC.

INFO [AntiEntropyStage:1] 2014-07-24 22:23:10,708 RepairSession.java:171 [repair #16499ef0-1381-11e4-88e3-c972e09793ca] Received merkle tree
for sessions from /192.168.2.101
INFO [RepairJobTask:1] 2014-07-24 22:23:10,740 RepairJob.java:145 [repair #16499ef0-1381-11e4-88e3-c972e09793ca] requesting merkle trees for events (to [/192.168.2.103, /192.168.2.101])

Below sample results of system.log is from our environment.

INFO [AntiEntropyStage:1] 2020-04-23 03:47:53,767 RepairSession.java:180 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] Received merkle tree for sessions from /10.235.112.238
INFO [AntiEntropyStage:1] 2020-04-23 03:47:53,794 RepairSession.java:180 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] Received merkle tree for sessions from /10.235.113.2
INFO [Repair#1:1] 2020-04-23 03:47:53,794 RepairJob.java:172 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] Requesting merkle trees for events (to [/10.235.112.238, sandbox306p/10.235.113.2])
INFO [RepairJobTask:1] 2020-04-23 03:47:53,803 SyncTask.java:66 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] Endpoints /10.235.112.238 and sandbox306p/10.235.113.2 are consistent for sessions
INFO [RepairJobTask:2] 2020-04-23 03:47:53,805 RepairJob.java:143 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] sessions is fully synced
INFO [AntiEntropyStage:1] 2020-04-23 03:47:53,885 RepairSession.java:180 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] Received merkle tree for events from /10.235.112.238
INFO [AntiEntropyStage:1] 2020-04-23 03:47:53,929 RepairSession.java:180 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] Received merkle tree for events from /10.235.113.2
INFO [RepairJobTask:1] 2020-04-23 03:47:53,930 SyncTask.java:66 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] Endpoints /10.235.112.238 and sandbox306p/10.235.113.2 are consistent for events
INFO [RepairJobTask:2] 2020-04-23 03:47:53,930 RepairJob.java:143 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] events is fully synced
INFO [RepairJobTask:2] 2020-04-23 03:47:53,932 RepairSession.java:270 – [repair #1e59b1d0-853f-11ea-8de8-5954b2c6b259] Session completed successfully

We can monitor the repair status from below command-

  • compactionstats
  • netstats
  • tpstats

nodetool compactionstats
pending tasks: 0
Active compaction remaining time : n/a

nodetool netstats
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 3459
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed
Commands n/a 0 1010910
Responses n/a 0 9126881

nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 99 0 0

Repair-Task 0 0 1 0 0
RequestResponseStage 0 0 2078 0 0

Once repair completed successfully i.e no Pending for Repair-Task(from tpstats) or from other two commands, we can proceed for next step.

Step 2) Change all keyspaces so they no longer reference the data-center being removed.

Login to cassandra cqlsh shell and alter the keyspace with below and remove the data-center which we want to remove.

ALTER KEYSPACE [keyspace_name(s)] WITH replication = {   ‘class’: ‘NetworkTopologyStrategy’,   ‘LONDON’: ‘3’ };

Step 3) Remove the nodes of the Datacenter (which we want to remove) one by one.

Login to the server(which we want to decommission) and execute nodetool decommission.

nodetool decommission

It won’t stream any data to remaining node as the datacenter it belongs contain no data from the keyspace definition. And it will complete quickly.

To monitor the decommission status check with nodetool netstats and status.

nodetool netstats
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 3459
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed
Commands n/a 0 1010910
Responses n/a 0 9126881

nodetool status

Datacenter: London

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
— Address Load Tokens Owns (effective) Host ID Rack
UN 10.235.112.238 338 KiB 256 0.0% 7b527224-efaa-4b43-88f5-d2a6ad683824 r1
UN 10.235.113.2 369.59 KiB 256 0.0% 987305dc-c9f4-4fed-9b70-e55a48701ea1 r1

Step 4)Stop the Cassandra process on each node, remove cronjob and auto-start script to monitor the Cassandra process.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.