Interpreting Cassandra repair logs and leveraging the OpsCenter repair service

Introduction to repairs and the Repair Service

Cassandra repairs consist of comparing data from between replica nodes, identifying inconsistencies, and streaming the latest value for mismatched data. We can't compare an entire cassandra database value by value so we create Merkle trees to identify inconsistencies and then we stream them.

Repairs are expensive; CPU is needed to generate the Merkle trees and networking / io is needed to stream missing data. Usually repairs also trigger lots of compactions if they have not been run for a while (especially when there's a lot of inconsistent data and leveled / date tiered compaction strategies are being used).

The OpsCenter repair service splits up the repair job into lots of little slices (256 per table) and runs them around the clock, turning a heavy, manual, weekly operation into an automatic, constant job. Clusters will see higher consistent cpu utilization / load when the repair service is on instead of a big spike once per week.

Steps in a repair

Each repair session will be identified by a UUID (for example #0d4544b0-8fc9-11e5-a498-4b9679ec178d). The following are the logs from a healthy repair. Notice all the messages are INFO messages, there are no WARN or ERROR.

Repair sessions have repair jobs and jobs which occur for each table in the session.

Repeat for every table:
1) RepairJob.java Request merkle trees
2) RepairSession.java Receive merkle trees
3) Differencer.java Check for inconsistencies
4) OptionalStreamingRepairTask.java Stream differences--if any
5) RepairSession.java is fully synced
6) StorageService.java Repair session for range (,] finished

Summarizing repair logs - clean run

To group the tasks use the following bash foo:

$ cat system.log| grep "11ff9870-8fc9-11e5-a498-4b9679ec178d" | sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Source/'|sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Target/' | awk '{ split($3,a,":"); $2=a[0] ; $3=""; $4=""; print }'|uniq -c

In this case there was no streaming and all the jobs complete successfully for the range.

   1 INFO    RepairSession.java:260 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] new session: will sync /Source, /Target1, /Target2, ...... on range (4393112290973329820,4394202908924102592] for OpsCenter.[rollups86400, events_timeline, rollups7200, events, bestpractice_results, backup_reports, settings, rollups60, rollups300, pdps]
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for rollups86400 (to [/Source, /Target, ...])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for rollups86400 from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for rollups86400
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] rollups86400 is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for events_timeline (to [/Source, /Target, ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for events_timeline from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for events_timeline
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] events_timeline is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for rollups7200 (to [/Source, /Target, ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for rollups7200 from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for rollups7200
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] rollups7200 is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for events (to [/Source, /Target,  ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for events from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for events
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] events is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for bestpractice_results (to [/Source, /Target,  ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for bestpractice_results from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for bestpractice_results
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] bestpractice_results is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for backup_reports (to [/Source, /Target,  ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for backup_reports from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for backup_reports
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] backup_reports is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for settings (to [/Source, /Target,  ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for settings from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for settings
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] settings is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for rollups60 (to [/Source, /Target,  ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for rollups60 from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for rollups60
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] rollups60 is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for rollups300 (to [/Source, /Target,  ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for rollups300 from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for rollups300
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] rollups300 is fully synced
   1 INFO    RepairJob.java:163 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for pdps (to [/Source, /Target,  ... ])
   9 INFO    RepairSession.java:171 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for pdps from /Source
  36 INFO    Differencer.java:67 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for pdps
   1 INFO    RepairSession.java:237 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] pdps is fully synced
   1 INFO    RepairSession.java:299 - [repair #11ff9870-8fc9-11e5-a498-4b9679ec178d] session completed successfully
   1 INFO    StorageService.java:3001 - Repair session 11ff9870-8fc9-11e5-a498-4b9679ec178d for range (4393112290973329820,4394202908924102592] finished

Summarizing repair logs - errors

Now let's look at a repair session with some errors.

$ cat system.log| grep "0fb1b0d0-8fc9-11e5-a498-4b9679ec178d" | sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Source/'|sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Target/' | awk '{ split($3,a,":"); $2=a[0] ; $3=""; $4=""; print }'|uniq -c

   1 INFO    RepairSession.java:260 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] new session: will sync /Source, /Target, ... on range (4393112290973329820,4394202908924102592] for keyspace.[table1, table2, ...]
   1 INFO    RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for table1 (to [/Source, /Target, ...])
   9 INFO    RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for table1 from /Source
  36 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for datasources
   1 INFO    RepairSession.java:237 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] datasources is fully synced
   1 INFO    RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for table1 (to [/Source, /Target, ...])
   9 INFO    RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for table1 from /Source
  36 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex_error
   1 INFO    RepairSession.java:237 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] tablex_error is fully synced
   1 INFO    RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for tablex (to [/Source, /Target, ])
   9 INFO    RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source
   9 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex
   2 INFO    Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex
   1 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex
   3 INFO    Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex
   1 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex
   3 INFO    Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex
   1 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex
   1 INFO    Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex
   3 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex
   3 INFO    Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex
   1 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex
   6 INFO    Differencer.java:74 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target have 1 range(s) out of sync for tablex
   2 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex
  11 INFO    StreamingRepairTask.java:81 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Forwarding streaming repair of 1 ranges to /Source (to be streamed with /Target)
   2 INFO    StreamingRepairTask.java:68 - [streaming task #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Performing streaming repair of 1 ranges with /Source
   1 INFO    StreamingRepairTask.java:81 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Forwarding streaming repair of 1 ranges to /Source (to be streamed with /Target)
   4 INFO    StreamingRepairTask.java:68 - [streaming task #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Performing streaming repair of 1 ranges with /Source
   1 INFO    RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for tablex (to [/Source, /Target, ...])
   1 INFO    RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source
   1 INFO    StreamingRepairTask.java:96 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] streaming task succeed, returning response to /Target
   1 INFO    RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source
   2 INFO    StreamingRepairTask.java:96 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] streaming task succeed, returning response to /Target
   7 INFO    RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source
  36 INFO    Differencer.java:67 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Endpoints /Source and /Target are consistent for tablex
   1 INFO    RepairSession.java:237 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] tablex is fully synced (1 remaining column family to sync for this session)
   1 INFO    StreamingRepairTask.java:96 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] streaming task succeed, returning response to /Target
   1 INFO    RepairJob.java:163 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] requesting merkle trees for tablex_processed (to [/Source, /Target, ...])
   2 INFO    StreamingRepairTask.java:96 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] streaming task succeed, returning response to /Target
   2 INFO    RepairSession.java:171 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Received merkle tree for tablex from /Source
   1 ERROR    RepairSession.java:303 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] session completed with the following error
   1 org.apache.cassandra.exceptions.RepairException:    keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target
   1 java.lang.RuntimeException:    on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target
   1 Caused    #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target
   1 ERROR    StorageService.java:3008 - Repair session 0fb1b0d0-8fc9-11e5-a498-4b9679ec178d for range (4393112290973329820,4394202908924102592] failed with error org.apache.cassandra.exceptions.RepairException: [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target
   1 java.util.concurrent.ExecutionException:    #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target
   1 Caused    [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target
   1 Caused    #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target

Notice the error message

ERROR    StorageService.java:3008 - Repair session 0fb1b0d0-8fc9-11e5-a498-4b9679ec178d for range (4393112290973329820,4394202908924102592] failed with error org.apache.cassandra.exceptions.RepairException: [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d on keyspace/tablex, (4393112290973329820,4394202908924102592]] Sync failed between /Source and /Target

Streaming for the range (4393112290973329820, 4394202908924102592] between nodes Source and Target failed and caused the repair session to fail. The OpsCenter repair service will retry this session again (up to a configurable threshold) on failure.

What happened?

The failure may have been due to a networking problem, an sstable corruption, etc. To get more information, we can 1) check our logs at the Target repair node for additional errors and 2) run the slice again and see if it works.

Check the target

Checking the logs at the target, there are no failures associated with this repair session which means there must have been a problem on the stream receiving end.

$ cat system.log| grep "0fb1b0d0-8fc9-11e5-a498-4b9679ec178d" | sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Source/'|sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/Target/' | awk '{ split($3,a,":"); $2=a[0] ; $3=""; $4=""; print }'|uniq -c
   1 INFO    Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
   1 INFO    Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
   1 INFO    Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
   2 INFO    StreamingRepairTask.java:68 - [streaming task #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Performing streaming repair of 1 ranges with /Source
   1 INFO    Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex
   1 INFO    Validator.java:257 - [repair #0fb1b0d0-8fc9-11e5-a498-4b9679ec178d] Sending completed merkle tree to /Source for keyspace/tablex

Run the slice

In this case we can run:

nodetool repair -par -st 4393112290973329820 -et 4394202908924102592

here is the result (notice the networking problem was temporary and now the repair slice succeeds!):

]# nodetool repair -par -st 4393112290973329820 -et 4394202908924102592
[2015-11-23 21:36:44,138] Starting repair command #1, repairing 1 ranges for keyspace dse_perf (parallelism=PARALLEL, full=true)
[2015-11-23 21:36:46,086] Repair session 4aa7a290-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished
[2015-11-23 21:36:46,086] Repair command #1 finished
[2015-11-23 21:36:46,095] Starting repair command #2, repairing 1 ranges for keyspace keyspace (parallelism=PARALLEL, full=true)
[2015-11-23 21:36:47,967] Repair session 4bc6f540-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished
[2015-11-23 21:36:47,968] Repair command #2 finished
[2015-11-23 21:36:47,979] Nothing to repair for keyspace 'system'
[2015-11-23 21:36:47,985] Starting repair command #3, repairing 1 ranges for keyspace keyspace (parallelism=PARALLEL, full=true)
[2015-11-23 21:36:53,100] Repair session 4ce92e20-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished
[2015-11-23 21:36:53,102] Repair command #3 finished
[2015-11-23 21:36:53,112] Starting repair command #4, repairing 1 ranges for keyspace dse_system (parallelism=PARALLEL, full=true)
[2015-11-23 21:36:53,979] Repair session 4fe50900-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished
[2015-11-23 21:36:53,979] Repair command #4 finished
[2015-11-23 21:36:53,987] Starting repair command #5, repairing 1 ranges for keyspace keyspace (parallelism=PARALLEL, full=true)
[2015-11-23 21:36:58,390] Repair session 507477c0-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished
[2015-11-23 21:36:58,390] Repair command #5 finished
[2015-11-23 21:36:58,399] Starting repair command #6, repairing 1 ranges for keyspace OpsCenter (parallelism=PARALLEL, full=true)
[2015-11-23 21:37:11,448] Repair session 531931f0-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished
[2015-11-23 21:37:11,448] Repair command #6 finished
[2015-11-23 21:37:11,458] Starting repair command #7, repairing 1 ranges for keyspace system_traces (parallelism=PARALLEL, full=true)
[2015-11-23 21:37:11,878] Repair session 5ae2e890-922a-11e5-ae1c-4b5d0d7247d3 for range (4393112290973329820,4394202908924102592] finished
[2015-11-23 21:37:11,878] Repair command #7 finished

There are a couple of known streaming issues to keep an eye on.
CASSANDRA-10791 and CASSANDRA-10012 which may cause streaming errors. If you are on an affected version, upgrade. If you encounter a reproducible streaming error and can't find the particular stack trace in an existing jira, open a new one.

A corruption

In a different repair session we see a different repair error, this time it refers to a specific sstable.

WARN  [STREAM-IN-/x.x.x.x] 2015-11-20 20:55:45,529  StreamSession.java:625 - [Stream #114cea40-8fc9-11e5-ae1c-4b5d0d7247d3] Retrying for following error  
java.lang.RuntimeException: Last written key DecoratedKey(4393675392884570836, 000b313032333432333334303000000343504600) >= current key DecoratedKey(918610503192973903, 00102941d767895c11e5b5dfaadf7c0db7b80000087765627369746573ff) writing into /cassandra/data/mykeyspace/table1-bd0990407e6711e5a4cae7fe9b813e81/mykeyspace-table1-tmp-ka-7890-Data.db  
    at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:164) ~[cassandra-all-2.1.11.908.jar:2.1.11.908]
    at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:261) ~[cassandra-all-2.1.11.908.jar:2.1.11.908]
    at org.apache.cassandra.streaming.StreamReader.writeRow(StreamReader.java:168) ~[cassandra-all-2.1.11.908.jar:2.1.11.908]
    at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:89) ~[cassandra-all-2.1.11.908.jar:2.1.11.908]
    at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48) [cassandra-all-2.1.11.908.jar:2.1.11.908]
    at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) [cassandra-all-2.1.11.908.jar:2.1.11.908]
    at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56) [cassandra-all-2.1.11.908.jar:2.1.11.908]
    at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250) [cassandra-all-2.1.11.908.jar:2.1.11.908]
    at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]

When there is an sstable corruption (due to disk failures or possibly even a bug), the procedure is to run nodetool scrub on the sstable which will correct the corruption.

In this case the issue was due to CASSANDRA-9133 which was fixed in 2.0.15 so an upgrade was also in order!

Summary, stay repaired!

Repairs can fail due to networking issues or sstable corruptions. The former are usually short lived and will go away on retry; the latter are more rare and require admin intervention in the form of running nodetool scrub.

Remember, repairs aren't things you run when your cluster is broken; they are a mandatory anti-entropy administrative task (like an oil change) that keeps your cluster healthy. In many cases, running a repair on an unhealthy cluster will just make things worse.

Hopefully this post will help you understand how repairs work, how to troubleshoot the repair service, and keep your production cluster happy and your database boring. Enjoy!