• Hints
  • Hinted Handoff
  • Application of Hints
  • Storage of Hints on Disk
  • Hints for Timed Out Write Requests
  • Configuring Hints
  • Configuring Hints at Runtime with
  • Make Hints Play Faster at Runtime
  • Allow a Node to be Down Longer at Runtime
  • Monitoring Hint Delivery

    Hints

    anti-entropy repair does. partitions data across the cluster using consistent hashing, and then replicates keys to multiple nodes along the hash ring. To guarantee availability, all replicas of a key can accept mutations without consensus, but this means it is possible for some replicas to accept a mutation while others do not. When this happens an inconsistency is introduced. Hints are one of the three ways, in addition to read-repair and full/incremental anti-entropy repair, that Cassandra implements the eventual consistency guarantee that all updates are eventually received by all replicas. Hints, like read-repair, are best effort and not an alternative to performing full repair, but they do help reduce the duration of inconsistency between replicas in practice.

    Hinted Handoff

    Hinted handoff is the process by which Cassandra applies hints to unavailable nodes. Consistency Level LOCAL_QUORUM against a keyspace with Replication Factor of 3. Normally the client sends the mutation to a single coordinator, who then sends the mutation to all three replicas, and when two of the three replicas acknowledge the mutation the coordinator responds successfully to the client. If a replica node is unavailable, however, the coordinator stores a hint locally to the filesystem for later application. New hints will be retained for up to max_hint_windowin_ms of downtime (defaults to 3 h). If the unavailable replica does return to the cluster before the window expires, the coordinator applies any pending hinted mutations against the replica to ensure that eventual consistency is maintained. Hinted Handoff in Action
  • t0): The write is sent by the client, and the coordinator sends it to the three replicas. Unfortunately replica_2 is restarting and cannot receive the mutation.
  • t1): The client receives a quorum acknowledgement from the coordinator. At this point the client believe the write to be durable and visible to reads (which it is).
  • t2): After the write timeout (default 2s), the coordinator decides that replica_2 is unavailable and stores a hint to its local disk.
  • t3): Later, when replica_2 starts back up it sends a gossip message to all nodes, including the coordinator.
  • t4): The coordinator replays hints including the missed mutation against replica_2. If the node does not return in time, the destination replica will be permanently out of sync until either read-repair or full/incremental anti-entropy repair propagates the mutation.

    Application of Hints

    Hints are streamed in bulk, a segment at a time, to the target replica node and the target node replays them locally. After the target node has replayed a segment it deletes the segment and receives the next segment. This continues until all hints are drained.

    Storage of Hints on Disk

    $CASSANDRA_HOME/data/hints directory. A hint includes a hint id, the target replica node on which the mutation is meant to be stored, the serialized mutation (stored as a blob) that couldn’t be delivered to the replica node, the mutation timestamp, and the Cassandra version used to serialize the mutation. By default hints are compressed using LZ4Compressor. Multiple hints are appended to the same hints file. Since hints contain the original unmodified mutation timestamp, hint application is idempotent and cannot overwrite a future mutation.

    Hints for Timed Out Write Requests

    write_request_timeout setting in cassandra.yaml configures the timeout for write requests.
    1. write_request_timeout: 2000ms
    write_request_timeout is 10 ms.

    Configuring Hints

    cassandra.yaml configuration file provides several settings for configuring hints: Table 1. Settings for Hints

    nodetool

    nodetool provides several commands for configuring hints or getting hints related information. The nodetool commands override the corresponding settings if any in cassandra.yaml for the node running the command. Table 2. Nodetool Commands for Hints

    Make Hints Play Faster at Runtime

    1024 kbps handoff throttle is conservative for most modern networks, and it is entirely possible that in a simple node restart you may accumulate many gigabytes hints that may take hours to play back. For example if you are ingesting 100 Mbps of data per node, a single 10 minute long restart will create 10 minutes * (100 megabit / second) ~= 7 GiB of data which at (1024 KiB / second) would take 7.5 GiB / (1024 KiB / second) = 2.03 hours to play back. The exact math depends on the load balancing strategy (round robin is better than token aware), number of tokens per node (more tokens is better than fewer), and naturally the cluster’s write rate, but regardless you may find yourself wanting to increase this throttle at runtime. hinted_handoff_throttle dynamically via the nodetool sethintedhandoffthrottlekb command.

    Allow a Node to be Down Longer at Runtime

    max_hint_window, (default of three hours), but the hardware and data itself will still be accessible. In such a case you may consider raising the max_hint_window dynamically via the nodetool setmaxhintwindow command added in Cassandra 4.0 (CASSANDRA-11720). This will instruct Cassandra to continue holding hints for the down endpoint for a longer amount of time. max_hint_window setting in cassandra.yaml followed by a rolling restart.

    Monitoring Hint Delivery

    CASSANDRA-13234). Hinted Handoff <handoff-metrics> and Hints Service <hintsservice-metrics> metrics.