When you need to reassign a partition to another broker or brokers, the kafka-reassign-partitions tool comes in handy. It's particularly effective for reassigning a small number of topics. However, I noticed some limitations when working with a cluster that had over 5,000 topics, especially after expanding it.
To accomplish the reassignment, one must execute three distinct commands and create two JSON files for each reassignment. Now, imagine doing this for 5,000 topics! While it's possible to construct a single JSON file with a list of all the topics and use the `--throttle` option to control bandwidth, this approach could potentially disrupt the environment. What I wanted was to reassign topics one at a time and retain control over which topics were being reassigned.
This was precisely the challenge that my colleague Bruno Costa and I faced last week. Our objective was to reassign one topic at a time, minimising any potential impact on the running producers and consumers, all while automating the process. Consequently, we developed a Bash script that streamlines the redistribution of Kafka partitions. This script can prove invaluable for various purposes, including cluster maintenance, scaling, or optimisation.
#!/bin/bash
set +xoe pipefail
#################################################################
#
# This script uses Kafka's partition reassignment tool to move partitions across brokers.
# Kafka Docs reference:
# https://kafka.apache.org/26/documentation.html#basic_ops_cluster_expansion
#
# Usage:
#
# Create a text file (kafka-topics.txt) with the list of Kafka topics to reassign
# chmod +x ./kafka-reassign.sh
# ./kafka-reassign.sh
#################################################################
output_dir="./output"
[ -d $output_dir ] || mkdir $output_dir
banner()
{
echo "+------------------------------------------+"
printf "| %-40s |\n" "`date`"
printf "| %s \n" "${@//[$'\t\r\n']}"
echo "+------------------------------------------+"
}
input="kafka-topics.txt"
while IFS= read -r line
do
banner "Starting Topic ${line//[$'\t\r\n']}"
echo -e "\nReplacing topic in JSON template"
sed -e "s/<TOPIC>/${line//[$'\t\r\n']}/g" template.json > $output_dir/topics.json
echo -e "\nGenerating and cleaning reassign.json"
kafka-reassign-partitions --bootstrap-server ${BOOTSTRAP_SERVER} --topics-to-move-json-file $output_dir/topics.json --broker-list "0,1,2,3,4" --generate | awk '/Proposed partition reassignment configuration/{c=NR+2}(NR<=c){print}' | awk NR\>1 > $output_dir/reassign.json
echo -e "\nExecuting reassignment"
kafka-reassign-partitions --bootstrap-server ${BOOTSTRAP_SERVER} --reassignment-json-file $output_dir/reassign.json --execute
echo -e "\nVerifying reassignment"
kafka-reassign-partitions --bootstrap-server ${BOOTSTRAP_SERVER} --reassignment-json-file $output_dir/reassign.json --verify
count=$(kafka-reassign-partitions --bootstrap-server ${BOOTSTRAP_SERVER} --reassignment-json-file $output_dir/reassign.json --verify | grep -o "still in progress" | wc -l)
echo "partition reassignment in progress: $count"
while [ $count -gt 0 ]
do
sleep 5
count=$(kafka-reassign-partitions --bootstrap-server ${BOOTSTRAP_SERVER} --reassignment-json-file $output_dir/reassign.json --verify | grep -o "still in progress" | wc -l)
echo "partition reassignment in progress: $count"
done
done < "$input"
You can download this script from here.
Here's a breakdown of what each part of the script does:
1. `set +xoe pipefail`: These are shell options. `+x` turns off the debugging output, `o` ensures that a script exits if any command returns a non-zero status, and `e` makes sure that if any part of a pipeline fails, the whole pipeline fails.
2. The script includes a brief description of its purpose and usage. It expects a file named `kafka-topics.txt` to contain a list of Kafka topics to reassign.
3. output_dir="./output": This sets the variable `output_dir` to the directory where the sc
ript will store temporary files and logs.
4. The `banner()` function prints a timestamped banner to the console to indicate when a new topic reassignment is starting.
5. The script reads the list of Kafka topics from the `kafka-topics.txt` file, one by one, and performs the following steps for each topic:
- Creates a JSON file named `topics.json` based on a template `template.json` by replacing `<TOPIC>` with the actual topic name.
- Uses the `kafka-reassign-partitions` tool to generate a reassignment plan and stores it in `reassign.json`. This plan specifies how Kafka partitions should be moved between brokers.
- Executes the reassignment, actually moving the partitions according to the plan.
- Verifies the reassignment status and waits until it's completed.
The script repeats these steps for each topic listed in `kafka-topics.txt`. It also keeps checking the reassignment status until it's no longer "in progress" before moving on to the next topic.
Results
Using this script, we successfully reassigned 4,923 topics in just under three days, all without causing any disruption to producers and consumers.
Conclusion
This marks the first version of the script, and we will continue to enhance it as needed. For more comprehensive details on the `kafka-reassign-partitions` tool and how to tailor its usage to your specific scenario, I recommend consulting the Kafka documentation. This will enable you to make further refinements and optimizations as you see fit.
Comments