Benchmarking etcd v2.2.0

Performance measures for etcd v2.2.0

Physical Machines

GCE n1-highcpu-2 machine type

  • 1x dedicated local SSD mounted as etcd data directory
  • 1x dedicated slow disk for the OS
  • 1.8 GB memory
  • 2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Bootstrap another machine, outside of the etcd cluster, and run the hey HTTP benchmark tool with a connection reuse patch to send requests to each etcd cluster member. See the benchmark instructions for the patch and the steps to reproduce our procedures.

The performance is calculated through results of 100 benchmark rounds.

Performance

Single Key Read Performance

key size in bytes number of clients target etcd server average read QPS read QPS stddev average 90th Percentile Latency (ms) latency stddev
64 1 leader only 2303 200 0.49 0.06
64 64 leader only 15048 685 7.60 0.46
64 256 leader only 14508 434 29.76 1.05
256 1 leader only 2162 214 0.52 0.06
256 64 leader only 14789 792 7.69 0.48
256 256 leader only 14424 512 29.92 1.42
64 64 all servers 45752 2048 2.47 0.14
64 256 all servers 46592 1273 10.14 0.59
256 64 all servers 45332 1847 2.48 0.12
256 256 all servers 46485 1340 10.18 0.74

Single Key Write Performance

key size in bytes number of clients target etcd server average write QPS write QPS stddev average 90th Percentile Latency (ms) latency stddev
64 1 leader only 55 4 24.51 13.26
64 64 leader only 2139 125 35.23 3.40
64 256 leader only 4581 581 70.53 10.22
256 1 leader only 56 4 22.37 4.33
256 64 leader only 2052 151 36.83 4.20
256 256 leader only 4442 560 71.59 10.03
64 64 all servers 1625 85 58.51 5.14
64 256 all servers 4461 298 89.47 36.48
256 64 all servers 1599 94 60.11 6.43
256 256 all servers 4315 193 88.98 7.01

Performance Changes

  • Because etcd now records metrics for each API call, read QPS performance seems to see a minor decrease in most scenarios. This minimal performance impact was judged a reasonable investment for the breadth of monitoring and debugging information returned.

  • Write QPS to cluster leaders seems to be increased by a small margin. This is because the main loop and entry apply loops were decoupled in the etcd raft logic, eliminating several blocks between them.

  • Write QPS to all members seems to be increased by a significant margin, because followers now receive the latest commit index sooner, and commit proposals more quickly.


Last modified June 14, 2021: Renaming content/en/next folder to content/en/v3.5. Updating redirects, links, and config as needed. (#363) (138926b)