Yahoo! Cloud Serving Benchmark (YCSB) is a benchmark primarily targeting systems with key-value like interface. It first populates the target system with key-value pairs and then measures the system with Insert, Update, Read, and Scan operations.
Some databases include a network stack to communicate with remote clients and some do not. Suppose the database includes a network stack. 1. With large key-value pairs, the bottleneck is likely to be network bandwidth. 2. With small key-value pairs, the bottleneck is likely to be the CPU to process network packets. 3. One can batch multiple small key-value pairs in a single request. In this case, the bottleneck is likely to be either the network bandwidth or the in-memory engine.
The turning points of course depend on both the hardware setting and software implemention. High network bandwidth is likely to push bottleneck to server CPUs. RDMA technique can reduce CPU utilization for network communication and push bottleneck to other components like index lookup.
High skewness (i.e., high zipf coefficient value) will cause YCSB to frequently access a small number of hot keys.
Key-value stores have two common mechanisms to handle concurrent requests to the same key-value pair. First, some allow any thread/process to access any key-value pairs and use locking or versioning to perform concurrency control, like a SQL database. In this case, high skewness will cause high contentions. Second, some shard data into multiple key ranges and tie each thread/process to certain key ranges, which eliminates concurrent access to the same key-value pair. In this case, there is no contention at run time and high skewness will cause load imbalance instead.
Both high contention and high load imbalance are typically bad for performance, but which one is better depends on the implementation of the key-value store and the workload.
The effect of this parameter is much like the number of warehouses in TPC-C. If the number is large so that data does not fit into DRAM, of course that will significantly change the performance profile. Even if all data fit into DRAM, larger number of key-value pairs may affect the speed of a tree-like index, page table lookup efficiency, etc.
In theory, contention level may be affected by the number of KVs. In practice, however, we find such impact is not significant. The reason is as follows: For workload with low skewness (e.g., uniform distribution), although changing the number of KVs will change the contention level significantly, the contention level will not reach a level that matters to throughput, unless we use, say, tens of key-value pairs. For highly skewed workload (e.g. zipf 0.99), the contention level is high, but it does not change proportionally with the number of key-value pairs. For example, with zipf 0.99, changing from one million KVs to 100 million KVs will only decrease the frequency to access the hottest key from 6.5% to 4.8% and decrease the frequency to access the hottest 32 keys from 27% to 20%. Therefore, in both cases,the change of contention level caused by the change of the number of KVs usually has no significant impact on performance.
When considering a single operation in DRAM, read and update usually do not have much difference in performce, since they both involve the same index lookup and DRAM read/write speed is not much different (PMEM is a different story). Insert is typically more expensive since it needs to allocate memory and may need to evict an existing key-value pair.
When considering concurrent operations, Read operations do not contend but Write operations may contend, so more Writes may cause a performance degradation.
In addition, in a replicated and/or persistent key-value store, since a Write needs to be replicated and/or persisted and a Read usually does not, more Writes may cause a performance degradation as well.
If we use large KVs, the bottleneck is likely to be network bandwidth; if we use small KVs, the bottleneck is likely to be the packet rate of the network stack; batching requests or increasing contention level will move the bottleneck to the in-memory engine but in different ways: batching with low contention is likely to stress test key lookup, DRAM speed, etc; high contention is likely to stress test concurrency control, if any.
If either network bandwidth or packet rate is the bottleneck, a higher Write ratio will incur more packets if the system needs to replicate Writes but has no significant impact if the system does not replicate Writes.
If contention is the bottleneck, a higher Write ratio or a higher skewness will incur a higher contention. The number of KVs does not have a significant impact.