Why You Should Benchmark With Production Services: Redis Edition

25 January 2017 on . 6 minutes to read

Learn How to Benchmark Redis with Ruby

Intro

There comes a time to benchmark your application and its services. However, if it’s not done properly, then your results can very misleading. In this post, I’ll throw some numbers at you to demonstrate the massive differences between benchmarking locally, on production, and locally pointing at a cloud resource. The numbers for each set of tests are different by an order of magnitude, demonstrating the importance of using the proper setup.

Setup

Each test will benchmark adding 1, 10, 100, 1000 or 10000 elements to a redis set via sadd. Redis version is 3.2.6. The cloud is powered by the free tier Redis Cloud 30MB instance. The tests run on Heroku are done from a Ruby on Rails 5 app, with Ruby 2.3.3.

Test Runs

Local Console, Local Redis

Benchmark.ips do |x|
  tests = [1,10,100,1000,10000]
  local = Redis.new
  tests.each do |t|
    num_elements = t
    arr = (1..num_elements).to_a

    x.report("cloud sadd: #{t} elements"){cloud.sadd "products_test:sadd", arr}
    x.report("local sadd: #{t} elements"){local.sadd "products_test:sadd", arr}
  end
  x.compare!
end

Comparison:
local sadd: 1 elements:       7328.4 i/s
local sadd: 10 elements:      6690.6 i/s - same-ish: difference falls within error
local sadd: 100 elements:     3454.8 i/s - 2.12x  slower
local sadd: 1000 elements:    574.7 i/s - 12.75x  slower
local sadd: 10000 elements:   53.5 i/s - 137.08x  slower

Local Console, Redis Cloud

# Local, pointing at Redis Cloud
Benchmark.ips do |x|
  x.time = 30
  tests = [1,10,100,1000,10000]
  cloud = Redis.new(url: ENV['CLOUD_REDIS_URL'])

  tests.each do |t|
    num_elements = t
    arr = (1..num_elements).to_a
    x.report("cloud sadd: #{t} elements"){cloud.sadd "products_test:sadd", arr}
  end
  x.compare!
end

cloud sadd: 1 elements:       36.9 i/s - 198.37x  slower
cloud sadd: 10 elements:      35.9 i/s - 203.93x  slower
cloud sadd: 100 elements:     34.7 i/s - 211.11x  slower
cloud sadd: 1000 elements:    24.4 i/s - 300.05x  slower
cloud sadd: 10000 elements:   2.2 i/s - 3268.97x  slower

Local Console, Redis Cloud

#Heroku 1x Rails Console, pointing at same Redis Cloud instance
Benchmark.ips do |x|
  x.time = 30
  tests = [1,10,100,1000,10000]
  cloud = Redis.new(url: ENV['CLOUD_REDIS_URL'])

  tests.each do |t|
    num_elements = t
    arr = (1..num_elements).to_a
    x.report("heroku sadd: #{t} elements"){cloud.sadd "products_test:sadd", arr}
  end
  x.compare!
end

Comparison:
heroku sadd: 1 elements:       352.3 i/s
heroku sadd: 10 elements:      311.2 i/s - same-ish: difference falls within error
heroku sadd: 100 elements:     255.9 i/s - same-ish: difference falls within error
heroku sadd: 1000 elements:    100.9 i/s - 3.49x  slower
heroku sadd: 10000 elements:   17.8 i/s - 19.82x  slower

Overall Standings

local sadd: 1 elements: 7328.4 i/s local sadd: 10 elements: 6690.6 i/s - same-ish: difference falls within error local sadd: 100 elements: 3454.8 i/s - 2.12x slower local sadd: 1000 elements: 574.7 i/s - 12.75x slower heroku sadd: 1 elements: 352.3 i/s - 20.80x slower heroku sadd: 10 elements: 311.2 i/s - 23.55x slower heroku sadd: 100 elements: 255.9 i/s - 28.64x slower heroku sadd: 1000 elements: 100.9 i/s - 72.63x slower local sadd: 10000 elements: 53.5 i/s - 137.08x slower cloud sadd: 1 elements: 36.9 i/s - 198.37x slower cloud sadd: 10 elements: 35.9 i/s - 203.93x slower cloud sadd: 100 elements: 34.7 i/s - 211.11x slower cloud sadd: 1000 elements: 24.4 i/s - 300.05x slower heroku sadd: 10000 elements: 17.8 i/s - 411.71x slower cloud sadd: 10000 elements: 2.2 i/s - 3268.97x slower

Take Away

Once you introduce calls to services outside your application, the numbers rapidly change. Single element operations locally vs heroku are a 20x difference. If you have a view which relies on redis to populate it (say fragment caching or retrieving stats), that could be the difference between spending 100ms and 2000ms just on Redis calls. What seems like a quick render to you locally would be unacceptably slow on production. Though this specific test is anecdotal, it handily illustrates why you need to do benching on a production like setup and thoroughly evaluate performance in order to keep your application snappy and efficient.

I’ll do a more in depth analysis on these results later, especially regarding the implications of the falloff between 1k and 10k element inserts.

If your application needs analysis or you’d like to see how it holds up under load and if it can handle some big spikes in requests to various services, drop me an email; I’ll be glad to work with you to ensure your systems are running as quick as possible.


If you enjoy having free time and the peace of mind that a professional is on your side, then you’d love to have me work on your project.

Contact or view a list of available services to see how I’ll make your life better, easier and bring satisfaction back into you running your business.