In my previous post, we laid out a few scenarios in which two (or more!) separate Rails applications need to access the same data.
Normally, shared data + lots of data === lots of expensive and slow database queries. One solution to this problem is to create a very special database that warehouses specific slices, or "views" of our shared data, indexed mindfully to maximize query time. In this post, we'll explore an alternative solution––using Redis as a shared data store between two (or more!) different Rails applications.
So, let's say your house has been invaded by an alien parasite that multiplies and disguises itself as your beloved friends and family.
Said parasite implants only pleasant memories of themselves in your brain. The only way to identify who is a horrifying parasite and who is a real friend or family member is to go through all of your memories of this individual and determine if there are any negative ones. If so, don't kill them. Otherwise, definitely kill them.
Luckily, we long ago built an application that stores information about our loved ones and our memories of them. Our main application is the Personal Memory Bank––a Rails app that stores each and every one of your friends and family members and your associated memories of this person. A LovedOne has many memories and a Memory belongs to a LovedOne. In order to help us stave off this alien invasion, we quickly build a secondary application, Alien Parasite Hunter––a Rails app that allows you the user to identify loved ones who are really alien parasites and "eliminate" them.
In order for our Alien Parasite Hunter app to determine whether or not an individual loved one is a parasite, it will have to ask our main application, the Personal Memory Bank.
The database that backs our Personal Memory Bank app is super large (all your memories of everyone from ever are in there, remember). It will take way to long to query the whole database for every memory of a given loved one to determine whether or not you have any negative memories of them. We don't have that kind of time––the alien parasites are multiplying really fast.
Here's what we'll do. Whenever a negative memory of someone gets created and stored in our Personal Memory Bank database, we'll store that memory in Redis, under a key of this loved one's ID + the date on which the memory occured. Then, when our Alien Parasite Hunter app needs to determine whether or not this loved one is really an alien parasite, it will simply ask Redis if it has this loved one's ID + date of a given memory stored as a key, indicating that we do have negative memories of them. This will be a much faster look up then having Alien Parasite Hunter tell Personal Memory Bank to search every single memory we have of a given loved one looking for any negative ones.
Let's get started!
Setting Up The Redis Client
We'll say that when the Alien Parasite Hunter app needs to determine if you have a negative memory of a given loved one, it will be provided with the ID of that loved one, and the date of the memory in question in order to perform the look-up. Our main application, Personal Memory Bank, will need to store any negative memories identifiable by their #loved_one_id and #date.
In order for the Personal Memory Bank app to be able to store data in Redis, and in order for our secondary application, the Alien Parasite Hunter, to be able to fetch data from Redis, we need to give both apps a way to communicate with Redis.
For this, we'll use the the redis-rb gem, a Redis client for Ruby. We'll initialize Redis clients in both apps and connect them to the same Redis instance, running locally on port 6379. We'll start by setting up the client in Personal Memory Bank.
# personal_memory_bank/config/redis.yml development: server: localhost port: 6379 password: test: server: localhost port: 6379 password:
Our redis.yml file stores our Redis configuration info. Important: If you don't have Redis installed and running on your machine, check out this resource to get yourself up and running.
Next up, we'll create an initializer for our client.
# personal_memory_bank/config/initializers/redis.rb
yml = YAML.load_file(File.join(Rails.root, "config", "redis.yml"))
config = trends_redis_yml_file["#{Rails.env}"]
namespace = "negative_memories".to_sym
$redis_connection = Redis.new(config)
$negative_memories_redis = Redis::Namespace.new(namespace, redis: redis_connection)
We accomplish a few things here:
- Load the YAML file that contains our Redis configuration
- Initialize a new instance of our Redis client, connected to the Redis server.
- Initialize a new instance of
Redis::Namespaceand connect it to Redis via our main Redis client. - Store this instance in a global variable,
$negative_memories_redis, to that we can interact with it throughout our application.
Redis Namespace
Redis is a simple but powerful key/value store. It can hold lots of data. Instead of dumping all of our loved-one-memory-related keys into the top-level of our Redis store, let's group, or namespace, them under a designation of "negative_memories". Essentially, we want all the keys we add to Redis to look something like this:
"negative_memories:loved_one_#{loved_one_id}:#{date_of_memory}"
This will keep the data that we put into Redis nice and organized, and make it really easy for us to query and clean out negative-memory-related-data from our Redis server as needed.
While it would be simple enough to create this namespace every time we add a new key to Redis, and use this namespace every time we query a key from Redis, that can get repetitive. Instead, we use the Redis Namespace gem to create a client that is namespaced for us. We can interact with our $negative_memories_redis client just like we would with any other Redis client, and it will automatically add that top level namespace to the key/value pair that is stored in Redis.
This way, we can be lazy, which we love, and set/get keys in the following format, without having to manually set the namespace:
"loved_one_#{loved_one_id}:#{date_of_memory}"
As a result, our top-level Redis client, $redis_connection, will have this key stored like this:
"negative_memories:loved_one_789:2017-01-01"
Now that our namespaced client is up and running, we're ready to actually store data in Redis.
Storing Data in Redis
When do we want to add a key/value pair to our Redis data store? Whenever a new memory record gets created and inserted into the Personal Memory Bank DB, we want to check if the memory is positive or negative. If it is negative, we want to to store a reference to it in Redis. It would be helpful for our Alien Parasite Hunter app to be able to quickly ask Redis for any negative memories associated to a given loved one that occurred on a given date. So, we'll store key/value pairs in Redis identifying the loved one by a key of their ID, combined with the date of the negative memory, alongside a value of the data describing that memory. Something like this*:
{
"loved_one_#{loved_id_id}:2017-01-01" => {id: 112, date: "2017-03-14", description: "that time you ate all my food at dinner after you finished your own meal.", negative: true}
}
{
"loved_one_#{loved_id_id}:2017-01-02" => {id: 112, date: "2017-03-14", description: "that time you purposefully sneezed all of my phone when you were sick.", negative: true}
}
*Real negative memories courtesy of my beloved sister.
Let's build a callback on our Memory model that inserts data into Redis if the newly created memory is a negative one.
Assume that an instance of Memory has the following attributes: #loved_one_id, #date, #negative where loved_one_id refers to the LovedOne record to which the memory is associated, the date refers to the date on which the memory occurred and negative is a boolean value indicating whether or not the memory was a good one.
class Memory
belongs_to :loved_one
after_create :store_in_redis, if: :negative
...
def store_in_redis
$negative_memories_redis.set(redis_key, redis_value)
end
private
def redis_key
"loved_one_#{self.loved_one.id}:#{self.date}"
end
def redis_value
self.attributes.to_json
end
end
Here, we create a key to store in Redis using the combination of the memory's #loved_one_id and #date and under that key we store a value of the memory instance's attributes, transformed into JSON.
Now that we are successfully storing data in Redis, let's teach our secondary application how to request that data from Redis.
Retrieving Data From Redis
Recall that our Alien Parasite Hunter App works like this:
Given the ID of a loved one and a date, return any negative memories. We set up our Redis keys to make this query as easy as possible. Alien Parasite Hunter simply needs to ask Redis: "do you have any keys that look like this:"
"loved_one_#{loved_one_id}:{date}"
Let's set up the Redis client for our secondary app. This process will exactly mirror the one we went through to connect our primary application to our Redis store.
# alient_parasite_hunter/config/redis.yml development: server: localhost port: 6379 password: test: server: localhost port: 6379 password:
Note that we are using the same configuration as we did for our Personal Memory Bank app. This is very important. We are connecting both applications to the same Redis server.
Let's set up our Alien Parasite Hunter app with its very own Redis client in an initializer:
# alient_parasite_hunter/config/initializers/redis.rb
yml = YAML.load_file(File.join(Rails.root, "config", "redis.yml"))
config = trends_redis_yml_file["#{Rails.env}"]
namespace = "negative_memories".to_sym
$redis_connection = Redis.new(config)
$negative_memories_redis = Redis::Namespace.new(namespace, redis: redis_connection)
What's that? It looks exactly the same as our Redis client initialization in our main application? You are really good at this, super perceptive.
We want to follow the same namespace pattern, giving our secondary application its very own client that will communicate with the same Redis server, in the same manner as our other app.
Now that we're all set up, let's teach Alien Parasite Hunter how to ask Redis for a specific key. Right now, I'm not too concerned with where this responsibility fits into the larger scope or flow of our application. Let's just build a simple service to determine what, if any, negative memories we have for a given loved one and date.
#alien_parasite_hunter/app/services/negative_memory_detector.rb
class NegativeMemoryDetector
def self.execute(loved_one_id, date)
detect_negative_memories_for(loved_one_id, date)
end
def self.detect_negative_memories_for(loved_one_id, date)
memories = $negative_memories_redis.get(key(loved_one_id, date))
JSON.parse(memories) if memories
end
def self.key(loved_one_id, date)
"loved_one_#{loved_one_id}:#{date}"
end
end
When we invoke our NegativeMemoryDetector, it will return the parsed negative memories if any, or it will return nil.
real_loved_one = LovedOne.find_by(name: "Zoe")
alien_parasite = LovedOne.find_by(name: "Mrs. Refridgerator")
NegativeMemoryDetector.execute(real_loved_one.id, "2017-01-1")
# => {id: 112, date: "2017-03-14", description: "that time you purposefully sneezed on my phone when you were sick.", negative: true}
NegativeMemoryDetector.execute(alien_parasite.id, "2017-01-1")
# => nil
Conclusion
This has been a simple and somewhat weird introduction to sharing Redis across Rails applications. If you like what you've read to far, consider subscribing below. I won't spam you––just send out an update when a new post have been published.