Caching 101: A Beginner's Guide
Caching is a very important strategy in computing and web development that involves storing copies of data or computational results in a temporary storage area, called a cache.
Cache doesn’t exist only in the software world, it exists in many places in our day-to-day life. Our brain caches frequently needed stuff in the hippocampus like where we keep our vehicle keys & etc.
In the kitchen, we keep frequently used stuff like salt, sugar, etc in nearby kitchen cabinet in smaller quantities.
This storage (Cache) allows for faster access to data & more efficient processing. Before we understand What is Caching and how to implement it, let’s understand why we need to use it.
Why Do We Need Caching?
An Application with a smaller user base always feels fast. But the moment you hit a decent user base, your application becomes slow.
Based on several studies conducted every second delay in your page leads to 11% fewer page views, a 16% decrease in customer satisfaction & 7% loss in conversions.
More than 10 years ago, Amazon found that every 100ms of latency cost them 1% in sales.
If your application is not performant you are losing your users/customers and revenue. There are many ways to make your application faster and one of the best ones is Caching.
I hope you are aware of The Pareto principle. Roughly 80% of consequences come from 20% of causes (the "vital few"). It applies to your Application as well.
Most of the users access the same data most of the time which can be Cached so that you don’t need to query from the database each time or run some business logic each time a user requests for that same data that was requested by another user a couple of milliseconds ago.
Because of Caching we can improve Application performance, Reduce network load & better user experience.
What is Cache?
According to Wikipedia, “A cache is a block of memory that stores data that's likely to be used again.”
I am guessing so far what I mentioned gave you a fair idea about What is Cache. Let’s understand from a Software application perspective.
Usually, a simple web application looks like this
Of course, in between load balancer & many things will come but just for simplicity's sake, we are sticking with it.
When a client sends a request to the application it goes to the server (our service), which then retrieves data from the database and sends it back to the client. In between any logic needs to be executed that will be done at our service/server level.
When there is a huge surge in users then it leads to slowness at the Database level because data needs to be retrieved from its storage which leads to performance degradation.
When we add Cache to this system it helps in overcoming this problem. When something is requested by a user our service checks the Cache if it is present then it gives it back to the user if not then retrieves it from the database, stores it in the cache (for future use) and gives it back to the user.
You might get doubt like even cache also stores somewhere and gives it back right why it is fast. As mentioned in the definition Cache stores the data in the memory(RAM) retrieval from it is much faster than accessing from storage systems(Hard disk).
Cache is like a Hashmap & Storage is like an ArrayList.
A couple of things to remember about Cache:
The cache uses memory so you can’t keep everything there because it’s expensive with respect to cost so we store only frequently accessed. If you don’t do this efficiently we will lose this advantage.
Data in the Cache is stored in memory so the moment we restart it that data is gone. Of course, there are ways to persist it but keeping this in mind is important to design a system better.
Where can Cache be used?
It can be used in multiple places. This pic will give you some idea.
Client Side Cache
Browsers usually keep the images after it downloaded at the time of page loaded. On the next page, if the same image is needed then it pics from the cache. Open any good website then open the Network tab in browser developer tools and go to the next page on that website you can observe that a bunch of images, fonts & etc picked from the memory cache.
By the way, we store some of the things like the user’s name, role & etc in Session level or cookies so that we can utilise the data instead of requesting from the server each time. Data at the Session level is usually stored in Local storage or Session storage or IndexedDB or cookies.
CDN Caching
CDNs are a geographically distributed network of servers (Edge Servers). These servers deliver content from the nearest server to the user's location. It reduces the time taken to access the content, resulting in a faster-loading website. Websites like Netflix use CDN caching extensively to give a great experience to the users.
CDN caching helps B2C sites especially. You can serve static pages, images, videos and many static assets from CDN which will give a great experience to the users by loading the website faster.
Application Server Cache
we can kind of say it is In-memory Caching. We either add this cache in in-memory alongside the application server or a separate server/service like Redis and Memcached.
If your application is very popular then you will use Distributed cache. In the Distributed cache each node(server) will have a part of the whole cache space and then using the consistent hashing function each request can be routed to where the cache request can be found.
Here we store frequently accessed data blocks, query results, metadata and session-specific information to reduce disk reads and query execution time, which results in faster response times.
Most of the developers spend a good amount of time on this.
Other caches are Database Caching, Web Server Caching, API Gateway Caching, etc.
Cache writing strategies
Writing data to the caching should be planned well based on your application needs, or else we will put unnecessary load on the Database or Cache which results in performance issues.
These are some of the popular writing strategies.
Write-around cache: In this strategy, we just write stuff in the Database directly. We don’t touch the cache at the time of write. In this strategy, Cache won’t be overwhelmed by write operations. Whenever the user wants some data application first reads from the cache if it’s not present it is called “Cache miss”, Then the application reads it from the Database and updates it in the Cache. Many people introduce Caching in their applications using this strategy.
Write-through cache: In this strategy, the Application writes the data in Cache & Database simultaneously. Write will be slow but read will be always fast and data will be consistent (same stays in Cache & Database).
Write-behind(Write-back) cache: In this strategy, the Application writes the data in the Cache. Data will be stored in the Database by Cache asynchronously (periodically or on certain events like eviction events). This strategy makes sense for write-heavy applications and ok with eventual consistency.
Cache reading strategies
Cache Aside: In this strategy, whenever the user wants some data application first reads from the cache if it’s not present it is called “Cache miss”, Then the application reads it from the Database and updates it in the Cache. This is a common read strategy. That’s why used this strategy at the time of making writing strategy images.
Read Through: In this strategy, the Application reads from the cache if the data is there it returns, if not there Cache gets it from the Database, sets it in the Cache and gives it back.
Cache eviction and Invalidation strategies
Removing data from the Cache is important or else we will face 2 problems.
Server memory space will get occupied slowly.
Stale data (Eventually every data gets old because sometimes we update data by SQL queries or some other way where we forget to update the cache).
Based on the strategy, the items will be removed whenever the server needs space. These are some of the popular strategies:
Least Recently Used (LRU): The most commonly used strategy. The title says it all. In this policy, it removes the items you didn’t use for a long time. Let me give you an example. You bought popcorn 3 months back by thinking you would eat it when you watch a movie at your home but you didn’t touch it at all. When the server needs space it removes these kinds of items in this strategy.
Least Frequently Used (LFU): It sounds similar to LRU but there is a difference. In this policy, it removes the items you use once in a blue moon. For example, Fancy dinnerware or Air Fryer (I bought it thinking of making healthy food but I use it once in 6 months so eventually my mom put it outside of the kitchen 😜).
First In, First Out (FIFO): The name says it all. It’s like a queue. The oldest item will get removed first when space is needed.
Window TinyLFU (W-TinyLFU): It is somewhat the best of both worlds (LRU & LFU). It keeps the most relevant cache entries in the cache by considering both how recent and how frequently the data has been accessed.
Time To Live (TTL): We put the expiry date (Time to live) with each time. Whenever that time touches item will be removed. It is like when any item hits the expiry date no matter how frequently used & favourite we will remove right like that. If you want your data regularly updated in Cache then this is a go-to strategy.
A Simple Example Application (How to implement it?)
We will take a simple use case which is the White labelling use case. An Organisation can customise the theme. If they don’t, the default theme will be considered for that organisation.
Multiple users belong to one organisation. When users log in and open any page, the application needs to show the data based on that organisation theme. Usually, these values won’t be changed often so there is no point in querying them on every request so it is good to cache them.
Whenever theme details are needed we will check the Cache if it’s there we will serve from it if not (It is called a Cache miss) we will query the database, store it in the Cache and return it to the user/FrontEnd/Originator.
If any one of the users of a company changes the theme value before we store it in the Database we will simply evict that organisation’s theme in the Cache so that new changes will be picked from the Database and reflected to all the users in that organisation.
I made a simple example application in Python. You can find the code here.
If you open the home page and choose a user then based on that user’s organisation the colour scheme and content will be changed. The theme data is picked from Cahe. If the theme values are changed then they will be deleted from the Cache. Here I used the Write-around strategy for writing & Cache Aside strategy for reading.
To create the required data you can check this SQL file. We have 3 tables(organisations, users, themes).
We used these Python packages for this application:
Flask
Flask-SQLAlchemy
redis
psycopg2-binary
python-dotenv
I am using the Flask framework for this application. For the UI part will use jinja2 templating, this comes by default with Flask so no need to install it.
Going to use SQLAlchemy for the DB part, Flask-SQLAlchemy is one of the best packages to handle SQLAlchemy stuff in the Flask application.
redis package is Redis Python Client. psycopg2-binary package is the most popular PostgreSQL database adapter for the Python programming language. I am using PostgreSQL if you want to use some other SQL database and don’t forget to change the adapter package of it. python-dotenv package is for reading the .env file, I stored SQL URI and Redis server details in the .env file.
On the home page, we can select a user. Based on the user’s organisation UI colour scheme and the content will be changed. Home page looks like this
If you select another user it looks like this
home page(index.html) code:
<!DOCTYPE html>
<html>
<head>
<title>{{ theme.title }}</title>
<style>
h1 {
color: {{ theme.styles.h1_color }};
}
h2 {
color: {{ theme.styles.h2_color }};
}
p {
color: {{ theme.styles.p_color }};
}
</style>
<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
<script>
$(document).ready(function() {
$('#organisation_id').change(function() {
var selectedOrgId = $(this).val();
$.ajax({
url: '/get-theme/' + selectedOrgId,
method: 'GET',
success: function(response) {
let new_theme = response;
updateTheme(new_theme);
},
error: function() {
console.log('Error in fetching organisation details');
}
});
});
function updateTheme(theme) {
$('h1').css('color', theme.styles.h1_color);
$('h2').css('color', theme.styles.h2_color);
$('p').css('color', theme.styles.p_color);
$('header h1').text(theme.header);
$('main h2').text('Welcome to the website!');
$('main p').text(theme.content);
}
});
</script>
</head>
<body>
<label for="organisation_id">Users:</label>
<select id="organisation_id" name="organisation_id">
<option value="">Select a User</option>
{% for user in users %}
<option value="{{ user.organisation_id }}">{{ user.name }}</option>
{% endfor %}
</select>
<header>
<h1>{{ theme.header }}</h1>
</header>
<main>
<h2>Welcome to the website!</h2>
<p>{{ theme.content }}</p>
</main>
</body>
</html>
At the start, it is powered by the default theme when we change the user from the dropdown using jQuery hitting the “get-theme” API based on the response changing the CSS & content.
We can change the theme values using the theme page
Based on the organisation you selected you can change the theme details.
Theme Page Code:
<!DOCTYPE html>
<html>
<head>
<title>Theme Selector</title>
<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
<script>
$(document).ready(function() {
$('#id').change(function() {
var selectedOrgId = $(this).val();
$.ajax({
url: '/get-theme/' + selectedOrgId,
method: 'GET',
success: function(response) {
if (response.organisation_id != selectedOrgId) {
$('#title').val('');
$('#header').val('');
$('#content').val('');
$('#h1_color').val('');
$('#h2_color').val('');
$('#p_color').val('');
} else {
$('#title').val(response.title);
$('#header').val(response.header);
$('#content').val(response.content);
$('#h1_color').val(response.styles.h1_color);
$('#h2_color').val(response.styles.h2_color);
$('#p_color').val(response.styles.p_color);
}
},
error: function() {
console.log('Error in fetching organisation details');
}
});
});
});
</script>
</head>
<body>
<h1>Please enter theme details of organisation</h1>
<form action="/set-theme" method="POST">
<label for="id">Organisation:</label>
<select id="id" name="id">
<option value="">Select an organisation</option>
{% for org in organisation_list %}
<option value="{{ org.id }}">{{ org.name }}</option>
{% endfor %}
</select>
<br/>
<label for="title">Title:</label>
<input type="text" id="title" name="title">
<br/>
<label for="header">Header:</label>
<input type="text" id="header" name="header">
<br/>
<label for="content">Content:</label>
<input type="text" id="content" name="content">
<br/>
<label for="h1_color">H1 Color:</label>
<input type="text" id="h1_color" name="h1_color">
<br/>
<label for="h2_color">H2 Color:</label>
<input type="text" id="h2_color" name="h2_color">
<br/>
<label for="p_color">P Color:</label>
<input type="text" id="p_color" name="p_color">
<br/>
<button type="submit">Apply Theme</button>
</form>
</body>
</html>
To get theme details similar to index.html. To set the theme we are using form which sends data to the “set-theme” API.
The whole Backend logic wrote in the app.py only for simplicity's sake.
At the start we need to set the SQL connection through SQLAlchemy & Redis Connection.
load_dotenv()
app = Flask(__name__)
# Connect to the database using the connection string from the environment variables
app.config['SQLALCHEMY_DATABASE_URI'] = os.getenv('DATABASE_URL')
db = SQLAlchemy(app)
# Connect to Redis using the connection details from the environment variables
redis_host = os.getenv('REDIS_HOST')
redis_port = os.getenv('REDIS_PORT')
redis_password = os.getenv('REDIS_PASSWORD')
redis_db = os.getenv('REDIS_DB')
redis = Redis(host=redis_host, port=redis_port, password=redis_password, db=redis_db)
We need a default organisation which will be for the organisations that didn’t choose the theme. Its id will be 1.
DEFAULT_ORGANISATION_ID = 1
We need all the users, organisations lists to show in the UI, that code looks like this
def get_users():
users = db.session.execute(text("SELECT * FROM users")).fetchall()
return users
def get_organisations():
organisations = db.session.execute(text("SELECT * FROM organisations")).fetchall()
return organisations
home page, theme page routes code
@app.route("/")
def home():
users = get_users()
theme = get_theme()
return render_template("index.html", users=users, theme=theme)
@app.route("/theme")
def theme():
organisations = get_organisations()
return render_template("theme.html", organisation_list=organisations)
To get the theme of an organisation we need to get it from Cache if it’s there or else we need to get it from the database and set it in the Cache. We are using the Cache Aside reading strategy. Getting it from the Cache and setting it in the Cache part is common for any functionality in the application where the Cache makes sense. Decorator makes a lot of sense for this part of the functionality. So created a cache_decorator for it
def cache_decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
# Check if the data is present in cache
cache_key = f"cache:{func.__name__}:{args}:{kwargs}"
cached_data = redis.get(cache_key)
if cached_data:
# Data is present in cache, return it
return json.loads(cached_data)
# Data is not present in cache, execute the function
result = func(*args, **kwargs)
# Set the result in cache
redis.set(cache_key, json.dumps(result))
return result
return wrapper
“get-theme” route code
@app.route("/get-theme/<organisation_id>")
@cache_decorator
def get_theme(organisation_id=DEFAULT_ORGANISATION_ID):
theme_db = db.session.execute(text(f"SELECT * FROM themes WHERE organisation_id = {organisation_id}")).fetchone()
theme = {}
if theme_db:
theme['organisation_id'] = theme_db[1]
theme['title'] = theme_db[2]
theme['header'] = theme_db[3]
theme['content'] = theme_db[4]
theme['styles'] = json.loads(theme_db[5])
else:
return get_theme()
return theme
At the time of setting the theme we need to store it in the database and also evict it from the Cache or else old data will be served. This functionality code you can find below
@app.route("/set-theme", methods=['POST'])
def set_theme():
organisation_id = request.form['id']
title = request.form['title']
header = request.form['header']
content = request.form['content']
h1_color = request.form['h1_color']
h2_color = request.form['h2_color']
p_color = request.form['p_color']
styles = json.dumps({
"h1_color": h1_color,
"h2_color": h2_color,
"p_color": p_color
})
theme = db.session.execute(text(f"SELECT * FROM themes WHERE organisation_id = {organisation_id}")).fetchone()
# Evict the cache for the get_theme function
# redis.delete('"cache:get_theme:():{' + "'organisation_id':'" + str(organisation_id) + "'}" + '"')
redis.delete(f"cache:get_theme:():{{'organisation_id': '{organisation_id}'}}")
if theme:
# Theme already exists, update it
db.session.execute(text(f"UPDATE themes SET title = '{title}', header = '{header}', content = '{content}', styles = '{styles}' WHERE organisation_id = {organisation_id}"))
else:
# Theme doesn't exist, insert it
db.session.execute(text(f"INSERT INTO themes (organisation_id, title, header, content, styles) VALUES ({organisation_id}, '{title}', '{header}', '{content}', '{styles}')"))
db.session.commit()
return "Theme set successfully"
You will find the whole application code here.
Cache Challenges you need to ponder about
If you think once you implement Cache your world will be filled with Rainbows & Unicorns then you are wrong. If you don’t implement it properly or your user base gets increased you will face some challenges due to Cache. Some of the popular ones are
Cache Consistency: In distributed systems, making sure data consistency across multiple caches can be challenging due to factors like network latency and node failures.
Let’s take an example. Let’s say we are showing the number of views of a page if there is inconsistency one user will see 10 and another user will get 8.
Ensure that the data seen by different parts of the system is up-to-date.Cache Pollution: When unnecessary or irrelevant data occupies cache space, it reduces the effectiveness of caching. Strategies such as eviction policies need to balance between keeping useful data and making room for new data.
Cache Size and Placement: Determining the appropriate size and placement of caches is important to maximize hit rates (accessing data already in the cache) while minimizing miss penalties (having to fetch data from slower memory like Disk/Database). Also, make sure you don’t create Large Cache keys(Big Key) which will lead to slower lookups.
Thundering Herd Problem: This occurs when multiple processes or threads are waiting for a particular Cache node(Server) to become available or to be updated. When it becomes available, all waiting processes or threads are awakened simultaneously, causing a sudden surge of requests. This can overwhelm caches and backend systems, leading to performance degradation or even system instability. Strategies to mitigate this problem include using techniques like throttling, caching responses or using queuing mechanisms to stagger requests.
Hot Key Problem: Hot keys are critical in caching because caching systems are designed to accelerate access to frequently accessed data. However, hot keys can also pose challenges. If not managed properly, Several hot keys come under one Cache Node and that node will get the most of the requests which leads to performance bottlenecks, increased latency and potential cache thrashing.
Monitoring
Hit Rate: Measure the percentage of requests that are served from the cache without needing to fetch data from slower storage.
Miss Rate: Track the percentage of requests that result in cache misses and require fetching data from slower storage. This will give you a good idea about whether you should improve your Cache strategy or take a new approach. If there is any unusual pattern analyse it because it will give you a fair idea of the Security aspect also such as unusual or malicious access attempts.
Eviction Rate: Monitor how often items are evicted from the cache due to space constraints or cache replacement policies.
Cache Size and Capacity: Keep an eye on the current size of the cache and its capacity limits to ensure efficient usage of resources.
Latency: Measure the time taken to serve cache hits and misses to understand performance implications.
Did I miss anything? If yes, Please help me make it better by sharing your thoughts in the comments.
👋 Let’s be friends! Follow me on Twitter and connect with me on LinkedIn. Don’t forget to Subscribe as well.
Inspired by
https://www.designgurus.io/blog/caching-system-design-interview
https://www.geeksforgeeks.org/caching-system-design-concept-for-beginners/
https://aws.amazon.com/caching/