How to set/get pandas.DataFrame to/from Redis?

Given a pandas dataframe, we have to set/get it to/from Redis.
Submitted by Pranit Sharma, on November 11, 2022

Pandas is a special tool that allows us to perform complex manipulations of data effectively and efficiently. Inside pandas, we mostly deal with a dataset in the form of DataFrame. DataFrames are 2-dimensional data structures in pandas. DataFrames consist of rows, columns, and data.

Redis is an open-source (BSD licensed), in-memory data structure used as a database, cache, message broker, and streaming engine.

Setting/Getting pandas.DataFrame to/from Redis

We can set pandas DataFrame to Redis, and then by getting it back, Redis returns a string but the fact that we stored our values to Redis in the form of DataFrame makes it strange that the stored values after retrieval become strings. We need to find out a way to convert these strings into our DataFrame so that we can restore our original DataFrame.

To successfully achieve this task, we need to first import redis and establish a connection to redis with the help of the following code snippet:

import redis
r = redis.StrictRedis(host='localhost', port=6379, db=0)

Once the connection is established and stored in a variable, this variable will act as an object which has several methods to call.

Now to set the DataFrame to Redis, we need another library called zlib which will help to compress the DataFrame to a message pack, hence we need to use pandas.DataFrame.to_msgpack() method inside which we will pass the compress parameter as 'zlib', following is the code snippet:

import zlib
r.set("key", df.to_msgpack(compress='zlib'))

Now our dataframe is set to Redis.

To successfully get the DataFrame, we can either use the read_msgpack() method or we can import the pickle package which has the loads method which will help us to decompress the Zlib compression applied on the DataFrame.

Below is the implementation:

Method 1:

import pickle
read = pickle.loads(zlib.decompress(r.get("key")))

Method 2:

pd.read_msgpack(r.get("key"))

Python Pandas Programs »

Pandas DataFrame Diagonal

Make pandas DataFrame to a dict and dropna