Home » Python » Python Programs

Dynamically filtering a pandas dataframe

Given a pandas dataframe, we have to dynamically filter it.
Submitted by Pranit Sharma, on November 15, 2022

Pandas is a special tool that allows us to perform complex manipulations of data effectively and efficiently. Inside pandas, we mostly deal with a dataset in the form of DataFrame. DataFrames are 2-dimensional data structures in pandas. DataFrames consist of rows, columns, and data.

Problem statement

Given a pandas dataframe, we have to dynamically filter it.

So, we are creating a DataFrame with multiple columns and then we need to filter the df using thresholds for three columns.

We can do this by simply applying all the conditions and if the data satisfies the condition, it will be the final result. However, we need to do this inside a function where the name of the thresholds and the columns are given in the form of a dictionary.

Dynamically filter a pandas DataFrame

For this purpose, we will use a quick and easy way of doing this. We will build a dynamic query using list comprehension and the join method of string.

Finally, we have a method called pandas.DataFrame.query() inside which we can pass any comprehended query and it will dynamically calculate all the results.

Let us understand with the help of an example,

Python program to dynamically filter a pandas DataFrame

# Importing pandas package
import pandas as pd

# Importing numpy package
import numpy as np

# Creating a dictionary
d ={
    "A" : [6, 2, 10, -5, 3],
    "B" : [2, 5, 3, 2, 6],
    "C" : [-5, 2, 1, 8, 2]
}

# Creating DataFrame
df = pd.DataFrame(d)

# Display dataframe
print('Original DataFrame:\n',df,'\n')

# Giving columns and thresholds
dic = {"A" : 0, "B" : 2, "C" : -1}

# Making a query
q = ' & '.join(['{}>{}'.format(k, v) for k, v in dic.items()])

# Running query
res = df.query(q)

# Display new DataFrame
print("Result:\n",res)

Output

The output of the above program is:

Python Pandas Programs »