Home » Python » Python Programs

How to remove rows in a Pandas dataframe if the same row exists in another dataframe?

Given a pandas dataframe, we have to remove rows in a Pandas dataframe if the same row exists in another dataframe. By Pranit Sharma Last updated : October 03, 2023

Pandas is a special tool that allows us to perform complex manipulations of data effectively and efficiently. Inside pandas, we mostly deal with a dataset in the form of DataFrame. DataFrames are 2-dimensional data structures in pandas. DataFrames consist of rows, columns, and data.

Problem statement

Suppose we are given two DataFrames d1 and d2 which contains several rows and we need to output the DataFrame that only contains the rows unique in df1 but not in df2 which means we must exclude all the common rows of df1 and df2.

Remove rows in a Pandas dataframe if the same row exists in another dataframe

For this purpose, we will use pandas.DataFrame.merge() method inside which we will pass both the DataFrames then we will define what type of join is this and most importantly we will then drop the common rows.

Let us understand with the help of an example,

Python program to remove rows in a Pandas dataframe if the same row exists in another dataframe

# Importing pandas package
import pandas as pd

# Creating two dictionaries
d1 = {'a':[1,2,3],'b':[10,20,30]}
d2 = {'a':[0,1,2,3],'b':[0,1,20,3]}

# Creating DataFrames
df1 = pd.DataFrame(d1)
df2 = pd.DataFrame(d2)

# Display original DataFrames
print("Original DataFrame 1:\n",df1,"\n")
print("Original DataFrame 2:\n",df2,"\n")

# merging and dropping common rows
res = pd.merge(df1,df2, indicator=True, how='outer').query('_merge=="left_only"').drop('_merge', axis=1)

# Display Result
print("Result:\n",res)

Output

The output of the above program will be:

Python Pandas Programs »