Home » Python » Python Programs

Merge a list of pandas dataframes

Given a list of pandas dataframes, we have to merge them. By Pranit Sharma Last updated : October 02, 2023

Pandas is a special tool that allows us to perform complex manipulations of data effectively and efficiently. Inside pandas, we mostly deal with a dataset in the form of DataFrame. DataFrames are 2-dimensional data structures in pandas. DataFrames consist of rows, columns, and data.

A list is a collection of heterogeneous elements and it is mutable in nature. Tuples are also another built-in data type of python used to store heterogeneous elements. Elements inside the list are encapsulated in square brackets whereas elements inside tuples are encapsulated in normal parenthesis.

Problem statement

We are given a list of DataFrames and we need to merge them together using a unique column ('X').

Merging a list of pandas dataframes

For this purpose, we can use the reduce() method which will compute the result of the merge method of pandas. Pandas merge is a method of combining or joining two DataFrames but the key point is merge method allows us to combine the DataFrames on the basis of specific columns instead of index values.

Let us understand with the help of an example,

Python program to merge a list of pandas dataframes

# Importing pandas package
import pandas as pd

# Creating a dictionary
d = {
    'X': [10,20,30,40], 
    'Y': [20,30,30,40]
}

# Creating a DataFrame
df = pd.DataFrame(d)

# Display original DataFrame
print("Original DataFrame:\n",df,"\n")

# Creating a list
l = [df,df]

# Display list
print("Created List:\n",l)

# Merging the list for combining DataFrames
df = df._reduce(lambda x, y: pd.merge(x, y, on = 'X'), l)

Output

The output of the above program is:

Python Pandas Programs »