Add Column to DataFrame Pandas
As data enthusiasts, we understand the pivotal role Pandas plays in data manipulation. One fundamental skill is adding columns to DataFrames. It is a task that can significantly impact your data analysis.
In this article, we will explore 5 different ways to add columns to Pandas DataFrames and will look at diverse scenarios and common mistakes.
- Direct Assignment
- Using Existing Columns
- Applying a Function
- Concatenating DataFrames
- Using the assign() Method
- Common Mistakes and How to Avoid Them
- Conclusion
Table of Contents
1. Direct Assignment
The most straightforward method involves directly assigning values to a new column. This method is suitable when you want to assign the same value to each row in the column.
For example, if you want to create a new column with label 'city' you can write df['City'] = 'New York'. This will create a new column with label 'city' and same values for all rows.
import pandas as pd
# Creating a simple DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35]}
df = pd.DataFrame(data)
print("Original Dataframe:")
print(df)
# 👉 Adding a new 'City' column with the same value for all rows
df['City'] = 'New York'
print("\nDataframe after adding new column 'City'")
print(df)Output:
Original Dataframe:
Name Age
0 Alice 25
1 Bob 30
2 Charlie 35
Dataframe after adding new column 'City'
Name Age City
0 Alice 25 New York
1 Bob 30 New York
2 Charlie 35 New York
If you want to learn how to create a DataFrame click here.
2. Using Existing Columns
You can create a new column based on existing columns, utilizing the flexibility of Python's arithmetic and logical operations.
The following example deriving a new column 'Birth Year' based on 'Age' column.
import pandas as pd
# Creating a simple DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35]}
df = pd.DataFrame(data)
print("Original Dataframe:")
print(df)
# 👉 Adding a new 'Birth Year' column based on 'Age'
df['Birth Year'] = 2024 - df['Age']
print("\nDataframe after adding new column 'Birth Year'")
print(df)Output:
Original Dataframe:
Name Age
0 Alice 25
1 Bob 30
2 Charlie 35
Dataframe after adding new column 'Birth Year'
Name Age Birth Year
0 Alice 25 1999
1 Bob 30 1994
2 Charlie 35 1989
3. Applying a Function
For more complex transformations, you can use the apply() function to apply a custom function to each row.
The following example creates a new column by applying a Python function on each element of other column.
import pandas as pd
# Creating a simple DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [15, 30, 35]}
df = pd.DataFrame(data)
print("Original Dataframe:")
print(df)
# 👉 Adding a new 'Status' column based on a custom function
def determine_status(age):
return 'Adult' if age >= 18 else 'Minor'
df['Status'] = df['Age'].apply(determine_status)
print("\nDataframe after adding new column 'Status'")
print(df)Output:
Original Dataframe:
Name Age
0 Alice 15
1 Bob 30
2 Charlie 35
Dataframe after adding new column 'Status'
Name Age Status
0 Alice 15 Minor
1 Bob 30 Adult
2 Charlie 35 Adult
4. Concatenating DataFrames
When working with multiple DataFrames, concatenation is a powerful method to combine them and add columns simultaneously.
To concatenate 2 or more dataframes use concat() method and pass list of all dataframes to concat, it returns a new concatenated DataFrame.
import pandas as pd
# Creating a simple DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35]}
df = pd.DataFrame(data)
print("Original Dataframe:")
print(df)
# Creating a second DataFrame
data2 = {'Name': ['David', 'Eva'],
'Age': [28, 22]}
df2 = pd.DataFrame(data2)
# 👉 Concatenating DataFrames along columns
df_concatenated = pd.concat([df, df2])
print("\nAfter concatenating")
print(df_concatenated)Output:
Original Dataframe:
Name Age
0 Alice 25
1 Bob 30
2 Charlie 35
After concatenating
Name Age
0 Alice 25
1 Bob 30
2 Charlie 35
0 David 28
1 Eva 22
You can reset index of Dataframe later.
5. Using the assign() Method
The assign() method allows you to add one or more columns in a single line, creating a new DataFrame with the added columns.
import pandas as pd
# Creating a simple DataFrame
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [25, 30, 35]}
df = pd.DataFrame(data)
print("Original Dataframe:")
print(df)
# 👉 Adding new 'Salary' and 'Experience' columns using assign()
df = df.assign(Salary=[60000, 70000, 80000], Experience=[2, 5, 8])
print("\nAfter adding 2 new columns")
print(df)Output:
Original Dataframe:
Name Age
0 Alice 25
1 Bob 30
2 Charlie 35
After adding 2 new columns
Name Age Salary Experience
0 Alice 25 60000 2
1 Bob 30 70000 5
2 Charlie 35 80000 8
6. Common Mistakes and How to Avoid Them
1. Iterative Appends
Mistake:
Iteratively appending rows or columns using methods like iterrows() can be inefficient and lead to performance issues, especially with large datasets.
Best Practice: Prefer direct assignment or vectorized operations to avoid iterative appends. They are more efficient and lead to cleaner code.
2. Inefficient Use of apply()
Mistake:
Using apply() without considering vectorized alternatives can lead to slower execution, especially on large datasets.
Best Practice: Leverage Pandas' vectorized operations whenever possible. They are optimized for efficiency and can significantly improve performance.
3. Ignoring Memory Efficiency
Mistake: Adding columns without considering memory usage can lead to increased overhead, affecting performance.
Best Practice: Be mindful of memory usage, especially with substantial datasets. Choose methods that minimize memory overhead, such as vectorized operations.
Conclusion
With above discussed methods you are now equipped to add columns to Pandas DataFrames. You can try out these methods on your own and see which one works best for you.
Also look at the common mistakes and best practices to avoid them. This will help you write more efficient code and improve your data analysis.
Happy coding! 🚀✨