Lecture 11
pandas Basics - Missing Values; Duplicates
February 21, 2025
Dealing with Missing Values
Dealing with Missing Values
Dealing with Missing Values
- Pandas often marks (1) missing text values and (2) missing numeric values with a
NaN(not a number);- It also marks missing datetime values with a
NaT(not a time).
- It also marks missing datetime values with a
Dealing with Missing Values: The isna() and notna() methods
- The
isna()method returns a BooleanSeriesin whichTruedenotes that an observation’s value is missing.- Is a value of a variable “XYZ” missing?
Dealing with Missing Values: The isna() and notna() methods
The
notna()method returns the inverseSeries, one in whichTrueindicates that an observation’s value is present.We use the tilde symbol (
~) to invert a BooleanSeries.Q. How can we pull out employees with non-missing
Teamvalues?
Dealing with Missing Values: The value_counts(dropna = False) method
- One way to missing data counts is to use the
isna().sum()on aSeries.Trueis 1 andFalseis 0.
- Another way to get missing data counts is to use the
.value_counts()method on aSeries.- If we use the
dropna = Falseoption, we can also get a missing value count.
- If we use the
Dealing with Missing Values: The dropna() method
- The
dropna()method removes observations that hold anyNaNorNaTvalues.
Dealing with Missing Values: The dropna() method with how
We can pass the
howparameter an argument of"all"to remove observations in which all values are missing.Note that the
howparameter’s default argument is"any".
Dealing with Missing Values: The dropna() method with subset
- We can use the
subsetparameter to target observations with a missing value in a specific variable.- The above example removes observations that have a missing value in the
Gendervariable.
- The above example removes observations that have a missing value in the
Dealing with Missing Values: The dropna() method with subset
- We can also pass the
subsetparameter a list of variables.
Dealing with Duplicates
Dealing with Duplicates with the duplicated() method
- Missing values are a common occurrence in messy data sets, and so are duplicate values.
- The
duplicated()method returns a BooleanSeriesthat identifies duplicates in a variable.
Dealing with Duplicates with the duplicated() method
emp["Team"].duplicated(keep = "first")
emp["Team"].duplicated(keep = "last")
~emp["Team"].duplicated()- The
duplicated()method’skeepparameter informs pandas which duplicate occurrence to keep.- Its default argument,
"first", keeps the first occurrence of each duplicate value. - Its argument,
"last", keeps the last occurrence of each duplicate value.
- Its default argument,
Dealing with Duplicates with the drop_duplicates() method
- The
drop_duplicates()method removes observations in which all values are equal to those in a previously encountered observations.
Dealing with Duplicates with the drop_duplicates() method
- Below is an example of the
drop_duplicates()method:
# Sample DataFrame with duplicate observations
data = {
'Name': ['John', 'Anna', 'John', 'Mike', 'Anna'],
'Age': [28, 23, 28, 32, 23],
'City': ['New York', 'Paris', 'New York', 'London', 'Paris']
}
# pd.DataFrame( Series, List, or Dict ) creates a DataFrame
df = pd.DataFrame(data)
df_unique = df.drop_duplicates()Dealing with Duplicates with the drop_duplicates() method
We can pass the
drop_duplicates()method asubsetparameter with a list of columns that pandas should use to determine an observation’s uniqueness.- The above example finds the first occurrence of each unique value in the Team variable.
Dealing with Duplicates with the drop_duplicates() method
- The above example uses a combination of values across the
GenderandTeamvariables to identify duplicates.
Dealing with Duplicates with the drop_duplicates() method
emp.drop_duplicates(subset = ["Team"], keep = "last")
emp.drop_duplicates(subset = ["Team"], keep = False)The
drop_duplicates()method also accepts akeepparameter.- We can pass it an argument of
"last"to keep the observations with each duplicate value’s last occurrence. - We can pass it an argument of
Falseto exclude all observations with duplicate values.
- We can pass it an argument of
Q. What does
emp.drop_duplicates(subset = ["First Name"], keep = False)do?Q. Find a subset of all employees with a First Name of “Douglas” and a Gender of “Male”. Then check which “Douglas” is in the DataFrame
emp.drop_duplicates(subset = ["Gender", "Team"]).
Pandas Basics
Let’s do Questions 7-8 in Classwork 6!
