Missing Values
Detect, count, and handle null values — the first thing you check in any new dataset.
You open a dataset of 10,000 customer orders. Before you write a single analysis query, you need to know one thing: what's missing?
A single NaN in the wrong column can silently corrupt an average, break a merge, or crash your model. In the real world, 5–20% of a dataset being missing is completely normal — data comes from humans, forms, and systems that fail.
Two lines diagnose any dataset:
import pandas as pd
df = pd.read_csv('orders.csv')
# How many nulls per column?
print(df.isnull().sum())
# What percentage is missing?
print(df.isnull().mean() * 100)isnull() returns a boolean DataFrame (True = missing). .sum() counts Trues per column. .mean() gives the fraction — multiply by 100 for percent.What does this print?
import pandas as pd
import numpy as np
df = pd.DataFrame({'a': [1, None, 3], 'b': [4, 5, 6]})
print(df.isnull().sum())- Aa 1 b 0 dtype: int64
- Ba 2 b 0 dtype: int64
- C1 0
- DTrue False
Three strategies for handling nulls — pick based on the column and context:
| Strategy | When to use |
|----------|------------|
| df.dropna() | Row has nulls in critical columns (can't impute) |
| df['col'].fillna(df['col'].mean()) | Numeric column, roughly symmetric distribution |
| df['col'].fillna(method='ffill') | Time-series — carry last known value forward |
| df['col'].fillna('Unknown') | Categorical column where null = missing category |
A 'signup_date' column is missing for 2% of users. You need this column for a retention analysis. Which approach is most appropriate?
- Afillna(0) — replace with zero to keep all rows
- Bdropna(subset=['signup_date']) — remove those rows since the date can't be imputed
- Cfillna(df['signup_date'].mean()) — fill with the average date
- Dfillna('Unknown') — mark them as unknown dates
dropna() options you'll actually use:
# Drop rows where ANY column is null
df.dropna()
# Drop rows missing a specific critical column
df.dropna(subset=['user_id', 'order_date'])
# Only keep rows with at least 3 non-null values
df.dropna(thresh=3)
# Drop columns where >50% of values are missing
df.dropna(axis=1, thresh=len(df) * 0.5)Always use inplace=False (the default) and assign to a new variable so you can compare before and after.
Fill missing values in the 'price' column with the column's median:
df['price'] = df['price'].fillna(df['price'].___())You fill a numeric column's nulls with its mean. Later you notice the column was highly skewed (many low values, a few very high ones). What went wrong?
- ANothing — mean is always the correct fill value
- BThe mean was pulled up by outliers, so you imputed an unrealistically high value for most missing rows
- Cfillna doesn't work on skewed data
- DYou should have used fillna(0) instead