Loading, cleaning and handling missing data honestly

Loading a file

import pandas as pd

df = pd.read_csv("students.csv")
print(df.head())
print(df.shape)

Useful arguments when the file fights back:

df = pd.read_csv(
    "students.csv",
    sep=",",              # use ";" or "\t" when needed
    encoding="utf-8",     # try "latin-1" if you get a UnicodeDecodeError
    na_values=["NA", "-", "?", "", "null"],
)

If the file has a title banner above the header row, add skiprows=2.

na_values matters more than it looks. Data entered by hand contains -, NA, n/a and blank cells meaning the same thing. Tell pandas about them on load and they all become one thing: NaN.

Other formats work the same way: pd.read_excel("file.xlsx"), pd.read_json("file.json").

Look before you clean

print(df.info())
print(df.isna().sum())                 # missing count per column
print(df.isna().sum() / len(df) * 100) # as a percentage
print(df.duplicated().sum())

A real result might look like this:

name        0
branch      3
sem         0
marks      47
fee_paid  312

Out of 500 rows, marks is missing 47 times (9.4 percent) and fee_paid is missing 312 times (62 percent). Those two columns need completely different decisions, and neither decision is "fill with zero".

Fixing types and text

df["marks"] = pd.to_numeric(df["marks"], errors="coerce")
df["joined"] = pd.to_datetime(df["joined"], errors="coerce")

df["branch"] = df["branch"].str.strip().str.upper()
df["name"] = df["name"].str.strip().str.title()

print(df["branch"].value_counts())

errors="coerce" turns anything unparseable into NaN instead of crashing. Check how many that created:

print(df["marks"].isna().sum(), "marks could not be read as a number")

The .str.strip().str.upper() line is often the single most valuable line in a cleaning script. Trailing spaces make "CSE " and "CSE" two different groups, and you will not see the difference on screen.

Duplicates

print(df[df.duplicated(subset=["roll_no"], keep=False)])
df = df.drop_duplicates(subset=["roll_no"], keep="first")

Look at the duplicates before dropping them. Sometimes two rows with the same roll number are a genuine data-entry error. Sometimes they are two different semesters and dropping one destroys real information.

The four honest options for missing data

1. Drop the rows.

df_clean = df.dropna(subset=["marks"])

Fine when the missing rows are few and missing for no particular reason. Always report how many you removed.

2. Drop the column.

df = df.drop(columns=["fee_paid"])

When 62 percent is missing, that column supports no conclusion. Say so and move on.

3. Fill, and record that you filled.

df["marks_filled"] = df["marks"].fillna(df["marks"].median())
df["marks_was_missing"] = df["marks"].isna()

Keep the flag column. Now anyone reading your work can check whether a result depends on the values you invented.

4. Leave it as NaN.

pandas skips NaN in mean(), sum() and groupby() on its own. Often the most honest choice is to change nothing and report the smaller sample size.

Why the value is missing changes what you should do

• Missing at random. A few marks lost in data entry. Dropping those

rows is usually safe.

• Missing for a reason. Only students who failed have a blank mark

column. Dropping them removes exactly the group you were studying, and your average jumps. This one is dangerous because the code runs fine and the number looks reasonable.

You cannot tell these apart from the file. You find out by asking whoever collected the data.

Finish with a cleaning report

before = len(df)
df = df.dropna(subset=["marks"]).drop_duplicates(subset=["roll_no"])
after = len(df)

print(f"Rows before: {before}")
print(f"Rows after : {after}")
print(f"Removed    : {before - after} ({(before - after) / before * 100:.1f}%)")

Next: asking real questions with groupby and merge.