• Follow Us On :

Python for Data Science Tutorial

Python for Data Science Tutorial: Master Analytics & Machine Learning from Scratch

Welcome to the ultimate Python for data science tutorial that will transform you from a beginner into a proficient data scientist. Python has emerged as the dominant programming language for data science, powering everything from exploratory data analysis to production machine learning systems at companies like Google, Netflix, and Spotify. This comprehensive Python for data science tutorial covers essential libraries, techniques, and real-world applications, providing you with the skills to extract insights from data, build predictive models, and communicate findings effectively.

Whether you’re a complete beginner taking your first steps in programming, an analyst transitioning from Excel or R, a software developer expanding into data science, or a student preparing for a career in analytics, this Python for data science tutorial will guide you through the entire data science workflow. You’ll learn data manipulation with Pandas, numerical computing with NumPy, visualization with Matplotlib and Seaborn, statistical analysis, and machine learning with scikit-learn. By the end of this tutorial, you’ll have the practical skills to tackle real-world data science projects confidently.

Why Python for Data Science? Understanding Its Dominance

Python’s rise to dominance in data science stems from multiple factors that make it uniquely suited for analytical work. Understanding why Python for data science has become the industry standard helps appreciate the ecosystem you’re entering.

The Python Advantage in Data Science

Simplicity and Readability: Python’s clean, readable syntax allows you to focus on solving problems rather than wrestling with complex language constructs. Code reads almost like English, making it accessible to non-programmers and facilitating collaboration between data scientists, domain experts, and engineers.

Comprehensive Ecosystem: Python boasts an extensive collection of data science libraries covering every aspect of analytical work. NumPy provides efficient numerical computing, Pandas offers powerful data manipulation, Matplotlib and Seaborn create stunning visualizations, scikit-learn implements machine learning algorithms, and specialized libraries handle deep learning (TensorFlow, PyTorch), natural language processing (NLTK, spaCy), and more.

Community and Resources: The massive, active Python community creates tutorials, documentation, Stack Overflow answers, and open-source libraries. Whatever problem you encounter, someone has likely faced and solved it, sharing their solution publicly.

Industry Adoption: Major tech companies, financial institutions, healthcare organizations, and research labs use Python for data science. Learning Python opens doors to opportunities across industries and roles.

Versatility: Python isn’t limited to data analysis. The same language handles web development (Django, Flask), automation, scripting, and production systems, enabling data scientists to build end-to-end solutions without switching languages.

Integration Capabilities: Python integrates seamlessly with databases (SQL, NoSQL), big data tools (Spark, Hadoop), cloud platforms (AWS, Azure, GCP), and other languages (R, Java, C++), fitting into existing technology stacks.

The Data Science Workflow

Data science projects typically follow a structured workflow that this Python for data science tutorial will help you master:

  1. Problem Definition: Understanding business questions and framing them as analytical problems
  2. Data Collection: Gathering data from databases, APIs, files, or web scraping
  3. Data Cleaning: Handling missing values, removing duplicates, correcting errors
  4. Exploratory Data Analysis (EDA): Understanding data distributions, patterns, and relationships
  5. Feature Engineering: Creating new features from existing data to improve models
  6. Modeling: Building statistical models or machine learning algorithms
  7. Evaluation: Assessing model performance using appropriate metrics
  8. Deployment: Putting models into production for real-world use
  9. Communication: Presenting findings through reports, dashboards, or presentations

Python excels at every stage of this workflow, making it the ideal choice for end-to-end data science.

Setting Up Your Python Data Science Environment

Before diving into coding in this Python for data science tutorial, you need a properly configured development environment.

Installing Python and Package Management

Python Installation: Download Python 3.8 or later from python.org. Most data scientists prefer the Anaconda distribution, which includes Python, Jupyter notebooks, and essential data science libraries pre-installed. Download Anaconda from anaconda.com and follow the installation wizard for your operating system.

Package Management with pip: Pip is Python’s package installer. Install libraries using:

bash
pip install numpy pandas matplotlib seaborn scikit-learn

Conda Package Manager: Anaconda users have conda, which manages packages and environments:

bash
conda install numpy pandas matplotlib seaborn scikit-learn

Virtual Environments for Project Isolation

Virtual environments isolate project dependencies, preventing version conflicts:

Using venv (built-in):

bash
python -m venv myenv
source myenv/bin/activate  # On Windows: myenv\Scripts\activate
pip install numpy pandas matplotlib

Using conda:

bash
conda create -n datasci python=3.10
conda activate datasci
conda install numpy pandas matplotlib seaborn

Jupyter Notebook: The Data Scientist’s IDE

Jupyter Notebook provides interactive development environments perfect for data exploration and analysis. Install with:

bash
pip install jupyter

Launch with:

bash
jupyter notebook

Jupyter notebooks combine code, visualizations, and markdown text in a single document, making them ideal for exploratory analysis and sharing results.

JupyterLab: The next-generation interface with enhanced features:

bash
pip install jupyterlab
jupyter lab

Essential Libraries Overview

This Python for data science tutorial focuses on core libraries:

  • NumPy: Numerical computing with efficient arrays and mathematical operations
  • Pandas: Data manipulation and analysis with DataFrames
  • Matplotlib: Fundamental plotting and visualization
  • Seaborn: Statistical data visualization built on Matplotlib
  • scikit-learn: Machine learning algorithms and tools
  • SciPy: Scientific computing and advanced mathematics
  • Statsmodels: Statistical modeling and hypothesis testing

NumPy: Foundation of Numerical Computing

NumPy (Numerical Python) is the foundation of data science in Python, providing fast, efficient operations on multi-dimensional arrays. Understanding NumPy is essential for this Python for data science tutorial.

NumPy Arrays: Better than Python Lists

NumPy arrays are more efficient than Python lists for numerical operations:

python
import numpy as np

# Creating arrays
arr1 = np.array([1, 2, 3, 4, 5])
arr2 = np.array([[1, 2, 3], [4, 5, 6]])

print(f"1D Array: {arr1}")
print(f"2D Array:\n{arr2}")
print(f"Array shape: {arr2.shape}")
print(f"Array dtype: {arr1.dtype}")

Array Creation Methods

python
# Zeros and ones
zeros = np.zeros((3, 4))
ones = np.ones((2, 3))

# Range of values
range_arr = np.arange(0, 10, 2)  # [0, 2, 4, 6, 8]
linspace_arr = np.linspace(0, 1, 5)  # 5 evenly spaced values from 0 to 1

# Random arrays
random_arr = np.random.random((3, 3))
random_int = np.random.randint(0, 100, (4, 4))
random_normal = np.random.randn(1000)  # Standard normal distribution

# Identity matrix
identity = np.eye(4)

Array Operations and Broadcasting

python
# Arithmetic operations (element-wise)
a = np.array([1, 2, 3, 4])
b = np.array([10, 20, 30, 40])

print(f"Addition: {a + b}")
print(f"Multiplication: {a * b}")
print(f"Power: {a ** 2}")

# Broadcasting: operations between arrays of different shapes
arr = np.array([[1, 2, 3], [4, 5, 6]])
scalar = 10
result = arr + scalar  # Adds 10 to each element

# Matrix multiplication
matrix1 = np.array([[1, 2], [3, 4]])
matrix2 = np.array([[5, 6], [7, 8]])
product = np.dot(matrix1, matrix2)  # or matrix1 @ matrix2

Indexing and Slicing

python
arr = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

# Basic indexing
print(arr[0])      # First element
print(arr[-1])     # Last element
print(arr[2:5])    # Elements from index 2 to 4

# 2D array indexing
arr2d = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr2d[0, 1])    # Row 0, column 1: 2
print(arr2d[:, 1])    # All rows, column 1: [2, 5, 8]
print(arr2d[1:, :2])  # Rows 1+, columns 0-1

# Boolean indexing
arr = np.array([1, 2, 3, 4, 5, 6])
mask = arr > 3
filtered = arr[mask]  # [4, 5, 6]

Statistical Operations

python
data = np.random.randn(1000)

print(f"Mean: {np.mean(data):.4f}")
print(f"Median: {np.median(data):.4f}")
print(f"Std Dev: {np.std(data):.4f}")
print(f"Variance: {np.var(data):.4f}")
print(f"Min: {np.min(data):.4f}")
print(f"Max: {np.max(data):.4f}")
print(f"Sum: {np.sum(data):.4f}")

# Percentiles
print(f"25th percentile: {np.percentile(data, 25):.4f}")
print(f"75th percentile: {np.percentile(data, 75):.4f}")

Array Manipulation

python
# Reshaping
arr = np.arange(12)
reshaped = arr.reshape(3, 4)

# Flattening
flattened = reshaped.flatten()

# Transposing
transposed = reshaped.T

# Concatenation
arr1 = np.array([[1, 2], [3, 4]])
arr2 = np.array([[5, 6], [7, 8]])
vertical = np.vstack([arr1, arr2])
horizontal = np.hstack([arr1, arr2])

# Splitting
split_arrays = np.split(np.arange(9), 3)

Pandas: Data Manipulation Powerhouse

Pandas is the cornerstone library for data manipulation in Python. This section of our Python for data science tutorial explores Pandas in depth.

Introduction to DataFrames and Series

python
import pandas as pd

# Series: 1D labeled array
s = pd.Series([10, 20, 30, 40, 50], index=['a', 'b', 'c', 'd', 'e'])
print(s)

# DataFrame: 2D labeled data structure
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
    'Age': [25, 30, 35, 28, 32],
    'Salary': [50000, 60000, 75000, 55000, 70000],
    'Department': ['HR', 'IT', 'Finance', 'IT', 'HR']
}

df = pd.DataFrame(data)
print(df)

Loading and Saving Data

python
# Reading CSV files
df = pd.read_csv('data.csv')

# Reading Excel files
df = pd.read_excel('data.xlsx', sheet_name='Sheet1')

# Reading from databases
import sqlite3
conn = sqlite3.connect('database.db')
df = pd.read_sql_query("SELECT * FROM table_name", conn)

# Reading JSON
df = pd.read_json('data.json')

# Saving data
df.to_csv('output.csv', index=False)
df.to_excel('output.xlsx', index=False)
df.to_json('output.json')

Data Exploration and Inspection

python
# Basic information
print(df.head())      # First 5 rows
print(df.tail())      # Last 5 rows
print(df.info())      # Data types and non-null counts
print(df.describe())  # Statistical summary
print(df.shape)       # (rows, columns)
print(df.columns)     # Column names
print(df.dtypes)      # Data types

# Unique values and counts
print(df['Department'].unique())
print(df['Department'].nunique())
print(df['Department'].value_counts())

Data Selection and Filtering

python
# Selecting columns
ages = df['Age']
subset = df[['Name', 'Salary']]

# Selecting rows by index
first_row = df.iloc[0]
first_three = df.iloc[0:3]

# Selecting by label
row = df.loc[0]
subset = df.loc[0:2, ['Name', 'Age']]

# Boolean filtering
high_earners = df[df['Salary'] > 60000]
it_dept = df[df['Department'] == 'IT']

# Multiple conditions
young_high_earners = df[(df['Age'] < 30) & (df['Salary'] > 55000)]

# Using query method
result = df.query('Age > 30 and Department == "IT"')

Data Cleaning and Preparation

python
# Handling missing values
df_copy = df.copy()

# Detecting missing values
print(df.isnull().sum())
print(df.isnull().any())

# Dropping missing values
df_dropped = df.dropna()  # Drop rows with any NaN
df_dropped = df.dropna(axis=1)  # Drop columns with any NaN

# Filling missing values
df_filled = df.fillna(0)
df_filled = df.fillna(df.mean())  # Fill with column mean
df_filled = df.fillna(method='ffill')  # Forward fill
df_filled = df.fillna(method='bfill')  # Backward fill

# Removing duplicates
df_unique = df.drop_duplicates()
df_unique = df.drop_duplicates(subset=['Name'])

# Renaming columns
df_renamed = df.rename(columns={'Salary': 'Annual_Salary'})

# Changing data types
df['Age'] = df['Age'].astype('int32')
df['Department'] = df['Department'].astype('category')

Data Transformation

python
# Creating new columns
df['Salary_K'] = df['Salary'] / 1000
df['Age_Group'] = pd.cut(df['Age'], bins=[0, 30, 40, 100], 
                         labels=['Young', 'Middle', 'Senior'])

# Applying functions
df['Name_Length'] = df['Name'].apply(len)
df['Salary_Doubled'] = df['Salary'].apply(lambda x: x * 2)

# String operations
df['Name_Upper'] = df['Name'].str.upper()
df['Name_Split'] = df['Name'].str.split(' ')

# Date operations
df['Date'] = pd.to_datetime(['2025-01-01', '2025-01-02', 
                              '2025-01-03', '2025-01-04', '2025-01-05'])
df['Year'] = df['Date'].dt.year
df['Month'] = df['Date'].dt.month
df['DayOfWeek'] = df['Date'].dt.dayofweek

Grouping and Aggregation

python
# GroupBy operations
dept_salary = df.groupby('Department')['Salary'].mean()
dept_stats = df.groupby('Department').agg({
    'Salary': ['mean', 'min', 'max'],
    'Age': ['mean', 'std']
})

# Multiple grouping
grouped = df.groupby(['Department', 'Age_Group'])['Salary'].sum()

# Pivot tables
pivot = df.pivot_table(values='Salary', 
                       index='Department', 
                       columns='Age_Group', 
                       aggfunc='mean')

Merging and Joining DataFrames

python
# Sample DataFrames
df1 = pd.DataFrame({
    'ID': [1, 2, 3, 4],
    'Name': ['Alice', 'Bob', 'Charlie', 'David']
})

df2 = pd.DataFrame({
    'ID': [1, 2, 3, 5],
    'Salary': [50000, 60000, 75000, 65000]
})

# Inner join (intersection)
inner = pd.merge(df1, df2, on='ID', how='inner')

# Left join (all from left, matching from right)
left = pd.merge(df1, df2, on='ID', how='left')

# Right join (all from right, matching from left)
right = pd.merge(df1, df2, on='ID', how='right')

# Outer join (union)
outer = pd.merge(df1, df2, on='ID', how='outer')

# Concatenation
combined = pd.concat([df1, df2], axis=0)  # Vertical
combined = pd.concat([df1, df2], axis=1)  # Horizontal

Data Visualization with Matplotlib and Seaborn

Visualization is crucial for understanding data and communicating insights. This Python for data science tutorial section covers essential plotting techniques.

Matplotlib Basics

python
import matplotlib.pyplot as plt
import numpy as np

# Line plot
x = np.linspace(0, 10, 100)
y1 = np.sin(x)
y2 = np.cos(x)

plt.figure(figsize=(10, 6))
plt.plot(x, y1, label='sin(x)', linewidth=2)
plt.plot(x, y2, label='cos(x)', linewidth=2, linestyle='--')
plt.xlabel('X axis')
plt.ylabel('Y axis')
plt.title('Sine and Cosine Functions')
plt.legend()
plt.grid(True, alpha=0.3)
plt.show()

Common Plot Types

python
# Scatter plot
x = np.random.randn(100)
y = 2 * x + np.random.randn(100)

plt.figure(figsize=(8, 6))
plt.scatter(x, y, alpha=0.6, c=y, cmap='viridis')
plt.colorbar(label='Y value')
plt.xlabel('X')
plt.ylabel('Y')
plt.title('Scatter Plot Example')
plt.show()

# Bar plot
categories = ['A', 'B', 'C', 'D', 'E']
values = [23, 45, 56, 78, 32]

plt.figure(figsize=(8, 6))
plt.bar(categories, values, color='skyblue', edgecolor='black')
plt.xlabel('Category')
plt.ylabel('Value')
plt.title('Bar Chart Example')
plt.show()

# Histogram
data = np.random.randn(1000)

plt.figure(figsize=(8, 6))
plt.hist(data, bins=30, edgecolor='black', alpha=0.7)
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.title('Histogram Example')
plt.show()

# Box plot
data = [np.random.normal(0, std, 100) for std in range(1, 4)]

plt.figure(figsize=(8, 6))
plt.boxplot(data, labels=['Group 1', 'Group 2', 'Group 3'])
plt.ylabel('Value')
plt.title('Box Plot Example')
plt.show()

Subplots and Multiple Figures

python
# Creating subplots
fig, axes = plt.subplots(2, 2, figsize=(12, 10))

# Plot 1: Line plot
axes[0, 0].plot(x, np.sin(x))
axes[0, 0].set_title('Sine Wave')

# Plot 2: Scatter plot
axes[0, 1].scatter(x, y)
axes[0, 1].set_title('Scatter Plot')

# Plot 3: Histogram
axes[1, 0].hist(np.random.randn(1000), bins=30)
axes[1, 0].set_title('Histogram')

# Plot 4: Bar plot
axes[1, 1].bar(categories, values)
axes[1, 1].set_title('Bar Chart')

plt.tight_layout()
plt.show()

Seaborn: Statistical Visualization

python
import seaborn as sns

# Set style
sns.set_style('whitegrid')

# Load sample dataset
tips = sns.load_dataset('tips')

# Distribution plot
plt.figure(figsize=(10, 6))
sns.histplot(data=tips, x='total_bill', kde=True, bins=30)
plt.title('Distribution of Total Bill')
plt.show()

# Box plot by category
plt.figure(figsize=(10, 6))
sns.boxplot(data=tips, x='day', y='total_bill', hue='sex')
plt.title('Total Bill by Day and Gender')
plt.show()

# Violin plot
plt.figure(figsize=(10, 6))
sns.violinplot(data=tips, x='day', y='total_bill', hue='time')
plt.title('Total Bill Distribution by Day and Time')
plt.show()

# Scatter plot with regression line
plt.figure(figsize=(10, 6))
sns.regplot(data=tips, x='total_bill', y='tip', scatter_kws={'alpha':0.5})
plt.title('Relationship between Total Bill and Tip')
plt.show()

# Pair plot
sns.pairplot(tips, hue='time', diag_kind='kde')
plt.show()

# Heatmap (correlation matrix)
plt.figure(figsize=(10, 8))
correlation = tips.corr(numeric_only=True)
sns.heatmap(correlation, annot=True, cmap='coolwarm', center=0)
plt.title('Correlation Heatmap')
plt.show()

# Count plot
plt.figure(figsize=(10, 6))
sns.countplot(data=tips, x='day', hue='sex')
plt.title('Count of Meals by Day and Gender')
plt.show()

Exploratory Data Analysis (EDA) Workflow

EDA is a critical phase in every data science project. This Python for data science tutorial section demonstrates a complete EDA workflow.

Loading and Initial Inspection

python
# Load dataset
df = pd.read_csv('dataset.csv')

# Initial inspection
print("Dataset shape:", df.shape)
print("\nFirst few rows:")
print(df.head())

print("\nDataset info:")
print(df.info())

print("\nStatistical summary:")
print(df.describe())

print("\nMissing values:")
print(df.isnull().sum())

print("\nData types:")
print(df.dtypes)

Univariate Analysis

python
# Numerical variables
numerical_cols = df.select_dtypes(include=[np.number]).columns

for col in numerical_cols:
    plt.figure(figsize=(12, 4))
    
    # Histogram
    plt.subplot(1, 3, 1)
    plt.hist(df[col].dropna(), bins=30, edgecolor='black')
    plt.title(f'{col} - Histogram')
    plt.xlabel(col)
    
    # Box plot
    plt.subplot(1, 3, 2)
    plt.boxplot(df[col].dropna())
    plt.title(f'{col} - Box Plot')
    plt.ylabel(col)
    
    # KDE plot
    plt.subplot(1, 3, 3)
    df[col].dropna().plot(kind='kde')
    plt.title(f'{col} - Density Plot')
    plt.xlabel(col)
    
    plt.tight_layout()
    plt.show()

# Categorical variables
categorical_cols = df.select_dtypes(include=['object', 'category']).columns

for col in categorical_cols:
    plt.figure(figsize=(10, 6))
    df[col].value_counts().plot(kind='bar')
    plt.title(f'{col} - Frequency Distribution')
    plt.xlabel(col)
    plt.ylabel('Count')
    plt.xticks(rotation=45)
    plt.show()

Bivariate Analysis

python
# Correlation analysis
correlation_matrix = df.corr(numeric_only=True)

plt.figure(figsize=(12, 10))
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0, 
            fmt='.2f', square=True, linewidths=1)
plt.title('Correlation Matrix')
plt.show()

# Scatter plots for highly correlated variables
high_corr = np.where(np.abs(correlation_matrix) > 0.5)
high_corr_pairs = [(correlation_matrix.index[x], correlation_matrix.columns[y]) 
                   for x, y in zip(*high_corr) if x != y and x < y]

for var1, var2 in high_corr_pairs[:5]:  # Top 5 pairs
    plt.figure(figsize=(8, 6))
    plt.scatter(df[var1], df[var2], alpha=0.5)
    plt.xlabel(var1)
    plt.ylabel(var2)
    plt.title(f'{var1} vs {var2}')
    plt.show()

# Categorical vs Numerical
for cat_col in categorical_cols[:2]:  # First 2 categorical
    for num_col in numerical_cols[:2]:  # First 2 numerical
        plt.figure(figsize=(10, 6))
        sns.boxplot(data=df, x=cat_col, y=num_col)
        plt.title(f'{num_col} by {cat_col}')
        plt.xticks(rotation=45)
        plt.show()

Outlier Detection

python
def detect_outliers_iqr(df, column):
    Q1 = df[column].quantile(0.25)
    Q3 = df[column].quantile(0.75)
    IQR = Q3 - Q1
    lower_bound = Q1 - 1.5 * IQR
    upper_bound = Q3 + 1.5 * IQR
    outliers = df[(df[column] < lower_bound) | (df[column] > upper_bound)]
    return outliers

# Check for outliers in numerical columns
for col in numerical_cols:
    outliers = detect_outliers_iqr(df, col)
    print(f"{col}: {len(outliers)} outliers detected")

Statistical Analysis with Python

Understanding statistics is essential for data science. This section of our Python for data science tutorial covers key statistical concepts.

Descriptive Statistics

python
import scipy.stats as stats

data = np.random.randn(1000)

# Central tendency
mean = np.mean(data)
median = np.median(data)
mode = stats.mode(data, keepdims=True)

# Dispersion
variance = np.var(data)
std_dev = np.std(data)
range_val = np.ptp(data)
iqr = stats.iqr(data)

# Shape
skewness = stats.skew(data)
kurtosis = stats.kurtosis(data)

print(f"Mean: {mean:.4f}")
print(f"Median: {median:.4f}")
print(f"Std Dev: {std_dev:.4f}")
print(f"Skewness: {skewness:.4f}")
print(f"Kurtosis: {kurtosis:.4f}")

Hypothesis Testing

python
# T-test: Comparing two groups
group1 = np.random.normal(100, 15, 100)
group2 = np.random.normal(105, 15, 100)

t_stat, p_value = stats.ttest_ind(group1, group2)
print(f"T-statistic: {t_stat:.4f}")
print(f"P-value: {p_value:.4f}")

if p_value < 0.05:
    print("Reject null hypothesis: Groups are significantly different")
else:
    print("Fail to reject null hypothesis: No significant difference")

# Chi-square test: Independence between categorical variables
contingency_table = pd.crosstab(df['category1'], df['category2'])
chi2, p_value, dof, expected = stats.chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.4f}")
print(f"P-value: {p_value:.4f}")

# ANOVA: Comparing multiple groups
group1 = np.random.normal(100, 15, 100)
group2 = np.random.normal(105, 15, 100)
group3 = np.random.normal(110, 15, 100)

f_stat, p_value = stats.f_oneway(group1, group2, group3)
print(f"F-statistic: {f_stat:.4f}")
print(f"P-value: {p_value:.4f}")

Correlation Analysis

python
# Pearson correlation
x = np.random.randn(100)
y = 2 * x + np.random.randn(100)

pearson_corr, p_value = stats.pearsonr(x, y)
print(f"Pearson correlation: {pearson_corr:.4f}")
print(f"P-value: {p_value:.4f}")

# Spearman correlation (non-parametric)
spearman_corr, p_value = stats.spearmanr(x, y)
print(f"Spearman correlation: {spearman_corr:.4f}")

Machine Learning with scikit-learn

Scikit-learn is the primary machine learning library in Python. This crucial section of our Python for data science tutorial introduces ML fundamentals.

Data Preparation for Machine Learning

python
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.impute import SimpleImputer

# Load and prepare data
# Assuming df is your DataFrame with features and target

# Separate features and target
X = df.drop('target', axis=1)
y = df['target']

# Handle categorical variables
le = LabelEncoder()
for col in X.select_dtypes(include=['object']).columns:
    X[col] = le.fit_transform(X[col].astype(str))

# Handle missing values
imputer = SimpleImputer(strategy='mean')
X = pd.DataFrame(imputer.fit_transform(X), columns=X.columns)

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Feature scaling
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Classification Models

python
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# Logistic Regression
lr_model = LogisticRegression(random_state=42)
lr_model.fit(X_train_scaled, y_train)
lr_pred = lr_model.predict(X_test_scaled)
print("Logistic Regression Accuracy:", accuracy_score(y_test, lr_pred))

# Decision Tree
dt_model = DecisionTreeClassifier(random_state=42)
dt_model.fit(X_train, y_train)
dt_pred = dt_model.predict(X_test)
print("Decision Tree Accuracy:", accuracy_score(y_test, dt_pred))

# Random Forest
rf_model = RandomForestClassifier(n_estimators=100, random_state=42)
rf_model.fit(X_train, y_train)
rf_pred = rf_model.predict(X_test)
print("Random Forest Accuracy:", accuracy_score(y_test, rf_pred))

# Detailed evaluation
print("\nClassification Report:")
print(classification_report(y_test, rf_pred))

print("\nConfusion Matrix:")
cm = confusion_matrix(y_test, rf_pred)
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.title('Confusion Matrix')
plt.show()

Regression Models

python
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

# Linear Regression
lr_reg = LinearRegression()
lr_reg.fit(X_train_scaled, y_train)
lr_pred = lr_reg.predict(X_test_scaled)

print("Linear Regression:")
print(f"R² Score: {r2_score(y_test, lr_pred):.4f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, lr_pred)):.4f}")
print(f"MAE: {mean_absolute_error(y_test, lr_pred):.4f}")

# Random Forest Regressor
rf_reg = RandomForestRegressor(n_estimators=100, random_state=42)
rf_reg.fit(X_train, y_train)
rf_pred = rf_reg.predict(X_test)

print("\nRandom Forest Regression:")
print(f"R² Score: {r2_score(y_test, rf_pred):.4f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y_test, rf_pred)):.4f}")

# Visualize predictions
plt.figure(figsize=(10, 6))
plt.scatter(y_test, rf_pred, alpha=0.5)
plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 
         'r--', lw=2)
plt.xlabel('Actual Values')
plt.ylabel('Predicted Values')
plt.title('Actual vs Predicted')
plt.show()

Model Evaluation and Cross-Validation

python
from sklearn.model_selection import cross_val_score, GridSearchCV

# Cross-validation
model = RandomForestClassifier(random_state=42)
cv_scores = cross_val_score(model, X_train, y_train, cv=5, 
                            scoring='accuracy')

print(f"Cross-validation scores: {cv_scores}")
print(f"Mean CV Score: {cv_scores.mean():.4f}")
print(f"Std CV Score: {cv_scores.std():.4f}")

# Hyperparameter tuning with Grid Search
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [None, 10, 20, 30],
    'min_samples_split': [2, 5, 10]
}

grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)

grid_search.fit(X_train, y_train)

print(f"Best parameters: {grid_search.best_params_}")
print(f"Best cross-validation score: {grid_search.best_score_:.4f}")

# Use best model
best_model = grid_search.best_estimator_
test_score = best_model.score(X_test, y_test)
print(f"Test set score: {test_score:.4f}")

Feature Importance and Selection

python
from sklearn.feature_selection import SelectKBest, f_classif

# Feature importance from Random Forest
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

feature_importance = pd.DataFrame({
    'feature': X_train.columns,
    'importance': model.feature_importances_
}).sort_values('importance', ascending=False)

plt.figure(figsize=(10, 6))
plt.barh(feature_importance['feature'][:10], 
         feature_importance['importance'][:10])
plt.xlabel('Importance')
plt.title('Top 10 Feature Importances')
plt.gca().invert_yaxis()
plt.show()

# Feature selection
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X_train, y_train)
selected_features = X_train.columns[selector.get_support()]
print(f"Selected features: {list(selected_features)}")

Real-World Data Science Project Example

Let’s complete a full data science project in this Python for data science tutorial, from data loading to model deployment.

Project: Customer Churn Prediction

python
# Step 1: Load and explore data
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, roc_auc_score, roc_curve

# Load data
df = pd.read_csv('customer_churn.csv')

print("Dataset shape:", df.shape)
print("\nFirst few rows:")
print(df.head())

# Step 2: Data cleaning
print("\nMissing values:")
print(df.isnull().sum())

# Remove duplicates
df = df.drop_duplicates()

# Handle missing values
df = df.fillna(df.median(numeric_only=True))

# Step 3: Exploratory Data Analysis
# Churn distribution
plt.figure(figsize=(8, 6))
df['Churn'].value_counts().plot(kind='bar')
plt.title('Churn Distribution')
plt.xlabel('Churn')
plt.ylabel('Count')
plt.show()

# Correlation analysis
plt.figure(figsize=(12, 10))
correlation = df.corr(numeric_only=True)
sns.heatmap(correlation, annot=True, cmap='coolwarm', center=0, fmt='.2f')
plt.title('Correlation Matrix')
plt.show()

# Step 4: Feature engineering
# Create new features
df['TenureGroup'] = pd.cut(df['tenure'], bins=[0, 12, 24, 48, 72], 
                           labels=['0-12', '13-24', '25-48', '49-72'])
df['AverageChargePerMonth'] = df['TotalCharges'] / (df['tenure'] + 1)

# Encode categorical variables
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()

categorical_cols = df.select_dtypes(include=['object']).columns
for col in categorical_cols:
    if col != 'Churn':
        df[col] = le.fit_transform(df[col].astype(str))

# Step 5: Prepare data for modeling
X = df.drop('Churn', axis=1)
y = df['Churn']

# Encode target variable
y = le.fit_transform(y)

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Step 6: Train model
model = RandomForestClassifier(
    n_estimators=100,
    max_depth=10,
    random_state=42
)

model.fit(X_train_scaled, y_train)

# Step 7: Evaluate model
y_pred = model.predict(X_test_scaled)
y_pred_proba = model.predict_proba(X_test_scaled)[:, 1]

print("\nClassification Report:")
print(classification_report(y_test, y_pred))

print(f"\nROC-AUC Score: {roc_auc_score(y_test, y_pred_proba):.4f}")

# ROC Curve
fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba)

plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, label=f'ROC Curve (AUC = {roc_auc_score(y_test, y_pred_proba):.2f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random Classifier')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()

# Feature importance
feature_importance = pd.DataFrame({
    'feature': X.columns,
    'importance': model.feature_importances_
}).sort_values('importance', ascending=False)

plt.figure(figsize=(10, 8))
plt.barh(feature_importance['feature'][:15], 
         feature_importance['importance'][:15])
plt.xlabel('Importance')
plt.title('Top 15 Feature Importances')
plt.gca().invert_yaxis()
plt.show()

# Step 8: Save model for deployment
import joblib
joblib.dump(model, 'churn_model.pkl')
joblib.dump(scaler, 'scaler.pkl')

print("\nModel saved successfully!")

Best Practices for Data Science in Python

Code Organization and Documentation

python
# Use meaningful variable names
customer_age = df['age']  # Good
ca = df['age']  # Bad

# Write docstrings for functions
def calculate_churn_rate(df, time_period):
    """
    Calculate customer churn rate for a given time period.
    
    Parameters:
    -----------
    df : pandas.DataFrame
        DataFrame containing customer data
    time_period : str
        Time period for calculation ('monthly', 'quarterly', 'yearly')
    
    Returns:
    --------
    float
        Churn rate as a percentage
    """
    # Function implementation
    pass

# Use type hints
def preprocess_data(df: pd.DataFrame) -> pd.DataFrame:
    """Preprocess raw data."""
    return df.dropna()

Performance Optimization

python
# Use vectorized operations instead of loops
# Bad
result = []
for value in df['column']:
    result.append(value * 2)

# Good
result = df['column'] * 2

# Use appropriate data types
df['category'] = df['category'].astype('category')  # Saves memory

# Chunk processing for large files
chunk_size = 10000
for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):
    process_chunk(chunk)

Reproducibility

python
# Set random seeds
np.random.seed(42)

# Version control with Git
# Create .gitignore for data files

# Document environment
# requirements.txt or environment.yml

# Use configuration files
import yaml

with open('config.yaml', 'r') as f:
    config = yaml.safe_load(f)

Conclusion: Your Data Science Journey with Python

This comprehensive Python for data science tutorial has equipped you with essential skills for data analysis, visualization, statistical analysis, and machine learning. You’ve learned NumPy for numerical computing, Pandas for data manipulation, Matplotlib and Seaborn for visualization, and scikit-learn for machine learning—the core toolkit for any data scientist.

Data science is a journey of continuous learning. New libraries, techniques, and best practices emerge constantly. The foundations covered in this Python for data science tutorial provide a solid base for exploring advanced topics like deep learning, natural language processing, time series analysis, and big data technologies.

Apply your knowledge through practical projects. Work with real datasets from Kaggle, UCI Machine Learning Repository, or your own domain of interest. Build a portfolio showcasing your skills through GitHub repositories and blog posts. Participate in data science competitions to learn from others and benchmark your abilities.

The demand for data scientists continues growing across industries. Your proficiency in Python for data science opens doors to exciting career opportunities in technology, finance, healthcare, e-commerce, and more. Continue learning, stay curious, and use data to solve real-world problems and create value.

Resources for Continued Learning

Online Courses:

  • Python for Data Science and Machine Learning (Udemy)
  • Applied Data Science with Python (Coursera)
  • DataCamp’s Data Scientist with Python track

Books:

  • “Python for Data Analysis” by Wes McKinney
  • “Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow” by Aurélien Géron
  • “Python Data Science Handbook” by Jake VanderPlas

Practice Platforms:

  • Kaggle (competitions and datasets)
  • DataCamp (interactive exercises)
  • LeetCode (Python programming)

Communities:

  • r/datascience and r/Python (Reddit)
  • Stack Overflow (Q&A)
  • Local data science meetups

Documentation:

  • Official documentation for NumPy, Pandas, Matplotlib, scikit-learn
  • Real Python tutorials
  • Towards Data Science blog

Your mastery of Python for data science will grow with practice and application. Start building projects today, and you’ll be amazed at how quickly you progress from beginner to proficient data scientist!

 

Tags :

Leave a Reply

Your email address will not be published. Required fields are marked *