Chapter 7: Applications of Python
You can use Python to analyze spreadsheets, run websites, control lab experiments, and a great deal more. This chapter is a quick tour of the main areas and the libraries people use in each, so you know where to look when you start a project of your own.
In this chapter you will learn:
- How to install third-party packages with pip
- Which libraries are used for data analysis and visualization
- What the popular web frameworks (Django, Flask, and Pyramid) look like
- How to automate tasks: running other programs, calling web APIs, and scheduling jobs
- Which libraries are used for numerical computing, machine learning, and simulation
Most examples in this chapter use third-party packages. These are not part of Python’s standard library, so you have to install them before you can import them. The usual tool is pip, which comes with Python. Install packages inside a virtual environment, so that each project keeps its own set of packages; Chapter 8 shows how to create one. Once the virtual environment is active, install a package like this:
python -m pip install pandas
You can install several packages at once, for example python -m pip install numpy matplotlib. If Python reports ModuleNotFoundError: No module named 'pandas', the package isn’t installed in the Python you are using to run your program.
Data analysis and visualization
Python is popular for data analysis and visualization because it has many libraries for the job. These are the ones you are most likely to meet.
pandas
The pandas library is the most widely used library for working with tables of data. Its main data structures are the DataFrame (a table with named columns, like a spreadsheet) and the Series (a single column). It has functions for reading data from files such as CSV and Excel, cleaning it, filtering it, and summarizing it, and it handles dates and missing values well.
This example builds a small table of sales from a dictionary, adds a calculated column, filters the rows, and adds up the revenue for each region:
import pandas as pd
# Build a table from a dictionary: each key becomes a column
sales = pd.DataFrame({
'region': ['East', 'West', 'East', 'West', 'East'],
'product': ['Pens', 'Pens', 'Paper', 'Paper', 'Pens'],
'units': [10, 4, 3, 8, 6],
'price': [1.5, 1.5, 4.0, 4.0, 1.5],
})
# Add a column calculated from two others (no loop needed)
sales['revenue'] = sales['units'] * sales['price']
# Print the first 5 rows of the table
print(sales.head())
# Output:
# region product units price revenue
# 0 East Pens 10 1.5 15.0
# 1 West Pens 4 1.5 6.0
# 2 East Paper 3 4.0 12.0
# 3 West Paper 8 4.0 32.0
# 4 East Pens 6 1.5 9.0
# Keep only the rows where more than 5 units were sold
big_orders = sales[sales['units'] > 5]
print(big_orders)
# Output:
# region product units price revenue
# 0 East Pens 10 1.5 15.0
# 3 West Paper 8 4.0 32.0
# 4 East Pens 6 1.5 9.0
# Group the rows by region and add up the revenue in each group
revenue_by_region = sales.groupby('region')['revenue'].sum()
print(revenue_by_region)
# Output:
# region
# East 36.0
# West 38.0
# Name: revenue, dtype: float64
# Save the result to a CSV file
revenue_by_region.reset_index().to_csv('revenue_by_region.csv', index=False)
To load real data from a file instead, use pd.read_csv('sales.csv'), which returns a DataFrame just like the one built here.
After groupby(), the region names become the index (the row labels) of the result, not a normal column. reset_index() turns them back into a column, so that to_csv(..., index=False) writes both the region and the revenue. Without reset_index(), index=False would leave the region names out of the file.
NumPy
Numerical computing in Python is built on the numpy library. Its main data structure is the array: a grid of numbers that all have the same type. You can do arithmetic on a whole array at once, without writing a loop, and NumPy does it much faster than a Python loop would. Many other libraries, including pandas, are built on top of NumPy.
import numpy as np
# Create an array from a Python list
scores = np.array([2, 4, 4, 4, 5, 5, 7, 9])
# Other ways to create arrays
print(np.arange(1, 6)) # Output: [1 2 3 4 5]
print(np.zeros(3)) # Output: [0. 0. 0.]
# Arithmetic applies to every element at once
print(scores * 10) # Output: [20 40 40 40 50 50 70 90]
print(scores - 1) # Output: [1 3 3 3 4 4 6 8]
# Statistics
print(scores.mean()) # Output: 5.0
print(scores.std()) # Output: 2.0
print(scores.max()) # Output: 9
# Boolean indexing: keep only the elements that match a condition
print(scores[scores > 4]) # Output: [5 5 7 9]
scores > 4 compares every element with 4 and gives an array of True and False values. Putting that array inside the square brackets keeps only the elements where it is True.
Matplotlib
For drawing charts, the most widely used library is matplotlib. It can create line plots, scatter plots, bar charts, histograms, and many other kinds of plots, and lets you customize every part of them. It is often used together with NumPy and pandas.
import numpy as np
import matplotlib.pyplot as plt
# Create an array with 100 evenly spaced values between 0 and 2*pi
x = np.linspace(0, 2 * np.pi, 100)
# Compute the sine and cosine of every value in the array
y1 = np.sin(x)
y2 = np.cos(x)
# Create a figure with one set of axes
fig, ax = plt.subplots()
# Plot the sine and cosine curves
ax.plot(x, y1, label='sin(x)')
ax.plot(x, y2, label='cos(x)')
# Add a legend and a title
ax.legend()
ax.set_title('Trigonometric functions')
# Show the plot in a window
plt.show()
plt.show() opens a window with the chart. To save the chart as an image file instead, call plt.savefig('trig.png') before plt.show().
seaborn
Built on top of Matplotlib, the seaborn library makes common statistical charts with less code and good-looking defaults. It works directly with pandas DataFrame objects.
This example uses tips, a small sample dataset of restaurant bills that seaborn can download for you, so it needs an internet connection the first time you run it:
import matplotlib.pyplot as plt
import seaborn as sns
# Load a sample dataset into a pandas DataFrame
tips = sns.load_dataset('tips')
# Scatter plot of tip against total bill, with a regression line
# and its 95% confidence interval, in a separate color for smokers
# and non-smokers
sns.lmplot(
data=tips,
x='total_bill',
y='tip',
hue='smoker', # Color the points by this column
ci=95, # Shade a 95% confidence interval
scatter_kws={'alpha': 0.5}, # Make the points semi-transparent
)
plt.show()
scikit-learn
Many people use the scikit-learn library (imported as sklearn) for machine learning. It has tools for common tasks such as classification (predicting a category), regression (predicting a number), and clustering (finding groups), and for measuring how well a model performs.
This example trains a RandomForestClassifier on the iris dataset, a small, classic dataset of flower measurements that comes with scikit-learn. Each flower has four measurements (the features) and a species (the label). The example first splits the data into a training set and a test set with the train_test_split() function from sklearn.model_selection. It then trains the classifier on the training set with the fit() method and predicts the species of the test flowers with the predict() method. Finally, it compares the predictions with the real species using the accuracy_score() function from sklearn.metrics.
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
# X holds the measurements (features), y holds the species (labels)
X, y = load_iris(return_X_y=True)
# Keep 20% of the flowers aside to test the model
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
# Train a random forest classifier
clf = RandomForestClassifier(random_state=42)
clf.fit(X_train, y_train)
# Predict labels for the test set
y_pred = clf.predict(X_test)
# Compute the share of correct predictions
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy:.2f}')
The accuracy is a number between 0 and 1, where 1.00 means every test flower was classified correctly. Setting random_state makes the split and the model the same on every run, so you get the same result each time.
Web development
Python has many libraries and frameworks that make web applications easier to build and deploy, from simple websites to large web-based systems. A web framework takes care of the common work, such as routing (matching a URL to the code that handles it), templating, and database integration, so you can focus on what your site does.
Django
Django is a high-level web framework for building web applications quickly. It includes a lot out of the box: an ORM (object-relational mapper) for working with a database through Python classes, an admin site, user accounts, templates, and form handling. Django organizes code using the model-template-view (MTV) pattern: models describe your data, templates describe how pages look, and views contain the logic that connects the two.
A simple Django view that returns an HTTP response looks like this:
from django.http import HttpResponse
def index(request):
return HttpResponse('Hello, world!')
A Django view doesn’t run on its own. It lives inside a Django project, which you create with the django-admin startproject command, and you connect it to a URL in the project’s urls.py file. The official Django tutorial walks you through the whole setup.
Flask
Flask is a microframework: it gives you only the essentials, such as URL routing and templates, and you add other features with extensions when you need them. It is lightweight and flexible, and a good choice for small to medium-sized web applications and APIs.
Here is a simple Flask app that returns an HTTP response:
from flask import Flask
app = Flask(__name__)
@app.route('/')
def index():
return 'Hello, world!'
if __name__ == '__main__':
app.run()
Save this as app.py, run it, and open http://127.0.0.1:5000 in your browser to see the page. Press Ctrl+C in the terminal to stop the server.
Pyramid
Pyramid is a web framework designed to be flexible and extensible. Like Flask, it starts small, but it is built to grow into larger applications or ones that need custom behavior.
This simple Pyramid app also returns an HTTP response. It uses Waitress, a separate package, as the web server, so install both with python -m pip install pyramid waitress:
from pyramid.config import Configurator
from pyramid.response import Response
from waitress import serve
def index(request):
return Response('Hello, world!')
if __name__ == '__main__':
config = Configurator()
config.add_route('index', '/')
config.add_view(index, route_name='index')
app = config.make_wsgi_app()
serve(app, host='127.0.0.1', port=8080)
Run it and open http://127.0.0.1:8080 in your browser.
Automation
A short Python script can handle many everyday chores on a computer: renaming files, running other programs, downloading data, or doing the same job every hour. Python’s standard library already includes most of the tools you need.
Running command-line programs
The subprocess module lets your program run other programs and capture their output, so you can automate tasks that involve running several programs or shell commands in a row.
This example runs a second Python program and captures what it prints:
import subprocess
import sys
# sys.executable is the path of the Python interpreter running this
# script, so this example works the same on Windows, macOS, and Linux
result = subprocess.run(
[sys.executable, '-c', "print('Hello from another program')"],
capture_output=True, # Collect the output instead of showing it
text=True, # Give the output as a str instead of bytes
check=True, # Raise an exception if the program fails
)
print(result.returncode) # Output: 0
print(result.stdout.strip()) # Output: Hello from another program
You pass the command as a list: the program name first, then each argument as a separate item. For example, subprocess.run(['ls', '-l'], capture_output=True, text=True, check=True) lists the files in the current folder on macOS and Linux. Windows has no ls program, though, so check that the commands you run exist on every system your script needs to support.
A program signals a problem by finishing with an exit code other than 0. With check=True, subprocess.run() turns that into a CalledProcessError exception, so a failure can’t go unnoticed:
import subprocess
import sys
try:
# This program exits with code 3 to report a problem
subprocess.run([sys.executable, '-c', 'import sys; sys.exit(3)'], check=True)
except subprocess.CalledProcessError as error:
print(f'The program failed with exit code {error.returncode}')
# Output: The program failed with exit code 3
Interacting with web APIs
Many websites offer an API (application programming interface): a set of URLs that return data for programs instead of web pages, usually in JSON format. Python can send HTTP requests to these APIs and process the responses, so a script can fetch or update data on a website for you.
The third-party requests library is the most popular way to make HTTP requests. (The standard library’s urllib.request module can do it too, but takes more code.) The example below asks the GitHub API for information about the repository that holds Python’s own source code:
import requests
url = 'https://api.github.com/repos/python/cpython'
try:
# timeout=10 stops waiting if the server doesn't answer within 10 seconds
response = requests.get(url, timeout=10)
# Raise an exception if the server answered with an error, such as 404
response.raise_for_status()
except requests.RequestException as error:
print(f'Request failed: {error}')
else:
data = response.json() # Turn the JSON text into dicts and lists
print(data['full_name']) # Output: python/cpython
print(data['description'])
Always pass a timeout; without one, requests can wait forever for a server that never answers. raise_for_status() raises an HTTPError when the response has an error status code (400 or higher), so you don’t have to check response.status_code yourself. Catching requests.RequestException handles both that and network problems such as a missing internet connection.
Scheduling tasks
Python can run a task at a specific time or at regular intervals, which suits jobs that need to happen on a schedule, such as data backups or system maintenance.
The sched module from the standard library runs functions after a delay. The scheduler runs each scheduled event once, so to repeat a task, the task schedules its own next run. This example runs a task three times, 0.1 seconds apart, and then stops:
import sched
import time
INTERVAL = 0.1 # Seconds between runs (use 3600 for once an hour)
RUNS = 3 # How many times to run the task
scheduler = sched.scheduler(time.monotonic, time.sleep)
def task(run_number):
print(f'Running task (run {run_number})')
# Schedule the next run, unless this was the last one
if run_number < RUNS:
scheduler.enter(INTERVAL, 1, task, argument=(run_number + 1,))
# Schedule the first run
scheduler.enter(INTERVAL, 1, task, argument=(1,))
# run() waits for each event in turn, and returns when none are left
scheduler.run()
print('All runs finished')
# Output:
# Running task (run 1)
# Running task (run 2)
# Running task (run 3)
# All runs finished
The enter() method takes the delay in seconds, a priority (used when two events are due at the same time; lower numbers run first), the function to call, and a tuple of arguments for it. The run() method starts the scheduler.
For a task that repeats forever, a loop with time.sleep() is often enough:
import time
while True:
print('Running task...')
time.sleep(3600) # Wait one hour
A Python script can only run scheduled tasks while it is running. For jobs that must keep running after you close the terminal or restart the computer, use your operating system’s scheduler to start the script for you: cron on macOS and Linux, or Task Scheduler on Windows.
Scientific computing
Python is widely used in science and engineering. Its libraries let you do heavy numerical work with short, readable code.
Numerical computing
The numpy and scipy libraries have functions for numerical computations such as linear algebra, optimization, integration, and statistics. NumPy provides the arrays and basic operations; SciPy builds on it with more specialized tools.
This example uses NumPy for a linear algebra operation, finding the inverse of a matrix:
import numpy as np
# Create a 2x2 matrix
A = np.array([[1, 2], [3, 4]])
# Calculate the inverse of the matrix
A_inv = np.linalg.inv(A)
print(A_inv)
# Output:
# [[-2. 1. ]
# [ 1.5 -0.5]]
Machine learning
Libraries such as scikit-learn, tensorflow, and pytorch provide tools for training and evaluating machine learning models.
The next example uses scikit-learn to train a simple linear regression model, which finds the straight line that best fits some data:
import numpy as np
from sklearn.linear_model import LinearRegression
# Generate 100 random data points that roughly follow y = 5 + 3x
X = np.random.rand(100, 1)
y = 5 + 3 * X + np.random.rand(100, 1)
# Create a linear regression model
model = LinearRegression()
# Fit the model to the data
model.fit(X, y)
# Print the coefficient (slope) and intercept of the line it found
print(f'Coefficient: {model.coef_[0][0]:.2f}')
print(f'Intercept: {model.intercept_[0]:.2f}')
The data is random, so the numbers change on every run. The coefficient should come out close to 3, and the intercept close to 5.5: 5 plus 0.5, the average of the random noise, which lies between 0 and 1.
Scientific simulation
A simulation imitates a real system so you can study it without building it. The simpy library is used for discrete-event simulation: modeling systems where things happen at particular moments, such as customers arriving at a shop and waiting for a free cashier, or cars waiting at a traffic light. SimPy describes each actor in the simulation as a generator function (see Chapter 6) that uses yield to wait for an event, such as some time passing or a resource becoming free.
The example below models a simple queue: five customers arrive at a shop with one counter, wait their turn, are served, and leave. Simulated time is measured in minutes.
import random
import simpy
random.seed(42) # Use the same random numbers on every run
def customer(env, name, counter):
arrived = env.now
print(f'{env.now:5.2f}: {name} arrives')
with counter.request() as request:
yield request # Wait until the counter is free
waited = env.now - arrived
print(f'{env.now:5.2f}: {name} is served after waiting {waited:.2f}')
yield env.timeout(random.uniform(1, 3)) # Serving takes 1 to 3 minutes
print(f'{env.now:5.2f}: {name} leaves')
def customer_arrivals(env, counter):
for number in range(1, 6):
env.process(customer(env, f'Customer {number}', counter))
yield env.timeout(random.uniform(0.5, 1.5)) # Next customer in 0.5 to 1.5 minutes
# Create the simulation environment and a counter that serves one customer at a time
env = simpy.Environment()
counter = simpy.Resource(env, capacity=1)
# Start generating customers, then run the simulation until nothing is left to do
env.process(customer_arrivals(env, counter))
env.run()
The program prints a line each time a customer arrives, is served, or leaves, with the simulated time in minutes at the start of the line. random.uniform(a, b) picks a random number between a and b. Customers arrive faster than the counter can serve them, so a queue builds up and later customers have to wait. Try changing the counter’s capacity to 2 and see how the waiting times change.
Key takeaways
- In each of these fields, most of what Python can do comes from third-party packages. Install them with
python -m pip installinside a virtual environment. - pandas works with tables of data, NumPy works with arrays of numbers, and Matplotlib and seaborn turn data into charts.
- Django is a full-featured web framework; Flask and Pyramid start smaller and let you add what you need.
- The standard library covers a lot of automation on its own:
subprocessruns other programs andschedruns tasks on a timer. For regular jobs, use cron or Task Scheduler. - When you call a web API, always set a timeout and check the response for errors.
Practice
Exercise 1: Sales totals with the csv module
The pandas example in this chapter adds up revenue by region. Do the same with only the standard library. Write the sales data below to a file named sales.csv with csv.writer, read the file back with csv.DictReader, and print the total revenue (units times price) for each region with two decimal places. Remember that the csv module gives you every value as a string.
region,product,units,price
East,Pens,10,1.5
West,Pens,4,1.5
East,Paper,3,4.0
West,Paper,8,4.0
East,Pens,6,1.5
Solution
import csv
rows = [
['region', 'product', 'units', 'price'],
['East', 'Pens', 10, 1.5],
['West', 'Pens', 4, 1.5],
['East', 'Paper', 3, 4.0],
['West', 'Paper', 8, 4.0],
['East', 'Pens', 6, 1.5],
]
# Write the data to a CSV file
with open('sales.csv', 'w', newline='') as file:
csv.writer(file).writerows(rows)
# Read the file back and add up the revenue for each region
revenue_by_region = {}
with open('sales.csv', newline='') as file:
for row in csv.DictReader(file):
revenue = int(row['units']) * float(row['price'])
region = row['region']
revenue_by_region[region] = revenue_by_region.get(region, 0) + revenue
for region, total in revenue_by_region.items():
print(f'{region}: {total:.2f}')
# Output:
# East: 36.00
# West: 38.00
Exercise 2: Read JSON data
Web APIs usually send their data as JSON text. The string below is shaped like a response from a weather API. Use json.loads() to turn it into Python dictionaries and lists, then print the names of the cities warmer than 20 degrees and the average temperature of all the cities.
response_text = '''
{
"unit": "celsius",
"readings": [
{"city": "Kathmandu", "temp": 22},
{"city": "Oslo", "temp": 9},
{"city": "Cairo", "temp": 31},
{"city": "Lima", "temp": 18}
]
}
'''
Solution
import json
response_text = '''
{
"unit": "celsius",
"readings": [
{"city": "Kathmandu", "temp": 22},
{"city": "Oslo", "temp": 9},
{"city": "Cairo", "temp": 31},
{"city": "Lima", "temp": 18}
]
}
'''
data = json.loads(response_text)
readings = data['readings']
warm_cities = [r['city'] for r in readings if r['temp'] > 20]
print(f"Warm cities: {', '.join(warm_cities)}")
average = sum(r['temp'] for r in readings) / len(readings)
print(f"Average: {average} {data['unit']}")
# Output:
# Warm cities: Kathmandu, Cairo
# Average: 20.0 celsius
Exercise 3: Run a program and check the result
Write a function run_python(code) that runs a piece of Python code in a separate process with subprocess.run() and returns what it printed, without the trailing newline. Use sys.executable so that it works on every operating system. If the code fails, return the text failed with exit code N instead, where N is the program’s exit code. Test it with the code print(2 ** 10) and with the code 1 / 0.
Solution
import subprocess
import sys
def run_python(code):
try:
result = subprocess.run(
[sys.executable, '-c', code],
capture_output=True,
text=True,
check=True,
)
except subprocess.CalledProcessError as error:
return f'failed with exit code {error.returncode}'
return result.stdout.strip()
print(run_python('print(2 ** 10)')) # Output: 1024
print(run_python('1 / 0')) # Output: failed with exit code 1