Programming for All: Python Edition

Welcome to the Python edition of “Programming for All”!

About the book

“Programming for All: Python Edition” is for anyone who wants to learn Python, whatever their background or experience. You can start as a complete beginner, or use it to pick up Python as a new language if you already program.

How to use this book

Read in order if you’re new. Each chapter builds on the ones before it. If you already program, skim Chapter 1 and jump to what you need.
Type the examples yourself. Every example is a complete program, and the # Output: comments show exactly what Python prints, so you can compare your result. All examples are checked with Python 3.13 and work with Python 3.10 or newer.
Do the practice exercises. Each chapter ends with key takeaways and a few exercises. Try them before opening the solutions.
Use the tools at the top of the page to search the whole book, switch between light and dark themes, or print a chapter.

Introduction

Found a mistake?

If you spot an error or have a suggestion, please get in touch through rupeshbhandari.com.np.

License

This book is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. You are free to share and adapt the content for non-commercial purposes, as long as you give appropriate credit and share your adaptations under the same license.

Acknowledgments

Special thanks to everyone who has helped make this book possible.

Introduction to Python

Python is a programming language known for code that is easy to read and write, which makes it a good first language.

In this chapter you will learn:

Where Python came from and the ideas behind its design
The features that make Python popular
Which version of Python this book uses
How to install Python and choose an editor
How to write and run your first program, and how to use the interactive prompt

History of Python

Python is a high-level programming language, first released in 1991. Guido van Rossum, a Dutch computer scientist, created it while he was working at the National Research Institute for Mathematics and Computer Science (CWI) in the Netherlands.

Van Rossum named the language after the British comedy group Monty Python, not the snake. He wanted a language that was easy to read and fun to use. “The Zen of Python”, a short list of guiding principles for the language written by long-time Python developer Tim Peters, sums up that goal. Two of its best-known lines are “Readability counts.” and “There should be one– and preferably only one –obvious way to do it.” In other words, Python code should be clear, and there should usually be one natural way to solve a problem. You can read the whole list by running this one-line program:

import this

Python soon became popular because it was simple and readable, and because its extensive standard library let programmers do common tasks without writing all the code themselves. A large and active community of developers has helped it keep growing.

Python 3 was released in 2008. It was not fully compatible with the older Python 2, which was retired in 2020. This book uses Python 3 only. If you find an old tutorial that writes print "hello" without parentheses, it was written for Python 2.

Today Python is used in many fields, including web development, scientific computing, data analysis, automation, and artificial intelligence. It is one of the most popular programming languages in the world, with wide use in industries such as finance, education, and government.

Key features of Python

Python is known for being simple, readable, and flexible, which appeals to beginners and experienced programmers alike. These are its main features:

Interpreted language: You run Python code directly, without a separate build step. Behind the scenes, the standard Python program (called CPython) first translates your code into bytecode, a simpler set of instructions, and then its interpreter runs that bytecode. This happens automatically every time you run a program, so you can change a line, run it again, and see the result right away. It also means you can type code into Python and get answers one line at a time.
Dynamic typing: You don’t have to declare what type of data a variable will hold. Python works out the type from the value you assign. This keeps code short and flexible.
High-level language: Python hides the details of the computer’s hardware and operating system. This makes code easier to write and understand, and the same program usually runs on Windows, macOS, and Linux without changes.
Extensive standard library: Python comes with modules for tasks such as reading and writing files, working with dates and times, and connecting to web servers.
Large and active community: There are many free resources for learning Python, including tutorials, books, and forums, and thousands of third-party packages that add extra features.
Object-oriented programming: Python lets you build reusable code with classes and objects. You will learn about this in Chapter 5.
Versatility: Python can be used for a wide range of tasks, from small scripts that automate a chore to large web applications and data science projects.

Setting up a Python development environment

To write and run Python programs, you need two things: the Python interpreter and a text editor or integrated development environment (IDE) to write your code in.

Which version to install

This book targets Python 3.10 or newer, and its examples are checked with Python 3.13. Any Python 3 release from 3.10 on will run them. When in doubt, install the newest stable release offered on the Python website.

Install Python

Download Python from the official website: https://www.python.org/downloads/. The site usually suggests the right download for your operating system.

Windows: Run the installer. If it offers an option to add Python to PATH (for example a checkbox labeled “Add python.exe to PATH”), turn it on before you continue. PATH is the list of folders your computer searches for programs, so this lets you run Python from any command prompt window. The installer also gives you the py launcher, which you can use to run Python.
macOS: Run the installer package you downloaded and follow the steps.
Linux: Most distributions already include Python 3. If yours doesn’t, install it with your package manager, for example sudo apt install python3 on Ubuntu or Debian.

To check that Python is installed, open a terminal (Terminal on macOS and Linux, Command Prompt or PowerShell on Windows) and ask Python for its version. On macOS and Linux, type:

python3 --version

On Windows, type:

py --version

Python answers with its version number, for example Python 3.13.1. Make sure it is 3.10 or newer.

Choose an editor

You can write Python in any plain text editor, but an editor made for code makes life easier. A good choice is Visual Studio Code (VS Code) with the Python extension from Microsoft, which you can install from the Extensions view inside VS Code. It colors your code, points out mistakes as you type, and can run your programs with one click.

IDLE, the simple editor that comes with Python, is also fine for the examples in this book. PyCharm is another popular IDE made just for Python.

Your first program

Your first program will print a message on the screen.

Create a folder for your Python files, for example python-practice.
In your editor, create a new file in that folder called hello.py. The .py ending tells your computer and your editor that the file contains Python code.
Type this line into the file and save it:

print("Hello, world!")

print() is a built-in function that shows a value on the screen. The text inside the quotes is a string, which you’ll learn about in Chapter 1.

To run the program, open a terminal and use the cd command to move to your folder:

cd python-practice

Then run the file. On macOS and Linux, type:

python3 hello.py

On Windows, type:

py hello.py

You should see:

Hello, world!

In VS Code, you can also run the file by clicking the Run Python File button (the triangle) in the top-right corner of the editor.

When something goes wrong

If you make a mistake, Python stops and prints an error message called a traceback. For example, if you misspell print as prnt, you will see something like this (the file path will match your own folder):

Traceback (most recent call last):
  File "/Users/you/python-practice/hello.py", line 1, in <module>
    prnt("Hello, world!")
    ^^^^
NameError: name 'prnt' is not defined. Did you mean: 'print'?

Read the last line first: it names the type of error and usually explains what went wrong. The lines above it tell you which file and line caused the problem. Fix the mistake, save the file, and run it again.

The interactive prompt

You can also use Python interactively. Type python3 (or py on Windows) in a terminal without a file name, and Python starts its interactive prompt, also called the REPL. The prompt looks like >>>. Type a line of code, press Enter, and Python runs it straight away and shows the result:

>>> 2 + 3
5
>>> print("Hello, world!")
Hello, world!

The interactive prompt is handy for trying out small ideas. Notice that it shows the value of 2 + 3 without print(). In a program saved in a file, Python only shows what you print. That is why the examples in this book use print() and show what it prints in a comment:

print(2 + 3)  # Output: 5

To leave the interactive prompt, type exit() and press Enter. On macOS and Linux you can also press Ctrl+D.

Key takeaways

Python was created by Guido van Rossum and first released in 1991. Its design puts readability first.
Python compiles your code to bytecode and runs it automatically, so you can edit and rerun programs quickly.
This book uses Python 3.10 or newer. Check your version with python3 --version on macOS and Linux or py --version on Windows.
Save programs in .py files and run them with python3 file.py on macOS and Linux or py file.py on Windows.
The interactive prompt (>>>) is a quick way to try out code. Leave it with exit().
When you get an error, read the last line of the traceback first.

Chapter 1: Python Basics

This chapter covers the parts of Python you will use in every program.

In this chapter you will learn:

The basic rules of Python syntax, such as indentation and comments
Python’s basic data types: numbers, strings, booleans and None
How to convert values from one type to another
How to print values and format them with f-strings
How to use variables with arithmetic, comparison, logical and bitwise operators

Basic syntax

Python’s syntax is the set of rules for how code is written. Here is a small program that shows the most important rules:

# This is a comment. Python ignores everything after the # sign.
temperature = 25  # This line creates a variable called temperature

if temperature > 20:
    print("It's warm today.")
    print("No coat needed.")
print("Have a nice day!")

# Output:
# It's warm today.
# No coat needed.
# Have a nice day!

These are the rules to remember:

Statements: Each line usually holds one instruction, called a statement. You don’t need a semicolon at the end of a line.
Indentation: Python uses indentation to group lines into blocks. In the example, the two indented print() lines belong to the if statement, so they only run when the condition is true. The last line isn’t indented, so it always runs. Use four spaces for each level of indentation, and don’t mix tabs and spaces. If the indentation is wrong, Python stops with an IndentationError.
Comments: Anything after a # sign on a line is a comment. Python ignores comments, so you can use them to explain your code to other people, and to your future self.
Variables: You create a variable by assigning a value to a name with =. For example, x = 5 creates a variable called x that holds the value 5. You’ll learn more in Variables and operations.
Case sensitivity: Python treats uppercase and lowercase letters as different, so name, Name and NAME are three different names. Keywords such as if and True must be written exactly as shown.
Data types: Python has several built-in data types, including integers, floating-point numbers, strings, and booleans. The next section covers them.
Operators: Python has operators to work with data, including arithmetic operators (+, -, *, /), comparison operators (==, !=, >, <, >=, <=), and logical operators (and, or, not).

Later chapters build on these basics. Chapter 2 covers control structures (if statements, for loops and while loops) and functions, which you define with the def keyword. Exception handling with try and except, which lets your program deal with errors gracefully, comes in Chapter 4. Chapter 5 introduces object-oriented programming with classes and objects, and Chapter 6 covers modules, including Python’s large standard library.

Data types

Every value in Python has a type, which decides what you can do with it. For example, you can add two numbers, but you can’t add a number to a piece of text. You can check the type of any value with the built-in type() function.

This section covers these basic types:

Type	Example	What it holds
`int`	`42`	Whole numbers
`float`	`3.14`	Numbers with a decimal point
`complex`	`3+4j`	Complex numbers, with a real and an imaginary part
`str`	`"hello"`	Text
`bool`	`True`	`True` or `False`
`NoneType`	`None`	No value at all

It also gives you a first look at the collection types list, tuple, dict and set, which hold many values at once.

Integers

Integers are whole numbers that can be positive, negative, or zero. For example, 0, 1, -2, and 100 are all integers. Python’s type for integers is int, and an integer can be as large as your computer’s memory allows.

# Declaring and using an integer
x = 10
print(x)  # Output: 10

# Checking the type
print(type(x))  # Output: <class 'int'>

# Adding integers
y = 20
z = x + y
print(z)  # Output: 30

# Dividing integers with / always gives a float
a = 10
b = 3
print(a / b)  # Output: 3.3333333333333335
print(10 / 2)  # Output: 5.0

# Floor division with // rounds down to a whole number
print(a // b)  # Output: 3

# The % operator (modulo) gives the remainder of a division
print(a % b)  # Output: 1

# Exponentiation: 10 to the power of 3
print(a ** b)  # Output: 1000

# Underscores make long numbers easier to read
population = 8_000_000_000
print(population)  # Output: 8000000000

Floats

Floats (short for floating-point numbers) are numbers with a decimal point. For example, 0.5, 3.14, and -1.0 are all floats. Python’s type for them is float.

# Declaring and using a float
x = 10.5
print(x)  # Output: 10.5
print(type(x))  # Output: <class 'float'>

# Adding floats
y = 20.7
print(x + y)  # Output: 31.2

# Dividing floats
a = 10.5
b = 3.1
print(a / b)  # Output: 3.3870967741935485

# Exponentiation
print(a ** 2)  # Output: 110.25

# The remainder of a division works with floats too
print(a % b)  # Output: 1.1999999999999997

The last result is 1.1999999999999997 rather than 1.2. That’s not a mistake in your code. Computers store floats in binary, and most decimal numbers, such as 3.1, can’t be stored exactly that way. The tiny errors this causes usually don’t matter, but sometimes they show up in results. Use round() to round a float to a number of decimal places, and be careful when you compare floats with ==:

print(0.1 + 0.2)  # Output: 0.30000000000000004
print(round(0.1 + 0.2, 2))  # Output: 0.3
print(round(10.5 % 3.1, 2))  # Output: 1.2
print(0.1 + 0.2 == 0.3)  # Output: False

Complex numbers

Complex numbers have a real part and an imaginary part. You will rarely need them unless you work with math, science or engineering. Python writes the imaginary part with a j (instead of the i used in math), so 3+4j is a complex number with a real part of 3 and an imaginary part of 4.

You can also use the complex() function to create complex numbers:

# A real part of 3 and an imaginary part of 4
z = complex(3, 4)
print(z)  # Output: (3+4j)
print(type(z))  # Output: <class 'complex'>

# A real part of 2.5 and an imaginary part of -1
print(complex(2.5, -1))  # Output: (2.5-1j)

# A real part of 0 and an imaginary part of 1
print(complex(0, 1))  # Output: 1j

You can do arithmetic with complex numbers just like with other numbers:

a = 3+4j
b = 1+2j

print(a + b)  # Output: (4+6j)
print(a - b)  # Output: (2+2j)
print(a * b)  # Output: (-5+10j)
print(a / b)  # Output: (2.2-0.4j)

The built-in abs() function gives the absolute value of a complex number, which is its distance from 0 in the complex plane. The .real and .imag attributes give its real and imaginary parts. They are attributes, not functions, so you write them without parentheses:

z = 3+4j

print(abs(z))  # Output: 5.0
print(z.real)  # Output: 3.0
print(z.imag)  # Output: 4.0

Strings

Strings are sequences of characters, such as words or sentences. For example, “hello”, “python”, and “The quick brown fox jumps over the lazy dog” are all strings. Python’s type for strings is str.

You write a string inside single or double quotes: 'hello' and "hello" are the same string. To put a quote character inside a string, surround the string with the other kind of quote, as in "It's warm today.". When you print a string, Python shows its characters without the quotes.

# Declaring and using a string
x = "hello"
print(x)  # Output: hello
print(type(x))  # Output: <class 'str'>

# Joining (concatenating) strings with +
y = "world"
print(x + y)  # Output: helloworld
print(x + " " + y)  # Output: hello world

# Getting the length of a string
print(len(x))  # Output: 5

# Accessing individual characters: positions start at 0
print(x[0])  # Output: h
print(x[-1])  # Output: o

# Slicing: characters at positions 1 and 2 (but not 3)
print(x[1:3])  # Output: el

# Repeating a string
print(x * 3)  # Output: hellohellohello

# Checking if a string contains another string
print("he" in x)  # Output: True
print("hi" in x)  # Output: False

# Changing the case of a string
print(x.upper())  # Output: HELLO
print("HeLLo".lower())  # Output: hello

A negative position counts from the end, so x[-1] is the last character. Strings can’t be changed after they are created. Methods such as upper() give you a new string and leave the original as it was.

Booleans

Booleans represent one of two values: True or False. Python’s type for them is bool. Note the capital letters: true and false aren’t valid Python. Comparisons give you a boolean result.

flag = True
print(flag)  # Output: True
print(type(flag))  # Output: <class 'bool'>

# Comparisons give a boolean result
print(5 > 3)  # Output: True
print(5 == 3)  # Output: False

# isinstance() checks whether a value has a given type
is_bool = isinstance(flag, bool)
print(is_bool)  # Output: True

Booleans are what if statements use to decide which code to run. You can combine them with and, or and not, which are covered in Logical operations below.

is_raining = True
has_umbrella = False

if is_raining:
    print("It's raining.")
else:
    print("It's dry.")

# and is True only if both sides are True; not flips True and False
if is_raining and not has_umbrella:
    print("You'll get wet!")

# Output:
# It's raining.
# You'll get wet!

None

None is a special value that means “no value” or “nothing here yet”. Its type is NoneType, and None is the only value of that type. You’ll often use it as a placeholder for a value you don’t know yet.

middle_name = None
print(middle_name)  # Output: None
print(type(middle_name))  # Output: <class 'NoneType'>

# Use "is" to check whether a value is None
print(middle_name is None)  # Output: True

middle_name = "Rose"
print(middle_name is None)  # Output: False

None isn’t the same as 0, False or an empty string "". It is a value of its own. You’ll also see it in Chapter 2: a function that doesn’t return anything gives back None.

Collections at a glance

So far, each value has been a single piece of data. Collections hold many values in one. Python has four main built-in collection types. Here is a quick look at each one.

A list is an ordered collection that you can change:

fruits = ["apple", "banana", "cherry"]
print(fruits[0])  # Output: apple

fruits.append("mango")
print(fruits)  # Output: ['apple', 'banana', 'cherry', 'mango']
print(len(fruits))  # Output: 4

When you print a whole list, Python shows the strings in it with quotes, the way it displays any string inside a collection.

A tuple is an ordered collection that you can’t change after you create it:

point = (3, 4)
print(point)  # Output: (3, 4)
print(point[0])  # Output: 3

A dictionary (dict) stores pairs of keys and values. You look up a value by its key:

person = {"name": "Asha", "age": 30}
print(person["name"])  # Output: Asha

person["age"] = 31
print(person)  # Output: {'name': 'Asha', 'age': 31}

A set is an unordered collection with no duplicate values:

numbers = {3, 1, 2, 3, 1}
print(len(numbers))  # Output: 3
print(2 in numbers)  # Output: True

# Sets have no fixed order, so sort them when you want ordered output
print(sorted(numbers))  # Output: [1, 2, 3]

Chapter 3 covers lists, tuples, dictionaries and sets in detail, along with frozen sets, bytes and byte arrays.

Type conversion

Sometimes you have a value of one type but need another. A common case is text that contains a number. For example, when a user types a number into your program (in a real program this might come from input()), it arrives as a string, and you have to convert it before you can do math with it.

Python’s conversion functions are named after the types they convert to: int() gives you an integer, float() a float, str() a string and bool() a boolean.

# Strings to numbers
age_text = "42"
age = int(age_text)
print(age + 1)  # Output: 43

price = float("19.99")
print(price)  # Output: 19.99

# Between floats and integers
print(int(3.9))  # Output: 3
print(float(7))  # Output: 7.0

# Numbers to strings
count = 3
message = "You have " + str(count) + " new messages"
print(message)  # Output: You have 3 new messages

# Values to booleans
print(bool(0))  # Output: False
print(bool(42))  # Output: True
print(bool(""))  # Output: False
print(bool("hello"))  # Output: True
print(bool(None))  # Output: False

Notice that int() doesn’t round a float: it drops the part after the decimal point, so int(3.9) is 3. Use round() if you want rounding. With bool(), zero, empty strings, empty collections and None count as False, and almost everything else counts as True.

You need str() in the example above because Python won’t join a string and a number with +. If you try, you get a TypeError:

count = 3
try:
    message = "You have " + count + " new messages"
except TypeError as error:
    print(error)  # Output: can only concatenate str (not "int") to str

If a string doesn’t contain a valid number, the conversion fails with a ValueError. Even int("3.5") fails, because “3.5” isn’t a whole number; use float() for that. The examples here use try and except to catch the error and print its message instead of stopping the program. You’ll learn how they work in Chapter 4.

text = "hello"
try:
    number = int(text)
except ValueError as error:
    print(error)  # Output: invalid literal for int() with base 10: 'hello'

Printing and f-strings

You have already used print() to show values. It has a few more options, and f-strings give you an easy way to build text from values. You’ll see both all through this book.

Printing several values

print() can take several values separated by commas. It prints them on one line with a space between each, and then moves to a new line. The sep argument changes the separator, and end changes what is printed at the end instead of the new line.

name = "Asha"
age = 30
print("Name:", name, "Age:", age)  # Output: Name: Asha Age: 30

print(2026, 10, 20, sep="-")  # Output: 2026-10-20

# end="... " keeps the next print on the same line
print("Loading", end="... ")
print("done")

# Output: Loading... done

f-strings

An f-string is a string with the letter f before the opening quote. Inside it, you can put any variable or expression in curly braces {}, and Python replaces it with its value. This is easier than joining strings with +, and you don’t need str().

name = "Asha"
price = 4.5
quantity = 3

print(f"Hello, {name}!")  # Output: Hello, Asha!
print(f"{quantity} apples cost {price * quantity}")  # Output: 3 apples cost 13.5

After the value, you can add a colon and a format specification to control how the value looks. The most useful one is .2f, which shows a number with exactly two digits after the decimal point:

total = 4.5 * 3
print(f"Total: {total}")  # Output: Total: 13.5
print(f"Total: {total:.2f}")  # Output: Total: 13.50
print(f"Pi is about {3.14159:.2f}")  # Output: Pi is about 3.14

Adding !r after a value shows it the way you would write it in Python code. For a string, that means with quotes, which helps you spot extra spaces when you’re tracking down a bug:

user_name = "Asha "
print(f"Hello, {user_name}!")  # Output: Hello, Asha !
print(f"Hello, {user_name!r}!")  # Output: Hello, 'Asha '!

Variables and operations

Variables are names that refer to values. You create one by assigning a value to a name with =. You can give a variable a new value later by assigning to it again, and you can use a variable anywhere you would use its value.

x = 10
y = 20

# Reassigning values to variables
x = 30
y = 40

# Using variables in expressions
z = x + y
print(z)  # Output: 70

The = sign means “assign this value to this name”. It doesn’t mean “is equal to”: to check whether two values are equal, you use ==.

A few rules and conventions for naming variables:

Names can contain letters, digits and underscores, but can’t start with a digit. total2 is fine, 2total isn’t.
Names are case-sensitive: score and Score are different variables.
You can’t use Python’s keywords, such as if, for or class, as names.
By convention, Python programmers write variable names in lowercase with underscores between words, like first_name. Choose names that say what the value means: price is clearer than p.

Python supports several kinds of operations, including arithmetic, comparison, logical and bitwise operations. Here are a few examples of each.

Arithmetic operations

x = 10
y = 20

# Addition
print(x + y)  # Output: 30

# Subtraction
print(x - y)  # Output: -10

# Multiplication
print(x * y)  # Output: 200

# Division
print(x / y)  # Output: 0.5

# Modulus (remainder)
print(x % y)  # Output: 10

# Exponentiation
print(x ** y)  # Output: 100000000000000000000

# Floor division (quotient without the remainder)
print(x // y)  # Output: 0

Python follows the usual order of operations from math: ** first, then *, /, // and %, and finally + and -. Use parentheses to change the order, or just to make it clearer:

print(2 + 3 * 4)  # Output: 14
print((2 + 3) * 4)  # Output: 20

To update a variable based on its current value, you can use a shortcut such as +=. Writing score += 5 does the same as score = score + 5. The other arithmetic operators have shortcuts too, such as -=, *= and /=.

score = 10
score += 5
print(score)  # Output: 15

score -= 3
print(score)  # Output: 12

score *= 2
print(score)  # Output: 24

Comparison operations

Comparison operators compare two values and give True or False.

x = 10
y = 20

# Equal to
print(x == y)  # Output: False

# Not equal to
print(x != y)  # Output: True

# Greater than
print(x > y)  # Output: False

# Less than
print(x < y)  # Output: True

# Greater than or equal to
print(x >= y)  # Output: False

# Less than or equal to
print(x <= y)  # Output: True

Logical operations

Logical operators combine booleans. and is True only if both sides are True, or is True if at least one side is True, and not flips True to False and back.

x = True
y = False

# AND
print(x and y)  # Output: False

# OR
print(x or y)  # Output: True

# NOT
print(not x)  # Output: False

You’ll often use them together with comparisons:

age = 25
print(age >= 18 and age < 65)  # Output: True
print(age < 13 or age > 64)  # Output: False

Bitwise operations

Bitwise operators work on the individual bits (binary digits) of integers. You won’t need them often as a beginner, but it helps to recognize them. The bin() function shows the binary form of a number, starting with 0b.

x = 10  # Binary: 01010
y = 20  # Binary: 10100

print(bin(x))  # Output: 0b1010
print(bin(y))  # Output: 0b10100

# AND: 1 only where both bits are 1 (01010 & 10100 = 00000)
print(x & y)  # Output: 0

# OR: 1 where either bit is 1 (01010 | 10100 = 11110)
print(x | y)  # Output: 30

# XOR: 1 where exactly one bit is 1 (01010 ^ 10100 = 11110)
print(x ^ y)  # Output: 30

# NOT: flips all the bits; for integers, ~x is the same as -(x + 1)
print(~x)  # Output: -11

# Left shift: moves the bits one place left, which doubles the number
print(x << 1)  # Output: 20

# Right shift: moves the bits one place right, which halves the number
print(y >> 1)  # Output: 10

Key takeaways

Python uses indentation, usually four spaces, to group code into blocks, and # starts a comment.
The basic data types are int, float, complex, str, bool and NoneType. Use type() to check a value’s type.
/ always gives a float, while // and % give the whole-number quotient and the remainder. Floats can have tiny rounding errors.
Convert between types with int(), float(), str() and bool(). Converting text that isn’t a valid number raises a ValueError.
f-strings such as f"Total: {total:.2f}" are the easiest way to put values into text.
= assigns a value to a variable, while == checks whether two values are equal.

Practice

Exercise 1: Temperature converter

Store a temperature of 21.5 degrees Celsius in a variable. Convert it to Fahrenheit with the formula celsius * 9 / 5 + 32, and print the result with one digit after the decimal point, like this: 21.5 degrees Celsius is 70.7 degrees Fahrenheit.

Solution

celsius = 21.5
fahrenheit = celsius * 9 / 5 + 32
print(f"{celsius} degrees Celsius is {fahrenheit:.1f} degrees Fahrenheit")

# Output:
# 21.5 degrees Celsius is 70.7 degrees Fahrenheit

Exercise 2: Minutes to hours

A movie is 135 minutes long. Use // and % to work out how many whole hours and leftover minutes that is, and print: 135 minutes is 2 hours and 15 minutes.

Solution

total_minutes = 135
hours = total_minutes // 60
minutes = total_minutes % 60
print(f"{total_minutes} minutes is {hours} hours and {minutes} minutes")

# Output:
# 135 minutes is 2 hours and 15 minutes

Exercise 3: Order total from text

Values that a user types arrive as text. Start with price_text = "4.75" and quantity_text = "3". Convert them to numbers, work out the total, and print it with two decimal places: Total: 14.25. Then set quantity_text to "three", try to convert it, and use try and except ValueError to print the error message instead of crashing.

Solution

price_text = "4.75"
quantity_text = "3"

price = float(price_text)
quantity = int(quantity_text)
total = price * quantity
print(f"Total: {total:.2f}")

# Text that isn't a number raises a ValueError
quantity_text = "three"
try:
    quantity = int(quantity_text)
except ValueError as error:
    print(error)

# Output:
# Total: 14.25
# invalid literal for int() with base 10: 'three'

Chapter 2: Control Flow

So far, your programs have run from top to bottom, one line after another. Control flow lets a program make decisions, repeat work, and group steps into reusable pieces.

In this chapter you will learn:

How to run code only when a condition is true, with if, elif, and else
How to write short conditions as one-line expressions and compare a value against several options with match
How to repeat code with for and while loops, and how break and continue change a loop’s flow
How to define functions with parameters, default values, and return values
How variable scope works inside and outside functions

Conditional statements

A conditional statement runs a block of code only if a condition is met. This section covers:

if statements
if-else statements
if-elif-else statements
The match statement
Conditional expressions (the ternary operator)
Combining conditions with and, or, and not

if statements

The simplest conditional statement is the if statement. Its syntax is:

if condition:
    # code to be executed if condition is True

Here, condition is an expression that evaluates to a boolean value (either True or False). If the condition is True, the code inside the if block runs. Otherwise, it is skipped.

The colon at the end of the if line and the indentation below it matter. Python uses indentation, by convention 4 spaces, to tell which lines belong to the block.

Here’s an example:

x = 5

if x > 0:
    print("x is positive")

# Output:
# x is positive

if-else statements

To run one block of code when the condition is True and a different block when it is False, use the if-else statement:

if condition:
    # code to be executed if condition is True
else:
    # code to be executed if condition is False

Here’s an example:

x = -5

if x > 0:
    print("x is positive")
else:
    print("x is not positive")

# Output:
# x is not positive

if-elif-else statements

When you need to check several conditions and run different code depending on which one is met, use the if-elif-else statement (elif is short for “else if”):

if condition1:
    # code to be executed if condition1 is True
elif condition2:
    # code to be executed if condition1 is False and condition2 is True
elif condition3:
    # code to be executed if condition1 and condition2 are False and condition3 is True
...
else:
    # code to be executed if all conditions are False

Python checks the conditions from top to bottom and runs only the first block whose condition is True. You can have as many elif branches as you need, and the final else is optional.

Here’s an example:

x = 0

if x > 0:
    print("x is positive")
elif x < 0:
    print("x is negative")
else:
    print("x is zero")

# Output:
# x is zero

The match statement

When you compare one value against several fixed options, a long if-elif-else chain can get repetitive. Python 3.10 and newer offer the match statement for this:

command = "exit"

match command:
    case "start":
        print("Starting the game")
    case "quit" | "exit":
        print("Goodbye")
    case _:
        print("Unknown command")

# Output:
# Goodbye

Python compares command against each case from top to bottom and runs the first one that matches. The | means “or”, so case "quit" | "exit": matches either word. The underscore _ matches anything, so case _: works like a final else. match can do much more, such as matching the shape of lists and dictionaries, but comparing a value against a few options is its most common use.

Conditional expressions (the ternary operator)

A conditional expression, also known as the ternary operator, picks one of two values based on a condition. It lets you write a simple if-else statement on a single line.

The syntax is:

value_if_true if condition else value_if_false

Here’s the same choice written both ways:

age = 20

# Traditional if-else statement
if age >= 18:
    status = "Adult"
else:
    status = "Minor"
print(status)  # Output: Adult

# Equivalent conditional expression
status = "Adult" if age >= 18 else "Minor"
print(status)  # Output: Adult

In this example, status is set to "Adult" if age is 18 or older, and "Minor" otherwise.

Another example

Suppose you want to describe whether a number is positive or not:

number = 5
result = "Positive" if number > 0 else "Not positive"
print(result)  # Output: Positive

Nested conditional expressions

You can also nest conditional expressions, although this can make code harder to read:

number = 0
result = "Positive" if number > 0 else "Negative" if number < 0 else "Zero"
print(result)  # Output: Zero

In this example, result is "Positive" if number is greater than 0, "Negative" if number is less than 0, and "Zero" if number is exactly 0.

Conditional expressions make code shorter, but use them only for simple choices. For anything more complex, a regular if-elif-else statement is easier to read.

Combining conditions with and, or, and not

You can combine conditions with the logical operators and, or, and not:

age = 25
has_ticket = True

if age >= 18 and has_ticket:
    print("You can enter")

if not has_ticket:
    print("Please buy a ticket")

# Output:
# You can enter

The truth table below shows the result of each operator for every combination of A and B:

A	B	A and B	A or B	not A
False	False	False	False	True
False	True	False	True	True
True	False	False	True	False
True	True	True	True	False

So and is True only when both sides are True, or is True when at least one side is True, and not flips a value.

Loops

Loops let you repeat a block of code multiple times. Python has two kinds: for loops and while loops.

for loops

A for loop goes through a sequence of elements, such as a list, tuple, or string, one element at a time. The syntax for a for loop is:

for element in sequence:
    # code to be executed for each element

Here’s an example of a for loop that goes through a list of numbers and prints each number:

numbers = [1, 2, 3, 4, 5]

for number in numbers:
    print(number)

# Output:
# 1
# 2
# 3
# 4
# 5

On each pass through the loop, the variable number holds the next item from the list.

Counting with range()

To repeat something a set number of times, use the range() function. range(5) produces the numbers 0 to 4: it starts at 0 and stops before 5.

for i in range(5):
    print(i)

# Output:
# 0
# 1
# 2
# 3
# 4

You can also give range() a start value, a stop value, and a step value. This loop starts at 2, stops before 6, and counts up by 2:

for i in range(2, 6, 2):
    print(i)

# Output:
# 2
# 4

Getting the position with enumerate()

If you need each item’s position as well as the item, use enumerate(). It gives you pairs of an index and an item, which you can unpack into two variables:

fruits = ["apple", "banana", "cherry"]

for index, fruit in enumerate(fruits):
    print(index, fruit)

# Output:
# 0 apple
# 1 banana
# 2 cherry

Counting starts at 0. To start at 1, write enumerate(fruits, start=1).

while loops

A while loop repeats a block of code as long as a condition is True. The syntax for a while loop is:

while condition:
    # code to be executed while condition is True

Here’s an example of a while loop that prints the numbers 1 to 5:

i = 1
while i <= 5:
    print(i)
    i += 1

# Output:
# 1
# 2
# 3
# 4
# 5

Make sure something inside the loop eventually makes the condition False. Here, i += 1 increases i each time. Without it, i would stay 1 and the loop would run forever, which is called an infinite loop. If that happens, press Ctrl+C to stop the program.

break and continue

You can use the break and continue statements to control the flow of a loop. break exits the loop completely, while continue skips the rest of the current pass and moves on to the next one. Both work in for and while loops.

i = 1
while True:
    if i > 5:
        break
    elif i % 2 == 0:
        i += 1
        continue
    print(i)
    i += 1

# Output:
# 1
# 3
# 5

while True would run forever on its own, so the break is what ends this loop once i goes past 5. For even numbers, continue jumps back to the top of the loop before print(i) runs, so only the odd numbers are printed.

Functions

A function is a named block of code that does one job. You define it once and then call it whenever you need that job done, instead of repeating the same code.

Defining and calling functions

To define a function, use the def keyword followed by the function name, a set of parentheses containing the function’s parameters, and a colon. The indented lines below form the function’s body:

def greet(name):
    print("Hello, " + name)

greet("John")  # Output: Hello, John
greet("Maria")  # Output: Hello, Maria

Defining a function doesn’t run it. The body runs only when you call the function by writing its name followed by parentheses containing the values you want to pass in, as in greet("John").

Parameters and arguments

These two words are easy to mix up:

A parameter is a variable name listed in the function definition. In def greet(name):, name is a parameter.
An argument is the actual value you pass when you call the function. In greet("John"), "John" is the argument.

When the function runs, each parameter takes the value of its argument. So during the call greet("John"), name holds "John", and the function prints Hello, John.

Default values and keyword arguments

You can give a parameter a default value, which is used when the caller leaves that argument out. You can also pass arguments by name, called keyword arguments, so their order doesn’t matter:

def greet(name, greeting="Hello"):
    print(f"{greeting}, {name}!")

greet("John")  # Output: Hello, John!
greet("John", "Good morning")  # Output: Good morning, John!
greet(greeting="Hi", name="Sam")  # Output: Hi, Sam!

The string is an f-string, which you met in Chapter 1: Python replaces {greeting} and {name} with their values. Parameters with default values must come after parameters without them in the definition.

Returning a value

Functions can also send a value back to the caller with the return keyword:

def add(x, y):
    """Return the sum of x and y."""
    return x + y

result = add(3, 4)
print(result)  # Output: 7
print(add(10, 5) * 2)  # Output: 30

The add function has two parameters, x and y, and returns their sum. When you call it with the arguments 3 and 4, it returns the value 7, which is stored in the result variable. You can use a returned value anywhere you could use the value itself, such as in a calculation.

The string in triple quotes right below the def line is a docstring. It describes what the function does, and tools such as help(add) display it. Adding one to each function you write is a good habit.

A function without a return statement returns the special value None. Printing inside a function is not the same as returning:

def greet(name):
    print("Hello, " + name)

value = greet("John")  # Output: Hello, John
print(value)  # Output: None

Scope and global variables

The scope of a variable is the part of the program where you can use it. There are two main kinds of scope:

A global variable is created outside any function. You can read it anywhere in the file, including inside functions.
A local variable is created inside a function, and parameters are local too. It exists only while the function runs.

For example:

x = 10  # global variable

def func1():
    print(x)  # reads the global x

def func2():
    x = 20  # creates a new local variable named x
    print(x)

func1()  # Output: 10
func2()  # Output: 20
print(x)  # Output: 10

The func1 function has no variable named x of its own, so it reads the global x and prints 10.

The func2 function assigns to x, which creates a local variable that shadows the global variable with the same name. Inside func2, x means the local variable, so it prints 20. Outside func2, the global x is unchanged and still has the value 10.

Changing a global variable

Assigning to a variable inside a function always creates a local variable, unless you say otherwise. To change a global variable from inside a function, declare it with the global keyword first:

count = 0

def increment():
    global count
    count += 1

increment()
increment()
print(count)  # Output: 2

Without the global line, count += 1 fails. Python sees the assignment, treats count as a local variable for the whole function, and then finds that this local variable has no value yet when it tries to add 1. The result is an UnboundLocalError. The try and except lines below catch the error and print its message instead of stopping the program. You’ll learn about them in Chapter 4.

count = 0

def increment():
    count += 1  # count is local here, but has no value yet

try:
    increment()
except UnboundLocalError as error:
    print(error)

# Output:
# cannot access local variable 'count' where it is not associated with a value

Python 3.10 words this message differently, but the error is the same.

Using global works, but it is usually better to pass values in as arguments and return the result. A function that only uses its parameters and return value is easier to understand, test, and reuse:

def increment(count):
    return count + 1

count = 0
count = increment(count)
count = increment(count)
print(count)  # Output: 2

Key takeaways

if, elif, and else run different code depending on conditions. Python runs only the first block whose condition is True.
match (Python 3.10+) compares one value against several options, and case _ catches everything else.
for loops go through each item in a sequence. range() produces a series of numbers, and enumerate() gives you each item’s index too.
while loops repeat as long as a condition is True, so make sure it eventually becomes False. break leaves a loop early, and continue skips to the next pass.
Functions receive arguments through their parameters, can have default values, and send results back with return.
Assigning to a variable inside a function makes it local. Prefer returning values over changing global variables with global.

Practice

Exercise 1: Letter grades

Store a test score from 0 to 100 in a variable named score. Print the letter grade: A for 90 and above, B for 80 to 89, C for 70 to 79, D for 60 to 69, and F below 60. Try several scores to check every branch.

Solution

score = 82

if score >= 90:
    grade = "A"
elif score >= 80:
    grade = "B"
elif score >= 70:
    grade = "C"
elif score >= 60:
    grade = "D"
else:
    grade = "F"

print(f"A score of {score} gets a {grade}")
# Output:
# A score of 82 gets a B

Exercise 2: FizzBuzz

Print the numbers from 1 to 15. For multiples of 3, print Fizz instead of the number. For multiples of 5, print Buzz. For multiples of both 3 and 5, print FizzBuzz.

Solution

for number in range(1, 16):
    if number % 3 == 0 and number % 5 == 0:
        print("FizzBuzz")
    elif number % 3 == 0:
        print("Fizz")
    elif number % 5 == 0:
        print("Buzz")
    else:
        print(number)
# Output:
# 1
# 2
# Fizz
# 4
# Buzz
# Fizz
# 7
# 8
# Fizz
# Buzz
# 11
# Fizz
# 13
# 14
# FizzBuzz

The check for both 3 and 5 has to come first. If it came after the check for 3, Python would print Fizz for 15 and never reach it.

Exercise 3: Counting vowels

Write a function count_vowels(text) with a docstring. It should return the number of vowels (a, e, i, o, u) in text, counting uppercase and lowercase letters. Test it on "Hello, World" and "Python".

Solution

def count_vowels(text):
    """Return the number of vowels in text."""
    count = 0
    for letter in text.lower():
        if letter in "aeiou":
            count += 1
    return count

print(count_vowels("Hello, World"))  # Output: 3
print(count_vowels("Python"))  # Output: 1

text.lower() converts the text to lowercase, so you only need to check lowercase vowels.

Chapter 3: Working with Data Structures

This chapter covers Python’s built-in data structures: lists, tuples, dictionaries, and sets, plus bytes and bytearray for raw binary data. You’ll need them whether you’re storing a list of favorite pizza toppings or building a complex data model.

In this chapter you will learn:

How to create, change, sort, and copy lists
How to use tuples for fixed data, unpack them, and return several values from a function
How to store and look up values by key with dictionaries
How to remove duplicates and compare groups of items with sets and frozen sets
How to work with raw binary data using bytes and bytearray
How to choose the right structure for the job

1. Lists

A list is an ordered, mutable collection. It can hold elements of any data type, and one list can mix several types.

1.1. Defining and initializing lists

Write a list as items separated by commas inside square brackets []:

# A simple list of integers
my_list = [1, 2, 3, 4]

# A mixed list (it can even hold another list)
mixed_list = [42, "hello", 3.14, True, [1, 2, 3]]

# An empty list, ready to be filled later
empty_list = []

print(my_list)  # Output: [1, 2, 3, 4]
print(mixed_list)  # Output: [42, 'hello', 3.14, True, [1, 2, 3]]
print(len(my_list))  # Output: 4
print(len(empty_list))  # Output: 0

1.2. Accessing and slicing

List indexes start at 0, so the first item is at index 0. Negative indexes count from the end, so -1 is the last item. To get several items at once, take a slice with my_list[start:stop:step]. The item at the stop index is not included.

my_list = [1, 2, 3, 4]

# Access the third element (index 2)
print(my_list[2])  # Output: 3

# Access the last element
print(my_list[-1])  # Output: 4

# Get a sublist from index 1 up to (but not including) index 3
print(my_list[1:3])  # Output: [2, 3]

# Reverse a list using slicing
print(my_list[::-1])  # Output: [4, 3, 2, 1]

1.3. Modifying lists

Lists are mutable, so you can change an element by assigning to its index. To add elements, use append(), insert(), or extend(). To remove them, use remove(), pop(), or del.

my_list = [1, 2, 3, 4]

# Change an element
my_list[3] = 5
print(my_list)  # Output: [1, 2, 3, 5]

# append() adds one item to the end
my_list.append(6)
print(my_list)  # Output: [1, 2, 3, 5, 6]

# insert() puts an item at a specific index (here, 10 goes in at index 1)
my_list.insert(1, 10)
print(my_list)  # Output: [1, 10, 2, 3, 5, 6]

# extend() adds every item from another list
my_list.extend([7, 8])
print(my_list)  # Output: [1, 10, 2, 3, 5, 6, 7, 8]

Watch the difference between append() and extend(): my_list.append([7, 8]) would add the whole list [7, 8] as a single item at the end.

The next example continues with the same list and takes items back out:

# remove() deletes the first item with a given value
my_list.remove(10)
print(my_list)  # Output: [1, 2, 3, 5, 6, 7, 8]

# pop() removes an item and returns it (the last one by default)
last_item = my_list.pop()
print(last_item)  # Output: 8
print(my_list)  # Output: [1, 2, 3, 5, 6, 7]

# del removes the item at an index without returning it
del my_list[0]
print(my_list)  # Output: [2, 3, 5, 6, 7]

You can also give pop() an index, as in my_list.pop(0), to remove and return the item at that position.

1.4. Sorting lists

There are two ways to sort a list, and they behave differently:

sorted(my_list) returns a new sorted list and leaves the original alone.
my_list.sort() sorts the list in place (it changes the original) and returns None.

scores = [42, 7, 19, 88, 3]

# sorted() gives you a new list
print(sorted(scores))  # Output: [3, 7, 19, 42, 88]
print(scores)  # Output: [42, 7, 19, 88, 3]

# sort() changes the list itself
scores.sort()
print(scores)  # Output: [3, 7, 19, 42, 88]

# Both accept reverse=True for largest-first order
scores.sort(reverse=True)
print(scores)  # Output: [88, 42, 19, 7, 3]

# A common mistake: sort() returns None, so don't save its result
result = scores.sort()
print(result)  # Output: None

Strings sort alphabetically. To sort by something else, pass a function as key. Here, key=len sorts words by their length:

words = ["banana", "kiwi", "apple"]

print(sorted(words))  # Output: ['apple', 'banana', 'kiwi']
print(sorted(words, key=len))  # Output: ['kiwi', 'apple', 'banana']

1.5. Looping and list comprehensions

A for loop goes through the items one at a time. A list comprehension is a one-line shortcut for a loop that builds a new list. You can add an if to keep only some items.

my_list = [1, 2, 3, 4]

# Looping through a list
for number in my_list:
    print(number)

# List comprehension: square each number
squared = [x ** 2 for x in my_list]
print(squared)

# List comprehension with a condition: keep only even numbers
evens = [x for x in my_list if x % 2 == 0]
print(evens)

# Output:
# 1
# 2
# 3
# 4
# [1, 4, 9, 16]
# [2, 4]

1.6. Common pitfalls & performance tips

Trying to access an index that doesn’t exist raises an IndexError:

my_list = [1, 2, 3]

try:
    print(my_list[3])
except IndexError as error:
    print(error)  # Output: list index out of range

Copying a list also catches people out. Assigning a list to a new name (same = original) doesn’t copy anything: both names point to the same list. copy() makes a shallow copy, which is a new outer list that still shares any lists nested inside it. To copy the nested lists too, make a deep copy with copy.deepcopy():

import copy

original = [[1, 2], [3, 4]]

same = original                 # not a copy: one list, two names
shallow = original.copy()       # new outer list, same inner lists
deep = copy.deepcopy(original)  # new outer list and new inner lists

original[0].append(99)

print(same)  # Output: [[1, 2, 99], [3, 4]]
print(shallow)  # Output: [[1, 2, 99], [3, 4]]
print(deep)  # Output: [[1, 2], [3, 4]]

As for speed, reading or replacing an item by index takes the same short time no matter how long the list is. Programmers write this as O(1). Adding or removing at the end with append() and pop() is also O(1) on average. But insert(0, x), pop(0), and remove() have to shift every item after the change, so they get slower as the list grows (O(n), where n is the length of the list). If you need to add and remove items at the front, as in a queue, use collections.deque, which is fast at both ends:

from collections import deque

queue = deque(["Alice", "Bob"])
queue.append("Charlie")  # join at the back
first = queue.popleft()  # leave from the front, quickly

print(first)  # Output: Alice
print(queue)  # Output: deque(['Bob', 'Charlie'])

2. Tuples

Tuples are like lists, except that they are immutable: once you create a tuple, it can’t change. That makes tuples a good fit for data that should stay the same.

2.1. Creating tuples

Write a tuple with parentheses (), or leave them out when the meaning is clear. A tuple with a single item needs a trailing comma. Without it, the parentheses are just grouping, like in math.

# Defining a tuple
my_tuple = (1, 2, 3)

# Parentheses are optional when the context is unambiguous
another_tuple = 4, 5, 6

# The comma makes the tuple, not the parentheses
single = (7,)
not_a_tuple = (7)

print(my_tuple)  # Output: (1, 2, 3)
print(another_tuple)  # Output: (4, 5, 6)
print(type(single))  # Output: <class 'tuple'>
print(type(not_a_tuple))  # Output: <class 'int'>

2.2. Accessing tuple elements

Indexing and slicing work just as they do for lists. So do len(), in, + (join), and * (repeat), although + and * build new tuples instead of changing the old ones.

my_tuple = (1, 2, 3, 4, 5)

print(my_tuple[1])  # Output: 2
print(my_tuple[1:4])  # Output: (2, 3, 4)
print(len(my_tuple))  # Output: 5
print(3 in my_tuple)  # Output: True

print((1, 2) + (3, 4))  # Output: (1, 2, 3, 4)
print((1, 2) * 3)  # Output: (1, 2, 1, 2, 1, 2)

2.3. Immutability

Any attempt to modify a tuple raises a TypeError:

my_tuple = (1, 2, 3)

try:
    my_tuple[1] = 10
except TypeError as error:
    print(error)  # Output: 'tuple' object does not support item assignment

If you need a different tuple, build a new one, for example my_tuple + (4,).

2.4. Unpacking tuples

Tuples are often used to keep related pieces of information together, such as a person’s name and age. Unpacking pulls the items out into separate variables in one step. You need exactly one variable per item, or Python raises a ValueError.

person = ("Alice", 25)

# One variable per item, in order
name, age = person
print(name)  # Output: Alice
print(age)  # Output: 25

# A handy trick: swap two variables without a temporary one
a, b = 1, 2
a, b = b, a
print(a, b)  # Output: 2 1

Unpacking is also how a function can return more than one value. When you write return quotient, remainder, Python packs the two values into a tuple, and the caller can unpack them:

def divide(x, y):
    quotient = x // y
    remainder = x % y
    return quotient, remainder

result = divide(10, 3)
print(result)  # Output: (3, 1)

# Unpack straight into two variables
quotient, remainder = divide(10, 3)
print(f"10 / 3 = {quotient} remainder {remainder}")  # Output: 10 / 3 = 3 remainder 1

2.5. When to use tuples

Tuples suit fixed collections such as coordinates, RGB values, or days of the week.

They can also be dictionary keys, which lists can’t. Dictionary keys (and set members) must be hashable. Python computes a number, called a hash, from the value and uses it to find things quickly. That only works if the value can’t change, so numbers, strings, and tuples are hashable, but lists are not. (A tuple is hashable only if everything inside it is hashable too.)

# Tuples work as dictionary keys
grid = {(0, 0): "start", (2, 3): "treasure"}
print(grid[(2, 3)])  # Output: treasure

# Lists don't, because they can change
try:
    grid[[0, 0]] = "oops"
except TypeError:
    print("A list can't be a dictionary key")  # Output: A list can't be a dictionary key

3. Dictionaries

Use a dictionary when you need a lookup table. It stores data as key-value pairs: you find a value quickly through its key, and the keys describe what each value is. Each key appears only once, and since Python 3.7, dictionaries remember the order in which you added the keys.

3.1. Creating dictionaries

Create a dictionary with curly braces {} or the dict() function. All three of these build the same dictionary:

# Using curly braces
my_dict = {'name': 'John', 'age': 30, 'city': 'New York'}
print(my_dict)  # Output: {'name': 'John', 'age': 30, 'city': 'New York'}

# Using dict() with keyword arguments
my_dict = dict(name='John', age=30, city='New York')
print(my_dict)  # Output: {'name': 'John', 'age': 30, 'city': 'New York'}

# Using dict() with a list of (key, value) tuples
my_dict = dict([('name', 'John'), ('age', 30), ('city', 'New York')])
print(my_dict)  # Output: {'name': 'John', 'age': 30, 'city': 'New York'}

If your keys and values are in separate lists, zip() pairs them up by position, and dict() turns the pairs into a dictionary:

keys = ['name', 'age', 'city']
values = ['John', 30, 'New York']

my_dict = dict(zip(keys, values))
print(my_dict)  # Output: {'name': 'John', 'age': 30, 'city': 'New York'}

3.2. Accessing and modifying data

Square brackets look up a value by its key, and the in operator checks whether a key exists:

my_dict = {'name': 'John', 'age': 30, 'city': 'New York'}

name = my_dict['name']
print(name)  # Output: John
print('name' in my_dict)  # Output: True

# A missing key raises a KeyError
try:
    my_dict['country']
except KeyError as error:
    print(f"KeyError: {error}")  # Output: KeyError: 'country'

For a key that might be missing, use the get() method. It returns a default value instead of raising an error, or None if you don’t give a default:

country = my_dict.get('country', 'United States')
print(country)  # Output: United States
print(my_dict.get('country'))  # Output: None

Assigning to an existing key updates its value, and assigning to a new key adds a new pair:

# Update a value
my_dict['age'] = 31

# Add a new key-value pair
my_dict['occupation'] = 'Engineer'

print(my_dict)  # Output: {'name': 'John', 'age': 31, 'city': 'New York', 'occupation': 'Engineer'}

You can also delete pairs:

# Using del
del my_dict['city']

# Using pop (also returns the value)
age = my_dict.pop('age')

print(age)  # Output: 31
print(my_dict)  # Output: {'name': 'John', 'occupation': 'Engineer'}

3.3. Looping through dictionaries

Looping over a dictionary gives you its keys. Loop over items() instead to get each key and value together. (There are also keys() and values() if you only need one side.)

my_dict = {'name': 'John', 'age': 30, 'city': 'New York'}

for key in my_dict:
    print(key, my_dict[key])

for key, value in my_dict.items():
    print(f"{key}: {value}")

# Output:
# name John
# age 30
# city New York
# name: John
# age: 30
# city: New York

3.4. Advanced methods

To merge two dictionaries, use the | operator (Python 3.9+), which builds a new dictionary, or update(), which merges into an existing one. The older {**dict1, **dict2} syntax also builds a new dictionary. When both dictionaries have the same key, the value from the second one wins:

defaults = {'theme': 'light', 'font_size': 12}
user_settings = {'font_size': 16}

settings = defaults | user_settings
print(settings)  # Output: {'theme': 'light', 'font_size': 16}

defaults.update(user_settings)
print(defaults)  # Output: {'theme': 'light', 'font_size': 16}

A dictionary comprehension builds a dictionary in a single expression:

# Create a dictionary with numbers and their squares
squares = {x: x**2 for x in range(5)}
print(squares)  # Output: {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

4. Sets

A set is an unordered collection of unique elements. Sets are good for membership testing, removing duplicates, and mathematical operations such as unions and intersections. Because they have no order, you can’t index a set (my_set[0] is an error).

4.1. Creating sets

Create a set with curly braces {} or the set() constructor. Duplicates are dropped automatically:

# Define a set
my_set = {1, 2, 3}
print(my_set)  # Output: {1, 2, 3}

# Duplicates disappear
numbers = set([1, 2, 2, 3, 3, 3])
print(numbers)  # Output: {1, 2, 3}

# An empty set must be created with set(), not {} (which creates an empty dict)
empty_set = set()
print(type(empty_set))  # Output: <class 'set'>
print(type({}))  # Output: <class 'dict'>

Turning a list into a set is the quickest way to remove duplicates. The set doesn’t keep the original order, so use sorted() if you want a predictable list back:

names = ["bob", "alice", "bob", "carol", "alice"]

unique_names = sorted(set(names))
print(unique_names)  # Output: ['alice', 'bob', 'carol']

4.2. Set operations

add() adds an item. remove() deletes one and raises a KeyError if it isn’t there, while discard() does nothing in that case:

my_set = {1, 2, 3}

# Add an item
my_set.add(4)
print(my_set)  # Output: {1, 2, 3, 4}

# Remove an item
my_set.remove(2)
print(my_set)  # Output: {1, 3, 4}

# Discard an item (no error if the item isn't present)
my_set.discard(10)
print(my_set)  # Output: {1, 3, 4}

# remove() complains about a missing item
try:
    my_set.remove(10)
except KeyError as error:
    print(f"KeyError: {error}")  # Output: KeyError: 10

The mathematical set operations are written as operators. Each operator also has a method version, such as set_a.union(set_b) or set_a.intersection(set_b):

set_a = {1, 2, 3}
set_b = {3, 4, 5}

# Union: All unique items from both sets
print(set_a | set_b)  # Output: {1, 2, 3, 4, 5}

# Intersection: Items common to both sets
print(set_a & set_b)  # Output: {3}

# Difference: Items in set_a but not in set_b
print(set_a - set_b)  # Output: {1, 2}

# Symmetric difference: Items in either set, but not in both
print(set_a ^ set_b)  # Output: {1, 2, 4, 5}

4.3. Membership testing

Sets are optimized for fast membership testing: checking in on a set stays fast however big the set gets. That makes a set a good choice for something like a guest list:

guest_list = {"Alice", "Bob", "Charlie"}

print("Alice" in guest_list)  # Output: True
print("Mallory" in guest_list)  # Output: False

4.4. Frozen sets

A frozen set is a set that can’t be changed after you create it. You make one with frozenset(). Everything that reads a set works on it (in, len(), union, intersection, and so on), but methods that change a set, such as add() and remove(), don’t exist.

frozen = frozenset([1, 2, 3, 4])
print(frozen)  # Output: frozenset({1, 2, 3, 4})

# Reading works just like a set
print(3 in frozen)  # Output: True
print(frozen | {5})  # Output: frozenset({1, 2, 3, 4, 5})

# Changing it doesn't
try:
    frozen.add(5)
except AttributeError as error:
    print(error)  # Output: 'frozenset' object has no attribute 'add'

Because a frozen set can’t change, it is hashable (see Section 2.5), so it can be a dictionary key or a member of another set. A regular set can be neither. Frozen sets are handy when the key is a group of things whose order doesn’t matter:

# Each key is a pair of friends; {"Alice", "Bob"} is the same pair as {"Bob", "Alice"}
friends_since = {
    frozenset({"Alice", "Bob"}): 2019,
    frozenset({"Bob", "Carol"}): 2022,
}
print(friends_since[frozenset({"Bob", "Alice"})])  # Output: 2019

# A regular set doesn't work as a key
try:
    friends_since[{"Carol", "Dave"}] = 2024
except TypeError:
    print("A set can't be a dictionary key")  # Output: A set can't be a dictionary key

5. Bytes and bytearrays: raw binary data

Everything a computer stores (text, images, sound) is made of bytes. A byte is a number from 0 to 255. Python has two types for working with bytes directly, which you’ll meet when reading files in binary mode or sending data over a network:

bytes: an immutable sequence of integers from 0 to 255
bytearray: the same thing, but mutable

5.1. Bytes

Create a bytes object from a list of numbers, or write a bytes literal with a b in front of the quotes. When Python displays bytes, it shows each byte that matches a printable ASCII character as that character, and the rest as escape codes like \x00. That is only a display convenience: each item is still a number.

data = bytes([72, 105, 33])
print(data)  # Output: b'Hi!'

# Indexing gives you a number, slicing gives you bytes
print(data[0])  # Output: 72
print(data[0:2])  # Output: b'Hi'
print(list(data))  # Output: [72, 105, 33]

# A bytes literal builds the same object
print(data == b"Hi!")  # Output: True

# Every value must fit in one byte
try:
    bytes([256])
except ValueError as error:
    print(error)  # Output: bytes must be in range(0, 256)

# And bytes can't be changed
try:
    data[0] = 74
except TypeError as error:
    print(error)  # Output: 'bytes' object does not support item assignment

5.2. Converting between text and bytes

A string holds characters; bytes hold numbers. To turn text into bytes, you encode it, and to turn bytes back into text, you decode them. UTF-8 is the most common encoding. Some characters take more than one byte:

text = "café"

encoded = text.encode("utf-8")
print(encoded)  # Output: b'caf\xc3\xa9'
print(len(text))  # Output: 4
print(len(encoded))  # Output: 5

decoded = encoded.decode("utf-8")
print(decoded)  # Output: café

The é became two bytes (\xc3 and \xa9), which is why the bytes version is one longer than the string.

5.3. Bytearray

A bytearray works like bytes, but you can change it in place, much like a list of small numbers. That makes it useful when you build up binary data piece by piece.

buffer = bytearray(b"hello")

# Replace the first byte (74 is the code for "J")
buffer[0] = 74
print(buffer)  # Output: bytearray(b'Jello')

# Add a byte to the end (33 is "!")
buffer.append(33)
print(buffer)  # Output: bytearray(b'Jello!')

# bytearray(n) makes n zero bytes
print(bytearray(3))  # Output: bytearray(b'\x00\x00\x00')

# Convert to bytes when you're done changing it
print(bytes(buffer))  # Output: b'Jello!'

6. Comparative overview & use cases

Data Structure	Mutable?	Ordered?	Use Case Example
List	Yes	Yes	Dynamic arrays, ordered collections
Tuple	No	Yes	Fixed collections, keys in dictionaries
Dictionary	Yes	Insertion-ordered (3.7+)	Fast lookups by unique keys, configuration data
Set	Yes	No	Membership testing, deduplication, set math
Frozen set	No	No	Sets used as dictionary keys or inside other sets
Bytes	No	Yes	Binary data read from files or networks
Bytearray	Yes	Yes	Building or editing binary data

Use a list for data that changes over time and where order matters, and a tuple for fixed data, including dictionary keys. If you need to look values up quickly by unique keys (like a mini-database), use a dictionary. When order doesn’t matter and you want to avoid duplicates or do set operations, use a set, or a frozen set when the set itself needs to be a key. Use str for text and bytes for raw binary data, and convert between them with encode() and decode().

7. Best practices and performance tips

A tuple uses a little less memory than a list with the same items, because a list needs extra bookkeeping (and often some spare room) so it can grow. The difference is small, so choose between them based on whether the data should change. Sets and dictionaries use noticeably more memory than a list of the same size. They trade that extra space for fast lookups.

Accessing elements by index in lists and tuples is O(1), and lookups in dictionaries and sets are also O(1) on average. Checking x in my_list, on the other hand, looks through the list item by item (O(n)), so if you test membership often on a large collection, use a set. Adding or removing at the front of a list is O(n); use collections.deque for queues.

A few habits prevent common errors. Be careful with indexes in lists and tuples, use the get() method for dictionaries to avoid KeyError, and use discard() for sets if you’re not sure whether an element exists.

8. Advanced tips and tricks

You can combine these structures to represent more complex data. For instance, a list of dictionaries can represent a table of data:

students = [
    {'name': 'Alice', 'age': 24, 'major': 'Physics'},
    {'name': 'Bob', 'age': 22, 'major': 'Mathematics'},
    {'name': 'Charlie', 'age': 23, 'major': 'Computer Science'}
]

for student in students:
    print(f"{student['name']} studies {student['major']}")

# Output:
# Alice studies Physics
# Bob studies Mathematics
# Charlie studies Computer Science

When you pass data to a function as a tuple, the function can’t change it by accident.

List, dictionary, and set comprehensions are a compact way to create new collections. They make code shorter and, once you get used to them, easier to read. Notice how the same expression gives three different results:

numbers = [1, 2, 2, 3, 3, 3]

print([n * n for n in numbers])  # Output: [1, 4, 4, 9, 9, 9]
print({n * n for n in numbers})  # Output: {1, 4, 9}
print({n: n * n for n in numbers})  # Output: {1: 1, 2: 4, 3: 9}

To inspect a data structure while debugging, use built-in functions like len() and type(), or plain print statements. When your data isn’t behaving as expected, Python’s debugger (pdb) can save you a lot of time.

Key takeaways

Lists are ordered and changeable. append() and pop() at the end are fast; sorted() returns a new list, while sort() changes the list and returns None.
Assigning a list to another name doesn’t copy it. copy() makes a shallow copy; use copy.deepcopy() for nested lists.
Tuples can’t change. Unpack them into variables, and use them to return several values from a function.
Dictionaries map unique keys to values, remember insertion order, and find values quickly. Use get() when a key might be missing.
Sets hold unique items and make membership tests fast. Frozen sets are the unchangeable, hashable version, so they can be dictionary keys or set members.
bytes and bytearray hold integers from 0 to 255; encode() and decode() convert between text and bytes.

Practice

Exercise 1: Tidy Up a Shopping List

You have this shopping list, with some items written down twice:

shopping = ["milk", "eggs", "bread", "milk", "apples", "eggs"]

Build a sorted list of the unique items, print it, and then print how many different items there are.

Solution

shopping = ["milk", "eggs", "bread", "milk", "apples", "eggs"]

unique_items = sorted(set(shopping))
print(unique_items)
print(f"{len(unique_items)} different items")

# Output:
# ['apples', 'bread', 'eggs', 'milk']
# 4 different items

Exercise 2: Count the Words

Count how many times each word appears in the sentence "the cat sat on the mat with the hat". Store the counts in a dictionary, then print each word and its count in the order the words first appear. Hint: sentence.split() turns the sentence into a list of words, and counts.get(word, 0) gives you the current count, or 0 for a new word.

Solution

sentence = "the cat sat on the mat with the hat"

counts = {}
for word in sentence.split():
    counts[word] = counts.get(word, 0) + 1

for word, count in counts.items():
    print(f"{word}: {count}")

# Output:
# the: 3
# cat: 1
# sat: 1
# on: 1
# mat: 1
# with: 1
# hat: 1

Exercise 3: Summarize Test Scores

Write a function summarize(scores) that returns three values: the lowest score, the highest score, and the average. Call it with [72, 95, 88, 64, 81], unpack the result into three variables, and print each one. You can use the built-in functions min(), max(), sum(), and len().

Solution

def summarize(scores):
    lowest = min(scores)
    highest = max(scores)
    average = sum(scores) / len(scores)
    return lowest, highest, average

low, high, avg = summarize([72, 95, 88, 64, 81])
print(f"Lowest: {low}")
print(f"Highest: {high}")
print(f"Average: {avg}")

# Output:
# Lowest: 64
# Highest: 95
# Average: 80.0

Chapter 4: Working with Data

Once your data is in lists, tuples, and dictionaries, you need to work with it: put it in order, pick out the parts you want, save it to files, and deal with the errors that come up along the way. This chapter covers those everyday tasks.

In this chapter you will learn:

How to sort, filter, and summarize lists and tuples
How to read, write, and append to text files with open() and pathlib
How to save and load data as JSON
How to handle errors with try, except, else, and finally
How to raise exceptions, including your own exception types

Manipulating data

Sorting data

You can use the sorted() function to sort a list or tuple in ascending order:

# Define a list of integers
my_list = [5, 2, 7, 1, 3]

# Sort the list in ascending order
sorted_list = sorted(my_list)
print(sorted_list)  # Output: [1, 2, 3, 5, 7]
print(my_list)  # Output: [5, 2, 7, 1, 3]

# Define a tuple of strings
my_tuple = ('c', 'a', 'b')

# Sort the tuple in ascending order
sorted_tuple = sorted(my_tuple)
print(sorted_tuple)  # Output: ['a', 'b', 'c']

sorted() returns a new list and leaves the original unchanged. It always returns a list, even when you give it a tuple.

You can also use the sort() method to sort a list in place, which changes the list itself:

# Define a list of integers
my_list = [5, 2, 7, 1, 3]

# Sort the list in ascending order
my_list.sort()
print(my_list)  # Output: [1, 2, 3, 5, 7]

sort() returns None, so don’t write my_list = my_list.sort(), or you’ll lose your list. Tuples can’t be changed, so they have no sort() method. Use sorted() for them.

Both sorted() and sort() accept two optional arguments. reverse=True sorts in descending order. key= takes a function that Python calls on each item, and the items are sorted by the results:

my_list = [5, 2, 7, 1, 3]
print(sorted(my_list, reverse=True))  # Output: [7, 5, 3, 2, 1]

words = ["banana", "fig", "apple", "kiwi"]
print(sorted(words))  # Output: ['apple', 'banana', 'fig', 'kiwi']
print(sorted(words, key=len))  # Output: ['fig', 'kiwi', 'apple', 'banana']

With key=len, the words are sorted by their length. Notice that you pass len without parentheses: you’re handing sorted() the function itself so it can call it on each word.

Filtering data

You can use a list comprehension with a condition to keep only the items you want:

# Define a list of integers
my_list = [5, 2, 7, 1, 8, 4]

# Filter the list to get only even numbers
even_numbers = [x for x in my_list if x % 2 == 0]
print(even_numbers)  # Output: [2, 8, 4]

To filter a tuple and get a tuple back, pass the same kind of expression to tuple():

# Define a tuple of strings
animals = ('cat', 'elephant', 'dog', 'giraffe')

# Filter the tuple to get only names longer than 3 letters
long_names = tuple(name for name in animals if len(name) > 3)
print(long_names)  # Output: ('elephant', 'giraffe')

Aggregating data

You can use the sum() function to add up the items in a list or tuple, and min(), max(), and len() to get the smallest item, the largest item, and the number of items:

# Define a list of integers
my_list = [5, 2, 7, 1, 3]

print(sum(my_list))  # Output: 18
print(min(my_list))  # Output: 1
print(max(my_list))  # Output: 7
print(len(my_list))  # Output: 5

# Combine them to get the average
print(sum(my_list) / len(my_list))  # Output: 3.6

# The same functions work on tuples
my_tuple = (5, 2, 7, 1, 3)
print(sum(my_tuple))  # Output: 18

Like sorted(), min() and max() accept a key= argument. For example, max(words, key=len) returns the longest word.

Reading and writing files

You can read and write files using the built-in open() function and the file object it returns. Each example below writes the file it needs first, so you can run it as is.

Writing to a file

To write to a file, open it in write mode:

# Open the file in write mode
with open('my_file.txt', 'w', encoding='utf-8') as f:
    # Write some text to the file
    f.write("Hello, world!\n")
    f.write("This is the second line.\n")

Here open() gets the file name and the mode 'w' (for write). It creates my_file.txt in the folder you run the program from, or empties the file if it already exists, and returns a file object, which is stored in the f variable. The file object’s write() method then writes each string to the file. write() doesn’t add a line break for you, so each string ends with \n.

Always pass encoding='utf-8' when you open a text file. Without it, Python uses your operating system’s default encoding, which differs between computers and can garble characters such as é or ü.

The with statement opens the file and automatically closes it when the indented block finishes, even if an error happens inside it. That is why it’s the recommended way to work with files: the file is always closed and released when you’re done with it.

Reading a file

To read a file, open it in read mode:

with open('my_file.txt', 'w', encoding='utf-8') as f:
    f.write("Hello, world!\nThis is the second line.\n")

# Open the file in read mode
with open('my_file.txt', 'r', encoding='utf-8') as f:
    # Read the contents of the file into a variable
    file_contents = f.read()

# Print the contents of the file
print(file_contents)
# Output:
# Hello, world!
# This is the second line.

This time open() gets the mode 'r' (for read). The file object’s read() method reads the whole file into the file_contents variable as one string, and print() displays it. Reading is the default mode, so open('my_file.txt', encoding='utf-8') does the same thing.

Reading line by line

You can loop over a file object with for to get one line at a time:

with open('shopping.txt', 'w', encoding='utf-8') as f:
    f.write("eggs\nmilk\nbread\n")

with open('shopping.txt', encoding='utf-8') as f:
    for line in f:
        print(line.strip())

# Output:
# eggs
# milk
# bread

Each line keeps its \n at the end. strip() removes it, so print() doesn’t add an extra blank line after each item.

The readlines() method reads all the lines at once and returns them as a list:

with open('shopping.txt', 'w', encoding='utf-8') as f:
    f.write("eggs\nmilk\nbread\n")

with open('shopping.txt', encoding='utf-8') as f:
    lines = f.readlines()

print(lines)  # Output: ['eggs\n', 'milk\n', 'bread\n']
print(len(lines))  # Output: 3

readlines() is handy when you need all the lines together, for example to count or sort them. Looping over the file reads one line at a time instead, so it uses less memory and works even for very large files.

Appending to a file

Mode 'w' replaces everything in the file. To add to the end of a file and keep what is already there, use mode 'a' (for append):

with open('log.txt', 'w', encoding='utf-8') as f:
    f.write("first entry\n")

with open('log.txt', 'a', encoding='utf-8') as f:
    f.write("second entry\n")

with open('log.txt', encoding='utf-8') as f:
    print(f.read())

# Output:
# first entry
# second entry

Different file modes in Python

Python has several file modes, depending on whether you want to read, write, or both. Modes without b open the file in text mode, which reads and writes strings. Modes with b open it in binary mode, which reads and writes bytes (used for images and other non-text files) and doesn’t take an encoding.

Mode	Description
`'r'`	Open for reading (the default). Raises an error if the file does not exist.
`'r+'`	Open for reading and writing. Raises an error if the file does not exist.
`'w'`	Open for writing. Truncates (empties) the file if it exists. Creates a new file if it doesn’t.
`'w+'`	Open for reading and writing. Truncates the file if it exists. Creates a new file if it doesn’t.
`'a'`	Open for appending. Writes go to the end of the file. Creates the file if it doesn’t exist.
`'a+'`	Open for reading and appending. Writes go to the end of the file. Creates the file if it doesn’t exist.
`'x'`	Create a new file and open it for writing. Raises an error if the file already exists.
`'rb'`	Open for reading in binary mode. Raises an error if the file doesn’t exist.
`'rb+'`	Open for reading and writing in binary mode. Raises an error if the file doesn’t exist.
`'wb'`	Open for writing in binary mode. Truncates the file if it exists. Creates a new one if not.
`'wb+'`	Open for reading and writing in binary mode. Truncates the file if it exists. Creates a new one if not.
`'ab'`	Open for appending in binary mode. Creates the file if it doesn’t exist.
`'ab+'`	Open for reading and appending in binary mode. Creates the file if it doesn’t exist.

Working with paths using pathlib

The pathlib module from Python’s standard library gives you Path objects, which represent a file or folder path. For small files, its write_text() and read_text() methods open, write or read, and close the file in one step:

from pathlib import Path

path = Path('notes.txt')
print(path.exists())  # Output: False

path.write_text("Buy milk\nCall Sam\n", encoding='utf-8')
print(path.exists())  # Output: True

text = path.read_text(encoding='utf-8')
print(text.splitlines())  # Output: ['Buy milk', 'Call Sam']

exists() tells you whether the file is there, which is useful before you try to read it. splitlines() splits a string into a list of lines without the \n characters. You can also build paths with /, as in Path('data') / 'notes.txt', and Python uses the right separator for your operating system.

Working with JSON

JSON (JavaScript Object Notation) is a text format for structured data. Many web services, programs, and configuration files use it. Python’s built-in json module converts dictionaries, lists, strings, numbers, booleans, and None to JSON text and back.

Use json.dump() to write data to a file and json.load() to read it back:

import json

student = {"name": "Asha", "age": 21, "courses": ["Math", "Physics"], "graduated": False}

# Save the dictionary to a file
with open('student.json', 'w', encoding='utf-8') as f:
    json.dump(student, f)

# Load it back
with open('student.json', encoding='utf-8') as f:
    loaded = json.load(f)

print(loaded["name"])  # Output: Asha
print(loaded["courses"])  # Output: ['Math', 'Physics']
print(loaded == student)  # Output: True

json.dumps() (the s stands for “string”) returns the JSON text as a string instead of writing it to a file. Pass indent=4 to make it easier to read. json.dump() accepts indent too:

import json

student = {"name": "Asha", "age": 21, "graduated": False}
print(json.dumps(student, indent=4))
# Output:
# {
#     "name": "Asha",
#     "age": 21,
#     "graduated": false
# }

Notice that JSON writes false in lowercase, uses double quotes for strings, and writes None as null. To turn a JSON string back into Python data, use json.loads().

Exception handling

An exception is an error that happens while a program is running, such as dividing by zero or converting "abc" to a number. If you don’t handle it, the program stops and prints an error message. Exception handling lets you catch the error and decide what to do instead, so your program can deal with unexpected input and keep going.

try and except

To handle an exception, put the code that might fail in a try block and the code that deals with the error in an except block:

text = "ten"  # this value might come from input()

try:
    number = int(text)
    print(f"Twice your number is {number * 2}")
except ValueError:
    print("Please enter a whole number, like 10.")

# Output:
# Please enter a whole number, like 10.

Python runs the code in the try block. int("ten") raises a ValueError, so Python skips the rest of the try block and runs the matching except block. If no exception is raised, the except block is skipped.

This is the most common error with user input: input() always returns a string, so int(input()) raises a ValueError whenever someone types something that isn’t a whole number. The examples here use variables so you can see the output, but the values could just as well come from input().

Handling different exceptions

A try block can be followed by several except clauses, one for each kind of error you expect:

def divide(a_text, b_text):
    try:
        result = int(a_text) / int(b_text)
        print(f"Result: {result}")
    except ValueError:
        print("Both values must be whole numbers.")
    except ZeroDivisionError:
        print("Cannot divide by zero.")

divide("10", "4")  # Output: Result: 2.5
divide("10", "zero")  # Output: Both values must be whole numbers.
divide("10", "0")  # Output: Cannot divide by zero.

Python checks the except clauses from top to bottom and runs the first one that matches. To handle several exceptions the same way, list them in one clause as a tuple: except (ValueError, ZeroDivisionError):.

Avoid a bare except: with no exception name. It catches every exception, even a typo in your own code (NameError) or pressing Ctrl+C (KeyboardInterrupt), which hides bugs and makes the program hard to stop. Catch the specific exceptions you expect.

Getting the error message

Add as and a name to an except clause to get the exception object. Printing it shows Python’s error message:

try:
    with open('missing.txt', encoding='utf-8') as f:
        print(f.read())
except FileNotFoundError as error:
    print(f"Could not open the file: {error}")

# Output:
# Could not open the file: [Errno 2] No such file or directory: 'missing.txt'

The [Errno 2] part is the error code your operating system reported.

else and finally

You can add an else block, which runs only if the try block raised no exception, and a finally block, which runs no matter what:

def check_number(text):
    try:
        number = int(text)
    except ValueError:
        print(f"{text} is not a number")
    else:
        print(f"Got the number {number}")
    finally:
        print("Done checking")

check_number("7")
check_number("seven")

# Output:
# Got the number 7
# Done checking
# seven is not a number
# Done checking

Putting the success code in else instead of try makes it clear which line you expect might fail. finally is the place for cleanup that must always happen, such as closing a connection or printing a final message.

Raising exceptions

You can raise an exception yourself with the raise statement when your code receives a value it can’t work with. The code that called it can then decide how to handle the problem:

def set_age(age):
    if age < 0:
        raise ValueError("Age cannot be negative")
    print(f"Age set to {age}")

set_age(30)  # Output: Age set to 30

try:
    set_age(-5)
except ValueError as error:
    print(error)  # Output: Age cannot be negative

Custom exceptions

You can define your own exception type by creating a class that inherits from Exception. You’ll learn about classes and inheritance in Chapter 5. For now, these two lines are all you need:

class InsufficientFundsError(Exception):
    """Raised when a withdrawal is larger than the balance."""

def withdraw(balance, amount):
    if amount > balance:
        raise InsufficientFundsError(f"Cannot withdraw {amount}, the balance is only {balance}")
    return balance - amount

print(withdraw(100, 30))  # Output: 70

try:
    withdraw(100, 500)
except InsufficientFundsError as error:
    print(error)  # Output: Cannot withdraw 500, the balance is only 100

A custom exception’s name says exactly what went wrong, and callers can catch it without also catching unrelated errors.

Built-in exceptions in Python

Python has many built-in exceptions, which it raises when errors occur while a program runs:

Exception	Description
`ArithmeticError`	Base class for arithmetic errors such as `OverflowError` and `ZeroDivisionError`.
`AssertionError`	Raised when an `assert` statement fails.
`AttributeError`	Raised when attribute reference or assignment fails.
`BaseException`	Base class for all built-in exceptions, including `KeyboardInterrupt` and `SystemExit`, which don’t inherit from `Exception`. You rarely catch it directly.
`EOFError`	Raised when the `input()` function hits an end-of-file condition (EOF).
`Exception`	Base class for ordinary errors. The exceptions you usually catch, and your own exception classes, inherit from it.
`FileExistsError`	Raised when you try to create a file or folder that already exists. A subclass of `OSError`.
`FileNotFoundError`	Raised when you try to open a file or folder that doesn’t exist. A subclass of `OSError`.
`GeneratorExit`	Raised when a generator is closed using `close()`.
`ImportError`	Raised when an `import` statement fails, for example when a name can’t be imported from a module.
`IndentationError`	Raised when indentation is not correct.
`IndexError`	Raised when an index of a sequence is out of range.
`KeyboardInterrupt`	Raised when the user interrupts the program, usually by pressing Ctrl+C.
`KeyError`	Raised when a key is not found in a dictionary.
`LookupError`	Base class for lookup errors like `IndexError` and `KeyError`.
`MemoryError`	Raised when the program runs out of memory.
`ModuleNotFoundError`	Raised when the module you import can’t be found. A subclass of `ImportError`.
`NameError`	Raised when a variable or other name is not defined.
`NotImplementedError`	Raised by a method that a subclass is meant to override but hasn’t.
`OSError`	Raised when a system-related operation fails, such as reading or writing a file.
`OverflowError`	Raised when the result of a calculation is too large to represent, such as `10.0 ** 1000`.
`PermissionError`	Raised when you don’t have permission to access a file or folder. A subclass of `OSError`.
`ReferenceError`	Raised when a weak reference is no longer valid.
`RuntimeError`	Raised for errors that don’t fit any other category.
`StopIteration`	Raised to signal the end of iteration.
`SyntaxError`	Raised when a syntax error is detected.
`SystemError`	Raised when Python itself hits an internal error.
`SystemExit`	Raised when `sys.exit()` is called.
`TabError`	Raised when inconsistent use of tabs and spaces is detected.
`TypeError`	Raised when an operation is applied to an object of inappropriate type.
`UnboundLocalError`	Raised when a local variable is used before it has a value (see Chapter 2).
`UnicodeDecodeError`	Raised when bytes can’t be decoded into text, such as when a file is read with the wrong encoding.
`UnicodeEncodeError`	Raised when text can’t be encoded into bytes.
`UnicodeError`	Base class for Unicode-related errors.
`UnicodeTranslateError`	Raised when a Unicode translation error occurs.
`ValueError`	Raised when a function receives the right type but an inappropriate value, such as `int("abc")`.
`ZeroDivisionError`	Raised when division or modulo by zero is attempted.

Key takeaways

sorted() returns a new sorted list, while list.sort() sorts a list in place. Both accept reverse=True and key=.
List comprehensions filter data, and sum(), min(), max(), and len() summarize it.
Open text files with with open(..., encoding='utf-8') so they close automatically and read the same way on every computer. Mode 'w' overwrites a file, and 'a' adds to the end.
pathlib.Path makes simple file tasks short, and the json module saves and loads dictionaries and lists.
Catch specific exceptions with try and except. Use else for code that should run only on success and finally for code that must always run.
Use raise to signal a problem, and create your own exception types by subclassing Exception.

Practice

Exercise 1: Test scores

Given the list scores = [72, 45, 90, 61, 38, 85], print the scores sorted from highest to lowest, a list of the passing scores (60 or higher), and the average score rounded to one decimal place with round().

Solution

scores = [72, 45, 90, 61, 38, 85]

print(sorted(scores, reverse=True))
passing = [score for score in scores if score >= 60]
print(passing)
average = sum(scores) / len(scores)
print(round(average, 1))
# Output:
# [90, 85, 72, 61, 45, 38]
# [72, 90, 61, 85]
# 65.2

Exercise 2: A to-do list file

Write three tasks to a file named todo.txt, one per line. Then append a fourth task without erasing the others. Finally, read the file and print each task with a number in front, starting at 1.

Solution

with open('todo.txt', 'w', encoding='utf-8') as f:
    f.write("Buy groceries\n")
    f.write("Call the bank\n")
    f.write("Water the plants\n")

with open('todo.txt', 'a', encoding='utf-8') as f:
    f.write("Read a chapter\n")

with open('todo.txt', encoding='utf-8') as f:
    for number, line in enumerate(f, start=1):
        print(f"{number}. {line.strip()}")
# Output:
# 1. Buy groceries
# 2. Call the bank
# 3. Water the plants
# 4. Read a chapter

Exercise 3: Safe number conversion

Write a function parse_numbers(items) that takes a list of strings and returns a list of the whole numbers they contain. If an item can’t be converted with int(), print a message saying it was skipped and move on. Test it with ["10", "abc", "7", "3.5", "-2"].

Solution

def parse_numbers(items):
    """Return the items that are whole numbers, converted to int."""
    numbers = []
    for item in items:
        try:
            numbers.append(int(item))
        except ValueError:
            print(f"Skipping {item}: not a whole number")
    return numbers

print(parse_numbers(["10", "abc", "7", "3.5", "-2"]))
# Output:
# Skipping abc: not a whole number
# Skipping 3.5: not a whole number
# [10, 7, -2]

Chapter 5: Object-Oriented Programming

So far you have kept data in variables, lists, and dictionaries, and written separate functions to work on that data. Object-oriented programming lets you bundle the data and the functions that use it into a single unit called an object.

In this chapter you will learn:

How to define a class and create objects from it
How to make objects print in a readable way with __str__ and __repr__
The difference between class attributes and instance attributes
How inheritance and polymorphism let you reuse code and treat different objects alike
How to protect an object’s data with encapsulation and @property
How @dataclass writes the repetitive parts of a simple class for you

Introduction to object-oriented programming

Object-oriented programming (OOP) is a way of organizing programs around objects. An object holds some data (its attributes) and the functions that work on that data (its methods). Objects often represent real-world things, such as a person, a bank account, or a product in a shop.

You have already been using objects. Every string in Python is an object, and upper() is one of its methods:

name = 'alice'
print(name.upper())  # Output: ALICE

With OOP, you keep each piece of data together with the code that knows how to handle it, instead of passing the same data from function to function. Code built this way is more modular, and easier to reuse and maintain.

Classes and objects

A class is a template that describes what data and methods its objects will have. Objects created from a class are called instances of that class.

Here is a simple class:

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def greet(self):
        print(f'Hello, my name is {self.name} and I am {self.age} years old.')

This class describes a Person with a name and an age attribute, and a greet method that prints a greeting message.

To create an instance, you call the class as if it were a function. Python creates a new, empty object and then automatically calls the special __init__ method to set it up, passing along the arguments you gave:

person = Person('John', 30)
person.greet()  # Output: Hello, my name is John and I am 30 years old.

The first parameter of every method, self, is the object the method is working on. When you write person.greet(), Python passes person as self for you, so self.name inside the method means person.name.

In this example, person is an instance of the Person class. You can read and change its attributes with a dot:

print(person.name)  # Output: John

person.age = 31
person.greet()  # Output: Hello, my name is John and I am 31 years old.

Objects can interact with each other through their methods. For example, a Dog object can call the greet method of a Person object that you pass to it:

class Dog:
    def bark(self, person):
        person.greet()
        print('Woof woof!')

dog = Dog()
dog.bark(person)
# Output:
# Hello, my name is John and I am 31 years old.
# Woof woof!

Printing objects with `str` and `repr`

If you print an object of the Person class above, Python shows something like <__main__.Person object at 0x10a3b2c50>: the class name and a memory address, which is not much help. You can tell Python how to show your objects by defining two special methods:

__str__ returns a friendly description for people. print() and str() use it.
__repr__ returns a precise description for programmers, ideally one that looks like the code that would create the object. Python uses it when it shows an object inside a list, and in the interactive shell.

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __str__(self):
        return f'{self.name}, {self.age} years old'

    def __repr__(self):
        return f'Person({self.name!r}, {self.age!r})'

person = Person('John', 30)
print(person)        # Output: John, 30 years old
print(repr(person))  # Output: Person('John', 30)
print([person])      # Output: [Person('John', 30)]

The !r inside the f-string uses the repr() of the value, which puts quotes around strings. If you only write one of the two methods, write __repr__: when a class has no __str__, print() uses __repr__ instead.

Class attributes and instance attributes

Attributes you set on self inside __init__ are instance attributes: each object gets its own copy. Attributes you define directly in the class body are class attributes: they belong to the class and are shared by all of its instances.

class Dog:
    species = 'Canis familiaris'  # class attribute: shared by every Dog
    count = 0                     # class attribute: how many dogs have been created

    def __init__(self, name):
        self.name = name          # instance attribute: one per Dog
        Dog.count += 1

rex = Dog('Rex')
fido = Dog('Fido')

print(rex.name)      # Output: Rex
print(fido.name)     # Output: Fido
print(fido.species)  # Output: Canis familiaris
print(Dog.count)     # Output: 2

Notice that __init__ updates Dog.count, not self.count. Assigning to self.count would create a new instance attribute on that one dog and leave the shared counter unchanged.

Inheritance and polymorphism

Inheritance

Inheritance lets a new class take on the attributes and methods of an existing class. The new class (the subclass or child class) can add its own attributes and methods, or replace the ones it inherited, without you having to rewrite the code that stays the same.

Here are a parent class and a subclass that inherits from it:

class Animal:
    def __init__(self, name, species):
        self.name = name
        self.species = species

    def make_sound(self):
        print('Some generic animal sound')

class Dog(Animal):
    def __init__(self, name, breed):
        super().__init__(name, species='Dog')
        self.breed = breed

    def make_sound(self):
        print('Woof woof!')

Writing class Dog(Animal) makes Dog a subclass of Animal. Dog inherits everything Animal has, adds a breed attribute, and defines its own version of make_sound, which replaces (overrides) the one in Animal.

Inside Dog.__init__, the call super().__init__(name, species='Dog') runs the __init__ method of the parent class, so the code that sets name and species is written only once.

You create a Dog the same way as any other object, by calling the class:

dog = Dog('Fido', 'Labrador')
print(dog.name)     # Output: Fido
print(dog.species)  # Output: Dog
print(dog.breed)    # Output: Labrador
dog.make_sound()    # Output: Woof woof!

print(isinstance(dog, Animal))  # Output: True

isinstance confirms that a Dog is also an Animal.

Polymorphism

Polymorphism means “many forms”. In practice it means that different classes provide a method with the same name, so the rest of your code can call that method without caring which class it is dealing with.

For example:

class Animal:
    def __init__(self, name):
        self.name = name

    def make_sound(self):
        print('Some generic animal sound')

class Dog(Animal):
    def make_sound(self):
        print('Woof woof!')

class Cat(Animal):
    def make_sound(self):
        print('Meow meow!')

def make_sounds(animals):
    for animal in animals:
        animal.make_sound()

dog = Dog('Fido')
cat = Cat('Fluffy')

make_sounds([dog, cat, Animal('Mystery')])
# Output:
# Woof woof!
# Meow meow!
# Some generic animal sound

Dog and Cat don’t define __init__, so they use the one they inherit from Animal. Each class has its own make_sound, and the make_sounds function simply calls make_sound on every object in the list. Python looks up the method on each object’s own class, so the dog barks, the cat meows, and a plain Animal makes the generic sound. If you add a Cow class later, make_sounds works with it without any changes.

Python doesn’t even require the objects to share a parent class. Any object that has a make_sound method will work:

class Robot:  # not an Animal at all
    def make_sound(self):
        print('Beep boop!')

make_sounds([dog, Robot()])
# Output:
# Woof woof!
# Beep boop!

This is called duck typing, from the saying “if it walks like a duck and quacks like a duck, it’s a duck”. Python only cares whether the object has the method you call, whatever class it comes from.

Encapsulation and data hiding

Encapsulation means bundling data together with the methods that work on it, and having the rest of the program use those methods instead of changing the data directly. The methods can then check that every change makes sense, for example that a bank account never goes below zero.

Data hiding is the related idea of keeping internal details out of reach of code outside the class. Unlike some languages, Python has no truly private attributes. Instead it uses naming conventions:

A single leading underscore, as in _balance, means “this is internal; please don’t use it from outside the class”. This is only a convention: Python does not stop anyone from using it, but other programmers will understand the warning.
A double leading underscore, as in __balance, makes Python rename the attribute behind the scenes (this is called name mangling).

Here is a bank account that uses the single-underscore convention and only changes its balance through methods:

class BankAccount:
    def __init__(self, owner, balance=0):
        self.owner = owner
        self._balance = balance  # internal: use the methods below

    def get_balance(self):
        return self._balance

    def deposit(self, amount):
        if amount <= 0:
            raise ValueError('Deposit amount must be positive')
        self._balance += amount

    def withdraw(self, amount):
        if amount > self._balance:
            raise ValueError('Insufficient funds')
        self._balance -= amount

account = BankAccount('Alice', 100)
account.deposit(50)
account.withdraw(30)
print(account.get_balance())  # Output: 120

try:
    account.withdraw(500)
except ValueError as error:
    print(error)  # Output: Insufficient funds

print(account.get_balance())  # Output: 120

withdraw raises a ValueError when there is not enough money, instead of just printing a message. That way the code that called it finds out that something went wrong and can decide what to do, and the balance stays unchanged.

Name mangling, the double-underscore form, works like this:

class BankAccount:
    def __init__(self, balance):
        self.__balance = balance

account = BankAccount(100)

try:
    print(account.__balance)
except AttributeError as error:
    print(error)  # Output: 'BankAccount' object has no attribute '__balance'

print(account._BankAccount__balance)  # Output: 100

Python stored the attribute as _BankAccount__balance. The main purpose of this is to stop a subclass from accidentally overwriting the attribute with one of its own that has the same name. It is not real protection, because the renamed attribute can still be reached. For everyday code, a single underscore is the usual choice.

Computed and validated attributes with `@property`

Calling get_balance() works, but Python has a neater way. The @property decorator turns a method into something you read like an attribute, without parentheses. You can also add a setter that runs whenever someone assigns a new value, which is a good place to validate it. (You will see how decorators work in Chapter 6.)

class Temperature:
    def __init__(self, celsius):
        self.celsius = celsius  # goes through the setter below

    @property
    def celsius(self):
        return self._celsius

    @celsius.setter
    def celsius(self, value):
        if value < -273.15:
            raise ValueError('Temperature below absolute zero')
        self._celsius = value

    @property
    def fahrenheit(self):
        return self._celsius * 9 / 5 + 32

t = Temperature(25)
print(t.celsius)     # Output: 25
print(t.fahrenheit)  # Output: 77.0

t.celsius = 100
print(t.fahrenheit)  # Output: 212.0

try:
    t.celsius = -300
except ValueError as error:
    print(error)  # Output: Temperature below absolute zero

The real value is stored in _celsius. Code outside the class reads and assigns t.celsius as if it were a plain attribute, but every assignment is checked. fahrenheit is computed from _celsius each time you read it, and because it has no setter, it is read-only.

Data classes

Many classes exist mainly to hold a few pieces of data. For those, writing __init__ and __repr__ by hand gets repetitive. The @dataclass decorator from the dataclasses module writes them for you, based on a list of fields:

from dataclasses import dataclass

@dataclass
class Product:
    name: str
    price: float
    quantity: int = 0

    def total(self):
        return self.price * self.quantity

pen = Product('Pen', 1.5, 4)
print(pen)          # Output: Product(name='Pen', price=1.5, quantity=4)
print(pen.total())  # Output: 6.0

notebook = Product('Notebook', 3.0)
print(notebook.quantity)  # Output: 0

print(pen == Product('Pen', 1.5, 4))  # Output: True

Each line such as name: str declares a field. The part after the colon is a type hint: it tells readers (and tools) what kind of value the field should hold, but Python does not enforce it. A field with a value, like quantity: int = 0, has a default.

From these fields, @dataclass generates an __init__ that takes the fields as arguments, a readable __repr__, and an __eq__ method so that two products with the same data compare as equal. (Two objects of an ordinary class are only equal if they are the very same object.) You can still add your own methods, such as total.

Key takeaways

A class is a template for objects. You create an object by calling the class, for example Person('John', 30), and Python then calls __init__ to set it up.
Define __str__ and __repr__ so your objects print in a useful way.
Instance attributes (set on self) belong to one object; class attributes (set in the class body) are shared by all instances.
Inheritance lets a subclass reuse and override a parent class’s code. Polymorphism and duck typing let the same code work with any object that has the right methods.
Python has no truly private attributes: _name marks something as internal by convention, and __name triggers name mangling. Use methods and @property to control how data changes.
@dataclass generates __init__, __repr__, and __eq__ for classes that mainly hold data.

Practice

Exercise 1: Rectangle

Write a Rectangle class that takes a width and a height. Give it an area() method, a perimeter() method, and a __str__ method so that print() shows something like Rectangle 3 x 4. Create a 3 by 4 rectangle and print it, its area, and its perimeter.

Solution

class Rectangle:
    def __init__(self, width, height):
        self.width = width
        self.height = height

    def area(self):
        return self.width * self.height

    def perimeter(self):
        return 2 * (self.width + self.height)

    def __str__(self):
        return f'Rectangle {self.width} x {self.height}'

rect = Rectangle(3, 4)
print(rect)
print(rect.area())
print(rect.perimeter())
# Output:
# Rectangle 3 x 4
# 12
# 14

Exercise 2: Shapes and polymorphism

Create a Shape class with a name attribute and an area() method that returns 0. Then create two subclasses: Square (with a side) and Circle (with a radius), each with its own area(). Put one of each in a list and print the name and area of every shape, rounded to two decimal places. Use 3.14159 for pi.

Solution

class Shape:
    def __init__(self, name):
        self.name = name

    def area(self):
        return 0

class Square(Shape):
    def __init__(self, side):
        super().__init__('Square')
        self.side = side

    def area(self):
        return self.side * self.side

class Circle(Shape):
    def __init__(self, radius):
        super().__init__('Circle')
        self.radius = radius

    def area(self):
        return 3.14159 * self.radius * self.radius

shapes = [Square(4), Circle(2)]
for shape in shapes:
    print(f'{shape.name}: {shape.area():.2f}')
# Output:
# Square: 16.00
# Circle: 12.57

Exercise 3: A validated age

Write a Person class with a name and an age. Make age a property whose setter raises a ValueError if the new age is negative. Create a person, change their age to a valid value, then try to set a negative age and print the error message.

Solution

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age  # uses the setter, so it is validated too

    @property
    def age(self):
        return self._age

    @age.setter
    def age(self, value):
        if value < 0:
            raise ValueError('Age cannot be negative')
        self._age = value

person = Person('Maya', 25)
person.age = 26
print(person.age)

try:
    person.age = -5
except ValueError as error:
    print(error)

print(person.age)
# Output:
# 26
# Age cannot be negative
# 26

Chapter 6: Advanced Python Features

This chapter covers features you will reach for as your programs grow: producing values on demand, adding behavior to functions, doing several things at once, splitting code into files, and using some of the most useful parts of Python’s standard library.

In this chapter you will learn:

How generators and iterators produce values one at a time
How decorators add behavior to a function without changing its code
The basics of running tasks concurrently with threads, processes, and asyncio
How to organize code into modules and packages
How to work with dates, times, and time spans
How to store data in a SQLite database and search text with regular expressions

Generators and iterators

A generator is a function that produces a sequence of values on demand, one at a time, instead of building the whole sequence and returning it at once. You write a generator like a normal function, but use the yield keyword to hand out each value.

This generator function produces a sequence of numbers:

def my_range(n):
    i = 0
    while i < n:
        yield i
        i += 1

# Use the generator
for i in my_range(5):
    print(i)
# Output:
# 0
# 1
# 2
# 3
# 4

In this example, my_range produces the numbers from 0 to n - 1. Calling my_range(5) does not run the function body yet; it returns a generator object. The for loop then asks the generator for one value at a time.

Each time the generator reaches yield, it hands out a value and pauses, keeping all its local variables. When the next value is requested, it carries on from exactly where it stopped. You can see the pausing if you add some print calls:

def countdown(n):
    while n > 0:
        print(f'About to yield {n}')
        yield n
        n -= 1

for number in countdown(3):
    print(f'Got {number}')
# Output:
# About to yield 3
# Got 3
# About to yield 2
# Got 2
# About to yield 1
# Got 1

Because a generator only produces values when asked, it uses very little memory, even for a very long sequence.

Generator expressions

For simple cases you can write a generator in one line with a generator expression. It looks like a list comprehension, but with parentheses instead of square brackets:

squares = (x * x for x in range(5))
print(list(squares))  # Output: [0, 1, 4, 9, 16]

# When a generator expression is the only argument, you can drop the extra parentheses
print(sum(x * x for x in range(5)))  # Output: 30

sum(x * x for x in range(5)) adds up the squares without ever building a list of them.

Iterators

Two related words come up whenever you loop in Python:

An iterable is any object you can loop over with for: lists, tuples, strings, dictionaries, files, and generators are all iterables.
An iterator is the object that does the actual stepping. It remembers its position and gives you the next value each time you call next() on it.

A list is an iterable, but it is not an iterator itself. The built-in iter() function gets an iterator from an iterable:

my_list = [1, 2, 3]

# Get an iterator from the list
it = iter(my_list)

print(next(it))  # Output: 1
print(next(it))  # Output: 2
print(next(it))  # Output: 3

# The iterator is now exhausted
try:
    next(it)
except StopIteration:
    print('No more values')  # Output: No more values

# Give next() a default to get that instead of an exception
print(next(it, 'done'))  # Output: done

When an iterator has no more values, next() raises a StopIteration exception. A for loop does all of this for you: it calls iter() on the object, calls next() until StopIteration is raised, and then stops without an error.

An iterator can only be used once. A list hands out a fresh iterator every time you loop over it, so you can loop over a list as often as you like. A generator is its own iterator, so once you have consumed it, it is empty:

numbers = [1, 2, 3]
print(list(numbers))  # Output: [1, 2, 3]
print(list(numbers))  # Output: [1, 2, 3]

squares = (n * n for n in numbers)
print(list(squares))  # Output: [1, 4, 9]
print(list(squares))  # Output: []

Decorators

In Python, functions are objects: you can pass a function to another function, and a function can create and return a new function. A decorator uses this to add behavior to a function without changing its code. It takes a function and returns a new function that usually calls the original one with some extra steps before or after.

The decorator below prints a message before and after the function it decorates:

import functools

def my_decorator(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print('Before calling the function')
        result = func(*args, **kwargs)
        print('After calling the function')
        return result
    return wrapper

@my_decorator
def add(a, b):
    print('Inside the function')
    return a + b

total = add(2, 3)
print(total)
# Output:
# Before calling the function
# Inside the function
# After calling the function
# 5

Here is what each part does:

my_decorator takes a function, func, and returns a new function, wrapper.
Writing @my_decorator above def add is a shortcut for add = my_decorator(add). From then on, calling add really calls wrapper.
wrapper(*args, **kwargs) accepts any positional and keyword arguments and passes them on to func, so the same decorator works for functions with any parameters.
wrapper returns func’s result, so the decorated function still returns its value. Without return result, add(2, 3) would return None.
@functools.wraps(func) copies the original function’s name and docstring onto wrapper, so add.__name__ is still 'add' instead of 'wrapper'. This makes error messages and debugging much clearer.

A common practical use is logging. This decorator prints each call with its arguments and the value it returned:

import functools

def log_calls(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        arguments = [repr(a) for a in args] + [f'{k}={v!r}' for k, v in kwargs.items()]
        print(f'Calling {func.__name__}({", ".join(arguments)})')
        result = func(*args, **kwargs)
        print(f'{func.__name__} returned {result!r}')
        return result
    return wrapper

@log_calls
def greet(name, greeting='Hello'):
    return f'{greeting}, {name}!'

greet('Ana')
greet('Ben', greeting='Hi')
# Output:
# Calling greet('Ana')
# greet returned 'Hello, Ana!'
# Calling greet('Ben', greeting='Hi')
# greet returned 'Hi, Ben!'

Decorators are often used to add logging, timing, caching, or permission checks to functions. You have already used two decorators from the standard library in Chapter 5: @property and @dataclass.

Concurrency

Concurrency means letting several tasks make progress during the same period of time, instead of finishing one completely before starting the next. Python has several ways to do this, including threading, multiprocessing, and asynchronous programming.

Threading

A thread is a separate flow of execution inside your program. All threads run in the same memory space, so they are lightweight and can share data. Python’s threading module lets you create and run threads.

import threading

def print_numbers():
    for i in range(5):
        print(i)

def print_letters():
    for letter in 'abcde':
        print(letter)

# Create the threads
thread1 = threading.Thread(target=print_numbers)
thread2 = threading.Thread(target=print_letters)

# Start the threads
thread1.start()
thread2.start()

# Wait for both threads to finish
thread1.join()
thread2.join()

start() begins running the target function in a new thread, and join() waits until that thread has finished. The two threads run concurrently, so the numbers and letters can come out interleaved, and the order can change from one run to the next.

Multiprocessing

Multiprocessing runs code in separate processes, each with its own memory space and its own Python interpreter. This approach is useful for CPU-bound tasks, such as heavy calculations, because the processes can run on different CPU cores at the same time. Python’s multiprocessing module creates and manages these processes.

import multiprocessing

def square_numbers():
    for i in range(5):
        print(i * i)

if __name__ == '__main__':
    # Create a process
    process = multiprocessing.Process(target=square_numbers)

    # Start the process and wait for it to finish
    process.start()
    process.join()
    print('Process finished')
# Output:
# 0
# 1
# 4
# 9
# 16
# Process finished

The if __name__ == '__main__': line is required on Windows and macOS, and on Linux since Python 3.14: there, a fresh Python interpreter imports your script to set up the new processes, and this check stops that import from starting new processes all over again. Older Python versions on Linux don’t need it, but it does no harm, so write it everywhere. (You will learn more about __name__ in Working with modules and packages.)

Asynchronous programming

Asynchronous programming lets one thread juggle many tasks that spend most of their time waiting, such as network requests or reading files. While one task waits, the others get to run. Python’s asyncio module provides the tools for this.

import asyncio

async def fetch_data():
    print('Start fetching')
    await asyncio.sleep(0.1)  # Simulate waiting for a network request
    print('Done fetching')

async def main():
    await asyncio.gather(
        fetch_data(),
        fetch_data(),
        fetch_data(),
    )

# Run the async function
asyncio.run(main())
# Output:
# Start fetching
# Start fetching
# Start fetching
# Done fetching
# Done fetching
# Done fetching

async def defines a coroutine, a function that can pause while it waits.
await asyncio.sleep(0.1) pauses this coroutine and lets the others run in the meantime.
asyncio.gather runs several coroutines at once and waits for all of them to finish.
asyncio.run starts everything and runs until main is done.

All three tasks start before any of them finishes, because their waits overlap. The whole program takes about 0.1 seconds, not 0.3. With waits of 2 seconds each, it would take about 2 seconds instead of 6.

The simplest option: `concurrent.futures`

If you just want to run the same function on many inputs at once, the concurrent.futures module is the easiest place to start. A ThreadPoolExecutor manages a group of threads for you:

import time
from concurrent.futures import ThreadPoolExecutor

def download(page):
    time.sleep(0.1)  # Pretend to wait for the network
    return f'Contents of {page}'

pages = ['home', 'about', 'contact']

with ThreadPoolExecutor(max_workers=3) as executor:
    results = list(executor.map(download, pages))

for result in results:
    print(result)
# Output:
# Contents of home
# Contents of about
# Contents of contact

executor.map works like the built-in map, but runs the calls in parallel threads. The three downloads wait at the same time, and the results come back in the same order as the inputs. For CPU-bound work, ProcessPoolExecutor from the same module works the same way but uses processes (and needs the if __name__ == '__main__': check).

Differences and use cases

Threading works best for I/O-bound tasks, where the program spends its time waiting for files or the network. In the standard Python interpreter, the Global Interpreter Lock (GIL) lets only one thread run Python code at a time, so threads don’t speed up CPU-bound work.
Multiprocessing is the best choice for CPU-bound tasks. Each process has its own interpreter and its own GIL, so processes can use several CPU cores at once. Starting processes and passing data between them costs more than using threads.
Asynchronous programming is best when you need to handle many I/O operations at once, such as hundreds of network connections. Libraries such as aiohttp (for HTTP requests) and Quart (for web applications) build on asyncio.

In practice, threading (for example with ThreadPoolExecutor) or asyncio works well for web scraping, where you wait for HTTP responses. Data processing that involves heavy computation can use all of your CPU cores with multiprocessing. And if you are building a server that handles many connections at once, asyncio is an efficient choice.

Working with modules and packages

Modules

A module is a Python file that contains functions, variables, classes, and other code that you can use in other programs. Modules help you organize and reuse code. Python comes with a large collection of ready-made modules called the standard library; math, datetime, sqlite3, and re, which you meet in this chapter, are all part of it.

To use a module, import it with the import statement. For example, the math module contains mathematical functions:

import math

result = math.sqrt(16)
print(result)  # Output: 4.0

You can also import specific functions or variables from a module using the from keyword. Then you can use them without the module name in front:

from math import sqrt

result = sqrt(16)
print(result)  # Output: 4.0

You can use the as keyword to give what you import a different name:

from math import sqrt as square_root

result = square_root(16)
print(result)  # Output: 4.0

Any .py file you write is a module too. If you save a file called greetings.py with a hello function in it, another file in the same folder can use it:

# main.py
import greetings

print(greetings.hello('Ana'))

Running a module as a script: `if name == 'main':`

Every module has a built-in variable called __name__. When you run a file directly, for example with python greetings.py, Python sets __name__ to '__main__'. When another file imports it, __name__ is the module’s name, 'greetings'.

This lets a file contain code that runs only when you run the file itself, not when it is imported:

# greetings.py
def hello(name):
    return f'Hello, {name}!'

if __name__ == '__main__':
    # Runs only when you run this file directly, not when it is imported
    print(hello('world'))  # Output: Hello, world!

This is useful for quick tests or demos of a module, and, as you saw in the multiprocessing example, it is sometimes required.

Packages

A package is a folder of modules. Packages let you divide a larger program into smaller, reusable pieces. A package folder usually contains an __init__.py file, which may be empty:

my_project/
├── main.py
└── mypackage/
    ├── __init__.py
    └── mymodule.py

Suppose mypackage/mymodule.py contains this function:

# mypackage/mymodule.py
def some_function():
    return 'Hello from mymodule'

In main.py, you import a module from a package by writing the package name and the module name, separated by a dot. All three of these forms work:

# main.py
import mypackage.mymodule
print(mypackage.mymodule.some_function())

from mypackage import mymodule
print(mymodule.some_function())

from mypackage import mymodule as mm
print(mm.some_function())

The code in __init__.py runs the first time the package is imported. You can use it to make names from inside the package available directly:

# mypackage/__init__.py
from .mymodule import some_function

With that line in place, main.py can write from mypackage import some_function. The dot in .mymodule means “the mymodule module in this same package”.

Besides the standard library, there are hundreds of thousands of third-party packages that you install with pip. You will learn how to install them, and how to keep each project’s packages separate with a virtual environment, in Chapter 8.

Working with dates and times

The datetime module handles dates and times. It has classes for dates (date), times of day (time), a date with a time (datetime), and lengths of time (timedelta).

This gets the current date and time, so the output depends on when you run it:

from datetime import date, datetime

now = datetime.now()
print(now)           # For example: 2022-12-26 14:47:59.438123
print(date.today())  # For example: 2022-12-26

You can also create datetime objects for specific dates and times, and format them as text with the strftime() method:

from datetime import datetime

# Create a date and time
dt = datetime(2022, 12, 26, 14, 50, 0)

# Format the date and time
formatted = dt.strftime('%B %d, %Y %I:%M %p')
print(formatted)  # Output: December 26, 2022 02:50 PM

The codes that start with % stand for parts of the date. Some common ones:

Code	Meaning	Example
`%Y`	Year	`2022`
`%m`	Month as a number	`12`
`%B`	Month name	`December`
`%d`	Day of the month	`26`
`%A`	Weekday name	`Monday`
`%H`	Hour (24-hour clock)	`14`
`%I`	Hour (12-hour clock)	`02`
`%M`	Minute	`50`
`%p`	AM or PM	`PM`

To go the other way, from text to a datetime, use strptime() with the same codes:

from datetime import datetime

dt = datetime.strptime('2024-03-15 09:30', '%Y-%m-%d %H:%M')
print(dt)        # Output: 2024-03-15 09:30:00
print(dt.year)   # Output: 2024
print(dt.month)  # Output: 3

Date arithmetic with `timedelta`

A timedelta is a length of time. Add one to a date to move forward, or subtract two dates to find out how far apart they are:

from datetime import date, datetime, timedelta

start = date(2024, 1, 15)
due = start + timedelta(days=30)
print(due)  # Output: 2024-02-14

days_left = date(2025, 1, 1) - start
print(days_left)       # Output: 352 days, 0:00:00
print(days_left.days)  # Output: 352

shift_start = datetime(2024, 3, 15, 9, 30)
shift_end = datetime(2024, 3, 15, 17, 45)
worked = shift_end - shift_start
print(worked)                          # Output: 8:15:00
print(worked.total_seconds() / 3600)   # Output: 8.25

Time zones

The datetime objects above are naive: they don’t know which time zone they are in. That is fine for many programs, but if you need to compare times from different places, use aware datetimes, which carry a time zone. The zoneinfo module provides time zones by name:

from datetime import datetime
from zoneinfo import ZoneInfo

meeting = datetime(2024, 7, 1, 9, 0, tzinfo=ZoneInfo('America/New_York'))
print(meeting)  # Output: 2024-07-01 09:00:00-04:00

in_kathmandu = meeting.astimezone(ZoneInfo('Asia/Kathmandu'))
print(in_kathmandu)  # Output: 2024-07-01 18:45:00+05:45

astimezone() converts a time to another time zone. On Windows, you may need to install the tzdata package with pip before zoneinfo can find the time zones.

Working with databases

A database stores data in a structured way so that you can search and update it efficiently. Several libraries let you work with databases from Python. Some popular ones:

sqlite3: Python’s built-in module for SQLite, a small database that keeps everything in a single file
psycopg (and the older psycopg2): for PostgreSQL databases
mysqlclient (imported as MySQLdb): for MySQL databases
SQLAlchemy: a flexible library that works with many databases, including MySQL, PostgreSQL, and SQLite

All except sqlite3 are third-party packages that you install with pip.

The following example uses sqlite3 to create a table, insert data into it, and query the data:

import sqlite3

# ':memory:' creates a temporary database that lives in memory.
# Use a filename such as 'mydatabase.db' instead to save it to a file on disk.
conn = sqlite3.connect(':memory:')
cursor = conn.cursor()

# 'with conn' saves (commits) the changes if the block succeeds,
# and undoes (rolls back) all of them if an error occurs
with conn:
    cursor.execute('CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT, email TEXT)')
    cursor.execute('INSERT INTO users (name, email) VALUES (?, ?)', ('John', '[email protected]'))
    cursor.execute('INSERT INTO users (name, email) VALUES (?, ?)', ('Jane', '[email protected]'))

# Query all the rows
cursor.execute('SELECT id, name, email FROM users ORDER BY id')
for row in cursor.fetchall():
    print(row)

# Query with a value supplied separately
cursor.execute('SELECT email FROM users WHERE name = ?', ('Jane',))
print(cursor.fetchone())

# Close the connection
conn.close()
# Output:
# (1, 'John', '[email protected]')
# (2, 'Jane', '[email protected]')
# ('[email protected]',)

The program connects to a database, creates a cursor to run SQL statements, creates a users table if it doesn’t exist yet, inserts two rows, and reads them back. Each row comes back as a tuple. fetchall() returns all the matching rows as a list, and fetchone() returns just the next one. The with conn: block takes care of committing, but it does not close the connection, so the program calls conn.close() at the end.

Notice the ? placeholders. Instead of putting values into the SQL text yourself, you pass them separately as a tuple, and sqlite3 inserts them safely. (A tuple with one item needs a trailing comma: ('Jane',).) Never build SQL by pasting user input into the string, for example with an f-string. A user could type text that changes the meaning of your SQL statement, an attack called SQL injection. Placeholders prevent it.

Working with regular expressions

Regular expressions are patterns for matching text. You can use them to check whether a string contains a pattern, to pull out the parts you need, and to replace parts of a string.

Python’s re module works with regular expressions. This example searches for a pattern in a string:

import re

string = 'The quick brown fox jumps over the lazy dog.'
pattern = r'quick'

match = re.search(pattern, string)
if match:
    print('Match found:', match.group())
else:
    print('Match not found')
# Output: Match found: quick

The r before the pattern makes it a raw string, in which a backslash is just a backslash. Regular expressions use a lot of backslashes, so it is a good habit to always write patterns as raw strings.

re.search() looks for the first place where the pattern matches. If it finds one, it returns a Match object, and group() gives you the text that matched. If there is no match, it returns None.

You can use groups to extract several pieces of information at once:

import re

string = 'The quick brown fox jumps over the lazy dog.'
pattern = r'(quick) (brown) (fox)'

match = re.search(pattern, string)
if match:
    print('Match found:', match.groups())
else:
    print('Match not found')
# Output: Match found: ('quick', 'brown', 'fox')

This pattern contains three groups, each enclosed in parentheses. When there is a match, the groups() method returns a tuple with the text matched by each group.

Patterns become much more useful with special characters. Here are a few:

Pattern	Matches
`\d`	Any digit, 0 to 9
`\w`	Any letter, digit, or underscore
`\s`	Any whitespace, such as a space or tab
`.`	Any character except a newline
`+`	One or more of the previous item
`{3}`	Exactly three of the previous item

re.findall() returns a list of every match, and re.sub() replaces every match with new text:

import re

text = 'Call 555-1234 or 555-9876 today.'
pattern = r'\d{3}-\d{4}'

print(re.findall(pattern, text))  # Output: ['555-1234', '555-9876']
print(re.sub(pattern, '[hidden]', text))  # Output: Call [hidden] or [hidden] today.

Regular expressions are a big topic. To learn them properly, see Regular Expressions for All, the companion book on this site.

Key takeaways

A generator uses yield to produce values one at a time, pausing in between. Iterables (such as lists) can be looped over again and again; iterators (including generators) are used up after one pass.
A decorator wraps a function to add behavior. Write the wrapper with *args, **kwargs, return the result, and use functools.wraps.
Use threads or asyncio for tasks that wait on I/O, and processes for CPU-heavy work. concurrent.futures is the simplest place to start.
Any .py file is a module, and a folder of modules is a package. Put code that should run only when the file is run directly under if __name__ == '__main__':.
The datetime module creates, formats (strftime), parses (strptime), and does arithmetic (timedelta) on dates and times.
With sqlite3, use ? placeholders for values to avoid SQL injection. Use the re module’s search, findall, and sub to work with text patterns.

Practice

Exercise 1: Fibonacci generator

Write a generator function fibonacci(n) that yields the first n Fibonacci numbers, starting with 0 and 1 (each number after that is the sum of the two before it). Print the first 10 as a list.

Solution

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

print(list(fibonacci(10)))
# Output:
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Exercise 2: An uppercase decorator

Write a decorator called uppercase that converts the string returned by a function to uppercase. It should work with functions that take any arguments. Test it on a function greet(name) that returns 'Hello, ' + name.

Solution

import functools

def uppercase(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        return result.upper()
    return wrapper

@uppercase
def greet(name):
    return 'Hello, ' + name

print(greet('Ana'))
print(greet.__name__)
# Output:
# HELLO, ANA
# greet

Exercise 3: A small contacts database

Using sqlite3 with an in-memory database, create a contacts table with name and city columns. Insert at least four contacts, then use a query with a ? placeholder to print the names of everyone who lives in one particular city, in alphabetical order.

Solution

import sqlite3

conn = sqlite3.connect(':memory:')
cursor = conn.cursor()

contacts = [
    ('Asha', 'Kathmandu'),
    ('Ben', 'London'),
    ('Bikash', 'Kathmandu'),
    ('Carla', 'Lima'),
]

with conn:
    cursor.execute('CREATE TABLE IF NOT EXISTS contacts (name TEXT, city TEXT)')
    for contact in contacts:
        cursor.execute('INSERT INTO contacts (name, city) VALUES (?, ?)', contact)

city = 'Kathmandu'
cursor.execute('SELECT name FROM contacts WHERE city = ? ORDER BY name', (city,))
for row in cursor.fetchall():
    print(row[0])

conn.close()
# Output:
# Asha
# Bikash

Chapter 7: Applications of Python

You can use Python to analyze spreadsheets, run websites, control lab experiments, and a great deal more. This chapter is a quick tour of the main areas and the libraries people use in each, so you know where to look when you start a project of your own.

In this chapter you will learn:

How to install third-party packages with pip
Which libraries are used for data analysis and visualization
What the popular web frameworks (Django, Flask, and Pyramid) look like
How to automate tasks: running other programs, calling web APIs, and scheduling jobs
Which libraries are used for numerical computing, machine learning, and simulation

Most examples in this chapter use third-party packages. These are not part of Python’s standard library, so you have to install them before you can import them. The usual tool is pip, which comes with Python. Install packages inside a virtual environment, so that each project keeps its own set of packages; Chapter 8 shows how to create one. Once the virtual environment is active, install a package like this:

python -m pip install pandas

You can install several packages at once, for example python -m pip install numpy matplotlib. If Python reports ModuleNotFoundError: No module named 'pandas', the package isn’t installed in the Python you are using to run your program.

Data analysis and visualization

Python is popular for data analysis and visualization because it has many libraries for the job. These are the ones you are most likely to meet.

pandas

The pandas library is the most widely used library for working with tables of data. Its main data structures are the DataFrame (a table with named columns, like a spreadsheet) and the Series (a single column). It has functions for reading data from files such as CSV and Excel, cleaning it, filtering it, and summarizing it, and it handles dates and missing values well.

This example builds a small table of sales from a dictionary, adds a calculated column, filters the rows, and adds up the revenue for each region:

import pandas as pd

# Build a table from a dictionary: each key becomes a column
sales = pd.DataFrame({
    'region': ['East', 'West', 'East', 'West', 'East'],
    'product': ['Pens', 'Pens', 'Paper', 'Paper', 'Pens'],
    'units': [10, 4, 3, 8, 6],
    'price': [1.5, 1.5, 4.0, 4.0, 1.5],
})

# Add a column calculated from two others (no loop needed)
sales['revenue'] = sales['units'] * sales['price']

# Print the first 5 rows of the table
print(sales.head())
# Output:
#   region product  units  price  revenue
# 0   East    Pens     10    1.5     15.0
# 1   West    Pens      4    1.5      6.0
# 2   East   Paper      3    4.0     12.0
# 3   West   Paper      8    4.0     32.0
# 4   East    Pens      6    1.5      9.0

# Keep only the rows where more than 5 units were sold
big_orders = sales[sales['units'] > 5]
print(big_orders)
# Output:
#   region product  units  price  revenue
# 0   East    Pens     10    1.5     15.0
# 3   West   Paper      8    4.0     32.0
# 4   East    Pens      6    1.5      9.0

# Group the rows by region and add up the revenue in each group
revenue_by_region = sales.groupby('region')['revenue'].sum()
print(revenue_by_region)
# Output:
# region
# East    36.0
# West    38.0
# Name: revenue, dtype: float64

# Save the result to a CSV file
revenue_by_region.reset_index().to_csv('revenue_by_region.csv', index=False)

To load real data from a file instead, use pd.read_csv('sales.csv'), which returns a DataFrame just like the one built here.

After groupby(), the region names become the index (the row labels) of the result, not a normal column. reset_index() turns them back into a column, so that to_csv(..., index=False) writes both the region and the revenue. Without reset_index(), index=False would leave the region names out of the file.

NumPy

Numerical computing in Python is built on the numpy library. Its main data structure is the array: a grid of numbers that all have the same type. You can do arithmetic on a whole array at once, without writing a loop, and NumPy does it much faster than a Python loop would. Many other libraries, including pandas, are built on top of NumPy.

import numpy as np

# Create an array from a Python list
scores = np.array([2, 4, 4, 4, 5, 5, 7, 9])

# Other ways to create arrays
print(np.arange(1, 6))  # Output: [1 2 3 4 5]
print(np.zeros(3))      # Output: [0. 0. 0.]

# Arithmetic applies to every element at once
print(scores * 10)  # Output: [20 40 40 40 50 50 70 90]
print(scores - 1)   # Output: [1 3 3 3 4 4 6 8]

# Statistics
print(scores.mean())  # Output: 5.0
print(scores.std())   # Output: 2.0
print(scores.max())   # Output: 9

# Boolean indexing: keep only the elements that match a condition
print(scores[scores > 4])  # Output: [5 5 7 9]

scores > 4 compares every element with 4 and gives an array of True and False values. Putting that array inside the square brackets keeps only the elements where it is True.

Matplotlib

For drawing charts, the most widely used library is matplotlib. It can create line plots, scatter plots, bar charts, histograms, and many other kinds of plots, and lets you customize every part of them. It is often used together with NumPy and pandas.

import numpy as np
import matplotlib.pyplot as plt

# Create an array with 100 evenly spaced values between 0 and 2*pi
x = np.linspace(0, 2 * np.pi, 100)

# Compute the sine and cosine of every value in the array
y1 = np.sin(x)
y2 = np.cos(x)

# Create a figure with one set of axes
fig, ax = plt.subplots()

# Plot the sine and cosine curves
ax.plot(x, y1, label='sin(x)')
ax.plot(x, y2, label='cos(x)')

# Add a legend and a title
ax.legend()
ax.set_title('Trigonometric functions')

# Show the plot in a window
plt.show()

plt.show() opens a window with the chart. To save the chart as an image file instead, call plt.savefig('trig.png') before plt.show().

seaborn

Built on top of Matplotlib, the seaborn library makes common statistical charts with less code and good-looking defaults. It works directly with pandas DataFrame objects.

This example uses tips, a small sample dataset of restaurant bills that seaborn can download for you, so it needs an internet connection the first time you run it:

import matplotlib.pyplot as plt
import seaborn as sns

# Load a sample dataset into a pandas DataFrame
tips = sns.load_dataset('tips')

# Scatter plot of tip against total bill, with a regression line
# and its 95% confidence interval, in a separate color for smokers
# and non-smokers
sns.lmplot(
    data=tips,
    x='total_bill',
    y='tip',
    hue='smoker',               # Color the points by this column
    ci=95,                      # Shade a 95% confidence interval
    scatter_kws={'alpha': 0.5}, # Make the points semi-transparent
)

plt.show()

scikit-learn

Many people use the scikit-learn library (imported as sklearn) for machine learning. It has tools for common tasks such as classification (predicting a category), regression (predicting a number), and clustering (finding groups), and for measuring how well a model performs.

This example trains a RandomForestClassifier on the iris dataset, a small, classic dataset of flower measurements that comes with scikit-learn. Each flower has four measurements (the features) and a species (the label). The example first splits the data into a training set and a test set with the train_test_split() function from sklearn.model_selection. It then trains the classifier on the training set with the fit() method and predicts the species of the test flowers with the predict() method. Finally, it compares the predictions with the real species using the accuracy_score() function from sklearn.metrics.

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

# X holds the measurements (features), y holds the species (labels)
X, y = load_iris(return_X_y=True)

# Keep 20% of the flowers aside to test the model
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train a random forest classifier
clf = RandomForestClassifier(random_state=42)
clf.fit(X_train, y_train)

# Predict labels for the test set
y_pred = clf.predict(X_test)

# Compute the share of correct predictions
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy:.2f}')

The accuracy is a number between 0 and 1, where 1.00 means every test flower was classified correctly. Setting random_state makes the split and the model the same on every run, so you get the same result each time.

Web development

Python has many libraries and frameworks that make web applications easier to build and deploy, from simple websites to large web-based systems. A web framework takes care of the common work, such as routing (matching a URL to the code that handles it), templating, and database integration, so you can focus on what your site does.

Django

Django is a high-level web framework for building web applications quickly. It includes a lot out of the box: an ORM (object-relational mapper) for working with a database through Python classes, an admin site, user accounts, templates, and form handling. Django organizes code using the model-template-view (MTV) pattern: models describe your data, templates describe how pages look, and views contain the logic that connects the two.

A simple Django view that returns an HTTP response looks like this:

from django.http import HttpResponse

def index(request):
    return HttpResponse('Hello, world!')

A Django view doesn’t run on its own. It lives inside a Django project, which you create with the django-admin startproject command, and you connect it to a URL in the project’s urls.py file. The official Django tutorial walks you through the whole setup.

Flask

Flask is a microframework: it gives you only the essentials, such as URL routing and templates, and you add other features with extensions when you need them. It is lightweight and flexible, and a good choice for small to medium-sized web applications and APIs.

Here is a simple Flask app that returns an HTTP response:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return 'Hello, world!'

if __name__ == '__main__':
    app.run()

Save this as app.py, run it, and open http://127.0.0.1:5000 in your browser to see the page. Press Ctrl+C in the terminal to stop the server.

Pyramid

Pyramid is a web framework designed to be flexible and extensible. Like Flask, it starts small, but it is built to grow into larger applications or ones that need custom behavior.

This simple Pyramid app also returns an HTTP response. It uses Waitress, a separate package, as the web server, so install both with python -m pip install pyramid waitress:

from pyramid.config import Configurator
from pyramid.response import Response
from waitress import serve

def index(request):
    return Response('Hello, world!')

if __name__ == '__main__':
    config = Configurator()
    config.add_route('index', '/')
    config.add_view(index, route_name='index')
    app = config.make_wsgi_app()
    serve(app, host='127.0.0.1', port=8080)

Run it and open http://127.0.0.1:8080 in your browser.

Automation

A short Python script can handle many everyday chores on a computer: renaming files, running other programs, downloading data, or doing the same job every hour. Python’s standard library already includes most of the tools you need.

Running command-line programs

The subprocess module lets your program run other programs and capture their output, so you can automate tasks that involve running several programs or shell commands in a row.

This example runs a second Python program and captures what it prints:

import subprocess
import sys

# sys.executable is the path of the Python interpreter running this
# script, so this example works the same on Windows, macOS, and Linux
result = subprocess.run(
    [sys.executable, '-c', "print('Hello from another program')"],
    capture_output=True,  # Collect the output instead of showing it
    text=True,            # Give the output as a str instead of bytes
    check=True,           # Raise an exception if the program fails
)

print(result.returncode)      # Output: 0
print(result.stdout.strip())  # Output: Hello from another program

You pass the command as a list: the program name first, then each argument as a separate item. For example, subprocess.run(['ls', '-l'], capture_output=True, text=True, check=True) lists the files in the current folder on macOS and Linux. Windows has no ls program, though, so check that the commands you run exist on every system your script needs to support.

A program signals a problem by finishing with an exit code other than 0. With check=True, subprocess.run() turns that into a CalledProcessError exception, so a failure can’t go unnoticed:

import subprocess
import sys

try:
    # This program exits with code 3 to report a problem
    subprocess.run([sys.executable, '-c', 'import sys; sys.exit(3)'], check=True)
except subprocess.CalledProcessError as error:
    print(f'The program failed with exit code {error.returncode}')
# Output: The program failed with exit code 3

Interacting with web APIs

Many websites offer an API (application programming interface): a set of URLs that return data for programs instead of web pages, usually in JSON format. Python can send HTTP requests to these APIs and process the responses, so a script can fetch or update data on a website for you.

The third-party requests library is the most popular way to make HTTP requests. (The standard library’s urllib.request module can do it too, but takes more code.) The example below asks the GitHub API for information about the repository that holds Python’s own source code:

import requests

url = 'https://api.github.com/repos/python/cpython'

try:
    # timeout=10 stops waiting if the server doesn't answer within 10 seconds
    response = requests.get(url, timeout=10)
    # Raise an exception if the server answered with an error, such as 404
    response.raise_for_status()
except requests.RequestException as error:
    print(f'Request failed: {error}')
else:
    data = response.json()  # Turn the JSON text into dicts and lists
    print(data['full_name'])  # Output: python/cpython
    print(data['description'])

Always pass a timeout; without one, requests can wait forever for a server that never answers. raise_for_status() raises an HTTPError when the response has an error status code (400 or higher), so you don’t have to check response.status_code yourself. Catching requests.RequestException handles both that and network problems such as a missing internet connection.

Scheduling tasks

Python can run a task at a specific time or at regular intervals, which suits jobs that need to happen on a schedule, such as data backups or system maintenance.

The sched module from the standard library runs functions after a delay. The scheduler runs each scheduled event once, so to repeat a task, the task schedules its own next run. This example runs a task three times, 0.1 seconds apart, and then stops:

import sched
import time

INTERVAL = 0.1  # Seconds between runs (use 3600 for once an hour)
RUNS = 3        # How many times to run the task

scheduler = sched.scheduler(time.monotonic, time.sleep)

def task(run_number):
    print(f'Running task (run {run_number})')
    # Schedule the next run, unless this was the last one
    if run_number < RUNS:
        scheduler.enter(INTERVAL, 1, task, argument=(run_number + 1,))

# Schedule the first run
scheduler.enter(INTERVAL, 1, task, argument=(1,))

# run() waits for each event in turn, and returns when none are left
scheduler.run()
print('All runs finished')
# Output:
# Running task (run 1)
# Running task (run 2)
# Running task (run 3)
# All runs finished

The enter() method takes the delay in seconds, a priority (used when two events are due at the same time; lower numbers run first), the function to call, and a tuple of arguments for it. The run() method starts the scheduler.

For a task that repeats forever, a loop with time.sleep() is often enough:

import time

while True:
    print('Running task...')
    time.sleep(3600)  # Wait one hour

A Python script can only run scheduled tasks while it is running. For jobs that must keep running after you close the terminal or restart the computer, use your operating system’s scheduler to start the script for you: cron on macOS and Linux, or Task Scheduler on Windows.

Scientific computing

Python is widely used in science and engineering. Its libraries let you do heavy numerical work with short, readable code.

Numerical computing

The numpy and scipy libraries have functions for numerical computations such as linear algebra, optimization, integration, and statistics. NumPy provides the arrays and basic operations; SciPy builds on it with more specialized tools.

This example uses NumPy for a linear algebra operation, finding the inverse of a matrix:

import numpy as np

# Create a 2x2 matrix
A = np.array([[1, 2], [3, 4]])

# Calculate the inverse of the matrix
A_inv = np.linalg.inv(A)

print(A_inv)
# Output:
# [[-2.   1. ]
#  [ 1.5 -0.5]]

Machine learning

Libraries such as scikit-learn, tensorflow, and pytorch provide tools for training and evaluating machine learning models.

The next example uses scikit-learn to train a simple linear regression model, which finds the straight line that best fits some data:

import numpy as np
from sklearn.linear_model import LinearRegression

# Generate 100 random data points that roughly follow y = 5 + 3x
X = np.random.rand(100, 1)
y = 5 + 3 * X + np.random.rand(100, 1)

# Create a linear regression model
model = LinearRegression()

# Fit the model to the data
model.fit(X, y)

# Print the coefficient (slope) and intercept of the line it found
print(f'Coefficient: {model.coef_[0][0]:.2f}')
print(f'Intercept: {model.intercept_[0]:.2f}')

The data is random, so the numbers change on every run. The coefficient should come out close to 3, and the intercept close to 5.5: 5 plus 0.5, the average of the random noise, which lies between 0 and 1.

Scientific simulation

A simulation imitates a real system so you can study it without building it. The simpy library is used for discrete-event simulation: modeling systems where things happen at particular moments, such as customers arriving at a shop and waiting for a free cashier, or cars waiting at a traffic light. SimPy describes each actor in the simulation as a generator function (see Chapter 6) that uses yield to wait for an event, such as some time passing or a resource becoming free.

The example below models a simple queue: five customers arrive at a shop with one counter, wait their turn, are served, and leave. Simulated time is measured in minutes.

import random

import simpy

random.seed(42)  # Use the same random numbers on every run

def customer(env, name, counter):
    arrived = env.now
    print(f'{env.now:5.2f}: {name} arrives')
    with counter.request() as request:
        yield request  # Wait until the counter is free
        waited = env.now - arrived
        print(f'{env.now:5.2f}: {name} is served after waiting {waited:.2f}')
        yield env.timeout(random.uniform(1, 3))  # Serving takes 1 to 3 minutes
        print(f'{env.now:5.2f}: {name} leaves')

def customer_arrivals(env, counter):
    for number in range(1, 6):
        env.process(customer(env, f'Customer {number}', counter))
        yield env.timeout(random.uniform(0.5, 1.5))  # Next customer in 0.5 to 1.5 minutes

# Create the simulation environment and a counter that serves one customer at a time
env = simpy.Environment()
counter = simpy.Resource(env, capacity=1)

# Start generating customers, then run the simulation until nothing is left to do
env.process(customer_arrivals(env, counter))
env.run()

The program prints a line each time a customer arrives, is served, or leaves, with the simulated time in minutes at the start of the line. random.uniform(a, b) picks a random number between a and b. Customers arrive faster than the counter can serve them, so a queue builds up and later customers have to wait. Try changing the counter’s capacity to 2 and see how the waiting times change.

Key takeaways

In each of these fields, most of what Python can do comes from third-party packages. Install them with python -m pip install inside a virtual environment.
pandas works with tables of data, NumPy works with arrays of numbers, and Matplotlib and seaborn turn data into charts.
Django is a full-featured web framework; Flask and Pyramid start smaller and let you add what you need.
The standard library covers a lot of automation on its own: subprocess runs other programs and sched runs tasks on a timer. For regular jobs, use cron or Task Scheduler.
When you call a web API, always set a timeout and check the response for errors.

Practice

Exercise 1: Sales totals with the csv module

The pandas example in this chapter adds up revenue by region. Do the same with only the standard library. Write the sales data below to a file named sales.csv with csv.writer, read the file back with csv.DictReader, and print the total revenue (units times price) for each region with two decimal places. Remember that the csv module gives you every value as a string.

region,product,units,price
East,Pens,10,1.5
West,Pens,4,1.5
East,Paper,3,4.0
West,Paper,8,4.0
East,Pens,6,1.5

Solution

import csv

rows = [
    ['region', 'product', 'units', 'price'],
    ['East', 'Pens', 10, 1.5],
    ['West', 'Pens', 4, 1.5],
    ['East', 'Paper', 3, 4.0],
    ['West', 'Paper', 8, 4.0],
    ['East', 'Pens', 6, 1.5],
]

# Write the data to a CSV file
with open('sales.csv', 'w', newline='') as file:
    csv.writer(file).writerows(rows)

# Read the file back and add up the revenue for each region
revenue_by_region = {}
with open('sales.csv', newline='') as file:
    for row in csv.DictReader(file):
        revenue = int(row['units']) * float(row['price'])
        region = row['region']
        revenue_by_region[region] = revenue_by_region.get(region, 0) + revenue

for region, total in revenue_by_region.items():
    print(f'{region}: {total:.2f}')
# Output:
# East: 36.00
# West: 38.00

Exercise 2: Read JSON data

Web APIs usually send their data as JSON text. The string below is shaped like a response from a weather API. Use json.loads() to turn it into Python dictionaries and lists, then print the names of the cities warmer than 20 degrees and the average temperature of all the cities.

response_text = '''
{
    "unit": "celsius",
    "readings": [
        {"city": "Kathmandu", "temp": 22},
        {"city": "Oslo", "temp": 9},
        {"city": "Cairo", "temp": 31},
        {"city": "Lima", "temp": 18}
    ]
}
'''

Solution

import json

response_text = '''
{
    "unit": "celsius",
    "readings": [
        {"city": "Kathmandu", "temp": 22},
        {"city": "Oslo", "temp": 9},
        {"city": "Cairo", "temp": 31},
        {"city": "Lima", "temp": 18}
    ]
}
'''

data = json.loads(response_text)
readings = data['readings']

warm_cities = [r['city'] for r in readings if r['temp'] > 20]
print(f"Warm cities: {', '.join(warm_cities)}")

average = sum(r['temp'] for r in readings) / len(readings)
print(f"Average: {average} {data['unit']}")
# Output:
# Warm cities: Kathmandu, Cairo
# Average: 20.0 celsius

Exercise 3: Run a program and check the result

Write a function run_python(code) that runs a piece of Python code in a separate process with subprocess.run() and returns what it printed, without the trailing newline. Use sys.executable so that it works on every operating system. If the code fails, return the text failed with exit code N instead, where N is the program’s exit code. Test it with the code print(2 ** 10) and with the code 1 / 0.

Solution

import subprocess
import sys

def run_python(code):
    try:
        result = subprocess.run(
            [sys.executable, '-c', code],
            capture_output=True,
            text=True,
            check=True,
        )
    except subprocess.CalledProcessError as error:
        return f'failed with exit code {error.returncode}'
    return result.stdout.strip()

print(run_python('print(2 ** 10)'))  # Output: 1024
print(run_python('1 / 0'))           # Output: failed with exit code 1

Chapter 8: Tips and Tricks

Once your code works, you still have to read it, test it, and fix it. This chapter covers the habits and tools that make that easier, along with the mistakes almost every beginner makes at least once.

In this chapter you will learn:

How to follow the PEP 8 style guide and let tools format and check your code
How to write useful comments, docstrings, and type hints
How to test your code with unittest and pytest
How to set up a virtual environment and install packages
How to track down bugs with print(), tracebacks, the debugger, assert, and logging
Which common pitfalls to watch out for, and how to avoid them

Best practices for writing Python code

These practices make your code easier to read and maintain, for other people and for yourself a few months from now.

Use a good editor and version control

Use a code editor or IDE that has syntax highlighting, code completion, and a built-in debugger, such as VS Code or PyCharm. A good editor makes code easier to write and catches many mistakes as you type.

Use version control software such as Git to track changes to your code. Git records every version of your files, so you can see what changed, undo a change that broke something, and collaborate with other developers.

Follow the PEP 8 style guide

PEP 8 is the official style guide for Python code. You can read it at peps.python.org/pep-0008. Following it makes your code consistent with most other Python code, so it is easier for anyone to read. The next few sections cover its most important points.

Indent with 4 spaces

Use 4 spaces for each level of indentation, and don’t use tabs. Tabs can be displayed at different widths in different editors, and mixing tabs and spaces leads to confusing errors. Most code editors can insert 4 spaces when you press the Tab key; check your editor’s settings.

def describe_numbers(limit):
    for i in range(limit):          # 4 spaces
        if i % 2 == 0:              # 8 spaces
            print(i, 'is even')     # 12 spaces
        else:
            print(i, 'is odd')

describe_numbers(3)
# Output:
# 0 is even
# 1 is odd
# 2 is even

Keep lines short

PEP 8 recommends a maximum line length of 79 characters. Long lines are hard to read, especially when you have two files side by side or are working on a small screen. Many teams use a slightly longer limit, such as 88 characters (the default of the Black and Ruff formatters) or 100. Whichever you choose, use the same limit across the whole project.

The preferred way to break a long line is to put the line break inside parentheses, brackets, or braces. Python knows the statement isn’t finished until the closing bracket, so you can continue on the next line:

def total_price(price, quantity, tax_rate, discount):
    return price * quantity * (1 + tax_rate) - discount

# One argument per line, inside the parentheses of the call
total = total_price(
    price=19.99,
    quantity=3,
    tax_rate=0.08,
    discount=5,
)
print(f'{total:.2f}')  # Output: 59.77

To split a long string, put several shorter strings next to each other inside parentheses. Python joins string literals that sit next to each other into one string:

message = (
    'Python joins string literals that sit next to each other, '
    'so you can split a long string over several lines.'
)
print(message)
# Output:
# Python joins string literals that sit next to each other, so you can split a long string over several lines.

You can also end a line with a backslash (\) to continue it on the next line, but PEP 8 prefers parentheses. A backslash is easy to miss, and inside a string it doesn’t do what you might expect: the indentation of the next line becomes part of the string.

# A backslash at the end of a line continues the statement...
total = 1 + 2 + \
    3
print(total)  # Output: 6

# ...but inside a string, the spaces at the start of the next line
# end up in the string
text = 'first part \
    second part'
print(text)  # Output: first part     second part

Pick one quote style and stick to it

In Python, single quotes and double quotes create exactly the same kind of string. PEP 8 doesn’t prefer either one; it only asks you to pick one style and use it consistently. (The examples in this book use both, so you can see they behave the same. In your own projects, pick one: the Black and Ruff formatters use double quotes.)

When a string contains a quote character, use the other kind of quote around it, so you don’t need a backslash:

# Single and double quotes make the same string
print('Hello, world!' == "Hello, world!")  # Output: True

# Use the other kind of quote when the string contains a quote
print('He said, "Hello!"')  # Output: He said, "Hello!"
print("It's a nice day")    # Output: It's a nice day

If you need to use the same kind of quote that surrounds the string, put a backslash (\) in front of it. This is called escaping the quote:

print('It\'s a nice day')       # Output: It's a nice day
print("He said, \"Hello!\"")    # Output: He said, "Hello!"

Triple-quoted strings, written with three single quotes or three double quotes, can span multiple lines and contain both kinds of quotes without escaping:

text = """This string spans
several lines and can contain 'single'
and "double" quotes without escaping."""
print(text)
# Output:
# This string spans
# several lines and can contain 'single'
# and "double" quotes without escaping.

Choose clear names

Use clear and descriptive names for variables, functions, and classes. A good name tells the reader what a value means or what a function or class is for, so the code explains itself. Avoid abbreviations unless they are widely known, and keep names as short as you can while still being descriptive.

PEP 8 also says how to write names:

Variable and function names use snake_case: lowercase words separated by underscores (for example my_variable, calculate_total).
Class names use CapWords, also called PascalCase: each word starts with a capital letter and there are no underscores (for example MyClass, StudentRecord). This is different from camelCase, which starts with a lowercase letter (myClass). PEP 8 allows camelCase only in code that already uses it, such as older standard library modules like logging (logging.getLogger), so don’t use it in new code.
Constants, values that never change while the program runs, are written in UPPER_CASE with underscores (for example MAX_SCORE).

Compare these two functions. They do the same thing, but only the second tells you what it is for:

# Hard to understand: what are f, a, b, and x?
def f(a):
    b = 0
    for x in a:
        b += x
    return b / len(a)

# Clear and descriptive names
MAX_SCORE = 100

def calculate_average_score(scores):
    total_score = 0
    for score in scores:
        total_score += score
    return total_score / len(scores)

class StudentRecord:
    def __init__(self, name, scores):
        self.name = name
        self.scores = scores

student = StudentRecord('Asha', [80, 92, 74])
print(calculate_average_score(student.scores))  # Output: 82.0
print(f(student.scores))                        # Output: 82.0

Write useful comments and docstrings

Use comments to explain why your code does something, especially when that isn’t obvious from the code itself. A comment that only repeats what the code says adds nothing, and it can become wrong if someone later changes the code but not the comment.

rows = ['name,score', 'Asha,80', 'Ben,92']

# Not useful: the comment repeats the code
data_rows = len(rows) - 1  # Subtract 1 from the length of rows

# Useful: the comment explains why
data_rows = len(rows) - 1  # The first row is a header, not data

print(data_rows)  # Output: 2

To describe what a function, class, or module does, write a docstring: a string on the first line of its body. Unlike a comment, a docstring is kept by Python, and tools such as help() and your editor show it to anyone who uses the function.

def shipping_cost(weight_kg):
    """Return the cost in dollars of shipping a parcel.

    The courier charges $2.50 per kilogram, with a minimum of $5.
    """
    return max(weight_kg * 2.5, 5.0)

print(shipping_cost(4))  # Output: 10.0
print(shipping_cost(1))  # Output: 5.0

Some general guidelines for comments:

Explain the purpose of a function or class in its docstring, and the logic of complex or non-obvious code in comments.
Don’t repeat what the code already says (for example, “Increment x by 1” next to x += 1).
Keep comments clear, short, and up to date.
Don’t leave old code commented out “just in case”. Delete it. Your version control system keeps every earlier version, so you can always get it back.

Let tools format and check your code

You don’t have to apply style rules by hand. A formatter rewrites your code in a consistent style, and a linter points out likely mistakes, such as unused imports or variables that are used before they are defined. Two popular choices are:

Ruff, which is both a fast linter and a formatter
Black, a formatter

Install Ruff with pip (see Virtual environments and packages below), then run it from the folder that holds your code:

python -m pip install ruff
ruff check .     # Look for problems in every Python file in this folder
ruff format .    # Reformat every Python file in this folder

Most code editors can run a formatter every time you save a file, so you never have to think about it.

Add type hints

Type hints say what types of values a function expects and what it returns. They are written after a colon for each parameter, and after -> for the return value:

def area(width: float, height: float) -> float:
    return width * height

print(area(3.0, 4.5))  # Output: 13.5

# Python doesn't check type hints when the program runs
print(area('ab', 3))   # Output: ababab

Type hints are documentation that tools can read. Python itself ignores them when it runs your code, as the second call shows. Your editor uses them to suggest completions and flag mistakes, and a type checker such as mypy checks the whole program before you run it:

python -m pip install mypy
mypy shapes.py

For the code above, saved as shapes.py, mypy reports an error for the second call: Argument 1 to "area" has incompatible type "str"; expected "float".

Handle errors with exceptions

Use try and except to handle errors that you expect could happen, such as a missing file or invalid input, instead of letting your program crash. The else block runs only if no exception was raised, and the finally block always runs:

x = 10
y = 0

try:
    result = x / y
except ZeroDivisionError:
    print('Error: Cannot divide by zero')
else:
    print(result)
finally:
    print('Finished')
# Output:
# Error: Cannot divide by zero
# Finished

Catch only the exceptions you expect and know how to handle, like ZeroDivisionError here. A bare except: catches every error, including bugs you would want to know about. Chapter 4 covers exception handling in more detail.

Test your code

Automated tests are small pieces of code that check that your code gives the right results. Run them after every change, and they tell you right away if the change broke something that used to work (this is called a regression).

A test calls your code with some input and checks that the result matches the output you expect. Python comes with the unittest module for writing tests:

import unittest

def add(x, y):
    return x + y

class TestAdd(unittest.TestCase):
    def test_positive_numbers(self):
        self.assertEqual(add(2, 3), 5)

    def test_negative_numbers(self):
        self.assertEqual(add(-1, -1), -2)

if __name__ == '__main__':
    unittest.main()

Each method whose name starts with test is one test. When you run the file, unittest runs every test and prints a dot for each one that passes, then a summary (the time will differ):

..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

If a test fails, unittest prints an F instead of a dot, followed by the name of the failing test and both values, for example AssertionError: -1 != 5.

Many Python projects use pytest instead, a third-party package that lets you write tests as plain functions with assert:

# test_calculator.py
def add(x, y):
    return x + y

def test_add_positive_numbers():
    assert add(2, 3) == 5

def test_add_negative_numbers():
    assert add(-1, -1) == -2

Install pytest, then run it from the folder that holds your tests. It finds every file whose name starts with test_ and runs every function in it whose name starts with test_:

python -m pip install pytest
pytest

pytest prints a dot for each passing test and ends with a summary such as 2 passed in 0.01s.

Virtual environments and packages

Python packages that aren’t in the standard library, such as requests or pandas, are installed with pip. By default, pip installs packages into the Python installation you run it with, so every project on your computer shares the same packages. That causes problems when two projects need different versions of the same package.

A virtual environment solves this. It is a folder inside your project with its own python command and its own packages. The python command runs the Python installation you created the environment with (on macOS and Linux it is a link to it), so the folder doesn’t hold a full copy of Python. Each project gets its own environment, and you can delete the folder and start over at any time without affecting anything else.

Create a virtual environment

Open a terminal in your project folder and create a virtual environment in a folder named .venv. On macOS and Linux:

python3 -m venv .venv

On Windows:

py -m venv .venv

Activate it

Before you use the environment, activate it. On macOS and Linux:

source .venv/bin/activate

On Windows (Command Prompt or PowerShell):

.venv\Scripts\activate

If PowerShell says that running scripts is disabled on your system, use Command Prompt instead.

Your prompt now starts with (.venv). While the environment is active, the python command runs the environment’s Python on every operating system, and the packages you install go into the .venv folder.

Install packages

Install packages with pip:

python -m pip install requests

Running pip as python -m pip makes sure it installs into the Python you are currently using.

Save your package list

To record the packages your project uses, save them in a file named requirements.txt:

python -m pip freeze > requirements.txt

Anyone who wants to run your project, including you on another computer, can then create a fresh environment and install the same versions:

python -m pip install -r requirements.txt

Don’t put the .venv folder itself in version control; add it to your .gitignore file and commit requirements.txt instead.

Deactivate it

When you are done working on the project, leave the environment:

deactivate

Debugging techniques

Debugging means finding and fixing errors in your code. These are some common techniques.

Print statements

One of the simplest and most effective techniques is to add print() calls that show the values of variables at different points in your code. They show you what the code is really doing and where it starts to differ from what you expected. An f-string with = after the expression prints both the expression and its value:

def average(numbers):
    total = sum(numbers)
    print(f'{total=}, {len(numbers)=}')  # Temporary debugging line
    return total / len(numbers)

print(average([4, 8, 6]))
# Output:
# total=18, len(numbers)=3
# 6.0

Remember to remove debugging prints when you have found the problem.

Read the traceback

When an exception isn’t handled, Python stops and prints a traceback. Suppose this program is saved as shop.py:

def get_price(prices, item):
    return prices[item]

prices = {'apple': 0.5, 'pear': 0.75}
print(get_price(prices, 'banana'))

Running it prints:

Traceback (most recent call last):
  File "/Users/you/python-practice/shop.py", line 5, in <module>
    print(get_price(prices, 'banana'))
          ~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/you/python-practice/shop.py", line 2, in get_price
    return prices[item]
           ~~~~~~^^^^^^
KeyError: 'banana'

Read it from the bottom up. The last line names the exception and gives a message: there is no key 'banana' in the dictionary. The lines above show the chain of calls that led there, ending with the line where the error happened: line 2, inside get_price().

Use a debugger

A debugger lets you pause your program and look around: run it one line at a time and inspect the values of variables at each step. Most code editors have a graphical debugger, where you click next to a line number to set a breakpoint, the place where the program pauses.

Python also has a built-in debugger called pdb. Add a call to breakpoint() where you want the program to pause:

def average(numbers):
    total = sum(numbers)
    breakpoint()  # The program pauses here and starts the debugger
    return total / len(numbers)

print(average([4, 8, 6]))

When the program reaches breakpoint(), it shows a (Pdb) prompt where you can type commands. The most useful ones are:

n          Run the next line (without stepping into function calls)
s          Step into the function called on the current line
p total    Print the value of a variable or expression
c          Continue running until the next breakpoint
q          Quit the debugger and stop the program

Remove the breakpoint() call when you are done.

Check assumptions with assert

The assert statement checks that a condition is true. If it is, the program continues. If it isn’t, Python raises an AssertionError with the message you give. This makes a wrong value stop the program right where the problem starts, instead of causing a confusing error later on.

def apply_discount(price, percent):
    assert 0 <= percent <= 100, f'percent must be 0-100, got {percent}'
    return price * (1 - percent / 100)

print(apply_discount(50, 10))  # Output: 45.0

try:
    apply_discount(50, 150)
except AssertionError as error:
    print(f'AssertionError: {error}')
# Output: AssertionError: percent must be 0-100, got 150

Use assert to catch mistakes in your own code, not to check user input. Python skips assert statements when it is run with the -O (optimize) option, so they can’t be relied on to protect your program.

Log instead of print

The logging module is a more flexible alternative to debugging prints. Each message has a level, such as DEBUG, INFO, WARNING, or ERROR, and you choose which levels to show. You can turn messages off by changing one line, without deleting them from your code, and you can send them to a file instead of the screen.

import logging
import sys

# Show messages of level INFO and above.
# By default, logging writes to the error stream (stderr);
# stream=sys.stdout sends the messages to the same place as print().
logging.basicConfig(
    level=logging.INFO,
    format='%(levelname)s: %(message)s',
    stream=sys.stdout,
)

def divide(a, b):
    logging.debug('divide() called with %s and %s', a, b)  # Hidden: below INFO
    if b == 0:
        logging.error('Cannot divide %s by zero', a)
        return None
    logging.info('Dividing %s by %s', a, b)
    return a / b

print(divide(10, 2))
print(divide(1, 0))
# Output:
# INFO: Dividing 10 by 2
# 5.0
# ERROR: Cannot divide 1 by zero
# None

Change level=logging.INFO to level=logging.DEBUG to see the debug messages too. To write the messages to a file, replace stream=sys.stdout with filename='app.log'.

Common pitfalls to avoid

Some mistakes come up again and again when you are learning Python, and knowing about them in advance saves you a lot of time.

Errors Python reports for you

These mistakes stop your program with an error message, so at least you know something is wrong:

Indentation errors: Indentation defines code blocks, such as the body of a function, loop, or if statement. If the indentation is wrong, Python raises an IndentationError, or your code runs in the wrong block. Use 4 spaces for each level, and never mix tabs and spaces.
Name errors: You have to define a variable, function, or class before you use it. Using a name that doesn’t exist, often because of a typo, raises a NameError.
Type errors: An operation that doesn’t work for a type raises a TypeError. For example, you can’t join a string and an integer with + (see the example below).
Syntax errors: Missing colons, brackets, or quotes raise a SyntaxError before your program even starts. The error message points to where Python got confused; the actual mistake is often on that line or the line before it.
Unhandled exceptions: If an exception isn’t handled, your program stops and prints a traceback. Use try and except for errors you can expect, such as a missing file or invalid input.

Here is the TypeError from item 3, and two ways to fix it:

age = 25

try:
    print('Age: ' + age)
except TypeError as error:
    print(f'TypeError: {error}')
# Output: TypeError: can only concatenate str (not "int") to str

# Fix: convert the number to a string, or use an f-string
print('Age: ' + str(age))  # Output: Age: 25
print(f'Age: {age}')       # Output: Age: 25

The pitfalls in the rest of this section are harder to spot, because they don’t raise an error. Your program runs, but gives the wrong result.

Mutable default arguments

A default argument value is created only once, when the function is defined, not each time the function is called. If the default is a mutable object, such as a list or dictionary, every call that uses the default shares the same object:

def add_item(item, items=[]):
    items.append(item)
    return items

print(add_item('apple'))   # Output: ['apple']
print(add_item('banana'))  # Output: ['apple', 'banana']

The second call still sees 'apple' from the first call. The fix is to use None as the default and create a new list inside the function:

def add_item(item, items=None):
    if items is None:
        items = []
    items.append(item)
    return items

print(add_item('apple'))   # Output: ['apple']
print(add_item('banana'))  # Output: ['banana']

Changing a list while looping over it

Removing items from a list while a for loop goes through it makes the loop skip items. Here the loop should remove every even number, but one is left behind:

numbers = [1, 2, 2, 3, 4]
for number in numbers:
    if number % 2 == 0:
        numbers.remove(number)

print(numbers)  # Output: [1, 2, 3]

When the first 2 is removed, every later item moves one position to the left, so the loop skips the second 2. The fix is to build a new list with the items you want to keep, or to loop over a copy of the list (numbers[:]):

numbers = [1, 2, 2, 3, 4]
odd_numbers = [number for number in numbers if number % 2 != 0]
print(odd_numbers)  # Output: [1, 3]

Confusing `is` with `==`

== checks whether two values are equal. is checks whether two names refer to the very same object in memory:

a = [1, 2, 3]
b = [1, 2, 3]

print(a == b)  # Output: True
print(a is b)  # Output: False

The lists have the same contents, so they are equal, but they are two separate objects. Use == to compare values. Use is only to check for None, as in if items is None:. Sometimes is seems to work for small numbers or short strings, because Python reuses some objects behind the scenes, but you can’t rely on that.

`/` versus `//`

The / operator always returns a float, even when both numbers are integers and the result is a whole number. The // operator (floor division) returns the result rounded down to a whole number. Rounding down means toward negative infinity, so -7 // 2 is -4, not -3:

print(7 / 2)    # Output: 3.5
print(6 / 2)    # Output: 3.0
print(7 // 2)   # Output: 3
print(-7 // 2)  # Output: -4

This matters when you use the result as a list index, which has to be an integer:

items = ['a', 'b', 'c', 'd', 'e']

try:
    print(items[len(items) / 2])
except TypeError as error:
    print(f'TypeError: {error}')
# Output: TypeError: list indices must be integers or slices, not float

print(items[len(items) // 2])  # Output: c

Shadowing built-in names

Python lets you use the names of built-in functions and types, such as list, str, dict, sum, max, input, or id, as variable names. But once you do, the built-in is hidden behind your variable:

list = [3, 1, 2]  # 'list' now means this list, not the built-in type

try:
    letters = list('abc')
except TypeError as error:
    print(f'TypeError: {error}')
# Output: TypeError: 'list' object is not callable

The fix is to choose a more descriptive name, which also makes your code clearer:

numbers = [3, 1, 2]
letters = list('abc')
print(letters)  # Output: ['a', 'b', 'c']

Key takeaways

Follow PEP 8: 4-space indentation, short lines, snake_case for functions and variables, and CapWords for classes. Let a formatter and linter such as Ruff do the work.
Write comments that explain why, docstrings that explain what a function does, and type hints that tools such as mypy can check.
Test your code automatically with unittest or pytest, so you find out right away when a change breaks something.
Give each project its own virtual environment, and record its packages in requirements.txt.
To find bugs, read tracebacks from the bottom up, and use print(), breakpoint(), assert, and logging to see what your code is really doing.
Watch out for mutable default arguments, changing a list while looping over it, is versus ==, / versus //, and variable names that hide built-ins.

Practice

Exercise 1: Fix the shared list

The function below is supposed to start a new list of scores for each student, unless it is given an existing list. Instead, every student’s scores end up in the same list. Find the bug and fix it so that the program prints [90] and then [75].

def add_score(score, scores=[]):
    scores.append(score)
    return scores

print(add_score(90))  # Output: [90]
print(add_score(75))  # Output: [90, 75]

Solution

def add_score(score, scores=None):
    # Create a new list on each call, instead of sharing one default list
    if scores is None:
        scores = []
    scores.append(score)
    return scores

print(add_score(90))
print(add_score(75))
# Output:
# [90]
# [75]

Exercise 2: Test a function with assert

Write a function celsius_to_fahrenheit(celsius) that converts a temperature using the formula celsius * 9 / 5 + 32. Then write a function test_celsius_to_fahrenheit() that uses assert to check at least three values: 0 degrees Celsius is 32 degrees Fahrenheit, 100 is 212, and -40 is -40. Call the test function and print All tests passed at the end. (This test function would also work unchanged with pytest.)

Solution

def celsius_to_fahrenheit(celsius):
    return celsius * 9 / 5 + 32

def test_celsius_to_fahrenheit():
    assert celsius_to_fahrenheit(0) == 32
    assert celsius_to_fahrenheit(100) == 212
    assert celsius_to_fahrenheit(-40) == -40

test_celsius_to_fahrenheit()
print('All tests passed')
# Output:
# All tests passed

Exercise 3: Clean up a function

The function below works, but it is hard to read. Rewrite it following the advice in this chapter: give the function and its variables descriptive names, add a docstring and type hints, and make sure it doesn’t shadow any built-in names. The function should return the total price of the items whose price is above a given limit.

def f(list, x):
    sum = 0
    for i in list:
        if i > x:
            sum = sum + i
    return sum

print(f([5.0, 20.0, 12.5, 3.0], 10))  # Output: 32.5

Solution

def total_above(prices: list[float], limit: float) -> float:
    """Return the sum of the prices that are greater than limit."""
    total = 0.0
    for price in prices:
        if price > limit:
            total += price
    return total

print(total_above([5.0, 20.0, 12.5, 3.0], 10))
# Output:
# 32.5

Chapter 9: Conclusion

You’ve reached the end of the book. This chapter recaps what each chapter covered, suggests projects to practice with, and points you to good places to keep learning.

Recap of key concepts learned

Introduction: where Python came from, what makes it popular, how to install it, and how to write and run your first program.
Chapter 1: Python Basics: Python’s basic syntax, the core data types (numbers, strings, booleans and None), converting between types, printing with f-strings, and arithmetic, comparison, logical and bitwise operators.
Chapter 2: Control Flow: making decisions with if, elif, else and match, repeating work with for and while loops, and organizing code into functions with parameters, return values and scope.
Chapter 3: Working with Data Structures: lists, tuples, dictionaries, sets and bytes, when to use each one, and comprehensions for building them concisely.
Chapter 4: Working with Data: sorting, filtering and aggregating data, reading and writing text and JSON files, and handling errors with try, except, else and finally.
Chapter 5: Object-Oriented Programming: classes and objects, inheritance and polymorphism, encapsulation, and modern conveniences such as properties and data classes.
Chapter 6: Advanced Python Features: generators and iterators, decorators, concurrency, modules and packages, dates and times, databases with sqlite3, and regular expressions.
Chapter 7: Applications of Python: how Python is used for data analysis, web development, automation and scientific computing, and the main libraries for each.
Chapter 8: Tips and Tricks: writing clean code that follows PEP 8, virtual environments and packages, testing, debugging, and the common mistakes to avoid.

Where to go from here

The fastest way to get better at Python is to build things you care about. Pick a small project, finish it, then make it a little bigger. A few ideas, roughly from easiest to hardest:

Expense tracker: read expenses from a CSV file, total them by category, and print a monthly summary.
Quiz game: store questions in a JSON file, ask them in a random order, and keep score.
File organizer: sort the files in a folder into subfolders by file type with pathlib.
Weather report: fetch a forecast from a public weather API with requests and print it neatly.
To-do app: save tasks in an SQLite database and add commands to add, complete and list them.
Web app: turn one of the projects above into a small website with Flask.

When you get stuck, read the error message carefully, search for it, and try the smallest piece of code that shows the problem.

Resources for further learning

Python documentation: the official documentation covers the whole language and standard library. Its tutorial is a good next step after this book.
Online tutorials and courses: Real Python has in-depth articles on almost every topic. Courses are also available on Codecademy, Coursera and edX.
Books: Python Crash Course by Eric Matthes (projects for beginners), Automate the Boring Stuff with Python by Al Sweigart (practical automation, free to read online), and Python Data Science Handbook by Jake VanderPlas (data analysis, free to read online).
Communities: ask questions and learn from others on the Python Discourse forum, the Python subreddit or the Python Discord server.
Other books on this site: Regular Expressions for All goes much deeper into the pattern matching introduced in Chapter 6, and Programming for All: SQL Edition covers the databases you met there.

Happy coding!

Keyboard shortcuts

Programming for All: Python Edition