Chapter 6: Advanced Python Features

This chapter covers features you will reach for as your programs grow: producing values on demand, adding behavior to functions, doing several things at once, splitting code into files, and using some of the most useful parts of Python’s standard library.

In this chapter you will learn:

• How generators and iterators produce values one at a time
• How decorators add behavior to a function without changing its code
• The basics of running tasks concurrently with threads, processes, and asyncio
• How to organize code into modules and packages
• How to work with dates, times, and time spans
• How to store data in a SQLite database and search text with regular expressions

Generators and iterators

A generator is a function that produces a sequence of values on demand, one at a time, instead of building the whole sequence and returning it at once. You write a generator like a normal function, but use the yield keyword to hand out each value.

This generator function produces a sequence of numbers:

def my_range(n):
    i = 0
    while i < n:
        yield i
        i += 1

# Use the generator
for i in my_range(5):
    print(i)
# Output:
# 0
# 1
# 2
# 3
# 4

In this example, my_range produces the numbers from 0 to n - 1. Calling my_range(5) does not run the function body yet; it returns a generator object. The for loop then asks the generator for one value at a time.

Each time the generator reaches yield, it hands out a value and pauses, keeping all its local variables. When the next value is requested, it carries on from exactly where it stopped. You can see the pausing if you add some print calls:

def countdown(n):
    while n > 0:
        print(f'About to yield {n}')
        yield n
        n -= 1

for number in countdown(3):
    print(f'Got {number}')
# Output:
# About to yield 3
# Got 3
# About to yield 2
# Got 2
# About to yield 1
# Got 1

Because a generator only produces values when asked, it uses very little memory, even for a very long sequence.

Generator expressions

For simple cases you can write a generator in one line with a generator expression. It looks like a list comprehension, but with parentheses instead of square brackets:

squares = (x * x for x in range(5))
print(list(squares))  # Output: [0, 1, 4, 9, 16]

# When a generator expression is the only argument, you can drop the extra parentheses
print(sum(x * x for x in range(5)))  # Output: 30

sum(x * x for x in range(5)) adds up the squares without ever building a list of them.

Iterators

Two related words come up whenever you loop in Python:

• An iterable is any object you can loop over with for: lists, tuples, strings, dictionaries, files, and generators are all iterables.
• An iterator is the object that does the actual stepping. It remembers its position and gives you the next value each time you call next() on it.

A list is an iterable, but it is not an iterator itself. The built-in iter() function gets an iterator from an iterable:

my_list = [1, 2, 3]

# Get an iterator from the list
it = iter(my_list)

print(next(it))  # Output: 1
print(next(it))  # Output: 2
print(next(it))  # Output: 3

# The iterator is now exhausted
try:
    next(it)
except StopIteration:
    print('No more values')  # Output: No more values

# Give next() a default to get that instead of an exception
print(next(it, 'done'))  # Output: done

When an iterator has no more values, next() raises a StopIteration exception. A for loop does all of this for you: it calls iter() on the object, calls next() until StopIteration is raised, and then stops without an error.

An iterator can only be used once. A list hands out a fresh iterator every time you loop over it, so you can loop over a list as often as you like. A generator is its own iterator, so once you have consumed it, it is empty:

numbers = [1, 2, 3]
print(list(numbers))  # Output: [1, 2, 3]
print(list(numbers))  # Output: [1, 2, 3]

squares = (n * n for n in numbers)
print(list(squares))  # Output: [1, 4, 9]
print(list(squares))  # Output: []

Decorators

In Python, functions are objects: you can pass a function to another function, and a function can create and return a new function. A decorator uses this to add behavior to a function without changing its code. It takes a function and returns a new function that usually calls the original one with some extra steps before or after.

The decorator below prints a message before and after the function it decorates:

import functools

def my_decorator(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print('Before calling the function')
        result = func(*args, **kwargs)
        print('After calling the function')
        return result
    return wrapper

@my_decorator
def add(a, b):
    print('Inside the function')
    return a + b

total = add(2, 3)
print(total)
# Output:
# Before calling the function
# Inside the function
# After calling the function
# 5

Here is what each part does:

• my_decorator takes a function, func, and returns a new function, wrapper.
• Writing @my_decorator above def add is a shortcut for add = my_decorator(add). From then on, calling add really calls wrapper.
• wrapper(*args, **kwargs) accepts any positional and keyword arguments and passes them on to func, so the same decorator works for functions with any parameters.
• wrapper returns func’s result, so the decorated function still returns its value. Without return result, add(2, 3) would return None.
• @functools.wraps(func) copies the original function’s name and docstring onto wrapper, so add.__name__ is still 'add' instead of 'wrapper'. This makes error messages and debugging much clearer.

A common practical use is logging. This decorator prints each call with its arguments and the value it returned:

import functools

def log_calls(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        arguments = [repr(a) for a in args] + [f'{k}={v!r}' for k, v in kwargs.items()]
        print(f'Calling {func.__name__}({", ".join(arguments)})')
        result = func(*args, **kwargs)
        print(f'{func.__name__} returned {result!r}')
        return result
    return wrapper

@log_calls
def greet(name, greeting='Hello'):
    return f'{greeting}, {name}!'

greet('Ana')
greet('Ben', greeting='Hi')
# Output:
# Calling greet('Ana')
# greet returned 'Hello, Ana!'
# Calling greet('Ben', greeting='Hi')
# greet returned 'Hi, Ben!'

Decorators are often used to add logging, timing, caching, or permission checks to functions. You have already used two decorators from the standard library in Chapter 5: @property and @dataclass.

Concurrency

Concurrency means letting several tasks make progress during the same period of time, instead of finishing one completely before starting the next. Python has several ways to do this, including threading, multiprocessing, and asynchronous programming.

Threading

A thread is a separate flow of execution inside your program. All threads run in the same memory space, so they are lightweight and can share data. Python’s threading module lets you create and run threads.

import threading

def print_numbers():
    for i in range(5):
        print(i)

def print_letters():
    for letter in 'abcde':
        print(letter)

# Create the threads
thread1 = threading.Thread(target=print_numbers)
thread2 = threading.Thread(target=print_letters)

# Start the threads
thread1.start()
thread2.start()

# Wait for both threads to finish
thread1.join()
thread2.join()

start() begins running the target function in a new thread, and join() waits until that thread has finished. The two threads run concurrently, so the numbers and letters can come out interleaved, and the order can change from one run to the next.

Multiprocessing

Multiprocessing runs code in separate processes, each with its own memory space and its own Python interpreter. This approach is useful for CPU-bound tasks, such as heavy calculations, because the processes can run on different CPU cores at the same time. Python’s multiprocessing module creates and manages these processes.

import multiprocessing

def square_numbers():
    for i in range(5):
        print(i * i)

if __name__ == '__main__':
    # Create a process
    process = multiprocessing.Process(target=square_numbers)

    # Start the process and wait for it to finish
    process.start()
    process.join()
    print('Process finished')
# Output:
# 0
# 1
# 4
# 9
# 16
# Process finished

The if __name__ == '__main__': line is required on Windows and macOS, and on Linux since Python 3.14: there, a fresh Python interpreter imports your script to set up the new processes, and this check stops that import from starting new processes all over again. Older Python versions on Linux don’t need it, but it does no harm, so write it everywhere. (You will learn more about __name__ in Working with modules and packages.)

Asynchronous programming

Asynchronous programming lets one thread juggle many tasks that spend most of their time waiting, such as network requests or reading files. While one task waits, the others get to run. Python’s asyncio module provides the tools for this.

import asyncio

async def fetch_data():
    print('Start fetching')
    await asyncio.sleep(0.1)  # Simulate waiting for a network request
    print('Done fetching')

async def main():
    await asyncio.gather(
        fetch_data(),
        fetch_data(),
        fetch_data(),
    )

# Run the async function
asyncio.run(main())
# Output:
# Start fetching
# Start fetching
# Start fetching
# Done fetching
# Done fetching
# Done fetching

• async def defines a coroutine, a function that can pause while it waits.
• await asyncio.sleep(0.1) pauses this coroutine and lets the others run in the meantime.
• asyncio.gather runs several coroutines at once and waits for all of them to finish.
• asyncio.run starts everything and runs until main is done.

All three tasks start before any of them finishes, because their waits overlap. The whole program takes about 0.1 seconds, not 0.3. With waits of 2 seconds each, it would take about 2 seconds instead of 6.

The simplest option: concurrent.futures

If you just want to run the same function on many inputs at once, the concurrent.futures module is the easiest place to start. A ThreadPoolExecutor manages a group of threads for you:

import time
from concurrent.futures import ThreadPoolExecutor

def download(page):
    time.sleep(0.1)  # Pretend to wait for the network
    return f'Contents of {page}'

pages = ['home', 'about', 'contact']

with ThreadPoolExecutor(max_workers=3) as executor:
    results = list(executor.map(download, pages))

for result in results:
    print(result)
# Output:
# Contents of home
# Contents of about
# Contents of contact

executor.map works like the built-in map, but runs the calls in parallel threads. The three downloads wait at the same time, and the results come back in the same order as the inputs. For CPU-bound work, ProcessPoolExecutor from the same module works the same way but uses processes (and needs the if __name__ == '__main__': check).

Differences and use cases

• Threading works best for I/O-bound tasks, where the program spends its time waiting for files or the network. In the standard Python interpreter, the Global Interpreter Lock (GIL) lets only one thread run Python code at a time, so threads don’t speed up CPU-bound work.
• Multiprocessing is the best choice for CPU-bound tasks. Each process has its own interpreter and its own GIL, so processes can use several CPU cores at once. Starting processes and passing data between them costs more than using threads.
• Asynchronous programming is best when you need to handle many I/O operations at once, such as hundreds of network connections. Libraries such as aiohttp (for HTTP requests) and Quart (for web applications) build on asyncio.

In practice, threading (for example with ThreadPoolExecutor) or asyncio works well for web scraping, where you wait for HTTP responses. Data processing that involves heavy computation can use all of your CPU cores with multiprocessing. And if you are building a server that handles many connections at once, asyncio is an efficient choice.

Working with modules and packages

Modules

A module is a Python file that contains functions, variables, classes, and other code that you can use in other programs. Modules help you organize and reuse code. Python comes with a large collection of ready-made modules called the standard library; math, datetime, sqlite3, and re, which you meet in this chapter, are all part of it.

To use a module, import it with the import statement. For example, the math module contains mathematical functions:

import math

result = math.sqrt(16)
print(result)  # Output: 4.0

You can also import specific functions or variables from a module using the from keyword. Then you can use them without the module name in front:

from math import sqrt

result = sqrt(16)
print(result)  # Output: 4.0

You can use the as keyword to give what you import a different name:

from math import sqrt as square_root

result = square_root(16)
print(result)  # Output: 4.0

Any .py file you write is a module too. If you save a file called greetings.py with a hello function in it, another file in the same folder can use it:

# main.py
import greetings

print(greetings.hello('Ana'))

Running a module as a script: if __name__ == '__main__':

Every module has a built-in variable called __name__. When you run a file directly, for example with python greetings.py, Python sets __name__ to '__main__'. When another file imports it, __name__ is the module’s name, 'greetings'.

This lets a file contain code that runs only when you run the file itself, not when it is imported:

# greetings.py
def hello(name):
    return f'Hello, {name}!'

if __name__ == '__main__':
    # Runs only when you run this file directly, not when it is imported
    print(hello('world'))  # Output: Hello, world!

This is useful for quick tests or demos of a module, and, as you saw in the multiprocessing example, it is sometimes required.

Packages

A package is a folder of modules. Packages let you divide a larger program into smaller, reusable pieces. A package folder usually contains an __init__.py file, which may be empty:

my_project/
├── main.py
└── mypackage/
    ├── __init__.py
    └── mymodule.py

Suppose mypackage/mymodule.py contains this function:

# mypackage/mymodule.py
def some_function():
    return 'Hello from mymodule'

In main.py, you import a module from a package by writing the package name and the module name, separated by a dot. All three of these forms work:

# main.py
import mypackage.mymodule
print(mypackage.mymodule.some_function())

from mypackage import mymodule
print(mymodule.some_function())

from mypackage import mymodule as mm
print(mm.some_function())

The code in __init__.py runs the first time the package is imported. You can use it to make names from inside the package available directly:

# mypackage/__init__.py
from .mymodule import some_function

With that line in place, main.py can write from mypackage import some_function. The dot in .mymodule means “the mymodule module in this same package”.

Besides the standard library, there are hundreds of thousands of third-party packages that you install with pip. You will learn how to install them, and how to keep each project’s packages separate with a virtual environment, in Chapter 8.

Working with dates and times

The datetime module handles dates and times. It has classes for dates (date), times of day (time), a date with a time (datetime), and lengths of time (timedelta).

This gets the current date and time, so the output depends on when you run it:

from datetime import date, datetime

now = datetime.now()
print(now)           # For example: 2022-12-26 14:47:59.438123
print(date.today())  # For example: 2022-12-26

You can also create datetime objects for specific dates and times, and format them as text with the strftime() method:

from datetime import datetime

# Create a date and time
dt = datetime(2022, 12, 26, 14, 50, 0)

# Format the date and time
formatted = dt.strftime('%B %d, %Y %I:%M %p')
print(formatted)  # Output: December 26, 2022 02:50 PM

The codes that start with % stand for parts of the date. Some common ones:

To go the other way, from text to a datetime, use strptime() with the same codes:

from datetime import datetime

dt = datetime.strptime('2024-03-15 09:30', '%Y-%m-%d %H:%M')
print(dt)        # Output: 2024-03-15 09:30:00
print(dt.year)   # Output: 2024
print(dt.month)  # Output: 3

Date arithmetic with timedelta

A timedelta is a length of time. Add one to a date to move forward, or subtract two dates to find out how far apart they are:

from datetime import date, datetime, timedelta

start = date(2024, 1, 15)
due = start + timedelta(days=30)
print(due)  # Output: 2024-02-14

days_left = date(2025, 1, 1) - start
print(days_left)       # Output: 352 days, 0:00:00
print(days_left.days)  # Output: 352

shift_start = datetime(2024, 3, 15, 9, 30)
shift_end = datetime(2024, 3, 15, 17, 45)
worked = shift_end - shift_start
print(worked)                          # Output: 8:15:00
print(worked.total_seconds() / 3600)   # Output: 8.25

Time zones

The datetime objects above are naive: they don’t know which time zone they are in. That is fine for many programs, but if you need to compare times from different places, use aware datetimes, which carry a time zone. The zoneinfo module provides time zones by name:

from datetime import datetime
from zoneinfo import ZoneInfo

meeting = datetime(2024, 7, 1, 9, 0, tzinfo=ZoneInfo('America/New_York'))
print(meeting)  # Output: 2024-07-01 09:00:00-04:00

in_kathmandu = meeting.astimezone(ZoneInfo('Asia/Kathmandu'))
print(in_kathmandu)  # Output: 2024-07-01 18:45:00+05:45

astimezone() converts a time to another time zone. On Windows, you may need to install the tzdata package with pip before zoneinfo can find the time zones.

Working with databases

A database stores data in a structured way so that you can search and update it efficiently. Several libraries let you work with databases from Python. Some popular ones:

• sqlite3: Python’s built-in module for SQLite, a small database that keeps everything in a single file
• psycopg (and the older psycopg2): for PostgreSQL databases
• mysqlclient (imported as MySQLdb): for MySQL databases
• SQLAlchemy: a flexible library that works with many databases, including MySQL, PostgreSQL, and SQLite

All except sqlite3 are third-party packages that you install with pip.

The following example uses sqlite3 to create a table, insert data into it, and query the data:

import sqlite3

# ':memory:' creates a temporary database that lives in memory.
# Use a filename such as 'mydatabase.db' instead to save it to a file on disk.
conn = sqlite3.connect(':memory:')
cursor = conn.cursor()

# 'with conn' saves (commits) the changes if the block succeeds,
# and undoes (rolls back) all of them if an error occurs
with conn:
    cursor.execute('CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT, email TEXT)')
    cursor.execute('INSERT INTO users (name, email) VALUES (?, ?)', ('John', '[email protected]'))
    cursor.execute('INSERT INTO users (name, email) VALUES (?, ?)', ('Jane', '[email protected]'))

# Query all the rows
cursor.execute('SELECT id, name, email FROM users ORDER BY id')
for row in cursor.fetchall():
    print(row)

# Query with a value supplied separately
cursor.execute('SELECT email FROM users WHERE name = ?', ('Jane',))
print(cursor.fetchone())

# Close the connection
conn.close()
# Output:
# (1, 'John', '[email protected]')
# (2, 'Jane', '[email protected]')
# ('[email protected]',)

The program connects to a database, creates a cursor to run SQL statements, creates a users table if it doesn’t exist yet, inserts two rows, and reads them back. Each row comes back as a tuple. fetchall() returns all the matching rows as a list, and fetchone() returns just the next one. The with conn: block takes care of committing, but it does not close the connection, so the program calls conn.close() at the end.

Notice the ? placeholders. Instead of putting values into the SQL text yourself, you pass them separately as a tuple, and sqlite3 inserts them safely. (A tuple with one item needs a trailing comma: ('Jane',).) Never build SQL by pasting user input into the string, for example with an f-string. A user could type text that changes the meaning of your SQL statement, an attack called SQL injection. Placeholders prevent it.

Working with regular expressions

Regular expressions are patterns for matching text. You can use them to check whether a string contains a pattern, to pull out the parts you need, and to replace parts of a string.

Python’s re module works with regular expressions. This example searches for a pattern in a string:

import re

string = 'The quick brown fox jumps over the lazy dog.'
pattern = r'quick'

match = re.search(pattern, string)
if match:
    print('Match found:', match.group())
else:
    print('Match not found')
# Output: Match found: quick

The r before the pattern makes it a raw string, in which a backslash is just a backslash. Regular expressions use a lot of backslashes, so it is a good habit to always write patterns as raw strings.

re.search() looks for the first place where the pattern matches. If it finds one, it returns a Match object, and group() gives you the text that matched. If there is no match, it returns None.

You can use groups to extract several pieces of information at once:

import re

string = 'The quick brown fox jumps over the lazy dog.'
pattern = r'(quick) (brown) (fox)'

match = re.search(pattern, string)
if match:
    print('Match found:', match.groups())
else:
    print('Match not found')
# Output: Match found: ('quick', 'brown', 'fox')

This pattern contains three groups, each enclosed in parentheses. When there is a match, the groups() method returns a tuple with the text matched by each group.

Patterns become much more useful with special characters. Here are a few:

re.findall() returns a list of every match, and re.sub() replaces every match with new text:

import re

text = 'Call 555-1234 or 555-9876 today.'
pattern = r'\d{3}-\d{4}'

print(re.findall(pattern, text))  # Output: ['555-1234', '555-9876']
print(re.sub(pattern, '[hidden]', text))  # Output: Call [hidden] or [hidden] today.

Regular expressions are a big topic. To learn them properly, see Regular Expressions for All, the companion book on this site.

Key takeaways

• A generator uses yield to produce values one at a time, pausing in between. Iterables (such as lists) can be looped over again and again; iterators (including generators) are used up after one pass.
• A decorator wraps a function to add behavior. Write the wrapper with *args, **kwargs, return the result, and use functools.wraps.
• Use threads or asyncio for tasks that wait on I/O, and processes for CPU-heavy work. concurrent.futures is the simplest place to start.
• Any .py file is a module, and a folder of modules is a package. Put code that should run only when the file is run directly under if __name__ == '__main__':.
• The datetime module creates, formats (strftime), parses (strptime), and does arithmetic (timedelta) on dates and times.
• With sqlite3, use ? placeholders for values to avoid SQL injection. Use the re module’s search, findall, and sub to work with text patterns.

Practice

Exercise 1: Fibonacci generator

Write a generator function fibonacci(n) that yields the first n Fibonacci numbers, starting with 0 and 1 (each number after that is the sum of the two before it). Print the first 10 as a list.

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

print(list(fibonacci(10)))
# Output:
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Exercise 2: An uppercase decorator

Write a decorator called uppercase that converts the string returned by a function to uppercase. It should work with functions that take any arguments. Test it on a function greet(name) that returns 'Hello, ' + name.

import functools

def uppercase(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        return result.upper()
    return wrapper

@uppercase
def greet(name):
    return 'Hello, ' + name

print(greet('Ana'))
print(greet.__name__)
# Output:
# HELLO, ANA
# greet

Exercise 3: A small contacts database

Using sqlite3 with an in-memory database, create a contacts table with name and city columns. Insert at least four contacts, then use a query with a ? placeholder to print the names of everyone who lives in one particular city, in alphabetical order.

import sqlite3

conn = sqlite3.connect(':memory:')
cursor = conn.cursor()

contacts = [
    ('Asha', 'Kathmandu'),
    ('Ben', 'London'),
    ('Bikash', 'Kathmandu'),
    ('Carla', 'Lima'),
]

with conn:
    cursor.execute('CREATE TABLE IF NOT EXISTS contacts (name TEXT, city TEXT)')
    for contact in contacts:
        cursor.execute('INSERT INTO contacts (name, city) VALUES (?, ?)', contact)

city = 'Kathmandu'
cursor.execute('SELECT name FROM contacts WHERE city = ? ORDER BY name', (city,))
for row in cursor.fetchall():
    print(row[0])

conn.close()
# Output:
# Asha
# Bikash