Chapter 4: Working with Data

Once your data is in lists, tuples, and dictionaries, you need to work with it: put it in order, pick out the parts you want, save it to files, and deal with the errors that come up along the way. This chapter covers those everyday tasks.

In this chapter you will learn:

• How to sort, filter, and summarize lists and tuples
• How to read, write, and append to text files with open() and pathlib
• How to save and load data as JSON
• How to handle errors with try, except, else, and finally
• How to raise exceptions, including your own exception types

Manipulating data

Sorting data

You can use the sorted() function to sort a list or tuple in ascending order:

# Define a list of integers
my_list = [5, 2, 7, 1, 3]

# Sort the list in ascending order
sorted_list = sorted(my_list)
print(sorted_list)  # Output: [1, 2, 3, 5, 7]
print(my_list)  # Output: [5, 2, 7, 1, 3]

# Define a tuple of strings
my_tuple = ('c', 'a', 'b')

# Sort the tuple in ascending order
sorted_tuple = sorted(my_tuple)
print(sorted_tuple)  # Output: ['a', 'b', 'c']

sorted() returns a new list and leaves the original unchanged. It always returns a list, even when you give it a tuple.

You can also use the sort() method to sort a list in place, which changes the list itself:

# Define a list of integers
my_list = [5, 2, 7, 1, 3]

# Sort the list in ascending order
my_list.sort()
print(my_list)  # Output: [1, 2, 3, 5, 7]

sort() returns None, so don’t write my_list = my_list.sort(), or you’ll lose your list. Tuples can’t be changed, so they have no sort() method. Use sorted() for them.

Both sorted() and sort() accept two optional arguments. reverse=True sorts in descending order. key= takes a function that Python calls on each item, and the items are sorted by the results:

my_list = [5, 2, 7, 1, 3]
print(sorted(my_list, reverse=True))  # Output: [7, 5, 3, 2, 1]

words = ["banana", "fig", "apple", "kiwi"]
print(sorted(words))  # Output: ['apple', 'banana', 'fig', 'kiwi']
print(sorted(words, key=len))  # Output: ['fig', 'kiwi', 'apple', 'banana']

With key=len, the words are sorted by their length. Notice that you pass len without parentheses: you’re handing sorted() the function itself so it can call it on each word.

Filtering data

You can use a list comprehension with a condition to keep only the items you want:

# Define a list of integers
my_list = [5, 2, 7, 1, 8, 4]

# Filter the list to get only even numbers
even_numbers = [x for x in my_list if x % 2 == 0]
print(even_numbers)  # Output: [2, 8, 4]

To filter a tuple and get a tuple back, pass the same kind of expression to tuple():

# Define a tuple of strings
animals = ('cat', 'elephant', 'dog', 'giraffe')

# Filter the tuple to get only names longer than 3 letters
long_names = tuple(name for name in animals if len(name) > 3)
print(long_names)  # Output: ('elephant', 'giraffe')

Aggregating data

You can use the sum() function to add up the items in a list or tuple, and min(), max(), and len() to get the smallest item, the largest item, and the number of items:

# Define a list of integers
my_list = [5, 2, 7, 1, 3]

print(sum(my_list))  # Output: 18
print(min(my_list))  # Output: 1
print(max(my_list))  # Output: 7
print(len(my_list))  # Output: 5

# Combine them to get the average
print(sum(my_list) / len(my_list))  # Output: 3.6

# The same functions work on tuples
my_tuple = (5, 2, 7, 1, 3)
print(sum(my_tuple))  # Output: 18

Like sorted(), min() and max() accept a key= argument. For example, max(words, key=len) returns the longest word.

Reading and writing files

You can read and write files using the built-in open() function and the file object it returns. Each example below writes the file it needs first, so you can run it as is.

Writing to a file

To write to a file, open it in write mode:

# Open the file in write mode
with open('my_file.txt', 'w', encoding='utf-8') as f:
    # Write some text to the file
    f.write("Hello, world!\n")
    f.write("This is the second line.\n")

Here open() gets the file name and the mode 'w' (for write). It creates my_file.txt in the folder you run the program from, or empties the file if it already exists, and returns a file object, which is stored in the f variable. The file object’s write() method then writes each string to the file. write() doesn’t add a line break for you, so each string ends with \n.

Always pass encoding='utf-8' when you open a text file. Without it, Python uses your operating system’s default encoding, which differs between computers and can garble characters such as é or ü.

The with statement opens the file and automatically closes it when the indented block finishes, even if an error happens inside it. That is why it’s the recommended way to work with files: the file is always closed and released when you’re done with it.

Reading a file

To read a file, open it in read mode:

with open('my_file.txt', 'w', encoding='utf-8') as f:
    f.write("Hello, world!\nThis is the second line.\n")

# Open the file in read mode
with open('my_file.txt', 'r', encoding='utf-8') as f:
    # Read the contents of the file into a variable
    file_contents = f.read()

# Print the contents of the file
print(file_contents)
# Output:
# Hello, world!
# This is the second line.

This time open() gets the mode 'r' (for read). The file object’s read() method reads the whole file into the file_contents variable as one string, and print() displays it. Reading is the default mode, so open('my_file.txt', encoding='utf-8') does the same thing.

Reading line by line

You can loop over a file object with for to get one line at a time:

with open('shopping.txt', 'w', encoding='utf-8') as f:
    f.write("eggs\nmilk\nbread\n")

with open('shopping.txt', encoding='utf-8') as f:
    for line in f:
        print(line.strip())

# Output:
# eggs
# milk
# bread

Each line keeps its \n at the end. strip() removes it, so print() doesn’t add an extra blank line after each item.

The readlines() method reads all the lines at once and returns them as a list:

with open('shopping.txt', 'w', encoding='utf-8') as f:
    f.write("eggs\nmilk\nbread\n")

with open('shopping.txt', encoding='utf-8') as f:
    lines = f.readlines()

print(lines)  # Output: ['eggs\n', 'milk\n', 'bread\n']
print(len(lines))  # Output: 3

readlines() is handy when you need all the lines together, for example to count or sort them. Looping over the file reads one line at a time instead, so it uses less memory and works even for very large files.

Appending to a file

Mode 'w' replaces everything in the file. To add to the end of a file and keep what is already there, use mode 'a' (for append):

with open('log.txt', 'w', encoding='utf-8') as f:
    f.write("first entry\n")

with open('log.txt', 'a', encoding='utf-8') as f:
    f.write("second entry\n")

with open('log.txt', encoding='utf-8') as f:
    print(f.read())

# Output:
# first entry
# second entry

Different file modes in Python

Python has several file modes, depending on whether you want to read, write, or both. Modes without b open the file in text mode, which reads and writes strings. Modes with b open it in binary mode, which reads and writes bytes (used for images and other non-text files) and doesn’t take an encoding.

Working with paths using pathlib

The pathlib module from Python’s standard library gives you Path objects, which represent a file or folder path. For small files, its write_text() and read_text() methods open, write or read, and close the file in one step:

from pathlib import Path

path = Path('notes.txt')
print(path.exists())  # Output: False

path.write_text("Buy milk\nCall Sam\n", encoding='utf-8')
print(path.exists())  # Output: True

text = path.read_text(encoding='utf-8')
print(text.splitlines())  # Output: ['Buy milk', 'Call Sam']

exists() tells you whether the file is there, which is useful before you try to read it. splitlines() splits a string into a list of lines without the \n characters. You can also build paths with /, as in Path('data') / 'notes.txt', and Python uses the right separator for your operating system.

Working with JSON

JSON (JavaScript Object Notation) is a text format for structured data. Many web services, programs, and configuration files use it. Python’s built-in json module converts dictionaries, lists, strings, numbers, booleans, and None to JSON text and back.

Use json.dump() to write data to a file and json.load() to read it back:

import json

student = {"name": "Asha", "age": 21, "courses": ["Math", "Physics"], "graduated": False}

# Save the dictionary to a file
with open('student.json', 'w', encoding='utf-8') as f:
    json.dump(student, f)

# Load it back
with open('student.json', encoding='utf-8') as f:
    loaded = json.load(f)

print(loaded["name"])  # Output: Asha
print(loaded["courses"])  # Output: ['Math', 'Physics']
print(loaded == student)  # Output: True

json.dumps() (the s stands for “string”) returns the JSON text as a string instead of writing it to a file. Pass indent=4 to make it easier to read. json.dump() accepts indent too:

import json

student = {"name": "Asha", "age": 21, "graduated": False}
print(json.dumps(student, indent=4))
# Output:
# {
#     "name": "Asha",
#     "age": 21,
#     "graduated": false
# }

Notice that JSON writes false in lowercase, uses double quotes for strings, and writes None as null. To turn a JSON string back into Python data, use json.loads().

Exception handling

An exception is an error that happens while a program is running, such as dividing by zero or converting "abc" to a number. If you don’t handle it, the program stops and prints an error message. Exception handling lets you catch the error and decide what to do instead, so your program can deal with unexpected input and keep going.

try and except

To handle an exception, put the code that might fail in a try block and the code that deals with the error in an except block:

text = "ten"  # this value might come from input()

try:
    number = int(text)
    print(f"Twice your number is {number * 2}")
except ValueError:
    print("Please enter a whole number, like 10.")

# Output:
# Please enter a whole number, like 10.

Python runs the code in the try block. int("ten") raises a ValueError, so Python skips the rest of the try block and runs the matching except block. If no exception is raised, the except block is skipped.

This is the most common error with user input: input() always returns a string, so int(input()) raises a ValueError whenever someone types something that isn’t a whole number. The examples here use variables so you can see the output, but the values could just as well come from input().

Handling different exceptions

A try block can be followed by several except clauses, one for each kind of error you expect:

def divide(a_text, b_text):
    try:
        result = int(a_text) / int(b_text)
        print(f"Result: {result}")
    except ValueError:
        print("Both values must be whole numbers.")
    except ZeroDivisionError:
        print("Cannot divide by zero.")

divide("10", "4")  # Output: Result: 2.5
divide("10", "zero")  # Output: Both values must be whole numbers.
divide("10", "0")  # Output: Cannot divide by zero.

Python checks the except clauses from top to bottom and runs the first one that matches. To handle several exceptions the same way, list them in one clause as a tuple: except (ValueError, ZeroDivisionError):.

Avoid a bare except: with no exception name. It catches every exception, even a typo in your own code (NameError) or pressing Ctrl+C (KeyboardInterrupt), which hides bugs and makes the program hard to stop. Catch the specific exceptions you expect.

Getting the error message

Add as and a name to an except clause to get the exception object. Printing it shows Python’s error message:

try:
    with open('missing.txt', encoding='utf-8') as f:
        print(f.read())
except FileNotFoundError as error:
    print(f"Could not open the file: {error}")

# Output:
# Could not open the file: [Errno 2] No such file or directory: 'missing.txt'

The [Errno 2] part is the error code your operating system reported.

else and finally

You can add an else block, which runs only if the try block raised no exception, and a finally block, which runs no matter what:

def check_number(text):
    try:
        number = int(text)
    except ValueError:
        print(f"{text} is not a number")
    else:
        print(f"Got the number {number}")
    finally:
        print("Done checking")

check_number("7")
check_number("seven")

# Output:
# Got the number 7
# Done checking
# seven is not a number
# Done checking

Putting the success code in else instead of try makes it clear which line you expect might fail. finally is the place for cleanup that must always happen, such as closing a connection or printing a final message.

Raising exceptions

You can raise an exception yourself with the raise statement when your code receives a value it can’t work with. The code that called it can then decide how to handle the problem:

def set_age(age):
    if age < 0:
        raise ValueError("Age cannot be negative")
    print(f"Age set to {age}")

set_age(30)  # Output: Age set to 30

try:
    set_age(-5)
except ValueError as error:
    print(error)  # Output: Age cannot be negative

Custom exceptions

You can define your own exception type by creating a class that inherits from Exception. You’ll learn about classes and inheritance in Chapter 5. For now, these two lines are all you need:

class InsufficientFundsError(Exception):
    """Raised when a withdrawal is larger than the balance."""

def withdraw(balance, amount):
    if amount > balance:
        raise InsufficientFundsError(f"Cannot withdraw {amount}, the balance is only {balance}")
    return balance - amount

print(withdraw(100, 30))  # Output: 70

try:
    withdraw(100, 500)
except InsufficientFundsError as error:
    print(error)  # Output: Cannot withdraw 500, the balance is only 100

A custom exception’s name says exactly what went wrong, and callers can catch it without also catching unrelated errors.

Built-in exceptions in Python

Python has many built-in exceptions, which it raises when errors occur while a program runs:

Key takeaways

• sorted() returns a new sorted list, while list.sort() sorts a list in place. Both accept reverse=True and key=.
• List comprehensions filter data, and sum(), min(), max(), and len() summarize it.
• Open text files with with open(..., encoding='utf-8') so they close automatically and read the same way on every computer. Mode 'w' overwrites a file, and 'a' adds to the end.
• pathlib.Path makes simple file tasks short, and the json module saves and loads dictionaries and lists.
• Catch specific exceptions with try and except. Use else for code that should run only on success and finally for code that must always run.
• Use raise to signal a problem, and create your own exception types by subclassing Exception.

Practice

Exercise 1: Test scores

Given the list scores = [72, 45, 90, 61, 38, 85], print the scores sorted from highest to lowest, a list of the passing scores (60 or higher), and the average score rounded to one decimal place with round().

scores = [72, 45, 90, 61, 38, 85]

print(sorted(scores, reverse=True))
passing = [score for score in scores if score >= 60]
print(passing)
average = sum(scores) / len(scores)
print(round(average, 1))
# Output:
# [90, 85, 72, 61, 45, 38]
# [72, 90, 61, 85]
# 65.2

Exercise 2: A to-do list file

Write three tasks to a file named todo.txt, one per line. Then append a fourth task without erasing the others. Finally, read the file and print each task with a number in front, starting at 1.

with open('todo.txt', 'w', encoding='utf-8') as f:
    f.write("Buy groceries\n")
    f.write("Call the bank\n")
    f.write("Water the plants\n")

with open('todo.txt', 'a', encoding='utf-8') as f:
    f.write("Read a chapter\n")

with open('todo.txt', encoding='utf-8') as f:
    for number, line in enumerate(f, start=1):
        print(f"{number}. {line.strip()}")
# Output:
# 1. Buy groceries
# 2. Call the bank
# 3. Water the plants
# 4. Read a chapter

Exercise 3: Safe number conversion

Write a function parse_numbers(items) that takes a list of strings and returns a list of the whole numbers they contain. If an item can’t be converted with int(), print a message saying it was skipped and move on. Test it with ["10", "abc", "7", "3.5", "-2"].

def parse_numbers(items):
    """Return the items that are whole numbers, converted to int."""
    numbers = []
    for item in items:
        try:
            numbers.append(int(item))
        except ValueError:
            print(f"Skipping {item}: not a whole number")
    return numbers

print(parse_numbers(["10", "abc", "7", "3.5", "-2"]))
# Output:
# Skipping abc: not a whole number
# Skipping 3.5: not a whole number
# [10, 7, -2]