Regular Expressions for All: Comprehensive Guide

This book is a thorough introduction to regular expressions, from the basic concepts through practical use cases to advanced techniques. It’s written for developers, data scientists, and anyone else who works with text processing and pattern matching.

Introduction
Meta Characters
Simple Use Cases
Advanced Use Cases
References and Further Reading

Trying the examples

Most patterns in chapters 2 to 4 have a Try it button under them. It opens a small tester with some sample text, where you can change the pattern or the text and see every match. The tester uses PCRE2, the engine behind PHP’s preg_ functions and regex101’s default flavor, so even the recursion, atomic groups and conditionals in chapter 4 work. Everything runs in your browser; the engine, about 105 KB, downloads the first time you open a tester.

To work on your own patterns, try the Regex Tester on the companion tools site. It highlights matches and capture groups as you type and previews replacements, using JavaScript’s regex engine, the one in browsers and Node.js. JavaScript has no recursion, atomic groups or conditionals, so use the Try it buttons for the chapter 4 patterns that need them.

Getting Started

To get started with the book, simply click on the links above to access each chapter. You can read the chapters directly on GitHub or clone the repository to your local machine to read offline.

Contributing

If you find any errors or have suggestions for improvement, feel free to open an issue or submit a pull request. Contributions are welcome and encouraged!

License

This book is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. You are free to share and adapt the content for non-commercial purposes, as long as you give appropriate credit and share your adaptations under the same license.

Acknowledgments

Special thanks to all contributors who have helped make this book possible.

Chapter 1: Introduction

What are regular expressions?

A regular expression (regex or regexp) is a sequence of characters that defines a search pattern. Regular expressions match and manipulate strings, which makes them powerful tools for searching, editing, and processing text. Many programming languages and tools support them, so they’re an essential skill for developers and data scientists.

History and evolution

The mathematician Stephen Cole Kleene introduced regular expressions in the 1950s to describe the algebraic properties of regular sets. Computer science later adopted them, and they’re now a basic part of text processing. Modern implementations have added a wide range of features and optimizations over the years.

Applications of regular expressions

Common uses include:

Text searching and matching: finding patterns in text, such as specific words or phrases.
Text editing and replacement: changing text based on patterns, such as replacing every instance of a word.
Input validation: checking that user input matches a format, such as an email address or a phone number.
Data extraction: pulling the information you need out of large text datasets, such as log files or web pages.
Syntax highlighting: highlighting code in text editors and IDEs according to the language’s rules.
Data transformation: converting data from one format to another with pattern-based rules.

Overview of regular expression syntax

A regular expression combines literal characters with special symbols. The basic pieces are:

Literal characters match themselves (a matches the character ‘a’).
Meta characters are special symbols that stand for classes of characters or control how matching works (. matches any single character except a newline).
Quantifiers say how many times an element should match (* matches zero or more occurrences).
Anchors mark positions in the text, such as the start or end of a line (^ matches the start of a line).
Character classes define a set of characters to match ([a-z] matches any lowercase letter).
Grouping and capturing let you treat several elements as one and keep the matched text for later processing ((abc) matches ‘abc’ and captures it).

The next chapters cover each of these in more detail.

Chapter 2: Meta Characters

Overview of meta characters

Meta characters are special symbols in a regular expression that stand for types of characters or control how matching works. You need to know them to write effective patterns, and this chapter introduces the main ones.

Character classes and sets

Example 1: Character class

A character class matches any single character within the brackets.

[abc]

This matches ‘a’, ‘b’, or ‘c’.

Example 2: Character range

A character range specifies a range of characters to match.

[a-z]

This matches any lowercase letter from ‘a’ to ‘z’.

Example 3: Negated character class

A negated character class matches any character not in the brackets.

[^abc]

This matches any character except ‘a’, ‘b’, or ‘c’.

Anchors and boundaries

Example 1: Start of string anchor

The caret (^) matches the start of a string.

^start

This matches ‘start’ only if it is at the beginning of a line.

Example 2: End of string anchor

The dollar sign ($) matches the end of a string.

end$

This matches ‘end’ only if it is at the end of a line.

Example 3: Word boundary

The word boundary (\b) matches the position between a word and a non-word character.

\bword\b

This matches ‘word’ as a whole word, not as part of another word like ‘wording’.

Quantifiers

Example 1: Zero or more

The asterisk (*) matches zero or more occurrences of the preceding element.

a*

This matches ‘’, ‘a’, ‘aa’, ‘aaa’, and so on.

Example 2: One or more

The plus sign (+) matches one or more occurrences of the preceding element.

a+

This matches ‘a’, ‘aa’, ‘aaa’, and so on, but not an empty string.

Example 3: Zero or one

The question mark (?) matches zero or one occurrence of the preceding element.

a?

This matches ‘’ or ‘a’.

Example 4: Specific number of occurrences

Braces ({n,m}) match at least n and at most m occurrences of the preceding element.

a{2,3}

This matches ‘aa’ or ‘aaa’.

Grouping and capturing

Example 1: Grouping

Parentheses (()) group elements together.

(abc)

This matches ‘abc’.

Example 2: Capturing groups

Parentheses also capture the matched text for later use.

(\d{3})-(\d{2})-(\d{4})

This matches a Social Security number format and captures each group of digits.

Escape sequences

Example 1: Escaping special characters

The backslash (\) escapes a special character so that it matches literally.

\.

This matches a literal dot.

Example 2: Common escape sequences

\d: Matches any digit, equivalent to [0-9].
\D: Matches any non-digit.
\w: Matches any word character (alphanumeric + underscore), equivalent to [a-zA-Z0-9_].
\W: Matches any non-word character.
\s: Matches any whitespace character (space, tab, newline).
\S: Matches any non-whitespace character.

Lookahead and lookbehind

Example 1: Positive lookahead

The positive lookahead ((?=...)) checks that the characters that follow match the given pattern, without consuming them.

a(?=b)

This matches ‘a’ only if it is followed by ‘b’.

Example 2: Negative lookahead

The negative lookahead ((?!...)) checks that the characters that follow do not match the given pattern.

a(?!b)

This matches ‘a’ only if it is not followed by ‘b’.

Example 3: Positive lookbehind

The positive lookbehind ((?<=...)) checks that the characters before the current position match the given pattern, without consuming them.

(?<=b)a

This matches ‘a’ only if it is preceded by ‘b’.

Example 4: Negative lookbehind

The negative lookbehind ((?<!...)) checks that the characters before the current position do not match the given pattern.

(?<!b)a

This matches ‘a’ only if it is not preceded by ‘b’.

Chapter 3: Simple Use Cases

Introduction

This chapter shows how to use regular expressions for everyday tasks. Each example is a simple solution to a common problem.

Basic pattern matching

Example 1: Finding a word

To find a specific word in a text, search for the word itself:

word

This pattern matches the exact word “word” in the text. It also matches those four letters inside longer words, such as the “word” in “wordy”. To match only the whole word, put the word boundary \b on each side: \bword\b. The section on finding words with a specific prefix, below, explains \b.

Example 2: Case-insensitive search

To perform a case-insensitive search, turn on the i flag. Writing (?i) at the start of the pattern does that:

(?i)word

This matches “Word”, “word”, “WORD”, etc. Most languages can also set the flag outside the pattern: /word/i in JavaScript, re.IGNORECASE in Python.

Searching for text

Example 1: Finding all digits

To find all digits in a text:

\d+

This pattern matches sequences of digits like “123”, “456”, etc.

Example 2: Finding words with specific prefix

To find words starting with “pre”:

\bpre\w*\b

This matches “prefix”, “prelude”, etc. The \b in the regex \bpre\w*\b is a word boundary anchor. It makes the pattern match only at the beginning and end of a word. Here’s how each part, including \b, works in this pattern:

\b: Asserts a word boundary at the start.
pre: Matches the literal string “pre”.
\w*: Matches zero or more word characters (letters, digits, or underscores).
\b: Asserts a word boundary at the end.

Because of \b, “pre” matches only at the beginning of a word and never inside one, so the pattern finds “prefix” but not “depression”.

Validating input

Example 1: Email address validation

To validate an email address:

^[\w\.-]+@[a-zA-Z\d\.-]+\.[a-zA-Z]{2,6}$

This pattern checks for a valid email format.

Here is the pattern, piece by piece:

^ : Asserts the position at the start of the string.
[\w\.-]+:

[ and ] : Define a character class.
\w : Matches any word character (equivalent to [a-zA-Z0-9_]).
\. : Matches a literal dot (.). The backslash is used to escape the dot, as a dot normally matches any character.
- : Matches a literal hyphen (-).
+ : Quantifier that matches one or more occurrences of the preceding element (in this case, any word character, dot, or hyphen).

Together, this part matches one or more word characters, dots, or hyphens: the local part of the email address (the part before the @).

@ : Matches the literal “@” character.
[a-zA-Z\d\.-]+ :

[ and ] : Define a character class.
a-zA-Z : Matches any uppercase or lowercase letter.
\d : Matches any digit (equivalent to [0-9]).
\.: Matches a literal dot (.).
- : Matches a literal hyphen (-).
+ : Quantifier that matches one or more occurrences of the preceding element (in this case, any letter, digit, dot, or hyphen).

Together, this part matches one or more letters, digits, dots, or hyphens: the domain part of the email address (the part after the @).

\. : Matches a literal dot (.), separating the domain name and the top-level domain (TLD).
[a-zA-Z]{2,6} :

[ and ] : Define a character class.
a-zA-Z : Matches any uppercase or lowercase letter.
{2,6} : Quantifier that matches between 2 and 6 occurrences of the preceding element (in this case, any letter).

Together, this part matches a top-level domain (TLD) of 2 to 6 letters, such as “com”, “org”, “net”, or “info”.

$ : Asserts the position at the end of the string.

Example 2: Phone number validation

To validate a US phone number:

^\(\d{3}\) \d{3}-\d{4}$

This matches phone numbers in the format “(123) 456-7890”.

Replacing text

Example 1: Replacing all spaces with underscores

To replace spaces with underscores, match each run of whitespace:

\s+

and replace every match with _. “my holiday photo.jpg” becomes “my_holiday_photo.jpg”.

Example 2: Removing punctuation

To remove all punctuation:

[^\w\s]+

Replacing each match with nothing removes all the punctuation marks.

Practical examples in everyday tasks

Example 1: Extracting URLs from text

To extract URLs from a text:

https?://[^\s]+

This pattern matches “http://” or “https://” followed by non-space characters.

Example 2: Finding dates in text

To find dates in the format “dd/mm/yyyy”:

\d{2}/\d{2}/\d{4}

This matches dates like “23/07/2024”.

Example 3: Identifying capitalized words

To find all capitalized words:

\b[A-Z][a-z]*\b

This matches words starting with a capital letter.

Chapter 4: Advanced Use Cases

This chapter covers advanced uses of regular expressions: complex patterns, recursion, conditional logic, and optimization techniques. The examples tackle harder text processing tasks than the ones in Chapter 3.

Some of these patterns use features that not every regex engine has. Recursion with (?R) works in PCRE (the engine behind PHP’s preg_ functions) and Perl, and in Python only through the third-party regex module. Atomic groups, (?>...), work in PCRE, Perl, Java, .NET, and Python 3.11 or newer. Conditionals such as (?(?=cat)cat|dog) work in PCRE, Perl, and .NET. JavaScript and Go’s standard regexp package support none of the three. Check your language’s documentation before you rely on them.

Complex pattern matching

Example 1: Matching balanced parentheses

Regex:

\(([^()]*|(?R))*\)

Explanation:

$ and $: Match a literal opening ( and closing ) parenthesis.
[^()]*: Matches any sequence of characters that are not parentheses. This part handles content within a single level of parentheses.
|: Acts as an OR operator, so the pattern after it is tried as an alternative.
(?R): This is a recursive pattern that allows the regex to match nested parentheses by calling itself.
*: Applies to the whole group, meaning zero or more occurrences of either content within parentheses or more nested parentheses.

Usage: Use it to check that parentheses are balanced and properly nested, for example in mathematical expressions or programming languages.

Example 2: Matching HTML tags

Regex:

<([a-zA-Z]+)([^<]+)*(?:>(.*)<\/\1>|\/>)

Explanation:

< and >: Match the opening and closing angle brackets of an HTML tag.
([a-zA-Z]+): Captures the tag name, consisting of one or more letters.
([^<]+)*: Matches any content within the tag that does not include a < character, capturing attributes.
(?: ... ): A non-capturing group that matches either a self-closing tag or a tag with content.
>(.*)<\/\1>: Matches the tag content, using .* to capture anything between the tags and \/\1 to ensure that the closing tag matches the opening tag captured in \1.
|\/>: Matches self-closing tags like <img />.

Usage: It extracts and validates HTML tags, which helps when you parse or sanitize HTML code.

Example 3: Matching delimited strings

Regex:

"(?:[^"\\]|\\.)*"

Explanation:

": Matches the opening and closing double quotation marks.
(?: ... ): A non-capturing group that matches the content within the quotes.
[^"\\]: Matches any character that is not a double quote " or a backslash \.
|\\.: Matches any escaped character within the string, ensuring that sequences like \" or \\ are handled correctly.
*: Matches zero or more occurrences of the previous group, covering the entire content within the quotes.

Usage: It matches strings in double quotes, including any escaped characters inside them, as you find in JSON or programming languages.

Nested and recursive patterns

Example 1: Matching nested structures

Regex:

\((?>[^()]+|(?R))*\)

Explanation:

$ and $: Match the literal parentheses.
(?> ... ): An atomic group that prevents backtracking, ensuring that once a match is made, the regex engine does not reconsider it.
[^()]+: Matches one or more characters that are not parentheses.
|(?R): Allows recursion, where (?R) refers to the entire regex pattern, enabling the matching of nested parentheses.
*: Matches zero or more occurrences, handling multiple nested structures.

Usage: Use it to validate and extract data from nested structures such as mathematical expressions, nested lists, or code with several levels of parentheses.

Example 2: Matching nested lists

Regex:

\[(?>[^\[\]]+|(?R))*\]

Explanation:

\[ and \]: Match the opening and closing square brackets of a list.
(?>[^\[\]]+): Matches one or more characters that are not square brackets, preventing backtracking within the group.
|(?R): Recursively matches nested lists by calling the entire pattern again.
*: Matches zero or more occurrences of the previous groups, handling multiple nested lists.

Usage: It handles nested lists, such as JSON arrays or other hierarchical data formats.

Conditional expressions

Example 1: Conditional matching

Regex:

(?(?=regex)then|else)

Explanation:

(?=regex): A lookahead assertion that checks if the specified pattern matches at the current position.
then: The pattern to match if the condition is true.
else: The pattern to match if the condition is false.

Example:

(?(?=cat)cat|dog)

(?=cat): Checks if “cat” is present.
cat: Matches “cat” if the lookahead is true.
dog: Matches “dog” if the lookahead is false.

Usage: A conditional helps when what you want to match depends on the context.

Example 2: Matching optional prefixes

Regex:

\.(?(?=\d)\d+|\w+)

Explanation:

\.: Matches a literal dot.
(?(?=\d)\d+|\w+): Conditional that checks if a digit follows the dot.
- (?=\d)\d+: If a digit is present, match one or more digits.
- \w+: If a digit is not present, match one or more word characters.

Usage: It parses strings with optional prefixes. For example, it can tell file extensions like .123 and .txt apart by the first character after the dot.

Performance optimization

Example 1: Avoiding catastrophic backtracking

Regex:

(?>\d+|\w+)

Explanation:

(?> ... ): An atomic group that prevents backtracking, ensuring that once a match is made, it is not reconsidered.
\d+: Matches one or more digits.
\w+: Matches one or more word characters.

Usage: In complex patterns, backtracking can slow matching down a lot, so avoiding it makes a big difference to performance.

Example 2: Using non-capturing groups

Regex:

(?:\d+|\w+)

Explanation:

(?: ... ): A non-capturing group that groups elements without storing the matched text.
\d+: Matches one or more digits.
\w+: Matches one or more word characters.

Usage: Use a non-capturing group when you need to group part of a regex, for logic or for a quantifier, but don’t need the matched text. That can make the pattern both faster and easier to read.

Example 3: Using anchors for efficiency

Regex:

^\d+$

Explanation:

^: Asserts the position at the start of the string.
\d+: Matches one or more digits.
$: Asserts the position at the end of the string.

Usage: Anchors like ^ and $ limit where the regex engine looks for a match, so it spends its time only on the parts of the text that matter.

Real-world examples and case studies

Example 1: Parsing log files

Regex:

\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b

Explanation:

\b: Word boundary ensures the IP address is matched as a whole.
\d{1,3}: Matches one to three digits, representing an octet of an IP address.
\.: Matches a literal dot, separating octets.
\d{1,3}: Repeated for each octet of the IP address.

Usage: Log parsers often use it to pull IP addresses out of server logs for later analysis of network traffic or security events.

Example 2: Extracting data from CSV files

Regex:

([^,]+),([^,]+),([^,]+)

Explanation:

([^,]+): Captures one or more characters that are not a comma, representing a CSV field.
,: Matches the literal comma separating fields.
([^,]+): Repeated for subsequent fields.

Usage: It extracts the individual fields from CSV files, which makes tabular data easy to parse and process.

Example 3: Validating complex passwords

Regex:

^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[\W_]).{8,}$

Explanation:

^: Asserts the position at the start of the string.
(?=.*[a-z]): Positive lookahead ensuring at least one lowercase letter.
(?=.*[A-Z]): Positive lookahead ensuring at least one uppercase letter.
(?=.*\d): Positive lookahead ensuring at least one digit.
(?=.*[\W_]): Positive lookahead ensuring at least one special character or underscore.
.{8,}: Ensures the password is at least 8 characters long.
$: Asserts the position at the end of the string.

Usage: Use it to enforce a strong password policy. Every password has to meet the complexity requirements, which makes passwords more secure.

Example 4: Web scraping

Regex:

href="(https?://[^"]+)"

Explanation:

href=": Matches the href attribute in an HTML tag.
https?://: Matches either “http://” or “https://”.
[^"]+: Matches any sequence of characters that are not double quotes, representing the URL.
": Matches the closing double quote of the href attribute.

Usage: Web scrapers often use it to pull URLs out of HTML anchor tags when they collect data from websites automatically.

Advanced text processing

Example 1: Text normalization

Regex:

\s+

Explanation:

\s+: Matches one or more whitespace characters (spaces, tabs, newlines).

Usage: Replacing each match with a single space normalizes the text, so later processing sees consistent spacing.

Example 2: Extracting keywords

Regex:

\b(?:keyword1|keyword2|keyword3)\b

Explanation:

\b: Word boundary ensures that the keywords are matched as whole words.
(?: ... ): Non-capturing group that groups the keywords without capturing them.
keyword1|keyword2|keyword3: Matches any of the specified keywords.

Usage: It picks specific keywords out of a text, which is handy in search engines, tagging systems, or text analysis.

Example 3: Advanced search and replace

Regex:

(?<=\bword1\b).+?(?=\bword2\b)

Explanation:

(?<=\bword1\b): Positive lookbehind that ensures the text is preceded by “word1”.
.+?: Matches any text between “word1” and “word2”, as few characters as possible due to the lazy quantifier ?.
(?=\bword2\b): Positive lookahead that ensures the text is followed by “word2”.

Usage: Use it for targeted search and replace, when you want to change only the text between two specific words.

Chapter 5: References and Further Reading

Recommended books

“Mastering Regular Expressions” by Jeffrey E. F. Friedl
Covers both basic and advanced topics, and is widely regarded as the definitive reference for anyone serious about learning regex.
“Regular Expressions Cookbook” by Jan Goyvaerts and Steven Levithan
Practical solutions and examples for a range of regex problems. It suits beginners as well as experienced users who want to improve their regex skills.
“Introducing Regular Expressions” by Michael Fitzgerald
A beginner-friendly book that explains the concepts and uses of regular expressions clearly.

Online resources

Regex101
An online regex tester and debugger that supports PCRE (PHP, JavaScript, Python), ECMAScript (JavaScript), and Python flavors. It’s a good place to learn and experiment with regular expressions.
Regular-Expressions.info
A detailed site for learning regular expressions. It goes from the basics to advanced topics and compares regex features across programming languages.
Regexr
An interactive tool for learning, testing, and debugging regular expressions. It’s easy to use and shows visually how a pattern matches.
Regex Crossword
Puzzles that turn regex practice into a game. They’re fun, and the harder ones will stretch your skills.

Articles and tutorials

“Understanding Regular Expressions” by Peter Norvig
Explains how regular expressions work under the hood, including their performance characteristics and common pitfalls.
“Regular Expressions: Now You Have Two Problems” by Jeff Atwood
A humorous blog post about the complexities of regular expressions and when and why to use them.
“Regular Expression Matching Can Be Simple And Fast” by Russ Cox
Shows why backtracking regex engines, like the ones in Perl, Python, and Java, can take exponentially long on some patterns, and how engines based on finite automata avoid the problem.

Community and forums

Stack Overflow
A popular forum where you can ask questions about regular expressions and find answers. The regex tag already has solutions to many common regex problems.
Reddit - r/regex
A community for regex enthusiasts, with discussions, tips, and challenges.
GitHub Repositories
Search GitHub for open-source projects related to regular expressions to see how regex is used in real applications.

Keyboard shortcuts

Regular Expressions for All: Comprehensive Guide