Chapter 3 Lists

3.1 Splitting

So far we’ve spent a lot of time understanding strings, and there’s a natural way to get from strings to lists via the split() method:

$ source venv/bin/activate
(venv) $ python3
>>> "ant bat cat".split(" ")     # Split a string into a three-element list.
['ant', 'bat', 'cat']

We see from this result that split() returns a list of the strings that are separated from each other by a space in the original string.

Splitting on space is one of the most common operations, but we can split on nearly anything else as well (Listing 3.1).

>>> "ant,bat,cat".split(",")
['ant', 'bat', 'cat']
>>> "ant, bat, cat".split(", ")
['ant', 'bat', 'cat']
>>> "antheybatheycat".split("hey")
['ant', 'bat', 'cat']

Many languages support this sort of splitting, but note that Python includes an empty string in the final case illustrated above, which some languages (such as Ruby) trim automatically. We can avoid this extra string in the common case of splitting on newlines using splitlines() instead (Listing 3.2).

>>> s = "This is a line.\nAnd this is another line.\n"
>>> s.split("\n")
['This is a line.', 'And this is another line.', '']
>>> s.splitlines()
['This is a line.', 'And this is another line.']

Many languages allow us to split a string into its component characters by splitting on the empty string, but this doesn’t work in Python:

>>> "badger".split("")
"badger".split("")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: empty separator

In Python, the best way to do this is using the list() function directly on the string:

>>> list("badger")
['b', 'a', 'd', 'g', 'e', 'r']

Because Python can naturally iterate over a string’s characters, this technique is rarely needed explicitly; instead, we’ll typically use iterators, which we’ll learn about in Section 5.3.

Perhaps the most common use of split() is with no arguments; in this case, the default behavior is to split on whitespace (such as spaces, tabs, or newlines):

>>> "ant bat cat".split()
['ant', 'bat', 'cat']
>>> "ant     bat\t\tcat\n    duck".split()
['ant', 'bat', 'cat', 'duck']

We’ll investigate this case more closely when discussing regular expressions in Section 4.3.

3.2 List access

Having connected strings with lists via the split() method, we’ll now discover a second close connection as well. Let’s start by assigning a variable to a list of characters created using list():

>>> a = list("badger")
['b', 'a', 'd', 'g', 'e', 'r']

Here we’ve followed tradition and called the variable a, both because it’s the first letter of the alphabet and as a nod to the array type that lists so closely resemble.

We can access particular elements of a using the same bracket notation we first encountered in the context of strings in Section 2.6, as seen in Listing 3.3.

>>> a[0]
'b'
>>> a[1]
'a'
>>> a[2]
'd'

We see from Listing 3.3 that, as with strings, lists are zero-offset, meaning that the “first” element has index 0, the second has index 1, and so on. This convention can be confusing, and in fact it’s common to refer to the initial element for zero-offset lists as the “zeroth” element as a reminder that the indexing starts at 0. This convention can also be confusing when using multiple languages (some of which start list indexing at 1), as illustrated in the xkcd comic strip “Donald Knuth”.¹

So far we’ve dealt exclusively with lists of characters, but Python lists can contain all types of elements (Listing 3.4).

>>> soliloquy = "To be, or not to be, that is the question:"
>>> a = ["badger", 42, "To be" in soliloquy]
>>> a
['badger', 42, True]
>>> a[2]
True
>>> a[3]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range

We see here that the square bracket access notation works as usual for a list of mixed types, which shouldn’t come as a surprise. We also see that trying to access a list index outside the defined range raises an error if we try to access an element that’s out of range.

Another convenient feature of Python bracket notation is supporting negative indices, which count from the end of the list:

>>> a[-2]
42

Among other things, negative indices give us a compact way to select the last element in a list. Because len() (Section 2.4) works on lists as well as strings, we could do it directly by subtracting 1 from the length (which we have to do because lists are zero-offset):

>>> a[len(a) - 1]
True

But it’s even easier like this:

>>> a[-1]
True

A final common case is where we want to access the final element and remove it at the same time. We’ll cover the method for doing this in Section 3.4.3.

By the way, starting in Listing 3.4, we used a literal square-bracket syntax to define lists by hand. This notation is so natural that you probably didn’t even notice it, and indeed it’s the same format the REPL uses when printing out lists.

We can use this same notation to define the empty list [], which just evaluates to itself:

>>> []
[]

You may recall from Section 2.4.2 that empty or nonexistent things like "", 0, and None are False in a boolean context. This pattern holds for the empty list as well:

>>> bool([])
False

3.3 List slicing

In addition to supporting the bracket notation described in Section 3.2, Python excels at a technique known as list slicing (Figure 3.1)² for accessing multiple elements at a time. In anticipation of learning to sort in Section 3.4.2, let’s redefine our list a to have purely numerical elements:

>>> a = [42, 8, 17, 99]
[42, 8, 17, 99]

Figure 3.1: Python is unusually good at slicing.

One way to slice a list is to use the slice() function and provide two arguments corresponding to the index number where the slice should start and where it should end. For example, slice(2, 4) lets us pull out the elements with index 2 and 3, ending at 4:

>>> a[slice(2, 4)]    # Not Pythonic
[17, 99]

This can be a little tricky to understand since there is no element with index 4 due to lists being zero-offset. We can understand this better by imagining a pointer that moves one element to the right as it creates the slice; it starts at 2, selects element 2 as it moves to 3, and then selects element 3 as it moves to 4.

The explicit slice() notation is rarely used in real Python code; far more common is the equivalent notation using colons, like this:

>>> a[2:4]    # Pythonic
[17, 99]

Note that the index convention is the same: to select elements with indices 2 and 3, we include a final range that is one more than the value of the final index in the slice (in this case, $3+1 = 4$).

In the case of our current list, 4 is the length of the list, so in effect we are slicing from the element with index 2 to the end. This is such a common task that Python has a special notation for it—we just leave the second index off entirely:

>>> a[2:]    # Pythonic
[17, 99]

As you might guess, the same basic notation works to slice from the front of the list:

>>> a[:2]    # Pythonic
[42, 8]

The general pattern here is that a[start:end] selects from index start to index end-1, where either can be omitted to select from the start or to the end. Python also supports an extension to this syntax taking the form a[start:end:step], which is the same as regular list slicing except taken step at a time. For example, we can select numbers from a range 3 at a time as follows:

>>> numbers = list(range(20))
>>> numbers
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> numbers[0:20:3]       # Not Pythonic
[0, 3, 6, 9, 12, 15, 18]

Or we could start at, say, index 5 and end at index 17:

>>> numbers[5:17:3]
[5, 8, 11, 14]

As with regular slicing, we can omit values if we want the start or the end:

>>> numbers[:10:3]    # Goes from the beginning to 10-1
[0, 3, 6, 9]
>>> numbers[5::3]     # Goes from 5 to the end
[5, 8, 11, 14, 17]

We can replicate the result of numbers[0:20:3] more Pythonically by omitting both 0 and 20:

>>> numbers[::3]      # Pythonic
[0, 3, 6, 9, 12, 15, 18]

We can even go backward using a negative step:

>>> numbers[::-3]
[19, 16, 13, 10, 7, 4, 1]

This suggests a (perhaps too clever) way to reverse a list, which is to use a step of -1. Applying this idea to our original list looks like this:

>>> a[::-1]
[99, 17, 8, 42]

You may encounter this [::-1] construction in real-life Python code, so it’s important to know what it does, but there are more convenient and readable ways to reverse a list, as discussed in Section 3.4.2.

3.4 More list techniques

There are many other things we can do with lists other than accessing and selecting elements. In this section we’ll discuss element inclusion, sorting and reversing, and appending and popping.

3.5 List iteration

One of the most common tasks with lists is iterating through their elements and performing an operation with each one. This might sound familiar, since we solved the exact same problem with strings in Section 2.6, and indeed the solution is virtually the same. All we need to do is adapt the for loop from Listing 2.27 to lists, i.e., replace soliloquy with a, as shown in Listing 3.8.

>>> a = ["ant", "bat", "cat", 42]
>>> for i in range(len(a)):    # Not Pythonic
...     print(a[i])
...
ant
bat
cat
42

That’s convenient, but it’s not the best way to iterate through lists, and Mike Vanier still wouldn’t be happy (Figure 3.2).

Figure 3.2: Mike Vanier is still annoyed by typing out for loops.

Luckily, looping the Right Way™ is easier than it is in most other languages, so we can actually cover it here (unlike in, e.g., Learn Enough JavaScript to Be Dangerous, when we had to wait until Chapter 5). The trick is knowing that, as with strings, the default behavior of for...in is to return each element in sequence, as shown in Listing 3.9.

>>> for e in a:    # Pythonic
...     print(e)
...
ant
bat
cat
42

Using this style of for loop, we can iterate directly through the elements in a list, thereby avoiding having to type out Mike Vanier’s bête noire, “for (i = 0; i < N; i++)”. The result is cleaner code and a happier programmer (Figure 3.3).

Figure 3.3: Avoiding range(len()) has made Mike Vanier a little happier.

By the way, we can use enumerate() if for some reason we need the index itself, as shown in Listing 3.10. (If you solved the exercise corresponding to Listing 2.29, the code in Listing 3.10 might look familiar.)

>>> for i, e in enumerate(a):    # Pythonic
...     print(f"a[{i}] = {e}")
...
a[0] = ant
a[1] = bat
a[2] = cat
a[3] = 42

Note the final results in Listing 3.10 aren’t quite right because we really should show, say, the first element as "ant" instead of as ant. Fixing this minor blemish is left as an exercise.

Finally, it’s possible to break out of a loop early using the break keyword (Listing 3.11).

>>> for i, e in enumerate(a):
...     if e == "cat":
...         print(f"Found the cat at index {i}!")
...         break
...     else:
...         print(f"a[{i}] = {e}")
...
a[0] = ant
a[1] = bat
Found the cat at index 2!
>>>

In this case the execution of the loop stops at index 2 and doesn’t proceed to any subsequent indices. We’ll see a similar construction using the return keyword in Section 5.1.

3.6 Tuples and sets

In addition to lists, Python also supports tuples, which are basically lists that can’t be changed (i.e., tuples are immutable). By the way, I generally say “toople”, though you will also hear “tyoople” and “tupple”.

We can create literal tuples in much the same way that we created literal lists. The only difference is that tuples use parentheses instead of square brackets:

>>> t = ("fox", "dog", "eel")
>>> t
('fox', 'dog', 'eel')
>>> for e in t:
...     print(e)
...
fox
dog
eel

We see here that iterating over a tuple uses the same for...in syntax used for lists (Listing 3.9).

Because tuples are immutable, trying to change them raises an error:

>>> t.append("goat")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'tuple' object has no attribute 'append'
>>> t.sort()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'tuple' object has no attribute 'sort'

Otherwise, tuples support many of the same operations as lists, such as slicing or non-mutating sorting:

>>> t[1:]
('dog', 'eel')
>>> sorted(t)
['dog', 'eel', 'fox']

Note in the second case that sorted() can take a tuple as an argument but that it returns a list.

By the way, we can also leave off parentheses when defining tuples:

>>> u = "fox", "dog", "eel"
>>> u
('fox', 'dog', 'eel')
>>> t == u
True

I think this notation is potentially confusing and generally prefer to use parentheses when defining tuples, but you should know about it in case you see it in other people’s code. The main exceptions are when simply displaying several variables in the REPL or when doing assignment via so-called tuple unpacking, which lets you make multiple assignments at once:

>>> a, b, c = t    # Very Pythonic; works for lists, too
>>> a
'fox'
>>> a, b, c        # Tuple to show the variable values

Finally, it’s worth noting that defining a tuple of one element requires a trailing comma because an object in parentheses alone is just the object itself:

>>> ("foo")
'foo'
>>> ("foo",)
('foo',)

Python also has native support for sets, which correspond closely to the mathematical definition and can be thought of as lists of elements where repeat values are ignored and the order doesn’t matter. Sets can be initialized literally using curly braces or by passing a list or a tuple (or in fact any iterable) to the set() function:

>>> s1 = {1, 2, 3, 4}
>>> s2 = {3, 1, 4, 2}
>>> s3 = set([1, 2, 2, 3, 4, 4])
>>> s1, s2, s3
({1, 2, 3, 4}, {1, 2, 3, 4}, {1, 2, 3, 4})

Set equality can be tested with == as usual:

>>> s1 == s2
True
>>> s2 == s3
True
>>> s1 == s3
True
>>> {1, 2, 3} == {3, 1, 2}
True

Sets can also mix types (and can be initialized with a tuple instead of a list):

>>> set(("ant", "bat", "cat", 1, 1, "cat"))
{'bat', 'ant', 'cat'}

Note that in all cases duplicate values are ignored.

Python sets support many common set operations, such as union and intersection:

>>> s1 = {1, 2, "ant", "bat"}
>>> s2 = {2, 3, "bat", "cat"}
>>> s1 | s2    # Set union
{'bat', 1, 2, 'ant', 3, 'cat'}
>>> s1 & s2    # Set intersection
{'bat', 2}

See “Sets in Python” for more information.

Because they are unordered, set elements can’t be selected directly (how would Python know which set element to pick?) but can be tested for inclusion or iterated over:

>>> s = {1, 2, 3, 4}
>>> s[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'set' object is not subscriptable
>>> 3 in s
True
>>> for e in s:
...     print(f"{e} is an element of the set")
...
1 is an element of the set
2 is an element of the set
3 is an element of the set
4 is an element of the set

Finally, it’s worth noting that, like the empty list, the empty tuple and the empty set are both False in a boolean context:

>>> bool(())
False
>>> bool(set())
False

Note here that, perhaps counterintuitively, we can’t use {} for the empty set because that combination is reserved for the empty dictionary, which we’ll discuss in Section 4.4. We also don’t have to include a trailing comma in (), which is the empty tuple as required.

We can confirm these statements using the type() function:

>>> type(())
<class 'tuple'>
>>> type({})
<class 'dict'>
>>> type(set())
<class 'set'>

Here we see that (), {}, and set() are of class tuple, dictionary, and set, respectively. (We’ll discuss more about what a class is in Chapter 7.)