Note: Click on "Kernel" > "Restart Kernel and Clear All Outputs" in JupyterLab before reading this notebook to reset its output. If you cannot run this file on your machine, you may want to open it in the cloud .

Chapter 2: Functions & Modularization¶

In Chapter 1 , we simply typed the code to calculate the average of the even numbers in a list of whole numbers into several code cells. Then, we executed them one after another. We had no way of reusing the code except for either executing cells multiple times. And, whenever we find ourselves doing repetitive manual work, we can be sure that there must be a way of automating what we are doing.

This chapter shows how Python offers language constructs that let us define functions ourselves that we may then call just like the built-in ones. Also, we look at how we can extend our Python installation with functionalities written by other people.

Built-in Functions

Python comes with plenty of useful functions built in, some of which we have already seen before (e.g., print() , sum() , len() , or id() ). The documentation has the full list. Just as core Python itself, they are mostly implemented in C and thus very fast.

Below, sum() adds up all the elements in the numbers list while len() counts the number of elements in it.

sum and len are no keywords like for or if but variables that reference objects in memory. Often, we hear people say that "everything is an object in Python" (e.g., this question ). While this phrase may sound abstract in the beginning, it simply means that the entire memory is organized with "bags" of $0$s and $1$s, and there are even bags for the built-in functions. That is not true for many other languages (e.g., C or Java) and often a source of confusion for people coming to Python from another language.

The built-in id() function tells us where in memory a particular built-in function is stored.

type() reveals that built-in functions like sum() or len() are objects of type builtin_function_or_method.

Python's object-oriented nature allows us to have functions work with themselves. While seemingly not useful from a beginner's point of view, that enables a lot of powerful programming styles later on.

To execute a function, we call it with the call operator () as shown many times in Chapter 1 and above.

If we are unsure whether a variable references a function or not, we can verify that with the built-in callable() function.

Abstractly speaking, any object that can be called with the call operator () is a so-called callable. And, objects of type builtin_function_or_method are just one kind of examples thereof. We will see another one already in the next sub-section.

list objects, for example, are not callable.

Constructors¶

The list of built-in functions in the documentation should really be named a list of built-in callables.

Besides the built-in functions, the list also features constructors for the built-in types. They may be used to cast (i.e., "convert") any object as an object of a given type.

For example, to "convert" a float or a str into an int object, we use the int() built-in. Below, new int objects are created from the 7.0 and "7" objects that are newly created themselves before being processed by int() right away without ever being referenced by a variable.

Casting an object as an int is different from rounding with the built-in round() function!

Notice the subtle difference compared to the behavior of the // operator in Chapter 1 that "rounds" towards minus infinity: int() always "rounds" towards 0.

Not all conversions are valid and runtime errors may occur as the ValueError shows.

We may also cast in the other direction with the float() or str() built-ins.

Constructors are full-fledged objects as well.

They are of type type, which is different from builtin_function_or_method above.

As already noted, constructors are callables. In that regard, they behave the same as built-in functions. We may call them with the call operator ().

The attentive student may already have discovered that we refer to builtin_function_or_method objects as "built-in functions" and type objects as just "built-ins." For a beginner, that difference is not so important. But, the ambitious student should already be aware that such subtleties exist.

Next, let's look at a third kind of callables.

Function Definitions¶

We may create so-called user-defined functions with the def statement (cf., reference ). To extend an already familiar example, we reuse the introductory example from Chapter 1 in its final Pythonic version and transform it into the function average_evens() below. We replace the variable name numbers with integers for didactical purposes in the first couple of examples.

A function's name must be chosen according to the same naming rules as ordinary variables since Python manages function names like variables. In this book, we further adopt the convention of ending function names with parentheses () in text cells for faster comprehension when reading (i.e., average_evens() vs. average_evens). These are not part of the name but must always be written out in the def statement for syntactic reasons.

Functions may define an arbitrary number of parameters as inputs that can then be referenced within the indented code block: They are listed within the parentheses in the def statement (i.e., integers below).

The code block is also called a function's body, while the first line starting with def and ending with a colon is the header.

Together, the name and the list of parameters are also referred to as the function's signature (i.e., average_evens(integers) below).

A function may specify an explicit return value (i.e., "result" or "output") with the return statement (cf., reference ): Functions that have one are considered fruitful; otherwise, they are void. Functions of the latter kind are still useful because of their side effects. For example, the built-in print() function changes what we see on the screen. Strictly speaking, print() and other void functions also have an implicit return value, namely the None object.

A function should define a docstring that describes what it does in a short subject line, what parameters it expects (i.e., their types), and what it returns, if anything. A docstring is a syntactically valid multi-line string (i.e., type str) defined within triple-double quotes """. Strings are covered in depth in Chapter 6 . Widely adopted standards for docstrings are PEP 257 and section 3.8 of Google's Python Style Guide .

Once defined, a function may be referenced just like any other variable by its name (i.e., without the parentheses).

This works as functions are full-fledged objects. So, average_evens is just a name referencing an object in memory with an identity, a type, namely function, and a value. In that regard, average_evens is no different from the variable numbers or the built-ins' names.

Its value may seem awkward at first: It consists of a location showing where the function is defined (i.e., __main__ here, which is Python's way of saying "in this notebook") and the signature wrapped inside angle brackets < and >.

The angle brackets are a convention to indicate that the value may not be used as a literal (i.e., typed back into another code cell). Chapter 11 introduces the concept of a text representation of an object, which is related to the semantic meaning of an object's value as discussed in Chapter 1 , and the angle brackets convention is one such way to represent an object as text. When executed, the angle brackets cause a SyntaxError because Python expects the < operator to come with an operand on both sides (cf., Chapter 3 ).

average_evens is, of course, callable. So, the function type is the third kind of callable in this chapter.

The built-in help() function shows a function's docstring.

Whenever we use code to analyze or obtain information on an object, we say that we introspect it.

In JupyterLab, we can just as well add a question mark ? to a function's name to achieve the same.

Two question marks ?? show a function's source code.

help() and the ?s also work for built-ins.

Function Calls¶

Once defined, we may call a function with the call operator () as often as we wish. The formal parameters are then filled in by passing expressions (e.g., literals or variables) as arguments to the function within the parentheses.

A function call's return value is commonly assigned to a variable for subsequent use. Otherwise, we lose access to the returned object right away.

Scoping Rules¶

Local Scope disappears¶

The parameters listed in a function's definition (i.e., integers in the example) and variables created inside it during execution (i.e., evens and average) are local to that function. That means they only reference an object in memory while the function is being executed and are dereferenced immediately when the function call returns. We say they go out of scope. That is why we see the NameErrors below.

PythonTutor visualizes what happens in memory: To be precise, in the exact moment when the function call is initiated and numbers passed in as the integers argument, there are two references to the same list object (cf., steps 4-5 in the visualization). We also see how Python creates a new frame that holds the function's local scope (i.e., "internal names") in addition to the global frame. Frames are nothing but namespaces to isolate the names of different scopes from each other. The list comprehension [n for n in integers if n % 2 == 0] constitutes yet another frame that is in scope as the list object assigned to evens is being created (cf., steps 6-20). When the function returns, only the global frame is left (cf., last step).

Global Scope is everywhere¶

On the contrary, while a function is being executed, it may reference the variables of enclosing scopes (i.e., "outside" of it). This is a common source of semantic errors. Consider the following stylized and incorrect example average_wrong(). The error is hard to spot with eyes: The function never references the integers parameter but the numbers variable in the global scope instead.

numbers in the global scope is, of course, not changed by merely defining average_wrong().

Sometimes a function may return a correct solution for some inputs ...

... but still be wrong in general.

PythonTutor is again helpful at visualizing the error interactively: Creating the list object evens eventually references takes 16 computational steps, namely two for managing the list comprehension, one for setting up an empty list object, twelve for filling it with elements derived from numbers in the global scope (i.e., that is the error), and one to make evens reference it (cf., steps 6-21).

The frames logic shown by PythonTutor is the mechanism with which Python not only manages the names inside one function call but also for many potentially simultaneous calls, as revealed in Chapter 4 . It is the reason why we may reuse the same names for the parameters and variables inside both average_evens() and average_wrong() without Python mixing them up. So, as we already read in the Zen of Python , "namespaces are one honking great idea" (cf., import this), and a frame is just a special kind of namespace.

Shadowing¶

Code gets even more confusing when variables by the same name from different scopes collide. In particular, what should we expect to happen if a function "changes" a globally defined variable in its body?

average_evens() below works like average_evens() above except that it rounds the numbers in integers with the built-in round() function before filtering and averaging them. round() returns int objects independent of its argument being an int or a float object. On the first line in its body, average_evens() introduces a local variable numbers whose name collides with the one defined in the global scope.

As a good practice, let's first "verify" that average_evens() is "correct" by calling it with inputs for which we can calculate the answer in our heads. Treating a function as a "black box" (i.e., input-output specification) when testing is also called unit testing and plays an important role in modern software engineering.

Such tests are often and conveniently expressed with the assert statement (cf., reference ): If the expression following assert evaluates to True, nothing happens.

However, if the expression evaluates to False, an AssertionError is raised.

Calling average_evens() leaves numbers in the global scope unchanged.

To add to the confusion, let's also pass the global numbers list as an argument to average_evens(). The return value is the same as before.

In summary, Python is smart enough to keep all the involved numbers variables apart. So, the global numbers variable is still referencing the same list object as before.

The reason why everything works is that every time we (re-)assign an object to a variable inside a function's body with the = statement, this is done in the local scope by default. There are ways to change variables existing in an outer scope from within a function, but this is a rather advanced topic.

PythonTutor shows how two numbers variables exist in different scopes referencing different objects (cf., steps 14-25) when we execute average_evens([40.0, 41.1, 42.2, 43.3, 44.4]).

Variables whose names collide with the ones of variables in enclosing scopes - and the global scope is just the most enclosing scope - are said to shadow them.

While this is not a problem for Python, it may lead to less readable code for humans and should be avoided if possible. But, as the software engineering wisdom goes, "naming things" is often considered a hard problem as well, and we have to be prepared to encounter shadowing variables.

Shadowing also occurs if a parameter in the function definition goes by the same name as a variable in an outer scope. Below, average_evens() is identical to the first version in this chapter except that the parameter integers is now called numbers as well.

PythonTutor reveals that in this example there are two numbers variables in different scope referencing the same list object in memory (cf., steps 4-23).

Positional vs. Keyword Arguments¶

So far, we have specified only one parameter in each of our user-defined functions. In Chapter 1 , however, we saw the built-in divmod() function take two arguments. And, the order in which they are passed in matters! Whenever we call a function and list its arguments in a comma separated manner, we say that we pass in the arguments by position or refer to them as positional arguments.

For many functions, there is a natural order to the arguments: For example, for any kind of division passing the dividend first and the divisor second seems intuitive. But what if that is not the case in another setting? For example, let's create a close relative of the above average_evens() function that also scales the resulting average by a factor. What is more natural? Passing in numbers first? Or scalar? There is no obvious way and we continue with the first alternative for no concrete reason.

As with divmod() , we may pass in the arguments by position.

Now, this function call is a bit harder to understand as we always need to remember what the 2 means. This becomes even harder with more parameters.

Luckily, we may also pass in arguments by name. Then, we refer to them as keyword arguments.

When passing all arguments by name, we may do so in any order.

We may even combine positional and keyword arguments in the same function call.

Unfortunately, there are ways to screw this up with a SyntaxError: If positional and keyword arguments are mixed, the keyword arguments must come last.

Similarly, we must always pass in the right number of arguments. Otherwise, a TypeError is raised.

Modularization¶

Defining average_evens() and scaled_average_evens() as above leads to a repetition of most of their code. That is not good as such a redundancy makes a code base hard to maintain in the long run: Whenever we change the logic in one function, we must not forget to do so for the other function as well. And, most likely, we forget about such issues in larger projects.

Below, three of four lines in the functions' bodies are identical!

A better way is to design related functions in a modular fashion such that they reuse each other's code.

For example, as not scaling an average is just a special case of scaling it with 1, we could redefine the two functions like below: In this version, the function resembling the special case, average_evens(), forwards the call to the more general function, scaled_average_evens(), passing a scalar argument of 1. As the name scaled_average_evens within the body of average_evens() is looked up each time the function is being executed, we may define average_evens() before scaled_average_evens().

After refactoring the functions, it is a good idea to test them again.

Default Arguments¶

Assuming that scaling the average occurs rarely, it may be a good idea to handle both cases in one function definition by providing a default argument of 1 for the scalar parameter.

Now, we call the function with or without passing a scalar argument.

If scalar is not passed in, it automatically takes the value 1.

If scalar is passed in, this may be done as either a positional or a keyword argument. Which of the two calls where scalar is 2 is faster to understand in a larger program?

Keyword-only Arguments¶

Because we assumed that scaling occurs rarely, we would prefer that our new version of average_evens() be called with a keyword argument whenever scalar is passed in. Then, a function call is never ambiguous when reading the source code.

Python offers a keyword-only syntax when defining a function that forces a caller to pass the scalar argument by name if it is passed in at all: To do so, we place an asterisk * before the arguments that may only be passed in by name. Note that the keyword-only syntax also works without a default argument.

As before, we may call average_evens() without passing in an argument for the scalar parameter.

If we call average_evens() with a scalar argument, we must use keyword notation.

If instead we pass in scalar as a positional argument, we get a TypeError.

Anonymous Functions¶

The def statement is a statement because of its side effect of creating a new name that references a new function object in memory.

We can thus think of it as doing two things atomically (i.e., either both of them happen or none). First, a function object is created that contains the concrete $0$s and $1$s that resemble the instructions we put into the function's body. In the context of a function, these $0$s and $1$s are also called byte code . Then, a name referencing the new function object is created.

Only this second aspect makes def a statement: Merely creating a new object in memory without making it accessible for later reference does not constitute a side effect because the state the program is not changed. After all, if we cannot reference an object, how do we know it exists in the first place?

Python provides a lambda expression syntax that allows us to only create a function object in memory without making a name reference it (cf., reference ). It starts with the keyword lambda followed by an optional listing of comma separated parameters, a mandatory colon, and one expression that serves as the return value of the resulting function object. Because it does not create a name referencing the object, we effectively create "anonymous" functions with it.

In the example, we create a function object that adds 3 to the only argument passed in as the parameter x and returns that sum.

If you think this is rather pointless to do, you are absolutely correct!

We created a function object, dit not call it, and Python immediately forgot about it. So what's the point?

To inspect the object created by a lambda expression, we use the simple = statement and assign it to the variable add_three, which is really add_three() as per our convention from above.

type() and callable() confirm that add_three is indeed a callable function object.

Now we may call add_three() as if we defined it with the def statement.

Alternatively, we could call an function object created with a lambda expression right away (i.e., without assigning it to a variable), which looks quite weird for now as we need two pairs of parentheses: The first one serves as a delimiter whereas the second represents the call operator.

The main point of having functions without a reference to them is to use them in a situation where we know ahead of time that we use the function only once.

Popular applications of lambda expressions occur in combination with the map-filter-reduce paradigm (cf., Chapter 8 ).