Python Review¶

Colab notebook

In the following sections, we will repeatedly use Python scripts. If you are less familiar with Python, here is a short tutorial on what you need to know. Also, please take a look here: Google’s Python Class

Introduction¶

Python is a general-purpose programming language that becomes a robust environment for scientific computing, combined with a few popular libraries (numpy, scipy, matplotlib).

!python --version

Python 3.7.12

Basics of Python¶

Python is a high-level, dynamically typed multiparadigm programming language. Python code is often almost like pseudocode since it allows you to express compelling ideas in a few lines of code while being very readable.

Basic data types¶

Numbers¶

Integers and floats work as you would expect from other languages:

x = 3
print(x, type(x))

3 <class 'int'>

print(x + 1)  #addition
print(x - 1)  #subtraction
print(x * 2)  #multiplication
print(x ** 2) #exponentiation

x = 10; x += 1
print(x)

x = 10; x *= 2
print(x)

11
20

y = 2.5
print(y, y+1, y*2, y *2, type(y))

2.5 3.5 5.0 5.0 <class 'float'>

Booleans¶

Python implements all of the usual operators for Boolean logic, but uses English words rather than symbols:

t, f = True, False; print(type(t))

<class 'bool'>

Now we let’s look at some of the operations:

print(t and f) # Logical AND;
print(t or f)  # Logical OR;
print(not t)   # Logical NOT;
print(t != f)  # Logical XOR;

False
True
False
True

Strings¶

hello = 'hello'   # single quotes or double quotes; it does not matter
world = "world"   
print(hello, len(hello))

hello 5

hw = hello + '-' + world+'!'  # String concatenation
print(hw)

hello-world!

hw12 = '{} {} {}'.format(hello, world, 12)  # string formatting
print(hw12)

hello world 12

String objects have a bunch of useful methods; for example:

s = "hello"
print(s.capitalize())  # Capitalize a string
print(s.upper())       # Convert a string to uppercase; prints "HELLO"
print(s.rjust(7))      # Right-justify a string, padding with spaces
print(s.center(7))     # Center a string, padding with spaces
print(s.replace('l', '(ell)'))  # Replace all instances of one substring with another
print('  world '.strip())  # Strip leading and trailing whitespace

Hello
HELLO
  hello
 hello 
he(ell)(ell)o
world

Lists¶

A list is the Python equivalent of an array, but is resizeable and can contain elements of different types:

xs = [3, 1, 2]   # Create a list
print(xs, xs[2])
print(xs[-1])     # Negative indices count from the end of the list

[3, 1, 2] 2
2

xs[2] = 'foo77'    # Lists can contain elements of different types
print(xs)

[3, 1, 'foo77']

xs.append('bar 87') # Add a new element to the end of the list
print(xs)  

[3, 1, 'foo77', 'bar 87']

x = xs.pop()     # Remove and return the last element of the list
print(x, xs)

bar 87 [3, 1, 'foo77']

Slicing¶

In addition to accessing list elements one at a time, Python provides concise syntax to access sublists; this is known as slicing:

nums = list(range(5))    # range is a built-in function that creates a list of integers
print(nums)         # Prints "[0, 1, 2, 3, 4]"
print(nums[2:4])    # Get a slice from index 2 to 4 (exclusive)
print(nums[2:])     # Get a slice from index 2 to the end
print(nums[:2])     # Get a slice from the start to index 2 (exclusive)
print(nums[:])      # Get a slice of the whole list
print(nums[:-1])    # Slice indices can be negative
nums[2:4] = [8, 9]; print(nums)  # Assign a new sublist to a slice      

[0, 1, 2, 3, 4]
[2, 3]
[2, 3, 4]
[0, 1]
[0, 1, 2, 3, 4]
[0, 1, 2, 3]
[0, 1, 8, 9, 4]

Loops¶

One can loop over the elements in a list like this:

animals = ['dog', 'cat', 'mouse']
for animal in animals:
    print(animal)

dog
cat
mouse

If you want access to the index of each element within the body of a loop, use the built-in enumerate function:

animals = ['cat', 'dog', 'monkey']
for idx, animal in enumerate(animals):
    print('#{}: {}'.format(idx + 1, animal))

#1: cat
#2: dog
#3: monkey

List comprehensions:¶

When programming, frequently we want to transform one type of data into another. As a simple example, consider the following code that computes square numbers:

nums = [0, 1, 2, 3, 4]
squares = []
for x in nums:
    squares.append(x ** 2)
print(squares)

[0, 1, 4, 9, 16]

You can make this code simpler using a list comprehension:

nums = [0, 1, 2, 3, 4]
squares = [x ** 2 for x in nums]
print(squares)

[0, 1, 4, 9, 16]

List comprehensions can also contain conditions:

nums = [0, 1, 2, 3, 4]
even_squares = [x ** 2 for x in nums if x % 2 == 0]
print(even_squares)

[0, 4, 16]

even_squares = [x ** 2 if x % 2 == 0 else -99 for x in nums ] # list comprehension with if/else condition
print(even_squares)

[0, -99, 4, -99, 16]

Dictionaries¶

A dictionary stores (key, value) pairs

d = {'cat': 'meow', 'dog': 'bark'}  # Create a new dictionary with some data
print(d['cat'])       # Get an entry from a dictionary
print('cat' in d)     # Check if a dictionary has a given key

meow
True

d['fish'] = 'wet'     # Set a new entry in the dictionary
print(d['fish'])      

wet

print(d)

{'cat': 'meow', 'dog': 'bark', 'fish': 'wet'}

del d['fish']        # Remove an element from a dictionary

print(d)

{'cat': 'meow', 'dog': 'bark'}

It is easy to iterate over the keys in a dictionary:

d = {'human': 2, 'dog': 4, 'spider': 8}
for animal, legs in d.items():
    print('A {} has {} legs'.format(animal, legs))

A human has 2 legs
A dog has 4 legs
A spider has 8 legs

Dictionary comprehensions: These are similar to list comprehensions but allow you to construct dictionaries easily. For example:

nums = [0, 1, 2, 3, 4]
even_num_to_square = {x: x ** 2 for x in nums if x % 2 == 0}
print(even_num_to_square)

{0: 0, 2: 4, 4: 16}

Sets¶

A set is an unordered collection of distinct elements. As a simple example, consider the following:

animals = {'cat', 'dog'}
print('cat' in animals)  
print('fish' in animals)  
print(animals)

True
False
{'dog', 'cat'}

animals.add('fish')      # Add an element to a set
print('fish' in animals)
print(len(animals))       # Number of elements in a set;
print(animals)

True
3
{'fish', 'dog', 'cat'}

animals.add('cat')       # Adding an element that is already in the set does nothing
print(len(animals))       

animals.remove('cat')    # Remove an element from a set
print(len(animals))    
print(animals)   

3
2
{'fish', 'dog'}

Loops: Iterating over a set has the same syntax as iterating over a list; however, since sets are unordered, you cannot make assumptions about the order in which you visit the elements of the set:

animals = {'cat', 'dog', 'mouse'}
for idx, animal in enumerate(animals):
    print('#{}: {}'.format(idx + 1, animal))

#1: dog
#2: cat
#3: mouse

Set comprehensions: Like lists and dictionaries, we can easily construct sets using set comprehensions:

from math import sqrt
print({int(sqrt(x)) for x in range(30)})

{0, 1, 2, 3, 4, 5}

Tuples¶

A tuple is an immutable ordered list of values. A tuple is similar to a list; one of the most important differences is that tuples can be used as keys in dictionaries and as elements of sets, while lists cannot. Here is a trivial example:

d = {(x, x + 1): x for x in range(7)}  # Create a dictionary with tuple keys
t = (5, 6)       # Create a tuple
print(type(t))
print(d)
print(d[t])       
print(d[(1, 2)])

<class 'tuple'>
{(0, 1): 0, (1, 2): 1, (2, 3): 2, (3, 4): 3, (4, 5): 4, (5, 6): 5, (6, 7): 6}
5
1

Functions¶

Python functions are defined using the def keyword. For example:

def sign(x):
    if x > 0:
        return 'positive'
    elif x < 0:
        return 'negative'
    else:
        return 'zero'

for x in [-1, 0, 1]:
    print(sign(x))

negative
zero
positive

We will often define functions to take optional keyword arguments, like this:

def hello(name, loud=False):
    if loud:
        print('HELLO, {}'.format(name.upper()))
    else:
        print('Hello, {}!'.format(name))

hello('Bob')
hello('Fred', loud=True)

Hello, Bob!
HELLO, FRED

Classes¶

The syntax for defining classes in Python is simple:

class Greeter:
    # Constructor
    def __init__(self, name):
        self.name = name  # Create an *instance* variable

    # Instance method
    def greet(self, loud=False):
        if loud:
          print('HELLO, {}'.format(self.name.upper()))
        else:
          print('Hello, {}!'.format(self.name))

g = Greeter('Fred')  # Construct an instance of the Greeter class
print(g.name)
g.greet()            # Call an instance method
g.greet(loud=True)   # Call an instance method

Fred
Hello, Fred!
HELLO, FRED

Numpy¶

Numpy is the core library for scientific computing in Python. It provides a high-performance multidimensional array object and tools for working with these arrays.

import numpy as np

Arrays¶

A numpy array is a grid of values, all of the same type, indexed by a tuple of non-negative integers. The number of dimensions is the rank of the array. The shape of an array is a tuple of integers giving the size of the array along each dimension.

We can initialize numpy arrays from nested Python lists and access elements using square brackets:

a = np.array([1, 2, 3])  # Create a rank 1 array
print(type(a), a.shape, a[0], a[1], a[2])

a[0] = 5                 # Change an element of the array
print(a)                  

<class 'numpy.ndarray'> (3,) 1 2 3
[5 2 3]

b = np.array([[1,2,3],[4,5,6]])   # Create a rank 2 array
print(b)

[[1 2 3]
 [4 5 6]]

print(b.shape)
print(b[0, 0], b[0, 1], b[1, 0])

(2, 3)
1 2 4

Numpy also provides many functions to create arrays:

a = np.zeros((2,2))  # Create an array of all zeros
print(a)

[[0. 0.]
 [0. 0.]]

b = np.ones((1,2))   # Create an array of all ones
print(b)

[[1. 1.]]

c = np.full((2,2), 7) # Create a constant array
print(c)

[[7 7]
 [7 7]]

d = np.eye(2)        # Create a 2x2 identity matrix
print(d)

[[1. 0.]
 [0. 1.]]

e = np.random.random((2,2)) # Create an array filled with random values between 0 and 1
print(e)

[[0.01725305 0.43419775]
 [0.40239247 0.55052041]]

Array indexing¶

Slicing: Similar to Python lists, numpy arrays can be sliced. Since arrays may be multidimensional, you must specify a slice for each dimension of the array:

import numpy as np

a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
print(a)

b = a[:2, 1:3]
print(b)

[[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]
[[2 3]
 [6 7]]

A slice of an array is a view into the same data, so modifying it will modify the original array.

print(a[0, 1])
b[0, 0] = 77   
print(a[0, 1]) 

2
77

Two ways of accessing the data in the middle row of the array. Mixing integer indexing with slices yields an array of lower rank, while using only slices yields an array of the same rank as the original array:

row_r1 = a[1, :]    # Rank 1 view of the second row of a  
row_r2 = a[1:2, :]  # Rank 2 view of the second row of a
row_r3 = a[[1], :]  # Rank 2 view of the second row of a
print(row_r1, row_r1.shape)
print(row_r2, row_r2.shape)
print(row_r3, row_r3.shape)

[5 6 7 8] (4,)
[[5 6 7 8]] (1, 4)
[[5 6 7 8]] (1, 4)

Same when accessing columns of an array:

col_r1 = a[:, 1]
col_r2 = a[:, 1:2]
print(col_r1, col_r1.shape)
print()
print(col_r2, col_r2.shape)

[77  6 10] (3,)

[[77]
 [ 6]
 [10]] (3, 1)

Integer array indexing: When you index into numpy arrays using slicing, the resulting array view will always be a subarray of the original array. In contrast, integer array indexing allows you to construct arbitrary arrays using the data from another array. Here is an example:

a = np.array([[1,2], [3, 4], [5, 6]])
print(a)

print(a[[0, 1, 2], [0, 1, 0]])

print(np.array([a[0, 0], a[1, 1], a[2, 0]]))

[[1 2]
 [3 4]
 [5 6]]
[1 4 5]
[1 4 5]

One useful trick with integer array indexing is selecting or mutating one element from each row of a matrix:

a = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
print(a)

[[ 1  2  3]
 [ 4  5  6]
 [ 7  8  9]
 [10 11 12]]

# Create an array of indices
b = np.array([0, 2, 0, 1])

# Select one element from each row of a using the indices in b
print(a[np.arange(4), b])  

[ 1  6  7 11]

a[np.arange(4), b] += 10
print(a)

[[11  2  3]
 [ 4  5 16]
 [17  8  9]
 [10 21 12]]

Boolean array indexing: Boolean array indexing lets you pick out arbitrary elements of an array. This type of indexing is frequently used to select elements of an array that satisfy some condition. Here is an example:

import numpy as np

a = np.array([[1,2], [3, 4], [5, 6]])

bool_idx = (a > 2)  
print(bool_idx)

[[False False]
 [ True  True]
 [ True  True]]

# Using boolean array indexing to construct a rank 1 array
print(a[bool_idx])

print(a[a > 2])

[3 4 5 6]
[3 4 5 6]

Datatypes¶

Every numpy array is a grid of elements of the same type. Numpy provides a large set of numeric datatypes that you can use to construct arrays. Numpy tries to guess a datatype when you create an array, but functions that construct arrays usually also include an optional argument to specify the datatype explicitly. Here is an example:

x = np.array([1, 2])  # Let numpy choose the datatype
y = np.array([1.0, 2.0])  # Let numpy choose the datatype
z = np.array([1, 2], dtype=np.int64)  # Force a particular datatype

print(x.dtype, y.dtype, z.dtype)

int64 float64 int64

Array math¶

Basic mathematical functions operate elementwise on arrays, and are available both as operator overloads and as functions in the numpy module:

x = np.array([[1,2],[3,4]], dtype=np.float64)
y = np.array([[5,6],[7,8]], dtype=np.float64)

# Elementwise sum; both produce the array
print(x + y)
print(np.add(x, y))

[[ 6.  8.]
 [10. 12.]]
[[ 6.  8.]
 [10. 12.]]

x = np.array([[1,2],[3,4]], dtype=np.float64)
y = np.array([[5,6],[7,8]], dtype=np.int64)  # now an int

# same result as before
print(x + y)
print(np.add(x, y))

[[ 6.  8.]
 [10. 12.]]
[[ 6.  8.]
 [10. 12.]]

# Elementwise difference; both produce the array
print(x - y)
print(np.subtract(x, y))

[[-4. -4.]
 [-4. -4.]]
[[-4. -4.]
 [-4. -4.]]

# Elementwise product; both produce the array
print(x * y)
print(np.multiply(x, y))

[[ 5. 12.]
 [21. 32.]]
[[ 5. 12.]
 [21. 32.]]

# Elementwise division; both produce the array
print(x / y)
print(np.divide(x, y))

[[0.2        0.33333333]
 [0.42857143 0.5       ]]
[[0.2        0.33333333]
 [0.42857143 0.5       ]]

# Elementwise square root;
print(np.sqrt(x))

[[1.         1.41421356]
 [1.73205081 2.        ]]

The dot function is used to compute the inner products of vectors, multiply a vector by a matrix, and multiply matrices. dot is available both as a function in the numpy module and as an instance method of array objects:

x = np.array([[1,2],[3,4]])
y = np.array([[5,6],[7,8]])

v = np.array([9,10])
w = np.array([11, 12])

# Inner product of vectors; both produce the same result
print(v.dot(w))
print(np.dot(v, w))

219
219

You can also use the @ operator which is equivalent to numpy’s dot operator.

print(v @ w)

# Matrix-vector product; all produce a rank 1 array
print(x.dot(v))
print(np.dot(x, v))
print(x @ v)

[29 67]
[29 67]
[29 67]

# Matrix-matrix product; both produce a rank 2 array
print(x.dot(y))
print(np.dot(x, y))
print(x @ y)

[[19 22]
 [43 50]]
[[19 22]
 [43 50]]
[[19 22]
 [43 50]]

Numpy provides many useful functions for performing computations on arrays; one of the most useful is sum:

x = np.array([[1,2],[3,4]])

print(np.sum(x))  # Compute sum of **all** elements
print(np.sum(x, axis=0))  # Compute sum of each **column**, collapsing all rows
print(np.sum(x, axis=1))  # Compute sum of each **row**, collapsing all columns

10
[4 6]
[3 7]

Apart from computing mathematical functions using arrays, we frequently need to reshape or otherwise manipulate data in arrays. The simplest example of this type of operation is transposing a matrix; to transpose a matrix, simply use the T attribute of an array object:

print(x)
print(x.T)

[[1 2]
 [3 4]]
[[1 3]
 [2 4]]

v = np.array([[1,2,3]])
print(v )
print(v.T)

[[1 2 3]]
[[1]
 [2]
 [3]]

Broadcasting¶

Broadcasting is a powerful mechanism that allows numpy to work with arrays of different shapes when performing arithmetic operations. Frequently we have a smaller array and a larger array, and we want to use the smaller array multiple times to perform some operation on the larger array.

For example, suppose that we want to add a constant vector to each matrix row. We could do it like this:

# We will add the vector v to each row of the matrix x,
# storing the result in the matrix y
x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
print(x)

v = np.array([1, 0, 1])
print(v)

y = np.zeros_like(x)   # an array of zeros with the same shape and type as x
print(y)

print(x.shape,v.shape,y.shape)

# Add the vector v to each row of the matrix x with an explicit loop
for i in range(4):
    y[i, :] = x[i, :] + v

print(y)

[[ 1  2  3]
 [ 4  5  6]
 [ 7  8  9]
 [10 11 12]]
[1 0 1]
[[0 0 0]
 [0 0 0]
 [0 0 0]
 [0 0 0]]
(4, 3) (3,) (4, 3)
[[ 2  2  4]
 [ 5  5  7]
 [ 8  8 10]
 [11 11 13]]

However, when the matrix x is huge, computing an explicit loop in Python can be slow.

Note that adding the vector v to each row of the matrix x is equivalent to forming a matrix vv by stacking multiple copies of v vertically, then performing an elementwise summation of x and vv. We could implement this approach like this:

print(v)
vv = np.tile(v, (4, 1))  # Stack 4 copies of v on top of each other
print(vv)                

[1 0 1]
[[1 0 1]
 [1 0 1]
 [1 0 1]
 [1 0 1]]

y = x + vv  # Add x and vv elementwise
print(y)

[[ 2  2  4]
 [ 5  5  7]
 [ 8  8 10]
 [11 11 13]]

Numpy broadcasting allows us to perform this computation without actually creating multiple copies of v. In example,

x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = np.array([1, 0, 1])

y = x + v  # Add v to each row of x using broadcasting
print(y)

[[ 2  2  4]
 [ 5  5  7]
 [ 8  8 10]
 [11 11 13]]

The line y = x + v works even though x has shape (4, 3) and v has shape (3,) due to broadcasting; this line works as if v actually had shape (4, 3), where each row was a copy of v, and the sum gets done elementwise.

When operating on two arrays, numPy compares their shapes element-wise. It starts with the rightmost dimensions and works its way left. Two dimensions are compatible when

they are equal, or
one of them is 1.

If these conditions are not met, a ValueError is thrown. Here are a few more examples:

Ex 1.	Shape	Ex 2.	Shape	Ex 3.	Shape
A	3 x 2	A	10 x 3 x 3	A	6 x 1 x 4 x 1
B	1 x 2	B	10 x 1 x 3	B	1 x 5 x 1 x 3
A+B	3 x 2	A+B	10 x 3 x 3	A+B	6 x 5 x 4 x 3

Here are some more applications of broadcasting:

# Compute outer product of vectors
v = np.array([1,2,3])  # v has shape (3,)
w = np.array([4,5])    # w has shape (2,)
print(v.shape, w.shape)

print(np.reshape(v, (3, 1)) * w)

(3,) (2,)
[[ 4  5]
 [ 8 10]
 [12 15]]

# Add a vector to each row of a matrix
x = np.array([[1,2,3], [4,5,6]])
v = np.array([1,2,3])  
print(x.shape, v.shape)

print(x + v)

(2, 3) (3,)
[[2 4 6]
 [5 7 9]]

# Add a vector to each column of a matrix
print(x.shape, w.shape)

print((x.T + w).T)

(2, 3) (2,)
[[ 5  6  7]
 [ 9 10 11]]

# Multiply a matrix by a constant:
# x has shape (2, 3). Numpy treats scalars as arrays of shape ();
# these can be broadcast together to shape (2, 3), producing:
print(x * 2)

[[ 2  4  6]
 [ 8 10 12]]

Pandas¶

Pandas is a software library in Python for data manipulation and analysis. It offers data structures and operations for manipulating numerical tables and time series. The main data structure is the DataFrame, which is an in-memory 2D table similar to a spreadsheet, with column names and row labels.

import pandas as pd

Series objects¶

A Series object is 1D array, similar to a column in a spreadsheet, while a DataFrame objects is a 2D table with column names and row labels.

s = pd.Series([1,2,3,4]); s

  1
  2
  3
  4
dtype: int64

Series objects can be passed as parameters to numpy functions

np.log(s)

  0.000000
  0.693147
  1.098612
  1.386294
dtype: float64

s = s + s + [1,2,3,4] # elementwise addition with a list
s

   3
   6
   9
  12
dtype: int64

s = s + 1 # broadcasting
s
 

   4
   7
  10
  13
dtype: int64

s >=10

  False
  False
   True
   True
dtype: bool

Index labels¶

Each item in a Series object has a unique identifier called index label. By default, it is simply the rank of the item in the Series (starting at 0), but you can also set the index labels manually.

s2 = pd.Series([1, 2, 3, 4], index=["a", "b", "c", "d"])
s2

a    1
b    2
c    3
d    4
dtype: int64

You can then use the Series just like a dict object

s2['b']

access items by integer location

s2[1]

accessing by label (L-oc)

s2.loc['b'] 

accessing by integer location (IL-oc)

s2.iloc[1] 

Slicing a Series

s2.iloc[1:3]

b    2
c    3
dtype: int64

s2.iloc[2:]

c    3
d    4
dtype: int64

Initializing from a dict¶

Create a Series object from a dict, keys are used as index labels

d = {"b": 1, "a": 2, "e": 3, "d": 4}
s3 = pd.Series(d)
s3

b    1
a    2
e    3
d    4
dtype: int64

s4 = pd.Series(d, index = ["c", "a"])
s4

c    NaN
a    2.0
dtype: float64

s5 = pd.Series(10, ["a", "b", "c"], name="series")
s5

a    10
b    10
c    10
Name: series, dtype: int64

Automatic alignment¶

When an operation involves multiple Series objects, pandas automatically aligns items by matching index labels.

print(s2.keys())
print(s3.keys())

s2 + s3

Index(['a', 'b', 'c', 'd'], dtype='object')
Index(['b', 'a', 'e', 'd'], dtype='object')

a    3.0
b    3.0
c    NaN
d    8.0
e    NaN
dtype: float64

DataFrame objects¶

DataFrame object represents a spreadsheet, with cell values, column names and row index labels.

d = {
    "feature1": pd.Series([1,2,3], index=["a", "b", "c"]),
    "feature2": pd.Series([4,5,6], index=["b", "a", "c"]),
    "feature3": pd.Series([7,8],  index=["c", "b"]),
    "feature4": pd.Series([9,10], index=["a", "b"]),
}
df = pd.DataFrame(d)
df

	feature1	feature2	feature3	feature4
a	1	5	NaN	9.0
b	2	4	8.0	10.0
c	3	6	7.0	NaN

accessing a column

df["feature3"] 

a    NaN
b    8.0
c    7.0
Name: feature3, dtype: float64

accessing ,multiple columns

df[["feature1", "feature3"]] 

	feature1	feature3
a	1	NaN
b	2	8.0
c	3	7.0

df

	feature1	feature2	feature3	feature4
a	1	5	NaN	9.0
b	2	4	8.0	10.0
c	3	6	7.0	NaN

Constructing a new DataFrame from an existing DataFrame

df2 = pd.DataFrame(
        df,
        columns=["feature1", "feature2", "feature3"],
        index=["b", "a", "d"]
     )
df2

	feature1	feature2	feature3
b	2.0	4.0	8.0
a	1.0	5.0	NaN
d	NaN	NaN	NaN

Creating a DataFrame from a list of lists

lol = [
        [11, 1,      "a",   np.nan],
        [12, 3,      "b",       14],
        [13, np.nan, np.nan,    15]
      ]
df3 = pd.DataFrame(
        lol,
        columns=["feature1", "feature2", "feature3", "feature4"],
        index=["a", "b", "c"]
     )
df3

	feature1	feature2	feature3	feature4
a	11	1.0	a	NaN
b	12	3.0	b	14.0
c	13	NaN	NaN	15.0

Creating a DataFrame with a dictionary of dictionaries

df5 = pd.DataFrame({
    "feature1": {"a": 15, "b": 1984, "c": 4},
    "feature2": {"a": "sentence", "b": "word"},
    "feature3": {"a": 1, "b": 83, "c": 4},
    "feature4": {"c": 2, "d": 0}
})
df5

	feature1	feature2	feature3	feature4
a	15.0	sentence	1.0	NaN
b	1984.0	word	83.0	NaN
c	4.0	NaN	4.0	2.0
d	NaN	NaN	NaN	0.0

Multi-indexing¶

If all columns/rows are tuples of the same size, then they are understood as a multi-index

df5 = pd.DataFrame(
  {
    ("features12", "feature1"):
        {("rows_ab","a"): 444, ("rows_ab","b"): 444, ("rows_c","c"): 666},
    ("features12", "feature2"):
        {("rows_ab","a"): 111, ("rows_ab","b"): 222},
    ("features34", "feature3"):
        {("rows_ab","a"): 555, ("rows_ab","b"): 333, ("rows_c","c"): 777},
    ("features34", "feature4"):
        {("rows_ab", "a"):333, ("rows_ab","b"): 999, ("rows_c","c"): 888}
  }
)

df5

		features12		features34
		feature1	feature2	feature3	feature4
rows_ab	a	444	111.0	555	333
rows_ab	b	444	222.0	333	999
rows_c	c	666	NaN	777	888

df5['features12']

		feature1	feature2
rows_ab	a	444	111.0
rows_ab	b	444	222.0
rows_c	c	666	NaN

df5['features12','feature1']

rows_ab  a    444
         b    444
rows_c   c    666
Name: (features12, feature1), dtype: int64

df5['features12','feature1']['rows_c']

c    666
Name: (features12, feature1), dtype: int64

df5.loc['rows_c']

	features12		features34
	feature1	feature2	feature3	feature4
c	666	NaN	777	888

df5.loc['rows_ab','features12']

	feature1	feature2
a	444	111.0
b	444	222.0

Dropping a level¶

df5.columns = df5.columns.droplevel(level = 0); 
df5

		feature1	feature2	feature3	feature4
rows_ab	a	444	111.0	555	333
rows_ab	b	444	222.0	333	999
rows_c	c	666	NaN	777	888

Transposing¶

swaping columns and indices

df6 = df5.T
df6

	rows_ab		rows_c
	a	b	c
feature1	444.0	444.0	666.0
feature2	111.0	222.0	NaN
feature3	555.0	333.0	777.0
feature4	333.0	999.0	888.0

Stacking and unstacking levels¶

expanding the lowest column level as the lowest index

df7 = df6.stack()
df7

		rows_ab	rows_c
feature1	a	444.0	NaN
	b	444.0	NaN
	c	NaN	666.0
feature2	a	111.0	NaN
feature2	b	222.0	NaN
feature3	a	555.0	NaN
	b	333.0	NaN
	c	NaN	777.0
feature4	a	333.0	NaN
	b	999.0	NaN
	c	NaN	888.0

df8 = df7.unstack()
df8

	rows_ab			rows_c
	a	b	c	a	b	c
feature1	444.0	444.0	NaN	NaN	NaN	666.0
feature2	111.0	222.0	NaN	NaN	NaN	NaN
feature3	555.0	333.0	NaN	NaN	NaN	777.0
feature4	333.0	999.0	NaN	NaN	NaN	888.0

If we call unstack again, we end up with a Series object

df9 = df8.unstack()
df9

rows_ab  a  feature1    444.0
            feature2    111.0
            feature3    555.0
            feature4    333.0
         b  feature1    444.0
            feature2    222.0
            feature3    333.0
            feature4    999.0
         c  feature1      NaN
            feature2      NaN
            feature3      NaN
            feature4      NaN
rows_c   a  feature1      NaN
            feature2      NaN
            feature3      NaN
            feature4      NaN
         b  feature1      NaN
            feature2      NaN
            feature3      NaN
            feature4      NaN
         c  feature1    666.0
            feature2      NaN
            feature3    777.0
            feature4    888.0
dtype: float64

Accessing rows¶

df

	feature1	feature2	feature3	feature4
a	1	5	NaN	9.0
b	2	4	8.0	10.0
c	3	6	7.0	NaN

df.loc["c"] #access the c row

feature1    3.0
feature2    6.0
feature3    7.0
feature4    NaN
Name: c, dtype: float64

df.iloc[2] #access the 2nd column by integer location

feature1    3.0
feature2    6.0
feature3    7.0
feature4    NaN
Name: c, dtype: float64

slice of rows

df.iloc[1:3] # by integer location

	feature1	feature2	feature3	feature4
b	2	4	8.0	10.0
c	3	6	7.0	NaN

slice rows using boolean array

df[np.array([True, False, True])]

	feature1	feature2	feature3	feature4
a	1	5	NaN	9.0
c	3	6	7.0	NaN

with boolean expressions:

df[df["feature2"] <= 5]

	feature1	feature2	feature3	feature4
a	1	5	NaN	9.0
b	2	4	8.0	10.0

Adding and removing columns¶

df

	feature1	feature2	feature3	feature4
a	1	5	NaN	9.0
b	2	4	8.0	10.0
c	3	6	7.0	NaN

df['feature5'] = 5 - df['feature2'] #adding a column
df['feature6'] = df['feature3'] > 5
df  

	feature1	feature2	feature3	feature4	feature5	feature6
a	1	5	NaN	9.0	0	False
b	2	4	8.0	10.0	1	True
c	3	6	7.0	NaN	-1	True

del df['feature6']
df

	feature1	feature2	feature3	feature4	feature5
a	1	5	NaN	9.0	0
b	2	4	8.0	10.0	1
c	3	6	7.0	NaN	-1

df["feature6"] = pd.Series({"a": 1, "b": 51, "c":1}) 
df

	feature1	feature2	feature3	feature4	feature5	feature6
a	1	5	NaN	9.0	0	1
b	2	4	8.0	10.0	1	51
c	3	6	7.0	NaN	-1	1

df.insert(1, "feature1b", [0,1,2]) #insert in the location of the 1st column
df

	feature1	feature1b	feature2	feature3	feature4	feature5	feature6
a	1	0	5	NaN	9.0	0	1
b	2	1	4	8.0	10.0	1	51
c	3	2	6	7.0	NaN	-1	1

Assigning new columns¶

create a new DataFrame with new columns

df

	feature1	feature1b	feature2	feature3	feature4	feature5	feature6
a	1	0	5	NaN	9.0	0	1
b	2	1	4	8.0	10.0	1	51
c	3	2	6	7.0	NaN	-1	1

df10 = df.assign(feature0 = df["feature1"] * df["feature2"] )
df10.assign(feature1 = df10["feature1"] +1)
df10

	feature1	feature1b	feature2	feature3	feature4	feature5	feature6	feature0
a	1	0	5	NaN	9.0	0	1	5
b	2	1	4	8.0	10.0	1	51	8
c	3	2	6	7.0	NaN	-1	1	18

Evaluating an expression¶

df = df[['feature1','feature2','feature3','feature4']]
df.eval("feature1 + feature2 ** 2")

a    26
b    18
c    39
dtype: int64

df

	feature1	feature2	feature3	feature4
a	1	5	NaN	9.0
b	2	4	8.0	10.0
c	3	6	7.0	NaN

use inplace=True to modify a DataFrame

df.eval("feature3 = feature1 + feature2 ** 2", inplace=True)
df

	feature1	feature2	feature3	feature4
a	1	5	26	9.0
b	2	4	18	10.0
c	3	6	39	NaN

use a local or global variable in an expression by prefixing it with @

threshold = 30
df.eval("feature3 = feature1 + feature2 ** 2 > @threshold", inplace=True)
df

	feature1	feature2	feature3	feature4
a	1	5	False	9.0
b	2	4	False	10.0
c	3	6	True	NaN

Querying a DataFrame¶

The query method lets you filter a DataFrame

df.query("feature1 > 2 and feature2 == 6")

	feature1	feature2	feature3	feature4
c	3	6	True	NaN

Sorting a DataFrame¶

df

	feature1	feature2	feature3	feature4
a	1	5	False	9.0
b	2	4	False	10.0
c	3	6	True	NaN

df.sort_index(ascending=False)

	feature1	feature2	feature3	feature4
c	3	6	True	NaN
b	2	4	False	10.0
a	1	5	False	9.0

df.sort_index(axis=0, inplace=True) #by row
df

	feature1	feature2	feature3	feature4
a	1	5	False	9.0
b	2	4	False	10.0
c	3	6	True	NaN

df.sort_index(axis=1, inplace=True) #by column
df

	feature1	feature2	feature3	feature4
a	1	5	False	9.0
b	2	4	False	10.0
c	3	6	True	NaN

df.sort_values(by="feature2", inplace=True)
df

/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

	feature1	feature2	feature3	feature4
b	2	4	False	10.0
a	1	5	False	9.0
c	3	6	True	NaN

Operations on DataFrame¶

a = np.array([[1,2,3],[4,5,6],[7,8,9]])
df = pd.DataFrame(a, columns=["q", "w", "e"], index=["a","b","c"])
df

	q	w	e
a	1	2	3
b	4	5	6
c	7	8	9

np.sqrt(df)

	q	w	e
a	1.000000	1.414214	1.732051
b	2.000000	2.236068	2.449490
c	2.645751	2.828427	3.000000

df + 1 #broadcasting

	q	w	e
a	2	3	4
b	5	6	7
c	8	9	10

df >= 5

	q	w	e
a	False	False	False
b	False	True	True
c	True	True	True

df.mean(), df.std(), df.max(), df.sum()

(q    4.0
 w    5.0
 e    6.0
 dtype: float64, q    3.0
 w    3.0
 e    3.0
 dtype: float64, q    7
 w    8
 e    9
 dtype: int64, q    12
 w    15
 e    18
 dtype: int64)

df

	q	w	e
a	1	2	3
b	4	5	6
c	7	8	9

(df > 2).all() #checks whether all values are True or not

q    False
w    False
e     True
dtype: bool

(df > 2).all(axis = 0) #executed vertically (on each column)

q    False
w    False
e     True
dtype: bool

(df > 2).all(axis = 1) #execute the horizontally (on each row).

a    False
b     True
c     True
dtype: bool

(df == 8).any(axis = 1)

a    False
b    False
c     True
dtype: bool

df - df.mean() 

	q	w	e
a	-3.0	-3.0	-3.0
b	0.0	0.0	0.0
c	3.0	3.0	3.0

df - df.values.mean() # subtracts the global mean elementwise

	q	w	e
a	-4.0	-3.0	-2.0
b	-1.0	0.0	1.0
c	2.0	3.0	4.0

Handling missing data¶

df10

	feature1	feature1b	feature2	feature3	feature4	feature5	feature6	feature0
a	1	0	5	NaN	9.0	0	1	5
b	2	1	4	8.0	10.0	1	51	8
c	3	2	6	7.0	NaN	-1	1	18

df11 = df10.fillna(0)
df11

	feature1	feature1b	feature2	feature3	feature4	feature5	feature6	feature0
a	1	0	5	0.0	9.0	0	1	5
b	2	1	4	8.0	10.0	1	51	8
c	3	2	6	7.0	0.0	-1	1	18

df11.loc["d"] = np.nan
df11.fillna(0,inplace=True)
df11

	feature1	feature1b	feature2	feature3	feature4	feature5	feature6	feature0
a	1.0	0.0	5.0	0.0	9.0	0.0	1.0	5.0
b	2.0	1.0	4.0	8.0	10.0	1.0	51.0	8.0
c	3.0	2.0	6.0	7.0	0.0	-1.0	1.0	18.0
d	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

Aggregating with groupby¶

Similar to SQL, pandas allows to compute over groups.

df5 = pd.DataFrame({
    "feature1": {"a": 3, "b": 11, "c": 14, 'd':4},
    "feature2": {"a": 2, "b": 2, "c": 4, 'd':4},
    "feature3": {"a": 32, "b": 4, "c": 3, 'd':35},
    "feature4": {"a": 5, "b": 11, "c": 2, 'd':13}
})

df5

	feature1	feature2	feature3	feature4
a	3	2	32	5
b	11	2	4	11
c	14	4	3	2
d	4	4	35	13

df5.groupby("feature2").mean()

	feature1	feature3	feature4
feature2
2	7.0	18.0	8.0
4	9.0	19.0	7.5

Pivot tables¶

pivot tables allows for quick summarization

df9 = df8.stack().reset_index()
df9

	level_0	level_1	rows_ab	rows_c
0	feature1	a	444.0	NaN
1	feature1	b	444.0	NaN
2	feature1	c	NaN	666.0
3	feature2	a	111.0	NaN
4	feature2	b	222.0	NaN
5	feature3	a	555.0	NaN
6	feature3	b	333.0	NaN
7	feature3	c	NaN	777.0
8	feature4	a	333.0	NaN
9	feature4	b	999.0	NaN
10	feature4	c	NaN	888.0

pd.pivot_table(df9, index="level_0", aggfunc=np.mean)

	rows_ab	rows_c
level_0
feature1	444.0	666.0
feature2	166.5	NaN
feature3	444.0	777.0
feature4	666.0	888.0

pd.pivot_table(df9, index="level_0", values=["rows_ab"], aggfunc=np.max)

	rows_ab
level_0
feature1	444.0
feature2	222.0
feature3	555.0
feature4	999.0

Functions¶

df = np.fromfunction(lambda x,y: (x+y)%7*11, (1000, 10))
big_df = pd.DataFrame(df, columns=list("1234567890"))
big_df.head(5)

	1	2	3	4	5	6	7	8	9	0
0	0.0	11.0	22.0	33.0	44.0	55.0	66.0	0.0	11.0	22.0
1	11.0	22.0	33.0	44.0	55.0	66.0	0.0	11.0	22.0	33.0
2	22.0	33.0	44.0	55.0	66.0	0.0	11.0	22.0	33.0	44.0
3	33.0	44.0	55.0	66.0	0.0	11.0	22.0	33.0	44.0	55.0
4	44.0	55.0	66.0	0.0	11.0	22.0	33.0	44.0	55.0	66.0

big_df[big_df % 3 == 0] = np.nan
big_df.insert(3,"feature1", 999)
big_df.head(5)

	1	2	3	feature1	4	5	6	7	8	9	0
0	NaN	11.0	22.0	999	NaN	44.0	55.0	NaN	NaN	11.0	22.0
1	11.0	22.0	NaN	999	44.0	55.0	NaN	NaN	11.0	22.0	NaN
2	22.0	NaN	44.0	999	55.0	NaN	NaN	11.0	22.0	NaN	44.0
3	NaN	44.0	55.0	999	NaN	NaN	11.0	22.0	NaN	44.0	55.0
4	44.0	55.0	NaN	999	NaN	11.0	22.0	NaN	44.0	55.0	NaN

big_df.tail(5)

	1	2	3	feature1	4	5	6	7	8	9	0
995	11.0	22.0	NaN	999	44.0	55.0	NaN	NaN	11.0	22.0	NaN
996	22.0	NaN	44.0	999	55.0	NaN	NaN	11.0	22.0	NaN	44.0
997	NaN	44.0	55.0	999	NaN	NaN	11.0	22.0	NaN	44.0	55.0
998	44.0	55.0	NaN	999	NaN	11.0	22.0	NaN	44.0	55.0	NaN
999	55.0	NaN	NaN	999	11.0	22.0	NaN	44.0	55.0	NaN	NaN

big_df.describe()

	1	2	3	feature1	4	5	6	7	8	9	0
count	572.00000	572.00000	571.000000	1000.0	571.000000	572.00000	571.000000	571.000000	572.00000	572.00000	571.000000
mean	33.00000	33.00000	33.038529	999.0	33.019264	33.00000	32.980736	32.961471	33.00000	33.00000	33.038529
std	17.40775	17.40775	17.398586	0.0	17.416910	17.40775	17.416910	17.398586	17.40775	17.40775	17.398586
min	11.00000	11.00000	11.000000	999.0	11.000000	11.00000	11.000000	11.000000	11.00000	11.00000	11.000000
25%	19.25000	19.25000	22.000000	999.0	16.500000	19.25000	16.500000	16.500000	19.25000	19.25000	22.000000
50%	33.00000	33.00000	44.000000	999.0	44.000000	33.00000	22.000000	22.000000	33.00000	33.00000	44.000000
75%	46.75000	46.75000	49.500000	999.0	49.500000	46.75000	49.500000	44.000000	46.75000	46.75000	49.500000
max	55.00000	55.00000	55.000000	999.0	55.000000	55.00000	55.000000	55.000000	55.00000	55.00000	55.000000

Save and Load¶

big_df.to_csv("my_df.csv")
big_df.to_csv("my_df.xlsx")

df0 = pd.read_csv("my_df.csv", index_col=0)

Combining DataFrames¶

city_loc = pd.DataFrame(
    [
        ["CA", "Murrieta", 33.569443, -117.202499],
        ["NY", "Cohoes", 42.774475, -73.708412],
        ["NY", "Rye", 40.981613,	-73.691925],
        ["CA", "Ojai", 34.456936,	-119.254440],
        ["AL", "Jasper", 33.834263,	-87.280708]
    ], columns=["state", "city", "lat", "lon"])

city_loc

	state	city	lat	lon
0	CA	Murrieta	33.569443	-117.202499
1	NY	Cohoes	42.774475	-73.708412
2	NY	Rye	40.981613	-73.691925
3	CA	Ojai	34.456936	-119.254440
4	AL	Jasper	33.834263	-87.280708

city_pop = pd.DataFrame(
    [
        [112941, "Murrieta", "California"],
        [16684, "Cohoes", "New York"],
        [15820, "Rye", "New York"],
        [13649, "Jasper", "Alabama"]
    ], index=[3,4,5,6], columns=["population", "city", "state"])
city_pop

	population	city	state
3	112941	Murrieta	California
4	16684	Cohoes	New York
5	15820	Rye	New York
6	13649	Jasper	Alabama

pd.merge(left=city_loc, right=city_pop, on="city", how="inner")

	state_x	city	lat	lon	population	state_y
0	CA	Murrieta	33.569443	-117.202499	112941	California
1	NY	Cohoes	42.774475	-73.708412	16684	New York
2	NY	Rye	40.981613	-73.691925	15820	New York
3	AL	Jasper	33.834263	-87.280708	13649	Alabama

all_cities = pd.merge(left=city_loc, right=city_pop, on="city", how="outer")
all_cities

	state_x	city	lat	lon	population	state_y
0	CA	Murrieta	33.569443	-117.202499	112941.0	California
1	NY	Cohoes	42.774475	-73.708412	16684.0	New York
2	NY	Rye	40.981613	-73.691925	15820.0	New York
3	CA	Ojai	34.456936	-119.254440	NaN	NaN
4	AL	Jasper	33.834263	-87.280708	13649.0	Alabama

pd.merge(left=city_loc, right=city_pop, on="city", how="right")

	state_x	city	lat	lon	population	state_y
0	CA	Murrieta	33.569443	-117.202499	112941	California
1	NY	Cohoes	42.774475	-73.708412	16684	New York
2	NY	Rye	40.981613	-73.691925	15820	New York
3	AL	Jasper	33.834263	-87.280708	13649	Alabama

Concatenation¶

result_concat = pd.concat([city_pop, city_loc], ignore_index=True)
result_concat

	population	city	state	lat	lon
0	112941.0	Murrieta	California	NaN	NaN
1	16684.0	Cohoes	New York	NaN	NaN
2	15820.0	Rye	New York	NaN	NaN
3	13649.0	Jasper	Alabama	NaN	NaN
4	NaN	Murrieta	CA	33.569443	-117.202499
5	NaN	Cohoes	NY	42.774475	-73.708412
6	NaN	Rye	NY	40.981613	-73.691925
7	NaN	Ojai	CA	34.456936	-119.254440
8	NaN	Jasper	AL	33.834263	-87.280708

pd.concat([city_loc, city_pop], join="inner", ignore_index=True)

	state	city
0	CA	Murrieta
1	NY	Cohoes
2	NY	Rye
3	CA	Ojai
4	AL	Jasper
5	California	Murrieta
6	New York	Cohoes
7	New York	Rye
8	Alabama	Jasper

pd.concat([city_loc.set_index("city"), city_pop.set_index("city")], axis=1, ignore_index=True)

	0	1	2	3	4
Murrieta	CA	33.569443	-117.202499	112941.0	California
Cohoes	NY	42.774475	-73.708412	16684.0	New York
Rye	NY	40.981613	-73.691925	15820.0	New York
Ojai	CA	34.456936	-119.254440	NaN	NaN
Jasper	AL	33.834263	-87.280708	13649.0	Alabama

city_loc.append(city_pop) #another way to append, but works on a copy and returns the modified copy.

	state	city	lat	lon	population
0	CA	Murrieta	33.569443	-117.202499	NaN
1	NY	Cohoes	42.774475	-73.708412	NaN
2	NY	Rye	40.981613	-73.691925	NaN
3	CA	Ojai	34.456936	-119.254440	NaN
4	AL	Jasper	33.834263	-87.280708	NaN
3	California	Murrieta	NaN	NaN	112941.0
4	New York	Cohoes	NaN	NaN	16684.0
5	New York	Rye	NaN	NaN	15820.0
6	Alabama	Jasper	NaN	NaN	13649.0

Categories¶

city_eco = city_pop.copy()
city_eco["edu"] = [17, 17, 34, 20]
city_eco

	population	city	state	edu
3	112941	Murrieta	California	17
4	16684	Cohoes	New York	17
5	15820	Rye	New York	34
6	13649	Jasper	Alabama	20

city_eco["education"] = city_eco["edu"].astype('category')
city_eco["education"].cat.categories

Int64Index([17, 20, 34], dtype='int64')

city_eco["education"].cat.categories = ["College", "High School", "Basic"]
city_eco

	population	city	state	edu	education
3	112941	Murrieta	California	17	College
4	16684	Cohoes	New York	17	College
5	15820	Rye	New York	34	Basic
6	13649	Jasper	Alabama	20	High School

city_eco.sort_values(by="education", ascending=False)

	population	city	state	edu	education
5	15820	Rye	New York	34	Basic
6	13649	Jasper	Alabama	20	High School
3	112941	Murrieta	California	17	College
4	16684	Cohoes	New York	17	College

OR/MS in Python

Python Review¶

Introduction¶

Basics of Python¶

Basic data types¶

Numbers¶

Booleans¶

Strings¶

Lists¶

Slicing¶

Loops¶

List comprehensions:¶

Dictionaries¶

Sets¶

Tuples¶

Functions¶

Classes¶

Numpy¶

Arrays¶

Array indexing¶

Datatypes¶

Array math¶

Broadcasting¶

Pandas¶

Series objects¶

Index labels¶

Initializing from a dict¶

Automatic alignment¶

DataFrame objects¶

Multi-indexing¶

Dropping a level¶

Transposing¶

Stacking and unstacking levels¶

Accessing rows¶

Adding and removing columns¶

Assigning new columns¶

Evaluating an expression¶

Querying a DataFrame¶

Sorting a DataFrame¶

Operations on DataFrame¶

Handling missing data¶

Aggregating with groupby¶

Pivot tables¶

Functions¶

Save and Load¶

Combining DataFrames¶

Concatenation¶

Categories¶