GAMS Jupyter Notebooks allow to use notebook technology in combination with GAMS. If you just want to learn GAMS there are probably better ways doing this. Notebooks allow you to combine GAMS and Python. The former works great with well structured data and optimization models, while the latter is very rich in features to retrieve, manipulate, and visualize data that comes in all sort of ways. Combining GAMS and Python in a notebook it is relatively easy to tell an optimization story with text, data, graphs, math, and models.
The GAMS Jupyter Notebook builds on top of the Python 3 kernel. So by default the notebook cells are Python cells. Cells can be turned into GAMS cells, i.e. cells with GAMS syntax, using the Jupyter magic facility (first line in a cell is %%gams). GAMS magic commands enable GAMS support in Python Jupyter notebooks. Beside running GAMS code, it is possible to transfer data between GAMS and Python. In order to enable the GAMS magic commands, it is required to load the extension gams_magic:
%load_ext gams_magic
There are a few other useful command in connection with running GAMS in a Jupyter notebook. Some transformation functions for pandas dataframes useful for exchange with GAMS have been collected in the DataTransform.ipynb. The next cell will execute that notebook and make such data transformation function, e.g. gt_from2dim (see below) available in this notebook. %%capture captures the output from the execution of the notebook and does not clutter your output.
%%capture
%run ~/share/DataTransform.ipynb
Somehow one output from a cell is sometimes not enough, e.g. if you want to display a couple of tables. The display function allows you to do this but needs to imported. As an example, we display a Python list:
from IPython.display import display
display([1,2,3])
Running GAMS code can be done by using either %gams (line magic) or %%gams (cell magic). While %gams can be used for running a single line of GAMS code, %%gams makes the whole cell a GAMS cell.
%gams set i;
%%gams
set j;
parameter p(i,j);
parameter p2(i,j);
The GAMS compiler and execution system has been adjusted so one can run a GAMS cell multiple time, even if it contains a declaration or an equation definition, which is normally not possible in the GAMS system. The execution of the next two cells does not create a problem, which mimics the execution, modification, and reexecution of a cell.
%%gams
set i / peter,paul,mary /, j / A,B,C /;
parameter p2(i,j) / set.i.set.j 1 /;
%%gams
set i / i1*i5 /, j /j1*j5 /;
parameter p2(i,j) / set.i.set.j 1 /;
You won't see any output from a GAMS cell (unless there is a solve executed in the cell, see below). All output goes to the log and lst file. If you really need to see this you can use magic command %gams_log and %gams_lst to display the content of the log and listing file of the most recent GAMS execution. The next cell displays the content of listing file of the last run GAMS cell or line magic. The -e only display the section of the listing file associated with the execution:
%gams display p2;
%gams_lst -e
There is a limit to the execution, modification, and reexecution of GAMS cells. If the type or the dimensionality of a symbol changes, you will need to execute the notebook from scratch and do a controlled reset of the entire GAMS database via %gams_reset. For example, since we declared parameter p2 already over (i,j) we cannot change our mind and redeclare p2 as parameter p2(i,i,j):
%gams parameter p2(i,i,j);
This will give you a compilation error and an exception in the cell execution (uncomment the line in the next cell to do so):
#%gams parameter p2(i,i,j);
#%gams_lst
With a %gams_reset we can reset the GAMS database and can declare symbols with a different type and domain/dimension. All other things in the GAMS database are gone, too. So we need to redeclare the sets i and j, too. The state of the GAMS database is kept in various files that can easily clutter your directory. The %gams_cleanup call helps you to clean the directory of temporary files. The option -k keeps the most recent GAMS database, hence the %gams_cleanup -k is a save call anywhere in your notebook.
%gams_reset
%gams set i,j; parameter p(i,j), p2(i,i,j);
%gams_cleanup -k
%gams_push transfers data from Python to GAMS. Supported data types for pushing data are lists, pandas.DataFrame and numpy arrays:
# Define Python lists with data
i = ['i1', 'i2', 'i3']
j = ['j1', 'j2']
p = [('i1', 'j1', 1.1), ('i1', 'j2', 2.2), ('i2', 'j1', 3.3), ('i2', 'j2', 4.4), ('i3', 'j1', 5.5), ('i3', 'j2', 6.6)]
%gams_push i j p
As mentioned above the execution of a %%gams cell or %gams and %gams_push line magic does not produce output. If one wants to verify that the data ended up in GAMS we can display the symbols in GAMS and output the corresponding part of the listing file:
%gams display i,j,p;
%gams_lst -e
The next cell turns a Python list into a pandas.DataFrame, multiplies the value by 2 and displays the dataframe with IPythons's display. We actually display the transformed p2 (via function gt_pivot2d found in the DataTransformation notebook run at the top of the notebook), so the table looks nicer. Next, we sends the pandas.DataFrame down to GAMS via the %gams_push command. Via the GAMS display and the output of the relevant part of the listing file we see that the %gams_push succeeded:
import pandas as pd
# turn the Python list p into a pandas.Dataframe p2 and send this down to GAMS
pp = pd.DataFrame(p)
# multiply the value by 2:
pp[2] = 2*pp[2]
# display a nicer version of the dataframe:
display(gt_pivot2d(pp))
%gams parameter pp(i,j)
%gams_push pp
%gams display pp;
%gams_lst -e
When using numpy arrays in order to push data into GAMS, the data is assumed to be dense. The correspondng sets are defined automatically from 1..n, 1..m, etc depending on the data that is pushed.
import numpy as np
data = [[[1.1,-1.1], [2.2,-2.2]], [[3.3,-3.3], [4.4,-4.4]], [[5.5,-5.5], [6.6,-6.6]]]
p3 = np.array(data)
%gams set i, j, k; parameter p3(i,j,k);
%gams_push p3
%gams display i,j,k,p3;
%gams_lst -e
The line magic %gams_pull transfers data from GAMS to Python in different formats. Supported formats are lists (default), pandas.DataFrame and numpy arrays. The following example pulls the sets i, j, and parameter p3 from GAMS into lists. For multi-dimensional symbols the records become Python tuples. Currently, the renaming functionality %gams_pull gamsSym=pySymbol is not yet supported.
%gams_pull p3 i j
display(i,j,p3)
The switch -d will populate pandas.DataFrames instead of lists with the GAMS data. The dataframes that are pushed into or pulled from GAMS have a very specific layout. There is a record index and the GAMS domains show up as columns in the dataframe. For parameters, there is an extra value column. For variables and equations we find extra columns level, marginal, lower, upper, and scale. The method head() used in the IPython display provides only the first 5 records of a pandas.DataFrame:
%gams variable x(i) /1.l 1, 2.m 3/;
%gams_pull -d i j p3 x
display(i, j, p3.head(), x)
The data transformation functions available from DataTransformations.ipynb help to convert between this format and formats more suitable for display of other transformations in Python. The following lines give a quick overview of the transformation functionality:
%gams parameter r(i,j); r(i,j) = uniformInt(1,10);
%gams_pull -d r
display(r,gt_pivot2d(r),gt_from2dim(gt_pivot2d(r),['i','j','value']))
The switch -n will populate numpy arrays instead lists with the GAMS parameters. This format works with parameters only! The GAMS data will be dropped into a dense numpy array:
%gams parameter p4(i,j) / 1.1 1, 2.2 2 /;
%gams_pull -n p4
display(p4)
%gams_cleanup or %gams_cleanup -k if you want to continue working in this session%cat path/to/listing.lst or %gams_lst. The path of the listing file can be found in the last line of the output of a failing cell.%gams Parameter p5(i,j,l);
#%cat /home/share/jupyter_examples/gamsJupyter16.lst
%gams_lst
%gams_cleanup