This Notebook will dive more into the Action class

Try me out interactively with: Binder

It is recommended to have a look at the 0_basic_functionalities and 1_Observation_Agents notebooks before getting into this one.

Objectives

This notebook will cover the basics of how to best use the Action class to modify the powergrid efficiently. Indeed, there are multiple concepts behind this class that may not be very clear at first glance.

This notebook will be focused on the manipulation of Actions from an expert system point of view in order to demonstrate how a desired action is fundamentally taken in the Grid2Op environment. We will give a more detailed example later of a more automatic way to handle actions (for example using machine learning, in the notebook 3_TrainingAnAgent).

In [ ]:
import os
import sys
import grid2op
In [ ]:
res = None
try:
    from jyquickhelper import add_notebook_menu
    res = add_notebook_menu()
except ModuleNotFoundError:
    print("Impossible to automatically add a menu / table of content to this notebook.\nYou can download \"jyquickhelper\" package with: \n\"pip install jyquickhelper\"")
res

I) A few comments on actions

It is recommended to have built locally the documentation, and to refer to the action pages of the documentation or to the Action.py file for a more detailed view on these two classes below.

I.A) "change" vs "set"

To manipulate a powergrid, we decided to introduce two distinct (yet close) concepts that will affect the objects differently:

  • change will:
    • connect a powerline if it was disconnected, or reconnect if it was connected
    • assign the object to the second bus if it was connected to bus 1 or assign it to bus 1 if it was connected to bus 2
  • set will:
    • connect a powerline (regardless of its previous status) or disconnect it (regardless of its previous status)
    • assign the object to a specific bus (regardless of its previous bus and status -- for powerline)

This is another change compared to the previous pypownet implementation, where only the change concept was implemented. Having these two things helps understand what is really going on in the powergrid and allows to represent better the intention of the Agent, especially in the debugging phase.

Of course, it is perfectly possible to use only the change capability and thus being closer to the original implementation.

Let's give a "concrete" example to highlight the differences between these two methods. Suppose we have a substation with 5 elements:

  • the origin of powerline $l_1$
  • the extremity of powerline $l_2$
  • the extremity of powerline $l_3$
  • a load $c_1$
  • a generator $g_1$

Let's also assume the original configuration (before the action is applied, ie the configuration of the observation at time t) is:

Object Name Original Bus Original Status
$l_1$ (origin) 1 connected
$l_2$ (extremity) 2 connected
$l_3$ (extremity) NA disconnected
$c_1$ 1 NA
$g_1 $ 2 NA

Let's say:

  • action $a_1$ is "change the status of the origin of powerline $l_1$". After applying this action, the status of the origin of powerline $l_1$ is: "disconnected", because it was "connected".
  • action $a_2$ is "change the status of the extremity of powerline $l_3$". After applying this action, the status of the extremity of powerline $l_3$ is: "connected", because it was "disconnected". ***.
  • action $a_3$ is "set the bus of $c_1$ to bus 1". It is equivalent to doing nothing since $c_1$ is already connected on bus 1.
  • action $a_4$ is "set the bus of $g_1$ to bus 1". It will change the bus of $g_1$ and assign it to bus 1.

* NB Another breaking change compared to the pypownet implementation is the introduction of "ambiguous" actions. When an action can be understood in different ways or have different meanings, then it will be replaced by a "do nothing" action by the environment.

In this situation, the previous actions are equivalent to:

  • $a_1 \to$ set status of $l_1$ to "disconnected"
  • $a_2 \to$ set status of $l_3$ to "connected"
  • $a_3 \to$ do nothing
  • $a_4 \to$ change bus of $g_1$

I.B) "Ambiguous" action

When some actions are "ambiguous" it means that they cannot be properly and / or univocally interpreted. These actions will be ignored if attempted to be used on the powergrid. This will be equivalent to doing nothing.

For a detailed list of ambiguous actions, the documentation is the only official source. Only some examples are presented here. The documentation is available at _check_for_ambiguity.

An action can be ambiguous in the following cases:

  • It affects the "injections" in an incorrect way:

    • it tries to modify a load (setting active or reactive values) that doesn't exist in the powergrid
    • it sets the values of a generator that doesn't exist (setting its voltage setpoint or active production)
  • It affects the "powerlines" in an incorrect manner:

    • it tries to change the status or to assign to a specific bus a powerline that doesn't exist
    • somes lines are reconnected but the action doesn't specify on which bus. It used to be like this, but now an "automaton" directly coded in the environment will assign the previous bus if that is the case. You can reconnect a powerline without specifying on which bus and in that case the last known buses when the powerline was connected will be used.
    • for some powerline, the status is both changed and set
  • It has an ambiguous behaviour concerning the topology of some substations

    • the state of some bus for some element is both changed and set
    • the bus is trying to be modified (set or changed) on a object that is not present in the powergrid

II) Action space

IMPORTANT NOTICE Each Agent has its own action space attribute that can be called with self.action_space. This is the only recommended way to create a valid Action. Using its constructor is strongly NOT recommended, as it requires a deep knowledge of all the elements in the powergrid, as well as their names, their type, the order in which they are used in the backend, etc. For performance reasons, no sanity check are performed to make sure the action that would be created this way is compatible with the backend.

In the next cell, we retrieve the action space used by the Agent.

Two main classes are useful when dealing with Actions in Grid2Op. The Action class is the most basic one. The ActionHelper is a tool that helps create and manipulate some actions.

As in most of our notebooks, we start by creating an Environment. We will use the case14_fromfile provided as an example. We will then extract the complete action space (action_space, that is, the actions that can be performed on the power grid) as a dictionary. When a specific action such as change or set is needed to be performed then we can apply this change to the action_space dictionary by accessing it with the relevant key as discussed below.

In [ ]:
# import the usefull class
env = grid2op.make(test=True)
action_space = env.action_space

As opposed to the previous plateform, pypownet, there are no restrictions on actions in Grid2Op. Generally speaking, an Action can modify production, loads, topology, etc. By default though, an Action that an Agent can perform is a TopologyAction, which is a specific type of action. A TopologyAction can :

  • change the status of a powerline (reconnect / disconnect it)
  • change the way the objects (end of a powerline, generator or load) are interconnected at substations.

We will focus on this class in this notebook.

Then best way to get an action is to give a dictionnary to the "action space" of the player. For example, to get the "do nothing" action, you can just pass the empty dictionnary.

In [ ]:
do_nothing = action_space({})

II.A) Line status modification

a) Modify multiple powerlines

If you want to change (or set) the status of most of the powerlines, you can create a vector having the same size as the number of powerlines in the grid, and pass it to the dictionnary with the relevant keys ("set_line_status", "change_line_status") and the proper values (a vector of booleans to set line status, or a vector of integers to change line status). An example is given below. Note that this example only modifies the status of a few powerlines, but this way of defining actions is more adapted when you need to modify the status of many powerlines.

The following code will:

  • change the status of powerlines with id 0,1,2
  • set the status "connected" for powerline with id 3,4
  • set the status "disconnected" for powerlines with id 5 and 6

Each powerline / load / generator has an ID and a name.

Keep in mind that the IDs of the powerlines are 0, 1, 2, ..., env.n_line - 1 (where env.n_line is the number of lines in the environment). The same goes for the loads and generators.

On the other hand, the names are more human-friendly identifiers for the different objects in the grid.

Therefore, for any vector containing information about the powerlines, the variable relative to the powerline of id i can be accessed or modified with vector[i] since it is the i+1th powerline in the grid.

This is how we will proceed in the next cell. However, the IDs are easy to use but less meaningful for a human (since it is the names of the powerlines that we see) and sometimes, when inspecting an observation for example, we may want to look at a specific powerline by specifying its name. This will be covered later in the notebook 7_PlottingCapabilities.

In [ ]:
change_status = action_space.get_change_line_status_vect()
change_status[0] = True
change_status[1] = True
change_status[2] = True

set_status = action_space.get_set_line_status_vect()
set_status[3] = 1
set_status[4] = 1
set_status[5] = -1
set_status[6] = -1

this_first_act = action_space({"set_line_status": set_status, "change_line_status": change_status})

NB even if it can handle different types, for performance reasons it's better to follow the type of data mentionned below : The dictionnary values should be:

  • for change actions, booleans :
    • True means "change"
    • False means "don't change"
  • for set actions, integers :
    • 0 means "do nothing"
    • 1 means "connect it"
    • -1 means "disconnect it"

For convenience, an Action object can be inspected easily by using the print method. It will summarize on which object it has an impact:

In [ ]:
print(this_first_act)
In [ ]:
this_first_act.is_ambiguous()

NB This action is ambiguous so it cannot be implemented on the powergrid. Indeed, powerlines 3 and 4 are reconnected, but we don't specify on which bus! Implementing this action on a grid will be equivalent to doing nothing.

b) Modify a single powerline, or a few powerlines

It's not always convenient to manipulate all the status of all the powerlines, or change it. For mor convenience, it's possible to modify only a few of them. The syntax is the following.

In [ ]:
the_same_act = action_space({"set_line_status": [(3,1), (4,1), (5,-1), (6,-1)],
                             "change_line_status": [0,1,2]
                            })
print(the_same_act)

We can check that the two actions are indeed equal:

In [ ]:
the_same_act == this_first_act

II.B) Substations reconfiguration / topology changes

One of the interesting aspects of Grid2Op is to be able to modify the topology of the powergrid. In other words it allows to reconfigure the way the objects (generators, loads, end of powerlines) are interconnected at their substations.

Comparable to the status change, topological change can be interpreted in two disctinct manners, as described above. Topologycal changes include some of the most interesting interactions with the environment.

In this section we study how to modify the topology of the powergrid.

The underlying way to represent the topology is through a integer vector, having the same dimension as the number of objects of the grid. For each object in the grid, this vector tells on which bus it's connected. Manipulating this vector can be done, but is absolutely not handy. We present here the way to change the topology through the helped, which can be done more easily.

To set the bus to which a load is connected, it is recommended to do:

In [ ]:
set_bus_load_0 = action_space({"set_bus": {"loads_id": [(0,2)]}})
print(set_bus_load_0)

To change the bus, a similar interface can be used:

In [ ]:
change_bus_load_0 = action_space({"change_bus": {"loads_id": [0]}})
print(change_bus_load_0)

The API is really similar for generator:

In [ ]:
change_bus_gen_0_and_1 = action_space({"change_bus": {"generators_id": [0,1]}})
set_bus_gen_3_and_4 = action_space({"set_bus": {"generators_id": [(3,2), (4,2)]}})
print(set_bus_gen_3_and_4)

The same goes for each ends of the powerlines:

In [ ]:
change_bus_lines_or_0 = action_space({"change_bus": {"lines_or_id": [0]}})
set_bus_lines_or_4 = action_space({"set_bus": {"lines_or_id": [(3,2)]}})
change_bus_lines_ex_15 = action_space({"change_bus": {"lines_ex_id": [15]}})
set_bus_lines_ex_18 = action_space({"set_bus": {"lines_ex_id": [(18,2)]}})
print(set_bus_lines_ex_18)

When reconnecting a powerline, if the bus to which this powerline is reconnected is not specified, the action is ambiguous and thus will not be implemented. It is, in that case, recommended to use the reconnect_powerline method as followed:

In [ ]:
reconnecting_line_1 = action_space.reconnect_powerline(line_id=1,bus_or=1,bus_ex=1)
print(reconnecting_line_1)

Finally, if you know how many objects are in a substation, you can also modify all their buses in one call:

In [ ]:
reconfigure_substation_id_1 = action_space({"set_bus": {"substations_id": [(1, (1,2,2,1,1,2))]}})

In the above code we knew that the substations of id 1 had 6 elements, and we assign to the elements of id 1,2 and 5 [second, third and sixth] the bus 2 and the others to bus 1.

Using this way does not make explicit which objects are modified. You need to know beforehand that the element of id 0 of this substation is the extremity of powerline 0, the element of id 1 of this substation is the origin of powerline 1 etc. We explain how to do so in the next paragraph.

Know the order of the objects on a substation

You can also have a look at the grid_objects_types object that is a representation, as a numpy array, of which element is connected to each substation. It counts as many rows as there are elements on the powergrid (here 56: 5 generators, 11 loads and 20 lines each having two ends - making 5 + 11 + 2*20 = 56 objects) and exactly 5 columns. The first column of this matrix of this matrix represents the id of the substation at which the object represented by the row is located.

Then the next 4 columns each account for an object type: either it's a load (column 1) or a generator (column 2) or the extremity of a powerline (column 3) or the extremity of a powerline (column 4). Let's take some example instead of getting lost in the details:

If at row 0 of grid_objects_types i see [ 0, -1, -1, 0, -1] this means that: the object of id 0 (remember we were looking for the row 0) is connected to substation 0 (first element of this vector). This is not a load nor a generator (there are -1 in columns 1 and 2 encoding for load and generators respectively. This is not an extremity of a powerline (there is a -1 in the 4th column encoding for the powerline extremity). We see a "0" on column 3, encoding for "powerline origin" this means that this is the origin of the powerline of id 0.

On the row 1 of grid_objects_types we see [ 0, -1, -1, 1, -1]. With the same reasoning we know that it corresponds to the origin of powerline 1 that is connected to substation 0.

On the row 2 we see [ 0, -1, 4, -1, -1]. This means that the third object of the substation 0 is the generator of id 4.

And a last example is the following. The row 12 of grid_objects_types is [ 2, 1, -1, -1, -1] (see below) this means that the 12th element of the grid is: connected to substation 2, is a load, this load has id 1.

In [ ]:
action_space.grid_objects_types[12]

This representation in terms of grid_objects_types is rather explicit for a human but is also particularly suited to look for information informatically thanks to numpy indexing. For example if you want to know where the loads are located you can do:

In [ ]:
is_load = action_space.grid_objects_types[:,action_space.LOA_COL] != -1
print("The substation with at least one load are: {}".format(
    action_space.grid_objects_types[is_load,action_space.SUB_COL]))

is_sub1 = action_space.grid_objects_types[:, action_space.SUB_COL] == 1
is_gen = action_space.grid_objects_types[:, action_space.GEN_COL] != -1
print("The generator ids connected to substations 1 are: {}".format(
    action_space.grid_objects_types[is_sub1 & is_gen, action_space.GEN_COL]))

print("The object connected to substation 1 are: {}".format(
     action_space.grid_objects_types[is_sub1,:]))
# etc.

For human reader however, the method "get_obj_connect_to" is clearer. This gives the following information:

In [ ]:
action_space.get_obj_connect_to(substation_id=1)

In this case it means on the substation 1 are connected:

  • the load of id 1
  • the generator of id 0
  • the origin of powerlines 2, 3 and 4
  • the extremity of powerline 0 And this substation counts 6 elements.

Note that this does not allow easily to know which object is assign to which bus with the action reconfigure_substation_id_1

II.C) Indexed actions

For convenience, it might be better sometimes to change the bus of an object from its name instead of its ID in case the ID is not known. Grid2Op allows to do that, but only for changing or setting a bus. These methods take longer than the methods shown above. If they are used at all, it's recommended NOT to use them for training an Agent. Their main goal aims at debugging and / or understanding the behaviour of an Agent.

These methods are:

Please refer to the official documentation for a complete description of their behaviour. To sum up, we can use them this way:

In [ ]:
my_act = action_space.set_bus("gen_1_0", # mandatory name of the element
                               new_bus=2, # mandatory the new bus to connect it too
                               type_element="gen", # optional the type of the element, one of "line", "gen" or "load"
                               previous_action=None  # optional: if you want to combine multiple action, you can do it with this
                              )
print(my_act)
In [ ]:
action_space.set_bus("1_3_3", # mandatory name of the element
                     extremity="or", # mandatory, which extrmity to change
                     new_bus=2, # mandatory the new bus to connect it too
                     type_element="line", # optional the type of the element, one of "line", "gen" or "load"
                     previous_action=my_act  # optional: if you want to combine multiple action, you can do it with this
                    )
print(my_act)

II.D) Important note on the powerline status

Direct impact on the status

As of grid2op version 1.2.0 the behavior of the platform with respect to line status modification has been clarified and rationalized (we hope).

The powerline status (connected / disconnected) can now be affected in two different ways:

  • by setting / changing its status directly (using the "set_line_status" or "change_line_status" keyword).
  • [NEW] by modifying the bus on any of the end (origin or extremity) of a powerline

In that later case, the behavior is:

  • if the bus of a powerline end (origin or extremity) is "set" to -1 and not modified at the other and if the powerline was connected, it will disconnect this powerline
  • if the bus of a powerline end (origin or extremity) is "set" to 1 or 2 at one end and not modified at the other and if the powerline was connected, it will reconnect the powerline
  • if the bus of a powerline end (origin or extremity) is "set" to -1 at one end and set to 1 or 2 at its other end the action is ambiguous.

The way to compute the impact of the action has also been adjusted to reflect these changes.

In the table below we try to summarize all the possible actions and their impact on the powerline. This table is made considering that "LINE_ID" is an id of a powerline and "SUB_OR" is the id of the origin of the substation. If a status is 0 it means the powerlines is disconnected, if the status is 1 it means it is connected.

action original status final status substations affected line status affected
{"set_line_status": [(LINE_ID, -1)]} 1 0 None LINE_ID
{"set_line_status": [(LINE_ID, +1)]} 1 1 None LINE_ID
{"set_line_status": [(LINE_ID, -1)]} 0 0 None LINE_ID
{"set_line_status": [(LINE_ID, +1)]} 0 1 None LINE_ID
{"change_line_status": [LINE_ID]} 1 0 None LINE_ID
{"change_line_status": [LINE_ID]} 0 1 None LINE_ID
{"set_bus": {"lines_or_id": [(LINE_ID, -1)]}} 1 0 None LINE_ID
{"set_bus": {"lines_or_id": [(LINE_ID, -1)]}} 0 0 SUB_OR None
{"set_bus": {"lines_or_id": [(LINE_ID, 2)]}} 1 1 SUB_OR None
{"set_bus": {"lines_or_id": [(LINE_ID, 2)]}} 0 1 None LINE_ID
{"change_bus": {"lines_or_id": [LINE_ID]}} 1 1 SUB_OR None
{"change_bus": {"lines_or_id": [LINE_ID]}} 0 0 SUB_OR None

of course we could have set {"set_bus": {"lines_ex_id": [(LINED_ID, 2)]}} (ie the extermity bus of the powerline) and it would have the same impact on its status. Assign the powerline extremity to bus 1 (instead of bus 2) by sending the dictionnaries {"set_bus": {"lines_or_id": [(LINED_ID, 1)]}} or {"set_bus": {"lines_ex_id": [(LINED_ID, 1)]}} would also lead to the same results.

Consequences on the bus at both powerline ends

In grid2op there is a convention that if an object is disconnected, then it is assigned to bus "-1". For a powerline this entails that a status changed affects the bus of

As we explained in the previous paragraph, some action on one end of a powerline can reconnect a powerline or disconnect it. This means they modify the bus of both the extremity of the powerline.

Here is a table summarizing how the buses are impacted. We denoted by "PREVIOUS_OR" the last bus at which the origin end of the powerline was connected and "PREVIOUS_EX" the last bus at which the extremity end of the powerline was connected. Note that for clarity when something is not modified by the action we decided to write on the table "not modified" (this entails that after this action, if the powerline is connected then "new origin bus" is "PREVIOUS_OR" and "new extremity bus" is "PREVIOUS_EX"). We remind the reader that "-1" encode for a disconnected object.

action original status final status new origin bus new extremity bus
{"set_line_status": [(LINE_ID, -1)]} 1 0 -1 -1
{"set_line_status": [(LINE_ID, +1)]} 1 1 Not modified Not modified
{"set_line_status": [(LINE_ID, -1)]} 0 0 Not modified Not modified
{"set_line_status": [(LINE_ID, +1)]} 0 1 PREVIOUS_OR PREVIOUS_EX
{"change_line_status": [LINE_ID]} 1 0 -1 -1
{"change_line_status": [LINE_ID]} 0 1 PREVIOUS_OR PREVIOUS_EX
{"set_bus": {"lines_or_id": [(LINE_ID, -1)]}} 1 0 -1 -1
{"set_bus": {"lines_or_id": [(LINE_ID, -1)]}} 0 0 Not modified Not modified
{"set_bus": {"lines_or_id": [(LINE_ID, 2)]}} 1 1 2 Not modified
{"set_bus": {"lines_or_id": [(LINE_ID, 2)]}} 0 1 2 PREVIOUS_EX
{"change_bus": {"lines_or_id": [(LINE_ID, 2)]}} 1 1 * Not modified
{"change_bus": {"lines_or_id": [(LINE_ID, 2)]}} 0 0 Not modified Not modified

* means that this bus is affected: if it was on bus 1 it moves on bus 2 and vice versa.

Extra note on the actions

As we can see here, but this is true in general in grid2op, each action you do is labeled to have at least an impact on some object. Some actions are considered to have impact on powerline status, some have impact on substations.

This is always the case regardless of the actual impact of this action on the powergrid. For example, if we look at the second and third row of the tables above we notice that:

  • these actions does not impact the grid (nothing is modified)
  • these actions are counted towards action affecting "powerline".

This is particularly important for what we call "cooldown" (see the notebook 0_Introduction section Introduction of "operational constraints" in grid2op for more information). You action can trigger a cooldown (preventing future action on the same element) while not impacting the grid at all.

III) Manipulating Actions

It is absolutely NOT recommended to use Actions outside of the action space.