Java allows you to configure the maximum heap size and various other settings with command line parameters when the JVM is started. BeakerX allows you to add parameters to the JVM with a control panel added to Jupyter's tree view.
The settings are stored in your ~/.jupyter/beakerx.json file.
Use the following cells to confirm the heap size and property values. Note that if you change the values in the UI, the changes will not take effect on any running kernel until it is restarted.
Runtime.getRuntime().maxMemory()/1e9
System.getProperty("myproperty")