# "Living in a noisy world...", using James Powell's (dutc) rwatch module¶

Goal : I want to write a context manager for Python 3.5+, so that inside the context manager, every number is seen noisy (with a white Gaussian noise, for instance).

It will be like being drunk, except that your it will be my Python interpretor and not me !

For instance, I will like to have this feature:

>>> x = 120193
>>> print(x)
120193
>>> np.random.seed(1234)
>>> with WhiteNoise():
>>>    print(x)
120193.47143516373249306


First, we will need numpy to have some random number generator, as numpy.random.normal to have some 1D Gaussian noise.

In [1]:
import numpy as np

In [2]:
np.random.seed(1234)
np.random.normal()

Out[2]:
0.47143516373249306

Then, the core part will be to install and import James Powell (dutc) rwatch module. If you don't have it installed :

1. Be sure to have CPython 3.5. rwatch patches the CPython eval loop, so it's fixed to specific versions of Python & the author lazy about keeping it updated.
2. Then pip install dutc-rwatch. It should work, but it fails for me.
3. (alternative) You can just cd /tmp/ && git clone https://github.com/dutc/rwatch && cd rwatch/src/ && make and copy the rwatch.so dynamic library wherever you need...
In [5]:
%%bash
tmpdir=$(mktemp -d) cd$tmpdir
git clone https://github.com/dutc/rwatch
cd rwatch/src/
make
ls -larth ./rwatch.so
file ./rwatch.so
# cp ./rwatch.so /where/ver/you/need/  # ~/publis/notebook/ for me

gcc python3-config --cflags python3-config --includes -fPIC -DPy_BUILD_CORE -c -o ceval.o ceval.c
gcc python3-config --cflags python3-config --includes -Lpython3-config --prefix/lib -Wl,--export-dynamic -fPIC -shared -o rwatch.so rwatch.c hook.c ceval.o -ldl python3-config --libs
-rwxrwxr-x 1 lilian lilian 453K mai   14 01:20 ./rwatch.so
./rwatch.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=f41ef744646e6247119bd702039725fc3a1c8e66, not stripped

Clonage dans 'rwatch'...
In file included from ceval.c:300:0:
ceval_gil.h:114:46: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
^~~~
ceval_gil.h:114:46: note: (near initialization for ‘gil_last_holder._value’)
ceval_gil.h: In function ‘create_gil’:
ceval_gil.h:145:5: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
_Py_atomic_store_relaxed(&gil_last_holder, NULL);
^~~~~~~~~~~~~~~~~~~~~~~~
ceval_gil.h: In function ‘drop_gil’:
ceval_gil.h:181:9: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
_Py_atomic_store_relaxed(&gil_last_holder, tstate);
^~~~~~~~~~~~~~~~~~~~~~~~
ceval_gil.h: In function ‘take_gil’:
ceval_gil.h:243:9: warning: initialization makes integer from pointer without a cast [-Wint-conversion]
_Py_atomic_store_relaxed(&gil_last_holder, tstate);
^~~~~~~~~~~~~~~~~~~~~~~~


Anyhow, if rwatch is installed, we can import it, and it enables two new functions in the sys module:

In [6]:
import rwatch
from sys import setrwatch, getrwatch

setrwatch({})  # clean any previously installed rwatch
getrwatch()

Out[6]:
{}

Finally, we need the collections module and its defaultdict magical datastructure.

In [7]:
from collections import defaultdict


## Defining a debugging context manager, just to try¶

This is the first example given in James presentation at PyCon Canada 2016.

We will first define and add a rwatch for just one object, let say a variable x, and then for any object using defaultdict. From there, writing a context manager that enables this feature only locally is easy.

### Watching just one object¶

1. Write the function, that needs two argument frame, obj, and should return obj,
2. Install it...
3. Check it!
In [15]:
def basic_view(frame, obj):
print("Python saw the object {} from frame {}".format(obj, frame))
return obj

In [16]:
x = "I am alive!"

In [17]:
setrwatch({
id(x): basic_view
})

In [18]:
print(x)

Python saw the object I am alive! from frame <frame object at 0x7fe313a70b88>
Python saw the object I am alive! from frame <frame object at 0x7fe32c005258>
I am alive!


That's awesome, it works!

### Can we delete the rwatch ?¶

Sure!

In [19]:
def delrwatch(idobj):
getrwatch().pop(idobj, None)

In [20]:
print(x)
delrwatch(id(x))
print(x)  # no more rwatch on this!
print(x)  # no more rwatch on this!

Python saw the object I am alive! from frame <frame object at 0x7fe313a7b9a8>
Python saw the object I am alive! from frame <frame object at 0x7fe32c01a558>
I am alive!
Python saw the object I am alive! from frame <frame object at 0x7fe313a7b7c8>
I am alive!
I am alive!


We can also delete rwatches that are not defined, without a failure:

In [21]:
y = "I am Zorro !"
print(y)
delrwatch(y)  # No issue!
print(y)

I am Zorro !
I am Zorro !


### More useful debuggin information¶

What is this frame thing? It is described in the documentation of the inspect module.

We can actually use it to display some useful information about the object and where was it called etc.

In [22]:
from inspect import getframeinfo

def debug_view(frame, obj):
info = getframeinfo(frame)
print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
return obj

In [23]:
setrwatch({})
setrwatch({
id(x): debug_view
})
getrwatch()

Out[23]:
{140613264650544: <function __main__.debug_view>}
In [24]:
print(x)

- Access to 'I am alive!' (@0x7fe313b0b130) at <ipython-input-24-81745ac23551>:1:<module>
I am alive!


That can be quite useful!

### Watching any object¶

We can actually pass a defaultdict to the setrwatch function, so that any object will have a rwatch!

Warning: obviously, this will crazily slowdown your interpreter!

So let be cautious, and only deal with strings here.

But I want to be safe, so it will only works if the frame indicate that the variable does not come from a file.

In [25]:
setrwatch({})

In [26]:
def debug_view_for_str(frame, obj):
if isinstance(obj, str):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
return obj

In [27]:
setrwatch(defaultdict(lambda: debug_view_for_str))

In [28]:
print(x)

- Access to 'I am alive!' (@0x7fe313b0b130) at <ipython-input-28-81745ac23551>:1:<module>
I am alive!


Clearly, there is a lot of strings involved, mainly because this is a notebook and not the simple Python interpreter, so filtering on the info.filename as I did was smart.

In [29]:
setrwatch({})


But obviously, having this for all objects is incredibly verbose!

In [30]:
def debug_view_for_any_object(frame, obj):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
print(msg.format(obj, hex(id(obj)), info.filename, info.lineno, info.function))
return obj


Let check that one, on a very simple example (which runs in less than 20 micro seconds):

In [52]:
print(x)
%time 123 + 134

I am alive!
CPU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 17.6 µs

Out[52]:
257
In [53]:
setrwatch({})
setrwatch(defaultdict(lambda: debug_view_for_any_object))
print(x)
%time 123 + 134
setrwatch({})

- Access to None (@0x55feae7373a0) at <ipython-input-53-734484c968c6>:2:<module>
I am alive!
- Access to <bound method InteractiveShell.get_ipython of <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7fe33b56d0b8>> (@0x7fe33b57d548) at <ipython-input-53-734484c968c6>:4:<module>
CPU times: user 8 ms, sys: 0 ns, total: 8 ms
Wall time: 10.8 ms

Out[53]:
257
- Access to None (@0x55feae7373a0) at <ipython-input-53-734484c968c6>:4:<module>


It seems to work very well!

But it slows down everything, obviously the filtering takes time (for every object!) Computing 123 + 134 = 257 took about 10 miliseconds! That's just CRAZY!

### A first context manager to have debugging for one object¶

It would be nice to be able to turn on and off this debugging tool whenever you want.

Well, it turns out that context managers are exactly meant for that! They are simple classes with just a __enter__() and __exit__() special methods.

First, let us write a context manager to debug ONE object.

In [54]:
class InspectThisObject(object):
def __init__(self, obj):
self.idobj = id(obj)

def __enter__(self):
getrwatch()[self.idobj] = debug_view

def __exit__(self, exc_type, exc_val, exc_tb):
delrwatch(self.idobj)


We can check it:

In [55]:
z = "I am Batman!"
print(z)

with InspectThisObject(z):
print(z)

print(z)

I am Batman!
I am Batman!
I am Batman!


The first debug information shows line 5, which is the line where print(z) is.

### A second context manager to debug any object¶

Easy:

In [56]:
class InspectAllObjects(object):
def __init__(self):
pass

def __enter__(self):
setrwatch(defaultdict(lambda: debug_view_for_any_object))

def __exit__(self, exc_type, exc_val, exc_tb):
setrwatch({})


It will probably break everything in the notebook, but works in a basic Python interpreter.

In [57]:
with InspectAllObjects():
print(0)

- Access to None (@0x55feae7373a0) at <ipython-input-56-7a0dc9fc43f8>:6:__enter__
0


The 5th debug information printed is Access to 0 (@0xXXX) at <ipython-input-41-XXX>:2:<module>, showing the access in line #2 of the constant 0.

In [58]:
with InspectAllObjects():
print("Luke -- I have a father? Yay! Let's eat cookies together!")

- Access to None (@0x55feae7373a0) at <ipython-input-56-7a0dc9fc43f8>:6:__enter__
- Access to "Luke -- I have a father? Yay! Let's eat cookies together!" (@0x7fe312b71f80) at <ipython-input-58-f38f4c4a3188>:3:<module>
Luke -- I have a father? Yay! Let's eat cookies together!


We also see here the None and {} objects being given to the context manager (see the __enter__ method at first, and __exit__ at the end).

## Defining a context manager to add white noise¶

Basically, we will do as above, but instead of debug information, a white noise sampled from a Normal distribution (i.e., $\sim \mathcal{N}(0, 1)$) will be added to any number.

### Capturing any numerical value¶

To capture both integers and float numbers, the numbers.Number abstract class is useful.

In [59]:
from numbers import Number


### Adding a white noise for numbers¶

This is very simple.

But I want to be safe, so it will only works if the frame indicate that the number does not come from a file, as previously.

In [60]:
def add_white_noise_to_numbers(frame, obj):
if isinstance(obj, Number):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
return obj + np.random.normal()
return obj


Let us try it out!

In [63]:
np.random.seed(1234)
setrwatch({})
x = 1234
print(x)
print(x)  # huhoww, that's noisy!
print(10 * x + x + x**2)  # and noise propagate!
setrwatch({})
print(x)
print(10 * x + x + x**2)

1234
1234.4714351637324
1535547.9958216753
1234
1536330


It seems to work! Let's do it for any number then...

... Sadly, it's actually breaking the interpreter, which obviously has to have access to non-noisy constants and numbers to work !

We can lower the risk by only adding noise to complex numbers. I guess the interpreter doesn't need complex numbers, write?

In [64]:
def add_white_noise_to_complex(frame, obj):
if isinstance(obj, complex):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
return obj + np.random.normal() + np.random.normal() * 1j
return obj

In [65]:
np.random.seed(1234)
setrwatch({})
y = 1234j
print(y)
print(y)  # huhoww, that's noisy!
setrwatch({})
print(y)

1234j
(0.47143516373249306+1232.8090243052936j)
1234j


Awesome!

« Now, the real world is non noisy, but the complex one is! »

That's one sentence I thought I would never say!

### WhiteNoiseComplex context manager¶

To stay cautious, I only add noise to complex numbers.

In [67]:
class WhiteNoiseComplex(object):
def __init__(self):
pass

def __enter__(self):

def __exit__(self, exc_type, exc_val, exc_tb):
setrwatch({})


And it works as expected:

In [69]:
np.random.seed(120193)
print(120193, 120193j)
with WhiteNoiseComplex():
print(120193, 120193j)  # Huhoo, noisy!
print(120193, 120193j)

print(0*1j)
with WhiteNoiseComplex():
print(0*1j)  # Huhoo, noisy!
print(0*1j)

120193 120193j
120193 (-1.4079099824891002+120192.24863460941j)
120193 120193j
0j
(1.5232719431619033-1.3234485447145865j)
0j


## Defining a generic noisy context manager¶

This will be a very simple change from the previous one, by letting the Noisy class accept any noisy function, which takes obj and return a noisy version of obj, only for complex-valued objects.

In [71]:
class Noisy(object):
def __init__(self, noise):
if isinstance(obj, complex):
info = getframeinfo(frame)
if '<stdin>' in info.filename or '<ipython-' in info.filename:
return noise(obj)
return obj

def __enter__(self):
setrwatch(defaultdict(lambda: self.rwatch))

def __exit__(self, exc_type, exc_val, exc_tb):
setrwatch({})

In [73]:
print(1j)
with Noisy(lambda obj: obj + np.random.normal()):
print(1j)
print(1j)

1j
(0.5850862095293242+1j)
1j

In [74]:
print(1j)
with Noisy(lambda obj: obj * np.random.normal()):
print(1j)
print(1j)

1j
(-0-0.582727025433589j)
1j

In [75]:
print(1j)
with Noisy(lambda obj: obj + np.random.normal(10, 0.1) + np.random.normal(10, 0.1) * 1j):
print(1j)
print(1j)

1j
(10.101006520511199+11.184818402012283j)
1j


## Conclusion¶

Clearly, that was a BAD idea, and not so useful.

But it was interesting!

I don't have any idea of a context where this could be useful, but still!