from __future__ import division, print_function, unicode_literals
import numpy as np
import tensorflow as tf
from tensorflow import keras
/Users/ageron/.virtualenvs/ml/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6 return f(*args, **kwds)
Some operations (like tf.reduce_sum()
) have favor performance over precision, and their outputs may vary slightly across runs. To get reproducible results, make sure TensorFlow runs on the CPU:
import os
os.environ["CUDA_VISIBLE_DEVICES"]=""
Because floats have limited precision, the order of execution matters:
2. * 5. / 7.
1.4285714285714286
2. / 7. * 5.
1.4285714285714284
You should make sure TensorFlow runs your ops on a single thread:
config = tf.ConfigProto(intra_op_parallelism_threads=1,
inter_op_parallelism_threads=1)
with tf.Session(config=config) as sess:
#... this will run single threaded
pass
The thread pools for all sessions are created when you create the first session, so all sessions in the rest of this notebook will be single-threaded:
with tf.Session() as sess:
#... also single-threaded!
pass
hash()
function¶print(set("Try restarting the kernel and running this again"))
print(set("Try restarting the kernel and running this again"))
{'n', 'k', 'l', 'h', 'r', 'a', 'i', 't', 'd', 's', 'g', 'T', ' ', 'y', 'e', 'u'} {'n', 'k', 'l', 'h', 'r', 'a', 'i', 't', 'd', 's', 'g', 'T', ' ', 'y', 'e', 'u'}
Since Python 3.3, the result will be different every time, unless you start Python with the PYTHONHASHSEED
environment variable set to 0
:
PYTHONHASHSEED=0 python
>>> print(set("Now the output is stable across runs"))
{'n', 'b', 'h', 'o', 'i', 'a', 'r', 't', 'p', 'N', 's', 'c', ' ', 'l', 'e', 'w', 'u'}
>>> exit()
PYTHONHASHSEED=0 python
>>> print(set("Now the output is stable across runs"))
{'n', 'b', 'h', 'o', 'i', 'a', 'r', 't', 'p', 'N', 's', 'c', ' ', 'l', 'e', 'w', 'u'}
Alternatively, you could set this environment variable system-wide, but that's probably not a good idea, because this automatic randomization was introduced for security reasons.
Unfortunately, setting the environment variable from within Python (e.g., using os.environ["PYTHONHASHSEED"]="0"
) will not work, because Python reads it upon startup. For Jupyter notebooks, you have to start the Jupyter server like this:
PYTHONHASHSEED=0 jupyter notebook
if os.environ.get("PYTHONHASHSEED") != "0":
raise Exception("You must set PYTHONHASHSEED=0 when starting the Jupyter server to get reproducible results.")
import random
random.seed(42)
print(random.random())
print(random.random())
print()
random.seed(42)
print(random.random())
print(random.random())
0.6394267984578837 0.025010755222666936 0.6394267984578837 0.025010755222666936
import numpy as np
np.random.seed(42)
print(np.random.rand())
print(np.random.rand())
print()
np.random.seed(42)
print(np.random.rand())
print(np.random.rand())
0.3745401188473625 0.9507143064099162 0.3745401188473625 0.9507143064099162
TensorFlow's behavior is more complex because of two things:
import tensorflow as tf
tf.set_random_seed(42)
rnd = tf.random_uniform(shape=[])
with tf.Session() as sess:
print(rnd.eval())
print(rnd.eval())
print()
with tf.Session() as sess:
print(rnd.eval())
print(rnd.eval())
0.63789964 0.8774011 0.63789964 0.8774011
Every time you reset the graph, you need to set the seed again:
tf.reset_default_graph()
tf.set_random_seed(42)
rnd = tf.random_uniform(shape=[])
with tf.Session() as sess:
print(rnd.eval())
print(rnd.eval())
print()
with tf.Session() as sess:
print(rnd.eval())
print(rnd.eval())
0.63789964 0.8774011 0.63789964 0.8774011
If you create your own graph, it will ignore the default graph's seed:
tf.reset_default_graph()
tf.set_random_seed(42)
graph = tf.Graph()
with graph.as_default():
rnd = tf.random_uniform(shape=[])
with tf.Session(graph=graph):
print(rnd.eval())
print(rnd.eval())
print()
with tf.Session(graph=graph):
print(rnd.eval())
print(rnd.eval())
0.5718187 0.6233171 0.32140207 0.46593904
You must set its own seed:
graph = tf.Graph()
with graph.as_default():
tf.set_random_seed(42)
rnd = tf.random_uniform(shape=[])
with tf.Session(graph=graph):
print(rnd.eval())
print(rnd.eval())
print()
with tf.Session(graph=graph):
print(rnd.eval())
print(rnd.eval())
0.63789964 0.8774011 0.63789964 0.8774011
If you set the seed after the random operation is created, the seed has no effet:
tf.reset_default_graph()
rnd = tf.random_uniform(shape=[])
tf.set_random_seed(42) # BAD, NO EFFECT!
with tf.Session() as sess:
print(rnd.eval())
print(rnd.eval())
print()
tf.set_random_seed(42) # BAD, NO EFFECT!
with tf.Session() as sess:
print(rnd.eval())
print(rnd.eval())
0.087068915 0.6322479 0.17158246 0.2868148
You can also set a seed for each individual random operation. When you do, it is combined with the graph seed into the final seed used by that op. The following table summarizes how this works:
Graph seed | Op seed | Resulting seed |
---|---|---|
None | None | Random |
graph_seed | None | f(graph_seed, op_index) |
None | op_seed | f(default_graph_seed, op_seed) |
graph_seed | op_seed | f(graph_seed, op_seed) |
f()
is a deterministic function.op_index = graph._last_id
when there is a graph seed, different random ops without op seeds will have different outputs. However, each of them will have the same sequence of outputs at every run.In eager mode, there is a global seed instead of graph seed (since there is no graph in eager mode).
tf.reset_default_graph()
rnd1 = tf.random_uniform(shape=[], seed=42)
rnd2 = tf.random_uniform(shape=[], seed=42)
rnd3 = tf.random_uniform(shape=[])
with tf.Session() as sess:
print(rnd1.eval())
print(rnd2.eval())
print(rnd3.eval())
print(rnd1.eval())
print(rnd2.eval())
print(rnd3.eval())
print()
with tf.Session() as sess:
print(rnd1.eval())
print(rnd2.eval())
print(rnd3.eval())
print(rnd1.eval())
print(rnd2.eval())
print(rnd3.eval())
0.95227146 0.95227146 0.55099714 0.8960779 0.8960779 0.54318357 0.95227146 0.95227146 0.6398845 0.8960779 0.8960779 0.24617589
In the following example, you may think that all random ops will have the same random seed, but rnd3
will actually have a different seed:
tf.reset_default_graph()
tf.set_random_seed(42)
rnd1 = tf.random_uniform(shape=[], seed=42)
rnd2 = tf.random_uniform(shape=[], seed=42)
rnd3 = tf.random_uniform(shape=[])
with tf.Session() as sess:
print(rnd1.eval())
print(rnd2.eval())
print(rnd3.eval())
print(rnd1.eval())
print(rnd2.eval())
print(rnd3.eval())
print()
with tf.Session() as sess:
print(rnd1.eval())
print(rnd2.eval())
print(rnd3.eval())
print(rnd1.eval())
print(rnd2.eval())
print(rnd3.eval())
0.4163028 0.4163028 0.96100175 0.033224702 0.033224702 0.17637014 0.4163028 0.4163028 0.96100175 0.033224702 0.033224702 0.17637014
Tip: in a Jupyter notebook, you probably want to set the random seeds regularly so that you can come back and run the notebook from there (instead of from the beginning) and still get reproducible outputs.
random.seed(42)
np.random.seed(42)
tf.set_random_seed(42)
If you use the Estimators API, make sure to create a RunConfig
and set its tf_random_seed
, then pass it to the constructor of your estimator:
my_config = tf.estimator.RunConfig(tf_random_seed=42)
feature_cols = [tf.feature_column.numeric_column("X", shape=[28 * 28])]
dnn_clf = tf.estimator.DNNClassifier(hidden_units=[300, 100], n_classes=10,
feature_columns=feature_cols,
config=my_config)
WARNING:tensorflow:Using temporary folder as model directory: /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmp2xxrubio INFO:tensorflow:Using config: {'_model_dir': '/var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmp2xxrubio', '_tf_random_seed': 42, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x11dba7da0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
Let's try it on MNIST:
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()
X_train = X_train.astype(np.float32).reshape(-1, 28*28) / 255.0
y_train = y_train.astype(np.int32)
Unfortunately, the numpy_input_fn
does not allow us to set the seed when shuffle=True
, so we must shuffle the data ourself and set shuffle=False
.
indices = np.random.permutation(len(X_train))
X_train_shuffled = X_train[indices]
y_train_shuffled = y_train[indices]
input_fn = tf.estimator.inputs.numpy_input_fn(
x={"X": X_train_shuffled}, y=y_train_shuffled, num_epochs=10, batch_size=32, shuffle=False)
dnn_clf.train(input_fn=input_fn)
INFO:tensorflow:Calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmp2xxrubio/model.ckpt. INFO:tensorflow:loss = 73.945915, step = 1 INFO:tensorflow:global_step/sec: 348.999 INFO:tensorflow:loss = 21.020527, step = 101 (0.287 sec) INFO:tensorflow:global_step/sec: 431.365 INFO:tensorflow:loss = 8.926933, step = 201 (0.232 sec) INFO:tensorflow:global_step/sec: 438.11 INFO:tensorflow:loss = 2.3184745, step = 301 (0.228 sec) INFO:tensorflow:global_step/sec: 437.696 INFO:tensorflow:loss = 10.654381, step = 401 (0.228 sec) INFO:tensorflow:global_step/sec: 452.808 INFO:tensorflow:loss = 4.2829914, step = 501 (0.221 sec) INFO:tensorflow:global_step/sec: 450.062 INFO:tensorflow:loss = 2.497019, step = 601 (0.222 sec) INFO:tensorflow:global_step/sec: 451.86 INFO:tensorflow:loss = 3.9215999, step = 701 (0.221 sec) INFO:tensorflow:global_step/sec: 442.86 INFO:tensorflow:loss = 3.8031044, step = 801 (0.226 sec) INFO:tensorflow:global_step/sec: 444.581 INFO:tensorflow:loss = 3.9209557, step = 901 (0.225 sec) INFO:tensorflow:global_step/sec: 439.603 INFO:tensorflow:loss = 5.506338, step = 1001 (0.227 sec) INFO:tensorflow:global_step/sec: 444.545 INFO:tensorflow:loss = 2.6690354, step = 1101 (0.225 sec) INFO:tensorflow:global_step/sec: 445.176 INFO:tensorflow:loss = 6.559507, step = 1201 (0.225 sec) INFO:tensorflow:global_step/sec: 443.365 INFO:tensorflow:loss = 5.707597, step = 1301 (0.225 sec) INFO:tensorflow:global_step/sec: 447.822 INFO:tensorflow:loss = 2.8663762, step = 1401 (0.223 sec) INFO:tensorflow:global_step/sec: 446.664 INFO:tensorflow:loss = 8.945509, step = 1501 (0.224 sec) INFO:tensorflow:global_step/sec: 439.348 INFO:tensorflow:loss = 0.55089784, step = 1601 (0.228 sec) INFO:tensorflow:global_step/sec: 446.299 INFO:tensorflow:loss = 0.38362896, step = 1701 (0.224 sec) INFO:tensorflow:global_step/sec: 444.764 INFO:tensorflow:loss = 3.543582, step = 1801 (0.225 sec) INFO:tensorflow:global_step/sec: 446.166 INFO:tensorflow:loss = 2.2667365, step = 1901 (0.224 sec) INFO:tensorflow:global_step/sec: 446.722 INFO:tensorflow:loss = 1.4623648, step = 2001 (0.224 sec) INFO:tensorflow:global_step/sec: 446.295 INFO:tensorflow:loss = 1.8456242, step = 2101 (0.224 sec) INFO:tensorflow:global_step/sec: 446.23 INFO:tensorflow:loss = 5.7156425, step = 2201 (0.224 sec) INFO:tensorflow:global_step/sec: 445.893 INFO:tensorflow:loss = 7.4256153, step = 2301 (0.224 sec) INFO:tensorflow:global_step/sec: 437.007 INFO:tensorflow:loss = 0.99539256, step = 2401 (0.229 sec) INFO:tensorflow:global_step/sec: 441.825 INFO:tensorflow:loss = 0.7815061, step = 2501 (0.226 sec) INFO:tensorflow:global_step/sec: 446.57 INFO:tensorflow:loss = 1.0568533, step = 2601 (0.224 sec) INFO:tensorflow:global_step/sec: 454.471 INFO:tensorflow:loss = 0.5744381, step = 2701 (0.220 sec) INFO:tensorflow:global_step/sec: 445.43 INFO:tensorflow:loss = 1.6298534, step = 2801 (0.225 sec) INFO:tensorflow:global_step/sec: 448.248 INFO:tensorflow:loss = 3.52447, step = 2901 (0.223 sec) INFO:tensorflow:global_step/sec: 448.121 INFO:tensorflow:loss = 0.66434807, step = 3001 (0.223 sec) INFO:tensorflow:global_step/sec: 444.626 INFO:tensorflow:loss = 2.757729, step = 3101 (0.225 sec) INFO:tensorflow:global_step/sec: 447.545 INFO:tensorflow:loss = 5.4983215, step = 3201 (0.223 sec) INFO:tensorflow:global_step/sec: 444.307 INFO:tensorflow:loss = 3.3224907, step = 3301 (0.225 sec) INFO:tensorflow:global_step/sec: 447.91 INFO:tensorflow:loss = 3.9577734, step = 3401 (0.223 sec) INFO:tensorflow:global_step/sec: 445.955 INFO:tensorflow:loss = 0.41272426, step = 3501 (0.224 sec) INFO:tensorflow:global_step/sec: 445.494 INFO:tensorflow:loss = 0.7215781, step = 3601 (0.225 sec) INFO:tensorflow:global_step/sec: 447.421 INFO:tensorflow:loss = 0.6514178, step = 3701 (0.224 sec) INFO:tensorflow:global_step/sec: 449.321 INFO:tensorflow:loss = 0.15730175, step = 3801 (0.222 sec) INFO:tensorflow:global_step/sec: 446.755 INFO:tensorflow:loss = 1.3917842, step = 3901 (0.224 sec) INFO:tensorflow:global_step/sec: 434.755 INFO:tensorflow:loss = 0.21514821, step = 4001 (0.230 sec) INFO:tensorflow:global_step/sec: 411.974 INFO:tensorflow:loss = 3.3500838, step = 4101 (0.243 sec) INFO:tensorflow:global_step/sec: 414.927 INFO:tensorflow:loss = 0.26301342, step = 4201 (0.241 sec) INFO:tensorflow:global_step/sec: 415.294 INFO:tensorflow:loss = 0.307297, step = 4301 (0.241 sec) INFO:tensorflow:global_step/sec: 428.301 INFO:tensorflow:loss = 1.3765923, step = 4401 (0.233 sec) INFO:tensorflow:global_step/sec: 454.43 INFO:tensorflow:loss = 0.8025591, step = 4501 (0.220 sec) INFO:tensorflow:global_step/sec: 438.331 INFO:tensorflow:loss = 0.8588423, step = 4601 (0.228 sec) INFO:tensorflow:global_step/sec: 443.007 INFO:tensorflow:loss = 5.4857945, step = 4701 (0.226 sec) INFO:tensorflow:global_step/sec: 447.41 INFO:tensorflow:loss = 1.2880125, step = 4801 (0.223 sec) INFO:tensorflow:global_step/sec: 437.684 INFO:tensorflow:loss = 0.54956007, step = 4901 (0.228 sec) INFO:tensorflow:global_step/sec: 446.877 INFO:tensorflow:loss = 4.3834505, step = 5001 (0.224 sec) INFO:tensorflow:global_step/sec: 449.585 INFO:tensorflow:loss = 0.4945531, step = 5101 (0.222 sec) INFO:tensorflow:global_step/sec: 451.96 INFO:tensorflow:loss = 1.7225018, step = 5201 (0.221 sec) INFO:tensorflow:global_step/sec: 441.325 INFO:tensorflow:loss = 0.4740196, step = 5301 (0.227 sec) INFO:tensorflow:global_step/sec: 403.846 INFO:tensorflow:loss = 0.11541156, step = 5401 (0.247 sec) INFO:tensorflow:global_step/sec: 421.468 INFO:tensorflow:loss = 0.5282439, step = 5501 (0.237 sec) INFO:tensorflow:global_step/sec: 397.467 INFO:tensorflow:loss = 0.11912455, step = 5601 (0.252 sec) INFO:tensorflow:global_step/sec: 379.736 INFO:tensorflow:loss = 1.1624157, step = 5701 (0.263 sec) INFO:tensorflow:global_step/sec: 418.64 INFO:tensorflow:loss = 0.08694888, step = 5801 (0.239 sec) INFO:tensorflow:global_step/sec: 439.798 INFO:tensorflow:loss = 0.84899473, step = 5901 (0.228 sec) INFO:tensorflow:global_step/sec: 426.445 INFO:tensorflow:loss = 0.68258023, step = 6001 (0.235 sec) INFO:tensorflow:global_step/sec: 439.694 INFO:tensorflow:loss = 3.325633, step = 6101 (0.227 sec) INFO:tensorflow:global_step/sec: 450.848 INFO:tensorflow:loss = 0.83453053, step = 6201 (0.222 sec) INFO:tensorflow:global_step/sec: 436.738 INFO:tensorflow:loss = 0.5704926, step = 6301 (0.229 sec) INFO:tensorflow:global_step/sec: 411.734 INFO:tensorflow:loss = 0.92547596, step = 6401 (0.243 sec) INFO:tensorflow:global_step/sec: 405.726 INFO:tensorflow:loss = 5.9035754, step = 6501 (0.246 sec) INFO:tensorflow:global_step/sec: 448.996 INFO:tensorflow:loss = 2.539691, step = 6601 (0.223 sec) INFO:tensorflow:global_step/sec: 393.599 INFO:tensorflow:loss = 0.1202597, step = 6701 (0.254 sec) INFO:tensorflow:global_step/sec: 444.148 INFO:tensorflow:loss = 1.2117217, step = 6801 (0.225 sec) INFO:tensorflow:global_step/sec: 450.242 INFO:tensorflow:loss = 0.7324259, step = 6901 (0.222 sec) INFO:tensorflow:global_step/sec: 430.006 INFO:tensorflow:loss = 4.3384786, step = 7001 (0.232 sec) INFO:tensorflow:global_step/sec: 437.595 INFO:tensorflow:loss = 7.8897605, step = 7101 (0.229 sec) INFO:tensorflow:global_step/sec: 447.848 INFO:tensorflow:loss = 0.5303372, step = 7201 (0.223 sec) INFO:tensorflow:global_step/sec: 449.926 INFO:tensorflow:loss = 2.6509504, step = 7301 (0.222 sec) INFO:tensorflow:global_step/sec: 452.495 INFO:tensorflow:loss = 0.91618377, step = 7401 (0.221 sec) INFO:tensorflow:global_step/sec: 448.898 INFO:tensorflow:loss = 0.7135824, step = 7501 (0.223 sec) INFO:tensorflow:global_step/sec: 442.401 INFO:tensorflow:loss = 0.20951754, step = 7601 (0.226 sec) INFO:tensorflow:global_step/sec: 456.171 INFO:tensorflow:loss = 0.6272392, step = 7701 (0.219 sec) INFO:tensorflow:global_step/sec: 448.783 INFO:tensorflow:loss = 0.16147618, step = 7801 (0.223 sec) INFO:tensorflow:global_step/sec: 454.218 INFO:tensorflow:loss = 1.6619877, step = 7901 (0.220 sec) INFO:tensorflow:global_step/sec: 448.047 INFO:tensorflow:loss = 0.8377149, step = 8001 (0.223 sec) INFO:tensorflow:global_step/sec: 448.666 INFO:tensorflow:loss = 0.05346997, step = 8101 (0.223 sec) INFO:tensorflow:global_step/sec: 424.041 INFO:tensorflow:loss = 0.105907656, step = 8201 (0.236 sec) INFO:tensorflow:global_step/sec: 428.866 INFO:tensorflow:loss = 0.3709848, step = 8301 (0.233 sec) INFO:tensorflow:global_step/sec: 411.811 INFO:tensorflow:loss = 0.55373275, step = 8401 (0.243 sec) INFO:tensorflow:global_step/sec: 435.513 INFO:tensorflow:loss = 1.4315906, step = 8501 (0.229 sec) INFO:tensorflow:global_step/sec: 458.526 INFO:tensorflow:loss = 0.5858322, step = 8601 (0.218 sec) INFO:tensorflow:global_step/sec: 446.98 INFO:tensorflow:loss = 0.7278558, step = 8701 (0.224 sec) INFO:tensorflow:global_step/sec: 429.608 INFO:tensorflow:loss = 1.044769, step = 8801 (0.233 sec) INFO:tensorflow:global_step/sec: 427.256 INFO:tensorflow:loss = 0.87007964, step = 8901 (0.234 sec) INFO:tensorflow:global_step/sec: 441.72 INFO:tensorflow:loss = 1.7463224, step = 9001 (0.227 sec) INFO:tensorflow:global_step/sec: 441.221 INFO:tensorflow:loss = 0.102753736, step = 9101 (0.227 sec) INFO:tensorflow:global_step/sec: 452.501 INFO:tensorflow:loss = 0.037403725, step = 9201 (0.221 sec) INFO:tensorflow:global_step/sec: 439.634 INFO:tensorflow:loss = 0.2352443, step = 9301 (0.228 sec) INFO:tensorflow:global_step/sec: 450.234 INFO:tensorflow:loss = 1.033936, step = 9401 (0.222 sec) INFO:tensorflow:global_step/sec: 450.269 INFO:tensorflow:loss = 0.17887983, step = 9501 (0.222 sec) INFO:tensorflow:global_step/sec: 437.656 INFO:tensorflow:loss = 0.7280132, step = 9601 (0.228 sec) INFO:tensorflow:global_step/sec: 453.474 INFO:tensorflow:loss = 0.90697455, step = 9701 (0.220 sec) INFO:tensorflow:global_step/sec: 442.149 INFO:tensorflow:loss = 1.7637879, step = 9801 (0.226 sec) INFO:tensorflow:global_step/sec: 437.817 INFO:tensorflow:loss = 0.6042792, step = 9901 (0.228 sec) INFO:tensorflow:global_step/sec: 441.359 INFO:tensorflow:loss = 0.21434633, step = 10001 (0.227 sec) INFO:tensorflow:global_step/sec: 450.379 INFO:tensorflow:loss = 0.6579297, step = 10101 (0.222 sec) INFO:tensorflow:global_step/sec: 438.496 INFO:tensorflow:loss = 0.29126522, step = 10201 (0.228 sec) INFO:tensorflow:global_step/sec: 424.247 INFO:tensorflow:loss = 0.24468184, step = 10301 (0.236 sec) INFO:tensorflow:global_step/sec: 450.012 INFO:tensorflow:loss = 0.24360758, step = 10401 (0.222 sec) INFO:tensorflow:global_step/sec: 441.997 INFO:tensorflow:loss = 0.15718684, step = 10501 (0.226 sec) INFO:tensorflow:global_step/sec: 435.172 INFO:tensorflow:loss = 0.7760467, step = 10601 (0.230 sec) INFO:tensorflow:global_step/sec: 446.09 INFO:tensorflow:loss = 0.844111, step = 10701 (0.224 sec) INFO:tensorflow:global_step/sec: 450.118 INFO:tensorflow:loss = 0.26858446, step = 10801 (0.222 sec) INFO:tensorflow:global_step/sec: 444.055 INFO:tensorflow:loss = 1.0553799, step = 10901 (0.225 sec) INFO:tensorflow:global_step/sec: 438.502 INFO:tensorflow:loss = 0.10617732, step = 11001 (0.228 sec) INFO:tensorflow:global_step/sec: 434.469 INFO:tensorflow:loss = 0.19385874, step = 11101 (0.230 sec) INFO:tensorflow:global_step/sec: 442.98 INFO:tensorflow:loss = 0.2478154, step = 11201 (0.226 sec) INFO:tensorflow:global_step/sec: 431.209 INFO:tensorflow:loss = 0.061364472, step = 11301 (0.232 sec) INFO:tensorflow:global_step/sec: 370.989 INFO:tensorflow:loss = 0.8201448, step = 11401 (0.270 sec) INFO:tensorflow:global_step/sec: 409.252 INFO:tensorflow:loss = 0.08134957, step = 11501 (0.244 sec) INFO:tensorflow:global_step/sec: 407.091 INFO:tensorflow:loss = 0.6539128, step = 11601 (0.246 sec) INFO:tensorflow:global_step/sec: 409.202 INFO:tensorflow:loss = 0.0966755, step = 11701 (0.244 sec) INFO:tensorflow:global_step/sec: 431.686 INFO:tensorflow:loss = 0.028790217, step = 11801 (0.232 sec) INFO:tensorflow:global_step/sec: 432.569 INFO:tensorflow:loss = 0.52753115, step = 11901 (0.231 sec) INFO:tensorflow:global_step/sec: 442.931 INFO:tensorflow:loss = 0.15677977, step = 12001 (0.226 sec) INFO:tensorflow:global_step/sec: 418.589 INFO:tensorflow:loss = 0.21374094, step = 12101 (0.239 sec) INFO:tensorflow:global_step/sec: 403.285 INFO:tensorflow:loss = 1.4326304, step = 12201 (0.248 sec) INFO:tensorflow:global_step/sec: 439.134 INFO:tensorflow:loss = 0.21316601, step = 12301 (0.227 sec) INFO:tensorflow:global_step/sec: 397.932 INFO:tensorflow:loss = 1.051509, step = 12401 (0.251 sec) INFO:tensorflow:global_step/sec: 450.451 INFO:tensorflow:loss = 0.5729082, step = 12501 (0.222 sec) INFO:tensorflow:global_step/sec: 433.685 INFO:tensorflow:loss = 0.22082293, step = 12601 (0.231 sec) INFO:tensorflow:global_step/sec: 440.441 INFO:tensorflow:loss = 0.36042464, step = 12701 (0.227 sec) INFO:tensorflow:global_step/sec: 456.517 INFO:tensorflow:loss = 0.036144137, step = 12801 (0.219 sec) INFO:tensorflow:global_step/sec: 450.497 INFO:tensorflow:loss = 0.03840449, step = 12901 (0.222 sec) INFO:tensorflow:global_step/sec: 458.22 INFO:tensorflow:loss = 0.11319007, step = 13001 (0.218 sec) INFO:tensorflow:global_step/sec: 451.138 INFO:tensorflow:loss = 0.017491136, step = 13101 (0.222 sec) INFO:tensorflow:global_step/sec: 448.907 INFO:tensorflow:loss = 0.267563, step = 13201 (0.223 sec) INFO:tensorflow:global_step/sec: 452.258 INFO:tensorflow:loss = 0.017946834, step = 13301 (0.221 sec) INFO:tensorflow:global_step/sec: 449.406 INFO:tensorflow:loss = 0.49629217, step = 13401 (0.222 sec) INFO:tensorflow:global_step/sec: 443.172 INFO:tensorflow:loss = 0.10338087, step = 13501 (0.226 sec) INFO:tensorflow:global_step/sec: 447.071 INFO:tensorflow:loss = 0.24734607, step = 13601 (0.224 sec) INFO:tensorflow:global_step/sec: 448.755 INFO:tensorflow:loss = 0.50354725, step = 13701 (0.223 sec) INFO:tensorflow:global_step/sec: 447.271 INFO:tensorflow:loss = 0.11354814, step = 13801 (0.223 sec) INFO:tensorflow:global_step/sec: 449.759 INFO:tensorflow:loss = 0.4873892, step = 13901 (0.222 sec) INFO:tensorflow:global_step/sec: 426.935 INFO:tensorflow:loss = 2.4945714, step = 14001 (0.234 sec) INFO:tensorflow:global_step/sec: 434.796 INFO:tensorflow:loss = 0.6957367, step = 14101 (0.230 sec) INFO:tensorflow:global_step/sec: 434.942 INFO:tensorflow:loss = 0.025100248, step = 14201 (0.230 sec) INFO:tensorflow:global_step/sec: 428.313 INFO:tensorflow:loss = 0.5430741, step = 14301 (0.233 sec) INFO:tensorflow:global_step/sec: 433.657 INFO:tensorflow:loss = 0.17433453, step = 14401 (0.231 sec) INFO:tensorflow:global_step/sec: 433.299 INFO:tensorflow:loss = 1.8630569, step = 14501 (0.230 sec) INFO:tensorflow:global_step/sec: 451.98 INFO:tensorflow:loss = 0.6306378, step = 14601 (0.221 sec) INFO:tensorflow:global_step/sec: 447.473 INFO:tensorflow:loss = 0.48561808, step = 14701 (0.224 sec) INFO:tensorflow:global_step/sec: 440.212 INFO:tensorflow:loss = 0.30988026, step = 14801 (0.227 sec) INFO:tensorflow:global_step/sec: 448.704 INFO:tensorflow:loss = 0.65663326, step = 14901 (0.222 sec) INFO:tensorflow:global_step/sec: 441.957 INFO:tensorflow:loss = 0.1834591, step = 15001 (0.226 sec) INFO:tensorflow:global_step/sec: 451.274 INFO:tensorflow:loss = 0.06628236, step = 15101 (0.222 sec) INFO:tensorflow:global_step/sec: 446.991 INFO:tensorflow:loss = 0.067467816, step = 15201 (0.224 sec) INFO:tensorflow:global_step/sec: 454.347 INFO:tensorflow:loss = 0.2939768, step = 15301 (0.220 sec) INFO:tensorflow:global_step/sec: 451.007 INFO:tensorflow:loss = 0.4629912, step = 15401 (0.221 sec) INFO:tensorflow:global_step/sec: 452.726 INFO:tensorflow:loss = 0.5365031, step = 15501 (0.221 sec) INFO:tensorflow:global_step/sec: 451.121 INFO:tensorflow:loss = 0.013298916, step = 15601 (0.222 sec) INFO:tensorflow:global_step/sec: 453.7 INFO:tensorflow:loss = 0.03741165, step = 15701 (0.221 sec) INFO:tensorflow:global_step/sec: 447.934 INFO:tensorflow:loss = 0.40824798, step = 15801 (0.223 sec) INFO:tensorflow:global_step/sec: 449.644 INFO:tensorflow:loss = 0.54686666, step = 15901 (0.222 sec) INFO:tensorflow:global_step/sec: 450.682 INFO:tensorflow:loss = 0.1791507, step = 16001 (0.222 sec) INFO:tensorflow:global_step/sec: 436.914 INFO:tensorflow:loss = 0.5521565, step = 16101 (0.229 sec) INFO:tensorflow:global_step/sec: 419.817 INFO:tensorflow:loss = 0.10045961, step = 16201 (0.238 sec) INFO:tensorflow:global_step/sec: 431.274 INFO:tensorflow:loss = 0.31248388, step = 16301 (0.231 sec) INFO:tensorflow:global_step/sec: 423.959 INFO:tensorflow:loss = 0.10864381, step = 16401 (0.236 sec) INFO:tensorflow:global_step/sec: 445.489 INFO:tensorflow:loss = 0.52347213, step = 16501 (0.224 sec) INFO:tensorflow:global_step/sec: 441.11 INFO:tensorflow:loss = 0.024003511, step = 16601 (0.227 sec) INFO:tensorflow:global_step/sec: 439.487 INFO:tensorflow:loss = 0.008060399, step = 16701 (0.227 sec) INFO:tensorflow:global_step/sec: 436.284 INFO:tensorflow:loss = 0.02893817, step = 16801 (0.229 sec) INFO:tensorflow:global_step/sec: 446.392 INFO:tensorflow:loss = 0.39498305, step = 16901 (0.224 sec) INFO:tensorflow:global_step/sec: 438.978 INFO:tensorflow:loss = 0.023037493, step = 17001 (0.228 sec) INFO:tensorflow:global_step/sec: 439.866 INFO:tensorflow:loss = 0.48648793, step = 17101 (0.227 sec) INFO:tensorflow:global_step/sec: 454.872 INFO:tensorflow:loss = 0.49331194, step = 17201 (0.220 sec) INFO:tensorflow:global_step/sec: 443.025 INFO:tensorflow:loss = 0.32060045, step = 17301 (0.226 sec) INFO:tensorflow:global_step/sec: 440.069 INFO:tensorflow:loss = 0.13167329, step = 17401 (0.227 sec) INFO:tensorflow:global_step/sec: 448.211 INFO:tensorflow:loss = 0.05688939, step = 17501 (0.223 sec) INFO:tensorflow:global_step/sec: 450.458 INFO:tensorflow:loss = 0.36213198, step = 17601 (0.222 sec) INFO:tensorflow:global_step/sec: 428.842 INFO:tensorflow:loss = 0.36243188, step = 17701 (0.233 sec) INFO:tensorflow:global_step/sec: 456.734 INFO:tensorflow:loss = 0.20977254, step = 17801 (0.219 sec) INFO:tensorflow:global_step/sec: 432.647 INFO:tensorflow:loss = 0.09754325, step = 17901 (0.231 sec) INFO:tensorflow:global_step/sec: 389.941 INFO:tensorflow:loss = 0.03494991, step = 18001 (0.256 sec) INFO:tensorflow:global_step/sec: 434.925 INFO:tensorflow:loss = 0.17031653, step = 18101 (0.230 sec) INFO:tensorflow:global_step/sec: 445.735 INFO:tensorflow:loss = 0.3200203, step = 18201 (0.224 sec) INFO:tensorflow:global_step/sec: 444.929 INFO:tensorflow:loss = 0.18385477, step = 18301 (0.225 sec) INFO:tensorflow:global_step/sec: 445.546 INFO:tensorflow:loss = 0.20921718, step = 18401 (0.225 sec) INFO:tensorflow:global_step/sec: 450.454 INFO:tensorflow:loss = 0.01868303, step = 18501 (0.222 sec) INFO:tensorflow:global_step/sec: 445.762 INFO:tensorflow:loss = 0.051421717, step = 18601 (0.224 sec) INFO:tensorflow:global_step/sec: 445.921 INFO:tensorflow:loss = 0.047041617, step = 18701 (0.224 sec) INFO:tensorflow:Saving checkpoints for 18750 into /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmp2xxrubio/model.ckpt. INFO:tensorflow:Loss for final step: 0.46282205.
<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11711b748>
The final loss should be exactly 0.46282205.
Instead of using the numpy_input_fn()
function (which cannot reproducibly shuffle the dataset at each epoch), you can create your own input function using the Data API and set its shuffling seed:
def create_dataset(X, y=None, n_epochs=1, batch_size=32,
buffer_size=1000, seed=None):
dataset = tf.data.Dataset.from_tensor_slices(({"X": X}, y))
dataset = dataset.repeat(n_epochs)
dataset = dataset.shuffle(buffer_size, seed=seed)
return dataset.batch(batch_size)
input_fn=lambda: create_dataset(X_train, y_train, seed=42)
random.seed(42)
np.random.seed(42)
tf.set_random_seed(42)
my_config = tf.estimator.RunConfig(tf_random_seed=42)
feature_cols = [tf.feature_column.numeric_column("X", shape=[28 * 28])]
dnn_clf = tf.estimator.DNNClassifier(hidden_units=[300, 100], n_classes=10,
feature_columns=feature_cols,
config=my_config)
dnn_clf.train(input_fn=input_fn)
WARNING:tensorflow:Using temporary folder as model directory: /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmpawwl1lf0 INFO:tensorflow:Using config: {'_model_dir': '/var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmpawwl1lf0', '_tf_random_seed': 42, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x137e1c6d8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} INFO:tensorflow:Calling model_fn. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmpawwl1lf0/model.ckpt. INFO:tensorflow:loss = 80.279686, step = 1 INFO:tensorflow:global_step/sec: 161.253 INFO:tensorflow:loss = 16.09288, step = 101 (0.621 sec) INFO:tensorflow:global_step/sec: 433.582 INFO:tensorflow:loss = 5.605775, step = 201 (0.231 sec) INFO:tensorflow:global_step/sec: 447.561 INFO:tensorflow:loss = 12.584702, step = 301 (0.224 sec) INFO:tensorflow:global_step/sec: 442.148 INFO:tensorflow:loss = 2.089463, step = 401 (0.226 sec) INFO:tensorflow:global_step/sec: 434.492 INFO:tensorflow:loss = 9.2258215, step = 501 (0.230 sec) INFO:tensorflow:global_step/sec: 447.994 INFO:tensorflow:loss = 8.11821, step = 601 (0.223 sec) INFO:tensorflow:global_step/sec: 442.723 INFO:tensorflow:loss = 0.653025, step = 701 (0.226 sec) INFO:tensorflow:global_step/sec: 425.438 INFO:tensorflow:loss = 4.331424, step = 801 (0.235 sec) INFO:tensorflow:global_step/sec: 444.471 INFO:tensorflow:loss = 1.55325, step = 901 (0.225 sec) INFO:tensorflow:global_step/sec: 436.037 INFO:tensorflow:loss = 5.208349, step = 1001 (0.229 sec) INFO:tensorflow:global_step/sec: 433.071 INFO:tensorflow:loss = 0.80289483, step = 1101 (0.231 sec) INFO:tensorflow:global_step/sec: 436.717 INFO:tensorflow:loss = 3.1879468, step = 1201 (0.229 sec) INFO:tensorflow:global_step/sec: 452.687 INFO:tensorflow:loss = 5.55963, step = 1301 (0.221 sec) INFO:tensorflow:global_step/sec: 446.2 INFO:tensorflow:loss = 12.830038, step = 1401 (0.224 sec) INFO:tensorflow:global_step/sec: 450.525 INFO:tensorflow:loss = 6.8311796, step = 1501 (0.222 sec) INFO:tensorflow:global_step/sec: 452.967 INFO:tensorflow:loss = 1.635078, step = 1601 (0.221 sec) INFO:tensorflow:global_step/sec: 453.743 INFO:tensorflow:loss = 1.9616288, step = 1701 (0.220 sec) INFO:tensorflow:global_step/sec: 450.01 INFO:tensorflow:loss = 1.4227519, step = 1801 (0.222 sec) INFO:tensorflow:Saving checkpoints for 1875 into /var/folders/wy/h39t6kb11pnbb0pzhksd_fqh0000gn/T/tmpawwl1lf0/model.ckpt. INFO:tensorflow:Loss for final step: 1.0556093.
<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x11dba7a20>
The final loss should be exactly 1.0556093.
indices = np.random.permutation(len(X_train))
X_train_shuffled = X_train[indices]
y_train_shuffled = y_train[indices]
input_fn = tf.estimator.inputs.numpy_input_fn(
x={"X": X_train_shuffled}, y=y_train_shuffled,
num_epochs=10, batch_size=32, shuffle=False)
dnn_clf.train(input_fn=input_fn)
If you use the Keras API, all you need to do is set the random seed any time you clear the session:
keras.backend.clear_session()
random.seed(42)
np.random.seed(42)
tf.set_random_seed(42)
model = keras.models.Sequential([
keras.layers.Dense(300, activation="relu"),
keras.layers.Dense(100, activation="relu"),
keras.layers.Dense(10, activation="softmax"),
])
model.compile(loss="sparse_categorical_crossentropy", optimizer="sgd",
metrics=["accuracy"])
model.fit(X_train, y_train, epochs=10)
Epoch 1/10 60000/60000 [==============================] - 5s 78us/step - loss: 0.5929 - acc: 0.8450 Epoch 2/10 60000/60000 [==============================] - 4s 75us/step - loss: 0.2804 - acc: 0.9199 Epoch 3/10 60000/60000 [==============================] - 4s 74us/step - loss: 0.2276 - acc: 0.9350 Epoch 4/10 60000/60000 [==============================] - 4s 74us/step - loss: 0.1933 - acc: 0.9449 Epoch 5/10 60000/60000 [==============================] - 4s 74us/step - loss: 0.1682 - acc: 0.9518 Epoch 6/10 60000/60000 [==============================] - 4s 74us/step - loss: 0.1490 - acc: 0.9573 Epoch 7/10 60000/60000 [==============================] - 4s 74us/step - loss: 0.1332 - acc: 0.9622 Epoch 8/10 60000/60000 [==============================] - 5s 75us/step - loss: 0.1202 - acc: 0.9658 Epoch 9/10 60000/60000 [==============================] - 4s 75us/step - loss: 0.1090 - acc: 0.9693 Epoch 10/10 60000/60000 [==============================] - 4s 75us/step - loss: 0.1000 - acc: 0.9716
<tensorflow.python.keras.callbacks.History at 0x1379fff98>
You should get exactly 97.16% accuracy on the training set at the end of training.
For example, os.listdir()
returns file names in an order that depends on how the files were indexed by the file system:
for i in range(10):
with open("my_test_foo_{}".format(i), "w"):
pass
[f for f in os.listdir() if f.startswith("my_test_foo_")]
['my_test_foo_1', 'my_test_foo_6', 'my_test_foo_8', 'my_test_foo_9', 'my_test_foo_7', 'my_test_foo_0', 'my_test_foo_5', 'my_test_foo_2', 'my_test_foo_3', 'my_test_foo_4']
for i in range(10):
with open("my_test_bar_{}".format(i), "w"):
pass
[f for f in os.listdir() if f.startswith("my_test_bar_")]
['my_test_bar_4', 'my_test_bar_3', 'my_test_bar_2', 'my_test_bar_5', 'my_test_bar_0', 'my_test_bar_7', 'my_test_bar_9', 'my_test_bar_8', 'my_test_bar_6', 'my_test_bar_1']
You should sort the file names before you use them:
filenames = os.listdir()
filenames.sort()
[f for f in filenames if f.startswith("my_test_foo_")]
['my_test_foo_0', 'my_test_foo_1', 'my_test_foo_2', 'my_test_foo_3', 'my_test_foo_4', 'my_test_foo_5', 'my_test_foo_6', 'my_test_foo_7', 'my_test_foo_8', 'my_test_foo_9']
for f in os.listdir():
if f.startswith("my_test_foo_") or f.startswith("my_test_bar_"):
os.remove(f)
I hope you enjoyed this notebook. If you do not get reproducible results, or if they are different than mine, then please file an issue on github, specifying what version of Python, TensorFlow, and NumPy you are using, as well as your O.S. version. Thank you!
If you want to learn more about Deep Learning and TensorFlow, check out my book Hands-On Machine Learning with Scitkit-Learn and TensorFlow, O'Reilly. You can also follow me on twitter @aureliengeron or watch my videos on YouTube at youtube.com/c/AurelienGeron.