Deep Learning Day 7.¶

High Level APIs¶

Estimator
Experiment

Estimator¶

sklearn의 estimator와 유사하게 설계된 framework
train, evaluate, predict 등을 쉽게 구현 할 수 있음.
Session을 사용자가 직접 구현 할 필요 없음
input pipeline만 작성하면 model 설계에 집중 할 수 있다.
model export를 제공하며 serving시 필요한 protocol buffer file을 생성해줌

input function¶

model에 투입되는 dataset을 load하는 pipeline을 구성하는 함수 정의
원하는 pipeline을 설계하고, 함수의 리턴은 train, labels로만 해주면 사용 가능
Estimator에서 이 input function을 함수형으로 받기 때문에 입력 argument 필요시 아래 코드와 같이 lambda 식 활용

In [1]:

import tensorflow as tf

In [2]:

def parser(serialized_example):
    features = {
        'age': tf.FixedLenFeature([1], tf.int64),
        'img': tf.FixedLenFeature([61*49], tf.int64)
    }
    
    parsed_feature = tf.parse_single_example(serialized_example, features)
    age = parsed_feature['age']
    img = parsed_feature['img']
    return age, img

In [3]:

tf.reset_default_graph()

def input_fn(dataset_dir, batch_size):
    dataset = tf.contrib.data.TFRecordDataset(dataset_dir).map(parser)
    dataset = dataset.batch(batch_size).shuffle(7777)
    
    itr = dataset.make_one_shot_iterator()
    
    age, img = itr.get_next()
    
    img = tf.reshape(img, [-1, 61, 49, 1])
    img = tf.cast(img, tf.float32)
    
    age = tf.reshape(age, [-1])
    age_onehot = tf.one_hot(age, depth=3, axis=-1)
    
    return {'img': img}, {'age': age, 'age_onehot': age_onehot}
    

def get_train_input_fn():
    return lambda: input_fn('./cnn_dataset/face_train.tfrecord', 10)

def get_test_input_fn():
    return lambda: input_fn('./cnn_dataset/face_test.tfrecord', 10)

model function¶

실제 모델이 들어가는 함수로써 입력인자(features, labels, mode) signature를 지켜주어야 한다.
이 model function도 마찬가지로 사용자의 hyper parameters를 외부에서 제어 할 수 있도록 lambda식을 사용하여 입력인자로 넘겨 받을 수 있다.
반드시 EstimatorSpec를 리턴시켜주어야 하며, TRAIN, EVAL, PREDICT 모드 별 트레인 모델을 분기문으로 나눠준다.

In [4]:

def model(features, labels, mode, **params):
    conv1 = tf.layers.conv2d(features['img'], filters=10, kernel_size=3, 
                             padding='SAME', activation=params['activation'])
    pool1 = tf.layers.max_pooling2d(conv1, pool_size=2, strides=2)
    conv2 = tf.layers.conv2d(pool1, filters=10, kernel_size=3, 
                             padding='SAME', activation=params['activation'])
    pool2 = tf.layers.max_pooling2d(conv2, pool_size=2, strides=2)
    conv3 = tf.layers.conv2d(pool2, filters=10, kernel_size=3, 
                             padding='SAME', activation=params['activation'])
    pool3 = tf.layers.max_pooling2d(conv3, pool_size=2, strides=2)

    flat_size = int(pool3.shape[1]) * int(pool3.shape[2]) * int(pool3.shape[3])
    flat = tf.reshape(pool3, [-1, flat_size])
    
    if mode == tf.estimator.ModeKeys.TRAIN:
        dropout_prob = params['dropout_prob']
    else:
        dropout_prob = 1
    

    dropout1 = tf.layers.dropout(flat, dropout_prob)
    fc1 = tf.layers.dense(dropout1, 1000)

    dropout2 = tf.layers.dropout(fc1, dropout_prob)
    out = tf.layers.dense(dropout2, 3) 
    
    if mode == tf.estimator.ModeKeys.TRAIN:
        global_step = tf.train.get_global_step()
        loss = tf.losses.softmax_cross_entropy(labels['age_onehot'], out)
        train_op = tf.train.GradientDescentOptimizer(1e-6).minimize(loss, global_step)
        estimator_spec = tf.estimator.EstimatorSpec(train_op=train_op, loss=loss, mode=mode)

        
    elif mode == tf.estimator.ModeKeys.EVAL:
        loss = tf.losses.softmax_cross_entropy(labels['age_onehot'], out)
        
        pred = tf.argmax(tf.nn.softmax(out), axis=1)
        accuracy = tf.metrics.accuracy(labels['age'], pred)
        eval_metric_ops = {"acc": accuracy} 
        estimator_spec = tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)
        
    else:
        estimator_spec = None
        
    return estimator_spec

def get_model_fn(activation, dropout_prob):
    return lambda features, labels, mode: model(features, labels, mode, 
                                                activation=activation, dropout_prob=0.7)

Estimator¶

아래와 같이 model 함수를 이용하여 Estimator를 생성해준다.
생성한 Estimator 이용하여 train, eval, predict 등을 수행 할 수 있다.

In [ ]:

est_config = tf.estimator.RunConfig()
est = tf.estimator.Estimator(model_fn=get_model_fn(activation=tf.nn.relu, dropout_prob=0.7), 
                             model_dir='./logs/estimator',
                             config=config)

for epoch in range(3):
    est.train(get_train_input_fn())
    est.evaluate(get_test_input_fn())
    

INFO:tensorflow:Using config: {'_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x115b742e8>, '_keep_checkpoint_every_n_hours': 10000, '_save_checkpoints_steps': None, '_tf_random_seed': None, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_save_checkpoints_secs': 600, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1
}
, '_log_step_count_steps': 100, '_save_summary_steps': 100, '_evaluation_master': '', '_master': '', '_session_config': None, '_environment': 'local', '_is_chief': True, '_num_worker_replicas': 0, '_model_dir': './logs/estimator', '_task_id': 0, '_task_type': None}
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Restoring parameters from ./logs/estimator/model.ckpt-6001
INFO:tensorflow:Saving checkpoints for 6002 into ./logs/estimator/model.ckpt.
INFO:tensorflow:loss = 1.17684, step = 6002
INFO:tensorflow:global_step/sec: 68.3415
INFO:tensorflow:loss = 1.27675, step = 6102 (1.464 sec)
INFO:tensorflow:global_step/sec: 67.5664
INFO:tensorflow:loss = 0.844532, step = 6202 (1.480 sec)
INFO:tensorflow:global_step/sec: 70.1978
INFO:tensorflow:loss = 1.04935, step = 6302 (1.425 sec)
INFO:tensorflow:global_step/sec: 70.1414
INFO:tensorflow:loss = 1.28345, step = 6402 (1.426 sec)
INFO:tensorflow:global_step/sec: 68.9065
INFO:tensorflow:loss = 0.820354, step = 6502 (1.452 sec)
INFO:tensorflow:Saving checkpoints for 6601 into ./logs/estimator/model.ckpt.
INFO:tensorflow:Loss for final step: 0.349178.
INFO:tensorflow:Starting evaluation at 2017-10-16-15:48:36
INFO:tensorflow:Restoring parameters from ./logs/estimator/model.ckpt-6601
INFO:tensorflow:Finished evaluation at 2017-10-16-15:48:37
INFO:tensorflow:Saving dict for global step 6601: acc = 0.492498, global_step = 6601, loss = 1.39988
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Restoring parameters from ./logs/estimator/model.ckpt-6601
INFO:tensorflow:Saving checkpoints for 6602 into ./logs/estimator/model.ckpt.
INFO:tensorflow:loss = 1.87919, step = 6602
INFO:tensorflow:global_step/sec: 65.7797
INFO:tensorflow:loss = 0.760529, step = 6702 (1.521 sec)
INFO:tensorflow:global_step/sec: 68.3511
INFO:tensorflow:loss = 1.00407, step = 6802 (1.463 sec)
INFO:tensorflow:global_step/sec: 69.2456
INFO:tensorflow:loss = 0.326808, step = 6902 (1.444 sec)
INFO:tensorflow:global_step/sec: 72.58
INFO:tensorflow:loss = 0.96353, step = 7002 (1.378 sec)
INFO:tensorflow:global_step/sec: 70.3555
INFO:tensorflow:loss = 0.414311, step = 7102 (1.421 sec)
INFO:tensorflow:Saving checkpoints for 7201 into ./logs/estimator/model.ckpt.
INFO:tensorflow:Loss for final step: 1.56244.
INFO:tensorflow:Starting evaluation at 2017-10-16-15:48:49
INFO:tensorflow:Restoring parameters from ./logs/estimator/model.ckpt-7201
INFO:tensorflow:Finished evaluation at 2017-10-16-15:48:50
INFO:tensorflow:Saving dict for global step 7201: acc = 0.484018, global_step = 7201, loss = 1.3787
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Restoring parameters from ./logs/estimator/model.ckpt-7201
INFO:tensorflow:Saving checkpoints for 7202 into ./logs/estimator/model.ckpt.
INFO:tensorflow:loss = 1.1452, step = 7202
INFO:tensorflow:global_step/sec: 68.1717
INFO:tensorflow:loss = 0.730702, step = 7302 (1.468 sec)
INFO:tensorflow:global_step/sec: 70.6265
INFO:tensorflow:loss = 1.26689, step = 7402 (1.416 sec)
INFO:tensorflow:global_step/sec: 71.3493
INFO:tensorflow:loss = 1.00017, step = 7502 (1.402 sec)
INFO:tensorflow:global_step/sec: 63.8502
INFO:tensorflow:loss = 0.831771, step = 7602 (1.566 sec)

Experiment¶

estimator를 기가막히게 돌려주는 framework
local distributed setting만 되어 있다면 분산러닝 자동 지원
training, inference, predict 모델을 따로 저장하기 때문에 serving 시 Experiment를 통한 export 권장
내부에 학습간 필요한 기능들을 다수 내장

In [50]:

import json
import os
def get_experiment(output_dir):
    os.environ['TF_CONFIG'] = json.dumps({'environment': 'local'})
    config = tf.contrib.learn.RunConfig()
    return tf.contrib.learn.Experiment(estimator=tf.estimator.Estimator(model_fn=get_model_fn(activation=tf.nn.relu,
                                                                                              dropout_prob=0.7), 
                                                                        model_dir=output_dir,
                                                                        config=config),
                                       train_input_fn=get_train_input_fn(),
                                       eval_input_fn=get_test_input_fn())

tf.contrib.learn.learn_runner.run(experiment_fn=get_experiment, output_dir='./logs/estimator')

INFO:tensorflow:Using config: {'_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x115b3e588>, '_keep_checkpoint_every_n_hours': 10000, '_save_checkpoints_steps': None, '_tf_random_seed': None, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_save_checkpoints_secs': 600, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1
}
, '_log_step_count_steps': 100, '_save_summary_steps': 100, '_evaluation_master': '', '_master': '', '_session_config': None, '_environment': 'local', '_is_chief': True, '_num_worker_replicas': 0, '_model_dir': './logs/estimator', '_task_id': 0, '_task_type': None}
WARNING:tensorflow:From /Users/chanwoo/anaconda/envs/tensorflow-py3/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:269: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Restoring parameters from ./logs/estimator/model.ckpt-5401
INFO:tensorflow:Saving checkpoints for 5402 into ./logs/estimator/model.ckpt.
INFO:tensorflow:Starting evaluation at 2017-10-16-15:45:29
INFO:tensorflow:Restoring parameters from ./logs/estimator/model.ckpt-5402
INFO:tensorflow:Evaluation [1/100]
INFO:tensorflow:Evaluation [2/100]
INFO:tensorflow:Evaluation [3/100]
INFO:tensorflow:Evaluation [4/100]
INFO:tensorflow:Evaluation [5/100]
INFO:tensorflow:Evaluation [6/100]
INFO:tensorflow:Evaluation [7/100]
INFO:tensorflow:Evaluation [8/100]
INFO:tensorflow:Evaluation [9/100]
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [11/100]
INFO:tensorflow:Evaluation [12/100]
INFO:tensorflow:Evaluation [13/100]
INFO:tensorflow:Evaluation [14/100]
INFO:tensorflow:Evaluation [15/100]
INFO:tensorflow:Evaluation [16/100]
INFO:tensorflow:Evaluation [17/100]
INFO:tensorflow:Evaluation [18/100]
INFO:tensorflow:Evaluation [19/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [21/100]
INFO:tensorflow:Evaluation [22/100]
INFO:tensorflow:Evaluation [23/100]
INFO:tensorflow:Evaluation [24/100]
INFO:tensorflow:Evaluation [25/100]
INFO:tensorflow:Evaluation [26/100]
INFO:tensorflow:Evaluation [27/100]
INFO:tensorflow:Evaluation [28/100]
INFO:tensorflow:Evaluation [29/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [31/100]
INFO:tensorflow:Evaluation [32/100]
INFO:tensorflow:Evaluation [33/100]
INFO:tensorflow:Evaluation [34/100]
INFO:tensorflow:Evaluation [35/100]
INFO:tensorflow:Evaluation [36/100]
INFO:tensorflow:Evaluation [37/100]
INFO:tensorflow:Evaluation [38/100]
INFO:tensorflow:Evaluation [39/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [41/100]
INFO:tensorflow:Evaluation [42/100]
INFO:tensorflow:Evaluation [43/100]
INFO:tensorflow:Evaluation [44/100]
INFO:tensorflow:Evaluation [45/100]
INFO:tensorflow:Evaluation [46/100]
INFO:tensorflow:Evaluation [47/100]
INFO:tensorflow:Evaluation [48/100]
INFO:tensorflow:Evaluation [49/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [51/100]
INFO:tensorflow:Evaluation [52/100]
INFO:tensorflow:Evaluation [53/100]
INFO:tensorflow:Evaluation [54/100]
INFO:tensorflow:Evaluation [55/100]
INFO:tensorflow:Evaluation [56/100]
INFO:tensorflow:Evaluation [57/100]
INFO:tensorflow:Evaluation [58/100]
INFO:tensorflow:Evaluation [59/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [61/100]
INFO:tensorflow:Evaluation [62/100]
INFO:tensorflow:Evaluation [63/100]
INFO:tensorflow:Evaluation [64/100]
INFO:tensorflow:Evaluation [65/100]
INFO:tensorflow:Evaluation [66/100]
INFO:tensorflow:Evaluation [67/100]
INFO:tensorflow:Evaluation [68/100]
INFO:tensorflow:Evaluation [69/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Evaluation [71/100]
INFO:tensorflow:Evaluation [72/100]
INFO:tensorflow:Evaluation [73/100]
INFO:tensorflow:Evaluation [74/100]
INFO:tensorflow:Evaluation [75/100]
INFO:tensorflow:Evaluation [76/100]
INFO:tensorflow:Evaluation [77/100]
INFO:tensorflow:Evaluation [78/100]
INFO:tensorflow:Evaluation [79/100]
INFO:tensorflow:Evaluation [80/100]
INFO:tensorflow:Evaluation [81/100]
INFO:tensorflow:Evaluation [82/100]
INFO:tensorflow:Evaluation [83/100]
INFO:tensorflow:Evaluation [84/100]
INFO:tensorflow:Evaluation [85/100]
INFO:tensorflow:Evaluation [86/100]
INFO:tensorflow:Evaluation [87/100]
INFO:tensorflow:Evaluation [88/100]
INFO:tensorflow:Evaluation [89/100]
INFO:tensorflow:Evaluation [90/100]
INFO:tensorflow:Evaluation [91/100]
INFO:tensorflow:Evaluation [92/100]
INFO:tensorflow:Evaluation [93/100]
INFO:tensorflow:Evaluation [94/100]
INFO:tensorflow:Evaluation [95/100]
INFO:tensorflow:Evaluation [96/100]
INFO:tensorflow:Evaluation [97/100]
INFO:tensorflow:Evaluation [98/100]
INFO:tensorflow:Evaluation [99/100]
INFO:tensorflow:Evaluation [100/100]
INFO:tensorflow:Finished evaluation at 2017-10-16-15:45:30
INFO:tensorflow:Saving dict for global step 5402: acc = 0.472, global_step = 5402, loss = 1.45123
INFO:tensorflow:Validation (step 5402): loss = 1.45123, global_step = 5402, acc = 0.472
INFO:tensorflow:loss = 1.12171, step = 5402
INFO:tensorflow:global_step/sec: 25.2029
INFO:tensorflow:loss = 2.16686, step = 5502 (1.649 sec)
INFO:tensorflow:global_step/sec: 62.801
INFO:tensorflow:loss = 0.589146, step = 5602 (1.592 sec)
INFO:tensorflow:global_step/sec: 63.1881
INFO:tensorflow:loss = 0.890765, step = 5702 (1.583 sec)
INFO:tensorflow:global_step/sec: 61.4764
INFO:tensorflow:loss = 2.06711, step = 5802 (1.626 sec)
INFO:tensorflow:global_step/sec: 65.2995
INFO:tensorflow:loss = 2.79375, step = 5902 (1.531 sec)
INFO:tensorflow:Saving checkpoints for 6001 into ./logs/estimator/model.ckpt.
INFO:tensorflow:Loss for final step: 2.11913.
INFO:tensorflow:Starting evaluation at 2017-10-16-15:45:41
INFO:tensorflow:Restoring parameters from ./logs/estimator/model.ckpt-6001
INFO:tensorflow:Evaluation [1/100]
INFO:tensorflow:Evaluation [2/100]
INFO:tensorflow:Evaluation [3/100]
INFO:tensorflow:Evaluation [4/100]
INFO:tensorflow:Evaluation [5/100]
INFO:tensorflow:Evaluation [6/100]
INFO:tensorflow:Evaluation [7/100]
INFO:tensorflow:Evaluation [8/100]
INFO:tensorflow:Evaluation [9/100]
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [11/100]
INFO:tensorflow:Evaluation [12/100]
INFO:tensorflow:Evaluation [13/100]
INFO:tensorflow:Evaluation [14/100]
INFO:tensorflow:Evaluation [15/100]
INFO:tensorflow:Evaluation [16/100]
INFO:tensorflow:Evaluation [17/100]
INFO:tensorflow:Evaluation [18/100]
INFO:tensorflow:Evaluation [19/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [21/100]
INFO:tensorflow:Evaluation [22/100]
INFO:tensorflow:Evaluation [23/100]
INFO:tensorflow:Evaluation [24/100]
INFO:tensorflow:Evaluation [25/100]
INFO:tensorflow:Evaluation [26/100]
INFO:tensorflow:Evaluation [27/100]
INFO:tensorflow:Evaluation [28/100]
INFO:tensorflow:Evaluation [29/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [31/100]
INFO:tensorflow:Evaluation [32/100]
INFO:tensorflow:Evaluation [33/100]
INFO:tensorflow:Evaluation [34/100]
INFO:tensorflow:Evaluation [35/100]
INFO:tensorflow:Evaluation [36/100]
INFO:tensorflow:Evaluation [37/100]
INFO:tensorflow:Evaluation [38/100]
INFO:tensorflow:Evaluation [39/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [41/100]
INFO:tensorflow:Evaluation [42/100]
INFO:tensorflow:Evaluation [43/100]
INFO:tensorflow:Evaluation [44/100]
INFO:tensorflow:Evaluation [45/100]
INFO:tensorflow:Evaluation [46/100]
INFO:tensorflow:Evaluation [47/100]
INFO:tensorflow:Evaluation [48/100]
INFO:tensorflow:Evaluation [49/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [51/100]
INFO:tensorflow:Evaluation [52/100]
INFO:tensorflow:Evaluation [53/100]
INFO:tensorflow:Evaluation [54/100]
INFO:tensorflow:Evaluation [55/100]
INFO:tensorflow:Evaluation [56/100]
INFO:tensorflow:Evaluation [57/100]
INFO:tensorflow:Evaluation [58/100]
INFO:tensorflow:Evaluation [59/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [61/100]
INFO:tensorflow:Evaluation [62/100]
INFO:tensorflow:Evaluation [63/100]
INFO:tensorflow:Evaluation [64/100]
INFO:tensorflow:Evaluation [65/100]
INFO:tensorflow:Evaluation [66/100]
INFO:tensorflow:Evaluation [67/100]
INFO:tensorflow:Evaluation [68/100]
INFO:tensorflow:Evaluation [69/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Evaluation [71/100]
INFO:tensorflow:Evaluation [72/100]
INFO:tensorflow:Evaluation [73/100]
INFO:tensorflow:Evaluation [74/100]
INFO:tensorflow:Evaluation [75/100]
INFO:tensorflow:Evaluation [76/100]
INFO:tensorflow:Evaluation [77/100]
INFO:tensorflow:Evaluation [78/100]
INFO:tensorflow:Evaluation [79/100]
INFO:tensorflow:Evaluation [80/100]
INFO:tensorflow:Evaluation [81/100]
INFO:tensorflow:Evaluation [82/100]
INFO:tensorflow:Evaluation [83/100]
INFO:tensorflow:Evaluation [84/100]
INFO:tensorflow:Evaluation [85/100]
INFO:tensorflow:Evaluation [86/100]
INFO:tensorflow:Evaluation [87/100]
INFO:tensorflow:Evaluation [88/100]
INFO:tensorflow:Evaluation [89/100]
INFO:tensorflow:Evaluation [90/100]
INFO:tensorflow:Evaluation [91/100]
INFO:tensorflow:Evaluation [92/100]
INFO:tensorflow:Evaluation [93/100]
INFO:tensorflow:Evaluation [94/100]
INFO:tensorflow:Evaluation [95/100]
INFO:tensorflow:Evaluation [96/100]
INFO:tensorflow:Evaluation [97/100]
INFO:tensorflow:Evaluation [98/100]
INFO:tensorflow:Evaluation [99/100]
INFO:tensorflow:Evaluation [100/100]
INFO:tensorflow:Finished evaluation at 2017-10-16-15:45:42
INFO:tensorflow:Saving dict for global step 6001: acc = 0.523666, global_step = 6001, loss = 1.31787

Out[50]:

({'acc': 0.52366567, 'global_step': 6001, 'loss': 1.3178735}, [])