Deeplearning Algorithms tutorial

谷歌的人工智能位于全球前列，在图像识别、语音识别、无人驾驶等技术上都已经落地。而百度实质意义上扛起了国内的人工智能的大旗，覆盖无人驾驶、智能助手、图像识别等许多层面。苹果业已开始全面拥抱机器学习，新产品进军家庭智能音箱并打造工作站级别Mac。另外，腾讯的深度学习平台Mariana已支持了微信语音识别的语音输入法、语音开放平台、长按语音消息转文本等产品，在微信图像识别中开始应用。全球前十大科技公司全部发力人工智能理论研究和应用的实现，虽然入门艰难，但是一旦入门，高手也就在你的不远处！ AI的开发离不开算法那我们就接下来开始学习算法吧！ 回归方法是对数值型连续随机变量进行预测和建模的监督学习算法。其特点是标注的数据集具有数值型的目标变量。回归的目的是预测数值型的目标值。

常用的回归方法包括:

  • 线性回归：使用超平面拟合数据集
  • 最近邻算法：通过搜寻最相似的训练样本来预测新样本的值
  • 决策树和回归树：将数据集分割为不同分支而实现分层学习
  • 集成方法：组合多个弱学习算法构造一种强学习算法，如随机森林（RF）和梯度提升树（GBM）等
  • 深度学习：使用多层神经网络学习复杂模型

逻辑回归

逻辑回归对应线性回归，但旨在解决分类问题，即将模型的输出转换为0/1值。逻辑回归直接对分类的可能性进行建模，无需事先假设数据的分布。

最理想的转换函数是单位阶跃函数（也称Heaviside函数），但单位阶跃函数是不连续的，没法在实际计算中使用。故而，在分类过程中更常使用对数几率函数（即sigmoid函数）：

逻辑回归 - 图1

这样，模型就变成了

逻辑回归 - 图2

如果将$y$看作是样本$x$作为正例的可能性，那么可以得到反应$x$作为正例的相对可能性对数几率（logit）

逻辑回归 - 图3

逻辑回归算法

可以使用极大似然估计法来估计参数$w$和$b$

假设逻辑回归 - 图4，似然函数为逻辑回归 - 图5，那么对数似然函数为逻辑回归 - 图6， 逻辑回归 - 图7 然后就可以使用梯度下降算法、牛顿法或者BFGS等拟牛顿法来求解了。

正则化

在模型过于复杂的情况下，模型会学习到很多特征，从而导致可能把所有训练样本都拟合到，这样就导致了过拟合。解决过拟合可以从两个方面入手，一是减少模型复杂度，一是增加训练集个数。而正则化就是减少模型复杂度的一个方法。

一般是在目标函数(经验风险)中加上一个正则化项逻辑回归 - 图8，即

逻辑回归 - 图9 而这个正则化项一般会采用L1范数或者L2范数。其形式分别为：逻辑回归 - 图10

类别不均衡问题

类别不均衡是指分类任务中不同类别的训练样例数目差别很大的情况。解决这类问题的基本思路是“再缩放（rescaling）”，即令

逻辑回归 - 图11 其中，$m^-$为反例数目，$m^+$为正例数目）

然而，这个方法的实际操作却很难。实际使用上通常使用的方法：

  • 欠采样：去除一些样例使得不同类别的训练样例数目平衡。注意随机丢弃样例可能会导致丢失一些重要信息。
  • 过采样：增加一些样例使得不同类别的训练样例数目平衡。注意不能简单对原样本重复采样，否则会导致严重的过拟合
  • 直接基于原始训练集进行学习，但在使用最终模型预测时使用再缩放（也称为阈值移动）

示例代码

  2. import os
  3. import tensorflow as tf
  5. # initialize variables/model parameters
  6. W = tf.Variable(tf.zeros([5, 1]), name="weights")
  7. b = tf.Variable(0., name="bias")
  9. def read_csv(batch_size, file_name, record_defaults):
  10.     filename_queue = tf.train.string_input_producer(
  11.         [os.path.dirname(__file__) + "/" + file_name])
  13.     reader = tf.TextLineReader(skip_header_lines=1)
  14.     key, value = reader.read(filename_queue)
  16.     # decode_csv will convert a Tensor from type string (the text line) in
  17.     # a tuple of tensor columns with the specified defaults, which also
  18.     # sets the data type for each column
  19.     decoded = tf.decode_csv(value, record_defaults=record_defaults)
  21.     # batch actually reads the file and loads "batch_size" rows in a single
  22.     # tensor
  23.     return tf.train.shuffle_batch(decoded,
  24.                                   batch_size=batch_size,
  25.                                   capacity=batch_size * 50,
  26.                                   min_after_dequeue=batch_size)
  28. def inference(X):
  29.     # compute inference model over data X and return the result
  30.     return tf.sigmoid(tf.matmul(X, W) + b)
  32. def loss(X, Y):
  33.     # compute loss over training data X and expected outputs Y
  34.     return tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(
  35.         tf.matmul(X, W) + b, Y))
  37. def inputs():
  38.     # data is downloaded from https://www.kaggle.com/c/titanic/data.
  39.     passenger_id, survived, pclass, name, sex, age, sibsp, parch, ticket, fare,\
  40.         cabin, embarked = read_csv(100,
  41.                                    "train.csv",
  42.                                    [[0.0], [0.0], [0], [""],
  43.                                     [""], [0.0], [0.0], [0.0],
  44.                                     [""], [0.0], [""], [""]])
  46.     # convert categorical data
  47.     is_first_class = tf.to_float(tf.equal(pclass, [1]))
  48.     is_second_class = tf.to_float(tf.equal(pclass, [2]))
  49.     is_third_class = tf.to_float(tf.equal(pclass, [3]))
  50.     gender = tf.to_float(tf.equal(sex, ["female"]))
  52.     # Finally we pack all the features in a single matrix;
  53.     # We then transpose to have a matrix with one example per row and one
  54.     # feature per column.
  55.     features = tf.transpose(
  56.         tf.pack([is_first_class,
  57.                  is_second_class,
  58.                  is_third_class,
  59.                  gender,
  60.                  age]))
  61.     survived = tf.reshape(survived, [100, 1])
  63.     return features, survived
  66. def train(total_loss):
  67.     # train / adjust model parameters according to computed total loss
  68.     learning_rate = 0.01
  69.     return tf.train.GradientDescentOptimizer(learning_rate).minimize(
  70.         total_loss)
  73. def evaluate(sess, X, Y):
  74.     # evaluate the resulting trained model
  75.     predicted = tf.cast(inference(X) > 0.5, tf.float32)
  77.     print sess.run(tf.reduce_mean(tf.cast(tf.equal(predicted, Y), tf.float32)))
  79. # Create a saver.
  80. # saver = tf.train.Saver()
  82. # Launch the graph in a session, setup boilerplate
  83. with tf.Session() as sess:
  84.     tf.initialize_all_variables().run()
  86.     X, Y = inputs()
  88.     total_loss = loss(X, Y)
  89.     train_op = train(total_loss)
  91.     coord = tf.train.Coordinator()
  92.     threads = tf.train.start_queue_runners(sess=sess, coord=coord)
  94.     # actual training loop
  95.     training_steps = 1000
  96.     for step in range(training_steps):
  97.         sess.run([train_op])
  98.         # for debugging and learning purposes, see how the loss gets decremented
  99.         # through training steps
  100.         if step % 100 == 0:
  101.             print "loss at step ", step, ":", sess.run([total_loss])
  102.         # save training checkpoints in case loosing them
  103.         # if step % 1000 == 0:
  104.         #     saver.save(sess, 'my-model', global_step=step)
  106.     evaluate(sess, X, Y)
  107.     coord.request_stop()
  108.     coord.join(threads)
  109.     # saver.save(sess, 'my-model', global_step=training_steps)
  2. loss at step  0 : [1.0275139]
  3. loss at step  100 : [1.389969]
  4. loss at step  200 : [1.4667224]
  5. loss at step  300 : [0.67178178]
  6. loss at step  400 : [0.568793]
  7. loss at step  500 : [0.48835525]
  8. loss at step  600 : [1.0899736]
  9. loss at step  700 : [0.84278578]
  10. loss at step  800 : [1.0500686]
  11. loss at step  900 : [0.89417559]
  12. 0.72

minst回归示例

  1. import tensorflow as tf
  2. import numpy as np
  3. import input_data
  5. # Import MINST data
  6. mnist = input_data.read_data_sets("../MNIST_data/", one_hot=True)
  7. Extracting ../MNIST_data/train-images-idx3-ubyte.gz
  8. Extracting ../MNIST_data/train-labels-idx1-ubyte.gz
  9. Extracting ../MNIST_data/t10k-images-idx3-ubyte.gz
  10. Extracting ../MNIST_data/t10k-labels-idx1-ubyte.gz
  13. # Parameters
  14. learning_rate = 0.01
  15. training_epochs = 25
  16. batch_size = 100
  17. display_step = 1
  19. # tf Graph Input
  20. x = tf.placeholder("float", [None, 784]) # mnist data image of shape 28*28=784
  21. y = tf.placeholder("float", [None, 10]) # 0-9 digits recognition => 10 classes
  23. # Create model
  24. def init_weights(shape):
  25.     return tf.Variable(tf.random_normal(shape, stddev=0.01))
  27. def model(X, w):
  28.     return tf.matmul(X, w)
  30. # like in linear regression, we need a shared variable weight matrix
  31. # for logistic regression
  32. w = init_weights([784, 10]) 
  34. # Construct model
  35. # compute mean cross entropy (softmax is applied internally)
  36. cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(model(x, w), y)) 
  37. train_op = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost) # construct optimizer
  38. predict_op = tf.argmax(model(x, w), 1) # at predict time, evaluate the argmax of the logistic regression
  40. # Launch the graph
  41. with tf.Session() as sess:
  42.     tf.initialize_all_variables().run()
  44.     # Training cycle
  45.     for epoch in range(training_epochs):
  46.         avg_cost = 0.
  47.         total_batch = int(mnist.train.num_examples/batch_size)
  48.         # Loop over all batches
  49.         for i in range(total_batch):
  50.             batch_xs, batch_ys = mnist.train.next_batch(batch_size)
  51.             # Fit training using batch data
  52.             sess.run(train_op, feed_dict={x: batch_xs, y: batch_ys})
  53.             # Compute average loss
  54.             avg_cost += sess.run(cost, feed_dict={x: batch_xs, y: batch_ys})/total_batch
  55.         # Display logs per epoch step
  56.         if epoch % display_step == 0:
  57.             print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(avg_cost)
  59.     print "Optimization Finished!"
  61.     # Test model
  62.     correct_prediction = tf.equal(predict_op, tf.argmax(y, 1))
  63.     # Calculate accuracy
  64.     accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  65.     print "Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})
  1. Epoch: 0001 cost= 1.181141054
  2. Epoch: 0002 cost= 0.664358092
  3. Epoch: 0003 cost= 0.553026987
  4. Epoch: 0004 cost= 0.499294951
  5. Epoch: 0005 cost= 0.466518660
  6. Epoch: 0006 cost= 0.443856266
  7. Epoch: 0007 cost= 0.427351894
  8. Epoch: 0008 cost= 0.414347254
  9. Epoch: 0009 cost= 0.403219846
  10. Epoch: 0010 cost= 0.394844531
  11. Epoch: 0011 cost= 0.387121435
  12. Epoch: 0012 cost= 0.380693078
  13. Epoch: 0013 cost= 0.375634897
  14. Epoch: 0014 cost= 0.369904718
  15. Epoch: 0015 cost= 0.365776612
  16. Epoch: 0016 cost= 0.361626607
  17. Epoch: 0017 cost= 0.358361928
  18. Epoch: 0018 cost= 0.354674878
  19. Epoch: 0019 cost= 0.351685582
  20. Epoch: 0020 cost= 0.349124772
  21. Epoch: 0021 cost= 0.346287186
  22. Epoch: 0022 cost= 0.344134942
  23. Epoch: 0023 cost= 0.341778976
  24. Epoch: 0024 cost= 0.340130984
  25. Epoch: 0025 cost= 0.337454195
  26. Optimization Finished!
  27. Accuracy: 0.9122

sklearn示例

  1. from sklearn import datasets
  2. from sklearn import metrics
  3. import tensorflow as tf
  4. import tensorflow.contrib.layers.python.layers as layers
  5. import tensorflow.contrib.learn.python.learn as learn
  7. iris = datasets.load_iris()
  9. def my_model(features, labels):
  10.   """DNN with three hidden layers."""
  11.   # Convert the labels to a one-hot tensor of shape (length of features, 3) and
  12.   # with a on-value of 1 for each one-hot vector of length 3.
  13.   labels = tf.one_hot(labels, 3, 1, 0)
  15.   # Create three fully connected layers respectively of size 10, 20, and 10.
  16.   features = layers.stack(features, layers.fully_connected, [10, 20, 10])
  18.   # Create two tensors respectively for prediction and loss.
  19.   prediction, loss = (
  20.       tf.contrib.learn.models.logistic_regression(features, labels)
  21.   )
  23.   # Create a tensor for training op.
  24.   train_op = tf.contrib.layers.optimize_loss(
  25.       loss, tf.contrib.framework.get_global_step(), optimizer='Adagrad',
  26.       learning_rate=0.1)
  28.   return {'class': tf.argmax(prediction, 1), 'prob': prediction}, loss, train_op
  30. classifier = learn.Estimator(model_fn=my_model)
  31. classifier.fit(iris.data, iris.target, steps=1000)
  33. y_predicted = [
  34.   p['class'] for p in classifier.predict(iris.data, as_iterable=True)]
  35. score = metrics.accuracy_score(iris.target, y_predicted)
  36. print('Accuracy: {0:f}'.format(score))