{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"# TF Learn Model Selection Demo\n",
"\n",
"We'll be following the [TF Learn Quickstart Tutorial](http://tflearn.org/tutorials/quickstart.html), but instead of building one model, we are going to build many to find the optimal network structure."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Packages and versions\n",
"As of Feb 22, 2017, there are some issues between the latest releases of tensorflow and tflearn. The following was tested on tflearn v0.2.1 and tensorflow v0.12.1. Through pip, that is 'pip3 install --upgrade tensorflow==0.12.1 tflearn==0.2.1'. If other versions are installed, you may have to run 'pip3 uninstall tensorflow tflearn' first. To see which versions are currently installed, run 'pip3 show tensorflow tflearn'.\n",
"\n",
"To complete this tutorial, you will need to update some of the package files.\n",
"- Download [titanic.py](https://github.com/tflearn/tflearn/blob/master/tflearn/datasets/titanic.py) and copy it into the tflearn/datasets/ directory (MacOS path '/usr/local/lib/python3.6/site-packages/tflearn/datasets/').\n",
"- Download [data_utils.py](https://github.com/tflearn/tflearn/blob/master/tflearn/data_utils.py) and copy it into the tflear directory (MacOS path '/usr/local/lib/python3.6/site-packages/tflearn/').\n",
"\n",
"This will get you through step 5, where I haven't continued debugging. (Mostly because it's not worth your time to train 1600 different models on a trivial problem). Let the notebook throw some errors and get a feel for how you might go about generating and testing candidate models for your own problems.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"import tflearn"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### Step 1 - Preprocess the dataset\n",
"In raw TensorFlow, we need train_x, test_x, train_y, test_y, and next_batch(n).\n",
"\n",
"In TF Learn, we need all but next_batch(n). It isn't a bad idea to have though.\n",
"\n",
"Because we're following a tutorial, TF Learn ships with a [script](https://github.com/tflearn/tflearn/blob/master/tflearn/datasets/titanic.py) that implements most of what we need. Tutorials on [more advanced data](https://github.com/tflearn/tflearn/tree/master/tflearn/datasets) are better examples of this stage. We won't even split this data into train/cross-validation/test. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"# Download the Titanic dataset\n",
"from tflearn.datasets import titanic\n",
"titanic.download_dataset('titanic_dataset.csv')\n",
"\n",
"# Load CSV file, indicate that the first column represents labels\n",
"from tflearn.data_utils import load_csv\n",
"data, labels = load_csv('titanic_dataset.csv', target_column=0,\n",
" categorical_labels=True, n_classes=2)\n",
"\n",
"# Preprocessing function\n",
"def preprocess(data, columns_to_ignore):\n",
" # Sort by descending id and delete columns\n",
" for id in sorted(columns_to_ignore, reverse=True):\n",
" [r.pop(id) for r in data]\n",
" for i in range(len(data)):\n",
" # Converting 'sex' field to float (id is 1 after removing labels column)\n",
" data[i][1] = 1. if data[i][1] == 'female' else 0.\n",
" return np.array(data, dtype=np.float32)\n",
"\n",
"# Ignore 'name' and 'ticket' columns (id 1 & 6 of data array)\n",
"to_ignore=[1, 6]\n",
"\n",
"# Preprocess data\n",
"data = preprocess(data, to_ignore)\n",
"print(data.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### Step 2 - Identify Candidate Models\n",
"This is a fairly simple dataset. A decision tree would probably be the best. But we're interested in deep learning, so let's only consider full-connected neural networks.\n",
"\n",
"### Step 3 - Implement Candidate Models\n",
"It's always a good idea to understand what's going on inside a model before trying to abstract its creation to a function and vary its parameters. We'll build a simple full-connected neural network and verify that it works."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"# Build neural network\n",
"net = tflearn.input_data(shape=[None, 6])\n",
"net = tflearn.fully_connected(net, 32)\n",
"net = tflearn.fully_connected(net, 32)\n",
"net = tflearn.fully_connected(net, 2, activation='softmax')\n",
"net = tflearn.regression(net)\n",
"\n",
"# Define model\n",
"model = tflearn.DNN(net)\n",
"# Start training (apply gradient descent algorithm)\n",
"model.fit(data, labels, n_epoch=10, batch_size=16, show_metric=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"# Evaluate the final product\n",
"print(model.evaluate(data, labels))"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### Step 4 - Model Generation Functions\n",
"Now we'll abstract the creation of a model to a function. That function will need to take as input any parameteres we'd like to vary. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"def make_fully_connected(input_shape, output_shape, activation, layers, nodes, dropout, optimizer, loss):\n",
" tflearn.init_graph()\n",
" net = tflearn.input_data(shape=[None, input_shape])\n",
" for l in range(layers):\n",
" net = tflearn.fully_connected(net, nodes)\n",
" if (dropout != 0) and (l%2==1):\n",
" net = tflearn.dropout(net, dropout)\n",
" net = tflearn.fully_connected(net, output_shape, activation=activation)\n",
" net = tflearn.regression(net, optimizer=optimizer, loss=loss)\n",
" return tflearn.DNN(net)"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### Step 5 - Dumb Candidate Evaluation\n",
"Just to make a point, let's try some brute force. Let's start by creating a dictionary to store our candidates. The key will be an arbitrary model number. The value will be a list of [[model_parameters], score] where all scores are initially zero"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"model_dict = {}\n",
"i = 1\n",
"for layers in range(1,5+1):\n",
" for n in range(1,8+1):\n",
" for d in range(0, 3+1):\n",
" for activate in ['linear', 'tanh', 'sigmoid', 'softmax', 'relu']:\n",
" for optimizer in ['adam', 'sgd']:\n",
" for loss in ['categorical_crossentropy']:\n",
" nodes = n*5\n",
" dropout = 0.25*d\n",
" model_parameters = [6, 2, activate, layers, nodes, dropout, optimizer, loss]\n",
" key = 'model_%d' % i\n",
" i += 1\n",
" model_dict[key]=[model_parameters, 0]\n",
"\n",
"print(len(model_dict.keys())) "
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"Next, we'll iterate through the dictionary, sending each [model_parameters] to make_fully_connected() and then training the returned model. Then we'll set the corresponding score value to the output of model.evaluate(). "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"for k in sorted(model_dict.keys()):\n",
" p = model_dict[k][0]\n",
" print('\\rModel: %s' % str(p))\n",
" model = make_fully_connected(p[0], p[1], p[2], p[3], p[4], p[5], p[6], p[7])\n",
" model.fit(data, labels, n_epoch=10, batch_size=16, show_metric=True)\n",
" score = model.evaluate(data, labels)\n",
" print('| Score: %.4f' % score, end='')\n",
" model_dict[k][1] = score"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"And now we can find the highest score in the dictionary, and report the 'best' model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"deletable": true,
"editable": true
},
"outputs": [],
"source": [
"best_score = 0\n",
"\n",
"for k in model_dict.keys():\n",
" if model_dict[k][1] > best_score:\n",
" best_score = model_dict[k][1]\n",
" best_model = model_dict[k][0]\n",
" \n",
"print('Best Score: %.4f' % best_score)\n",
"print('Best Model: %s' % str(best_model))"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": true,
"editable": true
},
"source": [
"### Step 6 - Smart Candidate Evaluation\n",
"Dumb evaluation wastes a lot of time on models that clearly won't work well. Plus, we evaluate models on more than just a single value. Maybe we want to consider test accuracy and train accuracy. Or time per batch (important in a more deployment focused environment). Or type-1 versus type-2 error. This forces us to spend a lot of time evaluating results. How can we make this easier? Consider two options, although a complete solution probably involves both to some degree.\n",
"\n",
"##### tpot\n",
"[GitHub]()\n",
"This library works with sklearn, not tflearn. But the general idea is the same. First, devise a binary encoding of your parameters. Then devise some evaluation function (that can be a mixture of various performance metrics). Finally, slap the whole thing into a genetic algorithm and let Darwin do the work.\n",
"\n",
"##### TensorBoard\n",
"[Dev Summit Demo](https://www.youtube.com/watch?v=eBbEDRsCmv4)\n",
"TensorBoard helps to visualize what's going on within a network. It allows you to group different candidate models into categories and compare their performance in a browser.\n",
"\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}