{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cbb752b22_hw2.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Biomedical Data Science 2022: Homework Assignment 2\n",
        "### CBB & CPSC & S&DS (programming) Assignment\n",
        "\n",
        "Due: May 11th (Wed) 11:59pm EST\n",
        "\n",
        "#### Name:  write your name here (double click to edit)"
      ],
      "metadata": {
        "id": "2KEaADgvRO0Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Instructions\n",
        "\n",
        "* You only need to write code between the `### START CODE HERE ###` and `### END CODE HERE ###` comments. \n",
        "* You may write code outside of these blocks, but it will not be graded.\n",
        "* If you make use of any online resource please cite the source in the code comments. \n",
        "* You may use some small utility functions directly, but notice that directly copying large chunks of codes (even with variable name replacement) are not allowed and will be considered as plagrism.\n",
        "* After writing your code, you can run the cell by either pressing \"SHIFT\"+\"ENTER\" or by clicking on \"Run Cell\" (denoted by a play symbol) in the upper bar of the notebook. \n",
        "* Some questions require a written answer. To edit a text block in Colab, double click the text cell.\n",
        "* After you are finished, turn in the .ipynb file to canvas (File->Download->Download .ipynb). "
      ],
      "metadata": {
        "id": "jm4IYMKizd-k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Part 1: Calculating dihedral angles of proteins\n"
      ],
      "metadata": {
        "id": "kPXgPj2MSKuk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1.1 (25pts) \n",
        "Ramachandran plots allow us to investigate the sterically allowed and disallowed backbone\n",
        "dihedral angle combinations φ and ψ in proteins. Using the file core_THR_residues.txt provided,\n",
        "produce a Ramachandran plot for threonine residues. The file core_THR_residues.txt contains atomic coordinates for \n",
        "1000 threonine dipeptides taken from a database of high-resolution protein crystal structures. The\n",
        "Cα, carboxyl carbon, and oxygen atoms on the prior amino acid are labelled pCa, pC, and pO. The N,\n",
        "Cα and H atoms on the subsequent amino acid are labeled: nN, nCa and nH. Using this file, calculate\n",
        "φ and ψ for each residue and produce a Ramachandran plot similar to that shown in below. See the\n",
        "lecture notes for definitions of φ and ψ. "
      ],
      "metadata": {
        "id": "nhyZoJ78zH44"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6V095gvORNBI",
        "outputId": "3b374b7c-d16e-473e-ff1f-e6016df8efae"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([['1', 'pCA', '0.796000', '-4.634000', '-35.564999'],\n",
              "       ['1', 'pC', '1.707000', '-5.823000', '-35.827999'],\n",
              "       ['1', 'pO', '1.546000', '-6.572000', '-36.794998'],\n",
              "       ...,\n",
              "       ['1000', 'nN', '-9.844000', '-788.841980', '-20.346001'],\n",
              "       ['1000', 'nCA', '-11.077000', '-788.564026', '-19.641001'],\n",
              "       ['1000', 'nH', '-9.906000', '-788.838989', '-21.204000']],\n",
              "      dtype='<U11')"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ],
      "source": [
        "import numpy as np\n",
        "import requests\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "core_THR_residues_URL = 'http://files2.gersteinlab.org/public-docs/2021/04.28/core_THR_residues.txt'\n",
        "core_THR_residues = requests.get(core_THR_residues_URL).text.splitlines()\n",
        "core_THR_residues = np.loadtxt(core_THR_residues, dtype= str)\n",
        "np.set_printoptions(threshold=40)\n",
        "\n",
        "core_THR_residues"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def calculate_backbone_dihedrals(data):\n",
        "  ### START CODE HERE ### \n",
        "\n",
        "  # replace this code, add comments:\n",
        "  phi = np.random.uniform(-180,180, 100) \n",
        "  psi = np.random.uniform(-180,180, 100)\n",
        "\n",
        "  ### END CODE HERE ###\n",
        "  return phi, psi\n",
        "\n",
        "\n",
        "phi, psi = calculate_backbone_dihedrals(core_THR_residues)\n",
        "\n",
        "plt.xticks(np.arange(-180, 181, step=60))\n",
        "plt.yticks(np.arange(-180, 181, step=60))\n",
        "plt.xlabel('$\\phi$')\n",
        "plt.ylabel('$\\psi$')\n",
        "_ = plt.scatter(phi, psi, s = 20, facecolors='none', edgecolors='k')"
      ],
      "metadata": {
        "id": "EgSC-B3RyT2b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Expected Output:**\n",
        "\n",
        "![](http://files2.gersteinlab.org/public-docs/2022/04.26/hw2/hw2_img1.png)"
      ],
      "metadata": {
        "id": "e-Bj7Gs9BsDU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1.2 (25pts) \n",
        "In the lecture notes, we not only discussed backbone dihedral angles φ and ψ, but we also discussed sidechain dihedral angles. As the side chains have different numbers of atoms, they can have different numbers of sidechain dihedral angles. In the case of threonine, there is only one sidechain dihedral angle χ1. Generate the observed side chain dihedral angle distribution from core_THR_residues.txt discussed in question 1. The observed distribution should be similar to that shown in below. See the lecture notes for the definition of χ1 in threonine."
      ],
      "metadata": {
        "id": "_GDO2OgZCH-W"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def calculate_threonine_dihedrals(data):\n",
        "  ### START CODE HERE ### \n",
        "\n",
        "  # replace this code, add comments:\n",
        "  chi = np.random.uniform(-180,180, 10000)\n",
        "\n",
        "  ### END CODE HERE ###\n",
        "  return chi\n",
        "\n",
        "chi = calculate_threonine_dihedrals(core_THR_residues)\n",
        "values, binedges = np.histogram(chi, bins = 60, density = True)\n",
        "x = 0.5*(binedges[:-1] + binedges[1:])\n",
        "\n",
        "plt.xticks(np.arange(-180, 181, step=60))\n",
        "plt.xlabel('$\\chi_1(\\circ)$')\n",
        "plt.ylabel('$P(\\chi_1)$')\n",
        "_ = plt.plot(x, values, 'k')"
      ],
      "metadata": {
        "id": "duP9igAoDWV6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Expected Output:**\n",
        "\n",
        "![](http://files2.gersteinlab.org/public-docs/2022/04.26/hw2/hw2_img2.png)"
      ],
      "metadata": {
        "id": "KZpDpCKFG2_J"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Part 2: A simple neural network for finding transcription factor binding sites\n",
        "\n",
        "The following protocol is adapted from a colab notebook for one of our discussion section papers: **A Primer on Deep Learning in Genomics** (*Nature Genetics, 2018*) by James Zou, Mikael Huss, Abubakar Abid, Pejman Mohammadi, Ali Torkamani & Amalio Telentil. [paper link](https://www.nature.com/articles/s41588-018-0295-5). \n",
        "\n",
        "We will design a neural network that can discover binding motifs in DNA based on the results of an assay that determines whether a longer DNA sequence binds to the protein or not. Here, the longer DNA sequences are our *independent variables* (or *predictors*), while the positive or negative response of the assay is the *dependent variable* (or *response*).\n",
        "\n",
        "We will use simulated data that consists of DNA sequences of length 50 bases (chosen to be artificially short so that the data is easy to play around with), and is labeled with 0 or 1 depending on the result of the assay. Our goal is to build a classifier that can predict whether a particular sequence will bind to the protein and discover the short motif that is the binding site in the sequences that are bound to the protein.\n",
        "\n",
        "(Spoiler alert: the true regulatory motif is *`CGACCGAACTCC`*. Of course, the neural network doesn't know this.)\n"
      ],
      "metadata": {
        "id": "Dr2K-XeOHUka"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "#### **Instructions for part 2**: Section 2.1-2.4 will preprocess data, set up a basic network, and run evaluation for you on that network. **You only need to read and run these cells.** In section 2.5 you will be doing hyperparameter tuning to optimize this model. You may edit code in all sections, but **only section 2.5 is graded.**"
      ],
      "metadata": {
        "id": "3STtpshr7K8G"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aK7wr8n8gzQ_"
      },
      "source": [
        "## 2.1 - Curate the Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QRMSFdUSubgX"
      },
      "source": [
        "In order to train the neural network, we must load and preprocess the data, which consists of DNA sequences and their corresponding labels.By processing this data, the network will learn to distinguish sequences that bind to the transcription factor from those that do not. We will split the data into three different sub-datasets:\n",
        "\n",
        "(1) Training dataset: a dataset used to fit the parameters of a model or to define the weights of connections between neurons of a neural network.\n",
        "\n",
        "(2) Validation dataset: a second dataset used to minimize overfitting. The weights of the network are not adjusted with this data set. After each training cycle, if the accuracy over the training data set increases, but the accuracy over the validation data set stays the same or decreases, then there is overfitting on the neural network.\n",
        "\n",
        "(3) Testing dataset: is a third dataset not included in the training nor validation data sets. After all the training and validation cycles are complete, this dataset is used only for testing the final solution in order to measure the actual predictive power of the neural network on new examples.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B_F7VoAMhLiX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "62d28872-7423-4c5f-b0a5-e7a27ed49468"
      },
      "source": [
        "# this notebook requires an earlier version of tensorflow \n",
        "# ignore any warnings created by this\n",
        "!pip install tensorflow==1.13.2 --quiet\n",
        "\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import requests\n",
        "\n",
        "tf.set_random_seed(0)\n",
        "np.random.seed(0) \n",
        "\n",
        "SEQUENCES_URL = 'https://raw.githubusercontent.com/abidlabs/deep-learning-genomics-primer/master/sequences.txt'\n",
        "\n",
        "sequences = requests.get(SEQUENCES_URL).text.split('\\n')\n",
        "sequences = list(filter(None, sequences))  # This removes empty sequences.\n",
        "\n",
        "# Let's print the first few sequences.\n",
        "pd.DataFrame(sequences, index=np.arange(1, len(sequences)+1), \n",
        "             columns=['Sequences']).head()"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                           Sequences\n",
              "1  CCGAGGGCTATGGTTTGGAAGTTAGAACCCTGGGGCTTCTCGCGGA...\n",
              "2  GAGTTTATATGGCGCGAGCCTAGTGGTTTTTGTACTTGTTTGTCGC...\n",
              "3  GATCAGTAGGGAAACAAACAGAGGGCCCAGCCACATCTAGCAGGTA...\n",
              "4  GTCCACGACCGAACTCCCACCTTGACCGCAGAGGTACCACCAGAGC...\n",
              "5  GGCGACCGAACTCCAACTAGAACCTGCATAACTGGCCTGGGAGATA..."
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-eaaac263-c2bb-41ac-80fc-9f806ccfc1ac\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Sequences</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>CCGAGGGCTATGGTTTGGAAGTTAGAACCCTGGGGCTTCTCGCGGA...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>GAGTTTATATGGCGCGAGCCTAGTGGTTTTTGTACTTGTTTGTCGC...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>GATCAGTAGGGAAACAAACAGAGGGCCCAGCCACATCTAGCAGGTA...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>GTCCACGACCGAACTCCCACCTTGACCGCAGAGGTACCACCAGAGC...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>GGCGACCGAACTCCAACTAGAACCTGCATAACTGGCCTGGGAGATA...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-eaaac263-c2bb-41ac-80fc-9f806ccfc1ac')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-eaaac263-c2bb-41ac-80fc-9f806ccfc1ac button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-eaaac263-c2bb-41ac-80fc-9f806ccfc1ac');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bzsbNHqWiFek"
      },
      "source": [
        "The next  step is to organize the data into a format that can be passed into a deep learning algorithm. Most deep learning algorithms accept data in the form of vectors or matrices (or more generally, tensors). \n",
        "\n",
        "To get each DNA sequence in the form of a matrix, we use _one-hot encoding_, which encodes every base in a sequence in the form of a 4-dimensional vector, with a separate dimension for each base. We place a \"1\" in the dimension corresponding to the base found in the DNA sequence, and \"0\"s in all other slots. We then concatenate these 4-dimensional vectors together along the bases in the sequence to form a matrix. \n",
        "\n",
        "In the cell below, we one-hot encode the simulated DNA sequences, and show an example of what the one-hot encoded sequence looks like:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IPJD6PuDnaS6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "402d4cc3-000b-4c18-c811-dff6ec94cce0"
      },
      "source": [
        "from sklearn.preprocessing import LabelEncoder, OneHotEncoder\n",
        "\n",
        "# The LabelEncoder encodes a sequence of bases as a sequence of integers.\n",
        "integer_encoder = LabelEncoder()  \n",
        "# The OneHotEncoder converts an array of integers to a sparse matrix where \n",
        "# each row corresponds to one possible value of each feature.\n",
        "one_hot_encoder = OneHotEncoder(categories='auto')   \n",
        "input_features = []\n",
        "\n",
        "for sequence in sequences:\n",
        "  integer_encoded = integer_encoder.fit_transform(list(sequence))\n",
        "  integer_encoded = np.array(integer_encoded).reshape(-1, 1)\n",
        "  one_hot_encoded = one_hot_encoder.fit_transform(integer_encoded)\n",
        "  input_features.append(one_hot_encoded.toarray())\n",
        "\n",
        "np.set_printoptions(threshold=40)\n",
        "input_features = np.stack(input_features)\n",
        "print(\"Example sequence\\n-----------------------\")\n",
        "print('DNA Sequence #1:\\n',sequences[0][:10],'...',sequences[0][-10:])\n",
        "print('One hot encoding of Sequence #1:\\n',input_features[0].T)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Example sequence\n",
            "-----------------------\n",
            "DNA Sequence #1:\n",
            " CCGAGGGCTA ... CGCGGACACC\n",
            "One hot encoding of Sequence #1:\n",
            " [[0. 0. 0. ... 1. 0. 0.]\n",
            " [1. 1. 0. ... 0. 1. 1.]\n",
            " [0. 0. 1. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AbBmrEVGrahN"
      },
      "source": [
        "Similarly, we can go ahead and load the labels (_response variables_). In this case, the labels are structured as follows: a \"1\" indicates that a protein bound to the sequence, while a \"0\" indicates that the protein did not. While we could use the labels as a vector, it is often easier to similarly one-hot encode the labels, as we did the features. We carry out that here:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IA9FJeQkr1Ze",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "36898cff-f5b8-4cad-ee44-9d3d0b575ae7"
      },
      "source": [
        "LABELS_URL = 'https://raw.githubusercontent.com/abidlabs/deep-learning-genomics-primer/master/labels.txt'\n",
        "\n",
        "labels = requests.get(LABELS_URL).text.split('\\n')\n",
        "labels = list(filter(None, labels))  # removes empty sequences\n",
        "\n",
        "one_hot_encoder = OneHotEncoder(categories='auto')\n",
        "labels = np.array(labels).reshape(-1, 1)\n",
        "input_labels = one_hot_encoder.fit_transform(labels).toarray()\n",
        "\n",
        "print('Labels:\\n',labels.T)\n",
        "print('One-hot encoded labels:\\n',input_labels.T)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Labels:\n",
            " [['0' '0' '0' ... '0' '1' '1']]\n",
            "One-hot encoded labels:\n",
            " [[1. 1. 1. ... 1. 0. 0.]\n",
            " [0. 0. 0. ... 0. 1. 1.]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MTokFzDZvQR-"
      },
      "source": [
        "We also go ahead and split the data into training and test sets. The purpose of the test set is to ensure that we can observe the performance of the model on new data, not seen previously during training. At a later step, we will further partition the training set into a training and validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P_7LKgvc3Lnn"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "train_features, test_features, train_labels, test_labels = train_test_split(\n",
        "    input_features, input_labels, test_size=0.25, random_state=42)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7LQp2ZFrg6dm"
      },
      "source": [
        "## 2.2 - Select the Architecture and Train"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xBT6Q3j-sjhh"
      },
      "source": [
        "![alt text](https://github.com/abidlabs/deep-learning-genomics-primer/blob/master/Screenshot%20from%202018-08-01%2020-31-49.png?raw=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "krHJgtK_rzif"
      },
      "source": [
        "Next, we choose a neural network architecture to train the model. In this tutorial, we choose a simple 1D convolutional neural network (CNN), which is commonly used in deep learning for functional genomics applications.\n",
        "\n",
        "A CNN learns to recognize patterns that are generally invariant across space, by trying to match the input sequence to a number of learnable \"filters\" of a fixed size. In our dataset, the filters will be motifs within the DNA sequences. The CNN may then learn to combine these filters to recognize a larger structure (e.g. the presence or absence of a transcription factor binding site). \n",
        "\n",
        "We will use the deep learning library `Keras`. As of 2017, `Keras` has been integrated into `TensorFlow`,  which makes it very easy to construct neural networks. We only need to specify the kinds of layers we would like to include in our network, and the dimensionality of each layer. The CNN we generate in this example consists of the following layers:\n",
        "\n",
        "- _Conv1D_: We define our convolutional layer to have 32 filters of size 12 bases.\n",
        "\n",
        "- _MaxPooling1D_: After the convolution, we use a pooling layer to down-sample the output of the each of the 32 convolutional filters. Though not always required, this is a typical form of non-linear down-sampling used in CNNs.\n",
        "\n",
        "- _Flatten_: This layer flattens the output of the max pooling layer, combining the results of the convolution and pooling layers across all 32 filters. \n",
        "\n",
        "- _Dense_: The first Dense tensor creates a layer (dense_1) that compresses the representation of the flattened layer, resulting in smaller layer with 16 tensors, and the second Dense function converges the tensors into the output layer (dense_2) that consists of the two possible response values (0 or 1).\n",
        "\n",
        "We can see the details of the architecture of the neural network we have created by running `model.summary()`, which prints the dimensionality and number of parameters for each layer in our network. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dU3imaIns80_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "12fa6aca-b110-45a9-cfa0-c179169aec54"
      },
      "source": [
        "from tensorflow.keras.layers import Conv1D, Dense, MaxPooling1D, Flatten\n",
        "from tensorflow.keras.models import Sequential\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Conv1D(filters=32, kernel_size=12, \n",
        "                 input_shape=(train_features.shape[1], 4)))\n",
        "model.add(MaxPooling1D(pool_size=4))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(16, activation='relu'))\n",
        "model.add(Dense(2, activation='softmax'))\n",
        "\n",
        "model.compile(loss='binary_crossentropy', optimizer='adam', \n",
        "              metrics=['binary_accuracy'])\n",
        "model.summary()"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv1d (Conv1D)              (None, 39, 32)            1568      \n",
            "_________________________________________________________________\n",
            "max_pooling1d (MaxPooling1D) (None, 9, 32)             0         \n",
            "_________________________________________________________________\n",
            "flatten (Flatten)            (None, 288)               0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 16)                4624      \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 2)                 34        \n",
            "=================================================================\n",
            "Total params: 6,226\n",
            "Trainable params: 6,226\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qSOUwoG_vHRA"
      },
      "source": [
        "Now, we are ready to go ahead and train the neural network. We will further divide the training set into a training and validation set. We will train only on the reduced training set, but plot the loss curve on both the training and validation sets. Once the loss for the validation set stops improving or gets worse throughout the learning cycles, it is time to stop training because the model has already converged and may be just overfitting."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LSOmHIM83hXO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 349
        },
        "outputId": "8121719c-7533-4294-b229-36499579d1a4"
      },
      "source": [
        "history = model.fit(train_features, train_labels, \n",
        "                    epochs=50, verbose=0, validation_split=0.5)\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(history.history['loss'])\n",
        "plt.plot(history.history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'validation'])\n",
        "plt.show()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXwV9dn//9d1TvaVkIRACJuCkrBIICCCKIpa3HAXq7baRbve1l+XX23vu9raere921pri23x1tZal1qsFlu9rShudSO4IBD2NUEgCWTfc67vHzMJhxhCApmcJHM9H4/TM/u5hsbznpnPmc+IqmKMMca/ApEuwBhjTGRZEBhjjM9ZEBhjjM9ZEBhjjM9ZEBhjjM9ZEBhjjM9ZEBjTTSLyRxH5UTeX3SEi5xzvdozpCxYExhjjcxYExhjjcxYEZlBxL8l8S0TWiEitiDwgIlki8pyIVIvIChFJC1t+kYisE5EKEXlZRHLD5uWLyLvuen8B4jp81kUi8r677hsiMvUYa75JRLaIyAERWS4i2e50EZFfish+EakSkQ9FZLI77wIRWe/WViIi3zymfzBjsCAwg9MVwLnAScDFwHPAd4FMnL/5WwBE5CTgMeBWd96zwDMiEiMiMcDTwMPAUOCv7nZx180HHgS+AKQDvweWi0hsTwoVkbOBHwNXAyOAncDj7uzzgDPc/Uh1lyl35z0AfEFVk4HJwEs9+VxjwlkQmMHo16q6T1VLgNeAt1X1PVVtAJ4C8t3lFgP/VNUXVLUZ+DkQD8wBZgPRwD2q2qyqy4BVYZ9xM/B7VX1bVVtV9SGg0V2vJ64DHlTVd1W1EfgOcJqIjAWagWRgIiCqWqSqH7nrNQN5IpKiqgdV9d0efq4x7SwIzGC0L2y4vpPxJHc4G+cIHABVDQG7gZHuvBI9vFfGnWHDY4BvuJeFKkSkAhjlrtcTHWuowTnqH6mqLwG/AZYA+0VkqYikuIteAVwA7BSRV0TktB5+rjHtLAiMn+3B+UIHnGvyOF/mJcBHwEh3WpvRYcO7gbtUdUjYK0FVHzvOGhJxLjWVAKjqvao6A8jDuUT0LXf6KlW9BBiGcwnriR5+rjHtLAiMnz0BXCgiC0QkGvgGzuWdN4A3gRbgFhGJFpHLgVlh694PfFFETnUbdRNF5EIRSe5hDY8BnxGRaW77wn/jXMraISIz3e1HA7VAAxBy2zCuE5FU95JWFRA6jn8H43MWBMa3VHUjcD3wa6AMp2H5YlVtUtUm4HLgRuAATnvC38LWLQRuwrl0cxDY4i7b0xpWAN8DnsQ5CzkRuMadnYITOAdxLh+VAz9z530K2CEiVcAXcdoajDkmYg+mMcYYf7MzAmOM8TkLAmOM8TkLAmOM8TkLAmOM8bmoSBfQUxkZGTp27NhIl2GMMQPK6tWry1Q1s7N5Ay4Ixo4dS2FhYaTLMMaYAUVEdh5pnl0aMsYYn7MgMMYYn7MgMMYYnxtwbQTGmMGlubmZ4uJiGhoaIl3KoBAXF0dOTg7R0dHdXseCwBgTUcXFxSQnJzN27FgO7+zV9JSqUl5eTnFxMePGjev2enZpyBgTUQ0NDaSnp1sI9AIRIT09vcdnVxYExpiIsxDoPcfyb+mbICjccYCfPLcB623VGGMO55sgWLenit+9spX91Y2RLsUY049UVFRw33339Xi9Cy64gIqKCg8q6nu+CYK8bOdRr+v3VEW4EmNMf3KkIGhpaelyvWeffZYhQ4Z4VVaf8k0QTBzuPEFw3Z7KCFdijOlPbrvtNrZu3cq0adOYOXMm8+bNY9GiReTl5QFw6aWXMmPGDCZNmsTSpUvb1xs7dixlZWXs2LGD3NxcbrrpJiZNmsR5551HfX19pHbnmPjm56PJcdGMSU9g/Ud2RmBMf/WDZ9b1+ll7XnYKd1w86Yjzf/KTn7B27Vref/99Xn75ZS688ELWrl3b/vPLBx98kKFDh1JfX8/MmTO54oorSE9PP2wbmzdv5rHHHuP+++/n6quv5sknn+T666/v1f3wkm+CACBvRIpdGjLGdGnWrFmH/Qb/3nvv5amnngJg9+7dbN68+WNBMG7cOKZNmwbAjBkz2LFjR5/V2xt8FQSTslN4bu1eahpbSIr11a4bMyB0deTeVxITE9uHX375ZVasWMGbb75JQkIC8+fP7/Q3+rGxse3DwWBwwF0a8k0bARxqMN5gl4eMMa7k5GSqq6s7nVdZWUlaWhoJCQls2LCBt956q4+r6xu+OizOG5EKOD8lLRg7NMLVGGP6g/T0dObOncvkyZOJj48nKyurfd7ChQv53e9+R25uLieffDKzZ8+OYKXe8VUQZKXEMjQxxtoJjDGHefTRRzudHhsby3PPPdfpvLZ2gIyMDNauXds+/Zvf/Gav1+c1X10aEhGnwdguDRljTDv/BMGWFfD0l8kbkczGfdU0t4YiXZExxvQL/gmCgzvh/UcoGFJDU0uIraU1ka7IGGP6Bf8EQXY+AFNkK2BdTRhjTBv/BEHWJAhEk1WzgdiogAWBMca4/BMEUbGQlUdg7/tMHJ5sDcbGGOPyTxAAjJgGe94nb4QTBPZsAmNMTyUlJQGwZ88errzyyk6XmT9/PoWFhV1u55577qGurq59PJLdWnsaBCKyUEQ2isgWEbntCMtcLSLrRWSdiHT+Y97ekp0PDRXMSquhoq6ZPZX2sGxjzLHJzs5m2bJlx7x+xyCIZLfWngWBiASBJcD5QB7wSRHJ67DMBOA7wFxVnQTc6lU9AGQ7nUJNDWwHrMHYGON0Q71kyZL28e9///v86Ec/YsGCBUyfPp0pU6bw97///WPr7dixg8mTJwNQX1/PNddcQ25uLpdddtlhfQ196UtfoqCggEmTJnHHHXcATkd2e/bs4ayzzuKss84CDnVrDXD33XczefJkJk+ezD333NP+eV51d+3lncWzgC2qug1ARB4HLgHWhy1zE7BEVQ8CqOp+D+uBYXkQjGFUw0ZEZrN+TxXn5mUdfT1jTN947jbY+2HvbnP4FDj/J0ecvXjxYm699Va+8pWvAPDEE0/w/PPPc8stt5CSkkJZWRmzZ89m0aJFR3we8G9/+1sSEhIoKipizZo1TJ8+vX3eXXfdxdChQ2ltbWXBggWsWbOGW265hbvvvpuVK1eSkZFx2LZWr17NH/7wB95++21UlVNPPZUzzzyTtLQ0z7q79vLS0Ehgd9h4sTst3EnASSLybxF5S0QWdrYhEblZRApFpLC0tPTYK4qKhWF5xOz7gHHpiaz/yB5SY4zf5efns3//fvbs2cMHH3xAWloaw4cP57vf/S5Tp07lnHPOoaSkhH379h1xG6+++mr7F/LUqVOZOnVq+7wnnniC6dOnk5+fz7p161i/fv2RNgPA66+/zmWXXUZiYiJJSUlcfvnlvPbaa4B33V1Huq+hKGACMB/IAV4VkSmqeliLiaouBZYCFBQUHF8Lb/Y0WPcUuaOSWVNiQWBMv9LFkbuXrrrqKpYtW8bevXtZvHgxjzzyCKWlpaxevZro6GjGjh3baffTR7N9+3Z+/vOfs2rVKtLS0rjxxhuPaTttvOru2sszghJgVNh4jjstXDGwXFWbVXU7sAknGLyTnQ8NlcxOq2L3gXoq65s9/ThjTP+3ePFiHn/8cZYtW8ZVV11FZWUlw4YNIzo6mpUrV7Jz584u1z/jjDPaO65bu3Yta9asAaCqqorExERSU1PZt2/fYR3YHan763nz5vH0009TV1dHbW0tTz31FPPmzevFvf04L4NgFTBBRMaJSAxwDbC8wzJP45wNICIZOJeKtnlYk/MTUmB6tPN/bJHdT2CM702aNInq6mpGjhzJiBEjuO666ygsLGTKlCn86U9/YuLEiV2u/6UvfYmamhpyc3O5/fbbmTFjBgCnnHIK+fn5TJw4kWuvvZa5c+e2r3PzzTezcOHC9sbiNtOnT+fGG29k1qxZnHrqqXz+858nPz+/93c6jHj5W3oRuQC4BwgCD6rqXSJyJ1CoqsvFaXn5BbAQaAXuUtXHu9pmQUGBHu33uV1qaYIfj6Q2/yYmvT6X2y/K47Onjzv6esYYTxQVFZGbmxvpMgaVzv5NRWS1qhZ0trynbQSq+izwbIdpt4cNK/B199U3omIgaxKJ5R+SkXS23WFsjPE9f91Z3CY7H/Z8QN6IZNbZvQTGGJ/zZxCMmAaNlcwZWsWW/dU0tdizCYyJJOvupfccy7+lP4PA7ZJ6ZsxOmluVzfs7f3C1McZ7cXFxlJeXWxj0AlWlvLycuLi4Hq0X6fsIImNYLgRjObF5MzCC9XuqmJSdGumqjPGlnJwciouLOa6bRU27uLg4cnJyerSOP4MgGA1Zk0itWEd89FnWYGxMBEVHRzNunP1yL5L8eWkIIDsf+egDcocnWoOxMcbXfBwE06CxitPTqynaY88mMMb4l4+DwGkwnhW7k+rGFooP9k6fHcYYM9D4NwgyJ0IwlgmtzsPs7fKQMcav/BsEwWgYPoWMqiJErM8hY4x/+TcIALKnEdz7AScMjbcgMMb4ls+DIB+aqjkjs5qivRYExhh/8ncQuF1Snxa3i90H6qlqsGcTGGP8x99BkDkRouKYqE6D8YaPrKsJY4z/+DsIglEwfApZNUWANRgbY/zJ30EAMGIaMaXrSIsPWhAYY3zJgiA7H2mq5uyMKgsCY4wvWRBkOw3GcxNL2LivmtaQdTVhjPEXC4KMkyEqnsmyjYbmENvLaiNdkTHG9CkLArfBeGTDRsAajI0x/mNBAJA9jYTydUQH1ILAGOM7ngaBiCwUkY0iskVEbutk/o0iUioi77uvz3tZzxGNmIY01XBmeqU9pMYY4zueBYGIBIElwPlAHvBJEcnrZNG/qOo09/W/XtXTJbfB+MzkPXZGYIzxHS/PCGYBW1R1m6o2AY8Dl3j4ecfObTA+JbidfVWNHKhtinRFxhjTZ7wMgpHA7rDxYndaR1eIyBoRWSYiozrbkIjcLCKFIlLoyQOu3QbjMQ2bAGswNsb4S6Qbi58BxqrqVOAF4KHOFlLVpapaoKoFmZmZ3lSSPY3kivUIIQsCY4yveBkEJUD4EX6OO62dqparaqM7+r/ADA/r6dqIaQSaaylIKrcGY2OMr3gZBKuACSIyTkRigGuA5eELiMiIsNFFQJGH9XTNbTBekPoRRdYLqTHGRzwLAlVtAb4KPI/zBf+Eqq4TkTtFZJG72C0isk5EPgBuAW70qp6jchuMZ0TvYMv+appaQhErxRhj+lKUlxtX1WeBZztMuz1s+DvAd7ysodvcBuNx9ZtpblW27K8hLzsl0lUZY4znIt1Y3L9kT2No1QZrMDbG+IoFQTi3wfjkqH0WBMYY37AgCOc2GJ87ZI89zN4Y4xsWBOHcBuNZsbso+qgaVXs2gTFm8LMgCOc2GE9o3cqB2ib2VzcefR1jjBngLAg6yp5GZs1GAoTsxjJjjC9YEHQ0YhrBllrGyUfWYGyM8QULgo7cBuMzkortDmNjjC9YEHTkNhjPiS9m/Z7KSFdjjDGesyDoyG0wzmUr28tqaWhujXRFxhjjKQuCzmRPY3jtJtAQG/fa5SFjzOBmQdCZEdOIaq2zBmNjjC9YEHTGbTCeFbOTtdZOYIwZ5CwIOuM2GM9LKuHDEjsjMMYMbhYEnXEbjKfINoo+qrJnExhjBjULgiPJnsaI+s20tLSwaZ81GBtjBi8LgiMJazD+sMTaCYwxg5cFwZG4Dcanxu60IDDGDGoWBEeScTLEpbIwYQMfFlsQGGMGLwuCIwlGwUkLmdn4Dlv2HqCxxe4wNsYMThYEXZl4EfGtVeRrEZv21kS6GmOM8YSnQSAiC0Vko4hsEZHbuljuChFRESnwsp4eG7+AUDCOTwRWsaakItLVGGOMJzwLAhEJAkuA84E84JMiktfJcsnA14C3varlmMUkIuPPZmHUatbuPhjpaowxxhNenhHMArao6jZVbQIeBy7pZLkfAj8FGjys5ZhJ7sVkcYC6nasjXYoxxnjCyyAYCewOGy92p7UTkenAKFX9Z1cbEpGbRaRQRApLS0t7v9KunLSQEEEmVrxiXVIbYwaliDUWi0gAuBv4xtGWVdWlqlqgqgWZmZneFxcuYSgHM2dyrqxig3VJbYwZhLwMghJgVNh4jjutTTIwGXhZRHYAs4Hl/a7BGAhOupjxgT3s3PBepEsxxphe52UQrAImiMg4EYkBrgGWt81U1UpVzVDVsao6FngLWKSqhR7WdExS8y8FIHrzsxGuxBhjep9nQaCqLcBXgeeBIuAJVV0nIneKyCKvPtcLkprDtpiTObF8ZaRLMcaYXhfl5cZV9Vng2Q7Tbj/CsvO9rOV4lQxfwLxd91Fftov4jNGRLscYY3qN3VncTZJ7EQClq/4W4UqMMaZ3WRB004l509kSyiZ6c5e/dDXGmAHHgqCbhqfE8WpwNsMOrIa6A5Euxxhjeo0FQTeJCCXDzyZIK2z6v0iXY4wxvcaCoAcSx83kIx1Ky/rlR1/YGGMGCAuCHpiak8a/WmcQ2PoSNNVGuhxjjOkVFgQ9MCUnledDMwm0NsKWFyNdjjHG9AoLgh7ISolje8Ip1AaSocguDxljBoduBYGIfE1EUsTxgIi8KyLneV1cfzRpVDovBedC0TNQb88oMMYMfN09I/isqlYB5wFpwKeAn3hWVT82ZeQQfl87D1oa4MNlkS7HGGOOW3eDQNz3C4CHVXVd2DRfmZqTytrQOGqHToLVD4FqpEsyxpjj0t0gWC0i/8IJgufdx0uGvCur/5o8MhWA9zMXwb4PYY91TW2MGdi6GwSfA24DZqpqHRANfMazqvqxzORYRqTG8VTLHIiKh3cfinRJxhhzXLobBKcBG1W1QkSuB/4LqPSurP5tzokZvLCtgVDepU47QWNNpEsyxphj1t0g+C1QJyKn4DxacivwJ8+q6ufOyR1GZX0zRSMuhaYaWPdUpEsyxphj1t0gaFFVBS4BfqOqS3AeNelL807KJCYY4OnyUZBxErzr20w0xgwC3Q2CahH5Ds7PRv/pPng+2ruy+rek2Chmn5jOixtKYfqnofgd2F8U6bKMMeaYdDcIFgONOPcT7MV5EP3PPKtqADgndxjbymrZnrMIAtF2VmCMGbC6FQTul/8jQKqIXAQ0qKqvv/nOnjgMgBd2tMDEC+GDx6ClMcJVGWNMz3W3i4mrgXeAq4CrgbdF5EovC+vvctISyB2Rwoqi/TDjBqe7iaJnIl2WMcb0WHcvDf0nzj0EN6jqp4FZwPeOtpKILBSRjSKyRURu62T+F0XkQxF5X0ReF5G8npUfWefkDqNwxwEOZs2BIaPt8pAxZkDqbhAEVHV/2Hj50dYVkSCwBDgfyAM+2ckX/aOqOkVVpwH/A9zdzXr6hQW5WYQUXt5cBvmfhu2vwIFtkS7LGGN6pLtB8H8i8ryI3CgiNwL/BJ49yjqzgC2quk1Vm4DHcX5+2s7tyK5NIjCgOu6ZOjKVzORY5/LQtGtBAvDenyNdljHG9Eh3G4u/BSwFprqvpar67aOsNhLYHTZe7E47jIh8RUS24pwR3NKdevqLQEBYMHEYr24spSlxBEw4D1b/0bqnNsYMKN1+MI2qPqmqX3dfvXYrraouUdUTgW/jdF3xMSJys4gUikhhaWlpb310r1iQm0V1YwvvbD8AZ/0n1FfAih9EuixjjOm2o13nrxaRqk5e1SJS1dW6QAkwKmw8x512JI8Dl3Y2Q1WXqmqBqhZkZmYe5WP71unjM4iNCrCiaB+MmAqzvwSr/wC73o50acYY0y1dBoGqJqtqSievZFVNOcq2VwETRGSciMQA1wCHPd9RRCaEjV4IbD6WnYik+Jggc8dn8OKGfagqzP8OpOTAP26F1uZIl2eMMUfl2TOLVbUF+CrwPFAEPKGq60TkThFZ5C72VRFZJyLvA18HbvCqHi+dk5vF7gP1bNpXA7FJcMHPYP96ePM3kS7NGGOOKsrLjavqs3T4dZGq3h42/DUvP7+vLMgdBk/BiqJ9nDw8GSZeABMvgpd/CpMug7SxkS7RGGOOyLMzAj/JSoljyshUXizad2ji+f8DgSD885v2OEtjTL9mQdBLzsnN4r3dFZTVuP0NpY6Es/8LtrwA65+ObHHGGNMFC4JesiB3GKrw0oawG7Bn3QwjToHnvg0Nvn2gmzGmn7Mg6CWTslMYkRp3+OWhQBAu/hXUlsKLP4xcccYY0wULgl4iIpybl8XLG0upqGs6NCM73zkzWHU//OP/g8bqyBVpjDGdsCDoRYtnjqKxJcSy1cWHzzj3Tjjtq1D4B7hvDmx7OSL1GWNMZywIetGk7FRmjEnjkbd3EQqF/VIoKhY+cRd89nmIioE/XQLP3AoNR7s52xhjvGdB0Ms+NXsM28tq+ffWso/PHH0qfPF1mPMf8O5D8Ns5sPWlvi/SGGPCWBD0svOnDGdoYgwPv7mz8wWi4+G8H7lnB3Hw8GWw4Wg9ehtjjHcsCHpZbFSQqwtGsaJoHx9V1h95wVGz4IuvQcZJsOL7EGrtsxqNMSacBYEHrjt1NAo89vaurheMjne6ri7bCB883ie1GWNMRxYEHhg1NIGzTh7GY6t209wa6nrhvEtgxDR4+cfQ0tg3BRpjTBgLAo9cP3s0pdWN/Gvdvq4XFIFz7oDK3c7PS40xpo9ZEHjkzJOGkZMWz8Nv7Tj6wiecBWPnwas/g8Yaz2szxphwFgQeCQaE604dw1vbDrB531HuJhaBBXdAXRm89du+KdAYY1wWBB66uiCHmGCAR47WaAwwaiacfCG8cS/UHfC+OGOMcVkQeCg9KZYLpgznydXF1Da2HH2Fs//L6Yvo9bu9L84YY1wWBB771GljqG5s4e/v7zn6wll5cMo18M79UNWN5Y0xphdYEHhs+ug0ckek8PBbO52H2x/N/O84N5e98lPvizPGGCwIPCci3HDaGIo+quL5o/2UFCBtDBR8Bt59GMq3el+gMcb3LAj6wJUzcjg5K5kf/mM9Dc3d6ErijG85dx3/5VNQ20nndcYY04s8DQIRWSgiG0Vki4jc1sn8r4vIehFZIyIvisgYL+uJlKhggO8vmkRJRT2/fbkbR/lJw+CaR+DANnjoYqgp9b5IY4xveRYEIhIElgDnA3nAJ0Ukr8Ni7wEFqjoVWAb8j1f1RNppJ6Zz8SnZ/PaVrewqrzv6CifMh2v/Age2w0MXQc3+o61hjDHHxMszglnAFlXdpqpNwOPAJeELqOpKVW37VnwLyPGwnoj77gUTiQoIP/zn+u6tcMKZcN0TcHAn/PEiqO5GG4MxxvSQl0EwEtgdNl7sTjuSzwHPdTZDRG4WkUIRKSwtHbiXSUakxnPLggm8sH4fKzd28wh/3Blw3V+dvogeugiq93pbpDHGd/pFY7GIXA8UAD/rbL6qLlXVAlUtyMzM7Nvietln547jhIxE7nxmPY0t3XwGwbh5cN0yqCxxzgwObIPu/BTVGGO6IcrDbZcAo8LGc9xphxGRc4D/BM5U1UHfD3NMlNNw/OkH3+GB17fz5fnju7fi2Llw/TL485Vwbz7EJEHa2MNfo0+D4ZO9K94YMyh5GQSrgAkiMg4nAK4Brg1fQETygd8DC1XVN62hZ5yUyScmZfHrF7dwWf5IRqTGd2/FMXPg5ped5xwf3AEHt0P5FtiyAloaIBAFNzzjLGeMMd0k3brb9Vg3LnIBcA8QBB5U1btE5E6gUFWXi8gKYArwkbvKLlVd1NU2CwoKtLCw0LOa+8ruA3Wcc/crnJuXxW+unX58GwuFnDaEP18O9RVw80oYMrp3CjXGDAoislpVCzqd52UQeGGwBAHAr1Zs5pcrNvHEF05j1rihx7/Bss1w/9nO3cmffR5iEo9/m8aYQaGrIOgXjcV+9YUzTyAzOZZ7VmzqnQ1mTIArH4S9a+HpL1uDsjGmWywIIiguOsgXzjiBN7aWs2pHLz2DYMK5cO4PYP3T8NrPe2ebxphBzYIgwq47dQwZSTHc++Lm3tvonFtg6mJ46Uew4dne264xZlCyIIiw+JggN807gdc2l/HuroO9s1ERuPhXkJ0Pf7sJ9hf1znaNMYOSBUE/cP3sMaQlRPPr3jwriI6Hax51GowfXQxFz0BrN56SZozxHQuCfiAxNorPzzuBlRtLWVNc0XsbTsmGax5zHnTzl+vhV6fAqz+zDuyMMYexIOgnPn3aGFLjo7n3xS29u+GcGfC1D2DxI5Ax3mk3uDsPln0Odr1lvywyxlgQ9BfJcdF87vRxrCjax9qSyt7deDAKci+CT/8dvloIMz8Pm1+ABz8BS06Ff//q6D2b1h2AdU/Dzjd6tzZjTMTZDWX9SGV9M6f/9CXmnpjB7z41w9sPa6qFD5fB+4/A7rdBgjDhPMi/DiZ8wlmmeJXTncXWl2DPe4D7t1LwOTjvRxCT4G2Nxphe09UNZV72NWR6KDU+ms/MHce9L25mw94qJg5P8e7DYhJhxg3Oq3STEwgfPA6bnoP4odDaBE01TkDkFMD825yH5Wz4B7zxa9jxGlx+P2RP865GY0yfsDOCfqaironTf7qSM0/OZMnx9kHUU60tztH/h3+F2CQ48WwYOw/ihxy+3NaV8PSXnOcpL/genPYfEOjBVcaWJti7BjInOp9jjDm6kNttfSB4TKvbGcEAMiQhhhvmjOG+l7eyeV81E7KS++7Dg1Fw0nnOqysnngVfegOe+Rq8cLvT3nDZ7yC1iwfMNdXClhedM4qN/weNlZA4zDnTmP5pCEb37r4YEwkNlVCxy3kd3Om8N9VAVBxExUIwxh2OcZavr4D6g9BQcWi4sco5WGrt8NIQXHQPFHym18u2M4J+6EBtE/N++hLjs5L5y82ziYs+tiMAz6nCe3+G574NzbWQlOWEQWoOpI5yXtFxTlBseRFa6p3LTidfAGNPh/cehp3/hvTxsOAOyL3YuRmuJ6r2QEMVDJvozT6agUEVavY5lzLj05yDmq6WbayGujLnb6el0enGvaURWhsPjTfXH3q11HcxXue8V3/kBEG46ESIS3G36W4/FHY/TzAG4oY4NccPcYbjUg+FRjDGOUgKxjjTJpzr3Ch6DKz30QHo+XV7+eKfV3PBlBH8+pp8AoEefkH2pQPb4MMnofrOAkcAABH5SURBVHIXVBZDxW7nvaXemZ+c7fxqKfdiGD3n0H+kqrDp/+CFO6BsI+TMgvN+CKNnd/459QedRuuSd53Xnned//gATjjLuUw10uNGdtP3VJ0v5qY65+i6oRIObIWyLVC2yXmVb3HmtYlNdb5YE4Y6Bx/aCrXlzpd/XblzhN0TwRiIincObKITnBs2o+Od4ag4Z3rScKf79/bXGOfzOx7chFqdUABnGz09+DlGFgQD1O9f2cqPn9vALWeP5+vnnRzpcnpG1fnJaf1BGHpC120IrS1OY/XK/4aavc5/uKhzKqxt7yHnyKtN+njInu588bc2Oj+BrSt3wubs70HmAPv3GsxaW5wv8vqDULvfuaGxZh/UlDrv9Qfdo+o658u+/Si7zrmk2FTj/P/fmdTRzv0xGSc5fxPg/t0dOPT3V38AJAAJGZCQDonph4bjhzhH2lFxYZdvYp336ATnCz4qvuszjAHCgmCAUlW+/eQanigs5p7F07g0f2SkS/JWUy0U/sF58poEDn+Bc3SVPd05Ne7YgN1YDW/e5/yiqbkWpl7jtD+kjen7/Yi01hbnKLOnjYqq7q/FasNeNc6164aKQ9exGyqdV/sXd23YF3jt4ZdaWhoOvxTSUXya84pOdI6OYxLCjrgTnEeyxiQ4v3KLSTr0PvQESD/RnrnRAxYEA1hTS4hPP/g27+6s4NGbTqVgbC88wGYwqy2H1++Gd+53zhSiE8KO+MLeA9HOoz0DUc4XZtswHDoD6fgKtTpfauq+h0LOe8dGvZYmZ3pSFgwZ5VwmSHXfh4xyPrv9yzLsC1NDTi0SPPw9EDz8enHAvWasIecS3MEdh7+qSgB1ArTjdWbEmad6+HtbAHT1pd0mKs65jh2T6H5pJ7hf4ImHLpl0/DcPxjjhnZQFScOc98RMZ77pExYEA1xFXROX3fcGlfXNPP3luYxOtxu5jqqyBD541DlyDT9Cba533kPN7hd725d6izMN+fjZSNsrEP4FHRX2JR176Ms2KtZ5l4Bzt3blbqfNpKrECRCvJA2HtLHOa8hop772cGo+NIy6+yiHvwdj3KPt8CNvdzjebcBsa8iMjvNuP4xnLAgGgW2lNVx23xsMS47lyS/PISXOfm45oLS2QPUe5wg+1NrhDMX9SaEEnHnaGvYeOhRSh32pNwN66EwjOj7Se2j6OQuCQeKNrWV8+oF3mDchgwdumNm/f0lkjOlX7JnFg8ScEzO4Y9EkVm4s5fevbot0OcaYQcLTIBCRhSKyUUS2iMhtncw/Q0TeFZEWEbnSy1oGi+tPHc2FU0fw839t7L3nHBtjfM2zIBCRILAEOB/IAz4pInkdFtsF3Ag86lUdg42I8JPLp5CTFs8tj73Hgdoe3hhjjDEdeHlGMAvYoqrbVLUJeBy4JHwBVd2hqmuAI9wtYjqTHBfNkmunU17TxDeeeJ9QaGC18xhj+hcvg2AksDtsvNid1mMicrOIFIpIYWlpaa8UN9BNHpnK9y7KZeXGUpa+Zu0FxphjNyAai1V1qaoWqGpBZmZmpMvpN66fPYYLp4zgZ89vpNDaC4wxx8jLICgBRoWN57jTTC8REX58hdNe8B+PvcdBay8wxhwDL4NgFTBBRMaJSAxwDbDcw8/zpZSw9oLP/HEVj769i837qq3dwBjTbZ7eUCYiFwD3AEHgQVW9S0TuBApVdbmIzASeAtKABmCvqk7qapt+vqGsK397t5j/fraIshrnrCAtIZoZY4Yyc2wac8dnMHlkaoQrNMZEkt1Z7BOqyo7yOlbtOEDhjgOs2nGQ7WW1AHzzvJP4ylnjkT7q+9wY07/Yoyp9QkQYl5HIuIxEri5wmmdKqxv572eL+Pm/NnGwrpn/vCDXuqYwxhzGgmCQy0yO5RdXnUJqfDQPvL6dyvpmfnL5FKKCA+IHY8aYPmBB4AOBgHDHxXkMSYjmnhWbqapv5t5P5vffZyEbY/qUHRb6hIhw6zkn8f2L8/jX+n185g+rqGnsxkNIjDGDngWBz9w4dxz3LJ7GOzsOcO39b7H7QN3RVzLGDGoWBD50af5Iln5qBpv2VXP2L17m+8vXUVrdGOmyjDERYkHgUwtys3jpG/O5ckYOD7+1kzP+ZyU/e34DlfXNkS7NGNPH7D4Cw/ayWu5+YRPPfLCHlLgovjj/RG6cM5aEGPstgTGDhd1QZrpl/Z4qfvGvjby4YT/DU+L41idO5rL8kXbfgTGDgD2q0nRLXnYKD9w4k79+8TSyUmL5xl8/4NL7/s07261nU2MGMwsC8zEzxw7lqS/P5ZeLT2F/VSNX//5NvvzIanaV2y+MjBmM7CKw6VQgIFyWn8PCSSNY+uo2fvfKVlas388VM0ayYGIWc8anWxuCMYOEtRGYbtlb2cAv/rWRf374EXVNrcQEA8wcl8b8k4Yx/+RMxg9Lsg7tjOnHrLHY9JrGllYKdxzklU2lvLxxP5v21QCQlRLLKTlDOGXUEKbmpDJ15BBSE6IjXK0xpo0FgfHMnop6XtlUyptby1lTXMGOsHaEsekJnDJqCKedkM7c8RmMGpoQwUqN8TcLAtNnKuua+bCkkg+KK1hTXMG7uyra71oek57AnBMzOH18BnNOTCctMSbC1RrjHxYEJmJUla2lNby+uYzXt5Tz1rby9s7uTshIZPLIVCaPTHHfU0mJs8tJxnjBgsD0Gy2tIdaUVPLGljLWFFeytqSSPZUN7fPHpCcwYVgyo4cmMHpoPKPTExiVlsCooQnWbbYxx8GeUGb6jahggOmj05g+Oq19WnlNI2v3VLG2xAmG7WW1vLG1jLqm1sPWzUiKYVhyHFkpsWSlxDEsxRnOTIolLTGGtIRoUuNjGJIQTbQ9eMeYbrMgMBGXnhTLmSdlcuZJme3TVJXy2iZ2Hahjt/sqqahnf1Uj+6obWLunirKaRo50QpscG8WQxGhGpMaTkxZPzpB4RqbFk5OWwMgh8aTERxMfHSQ2KmBdaBjfsyAw/ZKIkJEUS0ZS7GFnD+FaWkOU1TSxv7qBirpmDtY1UVHX3D58oLaJPRX1vLm1nL1VDUcMjbjoAPHRQeKjg6TER5ORFEt6Ukz752ckxZCWEENCTJC4mCAJMc6y8e57QkwUQQsTM4B5GgQishD4FRAE/ldVf9JhfizwJ2AGUA4sVtUdXtZkBo+oYIDhqXEMT4076rJNLSH2VjZQXFFHycF6ahpbaGgOUd/cSoP7qmtqpbK+mbKaRnbtqqOspvFjl6eOJCYqQGKMEwrxbljERgWIjQoSExUgNirQ/h4XHXReUQHiYoLERTnjMVEBooNCVCBAVFDah6ODgfZthgdQfHTQzmZMr/AsCEQkCCwBzgWKgVUislxV14ct9jngoKqOF5FrgJ8Ci72qyfhXTFSA0ekJjE7v2b0MdU0tlFU3UVHfRH1TK3XNrTQ0OaFR39zqTGtqpa65hfqmVmobW6lvbqGuqZVGN2gq6ptobA7R1BqisTlEQ0tb+ISOe79EIChCMOC+RAgGhZigEziHgudQKEUHhZiooPMedIImKihEBYRgIOC+u+NBITpwaH5UMOC+CwE59JmBts8OOMMBty4RZ9iZ5k5vH3aXFw4t506X9nUOTTtsfgAE2pcV3PfwYQ5t1+5675qXZwSzgC2qug1ARB4HLgHCg+AS4Pvu8DLgNyIiOtB+ymQGrYSYKEanRzGa3r8ZTlVpbAnR0OyESnOL0hwK0dKqNLeGaAkpLa0hmlqc8KhrcoKnLYDqm1tpDanzUqW11X0PqbNOc2v79htbQtQ1tVBRH3I+p9UJpubWEM2tSnNLiFZVWtq2Fxqc/wmKGybCodCgPTQOD5GAhM87FCjSYXthY2EB9PFACl+nbbSzZaT9f+jwWcItCyaw6JTsXvwXcXgZBCOB3WHjxcCpR1pGVVtEpBJIB8rCFxKRm4GbAUaPHu1Vvcb0KRFpv0w0JNLFdKBuoLSEnNBoDSnNrUqLG1RtgRHSQ8HRNhxSCKkScodVnYAKKe60Q8uHFFpDirqf6aznrq+KKu3rqrvNVndYFZS2ec46bbU78w5NV2fGx6a1bYP25T++bvs2OTTubu7QcPu4u7673fBtuXOdCYR93mHznWltw4d9ADAk3pv7bAZEY7GqLgWWgnMfQYTLMWbQE3Eu/0QFsfs3fMDLH1uXAKPCxnPcaZ0uIyJRQCpOo7Exxpg+4mUQrAImiMg4EYkBrgGWd1hmOXCDO3wl8JK1DxhjTN/y7NKQe83/q8DzOD8ffVBV14nInUChqi4HHgAeFpEtwAGcsDDGGNOHPG0jUNVngWc7TLs9bLgBuMrLGowxxnTNOmQxxhifsyAwxhifsyAwxhifsyAwxhifG3APphGRUmDnMa6eQYe7ln3Cr/sN/t13229/6c5+j1HVzM5mDLggOB4iUnikJ/QMZn7db/Dvvtt++8vx7rddGjLGGJ+zIDDGGJ/zWxAsjXQBEeLX/Qb/7rvtt78c1377qo3AGGPMx/ntjMAYY0wHFgTGGONzvgkCEVkoIhtFZIuI3BbperwiIg+KyH4RWRs2baiIvCAim933tEjW6AURGSUiK0VkvYisE5GvudMH9b6LSJyIvCMiH7j7/QN3+jgRedv9e/+L2xX8oCMiQRF5T0T+4Y4P+v0WkR0i8qGIvC8ihe604/o790UQiEgQWAKcD+QBnxSRvMhW5Zk/Ags7TLsNeFFVJwAvuuODTQvwDVXNA2YDX3H/Px7s+94InK2qpwDTgIUiMhv4KfBLVR0PHAQ+F8EavfQ1oChs3C/7fZaqTgu7d+C4/s59EQTALGCLqm5T1SbgceCSCNfkCVV9FefZDuEuAR5yhx8CLu3TovqAqn6kqu+6w9U4Xw4jGeT7ro4adzTafSlwNrDMnT7o9htARHKAC4H/dccFH+z3ERzX37lfgmAksDtsvNid5hdZqvqRO7wXyIpkMV4TkbFAPvA2Pth39/LI+8B+4AVgK1Chqi3uIoP17/0e4P8HQu54Ov7YbwX+JSKrReRmd9px/Z0PiIfXm96jqioig/Y3wyKSBDwJ3KqqVc5BomOw7ruqtgLTRGQI8BQwMcIleU5ELgL2q+pqEZkf6Xr62OmqWiIiw4AXRGRD+Mxj+Tv3yxlBCTAqbDzHneYX+0RkBID7vj/C9XhCRKJxQuARVf2bO9kX+w6gqhXASuA0YIiItB3oDca/97nAIhHZgXOp92zgVwz+/UZVS9z3/TjBP4vj/Dv3SxCsAia4vyiIwXk28vII19SXlgM3uMM3AH+PYC2ecK8PPwAUqerdYbMG9b6LSKZ7JoCIxAPn4rSPrASudBcbdPutqt9R1RxVHYvz3/NLqnodg3y/RSRRRJLbhoHzgLUc59+5b+4sFpELcK4pBoEHVfWuCJfkCRF5DJiP0y3tPuAO4GngCWA0ThfeV6tqxwblAU1ETgdeAz7k0DXj7+K0EwzafReRqTiNg0GcA7snVPVOETkB50h5KPAecL2qNkauUu+4l4a+qaoXDfb9dvfvKXc0CnhUVe8SkXSO4+/cN0FgjDGmc365NGSMMeYILAiMMcbnLAiMMcbnLAiMMcbnLAiMMcbnLAiM6UMiMr+tp0xj+gsLAmOM8TkLAmM6ISLXu/38vy8iv3c7dqsRkV+6/f6/KCKZ7rLTROQtEVkjIk+19QUvIuNFZIX7rIB3ReREd/NJIrJMRDaIyCMS3iGSMRFgQWBMByKSCywG5qrqNKAVuA5IBApVdRLwCs5d2wB/Ar6tqlNx7mxum/4IsMR9VsAcoK13yHzgVpxnY5yA02+OMRFjvY8a83ELgBnAKvdgPR6nE68Q8Bd3mT8DfxORVGCIqr7iTn8I+KvbH8xIVX0KQFUbANztvaOqxe74+8BY4HXvd8uYzlkQGPNxAjykqt85bKLI9zosd6z9s4T3fdOK/XdoIswuDRnzcS8CV7r9vbc9D3YMzn8vbT1bXgu8rqqVwEERmedO/xTwivuUtGIRudTdRqyIJPTpXhjTTXYkYkwHqrpeRP4L5ylQAaAZ+ApQC8xy5+3HaUcAp9vf37lf9NuAz7jTPwX8XkTudLdxVR/uhjHdZr2PGtNNIlKjqkmRrsOY3maXhowxxufsjMAYY3zOzgiMMcbnLAiMMcbnLAiMMcbnLAiMMcbnLAiMMcbn/h+kBZf8GQNXYAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P5eKDmX8ODBE"
      },
      "source": [
        "Similarly, we can plot the accuracy of our neural network on the binary classification task. The metric used in this example is the _binary accuracy_, which calculates the proportion of predictions that match labels or response variables. Other metrics may be used in different tasks -- for example, the _mean squared error_ is typically used to measure the accuracy for continuous response variables (e.g. polygenic risk scores, total serum cholesterol level, height, weight and systolic blood pressure)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J2Jdpa1i8zqM",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "06d14c69-4989-4c39-fbdc-e87a3d2ccab5"
      },
      "source": [
        "plt.figure()\n",
        "plt.plot(history.history['binary_accuracy'])\n",
        "plt.plot(history.history['val_binary_accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'validation'])\n",
        "plt.show()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxV9Z3/8dcnIRCWkABhDSCoCIhsgqhVZ9zq4L5Uq9a24lSZsTou1XZsp4s6dabza8daO1ZrW6qtVaRYlFbUqsWtihJW2UEFs0BAIBvZk8/vj3MClxDgArm5Sc77+Xjkwb1nu58Tbs7nfJfz/Zq7IyIi0ZWS7ABERCS5lAhERCJOiUBEJOKUCEREIk6JQEQk4pQIREQiTolAIsXMnjCzH8a57UYzOzfRMYkkmxKBiEjEKRGItENm1inZMUjHoUQgbU5YJfNNM1tuZrvM7Ddm1t/MXjKzMjN7zcx6xWx/iZmtNLNiM3vDzEbHrJtoZovD/Z4F0pt81kVmtjTc910zGxdnjBea2RIzKzWzPDO7t8n608PjFYfrp4XLu5rZ/5rZJjMrMbN3wmVnmll+M7+Hc8PX95rZbDN7ysxKgWlmNsXM3gs/Y7OZ/Z+ZdY7Zf4yZvWpmO8ysyMy+Y2YDzKzCzPrEbHeimW0zs7R4zl06HiUCaau+AHweOA64GHgJ+A7Ql+B7exuAmR0HPAPcEa6bB/zZzDqHF8Xngd8DvYE/hscl3HciMAP4F6AP8Etgrpl1iSO+XcBXgSzgQuBmM7ssPO5RYbw/D2OaACwN9/sJMAn4XBjTt4CGOH8nlwKzw8/8A1AP3AlkA6cC5wBfD2PIAF4DXgYGAccCr7v7FuAN4Isxx/0KMNPda+OMQzoYJQJpq37u7kXuXgC8Dbzv7kvcvQqYA0wMt7saeNHdXw0vZD8BuhJcaE8B0oCH3L3W3WcDC2M+YzrwS3d/393r3f1JoDrc74Dc/Q13/9DdG9x9OUEy+sdw9ZeA19z9mfBzt7v7UjNLAf4ZuN3dC8LPfNfdq+P8nbzn7s+Hn1np7ovcfYG717n7RoJE1hjDRcAWd/9fd69y9zJ3fz9c9yTwZQAzSwWuJUiWElFKBNJWFcW8rmzmfY/w9SBgU+MKd28A8oCccF2B7z2y4qaY10cBd4VVK8VmVgwMCfc7IDM72czmh1UqJcC/EtyZEx7jo2Z2yyaommpuXTzymsRwnJn9xcy2hNVF/xVHDAAvAMeb2XCCUleJu39wmDFJB6BEIO1dIcEFHQAzM4KLYAGwGcgJlzUaGvM6D3jA3bNifrq5+zNxfO7TwFxgiLtnAo8BjZ+TBxzTzD6fAVX7WbcL6BZzHqkE1Uqxmg4V/CiwBhjh7j0Jqs5iYzi6ucDDUtUsglLBV1BpIPKUCKS9mwVcaGbnhI2ddxFU77wLvAfUAbeZWZqZXQFMidn3V8C/hnf3Zmbdw0bgjDg+NwPY4e5VZjaFoDqo0R+Ac83si2bWycz6mNmEsLQyA3jQzAaZWaqZnRq2SawD0sPPTwO+CxysrSIDKAXKzWwUcHPMur8AA83sDjPrYmYZZnZyzPrfAdOAS1AiiDwlAmnX3H0twZ3tzwnuuC8GLnb3GnevAa4guODtIGhP+FPMvrnATcD/ATuBDeG28fg6cL+ZlQHfJ0hIjcf9FLiAICntIGgoHh+uvhv4kKCtYgfwP0CKu5eEx/w1QWlmF7BXL6Jm3E2QgMoIktqzMTGUEVT7XAxsAdYDZ8Ws/ztBI/Vid4+tLpMIMk1MIxJNZvY34Gl3/3WyY5HkUiIQiSAzOwl4laCNoyzZ8UhyqWpIJGLM7EmCZwzuUBIQUIlARCTyVCIQEYm4djdwVXZ2tg8bNizZYYiItCuLFi36zN2bPpsCtMNEMGzYMHJzc5MdhohIu2Jm++0mrKohEZGIUyIQEYk4JQIRkYhTIhARiTglAhGRiEtYIjCzGWa21cxW7Ge9mdnDZrbBgikJT0xULCIisn+JLBE8AUw9wPrzgRHhz3SCsdVFRKSVJew5And/y8yGHWCTS4HfhbNHLTCzLDMb6O6bExWTJN/28mqWF5SwqrCU6tr6ZIcj0q6cM7o/44dktfhxk/lAWQ57T72XHy7bJxGY2XSCUgNDhw5tulpayZaSKlZvKWVraRVFpdUUlVaFP9VU1NTRN6ML/XumM6BnOv16ptO/Zxe6d+nE2i1lLM8vZlleCQXFlbuPt9e8YSJyUP16pne4RBA3d38ceBxg8uTJGiWvlbg7a7aU8eqqIl5dVcSHBSV7re/dvTP9MrowIDOdIZ27sq2smsWf7qSotJqauoa9th3SuysThmZx/eeOYmxOFifk9CQjPa01T0dE9iOZiaCAYG7ZRoPDZZJExRU1LMsv4Y21W3l1VRH5OysxgwlDsvjW1JFMGdabAZnp9M3oQpdOqc0ew90pqaylqLSakspaRvTrQa/unVv5TEQkXslMBHOBW81sJnAyUKL2gda1q7qOFQUlLM8vYXlBCcvzi9m0vQKAzp1SOP3YbG4561jOGd2PfhnpcR/XzMjq1pmsbrr4i7QHCUsEZvYMcCaQbWb5wA+ANAB3fwyYRzCv6wagArghUbEIVNXWs3pzaXDRzw8u+hu2ldM4HUVOVlfG5mRy9UlDGJeTxcShWXTv0i5qDkXkCCWy19C1B1nvwC2J+vyoqK6r592PtvPaqiLWbml+sqny6jo2bC2nriG46mf36My4wVlcOG4g4wdncUJOJn0zurRm2CLShuiWrx0qqajlb2uDBtw3125jV0093TqnMjYnk06p+3bFyUhP5+xR/Rg3OJNxg7MYmJmOqcuOiISUCNqRhgbnP57/kFm5+dQ3OH0zunDJhBzOO74/px7Th/S05htvRUQORImgHfnhi6t55oM8rjt5KFdOGsz4wVmkpOjOXkSOjBJBO/Hrtz9mxt8/YdrnhvGDi49X1Y6ItBiNPtoO/GV5IT98cTXnnzCA712kJCAiLUuJoI1b8PF2vvHsMk4a1oufXj2BVFUFiUgLUyJow9YVlTH9d7kM6d2VX311shqDRSQhlAjaqC0lVUyb8QFd0lJ54oYpekpXRBJGiaANqqyp54YnFlJSWcsTN5zEkN7dkh2SiHRg6jXUBt07dyVrtpQyY9pJjBmUmexwRKSDUyJoY/60OJ9nc/O49axjOWtkv2SHI8lSVwPlW6BsC1QWN79Nahr0PwF69G3Zzy7Og7UvwdoXYfNyoJmR31M7w7DTYeQFMOLzkN7MDUtDPeTnwtp5sP6vkNYNRp4Poy6EvqPaz4QU7lCxHco2B//2Phoyh7Sf+OOgRNCGbNhaznefX8GUYb2549wRyQ6nfSv+FBY8Bv1GwXFToUcLJ9XGi2X+QvAWmGnNHarLggt/2Wao+Cz+fXsOhpyJMGgiDDoRBoyDuqrwWIV7jlm2JbiA9xwEGQMgY2D47yAozYc184KL9pblwXH7HAvHXxLs01RVCWx4HVY8BymdwqRwIRx7Dny2Dta8COtehl3bgvVHnRac39/+M/jpNTxICCPPh37HQ/nWPTE2xlyxg2aTUEoajLogSEKpB5nTYucmWPJ7qNm15/fT+2hIaaZWfNd2KFwChYuhaAWUhvGUb4H6mr237ZYdHm8i5JwY/JsxIK7/LratC5Js+VYY98Vg33jU7AJLgbSu8W1/CMy9fc3zMnnyZM/NzU12GC2usqaeyx75O9vKq5l32xkMyIx/2OcOr2JH8AeangWDJx18+03vwrNfCe7ecMBgyJTgojPyQuh7XHDhLckLjluwOPjjL1wGqZ32XDAa/8gzBgTbb1m+78Wy52BIa6H/q87dg4ty7EW65yDo2is4h6ZqymHLh2HsS2DHx/s/tqVCj/5QXx3+XprdKPw9XRBcpLMPcjPSUB8kwrXzgt/L9vV71nXpCceeGxzn2HOhazirVmlhWNqYB5+8te8FtlF6JnTrE1z4mqosDhJljwEw6Xo48XrIzNk7rvWvQu5vgn/NILUL1IWz43XJhEHjg//j9EzYvDT4/RV/uuf30PtoyBwckywHQs+Bwf/FZ+ugYEmwz7bV4OEkTBmDwu9MTFLu1nvP72nNi8F5b98QbJ/aOTj/nEkw+WtwwhX7XuTLt+75fX38Blz0EEw44Hie+2Vmi9x9crPrlAjahnueW87MhXk8+c9T+MfjWrionwzVZbD82aBq4LTbod/o+ParqQgubAWL99yd7dy4Z/3Yq+C8ByCjf/P7L3oCXrwbeh0F184M7ozXzAurOZYF2/QaHsTXeNedkgb9xwR/vA21wR950z9wSwnumjEYcnJ4R3ohZB97GL+cBKncCYVLoWhlmFRikkm3PpASdj+uq4byouBut7QwuBPvkgEjzjuyktNn64OLVe+jYdgZ0OkgPd2qy2DDa1BS0KSEMiCIf38a6mHdK8GFfsPrwf/NyPNh4ldg60rIfQJKPg0S34lfhUnTgqSxbc2e71TB4uD31FALWUftuasfdCIMHA/pPeM755pdQTKO/b42XugBeg2D6vLgu5aSBsPPCBLtyPOhcw9YNjM4j8/WBTc6E64Lkmf+B8H3Nn8h4JA5NPjOTfwyDBgbX2xNKBG0cc8vKeCOZ5dyy1nH8M1/GpXscI5M0UpY+JsgCdSUB3c9qV3gC78KvvwHsu6v8MLXg+oECOphG4vfgybCpwvgnQehU1c453sw+Z/3XNzq6+CVb8MHj8Mx58CVM/bchTYqKQjurD76W3Bn13jH3/8E6NRkGO6aiuCuv/EPvK4quFAeN7Xl6+Tl8O34BBb9FpY8taekM+wMOOlrMOqiA1cd1VVDbUVY4mpBVSVBQm4sqaV2gZFTg5JRc20p7rDxHVj4a1jzF2ioC5YPHB/cbIy6IPiOHmGbhBJBG/bRtnIu/vk7nDAok6dvOplOqe2wR29dDax6PkgAeQugUzqMuSL4Y8wYCDO/FNyNn/N9OP3Ofb/QtVXw2r3w/qPQb0xwkc+Z3PwF97MNMO+u4M5z0Ilw0U8hayj8cRp88iaceit8/v49CUKiobYKPp4flEb6jkx2NIevbEtQtTlkSlA11YKUCNqoXdV1fOHRd9laVs2Lt53OwMyWbwRKuI3vwF++AZ+thd7HBHfpE74U1I02qqmAubcGDYtjr4JLfr6nLnTrGnjua0Hj3Mn/Cufed/A6d/fgWC9/Oyhyd+8bVItc9BBMvC5x5yrSjh0oEajXUJJU1tTztScXsn5rOb+5fnL7SwLl2+DV78GyZ4I61mueCapNmuuN0bkbfOE3QT386/8Z1KFe83TQCPbKd4K60i/NguP+Kb7PNoOxVwZF7b/9MCgdXP1UcBclIodMJYIkqKqt56bf5fLOhs946OoJXDoh5+A7JUJDfVAfv7vb3uagy1z5FujeL6bXzMA91TkNDbDkd/DqD4KGstNugzPuDi728VgzD/50U1APWlcFx5wNlz22/8ZfEWkRKhG0IdV19dz81CLe2fAZP75yfHKSQMUOePfnQcNqTfne6ywl6CNdsX1P//ge/cPulBOCXhr5HwT9wi98MOinfyhGXQA3vgYv3hX0jjj55uZLESLSapQIWlFtfQO3Pr2E+Wu38d9XjOXKSS3bGHRQVSWw4FF475Gg696Yy4IHgWL7SnfvF/Slr60M+6gv2dPXft3LQd3/ZY/C+GsPvxdDv9Fww7yWPTcROWxKBK2krr6B22cu4dVVRdx/6RiunTK09T68Zhe8/0v4+8+gqjjoVnfWd4I6+/1J6xrUucfWu1eXBX2hW+oBKhFpE5QIWkF9g3PXH5cx78MtfPfC0Xz11GGt88ENDbD0KXj9/qAtYMR5QQKI95H2prpktGx8ItImKBG0gqc/+JQXlhbyrakjufGMo1vnQ4tWwl/uhLz3YcgpQS8d9aoRkWYoEbSCWQvzOCGnJ18/sxWGI6guhzd/BO/9IniK8dJHYPyX1CArIvulRJBga7aU8mFBCfdefHwrfNiLMO9bwZg4E78SPGEb+2CXiEgzlAgS7LlF+aSlGpckopuoezB0Q+Poj0UfBkP6XvkKDD2l5T9PRDokJYIEqq1vYM6SQs4Z1Z/e3VtozuH6Wtj4djii5kt7RsQcegpc8JNgpMWDjdEuIhJDiSCB3lq3jc/Kq1vueYF1r8C8u4Nx0zt1DZ7KPevbwdAO3bNb5jNEJHKUCBJo9qJ8+nTvzD+OPMJhi0sK4OV/h9V/huyR8MXfB+PsxDusg4jIASgRJMjOXTW8trqIr546jLTDHVq6vg7efwze+O9gXKBzfhAMs3ywCT9ERA6BEkGCzF1WSG29H361UH4u/PmOoAF4xHlwwY+D2Y5ERFqYEkGCzF6Uz5hBPRk9MM4p72Jt/wieuDCYXvCLv4fRFx/x7EQiIvujp4wSYO2WMj4sKDm80oA7zPtmMKbPja/D8ZcoCYhIQikRJMBzi4NnBw5riOnVc+Gj1+Hs/4CeA1s+OBGRJpQIWlhdfQN/WlzA2aP6HfqzA9XlwfSL/cfCSTclJkARkSbURtDC3lrf+OzAkEPf+c3/gdICuOqJYE4AEZFWoBJBC2t8duDMQ312oGgVLPgFnPhVjRIqIq1KiaAF7dxVw2urtnLZxJxDe3bAPZi6sUsGnHNvwuITEWlOQhOBmU01s7VmtsHM7mlm/VFm9rqZLTezN8ysledubFl/Xl5ITX0DXzixyWkUfwqv/ydsXt78jstmwqfvwrn3Qfc+iQ9URCRGwhKBmaUCjwDnA8cD15pZ07GYfwL8zt3HAfcD/52oeBKttr6BX7/9CeMHZ3L8oJhnBxoa4Lmb4O2fwC/PgF+fG1z4a6uC9ZU74a/fhcEnBUNHi4i0skS2SE4BNrj7xwBmNhO4FFgVs83xwDfC1/OB5xMYT0LNXpTPpzsquO+Sk/ZekfsbyFsAU/8HvCF4P+dfgt5BE6+DsiKo3AEXztHkMSKSFIlMBDlAXsz7fODkJtssA64AfgZcDmSYWR933x67kZlNB6YDDB3aipO+x6m6rp6fv76eiUOz9m4kLv4UXrsXjjkHTv6X4MGwU26GT94KEsJ7vwCvh5NvhoHjkha/iERbsvso3g38n5lNA94CCoD6phu5++PA4wCTJ0/21gwwHjM/yKOwpIr/d+V4rPEpYPdgrCB3uPihPU8Hm8HR/xj8lG6Gj/4GYy5PXvAiEnmJTAQFQGxn+sHhst3cvZCgRICZ9QC+4O7FCYypxVXV1vPI/A2cPLw3px0b09C7bGbwhPD5P4as/ZRieg4MqodERJIokZXSC4ERZjbczDoD1wBzYzcws2wza4zh28CMBMaTEE8t2MTWsmruOm/kntJA+VZ4+R4YcgqcdGNyAxQROYiEJQJ3rwNuBV4BVgOz3H2lmd1vZpeEm50JrDWzdUB/4IFExZMIu6rrePSNjzhjRDZThsdMEj/vm1BbCZf8XA3AItLmJbSNwN3nAfOaLPt+zOvZwOxExpBIT7y7ke27avjG54/bs3D1n2HV83DO96HvcfvfWUSkjdDt6mEqrarl8bc+5uxR/Zg4tFewsHJn8ITwgLHwuduSG6CISJyS3Wuo3ZrxzieUVNbuXRqY/1+w6zO47o+Qmpa84EREDoFKBIehuKKG37z9CVPHDOCEnMxgYcUOWPx7GH8tDByf3ABFRA6BEsFh+NXbH1NeU8edsaWBRb+Fuko49evJC0xE5DAoERyimroGnlrwKeefMICRAzKChXU18P7jcPRZ0H9McgMUETlESgSH6O312yiprN17PuKVc6B8C5x6S/ICExE5TEoEh+iFpYVkdUvj9GPDMYXcYcEjkH1cMKaQiEg7o0RwCCpq6nh1VRHnnzCQzp3CX92md2HzMjjl63p4TETaJV25DsFrq7dSWVvPJeMH7Vn43iPQtTeMvyZ5gYmIHAElgkMwd2kh/Xt22TOcxPaPYO08OOlrkNY1ucGJiBwmJYI4lVTU8ua6rVw0bhCpKeHgcu//ElI6aWA5EWnXlAji9PLKzdTW+55qocpiWPIUjL0SMgYkNzgRkSOgRBCnucsKOapPN8YNDp8kXvwk1O4KGolFRNoxJYI4bC2r4r2PtnPp+EHBnAP1tcEDZMPO0BSTItLuKRHE4cXlm2lwuGRCWC206gUozdcDZCLSISgRxGHuskJGD+zJsf3CISU+eBx6HwMj/im5gYmItAAlgoPI21HBkk+L9zQS7/gE8t6HE7+qB8hEpEPQlewg5i4rBODi8QODBSvCCdVO+EKSIhIRaVlKBAfx52WFTDqqF4N7dQsWfPgcDD0VsoYkNzARkRaiRHAA64rKWLOlbE+1UNFK2LZapQER6VCUCA5g7tJCUgwuGBtWC304GywVxlye3MBERFqQEsF+uDt/Xl7Iacdm0zejSzDc9Irn4OgzoXt2ssMTEWkxSgT7sXF7BZu2V/BPY8LhI/JzoXhTMKSEiEgHokSwHws37gDg5MaRRlfMhtQuMOqiJEYlItLylAj2Y9HGnWR2TeOYvj2goT6YjvK48yC9Z7JDExFpUXElAjP7k5ldaGaRSRy5m3Yw6ahepKQYbHwbyovgBFULiUjHE++F/RfAl4D1ZvYjMxuZwJiSbseuGj7atovJw3oFCz6cDZ0z4DgNKSEiHU9cicDdX3P364ATgY3Aa2b2rpndYGZpiQwwGRZt2gnA5KN6Q101rJ4Loy7ULGQi0iHFXdVjZn2AacCNwBLgZwSJ4dWERJZEuZt2kJZqwdwDG16HqhL1FhKRDqtTPBuZ2RxgJPB74GJ33xyuetbMchMVXLLkbtzJ2JxM0tNSg95CXXsHzw+IiHRAcSUC4GF3n9/cCnef3ILxJF1VbT0f5pcw7bRhULML1r4E46+B1A5XAyYiAsRfNXS8mWU1vjGzXmbWIedo/LCghJr6BiYd1StIArUV6i0kIh1avIngJncvbnzj7juBmxITUnLlbmxsKO4V9BbqmROMNioi0kHFmwhSzcwa35hZKtA5MSEl16JNOzg6uzt9Uitgw2vBAHOagEZEOrB4r3AvEzQMn2Nm5wDPhMs6lIYGJ3fTzqBaaM2L0FCrIadFpMOLt7H434F/AW4O378K/DohESXRx5+VU1xRy0nDegdDSmQdBYMmJjssEZGEiisRuHsD8Gj402E1tg+cNMDhpTfg1FthT42YiEiHFO9zBCOA/waOB9Ibl7v70QmKKykWbtxJ7+6dGbZ1PjTUaQIaEYmEeNsIfktQGqgDzgJ+BzyVqKCSZVE40Jyteh56DYeB45MdkohIwsWbCLq6++uAufsmd78XuPBgO5nZVDNba2YbzOyeZtYPNbP5ZrbEzJab2QWHFn7L2VZWzcbtFZw+EPj4zaA0oGohEYmAeBuLq8MhqNeb2a1AAdDjQDuEXUwfAT4P5AMLzWyuu6+K2ey7wCx3f9TMjgfmAcMO8RxaxKJNwUQ0Z/oH4PWqFhKRyIi3RHA70A24DZgEfBm4/iD7TAE2uPvH7l4DzAQubbKNA40zvWQChXHG0+JyN+6kc6cUhmx+GXofAwPGJisUEZFWddASQXhnf7W73w2UAzfEeewcIC/mfT5wcpNt7gX+amb/BnQHzt1PDNOB6QBDhw6N8+MPzcJNOzljkJOy8W04/RuqFhKRyDhoicDd64HTE/T51wJPuPtg4ALg983Ngubuj7v7ZHef3Ldv3xYPorKmnpUFJVzVfSl4g6qFRCRS4m0jWGJmc4E/ArsaF7r7nw6wTwEwJOb94HBZrK8BU8NjvWdm6UA2sDXOuFrEsvxi6hqcKbvehD4joP+Y1vx4EZGkireNIB3YDpwNXBz+XHSQfRYCI8xsuJl1Bq4B5jbZ5lPgHAAzGx1+zrY4Y2oxuRt3kE0JvbZ9oN5CIhI58T5ZHG+7QOw+dWEPo1eAVGCGu680s/uBXHefC9wF/MrM7iRoOJ7m7n6on3Wkcjft5CtZy7AqVQuJSPTE+2Txbwku1Htx938+0H7uPo+gS2jssu/HvF4FnBZXpAnS0OAs2rST73V/H7JHQr/RyQxHRKTVxdtG8JeY1+nA5SSxq2dLWr+1nPSqzziapTD531UtJCKRE2/V0HOx783sGeCdhETUygqLK5ma+gGGw5jLkh2OiEirO9wZV0YA/VoykGQprarlotQF1PRWtZCIRFO8bQRl7N1GsIVgjoJ2r6a4kJNsLZWjvtkxp1wTETmIeKuGMhIdSLKkb11Oijmdj2v2oWYRkQ4vrqohM7vczDJj3meZWYeoUE8rzw/+zR6e5EhERJIj3jaCH7h7SeMbdy8GfpCYkFpX14pCqugM3Vt+6AoRkfYg3kTQ3Hbxdj1t07pXbmZbSl91GxWRyIo3EeSa2YNmdkz48yCwKJGBtZbMmiJ2dOoQHaBERA5LvIng34Aa4FmCeQWqgFsSFVRr6lNXRGnnAckOQ0QkaeLtNbQL2GeqyXavtorevpPyrgOTHYmISNLE22voVTPLinnfy8xeSVxYraQ0GBW7qntOkgMREUmeeKuGssOeQgC4+046wJPF9TuDCdRqeygRiEh0xZsIGsxs9xyRZjaMZkYjbW+qPvskeJE55MAbioh0YPF2Af0P4B0zexMw4AzCOYTbs9rtm6h3o1OWSgQiEl3xNha/bGaTCS7+S4DngcpEBtYaGorzKKIXPbt3S3YoIiJJE++gczcCtxPMO7wUOAV4j2DqynYrpTSPAs+mZ9e0ZIciIpI08bYR3A6cBGxy97OAiUDxgXdp+9LKCynwbDKVCEQkwuJNBFXuXgVgZl3cfQ0wMnFhtYKGBtIrNlOoRCAiERdvY3F++BzB88CrZrYT2JS4sFpB+RZSvU4lAhGJvHgbiy8PX95rZvOBTODlhEXVGoqDZwi2WF/S0w53ojYRkfbvkEcQdfc3ExFIqysJEkFplwGYRh4VkQiL7q1wmAh2aZwhEYm46CaC4jzKUjLo0q1nsiMREUmq6CaCkjy2Wj96pquhWESiLbqJoDiPQtRjSEQkmonAHUryyKvvrUQgIpEXzURQVQw15XxS14eeXTvE1MsiIoctmokgfIYgr0FVQyIi0UwEYddRPVUsIhLVRFCsRCAi0iiaiaAkj/rUdHaQoe6jIhJ5kU0Eld0GAqa5CEQk8qKZCIrzKE8PhpZQ1ZCIRCV6dNMAAA+5SURBVF00E0FJHsVpAwDI7KZEICLRFr1EUFsJu7bxWWpfzKBHZz1HICLRFr1EUFIAQFFKMM5QSoqGoBaRaItgIvgUUNdREZFG0UsE4TMEn9b3USIQESHBicDMpprZWjPbYGb3NLP+p2a2NPxZZ2bFiYwHCJ4qthQ21mRqnCEREQ5jqsp4mVkq8AjweSAfWGhmc919VeM27n5nzPb/BkxMVDy7leRDxiCKq53jslQiEBFJZIlgCrDB3T929xpgJnDpAba/FngmgfEEivMgawgllbWqGhIRIbGJIAfIi3mfHy7bh5kdBQwH/raf9dPNLNfMcrdt23ZkUZV8CplBItBTxSIibaex+BpgtrvXN7fS3R9398nuPrlv376H/ykN9VBaSF3GYGrqGjTOkIgIiU0EBcCQmPeDw2XNuYbWqBYq2wwNdVR00/ASIiKNEpkIFgIjzGy4mXUmuNjPbbqRmY0CegHvJTCWQEk+AGVdwuEllAhERBKXCNy9DrgVeAVYDcxy95Vmdr+ZXRKz6TXATHf3RMWyW/gMwc40JQIRkUYJ7Ujv7vOAeU2Wfb/J+3sTGcNewqeKt6X2BbapsVhEhLbTWNw6ivOgWx+K64IEoBKBiEjUEkFJHmQOpqSiFlAiEBGByCWC/PAZgjoAMtI1xISISHQSgXv4VPFQSipr6d45lbTU6Jy+iMj+ROdKWLkTandB5hBKqzS8hIhIo+gkguKgxxCZgzW8hIhIjOgkgvBhssYB55QIREQCEUoE4fh3mUMp1cijIiK7RScR5EyCf/gmdOutRCAiEiM6/SeHTAl+QHMRiIjEiE6JIFRb38CumnoNQS0iEopcIiitbHyqODqFIRGRA4leIqgKnirO7KYSgYgIRDARlFRqnCERkViRTQRqIxARCUQ2EahEICISiFwiKFUiEBHZS+QSwe6qISUCEREggomgtLKWzp1SSE9LTXYoIiJtQvQSgYagFhHZS+SeqtLwEiJtS21tLfn5+VRVVSU7lA4hPT2dwYMHk5YW/3VOiUBEkio/P5+MjAyGDRuGmSU7nHbN3dm+fTv5+fkMHz487v0iVzVUUllLT81VLNJmVFVV0adPHyWBFmBm9OnT55BLV5FLBKWVdSoRiLQxSgIt53B+l5FLBKoaEhHZW6QSQUODq9eQiOyluLiYX/ziF4e83wUXXEBxcXECImp9kUoEZdV1uOthMhHZY3+JoK6u7oD7zZs3j6ysrESF1aoi1WpaqqeKRdq0+/68klWFpS16zOMH9eQHF4/Z7/p77rmHjz76iAkTJpCWlkZ6ejq9evVizZo1rFu3jssuu4y8vDyqqqq4/fbbmT59OgDDhg0jNzeX8vJyzj//fE4//XTeffddcnJyeOGFF+jatWuLnkciRapEoAHnRKSpH/3oRxxzzDEsXbqUH//4xyxevJif/exnrFu3DoAZM2awaNEicnNzefjhh9m+ffs+x1i/fj233HILK1euJCsri+eee661T+OIRLNEoCGoRdqkA925t5YpU6bs1Qf/4YcfZs6cOQDk5eWxfv16+vTps9c+w4cPZ8KECQBMmjSJjRs3tlq8LSFSiUAlAhE5mO7du+9+/cYbb/Daa6/x3nvv0a1bN84888xm++h36dJl9+vU1FQqKytbJdaWEqmqodKqMBFomkoRCWVkZFBWVtbsupKSEnr16kW3bt1Ys2YNCxYsaOXoWodKBCISaX369OG0007jhBNOoGvXrvTv33/3uqlTp/LYY48xevRoRo4cySmnnJLESBMncokgNcXo3llDUIvIHk8//XSzy7t06cJLL73U7LrGdoDs7GxWrFixe/ndd9/d4vElWqSqhhrHGdLj7CIie0QqEWicIRGRfUUqEWicIRGRfUUuEeipYhGRvSU0EZjZVDNba2YbzOye/WzzRTNbZWYrzaz5FpsWUqpEICKyj4T1GjKzVOAR4PNAPrDQzOa6+6qYbUYA3wZOc/edZtYvUfGA5isWEWlOIksEU4AN7v6xu9cAM4FLm2xzE/CIu+8EcPetiQrG3dVGICJHrEePHgAUFhZy5ZVXNrvNmWeeSW5u7gGP89BDD1FRUbH7fTKHtU5kIsgB8mLe54fLYh0HHGdmfzezBWY2NVHBVNbWU1vvGmdIRFrEoEGDmD179mHv3zQRJHNY62Q/UNYJGAGcCQwG3jKzse6+V1o0s+nAdIChQ4ce1geVVgZji6tEINKGvXQPbPmwZY85YCyc/6P9rr7nnnsYMmQIt9xyCwD33nsvnTp1Yv78+ezcuZPa2lp++MMfcumle1dobNy4kYsuuogVK1ZQWVnJDTfcwLJlyxg1atReYw3dfPPNLFy4kMrKSq688kruu+8+Hn74YQoLCznrrLPIzs5m/vz5u4e1zs7O5sEHH2TGjBkA3Hjjjdxxxx1s3LgxYcNdJ7JEUAAMiXk/OFwWKx+Y6+617v4JsI4gMezF3R9398nuPrlv376HFYyGlxCR5lx99dXMmjVr9/tZs2Zx/fXXM2fOHBYvXsz8+fO56667cPf9HuPRRx+lW7durF69mvvuu49FixbtXvfAAw+Qm5vL8uXLefPNN1m+fDm33XYbgwYNYv78+cyfP3+vYy1atIjf/va3vP/++yxYsIBf/epXLFmyBEjccNeJLBEsBEaY2XCCBHAN8KUm2zwPXAv81syyCaqKPk5EMEoEIu3AAe7cE2XixIls3bqVwsJCtm3bRq9evRgwYAB33nknb731FikpKRQUFFBUVMSAAQOaPcZbb73FbbfdBsC4ceMYN27c7nWzZs3i8ccfp66ujs2bN7Nq1aq91jf1zjvvcPnll+8eBfWKK67g7bff5pJLLknYcNcJSwTuXmdmtwKvAKnADHdfaWb3A7nuPjdcd56ZrQLqgW+6+76zPrSAkt2zkyW7NkxE2pqrrrqK2bNns2XLFq6++mr+8Ic/sG3bNhYtWkRaWhrDhg1rdvjpg/nkk0/4yU9+wsKFC+nVqxfTpk07rOM0StRw1wl9jsDd57n7ce5+jLs/EC77fpgE8MA33P14dx/r7jMTFUupSgQish9XX301M2fOZPbs2Vx11VWUlJTQr18/0tLSmD9/Pps2bTrg/v/wD/+we+C6FStWsHz5cgBKS0vp3r07mZmZFBUV7TWA3f6Gvz7jjDN4/vnnqaioYNeuXcyZM4czzjijBc92X5G5PVbVkIjsz5gxYygrKyMnJ4eBAwdy3XXXcfHFFzN27FgmT57MqFGjDrj/zTffzA033MDo0aMZPXo0kyZNAmD8+PFMnDiRUaNGMWTIEE477bTd+0yfPp2pU6fubitodOKJJzJt2jSmTJkCBI3FEydOTOisZ3agBpC2aPLkyX6w/rnN+evKLcxelM+jX55EaopGHxVpK1avXs3o0aOTHUaH0tzv1MwWufvk5raPTIngvDEDOG9M8w09IiJRFqlB50REZF9KBCKSdO2tirotO5zfpRKBiCRVeno627dvVzJoAe7O9u3bSU9PP6T9ItNGICJt0+DBg8nPz2fbtm3JDqVDSE9PZ/DgwYe0jxKBiCRVWloaw4cPT3YYkaaqIRGRiFMiEBGJOCUCEZGIa3dPFpvZNuDAA3/sXzbwWQuG015E9bwhuueu846WeM77KHdvdhz/dpcIjoSZ5e7vEeuOLKrnDdE9d513tBzpeatqSEQk4pQIREQiLmqJ4PFkB5AkUT1viO6567yj5YjOO1JtBCIisq+olQhERKQJJQIRkYiLTCIws6lmttbMNpjZPcmOJ1HMbIaZbTWzFTHLepvZq2a2Pvy3VzJjTAQzG2Jm881slZmtNLPbw+Ud+tzNLN3MPjCzZeF53xcuH25m74ff92fNrHOyY00EM0s1syVm9pfwfYc/bzPbaGYfmtlSM8sNlx3R9zwSicDMUoFHgPOB44Frzez45EaVME8AU5ssuwd43d1HAK+H7zuaOuAudz8eOAW4Jfw/7ujnXg2c7e7jgQnAVDM7Bfgf4KfufiywE/haEmNMpNuB1THvo3LeZ7n7hJhnB47oex6JRABMATa4+8fuXgPMBC5NckwJ4e5vATuaLL4UeDJ8/SRwWasG1QrcfbO7Lw5flxFcHHLo4OfugfLwbVr448DZwOxweYc7bwAzGwxcCPw6fG9E4Lz344i+51FJBDlAXsz7/HBZVPR3983h6y1A/2QGk2hmNgyYCLxPBM49rB5ZCmwFXgU+AordvS7cpKN+3x8CvgU0hO/7EI3zduCvZrbIzKaHy47oe675CCLG3d3MOmyfYTPrATwH3OHupcFNYqCjnru71wMTzCwLmAOMSnJICWdmFwFb3X2RmZ2Z7Hha2enuXmBm/YBXzWxN7MrD+Z5HpURQAAyJeT84XBYVRWY2ECD8d2uS40kIM0sjSAJ/cPc/hYsjce4A7l4MzAdOBbLMrPFGryN+308DLjGzjQRVvWcDP6PjnzfuXhD+u5Ug8U/hCL/nUUkEC4ERYY+CzsA1wNwkx9Sa5gLXh6+vB15IYiwJEdYP/wZY7e4Pxqzq0OduZn3DkgBm1hX4PEH7yHzgynCzDnfe7v5tdx/s7sMI/p7/5u7X0cHP28y6m1lG42vgPGAFR/g9j8yTxWZ2AUGdYioww90fSHJICWFmzwBnEgxLWwT8AHgemAUMJRjC+4vu3rRBuV0zs9OBt4EP2VNn/B2CdoIOe+5mNo6gcTCV4MZulrvfb2ZHE9wp9waWAF929+rkRZo4YdXQ3e5+UUc/7/D85oRvOwFPu/sDZtaHI/ieRyYRiIhI86JSNSQiIvuhRCAiEnFKBCIiEadEICIScUoEIiIRp0Qg0orM7MzGkTJF2golAhGRiFMiEGmGmX05HOd/qZn9MhzYrdzMfhqO+/+6mfUNt51gZgvMbLmZzWkcC97MjjWz18K5Ahab2THh4XuY2WwzW2Nmf7DYAZFEkkCJQKQJMxsNXA2c5u4TgHrgOqA7kOvuY4A3CZ7aBvgd8O/uPo7gyebG5X8AHgnnCvgc0Dg65ETgDoK5MY4mGDdHJGk0+qjIvs4BJgELw5v1rgSDeDUAz4bbPAX8ycwygSx3fzNc/iTwx3A8mBx3nwPg7lUA4fE+cPf88P1SYBjwTuJPS6R5SgQi+zLgSXf/9l4Lzb7XZLvDHZ8lduybevR3KEmmqiGRfb0OXBmO9944H+xRBH8vjSNbfgl4x91LgJ1mdka4/CvAm+Esaflmdll4jC5m1q1Vz0IkTroTEWnC3VeZ2XcJZoFKAWqBW4BdwJRw3VaCdgQIhv19LLzQfwzcEC7/CvBLM7s/PMZVrXgaInHT6KMicTKzcnfvkew4RFqaqoZERCJOJQIRkYhTiUBEJOKUCEREIk6JQEQk4pQIREQiTolARCTi/j/99EtftuUg3wAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Xy7VhhZg-hN"
      },
      "source": [
        "## 2.3 -  Evaluate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "btf7FyMVsnFA"
      },
      "source": [
        "![alt text](https://github.com/abidlabs/deep-learning-genomics-primer/blob/master/Screenshot%20from%202018-08-01%2020-32-12.png?raw=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eQ_xYCvfvFlE"
      },
      "source": [
        "The best way to evaluate whether the network has learned to classify sequences is to evaluate its performance on a fresh test set consisting of data that it has not observed at all during training. Here, we evaluate the model on the test set and plot the results as a confusion matrix. Nearly every test sequence should be correctly classified."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J1bvxV9J-EMT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 349
        },
        "outputId": "0cd86b05-6f6d-4c29-ec2d-8f91c3500235"
      },
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "import itertools\n",
        "\n",
        "predicted_labels = model.predict(np.stack(test_features))\n",
        "cm = confusion_matrix(np.argmax(test_labels, axis=1), \n",
        "                      np.argmax(predicted_labels, axis=1))\n",
        "print('Confusion matrix:\\n',cm)\n",
        "\n",
        "cm = cm.astype('float') / cm.sum(axis = 1)[:, np.newaxis]\n",
        "\n",
        "plt.imshow(cm, cmap=plt.cm.Blues)\n",
        "plt.title('Normalized confusion matrix')\n",
        "plt.colorbar()\n",
        "plt.xlabel('True label')\n",
        "plt.ylabel('Predicted label')\n",
        "plt.xticks([0, 1]); plt.yticks([0, 1])\n",
        "plt.grid('off')\n",
        "for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "    plt.text(j, i, format(cm[i, j], '.2f'),\n",
        "             horizontalalignment='center',\n",
        "             color='white' if cm[i, j] > 0.5 else 'black')"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Confusion matrix:\n",
            " [[249  10]\n",
            " [  7 234]]\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAATUAAAEWCAYAAAAHJwCcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxU1ZnG8d/TdHBhXwS1WQRBAXcEUdFoHDci6mQ0inGJiYY4RmPiFo2aqBmTyaqZiRmjxmiMUdG44DJCooKjgywxcUFFGWXpFkEQcImK4Dt/1O22aHqpgq6u27efL5/76bucOvXequbtc5dzriICM7OsqCh3AGZmLclJzcwyxUnNzDLFSc3MMsVJzcwyxUnNzDLFSS0lJE2TdHoyf6KkqS1c/3aSQlJlS9bbzHtK0u8krZQ0axPq2V/SvJaMrVwkDZD0nqQO5Y4lq9pNUpO0QNIySZ3y1p0uaVoZw2pQRNwWEYeWO44WsB9wCNAvIvba2Eoi4n8iYseWC6s0kt+xg5sqExGLIqJzRKxrrbjam3aT1BIdgHM2tZKkBdLePruNMRBYEBHvlzuQNGjNVnJ71t7+Y/4UOF9S94Y2StpX0mxJq5Of++ZtmybpKklPAf8ABieHc2dKelXSu5J+IGl7Sf8r6R1JkyR1TF7fQ9KDkt5KDscelNSvkThOlfRkMn9hcrhSO30s6eZkWzdJv5W0RFKNpH+rPayR1EHSzyQtl/QacERTH4yk/pLuSeJbIelXyfoKSZdKWpi0dH8vqVuyrfaQ9suSFiXvdUmy7TTgRmCfJO4r8vcr731D0pBk/vOSXkw+yxpJ5yfrD5RUnfea4cn3sUrSXElH5W27WdK1kh5K6pkpaftG9rk2/q9IWpx8L2dIGi3puaT+X+WV317SY8nns1zSbbW/S5JuBQYADyT7e2Fe/adJWgQ8lreuUlJPSdWSjkzq6CxpvqRTmvqurBkR0S4mYAFwMHAP8G/JutOBacl8T2AlcDJQCZyQLPdKtk8DFgE7Jds/AwRwP9A1Wf8R8CgwGOgGvAh8OXl9L+AYYEugC3AXcF9efNOA05P5U4EnG9iH/sAbwLhk+V7gN0AnoA8wC/h6su0M4OXkNT2Bx5N4KxuotwPwLHB1UtfmwH7Jtq8C85N96px8frcm27ZL6rwB2ALYLfkMhje0Hw3tV/L6Icn8EmD/ZL4HMDKZPxCoTuY/k8TzXaAjcBDwLrBjsv1mYAWwV/I93Qbc0cjvRG381yX7fCjwIXBf8nlWAcuAA5LyQ8gdTm8GbAU8AVxT/3esgfp/n3yuW+Stq0zKHAq8mbzfDcDd5f6/0tansgfQajv6aVLbGVid/FLmJ7WTgVn1XjMDODWZnwZcWW97AGPzlv8KfCdv+ef5v/T1Xrs7sDJveRpNJLXkP0Rd/UDfJIFskVfmBODxZP4x4Iy8bYfSeFLbB3irkW2PAmfmLe8IfJwkjNr/oP3yts8CJjS0H43sV35SWwR8Hehar8yBfJrU9k+SQEXe9tuBy5P5m4Eb87Z9Hni5ke+gNv6qvHUrgOPzlv8EfKuR1/8z8Lf6v2MN1D+4gXWVeev+E3geqCH5I+pp46f2dvhJRLwAPAhcVG/TtsDCeusWkvtrXWtxA1UuzZv/oIHlzgCStpT0m+Qw7h1yf+W7q/CrYL8F5kXEj5PlgeRaLUuSw6RV5FptffL2Jz/e+vuWrz+wMCLWNrCt/ueykFxC65u37s28+X+Q7PNGOIZcElooabqkfRqJZ3FEfFIvpvzvqdh4Cv0O+0q6Izk0fgf4A9C7mbqh4d+bfNeT+2N7c0SsKKA+a0K7S2qJ7wNfY/3/CG+QSxT5BpD761lrU4Y0OY9cK2dMRHQFPpusV3MvlHQRsANwWt7qxeRaar0jonsydY2InZLtS8glq1oDmniLxcAANXwiu/7nMgBYy/r/8Qv1PrnDbwAkbZ2/MSJmR8TR5BLzfcCkRuLpr/Uv1NT/nkrlh+R+B3ZJvsOTWP/7a+z3o9Hfm+SP2vXkDlHPrD2/aBuvXSa1iJgP3Al8M2/1w8AOkr6UnMQ9HhhBrlXXErqQ+6u/SlJPcom1WZLGJXF+ISI+yNuHJcBU4OeSuiYn9LeXdEBSZBLwTUn9JPVgw5ZpvlnkkuC/S+okaXNJY5NttwPfljRIUmdy/7HvbKRV15xngZ0k7S5pc+DyvP3sqNz9ed0i4mPgHeCTBuqYSa71daGkz0g6EDgSuGMj4ilWF+A9YLWkKuCCetuXkjv3WIzvkkt6XyV3Iev3RbTerQHtMqklriR38haApNk/nlyLagVwITA+Ipa30PtdQ+682HLgaeCRAl93PLnzfy/p0yug1yXbTiF3svxFchc17ga2SbbdAEwhl0ieIXeCv0GRu2fqSHInwhcB1cn7AtwE3ErucPl1cifSzy4w9vrv8wq5z/0vwKvAk/WKnAwsSA7tzgBObKCONUms48h9lr8GTomIlzcmpiJdAYwkd072ITb8TH8EXJqcDji/ucok7QmcSy7+dcCPySW4pv4AWTOUnKg0M8uE9txSM7MMclIzs0xxUjOzTHFSM7NMSVUHW1VuEdqsa7nDaHFb99yCN9/+oPmCbdDuw/o3X6gN+sf777Flp429hzi9Fi1cwPLly5u9N7IpHboOjFhb2O9zfPDWlIg4fFPer1jpSmqbdWWzYRPKHUaLO//0UVxy45xyh1EST864ptwhlMTTT01n77EHNF+wjdlvn9GbXEes/YDNdjyuoLIf/v3aQnpctKhUJTUzawsEKR55y0nNzIojoCK9nR6c1MyseNqk03Il5aRmZkXy4aeZZY1bamaWGcItNTPLErmlZmYZ46ufZpYdvlBgZlkifPhpZhnjlpqZZYcPP80sSwR08IUCM8sSn1Mzs+zw4aeZZY1bamaWKW6pmVlmyN2kzCxr3E3KzLLDFwrMLGt8+GlmmeHx1MwsW3z4aWZZ4wsFZpYpPqdmZpkhH36aWda4pWZmWSInNTPLitxo3k5qZpYVEqpwUjOzDHFLzcwyxUnNzDLFSc3MskPJlFJOamZWFKFUt9TSe1uwmaVWRUVFQVNzJB0uaZ6k+ZIuamD7AEmPS/qbpOckfb7Z2DZyn8ysHZNU0NRMHR2Aa4FxwAjgBEkj6hW7FJgUEXsAE4BfNxebk5qZFUdFTE3bC5gfEa9FxBrgDuDoemUC6JrMdwPeaK5Sn1Mzs6IVcU6tt6Q5ecvXR8T1yXwVsDhvWzUwpt7rLwemSjob6AQc3NwbOqmZWVGKvFCwPCJGbcLbnQDcHBE/l7QPcKuknSPik8Ze4KRmZkVroW5SNUD/vOV+ybp8pwGHA0TEDEmbA72BZY1V6nNqZlYctcyFAmA2MFTSIEkdyV0ImFyvzCLgnwAkDQc2B95qqlK31MysaC1xn1pErJV0FjAF6ADcFBFzJV0JzImIycB5wA2Svk3uosGpERFN1eukZmZFa6mbbyPiYeDheuu+lzf/IjC2mDqd1MysKGnvUeCkZmbFS29O84WCTXHIPsN49k/f5YX7LuX8Uze8fWbA1j14+L++wcnHjWfKb86iqk+3um39t+7BA9f+K3+7+2KeuetiBmzTszVDb7emTnmE3Xcexi7Dh/Kzn/77Bts/+ugjTjlxAiefdCIH7Lc3CxcsWG/74kWL6NOzC9f84metFHEKqeW6SZWCW2obqaJCXHPRFznizF9Ts3QVT956Hg9Of56XX19aV+ZH3z6a2x6axYitg6nPrubKs47ktO/9AYAbrziRH9/0Zx6bOY9OW3Tkk6bPfVoLWLduHeeecxYPPDyVqn792H/fvThi/FEMH/5pz5xbfvdbunfvzq1/uI3FNUu47JKL+P1td9Rtv+jC8zj0sHHlCD9V0nz46ZbaRhq900D+b/FbLKhZwcdr13HX1GcYf+Au65UZNmhrps9+FYDps19l/AG7JOv7UlnZgcdmzgPg/Q/W8MGHH7fuDrRDc2bPYvD2Qxg0eDAdO3bk2OOO58EH7l+vzIMPTObEk78MwBf+5VimPf4otRfbHrj/PgZutx3DR9TvntgOtUw3qZJwUttI2/bpRvXSVXXLNUtXUbVVt/XKPP/qGxx90G4AHP25XenaeXN6dtuSoQP7sOrdD7jjp19lxm0X8MNzjqIixWO+Z8Ubb9TQr3+/uuWqqn4sqanZsEy/3P2glZWVdO3ajRUrVvDee+/xi5//hO9e+v1WjTmtWug+tZIoaVJrbliRrLv46vvYf+T2nHTsEey/5xBqlq5i3bqgskMFY/cYzEXX3M9+p/ycQVW9OfnI+l3eLE2u+sHlnPXNb9G5c+dyh1J2hSa0ciW1kp1TyxtW5BByHVVnS5qc3HfS5r2xbDX9+navW67q252at1avV2bJ8neYcMFNXHX6KH5423P880G7sfq9D6hZuorn5tWwoGYFAJOnPcdeu2zHLesfCVkL23bbKqoXV9ct19RUs01V1YZlqnN9rNeuXcs776ymV69ezJk9i/vu/ROXfvc7rF61ioqKCjbffHPOOPOsVt2HtGiv59QKGVakzZrz4iKG9N+Kgdv25DOVHfjioSN5aPoL65Xp1b1T3Zd/wVcO4ZbJT9e9tluXLejdvRMAB47egZdfe7N1d6Ad2nPUaP5v/qsseP111qxZw92T7uSI8UetV+aI8Udy2623AHDvPXdzwIEHIYk/P/YEL73yOi+98jrfOPsczr/w4nab0CDX97OQqRxKefWzkGFFkDQRmAjQrUcvvnf6pnTob11/f/ZZZtz6bSTxwsvzOemgfuz7nSN4860VvLagmqGDB7DfmD2o7CAWVL/JY0/M4qpk/16Z9xLPTLoAAUvfepu3l8yv29aWPP3U9HKHUJSJEydy6MEH8Mm6Txg3bhzvrFzO10//MjvusCP7jh3L8GE7cN89d3Pil75Et27duPSyyzbYx+pFC9liiy3a3L63pDS31NRMN6qNr1g6Fjg8Ik5Plk8GxkREo3/eKjr1jc2GTShJPOV01emjuOTGOc0XbINWzLim3CGUxNNPTWfvsQeUO4wWt98+o3nmr3M2KSNttvXQ6HfifxRU9rVffP6vmzj0UNFK2VIrZFgRM2tjBKS4oVbSc2qFDCtiZm1OO7362diwIqV6PzNrPWm+r7Kk3aQaGlbEzNo4pfvw030/zawooh231Mwsm9xSM7NMSfN9ak5qZlYcn1MzsywRKtsAkIVwUjOzormlZmaZ4nNqZpYdPqdmZlmS6/uZ3qzmpGZmRUtxTnNSM7PiuUeBmWWHfPhpZhmS9vHUGk1qks5t6oUR8YuWD8fM0q98Y6UVoqmWWpdWi8LM2pQU57TGk1pEXNGagZhZG6F0XyhotgOXpB0kPSrphWR5V0mXlj40M0uj2vvU0jqcdyG9Um8ALgY+BoiI58g9b8DM2qk0J7VCrn5uGRGz6gW4tkTxmFkb0CbPqeVZLml7IKDueZ5LShqVmaVaW736WesbwPXAMEk1wOvAiSWNyszSq613aI+I14CDJXUCKiLi3dKHZWZplRskMr1ZrZCrn70k/QfwP8A0Sb+U1Kv0oZlZWlVIBU3NkXS4pHmS5ku6qJEyx0l6UdJcSX9srs5CDj/vAJ4AjkmWTwTuBA4u4LVmlkEtcfgpqQNwLXAIUA3MljQ5Il7MKzOU3N0XYyNipaQ+zdVbyC0d20TEDyLi9WT6N6Dvxu2GmbV1Uovd0rEXMD8iXouINeQaUEfXK/M14NqIWAkQEcuaq7SQpDZV0gRJFcl0HDClgNeZWUZVqLAJ6C1pTt40Ma+aKmBx3nJ1si7fDsAOkp6S9LSkw5uLrakO7e+Su41DwLeAP9TuD/AecH5zlZtZNhVxoWB5RIzahLeqBIYCBwL9gCck7RIRq5p6QYMiwh3azWwDIncFtAXUAP3zlvsl6/JVAzMj4mPgdUmvkEtysxurtKDx1CT1SCravHZdRDxRWNxmljUtdEfHbGCopEHkktkE4Ev1ytwHnAD8TlJvcoejrzVVabNJTdLpwDnksujfgb2BGcBBRe6AmWVBC/XrjIi1ks4id46+A3BTRMyVdCUwJyImJ9sOlfQisA64ICJWNFVvIS21c4DRwNMR8TlJw4AfbsrOmFnb1lI9CiLiYeDheuu+lzcfwLnJVJBCktqHEfFhcol2s4h4WdKOhb6BmWWLoKAba8ulkKRWLak7uWPbP0taCSwsbVhmlmZp7iZVSN/PLySzl0t6HOgGPFLSqMwstdRWO7RL6tnA6ueTn52Bt0sSkZmlXls9/Pwrn958W6t2OYDBJYzLzFIsvSmt6ZtvB7VmIGbWdrT1QSLNzOrkrn6WO4rGOamZWXGU7kEindTMrGht8vCzkaufdSLCVz/N2qG2fPiZf/VzALAyme8OLAJ8IcGsnUpzS63RQSIjYlBEDAb+AhwZEb0johcwHpjaWgGaWfqowKkcChn5du+k0ykAEfHfwL6lC8nM0kyCDhUqaCqHQi4UvCHpUj4d+fZE4I3ShWRmadcmDz/znABsBdwL3JPMn1DKoMws3Wr7fzY3lUMhHdrfBs6R1Cki3m+FmMwsxURhz/Qsl0IeZrxvMurkS8nybpJ+XfLIzCydCmylpbalBlwNHAZMBoiIZyV9thTB7DGsP0/N/GUpqi6rGU9OY+XMk8sdRkn0GH1WuUMoiasmjuGIc88udxgt7qN5i1qknjSfUyuoR0FELK63E+tKE46ZpZ2ADm08qS2WtC8Qkj5D7pkFL5U2LDNLs7bao6DWGcAvyT05uYbcjbdnljIoM0u3tp7UdoyIE/NXSBoLPFWakMwszXIXAdKb1Qq5T+0/C1xnZu1EhQqbyqGpUTr2IdcdaitJ+c/c60ruwaNm1k6luKHW5OFnR3IPWKkEuuStfwc4tpRBmVl6CahMcVZr6hkF04Hpkm6OCD/n08zqpDinFXRO7cbkYcYASOohaUoJYzKzFJNy3aQKmcqhkKufvSNiVe1CRKyU1KeEMZlZyrX1ltonkgbULkgaSG5EXDNrp9rk1c88lwBPSppO7hzh/sDEkkZlZqklKNsAkIUoZOihRySNBPZOVn0rIpaXNiwzS60ytsIK0ejhp6Rhyc+R5B688kYyDUjWmVk7pQL/lUNTLbXzgK8BP29gWwAHlSQiM0u1NvuIvIj4WvLzc60Xjpm1BW0yqUn6l6ZeGBH3tHw4ZtYWpLlDe1OHn0cmP/uQ6wP6WLL8OeB/yT2Exczamdwj8sodReOaOvz8CoCkqcCIiFiSLG8D3Nwq0ZlZKrXpB68A/WsTWmIpuauhZtYO1V4oaImbbyUdLmmepPmSLmqi3DGSQtKo5uos5ObbR5O+nrcny8cDfyngdWaWUS3RUJPUAbgWOASoBmZLmhwRL9Yr14XcYwRmFlJvsy21iDgLuA7YLZmuj4jsPWbHzAokKgqcmrEXMD8iXouINcAdwNENlPsB8GPgw0KiK+hpUsAzwLsR8RdJW0rqEhHvFvhaM8sQUVRLrbekOXnL10fE9cl8FbA4b1s1MGa998rd6N8/Ih6SdEEhb9hsUpP0NXJ9PXsC2yeBXAf8UyFvYGYZI6gs/Ea15RHR7HmwBt9GqgB+AZxazOsKuVDwDWAsuRFviYhXyd3mYWbtUG1LrQWe0F4D9M9b7pesq9UF2BmYJmkBuf7nk5u7WFDI4edHEbGm9mY7SZV46CGzdq2FbumYDQyVNIhcMpsAfKl2Y0SsBnrXLkuaBpwfEXNoQiEttemSvgtsIekQ4C7ggaLDN7PMaImWWkSsBc4CppB7QPqkiJgr6UpJR21sbIW01L4DnA48D3wdeBi4cWPf0MzaNlFYa6gQEfEwuZySv+57jZQ9sJA6m0xqyX0kcyNiGHBDYWGaWaapDfcoiIh1wLz84bzNrH3L9Sho2w9e6QHMlTQLeL92ZURs9DGvmbVt6W2nFZbULit5FGbWpqT46LPJ8dQ2B84AhpC7SPDb5GqFmbVrarPjqd0CfAz8DzAOGEGuU6mZtWMtefWzFJpKaiMiYhcASb8FZrVOSGaWdmm++tlUUvu4diYi1qa5uWlmrUhtdzjv3SS9k8yLXI+Cd5L5iIiuJY/OzFKnzR5+RkSH1gzEzNqOttpSMzNrUHpTmpOamRVJQAe31MwsS1Kc05zUzKxYQik+AHVSM7OiuaVmZpmRu6UjvVnNSc3MilPY8wfKxknNzIqW5m5Sab4xuE2YOuURdt1pR3YaNoSf/uTfN9j+0UcfceUVV7DTsCHsv+8YFi5YAMDsWbMYs+fujNlzd/YauRv333dvK0fePh2y73CevfcyXrj/+5z/lUM22D5gmx48fN3ZnHzceKbccA5VfboD8NlRQ3n6jovqppVPX82RB+7a2uGnQm6QyMKmcnBLbROsW7eOb33zGzz033+mql8/9tt7NOPHH8XwESPqytx802/p0qULc1+ez6Q77+CS736HP/zxTnbaeWeemjmHyspKlixZwpg9d+OI8UdSWemvpFQqKsQ1Fx3HEf/6K2qWruLJ2y7gwenP8/Jrb9aV+dG3v8BtD81ixDYw9W8rufLsozjtst/zxJxX2XtC7o9Wj65b8sLk7/OXp18q166UXZqvfrqltglmz5rF9tsPYdDgwXTs2JEvHj+BBx+4f70yDz5wP4cedhgA/3LMsUx77FEigi233LIugX304Yep7naSFaN33o7/W7ycBTUr+HjtOu6a8gzj67W2hg3ehumz5gEwffYrjD9wlw3q+cLBezD1qRf54MOPN9jWXrTQcz9LwkltE7zxRg39+n36LNaqqn7U1NRsUKZPn9yznysrK+narRsrVqwAYNbMmYzcbSdG7bEL/3HtdW6lldi2fbpRvXRl3XLN0pVUbdVtvTLPv1LD0QftDsDRB+1G185b0LNbp/XKfPGwkUx65K+lDzjFVOC/cihZUpN0k6Rlkl4o1Xu0dXuNGcMzz87lyRmz+emPf8SHH35Y7pDavYuvvpf99xzCSccewf57DqFm6UrWrfukbvvWvbuy09Bt+fOMF8sYZXml/ZxaKVtqNwOHl7D+stt22yqqqxfXLdfUVFNVVbVBmWXLlgGwdu1a3lm9ml69eq1XZtjw4XTu3Jm5Lzj/l9Iby1bTr2+PuuWqvj2oeWv1emWWvLWaCeffyB/ufojv/yr3zO7V731Qt/2YQ0Yy+bHnWLv2E9qtAp8kVa4rpCVLahHxBPB2qepPg1GjRzN//qsseP111qxZw1133sER49d/yNYR449i6pQpANzzp7s54HMHIYkFr7/O2rW5Rz4sXLiQefNeZuB227X2LrQrc+YuZMiArRi4bS8+U9mBLx42koemPbdemV7dO9Wd37zgq4dxy/1Pr7f9uMP3ZNIjc1ot5rRSgVM5lP0kjqSJwESAvn37MuPJaeUNqEgTJ07kkIM+y7pPPmHcuHGsXvkWE796CjvsuCNjx45l+LCh3DXpdrYfWEWXrl257LLLmPHkNKZOncrtf/wjlZWVqKKCM888k1dffoFXy71DRbpq4phyh1CUvz/7LDNuOw9JvPDyfE46eCD7jj6KN99awWsLqhk6eAD7jdmDyg5iQfWbPPbErLp97NqlEyMG9+WwkT05bGTb2u9a5583aZPrqH3uZ1opIkpXubQd8GBE7FxI+T33HBVPzczeX8EZT05jn/0OLHcYJdFj9FnlDqEkrpo4hkuun1nuMFrcR/Mm8ck/lm1SRhq+yx7xu3sfL6jsPkN7/DUiRm3K+xWr7C01M2uD0ttQc1Izs+Kl+fCzlLd03A7MAHaUVC3ptFK9l5m1rnZ5oSAiTihV3WZWZultqPnw08yKk2uFpTerOamZWXE8npqZZU2Kc5qTmpkVS6keVcZJzcyKluKc5qRmZsUp5+0ahfB4amZWvBa6UU3S4ZLmSZov6aIGtp8r6UVJz0l6VNLA5up0UjOzorXEIJGSOgDXAuOAEcAJkkbUK/Y3YFRE7ArcDfykudic1MysaC00nPdewPyIeC0i1gB3AEfnF4iIxyPiH8ni00C/5ip1UjOz4hSY0JKk1lvSnLxpYl5NVcDivOXqZF1jTgP+u7nwfKHAzIpWRI+C5S0x9JCkk4BRwAHNlXVSM7OiiBa7paMG6J+33C9Zt/77SQcDlwAHRMRHzVXqw08zK1oLXfycDQyVNEhSR2ACMHm995H2AH4DHBURywqJzUnNzIrXAlktItYCZwFTgJeASRExV9KVkmof9vFToDNwl6S/S5rcSHV1fPhpZkVrqUEiI+Jh4OF6676XN39wsXU6qZlZ0dLco8BJzcyKl+Ks5qRmZkXxIJFmli0eJNLMsibFOc1JzcyK5UEizSxjUpzTnNTMrDhpHyTSSc3MipfirOakZmZF8y0dZpYpPqdmZtkhqHBSM7NsSW9Wc1Izs6K04CCRJeGkZmZFS3FOc1Izs+K5pWZmmeJuUmaWKelNaU5qZlakAh9UXDZOamZWNPcoMLNsSW9Oc1Izs+KlOKc5qZlZsdRij8grBSc1MytK2nsU+AntZpYpbqmZWdHS3FJzUjOzovmWDjPLDt98a2ZZkvYLBU5qZlY0H36aWaa4pWZmmZLinOakZmYbIcVZzUnNzIoiSHU3KUVEuWOoI+ktYGG54yiB3sDycgdhRcnqdzYwIrbalAokPULu8ynE8og4fFPer1ipSmpZJWlORIwqdxxWOH9nbZf7fppZpjipmVmmOKm1juvLHYAVzd9ZG+VzamaWKW6pmVmmOKmZWaY4qZWQpMMlzZM0X9JF5Y7HmifpJknLJL1Q7lhs4ziplYikDsC1wDhgBHCCpBHljcoKcDPQqjeLWstyUiudvYD5EfFaRKwB7gCOLnNM1oyIeAJ4u9xx2MZzUiudKmBx3nJ1ss7MSshJzcwyxUmtdGqA/nnL/ZJ1ZlZCTmqlMxsYKmmQpI7ABGBymWMyyzwntRKJiLXAWcAU4CVgUkTMLW9U1hxJtwMzgB0lVUs6rdwxWXHcTcrMMsUtNTPLFCc1M8sUJzUzyxQnNTPLFCc1M8sUJ7UMk9RL0t+T6U1JNXnLHVvoPaZJavIBJZIWSCr06UNIOlXSrzY9OmuP/NzPDIuIFcDuAJIuB96LiJ/VbpdUmdxPZ5YZbqm1M5JulnSdpJnATyRdLuFzTyMAAAGVSURBVOn8vO0vSNoumT9J0qykZfebZDilpur+L0lzJM2VdEW9zRdKej6pb0hSfitJf5I0O5nGtuzeWnvkpNY+9QP2jYhzGysgaThwPDA2InYH1gEnNlPvJcmzMncFDpC0a9621RGxC/Ar4Jpk3S+BqyNiNHAMcONG7Y1ZHh9+tk93RcS6Zsr8E7AnMFsSwBbAsmZec5ykieR+r7YhNzjmc8m22/N+Xp3MHwyMSOoH6Cqpc6E7YdYQJ7X26f28+bWs32LfPPkp4JaIuLiQCiUNAs4HRkfESkk359UFEA3MVwB7R8SH9eoq5C3NGuTDT1sAjASQNBIYlKx/FDhWUp9kW09JA5uopyu5ZLlaUl9yw5jnOz7v54xkfipwdm0BSbtv/G6Y5bilZn8CTpE0F5gJvAIQES9KuhSYKqkC+Bj4BrCwoUoi4llJfwNeJjfi71P1ivSQ9BzwEXBCsu6bwLXJ+krgCeCMltw5a388SoeZZYoPP80sU5zUzCxTnNTMLFOc1MwsU5zUzCxTnNTMLFOc1MwsU/4flJ0qQ8Vik6wAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "cm"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hD-jrVWK-0G6",
        "outputId": "873649cc-808c-4a83-cc87-a3ec28c43077"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.96138996, 0.03861004],\n",
              "       [0.02904564, 0.97095436]])"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UBdJQC1Ug__3"
      },
      "source": [
        "## 2.4 - Interpret"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6qmKi1ResqHo"
      },
      "source": [
        "![alt text](https://github.com/abidlabs/deep-learning-genomics-primer/blob/master/Screenshot%20from%202018-08-01%2020-32-31.png?raw=true)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UpAwoK9SwAbb"
      },
      "source": [
        "Your results so far should allow you to conclude that the neural network is quite effective in learning to distinguish sequences that bind the protein from sequences that do not. But can we understand _why_ the neural network classifies a training point in the way that it does? To do so, we can compute a simple _saliency map_, which is the gradient of the model's prediction with respect to each individual nucleotide. \n",
        "\n",
        "In other words, the saliency maps shows how the output response value changes with respect to a small changes in input nucleotide sequence. All the positive values in the gradients tell us that a small change to that nucleotide will change the output value. Hence, visualizing these gradients for a given input sequence, should provide some clues about what nucleotides form the binding motive that we are trying to identify."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WNT_Au-dAP8a"
      },
      "source": [
        "import tensorflow.keras.backend as K\n",
        "\n",
        "def compute_salient_bases(model, x):\n",
        "  input_tensors = [model.input]\n",
        "  gradients = model.optimizer.get_gradients(model.output[0][1], model.input)\n",
        "  compute_gradients = K.function(inputs = input_tensors, outputs = gradients)\n",
        "  \n",
        "  x_value = np.expand_dims(x, axis=0)\n",
        "  gradients = compute_gradients([x_value])[0][0]\n",
        "  sal = np.clip(np.sum(np.multiply(gradients,x), axis=1),a_min=0, a_max=None)\n",
        "  return sal"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uObq5iC4BG1g",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        },
        "outputId": "ceaa8fa8-067f-4dce-f5de-92f1f9311c1a"
      },
      "source": [
        "sequence_index = 1999  # You can change this to compute the gradient for a different example. But if so, change the coloring below as well.\n",
        "sal = compute_salient_bases(model, input_features[sequence_index])\n",
        "\n",
        "plt.figure(figsize=[16,5])\n",
        "barlist = plt.bar(np.arange(len(sal)), sal)\n",
        "[barlist[i].set_color('C1') for i in range(5,17)]  # Change the coloring here if you change the sequence index.\n",
        "plt.xlabel('Bases')\n",
        "plt.ylabel('Magnitude of saliency values')\n",
        "plt.xticks(np.arange(len(sal)), list(sequences[sequence_index]));\n",
        "plt.title('Saliency map for bases in one of the positive sequences'\n",
        "          ' (orange indicates the actual bases in motif)');"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1152x360 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA8kAAAFNCAYAAAAzci1+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdebhkVX3v//fHbkFFBGWIMjZKE9M4RG1Fo7nOAZwwCgpxDkq8ETVXQ8RcY5BIIjFKrj9FRXE2NgSnvoriNYjGgVENBpDYNiiNGqBtEFHAxu/vj71aqsuqc+p0nzp1+vT79Tz1nL3X3mvt7x5q1/nW3ntVqgpJkiRJkgR3mHQAkiRJkiTNFybJkiRJkiQ1JsmSJEmSJDUmyZIkSZIkNSbJkiRJkiQ1JsmSJEmSJDUmydI8kuTKJE9ow3+d5L2Tjmk+SPK7Sb6d5MYkr5iF9o5L8pHZiG22JXlOki9MOo7NkeSPk1yV5OdJHjTC/I9JsmYuYtsUSfZq67Joinl+nuTecxnX1irJ10Y5ruazJH+Y5PLNqF9J9m3D70ryN7MX3XglWdLiXzzpWGZLkhcm+eqQafN6fefruav/szDJI5N8r8X79CQfT3LwJGPUwmaSLM2yJI9K8vUkNyT5afuH7qEzbaeq/r6qXjyOGLdAfwV8qaq2r6q3TTqYcaqqj1bVH006js30T8DRVXXXqvpW/8Tef/C3BFX1w7YutwEkOSfJi/vmuWtVrZ5MhFuPJE8Fbhx0XG1Jqurfq+p3Z6mtl1bV321OG+P8oqr3y9/5Yr5/MTeX5sO5a9AXCQM+C48H3t7i/RRwIvDGuY5VWw+TZGkWJbkb8Bng/wPuAewOvAG4ZZJxLQB7A5dsSsX5+u39ArfJ+0uaxkuBD29KRc8F0hZto8+VqjofuFuS5ZMLSQuZSbI0u/YDqKqPVdVtVfXLqvpCVV0MkOQ+Sc5OsjbJdUk+mmTHQQ313xKc5OHtCvX1Sf4jyWN6pp2T5O/aVesbk3whyc490x/VU/eqdmvYQ5P8d+8tpEmekeQ/hsTzgSQnJ/lcu93pa0numeSfk6xL8t3eWyCTHJvk+y2eS5P8cc+0F7b6b29X3L+b5PFDlns28Fjg7W25+yXZIcmHklyb5AdJXpfkDn1tn5RkLXDckH11pySntfi+meSBI8a+b5Ivt7ivS3Jaz7T7Jvl/7Q6Cy5M8q2fak1pbNya5OslfDlnfjW7ba9+uv7TdZnZ9knckyZC627b98aP2+uck27Zpj0myJsmrk1yT5MdJXtRX95+S/LAdF+9Kcuchy7lD2+Y/aG19qO2TbZP8HFgE/EeS7w+o+5U2+B9tfz67Z9psxDblsZVktyQr2z5aleQlPdMeluTCJD9ry3lrK//NVY4kJwB/yO3H49t79tO+SQ5I8pNs/L764yQbzgF36Dm+1iY5Pck9hqzLzkk+0/b7T5P8e89xvlu62w2vTXJFeh5DSHLndO/Xde2YOyY9V83SdyW/zfvGnvGnpHu84fp0540H9Ey7MslfJrm4bd/TktypZ/ohre7P2joe1Mp3SHJq27dXJ3njhm2UKd5TfdtjG+BxwJd7ykY55l+T5CfA+5PcvW3Ta9v2+UySPXram+5c+vx0x/3aJH+TjR+Rmcm+3ehK5gjb9Zi27X6U5E/72urff8P2wYuSXNbWa3WSP2vl2wGfA3Zrx/TP2/E1dH2S3CnJR1r59UkuSPI7A9bzw8BewP9t7f5Vz+TnpHtPX5fkf/fUmcl2nG5/3iPJ+9t2W5fkU1Osb/927N9HQz8XRvSnLY4fp+f8n+688422HX+c7ty1TZuWdJ9l17T9+Z0k92vThp4XM8W5Y8A27L11/wPpPmM+29bzvCT3GVJvw3nxRen+r1iX7rPqoe04vj7t/NjmH/i50SZv+Fy4vu2PR6TnszDdZ8m9uf042rbNfw7w5BnuB2k0VeXLl69ZegF3A9YCHwQOBu7eN31f4InAtsAudB8M/9wz/UrgCW34OOAjbXj31u6T6L7cemIb36VNPwf4Pl2Sfuc2/qY2bW/gRuAI4I7ATsDvt2mXAgf3LP+TwKuHrNsHgOuAhwB3As4GrgCeT5cUvZHulugN8x8G7NbifTZwE3CvNu2FwHrgf7WYng3cANxjyLLPAV7cM/4h4NPA9sAS4L+AI/vafjmwGLjzgPaOA34FHNqW/5dtXe44QuwfA/53m3Yn4FGtfDvgKuBFbbkPattrWZv+Y+AP2/DdgQcPWdcXAl/tGS+6uxN2pPtn81rgoCF1jwfOBXalO76+Dvxdm/aYtl2Ob+v8JOAXtGMUOAlYSXcHxPbA/wX+Ychy/hRYRfdPy12BTwAf7ot53yneJxtNn+XYpjy26N5zJ7d99/ttez6uTfsG8Lw2fFfg4W14SYt58aDjsX+d6N6LT+yZ9q/AsW34lW0f7UF3Hng38LEh6/IPwLvaetyRLjkP3bF3EfB6YJu2H1YDB7Z6bwL+vW2vPYH/BNZMsf0/ALyxDT8IuAY4gO59/QK689K2Peeo8+neH/cALgNe2qY9rG3rJ7YYdwfu23NueTfd+2TX1safTfWeGrA99gdu2oRj/sS2re9Md/57JnAXumPpX4FP9Z1rhp1LlwE/Bx7Vtvs/0Z1HnrAJ+/Yxfftkqu16EPDfwP3a9vsXNj7eevffVPvgycB96I6hR9O9xx48KJ7p1gf4M7r34V3ojpOHAHcbsq5XbthGfe+n97Rt/EC6u61+bxO243T787PAaXTn3DsCj55ifX+zHYfso+k+0746JMYN6/uxtv/uT3fe2XDcPAR4ON3nxpK27/+iTTuQ7r2+Y9tvv9ezzKHnRYacO6Y7H7dtsJbuOFoMfBRYMc16vYvufftHwM3Ap+jej7vTnUs2bPOhnxv0nWMHbVP6jqNW9irgE8M+a3z52pzXxAPw5WuhvdqH2AeANXT/oK0EfmfIvE8HvtUz/psPATZOkl9DTxLSys4CXtCGzwFe1zPtz4HPt+HXAp8csvzXAB9tw/eg+6fpXkPm/QDwnp7xlwOX9YzfH7h+iu3ybeCQNvxC4Ee9H9p0/yA+b0jdc2hJCd0/ZLfSks9W9mfAOT1t/3CafXQccG7P+B3oSWKnif1DwCnAHn3zPBv4976ydwN/24Z/2OIc+I9kT53+fwyKnqQBOJ2WcA2o+33gST3jBwJXtuHHAL9k439CrqH75yx0//Ddp2faI4Arhizn34A/7xn/XbpkYXFPzDNNkmcrtqHHFl3CeBuwfc+0fwA+0Ia/Qvd4xM59bS5hZknyG4H3teHtW/x7t/HLgMf31LtX77bra/N4ui+D9u0rP4C+Y5zuff7+Nryani9SgKMYPUl+Jy3J7Jl+Obf/o3sl8Nyeaf8IvKvneD9pwHr8Dl0SdOeesiNoX6ox5D01oJ1HAj+Z4TF/K3CnKdr8fWBdz/g5DD+Xvp6eZI0uMbuV28/ZM9m3j+G3k+Rh2/V9tES9je/H8CR54D4Ysu6fAl45KJ7p1ocu4fk68IARlnMlg5PkPXrKzgcOn+l2nGp/tnq/pu/L6inW9zfbcdg8ffP3f6ZNlyTft2//njpk/r+gfWbT3TnxX3Tnwjv0zDPleZEh544hy+s/lt7bM+1JwHenWa/de8rWAs/uGf84tyf8Qz832PQk+SXA2aMc7758zfTl7dbSLKuqy6rqhVW1B903/7sB/wyQ5HeSrEh3u+HPgI8AO0/R3AZ7A4e125euT3I93dWMe/XM85Oe4V/QfVMLXWLwW7e9Nh8BntpuP3sWXZL34yni+O+e4V8OGN+wzA23JX67J977sfG6Xl1V1TP+A7ptNZ2d6b4Z/0Ff3d17xq8aoZ3fzFNVv6b7UmO3EWL/K7p/UM5Pckluv/Vxb+CAvn30HOCebfoz6f7h+EG6W0sfMUKMGwzbt/1247e3S+82XVtV6we0tQvdP/wX9cT++VY+6nIW0yVDm2q2YoPhx9ZuwE+r6sa+aRuOnSPpEpDvprt99CmbuC7/Ajyj3RL4DOCbVbVhe+0NfLJnXS6jS9wHbbs30115+UK622OP7Wljt75j7a972tiNjd8DvftqOnsDr+5re082Po5meq7Zm+49++OeNt9Nd7UJhr+n+q2j+9Kh13TH/LVVdfOGkSR3SfLudsvnz+i+GNkxG/dcPmz9NtquVfULuqSgdz1H3beDjLRcpt6fQ8/3SQ5Ocm67/fZ6uvPRVJ8/U63Ph+m+qF3RbiH+xyR3nKKtQYat78jbcZr9uSfd+33dDOMaaITPtOn078MNnzf7tVujf9LW4e83tFtVZwNvB94BXJPklHR9n0x3Xhx27hjFqJ83G4z6f8E4Pje2B67fjPrSUCbJ0hhV1Xfpvpm9Xyv6e7pvS+9fVXcDnkv3z+F0rqK7krxjz2u7qnrTiHUHPlNUVVfT3WL6DLorbZvUIU6/JHvT3Up3NLBTVe1Id8tn77runmz0bO1edFcAp3Md3bfPe/fVvbpnvJjenj3x3oHu1r4fTRd7Vf2kql5SVbvRXRk+uT3PdRXw5b59dNeq+p+t3gVVdQhdYvApuivCs+1H/PZ2GXWb/hLYvyf2Hapq2D9Hg5azno3/OZotM40Nhh9bPwLukWT7vmlXA1TV96rqCLp9dCJwRvsCqd+Ux1dVXUr3D+DBwJ/QJc0bXEX3iEPvcXKn9l7sb+fGqnp1Vd0beBrwqnTPV19Fd8Wot43tq+pJreqP6Tm+2zr2+gXdP9gb3LNn+CrghL6271JVH5tqnXvqDjrXXEV3JXnnnjbvVlX7t/Uc9p7qt4ruMc3eL8SmO+b799Wr6a5gHdDOwf+jlY9yHv4x3Xmiq9A9/7lT33qOtG9naLr92WvgPmhf2Hyc7hbx32nntTO5fb0HHdND16eqflVVb6iqZcAfAE+he/RmkFHOxyMtd8C8U+3Pq+je74P6/RgU000MeV+M+Jk2nf59uOE4fSfwXWBpW4e/7m23qt5WVQ+hu91/P+AYpjkvTnHumKSpPjdmeoxs8HvAwH5UpM1lkizNonQdN706reOQJHvS3VZ4bptle7pn2m5o/+gdM2LTG674HphkUbpOUx6Tng5KpvBR4AlJnpWu46Gdkvx+z/QP0V3JuT/dM0KzYTu6D71roeswhtu/KNhgV+AVSe6Y5DC6D7szp2u4up/hOR04Icn27Z+XV9Fto5l4SLqOyhbT3d52C91+mjL2JIf1bPd1bd5f0z03vF+S57V1umO6Dkx+L8k26X7zcYeq+hXws1Zntn0MeF2SXdJ1NvR6Rtgu7Ur6e4CTkuwKkGT3JAdOsZz/lWSfJHel+/LntL4rwVP5b7rn0qa1CbHBkGOrqq6iu0X0H9p76AF0V48/0tp9bpJd2jI3XJ0YtJ9Gif9f6J6t/B90z0lu8C66Y3fvtsxdkhwyqIF0HWjt2xL+G+iupv2a7tbUG9N1SHXndk64X27/qbnTgdem69RoD7pHI3p9G/iTVu8guudTN3gP8NJ0HZAlyXZJntz3xcIwpwIvSvL4dJ307J7kvu3ulC8Ab0lytzbtPkke3dZz2HtqI1V1K/DFvnhnesxvT5dcXJ+uM6i/HWG9NjiD7jz8B+k6VjqOjZOkkfftDJ0OvDDJsiR3mSbmgfuA7hnqbenOa+vT/b5s78/r/DewU27vSGnK9Uny2CT3T3fF9md0X1wOO6eN/H6fbrkDDN2f7bj7HN2XLndv54MNSfSg9f028KR0nX3dk+5zYYNRPtOm8zfprnzvT9d3xYYO6ran24Y/b/vqf26o0D5DDkh3lf4mumd+fz3deXGKc8ckTfW5cS1dfDP9veZH0+1jadaZJEuz60a65wXPS3ITXdL1n3TfdkP3vOOD6T60PsuISWn75/4Qum+Yr6X7hvwYRngPV9UP6W6rezXwU7p/BB7YM8snabe3tdsHN1u7kvYWuqvU/02XgH+tb7bzgKV034ifABxaVWsZzcvp/mFYDXyVLiF53wzD/DTdc8Tr6K6iP6NdHZku9ofS7d+f0z1v/sqqWt1u4f0j4HC6b8x/wu0dBtGWcWW62+leSncr9mx7I3AhcDHwHeCbjP47kq+hu1J3bovxi3RXaAZ5H91dB1+h6/DsZn47EZvKccAH090m+KzpZp5hbDD1sXUE3fNvP6I79v+2qr7Yph0EXNL27f+he0bylwPa/z/Aoel6cx32u90fo/sH7uyquq6v7kq62yBvpDtHHDCkjaVtXX9OdzyeXFVfal8UPYXu+csr2nq+F9jwD/8b6K5kX0GXnPbfIfJK4Kl0XwQ8h+7OBgCq6kK65/zeTvfeWEX3bOC0qvtJlhfRdSh0A10v1BuuHD2fLlG7tLV7Brc/LjLwPTVkMe+mey9tMNNj/p/pOou6jm7bf36UdWvrdwndcb6C7uruz+mend/wE38z2bcjq6rPtbjPptsfZ08x78B90M5Pr6BLuNfR3eGwsqfed+mO2dXtfbnbNOtzT7p9+DO626G/zPA7kf6B7ouM6zOkV/8+M9mO0+3P59El8N+l21d/McX6fpjuquSVdO+b3/SyPuJn2nS+TLf//g34p6r6Qiv/S7r9cSNd4tvbu/vdWtk6uvf0WrpbqWHq8+LAc8cM451tQz832v8eJwBfa/vj4dM11r4U/Hk75qVZl6pNvcNB0kKR7ucV/qwnWRj38l5I1/HRo+Ziedp6eGz9tnQ/F/eR6vpJ2OIl+RpwdFV9a8Jx3JXui4alVXXFJGORtjZJPk7X+dm0d6BJm2LxpAOQNFlJnkl3G9nQqxOSNF9U1SMntewkT6W7Ehi653u/Q3flUdIcqqpnTjoGLWwmydJWLMk5dJ2BPK894yRJGu4QultGQ3eb9+HlLXmStOB4u7UkSZIkSY0dd0mSJEmS1JgkS5IkSZLU+EzyADvvvHMtWbJk0mFIkiRJksbgoosuuq6qdhk0zSR5gCVLlnDhhRdOOgxJkiRJ0hgk+cGwad5uLUmSJElSY5IsSZIkSVJjkixJkiRJUmOSLEmSJElSY5IsSZIkSVJjkixJkiRJUmOSLEmSJElSY5IsSZIkSVJjkixJkiRJUmOSLEmSJElSY5IsSZIkSVKzeNIBSJJmwZuXwk3XzKzOdrvCMd8bTzySJElbKK8kS9JCMNMEeVPrSJIkLXAmyZIkSZIkNSbJkiRJkiQ1JsmSJEmSJDUmyZIkSZIkNWNNkpMclOTyJKuSHDtg+rZJTmvTz0uypGfaa1v55UkOnK7NJEe3skqyc99yHpPk20kuSfLl8aytJEmSJGlLN7YkOcki4B3AwcAy4Igky/pmOxJYV1X7AicBJ7a6y4DDgf2Bg4CTkyyaps2vAU8AftAXx47AycDTqmp/4LDZXldJkiRJ0sIwzivJDwNWVdXqqroVWAEc0jfPIcAH2/AZwOOTpJWvqKpbquoKYFVrb2ibVfWtqrpyQBx/Anyiqn7Y5vM3TyRJkiRJA40zSd4duKpnfE0rGzhPVa0HbgB2mqLuKG322w+4e5JzklyU5PkzXA9JkiRJ0lZi8aQDmAOLgYcAjwfuDHwjyblV9V+9MyU5CjgKYK+99przICVJkiRJkzfOK8lXA3v2jO/RygbOk2QxsAOwdoq6o7TZbw1wVlXdVFXXAV8BHtg/U1WdUlXLq2r5LrvsMk2TkiRJkqSFaJxJ8gXA0iT7JNmGriOulX3zrARe0IYPBc6uqmrlh7fer/cBlgLnj9hmv08Dj0qyOMldgAOAy2Zh/SRJkiRJC8zYbreuqvVJjgbOAhYB76uqS5IcD1xYVSuBU4EPJ1kF/JQu6aXNdzpwKbAeeFlV3QbdTz31t9nKXwH8FXBP4OIkZ1bVi6vqsiSfBy4Gfg28t6r+c1zrLUmSJAmWHPvZGde58k1PHkMk0syku3CrXsuXL68LL7xw0mFI0uiO22ET690wu3FIktSYJGs+S3JRVS0fNG2ct1tLkiRJkrRFMUmWJEmSJKkxSZYkSZIkqTFJliRJkiSpGVvv1pK2AG9eCjddM/N62+0Kx3xv9uORJEmSJswrydLWbFMS5M2pJ0mSJM1zJsmSJEmSJDUmyZIkSZIkNSbJkiRJkiQ1JsmSJEmSJDUmyZIkSZIkNSbJkiRJkiQ1JsmSJEmSJDUmyZIkSZIkNSbJkiRJkiQ1JsmSJEmSJDUmyZIkSZIkNYsnHYCkrdybl8JN18yszna7wjHfG088kiRJ2qp5JVnSZM00Qd7UOpIkSdIITJIlSZIkSWpMkiVJkiRJakySJUmSJElqxpokJzkoyeVJViU5dsD0bZOc1qafl2RJz7TXtvLLkxw4XZtJjm5llWTnAct6aJL1SQ6d/TWVJEmSJC0EY0uSkywC3gEcDCwDjkiyrG+2I4F1VbUvcBJwYqu7DDgc2B84CDg5yaJp2vwa8ATgB0NiORH4wqyupCRJkiRpQRnnleSHAauqanVV3QqsAA7pm+cQ4INt+Azg8UnSyldU1S1VdQWwqrU3tM2q+lZVXTkklpcDHwfsEleSJEmSNNQ4k+Tdgat6xte0soHzVNV64AZgpynqjtLmRpLsDvwx8M4Zr4EkSZIkaauyNXTc9c/Aa6rq11PNlOSoJBcmufDaa6+do9AkSZIkSfPJ4jG2fTWwZ8/4Hq1s0DxrkiwGdgDWTlN3ujb7LQdWdHdxszPwpCTrq+pTvTNV1SnAKQDLly+vadqUJEmSJC1A47ySfAGwNMk+Sbah64hrZd88K4EXtOFDgbOrqlr54a33632ApcD5I7a5karap6qWVNUSuuee/7w/QZYkSZIkCcaYJLdnjI8GzgIuA06vqkuSHJ/kaW22U4GdkqwCXgUc2+peApwOXAp8HnhZVd02rE2AJK9Isobu6vLFSd47rnWTJEmSJC1M47zdmqo6Ezizr+z1PcM3A4cNqXsCcMIobbbytwFvmyaeF44StyRJkiRp67Q1dNwlSZIkSdJITJIlSZIkSWpMkiVJkiRJakySJUmSJElqTJIlSZIkSWpmlCQnuXuSB4wrGEmSJEmSJmnaJDnJOUnuluQewDeB9yR56/hDkyRJkiRpbo1yJXmHqvoZ8AzgQ1V1APCE8YYlSZIkSdLcGyVJXpzkXsCzgM+MOR5JkiRJkiZmlCT5eOAs4PtVdUGSewPfG29YkiRJkiTNvcXTzVBV/wr8a8/4auCZ4wxKkiRJkqRJGKXjrv2S/FuS/2zjD0jyuvGHJkmSJEnS3Brlduv3AK8FfgVQVRcDh48zKEmSJEmSJmGUJPkuVXV+X9n6cQQjSZIkSdIkjZIkX5fkPkABJDkU+PFYo5IkSZIkaQKm7bgLeBlwCnDfJFcDVwDPHWtUkiRJkiRNwCi9W68GnpBkO+AOVXXj+MOSJEmSJGnuTZskJ3l93zgAVXX8mGKSJEmSJGkiRrnd+qae4TsBTwEuG084kiRJkiRNzii3W7+ldzzJPwFnjS0iSZIkSZImZJTerfvdBdhjtgORJEmSJGnSRnkm+Tu0n38CFgG7AD6PLEmSJElacEa5kvwU4Knt9UfAblX19lEaT3JQksuTrEpy7IDp2yY5rU0/L8mSnmmvbeWXJzlwujaTHN3KKsnOPeXPSXJxku8k+XqSB44SuyRJkiRp6zP0SnKSe7TB/p98ulsSquqnUzWcZBHwDuCJwBrggiQrq+rSntmOBNZV1b5JDgdOBJ6dZBlwOLA/sBvwxST7tTrD2vwa8BngnL5QrgAeXVXrkhxM95vPB0wVuyRJkrSlW3LsZ2dc58o3PXkMkUhblqlut76I7jbrDJhWwL2nafthwKr2O8skWQEcAvQmyYcAx7XhM4C3p/uNqUOAFVV1C3BFklWtPYa1WVXfamUbB1r19Z7Rc/F5akmSJEnSEEOT5KraZzPb3h24qmd8Db99Bfc381TV+iQ3ADu18nP76u7ehqdrcypHAp+bwfySJEmSpK3IKL+TTJK7A0vpficZgKr6yriCGockj6VLkh81ZPpRwFEAe+211xxGJkmSJEmaL0bp3frFwCvpblP+NvBw4BvA46apejWwZ8/4Hq1s0DxrkiwGdgDWTlN3ujYHrcMDgPcCB1fV2kHzVNUpdM8rs3z58ho0jyRJkiRpYRuld+tXAg8FflBVjwUeBFw/Qr0LgKVJ9kmyDV1HXCv75lkJvKANHwqcXVXVyg9vvV/vQ3cV+/wR29xIkr2ATwDPq6r/GiFuSZIkSdJWapQk+eaquhm6n2yqqu8CvztdpapaDxwNnAVcBpxeVZckOT7J09pspwI7tY65XgUc2+peApxO18nX54GXVdVtw9pssb0iyRq6q8sXJ3lvW8br6Z5zPjnJt5NcOMI6S5IkSZK2QqM8k7wmyY7Ap4D/l2Qd8INRGq+qM4Ez+8pe3zN8M3DYkLonACeM0mYrfxvwtgHlLwZePEq8kiRJkqSt27RJclX9cRs8LsmX6J4b/vxYo5IkSZIkaQJG6bjrbXS/Wfz1qvryHMQkSZIkSdJEjPJM8kXA65J8P8k/JVk+7qAkSZIkSZqEaZPkqvpgVT2Jrofry4ETk3xv7JFJkiRJkjTHRum4a4N9gfsCe9P1LC1pc715Kdx0zczqbLcrHOP3VJIkSdI4THslOck/tivHxwPfAZZX1VPHHpm0NZhpgrypdSRJkiSNZJQryd8HHlFV1407GEmSJEmSJmmUn4B691wEIkmSJEnSpI3Su7UkSZIkSVsFk2RJkiRJkppROu56S5L95yIYSZIkSZImaZSOuy4DTkmyGHg/8LGqumG8YUkaiT8hJUmSJM2qaa8kV9V7q+qRwPOBJcDFSf4lyWPHHZykafgTUpIkSdKsGumZ5CSLgPu213XAfwCvSrJijLFJkiRJkjSnpr3dOslJwFOAs4G/r6rz26QTk1w+zuAkSZIkSZpLozyTfDHwuqq6acC0h81yPJIkSZIkTcwot1tfT08ynWTHJE8HsAMvSZIkSdJCMkqS/Le9yXBVXQ/87fhCkiRJkiRpMkZJkgfNM8pt2pIkSZIkbVFGSZIvTPLWJPdpr7cCF407MEmSJEmS5tooSfLLgVuB09rrFuBl4wxKkiRJkqRJmPa26dar9bFzEIukLdGbl8JN18yszna7wjHfG088kiRJ0maY9kpykv2SnJLkC0nO3vAapfEkB2SkBBMAAB34SURBVCW5PMmqJL+VaCfZNslpbfp5SZb0THttK788yYHTtZnk6FZWSXbuKU+St7VpFyd58CixSxrRTBPkTa0jSZIkzYFROuD6V+BdwHuB20ZtOMki4B3AE4E1wAVJVlbVpT2zHQmsq6p9kxwOnAg8O8ky4HBgf2A34ItJ9mt1hrX5NeAzwDl9oRwMLG2vA4B3tr+SJEmSJG1klCR5fVW9cxPafhiwqqpWAyRZARwC9CbJhwDHteEzgLcnSStfUVW3AFckWdXaY1ibVfWtVtYfxyHAh6qqgHPb7zzfq6p+vAnrJEmSJElawEbpuOv/JvnzJPdKco8NrxHq7Q5c1TO+ppUNnKeq1gM3ADtNUXeUNjclDkmSJEmSRrqS/IL295iesgLuPfvhTE6So4CjAPbaa68JRyNJkiRJmoRRerfeZxPbvhrYs2d8j1Y2aJ41SRYDOwBrp6k7XZubEgdVdQpwCsDy5ctrmjYlSZIkSQvQKL1b3yXJ65Kc0saXJnnKCG1fACxNsk+Sbeg64lrZN89Kbr9SfShwdnt2eCVweOv9eh+6TrfOH7HNfiuB57derh8O3ODzyJIkSZKkQUZ5Jvn9wK3AH7Txq4E3TlepPWN8NHAWcBlwelVdkuT4JE9rs50K7NQ65noV7feYq+oS4HS6Tr4+D7ysqm4b1iZAklckWUN3pfjiJO9tyzgTWA2sAt4D/PkI6yxJkiRJ2gqN8kzyfarq2UmOAKiqX2RAF9KDVNWZdElqb9nre4ZvBg4bUvcE4IRR2mzlbwPeNqC8gJeNEq8kSZIkaes2ypXkW5Pcma6zLpLcB7hlrFFJkiRJkjQBo1xJ/lu6W573TPJR4JHAC8cZlCRJkiRJkzBK79b/L8k3gYcDAV5ZVdeNPTJJkiRJkubY0Nutk9y3/X0wsDfwY+BHwF6tTJIkSZKkBWWqK8mvBl4CvGXAtAIeN5aIJEmSJEmakKFJclW9pP197NyFI0mSJEnS5AxNkpM8Y6qKVfWJ2Q9HkiRJkqTJmep266dOMa0Ak2RJkiRJ0oIy1e3WL5rLQCRJkiRJmrRRfieZJE8G9gfutKGsqo4fV1CSJEmSJE3C0J+A2iDJu4BnAy+n+53kw+h+EkqSJEmSpAVl2iQZ+IOqej6wrqreADwC2G+8YUmSJEmSNPdGSZJ/2f7+IsluwK+Ae40vJEmSJEmSJmOUZ5I/k2RH4M3AN+l6tn7vWKOSpLn05qVw0zUzq7PdrnDM98YTjyRJkiZm2iS5qv6uDX48yWeAO1XVDeMNS5Lm0EwT5E2tI0mSpHlvlI67DkuyfRs9Bnh/kgeNNyxJkiRJkubeKM8k/01V3ZjkUcATgFOBd403LEmSJEmS5t4oSfJt7e+TgVOq6rPANuMLSZIkSZKkyRglSb46ybvpfiv5zCTbjlhPkiRJkqQtyijJ7rOAs4ADq+p64B50zyZLkiRJkrSgjNK79S+AT/SM/xj48TiDkqQtij8htSAsOfazM65z5ZuePIZIJEnSJHnbtCRtLn9CSpIkacEYeiU5ybZVdcvmNJ7kIOD/AIuA91bVm/qXAXwIeAiwFnh2VV3Zpr0WOJKu47BXVNVZU7WZZB9gBbATcBHwvKq6NclewAeBHVudY6vqzM1ZLy0QXv2TJEmS1GeqK8nfAEjy4U1pOMki4B3AwcAy4Igky/pmOxJYV1X7AicBJ7a6y4DDgf2Bg4CTkyyaps0TgZNaW+ta2wCvA06vqge1Nk/elPXRAuTVP0mSJEl9pnomeZskfwL8QZJn9E+sqk8MqNPrYcCqqloNkGQFcAhwac88hwDHteEzgLcnSStf0a5kX5FkVWuPQW0muQx4HPAnbZ4PtnbfCRRwt1a+A/CjaeKWJEmSJG2lpkqSXwo8h+425af2TSt6OvMaYnfgqp7xNcABw+apqvVJbqC7XXp34Ny+uru34UFt7gRcX1XrB8x/HPCFJC8HtgOeME3ckiRJkqSt1NAkuaq+Cnw1yYVVdeocxjTbjgA+UFVvSfII4MNJ7ldVv+6dKclRwFEAe+211wTClCRJkiRN2ii9W384ySuSnNFeL09yxxHqXQ3s2TO+RysbOE+SxXS3Q6+dou6w8rXAjq2N/mUdCZwOUFXfAO4E7NwfbFWdUlXLq2r5LrvsMsLqSZIkSZIWmlGS5JPpep8+ub0eTPes73QuAJYm2SfJNnSdZq3sm2cl8II2fChwdlVVKz88ybat1+qlwPnD2mx1vtTaoLX56Tb8Q+DxAEl+jy5JvnaE+CVJkiRJW5mpnkne4KFV9cCe8bOT/Md0ldozxkcDZ9H99NL7quqSJMcDF1bVSuBUuivVq4Cf0iW9tPlOp+vkaz3wsqq6DWBQm22RrwFWJHkj8K3WNsCrgfck+V90z1K/sCXVkiRJkiRtZJQk+bYk96mq7wMkuTfdbxdPq/0e8Zl9Za/vGb4ZOGxI3ROAE0Zps5Wv5vYesHvLLwUeOUq8kiRJkqSt2yhJ8jHAl5KsBgLsDbxorFFJkiRJkjQB0ybJVfVvSZYCv9uKLm+/XyxJkiRJ0oIyypVkWlJ88ZhjkSRJkiRpokbp3VqSJEmSpK2CSbIkSZIkSc20SXI6z03y+ja+V5Lf6kVakiRJkqQt3SjPJJ8M/Bp4HHA8cCPwceChY4xLkiRpzi059rMzrnPlm548hkgkSZMySpJ8QFU9OMm3AKpqXZJtxhyXJEmSJElzbpRnkn+VZBFQAEl2obuyLEmSJEnSgjJKkvw24JPArklOAL4K/P1Yo5IkSZIkaQKmvd26qj6a5CLg8UCAp1fVZWOPTJIkSZKkOTY0SU5yj57Ra4CP9U6rqp+OMzBJkiRJkubaVFeSL6J7DjnAXsC6Nrwj8ENgn7FHJ0mSJEnSHBr6THJV7VNV9wa+CDy1qnauqp2ApwBfmKsAJUmSJEmaK6P8BNTDq+olG0aq6nNJ/nGMMUnS6N68FG66ZmZ1ttsVjvneeOKRJEnSFm2UJPlHSV4HfKSNPwf40fhCkqQZmGmCvKl1JEmStFUY5SegjgB2ofsZqE8Cu7YySZIkSZIWlFF+AuqnwCvnIBZJkiRJkiZq2iQ5yZfoerneSFU9biwRSZIkSZI0IaM8k/yXPcN3Ap4JrB9POJIkSZIkTc4ot1tf1Ff0tSTnjykeSZIkSZImZpTbre/RM3oH4CHADmOLSJIkSZKkCRnlduuL6J5JDt1t1lcAR44zKEmSJEmSJmGUn4D6vaq6d1XtU1VLq+qPgAtGaTzJQUkuT7IqybEDpm+b5LQ2/bwkS3qmvbaVX57kwOnaTLJPa2NVa3ObnmnPSnJpkkuS/MsosUuSJEmStj6jJMlfH1D2jekqJVkEvAM4GFgGHJFkWd9sRwLrqmpf4CTgxFZ3GXA4sD9wEHBykkXTtHkicFJra11rmyRLgdcCj6yq/YG/GGGdJUmSJElboaFJcpJ7JnkIcOckD0ry4PZ6DHCXEdp+GLCqqlZX1a3ACuCQvnkOAT7Yhs8AHp8krXxFVd1SVVcAq1p7A9tsdR7X2qC1+fQ2/BLgHVW1DqCqrhkhdkmSJEnSVmiqZ5IPBF4I7AG8taf8RuCvR2h7d+CqnvE1wAHD5qmq9UluAHZq5ef21d29DQ9qcyfg+qpaP2D+/QCSfA1YBBxXVZ/vDzbJUcBRAHvttdcIqydJkiRJWmiGJslV9UHgg0meWVUfn8OYZttiYCnwGLqE/ytJ7l9V1/fOVFWnAKcALF++vOY6SEmSJEnS5A1NkpM8t6o+AixJ8qr+6VX11gHVel0N7NkzvkcrGzTPmiSL6X5aau00dQeVrwV2TLK4XU3unX8NcF5V/Qq4Isl/0SXNI3U+JkmSJEnaekzVcdd27e9dge0HvKZzAbC09Tq9DV1HXCv75lkJvKANHwqcXVXVyg9vvV/vQ5fUnj+szVbnS60NWpufbsOforuKTJKd6W6/Xj1C/JIkSZKkrcxUt1u/u/19w6Y03J4xPho4i+5Z4PdV1SVJjgcurKqVwKnAh5OsAn5Kl/TS5jsduJTut5lfVlW3AQxqsy3yNcCKJG8EvtXaps37R0kuBW4DjqmqtZuyTpIkSZKkhW2qjrsASLILXQ/RS3rnr6o/na5uVZ0JnNlX9vqe4ZuBw4bUPQE4YZQ2W/lqut6v+8sLeFV7SZIkSZI01LRJMt1ty/8OfJHuSqwkSZIkSQvSKEnyXarqNWOPRJIkSZKkCZuq464NPpPkSWOPRJIkSZKkCRslSX4lXaL8yyQ/S3Jjkp+NOzBJkiRJkubatLdbV9UoP/ckSZIkSdIWb5TerR88oPgG4AdVtX72Q5IkSZIkaTJG6bjrZODBwHfa+P2B/wR2SPI/q+oL4wpOkiRJkqS5NMozyT8CHlRVD6mqhwC/D6wGngj84ziDkyRJkiRpLo2SJO9XVZdsGKmqS4H7VtXq8YUlSZIkSdLcG+V260uSvBNY0cafDVyaZFvgV2OLTJIkSZKkOTbKleQXAquAv2iv1a3sV8BjxxWYJEmSJElzbZSfgPol8Jb26vfzWY9IkiRJkqQJGeUnoJYC/wAsA+60obyq7j3GuCRJkiRJmnOj3G79fuCdwHq626s/BHxknEFJkiRJkjQJoyTJd66qfwNSVT+oquOAJ483LEmSJEmS5t4ovVvfkuQOwPeSHA1cDdx1vGFJkiRJkjT3RrmS/ErgLsArgIcAzwNeMM6gJEmSJEmahFF6t76gDf4ceNF4w5EkSZIkaXKGJslJVk5VsaqeNvvhSJIkSZI0OVNdSX4EcBXwMeA8IHMSkSRJkiRJEzJVknxP4InAEcCfAJ8FPlZVl8xFYJIkSZIkzbWhHXdV1W1V9fmqegHwcGAVcE7r4VqSJEmSpAVnyt6tk2yb5BnAR4CXAW8DPjlq40kOSnJ5klVJjh3S/mlt+nlJlvRMe20rvzzJgdO1mWSf1saq1uY2fct6ZpJKsnzU+CVJkiRJW5ehSXKSDwHfAB4MvKGqHlpVf1dVV4/ScJJFwDuAg4FlwBFJlvXNdiSwrqr2BU4CTmx1lwGHA/sDBwEnJ1k0TZsnAie1tta1tjfEsj3dT1mdN0rskiRJkqSt01RXkp8LLKVLLr+e5GftdWOSn43Q9sOAVVW1uqpuBVYAh/TNcwjwwTZ8BvD4JGnlK6rqlqq6gu5W74cNa7PVeVxrg9bm03uW83d0SfTNI8QtSZIkSdpKTfVM8h2qavv2ulvPa/uqutsIbe9O1zv2Bmta2cB5qmo9cAOw0xR1h5XvBFzf2thoWUkeDOxZVZ+dKtgkRyW5MMmF11577QirJ0mSJElaaKZ8JnlLl+QOwFuBV083b1WdUlXLq2r5LrvsMv7gJEmSJEnzzlQ/AbW5rgb27Bnfo5UNmmdNksXADsDaaeoOKl8L7JhkcbuavKF8e+B+dL1yQ/ezViuTPK2qLtzsNZQkSZIWqCXHTnkj5kBXvunJY4hEmlvjvJJ8AbC09Tq9DV1HXCv75lkJvKANHwqcXVXVyg9vvV/vQ/ds9PnD2mx1vtTaoLX56aq6oap2rqolVbUEOBcwQZYkSZIkDTS2K8lVtb79pvJZwCLgfVV1SZLjgQuraiVwKvDhJKuAn9IlvbT5TgcuBdYDL6uq2wAGtdkW+RpgRZI3At9qbUuSJEmSNLJx3m5NVZ0JnNlX9vqe4ZuBw4bUPQE4YZQ2W/lqut6vp4rnMaPELUmSJEnaOi3ojrskSZIkSZoJk2RJkiRJkhqTZEmSJEmSGpNkSZIkSZIak2RJkiRJkpqx9m4tLXhvXgo3XTOzOtvtCsd8bzzxSJIkSdosXkmWNsdME+RNrSNJkiRpTpgkS5IkSZLUmCRLkiRJktSYJEuSJEmS1JgkS5IkSZLU2Lu1JElaMJYc+9kZ17nyTU8eQySSpC2VV5IlSZIkSWpMkiVJkiRJakySJUmSJElqTJIlSZIkSWrsuEuSpAXCTqskSdp8XkmWJEmSJKkxSZYkSZIkqTFJliRJkiSpMUmWJEmSJKkxSZYkSZIkqRlrkpzkoCSXJ1mV5NgB07dNclqbfl6SJT3TXtvKL09y4HRtJtmntbGqtblNK39VkkuTXJzk35LsPc51liRJkiRtucaWJCdZBLwDOBhYBhyRZFnfbEcC66pqX+Ak4MRWdxlwOLA/cBBwcpJF07R5InBSa2tdaxvgW8DyqnoAcAbwj+NYX0mSJEnSlm+cV5IfBqyqqtVVdSuwAjikb55DgA+24TOAxydJK19RVbdU1RXAqtbewDZbnce1NmhtPh2gqr5UVb9o5ecCe4xhXSVJkiRJC8A4k+Tdgat6xte0soHzVNV64AZgpynqDivfCbi+tTFsWdBdXf7cJqyLJEmSJGkrsHjSAcyVJM8FlgOPHjL9KOAogL322msOI5MkSZIkzRfjTJKvBvbsGd+jlQ2aZ02SxcAOwNpp6g4qXwvsmGRxu5q80bKSPAH438Cjq+qWQcFW1SnAKQDLly+v0VdTkiRJ0kK05NjPzrjOlW968hgi0Vwa5+3WFwBLW6/T29B1xLWyb56VwAva8KHA2VVVrfzw1vv1PsBS4PxhbbY6X2pt0Nr8NECSBwHvBp5WVdeMaV0lSZIkSQvA2K4kV9X6JEcDZwGLgPdV1SVJjgcurKqVwKnAh5OsAn5Kl/TS5jsduBRYD7ysqm4DGNRmW+RrgBVJ3kjXo/WprfzNwF2Bf+369+KHVfW0ca23JEmSJGnLNdZnkqvqTODMvrLX9wzfDBw2pO4JwAmjtNnKV9P1ft1f/oQZBy5JkiRJ2iqN83ZrSZIkSZK2KFtN79aSJM13dhAj8DiQpEnzSrIkSZIkSY1JsiRJkiRJjUmyJEmSJEmNzyRLkiRJ84zPpkuTY5K8hfLEKUmSJEmzzyRZkrQg+OWhJEmaDT6TLEmSJElSY5IsSZIkSVJjkixJkiRJUmOSLEmSJElSY5IsSZIkSVJj79aSNGlvXgo3XTOzOtvtCsd8bzzxSJIkbcW8kixJkzbTBHlT60iSJGlaJsmSJEmSJDUmyZIkSZIkNSbJkiRJkiQ1JsmSJEmSJDUmyZIkSZIkNSbJkiRJkiQ1JsmSJEmSJDVjTZKTHJTk8iSrkhw7YPq2SU5r089LsqRn2mtb+eVJDpyuzST7tDZWtTa3mW4ZkiRJkiT1GluSnGQR8A7gYGAZcESSZX2zHQmsq6p9gZOAE1vdZcDhwP7AQcDJSRZN0+aJwEmtrXWt7aHLkCRJkiSp3zivJD8MWFVVq6vqVmAFcEjfPIcAH2zDZwCPT5JWvqKqbqmqK4BVrb2BbbY6j2tt0Np8+jTLkCRJkiRpI+NMkncHruoZX9PKBs5TVeuBG4Cdpqg7rHwn4PrWRv+yhi1DkiRJkqSNpKrG03ByKHBQVb24jT8POKCqju6Z5z/bPGva+PeBA4DjgHOr6iOt/FTgc63ab7XZM/++rXxP4HNVdb9hy6iq6/riPQo4qo3+LnD5LG6OubQzcN20c423jS29/nyIwXWYHzFMuv58iMF1mB8xTLr+fIjBdZgfMUy6/nyIwXWYHzFMuv58iGEhrMMk7V1VuwyasHiMC70a2LNnfI9WNmieNUkWAzsAa6epO6h8LbBjksXtanHv/MOWsZGqOgU4ZYbrOO8kubCqlk+yjS29/nyIwXWYHzFMuv58iMF1mB8xTLr+fIjBdZgfMUy6/nyIwXWYHzFMuv58iGEhrMN8Nc7brS8AlrZep7eh64hrZd88K4EXtOFDgbOru7S9Eji89Uy9D7AUOH9Ym63Ol1obtDY/Pc0yJEmSJEnayNiuJFfV+iRHA2cBi4D3VdUlSY4HLqyqlcCpwIeTrAJ+Spf00uY7HbgUWA+8rKpuAxjUZlvka4AVSd4IfKu1zbBlSJIkSZLUb5y3W1NVZwJn9pW9vmf4ZuCwIXVPAE4Ypc1Wvpqu9+v+8qHLWKBm45bxzW1jS68/H2JwHeZHDJOuPx9icB3mRwyTrj8fYnAd5kcMk64/H2JwHeZHDJOuPx9iWAjrMC+NreMuSZIkSZK2NON8JlmSJEmSpC2KSfICkuTpSSrJfee6fpKdkny7vX6S5Oqe8W1GbON3kvxLktVJLkryjSR/PFfLb+3cM8mKJN9vMZyZZL8Z1N/kdehpY3P34+auwxa/DTa3jc3ZBvPoWJwP+3GzYmhtbLH7cdLbcB6tw2a1MUvH0Rb9fpp0/dbGpLfhbJ1bN+mcMunlz1YMkz6WWxub+3/ORGOYhffjfDk3T/T/zXmvqnwtkBdwGvDvwBsmUb+nneOAv5xhnQDfAF7aU7Y38PK5WP4UMTwQ+MO5XIfN2Q9jWoctahtsbhubuw3m8bE4p/txtrbjlrof59M2nIfrMHIbs7ENtvT306Trz4dtOBvHc0/d2fh8mejyNzWGSR/Ls7ENJh3DbC1/DPtxPuyHTX5Pz8fXxAPwNUs7Eu5K95vQ+wGXz3X9vrY25Q3/eODLs7QtNunDC3gc8JXNWO5mr8Ms7MfNXYctfhtsbhubuw362prUsTgf9uNmb8cteT/Ol204D9Zhs9qYpeNoi34/Tbr+fNiGA9rb1HPrrPyvM+nlb2oMkz6WZ2MbTDqG2Vj+LOzH+bAfZvU9PR9f3m69cBwCfL6q/gtYm+Qhc1x/c+0PfHOOl/n/t3cHoZpWdRzHv7+80mSDiujCYLBZhIIzLnRlUFJEE0lM5UC4CgQFEdyJUFvBhQtFaNcmMpgWoVCBBiYimJgT6jhRs0hXGokoMghqM/8W91yQwYXOc973PM/T97O5733uPec5//95n/ee/z3Pe+/5DgEnJrTvEcPUeZgawxpyMLWPqTnoYQ3z2COPS57HueRwih4xTO2jRw6Wfj2Nbg/jc9jL6LXO6POPfi7D9ByMHsMc1qujcwDzuaY3xiJ5PW4HjrfHx9vn22zfVZJfJHklyV9HjmOKC4xhVvMw1cAcrCqPow28Hlczj//Hr2nd+1i6qTkY3X7hRr+mjD5/V3NY54wewxyup9E5WKuN/p9kbUeSK9i97eFwkgIuAirJfdXuidhk+05OAbftfVJV9yS5EnhpS+ffG8Oxie0vOIZO89AjhkXnoEMfU3PQw6LnsccYVjCPw3PYQa8YpvTRIwdLv55Gt9/rY2QOJxu91hl9/mbxP+NnMIa5rFfnMA+j10ob5U7yOhwDfl1V11TVV6vqAPA68I0tte/hz8C+JHd/4tglWzz/3hi+mOSuvQNJbkjyWfMwNYYe89AjhqXnYGofU3PQw9LnsccYlj6Pc8jhVL1imNJHjxws/Xoa3X6vj5E57GH0Wmf0+WH8c7nXOmfkGOayXp3DPIy+pjfKInkdbgceP+/Y7/jst05MbT9Z+83VD4Fbkrye5EXgV8D9Wx7Dj4DvZPfP2Z8CHgT+/TnaT4lh8jx0imHROZjax9Qc9LCCeeyRx0XP40xyOEnHGC64jx45WPr1NLr9J/oYlsNORq91Rp9/+HOZfuucYWOY0Xp1DvMw+preqGzvDg9JkiRJkubNnWRJkiRJkhqLZEmSJEmSGotkSZIkSZIai2RJkiRJkhqLZEmSJEmSGotkSZJWJMnZJC8neSXJ35J8ffSYJElaEv8FlCRJK5LkTFXtb4+PAD+rqlsGD0uSpMVwJ1mSpPW6FHgXIMn+JE+33eWTSY62419O8se28/xakp+04zcleTbJiSRPJbm6Hb83yd+TvJrk+LDIJEnaEHeSJUlakSRngZPAPuBq4NtVdSLJDnBJVb2f5ErgBeBrwI+B71XVna39ZcAHwLPA0ap6uxXOR6rqjiRvAger6sMkl1fVe9uPUpKkzbFIliRpRc673fpm4JfAIWAHeBj4JnAOuBY4yO5u85+A3wJ/qKrnkhwCngf+1bq9CHirqr6b5EngDPAE8ERVndlacJIkbcHO6AFIkqTNqKq/tF3jq4Dvt483VdXHSd4A9lXV6SQ3tq8/kORp4HHgVFXd/Cnd3spuof0D4OdJDlfVf7cRjyRJ2+B7kiVJWqkk17G7C/wOcBnwn1Ygfwu4pn3PV4APquox4CHgRuCfwFVtJ5okFye5PskXgANV9Qxwf+tz/7bjkiRpk9xJliRpXb6U5OX2OMBPq+pskt8Av09yEngJ+Ef7nsPAQ0nOAR8Dd1fVR0mOAY+29yjvAI8Ap4HH2rEAj/qeZEnS2vieZEmSJEmSGm+3liRJkiSpsUiWJEmSJKmxSJYkSZIkqbFIliRJkiSpsUiWJEmSJKmxSJYkSZIkqbFIliRJkiSpsUiWJEmSJKn5H7Z7ILkCjtO3AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "voPIrz9TPtIH"
      },
      "source": [
        "The results above should show high saliency values for the bases _CGACCGAACTCC_ appearing in the DNA sequence. If you recall from the top of the document, this is exactly the motif that we embedded in the positive sequences! The raw saliency values may be non-zero for other bases as well -- the gradient-based saliency map method is not perfect, and there other more complex interpretation methods that are used in practice to obtain better results.  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2.5 - Hyperparameter tuning (graded portion)\n",
        "\n"
      ],
      "metadata": {
        "id": "uqDt9AeMvmCE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "For the following problems, you can (and should) copy/reuse any of the code from above."
      ],
      "metadata": {
        "id": "zXyCTman8tpF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### a. (10 pts) \n",
        "The base model above uses `'binary_crossentropy'`(BCE) as its loss function. Re-train the model using `'mean_squared_error'` (MSE) and `'hinge'` as loss instead. Show, in a plot, how the precision and recall are affected on the test set for each of the losses (BCE,MSE, and hinge). Hint: simply replace the loss when doing `model.compile `"
      ],
      "metadata": {
        "id": "TwUw8NKS8EQF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### START CODE HERE ### \n",
        "\n",
        "### END CODE HERE ###"
      ],
      "metadata": {
        "id": "IUpHywIS7fWx"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### b. (10 pts) \n",
        "Explain why the binary cross entropy loss is more appropriate than MSE or hinge loss for this classification problem\n"
      ],
      "metadata": {
        "id": "MrWkNDZY9hgu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**type your answer here (double click to edit)**"
      ],
      "metadata": {
        "id": "MRrB0FBV_AaR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### c. (10 pts)\n",
        "\n",
        "Keeping all other parameters equal, vary the number of filters in the first convolutional layer from 1 to 64, in powers of 2, and show in a plot how the precision and recall are affected on the test set."
      ],
      "metadata": {
        "id": "dTjwnayj-IsA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "filters = 2**np.arange(7)\n",
        "\n",
        "### START CODE HERE ### \n",
        "for filter in filters:\n",
        "### END CODE HERE ###"
      ],
      "metadata": {
        "id": "mf912ZiwOcr8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### d. (10 pts)\n",
        "\n",
        "Keeping all other parameters equal, vary the size of the filter (kernel size) in the first convolutional layer from 1 to 32, in powers of 2, and show in a plot how the precision and recall are affected on the test set."
      ],
      "metadata": {
        "id": "nMnWyFJE_Air"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "kernel_sizes = 2**np.arange(6)\n",
        "\n",
        "### START CODE HERE ### \n",
        "for kernel_size in kernel_sizes:\n",
        "### END CODE HERE ###\n"
      ],
      "metadata": {
        "id": "9g6yMjvDOvNS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### e. (10 pts)\n",
        "Explain how precision/recall are affected by the number of filters and the filter size. Explain why this might be happening."
      ],
      "metadata": {
        "id": "XcCcOZbzPOJa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**type your answer here (double click to edit)**"
      ],
      "metadata": {
        "id": "St2N4-lqPnx6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### f. extra credit (10 pts)\n",
        "\n",
        "Change the architecture of the convolutional neural network model to see if you can get a better validation accuracy. You can try adding some more convolutional layers or dense layers. \n",
        "\n",
        "Show the saliency map for your new model. How do the saliency values compare to before?\n"
      ],
      "metadata": {
        "id": "4xSTR0ivPrdF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### START CODE HERE ### \n",
        "\n",
        "### END CODE HERE ###"
      ],
      "metadata": {
        "id": "DH1y8vtMSsrk"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}