{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Hopfield Network model of associative memory" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "first name: ...\n", "\n", "last name: ...\n", "\n", "sciper: ...\n", "\n", "date: ...\n", "\n", "*Your teammate*\n", "\n", "first name of your teammate: ...\n", "\n", "last name of your teammate: ...\n", "\n", "sciper of your teammate: ...\n", "\n", "\n", "** Remember **\n", "\n", "If you are asked for plots: The appearance of the plots (labelled axes, ** useful scaling **, etc.) is important!\n", "\n", "If you are asked for discussions: Answer in a precise way and try to be concise. \n", "\n", "\n", "** Submission **\n", "\n", "Rename this notebook to Ex2_FirstName_LastName_Sciper.ipynb and upload that single file on moodle before the deadline.\n", "\n", "Exercise instructions are given in this notebook file.\n", "\n", "** Rules: **\n", "\n", "1) You are strongly encouraged to work in groups of 2. You are allowed to work alone. Groups of 3 or more are NOT allowed\n", "\n", "2) If you work in a group of 2, BOTH people should upload the same notebook file \n", "\n", "3) If you work alone, you can't share your notebook file with anyone else\n", "\n", "4) Discussion between groups is encouraged, but you can't share your code or text\n", "\n", "5) The points assigned to each exercise are indicated in the notebook file\n", "\n", "6) You should upload a jupyter notebook file with all code run and picture visible. We are not going to run your notebook.\n", "\n", "7) Read carefully the instructions at the beginning of the notebook file, answer in a clear and concise way to open questions\n", "\n", "8) You have to understand every line of code you write in this notebook. We will ask you questions about your submission during a fraud detection session during the last week of the semester\n", "\n", "## \\* \\* \\* \\* Programming Notes : Copying and slicing lists and numpy arrays \\* \\* \\* \\*\n", "We would like to take the opportunity to bring to your attention certain features of Python, that might lead to unwanted behaviour and serious mistakes, if one is not aware of them. \n", "Please check the Python Cheat Sheet file on the moodle (https://moodle.epfl.ch/mod/page/view.php?id=981134 Part 4 of the ipynb file) for some examples of the following notes:\n", "\n", "* Assigning a list to a new variable does not create a copy of the list, but creates a variable that points to the list. This means that modifying the second variable, also modifies the original list.\n", "* Assigning a slice of a list to a new variable, creates a copy of the list. Any modification to the sliced list does not modify the original.\n", "\n", "Now when it comes to numpy arrays:\n", "* Assigning a numpy array to a new variable does not create a copy of the array, but creates a variable that points to the array. This means that modifying the second variable, also modifies the original array. (same as above)\n", "* Assigning a slice of a numpy array to a new variable creates a variable that points to the corresponding elements of the original array as well! (contrary to what we saw above!) This means that modifying the second variable, also modifies the original array!\n", "* To copy the original array and ensure that it is not modified by any modification of its copied version, the method copy() should be used." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.1. Getting started:\n", "\n", "See [Chapter 17](https://neuronaldynamics-exercises.readthedocs.io/en/latest/exercises/hopfield-network.html) Section 2 for an introduction to Hopfield networks.\n", "\n", "We provide a couple of functions to easily create patterns, store them in the network and visualize the network dynamics. Check the modules [`hopfield_network.network`](https://neuronaldynamics-exercises.readthedocs.io/en/latest/modules/neurodynex.hopfield_network.html#module-neurodynex.hopfield_network.network), [`hopfield_network.pattern_tools`](https://neuronaldynamics-exercises.readthedocs.io/en/latest/modules/neurodynex.hopfield_network.html#module-neurodynex.hopfield_network.pattern_tools) and [`hopfield_network.plot_tools`](https://neuronaldynamics-exercises.readthedocs.io/en/latest/modules/neurodynex.hopfield_network.html#module-neurodynex.hopfield_network.plot_tools) to learn the building blocks we provide.\n", "\n", "***Note:*** If you instantiate a new object of class `network.HopfieldNetwork` it’s default dynamics are deterministic and synchronous. That is, all states are updated at the same time using the sign function. We use this dynamics in all exercises described below.\n", "\n", "Run the following code. Read the inline comments and check the documentation. The patterns and the flipped pixels are randomly chosen. Therefore the result changes every time you execute this code. Run it several times and change some parameters like nr_patterns and nr_of_flips." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAB4CAYAAADSWhi9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAABZpJREFUeJzt3U2obXUZx/Hfc1MyMsNhg2pQgWAgSAUJDYSwFxKqQYPC\nl0HhwFmD0BDKiYg6KBwUSCEVjZ1ZUKAUQiNxEBQpJRXpwEzvTRv1b3B2dO7h3tvZW9fLs/x84MK5\n5+x91rP/e50v6272+d8aYwSAPs4sPQAA+xFugGaEG6AZ4QZoRrgBmhFugGaEG6CZTYW7qv5UVa9X\n1bmqerGqHq2qK6vq7VX1w6p6tapeqKqvLz1rN5dY2y9V1VNV9VpVPbH0nB1dYm0fqqo/VNXZqvpd\nVd269KwdXWJ9H6iqP++68HxVfXPpWU9rU+HeuXmMcWWS65N8JMk9Sb6d5ENJ3p/kxiTfqKpPLzZh\nXxda278n+U6S+5ccbAMutLb/THJzkncnuS3Jd6vqhuVGbO1C6/uDJNeMMa5KckOSr1TVFxec8dQu\nW3qAqYwx/lpVjyf5cI6erNvHGC8nebmqHklye5KfLThiW8fXdoxxV5JU1VcXHmsTTqzt54596TdV\n9askH0/y1DLT9XdifX9/4sv/TvLBBcba2xavuJMkVfXeJJ9N8sck70nyzLEvP5Pk2iXm2oJja/v0\n0rNszcXWtqrekeSjSX67xFxbcXJ9q+quqjqX5C9J3pnkpwuOd2pbDPdjVfWPJL9O8mSSB3aff+XY\nbV5J8q65B9uAk2t738LzbMn/W9vv5+iC4+dzD7YRF1zfMcb9OWrB9Ul+nPM7sVpbfKnk82OMX/z3\nL1V19e7Dq5L869jHZ+cebAPOW1veVBdd26p6MEcv+d047Ap3qIuu725Nn66qTyW5N8nq37ywxSvu\n8+xe1/5bkuuOffq6+CcnDVTVvUk+k+SmMcarS8+zcZcl+cDSQ5zG5sO986Mk91TV1VV1TZKvJXl0\n2ZG2oareVlVX5OikP1NVV1TV5UvPtQVVdXeSLyf55BjjpaXn2ZKqOlNVd+yaUFX1sSR3Jvnl0rOd\nxlsl3N9K8lyS53P0+taDYwzvKHlz3JLk9STfS/KJ3cePLDrRdtyX5H1Jnt29B/lcp/caN/CFHHXh\nbJKfJHl492f1yktmAL28Va64ATZDuAGaEW6AZoQboBnhBmhmkt+crNR+b1XZ8+YZtd/t9/3+Mxxj\njOx5gN1hDnowzc21tpV1re2+5+AhZlrb5ID1nePnfGVOu76uuAGaEW6AZoQboBnhBmhGuAGaEW6A\nZoQboBnhBmhGuAGaEW6AZoQboJlp/pf3te09csieD5Mfo/++Cknm2QdmrrWdY2+QfWxg7403ZAuP\nf6Jz1xU3QDPCDdCMcAM0I9wAzQg3QDPCDdCMcAM0I9wAzQg3QDPCDdCMcAM0M81eJWvbe2SO/TT2\nfgz73fzw48ywVlsx9WOffI+WOaz4/FjjuT5RF1xxAzQj3ADNCDdAM8IN0IxwAzQj3ADNCDdAM8IN\n0IxwAzQj3ADNCDdAM9PsVbK2/TQO2fNh8mPMtOfD2p6LQ++zj0O//Rr3upjaXHvsJNPvvbLGLkzE\nFTdAM8IN0IxwAzQj3ADNCDdAM8IN0IxwAzQj3ADNCDdAM8IN0IxwAzQj3ADNTLPJ1No2jTpkY5jJ\nH8N+N//f/WzUM5k1nof72nemtW6Odog1Ph8TccUN0IxwAzQj3ADNCDdAM8IN0IxwAzQj3ADNCDdA\nM8IN0IxwAzQj3ADNTLNXydr2fJhjP4259nxY2/4Kc+wDs/8BDrzbys7DOZ7rufbYOehYK3s+DjnG\nRFxxAzQj3ADNCDdAM8IN0IxwAzQj3ADNCDdAM8IN0IxwAzQj3ADNCDdAMzXGOn73HoDTccUN0Ixw\nAzQj3ADNCDdAM8IN0IxwAzQj3ADNCDdAM8IN0IxwAzQj3ADNCDdAM8IN0IxwAzQj3ADNCDdAM8IN\n0IxwAzQj3ADNCDdAM8IN0IxwAzQj3ADN/AcUR0qlQqi4uAAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUQAAAEWCAYAAAAerO46AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGxlJREFUeJzt3X+8VXWd7/HXWxTxZ9Lgo1BQsaxEazCPaDlTZv4g80aP\nqQymHzhXL3fuqKP9mtGpa47lnZrurbGrlVTkj0wtcIoxyqGE6yN/IGiEgGloGqCjIoKIP4HP/WN9\nz7g4nX3OXmfvtffZa7+fj8d6nPVrr+9nHeDDZ63vWt+tiMDMzGCndgdgZjZcOCGamSVOiGZmiROi\nmVnihGhmljghmpklTohWOklXSvpiu+MAkDRR0lJJSssrJR1XY99Fks6ssW2upPeUGKq1gRNiG/T3\nD01SSHp9u2LqIl8A/nekB3Aj4rCIWDSE43wZGBZJ3prHCbECJO3c7hhqkTSi3TH0kjQWeBfw40aP\nFRF3AXtL6mk4MBs2nBCHSNLDki6QtErS05K+J2lU2jZa0k2SnkzbbpI0Lm27BPhz4DJJz0q6TNKt\n6bC/Ses+nPY9VdIySRsl3S7pLX3a/3tJy4EtknZO6z4tabmkTZJu6I2pn/h3kvQ5SY9IekLS1ZJe\nlbb9TNLZffb/jaS/SPNvkrRA0gZJ90s6LbfflZK+KWm+pC1kCSh/nJq/m7R9kaR/knSXpGck/UTS\nq2ucw3GS1kr6u3QOj0l6v6RTJD2Q4vuH3EdOBO6JiBf6/B5P6PcPece2xqbf62dyqxcB7x3ss9ZB\nIsLTECbgYWAFMB54NXAb8MW07U+ADwC7A3sBPwJ+nPvsIuDMPscL4PW55SOAJ4CjgRHAjNTmrrn2\nl6X2d8utuwvYL8V0H/DXNeL/r8Bq4GBgT+BG4Jq07ePAbbl9JwIbgV2BPYA1wF8BO6c41wMT075X\nApuAY8n+wx2V1hX53awDDk9tzQW+X+McjgO2AhcCuwD/DXgS+EE69mHA88CEtP9XgMv7+XM8ocbx\nFwFnAhOAB4CZfbZ/Erix3X8XPTVvcoXYmMsiYk1EbAAuAaYDRMRTETE3Ip6LiM1p2zsLHnsmcEVE\nLI6IbRFxFfAicExun6+n9p/vs+7RFNO/AZNqHP8jwFcj4qGIeBa4AJiWLr//FZgk6cDcvjdGxIvA\nqcDDEfG9iNgaEb8mS1ofyh37JxFxW0Rsj1w1VuB3c01ErIiILcD/BE4b4NL7ZeCSiHgZuB4YA1wa\nEZsjYiWwCvjTtO8+wOYax6llIrAQ+HxEzOqzbXM6plWEE2Jj1uTmHyGrzJC0u6Qr0uXoM8CtwD4F\n76cdCHwqXS5vlLSRrBrcr0b7vf4jN/8cWfXXn/1SzPn4dwZekxLVT4Fpadt04NpcXEf3iesjwGsH\niQuo+3fT9/e6C1mi689TEbEtzff+x/B4bvvzvPI7eJqsciziI2QV65x+tu1FVjlbRTghNmZ8bv4A\n4NE0/yngjcDREbE38I60XulnPUMMrSGrfPbJTbtHxHW5fRoZquhRsuTW6wCyy8/eZHIdMF3S28gu\nexfm4vp/feLaMyL+R51xDfa7gT/+vb5MdlneqOXAGwp+5qLU9g/6+Q/tUOA3TYjLhgknxMacJWlc\nuun/WeCGtH4vsspkY9r2+T6fe5zs3t1A674N/LWko5XZQ9J7JRWtcGq5DviEpAmS9gT+F3BDRGxN\n2+eTJcyL0/rtaf1NwBskfUzSLmk6StKhdbY72O8G4KPKnhfcPbU/J1cFNmIB8NYBOpqOk9Q3mb9M\ndjtgD+BqSfl/M+8EftaEuGyYcEJszA+AfwceAh7klefS/gXYjayyuBP4eZ/PXQp8MPWyfj2tuwi4\nKl2GnhYRS8k6CS4ju9RbDZzexNhnA9eQXbL+HngBOKd3Y7pfeCNwQjrP3vWbgZPILqcfJbtE/zJZ\nh0s9BvvdkOK6Mh17FPC3dZ/VACLiceAWYGqNXcYDt/fzuZeAvwBeA8xOPfRHAc9G9viNVYQiPEDs\nUEh6mKyn+BftjqVKJC0i61X+TknHnwhcBUyOPn/5JX0H+FFE3FzHceYC342I+WXEae0xbB/oNStD\nRKwCjqqxrd/X9Grs+4GmBWXDhi+ZzaxtJM1OD9WvqLFdkr4uaXV6MP6tuW0zJP0uTTOaEo8vmc2s\nXSS9A3gWuDoiDu9n+ylk97ZPIXtJ4dKIODp1yC0FesieargbODIinm4kHleIZtY2EXErsGGAXaaS\nJcuIiDvJnlkdC5wMLIiIDSkJLgCmNBrPsLqHOEaKg9odRAmen3hku0MozZYt7Y6gHGNGPtPuEErx\n8OOPs37TJg2+Z21TpKj3odC7YSXZEwy9ZvXzxs9A9mfHB/XXpnW11jdkWCXEg8hq4KpZcUMVzyqz\neHG7IyjHGeMG7WjuSD3nnDP4ToNYT/3/TgUvRETHjAjkS2YzK26nneqbGreOHd9cGpfW1VrfECdE\nMyuudQlxHvDx1Nt8DLApIh4DbgZOSsPJjSZ7WaDhsn5YXTKbWQeQmpXskHQd2TBuYyStJXuVcxeA\niPgW2Sukp5C9qfUc2bBzRMQGSV8AlqRDXZxGeGqIE6KZtU1ETB9kewBn1dg2m+wV1KZxQjSz4ppU\nIQ43TohmVkwTL5mHm2qelZnZELhCNLPidq5m6nCFaGaWVDPNm1l5KnwP0QnRzIpzQjQzwxWimdkO\nnBDNzJKKJsRqnpWZ2RC4QjSzYnwP0cwsp6IJsZpnZWY2BK4QzawYqbKv7lXzrMysXL5kNjOrNleI\nZlZMhXuZSz0rSVMk3S9ptaTzy2zLzFqodV8y1VKlVYiSRgCXAyeSfYn0EknzImJVWW2aWQu4QhyS\nycDqiHgoIl4CrgemltiembVKRSvEMiPeH1iTW16b1pmZDUtt71SRNBOYCXBAm2Mxszp1YPVXjzIT\n4jpgfG55XFq3g4iYBcwC6JGixHjMrBl8D3FIlgCHSJogaSQwDZhXYntmZg0prUKMiK2SzgZuBkYA\nsyNiZVntmVmL+NW9oYmI+cD8MtswszbwJbOZWbVVs+41s3JVtEJ0QjSzYtzLbGbWfIONdyDpa5KW\npekBSRtz27bltjXlCRZXiGZWTJN6mesZ7yAiPpHb/xzgiNwhno+ISQ0HkuMK0czapeh4B9OB68oM\nyAnRzIqrf3CHMZKW5qaZuaPUPd6BpAOBCcAtudWj0jHvlPT+ZpyWL5nNrLj6O1XWR0RPE1qcBsyJ\niG25dQdGxDpJBwO3SLo3Ih5spBFXiGbWLnWNd5BMo8/lckSsSz8fAhax4/3FIXFCNLNiejtV6pkG\nVtd4B5LeBIwG7sitGy1p1zQ/BjgWaHjwaV8ym1kxTXoOsdZ4B5IuBpZGRG9ynAZcHxH50bAOBa6Q\ntJ2ssPtSM0bjd0I0s7bpb7yDiLiwz/JF/XzuduDNzY7HCdHMiqvomypOiGZWXEUTYjXPysxsCFwh\nmlkxFR4g1hWimVlSzTRvZuWp8PBfTohmVpwTopkZla4Qq3lWZmZDMKwqxOcnHsmKG5a2O4ymO/zN\nancIpTl8+/Z2h1CKO+48ud0hlGLLiL2bcyD3MpuZVVs107yZlafC9xCdEM2suIomxGqelZnZELhC\nNLNi/OqemVn1VTPNm1m5KnoP0QnRzIpxL7OZWU5FE2I1z8rMbAhcIZpZMe5lNjOrvmqmeTMrV0Xv\nITohmlkxFe5lruZZmZkNgStEMyuuohWiE6KZFVPhS2YnRDMrzgnRzCxxQjQzo9KXzNU8KzPrCJKm\nSLpf0mpJ5/ez/XRJT0palqYzc9tmSPpdmmY0Ix5XiGZWTJNe3ZM0ArgcOBFYCyyRNC8iVvXZ9YaI\nOLvPZ18NfB7oAQK4O3326UZicoVoZu0yGVgdEQ9FxEvA9cDUOj97MrAgIjakJLgAmNJoQE6IZlbc\nTjvVNw1sf2BNbnltWtfXByQtlzRH0viCny2ktIQoabakJyStKKsNM2uT+hPiGElLc9PMgi39G3BQ\nRLyFrAq8qtmnkldmhXglTShhzayjrY+Intw0K7dtHTA+tzwurftPEfFURLyYFr8DHFnvZ4eitIQY\nEbcCG8o6vpm1Se9jN41fMi8BDpE0QdJIYBowb8emNDa3+D7gvjR/M3CSpNGSRgMnpXUNaXsvcyqh\nZwKMHXtAm6Mxs0E1qZc5IrZKOpsskY0AZkfESkkXA0sjYh7wt5LeB2wlK7BOT5/dIOkLZEkV4OKI\naLgAa3tCTCX0LIDDDuuJNodjZi0UEfOB+X3WXZibvwC4oMZnZwOzmxlP2xOimXWgir6p4oRoZsVV\nNCGW+djNdcAdwBslrZV0RlltmZk1Q2kVYkRML+vYZtZG/tY9M7Pqq2aaN7PyVHj4r5oJUdIPI+I0\nSfeSjSbxn5uASK/SmFk36raECJybfp7aikDMzNqtZkKMiMfSz0daF46ZdYQurBDNzP6Ye5nNzKqv\nmmnezMrTjb3MvSQdC1wEHJj27+1lPrjc0Mxs2OrWhAh8F/gEcDewrdxwzMzap56EuCkiflZ6JGbW\nObq4Qlwo6SvAjUDvUN5ExD2lRWVmw1c330MEjk4/e3LrAji++eGYWUfoxoQoaSfgmxHxwxbFY2bD\nXYUrxAHPKiK2A3/XoljMzNqqnkvmX0j6NHADsKV3ZTO+0MXMOlRFK8R6EuKH08+zcusC8HOIZt2o\nwq/uDXpWETGhFYGYmbXboHWvpN0lfU7SrLR8iCQPCWbWzZrzRfXDTj0Rfw94CXh7Wl4HfLG0iMxs\neOvtZe7ShPi6iPhn4GWAiHiO7H1mM7NKqefO6EuSdiN9jYCk15F7Y6WZtmyBxYvLOHJ7Hb59e7tD\nKE8HVgF1uT0G36ebVfTPvZ6EeBHwc2C8pGuBY4G/KjMoMxvGKvxgdj29zP8u6W7gGLJL5XMjYn3p\nkZnZ8NWtCVHSLyPi3cBP+1lnZt2o2xKipFHA7sAYSaN5pSNlb2D/FsRmZsNRl14y/3fgPGA/ssFh\nexPiM8BlJcdlZtZyNdN8RFya3lL5TEQcHBET0vSnwLdbF6KZDTtNeg5R0hRJ90taLen8frZ/UtIq\nScsl/VLSgblt2yQtS9O8ppxWHfuc3s+6O5rRuJl1oCY9mC1pBHA58B5gIjBd0sQ+u/0a6ImItwBz\ngH/ObXs+Iial6X3NOLWB7iG+luxe4W6SjmDHe4i7N6NxM+tQzbmHOBlYHREPAUi6HpgKrOrdISIW\n5va/E/hoMxquZaB7iCeTVYfjgK/m1m8G/qHEmMxsuKs/IY6RtDS3PCsiZqX5/YE1uW1reWWE/v6c\nAeS/32lUOvZW4EsR8eN6g6qlZkKMiKuAqyR9ICLmNtqQmXWl9RHRM/huA5P0UbKvMXlnbvWBEbFO\n0sHALZLujYgHG2mnngez50p6L3AYMCq3/uJGGjazDtW8x27WAeNzy+PSuj7N6QTgs8A7IyL/RXfr\n0s+HJC0CjgAaSoj1DP/1LbJBYs8hu4/4IbIvrTezbtWcXuYlwCGSJkgaCUwDdugtTv0XVwDvi4gn\ncutHS9o1zY8he6V4FQ2q513mt0fEWyQtj4h/lPR/2PE63sy6SZMqxIjYKuls4GZgBDA7IlZKuhhY\nGhHzgK8AewI/kgTwh9SjfChwhaTtZIXdlyKiJQnx+fTzOUn7AU8BYxtt2Mw6WJPeVImI+cD8Pusu\nzM2fUONztwNvbkoQOfUkxJsk7UOWqe8hGwbMD2abdbMufHUPgIj4QpqdK+kmYFREbCo3LDOz1qtn\ntJtRwN8Af0ZWHf5K0jcj4oWygzOzYahLB3fodTXZw9j/Ny3/JXANWW+zmXWjLk6Ih0dE/v3ChZIa\n7s0xsw5V4QqxnrO6R9IxvQuSjgaWDrC/mVVdRb91r54K8Ujgdkl/SMsHAPdLuheINAqFmVnHqych\nTik9CjPrLB1Y/dWjnsduHmlFIGbWIbr8HuKQSBovaWEa7XalpHPLasvMWqyL7yEO1VbgUxFxj6S9\ngLslLWjG+4Zm1kYVrhBLS4gR8RjwWJrfLOk+sgEhnRDNOl1FE2JLzkrSQWRjlS3uZ9tMSUslLd28\n+clWhGNm1q/SE6KkPYG5wHkR8Uzf7RExKyJ6IqJnr732LTscM2sG30MsTtIuZMnw2oi4scy2zKxF\nfA+xOGWjOX4XuC8ivjrY/mbWQSqaEMs8q2OBjwHH575M+pQS2zOzVmjS9zIPR2X2Mv+KV77L2cxs\n2Cv1HqKZVVQHVn/1cEI0s+KcEM3McC+zmdkOnBDNzKh0hVjNszIzGwJXiGZWXEUrRCdEMyumwpfM\nTohmVpwToplZUtGEWM2zMjMbAleIZlZMhe8hVvOszKxcTRrtRtIUSfdLWi3p/H627yrphrR9cRp9\nv3fbBWn9/ZJObsppNeMgZtZFmjT8l6QRwOXAe4CJwHRJE/vsdgbwdES8Hvga8OX02YnANOAwsu+O\n/0Y6XkOcEM2suOZUiJOB1RHxUES8BFwPTO2zz1TgqjQ/B3h3Gnx6KnB9RLwYEb8HVqfjNXZajR7A\nzLpPoLomYEzvl8ilaWbuMPsDa3LLa9M6+tsnIrYCm4A/qfOzhblTxczKtD4ietodRL2cEM2ssO3b\nm3KYdcD43PK4tK6/fdZK2hl4FfBUnZ8tzJfMZlZIRJYQ65kGsQQ4RNIESSPJOknm9dlnHjAjzX8Q\nuCUiIq2flnqhJwCHAHc1em6uEM2ssGZUiBGxVdLZwM3ACGB2RKyUdDGwNCLmkX1z5zWSVgMbyJIm\nab8fAquArcBZEbGt0ZicEM2skN4KsTnHivnA/D7rLszNvwB8qMZnLwEuaU4kmWGVEMeMfIYzxt3c\n7jCa7o47m/LM6PB0e7Q7glK87e3V/MLIPdodwDA3rBKimXWGZlWIw40TopkV5oRoZkZz7yEON06I\nZlaYE6KZGdWuEP1gtplZ4grRzAqraoXohGhmhVT5ktkJ0cwKc0I0M0ucEM3MqPYls3uZzcwSV4hm\nVlhVK0QnRDMrpMqXzE6IZlaYE6KZWVLVhOhOFTOzxBWimRXie4hmZjlOiGZmuEI0M9uBE6KZWVLV\nhOheZjOzpLQKUdIo4FZg19TOnIj4fFntmVlr+B7i0LwIHB8Rz0raBfiVpJ9FxJ0ltmlmLeCEWFBE\nBPBsWtwlTVFWe2bWGlWuEEu9hyhphKRlwBPAgohYXGZ7ZtYa27fXN3WaUhNiRGyLiEnAOGCypMP7\n7iNppqSlkpY+uWlTmeGYmQ2oJb3MEbERWAhM6WfbrIjoiYiefV/1qlaEY2YNcoVYkKR9Je2T5ncD\nTgR+W1Z7ZtYavfcQy06Ikl4taYGk36Wfo/vZZ5KkOyStlLRc0odz266U9HtJy9I0abA2y6wQxwIL\nJS0HlpDdQ7ypxPbMrFrOB34ZEYcAv0zLfT0HfDwiDiO7Av2X3kIs+UxETErTssEaLLOXeTlwRFnH\nN7P2iICtW1vS1FTguDR/FbAI+PsdY4kHcvOPSnoC2BfYOJQG/aaKmRVW4JJ5TG+naZpmFmjmNRHx\nWJr/D+A1A+0saTIwEngwt/qSdCn9NUm7Dtag32U2szKtj4ieWhsl/QJ4bT+bPptfiIiQVPM5Zklj\ngWuAGRHRe/fyArJEOhKYRVZdXjxQsE6IZlZYs3qQI+KEWtskPS5pbEQ8lhLeEzX22xv4KfDZ/Jtw\nueryRUnfAz49WDy+ZDazQlrVywzMA2ak+RnAT/ruIGkk8K/A1RExp8+2semngPcDKwZr0AnRzApr\nUUL8EnCipN8BJ6RlJPVI+k7a5zTgHcDp/Txec62ke4F7gTHAFwdr0JfMZlZIq95ljoingHf3s34p\ncGaa/z7w/RqfP75om64QzcwSV4hmVlgnvpZXDydEMyvMCdHMjGqPh+iEaGaFOSGamVHtCtG9zGZm\niStEMyusqhWiE6KZFVbVhOhLZjOzxBWimRXSwgFiW84VoplZ4grRzAqp8mM3TohmVpgTopkZrhDN\nzHZQ1YSoiJrf29Jykp4EHmlRc2OA9S1qq5V8Xp2nled2YETs28gBJP2cLOZ6rI+IKY2010rDKiG2\nkqSlA30bWKfyeXWeKp9bp/FjN2ZmiROimVnSzQlxVrsDKInPq/NU+dw6StfeQzQz66ubK0Qzsx04\nIZqZJV2XECVNkXS/pNWSzm93PM0iabakJyStaHcszSRpvKSFklZJWinp3HbH1AySRkm6S9Jv0nn9\nY7tjsi67hyhpBPAAcCKwFlgCTI+IVW0NrAkkvQN4Frg6Ig5vdzzNImksMDYi7pG0F3A38P5O/zOT\nJGCPiHhW0i7Ar4BzI+LONofW1bqtQpwMrI6IhyLiJeB6YGqbY2qKiLgV2NDuOJotIh6LiHvS/Gbg\nPmD/9kbVuMg8mxZ3SVP3VCfDVLclxP2BNbnltVTgH1e3kHQQcASwuL2RNIekEZKWAU8ACyKiEufV\nybotIVqHkrQnMBc4LyKeaXc8zRAR2yJiEjAOmCypMrc6OlW3JcR1wPjc8ri0zoaxdI9tLnBtRNzY\n7niaLSI2AguBjhkEoaq6LSEuAQ6RNEHSSGAaMK/NMdkAUufDd4H7IuKr7Y6nWSTtK2mfNL8bWUff\nb9sblXVVQoyIrcDZwM1kN+d/GBEr2xtVc0i6DrgDeKOktZLOaHdMTXIs8DHgeEnL0nRKu4NqgrHA\nQknLyf6jXhARN7U5pq7XVY/dmJkNpKsqRDOzgTghmpklTohmZokToplZ4oRoZpY4IdofkXS6pP1y\ny+dJ2r2kto6T5MdNbFhwQrT+nA7sl1s+DyiUENPIQmYdxQmx4iQdJOm3kq6VdJ+kOb3VnqQLJS2R\ntELSLGU+CPQA16aHoM8lS44LJS1MnztJ0h2S7pH0o/SeMZIelvRlSfcAH5K0KC3fJekBSX8+SKxH\nSfq1pNeV+ksxq8EJsTu8EfhGRBwKPAP8TVp/WUQclcZP3A04NSLmAEuBj0TEpIi4FHgUeFdEvEvS\nGOBzwAkR8da07ydzbT0VEW+NiOvT8s4RMZmsyvx8rQAlvR34FjA1Ih5s1ombFeGE2B3WRMRtaf77\nwJ+l+XdJWizpXuB44LA6jnUMMBG4LQ1dNQM4MLf9hj779w7GcDdwUI1jHkr2zXP/JSL+UEcMZqXY\nud0BWEv0fT8zJI0CvgH0RMQaSRcBo+o4lsjeu51eY/uWPssvpp/bqP337bHU9hFk1ahZW7hC7A4H\nSHpbmv9LsuHqe5Pf+nQP8IO5/TcDe9VYvhM4VtLrASTtIekNDca3EXgv8E+SjmvwWGZD5oTYHe4H\nzpJ0HzAa+GYag+/bwAqy0X+W5Pa/EvhW6lTZjexy9ueSFkbEk2S90NelkVruAN7UaIAR8ThwKnC5\npKMbPZ7ZUHi0m4pLw+7fVKUvnjIriytEM7PEFaKZWeIK0cwscUI0M0ucEM3MEidEM7PECdHMLPn/\nJhmx7Pbxwn8AAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEVCAYAAADpbDJPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGcxJREFUeJzt3Xm4HXWd5/H3pxMWTUBEIoaQEFYx2A9grki3QCPqNKgY\nBemG2IgLxtahtR2XxnYGG9pxEJ9He7rbthsFARfAUZaAoRlFAsqiSVgT1siAJARCwGyAQOA7f9Qv\nUh7Ocu+tume5v8/reerJqapfVf2+t07O59Zy6ioiMDOz/PxRrztgZma94QAwM8uUA8DMLFMOADOz\nTDkAzMwy5QAwM8uUA8CyI2mhpBNHuew5kr5Yd5+qkHSwpLt73Q8bPA4Aq4Wk+yWtljSpNO1ESQuH\nuXzffbAOioj4eUS8utf9sMHjALA6TQA+0etOtKKC3/Nmif8zWJ2+Anxa0nbNZkraW9JPJD0u6W5J\nf5GmzwPeC3xW0kZJl0n6gKTLSsveK+n/lMYflLRfev2nkhZJWpf+/dNSu4WS/qek64Angd0a+jRV\n0m2SPtOiz/tLuknSBkkXAluX5i2VdGRpfAtJa9IyMyWFpBMk/SZN/3yp7QGSbpC0VtIqSf8qacvS\n/JD0sVT3Bkn/KGl3SddLWi/pB5vbSzpU0orSstMlXSTpUUmPSfrXNH0PSdekn9OaVI/lLCI8eKg8\nAPcDbwEuAr6Ypp0ILEyvJwEPAh8AJgL7A2uAWWn+OZuXS+O7AWspfknZCXgAWFGa99s0b/v0+vi0\n3uPS+CtS24XAb4B90vwt0rQTgV2Be4B5LWraMm33k2m59wDPlur7LHBhqf0c4Pb0eiYQwDeBlwD7\nAk8Dr0nzZwMHpj7NBO4E/ra0rgAuBbZNfX8auCrV/jLgDuCE1PbQ0s9mAnAr8LX0M98aOCjNOx/4\nfPq5/X66h3wHHwFY3U4B/kbSlIbp7wDuj4hvR8SmiLgZ+BFwTLOVRMR9wAZgP+AQ4ErgIUl7A38G\n/DwingfeDtwbEd9J6z0fuAs4srS6cyJiWZr/bJo2C7ga+EJEnNmilgMpPvj/KSKejYgfAotK878L\nvE3Stmn8eOA7Des4NSKeiohbKT6Y9031LYmIG1Of7gf+I9VVdkZErI+IZcBS4P9GxH0RsQ64giJE\nGx1AEZifiYgnIuJ3EfGLNO9ZYBdgp4bplikHgNUqIpYClwMnN8zaBXhDOuWxVtJaitM+r2qzumso\nfrs9JL1eSPEh+WdpHF44Oih7AJhWGn+wybrfC6wEfthm+zsBKyOi/MTE328rIh4CrgOOTqe9jgC+\n17COh0uvnwQmA0jaS9Llkh6WtB74ErBDw7KPlF4/1WR8cpM+TwceiIhNTeZ9FhDwK0nLJH2wSRvL\niAPAxsIXgA/z4g/hayJiu9IwOSI+muY3eyzt5gA4OL2+hhcHwEMU4VI2g+LDfbNm6/4HilNQ35c0\noUUdq4BpktSw7rJzgb+iOJK5ISJWMjzfoDhS2TMitgX+nuLDuaoHgRmSJjbOiIiHI+LDEbET8BHg\n3yTtUcM2bUA5AKx2EbEcuBD4eGny5cBeko5PF0u3kPR6Sa9J8x+h4QItxYf8m4CXRMQK4OfA4cAr\ngJtTmwVpvXMlTZT0lxSndy7v0M1nKT60JwHntbg76AZgE/Dx1N+jKE6xlF0CvI7i7qfzOmyzbBtg\nPbAxndb6aIf2w/UriuA6XdIkSVtLeiOApGMk7Zza/ZYiGJ+vabs2gBwANlZOo/hwBSAiNgD/BTiW\n4rf2h4EvA1ulJmcBs9LpoUvSMvcAGyk++ImI9cB9wHUR8Vya9hjF9YVPAY9RnOZ4R0Ss6dTBiHgG\nOArYETi7MQRK898PPA78JcVF7nKbpyiuZezaOK+DTwNzKa5zfJMiMCtLP5cjgT0oLn6voOg3wOuB\nX0raCMwHPpGutVim9IenN81spCSdAuwVEX/V676YjcSLzhOa2fBJ2h74EMUdQGYDxaeAzEZJ0ocp\nLrpeERHX9ro/ZiPlU0BmZpnyEYCZWaYcAGZmmXIAmJllygFgZpYpB4CZWaYcAGZmmXIAmJllygFg\nZpYpB4CZWaYcAGZmmXIAmJllygFgZpYpB4CZWaYcAGZmmXIAmJllygFgZpYpB4CZWaYcAGZmmXIA\nmJllygFgZpYpB4CZWaYcAGZmmXIAmJllygFgZpYpB4CZWaYcAGZmmcoiACQdJOl6SeskPS7pOkmv\nT/PmSnpA0hOSLpG0fa/7OxqtapQ0VdJ8SQ9JCkkze93X0WpT49sl/ULSWkkPS/qWpG163d/RalPn\nmyTdnup8TNLFkqb1ur+j0e7/ZKnN2ek9u0ev+llVm315qKTnJW0sDSd0u3/jPgAkbQtcDvwLsD0w\nDTgVeFrSPsB/AMcDOwJPAv/Wo66OWrsageeB/wSO7lkHa9ChxpcBXwR2Al6T5n2lNz2tpkOddwB/\nHhHbUdR6L/CNHnV11DrUuLnNQcDuPelgTYZR50MRMbk0nNv1PkZEt7fZVZKGgJ+m/zSN874EzIyI\nuWl8d+BO4BURsaG7PR29djWW2kwEngV2jYj7u9W3ugynxlLbo4BTI+KPx75n9RpunZK2Av4BmBMR\ns7rRt7p0qjG9VxcBJwC3AntGxPIudrEWHT57DgW+GxE7d71jJeP+CAC4B3hO0rmSjpD08tK8fSje\nYABExK+BZ4C9utzHqtrVOF6MpMZDgGVd6lfd2tYpaYaktcBTwKeBM3rRyYo67ctPAtdGxG096Fud\nOtX5SkmPSPp/kr4maVK3OzjuAyAi1gMHAQF8E3g0nRPfEZgMrGtYZB0wUOePO9Q4Lgy3RklvpfjN\n8ZTu97K6TnVGxG/Sb5Q7AP8duKtnnR2ldjVKmg58hAHdf2Ud9uVdwH7AVOAwYDbw1W73cdyfAmok\naW/guxTnT18KXBcRZ5TmbwAOjYglPepiZeUaI+K4NG2gTwE1alHjgcBlwLERcVUv+1eXZnWW5r2K\n4gh2WkRs6kX/6tDwf3JL4NKIOC/NCwb0FFCjDvvyQODyiNihm30a90cAjSLiLuAc4LUUpwn23TxP\n0m7AVhSHbgOrocZxqbFGSfsD84EPjpcPf+i4LycCrwS27Waf6tZQ45uBr6S7uR5OTW6QNLdX/atL\nh30Z9ODzeNwHgKS9JX1K0s5pfDpwHHAj8D3gSEkHp/NvpwEXDdIFYOhYI5K2pgg2gK3S+EBpV6Ok\n11Lc6fQ3EXFZL/tZVYc6j5L0akl/JGkKxSmDmyPi8V72eaQ6vF/3ovilbL80ABwJXNyLvlbRYV++\nSdIuKkwHTgcu7XYfx30AABuANwC/lPQExZtsKfCpiFgG/DVFEKymOPf/sV51tIKWNab5TwEb0+u7\n0vigaVfjp4ApwFl64Z7qQb0I3K7OaRRBtwG4neIW33f3qJ9VtPs/uToiHt48pPZrImK8vWf3B64H\nnkj/3g58vNsdzO4agJmZFXI4AjAzsyYcAGZmmXIAmJllygFgZpapib3uQCtCba9OL9br2i4/FDd1\n2ECHi9+hSstH0GEFaTWdVjTG/VzM7Lbzh2j/fbjh1NnrfdmNGsH7EsbHvsxhP27mIwAzs0w5AMzM\nMuUAMDPLlAPAzCxTDgAzs0w5AMzMMuUAMDPLVN9+D6DTvbJDFe+1rXqvbsflGeZD9qreE1yxn1WX\nH1adnbbRocaq+6IrNYL3JTAu9mUO+zHxEYCZWaYcAGZmmXIAmJllygFgZpYpB4CZWaYcAGZmmXIA\nmJllqn+/B9DhXtfOzx0f42dyd7xXt/3sF9qN8T3BY758+9nD2cZY15jLvuxKna5xfNSY+AjAzCxT\nDgAzs0w5AMzMMuUAMDPLlAPAzCxTDgAzs0w5AMzMMqWIEdw02kVSh7tZe37/e/vlo/NDu4vNZFBn\nDjVCHnW6xs7bGIQaN/MRgJlZphwAZmaZcgCYmWXKAWBmlikHgJlZphwAZmaZcgCYmWVqYP8eQNV7\nZcf67wnU9Qz5sb6nuB+er97r+6a9Lzdvv/3sok2v90Xv/37FuNiPiY8AzMwy5QAwM8uUA8DMLFMO\nADOzTDkAzMwy5QAwM8uUA8DMLFP+ewAtdP6ewE3tV+9nyL+wiQxqhDzqdI2dtzEINW7mIwAzs0w5\nAMzMMuUAMDPLlAPAzCxTDgAzs0w5AMzMMuUAMDPLVN9+D8DMzMaWjwDMzDLlADAzy5QDwMwsUw4A\nM7NMOQDMzDLlADAzy5QDwMwsUw4AM7NMOQDMzDLlADAzy5QDwMwsUw4AM7NMOQDMzDLlADAzy5QD\nwMwsUw4AM7NMOQDMzDLlADAzy5QDwMwsUw4AM7NMOQDMzDLlADAzy5QDwMwsUw4AM7NMOQDMzDLl\nADAzy5QDwMwsUw4AM7NMOQDMzDLlADAzy5QDAJA0V9IDkp6QdImk7Vu0O1jSxoYhJB2d5p8gaYmk\n9ZJWSDpD0sTuVtPacOtMbQ+TdFOq5T5J80rz3iTpdklrJT0m6WJJ07pTRXs17ktJ+qKklZLWSVoo\naZ/uVtNcDjVCHnX2vMaIyHoA9gE2AIcAk4HvAxcMc9lD07KT0vhHgYOBLYFpwBLg5F7XONI6gS2A\ndcBHAAGvBzYC+6b5OwI7pddbAWcA8wepxmHsy78AHgJ2AyYA/wu4yTW6zvFUY893dENR9wOfAW4D\nngDOSh82V6Rifwq8vOZtfgn4fml8d+AZYJthLPtt4Ntt5v834LJBqzP1JYCXlqYtAo5r0nar9Ga7\nY5Bq7LQvgb8DflAa3wf4XW415lJnDjU2XU+dBdXwA7kfuDH94KcBq4GbgP2BrYGfAV9osewMYG2b\nYW6L5S4F/q5h2kZgdoe+TkpvjEPbtLkEOH0Q66T4beS/Uvw28Sepj9Ob9ON54Fng/YNWY7t9CexC\ncQS3F8UR0RnAJbnVmEudOdTYbOib89Ml/xIRjwBI+jmwOiJuTuMXA29utlBE/AbYbhTbm0xxuqNs\nHbBNh+WOAtYA1zSbKemDwBBwYovl+73O84FvAf87jX80Ih5s7Ec6Z/lh4K4m6+j3Gjdrti9XAb8A\n7gaeAx4EDmuybA41Qh515lDjH+jHi8CPlF4/1WR8cs3b2whs2zBtW4qEbecE4LxI8Vsm6V0Up0WO\niIg1LZbv2zol7Q1cALyP4nrGPsBnJb29sW1EPA6cC1za5IJ339bYoNm+PIXi2sd0it8ATwV+Juml\nDcvmUCPkUWcONf6BfgyAUZE0o8lV8vLw3haLLgP2La1nN4rz2ve02dZ0iosw5zWZdzjwTeDIiLi9\nQkmttt2NOl8L3BMRV0bE8xFxN/Bj4IgW654IvJIXv5lHpU/25X7AhRGxIiI2RcQ5wMuBWaOv7A+2\nO+5rTNse93UOco39eApoVNJh2GgS+nvADZIOpjjndxpwUUS0S+Hjgesj4tfliZIOS+t7d0T8ahR9\n6ahLdd4M7JnquZrizoJ3UJxXRNJRFG/ee4FXAF8Fbk5HA5X1w76kuOh9jKQLgEeB91KcW10+in69\nSA41Qh51DnKN4+YIYLQiYhnw1xQ7YzXF+bePbZ4v6QpJf9+w2PsoTns0+h/Ay4AFpfS/Ymx6PjIj\nqTO9uT4I/DOwnuJc448orglAcZHsPykOVW+nuBD87q4U0kbN+/LLwK3ALRQX8j4JHB0Ra8eg68OW\nQ42QR539UKOanMI2M7MMZH8EYGaWq1oCQNLZklZLWtpiviT9s6Tlkm6T9Lo6tmtmZqNX1xHAOcDh\nbeYfAeyZhnnAN2rarpmZjVItARAR1wLt7gCZQ7pvNSJupPgC0dQ6tm1mZqPTrdtAp1F8M22zFWna\nqnIjFU+cnAcwadKk2XvvvXeXule/JUuWrImIKc3m5VBnDjXC+KkzhxrB79cX6fSsiOEOwExgaYt5\nlwMHlcavAobarW/27NkxyIDFMYyfWw515lBjDHidOdQY4fdr49Ctu4BWUnxFebOd0zQzM+uRbgXA\nfOB96W6gA4F1EbGq00JmZjZ2arkGIOl8iudT7CBpBfAFiq8hExH/DiwA3kbxteQngQ/UsV0zMxu9\nWgIgIo7rMD8oni1vZmZ9wt8ENjPLlAPAzCxT4+Zx0INu5sk/7tjm/tNf9PdYzMxGzUcAZmaZcgCY\nmWXKp4Csa4ZzmgsG/1RXDqfzcqgRxn+dPgIwM8uUA8DMLFMDcQpovB+GmZn1go8AzMwy5QAwM8uU\nA8DMLFMOADOzTDkAzMwy5QAwM8tULQEg6XBJd0taLunkJvPfL+lRSbek4cQ6tmtmZqNX+XsAkiYA\nXwfeCqwAFkmaHxF3NDS9MCJOqro9MzOrRx1HAAcAyyPivoh4BrgAmFPDes3MbAzVEQDTgAdL4yvS\ntEZHS7pN0g8lTa9hu2ZmVkG3HgVxGXB+RDwt6SPAucBhjY0kzQPmAcyYMaNLXeu+HOrs9xrrerxI\nP9eZQ41QT5051NhMHUcAK4Hyb/Q7p2m/FxGPRcTTafRbwOxmK4qIMyNiKCKGpkyZUkPX+lMOdeZQ\nI+RRp2scv+o4AlgE7ClpV4oP/mOBueUGkqZGxKo0+k7gzhq2a2PAD94zy0flAIiITZJOAq4EJgBn\nR8QySacBiyNiPvBxSe8ENgGPA++vul0zM6umlmsAEbEAWNAw7ZTS688Bn6tjW2ZmVg9/E9jMLFMO\nADOzTDkAzMwy5QAwM8uUA8DMLFMOADOzTDkAzMwy5QAwM8tUtx4GNy74MQlmNp74CMDMLFMOADOz\nTDkAzMwy5QAwM8uUA8DMLFMOADOzTDkAzMwyVUsASDpc0t2Slks6ucn8rSRdmOb/UtLMOrZrZmaj\nVzkAJE0Avg4cAcwCjpM0q6HZh4DfRsQewNeAL1fdrpmZVVPHEcABwPKIuC8ingEuAOY0tJkDnJte\n/xB4syTVsG0zMxslRUS1FUjvAQ6PiBPT+PHAGyLipFKbpanNijT+69RmTcO65gHzAGbMmDH7gQce\nqNS34RirxztIWhIRQy3mVapzOH2GF/o9lo+waFVnHftyJP3uRY1p3pjvy3K/B/H9Cv2/L/uxxnL7\nkWi3Lxv11UXgiDgzIoYiYmjKlCm97s6YyaHOHGqEPOp0jeNXHQGwEpheGt85TWvaRtJE4GXAYzVs\n28zMRqmOAFgE7ClpV0lbAscC8xvazAdOSK/fA/wsqp57MjOzSio/DjoiNkk6CbgSmACcHRHLJJ0G\nLI6I+cBZwHckLQcepwgJMzProVr+HkBELAAWNEw7pfT6d8AxdWzLzMzq0VcXgc3MrHscAGZmmXIA\nmJllygFgZpYpB4CZWaZquQvIumu0X4M3MyvzEYCZWaYcAGZmmfIpIOtbOZzqyqFGyKPOQazRRwBm\nZplyAJiZZcoBYGaWKQeAmVmmHABmZplyAJiZZcoBYGaWqUoBIGl7ST+RdG/69+Ut2j0n6ZY0NP65\nSDMz64GqRwAnA1dFxJ7AVWm8maciYr80vLPiNs3MrAZVA2AOcG56fS7wrorrMzOzLqkaADtGxKr0\n+mFgxxbttpa0WNKNklqGhKR5qd3iRx99tGLX+lcOdeZQI+RRp2scvzoGgKSfSlraZJhTbhcRAUSL\n1ewSEUPAXOCfJO3erFFEnBkRQxExNGXKlJHWMjByqDOHGiGPOl3j+NXxYXAR8ZZW8yQ9ImlqRKyS\nNBVY3WIdK9O/90laCOwP/Hp0XTYzszpUfRrofOAE4PT076WNDdKdQU9GxNOSdgDeCJxRcbu1GcQn\n+I1UDjWa2chVvQZwOvBWSfcCb0njSBqS9K3U5jXAYkm3AlcDp0fEHRW3a2ZmFVU6AoiIx4A3N5m+\nGDgxvb4e+OMq2zEzs/r5D8LYuJDLaa4c6nSN3eNHQZiZZcoBYGaWKQeAmVmmHABmZplyAJiZZcoB\nYGaWKQeAmVmmHABmZplyAJiZZcoBYGaWKQeAmVmmHABmZplyAJiZZcoBYGaWqUoBIOkYScskPS9p\nqE27wyXdLWm5pJOrbNPMzOpR9QhgKXAUcG2rBpImAF8HjgBmAcdJmlVxu2ZmVlHVvwh2J4Ckds0O\nAJZHxH2p7QXAHMB/FtLMrIe6cQ1gGvBgaXxFmmZmZj3U8QhA0k+BVzWZ9fmIuLTOzkiaB8xLoxsl\n3d2i6Q7AmhGseiTt61r3Lq0WGKM6e1EjtKizD2qsc91V96Xfr9Xa1rnubr5fR9p+zN+vLxIRlQdg\nITDUYt6fAFeWxj8HfK7i9haPVfuxXPdY1ukau7vusfz59cvPxPuyf/tdV43dOAW0CNhT0q6StgSO\nBeZ3YbtmZtZG1dtA3y1pBcVv+T+WdGWavpOkBQARsQk4CbgSuBP4QUQsq9ZtMzOrqupdQBcDFzeZ\n/hDwttL4AmBBlW01OHMM24/lukdqrPqdQ41jve6R6Kd+90ud/fQzGYl+6XctNSqdTzIzs8z4URBm\nZpkauAAYyWMlJJ0tabWkpcNY73RJV0u6Iz3e4hNt2m4t6VeSbk1tTx1NLWZmvTRQp4DSYyXuAd5K\n8YWyRcBxEdH0W8WSDgE2AudFxGs7rHsqMDUibpK0DbAEeFezdav46vOkiNgoaQvgF8AnIuLGCuWZ\nmXXVoB0B/P6xEhHxDLD5sRJNRcS1wOPDWXFErIqIm9LrDRR3LDX9xnIUNqbRLdIwOElqZsbgBUBX\nHishaSawP/DLNm0mSLoFWA38JCJatjUz60eDFgBjTtJk4EfA30bE+lbtIuK5iNgP2Bk4QFLbU0xm\nZv1m0AJgJTC9NL5zmlaLdD7/R8D3IuKi4SwTEWuBq4HD6+qHmVk3DFoAjNljJdKF3bOAOyPiqx3a\nTpG0XXr9EoqL0nfV0Q8zs24ZqAAY6WMlJJ0P3AC8WtIKSR9qs/o3AscDh0m6JQ1va9F2KnC1pNso\nQuknEXH5KEoyM+uZgboN1MzM6jNQRwBmZlYfB4CZWaYcAGZmmXIAmJllygFgZpYpB4CZWaYcAGZm\nmXIAmJll6v8DkUipuCgZO74AAAAASUVORK5CYII=\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%matplotlib inline\n", "from neurodynex.hopfield_network import network, pattern_tools, plot_tools\n", "\n", "pattern_size = 6\n", "\n", "# create an instance of the class HopfieldNetwork\n", "hopfield_net = network.HopfieldNetwork(nr_neurons= pattern_size**2)\n", "# instantiate a pattern factory\n", "factory = pattern_tools.PatternFactory(pattern_size, pattern_size)\n", "# create a checkerboard pattern and add it to the pattern list\n", "checkerboard = factory.create_checkerboard()\n", "pattern_list = [checkerboard]\n", "\n", "# add random patterns to the list\n", "pattern_list.extend(factory.create_random_pattern_list(nr_patterns=3, on_probability=0.5))\n", "plot_tools.plot_pattern_list(pattern_list)\n", "# how similar are the random patterns and the checkerboard? Check the overlaps\n", "overlap_matrix = pattern_tools.compute_overlap_matrix(pattern_list)\n", "plot_tools.plot_overlap_matrix(overlap_matrix)\n", "\n", "# let the hopfield network \"learn\" the patterns. Note: they are not stored\n", "# explicitly but only network weights are updated !\n", "hopfield_net.store_patterns(pattern_list)\n", "\n", "# create a noisy version of a pattern and use that to initialize the network\n", "noisy_init_state = pattern_tools.flip_n(checkerboard, nr_of_flips=5)\n", "hopfield_net.set_state_from_pattern(noisy_init_state)\n", "\n", "# from this initial state, let the network dynamics evolve.\n", "states = hopfield_net.run_with_monitoring(nr_steps=5)\n", "\n", "# each network state is a vector. reshape it to the same shape used to create the patterns.\n", "states_as_patterns = factory.reshape_patterns(states)\n", "# plot the states of the network\n", "plot_tools.plot_state_sequence_and_overlap(states_as_patterns, pattern_list, reference_idx=0, suptitle=\"Network dynamics\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***Note:*** The network state is a vector of N neurons. For visualization we use 2d patterns which are two dimensional numpy.ndarrays of size = (length, width). To store such patterns, initialize the network with N = length x width neurons." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.2. Introduction: Hopfield-networks\n", "\n", "This exercise uses a model in which neurons are pixels and take the values of -1 (off) or +1 (on). The network can store a certain number of pixel patterns, which is to be investigated in this exercise. During a retrieval phase, the network is started with some initial configuration and the network dynamics evolves towards the stored pattern (attractor) which is closest to the initial configuration.\n", "\n", "The dynamics is that of equation:\n", "\\begin{equation}\n", " S_i(t+1) = sgn\\left(\\sum_j w_{ij} S_j(t)\\right)\n", "\\end{equation}\n", "In the Hopfield model each neuron is connected to every other neuron (full connectivity). The connection matrix is\n", "\\begin{equation}\n", " w_{ij} = \\frac{1}{N}\\sum_{\\mu} p_i^\\mu p_j^\\mu\n", "\\end{equation}\n", "where N is the number of neurons, $p^{\\mu}_i$\n", " is the value of neuron $i$\n", " in pattern number $\\mu$\n", " and the sum runs over all patterns from $\\mu=1$\n", " to $\\mu=P$. This is a simple correlation based learning rule (Hebbian learning). Since it is not a iterative rule it is sometimes called one-shot learning. The learning rule works best if the patterns that are to be stored are random patterns with equal probability for on (+1) and off (-1). In a large networks (N to infinity) the number of random patterns that can be stored is approximately 0.14 times N.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.3. Exercise: N=6x6 Hopfield-network\n", "\n", "We study how a network stores and retrieve patterns. Using a small network of only 36 neurons allows us to have a close look at the network weights and dynamics." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.3.1. Question: Storing a single pattern\n", "#### [2 + 2 + 3 points]\n", "\n", "Modify the Python code given above to implement this exercise:\n", "\n", "Create a network with N=36 neurons.\n", "Create a single 6 by 6 checkerboard pattern.\n", "Store the checkerboard in the network.\n", "Set the initial state of the network to a noisy version of the checkerboard (nr_flipped_pixels = 5).\n", "Let the network dynamics evolve for 5 iterations.\n", "Plot the sequence of network states along with the overlap of network state with the checkerboard.\n", "Now test whether the network can still retrieve the pattern if we increase the number of flipped pixels. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# write your code here " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Plot the sequence of network states along with the overlap of network state with the checkerboard" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What happens at nr_flipped_pixels = 18, what if nr_flipped_pixels > 18 ?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.3.2. Question: the weights matrix\n", "#### [1 + 3 + 2 points]\n", "\n", "The patterns a Hopfield network learns are not stored explicitly. Instead, the network learns by adjusting the weights to the pattern set it is presented during learning. Let’s visualize this.\n", "\n", "1. Create a new 6x6 network. Do not yet store any pattern.\n", "2. What is the size of the network matrix?\n", "3. Visualize the weight matrix using the function [`plot_tools.plot_nework_weights()`](https://neuronaldynamics-exercises.readthedocs.io/en/latest/modules/neurodynex.hopfield_network.html#neurodynex.hopfield_network.plot_tools.plot_nework_weights). It takes the network as a parameter.\n", "4. Create a checkerboard, store it in the network.\n", "5. Plot the weights matrix. What weight values do occur?\n", "6. Create a new 6x6 network\n", "7. Create an L-shaped pattern (look at the pattern factory doc), store it in the network\n", "8. Plot the weights matrix. What weight values do occur?\n", "9. Create a new 6x6 network\n", "10. Create a checkerboard and an L-shaped pattern. Store both patterns in the network\n", "11. Plot the weights matrix. What weight values do occur? How does this matrix compare to the two previous matrices?\n", "\n", "***Note:*** The mapping of the 2 dimensional patterns onto the one dimensional list of network neurons is internal to the implementation of the network. You cannot know which pixel (x,y) in the pattern corresponds to which network neuron i." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# write your code here: Bullet points 1-5 " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# write your code here: Bullet points 6-11" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How does this matrix compare to the two previous matrices?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.3.3. Question: Weights Distribution\n", "#### *** Note:*** this exercise is NOT a bonus and it is graded\n", "#### [2 points]\n", "\n", "It’s interesting to look at the weights distribution in the three previous cases. You can easily plot a histogram by adding the following two lines to your script. It assumes you have stored your network in the variable ‘hopfield_net’. Explain the observed distribution (max 3 lines)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "plt.figure()\n", "plt.hist(hopfield_net.weights.flatten())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.4. Exercise: Capacity of an N=100 Hopfield-network\n", "\n", "Larger networks can store more patterns. There is a theoretical limit: the capacity of the Hopfield network. Read chapter [“17.2.4 Memory capacity”](http://neuronaldynamics.epfl.ch/online/Ch17.S2.html) to learn how memory retrieval, pattern completion and the network capacity are related." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.4.1. Question:\n", "#### [2 points]\n", "\n", "A Hopfield network implements so called associative or content-adressable memory. Explain what this means. (max 4 lines)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.4.2. Question:\n", "#### [1 points]\n", "\n", "Using the value $C_{store}$\n", " given in the book, how many patterns can you store in a N=10x10 network? Use this number K in the next question:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.4.3. Question:\n", "#### [3+2 points]\n", "\n", "Create an N=10x10 network and store a checkerboard pattern together with (K-1) random patterns. Then initialize the network with the unchanged checkerboard pattern. Let the network evolve for five iterations." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# write your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Rerun your script a few times. What do you observe?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.5. Exercise: Non-random patterns\n", "\n", "In the previous exercises we used random patterns. Now we us a list of structured patterns: the letters A to Z. Each letter is represented in a 10 by 10 grid." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.5.1. Question:\n", "\n", "Run the following code. Read the inline comments and look up the doc of functions you do not know." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "from neurodynex.hopfield_network import network, pattern_tools, plot_tools\n", "import numpy\n", "\n", "# the letters we want to store in the hopfield network\n", "letter_list = ['A', 'B', 'C', 'S', 'X', 'Y', 'Z']\n", "\n", "# set a seed to reproduce the same noise in the next run\n", "# numpy.random.seed(123)\n", "\n", "abc_dictionary =pattern_tools.load_alphabet()\n", "print(\"the alphabet is stored in an object of type: {}\".format(type(abc_dictionary)))\n", "# access the first element and get it's size (they are all of same size)\n", "pattern_shape = abc_dictionary['A'].shape\n", "print(\"letters are patterns of size: {}. Create a network of corresponding size\".format(pattern_shape))\n", "# create an instance of the class HopfieldNetwork\n", "hopfield_net = network.HopfieldNetwork(nr_neurons= pattern_shape[0]*pattern_shape[1])\n", "\n", "# create a list using Pythons List Comprehension syntax:\n", "pattern_list = [abc_dictionary[key] for key in letter_list ]\n", "plot_tools.plot_pattern_list(pattern_list)\n", "\n", "# store the patterns\n", "hopfield_net.store_patterns(pattern_list)\n", "\n", "# # create a noisy version of a pattern and use that to initialize the network\n", "noisy_init_state = pattern_tools.get_noisy_copy(abc_dictionary['A'], noise_level=0.2)\n", "hopfield_net.set_state_from_pattern(noisy_init_state)\n", "\n", "# from this initial state, let the network dynamics evolve.\n", "states = hopfield_net.run_with_monitoring(nr_steps=4)\n", "\n", "# each network state is a vector. reshape it to the same shape used to create the patterns.\n", "states_as_patterns = pattern_tools.reshape_patterns(states, pattern_list[0].shape)\n", "\n", "# plot the states of the network\n", "plot_tools.plot_state_sequence_and_overlap(\n", " states_as_patterns, pattern_list, reference_idx=0, suptitle=\"Network dynamics\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.5.2. Question:\n", "#### [2 + 3 points]\n", "\n", "Add the letter ‘R’ to the letter list and store it in the network." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# write your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Is the pattern ‘A’ still a fixed point? Does the overlap between the network state and the reference pattern ‘A’ always decrease? (max 3 lines)" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.5.3. Question:\n", "#### [3 + 3 points]\n", "\n", "Make a guess of how many letters the network can store. Then create a (small) set of letters. Check if all letters of your list are fixed points under the network dynamics. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# write your code here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Explain the discrepancy between the network capacity C (computed above) and your observation. (max 4 lines)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 8.6 Low activity attractor network\n", "\n", "Hopfield networks are a useful abstraction in order to understand how associative memory works. \n", "However they are quite far from biological plausibility. One step towards a more bioplausible attractor network is to consider sparse patterns. In the traditional Hopfield network you have analysed so far, each pattern counts 50% of active units, now we consider patterns that only have a small fraction `f` of active units. \n", "\n", "A simple low activity attractor network is described in this [paper](https://www.mitpressjournals.org/doi/full/10.1162/NECO_a_00499) (\"Scaling Laws of Associative Memory Retrieval\" by Sandro Romani, Itai Pinkoviezky, Alon Rubin and Misha Tsodyks). \n", "The goal of this exercise is to understand the network model proposed in the first part of the paper and reproduce Fig.1.\n", "\n", "Start by reading the introduction, ***section 2.1*** and ***A.1*** of the paper (that's all you'll need for the exercise).\n", "\n", "***Note:*** In what follows we stick to the same notation of the paper. If you are confused about the meaning of a variable, you know were to find it!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.6.1 Patterns and weight matrix\n", "#### [1 + 2 + 2 points]\n", "\n", "Start by implementing L sparse patterns. You can store them into a $L\\times P$ matrix called `items`. In what follows, use the indications given in the comments." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Initialize the parameters like in the paper ## \n", "N = ???\n", "L = ???\n", "f = ???\n", "T = ???\n", "Dth = ???\n", "Tth = ???\n", "TJ0 = ???\n", "min_J0 = ???\n", "max_J0 = ???\n", "timesteps = ??? ## for this one, check out Fig.1\n", "np.random.seed(26) ## KEEP THE SEED FIXED FOR CORRECTION ##" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Initialize patterns and weights (!!! for loops = -1 point !!!)\n", "\n", "items = ??? ## find the right probability distibution within the numpy.random library\n", "## Check that items has the right shape\n", "print \"Shape of items\", np.shape(items)\n", "\n", "J = ??? ## implement the definition of the weight matrix, find the right numpy function for matrix multiplication\n", "## Check that J has the right shape\n", "print \"Shape of items\", np.shape(J)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Explain why we need to subtract `f` in the weight formula. (max 5 lines)\n", "#### [4 points]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now initialize the threshold.\n", "#### [2 points]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Initilize the thtrshold\n", "thresh = ??? ## find the right probability distibution within the numpy.random library\n", "thresh_0 = thresh.copy() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.6.2 Dynamical evolution\n", "Now implement a function that let the state of the system evolve (for Fig.1A) and a second one which computes the overlaps between items (for Fig.1B) (use again the right numpy function for matrix multiplication and check out a numpy function for $\\Theta$).\n", "\n", "***Note:*** Don't confuse the overlap between the current state and the stored patterns and the overlap between patterns.\n", "#### [4 + 1 points]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def evolve(V_,thresh_,J0_):\n", " overlap_ = ??? ## write the overlap formula at time t \n", " V_old = ??? \n", " V_ = ??? ## write the update formula for time t+1\n", " thresh_ += ??? ## update the threshold for time t+1\n", " return V_,thresh_, overlap_\n", "\n", "def overlap_mat(a,b):\n", " return ??? ## if a and b are two patterns, what's the overlap between them?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Run the dynamics and plot $J_0(t)$.\n", "#### [4 + 1 points]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Here we prepare a dictionary in which to store the state, J0, threshold and overlap for each time step ## \n", "dic_evolution = {}\n", "dic_evolution['J0'] = []\n", "dic_evolution['V'] = []\n", "dic_evolution['thresh'] = []\n", "dic_evolution['overl'] = []\n", "\n", "## Set the initial state of the matrix equal to the first pattern\n", "V = ???\n", "\n", "## Now we are ready to run the dynamics \n", "for i in range(timesteps):\n", " J0 = ??? ## use numpy.sin() to have the correct osclillatory behavior. Don't forget to scale it correctly\n", " ??? = evolve(???)\n", " \n", " dic_evolution['J0'] += ???\n", " dic_evolution['V'] += ???\n", " dic_evolution['thresh'] += ???\n", " dic_evolution['overl'] += ???" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## check that you have the right evolution of J0\n", "plt.plot(???)\n", "plt.xlim([0,???]) ## plot for one period\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 8.6.3 Plot the figures\n", "#### [2 + 4 points]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Plot Fig. 1A \n", "plt.figure( ??? ) ## check out the inputs, how do you fix the figure size?\n", "plt.imshow( ??? , interpolation = 'None', aspect = 'auto', cmap = 'jet') ## Insert the right dictionary \n", "## add a colorbar here\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "## Plot Fig. 1B\n", "## first you need to create a matrix filled with the overlaps between patterns.\n", "overlap_matrix = ??? ## declare a matrix of 0s with the right shape. One more time, numpy is going to be useful.\n", "for i in range( ??? ):\n", " for j in range(i, ??? ):\n", " if i != j:\n", " overlap_matrix[i,j] = overlap_matrix[j,i] = ??? ## insert the right expression for the element of overlap_matrix\n", " \n", "## Set the diagonal of overlap_matrix = to average of other entries ## \n", "overlap_matrix = ??? \n", "plt.imshow( ??? , aspect='auto',cmap='jet')\n", "## add a colorbar here\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Comment your first figure: do you notice any recurrence in the item retrieval? How do they interpret it in the paper? \n", "Now explain the link between the first and the second figure. (max 12 lines)\n", "#### [8 points]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "your answer" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.1" } }, "nbformat": 4, "nbformat_minor": 2 }