164 lines
18 KiB
Plaintext
164 lines
18 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "70f73980",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import numpy as np"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e8012b5c",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Numerical Optimization\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bea6fa0f",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Linear Classifier\n",
|
|
"\n",
|
|
"$D = \\set{x_i, y_i }^N_{i=1}$ is given. ($x_i \\in \\R^D,\\, y_i\\in \\set{0, 1}$)\n",
|
|
"\n",
|
|
"like"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 37,
|
|
"id": "3ad846b8",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi4AAAGiCAYAAADA0E3hAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjYsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvq6yFwwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAKdtJREFUeJzt3QtQ1XX+//E3ooBOgZWrKKKUlXYxNW+r5gS7bEw2hlO2bDje8rJdthGdNjVLs5vdB7cwJ7tYU6ktP2ucdKjWYqqNXSfMmWrTptRAVlDXBNMCw+9/3p/zP8Q5HJAD5/Y55/mYOR2/n/P5cr7wFc+rzzXOcRxHAAAALNAl3BcAAADQXgQXAABgDYILAACwBsEFAABYg+ACAACsQXABAADWILgAAABrEFwAAIA1CC4AAMAaBBcAABC9weWjjz6SyZMnS79+/SQuLk7efvvtM55TWloqV155pSQmJsqFF14o69ev7+j1AgCAGOZ3cDlx4oQMGzZMioqK2lV/3759ct1110lWVpbs2rVLCgoKZO7cufLuu+925HoBAEAMi+vMJova4vLWW2/JlClTWq2zePFi2bp1q3z55ZdNZX/605/k2LFjUlJS0tG3BgAAMahrsN+grKxMsrOzPcpycnJMy0tr6uvrzcPt9OnTcvToUTnvvPNMWAIAAJFP20aOHz9uhpd06dLFjuBSXV0tffr08SjT47q6Ovnpp5+ke/fuLc5ZtWqVrFy5MtiXBgAAQqCyslL69+9vR3DpiKVLl8qiRYuajmtra2XAgAHmG09OTg7rtQEAgPbRRor09HQ5++yzJVCCHlxSU1OlpqbGo0yPNYD4am1ROvtIH970HIILAAB2CeQwj6Cv4zJu3DjZvn27R9n7779vygEAAIIaXH788UczrVkf7unO+ueKioqmbp4ZM2Y01b/11ltl7969cvfdd8vu3btlzZo18uabb8rChQv9fWsAABDj/A4un332mYwYMcI8lI5F0T8vX77cHB88eLApxKjzzz/fTIfWVhZd/+Wpp56SF154wcwsAgAACNk6LqEc3JOSkmIG6TLGBQDQEfpx98svv0hjY2O4LyVqxMfHS9euXVsdwxKMz++InFUEAEAgNTQ0mB6BkydPhvtSok6PHj2kb9++kpCQEJL3I7gAAKKaLmKq4zG1dUAXQtMPWBYzDUwLlgbCw4cPm5/vRRddFLBF5tpCcAEARDX9cNXwouuJaOsAAkeXNenWrZt8//335ueclJQkwRb8aAQAQAQIRWtALOoS4p8rdxEAAFiD4AIAAKxBcAEAANYguAAAgCZHjx6VadOmmXVXevbsKXPmzDGr5kcKggsAAO2li9eVlops2OB6jsLF7KZNmyZfffWVWfH+nXfekY8++kjmz58vkYLgAgBAe2zeLJKRIZKVJZKf73rWYy0PgldffVXOO+88qa+v9yifMmWKTJ8+PSjv+fXXX0tJSYnZmmfs2LFy1VVXyTPPPCMbN26U//73vxIJCC4AAJyJhpOpU0UOHPAsr6pylQchvNx0001me4ItW7Y0lR06dMjs/3fLLbe0et5ll10mZ511VquPa6+9ttVzy8rKTPfQqFGjmsqys7PNlOd///vfEglYgA4AgLZod9CCBbpUbMvXtExX4S0oEMnN1c17Arq4W35+vrz88ssmxKjXXntNBgwYIJmZma2et23bNjl16lSbX7c11dXV0rt3b48y3Yvo3HPPNa9FAoILAABt+fjjli0t3uGlstJVr41A0RHz5s2T0aNHS1VVlaSlpcn69etl1qxZbW5ZMHDgQIlmdBUBANCWgwcDW88PI0aMkGHDhpnxLuXl5WbQrAaXtnSmqyg1NdV0RzWnO2rrTCN9LRLQ4gIAQFv69g1sPT/NnTtXCgsLTatLdna22XOpLZ3pKho3bpwcO3bMhKSRI0easg8++MDs9aSDdSNBnKPbO0a4uro6SUlJkdraWjOvHACA9vr555/N7sXnn39+xzYB1DEuOntIB+L6+sjUbpv+/UX27QvoGBc3/ezTXa215ePVV1+VvLw8CSZtkampqZG1a9eaADR79mwzWPeNN97w++cbjM9vuooAAGiLhpHVq11/9h5b4j4uLAxKaFH6wX/jjTeabp4pU6ZIsL3++usyZMgQ+f3vfy+TJk0yU6Kff/55iRR0FQEAcCY33CBSXOyaXdR8oK62tGho0deDSLuJdGG4xMRECTadQdRa60okILgAANAeGk50yrPOHtKBuDqmZeLEoLW0qB9++EFKS0vNY82aNUF7H5sQXAAAaC8NKQGe8nymWUUaXh577DEZPHhwyN43khFcAACIUPv37w/3JUQcBucCAABrEFwAAIA1CC4AAMAaBBcAAGANggsAALAGwQUAAFiD4AIAAKxBcAEAwI/9FktLRTZscD3rcbR5+OGHZfz48dKjRw/p2bOnRBqCCwAA7bB5s2uT6Kwskfx817Mea3k0aWhokJtuukluu+02iUQEFwAAzkDDydSpnvsrqqoqV3kwwsurr74q5513ntTX13uUT5kyRaZPny7BsnLlSlm4cKEMHTpUIhHBBQCANmh3kG4K7TgtX3OXFRQEvttIWz0aGxtly5YtTWWHDh2SrVu3yi233NLqeZdddpmcddZZrT6uvfZasRl7FQEA0AbdDNq7pcU7vFRWuuoFcv/F7t27S35+vrz88ssmxKjXXntNBgwYIJltvNG2bdvk1KlTbX5dmxFcAABow8GDga3nj3nz5sno0aOlqqpK0tLSZP369TJr1iyJi4tr9ZyBAwdKNKOrCACANvTtG9h6/hgxYoQMGzbMjHcpLy+Xr776ygSXttBVBABADJs4UaR/f9dAXF/jXLTxQ1/XesEwd+5cKSwsNK0u2dnZkp6e3mZ9uooAAIhh8fEiq1e7Zg9pSGkeXtw9NoWFrnrBoONc7rrrLlm3bp1peTmTznYVVVRUyNGjR82zDg7etWuXKb/wwgtNi0240VUEAAgbWxZ0u+EGkeJikbQ0z3JtadFyfT1YUlJS5MYbbzShYcqUKRJsy5cvN11UK1askB9//NH8WR+fffaZRAJaXAAAYaFrn+g04+YzdjQIaOtGMINAR+k15ea6Zg/pQFwd06LdQ8FqaWlOu4mmTZsmiYmJQX8vHQCsj0hFcAEAhG1BN+8xI+4F3YLditFRGlICOeX5TH744QcpLS01jzVr1oTujSMYwQUAEFELuum4EV3QTVs3QtGaEcm0i0bDy2OPPSaDBw8O9+VEBIILACAmFnSz0f79+8N9CRGHwbkAgJhZ0A32I7gAAGJiQTfHV98UrPu5ElwAAGFZ0K21Veu1XNdYC9SCbt26dTPPJ0+eDMwXhAf3z9X9cw42xrgAAKJ6Qbf4+Hjp2bOn2VlZ9ejRo829ftD+lhYNLfpz1Z+v/pxDgeACAAjbgm6+1nHR0BLoqdCpqanm2R1eEDgaWtw/31CIcyzo9KurqzMrB9bW1kpycnK4LwcAEMCp0aFc0E2XsG9rHx/4R7uH2mppCcbnNy0uAICYWdBNP2RD1aWB4CC4AAAQhtYfdAzBBQAQ82zbNymWMR0aABDT3Psmea/m6943SV9H5CC4AABi1pn2TVK6b5LWQ2QguAAAYpY/+yYhMhBcAAAxi32T7ENwAQDErHDtm4SOI7gAAGJWqPdNQucRXAAAEuv7Jinv8BKMfZPQeQQXAEBMc++blJbmWa4tMVrOOi6RhQXoAAAxT8NJbi4r59qA4AIAQBj2TULH0FUEAACsQXABAADRHVyKiookIyNDkpKSZOzYsbJjx4426xcWFsrgwYOle/fukp6eLgsXLpSff/65o9cMAABilN9jXDZt2iSLFi2StWvXmtCioSQnJ0f27NkjvXv3blH/jTfekCVLlshLL70k48ePl2+++UZmzZolcXFx8vTTTwfq+wAAhIHu4cOAVkR0i4uGjXnz5sns2bPl0ksvNQGmR48eJpj48umnn8qECRMkPz/ftNJcc801cvPNN5+xlQYAENl01+SMDJGsLJH8fNezHrObMiImuDQ0NEh5eblkZ2f/+gW6dDHHZWVlPs/RVhY9xx1U9u7dK9u2bZNJkya1+j719fVSV1fn8QAARA4NJ1OnttygsKrKVU54QUQElyNHjkhjY6P06dPHo1yPq6urfZ6jLS0PPPCAXHXVVdKtWzcZNGiQZGZmyj333NPq+6xatUpSUlKaHjouBgAQOd1DCxa4dk725i4rKHDVA6ybVVRaWiqPPPKIrFmzRnbu3CmbN2+WrVu3yoMPPtjqOUuXLpXa2tqmR6XuKQ4AiAg6psW7pcU7vOg/21oPCOvg3F69ekl8fLzU1NR4lOtxamqqz3Puu+8+mT59usydO9ccDx06VE6cOCHz58+XZcuWma4mb4mJieYBAIi8gbj/93/tq68DdoGwtrgkJCTIyJEjZfv27U1lp0+fNsfjxo3zec7JkydbhBMNP8rx1c4IAIjogbjPPtu+c3SWERD26dA6FXrmzJkyatQoGTNmjJkOrS0oOstIzZgxQ9LS0sw4FTV58mQzE2nEiBFm+vS3335rWmG03B1gAACRPxC3vf+vqbsq6waFOjUaCHtwycvLk8OHD8vy5cvNgNzhw4dLSUlJ04DdiooKjxaWe++916zZos9VVVXym9/8xoSWhx9+OLDfCQAgpANxWwstqrCQ9VwQHHGOBf01Oh1aZxfpQN3k5ORwXw4AxIzSUlf3UHvpJFANLbrbMlAXhM9vdocGAHR6gO1f/iJy442snIvgI7gAADo9wFZDS2ZmsK8GYHdoAEAbtAVFB9q6x65403LtHmIgLkKF4AIAaJV2+6xe7fqzd3hhIC7CgeACAGiTDrQtLhZJS/Ms15YYLWcgLkKJMS4AgDPScJKb61o5Vwfs6tgXBuIiHAguAIB20ZDCAFyEG11FAADAGgQXAABgDYILAACwBsEFAABYg8G5AICwb+TIbCW0F8EFABA2mze7dp8+cMBzfRhd9I71YeALXUUAgLCFlqlTPUOLqqpylevrgDeCCwAgLN1D2tLiOC1fc5cVFLjqAc0RXAAAIadjWrxbWrzDS2Wlqx7QHGNcACAMYn1Aqn7fgayH2EFwAYAQY0CqK6wFsh5iB11FABBCDEh10RYmDWtxcb5f1/L0dFc9oDmCCwCECANSf6XdYtrCpLzDi/u4sDC2us/QPgQXAAgRBqR60m6x4mKRtDTPcm2J0fJY6TaDfxjjAgAhwoDUljSc5ObG9kBl+IfgAgAhwoBU3zSkZGaG+ypgC7qKACBEGJAKdB7BBQBChAGpQOcRXAAghBiQCnROnOP4mpgXWerq6iQlJUVqa2slOTk53JcDAAFdObd3b1fZoUMMTkV0qQvC5zeDcwEgjANSdcG5WbNiexVdwB90FQFAmETSKrraAlRaKrJhg+s5FhbBg50ILgAQ46voakDKyBDJyhLJz3c963GsbD8AuxBcACCGV9GNpFYfoD0ILgAQo6voRlKrD9BeBBcAiNFVdCOl1QfwB8EFAGJ0Fd1IaPUB/EVwAYAYXUU3Elp9AH8RXAAgRlfRjYRWH8BfLEAHAGGk4SQ399dVdEO5cq671UdnD2lIaT5Il72TEKkILgAQIavohrPVR2cXea/eq6GF1XsRaQguABDjwtnqA/iL4AIACGurD+APBucCAABrEFwAAIA1CC4AAMAaBBcAAGANggsAALAGwQUAAFiD4AIAAKxBcAEAANYguAAAAGsQXAAAgDUILgAAwBoEFwAAYA2CCwAAsAbBBQAAWIPgAgAArNE13BcAAEBHNDaKfPyxyMGDIn37ikycKBIfH+6rQrARXAAA1tm8WWTBApEDB34t699fZPVqkRtuCOeVIdjoKgIAWBdapk71DC2qqspVrq8jehFcAABWdQ9pS4vjtHzNXVZQ4KqH6ERwAQBYQ8e0eLe0eIeXykpXPUQnggsAwBo6EDeQ9WAfggsAwBo6eyiQ9WAfggsAwBo65VlnD8XF+X5dy9PTXfUQnToUXIqKiiQjI0OSkpJk7NixsmPHjjbrHzt2TO644w7p27evJCYmysUXXyzbtm3r6DUDAGKUrtOiU56Vd3hxHxcWsp5LNPM7uGzatEkWLVokK1askJ07d8qwYcMkJydHDh065LN+Q0OD/OEPf5D9+/dLcXGx7NmzR9atWydpaWmBuH4AQIzRdVqKi0W8P0a0JUbLWcclusU5jq9JZa3TFpbRo0fLs88+a45Pnz4t6enpcuedd8qSJUta1F+7dq088cQTsnv3bunWrVuHLrKurk5SUlKktrZWkpOTO/Q1AADRhZVzI18wPr/9anHR1pPy8nLJzs7+9Qt06WKOy8rKfJ6zZcsWGTdunOkq6tOnj1x++eXyyCOPSGMbk+zr6+vNN9v8AQBAcxpSMjNFbr7Z9UxoiQ1+BZcjR46YwKEBpDk9rq6u9nnO3r17TReRnqfjWu677z556qmn5KGHHmr1fVatWmUSmvuhLToAgMik/x9aWiqyYYPrmcXfYPWsIu1K6t27tzz//PMycuRIycvLk2XLlpkupNYsXbrUNCu5H5W6mhAAIOLo8voZGSJZWSL5+a5nPWbZfUTEJou9evWS+Ph4qamp8SjX49TUVJ/n6EwiHdui57ldcsklpoVGu54SEhJanKMzj/QBAIj8PYO8R0q69wxioCzC3uKiIUNbTbZv3+7RoqLHOo7FlwkTJsi3335r6rl98803JtD4Ci0AgMjHnkGwpqtIp0LrdOZXXnlFvv76a7ntttvkxIkTMnv2bPP6jBkzTFePm75+9OhRWbBggQksW7duNYNzdbAuAMBO7BkEK7qKlI5ROXz4sCxfvtx09wwfPlxKSkqaBuxWVFSYmUZuOrD23XfflYULF8oVV1xh1m/RELN48eLAficAgJBhzyBYs45LOLCOCwBEFp09pANxz+TDD11TlRGb6sK9jgsAAIo9gxAuBBcAgN/YMwjhQnABAHQIewbBisG5AAC4aTjJzWXPIIQOwQUAEJA9g4BQoKsIAABYg+ACAACsQXABAADWILgAAABrEFwAAIA1CC4AAMAaBBcAAGANggsAALAGwQUAAFiD4AIAAKxBcAEAANYguAAAAGsQXAAAgDUILgAAwBoEFwAAYA2CCwAAsAbBBQAAWIPgAgAArEFwAQAA1iC4AAAAaxBcAACANQguAADAGgQXAABgDYILAACwBsEFAABYg+ACAACsQXABAADWILgAAABrEFwAAIA1CC4AAMAaBBcAAGANggsAALAGwQUAAFiD4AIAAKxBcAEAANYguAAAAGsQXAAAgDUILgAAwBoEFwAAYA2CCwAAsAbBBQAAWIPgAgAArEFwAQAA1iC4AAAAaxBcAACANbqG+wKs0Ngo8vHHIgcPivTtKzJxokh8fLivCgCAmENwOZPNm0UWLBA5cODXsv79RVavFrnhhnBeGQAAMYeuojOFlqlTPUOLqqpylevrAAAgZAgubXUPaUuL47R8zV1WUOCqBwAAQoLg0hod0+Ld0uIdXiorXfUAAEBIEFxaowNxA1kPAAB0GsGlNTp7KJD1AABApxFcWqNTnnX2UFyc79e1PD3dVQ8AAIQEwaU1uk6LTnlW3uHFfVxYyHouAACEEMGlLbpOS3GxSFqaZ7m2xGg567gAABBSLEB3JhpOcnNZORcAgAhAcGkPDSmZmeG+CgAAYh5dRQAAILqDS1FRkWRkZEhSUpKMHTtWduzY0a7zNm7cKHFxcTJlypSOvC0AAIhxfgeXTZs2yaJFi2TFihWyc+dOGTZsmOTk5MihQ4faPG///v1y1113yUSmDwMAgFAFl6efflrmzZsns2fPlksvvVTWrl0rPXr0kJdeeqnVcxobG2XatGmycuVKueCCC874HvX19VJXV+fxAFrQfaJKS0U2bHA9s28UAEQ9v4JLQ0ODlJeXS3Z29q9foEsXc1xWVtbqeQ888ID07t1b5syZ0673WbVqlaSkpDQ90nWhN6A53Zk7I0MkK0skP9/1rMfs2A0AUc2v4HLkyBHTetKnTx+Pcj2urq72ec4nn3wiL774oqxbt67d77N06VKpra1telTqZoaAm4aTqVNbboJZVeUqJ7wAQNQK6qyi48ePy/Tp001o6dWrV7vPS0xMlOTkZI8HYGh30IIFrt25vbnLCgroNgKAKOXXOi4aPuLj46WmpsajXI9TU1Nb1P/uu+/MoNzJkyc3lZ0+fdr1xl27yp49e2TQoEEdv3rEHl0I0LulxTu8aAud1mPtHQCI7RaXhIQEGTlypGzfvt0jiOjxuHHjWtQfMmSIfPHFF7Jr166mx/XXXy9ZWVnmz4xdgd909eJA1gMARPfKuToVeubMmTJq1CgZM2aMFBYWyokTJ8wsIzVjxgxJS0szA2x1nZfLL7/c4/yePXuaZ+9yoF10y4VA1gMARHdwycvLk8OHD8vy5cvNgNzhw4dLSUlJ04DdiooKM9MICApdB0g3udSBuL7GuejO3fo66wUBQFSKcxxf//pHFl3HRadF6wwjBuqiaVaRav7XV0OLYuduAIjaz2+aRmAfDSUaTtLSPMu1pYXQAgBRjd2hYScNJ7m5rtlDOhBXx7Ro95Du5A0AiFoEF9hLQwpTngEgptBVBAAArEFwAQAA1iC4AAAAaxBcAACANQguAADAGgQXAABgDYILAACwBsEFAABYgwXoEHqNjax4CwDoEIILQr9B4oIFIgcOeO4xtHo1ewwBAM6IriKEflfn5qFFVVW5yvV1AADaQHBB6LqHtKXFcVq+5i4rKHDVAwCgFQQXhIaOafFuafEOL5WVrnoAALSC4ILQ0IG4gawHAIhJBBeEhs4eCmQ9AEBMYlYRQkOnPOvsIR2I62ucS1yc63WtF+uYLg4AraLFBaGhH7w65dkdUppzHxcW8gGtM6syMkSyskTy813PesyMKwAwCC4IHV2npbhYJC3Ns1xbWrQ81tdxYbo4AJxRnOP4arePLHV1dZKSkiK1tbWSnJwc7stBZ9EV4vtnoi0rrc28cnel7dvHzwqANYLx+c0YF4SefvBmZob7KuydLs7PDkAMo6sIiARMFweAdiG4AJGA6eIA0C4EFyCSpot7z7hy0/L0dKaLA4h5BBcEfpBpaanIhg2uZ/Yeah+miwNAuxBcEDisQdI5TBcHgDNiOjQCuwaJ918nd2sBH7ztx3RxAFGiLgif3wQXdB5rkAAAQvT5TVcRQrsGCQAAnUBwQeexBgkAIEQILug81iABAIQIwQWdxxokAIAQIbig81iDBAAQIgQXBAZrkAAAQoDdoRE4Gk5yc1mDBAAQNAQXBJaGlMzMcF8FACBK0VUEAACsQXABAADWILgAAABrEFwAAIA1CC4AAMAaBBcAAGANggsAALAGwQUAAFiD4AIAAKxBcAEAANYguAAAAGsQXAAAgDUILgAAwBoEFwAAYA2CCwAAsAbBBQAAWIPgAgAArNE13BcAhFRjo8jHH4scPCjSt6/IxIki8fHhvioAQDsRXBA7Nm8WWbBA5MCBX8v69xdZvVrkhhvCeWUAgHaiqwixE1qmTvUMLaqqylWurwMAIh7BBbHRPaQtLY7T8jV3WUGBqx4AIKIRXBD9dEyLd0uLd3iprHTVAwBENIILop8OxA1kPQBA2BBcEP109lAg6wEAwobgguinU5519lBcnO/XtTw93VUPABB9waWoqEgyMjIkKSlJxo4dKzt27Gi17rp162TixIlyzjnnmEd2dnab9YGA03VadMqz8g4v7uPCQtZzAYBoDC6bNm2SRYsWyYoVK2Tnzp0ybNgwycnJkUOHDvmsX1paKjfffLN8+OGHUlZWJunp6XLNNddIlU5DBUJF12kpLhZJS/Ms15YYLWcdFwCwQpzj+Joj2jptYRk9erQ8++yz5vj06dMmjNx5552yZMmSM57f2NhoWl70/BkzZvisU19fbx5udXV15j1qa2slOTnZn8sFYmvl3Gj//gBYRT+/U1JSAvr57dfKuQ0NDVJeXi5Lly5tKuvSpYvp/tHWlPY4efKknDp1Ss4999xW66xatUpWrlzpz6UB7aMf4pmZEpVYGRhADPCrq+jIkSOmxaRPnz4e5XpcXV3drq+xePFi6devnwk7rdFgpOnM/ajUNTYAtI6VgQHEiJDuVfToo4/Kxo0bzbgXHdjbmsTERPMAEICVgXUAsq4MnJtLtxGA2Gpx6dWrl8THx0tNTY1HuR6npqa2ee6TTz5pgst7770nV1xxRceuFkBLrAwMIIb4FVwSEhJk5MiRsn379qYyHZyrx+PGjWv1vMcff1wefPBBKSkpkVGjRnXuigF4YmVgADHE764inQo9c+ZME0DGjBkjhYWFcuLECZk9e7Z5XWcKpaWlmQG26rHHHpPly5fLG2+8YdZ+cY+FOeuss8wDQCexMjCAGOJ3cMnLy5PDhw+bMKIhZPjw4aYlxT1gt6Kiwsw0cnvuuefMbKSpOkCwGV0H5v777w/E9wDENvfKwDoQ19c4Fx3joq+zMjCAWFzHJVrmgQNROatINf+Vdq8MzCJ7AKLk85u9ioBowMrAAGJESKdDAwgiDSc65ZmVcwFEMYILEE2ieWVgAKCrCAAA2IQWF8QGNh8EgKhAcEH0Y/NBAIgadBUhurH5IABEFYILYnfzQaWbD2o9AIAVCC6IXmw+CABRh+CC6MXmgwAQdQguiF5sPggAUYfggujffNC9X483LU9PZ/NBALAIwQXRS9dp0SnPyju8uI8LC1nPBQAsQnBBdGPzQQCIKixAh+jH5oMAEDUILogNbD4IAFGBriIAAGANggsAALAGwQUAAFiDMS6IDLpfEINnAQBnQHBB+OkOzboZYvN9hXS6sq7BwnRlAEAzdBUh/KFl6tSWmyFWVbnK9XUAAP4/gotNXSmlpSIbNrie9dh2+j1oS4vu0uzNXVZQEB3fKwAgIAguNtBWh4wMkawskfx817Me294aoWNavFtavMNLZaWrHgAABBcLRHNXig7EDWQ9AEDUI7hEsmjvStHZQ4GsBwCIegSXSBbtXSk65VlnD3nv3Oym5enprnoAABBcIly0d6XoOi065Vl5hxf3cWEh67kAAJoQXCJZLHSl6DotxcUiaWme5doSo+Ws4wIAaCbOcXwNoIgsdXV1kpKSIrW1tZKcnCwxQ8eu6OwhHYjr6zZpq4R+wO/bZ3+rBCvnAkDUqQvC5zcr59rQlaKzhzSkNA8v0daVot9DZma4rwIAEOHoKop0dKUAANCEFhcbaDjJzaUrBQAQ8wgutqArBQAAuooAAIA9CC4AAMAaBBcAAGANggsAALAGwQUAAFiD4AIAAKxBcAEAANYguAAAAGsQXAAAgDUILgAAwBoEFwAAYA2CCwAAsAbBBQAAWIPgAgAArEFwAQAA1iC4AAAAaxBcAACANQguAADAGgQXAABgDYILAACwBsEFAABYg+ACAACsQXABAADWILgAAABrEFwAAIA1CC4AAMAaBBcAAGANggsAALAGwQUAAER3cCkqKpKMjAxJSkqSsWPHyo4dO9qs//e//12GDBli6g8dOlS2bdvW0esFAAAxzO/gsmnTJlm0aJGsWLFCdu7cKcOGDZOcnBw5dOiQz/qffvqp3HzzzTJnzhz5/PPPZcqUKebx5ZdfBuL6AQBADIlzHMfx5wRtYRk9erQ8++yz5vj06dOSnp4ud955pyxZsqRF/by8PDlx4oS88847TWW//e1vZfjw4bJ27Vqf71FfX28ebrW1tTJgwACprKyU5ORkfy4XAACESV1dnckIx44dk5SUlIB8za7+VG5oaJDy8nJZunRpU1mXLl0kOztbysrKfJ6j5dpC05y20Lz99tutvs+qVatk5cqVLcr1mwcAAHb53//+F57gcuTIEWlsbJQ+ffp4lOvx7t27fZ5TXV3ts76Wt0aDUfOwo0lt4MCBUlFREbBvHJ1Lz7R+hR/3InJwLyIL9yNyuHtMzj333IB9Tb+CS6gkJiaahzcNLfwljAx6H7gXkYF7ETm4F5GF+xE5tHcmYF/Ln8q9evWS+Ph4qamp8SjX49TUVJ/naLk/9QEAAAISXBISEmTkyJGyffv2pjIdnKvH48aN83mOljevr95///1W6wMAAASsq0jHnsycOVNGjRolY8aMkcLCQjNraPbs2eb1GTNmSFpamhlgqxYsWCBXX321PPXUU3LdddfJxo0b5bPPPpPnn3++3e+p3UY6/dpX9xFCi3sRObgXkYN7EVm4H9F9L/yeDq10KvQTTzxhBtjqtOa//e1vZpq0yszMNIvTrV+/3mMBunvvvVf2798vF110kTz++OMyadKkgH0TAAAgNnQouAAAAIQDexUBAABrEFwAAIA1CC4AAMAaBBcAAGCNiAkuRUVFZjZSUlKSmaG0Y8eONuvrTKUhQ4aY+kOHDpVt27aF7FqjnT/3Yt26dTJx4kQ555xzzEP3rTrTvUPwfi/cdNmBuLg4sxM7wnMvdKuSO+64Q/r27Wumgl588cX8OxWme6HLdgwePFi6d+9utgJYuHCh/PzzzyG73mj10UcfyeTJk6Vfv37m35u29iB0Ky0tlSuvvNL8Tlx44YUeM5DbzYkAGzdudBISEpyXXnrJ+eqrr5x58+Y5PXv2dGpqanzW/+c//+nEx8c7jz/+uPOf//zHuffee51u3bo5X3zxRcivPdr4ey/y8/OdoqIi5/PPP3e+/vprZ9asWU5KSopz4MCBkF97rN8Lt3379jlpaWnOxIkTndzc3JBdbzTz917U19c7o0aNciZNmuR88skn5p6UlpY6u3btCvm1x/q9eP31153ExETzrPfh3Xffdfr27essXLgw5NcebbZt2+YsW7bM2bx5s85Odt5666026+/du9fp0aOHs2jRIvPZ/cwzz5jP8pKSEr/eNyKCy5gxY5w77rij6bixsdHp16+fs2rVKp/1//jHPzrXXXedR9nYsWOdP//5z0G/1mjn773w9ssvvzhnn32288orrwTxKmNDR+6F/vzHjx/vvPDCC87MmTMJLmG6F88995xzwQUXOA0NDSG8ytjg773Qur/73e88yvSDc8KECUG/1lgi7Qgud999t3PZZZd5lOXl5Tk5OTl+vVfYu4oaGhqkvLzcdDE034xJj8vKynyeo+XN66ucnJxW6yN498LbyZMn5dSpUwHdCTQWdfRePPDAA9K7d2+ZM2dOiK40+nXkXmzZssVsa6JdRX369JHLL79cHnnkEWlsbAzhlUefjtyL8ePHm3Pc3Ul79+41XXYsghp6gfrsDvvu0EeOHDG/zPrL3Zwe79692+c5umKvr/pajtDeC2+LFy82/Z3efzkR/HvxySefyIsvvii7du0K0VXGho7cC/1w/OCDD2TatGnmQ/Lbb7+V22+/3YR6Xf4cobsX+fn55ryrrrpKexjkl19+kVtvvVXuueeeEF01zvTZXVdXJz/99JMZg9QeYW9xQfR49NFHzaDQt956ywyaQ+gcP35cpk+fbgZL6y7uCC/dfFZbvnRPNt2YNi8vT5YtWyZr164N96XFHB0Mqq1da9askZ07d8rmzZtl69at8uCDD4b70tBBYW9x0X9k4+PjpaamxqNcj1NTU32eo+X+1Efw7oXbk08+aYLLP/7xD7niiiuCfKXRz9978d1335m9wHSEf/MPT9W1a1fZs2ePDBo0KARXHn068nuhM4m6detmznO75JJLzP9xandHQkJC0K87GnXkXtx3330m1M+dO9cc6yxU3Rh4/vz5JkxqVxNCo7XP7uTk5Ha3tqiw3zH9Bdb/I9m+fbvHP7h6rH3Evmh58/rq/fffb7U+gncvlG6aqf/3UlJSYnYNR+jvhS4N8MUXX5huIvfj+uuvl6ysLPNnnQKK0P1eTJgwwXQPucOj+uabb0ygIbSE9l7ouDvvcOIOlGzVF1oB++x2ImR6m05XW79+vZkiNX/+fDO9rbq62rw+ffp0Z8mSJR7Tobt27eo8+eSTZgruihUrmA4dpnvx6KOPmqmJxcXFzsGDB5sex48fD+N3EZv3whuzisJ3LyoqKszsur/85S/Onj17nHfeecfp3bu389BDD4Xxu4jNe6GfD3ovNmzYYKbjvvfee86gQYPM7FR0jv47r0th6EPjxNNPP23+/P3335vX9T7o/fCeDv3Xv/7VfHbrUhrWTodWOp97wIAB5kNQp7v961//anrt6quvNv8IN/fmm286F198samv06u2bt0ahquOTv7ci4EDB5q/sN4P/ccCof+9aI7gEt578emnn5plGvRDVqdGP/zww2a6OkJ7L06dOuXcf//9JqwkJSU56enpzu233+788MMPYbr66PHhhx/6/Pff/fPXZ70f3ucMHz7c3Dv9vXj55Zf9ft84/U9gG4MAAACCI+xjXAAAANqL4AIAAKxBcAEAANYguAAAAGsQXAAAgDUILgAAwBoEFwAAYA2CCwAAsAbBBQAAWIPgAgAArEFwAQAAYov/B2PvliRz5FFdAAAAAElFTkSuQmCC",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"np.random.seed(7)\n",
|
|
"\n",
|
|
"y = np.random.randint(0, 2, size=20)\n",
|
|
"\n",
|
|
"x = np.random.normal(0, 0.08, size=(2, 20)) + np.vstack([y / 2 + 0.25, y / 2 + 0.25])\n",
|
|
"\n",
|
|
"plt.xlim(0, 1)\n",
|
|
"plt.ylim(0, 1)\n",
|
|
"\n",
|
|
"plt.scatter(x[0, :][y == 0], x[1, :][y == 0], color=\"red\", label=\"y = 0\")\n",
|
|
"plt.scatter(x[0, :][y == 1], x[1, :][y == 1], color=\"blue\", label=\"y = 1\")\n",
|
|
"\n",
|
|
"plt.legend()\n",
|
|
"\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8b9e393c",
|
|
"metadata": {},
|
|
"source": [
|
|
"we want to create a **classifier**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8bb1768a",
|
|
"metadata": {},
|
|
"source": [
|
|
"first, set a target function\n",
|
|
"\n",
|
|
"$$L(w) = \\frac{1}{2}\\sum_{i}^{N}{| f(x_i; w) - y_i|^2}$$\n",
|
|
"\n",
|
|
"then how to get $w^* = \\text{argmin} L(w) $: $w$ to get lowest $L(w)$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "18da07bf",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Learning (Optimization)\n",
|
|
"\n",
|
|
"$$f(x; w) = w^Tx$$\n",
|
|
"\n",
|
|
"$$L(w) = \\frac{1}{2}\\sum{|w^Tx_i - y_i|^2} \\\\=\\frac{1}{2} (w^Tx - y)^2\\\\= \\frac{1}{2}{(w^Tx - y)}^T{(w^Tx - y)}$$\n",
|
|
"\n",
|
|
"$$\\frac{dL}{dw} = x(x^Tw-y)$$\n",
|
|
"\n",
|
|
"in order to minimize $L(w)$, \n",
|
|
"$$\\left.\\frac{dL}{dw}\\right|_{w=w^*} = 0$$\n",
|
|
"\n",
|
|
"therefore,\n",
|
|
"\n",
|
|
"$$\\frac{dL}{dw} = $$\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bcb16cc8",
|
|
"metadata": {},
|
|
"source": [
|
|
"Analytic Method\n",
|
|
"\n",
|
|
"$$\\frac{dL}{dw} = 0 \\to$$\n",
|
|
"\n",
|
|
"Numerical Method\n",
|
|
"\n",
|
|
"* gradient descent\n",
|
|
"\n",
|
|
"$$w_{t+1} = w_t - \\eta \\left . \\frac{dL}{dw} \\right |_{w=w_t}$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "71cb6971",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6577fc8a",
|
|
"metadata": {},
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "2025-02-AI (3.12.11)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.12.11"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|