{ "cells": [ { "cell_type": "code", "execution_count": 37, "id": "cce80b24", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "plt.rcParams[\"font.family\"] = \"D2Coding\"" ] }, { "cell_type": "markdown", "id": "d2df6767", "metadata": {}, "source": [ "# Geometric Interpretation of LR\n", "\n", "## Definition\n", "$$X, W\\in \\R^D$$\n", "\n", "$$W^TX = \\lVert W \\rVert \\lVert X \\rVert \\cos \\theta$$\n", "\n", "$$\\lVert X-W \\rVert^2 = \\lVert X \\rVert ^2 + \\lVert W \\rVert - 2 W^TX$$\n", "\n", "\n", "$$\\lVert X \\rVert \\cos \\theta = \\frac{W^T X} {\\lVert W \\rVert} $$\n", "it means length of projected vector." ] }, { "cell_type": "markdown", "id": "28627e12", "metadata": {}, "source": [ "$\\begin{cases}\n", "W^TX_i \\geq b &\\to y_i = 1\\\\\n", "W_TX_i < b &\\to y_i = 2\n", "\\end{cases}$" ] }, { "cell_type": "markdown", "id": "7d9d80da", "metadata": {}, "source": [ "$\\sum \\frac{1}{2} \\left\\Vert f(x_i;w) - y_i\\right\\Vert$" ] }, { "cell_type": "markdown", "id": "d0086883", "metadata": { "vscode": { "languageId": "powershell" } }, "source": [ "## Likelihood\n", "\n" ] }, { "cell_type": "code", "execution_count": 38, "id": "81be7a4c", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAAGhCAYAAACtc4RMAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjYsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvq6yFwwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAJxpJREFUeJzt3X9MXfX9x/H3LUaYWmgH1baANKRrgxZ1Moc4ZW3M/lmy4CqprgYaky3WdAY0YYjR4LKY2qiVus5lMfk2dbXEQVpr5mbWJiX2H/6ZzjVlOof9QSmoUIUWC9X2fvP+XC/C7aVw7z33nvM55/lIrpd77rmXA4d6Xvfz4/0JhcPhsAAAAHjcPLcPAAAAYC4ILQAAwAqEFgAAYAVCCwAAsAKhBQAAWIHQAgAArEBoAQAAViC0AAAAKxBaAACAFQgtAADAn6Fl//79smTJEqmtrZ3T/r29vVJVVSU5OTlSXV0tJ06cSOY4AQBAwCUUWh566CHZsGGDrFq1as6vqaurk7KyMhkcHJSKigrzegAAgLSGlp/+9Kfy0UcfyY9+9KM57a+tKt3d3bJlyxZZsGCBLF++XLq6uqS/vz/hAwUAAMF2RSI719TUJPTmR44ckaKiIlm0aJEJKlu3bpWCggI5fPiwFBYWxn3NxMSEuUVdvHhRTp8+Lfn5+RIKhRL6/gAAwB3hcFjOnDkjS5culXnz5mU+tCQqGjbUpk2bpLW1VV566SWzfSabN2+W3/72t+k8LAAAkCF9fX2mAcPzoUVTVlZWlnR0dMjY2JjU19fLiy++eNnXtLS0yGOPPTb5eGRkRK6//nrzQ+fm5qbzcAEAgENGR0eluLhY5s+f79Rbpje05OXlyalTp6S5uVkOHDhgtmkry8KFC2d8TXZ2trnF0sBCaAEAwC5ODu1wtE6LBpSpVq5cKQMDA7Jx40YpLS2VoaEh02KSyOwjAACAhEPLuXPnzNTls2fPmsGy0a+jY1F0cG17e/vk/itWrJDy8nIZHx+X4eFhaWpqksrKStNcBAAAkLbQ8vrrr5vCci+88IL89a9/NV8///zz5rnFixeb7hudHTTVrl27ZO/evWb0cE9Pj+zcuTOhAwQAAFChsI6W9fhAHh0bowNyGdMCAIAd0nH9Zu0hAABgBUILAACwAqEFAABYgdACAACsQGgBAABWILQAAAArpLWMPwDY5sIFkUOHRAYGRJYsEbnrLpGsLLePCoAitADAN/bsEWloEDl58tttujjttm0ia9e6eWQAFN1DAPBNYKmtnR5YVH9/ZLs+D8BdhBYAgaddQtrCEq8+eHRbY2NkPwDuIbQACDwdwxLbwhIbXPr6IvsBcA+hBUDg6aBbJ/cDkB6EFgCBp7OEnNwPQHoQWgAEnk5r1llCoVD853V7cXFkPwDuIbQACDytw6LTmlVscIk+bmujXgvgNkILAEikDktnp0hh4fTt2gKj26nTAriP4nIA8A0NJjU1VMQFvIrQAgBTaEBZvdrtowAQD91DAADACoQWAABgBUILAACwAqEFAABYgYG4gI/pAn/MhAHgF4QWwKf27ImsXDx1IUCtOaJF1Kg5AsBGdA8BPg0stbWXrlzc3x/Zrs8DgG0ILYAPu4S0hSUcvvS56LbGxsh+AGATQgvgMzqGJbaFJTa49PVF9gMAmxBaAJ/RQbdO7gcAXkFoAXxGZwk5uR8AeAWhBfAZndass4RCofjP6/bi4sh+AGATQgvgM1qHRac1q9jgEn3c1ka9FgD2IbQAPqR1WDo7RQoLp2/XFhjdTp0WADaiuBzgUxpMamqoiAvAPwgtgI9pQFm92u2jAABn0D0EAACsQGgBAABWILQAAAArEFoAAIAVCC0AAMAKhBYAAGAFQgsAALACoQUAAFiB0AIAAKxAaAEAAFYgtAAAACsQWgAAgBUILQAAwAqs8gzAURcuiBw6JDIwILJkichdd0VWmwaAVBFaADhmzx6RhgaRkye/3VZUJLJtm8jatW4eGQA/oHsIgGOBpbZ2emBR/f2R7fo8AKSC0ALAkS4hbWEJhy99LrqtsTGyHwAki9ACIGU6hiW2hSU2uPT1RfYDgGQRWgCkTAfdOrkfAMRDaAGQMp0l5OR+ABAPoQVAynRas84SCoXiP6/bi4sj+wFAsggtAFKmdVh0WrOKDS7Rx21t1GsBkBpCCwBHaB2Wzk6RwsLp27UFRrdTpwVAqiguB8AxGkxqaqiICyA9CC0AHKUBZfVqt48CgB/RPQQAAKxAaAEAAFYgtAAAACsQWgAAgD9DS29vr1RVVUlOTo5UV1fLiRMnZn3NwYMH5aabbpKrrrpKbr75ZvMYAAAgraGlrq5OysrKZHBwUCoqKmTDhg2X3T8cDst9990n999/v3z22Wfy4IMPyr333ivnz59P9FsjYHRF4K4ukfb2yD0rBANAsCUUWrRVpbu7W7Zs2SILFiyQ5cuXS1dXl/T398/4Gg0qevvlL38pV199tfzqV7+Szz///LKvAfbsEVm2TGTNGpH16yP3+li3AwCCKaHQcuTIESkqKpJFixaZ0LF161YpKCiQw4cPz/ga3ffGG2+UHTt2yJdffimvvPKKrFy5Uop1IZI4JiYmZHR0dNoNwaLBpLZW5OTJ6ds15+p2gkvw0OoGIOHQcvr0acnPzzdfb9q0SVpbW6WkpMRsn0koFJLXXntNnnnmGdPS8uSTT8ru3bvliivi17XbvHmz5OXlTd5mCjfwJ70YNTRot+Klz0W3NTZy0QoSWt0AJBVadHxKVlaWdHR0yNjYmNTX18uFWa4eX331lRnT0tzcLF988YUJJevWrZNz587F3b+lpUVGRkYmb319fYkcIiyn5d9jW1hig4v+Seh+8D9a3QAkHVq05ePUqVMmgPzpT38y27SVZeHChTO+RmcKnTlzRp544gnz+kceecS0vrz99ttx98/Ozpbc3NxpNwSHrlfj5H6wF61uAFIKLToWZWBgQDZu3CilpaUyNDRkWkJWrVo1uY+GmqnOnj1rgogGlSid+qzbgVi6wJ6T+8FetLoBSCm0rFixQsrLy2V8fFyGh4elqalJKisrJ8edaNdPYWGhtOtouW/cdtttJsi8/PLLpktp165d8sEHH8jtt9+eyLdGQOiKwEVFOhYq/vO6Xf/cdD/4G61uAFKu06KhY+/evbJ06VLp6emRnTt3Tj63ePFi052jM4qiNND85S9/kT/+8Y9mJpEGGx2I+73vfS/Rb42ArBC8bVvk69jgEn3c1hbZD/5GqxuAWKGwjq71MJ3yrGNhdFAu41uCQwdY6niGqd0D2sKigWXtWjePDJmiY1V0lpAOuo33fykNsdoqd/QoIRYIyvU7/rxjwGUaTGpqIuMVtPlfP01rlxAXp+C1uuksIQ0oU4MLrW5AMBFa4Fl6MVq92u2jgNvhtbPz0lY3bWGh1Q0IHkILAE+j1Q1AFKEFgOfR6gYgqdlDAAAAbiC0AAAAKxBaAACAFQgtAADACoQWAABgBUILAACwAlOeAThefp+aKgDSgdACIK1rRmn1Wi3HT/VaAKmiewiAY4FF1wmaGliULnio2/V5AEgFoQWAI11C2sISbzXm6LbGxsh+AJAsQguAlOkYltgWltjg0tcX2Q8AkkVoAZAyHXTr5H4AEA+hBUDKdJaQk/sBQDyEFgAp02nNOksoFIr/vG4vLo7sBwDJIrQASJnWYdFpzSo2uEQft7VRrwVAaggtgMN0hkxXl0h7e+Q+KDNmtA5LZ6dIYeH07doCo9up0wIgVRSXAxwU9OJq+jPW1FARF0B6hMLheJUVvGN0dFTy8vJkZGREcnNz3T4cYNbiarH/oqLdI7Q2AAiS0TRcv+keAhxAcTUASD9CC+AAiqsBQPoRWgAHUFwNANKP0AI4gOJqAJB+hBbAARRXA4D0I7QADqC4GgCkH6EFcAjF1QAgvSguBziI4moAkD6EFsBhGlBWr3b7KADAf+geAgAAViC0AAAAKxBaAACAFQgtAADACoQWAABgBUILAACwAqEFAABYgdACAACsQGgBAABWILQAAAArEFoAAIAVCC0AAMAKhBYAAGAFVnkGEFgXLogcOiQyMCCyZInIXXdFVukG4E2EFgCBtGePSEODyMmT324rKhLZtk1k7Vo3jwzATOgeAhDIwFJbOz2wqP7+yHZ9HoD3EFoABK5LSFtYwuFLn4tua2yM7AfAWwgtAAJFx7DEtrDEBpe+vsh+ALyF0AIgUHTQrZP7AcgcQguAQNFZQk7uByBzCC0AAkWnNessoVAo/vO6vbg4sh8AbyG0AAgUrcOi05pVbHCJPm5ro14L4EWEFgCBo3VYOjtFCgunb9cWGN1OnRbAmyguByCQNJjU1FARdyoqBMPrCC0AAksvyKtXu30U3kCFYNiA7iEACDgqBMMWhBYACDAqBMMmhBYACDAqBMMmjGkJEAbZAYhFhWDYhNASEAyyAxAPFYJhE7qHAiCdg+y09aarS6S9PXJPvzdgFyoEwyaEFp9L5yA7DTvLlomsWSOyfn3kXh8z0wCwBxWC4evQ0tvbK1VVVZKTkyPV1dVy4sSJWV9z8eJFeeqpp2ThwoWyYMECeeihh5I9XnhkkB1TJAH/oEIwfBta6urqpKysTAYHB6WiokI2bNgw62u2b98ub7/9thw+fFiOHTtm3gP2DrJjiiTgPxpMjh0TOXhQZPfuyP3RowQWWDwQV1tVuru7Zd++fabFZPny5dLW1ib9/f1SGBvRp9i6davs3btXijS2i8idd96Z+pHDtUF2ibTeUG0UsAcVguGrlpYjR46Y4LFo0SITVDSMFBQUmBaUy3UnjY6OyquvvipXXXWVaaV57733Ztx/YmLC7D/1Bm8NsmOKJADA86Hl9OnTkp+fb77etGmTtLa2SklJidk+k48//lg+//xzOXv2rGmpueeee2TdunVmnEs8mzdvlry8vMlbsV5R4alBdkyRBAB4PrSEw2HJysqSjo4OGRsbk/r6erkwy8AFbTmJhhFtlXnyySfl+PHjptUmnpaWFhkZGZm89Wk/Azw1yI4pknAaU+cBOD6mRVs+Tp06Jc3NzXLgwAGzTVtZdFbQTK655hrTLaSBRV199dWmtebTTz+Nu392dra5wVkaTGpqnKmIG2290VlCGlCmDshliiQSReFDAGlpaVm5cqUMDAzIxo0bpbS0VIaGhkxLyKpVqyb30VAz1Q033CDnzp0z+6ozZ86Yr5cuXZrIt4aDg+x+8YvIfSqhgimScAJT5wGkLbSsWLFCysvLZXx8XIaHh6WpqUkqKysnx51oF5DOImrXNt5vXHvttWa20NNPP21aZfT+xhtvNANyYTemSCIVTJ0HkPY6Lbt27TLTl7WlpKenR3bu3Dn53OLFiyU3N3eyKyhqx44d8s9//tO85tChQ9KpH8XhC0623iBYWF0YQKJCYR1d62E65VnH0uigXA1EAPxBG2R1+YfZaCuehmIAdknH9Zu1hwC4gqnzABJFaAHgCqbOA0gUoQWAK1hdGECiCC0AXMPUeQBpKy4HAF4ufAjA3wgtAFzH6sIA5oLQAsDTtLgcrTAAFKEFgGexLhH8hACeOgbiAvAk1iWCn+jf67JlImvWRIoq6r0+5u84MYQWAJ7DukTwEwK4cwgtADyHdYngFwRwZxFaAHiO9vk7uR/gFgK4swgtADyHdYngFwRwZzF7CIBn1yXSPv94zepa5l+fZ10ieH1WDwHcWbS0APAc1iWCX2b1sDCoswgtADzJ5nWJ9JN7V5dIe3vknkGWwZ3VQwB3Vigcjtf46h2jo6OSl5cnIyMjkpub6/bhAMgw2wpyURDPnr8rbVGZaZBstAvy6FFn/t7i/V1oC4sGFr/+XYym4fpNaAEAhz+5x/5fNfqJ2ustREGiLWDaFTSbgwedWxfLtgDuxes3A3EBIAP1ODS4aD0OXdHazxcqW7gxq4eFQVPHmBYAcAD1OOzCrB47EVoAwAHU47ALs3rsRGgBAAfwyd0uzOqxE6EFABzAJ3f72DytPqgYiAsADn5y19lDGlCmDsjlk7t3aTDRwdFBmtVjM0ILkIKgTWHE3D65x6vT4ud6HLZjVo89CC1Akigihnj45A6kD8XlgCRQRAwAMn/9ZiAu4HARMaVFxFhvBgCcRWgBEkQRMQBwB6EFSBBFxADAHYQWIEEUEQMAdxBagARRRAwA3EFoARJE+W8AcAehBUgC5b8BIPMoLgckiSJiAJBZhBYgBZT/BoDMoXsIAABYgZYWwGUsuggAc0NoAVzEootwAsEXQUH3EODyoouxSwL090e26/PAbPTvZNkykTVrRNavj9zrY/5+4Ees8gy49MlYLywzrWGk9V60xeXoUT4xY2asNj4dLU7ewirPgE+w6CJSxWrj09HiFAyEFsAFLLqIVBF8v0VXa3AQWgAXsOgiUkXwjaDFKVgILYALWHTRP/Ri2NUl0t4euc/UxZHgG0GLU7AQWgAXsOiiP7g5joLgG0GLU7AQWgCXsOii3dweR0HwjaDFKViY8gy4zA/TNP3wM9g6ZT1egUJtYdHAEoTg29Ehsm7d5ffR3wflA/xx/Sa0AEhJEKv66tgV7QqazcGDmVlQM2ihca7hcWqw0dYv2H/9pow/AMeLm0W7SPzazeW1cRRBXW18tkG4UQUFzn7foIZEL2BMC4CkBHmqKeMoghseKWLnLkILgKQEeaopM3eCGR7dHnwNQgsAn3SRZBIzd4IXHoPcsuglhBYASQl6FwlT1oMVHoPcsuglhBYASaGLJBJMjh2LzBLavTtyr1NrCSz+C49Bbln0EmYPAUjpU6725WtAmdpsHqQuEr/O3LFphowGk5qa9B5v0FsWvYI6LQBSEvTiZn4UxNo7c60Jo4Nu4101M1lQ0BYUlyO0AJ5k06dyJFd7J9p6FuTxOtHfjYrXshjk3008hBZCCwAEYnkCr6Jlce6oiAsASJtEZsj4cRzPXFoOMzF+BjMjtAAADGbIzG08j18HX9uAKc8AACPoM2SoeOt9hBYAgAS99g4Vb30aWnp7e6WqqkpycnKkurpaTpw4MefXvvPOOxIKhWT79u2JflsASIpeZLq6RNrbI/dcdGYW5OUJqHjr09BSV1cnZWVlMjg4KBUVFbJhw4Y5ve7rr7+WTZs2yXXXXZfMcQJAwliRN3FBXZ6A8Tw+HIirrSrd3d2yb98+WbBggSxfvlza2tqkv79fCmP/wmPoft///vdnbZmZmJgwt6lTpgDAqXoj0fEJfr4ApyqIM2SCPp7Hly0tR44ckaKiIlm0aJEJKlu3bpWCggI5fPjwZV+n+77wwgvy7LPPzvo9Nm/ebOZ1R2/F2oEKAAlgfELqojNkfvGLyL2fA0vQx/P4NrScPn1a8vPzzdfa1dPa2iolJSVm++U8+uij0tDQIEuXLp31e7S0tJhCNNFbn3YiAkACGJ+ARAV5PI9vQ4sWz83KypKOjg4ZGxuT+vp6uTDLR5X9+/fLu+++a4LLXGRnZ5vKeVNvAJAIxicgGUEdz+PbMS3aXXPq1Clpbm6WAwcOmG3ayrJw4cIZX/PWW2/JwMCAaZGJ7q8h5v3335dXXnkl1eMHgEswPgHJCuJ4HpsktPbQf//7X1m5cqVs2bJFfvOb38jQ0JBce+21cvz48cmxJxpqpnYDnTlzxrTKRK1du1Z+9rOfycMPP2wG886GtYcAJIoVeQH3peP6nVD30IoVK6S8vFzGx8dleHhYmpqapLKycjKw6CBanUXUrgURvjF//nxZvHjx5O3KK6802+YSWAAgGYxPAPwp4Totu3btkr1795rWlJ6eHtm5c+fkcxpKNE3pjCIAcBPjE4CAdw+5wZbuocutCgrAPfzbBPxz/WaV5wytCgrAHazIC/gHCyamiFVBAQDIDEJLCqi6CQBA5hBaUkDVTQAAMofQkgKqbgIAkDmElhRQdRMAgMxh9pADq4LOVnWTVUGB4GGqNeA8WlpSQNVNAPHorEFdRmDNGpH16yP3+pjZhEBqCC0pouomgKkogwCkDxVxHUJTMIDoQo0zzSpkoUYEySgVcb2LqpsAEimDwP8vgMTRPQQADqEMApBehBYAcAhlEID0IrQAgMNlEGJnE0bp9uJiyiAAySK0AIBDKIMApBehBQAcRBkEIH2YPQT4HNPxM0+DSU0Nv3fAaYQWwMe0kFlDw/RpuPqJX7sw+MSfXpRBAJxH9xDgU1RmBeA3hBbAp11C2sISr951dFtjY2Q/ALAFoQUIeGVWALAFoQXwISqzAvAjQgvgQ1RmBeBHhBbAh6jMCsCPCC2AD1GZFYAfEVoAn6IyKwC/obgc4GNUZgXgJ4QWwOeozArALwgt8DTWzQEARBFa4FmsmwMAmIqBuPAk1s0BAMQitMBzWDcHABAPoQWew7o5AIB4CC3wHNbNAQDEQ2iB57BuDgAgHkILPId1cwAA8RBa4DmsmwMAiIfQAk9i3RwAQCyKy8GzWDcnM6g6DMAWhBZ4GuvmpBdVhwHYhO4hIKCoOgzANoQWIICoOgzARoQWIICoOgzARoQWIICoOgzARoQWIICoOgzARoQWIICoOgzARoQWIICoOgzARoQWIKCoOgzANhSXAwKMqsMAbEJoAQKOqsMAbEH3EAAAsAKhBQAAWIHQAgAArEBoAQAAViC0AAAAKxBaAACAFQgtAADACoQWAABgBYrLAYjrwgXnKuU6+V4AgovQAuASe/aINDSInDw5fU0iXWQx0TWJnHwvAMFG9xCAS0JGbe30kKH6+yPb9Xk33gsAQuFwOCweNjo6Knl5eTIyMiK5ubluHw7ga9qNs2zZpSEjKhSKtJIcPTp7946T7wXAPum4ftPSAmCSjjuZKWQo/YjT1xfZL5PvBQBJhZbe3l6pqqqSnJwcqa6ulhMnTlx2//Pnz8vjjz8uy5cvl/nz58vdd98tH374Ib99wIN0oKxT+zn5XqnSVp+uLpH29si9PgYQgNBSV1cnZWVlMjg4KBUVFbJhw4bL7n/69Glze/PNN81rbr31VnnggQdSOWYAaaIze5zaz8n3SoWOm9FuqjVrRNavj9zrY8bTAD4f06KtKsuWLZNPPvlEFi1aJH/4wx/k17/+tZw8eVIKCwvn9B7/+c9/ZNWqVaYFJitOR/bExIS5Te0TKy4uZkwLkAHRcSg6UDbe/xmSGdPixHslKzoQOPb76/dWnZ3MYAJ8O6blyJEjUlRUZAJLf3+/bN26VQoKCuTw4cNzfo/jx4/L4sWL4wYWtXnzZvNDRm8aWABkhv6z1KnIUy/sUdHHbW1zCxlOvlcyNDTpVOt4gSm6rbGRriLAJgmFFu3myc/PN19v2rRJWltbpaSkxGyfq+eff14efPDBGZ9vaWkxqSx669ORegAyRlsetAUitvFUW0USbZlw8r0SxUBgIODF5bQnSVtIOjo6ZGxsTOrr6+XFF1+c8+u3bdtmWlreeOONGffJzs42NwDu0TBRU+NMFVsn3ysRXhoIDMCF0KLdNadOnZLm5mY5cOCA2aatLAsXLpz1td3d3fLUU09JV1eXXHPNNckfMYCM0FCxerX33muuvDIQGIBL3UMrV66UgYEB2bhxo5SWlsrQ0JDpvtGBtVEaamLpfuvWrZMtW7aY2UMAkG7amqPdULHjaaJ0uw6Z0/0A+DC0rFixQsrLy2V8fFyGh4elqalJKisrJwfL6iBanUXUrsUQvnHx4kVZv3693HHHHfLwww87/xMAgAcHAgPwQJ2WXbt2yd69e2Xp0qXS09MjO3funHxOZwXptCadUTR1mvT+/fvl9ddfl1AoNHnTbiIASCc3BwIDcB5rDwHwPZ3WnOmBwEDQjabh+p3QQFwAsJEbA4EBOI8FEwEAgBUILQAAwAqEFgAAYAVCCwAAsAKhBQAAWIHZQwBcwTRkAIkitADIuD17RBoapq/CrAXftIItBd8AzITuIQAJtY5oMWtdqUPv9XEygaW2dnpgUf39ke36PADEQ2gBMCcaJpYtE1mzRmT9+si9Pk4kZGjI0RaWeHW4o9saG5MLQwD8j9DiQ058GgbS0TqiY1hi3yM2uPT1RfYDgFiEFp9x4tMwkK7WER10Oxdz3Q9AsBBafISxAkgHJ1tHdJbQXMx1PwDBQmjxCcYKIF2cbB3Rac06SygUiv+8bi8ujuwHALEILT7BWAGki5OtI1qHRac1q9jgEn3c1ka9FgDxEVp8grECSBenW0e0Dktnp0hh4fTt+j10O3VaAMyE4nI+wVgBpEu0dUTHRWlAmdoFmWzriAaTmhoq4gJITCgcjjcKwjtGR0clLy9PRkZGJDc31+3D8Swdq6KzhHTQbbwzqhcX/SR79CgXhiByomR+vCq22sKigYXWEQCZuH7T0uIT6fg0DH9wqmQ+rSMA3EZLi8/waRjxpsHH/iuPBlnGkACw6fpNaPEhVs/F1C7DmWaV0WUIIJ3oHsKc6AVo9Wq3jwI2TYPn7wWADZjyDPgU0+AB+A2hBfAppsED8BtCC+BTlMwH4DeEFsCnKJkPwG8ILYCPUTIfgJ8wewjwOYrCAfALQgsQAEyDB+AHdA8BAAArEFoAAIAVCC0AAMAKhBYAAGAFQgsAALACoQUAAFiB0AIAAKxAaAEAAFYgtAAAACsQWgAAgBUILQAAwAqEFgAAYAVCCwAAsAKhBQAAWIHQAgAArEBoAQAAViC0AAAAK1zh9gEASI8LF0QOHRIZGBBZskTkrrtEsrLcPioASB6hBfChPXtEGhpETp78dltRkci2bSJr17p5ZACQPLqHAB8Gltra6YFF9fdHtuvzAGAjQgvgI9olpC0s4fClz0W3NTZG9gMA2xBaAB/RMSyxLSyxwaWvL7IfANiG0AL4iA66dXI/APASQgvgIzpLyMn9AMBLCC2Aj+i0Zp0lFArFf163FxdH9gMA2xBaAB/ROiw6rVnFBpfo47Y26rUAsBOhBfAZrcPS2SlSWDh9u7bA6HbqtACwFcXlAB/SYFJTQ0VcAP5CaAF8SgPK6tVuHwUAOIfuIQAAYAVCCwAAsAKhBQAAWIHQAgAArEBoAQAA/gwtvb29UlVVJTk5OVJdXS0nTpxIy2sAAABSCi11dXVSVlYmg4ODUlFRIRs2bEjLawAAAJKu06ItJN3d3bJv3z5ZsGCBLF++XNra2qS/v18KY8tvJvmaiYkJc4saGRkx96Ojo4kcKgAAcFH0uh0Oh51703AC/va3v4WLi4vN1ydPngyXlpaGCwoKwn//+98de01ra6v+dNy4cePGjRs3H9x6e3vDTkmopeX06dOSn59vvt60aZO0trbKSy+9ZLY79ZqWlhZ57LHHJh9/8cUXUlJSYlps8vLyEjlcpCE1FxcXS19fn+Tm5rp9OIHGufAOzoW3cD68Q3tKrr/+evnud7/r2HsmFFq0iScrK0s6OjpkbGxM6uvr5cUXX3T0NdnZ2eYWSwMLf4DeoOeBc+ENnAvv4Fx4C+fDO+bNm+dOaNHgcOrUKWlubpYDBw6YbdpisnDhQkdfAwAAECuh+LNy5UoZGBiQjRs3SmlpqQwNDZkmuFWrVk3uowEl0dcAAAA4GlpWrFgh5eXlMj4+LsPDw9LU1CSVlZWm/1Bt3rzZzAhqb2+f82tmo11FOg4mXpcRMotz4R2cC+/gXHgL58Pf5yKko3ETecG///1vU2elp6dHbrnlFvnzn/9sgonasWOHNDY2Smdnp/zkJz+Z02sAAADSEloAAADcwNpDAADACoQWAABgBUILAACwAqEFAABYwfXQ0tvbK1VVVZKTkyPV1dWmXH86XgNx/Pd6/vx5efzxx80imPPnz5e7775bPvzww4wdr5+l8jf+zjvvSCgUku3bt6f1GIMimXNx8eJFeeqpp0wRTV0o9qGHHsrIsQZBMufj4MGDctNNN8lVV10lN998s3mM1O3fv1+WLFkitbW1c9rfiWu366Glrq5OysrKZHBwUCoqKszU6HS8BuL471UrG+vtzTffNK+59dZb5YEHHsjY8fpZsn/jX3/9tVnj67rrrkv7MQZFMudCA+Pbb78thw8flmPHjpn3gDvnQyfI3nfffXL//ffLZ599Jg8++KDce++95kMXkqdBXH/3iRSKdeTaHXbR8ePHw6FQKPzpp5+ax9u3bzcrQupq0E6+BrNz4vfa09MTnjdvXvjrr79O45H6Xyrn4rnnngvX1dWFf/zjH4d///vfZ+Bo/S3Zc1FSUhJ+9913M3SUwZHM+fjkk0/MPnqvzp49ax5//PHHGTtuP3rjjTfM77K1tTV87733zrq/U9duV1tajhw5IkVFRbJo0SLp7++XrVu3SkFBgfl04uRrMDsnfq/Hjx+XxYsXmwUykflzofu+8MIL8uyzz2bsWP0umXOhTeC60vCrr75quiP0k+V7772X0eP2q2TOh+574403muKnX375pbzyyitmeZm5VmVHfDU1NXL11VfLXDl17XY1tGjXQn5+vvlam7S13G9JSYnZ7uRrMDsnfq/PP/+8aXqFO+fi0UcflYaGBlm6dGmGjtT/kjkXH3/8sXz++edy9uxZ02d/zz33yLp168w4F2T+fOj4rtdee02eeeYZc5F98sknZffu3XLFFQmtF4wUOXXtdjW0aF+jfirv6OiQsbExqa+vlwsXLjj+Gswu1d/rtm3bTEuLDsxF5s+FDoh79913TXCBu+diYmJici02/SSpF0n9t6GfNJH58/HVV1+ZMS3Nzc3yxRdfmPOiIfLcuXMZO26IY9duV6NmXl6eWRVa/5gOHDhgtmnq0hH3Tr4Gs0vl99rd3W1mSnR1dck111yTgaP1t2TOxVtvvWVWU9dPLtH9NcS8//77pjkcmTsX+m9Au4U0sCj9dK+fMD/99NOMHbdfJXM+dKbQmTNn5IknnjCtLo888oi89NJLZqD0z3/+8wwefbDlOXTtdrWlRfsV9X+0GzdulNLSUhkaGpK+vr5po5H1h0z0NcjMuVC6n35q2bJli5k9BHfOxe9+9zszluJf//qXuf3gBz+QlpYWee6551z4CYJ9Lm644QbzKV73VXrB1K/ptnPnfGg3na4yrIElSkOlbkf6pOva7Wpo0ZWey8vLZXx8XIaHh6WpqUkqKysnB0hpM15hYaG0t7fP+TXI3LnQPvr169fLHXfcIQ8//LCLR+8vyZwLrZOjg6CjtyuvvNJs0xohyOy5uPbaa+XOO++Up59+2nyS1HsdCKoDcpH583HbbbeZC+jLL79suiV27dolH3zwgdx+++0u/iT2O3funJm6rOFPu0SjX6f92h122fvvvx++5ZZbwldeeWX4hz/8YfjDDz+cfO7//u//wrm5ueF//OMfc34NMncujh49aqasxd4OHjzo0k/gH8n8u5iKKc/unov//e9/4dtvvz2cnZ0dvu2228IfffSRC0fuT8mcj3379oVXrVoV/s53vhO+4YYbwp2dnS4cub/s2LHjkv/36/TndF+7Q/ofpxMYAACA01yviAsAADAXhBYAAGAFQgsAALACoQUAAFiB0AIAAKxAaAEAAFYgtAAAACsQWgAAgBUILQAAwAqEFgAAIDb4fwRNjdbIN8icAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "np.random.seed(7)\n", "\n", "#y = np.random.randint(0, 2, size=20)\n", "\n", "#x = np.random.normal(0, 0.08, size=(2, 20)) + np.vstack([y / 2 + 0.25, y / 2 + 0.25])\n", "\n", "\n", "x = np.random.normal(0.5, 0.2, size=(40, 2), )\n", "plt.xlim(0, 1)\n", "plt.ylim(0, 1)\n", "\n", "plt.scatter(x[:, 0], x[:, 1], color='blue')\n", "#plt.scatter(x[0, :][y == 0], x[1, :][y == 0], color=\"red\", label=\"y = 0\")\n", "#plt.scatter(x[0, :][y == 1], x[1, :][y == 1], color=\"blue\", label=\"y = 1\")\n", "\n", "#plt.legend()\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "df8f9d0b", "metadata": {}, "source": [ "Maximize the probability" ] }, { "cell_type": "code", "execution_count": null, "id": "417911ef", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 40, "id": "620ef30c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MLE estimated mu: [0.50995502 0.46521124]\n", "MLE estimated sigma: [0.19530993 0.22297348]\n" ] } ], "source": [ "# X가 2차원 데이터이므로 각 차원별로 평균(mu)과 표준편차(sigma)를 추정합니다.\n", "# MLE: mu = sample mean, sigma = sample std (biased=False)\n", "mu_mle = np.mean(x, axis=0)\n", "sigma_mle = np.std(x, axis=0, ddof=0) # ddof=0 for MLE\n", "\n", "print(\"MLE estimated mu:\", mu_mle)\n", "print(\"MLE estimated sigma:\", sigma_mle)" ] }, { "cell_type": "markdown", "id": "c51e029e", "metadata": {}, "source": [ "to assign high probability to the observed data" ] }, { "cell_type": "markdown", "id": "36b7c733", "metadata": {}, "source": [ "## Logistic Regression RV\n", "\n", "$y_i \\in \\set{0, 1}$\n", "$x_i \\in \\R^D$\n", "\n", "logistic regression function\n", "$$f(X;W) = \\frac{1}{1+\\exp(-W^TX)}$$\n", "\n", "$$\\text{max}\\; L(w) = \\prod_{i=1}^{N}f(x_i)^{y_i} (1-f(x_i))^{1-y_i}$$\n", "where\n", "* $f(x_i)$: prob of choosing $y_i = 1$\n", "* $1-f(x_i)$: prob of choosing $y_i = 0$\n", "\n", "and then update like\n", "$$w_{t+1} = w_t + \\eta \\left . \\frac{dL}{dw}\\right\\vert_{w=w_t}$$\n", "\n", "how to get:\n", "\n", "$$w^* = \\argmax L(w) = \\argmax \\ln L(w)$$\n", "\n", "because function $\\ln$ is monotonically increasing function\n", "\n", "so $\\ln L(w) = \\sum_{i=1}^{N}{(y_i\\ln f(x_i) + (1-y_i)\\ln(1-f(x_i)) )}$\n", "\n", "미분하면 된다" ] }, { "cell_type": "code", "execution_count": null, "id": "6b19eb64", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "cba17457", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "id": "0b189852", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "2025-02-AI (3.12.11)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.11" } }, "nbformat": 4, "nbformat_minor": 5 }