통계 (18) 썸네일형 리스트형 Measure theoretic description of change-of-variables 1. SettingsTwo measurable spaces:\[(X,\mathcal{A},\mu)\quad\text{and}\quad (Y,\mathcal{B},\mu_T)\]Consider a transform \(T\) such that:\[T:X\rightarrow Y\]where T is a measurable map. i.e., \(T^{-1}(B)\in \mathcal{A}\quad \forall B\in\mathcal{B}\).\(T\) induces a pushforward measure \(\mu_T\):\[\mu_T(B)=\mu(T^{-1}(B))\]2. Change-of-variablesLet \(\nu\) and \(\mu_T\) are measures defined on \(Y\).. Measure theoretic description of KL divergence In this writing, I consider the KL divergence with measure theoretic approach. Suppose \(P\) and \(Q\) are probability measures on a measurable space \((\Omega, \mathcal{F})\), and \(P\ll Q\). That is, \(P\) is absolutely continuous with respect to \(Q\). (Or, equivalently, \(Q\) dominates \(P\)) KL divergence is defined as below:\[\begin{aligned} \mathcal{D}_{KL}(P||Q)&:=E_P\left[\log \frac{dP}.. limsup <=> inf{sup} 원래 알던 정의:$$\limsup_n x_n := \lim_{n\rightarrow \infty} \sup_{k \geq n} x_k$$새로 발견한 정의:$$\limsup_n x_n := \inf_m \{ \sup_{n\geq m}x_n\}$$둘이 동치임을 증명:$$X_m = \sup_{n \geq m} x_n$$이라 하면 \(X_m\)은 \(X_{m+1} \subset X_m\) 이므로 decreasing set임.$$\Rightarrow \lim_{m\rightarrow \infty} X_m = \inf_m X_m$$$$\Rightarrow\lim_{m\rightarrow \infty} \sup_{n \geq m} x_n= \inf_m \{ \sup_{n\geq m}x_n\}$$ 마찬가지로 limin.. 급수 수렴 판정법 1. Partial Sum$$ \left\{S_n=\sum_{i=1}^n a_i\right\}_{n=1}^\infty\quad \mathrm{converges}\quad\rightarrow \quad \sum_{i=1}^n a_i\quad \mathrm{converges} $$$$\left\{S_n=\sum_{i=1}^n a_i\right\}_{n=1}^\infty\quad \mathrm{diverges}\quad\rightarrow \quad \sum_{i=1}^n a_i \quad \mathrm{diverges}$$2. Cauchy Criterion$$\forall \epsilon >0,\;\exists N \quad s.t.$$$$n>m>N\;\rightarrow\;\left|\sum_{i=m+1}.. 수열 수렴 판정법 1. Direct Method (epsilon-delta)$$\forall \epsilon > 0,\;\exists N \in \mathbb{Z}^+\quad s.t.$$$$n>N\;\rightarrow\;|a_n-c|2. Box Method(i) Bounded below (above)(ii) Monotone decreasing (increasing)\(\rightarrow\) \(a_n\) converges.3. Cauchy Criterion$$\forall \epsilon>0,\;\exists N \in \mathbb{Z}^+\quad s.t.$$$$m,n>N\;\rightarrow\;|a_m-a_n| LASSO, Ridge regression LASSO$$\min_{\beta}(y-X\beta)^T(y-X\beta)$$$$\mathrm{subject\;to}\quad||\beta||_1\leq t$$\(\beta_i\)가 0이 되는 걸 허용한다. 0이 됨으로써 variable selection도 되는 듯.Ridge regression$$\min_{\beta}(y-X\beta)^T(y-X\beta)$$$$\mathrm{subject\;to}\quad||\beta||_2= c$$ Lagrangian당연히 둘 다 constraint를 Lagrangian을 이용해 objective에 포함시킬 수 있다. Referenceshttps://en.wikipedia.org/wiki/Lasso_(statistics)https://en.wikipedia.org/.. 2d symmetric KL-divergence Implementation import numpy as np import matplotlib.pyplot as plt from mpl_toolkits.mplot3d import Axes3D def normal2d(mu:np.array, sigma): def normal2d_(x, y): x = np.array([x,y]) x = x.reshape((2,1)) I = np.eye(2) V = sigma**2*I V_inv = np.linalg.inv(V) mul = np.linalg.det(2*np.pi*V)**(-0.5) px = -0.5*(x-mu).T@V_inv@(x-mu) result = mul*np.exp(px[0]) return result[0] return normal2d_ def calcul.. Metric vs distance measure Metric은 symmetry를 만족해야함. d(x,y)=d(y,x) Distance measure는 symmetry를 만드시 만족할 필요 없음. ex) KL-divergence. 그러면 \(Metric \subset Distance \: Measure\)인가? Metric은 distance measure의 특수한 경우가 되는건가?? ------------------------------------------------------------------------ (2024.01.31 수정) "Although the KL divergence measures the “distance” between two distributions, it is not a distance measure. This is beca.. 이전 1 2 3 다음
