본문 바로가기

통계

(19)

Monotone Mapping A mapping

ϕ

is monotone if for all

x, y \in dom ϕ

(x - y)^{T} (ϕ (x) - ϕ (y)) \geq 0

Reference:Boyd, S. P., & Vandenberghe, L. (2004). Convex optimization. Cambridge university press.Khanh, P., Luong, H. C., Mordukhovich, B., & Tran, D. (2024). Fundamental convergence analysis of sharpness-aware minimization. Advances in Neural Information Processing Systems, 37, 13149-13182.

Measure theoretic description of change-of-variables 1. SettingsTwo measurable spaces:

(X, A, μ) and (Y, B, μ_{T})

Consider a transform

T

T : X \to Y

where T is a measurable map. i.e.,

T^{- 1} (B) \in A \forall B \in B

T

induces a pushforward measure

μ_{T}

μ_{T} (B) = μ (T^{- 1} (B))

2. Change-of-variablesLet

ν

μ_{T}

are measures defined on

Y

Measure theoretic description of KL divergence In this writing, I consider the KL divergence with measure theoretic approach. Suppose

P

Q

are probability measures on a measurable space

(Ω, F)

P ≪ Q

P

is absolutely continuous with respect to

Q

. (Or, equivalently,

Q

P

) KL divergence is defined as below:\[\begin{aligned} \mathcal{D}_{KL}(P||Q)&:=E_P\left[\log \frac{dP}..

limsup <=> inf{sup} 원래 알던 정의:

\underset{n}{lim sup} x_{n} := lim_{n \to \infty} sup_{k \geq n} x_{k}

새로 발견한 정의:

\underset{n}{lim sup} x_{n} := inf_{m} {sup_{n \geq m} x_{n}}

둘이 동치임을 증명:

X_{m} = sup_{n \geq m} x_{n}

X_{m}

X_{m + 1} \leq X_{m}

이므로 non-increasing sequence임. decreasing 하거나 그대로거나.

\Rightarrow lim_{m \to \infty} X_{m} = inf_{m} X_{m}

$$\Rightarrow\lim_{m\rightarrow \infty} \sup_{n \geq m} x_n= \inf_m \{ \sup_..

급수 수렴 판정법 1. Partial Sum

{S_{n} = \sum_{i = 1}^{n} a_{i}}_{n = 1}^{\infty} converges \to \sum_{i = 1}^{n} a_{i} converges

{S_{n} = \sum_{i = 1}^{n} a_{i}}_{n = 1}^{\infty} diverges \to \sum_{i = 1}^{n} a_{i} diverges

2. Cauchy Criterion

\forall ϵ > 0, \exists N s . t .

$$n>m>N\;\rightarrow\;\left|\sum_{i=m+1}..

수열 수렴 판정법 1. Direct Method (epsilon-delta)

\forall ϵ > 0, \exists N \in Z^{+} s . t .

n > N \to | a_{n} - c | 2. B o x M e t h o d (i) B o u n d e d b e l o w (a b o v e) (i i) M o n o t o n e d e c r e a s i n g (i n c r e a s i n g) \(\to \) \(a_{n} \) c o n v e r g e s .3 . C a u c h y C r i t e r i o n

\forall \epsilon>0,\;\exists N \in \mathbb{Z}^+\quad s.t.

m,n>N\;\rightarrow\;|a_m-a_n|

LASSO, Ridge regression LASSO

min_{β} (y - X β)^{T} (y - X β)

subject to | | β | |_{1} \leq t

β_{i}

가 0이 되는 걸 허용한다. 0이 됨으로써 variable selection도 되는 듯.Ridge regression

min_{β} (y - X β)^{T} (y - X β)

subject to | | β | |_{2} = c

Lagrangian당연히 둘 다 constraint를 Lagrangian을 이용해 objective에 포함시킬 수 있다. Referenceshttps://en.wikipedia.org/wiki/Lasso_(statistics)https://en.wikipedia.org/..

2d symmetric KL-divergence Implementation import numpy as np import matplotlib.pyplot as plt from mpl_toolkits.mplot3d import Axes3D def normal2d(mu:np.array, sigma): def normal2d_(x, y): x = np.array([x,y]) x = x.reshape((2,1)) I = np.eye(2) V = sigma**2*I V_inv = np.linalg.inv(V) mul = np.linalg.det(2*np.pi*V)**(-0.5) px = -0.5*(x-mu).T@V_inv@(x-mu) result = mul*np.exp(px[0]) return result[0] return normal2d_ def calcul..

목록 더보기

티스토리툴바