二项逻辑斯谛回归模型是如下的条件概率分布: \begin{align*} \\& P \left( Y = 1 | x \right) = \dfrac{1}{1+\exp{-\left(w \cdot x + b \right)}} \\ & \quad\quad\quad\quad = \dfrac{\exp{\left(w \cdot x + b \right)}}{\left( 1+\exp{-\left(w \cdot x + b \right)}\right) \cdot \exp{\left(w \cdot x + b \right)}} \\ & \quad\quad\quad\quad = \dfrac{\exp{\left(w \cdot x + b \right)}}{1+\exp{\left( w \cdot x + b \right)}}\\& P \left( Y = 0 | x \right) = 1- P \left( Y = 1 | x \right) \\ & \quad\quad\quad\quad=1- \dfrac{\exp{\left(w \cdot x + b \right)}}{1+\exp{\left( w \cdot x + b \right)}} \\ & \quad\quad\quad\quad=\dfrac{1}{1+\exp{\left( w \cdot x + b \right)}}\end{align*} 其中,$x \in R^{n}$是输入,$Y \in \left\{ 0, 1 \right\}$是输出,$w \in R^{n}$和$b \in R$是参数,$w$称为权值向量,$b$称为偏置,$w \cdot x$为$w$和$b$的内积。
可将权值权值向量和输入向量加以扩充,即$w = \left( w^{\left(1\right)},w^{\left(2\right)},\cdots,w^{\left(n\right)},b \right)^{T}$,$x = \left( x^{\left(1\right)},x^{\left(2\right)},\cdots,x^{\left(n\right)},1 \right)^{T}$,则逻辑斯谛回归模型: \begin{align*} \\& P \left( Y = 1 | x \right) = \dfrac{\exp{\left(w \cdot x \right)}}{1+\exp{\left( w \cdot x \right)}}\\& P \left( Y = 0 | x \right) =\dfrac{1}{1+\exp{\left( w \cdot x \right)}}\end{align*}
一个事件的几率是指事件发生的概率$p$与事件不发生的概率$1-p$的比值,即
\begin{align*} \\& \dfrac{p}{1-p}\end{align*}
该事件的对数几率(logit函数)
\begin{align*} \\& logit\left( p \right) = \log \dfrac{p}{1-p}\end{align*}
对于逻辑斯谛回归模型
\begin{align*} \\& \log \dfrac{P \left( Y = 1 | x \right)}{1-P \left( Y = 1 | x \right)} = w \cdot x\end{align*}
即输出$Y=1$的对数几率是输入$x$的线性函数。
给定训练数据集
\begin{align*} \\& T = \left\{ \left( x_{1}, y_{1} \right), \left( x_{2}, y_{2} \right), \cdots, \left( x_{N}, y_{N} \right) \right\} \end{align*}
其中,$x_{i} \in R^{n+1}, y_{i} \in \left\{ 0, 1 \right\}, i = 1, 2, \cdots, N$。
设:
\begin{align*} \\& P \left( Y =1 | x \right) = \pi \left( x \right) ,\quad P \left( Y =0 | x \right) = 1 - \pi \left( x \right) \end{align*}
似然函数
\begin{align*} \\& l \left( w \right) = \prod_{i=1}^{N} P \left( y_{i} | x_{i} \right)
\\ & = P \left( Y = 1 | x_{i} , w \right) \cdot P \left( Y = 0 | x_{i}, w \right)
\\ & = \prod_{i=1}^{N} \left[ \pi \left( x_{i} \right) \right]^{y_{i}}\left[ 1 - \pi \left( x_{i} \right) \right]^{1 - y_{i}}\end{align*}
对数似然函数 \begin{align*} \\& L \left( w \right) = \log l \left( w \right) \\ & = \sum_{i=1}^{N} \left[ y_{i} \log \pi \left( x_{i} \right) + \left( 1 - y_{i} \right) \log \left( 1 - \pi \left( x_{i} \right) \right) \right] \\ & = \sum_{i=1}^{N} \left[ y_{i} \log \dfrac{\pi \left( x_{i} \right)}{1- \pi \left( x_{i} \right)} + \log \left( 1 - \pi \left( x_{i} \right) \right) \right] \\ & = \sum_{i=1}^{N} \left[ y_{i} \left( w \cdot x_{i} \right) - \log \left( 1 + \exp \left( w \cdot x \right) \right) \right]\end{align*}
假设$w$的极大似然估计值是$\hat{w}$,则学得得逻辑斯谛回归模型 \begin{align*} \\& P \left( Y = 1 | x \right) = \dfrac{\exp{\left(\hat{w} \cdot x \right)}}{1+\exp{\left( \hat{w} \cdot x \right)}}\\& P \left( Y = 0 | x \right) =\dfrac{1}{1+\exp{\left( \hat{w} \cdot x \right)}}\end{align*}
假设离散型随机变量$Y$的取值集合$\left\{ 1, 2, \cdots, K \right\}$,则多项逻辑斯谛回归模型 \begin{align*} \\& P \left( Y = k | x \right) = \dfrac{\exp{\left(w_{k} \cdot x \right)}}{1+ \sum_{k=1}^{K-1}\exp{\left( w_{k} \cdot x \right)}}, \quad k=1,2,\cdots,K-1 \\ & P \left( Y = K | x \right) = 1 - \sum_{k=1}^{K-1} P \left( Y = k | x \right) \\ & = 1 - \sum_{k=1}^{K-1} \dfrac{\exp{\left(w_{k} \cdot x \right)}}{1+ \sum_{k=1}^{K-1}\exp{\left( w_{k} \cdot x \right)}} \\ & = \dfrac{1}{1+ \sum_{k=1}^{K-1}\exp{\left( w_{k} \cdot x \right)}}\end{align*}