lemga::LPBoost

LPBoost Class Reference

LPBoost (Linear-Programming Boosting). More...

#include <lpboost.h>

Inheritance diagram for LPBoost:

Inheritance graph
[legend]
Collaboration diagram for LPBoost:

Collaboration graph
[legend]
List of all members.

Public Member Functions

 LPBoost (REAL _C=1.0)
 LPBoost (std::istream &is)
virtual const id_t & id () const
virtual LPBoost * create () const
 Create a new object using the default constructor.
virtual LPBoost * clone () const
 Create a new object by replicating itself.
virtual REAL train ()
 Train with preset data set and sample weight.
REAL C () const
 The regularization constant C.
void set_C (REAL _C)
 Set the regularization constant C.

Detailed Description

LPBoost (Linear-Programming Boosting).

With a similar idea to the original LPBoost [1], which solves

\begin{eqnarray*} \min && -\rho + D \sum_i \xi_i \\ \textrm{s.t.}&& y_i \left(\sum_t w_t h_t(x_i)\right) \ge \rho - \xi_i,\\ && \xi_i \ge 0, \quad w_t \ge 0, \quad \sum_t w_t = 1. \end{eqnarray*}

we instead implement the algorithm to solve

\begin{eqnarray*} \min && \sum_t w_t + C \sum_i \xi_i \\ \textrm{s.t.}&& y_i \left(\sum_t w_t h_t(x_i)\right) \ge 1 - \xi_i,\\ && \xi_i \ge 0, \quad w_t \ge 0. \end{eqnarray*}

by column generation. Note that the dual problem is

\begin{eqnarray*} \max && \sum_i u_i \\ \textrm{s.t.}&& \sum_i u_i y_i h_t(x_i) \ge 1, \qquad (*)\\ && 0 \le u_i \le C. \end{eqnarray*}

Column generation corresponds to generating the constraints (*). We actually use individual upper bound $C_i$ proportional to example's initial weight.

If we treat $v_i$, the normalized version of $u_i$, as the sample weight, and $\Sigma_u = \sum_i u_i$ as the normalization constraint, (*) is the same as

\[ \Sigma_u (1 - 2 e(h_t, v)) \ge 1, \]

which means

\[ e(h_t, v) \ge \frac12 (1 - \Sigma_u^{-1}).\qquad (**) \]

Assume that we have found $h_1, \dots, h_T$ so far, solving the dual problem with $T$ (*) constraints gives us $\Sigma_u$. If for every remaining $h$ in $\cal{H}$, $ e(h, v) \ge \frac12 (1 - \Sigma_u^{-1}),$ the duality condition tells us that even if we set $w=0$ for those remaining $h$, the solution is still optimal. Thus, we can train the weak learner with sample weight $v$ in each iteration, and terminate if the best hypothesis has satisfied (**).

[1] A. Demiriz, K. P. Bennett, and J. Shawe-Taylor. Linear programming boosting via column generation. Machine Learning, 46(1-3):225-254, 2002.

Definition at line 58 of file lpboost.h.

Constructor & Destructor Documentation

LPBoost (  REAL  _C = 1.0  )  [inline, explicit]
 

Definition at line 62 of file lpboost.h.

References LPBoost::set_C().

Referenced by LPBoost::clone(), and LPBoost::create().

LPBoost (  std::istream &  is  )  [inline, explicit]
 

Definition at line 63 of file lpboost.h.

Member Function Documentation

REAL C (   )  const [inline]
 

The regularization constant C.

Definition at line 72 of file lpboost.h.

virtual LPBoost* clone (   )  const [inline, virtual]
 

Create a new object by replicating itself.

Returns:
A pointer to the new copy.
The code for a derived class Derived is always 
 return new Derived(*this);
Though seemingly redundant, it helps to copy an object without knowing the real type of the object.
See also:
C++ FAQ Lite 20.6

Reimplemented from Boosting.

Definition at line 67 of file lpboost.h.

References LPBoost::LPBoost().

virtual LPBoost* create (   )  const [inline, virtual]
 

Create a new object using the default constructor.

The code for a derived class Derived is always 

 return new Derived();

Reimplemented from Boosting.

Definition at line 66 of file lpboost.h.

References LPBoost::LPBoost().

virtual const id_t& id (   )  const [virtual]
 

Returns:
Class ID string (class name)

Reimplemented from Boosting.

void set_C (  REAL  _C  )  [inline]
 

Set the regularization constant C.

Definition at line 74 of file lpboost.h.

Referenced by LPBoost::LPBoost().

REAL train (   )  [virtual]
 

Train with preset data set and sample weight.

Returns:
Probably the training error.
Todo:
Make the return type void

Reimplemented from Boosting.

Definition at line 20 of file lpboost.cpp.

References Aggregating::empty(), Boosting::grad_desc_view, Aggregating::lm_base, Aggregating::n_in_agg, LearnModel::n_samples, LearnModel::ptd, LearnModel::ptw, and U.

The documentation for this class was generated from the following files:
Generated on Mon Jan 9 23:44:50 2006 for LEMGA by  doxygen 1.4.6