lemga::_gd_weightdecay

_gd_weightdecay Struct Template Reference

Gradient descent with weight decay. More...

#include <optimize.h>

Inheritance diagram for _gd_weightdecay:

Inheritance graph
[legend]
Collaboration diagram for _gd_weightdecay:

Collaboration graph
[legend]
List of all members.

Public Member Functions

 _gd_weightdecay (LM *lm, const Step &lr, const Step &dcy)
void update_weight (Dir d, const Step &s)

Public Attributes

Step decay

Detailed Description

template<class LM, class Dir, class Step>
struct lemga::_gd_weightdecay< LM, Dir, Step >

Gradient descent with weight decay.

The cost function includes a regularization term which prefers smaller weight values. That is, the cost to be minimized is

\[ \hat{E} = E + \frac{\lambda}2 \Vert w\Vert^2. \]

The updating rule thus becomes

\[ w \leftarrow w - \eta \frac{\partial\hat{E}}{\partial w} = (1-\eta\lambda)w - \eta\frac{\partial E}{\partial w}. \]

$\lambda$ usually ranges from 0.01 to 0.1.

Todo:
Well, I guess weight decay is more a term in the cost than an optimization technique. Though it is easy to add it here (as an optimization technique).

Definition at line 140 of file optimize.h.

Constructor & Destructor Documentation

_gd_weightdecay (  LM *  lm,
const Step &  lr,
const Step &  dcy
)  [inline]
 

Definition at line 143 of file optimize.h.

Member Function Documentation

void update_weight (  Dir  d,
const Step &  s
)  [inline]
 

Reimplemented from _gradient_descent.

Definition at line 146 of file optimize.h.

Member Data Documentation

Step decay
 

Definition at line 141 of file optimize.h.

The documentation for this struct was generated from the following file:
Generated on Wed Nov 8 08:17:11 2006 for LEMGA by  doxygen 1.4.6