Neural Networks

Lecturer: Dr. Thomas Stibor

2 SWS Vorlesung, 3 CP

Prerequisite: Vordiplom/Bachelor
Course description: Neural networks are computational models for information processing problems such as pattern recognition, clustering, novelty detection and regression analysis. This course will cover the following topics:
  • Single-Layer Networks
  • Multi-Layer Networks
  • Error Functions
  • Weights Optimization Algorithms and Regularization
  • Hopfield Network
  • Boltzmann Machine
  • Winner Take All Neural Network (Unsupervised Learning)
  • Evolving Neural Networks
  • Comparison to Support Vector Machine
Meeting times and location: Friday 9:50 - 11:30, S2|02 C110, Start 04.04.2008
Final exam: Tuesday, 26.08.2008, 10:00 - 12:00, S2|02 C120
Topics and lecture dates
Topics & slides Lecture date:
Introduction to Neural Computation & Optimization, Single-Layer Networks, Lecture 1 (PDF), Lecture 1 (PS.GZ) 04.04.2008
Bayesian Decision, Perceptron, LMS, Lecture 2 (PDF), Lecture 2 (PS.GZ)(revised) 11.04.2008
Multi-Layer Networks, Backpropagation, Lecture 3 (PDF), Lecture 3 (PS.GZ) 18.04.2008
Regularization, Over/Underfitting, Momentum, Lecture 4 (PDF), Lecture 4 (PS.GZ) 25.04.2008
Learning Rate, Hopfield Network, Lecture 5 (PDF), Lecture 5 (PS.GZ) 09.05.2008
Boltzmann Machine, Lecture 6 (PDF), Lecture 6 (PS.GZ) 16.05.2008
PCA (with Neural Networks), Feature Extraction, Lecture 7 (PDF), Lecture 7 (PS.GZ) 30.05.2008
Winner-Take-All, K-Means, SOM, Lecture 8 (PDF), Lecture 8 (PS.GZ) 06.06.2008
Evolving Neural Networks, Lecture 9 (PDF), Lecture 9 (PS.GZ) 13.06.2008
Support Vector Machine, Lecture 10 (PDF), Lecture 10 (PS.GZ) 20.06.2008
Neural Networks for Time Series Prediction, Support Vector Regression, Lecture 11 (PDF), Lecture 11 (PS.GZ) 27.06.2008
Neural Networks for Pattern Recognition,
Christopher M. Bishop,
Oxford University Press, 1995, ISBN: 0198538642
Neural Networks: A Comprehensive Foundation (2nd Edition),
Simon Haykin,
Prentice Hall Publishers, 1998, ISBN: 0132733501
Introduction to the Theory of Neural Computation,
John Hertz, Andreas Krogh, Richard Palmer,
Addison Wesley, 1991, ISBN: 0201515601
Information Theory, Inference and Learning Algorithms,
David MacKay,
Cambridge University Press, 2003, ISBN: 0521642981

R Src Code:
Single Layer Network (Perceptron, LMS, Pseudoinverse), Lecture 1 & 2,
download source code (single.layer.nn.R)
Network Output as Posterior Probability, Lecture 1 & 2,
download source code (logistic.discr.R)
Multi-Layer Network, Decision Regions, Lecture 3,
download source code (multi.layer.nn.R)
Momentum, Lecture 4,
download source code (rosenbrock.function.momentum.R)
Hopfield Network, Lecture 5,
download source code (
PCA, Oja's update rule, Lecture 7,
download source code (PCA.Oja.R)
PCA example (projection + reconstruction), Lecture 7,
download source code (PCA.image.R,charly.pgm)
Winner-Take-All, Lecture 8,
download source code (WTA.R)
Support Vector Machine, Lecture 10,
download source code (SVM.R)
R Literature:
Modern Applied Statistics with S (4th Edition),
W.N. Venables, B.D. Ripley,
Springer-Verlag, 2002, ISBN: 0387954570,
(Note: this book covers also R).
Programmieren mit R (Statistik und ihre Anwendungen) (2. Auflage),
Uwe Ligges,
Springer-Verlag, 2006, ISBN: 3540363327