Deep Learning Resources

• ImageNet
  • AlexNet
  • GoogLeNet
  • VGGNet
  • Inception-v3
  • ResNet
  • Inception-V4
• Tensor
• Deep Learning And Bayesian
• Autoencoders
• Semi-Supervised Learning
• Unsupervised Learning
• Deep Learning Networks
• Distributed System
• Deep Learning For Driving
• Deep Learning’s Accuracy
• Papers
  • STDP
• Codes
• Readings and Questions
• Resources
• Tools
• Books
• Blogs
• Competitions

ImageNet

AlexNet

ImageNet Classification with Deep Convolutional Neural Networks

• nips-page: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-
• paper: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
• slides: http://www.image-net.org/challenges/LSVRC/2012/supervision.pdf

GoogLeNet

Going Deeper with Convolutions

• paper: http://arxiv.org/abs/1409.4842
• code: https://github.com/google/inception
• blog(“Building a deeper understanding of images”): http://googleresearch.blogspot.jp/2014/09/building-deeper-understanding-of-images.html

VGGNet

Very Deep Convolutional Networks for Large-Scale Image Recognition

• homepage: http://www.robots.ox.ac.uk/~vgg/research/very_deep/
• arxiv: http://arxiv.org/abs/1409.1556
• slides: http://llcao.net/cu-deeplearning15/presentation/cc3580_Simonyan.pptx
• slides: http://www.robots.ox.ac.uk/~karen/pdf/ILSVRC_2014.pdf
• slides: http://deeplearning.cs.cmu.edu/slides.2015/25.simonyan.pdf

Tensorflow VGG16 and VGG19

• github: https://github.com/machrisaa/tensorflow-vgg

Inception-v3

Rethinking the Inception Architecture for Computer Vision

• intro: “21.2% top-1 and 5.6% top-5 error for single frame evaluation using a network; 3.5% top-5 error and 17.3% top-1 error With an ensemble of 4 models and multi-crop evaluation.”
• arXiv: http://arxiv.org/abs/1512.00567
• github(“Torch port of Inception V3”): https://github.com/Moodstocks/inception-v3.torch

ResNet

Deep Residual Learning for Image Recognition

• arxiv: http://arxiv.org/abs/1512.03385
• slides: http://research.microsoft.com/en-us/um/people/kahe/ilsvrc15/ilsvrc2015_deep_residual_learning_kaiminghe.pdf
• gitxiv: http://gitxiv.com/posts/LgPRdTY3cwPBiMKbm/deep-residual-learning-for-image-recognition
• github(by KaimingHe): https://github.com/KaimingHe/deep-residual-networks
• github: https://github.com/alrojo/lasagne_residual_network
• github: https://github.com/shuokay/resnet
• github: https://github.com/gcr/torch-residual-networks
• blog(“Highway Networks and Deep Residual Networks”): http://yanran.li/peppypapers/2016/01/10/highway-networks-and-deep-residual-networks.html
• blog(“Interpretating Deep Residual Learning Blocks as Locally Recurrent Connections”): https://matrixmashing.wordpress.com/2016/01/29/interpretating-deep-residual-learning-blocks-as-locally-recurrent-connections/
• github: https://github.com/apark263/cfmz
• github: https://github.com/NervanaSystems/neon/blob/master/examples/cifar10_msra.py
• github: https://github.com/facebook/fb.resnet.torch

Training and investigating Residual Nets

http://torch.ch/blog/2016/02/04/resnets.html

Resnet in Resnet: Generalizing Residual Architectures

• paper: http://beta.openreview.net/forum?id=lx9l4r36gU2OVPy8Cv9g
• arxiv: http://arxiv.org/abs/1603.08029

Identity Mappings in Deep Residual Networks (by Kaiming He)

• arxiv: http://arxiv.org/abs/1603.05027

Inception-V4

Inception-V4, Inception-Resnet And The Impact Of Residual Connections On Learning (Workshop track - ICLR 2016)

• intro: “achieve 3.08% top-5 error on the test set of the ImageNet classification (CLS) challenge”
• arxiv: http://arxiv.org/abs/1602.07261
• paper: http://beta.openreview.net/pdf?id=q7kqBkL33f8LEkD3t7X9

---

Network In Network

• arxiv: http://arxiv.org/abs/1312.4400

Striving for Simplicity: The All Convolutional Net

• arxiv: http://arxiv.org/abs/1412.6806

Batch-normalized Maxout Network in Network

• arxiv: http://arxiv.org/abs/1511.02583

Tensor

Tensorizing Neural Networks

• paper: http://arxiv.org/abs/1509.06569v1
• github(TensorNet): https://github.com/Bihaqo/TensorNet

On the Expressive Power of Deep Learning: A Tensor Analysis

• paper: http://arxiv.org/abs/1509.05009

Convolutional neural networks with low-rank regularization

• arxiv: http://arxiv.org/abs/1511.06067
• github: https://github.com/chengtaipu/lowrankcnn

Deep Learning And Bayesian

Scalable Bayesian Optimization Using Deep Neural Networks (ICML 2015)

• paper: http://jmlr.org/proceedings/papers/v37/snoek15.html
• arxiv: http://arxiv.org/abs/1502.05700
• github: https://github.com/bshahr/torch-dngo

Bayesian Dark Knowledge

• paper: http://arxiv.org/abs/1506.04416
• notes: Notes on Bayesian Dark Knowledge

Memory-based Bayesian Reasoning with Deep Learning(2015.Google DeepMind)

• slides: http://blog.shakirm.com/wp-content/uploads/2015/11/CSML_BayesDeep.pdf

Autoencoders

Importance Weighted Autoencoders

• paper: http://arxiv.org/abs/1509.00519
• code: https://github.com/yburda/iwae

Review of Auto-Encoders(by Piotr Mirowski, Microsoft Bing London, 2014)

• slides: https://piotrmirowski.files.wordpress.com/2014/03/piotrmirowski_2014_reviewautoencoders.pdf
• github: https://github.com/piotrmirowski/Tutorial_AutoEncoders/

Stacked What-Where Auto-encoders

• arxiv: http://arxiv.org/abs/1506.02351

Semi-Supervised Learning

Semi-Supervised Learning with Graphs (Label Propagation)

• paper: http://pages.cs.wisc.edu/~jerryzhu/pub/thesis.pdf
• blog(“标签传播算法（Label Propagation）及Python实现”): http://blog.csdn.net/zouxy09/article/details/49105265

Unsupervised Learning

Unsupervised Learning of Spatiotemporally Coherent Metrics

• paper: http://arxiv.org/abs/1412.6056
• code: https://github.com/jhjin/flattened-cnn

Unsupervised Learning on Neural Network Outputs

• intro: “use CNN trained on the ImageNet of 1000 classes to the ImageNet of over 20000 classes”
• arXiv: http://arxiv.org/abs/1506.00990
• github: https://github.com/yaolubrain/ULNNO

Deep Learning Networks

Deeply-supervised Nets (DSN)

• arxiv: http://arxiv.org/abs/1409.5185
• homepage: http://vcl.ucsd.edu/~sxie/2014/09/12/dsn-project/
• github: https://github.com/s9xie/DSN
• notes: http://zhangliliang.com/2014/11/02/paper-note-dsn/

Striving for Simplicity: The All Convolutional Net

• arxiv: http://arxiv.org/abs/1412.6806

Highway Networks

• arxiv: http://arxiv.org/abs/1505.00387
• blog(“Highway Networks with TensorFlow”): https://medium.com/jim-fleming/highway-networks-with-tensorflow-1e6dfa667daa#.71fgztsb6

Training Very Deep Networks (highway networks)

• arxiv: http://arxiv.org/abs/1507.06228

Very Deep Learning with Highway Networks

• homepage(papers, code, FAQ): http://people.idsia.ch/~rupesh/very_deep_learning/

Rectified Factor Networks

• arXiv: http://arxiv.org/abs/1502.06464
• github: https://github.com/untom/librfn

Correlational Neural Networks

• arxiv: http://arxiv.org/abs/1504.07225

Semi-Supervised Learning with Ladder Networks

• arxiv: http://arxiv.org/abs/1507.02672
• github: https://github.com/CuriousAI/ladder

Diversity Networks

• arxiv: http://arxiv.org/abs/1511.05077

A Unified Approach for Learning the Parameters of Sum-Product Networks (SPN)

• intro: “The Sum-Product Network (SPN) is a new type of machine learning model with fast exact probabilistic inference over many layers.”
• arxiv: http://arxiv.org/abs/1601.00318
• homepage: http://spn.cs.washington.edu/index.shtml
• code: http://spn.cs.washington.edu/code.shtml

Bitwise Neural Networks

• paper: http://paris.cs.illinois.edu/pubs/minje-icmlw2015.pdf
• demo: http://minjekim.com/demo_bnn.html

Learning Discriminative Features via Label Consistent Neural Network

• arxiv: http://arxiv.org/abs/1602.01168

Binarized Neural Networks

• arxiv: http://arxiv.org/abs/1602.02505

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

• paper: http://papers.nips.cc/paper/5647-shape-and-illumination-from-shading-using-the-generic-viewpoint-assumption
• github: https://github.com/MatthieuCourbariaux/BinaryConnect

BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

• arxiv: http://arxiv.org/abs/1602.02830
• github: https://github.com/MatthieuCourbariaux/BinaryNet

A Theory of Generative ConvNet

• arxiv: http://arxiv.org/abs/1602.03264
• project page: http://www.stat.ucla.edu/~ywu/GenerativeConvNet/main.html

Value Iteration Networks

• arxiv: http://arxiv.org/abs/1602.02867

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

• arxiv: http://arxiv.org/abs/1602.02282

Group Equivariant Convolutional Networks (G-CNNs)

• arxiv: http://arxiv.org/abs/1602.07576

Deep Spiking Networks

• arxiv: http://arxiv.org/abs/1602.08323
• github: https://github.com/petered/spiking-mlp

Low-rank passthrough neural networks

• arxiv: http://arxiv.org/abs/1603.03116
• github: https://github.com/Avmb/lowrank-gru

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

• arxiv: http://arxiv.org/abs/1603.05279

Distributed System

SparkNet: Training Deep Networks in Spark

• arXiv: http://arxiv.org/abs/1511.06051
• github: https://github.com/amplab/SparkNet
• blog: http://www.kdnuggets.com/2015/12/spark-deep-learning-training-with-sparknet.html

A Scalable Implementation of Deep Learning on Spark (Alexander Ulanov)

• page: http://www.slideshare.net/AlexanderUlanov1/a-scalable-implementation-of-deep-learning-on-spark-alexander-ulanov
• slides: http://pan.baidu.com/s/1jHiNW5C

Deep Learning For Driving

• project: http://deepdriving.cs.princeton.edu/
• paper: http://deepdriving.cs.princeton.edu/paper.pdf
• code: http://deepdriving.cs.princeton.edu/DeepDriving.zip

Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture

• paper: http://arxiv.org/abs/1509.05016
• homepage: http://www.brain4cars.com/

Eyes on the Road: How Autonomous Cars Understand What They’re Seeing

• blog: http://blogs.nvidia.com/blog/2016/01/05/eyes-on-the-road-how-autonomous-cars-understand-what-theyre-seeing/

Deep Learning’s Accuracy

• blog: http://deeplearning4j.org/accuracy.html

Papers

Reweighted Wake-Sleep

• paper: http://arxiv.org/abs/1406.2751
• code: https://github.com/jbornschein/reweighted-ws

Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks

• paper: http://arxiv.org/abs/1502.05336
• code: https://github.com/HIPS/Probabilistic-Backpropagation

Deeply-Supervised Nets

• paper: http://arxiv.org/abs/1409.5185
• code: https://github.com/mbhenaff/spectral-lib

STDP

A biological gradient descent for prediction through a combination of STDP and homeostatic plasticity

• arxiv: http://arxiv.org/abs/1206.4812

An objective function for STDP(Yoshua Bengio)

• arxiv: http://arxiv.org/abs/1509.05936

Towards a Biologically Plausible Backprop

• arxiv: http://arxiv.org/abs/1602.05179

---

Understanding and Predicting Image Memorability at a Large Scale (MIT. ICCV2015)

• homepage: http://memorability.csail.mit.edu/
• paper: https://people.csail.mit.edu/khosla/papers/iccv2015_khosla.pdf
• code: http://memorability.csail.mit.edu/download.html
• reviews: http://petapixel.com/2015/12/18/how-memorable-are-times-top-10-photos-of-2015-to-a-computer/

A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction

• arxiv: http://arxiv.org/abs/1512.06293

Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition

• arxiv: http://arxiv.org/abs/1601.02970
• demo: http://brainmodels.csail.mit.edu/dnn/drawCNN/

Deep-Spying: Spying using Smartwatch and Deep Learning

• arxiv: http://arxiv.org/abs/1512.05616
• github: https://github.com/tonybeltramelli/Deep-Spying

A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction

• arxiv: http://arxiv.org/abs/1512.06293

Understanding Deep Convolutional Networks

• arxiv: http://arxiv.org/abs/1601.04920

DeepCare: A Deep Dynamic Memory Model for Predictive Medicine

• arxiv: http://arxiv.org/abs/1602.00357

Exploiting Cyclic Symmetry in Convolutional Neural Networks

• arxiv: http://arxiv.org/abs/1602.02660
• github(Winning solution for the National Data Science Bowl competition on Kaggle (plankton classification)): https://github.com/benanne/kaggle-ndsb
• ref(use Cyclic pooling): http://benanne.github.io/2015/03/17/plankton.html

Cross-dimensional Weighting for Aggregated Deep Convolutional Features

• arxiv: http://arxiv.org/abs/1512.04065
• github: https://github.com/yahoo/crow

Understanding Visual Concepts with Continuation Learning

• project page: http://willwhitney.github.io/understanding-visual-concepts/
• arxiv: http://arxiv.org/abs/1602.06822
• github: https://github.com/willwhitney/understanding-visual-concepts

Learning Efficient Algorithms with Hierarchical Attentive Memory

• arxiv: http://arxiv.org/abs/1602.03218

Convergent Learning: Do different neural networks learn the same representations?

• arxiv: http://arxiv.org/abs/1511.07543
• github: https://github.com/yixuanli/convergent_learning

Do Deep Convolutional Nets Really Need to be Deep (Or Even Convolutional)?

• arxiv: http://arxiv.org/abs/1603.05691

Harnessing Deep Neural Networks with Logic Rules

• arxiv: http://arxiv.org/abs/1603.06318

A guide to convolution arithmetic for deep learning

• arxiv: http://arxiv.org/abs/1603.07285

Degrees of Freedom in Deep Neural Networks

• arxiv: http://arxiv.org/abs/1603.09260

Deep Networks with Stochastic Depth

• arxiv: http://arxiv.org/abs/1603.09382
• github: https://github.com/yueatsprograms/Stochastic_Depth
• notes(“Stochastic Depth Networks will Become the New Normal”): http://deliprao.com/archives/134

LIFT: Learned Invariant Feature Transform

• arxiv: http://arxiv.org/abs/1603.09114

Codes

deepnet: Implementation of some deep learning algorithms

• github: https://github.com/nitishsrivastava/deepnet

DeepNeuralClassifier(Julia): Deep neural network using rectified linear units to classify hand written digits from the MNIST dataset

• github: https://github.com/jostmey/DeepNeuralClassifier

Using deep learning to break a Captcha system

• intro: “Using Torch code to break simplecaptcha with 92% accuracy”
• blog: https://deepmlblog.wordpress.com/2016/01/03/how-to-break-a-captcha-system/
• github: https://github.com/arunpatala/captcha

Breaking reddit captcha with 96% accuracy

• blog: https://deepmlblog.wordpress.com/2016/01/05/breaking-reddit-captcha-with-96-accuracy/
• github: https://github.com/arunpatala/reddit.captcha

Clarifai Node.js Demo

• github: https://github.com/patcat/Clarifai-Node-Demo
• blog(“How to Make Your Web App Smarter with Image Recognition”): http://www.sitepoint.com/how-to-make-your-web-app-smarter-with-image-recognition/

Visual Search Server

• intro: “A simple implementation of Visual Search using features extracted from Tensor Flow inception model”
• github: https://github.com/AKSHAYUBHAT/VisualSearchServer

Deep Learning in Rust: baby steps

• blog: https://medium.com/@tedsta/deep-learning-in-rust-7e228107cccc#.t0pskuwkm
• github: https://github.com/tedsta/deeplearn-rs

Readings and Questions

What are the toughest neural networks and deep learning interview questions?

https://www.quora.com/What-are-the-toughest-neural-networks-and-deep-learning-interview-questions

What you wanted to know about AI

http://fastml.com/what-you-wanted-to-know-about-ai/

Epoch vs iteration when training neural networks

• stackoverflow: http://stackoverflow.com/questions/4752626/epoch-vs-iteration-when-training-neural-networks

Questions to Ask When Applying Deep Learning

http://deeplearning4j.org/questions.html

Resources

Awesome Deep Learning

• github: https://github.com/ChristosChristofidis/awesome-deep-learning

Deep Learning Libraries by Language

• website: http://www.teglor.com/b/deep-learning-libraries-language-cm569/

Deep Learning Resources

http://yanirseroussi.com/deep-learning-resources/

Turing Machine: musings on theory & code(DEEP LEARNING REVOLUTION, summer 2015, state of the art & topnotch links)

https://vzn1.wordpress.com/2015/09/01/deep-learning-revolution-summer-2015-state-of-the-art-topnotch-links/

BICV Group: Biologically Inspired Computer Vision research group

http://www.bicv.org/deep-learning/

Learning Deep Learning

http://rt.dgyblog.com/ref/ref-learning-deep-learning.html

Summaries and notes on Deep Learning research papers

• github: https://github.com/dennybritz/deeplearning-papernotes

Deep Learning Glossary

• intro: “Simple, opinionated explanations of various things encountered in Deep Learning / AI / ML.”
• author: Ryan Dahl, author of NodeJS.
• github: https://github.com/ry/deep_learning_glossary

The Deep Learning Playbook

https://medium.com/@jiefeng/deep-learning-playbook-c5ebe34f8a1a#.eg9cdz5ak

Deep Learning Study: Study of HeXA@UNIST in Preparation for Submission

• github: https://github.com/carpedm20/deep-learning-study

Tools

DNNGraph - A deep neural network model generation DSL in Haskell

• homepage: http://ajtulloch.github.io/dnngraph/

Books

Deep Learning (by Ian Goodfellow, Aaron Courville and Yoshua Bengio)

• homepage: http://www.deeplearningbook.org/
• website: http://goodfeli.github.io/dlbook/

Blogs

Neural Networks and Deep Learning

http://neuralnetworksanddeeplearning.com

Deep Learning Reading List

http://deeplearning.net/reading-list/

WILDML: A BLOG ABOUT MACHINE LEARNING, DEEP LEARNING AND NLP.

http://www.wildml.com/

Andrej Karpathy blog

http://karpathy.github.io/

Rodrigob’s github page

http://rodrigob.github.io/

colah’s blog

http://colah.github.io/

Competitions

Classifying plankton with deep neural networks

• blog: http://benanne.github.io/2015/03/17/plankton.html
• github: https://github.com/benanne/kaggle-ndsb