CNN


        require(R);
                               

#' feed-forward phase of deep Convolutional Neural Networks
imports "CNN" from "MLkit";

.NET clr type export
cnn: LayerBuilder

.NET clr function exports
n_threads	get/set of the CNN parallel thread number
cnn	Create a new CNN model Convolutional neural network (CNN) is a regularized type of feed-forward neural network that learns feature engineering by itself via filters (or kernel) optimization. Vanishing gradients and exploding gradients, seen during backpropagation in earlier neural networks, are prevented by using regularized weights over fewer connections.
input_layer	The input layer is a simple layer that will pass the data though and create a window into the full training data set. So for instance if we have an image of size 28x28x1 which means that we have 28 pixels in the x axle and 28 pixels in the y axle and one color (gray scale), then this layer might give you a window of another size example 24x24x1 that is randomly chosen in order to create some distortion into the dataset so the algorithm don't over-fit the training.
regression_layer
conv_layer	This layer uses different filters to find attributes of the data that affects the result. As an example there could be a filter to find horizontal edges in an image.
conv_transpose_layer
lrn_layer	This layer is useful when we are dealing with ReLU neurons. Why is that? Because ReLU neurons have unbounded activations and we need LRN to normalize that. We want to detect high frequency features with a large response. If we normalize around the local neighborhood of the excited neuron, it becomes even more sensitive as compared to its neighbors. At the same time, it will dampen the responses that are uniformly large in any given local neighborhood. If all the values are large, then normalizing those values will diminish all of them. So basically we want to encourage some kind of inhibition and boost the neurons with relatively larger activations. This has been discussed nicely in Section 3.3 of the original paper by Krizhevsky et al.
tanh_layer	Implements Tanh nonlinearity elementwise x to tanh(x) so the output is between -1 and 1.
softmax_layer	[*loss_layers] This layer will squash the result of the activations in the fully connected layer and give you a value of 0 to 1 for all output activations.
relu_layer	This is a layer of neurons that applies the non-saturating activation function f(x)=max(0,x). It increases the nonlinear properties of the decision function and of the overall network without affecting the receptive fields of the convolution layer.
leaky_relu_layer
maxout_layer	Implements Maxout nonlinearity that computes x to max(x) where x is a vector of size group_size. Ideally of course, the input size should be exactly divisible by group_size
sigmoid_layer	Implements Sigmoid nonlinearity elementwise x to 1/(1+e^(-x)) so the output is between 0 and 1.
pool_layer	This layer will reduce the dataset by creating a smaller zoomed out version. In essence you take a cluster of pixels take the sum of them and put the result in the reduced position of the new image.
dropout_layer	This layer will remove some random activations in order to defeat over-fitting.
full_connected_layer	Neurons in a fully connected layer have full connections to all activations in the previous layer, as seen in regular Neural Networks. Their activations can hence be computed with a matrix multiplication followed by a bias offset.
gaussian_layer
sample_dataset
sample_dataset.image
auto_encoder
training	Do CNN network model training
ada_delta	Adaptive delta will look at the differences between the expected result and the current result to train the network.
ada_grad	The adaptive gradient trainer will over time sum up the square of the gradient and use it to change the weights.
adam	Adaptive Moment Estimation is an update to RMSProp optimizer. In this running average of both the gradients and their magnitudes are used.
nesterov	Another extension of gradient descent is due to Yurii Nesterov from 1983,[7] and has been subsequently generalized
sgd	Stochastic gradient descent (often shortened in SGD), also known as incremental gradient descent, is a stochastic approximation of the gradient descent optimization method for minimizing an objective function that is written as a sum of differentiable functions. In other words, SGD tries to find minimums or maximums by iteration.
window_grad	This is AdaGrad but with a moving window weighted average so the gradient is not accumulated over the entire history of the run. it's also referred to as Idea #1 in Zeiler paper on AdaDelta.
predict
CeNiN	load a CNN model from file
detectObject	classify a object from a given image data
saveModel	save the CNN model into a binary data file