4D tensor -> ND tensor (for layers)

[pluskid]

This is a working-in-progress PR that upgrade Mocha's 4D-tensor to general ND-tensor.

• The interface will be almost backward-compatible, meaning that old code using 4D tensor will still run as before.
  • Some breaks will still be introduced. For example, old ReshapeLayer explicitly specify the new width, height, and channels, but now we need to specify a general target dimension
• Some layers will require the ND-tensor to be 4D explicitly (for now) because some of the implementations rely on the 4D-tensor shape
  • It is expected those requirements will be relaxed gradually when the implementations get generalized / re-written

Behavior changes for existing layers

• HDF5DataLayer, MemoryDataLayer, HDF5OutputLayer: support general ND tensor datasets. The last dimension is treated as num dimension, and split for mini-batch.
• ConcatLayer: supports general ND tensor and could concat along any dimension.
• CropLayer: requires the input to be 4D tensor, as crop layer is designed specific for image data.
• DropoutLayer, SplitLayer: supports general ND tensor out of the box.
• ElementwiseLayer, PowerLayer: support general ND tensor.
• InnerProductLayer: supports general ND tensor. For tensor with ndims tensor dimension, the first ndims-1 dimensions are "collapsed" to form the input feature dimension. The output is now 2D tensor (output-dim x mini-batch) instead of 4D tensor.
• ConvolutionLayer, PoolingLayer: requires the input to be 4D tensor due to the current implementation.
• ReshapeLayer: is now more useful to deal with some layers that has special requirement for the tensor dimensions.
• ArgmaxLayer, SoftmaxLayer, MultinomialLogisitcLayer, SoftmaxLossLayer, AccuracyLayer: all of them rely on the concept of the channel dimension.
  • Current behavior: treat the ND-tensor as 4D. If the tensor dimension is larger than 4, all the extra dimensions are collapsed to form the channel dimension.
  • Future work: modify the behavior by allowing the user to specify which dimension to operate on.
• LRNLayer: require the input to be 4D tensor for now.
• ChannelPooling: require the input to be 4D tensor for now. Could possibly be modified to do 1-dimensional pooling along any user-specified tensor dimension.

Related Issues (will be addressed in other PRs)

• Loading network parameters now need to check parameter dimensions in a smarter way (e.g. InnerProductLayer weights saved by old version of Mocha are 4D-tensors, but they are now 2D tensors)
• Generalize other components if Mocha to support ND-tensor (notably norm constraints)
• Implement future behavior of some layers mentioned above
• Update documentation

[coveralls]

Coverage increased (+0.14%) when pulling 6dd5d75 on nd-tensor into c1fd5af on master.

[coveralls]

Coverage increased (+0.07%) when pulling d21d4ec on nd-tensor into c1fd5af on master.

[coveralls]

Coverage increased (+0.16%) when pulling d440c49 on nd-tensor into c1fd5af on master.

[coveralls]

Coverage increased (+0.16%) when pulling b9319cc on nd-tensor into c1fd5af on master.

[coveralls]

Coverage increased (+0.17%) when pulling f51a624 on nd-tensor into c1fd5af on master.