Design and implementation of a convolutional neural network using tensor-train decomposition

Abstract

Neural networks show state-of-the-art performance in different fields. However, this technique suffers a memory consumption issue as we are handling high-dimensional data more and more often. In this thesis, we introduce a new formulation of the convolutional layer and verify a new training algorithm using Bayesian inference. Here we refer to any neural networks with any tensor-train-layers and trained by Bayesian training algorithm as a Bayesian TensorNet (BTN). The BTN provides a compressed network size and simplifies the operation in the neural network forward computation. We developed a novel tensor-train formulation of a convolutional neural network and trained it with a Bayesian training algorithm for a plant classification problem. We used the idea of representing the fully connected layer given by Novikov, and our novel tensor-train representation for the convolutional layer which is more general and straight than the tensor-train representation given by Garipov. We tested our BTN with a Bayesian training algorithm, which is an algorithm completely different than the backpropagation training algorithm where we do not need to compute any gradient of the network's weights. The training of our BTN was done with a dataset of plant images from the TerraByte project, an academic agriculture project focusing on machine learning application development in modern digital agriculture. We have tested the training result by achieving a 67% accuracy in the plant classification problem. Currently, the BTN developed here is still computationally expensive. It could benefit from further optimization, graphics processing unit (GPU) acceleration support and new development of neural network architectures. Suggested future work includes the exploration of another numerical integration method and a fair comparison to the backpropagation training algorithm.