|
Article information
2019 , Volume 24, ¹ 6, p.69-78
Lukyanova O.A., Nikitin O.Y., Kunin A.S.
Application of matrix filters and braid theory for the procedural generation of neural network architectures
There are various approaches to the algorithmic specification of the network structure in the deep learning problems, which are successfully used in applications. These methods can be generalized by the concept of procedural generation of neural network architectures. Methodology. In the work, we use binary matrix filters. The filters are obtained with the help of the Hadamard product. Such filters define active network modules, thereby changing the way information is transmitted between layers. To build various architectures, the theory of braids is used in the work. The article reproduces the wellknown PathNet architecture. Examples of generating three new deep neural network architectures (3DNN, GraphNet, and BraidNet) are examined. Findings. The paper shows how the procedural generation of neural network architectures allows avoiding manually setting the network structure and automatically forming it. The use of matrix filters simplifies the process of generating network architecture due to a large number of possible combinations of modules and connections between them. Using the MNIST classification problem as an example, it is shown how the architectures presented in the article solve real-world pattern recognition problems. The results of application of neural networks indicate their diminishing tendency to retraining due to the subsequent convergence and the presence of stochastic dynamics in the learning process. Originality/value. Learning methods with dynamic adaptive changes in the network architecture allows achieving satisfactory accuracy faster and should also be less prone to retraining. The BraidNet algorithm presented in the article is applicable for a convenient brief record of the structure of a neural network in genetic algorithms. Such features make BraidNet a promising algorithm for further application and research in complex problems of pattern recognition, including using neuroevolutionary approaches.
[full text] [link to elibrary.ru]
Keywords: neural networks, neural network architectures, procedural generation, low-dimensional topology, braid theory, information transfer, deep learning
doi: 10.25743/ICT.2019.24.6.009.
Author(s):
Lukyanova Olga Alexandrovna
Position: Research Scientist
Office: Federal State Budgetary Institution of Science Computing Center of the Far Eastern Branch of the Russian Academy of Sciences
Address: 680000, Russia, Vladivostok, 65, Kim Yu Chen, str.
Phone Office: (924) 411-6656
E-mail: ollukyan@gmail.com
SPIN-code: 5347-8092Nikitin Oleg Yur`evich
Position: Research Scientist
Office: Institution of Science Computing Center of the Far Eastern Branch of the Russian Academy of Sciences
Address: 680000, Russia, Khabarovsk, 65, Kim U Chen st.
Phone Office: (4212) 703913
E-mail: olegioner@gmail.com
SPIN-code: 8499-0846Kunin Alexey Sergeevich
Position: Junior Research Scientist
Office: Institution of Science Computing Center of the Far Eastern Branch of the Russian Academy of Sciences
Address: 680000, Russia, Khabarovsk, 65, Kim U Chen st.
Phone Office: (4212) 703913
E-mail: alexkunin88@gmail.com
References:
[1] Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A.A., Pritzel, A., Wierstra, D. Pathnet: Evolution channels gradient descent in super neural networks. Available at: https://arxiv.org/abs/1701.08734 (accessed 30.09.2019).
[2] Larsson, G., Maire, M., Shakhnarovich, G. FractalNet: Ultra-deep neural networks without residuals. Available at: https://arxiv.org/abs/1605.07648 (accessed 30.09.2019).
3] Saxena, S., Verbeek, J. Convolutional neural fabrics. Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, 5–10 December 2016, Barcelona, Spain. 2016:4053–4061.
[4] Zoph, B., Le, Q.V. Neural architecture search with reinforcement learning. Available at: https://arxiv.org/abs/1611.01578 (accessed 30.09.2019).
[5] Togelius, J., Kastbjerg, E., Schedl, D., Yannakakis, G.N. What is procedural content generation? Mario on the borderline. Proceedings of the 2nd International Workshop on Procedural Content Generation in Games, Bordeaux, France, 28 June 2011. New York, USA: ACM; 2011: 8.
[6] Murasugi, K., Kurpita, B.I. Isotopy of braids. A Study of Braids. Part of the Mathematics and Its Applications book series. 1999; (484):96–112.
[7] Lukyanova, O., Nikitin, O. Modeling of extrasynaptic information transfer in neural networks using braid theory. Computer Science. 2018; (145C):306–311. DOI:10.1016/j.procs.2018.11.076
[8] Lukyanova, O., Nikitin, O. Isotopic inheritance: A topological approach to genotype transfer. International Conference on Simulation of Adaptive Behavior. SAB 2018: From Animals to Animats 15. Part of the Lecture Notes in Computer Science book series. 2018; (10994):27–38.
[8] Lukyanova, O., Nikitin, O. Neuronal topology as set of braids: information processing, transformation and dynamics. Optical Memory and Neural Networks. 2017; 26(3):172–181.
[9] LeCun, Y., Cortes, C., Burges, C.J.C. The MNIST database of handwritten digits. Available at: http://yann.lecun.com/exdb/mnist/ (accessed 30.09.2019).
[10] Sorokin, A.A., Makogonov, S.V., Korolev, S.P. Information infrastructure for the collective work of scientists of the Russian Far East. Nauchno-tekhnicheskaya informatsiya. Seriya 1: Organizatsiya i metodika informatsionnoy raboty. 2017; (12):14–16. (In Russ.)
Bibliography link:
Lukyanova O.A., Nikitin O.Y., Kunin A.S. Application of matrix filters and braid theory for the procedural generation of neural network architectures // Computational technologies. 2019. V. 24. ¹ 6. P. 69-78
|
