|
Article information
2026 , Volume 31, ¹ 4, p.123-131
Shipitsyn M.N., Fedorov R.K.
Vector representation of source code based on hashing for abstract syntax tree
This paper proposes an approach to source code analysis based on identifying recurring fragments of abstract syntax trees (ASTs) in Python programs. The aim is to transform code into a compact vector representation suitable for machine learning tasks. The proposed method involves hashing AST fragments while considering control flow structures and external function calls. An extended AST representation is developed, augmenting the standard Python AST module with metadata such as subtree hash codes. Based on the obtained hashes and their occurrences, a binarized bag-ofwords (BoW) vector representation of the source code is constructed. Experimental validation was conducted on the Python75 dataset, containing over 200 000 source code files across 75 programming tasks. Using the binary vectors as input, a Random Forest classifier achieved a classification accuracy of 89.8 %. The transformation to sparse binary matrices significantly reduced memory requirements while maintaining high performance. The key contributions include a specialized hashing algorithm and an effective code vectorization method based on binary bag-of-words. The approach demonstrates practical value by enabling efficient code analysis with low memory overhead and high accuracy, opening up possibilities for applications in code classification and similar code search.
Keywords: abstract syntax tree, hash-function, bag of words, feature vector, classification
Author(s): Shipitsyn Mark Nikolaevich Position: Student Office: Matrosov Institute for System Dynamics and Control Theory of Siberian Branch of Russian Academy of Sciences Address: 664033, Russia, Irkutsk, Lermontov Str. 134
E-mail: lmarkus243@gmail.com Fedorov Roman Konstantinovich Dr. Position: Leading research officer Office: Matrosov Institute for System Dynamics and Control Theory of Siberian Branch of Russian Academy of Sciences Address: 664033, Russia, Irkutsk, Lermontov Str., 134
Phone Office: (3952) 45-31-08 E-mail: fedorov@icc.ru SPIN-code: 5344-2226 Bibliography link: Shipitsyn M.N., Fedorov R.K. Vector representation of source code based on hashing for abstract syntax tree // Computational technologies. 2026. V. 31. ¹ 4. P. 123-131
|