EMONAS : Evolutionary Multi-objective Neuron Architecture Search of Deep Neural Network

Sammanfattning: Customized Deep Neural Network (DNN) accelerators have been increasingly popular in various applications, from autonomous driving and natural language processing to healthcare and finance, etc. However, deploying them directly on embedded system peripherals within real-time operating systems (RTOS) is not easy due to the paradox of the complexity of DNNs and the simplicity of embedded system devices. As a result, DNN implementation on embedded system devices requires customized accelerators with tailored hardware due to their numerous computations, latency, power consumption, etc. Moreover, the computational capacity, provided by potent microprocessors or graphics processing units (GPUs), is necessary to unleash the full potential of DNN, but these computational resources are often not easily available in embedded system devices. In this thesis, we propose an innovative method to evaluate and improve the efficiency of DNN implementation within the constraints of resourcelimited embedded system devices. The Evolutionary Multi-Objective Neuron Architecture Search-Binary One Optimization (EMONAS-BOO) optimizes both the image classification accuracy and the innovative Binary One Optimization (BOO) objectives, with Multiple Objective Optimization (MOO) methods. The EMONAS-BOO automates neural network searching and training, and the neural network architectures’ diversity is also guaranteed with the help of an evolutionary algorithm that consists of tournament selection, polynomial mutation, and point crossover mechanisms. Binary One Optimization (BOO) is used to evaluate the difficulty in implementing DNNs on resource-limited embedded system peripherals, employing a binary format for DNN weights. A deeper implementation of the innovative Binary One Optimization will significantly boost not only computation efficiency but also memory storage, power dissipation, etc. It is based on the reduction of weights binary 1’s that need to be computed and stored, where the reduction of binary 1 brings reduced arithmetic operations and thus simplified neural network structures. In addition, analyzed from a digital circuit waveform perspective, the embedded system, in interpreting the neural network, will register an increase in zero weights leading to a reduction in voltage transition frequency, which, in turn, benefits power efficiency improvement. The proposed EMONAS employs the MOO method which optimizes two objectives. The first objective is image classification accuracy, and the second objective is Binary One Optimization (BOO). This approach enables EMONAS to outperform manually constructed and randomly searched DNNs. Notably, 12 out of 100 distinct DNNs maintained their image classification accuracy. At the same time, they also exhibit superior BOO performance. Additionally, the proposed EMONAS ensures automated searching and training of DNNs. It achieved significant reductions in key performance metrics: Compared with random search, evolutionary-searched BOO was lowered by up to 85.1%, parameter size by 85.3%, and FLOPs by 83.3%. These improvements were accomplished without sacrificing the image classification accuracy, which saw an increase of 8.0%. These results demonstrate that the EMONAS is an excellent choice for optimizing innovative objects that did not exist before, and greater multi-objective optimization performance can be guaranteed simultaneously if computational resources are adequate.