Towards Leveraging Structure for Neural Predictor in NAS

Document Type : Machine Learning - Monsefi

Authors

• Saeedeh Eslami ¹
• Reza Monsefi ¹
• Mohammad Akbari ²

¹ Department of Computer Engineering, Engineering Faculty of Ferdowsi University, Mashhad, Iran...

² Department of Mathematics and Computer Science, Amirkabir University, Tehran, Iran

10.22067/cke.2022.73356.1031

Abstract

Neural Architecture Search (NAS), which automatically designs a neural architecture for a specific task, has attracted much attention in recent years. Properly defining the search space is a key step in the success of NAS approaches, which allows us to reduce the required time for evaluation. Thus, late strategies for searching a NAS space is to leverage supervised learning models for ranking the potential neural models, i.e., surrogate predictive models. The predictive model takes the specification of an architecture (or its feature representation) and predicts the probable efficiency of the model ahead of training. Therefore, proper representation of a candidate architecture is an important factor for a predictor NAS approach. While several works have been devoted to training a good surrogate model, there exits limited research focusing on learning a good representation for these neural models. To address this problem, we investigate how to learn a representation with both structural and non-structural features of a network. In particular, we propose a tree structured encoding which permits to fully represent both networks’ layers and their intra-connections. The encoding is easily extendable to larger or more complex structures. Extensive experiments on two NAS datasets, NasBench101 and NasBench201, demonstrate the effectiveness of the proposed method as compared with the state-of-the-art predictors.

Keywords

• Neural Architecture Search
• Search Space Pruning
• Network Architecture
• Representation Learning

Main Subjects

• Machine learning

---

References

• [1] B. Zoph, Q. V. Le, Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611.01578 (2016).
• [2] Baker, O. Gupta, N. Naik, R. Raskar, Designing neural network architectures using reinforcement learning, arXiv preprint arXiv:1611.02167, (2016).
• [3] Brock, T. Lim, J. M. Ritchie, N. Weston, Smash: one-shot model architecture search through hypernetworks, arXiv preprint arXiv:1708.05344 (2017).
• [4] Liu, K. Simonyan, Y. Yang, Darts: Differentiable architecture search, arXiv preprint arXiv:1806.09055 (2018).
• [5] Wang, Y. Zhao, Y. Jinnai, Y. Tian, R. Fonseca, Neural architecture search using deep neural networks and monte carlo tree search, Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, (2020)
• [6] Deng, J. Yan, D. Lin, Peephole: Predicting network performance before training, arXiv preprint arXiv:1712.03351 (2017)
• [7] Luo, F. Tian, T. Qin, E. Chen, T.-Y. Liu, Neural architecture optimization, arXiv preprint arXiv:1808.07233 (2018)
• [8] Baker, O. Gupta, R. Raskar, N. Naik, Accelerating neural architecture search using performance prediction, arXiv preprint arXiv:1705.10823 435, (2017)
• [9] Zhong, J. Yan, W. Wu, J. Shao, C.-L. Liu, Practical block-wise neural network architecture generation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, (2018).
• Zoph, V. Vasudevan, J. Shlens, Q. V. Le, Learning transferable architectures for scalable image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, (2018).
• Luo, X. Tan, R. Wang, T. Qin, E. Chen, T.-Y. Liu, Neural architecture search with gbdt, arXiv preprint arXiv:2007.04785 (2020)
• Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, F. Hutter, Nas-bench-101: Towards reproducible neural architecture search, in: International Conference on Machine Learning, PMLR, (2019).
• Dong, Y. Yang, Nas-bench-201: Extending the scope of reproducible neural architecture search, arXiv preprint arXiv:2001.00326 (2020).
• Veit, M. J. Wilber, S. Belongie, Residual networks behave like ensembles of relatively shallow networks, Advances in neural information processing systems 29 (2016).
• Wen, H. Liu, Y. Chen, H. Li, G. Bender, P.-J. Kindermans, Neural predictor for neural architecture search, in: European Conference on Computer Vision, Springer, (2020)
• Real, A. Aggarwal, Y. Huang, Q. V. Le, Regularized evolution for image classifier architecture search, in: Proceedings of the aaai conference on artificial intelligence, Vol. 33, (2019)
• White, W. Neiswanger, Y. Savani, Bananas: Bayesian optimization with neural architectures for neural architecture search, arXiv preprint arXiv:1910.11858 1 (2) (2019).
• Li, A. Talwalkar, Random search and reproducibility for neural architecture search, in: arXiv preprint arXiv:1902.07638, (2019).
• Wu, X. Dai, D. Chen, Y. Chen, M. Liu, Y. Yu, Z. Wang, Z. Liu, M. Chen, L. Yuan, Weak nas predictors are all you need, arXiv preprint arXiv:2102.10490 (2021).
• White, W. Neiswanger, S. Nolen, Y. Savani, A study on encodings for neural architecture search, arXiv preprint arXiv:2007.04965 (2020)
• Wei, C. Niu, Y. Tang, Y. Wang, H. Hu, J. Liang, Npenas: Neural predictor guided evolution for neural architecture search, arXiv preprint arXiv:2003.12857 (2020).
• Sun, H. Wang, B. Xue, Y. Jin, G. G. Yen, M. Zhang, Surrogate-assisted evolutionary deep learning using an end-to-end random forest-based performance predictor, IEEE Transactions on Evolutionary Computation 24 (2), (2019)
• Istrate, F. Scheidegger, G. Mariani, D. Nikolopoulos, C. Bekas, A. C. I. Malossi, Tapas: Train-less accuracy predictor for architecture search, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, (2019)
• Ning, Y. Zheng, T. Zhao, Y. Wang, H. Yang, A generic graph-based neural architecture encoding scheme for predictor-based nas, in: Computer Vision–ECCV 2020: 16th European Conference, 28, 2020, Proceedings, Part XIII 16, Springer, (2020)
• -G. Talbi, Optimization of deep neural networks: a survey and unified taxonomy (2020).
• Cai, T. Chen, W. Zhang, Y. Yu, J. Wang, Efficient architecture search by network transformation, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, (2018)
• Ning, Y. Zheng, T. Zhao, Y. Wang, and H. Yang. A generic graph-based neural architecture encoding scheme for predictor-based nas. arXiv preprint arXiv:2004.01899, (2020)
• Wang, L., Zhao, Y., Jinnai, Y., Fonseca, R.: Alphax: exploring neural architec- tures with deep neural networks and monte carlo tree search. arXiv preprint arXiv:1805.07440 (2018)
• Shi, H., Pi, R., Xu, H., Li, Z., Kwok, J.T., Zhang, T.: Multi-objective neural architecture search via predictive network performance optimization. arXiv preprint arXiv:1911.09336 (2019) 

• Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.J., Fei-Fei, L., Yuille, A., Huang, J., Murphy, K.: Progressive neural architecture search. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 19–34 (2018) 

• Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018) 

• Minghao Guo, Yuzhe Yang, Rui Xu, Ziwei Liu, Dahua Lin; When NAS Meets Robustness: In Search of Robust Architectures Against Adversarial Attacks, Proc. Of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), (2020)
• Liu J., Jin Y., Multi-Objective search of robust neural architectures against multiple types of adversarial attacks, Neurocomputing Vol. 453 (2021)