Computer and Knowledge Engineering

Computer and Knowledge Engineering

Enhancing Channel Selection in 5G with Decentralized Federated Multi-Agent Deep Reinforcement Learning

Document Type : Original Article

Authors
Taghi Shahgholi 1
Keyhan Khamforoosh 2
Amir Sheikhahmadi 2
Sadoon Azizi 3
1 Department of Computer Engineering, Sanandaj Branch, Islamic Azad University ,Sanandaj, Iran
2 Department of Computer Engineering, Sanandaj Branch, Islamic Azad University,Sanandaj, Iran
3 Department of Computer Engineering and IT, University of Kurdistan,Sanandaj,Iran
10.22067/cke.2024.88900.1119
Abstract
The increasing popularity of vehicular communication systems necessitates efficient and autonomous decision-making to address the challenges of vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications. In this paper, we present a comprehensive study on channelization in Cellular Vehicle-to-Everything (C-V2X) communication and propose a novel two-layer multi-agent approach that integrates deep reinforcement learning (DRL) and federated learning (FL) to enhance the decision-making process in channel utilization.

Our approach leverages the autonomy of each vehicle, treating it as an independent agent capable of making channel selection decisions based on its local observations in its own cluster. Simultaneously, a centralized architecture coordinates nearby vehicles to optimize overall system performance. The DRL-based decision-making model considers crucial factors, such as instantaneous channel state information and historical link selections, to dynamically allocate channels and transmission power, leading to improved system efficiency.

By incorporating federated learning, we enable knowledge sharing and synchronization among the decentralized vehicular agents. This collaborative approach harnesses the collective intelligence of the network, empowering each agent to gain insights into the broader network dynamics beyond its limited observations. The results of our extensive simulations demonstrate the superiority of the proposed approach over existing methods, as it achieves higher data rates, success rates, and superior interference mitigation.
Keywords
C-V2X Optimization
Multi-Agent Learning
DRL-based Channel Access
Federated Learning Integration
Subjects
  1. Obiodu, A. Raman, A. K. Abubakar, S. Mangiante, N. Sastry and A. H. Aghvami. (2022, Feb.). DSM-MoC as Baseline: Reliability Assurance via Redundant Cellular Connectivity in Connected Cars. IEEE Transactions on Network and Service Management. [Online]. 19(3), pp. 2178-2194. Available: 10.1109/TNSM.2022.3153452
  2. Zhang, M. Peng, S. Yan and Y. Sun. (2019, Dec.). Deep-Reinforcement-Learning-Based Mode Selection and Resource Allocation for Cellular V2X Communications. IEEE Internet of Things Journal. [Online]. 7(7), pp. 6380-6391. Available: 10.1109/JIOT.2019.2962715
  3. Yang, Y. Ju, L. Liu, Q. Pei, K. Yu and J. J. P. C. Rodrigues, “Secure mmWave C-V2X Communications Using Cooperative Jamming," in GLOBECOM 2022 - 2022 IEEE Global Communications Conference, Rio de Janeiro, Brazil, 2022, pp. 2686-2691
  4. -X. Zheng et al. and K. K. W. (2022, Jun.). Physical-Layer Security of Uplink mmWave Transmissions in Cellular V2X Networks. IEEE Transactions on Wireless Communications. [Online]. 21(11), pp. 9818-9833. Available: https://doi.org/10.1109/TWC.2022.3179706
  5. S. Bahbahani, E. Alsusa and A. Hammadi. (2022, Nov.). A Directional TDMA Protocol for High Throughput URLLC in mmWave Vehicular Networks. IEEE Transactions on Vehicular Technology. [Online]. 72(3), pp. 3584-3599. Available: 10.1109/TVT.2022.3219771
  6. Xiang, H. Shan, Z. Su, Z. Zhang, C. Chen and E. -P. Li. (2022, Oct.). Multi-Agent Reinforcement Learning-Based Decentralized Spectrum Access in Vehicular Networks with Emergent Communication. IEEE Communications Letters. [Online]. 27(1), pp. 195-199. Available: https://doi.org/10.1109/LCOMM.2022.3214792
  7. Sehla, T. M. T. Nguyen, G. Pujolle and P. B. Velloso. (2022, Mar.). Resource Allocation Modes in C-V2X: From LTE-V2X to 5G-V2X. IEEE Internet of Things Journal. [Online]. 9(11), pp. 8291-8314. Available: https://doi.org/10.1109/JIOT.2022.3159591
  8. Bagheri et al. and K. Moessner. (2021, Mar.). 5G NR-V2X: Toward Connected and Cooperative Autonomous Driving. IEEE Communications Standards Magazine. [Online]. 5(1), pp. 48-54. Available: https://doi.org/10.1109/MCOMSTD.001.2000069
  9. Twardokus and H. Rahbari. (2023, Mar.). Towards Protecting 5G Sidelink Scheduling in C-V2X Against Intelligent DoS Attacks. IEEE Transactions on Wireless Communications. [Online]. 22(11), pp. 7273-7286. Available: https://doi.org/10.1109/TWC.2023.3249665
  10. Sehla, T. M. T. Nguyen, G. Pujolle and P. B. Velloso. (2022, Mar.). Resource Allocation Modes in C-V2X: From LTE-V2X to 5G-V2X. IEEE Internet of Things Journal. [Online]. 9(11), pp. 8291-8314. Available: https://doi.org/10.1109/JIOT.2022.3159591
  11. -H. Wu, R. -H. Hwang, C. -Y. Wang and C. -H. Chou. “Deep Reinforcement Learning Based Resource Allocation for 5G V2V Groupcast Communications,” in 2023 International Conference on Computing, Networking and Communications (ICNC), Honolulu, HI, USA, 2023, pp. 1-6.
  12. Lin, J. G. Andrews, A. Ghosh and R. Ratasuk. (2014, Apr.). An overview of 3GPP device-to-device proximity services.  IEEE Communications Magazine. [Online]. 52(4), pp. 40-48. Available: https://doi.org/10.1109/MCOM.2014.6807945
  13. Chen et al. and R. Zhao. (2017, Jul.). Vehicle-to-Everything (v2x) Services Supported by LTE-Based Systems and 5G. IEEE Communications Standards Magazine. [Online]. 1(2), pp. 70-76. Available: https://doi.org/10.1109/MCOMSTD.2017.1700015
  14. Ran, “Study on Evaluation Methodology of New V2X Use Cases for LTE and NR,” Dubrovnik, Croatia, 2017.
  15. H. C. Garcia et al. and T. Şahin. (2021, Feb.). A Tutorial on 5G NR V2X Communications. IEEE Communications Surveys & Tutorials. [Online]. 23(3), pp. 1972-2026. Available: https://doi.org/10.1109/COMST.2021.3057017
  16. Shahgholi, A. Sheikhahmadi, K. Khamforoosh, and S. Azizi. (2021, Feb.). LPWAN-based hybrid backhaul communication for intelligent transportation systems. Architecture and performance evaluation. EURASIP Journal on Wireless Communications and Networking. [Online]. (35). Available: https://doi.org/10.1186/s13638-021-01918-2
  17. Yang, L. Zhao, L. Lei and K. Zheng, “A two-stage allocation scheme for delay-sensitive services in dense vehicular networks,” IEEE International Conference on Communications Workshops (ICC Workshops), Paris, France, 2017, pp. 1358-1363. Available: https://doi.org/10.1109/ICCW.2017.7962848
  18. Liang, S. Xie, G. Y. Li, Z. Ding and X. Yu. (2018, Feb.). Graph-Based Resource Sharing in Vehicular Communication. IEEE Transactions on Wireless Communications. [Online]. 17(7), pp. 4579-4592. Available: https://doi.org/10.1109/TWC.2018.2827958
  19. Chen, B. Wang and R. Zhang. (2018, Oct.). Interference Hypergraph-Based Resource Allocation (IHG-RA) for NOMA-Integrated V2X Networks. IEEE Internet of Things Journal. [Online]. 6(1), pp. 161-170. Available: https://doi.org/10.1109/JIOT.2018.2875670
  20. Bai, W. Chen, K. B. Letaief and Z. Cao. (2010, Dec.). Low Complexity Outage Optimal Distributed Channel Allocation for Vehicle-to-Vehicle Communications. IEEE Journal on Selected Areas in Communications. [Online]. 29(1), pp. 161-172. Available: https://doi.org/10.1109/JSAC.2011.110116
  21. Liang, J. Kim, S. C. Jha, K. Sivanesan and G. Y. Li. (2017, May.). Spectrum and Power Allocation for Vehicular Communications With Delayed CSI Feedback. IEEE Wireless Communications Letters. [Online]. 6(4), pp. 458-461. Available: https://doi.org/10.1109/LWC.2017.2702747
  22. Ye, G. Y. Li and B. -H. F. Juang. (2019, Feb.). Deep Reinforcement Learning Based Resource Allocation for V2V Communications. IEEE Transactions on Vehicular Technology. [Online]. 68(4), pp. 3163-3173. Available: https://doi.org/10.1109/TVT.2019.2897134
  23. Wang, X. Zheng and X. Hou, “A Novel Semi-Distributed Transmission Paradigm for NR V2X,”  in IEEE Globecom Workshops (GC Wkshps), Waikoloa, HI, USA, 2019, pp. 1-6.
  24. Mosavat-Jahromi, Y. Li, L. Cai and L. Lu, “NC-MAC: Network Coding-based Distributed MAC Protocol for Reliable Beacon Broadcasting in V2X,” GLOBECOM 2020 - 2020 IEEE Global Communications Conference, Taipei, Taiwan, 2020, pp. 1-6.
  25. Zappone, M. Di Renzo and M. Debbah. (2019, Jun.). Wireless Networks Design in the Era of Deep Learning: Model-Based, AI-Based, or Both?. IEEE Transactions on Communications. [Online]. 67(10), pp. 7331-7376. Available: https://doi.org/10.1109/TCOMM.2019.2924010
  26. Wang, H. Ye, L. Liang and G. Y. Li. (2020, Mar.). Learn to Compress CSI and Allocate Resources in Vehicular Networks. IEEE Transactions on Communications. [Online]. 68(6), pp. 3640-3653. Available: https://doi.org/10.1109/TCOMM.2020.2979124
  27. Lu, Yan, Ping Wang, Shuai Wang, and Wangding Yao, “A Q-learning based SPS resource scheduling algorithm for reliable C-V2X communication,” in 5th International Conference on Digital Signal Processing, 2021, pp. 201-206.
  28. Busoniu, R. Babuska and B. De Schutter. (2008, Feb.). A Comprehensive Survey of Multiagent Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews). [Online]. 38(2), pp. 156-172. Available: https://doi.org/10.1109/TSMCC.2007.913919
  29. Gu, W. Chen, M. Alazab, X. Tan and M. Guizani. (2022, Jul.). Multiagent Reinforcement Learning-Based Semi-Persistent Scheduling Scheme in C-V2X Mode 4. IEEE Transactions on Vehicular Technology. [Online]. 71(11), pp. 12044-12056. Available: https://doi.org/10.1109/TVT.2022.3189019
  30. Cao and H. Yin, “Resource Allocation for Vehicle Platooning in 5G NR-V2X via Deep Reinforcement Learning,” in 2021 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom), Bucharest, Romania, 2021, pp. 1-7.
  31. Yuan, G. Zheng, K. -K. Wong and K. B. Letaief. (2021, Jul.). Meta-Reinforcement Learning Based Resource Allocation for Dynamic V2X Communications. IEEE Transactions on Vehicular Technology. [Online]. 70(9), pp. 8964-8977. Available: https://doi.org/10.1109/TVT.2021.3098854
  32. He, L. Wang, H. Ye, G. Y. Li and B. -H. F. Juang, “Resource Allocation based on Graph Neural Networks in Vehicular Communications,” GLOBECOM 2020 - 2020 IEEE Global Communications Conference, Taipei, Taiwan, 2020, pp. 1-5.
  33. Wang, X. Jiang, Y. Zhou, Z. Li, D. Wu, T. Tang, A. Fedotov and V. Badenko. (2022, Jun.). Multi-agent reinforcement learning for edge information sharing in vehicular networks. Digital Communications and Networks. [Online]. 8(3), pp. 267-277. Available: https://doi.org/10.1016/j.dcan.2021.08.006
  34. Wu, Z. Liu, F. Liu, T. Yoshinaga, Y. Ji and J. Li. (2020, Jun.). Collaborative Learning of Communication Routes in Edge-Enabled Multi-Access Vehicular Environment. IEEE Transactions on Cognitive Communications and Networking. [Online]. 6(4), pp. 1155-1165. Available: https://doi.org/10.1109/TCCN.2020.3002253
  35. Gu, X. Yang, Z. Lin, W. Hu, M. Alazab and R. Kharel. (2020, Sep.). Multiagent Actor-Critic Network-Based Incentive Mechanism for Mobile Crowdsensing in Industrial Systems. IEEE Transactions on Industrial Informatics. [Online]. 17(9), pp. 6182-6191. Available: https://doi.org/10.1109/TII.2020.3024611
  36. Liang, H. Ye and G. Y. Li. (2019, Aug.). Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning. IEEE Journal on Selected Areas in Communications. [Online]. 37(10), pp. 2282-2292. Available: https://doi.org/10.1109/JSAC.2019.2933962
  37. Tian, Q. Liu, H. Zhang and D. Wu. (2021, Jun.). Multiagent Deep-Reinforcement-Learning-Based Resource Allocation for Heterogeneous QoS Guarantees for Vehicular Networks. IEEE Internet of Things Journal. [Online]. 9(3), pp. 1683-1695. Available: https://doi.org/10.1109/JIOT.2021.3089823
  38. Chen et al. and Y. Zhang. (2020, Jan.). Age of Information Aware Radio Resource Management in Vehicular Networks: A Proactive Deep Reinforcement Learning Perspective. IEEE Transactions on Wireless Communications. [Online]. 19(4), pp. 2268-2281. Available: https://doi.org/10.1109/TWC.2019.2963667
  39. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, 2017,pp. 1273-1282.
  40. Niknam, H. S. Dhillon and J. H. Reed. (2020, Jun.). Federated Learning for Wireless Communications: Motivation, Opportunities, and Challenges. IEEE Communications Magazine. [Online]. 58(6), pp. 46-51. Available: https://doi.org/10.1109/MCOM.001.1900461
  41. Qi, Q. Zhou, L. Lei and Kan Zheng, “Federated reinforcement learning: Techniques, applications, and open challenges,”  2021.
  42. Valente, C. Senna, P. Rito and S. Sargento. (2023, Feb.). Embedded Federated Learning for VANET Environments. Applied Sciences. [Online]. 13(4), p. 2329. Available: https://doi.org/10.3390/app13042329
  43. Wang, X. Fangmin, H. Zhang and C. Zhao. (2022, Apr.). Joint resource management for mobility supported federated learning in Internet of Vehicles. Future Generation Computer Systems.[Online]. 129, pp. 199-211. Available: https://doi.org/10.1016/j.future.2021.11.020
  44. Guo, F. Tang and N. Kato. (2022, Sep.). Federated Reinforcement Learning-Based Resource Allocation in D2D-Enabled 6G. IEEE Network. [Online]. 37(5), pp. 89-95. Available: https://doi.org/10.1109/MNET.122.2200102
  45. Kandukuri and S. Boyd. (2002, Jan.). Optimal power control in interference-limited fading wireless channels with outage-probability specifications. IEEE Transactions on Wireless Communications. [Online]. 1(1), pp. 46-55. Available: https://doi.org/10.1109/7693.975444
  46. Li, L. Lu, W. Ni, A. Jamalipour, D. Zhang and H. Du. (2022, May.). Federated Multi-Agent Deep Reinforcement Learning for Resource Allocation of Vehicle-to-Vehicle Communications. IEEE Transactions on Vehicular Technology. [Online]. 71(8), pp. 8810-8824. Available: https://doi.org/10.1109/TVT.2022.3173057
  47. M. Meredith, “Technical Specification Group Radio Access Network; Study on LTEBased V2X Services; (Release 14),” 3rd Generation Partnership Project, 2016.
  48. Guo, L. Liang and G. Y. Li. (2019, Feb.). Resource Allocation for Low-Latency Vehicular Communications: An Effective Capacity Perspective. IEEE Journal on Selected Areas in Communications. [Online]. 37(4), pp. 905-917. Available: https://doi.org/10.1109/JSAC.2019.2898743
  49. Zeng et al and Y. Wu, “Multi-D3QN: A multi-strategy deep reinforcement learning for service composition in cloud manufacturing,” International Conference on Collaborative Computing: Networking, Applications and Worksharing. Cham: Springer International Publishing, 2021, pp. 225-240.
  50. Yang, X. Xie and M. Kadoch. (2019, Jan.). Intelligent Resource Management Based on Reinforcement Learning for Ultra-Reliable and Low-Latency IoV Communication Networks. IEEE Transactions on Vehicular Technology. [Online]. 68(5), pp. 4157-4169. Available: https://doi.org/10.1109/TVT.2018.2890686
  51. Chen, J. Hu, Y. Shi and L. Zhao. (2016, Sep.). LTE-V: A TD-LTE-Based V2X Solution for Future Vehicular Network. IEEE Internet of Things Journal. [Online]. 3(6), pp. 997-1005. Available: https://doi.org/10.1109/JIOT.2016.2611605
  52. L. Liang, S. Xie, G. Y. Li, Z. Ding and X. Yu. (2018, Apr.). Graph-Based Resource Sharing in Vehicular Communication. IEEE Transactions on Wireless Communications. [Online]. 17(7), pp. 4579-4592. Available: https://doi.org/10.1109/TWC.2018.2827958
Send comment about this article
Enter Name.
Enter a valid email address.
Enter a vaid affiliation.
Enter comments (At leaset 10 words)
CAPTCHA Image
Enter Security Code Correctly.