RIS-Assisted Robust Beamforming for UAV Anti-Jamming and Eavesdropping Communications: A Deep Reinforcement Learning Approach

Abstract

The reconfigurable intelligent surface (RIS) has been widely recognized as a rising paradigm for physical layer security due to its potential to substantially adjust the electromagnetic propagation environment. In this regard, this paper adopted the RIS deployed on an unmanned aerial vehicle (UAV) to enhance information transmission while defending against both jamming and eavesdropping attacks. Furthermore, an innovative deep reinforcement learning (DRL) approach is proposed with the purpose of optimizing the power allocation of the base station (BS) and the discrete phase shifts of the RIS. Specifically, considering the imperfect illegitimate node’s channel state information (CSI), we first reformulated the non-convex and non-conventional original problem into a Markov decision process (MDP) framework. Subsequently, a noisy dueling double-deep Q-network with prioritized experience replay (Noisy-D3QN-PER) algorithm was developed with the objective of maximizing the achievable sum rate while ensuring the fulfillment of the security requirements. Finally, the numerical simulations showed that our proposed algorithm outperformed the baselines on the system rate and at transmission protection level.

1. Introduction

Recently, the advancement of next-generation wireless communications has led to exponential growth in data transmission and connected nodes [1]. However, owing to the open nature of wireless channels, wireless communications are progressively susceptible to active jamming and passive eavesdropping [2,3]. With this as the focus, the academic community has studied various techniques to combat jamming and eavesdropping attacks, e.g., power control [4], frequency hopping [5], artificial-noise-aided beamforming [6], and cooperative relaying scheme [7]. However, power control cannot handle the jamming attacks with high power, and frequency hopping consumes additional spectrum resources. On the other hand, releasing artificial noise consumes extra power, and employing relays may incur additional hardware cost [4,5,6,7].

The above-mentioned shortcomings have motivated a new paradigm called the reconfigurable intelligent surface (RIS) [8]. This technology has recently been regarded as a promising solution for enhancing the power/spectral efficiency of wireless communication systems [8,9,10,11]. Specifically, the RIS consists of massive passive elements, which can dynamically adjust the reflection coefficient on the elements according to the needs of different communication scenarios to increase the received signal power or significantly reduce the impact of interference in the network [9,10,11]. Therefore, the RIS has garnered extensive research attention in the domain of secure communications [12,13,14,15,16,17,18,19,20,21,22,23,24,25]. However, in the face of increasingly complex electromagnetic environments, there is an urgent need for highly efficient and reliable beamforming algorithms for RIS-aided secure communications.

1.1. Related Works

In recent years, several fundamental technical challenges of RIS-assisted secure communication systems have been addressed [12,13,14]. In [12], the joint beamforming scheme was proposed to protect secure transmission from eavesdropping attacks, where several optimization algorithms were applied, including alternating optimization (AO) and semidefinite relaxation (SDR). To maximize the secrecy rate of the RIS-assisted Gaussian multiple-input multiple-output (MIMO) channel, the authors in [14] used the AO algorithm to jointly optimize the transmit covariance at the transmitter and the phase shift coefficient at the RIS and further proposed the minimization–maximization (MM) algorithm to optimize the local optimal phase shift. However, these works assumed that the base station (BS) can acquire the ideal channel state information (CSI) of all nodes, which is impractical due to the uncooperative relationship between the BS and the illegitimate nodes. To tackle this matter, a robust algorithm has been developed to jointly optimize active beamforming and passive reflecting beamforming to secure the wireless transmission system against jammer attacks, where the CSI of illegitimate nodes at the BS is not completely known [15,16,17]. In addition, the authors in [18] iteratively solved an energy-efficient secure transmission problem with the probabilistic outage constraint by low-complexity first-order algorithms in the presence of imperfect information about the eavesdropper’s channel state.

Considering that the actual communication environment may become increasingly complex, such as in densely populated areas with clusters of buildings, the links between the RIS and various nodes may encounter obstacles. Unmanned aerial vehicles (UAVs) have been widely used in complex communication networks due to their low cost and flexible maneuverability [19,20,21,22,23]. In addition, when we mount the RIS on a UAV, the channel attenuation of the ground-to-air channel is much lower than that of the ground channel, which can significantly reduce the energy loss of passive reflection. The authors in [21] utilized UAVs carrying reflective surfaces to facilitate power delivery to intelligent devices, while simultaneously transmitting information. Liu et al. used an AO framework to study a multi-controllable system for RIS-aided UAV communication [22]. In [23], the authors studied the secrecy problem of RIS-based integrated satellite UAV relay networks with multiple eavesdroppers.

The obvious challenge is that traditional convex optimization algorithms may be less efficient for large-scale communication systems. Besides, the practical RIS’s coefficient adjustment is discrete, which makes the traditional algorithms no longer applicable. Benefiting from the rapid development of artificial intelligence (AI), reinforcement learning (RL) has attracted much interest in beamforming design in RIS-assisted wireless communication systems [24,25,26,27,28,29,30], which can effectively deal with the large-scale discrete RIS’s coefficients. The authors in [24] proposed a passive phase shift design to maximize the downlink received signal-to-noise ratio based on deep reinforcement learning (DRL). In [25], DRL and extremum seeking control were incorporated for the purposes of model-free control of the RIS. In response to increased network demand and interference challenges from nearby UAV cells, a direct collaborative-communication-enabled multi-agent decentralized double-deep Q-network (CMAD–DDQN) approach facilitates direct collaboration among UAVs, optimizing their 3D flight trajectories to maximize energy efficiency while outperforming existing methods by up to 85% [26]. However, these works did not explore the issue of the security of AI in RIS-enhanced communication systems. In [27,28], the authors proposed secure DRL-based beamforming methods for protecting RIS-assisted wireless communications from active jamming or passive eavesdropping. Furthermore, in order to maximize the energy efficiency of multi-UAV-assisted wireless coverage, the authors in [29] proposed a cooperative multi-agent decentralized double-deep Q-network (MAD-DDQN) approach, but the algorithm could not be directly applied to optimize the reflecting beamforming for the RIS. To the best of our knowledge, no exiting work has considered the design of DRL in RIS-assisted secure transmission strategies in the presence of both jammers and eavesdroppers and imperfect CSI conditions.

1.2. Contributions

In this paper, we aimed to delve into the anti-jamming and anti-eavesdropping problems in an RIS-assisted UAV transmission system and introduce an innovative robust DRL-based approach to design discrete RIS coefficients in the presence of imperfect CSI from illegitimate nodes. In conclusion, our principal contributions are itemized as follows:

• Considering the illegitimate nodes’ imperfect CSI, the joint optimization problem of power allocation at the BS and reflecting beamforming at the RIS is formulated to maximize the achievable system rate, while ensuring fulfillment of the security requirements.
• To cope with the non-convex and non-conventional optimization problem, we first used a robust method to process the imperfect CSI, and subsequently, the optimization problem was reformulated into a Markov decision process (MDP) framework. Then, a noisy dueling double-deep Q-network with prioritized experience replay (Noisy-D3QN-PER) algorithm with safety performance awareness is proposed, where the D3QN is the improvement of the DQN, the NoisyNet can be encouraged to avoid falling into local optima, and the PER accelerates the convergence.
• The numerical results indicated that the Noisy-D3QN-PER algorithm outperformed conventional approaches in improving the safety performance protection level and achievable sum rate. For example, the proposed algorithm improved the system rate and transmission protection level by 27.43% and 11.11%, respectively, compared to the conventional DQN of the benchmark scheme.

2. System Model and Problem Formulation

2.1. System Model

Figure 1 depicts the secure transmission scenario under consideration, wherein a BS aided by a fixed aerial RIS-UAV seeks to establish dependable links with K single-antenna users in the presence of a smart jammer and a single-antenna eavesdropper. Here, we assumed that the BS and the jammer are equipped with N, $NJ$ antennas, respectively, and the RIS deployed on the UAV has L reflecting units. For the ease of exposition, we further denote the channel matrix between the BS and the RIS-UAV, the smart jammer and the RIS-UAV, the BS and the k-th user, the RIS-UAV and the k-th user, the smart jammer and the k-th user, the BS and the eavesdropper, and the RIS-UAV and the eavesdropper by ${\mathbf{G}}_{\mathrm{BR}}\in {\mathbb{C}}^{L\times N}$, ${\mathbf{G}}_{\mathrm{JR}}\in {\mathbb{C}}^{L\times NJ}$, ${\mathbf{h}}_{\mathrm{BU},k}^H\in {\mathbb{C}}^{1\times N}$, ${\mathbf{h}}_{\mathrm{RU},k}^H\in {\mathbb{C}}^{1\times L}$, ${\mathbf{h}}_{\mathrm{JU},k}^H\in {\mathbb{C}}^{1\times NJ}$, ${\mathbf{h}}_{\mathrm{BE}}^H\in {\mathbb{C}}^{1\times N}$, and ${\mathbf{h}}_{\mathrm{RE}}^H\in {\mathbb{C}}^{1\times L}$, respectively. Due to the cooperation between the legitimate nodes, we assumed that the CSI of the involved legitimate channel $\left\{{\mathbf{G}}_{\mathrm{BR}},{\mathbf{h}}_{\mathrm{BU},k},{\mathbf{h}}_{\mathrm{RU},k}\right\}$ is accurately available at the BS. However, in light of the expectation that illegitimate nodes will not collaborate with the BS to perform channel estimation, we took the practical assumption into account that the CSI of illegitimate channels, namely $\left\{{\mathbf{G}}_{\mathrm{JR}},{\mathbf{h}}_{\mathrm{JU},k},{\mathbf{h}}_{\mathrm{BE}},{\mathbf{h}}_{\mathrm{RE}}\right\}$, cannot be perfectly obtained. To elaborate on this, considering a more-practical and more-general situation, rather than using a statistical or bounded uncertainty model [15], we further characterized the illegitimate CSI as a given angle-based range, i.e.,

$$\begin{array}{ll}\hfill {\Delta }_{J,G} = & \left\{{\mathbf{h}}_{\mathrm{JR}}\right|{\theta }_G^{J,R}\in \left[{\theta }_{G,L}^{J,R},{\theta }_{G,U}^{J,R}\right],{\phi }_G^{J,R}\in \left[{\phi }_{G,L}^{J,R},{\phi }_{G,U}^{J,R}\right],\hfill \\ & \left|g_G^J\right|\in \left[\left|g_{G,L}^J\right|,\left|g_{G,U}^J\right|\right]\},\hfill \end{array}$$

(1)

$$\begin{array}{ll}{\Delta }_{J,h} = & \left\{{\mathbf{h}}_{\mathrm{JU},k}\right|{\theta }_k^J\in \left[{\theta }_{k,L}^J,{\theta }_{k,U}^J\right],{\phi }_k^J\in \left[{\phi }_{k,L}^J,{\phi }_{k,U}^J\right],\hfill \\ & \left|g_k^J\right|\in \left[\left|g_{k,L}^J\right|,\left|g_{k,U}^J\right|\right],\forall k\in K\},\end{array}$$

(2)

$$\begin{array}{ll}{\Delta }_E=& \left\{{\mathbf{h}}_i\right|{\theta }_i^E\in \left[{\theta }_{i,L}^E,{\theta }_{i,U}^E\right],{\phi }_i^E\in \left[{\phi }_{i,L}^E,{\phi }_{i,U}^E\right],\hfill \\ & \left|g_i^E\right|\in \left[\left|g_{i,L}^E\right|,\left|g_{i,U}^E\right|\right],i\in \left(\mathrm{BE},\mathrm{RE}\right)\},\end{array}$$

(3)

where ${\Delta }_J=\left\{{\Delta }_{J,h},{\Delta }_{J,G}\right\}$, ${\theta }_L$ represents the minimum vertical angle of AoD (AoA), while ${\theta }_U$ represents the maximum vertical angle of AoD (AoA). Similarly, ${\phi }_L$ represents the minimum horizontal angle of AoD (AoA), while ${\phi }_U$ represents the maximum horizontal angle of AoD (AoA). Finally, $g_L$ and $g_U$ represent the lower and upper limits of the channel gain amplitude, respectively.

Let $s_k$ be defined as the information symbol transmitted to the k-th user, satisfying $\mathrm{E}\left[s_k\right]$ and $\mathrm{E}\left[{\left|s_k\right|}^2\right]=1$. Before transmission, $s_k$ should be multiplied by the beamforming vector ${\mathbf{w}}_k\in {\mathbb{C}}^{N\times 1}$ satisfying ${∥{\mathbf{w}}_k∥}^2=1$. Consequently, the total transmitted signal at the BS can be written as $\mathbf{x}={\sum }_{k=1}^K\sqrt{P_k}{\mathbf{w}}_k{s}_k$, where $P_k$ denotes the allocated transmit power assigned to the k-th user. Meanwhile, the smart jammer endeavors to disrupt the legitimate communication by transmitting jamming signal ${\mathbf{w}}_{\mathrm{J}}{s}_{\mathrm{J}}\in {\mathbb{C}}^{NJ\times 1}$. As such, the RIS receives the superimposed signals and imposes the phase shift coefficient $\mathbf{\Phi }=diag\left({\beta }_1{e}^{j{\varphi }_1},\dots ,{\beta }_l{e}^{j{\varphi }_l},\dots ,{\beta }_L{e}^{j{\varphi }_L}\right)$ on them, where ${\varphi }_l\in [0,2\pi ]$ and ${\beta }_l\in [0,1]$ represent the phase shift and the amplitude of the l-th RIS reflective element, respectively. Hence, the received signal at the k-th user and the eavesdropper can be, respectively, expressed by

$$\begin{array}{cc}\hfill y_{\mathrm{U},k}=& {\overline{\mathbf{h}}}_{\mathrm{BU},k}\sqrt{P_k}{\mathbf{w}}_k{s}_k+\sum _{i\ne k}{\overline{\mathbf{h}}}_{\mathrm{BU},k}\sqrt{P_i}{\mathbf{w}}_i{s}_i\hfill \end{array}$$

$$\begin{array}{cc}\hfill & +{\overline{\mathbf{h}}}_{\mathrm{JU},k}{\mathbf{w}}_{\mathrm{J}}{s}_{\mathrm{J}}+n_{\mathrm{U},k},\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill y_{\mathrm{E}}=& {\overline{\mathbf{h}}}_{\mathrm{BE}}\sqrt{P_k}{\mathbf{w}}_k{s}_k+\sum _{i\ne k}{\overline{\mathbf{h}}}_{\mathrm{BE}}\sqrt{P_i}{\mathbf{w}}_i{s}_i+n_{\mathrm{E}},\hfill \end{array}$$

(5)

where ${\overline{\mathbf{h}}}_{\mathrm{BU},k}={\mathbf{h}}_{\mathrm{RU},k}^H\mathbf{\Phi }{\mathbf{G}}_{\mathrm{BR}}+{\mathbf{h}}_{\mathrm{BU},k}^H$, ${\overline{\mathbf{h}}}_{\mathrm{JU},k}={\mathbf{h}}_{\mathrm{RU},k}^H\mathbf{\Phi }{\mathbf{G}}_{\mathrm{BR}}+{\mathbf{h}}_{\mathrm{JU},k}^H$, ${\overline{\mathbf{h}}}_{\mathrm{BE}}={\mathbf{h}}_{\mathrm{RE},k}^H\mathbf{\Phi }{\mathbf{G}}_{\mathrm{BR}}+{\mathbf{h}}_{\mathrm{BE}}^H$. The symbol $n_{\mathrm{U},k}\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }_{\mathrm{U},k}^2\right)$ represents the additive white Gaussian noise (AWGN) at the k-th user, and $n_{\mathrm{E}}\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }_{\mathrm{E}}^2\right)$ is the AWGN at the eavesdropper. Hence, the achievable system rate of the k-th user and the wiretap rate of the eavesdropper can be, respectively, expressed as

$$\begin{array}{cc}\hfill & R_{\mathrm{U},k}={log}_2\left(1+\frac{P_k{\left|{\tilde{\mathbf{h}}}_{\mathrm{BU},k}{\mathbf{w}}_k\right|}^2}{{\displaystyle \sum _{i\ne k}}{P}_i{\left|{\tilde{\mathbf{h}}}_{\mathrm{BU},k}{\mathbf{w}}_i\right|}^2+{\left|{\tilde{\mathbf{h}}}_{\mathrm{JU},k}{\mathbf{w}}_{\mathrm{J}}\right|}^2+{\sigma }_{U,k}^2}\right),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & R_{\mathrm{E},k}={log}_2\left(1+\frac{P_k{\left|{\tilde{\mathbf{h}}}_{\mathrm{BE}}{\mathbf{w}}_k\right|}^2}{{\displaystyle \sum _{i\ne k}}{P}_i{\left|{\tilde{\mathbf{h}}}_{\mathrm{BE}}{\mathbf{w}}_i\right|}^2+{\sigma }_E^2}\right).\hfill \end{array}$$

(7)

The secrecy rate of the k-th user can be written as

$$\begin{array}{c}\hfill R_{sec,k}={\left[R_{\mathrm{U},k}-R_{\mathrm{E},k}\right]}^{+},\end{array}$$

(8)

where ${\left[z\right]}^{+}=max(z,0)$.

2.2. Problem Formulation

Our objective is to maximize the achievable sum rate through jointly optimizing the transmit power allocation ${\left\{P_k\right\}}_{k\in \mathrm{K}}$ and the reflecting beamforming matrix $\mathbf{\Phi }$ under the imperfect illegitimate node’s CSI, while meeting the worst-case secrecy/achievable rate constraints. As such, the optimization problem can be formulated as

$$\begin{array}{cc}\hfill \mathcal{F}:& \underset{{\left\{P_k\right\}}_{k\in K},\mathbf{\Phi }}{\max }\underset{\Delta J}{min}\sum _{k\in K}{R}_{\mathrm{U},k},\hfill \\ \hfill & s.t.\mathrm{C}1:\underset{\Delta E}{min}{R}_{sec,k}\ge R_{{}_{sec,k}}^{min},\forall k\in K,\hfill \\ \hfill & \mathrm{C}2:\underset{\Delta J}{min}{R}_{\mathrm{U},k}\ge R_{{}_k}^{min},\forall k\in K,\hfill \\ \hfill & \mathrm{C}3:{\sum }_{k=1}^K{P}_k\le P_{max},\hfill \\ \hfill & \mathrm{C}4:\left|{\beta }_l{e}^{j{\varphi }_l}\right|=1,0\le {\theta }_l\le 2\pi ,\forall l\in L,\hfill \end{array}$$

(9)

where $R_{{}_{sec,k}}^{min}$ and $R_{{}_k}^{min}$ represent the minimum secrecy rate and the target rate of the k-th user. The power allocation is restricted to $\mathrm{C}3$ due to the limited energy supply at the BS, and $P_{\max }$ is the BS’s maximum transmit power. Note that, due to the non-convexity of both the objective function and the constraints, (9) is a non-convex and non-trivial problem. Many existing traditional optimization methods, such as the SDR algorithm and the AO algorithm, obtain the solution in each time slot, where the correlation of consecutive instants is ignored, and phase adjustment is usually discrete in form on practical RIS elements, which leads traditional methods to no longer be applicable. In addition, in the scenario we are considering, the jammer is intelligent and can change the unknown jamming strategy in real-time. In order to be able to optimize in real-time and from the perspective of long-term interests, instead of directly solving this problem mathematically, we propose a robust DRL-based approach that can constantly interact with the environment that contains eavesdroppers and smart jammers to learn the optimal solution.

3. DRL-Based Algorithm Design

3.1. Robust Channel Processing

As stated in Section 2, the imperfect CSI results in infinite non-convexity in both the objective function and constraints. With this as the focus, according to the works [28,29,30], the equivalent worst-case CSI of the illegitimate channel that can be obtained through utilizing the discretization method is given, respectively, by

$$\begin{array}{cc}\hfill & {\tilde{\mathbf{G}}}_{\mathrm{JR}}={\sum }_{i_1=1}^{N{J}_1}{\sum }_{i_2=1}^{N{J}_2}{\sum }_{j_1}^{N_1}{\sum }_{j_2}^{N_2}(1/\left(Nj+N\right)){\mathbf{G}}_{\mathrm{JR}}^{\left(i_1,i_2\right)},\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & {\tilde{\mathbf{h}}}_{\mathrm{M}}={\sum }_{i_1=1}^{M_{N_1}}{\sum }_{i_2=1}^{M_{N_2}}(1/M_N){\mathbf{h}}_{\mathrm{M}}^{\left(i_1,i_2\right)},\hfill \end{array}$$

(11)

where $\mathrm{M}\in \left(\mathrm{BE},\mathrm{RE},\{\mathrm{JU},k\}\right)$, $M_N\in \left(N,L,NJ\right)$, and ${\mathbf{G}}_{\mathrm{JR}}^{\left(i_1,i_2\right)}$, ${\mathbf{h}}_{\mathrm{JU},k}^{\left(i_1,i_2\right)}$, ${\mathbf{h}}_{\mathrm{BE}}^{\left(i_1,i_2\right)}$, ${\mathbf{h}}_{\mathrm{RE}}^{\left(i_1,i_2\right)}$ are the discrete CSI by uniformly discretizing all the angles in the set of ${\Delta }_J$ and ${\Delta }_U$, respectively, i.e.,

$$\begin{array}{cc}\hfill {\theta }^{\left(i_1\right)}& ={\theta }_L+(i_1-1)\left({\theta }_U-{\theta }_L\right)/\left(Q_1-1\right),i_1=1,\dots ,Q_1,\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\phi }^{\left(i_2\right)}& ={\phi }_L+(i_2-1)\left({\phi }_U-{\phi }_L\right)/\left(Q_2-1\right),i_2=1,\dots ,Q_2,\hfill \end{array}$$

(13)

where $Q_1$ and $Q_2$ are the sample numbers of $\theta$ and $\phi$. Here, the detail is omitted for brevity, which can be referenced in [31,32].

3.2. Overview of DRL

DRL amalgamates the feature acquisition prowess inherent to deep learning (DL) with the decision-making capabilities intrinsic to RL. It comprises two fundamental constituents: the agent and the environment. The agent continuously improves its strategy by receiving feedback through interactions with the environment to achieve maximum return. This learning process is described as an MDP [33]. The MDP framework can be defined by a tuple $\left\{S,A,P,R\right\}$. Herein, S represents the state space denoting the set of observations characterizing the environment. A denotes the set of potential choices. P is the state transition probability denoting the distribution of the next state $s_{t+1}$ given the action $a_t$ taken in the current state $s_t$. Lastly, R is the immediate reward, which provides the quality evaluation $r_t\left(s_t,a_t\right)$ of the state–action pair $\left(s_t,a_t\right)$. At each time step t, the agent obtains the state $s_t\in S$ from the environment and executes an action $a_t\in A$ according to the policy function $\pi \left(a_t|s_t\right)=\mathrm{Pr}\left(A_t=a_t|S_t=s_t\right)$. Subsequently, the environment will transit to a new state $s_{t+1}$ with probability $\mathrm{P}\left(s_{t+1}|s_t,a_t\right)=\mathrm{Pr}\left(S_{t+1}=s_{t+1}|S_t=s_t,A_t=a_t\right)$; in the meantime, the agent will receive the immediate reward $r_t\in R$. The agent aims at learning strategies maximizing the long-term reward, i.e., the cumulative discounted future reward $U_t={\sum }_{\tau =0}^{\infty }{\gamma }^{\tau }{R}_{t+\tau +1}$, where $\gamma \in \left[0,1\right]$ is the discount factor. Therefore, the tuples $\left(s_1,a_1,r_1,s_2,\dots ,s_{t-1},a_{t-1},r_{t-1},s_t\right)$ constitute the trajectory in an episode used for the iterative updating of the agent.

To accommodate the proposed algorithm in our problem, we first reformulated Problem (9) into an MDP framework. The corresponding elements of the MDP problem are specified as follows:

$\mathbf{State}$ $\mathcal{S}$: The state $s_t$ fed back from the RIS-UAV-assisted communication system is given as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{{\left\{{\mathbf{h}}_k^t\right\}}_{k\in K},\left\{{\mathbf{h}}_e^t\right\},{\left\{R_{\mathrm{U},k}^{t-1}\right\}}_{k\in K}\right\},\end{array}\end{array}$$

(14)

where ${\mathbf{h}}_k$ and ${\mathbf{h}}_e$ denote the composite channel coefficients of the k-th user and eavesdropper, respectively.

$\mathbf{Action}$ $\mathcal{A}$: Based on the current state $s_t$, the agent needs to make a coordinated decision on the phase shift at the RIS and the power allocation at the BS. Hence, the action $a_t$ at each time step t is given as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill a_t=\left\{{\left\{\Delta {\varphi }_l\right\}}_{l\in L},{\left\{\Delta P_k\right\}}_{k\in K}\right\},\end{array}\end{array}$$

(15)

where $\Delta {\varphi }_l\in \left\{-\frac{\pi }{4},0,\frac{\pi }{4}\right\}$ is the variable for the phase shift of the l-th reflection element and $\Delta P_k\in \left\{-\tilde{p},0,\tilde{p}\right\}$ is the variable for assigning the k-th user’s transmit power.

$\mathbf{Reward}$ $\mathcal{R}$: Our goal was not only to maximize the achievable rate, but also to ensure the system safety performance requirements. Therefore, we designed a composite reward function expressed as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill r_t=\underset{\mathrm{basic}}{\underbrace{{\displaystyle \sum _{k\in K}}{R}_{\mathrm{U},k}}}-\underset{\mathrm{penalty}}{\underbrace{{\displaystyle \sum _{k\in K}}{\rho }_1{p}_{\mathrm{U},k}-{\displaystyle \sum _{k\in K}}{\rho }_2{p}_{\mathrm{E},k}}},\end{array}\end{array}$$

(16)

where

$$\begin{array}{c}\hfill p_{\mathrm{E},\mathrm{k}}=\left\{\begin{array}{c}1, if R_{sec,k}<R_{sec,k}^{min},\forall k\in K,\hfill \\ 0, otherwise.\hfill \end{array}\right.,\end{array}$$

(17)

$$\begin{array}{c}\hfill p_{\mathrm{U},\mathrm{k}}=\left\{\begin{array}{c}1, if R_{\mathrm{U},\mathrm{k}}<R_{\mathrm{U},\mathrm{k}}^{min},\forall k\in K,\hfill \\ 0, otherwise.\hfill \end{array}\right..\end{array}$$

(18)

In (16), the base reward is the sum of the rates of all users, and when the constraints in (17) or (18) are not satisfied, we add a penalty term to encourage the agent’s behavioral strategy to be closer to our needs. The coefficients ${\rho }_1$ and ${\rho }_2$ are the positive constants.

With DRL, a well-known function measuring the expected return for the agent to execute action $a_t$ in the state $s_t$ under the policy $\pi$ is the action value function Q:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{\pi }\left(s_t,a_t;w\right)=\mathrm{E}\left[U_t|S_t=s_t,A_t=a_t\right],\end{array}\end{array}$$

(19)

where $w$ represents the parameters of the deep neural networks (DNNs). In the learning process, the agent intends to find optimal policy ${\pi }^{*}$. Thus, the optimal Q function is expressed as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{*}\left(s,a\right)=\underset{\pi }{max}{Q}_{\pi }\left(s,a\right),\forall s\in S,a\in A.\end{array}\end{array}$$

(20)

In order to obtain the above equation, the optimal Q function can be constantly approximate by updating the parameter $w$ using the temporal difference (TD) algorithm:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill w_{t+1}=w_t-\alpha {\nabla }_{w}L\left(w\right),\end{array}\end{array}$$

(21)

where $\alpha \in \left(0,1\right)$ is the learning rate for the update on $w$ and ${\nabla }_{w}L\left(w\right)$ is the gradient of the loss function $L\left(w\right)$ with respect to $w$, which is given by

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L\left(w\right)={\left[r_t+\gamma \underset{a\in A}{max}Q\left(s_{t+1},a;w\right)-Q\left(s_t,a_t;w\right)\right]}^2,\end{array}\end{array}$$

(22)

where $r_t+\gamma \underset{a\in A}{max}Q\left(s_{t+1},a;w\right)$ refers to the TD target value.

3.3. Joint Power Allocation and Reflecting Beamforming Using Noisy-D3QN-PER

Prevailing reinforcement learning techniques, such as Q-learning, the policy gradient, and the deep Q-network (DQN), have demonstrated notable accomplishments in diverse control tasks. However, regarding the safety beamforming policy requirements discussed in Section 2, the policy gradient algorithm is inadequate for addressing Problem (9), as it involves continuous action space optimization and may converge to suboptimal solutions [34]. Furthermore, although the DQN performs well in in environments characterized by high-dimensional continuous state spaces and discrete action spaces, it remains plagued by several inherent limitations, which adversely affect algorithmic efficacy [35]. Therefore, the Noisy-D3QN-PER algorithm was developed to deal with the challenges in this paper, as shown in Figure 2, which can overcome the constraints associated with the aforementioned methods and significantly enhance the attainable performance.

It is noteworthy that a significant disadvantage inherent to the DQN algorithm is over-estimation of the Q function value. The overestimation issue is primarily attributable to two principal factors. First, the process of maximization causes the target value to overestimate the value of the true value. Second, bootstrapping engenders the propagation of bias. In order to address this issue, the double-DQN was adopted in the algorithm [36]. We applied another neural network, i.e., the target network $Q_{\pi }(s_t,a_t;w^{-})$, whose neural network architecture is identical to that of the primary network, but the parameter $w^{-}$ is different from $w$. Specifically, the primary network was used to choose an action that maximizes the output of the Q function $a^{*}=\underset{a\in A}{argmax}Q(s_{t+1},a;w)$, and then, the target network calculates the TD target value $r_t+\gamma Q\left(s_{t+1},a^{*};w^{-}\right)$ with the selected action. Thus, the primary network parameter is updated with the following loss function:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill L\left(w\right)={\left[r_t+\gamma Q\left(s_{t+1},a^{*};w^{-}\right)-Q\left(s_t,a_t;w\right)\right]}^2.\end{array}\end{array}$$

(23)

Subsequently, the parameter of the target network is updated with $w$ and $w^{-}$ every regular interval.

In order to further enhance the algorithm’s performance, we incorporated the dueling layer [37], resulting in the formation of the dueling double-DQN (D3QN). The core concept underlying the dueling layer is the decomposition of the optimal action value $Q_{*}$ into the optimal state value $V_{*}$ and the optimal advantage $D_{*}$. As such, the expression of the optimal advantage function is formulated as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill D_{*}\left(s,a\right)\stackrel{\Delta }{=}{Q}_{*}\left(s,a\right)-V_{*}\left(s\right).\end{array}\end{array}$$

(24)

The advantage of modeling the state value function and the advantage function separately is that, in some specific situations, agents only pay attention to the value of the state and do not care about the differences caused by different actions. More specifically, in the optimization problem we are considering, the state values differ greatly, while the action in the same state differs little. The agent pays attention to the difference in the advantage value of different actions, which makes the algorithm converge more stably. As shown in Figure 3, the dueling layer comprises two distinct neural networks. The neural network denoted by $D\left(s,a;w^D\right)$ is an approximation of the optimal advantage function $D_{*}\left(s,a\right)$, and the other neural network is $V(s;w^V)$, which is an approximation of the optimal state value function $V_{*}\left(s\right)$. The corresponding optimal action value function can be approximated as the following neural network:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q(s,a;w)\stackrel{\Delta }{=}V(s;w^V)+D(s,a;w^D)-\underset{a\in A}{\mathrm{mean}}D(s,a;w^D)\end{array}\end{array}$$

(25)

where $\underset{a\in A}{\mathrm{mean}}D(s,a;w^D)$ ensures the stability of the parameters in the training process and $w\stackrel{\Delta }{=}(w^V;w^D)$, since, at each iteration, the function $V(s;w^V)$ is updated, which also affects the action value of the other actions.

In addition, there is a dilemma of exploration and exploitation in RL that greatly affects the performance of the algorithm. By gathering more information, or sufficient information, the agent can achieve the optimal long-term strategies on a macro-level at the expense of some short-term benefits. In an effort to attain a good tradeoff between exploration and exploitation, several basic strategies have been proposed, such as Boltzmann exploration and the $\epsilon$-greedy policy. However, these methods only utilize action dithering, which results in a low exploration rate, especially in complex and unstable environments. Therefore, we propose a NoisyNet technique to improve the exploration efficiency, i.e., adding parameterized noise to the DNN layer [38]. Specifically, as shown in Figure 4, the weight parameter $w$ of the DNN is replaced with

$$\begin{array}{c}\hfill w=\mathit{\mu }+\mathit{\sigma }\circ \mathit{\xi },\end{array}$$

(26)

where $\mathit{\mu }$ and $\mathit{\sigma }$ are learnable parameters and denote the mean and standard deviation, respectively, and $\mathit{\xi }\sim \mathcal{N}\left(0,1\right)$ is the noise. Here, the term ∘ denotes the multiplication of the corresponding elements, i.e.,

$$\begin{array}{c}\hfill w_{ij}={\mathit{\mu }}_{ij}+{\mathit{\sigma }}_{ij}{\mathit{\xi }}_{ij}.\end{array}$$

(27)

Hence, the Q function is written as

$$\begin{array}{c}\hfill \tilde{Q}(s,a,\xi ;\mu ,\sigma )\stackrel{\Delta }{=}Q(s,a;\mu +\sigma \circ \xi ).\end{array}$$

(28)

The loss function can be further rewritten as

$$\begin{array}{c}\hfill L(\mathit{\mu },\mathit{\sigma })={(r_t+\gamma \tilde{Q}(s_{t+1},a^{*},{\mathit{\xi }}^{\mathbf{\prime }};{\mathit{\mu }}^{-},{\mathit{\sigma }}^{-})-\tilde{Q}(s_t,a_t,\xi ;\mathit{\mu },\mathit{\sigma }))}^2,\end{array}$$

(29)

where $\underset{a\in A}{a^{*}=argmax}\tilde{Q}(s_{t+1},a,\mathit{\xi };\mathit{\mu },\mathit{\sigma })$, and the noise value $\mathit{\xi }$ is different from ${\mathit{\xi }}^{\mathbf{\prime }}$. In the training process, noise is added to the training parameters to force the algorithm to minimize the error in the case of parameters with noise, which means that it is forced to tolerate the disturbance of the parameters. It does not matter if the parameters are not strictly equal to the mean; as long as the parameters are in the neighborhood of the mean, the prediction made by the agent can be reasonable. Therefore, the NoisyNet is not only beneficial to enhance exploration, but also to enhance the robustness of the algorithm.

Experience replay is often utilized in the classical DQN to store and uniformly sample experience transitions, which help in reusing experiences and breaking the correlation of experience transition sequences. However, due to the uncertainty of the jamming strategy of the jammer, the importance of different transitions is different, and uniformly sampling may be ineffective. Hence, we adopted prioritized experience replay (PER) to make the algorithm learn more efficiently and converge faster [39]. PER non-uniformly samples each transition, where the priority of the transition is proportional to its TD error value. Therefore, the sampling probability of transition j is given by

$$\begin{array}{c}\hfill P\left(j\right)={\left|{\delta }_i\right|}^{{\alpha }_j}/{\sum }_n{\left|{\delta }_n\right|}^{{\alpha }_j},\end{array}$$

(30)

where ${\alpha }_j$ adjusts the importance of the priority. In addition, the loss function needs to be multiplied by importance sampling weights to counteract the bias caused by varying the sampling probabilities. Thus, the parameters of the proposed algorithm are update by a mini-batch transition:

$$\begin{array}{c}\hfill {\mathit{\sigma }}_{t+1}={\mathit{\sigma }}_t-{\alpha }_{\sigma }{\nabla }_{\mathit{\sigma }}\frac{1}{m}\sum _{j=1}^m\left({(N·P(j\left)\right)}^{-\varpi }{L}_j(\mathit{\mu },\mathit{\sigma })\right),\end{array}$$

(31)

$$\begin{array}{c}\hfill {\mathit{\mu }}_{t+1}={\mathit{\mu }}_t-{\alpha }_{\mu }{\nabla }_{\mathit{\mu }}\frac{1}{m}\sum _{j=1}^m\left({(N·P(j\left)\right)}^{-\varpi }{L}_j(\mathit{\mu },\mathit{\sigma })\right),\end{array}$$

(32)

where ${\alpha }_{\sigma }$ and ${\alpha }_{\mu }$ are the learning rate, m is the mini-batch size, N represents the number of samples in the buffer, and $\varpi \in \left(0,1\right)$ is a hyperparameter that determines the extent to which PER affects the convergence result.

The detailed training process of the Noisy-D3QN-PER algorithm is shown in Algorithm 1. At the beginning of the training, we sample new channel realizations and randomly choose the phase shifts and power allocation to compute the first state $s_0$. Since the NoisyNet is inherently random, exploration can be encouraged. Based on the current state $s_t$, the $\epsilon$-greedy policy is implemented to select action $a_t$ and, subsequently, receive feedback reward $r_t$ and the next state $s_t+1$. The transition sequence $(s_t,a_t,r_t,s_{t+1})$ is saved in the experience replay buffer $\mathcal{D}$. After storing enough experiences transitions, the training of the primary networks starts, and mini-batch transitions are selected according to the PER principle and put into the neural networks to obtain the loss function according to Equation (29). Then, the parameters of the primary networks are updated by the Adam optimizer according to Equations (31) and (32), and the target network copies the parameters of the primary networks in every $T_{NET}$ time interval. In addition, each time the experience transitions are sampled, the selected transitions need to update the priority with the new TD error.

Algorithm 1 Noisy-D3QN-PER algorithm

Require: environment simulator, experience replay buffer $\mathcal{D}$, learning rate ${\alpha }_{\sigma }$ and ${\alpha }_{\mu }$, mini-batch size m. 1: Initialize: experience replay buffer $\mathcal{D}$ with size D, mini-batch size m, primary network parameters $(\mu ,\sigma )$, target network parameters $({\mu }^{-}=\mu ,{\sigma }^{-}=\sigma )$. 2: for each episode = 1, 2, …, $N^{\mathrm{epi}}$ do 3:    Perceive an initial system state s. 4:    for each step = 1, 2, …, T do 5:      Select action $a_t$ using $\epsilon$-greedy policy, i.e., select the action that yields the largest action value with a probability of $1-\epsilon$, or randomly select from all the possible actions with the probability of $\epsilon$. 6:      Receive an instantaneous reward $r_t$, and obtain the next state $s_{t+1}$. 7:      Store the experience transitions $(s_t,a_t,r_t,s_{t+1})$. 8:      if $\left|D\right|\ge m$ then 9:         Sample mini-batch transitions based on PER using (30), and then, update the priority of the selected transition based on its TD error. 10:         Calculate the loss function for the mini-batch according to (29). 11:         Perform gradient descent, and update the parameters of the primary networks using (31) and (32). 12:         if $tmod{T}_{NET}=0$ then 13:           target network copies the parameters of the primary networks. 14:         end if 15:      end if 16:    end for 17: end for Ensure: joint power allocation and RIS phase shift design strategy.

4. Simulation Results

This section presents an evaluation of the Noisy-D3QN-PER algorithm. We varied the maximum transmission power $Pmax$ between 10 dBm and 30 dBm. The number of antennas on both the BS and the jammer were $N=NJ=64$, and the number of users was $K=2$. The fixed deployment height of the RIS-UAV was 100 m. The minimum secrecy rate and target data rate were $R_{sec,k}^{min}=0.5$ bits/s/Hz and $R_k^{min}=1$ bits/s/Hz, respectively. The background noise at each user and eavesdropper was set to ${\sigma }_{\mathrm{U},k}^2={\sigma }_{\mathrm{E}}^2=-90$ dBm. All involved neural networks were considered to be fully connected. The learning rates ${\alpha }_{\sigma }$ and ${\alpha }_{\mu }$ were set as $\alpha =0.001$. The initial exploration rate $\epsilon$ was 1, then was linearly annealed to 0.1. The parameters ${\rho }_1$ and ${\rho }_2$ in (12) were set to ${\rho }_1={\rho }_2=2$. The replay buffer size was $D=$ 100,000, and the mini-batch size was $m=32$. In addition, the jammer chooses power was from 10 dBm to 30 dBm based on its own jamming strategy, which the BS could not access. Besides, we chose three conventional approaches as benchmarks, namely the classical DQN, the DDQN, and the optimal transmit power allocation without the RIS approach. All of the displayed illustrations are the average results of over 100 independently executed implementations.

Figure 5 shows the average gain graph of the Noisy-D3QN-PER algorithm and the benchmark algorithm. It can be observed that, in the initial phase of training, the algorithms obtained approximately the same reward gain. However, after 100 episodes of training, the Noisy-D3QN-PER algorithm significantly achieved higher gains and faster convergence compared to the benchmark algorithm. This was due to the fact that the preferred empirical playback and competition layers included in the proposed algorithm were better able to adapt to the dynamic and complex interference environment. Specifically, the dueling layer helps to analyze the state bias due to unknown jammer power and unknown location information, and the NoisyNet encourages the exploration of more reflecting beamforming strategies for higher long-term benefits. Moreover, it can be observed that both the DDQN and the proposed algorithm outperformed the classical DQN, which suggests that the use of the DDQN can effectively mitigate the overestimation problem.

Figure 6 shows the achievable sum rate with varying maximum transmit power $P_{max}$. Here, we set $L=64$. As expected, the proposed algorithm outperformed other approaches. This was because the dueling layers modeling the advantage function and the state value function separately can better focus on states that are less correlated with the current strategy–action relationship and better predict the jammer’s strategy when the transmit power changes. Besides, the NoisyNet can prevent the proposed algorithm from becoming stuck at the undesired suboptimal solutions. It can be also observed that the three RIS-UAV-assisted approaches can obtain a much higher achievable rate than that without the RIS, which indicates that deploying the RIS-UAV can efficiently enhance the secure performance. To elaborate on this, the system can enhance the desired signals at the users and eliminate the jamming signal by adjusting the reflecting beamforming at the RIS.

To further highlight the security performance enhancement of the proposed algorithm, the security requirement satisfaction probability (the probability of the satisfaction of the rate constraints [27,28]) of different approaches is shown in Figure 7. It is evident from the figure that the security performance of the optimal PA without the RIS approach cannot be guaranteed when the $P_{max}$ is low, and the security performance protection improved until $P_{max}$ was raised to a certain value. However, the other approaches with the RIS-UAV can obtain satisfactory performance at different $P_{max}$, which further confirmed the superiority of deploying the RIS-UAV in wireless communication systems. Furthermore, it is noteworthy that the proposed algorithm achieved the best result as compared to other conventional approaches. This can be explained by the fact that the comparison approaches usually fell into the suboptimal solution, which only increased the achievable sum rate, but ignored the security performance requirement. However, due to the adopted NoisyNet and the security-aware reward function, the proposed Noisy-D3QN-PER algorithm can explore strategies and make a desirable balance between the security performance and the achievable rate.

5. Conclusions

This paper delved into the optimization of joint power allocation and reflecting beamforming regarding secure communication via RIS-UAV assistance with imperfect CSI. Specifically, the original optimization problem was formulated into an MDP framework and solved by a Noisy-D3QN-PER algorithm, in which the agent can estimate the unknown jamming strategy through constantly interacting with the environment to quickly adapt to the dynamic environment and, finally, obtain the optimal policy that maximizes the achievable rate and meets the requirements of system security performance, which provides technical support for the realization of the intelligence of the RIS-assisted robust beamforming system. The numerical results confirmed the predominance of the proposed Noisy-D3QN-PER algorithm over other existing conventional approaches in improving the achievable sum rate and system security performance. Although the method proposed in this paper can effectively resist the jamming attack with the uncertainty of the CSI, it is still necessary to know the variation range of interference. The next step needs to focus on the following two aspects of research: one is to study the anti-jamming method without any interference information; the other is to explore the AI interpretability, to improve the trustworthiness and effectiveness of the AI method.