Reinforcement Learning-Based Joint Cooperation Clustering and Content Caching paper reading notes

I. Introduction

Cell-free massive multiple -input multiple-output (CF-mMIMO ) [1] has recently been proposed as a promising fifth-generation (B5G) wireless communication technology. In the CF-mMIMO network, there are a large number of access points (APs). Each access point has an antenna, distributed over a large area, and uses the same time-frequency resources to jointly serve users. This scenario is similar to operating ubiquitous small cell base stations (BSs), but without cell boundary constraints (i.e., cell freedom). By taking advantage of distributed MIMO and massive MIMO, CF-mMIMO has good propagation properties and higher performance than traditional co-located MIMO (co-located mMIMO) [1] - [ 3 ] . Energy efficiency ( enhanced energy efficiency, EE ) performance. It only requires simple signal processing, such as local conjugate beamforming based on partial channel state information (CSI) , to achieve interference mitigation [3].

In CF-mMIMO without cell boundaries, users can freely connect to one or more APs. The task of determining user-AP associations is called cooperation cluster formation . Dynamic collaborative clustering of CF-mMIMO systems has been studied [3]-[6]. Ngo et al. [3] proposed a user -centric AP selection scheme to reduce backhaul power consumption and improve the energy efficiency of the network. Riera-Palou et al. [4] used K-means clustering algorithm to develop the optimal connectivity pattern between AP and users to minimize pilot contamination . Le et al. [5] developed a learning scheme based on K-means clustering, whose goal is to maximize the spectral efficiency . Björnson and Sanguinetti [6] proposed a cooperation clustering framework that can scale as the number of users increases.

Content caching is a technology that stores popular content files at the edge of the network to improve network throughputs and reduce service delays . Caching policies and user association have been studied in various cache-enabled networks [7]–[13]. Wei et al. [7] considered a joint user scheduling and caching strategy to minimize transmission delays in cellular networks using deep reinforcement learning (DRL) algorithms. However, only the centralized caching capability of macro BS is considered, and each user can only be associated with one BS. Li et al. [8] believe that all small BSs and macro BSs can cache files, aiming to find the optimal caching strategy through deep Q-learning to maximize the overall energy efficiency. However, only random user-bs associations are considered. Sadeghi et al. [9] use Q-learning to optimize caching strategies for individual units instead of the entire network. Yang et al. [10] studied content caching in heterogeneous networks composed of BSs, relays, and device-to-device (D2D) communication devices. However, the requesting user is only served by the nearest node with the maximum received power. Jing et al. [11] considered caching and user association in cache-enabled small cell networks and solved the problem by decomposing it into two sub-problems to optimize the effective capacity. However, under cell restrictions, each user can only be associated with one small cell BS. Chan and Chien [12] introduced the concept of joint caching and user association design in hierarchical heterogeneous networks, where each user is only allowed to associate with a single-antenna BS. Lin et al. [13] studied caching strategies in coordinated multipoint (CoMP)-enabled cellular networks , where edge users are allowed to connect to nearby BSs, but still Limited by the honeycomb structure.

In this work, we consider the joint cooperative clustering and content caching problem in CF-mMIMO networks with previously unexamined cell structure constraints . The goal is to determine content caching and user-AP association on the AP such that network EE, defined by the achievable network sum rate divided by the network power consumption , is maximized.

Note that this question involves trade-offs. User-AP association based on channel quality can increase the overall network rate, but may produce lower content hit rates ( content hit rates ), thus requiring additional power consumption when retrieving content from the backbone/backhaul. In contrast, user-AP association based on AP cache status can reduce power consumption related to content retrieval ( content retrieval ), but may not provide good network rates. This motivates joint design, as considered in this article. The main contributions of the paper are summarized as follows:

• 我们提出了基于深度确定性策略梯度(deep deterministic policy gradient, DDPG)的方法来解决 CF-mMIMO 网络中复杂的联合合作聚类（joint cooperation clustering）和内容缓存问题，其中在大型网络中难以找到全局最优解。DDPG算法基于参与者-评论家网络体系结构（actor-critic network architecture），其中参与者由奖励(网络EE)训练，以输出良好的策略值（good policy values），即合作聚类和内容缓存策略（cooperation clustering and content caching strategies）。

• 所提出的方法展示了良好的性能，在小型网络中接近最优的穷举搜索方法（optimal exhaustive search approach），并由于明智的联合设计（judicious joint design），在大型网络中优于基准（benchmarks），如基于信噪比(SNR)的聚类和基于流行度的缓存（popularity-based caching）。

论文的其余部分组织如下。Sec. II 描述了CF-mMIMO的系统模型和问题的表述。Sec. III 介绍了提出的 DRL-based 的方法。Sec. IV 给出了仿真结果和讨论。最后，Sec. V 对全文进行总结。

---

II. SYSTEM MODEL

A. 信号模型

我们考虑一个具有 $M$ 个AP和 $K$ 个用户（或用户设备，UE）的下行单元的大规模MIMO网络。$\mathcal{M}$ 和 $\mathcal{K}$ 分别表示所有 AP 和 UE 的集合。图1描绘了一个拓扑的例子（example topology）。AP和UE各配置一根天线。每个AP通过物理光纤链路连接到地理上最近的中央处理单元(CPU)。在时分双工(TDD)操作下，选定的AP子集使用相同的时频资源共同服务于选定的UE子集[3]。设第 $m$ 个AP与第 $k$ 个UE之间的信道为 $g_{mk}={\left(d_{mk}/d_0\right)}^{-a}{h}_{mk}$, where $d_{mk}$is $m$ AP andkkthThe distance between$k$$d_0={\min }_{m,k}{d}_{mk}$is the reference distance, $\alpha$ is the path loss index ($a\ge 2$），$h_{mk}\sim \mathcal{C}\mathcal{N}(0,1)$ represents small-scale fading. Let${\mathcal{S}}_k$For the $The\; serving\; AP\; set\; of\; k$ UEs,${\mathcal{C}}_m$MM _The set of UEsserved by$M$ APs . We believe that all UEs are guaranteed to be no more than$L(L<M)$ APs (i.e.$|S_k|\le L,\forall k$ ), but not necessarily all APs are serving UE. Therefore, the set of all serving (active) APs and all UEs is$\mathcal{S}={\bigcup }_{k=1}^K{S}_k$ 和 $\mathcal{C}={\bigcup }_{m=1}^M{C}_m$. Let $q_k$The APs serving the UE are sent to the $Symbols\; of\; k$ UEs, where$\mathbb{E}\left[{\left|q_k\right|}^2\right]=1$ 且 $E\left[q_k\right]=0,\forall k$，$E\left[q_k{q}_l^{\ast }\right]=0,\forall k\ne l$ (i.e. symbols used for different UEs are uncorrelated). ThenmmThe transmitted signal of$m$ APs using conjugate beamforming[

$$x_m=\sum _{k\in {\mathcal{C}}_m}\sqrt{r_{mk}}{\hat{g}}_{mk}^{\ast }{q}_k\text{\hspace{1em}(1)}$$

Among them, ${\hat{g}}_{mk}$is the channel $g_{mk}$estimated value.

$$\begin{array}{rl}{r}_k& =\sum _{m\in \mathcal{S}}{g}_{mk}{x}_m+w_k\\ =& \sum _{m\in S_k}{g}_{mk}{x}_m+\sum _{m\in S_k^c}{g}_{mk}{x}_m+w_k\\ =& \sum _{m\in S_k}\sum _{k^{\prime }\in {\mathcal{C}}_m}\sqrt{r_{m{k}^{\prime }}}{g}_{mk}{g}_{m{k}^{\prime }}^{\ast }{q}_{k^{\prime }}+\sum _{m\in S_k^c}{g}_{mk}{x}_m+w_k\\ =& \underset{\text{useful signal }}{\underbrace{\sum _{m\in S_k}\sqrt{r_{mk}}{\left|g_{mk}\right|}^2{q}_k}}\\ & +\underset{\text{interference plus noise }}{\underbrace{\sum _{m\in S}\sum _{k^{\prime }\in C_m,k^{\prime }\ne k}\sqrt{r_{m{k}^{\prime }}}{g}_{mk}{g}_{m{k}^{\prime }}^{\ast }{q}_{k^{\prime }}+w_k}}\end{array}\text{\hspace{1em}(2)}$$

$$R_k={\log }_2\left(1+\frac{{{\left|{\sum }_{m\in {\mathcal{S}}_k}\sqrt{r_{mk}}\right|g_{mk}|}^2|}^2}{{\sum }_{k^{\prime }\in \mathcal{C},k^{\prime }\ne k}{\left|{\sum }_{m\in \mathcal{S}}\sqrt{r_{m{k}^{\prime }}}{g}_{mk}{g}_{m{k}^{\prime }}^{\ast }\right|}^2+p_w^2}\right).\text{\hspace{1em}(3)}$$

There is a big problem with this formula (3).

The maximum achievable sum rate of the network is:

$$R_{\mathrm{sum}}=\sum _{k\in \mathcal{C}}{R}_k.\text{\hspace{1em}(4)}$$

B. Caching Model

C. Power Consumption Model

The total network power consumption consists of three parts [14]:

• i) Transmit power of all serving APs, including the power required to provide content in the AP cache directly to UEs,
• ii) the power required by APs to retrieve missing content files from the CPU, and
• iii) The power required by the CPU to download the missing content from the backbone .

For i) , the sum of the transmit power of all service ap is given by ${\sum }_{m\in \mathcal{S}}{P}_m$given.

For ii) , we believe that the power required by the AP to retrieve missing content files is proportional to the number of different missing files. When multiple terminals associated with an AP request the same content file, the AP only needs to retrieve the missing file from the CPU once. Let F mmiss = ( ⋃ k ∈ C m { fk } ) ∩ F mc \mathcal{F}_{m}^{\mathrm{miss}}=\left(\bigcup_{k \in \mathcal{C}_ {m}}\left\{f_{k}\right\}\right) \cap \mathcal{F}_{m}^{c}Fmmiss​=(⋃k∈Cm​​{ fk​})∩Fmc​is $m$ APs are needed but are not in the$A\; set\; of\; content\; files\; cached\; by\; m$ APs and jointly requested by terminals,$P_{\text{backhaul }}$ 为每个 AP 向CPU请求一个缺失内容文件所需的功率，其中包括 backhaul 功率（在AP - CPU链路中获取内容所需的功率）和电路功率。那么，所有从CPU检索内容的总功耗由 $P_{\text{backhaul }}\cdot {\sum }_{m\in \mathcal{S}}\left|{\mathcal{F}}_m^{\text{miss }}\right|$ 给出。

对于 iii)，我们认为，如果与不同 APs 关联的两个终端请求相同的内容，而这些内容没有在各自的服务 APs上缓存，CPU 只需要下载此内容文件一次。设 $P_{\text{backbone}}$ 为CPU从主干下载一个内容文件所需的功率。则 CPU 从骨干网下载内容的功耗为 $P_{\text{backbone }}\cdot \left|{\bigcup }_{m\in \mathcal{S}}{\mathcal{F}}_m^{\text{miss }}\right|$。

Note that this consideration can be implemented practically by caching specific content on the CPU, which is equipped with a cache whose storage is large enough to hold the missing content, but not large enough to hold everything.

Summarizing i) -iii) , the total power consumption of the network

$$\begin{array}{rl}{P}_{\text{total }}=& \sum _{m\in \mathcal{S}}{P}_m+P_{\text{backhaul }}\cdot \sum _{m\in S}\left|F_m^{\text{miss }}\right|\\ & +P_{\text{backbone }}\cdot \left|\bigcup _{m\in S}{F}_m^{\text{miss }}\right|\end{array}\text{\hspace{1em}(5)}$$

---

D. Problem Formulation

我们的设计目标是共同设计合作聚类，${\mathcal{S}}_1,{\mathcal{S}}_2,\dots ,{\mathcal{S}}_K$，还有内容缓存，${\mathcal{F}}_1,{\mathcal{F}}_2,\dots ,{\mathcal{F}}_M$，使得 $R_{\text{sum }}/P_{\text{total }}$ 定义的网络能效（energy efficiency，EE）最大化。从数学上讲，设计问题是

$$\begin{array}{rl}\underset{\begin{array}{c}{\mathcal{S}}_1,{\mathcal{S}}_2,\dots ,{\mathcal{S}}_K\\ {\mathcal{F}}_1,{\mathcal{F}}_2,\dots ,{\mathcal{F}}_M\end{array}}{\max }& \frac{R_{\text{sum }}}{P_{\text{total }}}\\ \text{ s.t. }& \left|{\mathcal{S}}_k\right|\le L,\forall k\in \mathcal{C},\\ & |{\mathcal{F}}_m|\le N,\forall m\in \mathcal{S}.\end{array}$$

Note that there are design tradeoffs in solving problem (6) . Completely based on ${\mathcal{S}}_1,S_2,\dots ,S_K$The design may tend to use the $k$ UEsthe subsetAPsthebest channel conditionsS k \mathcal{S}_k$S_k$associated as this may increase $R_k$, thereby increasing $R_{\text{sum}}$。

In contrast, a design based entirely on ${\mathcal{F}}_1,F_2,\dots ,F_M$There may be a tendency to regard the $k$ UEsa subset ofAPsthat best aligns content cache status and user requestsS k \mathcal{S}_k${\mathcal{S}}_k$associated, as this may increase hit events and thus reduce $P_{\text{total }}$。

This motivates joint design. Furthermore, the problem becomes intractable for large networks with a large number of APs and terminals. Therefore, we developed a cooperative clustering and content caching strategy based on reinforcement learning ( RL ), which is discussed in the next section.

---

III. REINFORCEMENT LEARNING METHOD

In this section, we elaborate on how to solve the joint problem of clustering and caching through the DDPG algorithm . Defines the three basic elements ( action, state and reward ) in the RL problem under consideration. Since the dynamic evolutions of clustering and caching strategies are particularly important when formulating RL problems, relevant parameters will be indexed by time in the following.

A. Action, Observation State, and Reward Function

在 $t\text{th}$ time slot, agent’s action$a_t$Both clustering and caching are involved. Let the index amk , t ∈ { 0 , 1 } a_{mk,t}∈\{0,1\}amk,t​∈{ 0,1 }和$a_{mf,}\in \{0,1\}$ respectively represents the$m$ APs—kth_$k$ UEstatusandmmth$m$ APs—th$f$ file cache status, where "1" means successful association or caching, and "0" means the opposite.

Then, the action of the agent $a_t$ 可以定义为 $a_t=\left\{a_t^{cl},a_t^{ca}\right\}$，其中集合 $a_t^{cl}=\left\{a_{mk,t}^{cl}:m\in \mathcal{M},k\in \mathcal{K}\right\}$ 和 $a_t^{ca}=\left\{a_{mf,t\_}^{ca}:m\in \mathcal{M},f\in F\right\}$ respectively contains thettthMetrics for the overall results of clustering and caching for$t\; time\; slots.$Note that action$a_t$Uniquely determines the set ${\mathcal{S}}_k$、${\mathcal{C}}_m$and ${\mathcal{F}}_m$，即 $S_k=\left\{m:a_{mk,t}^{cl}=1,m\in \mathcal{M}\right\}$， ${\mathcal{C}}_m=\left\{k:a_{mk,t}^{cl}=1,k\in \mathcal{K}\right\}$，${\mathcal{F}}_m=\left\{f:a_{mf,t\_}^{ca}=1,f\in \mathcal{F}\right\}$。

The state in RL considered should be the collectable information set of the CPU that can be used to calculate the reward function. In this work, the $t$ 个时隙的观察状态（observation state）定义为信道增益 G t = { g m k , t : m ∈ M , k ∈ K } G_t=\left\{g_{m k, t}: m \in \mathcal{M}, k \in \mathcal{K}\right\} Gt​={ gmk,t​:m∈M,k∈K}，前一个时隙的聚类和缓存动作，以及每个终端的文件请求历史的集合。用户请求统计 $e_t$ 定义为 e t = { e k f , t : k ∈ K , f ∈ F } e_{t}=\left\{e_{k f, t}: k \in\right. \mathcal{K}, f \in \mathcal{F}\} et​={ ekf,t​:k∈K,f∈F}，其中 $e_{kf,t}={\sum }_{t^{\prime }=1}^t{1}_{f_{k,t^{\prime }}=f}$For the $k$ terminals to timettffat time$t$$The\; number\; of\; downloads\; of\; f$ files. The observation state is expressed as

s t ≜ { G t , e t , a t − 1 } (7) s_{t} \triangleq\left\{G_{t}, e_{t}, a_{t-1}\right\}\tag{7} st​≜{ Gt​,et​,at−1​}(7)

Following the problem objective in (6a), at $The\; reward\; function\; for\; t$ time slots is defined as

$$r\left(s_t,a_t\right)\triangleq \frac{R_{\mathrm{sum},t}}{P_{\mathrm{total},t}}$$

其中，$R_{\mathrm{sum},t}$ 和 $P_{\mathrm{total},t}$are given by equations (4) and (5) respectively, with the appended subscript $t$ emphasizes dynamic behaviors. Note that the total rate$R_{sum,t}$Depends on channel conditions $G_t$and clustering results $a_t^{cl}$, and the total power $P_{total,t}$Depends on cached result $a_t^{ca}$。

B. Deep Deterministic Policy Gradient Approach

Since the potentially large number of UEs and APs in the CF-mMIMO network requires considerable action spaces , the traditional deep Q network ( DQN ) requires a long training time to learn joint clustering and caching strategies, so it cannot Effectively capture system dynamics . In this work, the deep deterministic policy gradient ( DDPG ) algorithm is adopted to achieve fast and stable learning [15]. Essentially, the DDPG network follows the actor -critic approach by combining an additional target -network with the original evaluation -network to improve convergence speed and stability.

The purpose of the player part is to use a deterministic strategy $m\left(s_t\mid i^{\mu }\right)$produces an action in each time slot, and the policy consists of a weight$i^{}$Deep neural network (DNN) learning of${}^{\mu \; .}$Weight$i^{\mu }$ update is to find the best deterministic strategy$m\left(s_t\mid i^{\mu }\right)$is based on the action-value function, that is, the expected long-term reward is defined as (expected long-term reward):

$$Q\left(s_t,a_t\right)\triangleq \mathbb{E}\left[R_t|s_t,a_t\right]\text{\hspace{1em}(9)}$$

where $R_t$is the cumulative discounted future reward (cumulative discounted future reward ) $R_t\triangleq {\sum }_{i=t}^{\infty }{c}^{i-t}r\left(s_i,a_i\right)$，$c\in [0,1]$ is the discount factor. Normally,$i^{\mu }$ is updated by the gradient ascent method

$$i^m\leftarrow i^m+{a^{\mu }{\nabla }_{}J(\theta )|}_{\theta ={\theta }^{}}\text{\hspace{1em}(10)}$$

where $a^{\mu }$ is the learning rate and

$$J(\theta )\triangleq {\mathbb{E}}_{s_t}\left[Q\left(s_t,m\left(s_t\mid i\right)\right)\right]\text{\hspace{1em}(11)}$$

is the objective with the expectation taken with respect to $s_t$. Note that the action-value function in (9) allows recursive relations

$$Q\left(s_t,a_t\right)=r\left(s_t,a_t\right)+cE\left[Q\left(s_{t+1},a_{t+1}\right)\right]\text{\hspace{1em}(12)}$$

In the critic part, the above formula needs to be determined $Q\left(s_t,m\left(s_t|i^{\mu }\right)\right)$。更具体地说，评论家评估动作值函数 $Q\left(s_t,\mu \left(s_t\mid {\theta }^{\mu }\right)\mid {\theta }^Q\right)$ 使用权重为 ${\theta }^Q$ 的单独DNN。通常，权重 ${\theta }^Q$ 使用更新

$${\theta }^Q\leftarrow {\theta }^Q-{{\alpha }^Q{\nabla }_{\theta }L(\theta )|}_{\theta ={\theta }^Q}\text{\hspace{1em}(13)}$$

其中 ${\alpha }^Q$ 是学习率并且

$$L\left(i\right)=\mathbb{E}\left[{\left(Q\left(s_t,a_t\mid i\right)-y_t\right)}^2\right]\text{\hspace{1em}(14)}$$

is the mean square value of the mean square Bellman error function, where the target is

$$y_t=r\left(s_t,a_t\right)+\gamma Q^{\prime }\left(s_{t+1},m^{\prime }\left(s_{t+1}|i^{m^{\prime }}\right)|i^{Q^{\prime }}\right)\text{\hspace{1em}(15)}$$

Adapted from the recurrence relation in (12). Superscript $\prime$ 用来表示(15)中产生 $y_t$ 的动作-价值函数是由单独的目标网络（target-network）提供的。相比之下，(14)中的 $Q\left(s_t,a_t\mid \theta \right)$ 则由原始的评估网络（original evaluation-network）提供。评估网络和目标网络的结合使DDPG算法趋于稳定[15]。

请注意，实际上，(11)和(14)中的期望可以用样本均值近似（approximated by sample mean），因为很难知道 $s_t$ 的精确概率分布，其中随机样本是从存储元组 $b_i\triangleq \left(s_i,a_i,r_i,s_{i+1}\right)$ 的 replay buffer D \mathcal{D} Extracted from$D$$i=t-D+1,\dots ,t$。replay buffer$D$ has limited buffer size$D$ , where DDis cached$D$ recent actions (actions) and corresponding status and rewards.

Finally, soft updates are performed to further stabilize the target critic network $i^{Q^{\prime }}\leftarrow t{i}^{Q^{\prime }}+(1-t)i^Q$ also quantifies the function(target actor network)$i^{m^{\prime }}\leftarrow t{i}^{m^{\prime }}+(1-t)i^{\mu }$ , where$t\ll 1$ . The complete DDPG process is summarized in Algorithm 1.

---

IV. SIMULATION RESULTS

A. Simulation Settings

We consider two scenarios as shown in Table 1, where UEs and APs are evenly distributed over $1km{}^2$ range, one of the APs is anchored at the reference coordinates$(0,0)$ . During the evaluation phase,UEsandAPsare fixed. A content preference vectoris randomly generatedfor eachUE, and the user request is determined bythe Zipffactorβ = 1 β = 1$b=1$ Content preference vector generation [9], [10], [16]. During the evaluation phase, user requests remain fixed. The path loss exponent is$a=2$ , small-scale fading coefficient$h_{mk}\sim \mathcal{C}\mathcal{N}(0,1)$ obey the time-varying model [17].

$$h_{mk}(t+1)=\sqrt{1-{ϵ}^2}\times h_{mk}(t)+ϵ\times n_{mk}\left(t\right)\text{\hspace{1em}(16)}$$

Where $n_{mk}\left(t\right)\sim CN(0,1)$，$ϵ\in [0,1]$ is the channel variation coefficient, which is set to ϵ = 0.01 \epsilon=0.01in all simulations$ϵ=0.01$。

We set the transmit power of all APs to $P_m=10\mathrm{mW}$, $P_{\text{backhaul }}=P_{\text{backbone }}=500mW$ [8]. The thermal noise power at each terminal is derived from [1]

$$p_w^2=\text{ Bandwidth }\times K_B\times T_0\times \text{ noise figure (W) }\text{\hspace{1em}(17)}$$

where $\text{ Bandwidth }$ is set to$20\text{ MHz}$，$K_B=1.381\times 10^{-23}$，$T_0=300\text{ K}$，噪声系数 $9\text{ dB}$，这导致 ${\sigma }_w^2=7.457\times 10^{-13}\mathrm{W}$。actor 和 critic 网络都使用了两个隐藏层的DNNs，分别有400和300个神经元，以及tanh激活函数。

我们将所提出的RL方法与三种固定策略基准（fixed strategy benchmarks），称为BM1、BM2和BM3，以及仅适用于小场景(即场景1)的最优蛮力(BF)算法进行比较:

• 1. BM1：基于信噪比的聚类策略，SNR-based clustering policy（第 $k$ 个 UE 连接到所有AP中 ${\left|g_{mk}\right|}^2$ 最高的 $l\le L$ 个 APs）和基于本地流行度的缓存策略，local-popularity-based caching policy（连接到第 $m$ 个AP的特定 UEs 中最流行的 $N$ 个文件缓存在第 $m$ 个AP上）。
• 2. BM2：基于信噪比的集群策略(与BM1相同)和基于全局流行度的缓存策略（所有终端中最流行的 $N$ 文件在所有AP上缓存）。
• 3. BM3：基于缓存的集群策略（一个终端连接到 $l\le L$ 个AP，这些 APs 的前一个时隙的缓存最符合终端的文件请求）和基于全局流行度的缓存策略(与BM2相同)。
• 4. BF：对所有集群/缓存组合进行穷举搜索，并找到产生最高EE的组合。

B. Results and Discussion

图2显示了场景1的结果。可以看出，BF、提出的RL和BM1（$l=1$ ) Exhibits comparable EE performance, significantly higher than all other schemes. This difference is due to nearly two orders of magnitude difference in power consumption due to varying degrees of missing files and different values ​​of the second and/or third terms of (5) in different schemes (Figure 2(c) )).

The proposed RL outperforms all benchmarks, and unlike these benchmarks, it does not require user preference information .

BM1 outperforms BM2 in terms of EE because although they use the same clustering strategy to achieve the same summing rate, BM1's local popularity-based caching strategy results in fewer file losses and thus lower power consumption. More specifically, in BM1, different APs can be in the AP's own service network ${\mathcal{C}}_m$Different files are cached based on their popularity in BM2, while in BM2, all APs cache the same files based on their popularity in the entire network.

BM2 is better than BM3 in terms of EE because in this case, BM2 is superior in achieving higher sum rate due to signal-to-noise ratio based clustering than BM3 is in achieving lower network efficiency due to cache-based clustering. consumption advantages. Fluctuations in the BM3 summation rate are due to its random correlation in selecting the AP that best matches the terminal file request $l$ AP mechanism. Compare BM1 ($l=1$ ) and BM1 ($l=2$ ), the EE performance of BM1 (l = 2) will degrade because it becomes increasingly difficult for anAPto match its cached files with file requests from all relevant endpoints, and more files are lost and Higher power consumption dominates performance.

Figure 2 Snapshots of BF, RL and BM1 (l = 1) at t = 5 are shown in Figure 3. BF finds the optimal strategy of ee. Similar strategies were found for RL and BM1, where RL prescribed additional UE2-AP4 associations, which resulted in slight increases in sum rate and EE. RL may converge to a local optimum, or the optimal action may not be explored during the exploration process in the training phase, resulting in the observed gap between RL and BF.

---

V. CONCLUSION

In this paper, we propose a DDPG-based approach to solve the joint cooperative clustering and content caching problem in cache-enabled CF-mMIMO networks. The DDPG framework follows the actor-critic network architecture to enhance convergence speed and stability. The actor produces an action corresponding to the clustering and caching policy at each time slot, and the critic evaluates the expected cumulative reward corresponding to the action. The proposed method demonstrates verifiable, near-optimal performance in small networks and outperforms fixed-policy benchmarks in actual, potentially large networks.