1. Intrinsic Reward for Circuit Optimization

In our evaluation, conventional methods that rely solely on extrinsic rewards showed low sample efficiency and tended to converge to suboptimal local minima, resulting in low-quality circuit designs. To address this, we combine extrinsic rewards with an intrinsic reward that encourages the agent to explore less-visited regions of the circuit design space rather than familiar regions. Intrinsic reward mechanisms can be categorized into count-based and prediction-based methods. Since analog circuits have a large and continuous state space, count-based methods for estimating state visitation counts are impractical. Therefore, we propose employing prediction-based methods that enable more effective exploration in analog circuits. The prediction-based intrinsic reward is formulated as ${r}^{\mathrm{i}}_{t} = \|{\mathbf{s}}_{t+1}-{f}(\mathbf{s}_{t})\|_{2}$, where $f$ represents the prediction network, which is used to predict the next circuit state ${\mathbf{s}}_{t+1}$.

Algorithm 1 summarizes the proposed circuit optimization framework using prediction-based intrinsic reward. The policy network $\pi_\theta$, the prediction network $f$, and the initial circuit parameters ${\mathbf{s}}_0$ are first initialized. For each episode, the agent interacts with the environment and collects the trajectory (lines 1-3). Given the current circuit state ${\mathbf{s}}_{t}$, the agent samples an action ${\mathbf{a}}_{t} \sim \pi_\theta ({{\mathbf{a}}_{t}\mid \mathbf{s}}_{t})$ (line 4). The next circuit state ${\mathbf{s}}_{t+1} = {\mathbf{s}}_{t} + {\mathbf{a}}_{t}$ is obtained by applying the action to the circuit state (line 5). Here, ${\mathbf{a}}_{t}$ corresponds to a vector ${\delta \mathbf{s}}_{t}$, which is added to ${\mathbf{s}}_{t}$ to generate ${\mathbf{s}}_{t+1}$. Then, a SPICE simulation is conducted using ${\mathbf{s}}_{t+1}$, and an extrinsic reward ${r}^{\mathrm{e}}_{t}$ is computed based on FOM (lines 6-7). The prediction network f predicts the next circuit state by ${\widehat{\mathbf{s}}}_{t+1} \leftarrow f ({\mathbf{s}}_{t})$, and the intrinsic reward ${r}^{\mathrm{i}}_{t}$ is computed using the $L_{2}$ norm of the prediction error (lines 8-9). Therefore, the total reward ${r}_{t}$ is computed as the summation of ${r}^{\mathrm{e}}_{t}$ and ${r}^{\mathrm{i}}_{t}$ (line 10). Finally, $\pi_\theta$ and $f$ are jointly updated using the collected trajectory in the buffer ${D}$ (line 14).

By assigning larger intrinsic rewards to novel regions of circuit design spaces, we hypothesize that this approach improves sample efficiency and helps the agent discover high-performance designs that might otherwise be overlooked. However, advances in technology and complex device characteristics increase the sensitivity of analog circuit behavior, making accurate modeling and optimization more challenging. Thus, RL agents using predictionbased intrinsic reward can be trapped in design parameters that are highly sensitive to small variations, and minor changes can lead to abrupt performance shifts.

2. Enhancing RL Agent with ReconstructionBased Intrinsic Reward

To overcome the limitation of prediction-based intrinsic reward, we propose a {reconstruction-based intrinsic reward framework}, as shown in Fig. 1. The circuit reconstruction network is implemented using a CVAE, which enables structured encoding of circuit states conditioned on actions. Compared with the next circuit state prediction in Algorithm 1, the proposed approach evaluates how well a circuit state can be reconstructed from past experiences with the intrinsic reward defined as ${r}^{\mathrm{i}}_{t}=\|\mathbf{s}_{t+1}-g(f(\mathbf{s}_{t+1}, \mathbf{a}_{t+1}))_2$. Here, $f$ and $g$ denote the encoder and decoder of the proposed circuit reconstruction network, respectively. Thus, ${g}(f(\mathbf{s}_{t+1}, \mathbf{a}_{t+1}))$ represents the reconstructed circuit state ${\hat{\mathbf{s}}}_{t+1}$, and a higher reconstruction error indicates a novel circuit state. The overall procedure of the circuit optimization follows Algorithm 1, except for how intrinsic reward is defined. The policy network $\pi_\theta$, the encoder network ${f }$, the decoder network ${g}$, and the initial circuit parameters ${\mathbf{s}}_0$ are first initialized. A latent representation $\mathbf{z}_{t+1}$ is obtained from $f(\mathbf{s}_{t+1}, \mathbf{a}_{t+1})$ in Fig. 1. Then, $g$ reconstructs the next circuit state ${\widehat{\mathbf{s}}}_{t+1}$ using $\mathbf{z}_{t+1}$ and $\mathbf{a}_{t+1}$. Thus, the intrinsic reward ${r}^{\mathrm i}$ and the total reward $r_{t}$ are computed in the same manner. Finally, $\pi_\theta$, $f$, and $g$ are updated jointly using the trajectory in buffer $D$.

The underlying principle that exploration can be effectively guided by the state estimation error is similar to prediction-based methods. However, instead of relying on prediction errors, the proposed method utilizes reconstruction errors, allowing the RL agent to efficiently explore diverse designs while avoiding misleading configurations. Experimental results demonstrate that the proposed approach significantly improves sample efficiency and enables the discovery of optimal circuit designs.

Fig. 1. Circuit optimization method using intrinsic reward.

1. Environment and Evaluation

We adopted proximal policy optimization (PPO) ^[18] as the base RL algorithm and Adam ^[19] as the optimizer. To evaluate the effectiveness of RL-based optimization, we evaluated a random policy that selects actions uniformly at random without any learning mechanism. For comparison, we used BO ^[20] and strong intrinsic reward mechanisms, including the intrinsic curiosity module (ICM) ^[13], which computes intrinsic rewards based on forward dynamics prediction errors, and random network distillation (RND) ^[14], which measures the prediction errors of a randomly initialized network as intrinsic rewards. Additionally, we adopted traditional exploration methods such as greedy and $\epsilon$-greedy algorithms. We also evaluated NovelD ^[21], which is a state-of-the-art countbased method. However, it failed to learn the policy due to the large and continuous state space of analog circuits, thereby being excluded from the results. All models used the same base RL algorithm and neural network architecture for both the policy and value functions. The only difference among them was how intrinsic rewards were defined. Besides, PPO has approximately 0.210 M parameters, while the CVAE in the proposed method has about 0.035 M parameters, accounting for only 14\% of the total model size. To demonstrate the effectiveness of the proposed method, we employed two-stage and three-stage OpAmps with the same structure as those used in ^[22]. Besides, a folded cascode OpAmp was included to evaluate the generalization capability across different circuit topologies. The structure of the folded cascode OpAmp follows the design used in ^[4]. The circuit simulation was performed using Ngspice, while a commercial 250 nm CMOS technology was used for circuit design. Ngspice is linked with the ngspyce Python interface ^[23], which can be used to alter parameters and gather simulation results. Besides, node voltage, resistance, capacitance, and width of the transistor were used as design parameters. We measured the DC gain, 3 dB bandwidth, and power consumption during training to compute the FOM. The target specification values for each circuit are summarized in Table 1.

2. Experimental Results and Analysis

Figs. 2 and 3 show the performance of the proposed method compared with different methods for the twostage and three-stage OpAmps, respectively(Note that the greedy and $\epsilon $-greedy algorithms were not visualized due to high variance affecting visibility). Compared to PPO without adopting intrinsic rewards, the power consumption, gain-bandwidth product (GBW), and FOM were significantly improved when agents used an intrinsic reward during optimization. Notably, compared to other intrinsic reward methods such as ICM and RND, the proposed method demonstrates more stable convergence and higher performance. Conventional approaches, which rely on prediction errors to design intrinsic rewards, are wellsuited for game environments where state transitions are relatively smooth and predictable. However, they are less effective for the circuit optimization, where high sensitivity and nonlinear state variations make prediction-based rewards unreliable, often leading to unstable exploration. In contrast, the proposed method leverages reconstruction errors to evaluate structural differences in circuit states, enabling more reliable exploration and faster convergence while maintaining high sample efficiency.

On the other hand, Tables 2 and 3 show the average of the FOM, which represents a combined measure of three performance metrics, providing a comprehensive evaluation of optimization effectiveness. To ensure a fair evaluation, all methods were compared using an equal number of samples collected within the same runtime. Terms ${S}(\alpha_{1})$ and ${S}(\alpha_2)$ denote the number of samples required to reach the target FOM values $\alpha_{1}$ and $\alpha_{2}$, respectively. To compare sample efficiency, we set $\alpha_{1}= -3$ and $\alpha_{2} = -1$ for the two-stage OpAmp. Besides, given the increased optimization difficulty, we set $\alpha_{1} = -10$ and $\alpha_{2} = -5$ for the three-stage and folded-cascode OpAmps. The results show that the proposed method consistently outperforms all baseline approaches in both optimization performance and sample efficiency. The lower values of ${S}(\alpha_{1})$ and ${S}(\alpha_{2})$ demonstrate that the proposed method reaches the desired performance levels with significantly fewer SPICE simulations compared to baselines. Notably, while ICM and RND show minimal improvements over standard PPO and BO, they fail to learn the optimal policy effectively due to the high-dimensional and continuous space of the analog circuit. In contrast, the proposed method allows the agent to explore the design space more efficiently, leading to more stable convergence and better optimization ability.

Fig. 2. Performance comparison on a two-stage OpAmp in terms of power consumption, GBW, and FOM.

Fig. 3. Performance comparison on a three-stage OpAmp in terms of power consumption, GBW, and FOM.