3.1 Model Architecture

This paper proposes the EFSOD to address the shortcomings of existing small object detection algorithms based on super-resolution reconstruction. The EESRM and the CMFFM improve image quality and enhance target edge features, increasing detection accuracy. EFSOD shares information at the feature level, allowing super-resolution reconstruction results to optimize the detection task more directly. The overall structure of the EFSOD network is shown in Figure 1.

In EFSOD, the generator mainly consists of the EESRM, which generates high-quality super-resolved images $I^{SR}$ from low-resolution remote sensing images $I^{LR}$. To better capture and preserve edge information in the images, this paper proposes the Edge Enhanced Dense Residual Module (EEDRM), which extracts edge information from the feature maps of each layer within the residual blocks and connects these edge details through residual connections.

In the generator's super-resolution reconstruction network, the low-resolution remote sensing image $I^{LR}$ is first processed through an initial convolutional layer ($Conv$) to extract the basic features $F_0$. Then, the extracted features are applied to three EEDRM to learn deeper features within the feature maps, using residual connections to facilitate the flow of information and obtaining the feature map $F_d$ after deep feature learning. Subsequently, the feature map is further processed through a convolutional layer ($Conv$) to extract the fused deep features $F_{\text{deep}}$. Finally, the learned feature map is upsampled using pixel shuffle technology to increase the image to the target resolution. Additional convolutional layers further enhance and refine the features, and a final convolutional layer converts the feature map into a high-resolution image $I^{SR}$. The $I^{SR}$ undergoes feature extraction using the convolutional neural network ResNet50 [30] to obtain the feature map.

The object detection network is responsible for detecting and identifying objects in the generated images. We propose the CMFFM module to extend the object detection component. Firstly, to ensure dimensional consistency between the feature map $F_{sr}\in {ℝ}^{H_s\times W_s\times C_s}$ extracted from the super-resolution reconstruction network and the feature map $F_{ob}\in {ℝ}^{H_d\times W_d\times C_d}$ in the object detection network, we use a Feature Alignment function $FA(\cdot )$ to adjust $F_{sr}$ to match the size of $F_{ob}$ as a feature map:

Then, ${\tilde{F}}_{sr}$ and $F_{ob}$ are concatenated at the channel level, denoted as:

where $H_d$ , $W_d$ and $C_s$+$C_d$ respectively denote the height, width, and channels of the feature maps. ${\tilde{F}}_{sr}$ is the super-resolution feature map after feature alignment. $F_{cat}$ represents the feature map obtained by channel-wise concatenation and fusion of the aligned super-resolution feature map ${\tilde{F}}_{sr}$ with the object detection feature map Fob.

This step effectively combines the features of the two networks, where the super-resolution feature map adds more detailed information and the object detection feature map incorporates richer high-level semantic information. The number of channels in the concatenated feature map will increase, and to meet the input requirements of the subsequent layers of the object detection network, it is necessary to adjust the number of channels in the feature map using a convolution operation $Conv(\cdot )$, convert $F_{cat}$ into a feature map with $C^{{\prime }^{}}$ target channels.

where $C^{{\prime }^{}}$ respectively denotes the adjusted target number of channels to match the input requirements of the object detection network.

This convolutional layer adjusts the number of channels and helps further integrate features from two different networks. It is then followed by a batch normalization layer (BN) to standardize the features, adjusting and scaling the incoming features to promote rapid convergence of the model.

where $F_{bn}$ respectively denotes the feature map after batch normalization, $BN(\cdot )$ denotes the batch normalization operation, standardizing the input features so that their mean is close to 0 and variance is close to 1.

Subsequently, the ReLU activation function is applied to maintain non-linear characteristics.

where $F_{relu}$ is the feature map after RELU activation, the ReLU activation function is applied to enhance the features' nonlinear expressive capability.

Finally, the fused features output by the CMFFM module are:

where $F_{fusion}$ denotes the fused feature map output by the CMFFM.

This feature effectively integrates the detailed edge and texture information provided by the super-resolution network with the high-level semantic information from the object detection network, thereby enhancing the detection accuracy of small targets in remote sensing images.

3.2 Edge Enhancement Super-Resolution Reconstruction Module

This paper proposes the EESRM to better capture and preserve edge information in images. The EESRM primarily comprises the Edge-Enhanced Dense Residual Module (EEDRM), whose network structure is shown in Figure 2. Based on the dense residual structure, EEDRM incorporates an Edge Enhance Block (EEB) consisting of a convolution layer, an activation layer, and a Laplacian operator.

Figure 2 Diagram of the EEDRM structure.

The EEB extracts high-frequency components from the feature layers extracted by the residuals using a Laplacian operator [31] and connects them with the original feature layers through residual connections. This process can be expressed as:

where $F^i$ represents the feature map extracted by the ith dense residual module, $F_{le}^i$ represents the edge information extracted by the edge enhancement module of $F^i$, $\alpha$ denotes the adjustment factor, and $F_e^i$ represents the edge-enhanced feature layer output through residual connection.

Figure 3 Structure of the Laplace Block.

Motivated by the Laplacian operator, The structure is illustrated in Figure 3, the original image undergoes low-pass filtering and downsampling, followed by upsampling to restore the dimensions. Finally, the original image and the upsampled result are subtracted, a process that can be expressed as:

where $PyrUp$ is the upsampling, $PyrDown$ is the downsampling.we employ the Dense Residual Laplacian Block (DLB) as the basic unit to extract features from the original image. This approach preserves more high-frequency edge information without incurring significant computational overhead, thereby improving the accuracy of small object detection. The $i$th DLB, the feature map $F^i$ is obtained through dense residual feature extraction. Due to repeated convolution operations, the edge information of small objects becomes weakened, resulting in ghosting artifacts in the generated images. Therefore, we embed an additional computation for edge information extraction within each dense residual block to enhance the high-frequency details of the image better, thereby achieving a more precise and refined super-resolution reconstruction.

For the feature extraction network of the super-resolution model, a 3×3 convolutional layer is first used to extract the shallow feature map $F^S$ from the image. This operation reduces the spatial dimensions of the feature maps, increases the receptive field of the network layers—thus enabling the capture of a broader range of features—and simultaneously decreases the volume of data subsequent layers need to process. This process can be expressed as:

where $Con{v}_{3\times 3}$ denotes the operation of a 3×3 convolutional layer.$I_{LR}$ x denotes the low-resolution image.

Next, the shallow feature map $F^S$ is passed through a series of EEDRM to extract deep features containing edge information. In this process, the number of EEDRM blocks is set to three. This module employs dense residual connections through a multi-scale residual network and an edge feature extraction module to extract rich semantic information, thereby enhancing the feature representation capability of the target feature extraction module.Simultaneously, the output of each convolutional layer is connected via residual connections with all preceding convolutional layers to prevent the loss of feature information. Furthermore, the EEDRM employs the Laplacian operator to extract edge information from the image, and this extracted edge information is then integrated through residual connections with the previously extracted features, significantly enhancing the utilization of edge features. After the series of EEDRM blocks, an additional convolutional layer is applied, and its output is added to the shallow feature map to obtain the final deep feature $F^d$. This process can be expressed as:

Then, the extracted deep features undergo a feature upsampling operation to increase their resolution to the desired size. A convolutional layer extracts features while preserving spatial resolution, followed by an activation function to introduce non-linearity for learning complex features.

Finally, a second convolution and activation operation is performed to refine the target features further, enhancing detail restoration and deep feature extraction. The ultimately extracted target features $F^{SR}$ can be expressed as:

3.3 Cross-Model Feature Fusion Module

To integrate super-resolution features with target detection features and obtain superior detection performance, this paper proposes the CMFFM. The core objective of this module is to effectively fuse the target features extracted by the super-resolution reconstruction network into the feature maps extracted by the target detection network. By leveraging the detail enhancement capability of the super-resolution network and the recognition and localization expertise of the target detection network, the proposed method jointly improves detection performance. This approach introduces the super-resolution network's sensitivity to fine details without compromising the target detection network's spatial perception capability, thereby achieving effective feature fusion between the two networks and enhancing overall target detection performance.

The network structure of CMFFM is shown in Figure 4. $C_{sr}$ represents the number of channels in the super-resolution feature map, and $C_{ob}$ represents the number of channels in the target detection feature map. H and W denote the height and width of the feature maps, respectively. $Conv-1$ refers to a 1×1 convolutional layer, BN represents Batch Normalization, and Relu denotes the nonlinear activation function.

Figure 4 Cross-model feature fusion module.

During the cross-model feature fusion process, the given inputs include the target features $F_{sr}ϵF^{C_{sr}\times H\times W}$ extracted from the super-resolution reconstruction network and the features $F_{ob}ϵF^{C_{ob}\times H\times W}$ extracted from the backbone of the target detection network.First, it is necessary to ensure that the target features generated by the super-resolution model match the spatial dimensions of the final output layer of the target detection backbone network. To achieve this, the size of the additional features is adjusted using bilinear interpolation to align with the dimensions of the backbone features. The proposed feature fusion network employs a series of feature fusion operations using concatenation $Concat$ to merge cross-model features along the channel dimension. This process can be expressed as:

where $Concat$ is the feature concatenation operation along the channel dimension.

Then, the merged features are transformed into the target number of channels through the 1×1 convolutional layers in CMFFM. Subsequently, batch normalization and the ReLU activation function are applied to enhance feature representation and the network's nonlinear capability. The final fused feature $F_{multi}$ can be expressed as:

where $Con{v}_{1\times 1}$ is the 1×1 convolution operation, BN denotes batch normalization processing, and $\delta$ represents the application of Relu as the nonlinear activation function.

After the CMFFM processing, the fused features effectively combine multi-level information, incorporating rich high-level semantic representations. This mechanism successfully integrates critical attributes such as target categories and properties into the final feature encoding, enabling object detection models to leverage these advanced semantic cues for more precise localization and classification. Notably, the feature fusion module preserves sufficient spatial resolution throughout the fusion process, ensuring the retention of spatial fidelity at the feature level without compromising detection accuracy. The resultant fused features, enriched with comprehensive detail and semantic depth, significantly enhance the model's capability to capture subtle characteristics and small targets. By synergistically integrating heterogeneous feature sources, the framework improves the model's contextual understanding of inter-object relationships in complex scenes. It demonstrates outstanding performance advantages in detecting small targets within challenging environmental contexts.

3.4 Loss function

The network involved in this paper consists of a super-resolution network based on Generative Adversarial Networks (GANs) and a target detection network. During the training process, an end-to-end training approach is used, where the loss from the target detection is backpropagated to the generation network, guiding the generation network to reconstruct images that are more conducive to target detection.

We introduce a new loss in addition to the traditional adversarial loss, perceptual loss $L_{per}$, and L1 loss to prevent the edge enhancement module from over-enhancing the edges. Specifically, we use Charbonnier loss, i.e., the consistency loss $L_{char}$, between the super-resolved image (SR) and the high-resolution image (HR).

where $E_{xf}$ is the average value of all generated images in a batch. At the same time, $vg{g}_{fea}\left(x_f\right)$ and $vgg\left(x_r\right)$ denote the feature representations extracted by the convolutional neural network when the actual image $x_r$ and the super-resolved image $(x_f$ are fed into the network, respectively. $\rho (\cdot )$ is Charbonnier function.

Finally, we obtain the total loss of the edge-enhanced super-resolution generation network by adding the loss of the edge enhancement module to the original generative network's loss, with the empirical value set to ${\gamma }_1=1$, ${\gamma }_2=0.001$, ${\gamma }_3=0.01$, ${\gamma }_4=5$

where $L_G^{Ra}$ is the adversarial loss of the generator.

In the target detection network, Faster R-CNN, regression loss, and localization loss exist for the detected objects. Both losses are computed using the smoothed L1 loss. The classification loss $L_{cls}$, regression loss $L_{reg}$, and total detection loss $L_{det}$ can be expressed as:

where $T_{\ast }$ is the ground truth target coordinates, $(G_G(\cdot )$ denotes the super-resolution reconstruction network, $De{t}_{cls}$ is the classification loss in the target detection network, $De{t}_{reg}$ denotes the regression loss in target detection, $smoot{h}_{L_1}(\cdot )$ refers to the smoothed $L_1$ loss, and $\alpha$ is the balancing parameter. In the proposed network,

The total loss of the entire discriminator $L_{D\_det}$ can be expressed as:

where $L_D^{Ra}$ is the adversarial loss, and $\eta$ is the balancing parameter for the discriminator loss, which measures the contribution of the target detection network to the discriminator. Based on empirical experience, $\eta$ is set to 1.

Finally, the overall loss of the entire network architecture, $L_{overall}$, can be expressed as:

4.1 Dataset

Currently, the two mainstream datasets in the field of super-resolution target detection are the COWC [32] and RSOD [33] datasets. The COWC (Cars Overhead With Context) dataset consists of satellite images collected from six different geographic locations, with an image size of 256×256 pixels. The average target length ranges from 24 to 48 pixels, while the width ranges from 10 to 20 pixels. This dataset focuses solely on small targets of the "car" category and includes 3954 images for training and testing. 3164 images were randomly selected as the training set, while 790 were used as the test set. The RSOD dataset includes four types of objects: airplanes, sports fields, overpasses, and oil tanks. These objects exhibit diverse characteristics and have targets of varying sizes. The experiments divided this dataset into training and test sets in an 8:2 ratio.

4.2 Degradation processing of remote sensing image

Most methods for small target detection in super-resolution remote sensing rely on downsampling to create low-resolution datasets without considering the imaging characteristics of remote sensing images in practical scenarios. This leads to models that perform well on specific datasets but fail to generalize well on others, undermining their robustness and generalization ability. Figure 5 shows remote sensing image degradation methods, including blurring, downsampling, noise, and image compression.

Figure 5 remote sensing image degradation module.

Before model training, to obtain paired high/low-resolution images, it is necessary to preprocess the images in the dataset. A crucial part of this process is degrading the high-resolution remote sensing images to simulate the real remote sensing image generation environment. Let $I^{HR}\in {ℛ}^{C_{in}\times H\times W}$ and $I^{LR}\in {ℛ}^{C_{in}\times H\times W}$ represent the initial high-resolution image and its degraded low-resolution version, respectively. $C_{in}$, $H$, and $W$ correspond to the number of channels, height, and width of the input image.

First, Gaussian blurring is applied to the input high-resolution image $I^{HR}$. The blur kernel is based on a Gaussian probability density function with zero mean to achieve $N(0,\Sigma )$, resulting in $I_{blur}^{HR}\in {ℛ}^{C_{in}\times H\times W}$. This process can be represented as:

where $G_{blur}(\cdot )$ is the Gaussian blur function.

Secondly, considering that remote sensing images are inevitably affected by noise during the generation process, this paper focuses on two types of noise: additive noise in optical imaging systems and multiplicative features in remote sensing image imaging systems. Additive and multiplicative features are sequentially added to the blurred image. This process can be expressed as:

where $N_1$ is additive noise, which follows a normal distribution $X\sim N(0,{\sigma }^2)$. $N_2$ represents multiplicative features, where each image pixel is multiplied by a gain factor $N_2$. This gain factor follows an exponential distribution with unit mean $N_2\sim Exp\left(1\right)$.

The resolution of the image is reduced through downsampling to simulate the situation of a low-resolution remote sensing image, resulting in the low-resolution remote sensing image $I_I^{LR}\in R^{C_{in}\times H\times W}$. This process can be expressed as:

where $D(\cdot )$ represents the downsampling operation applied to the image.

Finally, the image parameters are adjusted to simulate remote sensing images $I_B^{LR}\in {ℛ}^{C_{in}\times H\times W}$ under different sunlight conditions. This process can be expressed as:

where $\Delta I$ is the brightness adjustment factor, $\alpha$ is the contrast adjustment factor, and $\overline{I}$ is the average brightness of the image.

In addition, to account for light attenuation caused by atmospheric scattering and the increase in ambient light due to scattering, the image after atmospheric scattering processing is the final low-resolution remote sensing image $I^{LR}$. The effect of atmospheric scattering on the image can be expressed as:

where $I^{LR}\left(x\right)$ is the image brightness at position $x$ after the influence of atmospheric scattering, $I_B^{LR}\left(x\right)$ is the brightness of the original image at position $x$, $\beta$ is the atmospheric scattering coefficient, $d\left(x\right)$ is the distance at position $x$ in the scene, and $A$ is the ambient light intensity.

Through the above degradation processing, this paper simulates more realistic low-resolution remote sensing images in the data processing, providing a solid foundation for the subsequent training and testing of the network.

4.3 Experimental Setup

For the images in the dataset, this paper first processes the original images using the proposed degradation method to obtain low-resolution images, forming high/low-resolution image pairs. During training, the low-resolution and high-resolution images are fed into the network for training. The parameter settings are as follows: the batch_size is set to 5, the learning rate for the super-resolution reconstruction network is initialized to 500, and the Adam optimizer parameters are set to ${\delta }_1=0.9$ and ${\delta }_2=0.99$. For the object detection network, the initial learning rate is set to 0.005, and the momentum parameter of the SGD optimizer is set to 0.9. The network is trained in an end-to-end manner. During the testing phase, only the low-resolution images must be input for super-resolution object detection without requiring high-resolution images.

This paper uses the MS COCO [22] evaluation matrix to evaluate the object detection results. It selects $A{P}^{50}$, $A{P}^S$, $A{P}^M$, $A{P}^L$, $AR$, $A{R}^S$, $A{R}^M$ and $A{R}^L$ as the verification criteria for the effectiveness of the object detection experiments to assess the performance of the degradation model.

4.4 Performance Evaluation

This paper evaluates the proposed EFSOD's object detection performance on remote sensing images from both subjective and objective perspectives. Considering the specific characteristics of remote sensing images, the original images are processed with blurring, downsampling, noise, simulated lighting changes, and atmospheric scattering when generating low-resolution images. In the comparative experiments with the baseline networks, ESRGAN+FRCNN and EESRGAN networks are compared with the proposed network, evaluating the networks from both object detection metrics and parameter/computation complexity aspects. EESRGAN adopts an end-to-end structure similar to our method. Additionally, this paper compares current mainstream object detection algorithms such as Faster RCNN++ [34], RetinaNet [35], CenterNet [36], and PP-YOLOv2 [37], as well as remote sensing small target detection algorithms like AVDNet [38], the target feature super-resolution vehicle detection algorithm TGFSR-VD [39], and ASAHI [40]. All of the above algorithms are based on the official code provided, and experiments are conducted with consistent experimental parameters as described in the papers, trained, and tested on the COWC and RSOD datasets.

Figure 6 Representative test results based on the COWC dataset.

Figure 7 Representative test results based on the RSOD dataset.

4.5 Ablation analysis

To verify the proposed CMFFM and EEDRM contribution to target detection accuracy, this section designs four comparative experiments on the baseline ESRGAN+FRCNN network using two different datasets, COWC and RSOD. Tables 7 and 8 show the performance comparison of each module on target detection.

From the performance shown in Tables 7 and 8, in the COWC dataset, both the CMFFM and EEDRM modules contribute significantly to improving the target detection accuracy of the baseline network ESRGAN+FRCNN. Regarding various metrics, the contribution of CMFFM is more significant than that of EEDRM. Specifically, in terms of accuracy, the most critical metric, $A{P}^{50}$ improves by 5.8%. In contrast, the small target detection accuracy, an important metric in this paper, $A{P}^S$, improves by 9.0%, and $A{P}^M$ improves by 13.0%. There is also a substantial improvement in recall, with $AR$ increasing by 11.7%, $A{R}^S$ by 9.8%, and $A{R}^M$ by 7.7%. In the RSOD dataset, similar to the experimental results in the COWC dataset, both modules, when used individually, effectively enhance the target detection performance compared to the baseline network ESRGAN+FRCNN. However, fully integrating the edge-enhanced dense residual block and the cross-model feature fusion module leads to better performance. Regarding accuracy, when $\sigma =0.5$, $A{P}^{50}$ increases by 6.3%, $A{P}^S$ by 5.3%, $A{P}^M$ by 3.7%, and $A{P}^L$L by 8.3%. In terms of recall, $AR$ increases by 11.9%, $A{R}^S$ by 11.7%, $A{R}^M$ by 10.2%, and $A{R}^L$L by 6.3%.

Introducing the CMFFM module in the target detection network dramatically enhances the feature extraction and fusion capabilities, significantly improving overall performance compared to the baseline network. The success of this improvement can be attributed to the effective combination of two key factors: First, the edge-enhanced super-resolution reconstruction network effectively restores high-frequency edge details of the image and fully exploits the relevant features beneficial for target detection within the super-resolution reconstruction network, significantly improving the quality of super-resolution target features. Second, the CMFFM module precisely and effectively integrates these excellent super-resolution target features with target detection-related features, further enhancing the performance of the target detection feature extraction network while maintaining the superior features of the super-resolution reconstructed image. This fusion strategy not only helps improve the accuracy of small target detection but also enhances the robustness and generalization ability of the network under challenging conditions such as complex backgrounds and lighting variations.