Lens Flare Artifact Removal: Bridging Traditional Methods with Deep Learning Models (Part-2)
This study comprehensively examines how flare removal techniques have progressed over time. We start with the basics, like hardware fixes, think anti-reflective coatings and lens hoods, and move on to older computational tricks like PSF-based deconvolution and image inpainting. From there, we get into the newer world of deep learning models, including U-Net, Generative Adversarial Networks (GANs), and Transformer-based architectures, which are pushing the envelope in automatically reducing flare effects.
Series: Lens Flare Artifact Removal: Bridging Traditional Methods with Deep Learning Models (4 articles)
- 2Lens Flare Artifact Removal: Bridging Traditional Methods with Deep Learning Models (Part-2) This article
DEEP LEARNING-BASED APPROACHES
Deep learning boosts the flare removal process by automatically removing both spotting and removing flare artifacts. Unlike the older methods that need manual tweaks or rely on assumptions, these models can adapt to different lighting setups, types of images, and flare styles. That makes them super flexible and easy to scale. They use neural networks to learn the difference between flare spots and real image details, so they can clean up even complicated scenes and restore the image accurately.
U-NET-BASED APPROACHES
Widely used for Flare Removal by utilizing multi-scale feature extraction, U-Net is a Convolutional Neural Network (CNN) architecture, which is designed for the segmentation of images. By utilizing the ability to extract multi-scale image features, it can be used for flare removal, unlike classical pipelines that required manual artifact segmentation. Its encoder-decoder structure allows for the extraction of both local and global image details, making it effective in reconstructing flare-free images.
The encoder progressively reduces spatial resolution while increasing feature complexity, while the decoder upscales features to restore fine details. Variants of U-Net, such as ResU-Net and Attention U-Net, improve performance by integrating residual connections and attention mechanisms to focus on essential image regions. These modifications enhance the network’s ability to remove flares without over-smoothing or distorting fine details. Datasets used for training U-Net-based flare removal models typically consist of real-world and synthetic flare-corrupted images.
The models are trained using supervised learning, where clean images serve as ground truth. Loss functions such as Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM) guide the training process. Some advanced U-Net variants integrate adversarial training by incorporating a discriminator network to refine results further. The adaptability of U-Net across different flare conditions makes it a strong baseline for deep-learning-based flare removal solutions.
U-Net effectively removes flares by preserving high-resolution details and world context through its encoder-decoder structure and skip connections. Variants like ResU-Net and Attention U-Net improve accuracy by focusing on prominent regions and preserving features. The flexible model accommodates various flare types and can be robust by adversarial training for enhanced visual quality.
U-Net models require a significant amount of paired training data, which is hard to come by in the real world. They are also computationally expensive, especially with higher-end versions, which makes mobile or real-time deployment challenging. Overfitting can occur if the model is not trained on varied flare examples.
ResU-Net, which combines the benefit of U-Net with residual learning, is efficient for removing flares because it can simultaneously learn both local and global features and preserve fine details of the images. Its architecture allows it to handle complex and variant flare patterns more stable than the traditional methods, so it can be used to deal with challenging real-scene cases.
Attention U-Net enhances the base U-Net architecture with learnable attention gates that automatically focus on flare-affected regions and de-emphasize background features that are unnecessary. By pointing its feature extraction and skip-connection procedure at lens flare-affected areas most impacted, it achieves better localization and eradication of complex, changing flare patterns—without losing fine image details.
The Daytime Flare Removal Using U-Net Approach introduced one of the early deep learning models used for daytime flare removal utilizing a U-Net based architecture. They focused on a data-driven approach by considering images taken under strong sunlight to eliminate the respective lens flares, which can be viewed as a milestone in photography advancement. They used a U-Net model which is trained on a dataset containing paired images in which one is a flare-corrupted image and the other is it's respective clean image, by using the dataset of that kind it is ensured that the network will be able to learn to reconstruct flare-free images while the image details are preserved.
The dataset was made sure to include images with different light conditions and different flare intensities to make the model more robust. The method involves the U-Net by skipping connections to ensure that the fine details that might be lost during the encoding are preserved. To compare feature representations at a high level rather than differentiating pixel-wise, they applied perceptual loss functions, which in turn led to better results. They utilized methods like random cropping and brightness adjustments to augment the data and enhance generalization.
The model is significantly better than the traditional approaches in flare removal, thus highlighting deep learning's potential. This method was one of the first successful applications of deep learning, in the guise of a U-Net model, to daytime flare removal. By training with corresponding flare and clean images, the model learns to remove flares without erasing scene details. Perceptual loss improved image quality over pixel-wise methods, and data augmentation aided generalization.
Their approach outperformed traditional flare removal techniques, showing the capability of deep learning to achieve it. The method relies on a paired set of flare-damaged and clean images, which in real-world conditions with varying diversity may be difficult to acquire. Further, while the U-Net model is effective, it could be very computationally intensive when trained. The method may not function very effectively with extreme cases of flare not present in the training set and could function only under common lighting conditions.
The Nighttime Flare Removal with U-Net introduced a dataset that was created explicitly for nighttime flare removal to address the significant gap in existing research; the dataset was named Flare7K. Previous works only focused on daytime flare removal. It is essential to consider the nighttime images, as they contain different flare patterns that might be occurring due to artificial light sources like vehicle headlights and streetlights. The Flare7K dataset contains 7,000 nighttime images comprising both real-world and synthetic images to train the deep learning models on a diverse dataset.
They optimized a U-Net based model for the unique features of nighttime light. They proposed a modified version of the U-Net architecture with the implementation of spectral normalization and adaptive feature enhancement to differentiate the flare from the background light. They included methods like random rotations and contrast adjustments in the training pipeline to augment the data to improve generalization. They utilized metrics like Peak Signal-to-Noise Ratio (PSNR) and SSIM to evaluate the results, thus showing an improvement over the traditional method.
The introduction of Flare7K has paved the way for the development of more robust solutions. This bridged an important research gap by introducing Flare7K, the first large-scale dataset solely intended for nighttime flare removal. Their dataset features real and synthesized nighttime images that capture flares of varying patterns produced by artificial light. They enhanced a U-Net model with spectral normalization and feature-adaptive enhancement, boosting the network in its ability to separate flares from backgrounds with clutter.
Despite its strengths, the method depends on the quality and completeness of the Flare7K dataset, which might not cover all kinds of nighttime scenarios in the real world. Using a U-Net variant architecture contributes to higher computational complexity, which can restrain deployment on low-power platforms. Lastly, synthetic images are approximations, albeit useful ones, of genuine nighttime flares, which can undermine model generalizability under highly fluctuating situations.
As shown in Table I, Baseline U-Net applies skip connections to normal flares but degrades details, ResU-Net utilizes residual blocks for intense glare at increased complexity, whereas Attention U-Net addresses reflective flares with attention gates but needs fine-tuning. The baseline model delivers basic structure preservation without refinement. ResU-Net adds residual learning to address extreme glare at the expense of simplicity for robustness. Attention U-Net refines further with selective emphasis on features, delivering accuracy at the expense of tuning complexity–illustrating how architectural developments deal with varying flare characteristics.
TABLE I: Comparative Analysis of U-Net-Based Flare Removal Approaches
| Features | Baseline U-Net | ResU-Net | Attention U-Net |
|---|---|---|---|
| Core Innovation | Encoder-decoder with skip connections | Residual blocks in each layer | Attention gates in skip connections |
| Flare Target | General daytime flares | Large scattering flares | Localized reflective flares |
| Datasets Used | Semi-Synthetic Data | Flare7K++ | Flare7K++ |
| Strenghts | Preserves structures | Handles heavy glare | Avoids over-smoothing |
| Challenges | Blurs fine details | More parameters | Needs careful tuning |
TRANSFORMER-BASED APPROACHES
Transformers for Global Feature Extraction, Improving Generalization and Robustness, although originally developed for Natural Language Processing (NLP), Transformers have gained attention in Computer Vision (CV) because of their ability to capture long-range dependencies in an image. Transformers use self-attention mechanisms to establish relations between image regions rather than using local receptive fields like CNNs. This ability is specifically crucial for flare removal, where artifacts can affect large portions of an image.
Transformers improve the generalization for different light conditions and flare patterns by considering the entire image. The self-attention mechanism helps differentiate actual scene content from flare artifacts. The CNN models struggle with spatial invariance, where transformers come into the picture since they can dynamically adapt to different flare intensities and positions. In high-resolution image restoration tasks, transformers outperform traditional CNNs.
As shown in Table II, transformer-based flare removal models show progressive improvement, with newer methods achieving higher PSNR and SSIM and lower Learned Perceptual Image Patch Similarity (LPIPS), indicating better reconstruction and perceptual quality.
Transformers learn a worldwide context using self-attention, effectively detecting and removing prevalent flare artifacts. They robustly generalize between different lights and projects to outperform CNNs for the restoration of high-resolution images. They are computationally costly in the form of additional data, computation, and training. Transformers are also susceptible to overfitting in small data sets and lack the spatial inductive biases that make CNNs perform efficiently on certain image tasks.
According to Figure 3, Flare-Free Vision achieves the highest PSNR and SSIM, and the lowest LPIPS, indicating superior perceptual and structural restoration quality. Flare-Free Vision demonstrates the highest PSNR, SSIM, and the lowest LPIPS, confirming its effectiveness across all performance metrics.
Uformer is a U-shaped Transformer constructed for image restoration tasks, including flare removal. It has the encoder-decoder structure of U-Net while replacing the convolution layers with self-attention mechanisms. This allows Uformer to model both the local and global dependencies in images, making it more effective. The model is trained on a dataset containing paired images: one is a flare-corrupted image, and the other is its respective clean image, using supervised learning.
It uses Multi-Head Self-Attention (MHSA) and shifted window attention to improve efficiency while preserving fine details. L1 loss and perceptual loss are the loss functions used to optimize the quality of the image. The results have shown that it outperformed CNN models like U-Net, thus stating the advantages of transformers in the restoration of images. Uformer incorporates the strengths of the architecture of U-Net with Transformers in a manner that makes it possible to model local and global features with high efficiency.
It utilizes shifted window attention to balance efficiency with accuracy, and it outperforms CNN-based models like U-Net in image restoration tasks like flare removal. While with improved performance, Uformer is computation-intensive due to using self-attention layers. It also requires big, paired datasets for training and may be harder to deploy in real time or on low-end devices.
Restormer is an encoder-decoder transformer that is optimized for the purpose of restoring high-resolution images. It is an improved version of existing transformer models using multi-Dconv head Transposed Attention (MDTA), which allows features to be extracted at different levels. The dataset used to train Restormer has diverse real-world and synthetic images of various kinds of lens flares. Methods like random noise injection and exposure variations are included in the training pipeline to advance the augmentation and make the model robust.
Restormer is designed for high-resolution image restoration with MDTA to effectively extract multi-scale details. Restormer is good at handling complex flare patterns and has robust training with real-world and synthetic datasets, which enables better generalization. The model is computationally expensive, especially for bigger images, since it is transformer-based. The model also requires a lot of data augmentation and longer training times, and therefore may not be ideal for low-resource or real-time applications.
FF-Former is an image de-raining network based on transformers that offers sparse attention through retaining only the top-k most informative attention scores, as decided by a learnable selection mechanism. It combines a mixed-scale feed-forward network for feature extraction from local and global levels with a CNN-based feature compensator to help improve global modelling. Such a configuration enables FF-Former to recover rain-degraded images effectively since it yields optimal performance on conventional de-raining benchmarks.
TABLE II: Performance Comparison of Transformer-Based Flare Removal Models
| Method /Transformer | PSNR (dB) | SSIM (x25) | LPIPS (x100) |
|---|---|---|---|
| Uformer | 23.12 | 0.850 | 0.120 |
| Sparse-UFormer | 23.43 | 0.859 | 0.111 |
| FF-Former | 23.75 | 0.868 | 0.105 |
| Flare-Free Vision | 24.10 | 0.872 | 0.098 |
Sparse-UFormer is a transformer-based model that introduces sparse multi-scale feature extraction to reduce the computational complexity and increase performance. It uses a self-attention mechanism in a dynamic way, focusing only on relevant image regions. The model is trained using contrastive learning to improve its ability to differentiate flare and scene content in an image.
Results show that it significantly reduces the processing time while not harming the image quality. Sparse-UFormer utilizes sparse multi-scale self-attention to focus only on the significant areas, significantly reducing computational overhead and boosting processing. Sparse-UFormer makes use of contrastive learning to demarcate flares even more distinctly from the scene elements, providing maximum image quality even for real-time tasks.
The dynamic attention mechanism adds model complexity and demands careful design and fine-tuning. Contrastive learning demands careful data sampling and could be sensitive to training stability. Real-time improvement may have some cost in terms of extreme flare pattern complexity.
Flare-Free Vision offers an effective method for training neural networks to remove diverse lens flare patterns like halos, streaks, and haze. Instead of using synthetic data, it uses a physically informed dataset created by capturing real flare artifacts under controlled lighting and combining them with clean images. By separating light sources from flare during training, the model learns to remove flare while preserving brightness, enabling strong generalization across different cameras and flare types.
Figure 3. Bar chart comparing flare removal methods using PSNR, SSIM (×25), and LPIPS (×100), with SSIM and LPIPS scaled for visual comparability.
CONCLUSION
Overall, deep learning approaches have improved lens flare removal by learning to distinguish artifacts from image details. However, challenges such as data requirements and computational complexity remain. The next part we are going to discuss GAN-based approaches, which aims to further improve the quality and realism of flare-free images.
Series: Lens Flare Artifact Removal: Bridging Traditional Methods with Deep Learning Models (4 articles)
- 2Lens Flare Artifact Removal: Bridging Traditional Methods with Deep Learning Models (Part-2) This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article