A Pair-Weighing Strategy for Enhancing Clip Zero-Shot Classification for Chest X-Rays

Chest radiography (CXR) is essential for diagnosing life-threatening diseases, yet the growing demand exceeds the availability of expert radiologists. Although deep learning models have achieved expertlevel performance, they require large labelled datasets, which are costly to obtain in the medical domain. Recent advances in selfsupervised representation learning, particularly contrastive visionlanguage models such as CLIP, enable label-efficient pre-training for zero-shot classification. In this work, we propose an optimized CLIP pre-training pipeline for chest X-ray interpretation that reduces computational costs while improving efficiency. Our method pretrains CLIP on chest X-rays and corresponding clinical reports, introducing two key modifications: a weighted contrastive loss to mitigate false negatives and prevent suboptimal representations, and a parameter-efficient fine-tuning strategy using LoRA to reduce computation requirements. We evaluate our approach on radiologistlabelled datasets across multiple pathology groups, demonstrating improved performance over standard CLIP pre-training and strong generalization to unseen pathologies.