The prediction of pelvic lymph node metastasis (PLNM) based on quantitative preoperative magnetic resonance imaging features of prostate cancer primary tumor is an important reference for treatment planning. However, current prediction methods inadequately capture the heterogeneity within the primary tumor, resulting in a weak correlation between extracted quantitative image features and PLNM prediction. To address the aforementioned issues, an attention-guided multi-task learning network with tumor segmentation as an auxiliary task is proposed for PLNM prediction. First, within the tumor segmentation network, a multi-branch anisotropic large kernel attention module is introduced, where a larger receptive field is obtained through different branches and anisotropic large convolutional kernels, effectively capturing both local and global tumor information. Then, within the PLNM prediction network, a multi-scale feature interaction fusion attention module is introduced to hierarchically fuse and select features from multiple scales. The experimental results on a dataset of 320 cases demonstrate that the area under the precision-recall curve and the area under the receiver operating characteristic curve of the method proposed are (85.44±2.04)% and (91.86±2.18)%, which are superior to state-of-the-art methods and multi-task approaches.