With the continuous advancement of sensors and algorithms, an increasing number of deep learning methods have been applied to fine-grained upper limb motion intention recognition using multimodal physiological signals. However, effectively and quantifiably integrating correlations between electroencephalogram (EEG) and electromyogram (EMG) signal channels as well as within EEG signal channels as a clue to improve performance remained challenging. In this paper, we proposed a novel framework that achieved accurate prediction of upper limb motion intentions via fusing EEG and EMG signals. Firstly, the raw input signals were fed into the feature extraction module, respectively, enabling feature decomposition in the channel dimension. Secondly, the graph convolution module with learnable edge weights was proposed to adaptively learn correlations between different modalities. Thirdly, we designed a self-attention graph pooling module that employed the self-attention mechanism to compute the attention score for each node as the basis for pooling. Compared with calculation methods using the mean or maximum value, this approach was more likely to retain nodes with stronger correlations to motor intentions. Finally, the prediction results were obtained through a classifier. We validated the effectiveness of our method on a publicly available multimodal upper limb dataset, achieving an accuracy of 93.17%.