Low-voltage distribution systems frequently experience leakage current accidents. However, it is challenging to distinguish fault characteristics in abnormal operating conditions, such as electric shock to biological organisms and three-phase imbalance, which leads to the improper operation of existing residual current protection devices, making it difficult to balance both the safety and continuity of power supply in low-voltage distribution systems. Therefore, an optimization method is proposed for combination of the residual current protection and the coincidence reclosing logic to address the ground fault caused by the leakage of the low-voltage distribution system. First, the residual current protection device and the automatic reclosing switch are installed in the low-voltage side outlet of the transformer. When the over-threshold residual current of the fault phase is detected, the instantaneous skip circuit breaker is used to ensure the safety of the power supply. Then, the robust empirical mode decomposition-approximate entropy value and the intrinsic mode function-time domain eigenvalue of the residual current signal are extracted, followed by the establishment of a genetic algorithm-backpropagation over-threshold status identification neural network. Finally, based on the neural network model, the electrical and three-phase imbalance characteristics are effectively separated. The action logic of the residual current protection value setting algorithm and the automatic coincidence brake is optimized to ensure the continuity of power supply. The simulation analysis shows that the error of the classification results of the model after seven iterations is only 9.49×10^-9 with a correlation coefficient of 0.99, confirming the feasibility of the proposed method.