In multi-operation manufacturing systems, production and maintenance interact, and machine condition is coupled with performance loss. To quantify the effects of maintenance and improve the economic performance of maintenance policies, this study considers the state evolution of machines and proposes a production-line maintenance strategy from a multi-cost perspective. Maintenance timing and level are taken as decision variables, and an integrated cost optimization model is built that includes preventive maintenance cost, downtime cost, delivery gap penalty, and performance loss cost on both product and equipment sides. To handle the high-dimensional and time-dependent action space, maintenance decision units are represented as graph nodes, and a graph-attention-based proximal policy optimization algorithm (GAT-PPO) is developed. Case studies show that the obtained policy can automatically identify high-risk machines and critical periods. Comparative experiments further indicate that the proposed algorithm achieves a favorable trade-off between solution quality and online computation time under different maintenance planning horizons, demonstrating strong potential for engineering applications and extension.