Container collection should be completed within the specified time limit to prevent the arriving ships from being unable to leave the port on schedule. The storage yard should formulate a container allocation plan and yard crane scheduling plan based on the scheduled arrival time of container delivery and loading. Otherwise, the plan will become invalid, which will lead to the problem of container turnover and loading inefficiency. Aimed at the impact of the non-engagement of container delivery and truck collection on the container location allocation plan, yard crane scheduling plan, and the waiting time of the truck, an optimization problem of container location allocation and yard crane scheduling in multi-container areas under non-engagement of container delivery and truck collection is proposed. Considering the constraints such as the booking period of the truck and the safe distance between the two yard cranes, a mixed integer programming model is established with the goal of minimizing the time of port concentration. The hybrid genetic variable neighborhood algorithm is used to solve the model. A comparison of the experimental results of different algorithms indicates that the algorithm in this paper has a fast convergence speed and excellent solution results. Based on the actual arrival time of the container truck, a disturbance recovery strategy is proposed to measure the impact of non-engagement on the completion time and container truck waiting time in different scenarios, an interference recovery strategy is proposed. Experiments show that the interference recovery strategy proposed in this paper not only shortens the job completion time and the number of containers overturned due to box delivery breach, but also reduces the waiting time of container delivery trucks.