DataLoader state is empty for different ranks ?

[ahatamiz]

Thanks for your amazing work !

We have been testing the llama3_8b model on slimpajama dataset. The training seem to be fine based on loss curves.

However, upon resuming the model from a previous checkpoint, we see the following warnings:

16: 2024-06-17 01:22:16,614 - root - WARNING - DataLoader state is empty for dp rank 16, expected key dp_rank_16.
28: 2024-06-17 01:22:16,614 - root - WARNING - DataLoader state is empty for dp rank 28, expected key dp_rank_28.
 5: 2024-06-17 01:22:16,614 - root - WARNING - DataLoader state is empty for dp rank 5, expected key dp_rank_5.
20: 2024-06-17 01:22:16,614 - root - WARNING - DataLoader state is empty for dp rank 20, expected key dp_rank_20.
27: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 27, expected key dp_rank_27.
 2: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 2, expected key dp_rank_2.
19: 2024-06-17 01:22:16,614 - root - WARNING - DataLoader state is empty for dp rank 19, expected key dp_rank_19.
30: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 30, expected key dp_rank_30.
23: 2024-06-17 01:22:16,614 - root - WARNING - DataLoader state is empty for dp rank 23, expected key dp_rank_23.
21: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 21, expected key dp_rank_21.
17: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 17, expected key dp_rank_17.
18: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 18, expected key dp_rank_18.
 1: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 1, expected key dp_rank_1.
26: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 26, expected key dp_rank_26.
31: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 31, expected key dp_rank_31.
12: 2024-06-17 01:22:16,614 - root - WARNING - DataLoader state is empty for dp rank 12, expected key dp_rank_12.
10: 2024-06-17 01:22:16,614 - root - WARNING - DataLoader state is empty for dp rank 10, expected key dp_rank_10.
11: 2024-06-17 01:22:16,615 - root - WARNING - DataLoader state is empty for dp rank 11, expected key dp_rank_11.
14: 2024-06-17 01:22:16,616 - root - WARNING - DataLoader state is empty for dp rank 14, expected key dp_rank_14.
15: 2024-06-17 01:22:16,616 - root - WARNING - DataLoader state is empty for dp rank 15, expected key dp_rank_15.
13: 2024-06-17 01:22:16,616 - root - WARNING - DataLoader state is empty for dp rank 13, expected key dp_rank_13.
29: 2024-06-17 01:22:16,616 - root - WARNING - DataLoader state is empty for dp rank 29, expected key dp_rank_29.
 7: 2024-06-17 01:22:16,617 - root - WARNING - DataLoader state is empty for dp rank 7, expected key dp_rank_7.
 8: 2024-06-17 01:22:16,617 - root - WARNING - DataLoader state is empty for dp rank 8, expected key dp_rank_8.
 4: 2024-06-17 01:22:16,617 - root - WARNING - DataLoader state is empty for dp rank 4, expected key dp_rank_4.
 3: 2024-06-17 01:22:16,618 - root - WARNING - DataLoader state is empty for dp rank 3, expected key dp_rank_3.
 9: 2024-06-17 01:22:16,618 - root - WARNING - DataLoader state is empty for dp rank 9, expected key dp_rank_9.
 6: 2024-06-17 01:22:16,619 - root - WARNING - DataLoader state is empty for dp rank 6, expected key dp_rank_6.
22: 2024-06-17 01:22:16,619 - root - WARNING - DataLoader state is empty for dp rank 22, expected key dp_rank_22.
24: 2024-06-17 01:22:16,619 - root - WARNING - DataLoader state is empty for dp rank 24, expected key dp_rank_24.
25: 2024-06-17 01:22:16,619 - root - WARNING - DataLoader state is empty for dp rank 25, expected key dp_rank_25.

What can be the reason for DataLoader state being empty when loading the model ?

Also noting that checkpoints are loaded properly.

[tianyu-l]

@ahatamiz Thanks for the question.

I wonder if you have changed the number of GPUs / number of data_parallel_degree/tensor_parallel_degree when resuming from the previous checkpoint. Currently resharding is not supported by the data loader.