LuWu: An End-to-End In-Network Out-of-Core Optimizer for 100B
The key idea of LuWu is to offload the entire optimizer states and parameters from GPU workers onto an in-network optimizer node (C2) and to offload the entire collective communication from GPU
Mehr lesen