Implement tensor parallelism for linear layers using PyTorch.
Implement tensor parallelism for linear layers using PyTorch.: a task in terminal-bench-2.0 (Harbor dataset). ColumnParallelLinear splits the weight matrix by columns; the output should be concatenated along the last dimension as if using all gather; the bias should be sharded in the same way as…
The task
ColumnParallelLinear splits the weight matrix by columns; the output should be concatenated along the last dimension as if using all_gather; the bias should be sharded in the same way as the output dimension. RowParallelLinear splits the weight matrix by rows; the partial outputs should be summed together as if using…
Part of introvoyz041/terminal-bench-2.0.