https://www.nature.com/articles/s41550-021-01323-1

Wolszczan, A. 1991, Nature, 350, 688,
doi: 10.1038/350688a0

[Difx-users] HPC using Ethernet vs Infiniband
From 	Walter Brisken via Difx-users
Sender 	Difx-users
To 	difx-users@nrao.edu
Reply-To 	Walter Brisken
Date 	Today 15:28
Contact photo
Message Body

Hi DiFX Users,

In the not so distant future we at VLBA may be be in the position to upgrade the network backbone of the VLBA correlator.  Currently we have a 40 Gbps Infiniband system dating back about 10 years.  At the time we installed that system, Infiniband showed clear advantages, likely driven by RDMA capability which offloads a significant amount of work from the CPU.  Now it seems Ethernet has RoCE (RDMA over Converged Ethernet) which aims to do the same thing.

1. Does anyone have experience with RoCE?  If so, is this as easy to configure as the OpenMPI page suggests?  Any drawbacks of using it?

2. Has anyone else gone through this decision process recently?  If so, any thoughts or advice?

3. Has anyone run DiFX on an RoCE-based network?

    -Walter

-------------------------
Walter Brisken
NRAO
Deputy Assistant Director for VLBA Development
(505)-234-5912 (cell)
(575)-835-7133 (office; not useful during COVID times)