drbd-user May 2010 archive
Main Archive Page > Month Archives  > drbd-user archives
drbd-user: Re: [DRBD-user] Infiniband card support and help

Re: [DRBD-user] Infiniband card support and help

From: Michael Iverson <miverson_at_nospam>
Date: Mon May 31 2010 - 11:45:22 GMT
To: Igor Neves <igor@3gnt.net>


I'm basically doing the same thing, only with MHEA28-XTC cards. I wouldn't
think you'll have any problems creating a similar setup with the MHES cards.

I've not attempted to use infiniband sdr, just ipoib. I am running opensm on
one of the nodes. I'm getting throughput numbers like this:

cirrus:~$ netperf -H stratus-ib
TCP STREAM TEST from ( port 0 AF_INET to
stratus-ib.focus1.com ( port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

 87380 16384 16384 10.00 7861.61

A couple of things to watch out for:

1. Upgrade the firmware on the cards to the latest and greatest version. I
saw about a 25% increase in throughput as a result. The firmware updater was
a pain to compile, but that was mostly due to Ubuntu's fairly rigid default
compiler flags.

2. Run the cards in connected mode, rather than datagram mode, and put the
MTU at the max value of 65520. My performance benchmarks of drbd show that
this is the best setup.

The replication rate on my setup is completely limited by the bandwidth of
my disk subsystem, which is about 200 MB/s for writes. I can share some
performance comparisons between this and bonded gigabit ethernet, if you
would like. However, I won't be able to provide it until tomorrow, as it is
a holiday in the US today, and I don't have ready access to the data.

On Mon, May 31, 2010 at 6:17 AM, Igor Neves <igor@3gnt.net> wrote:

> Hi,
> I'm looking for a 10Gbit backend for storage drbd replication. I'm
> expecting to setup infiniband solution connected back to back, this means
> both nodes will be connected together without a switch.
> I wonder if I bought two of this cards MHES14-xtc and a cable, I will be
> able to produce such setup?
> Link to the cards:
> http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=19&menu_section=41
> Another question, I intend to use this with infiniband sdr support added to
> drbd in 8.3.3, and I found this on the spec's of the card.
> "In addition, the card includes internal Subnet Management Agent (SMA) and
> General Service Agents, eliminating the requirement for an external
> management agent CPU."
> This means I don't need to run openSM in any nodes? I will just need to
> setup two cards, a cable, connect them, and setup IPoIB to start replicating
> in 10Gbit?
> Thanks very much,
> --
> Igor Neves<igor.neves@3gnt.net>
> 3GNTW - Tecnologias de Informação, Lda
> SIP: igor@3gnt.net
> MSN: igor@3gnt.net
> JID: igor@3gnt.net
> PSTN: 00351 252377120
> _______________________________________________
> drbd-user mailing list
> drbd-user@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

-- Dr. Michael Iverson Director of Information Technology Hatteras Printing

drbd-user mailing list