The way to Construct a Multi-GPU System for Deep Studying in 2023 #Imaginations Hub

The way to Construct a Multi-GPU System for Deep Studying in 2023 #Imaginations Hub
Image source -

My deep studying construct — work in progress :).

This story supplies a information on the right way to construct a multi-GPU system for deep studying and hopefully prevent some analysis time and experimentation.


Construct a multi-GPU system for coaching of pc imaginative and prescient and LLMs fashions with out breaking the financial institution! 🏦

Step 1. GPUs

Let’s begin with the enjoyable (and costly 💸💸💸) half!

The H100 beast! Picture from NVIDIA.

The primary concerns when shopping for a GPU are:

  • reminiscence (VRAM)
  • efficiency (Tensor cores, clock velocity)
  • slot width
  • energy (TDP)


For deep studying duties these days we’d like a loooot of reminiscence. LLMs are large even to fine-tune and pc imaginative and prescient duties can get memory-intensive particularly with 3D networks. Naturally crucial facet to search for is the GPU VRAM. For LLMs I like to recommend at the least 24 GB reminiscence and for pc imaginative and prescient duties I wouldn’t go beneath 12 GB.


The second criterion is efficiency which might be estimated with FLOPS (Floating-point Operations per Second):

The essential quantity previously was the variety of CUDA cores within the circuit. Nevertheless, with the emergence of deep studying, NVIDIA has launched specialised tensor cores that may carry out many extra FMA (Fused Multiply-Add) operations per clock. These are already supported by the principle deep studying frameworks and are what you must search for in 2023.

Beneath you’ll find a chart of uncooked efficiency of GPUs grouped by reminiscence that I compiled after fairly some handbook work:

Uncooked efficiency of GPUs based mostly on the CUDA and tensor cores (TFLOPs).

Observe that you need to be further cautious when evaluating efficiency of various GPUs. Tensor cores of various generations / architectures usually are not comparable. As an illustration, the A100 performs 256 FP16 FMA operations / clock whereas the V100 “solely” 64. Moreover, older architectures (Turing, Volta) don’t help 32-bit tensor operations. What makes the comparability harder is that NVIDIA doesn’t at all times report the FMA, not even within the whitepapers, and GPUs of the identical structure can have totally different FMAs. I saved banging my head with this 😵‍💫. Additionally notice that NVIDIA usually advertises the tensor FLOPS with sparsity which is a characteristic usable solely at inference time.

In an effort to determine the very best GPU with respect to cost, I collected the ebay costs utilizing the ebay API and computed the relative efficiency per greenback (USD) for brand spanking new playing cards:

Relative efficiency per USD of GPUs based mostly on the CUDA and tensor cores (TFLOPs / USD). Costs are based mostly on present ebay costs (September 2023).

I did the identical for used playing cards however because the rankings don’t change an excessive amount of I omit the plot.

To pick the very best GPU on your funds, you’ll be able to decide one of many prime GPUs for the most important reminiscence you’ll be able to afford. My advice would be:

Suggestion of GPUs for various budgets based mostly on present ebay costs (September 2023).

If you wish to dive into extra technical elements I counsel to learn Tim Dettmers’ wonderful information on Which GPU(s) to Get for Deep Studying.

Slot width

When constructing a multi-GPU system, we have to plan the right way to bodily match the GPUs right into a PC case. Since GPUs develop bigger and bigger, particularly the gaming collection, this turns into extra of a problem. Client motherboards have as much as 7 PCIe slots and PC instances are constructed round this setup. A 4090 can simply take up 4 slots relying on producer, so you’ll be able to see why this turns into a problem. Moreover we should always depart at the least 1 slot between GPUs that aren’t blower fashion or watercooled to keep away from overheating. We’ve got the next choices:

Watercooled variants will take as much as 2 slots however they’re dearer. You may alternatively convert an air-cooled GPU however this can void the guarantee. If you happen to don’t get All-in-One (AIO) options you’ll need to construct a customized watercooling loop. That is additionally true if you wish to match a number of watercooled GPUs because the AIO radiators might not match within the case. Constructing your individual loop is dangerous and I wouldn’t personally do it with costly playing cards. I might solely purchase AIO options straight from the manufactures (danger averse 🙈).

Aircooled 2–3 slot playing cards and PCIe risers
On this state of affairs you interleave playing cards on PCIe slots and playing cards related with PCIe riser cables. The PCIe riser playing cards might be positioned someplace contained in the PC case or within the open air. In both case you must make certain the GPUs are secured (see additionally the part about PC instances).

Energy (TDP)

Fashionable GPUs get increasingly energy hungry. As an illustration, A 4090 requires 450 W whereas a H100 can rise up to 700 W. Other than the facility invoice, becoming three or extra GPUs turns into a problem. That is very true within the US that the facility sockets can ship as much as round 1800w.

An answer to this downside in case you are getting near the max energy you’ll be able to draw out of your PSU / energy socket is power-limiting. All that you must cut back the max energy a GPU can draw is:

sudo nvidia-smi -i <GPU_index> -pl <power_limit>

the place:
GPU_index: the index (quantity) of the cardboard because it proven with nvidia-smi
power_limit: the facility in W you need to use

Energy-limiting by 10-20% has been proven to scale back efficiency by lower than 5% and retains the playing cards cooler (experiment by Puget Programs). Energy-limiting 4 3090s as an example by 20% will cut back their consumption to 1120w and may simply slot in a 1600w PSU / 1800w socket (assuming 400w for the remainder of the elements).

Step 2. Motherboard and CPU

The subsequent step of the construct is to choose a motherboard that permits a number of GPUs. Right here the principle consideration is the PCIe lanes. We’d like at minimal PCIe 3.0 slots with x8 lanes every for every of the playing cards (see Tim Dettmers’ put up). PCIe 4.0 or 5.0 are rarer and never wanted for many deep studying usecases.

Other than the slot kind, the spacing of the slots will decide the place you’ll be able to place the GPUs. Be sure you have checked the spacing and that your GPUs can truly go the place you need them to. Observe that the majority motherboards will use x8 configuration for some x16 slots while you use a number of GPUs. The one actual option to get this data is on the handbook of the card.

The best option to not spend hours of analysis and likewise future-proof your system is to choose a motherboard that has x16 slots all over the place. You should utilize PCPartPicker and filter motherboards which have 7+ PCIe x16 slots. This provides us 21 merchandise to select from. We then cut back the checklist by deciding on the minimal quantity of RAM we would like (e.g. 128 GB) with DDR4 / DDR5 kind to convey it right down to 10 merchandise:

Motherboards with at the least 7 PCIe x16 slots and 128 GB DDR4/DDR5 RAM based mostly on PCPartPicker.

The supported CPU sockets of the above checklist are LGA2011–3 and LGA2066. We then transfer to the CPU choice and choose CPUs with the specified variety of cores. These are primarily wanted for knowledge loading and batch preparation. Purpose to have at the least 2 cores / 4 threads per GPU. For the CPU we also needs to examine the PCIe lanes it helps. Any CPU of the final decade ought to help at the least 40 lanes (overlaying 4 GPUs at x8 lanes) however higher be protected than sorry. With a filtering of e.g. 16+ cores with the above sockets we get the next CPUs:

  • Intel Xeon E5 (LGA2011–3): 8 outcomes
  • Intel Core i9 (LGA2066): 9 outcomes

We then decide our favourite mixture of motherboard and CPU based mostly on the variety of cores, availability and worth.

Each LGA2011–3 and LGA2066 sockets are very previous (2014 and 2017 respectively), and due to this fact you’ll find good offers on ebay for each the motherboard and CPU. An ASRock X99 WS-E motherboard and a 18-core Intel Xeon E5–2697 V4 can price you lower than 300$ in used situation. Don’t purchase the cheaper ES or QS variations for CPUs as these are engineering samples and should fail ⚠️️.

If you wish to purchase one thing extra highly effective and/or more moderen and/or an AMD CPU you’ll be able to look into motherboards with e.g. 4+ PCIe x16 slots however be sure to examine the slot spacings.

At this stage it’s a good suggestion to begin a PCPartPicker construct. 🛠️
PCPartPicker will examine compatibilities between elements for you and can make your life simpler.

Step 3. RAM 🐏

Right here crucial facet is the quantity of RAM. RAM is used elsewhere of the deep studying cycle: loading knowledge from disk for batch creation, loading the mannequin and naturally prototyping. The quantity wanted relies upon so much in your utility (e.g. 3D picture knowledge will want rather more extra RAM) however you must purpose for 1x–2x the whole quantity of VRAM of your GPUs. The kind ought to be at the least DDR4 however the RAM clock shouldn’t be essential, so don’t spend your cash there 🕳️.

When shopping for RAM you must guarantee that the shape issue, kind, variety of modules and reminiscence per module all agree together with your motherboard specs (PCPartPicker is your buddy!).

Step 4. Disks

One other element that you could save on is the disks 😌. Once more the quantity of disk house is vital and depends upon the applying. You don’t essentially want ultra-fast disks or NVMEs as they gained’t have an effect on your deep studying efficiency. The info will probably be anyway loaded to RAM and in an effort to not create a bottleneck you’ll be able to merely use extra parallel CPU employees.

Step 5. Energy provide (PSU) 🔌

As we noticed GPUs are power-hungry elements. When organising a multi-GPU system, the choice of the PSU turns into an vital consideration. Nearly all of PSUs can ship as much as 1600w — that is in keeping with the facility limits of US sockets. There are a couple of PSUs that may ship greater than that however want some analysis they usually goal particularly miners.

Estimated wattage offered by PCPartPicker on your builds.

To find out the wattage of your system, you should utilize once more PCPartPicker that computes the whole quantity of your construct. To this we have to add an additional 10%+ for peace of thoughts since GPUs could have spikes of energy greater than what’s on their specs.

An vital criterion is the PSU effectivity that’s marked with the 80 PLUS score. The provision will attain the wattage it advertises however will lose some energy within the course of. 80 PLUS Bronze provides are rated with 82% effectivity vs e.g. a Gold that may attain 87% effectivity. If we have now a system that attracts 1600w and we use it 20% of the time, we might save 22$ per yr with a GPU with Gold score, assuming a value of 0.16$ / KWh. When evaluating costs take that under consideration in your calculations.

PSU effectivity scores. Desk from techguided.

When operating at full load some PSUs are extra noisy than others since they use a fan at excessive RPMs. If you’re working (or sleeping!) near your case this may have some impact, so it’s a good suggestion to examine the decibels from the handbook 😵.

When deciding on a provide, we have to confirm that it has sufficient connectors for all our components. GPUs specifically use 8 (or 6+2) pin cables. One vital notice right here is that for every energy slot of the GPU we should always use a separate 8 pin cable and never use a number of outputs of the identical cable (daisy-chaining). 8 pin cables are typically rated to ~150w. When utilizing a single cable for multiple energy slot the GPU might not get sufficient energy and throttle.

Step 6. PC case

Final however not least, deciding on a PC case shouldn’t be trivial. GPUs can get humongous and a few instances won’t match them. A 4090 as an example can attain 36 cm size 👻!

On prime of that, mounting GPUs with PCIe risers might require some hacks. There are some some newer instances that permit to mount an extra card, particularly twin system instances just like the Phanteks Enthoo 719. Another choice is the Lian-Li O11D EVO that may home a GPU in upright place with the Lian-Li Upright GPU Bracket. I don’t have these instances so I’m unsure how properly they’d match e.g. a number of 3090 / 4090. Nevertheless you’ll be able to nonetheless mount a GPU upright even when your PC case doesn’t straight help it with the Lian-Li bracket. You will want to drill 2–3 holes to the case however shouldn’t be loopy (information to observe!).

Mounting a Titan Xp in an upright place with the Lian Li upright bracket.

The top

I hope you loved studying this information and that you just discovered some helpful ideas. The information is aimed to assist in your analysis on constructing a multi-GPU system, and never substitute it. Be at liberty to ship me any questions or feedback you will have. If I’m mistaken on something within the above I might actually admire a remark or DM to make it even higher 🙏!

Observe: Until in any other case famous, all pictures are by the creator. I’ve included some affiliate Amazon hyperlinks. Shopping for an merchandise by way of the hyperlinks comes at no further price and I might doubtlessly obtain a small fee.

The way to Construct a Multi-GPU System for Deep Studying in 2023 was initially printed in In the direction of Knowledge Science on Medium, the place individuals are persevering with the dialog by highlighting and responding to this story.

Related articles

You may also be interested in