cross-posted from: https://ani.social/post/16779655

GPU VRAM Price (€) Bandwidth (TB/s) TFLOP16 €/GB €/TB/s €/TFLOP16
NVIDIA H200 NVL 141GB 36284 4.89 1671 257 7423 21
NVIDIA RTX PRO 6000 Blackwell 96GB 8450 1.79 126.0 88 4720 67
NVIDIA RTX 5090 32GB 2299 1.79 104.8 71 1284 22
AMD RADEON 9070XT 16GB 665 0.6446 97.32 41 1031 7
AMD RADEON 9070 16GB 619 0.6446 72.25 38 960 8.5
AMD RADEON 9060XT 16GB 382 0.3223 51.28 23 1186 7.45

This post is part “hear me out” and part asking for advice.

Looking at the table above AI gpus are a pure scam, and it would make much more sense to (atleast looking at this) to use gaming gpus instead, either trough a frankenstein of pcie switches or high bandwith network.

so my question is if somebody has build a similar setup and what their experience has been. And what the expected overhead performance hit is and if it can be made up for by having just way more raw peformance for the same price.

  • hendrik@palaver.p3x.deEnglish
    3·
    10 hours ago

    I think there are some posts out there (on the internet / Reddit / …) with people building crazy rigs with old 3090s or something. I don’t have any experience with that. If I were to run such a large model, I’d use a quantized version and rent a cloud server for that.

    And I don’t think computers can fit infinitely many GPUs. I don’t know the number, let’s say it’s 4. So you need to buy 5 computers to fit your 18 cards. So add a few thousand dollars. And a fast network/interconnect between them.

    I can’t make any statement for performance. I’d imagine such a scenario might work for MoE models with appropriate design. And for the rest performance is abysmal. But that’s only my speculation. We’d need to find people who did this.

    Edit: Alternatively, buy a Apple Mac Studio with 512GB of unified RAM. They’re fast as well (probably way faster than your idea?) and maybe cheaper. Seems an M3 Ultra Mac Studio with 512GB costs around $10,000. With half that amount, it’s only $7,100.