I think there are some posts out there (on the internet / Reddit / …) with people building crazy rigs with old 3090s or something. I don’t have any experience with that. If I were to run such a large model, I’d use a quantized version and rent a cloud server for that.

And I don’t think computers can fit infinitely many GPUs. I don’t know the number, let’s say it’s 4. So you need to buy 5 computers to fit your 18 cards. So add a few thousand dollars. And a fast network/interconnect between them.

I can’t make any statement for performance. I’d imagine such a scenario might work for MoE models with appropriate design. And for the rest performance is abysmal. But that’s only my speculation. We’d need to find people who did this.

Edit: Alternatively, buy a Apple Mac Studio with 512GB of unified RAM. They’re fast as well (probably way faster than your idea) and maybe cheaper. Seems an M3 Ultra Mac Studio with 512GB costs around $10,000. With half that amount, it’s only $7,100.

Well, I wouldn’t call them a “scam”. They’re meant for a different use-case. In a datacenter, you also have to pay for rack space and all the servers which accomodate all the GPUs. And you can now pay for 32 times as many servers with Radeon 9060XT or you buy H200 cards. Sure, you’ll pay 3x as much for the cards itself. But you’ll save on the amount of servers and everything that comes with it, hardware cost, space, electricity, air-con, maintenance… Less interconnect makes everything way faster…

Of course at home different rules apply. And it depends a bit how many cards you want to run, what kind of workload you have… If you’re fine with AMD or you need Cuda…

Thanks for the random suggestion! Installed it already. Sadly as a drop-in replacement it doesn’t provide any speedup on my old machine, it’s exactly the same number of tokens per second… Guess I have to learn about the ik_llama.cpp and pick a different quantization of my favourite model.

Thanks. I’ll factor that in next time someone asks me for a recommendation. I personally have Kobold.CPP on my machine, that seems to be more transparent toward such things.

Is there any background information available on ollama becoming less open? It’s marked MIT licensed in the repo of my Linux distribution and on their Github.

Wasn’t Mistral AI supposed to be one of the European (French) answers to mostly US companies doing AI? From that perspective it wouldn’t be super great if an US company were to buy them. And the company goals don’t match either as Apple is mostly concerned with their own products. So I’d say they’d likely dismantle the company and we have one less somewhat open AI company.

I like YunoHost. That’s an all-in-one solution to do the selfhosting for you. So you won’t learn a lot about the intricate details of the tech, but you can install things with a few clicks. That’s nice if you just want to use stuff. And that project has some track-record. I’m using it for years to self-host Peertube, Immich a Nextcloud and a few other things.

Oh man, I’m a bit late to the party here.

He really believes the far-right Trump propaganda, and doesn’t understand what diversity programs do. It’s not a war between white men an all the other groups of people… It’s just that is has proven to be difficult to for example write a menstrual tracker with a 99.9% male developer base. It’s just super difficult to them to judge how that’s going to be used in real-world scenarios and what some specific challenges and nice features are. That’s why you listen to minority opinions, to deliver a product that caters to all people. And these minority opinions are notoriously difficult to attract. That’s why we do programs for that. They are task-forces to address things aside from what’s mainstream and popular. It’ll also benefit straight white men. Liteally everyone because it makes Linux into a product that does more than just whatever is popular as of today. Same thing applies to putting effort into screen readers and disabled people and whatever other minorities need.

If he just wants what is majority, I’d recommend installing Windows to him. Because that’s where we’re headed with this. That’s the popular choice, at least on the desktop. That’s what you’re supposed to use if you dislike niche.

Also his hubris… Says Debian should be free from politics. And the very next sentence he talks his politics and wants to shove his Trump anti-DEI politics into Debian… Yeah, sure dude.

Ah thanks. That makes sense.

Nice theory. Was there a plan to construct high speed trains across the USA? I heard people there don’t really want a good train system in the first place, and I must have missed it if any political party put that on their agenda.

To be fair, this is a 1.1-tonne test vehicle. While the Japanese maglev is an entire train. I guess the real issue, though, is to get it from a tech demonstration into a prototype and then into an actual product. China doesn’t even have a prototype with that. And the maglev in Japan is just a prototype. I think it’s on a test track to show off a few times a day, but doesn’t transport people anywhere. And we had other people fail early on with the entire vacuum tube idea. Like Elon Musk also promised that a decade ago and it’s scrapped now.

Hmm… Would be interesting to find out what kind of effect that has on the average marriage or relationship 😅

Likely everyday stuff… Meeting minutes, phone or video conferences and such…

I think a dual-channel system with DDR3-1600 isn’t what we call fast any more. So you should try to avoid offloading with that. But I’m not an expert on the numbers, and it depends a bit on the specific use-case whether it makes sense to invest in old hardware, or buy a new machine along with a graphics card, since that’s quite some money.

Sure. I mean we seem to be a bit different and have different visions. So I’m not sure if I’m the correct person to take your idea to pieces and add my spin on it… That could take away from a clear vision and turn it into a mess. Maybe it’s better if I do my thing and you do yours… But I’m not sure about that. My DMs are open, so feel free to DM me. I’m just not sure whether I’m able to contribute.

I meant both sex and gender. They regularly fail to tell me a lot for my own real life. I like some people and dislike others and it’s easier for me to talk to / work with / collaborate or empathize depending on various circumstances. Personality traits, shared goals… Maybe sharing something or it’s the opposite of that. I believe gender or sex or identity is a bit overrated and so is stereotyped thinking for a lot of applications. Or the need to conform to a stereotype. Dress and identify however you like, make sure to give your children an electronics kit, a plastic excavator and a princess dress… And unless that’s really important for some niche application, don’t feel the urge to look into people’s pants and check what’s in there.

You’re welcome. I mean it’s kind of a factual question. Is gender an indicator on its own? If yes, then the rest is just how statistics and probability work… And that’s not really a controversy. Maths in itself works 🥹

I’d also welcome if we were to cut down on unrelated stuff, stereotypes and biases. Just pick what you like to optimize for and then do that. At least if you believe in the free market in that way. Of course it also has an impact on society, people etc and all of that is just complex. And then women and men aren’t really different, but at the same time they are. And statistics is more or less a tool. Highly depends on what you do with it and how you apply it. It’s like that with most tools. (And LLMs in the current form are kind of a shit tool for this if you ask me.)

Right. If it’s true that women statistically outperform men (with same application documents), it’d be logical to prefer them just on gender alone. Because they likely turn out to be better.

Issue is they probably want to pattern-recognize something like merit / ability / competence here. And ignore other factors. Which is just hard to do.

LLMs reproducing stereotypes is a well researched topic. They do that due to what they are. Stereotypes and bias in (in the training data), bias and stereotypes out. That’s what they’re meant to do. And all AI companies have entire departments to tune that, measure the biases and then fine-tune it to whatever they deem fit.

I mean the issue aren’t women or anything, it’s using AI for hiring in the first place. You do that if you want whatever stereotypes Anthropic and OpenAI gave to you.