• 2 Posts
  • 6 Comments
Joined 10 months ago
cake
Cake day: March 22nd, 2024

help-circle

  • “Don’t feed the trolls” and defaulting to skepticism were part of the old internet. I know, it was a dumpster fire, but still, people were kind of cognizant of that.

    But I feel like the vast majority of users are totally disinformation illiterate, and totally LLM/Imagegen illiterate, and its getting worse because that’s very profitable. Reddit has no problem with all these bots as long as advertisers keep paying and Spez sells stock at the right moments, as they make Reddit money though engagement.


  • Unfortunately Nvidia is, by fair, the best choice for local LLM coder hosting, and there are basically two tiers:

    • Buy a used 3090, limit the clocks to like 1400 Mhz, and then host Qwen 2.5 coder 32B.

    • Buy a used 3060, host Arcee Medius 14B.

    Both these will expose an OpenAI endpoint.

    Run tabbyAPI instead of ollama, as it’s far faster and more vram efficient.

    You can use AMD, but the setup is more involved. The kernel has to be compatible with the rocm package, and you need a 7000 card and some extra hoops for TabbyAPI compatibility.

    Aside from that, an Arc B570 is not a terrible option for 14B coder models.