Back to @jack_bauer111's timeline
jack_bauer111 shared this post · 58m ago
YouTube
Alex Ziskind
Alex Ziskind

I Gave Local AI and the Cloud the Exact Same Job

I used my $60,000 512GB Mac Studio cluster running Kimi K3 to see if local AI can beat a $10/month cloud coding agent, and the result was way not what I expected.
🔹 Try out Supercomputer - https://supercomputer.abacus.ai/ltf

🛒 Gear Links 🛒

🪛🪛Highly rated precision driver kit: https://amzn.to/4fkMVfg
💻☕ Favorite 15" display with magnet: https://amzn.to/3zD1DhQ
🎧⚡ Great 40Gbps T4 enclosure: https://amzn.to/3JNwBGW
🛠️🚀 My nvme ssd: https://amzn.to/3YLEySo
📦🎮 My gear: https://www.amazon.com/shop/alexziskind

🎥 Related Videos 🎥

🧱🖥️ Mac Studio Cluster - https://youtu.be/25xVqvL5j4g
🧳🧰 Mini PC portable setup - https://youtu.be/4RYmsrarOSw
🍎💻 Dev setup on Mac - https://youtu.be/KiKUN4i1SeU
💸🧠 Cheap mini runs a 70B LLM 🤯 - https://youtu.be/xyKEQjUzfAk
🧪🔥 RAM torture test on Mac - https://youtu.be/l3zIwPgan7M
🍏⚡ FREE Local LLMs on Apple Silicon | FAST! - https://youtu.be/bp2eev21Qfo
🧠📉 REALITY vs Apple’s Memory Claims | vs RTX4090m - https://youtu.be/fdvzQAWXU7A
🧬🐍 Set up Conda - https://youtu.be/2Acht_5_HTo
⚡💥 Thunderbolt 5 BREAKS Apple’s Upcharge - https://youtu.be/nHqrvxcRc7o
🧠🚀 INSANE Machine Learning on Neural Engine - https://youtu.be/Y2FOUg_jo7k
🧱🖥️ Mac Mini Cluster - https://youtu.be/GBR6pHZ68Ho

My Silicon Exchange site prompt: https://gist.github.com/alexziskind1/fe55f03892f3fe1d6d8cf6065c631bb8

— — — — — — — — —

❤️ SUBSCRIBE TO MY YOUTUBE CHANNEL 📺
Click here to subscribe: https://www.youtube.com/@AZisk?sub_confirmation=1

— — — — — — — — —

Join this channel to get access to perks:
https://www.youtube.com/channel/UCajiMK_CY9icRhLepS8_3ug/join

— — — — — — — — —
📱 ALEX on X: https://x.com/digitalix

#macstudio #gdxspark #strixhalo

718 1.1M
@rikachiu Alternatively, I can use an enterprise Claude Max plan with contractual data-retention terms for roughly $2,000 a year. With the available discounts, that also gives me around $1,500 in monthly API credits. Using Fable 5, I can have the same project built, deployed to my custom sandbox, and configured with OAuth and the security controls I need in under 45 minutes.

When better models arrive, I can switch to them immediately, or use their APIs as needed at a 30% discount.

It would take roughly 30 years of subscriptions to equal the cost of that Mac Studio cluster.

Sure, the cluster is something I can say I own.

I’m pro–open source, but for enterprise work, it is currently too slow and doesn’t provide meaningful savings. For personal projects, though, it can make sense—if you have $60K+ to invest. 33 likes
@DejayClayton I'd really like to know what "enterprise" Claude Max plan you have access to for $2k a year, that also provides $18k a year in API credits. 8 likes
@rikachiu  @DejayClayton the beauty of being an NGO
@DejayClayton  @rikachiu  ok, so in other words, you’re in a situation that only applies to 0.0001% of people. Got it. 8 likes
@rikachiu  @DejayClayton  Maybe, but local AI is unlikely to ever going to match cutting-edge frontier models. One individual simply cannot compete with the scale of a full data center; especially once quantum computing and advanced world models become mainstream. It’s an unrealistic expectation and a power/compute restraint no matter how much we want it.

But, I am ready to be proven wrong. 1 like
@DejayClayton  @rikachiu I agree, but one always has to consider the cost when evaluating the value proposition of local AI vs hosted AI, and in many cases, the usage costs of hosted AI can be out of reach for some people. 1 like
@FlareBlossom  @rikachiu  what is an NGO? 1 like
@rikachiu  @FlareBlossom  A non-profit group that operates independently of any government to address social, environmental, and humanitarian issues

I rather keep the org I work for private. 1 like
@tomcanham9218  @DejayClayton  and there's something to be said for owning your own compute infrastructure, especially when cloud providers are doing so much fuckery. I'll keep going down the route I'm going down, where local/sovereign AI is the main driver, and cloud inference is an expensive add-on.

Also, everyone is still stuck in the "bigger is better" mindset. That's wrong. Sometimes bigger isn't just not better, but actively worse. 2 likes
@UnknownIllusionist  @rikachiu  the monthly plans are subsidized compared to the api rates. Chatgpt announced they are cutting $200 plan in half. Your assumption assumes the cheap cloud rates will continue to be cheap and local Ai software won't improve at the same time 3 likes
@rikachiu  @UnknownIllusionist  Sure, but you get more intelligence and cheaper per task completion. Don't forget you have to also spend on power to use anything locally. Local is still for pet personal projects, not to run an enterprise stack.

I also use local for mundane tasks that I can run 24/7, but no matter the hype, local just doesn't compare to bleeding edge frontier no matter how much people want it.
@mdzaid5925 My understanding is that smaller models with very high thinking mode performs better than bigger models on lower thinking level. 1 like