Apple’s big AI week ends with Mac + Muse AI supercomputers

17/10 07:15 - Apple’s big AI week ends with Mac + Muse AI supercomputers
Welcome to the week Apple became even more serious about artificial intelligence on Macs, with the company continuing to make moves to deliver the tech in the most cost effective, responsible way. While it’s true that today isiPhone Duo pre-order day, that’s arguably not really the big news. It is not the big news because Apple this week made huge steps in AI. Not only has it come up with a nifty tool to turn Macs into distributed inferencing data centers; not only does it now have a plan for Apple Intelligence in China; but it has also worked with Meta to make it possible to run on-premises instances of Muse Spark Open on a couple of Mac Studios. While working on that, its machine learning team also introduced a new version of MLX that enables AI researchers and developers to run large language models (LLMs) locally on Mac Studio with Apple silicon. What’s new to MLX from Apple?In case you’d missed it, MLX leverages the unified memory architecture of Apple silicon, which is invaluable if you want to build new inference and fine-tuning methods. Over the past year, the ML team has worked on “several key innovations” to make it easier and more efficient to “run state-of-the-art LLMs locally on recent Apple silicon.” These improvements include:Software support for the Neural Accelerators in the GPU of the M5 family of chips, Low-latency communication across several Macs using Remote Direct Memory Access (RDMA) over Thunderbolt.Efficient implementations of recent popular architectures such as mixture of experts (MoE). The release leans heavily into what Apple’s teams have been doing with Muse, and once again demonstrates how the very architecture of Apple silicon supports AI inferencing. That’s no surprise to anybody at this point as not only have I been banging on that drum regularly, but almost anyone who is anyone in AI development and research already works on a Mac.LLM Inference with MLX and RDMA on Mac StudioA new white paper from Apple’s machine learning team shares what it has been doing with MLX as well as significant statistics concerning how well its systems support Muse Spark Open. “On a cluster of four Mac Studios, Muse Spark Open generates over 36 tokens per second, well above average human reading speed,” they said. That’s impressive. It means for the cost of four Mac Studios a business can run a full-strength, frontier class model with almost a trillion parameters, on-premises, securing company data while extracting whatever benefits there are to be extracted from the use of it.Mac Meta mattersApple and Meta collaborated on making it possible to run Muse Spark Open AI models on Macs. It’s a shrewd move for both companies, given the growing desire to run powerful AI models locally rather than in the cloud. As a result of this work the collaboration has made it possible to use Muse Spark Open on two or four Mac Studios to engage in all kinds of agentic tasks. Commenting on this on X, new Apple CEO, John Ternus wrote:  “Amazing to run a model of Muse Spark Open’s caliber and scale this efficiently, right from your desk. Mac Studio keeps pushing the boundaries of on-device AI!” Meta’s AI chief, Alexandr Wang pointed out that this is in part made possible by Apple’s Unified Memory architecture, which can provide sufficient memory to run these models.Your Macs. Your supercomputer“Apple’s unified memory architecture lets the M5 Ultra's integrated GPU use up to 512 GB per Mac Studio, so two machines can hold an entire model,” Meta explained. “And the chip’s 1.2 TB/s of memory bandwidth is fast enough to read the small slice each token needs. The result is two Mac Studios behave like one supercomputer with a terabyte of unified memory.”Meta says it tested the system using agentic workflows running entirely on-device, including applying bug fixes and new features to an open source Mastodon client and driving DaVinci Resolve to review footage and transact tasks. The use of DaVinci Resolve makes this commentator curious, given Apple is expected to introduce new pro Macs this month. The idea that you may be able to run tasks like this on other Macs will surely be something Apple’s developers are thinking about.One more thingStep by step Apple is defining the playground for AI. In doing so it is attempting, as Apple’s VP Worldwide Marketing told TIME, not to make AI for “the lunatic fringe”, but to create an ecosystem for artificial intelligence that will actually help people — a strategic approach that, if it comes together, will help maintain Apple’s position at the intersection of computing and the liberal arts. Read my Mac Studio review here.Now please subscribe to my daily, human-curated Apple-related news headline feed at The Core, or follow me on BlueSky, LinkedIn, or Mastodon.Welcome to the week Apple showed us how serious it really is about artificial intelligence on Macs, with the company making more moves to deliver the tech in the most cost effective, responsible way. While it’s true that today is iPhone Duo pre-order day, that’s arguably not the big news.  It is not the big news because Apple this week made huge steps in AI. Not only have its researchers come up with a nifty tool to turn Macs into distributed inferencing data centers, not only does the company now have a plan for Apple Intelligence in China, but it has also worked with Meta to make it possible to run on-premises instances of Muse Spark Open on a couple of Mac Studios. Its machine learning team also introduced a new version of MLX that enables AI researchers and developers to run large language models (LLMs) locally on Mac Studio with Apple silicon.  What’s new to MLX from Apple? In case you’d missed it, MLX leverages the unified memory architecture of Apple silicon, which is invaluable if you want to build new inference and fine-tuning methods. Over the past year, the ML team has worked on “several key innovations” to make it easier and more efficient to “run state-of-the-art LLMs locally on recent Apple silicon.” These improvements include: Software support for the Neural Accelerators in the GPU of the M5 family of chips Low-latency communication across several Macs using Remote Direct Memory Access (RDMA) over Thunderbolt Efficient implementations of recent popular architectures such as mixture of experts (MoE) The release leans heavily into what Apple’s teams have been doing with Muse, and once again demonstrates how the very architecture of Apple silicon supports AI inferencing. That’s no surprise to anybody at this point: not only have I been banging on that drum regularly, but almost anyone who is anyone in AI development and research already works on a Mac. LLM inference with MLX on Mac Studio A new white paper from Apple’s machine learning team shares what it has been doing with MLX, along with significant statistics concerning how well its systems support Muse Spark Open, Meta’s new open-weight frontier model.  “On a cluster of four Mac Studios, Muse Spark Open generates over 36 tokens per second, well above average human reading speed,” they said.  That’s impressive. It means for the cost of four Mac Studios, a business can run a full-strength, frontier-class model with almost a trillion parameters, on-premises, securing company data while extracting whatever benefits there are to be extracted from the use of it. Mac Meta matters Apple and Meta collaborated on making it possible to run Muse Spark Open AI models on Macs. It’s a shrewd move for both companies, given the growing desire to run powerful AI models locally rather than in the cloud. As a result of this work, it’s possible to use Muse Spark Open on two or four Mac Studios to engage in all kinds of agentic tasks.  Commenting on this on X, new Apple CEO John Ternus wrote: “Amazing to run a model of Muse Spark Open’s caliber and scale this efficiently, right from your desk. Mac Studio keeps pushing the boundaries of on-device AI!”  Meta’s AI chief Alexandr Wang pointed out that this is in part made possible by Apple’s Unified Memory architecture, which can provide sufficient memory to run these models. Your Macs. Your supercomputer “Apple’s unified memory architecture lets the M5 Ultra’s integrated GPU use up to 512 GB per Mac Studio, so two machines can hold an entire model,” Meta explained. “And the chip’s 1.2 TB/s of memory bandwidth is fast enough to read the small slice each token needs. The result is two Mac Studios behave like one supercomputer with a terabyte of unified memory.” Meta says it tested the system using agentic workflows running entirely on-device, including applying bug fixes and new features to an open-source Mastodon client and driving DaVinci Resolve to review footage and transact tasks. The use of DaVinci Resolve makes this commentator curious, given that Apple is expected to introduce new pro Macs this month. The idea that you may be able to run tasks like this on other Macs will surely be something Apple’s developers are thinking about. One more thing Step by step, Apple is defining the playground for AI. In doing so it is attempting, as Greg Joswiak, Apple’s senior VP of worldwide marketing, told TIME, not to make AI for “the lunatic fringe,” but to create an ecosystem for artificial intelligence that will actually help people — a strategic approach that, if it comes together, will help maintain Apple’s position at the intersection of computing and the liberal arts. (Read my Mac Studio review here.) Now please subscribe to my daily, human-curated Apple-related news headline feed at The Core, or follow me on BlueSky, LinkedIn, or Mastodon. ...