Amazing article. It just goes to show how many problems they are solving just to work around bugs and limitations within Apple itself. I’d wager a guess that if Apple made a new Xserve with the same performance as a Mac mini but twice the price people would still buy them, because figuring out data center operations is expensive.
I feel like the ideal would be a 2U, 8 mac mini with dual 10Gb ethernet, shared dual power supplies, actual datacenter heatsink layout would be such a great CI platform.
I have no idea if this is true, but I would not be surprised if Apple has its own unreleased server variant based on the Mac Mini or Mac Studio platform that is deployed in data centers for specific clients.
Apple still lives in the world where software developed in IDE on a personal computer and shipped as a boxed software on a disquette with a licence key. Xcode cloud is a joke. All their automation is a joke. It’s not surprising even their own software development stuck because of their ecosystem because they use same tools, same cryptic software signing with their plists, p8, chains, certificates, lack of ui testing, horrible cpp, objc legacy. You just drowning in this environment. How I wish we have proper web app apis at last
In my experience this is only partially true. Apple has come a long way and their code signing experience is leaps and bounds better than Android at this point. Their UI testing is actually pretty solid, and Swift is a wonderful language to program in. The IDE part is partially true, but to be fair they are opening things up to coding agents (they're using Claude Code themselves) so I feel like this is only going to get better from here.
> their code signing experience is leaps and bounds better than Android at this point.
In which universe ? You generate a signing key, its password, configure the alias, the alias password and then just modify your gradle build script to say signingConfig.set(file("file/path.jks"))
Hell, if you want to go even further, give up on signing, use the debug signing certificate and let google sign for you with their own keys.
I've stopped counting how many issues you get on trying to build iOS apps because "whoops, sorry, xcodebuild just farted and can't sign"
I used to drown in this stuff in the when I first did IOS dev back in 2017. I recently got back into it and was pleasantly surprised at how quickly I could setup a ci/cd pipeline. What used to take hours figuring out certs, debugging build issues, I can now even test my build against prod without even deploying.
I'm not building anything complicated, just a iPhone / watch app, but when I do get stuck with env its codex to the rescue. IMO relative to my experience many years ago its amazing.
I assume you hit some limitations if you try to use Xcode Cloud for a larger project, but for small-scale use it’s pretty great. The setup is instant and seamless, builds are fast, and you get a bunch of minutes with your developer membership, which has been fixed at $99 for as long as I can remember.
Android tooling is cross-platform and easily executed in your choice of CI provider with cheap Linux ARM minutes. Simulator-based UI tests are headless (there are headless tests for iOS but not macOS). It's not even remotely a close comparison.
No, the apple ecosystem will always hold a monopoly over stupid upper class people to fleece. Apple will say "jump" and Developers will ask how high (if they know what's good for their wallet)
They might still be living in that world, but they also live in a world where they can do exactly $0 of R&D of actual rack server SKUs and still have people buying their high-end MBP SKUs at halfway-decent used car prices.
This is quite an amazing setup and shows the current limitations of Apple supporting their enterprise customers, which end up buying MacBooks to make their products available...
Dang, I hope they sell the older laptops for cheap (less than what they were at time of purchase obviously) on Ebay once they're phased out, wouldn't mind buying a "datacenter mac" if its still viable...
I could see them flooding the used Macbook market and selling like hot cakes, but also giving them back "rack" space.
What interests me is the miniaturization and thinness of the mainboard. Imagine if the mainboards were produced en masse and mated with a perfectly-fitting (liquid-cooled) heatspreader. That could be a very space-efficient blade server for very high density compute.
It is mind blowing to me that a 1 trillion dollar company don’t have a way to support large customers with enterprise servers. Fun fact: they used to have.
Yeah Apple needs to fix this by either making Xcode Cloud better, make more Mac minis (or just dedicated server hardware), or even loosening the two-VM limit. This kind of thing is bad for the average consumer and just plain wasteful.
They literally don't care. To them, either it's 1999 and your Macs should all be on desks at your physical office that you definitely have, or, you can "just" use Xcode Cloud[1], which is definitely fully featured and surely addresses the only possible 'headless' use case.
Plus, since the desperate are willing to do crazy-looking things like this, bulk-buying $6,000 premium laptops with batteries and screens, at enormous margins, Apple is even more incentivized to keep (not) doing exactly what it's (not) doing.
With that kind of airflow and consistent temperature I would suspect the failure rates to be incredibly small on those batteries - typically what kills them is extreme temperatures and high C charge/discharge at high temperature deltas (as measured from 25'C) which is where the cells have lowest degredation, which they should see very little of in this environment.
What would be more pertinent is changing the charging logic so the packs are held at 50% state of charge vs being always charged up to 100%, that will significantly decrease the chance of pack issues while greatly increasing the longevity of the battery itself. If you hold a lithium ion cell at 50% SOC for years without varying it and keep the temperatures flat, it can last for a very long time without capacity degredation. If you leave it at 100% it will be degrade in a few short years and can swell, at which point it has twice the chemical potential to flame out. Also if it's charged to 100% when the datacenter is at 20'C ambient and datacenter cooling fails and temps go up, having the pack warm up to 40'C can overvolt it and that is usually where they swell/fail.
Also fair to note here that Apple is probably at the frontier in terms of using the highest safety cells and battery management logic so I'd feel better about this than I would putting a lesser brand's lipo pouches in datacenter environment.
> With that kind of airflow and consistent temperature I would suspect ....
A lot of words, many of which might well be true. But it ignores the fundamental point.
I think we can safely assume its not their building, not their datacentre.
As a result, they need to play by the rules of the datacentre operator.
The datacentre operator has fire suppression systems build to a certain spec.
The datacentre has risk assessments and insurance covers to think about.
The datacentre has other customers to think about.
The datacentre has its own staff to think about.
Yes, datacentres have some of the most advanced multi-layer fire detection and suppression systems in the world.
But a fire caused by multiple racks stuffed full of LiPo batteries will likely result in very hard to control results, especially if it becomes a cascading event.
Yes, it is arguably highly unlikely to ever happen, especially with Apple hardware as you say.
However at the very least the guy should fess up and have a chat to the datacentre operator to at least give them a chance to say (preferably in writing) "ok, no problem, we'll just add it to our risk register".
I'm a little surprised that people would rather rent a Mac than buy one. Less so now that Apple raised the prices but for a long time a M4 Mac Mini was $599 which only works out to about 6 months at these prices.
Especially since they're not like a regular server, loud and power hungry. They're the definition of quiet and power sipping. I have one do all my background coding work when I want to keep my laptop clear (being headless helps with me moving around too).
the issue is the internal power supply is not redundant, even if you supply it with an ATS or UPS, that component can still die taking out the entire thing.
> the issue is the internal power supply is not redundant, even if you supply it with an ATS or UPS, that component can still die taking out the entire thing.
I mean, its basically the same thing in the MacBook, no ? The SMC ?
After all, there needs to be a magical chip inside to make sure you don't, for example, fry the laptop because it accepts charge via both USB-C and the MagSafe port.
Sure SMC failure is incredibly rare, but it doesn't mean the theoretical SPoF is not there.
Also when was the last time you had an internal PSU on a Mac fail ? Its been a very long time since I've ever seen one, and that's even thinking back to the old days with the old-school canister capacitors. Say what you like about Apple, but on the whole they do a decent job on their BOM selection.
Amazon also has much tighter integration and offers the Macs bare-metal. This has the downside that you can only gets Macs as dedicated hosts, so it's another layer to manage, and that it takes around an hour to wipe the Mac whenever you shut it down, even if you're not going to release the underlying dedicated host. Also for the last year, even grabbing a Mac dedicated hosts often runs into "None available" for hours at a time. Amazon tells you "try this other zone" but even that is no guarantee. I wasn't even able to get our TAMs to tell me whether trying other regions was likely to help. Besides that, it takes 5-10 minutes to launch a Mac, and once you acquire one, there's a 24 hour minimum billing period due to Apple licensing restrictions. Which is all to say, as a single customer, you can't really build an elastically scaled Mac CI/CD farm on top of AWS. You really need to size for your peak load. Which is expensive compared to just buying your own Macs, but $former_employer didn't want to be in the datacenter business.
FWIW, I believe MacStadium was the first to offer Macs in datacenters as a service:
> A well-known limitation is that Apple only allows two macOS VMs per physical Mac. It can be a Mac Pro with 24 cores and 192 GB memory, yet it is limited to running only two guest VMs. This limitation leaks into our product offering: we can only slice Mac resources in two halves. So we run instances with 6 or 12 cores on 14-core Mac Minis and 8/16-core instances on 18-core MacBooks (we reserve a couple cores for management overhead).
I don't know about selling, but Apple permits up to 2 virtualized Macs to run on a Mac. If they are circumventing this limit that would be an issue, but it doesn't sound like they are.
Ah okay - and the 24 hour minimum rental time doesn't apply to VMs? Im just curious how they are getting around the limits that Mac Stadium and AWS have.
They aren't getting around any limitations. How you decide to bill for meeting the limitations is a business decision, and you can do it via either higher average rates and allow more granular rentals to be financially covered that way or by requiring a 24 hour minimum.
I.e. just because Apple requires you to do things in 24 hour increments does not mean your billing has to expose it in exactly the same way.
If a business reserves a Mac for 24 hours (and allows no other users for that window), it complies with the EULA, no?
That doesn't mean the business has to charge the user for 24 hours. You could charge for 6 hours and then just let the Mac sit idle and unused the remaining 18.
You're eatting the cost of that unused time, but if its a differentiator that leads to more business than it might still be worth it.
EDIT: To be clear, imagine a case where the user has 24 hour sole access to the Mac. They could use up 24 hours if they wanted. But usage is tracked per hour, and they're only billed after the fact based on how much time they actually were running something during that 24 hour window.
I really wonder if Apple's logic to require a sale of a unit for 2 VMs is fiscally correct. Seems like it would be better to open it up to virtualization such that more devs can make iOS and macOS apps. Is the community of devs being forced to buy mac hw just do dev a iOS/macOs app so large that its worth putting us through all this HASSLE? Maybe they are worried about clone machine/farms that will eat into their sales. Idk. Because they are so VM unfriendly, I refuse to deal with Apple's bs.
Because the target market (mobile) is very profitable, if you have the skill, I also don’t understand that stance because Mac powerbooks, Mac minis, Mac studios last a long time and have high resale value if you decide to move on, particularly if you are a smaller development company.
I could see Mac minis being used for data center type setups. But MacBooks are distinctly consumer devices. I know your company is trying to solve a niche. But if it stops becoming a niche, consumers will be impacted because of operations like yours.
The article addresses that. In the very beginning.
> But still, why laptops? Isn’t it a major waste of screen pixels and keyboard buttons and battery ions? Well yes, Mac Mini would be a better choice for a datacenter. Sure, it has very rudimentary remote management, no redundant power, and a beautiful case that ensures that cooling will be difficult regardless of which side you lay it on. But these are all known challenges that we have learned to manage.
> But have you actually tried to order a couple hundred Minis? We have. Ordered in January, delivered in July. Welcome to the world of 2026 when the supply chains are in disarray. Never mind that Apple doesn’t release updated CPU models to all devices at once. So Mac Mini didn’t get M5 cores for 10 months (and then suddenly got M6 before any other device model!). We promise the best and the newest to our customers so we had to be inventive. That’s how the MacBook-in-Datacenter project was born.
xtool [0] came up on here a while ago [1]. I think it still requires you get a big xcode app bundle off a macos machine and drop that into xtool, since there's no other way to get the iOS library header files from Apple. The other parts of xcode just aren't used as far as I know. Other than that caveat, you're running builds in xtool's own environment on whatever platform you choose. You can even do code signing.
That's one of the reasons to keep it as far as possible from anywhere you need trust (like prod): software, like soylent green, is made of people. And you can't trust people like him.
"Luckily by now we found almost perfect cables that bend exactly as we want them to bend. And we bought the entire supply of them. There are zero more of them available to order on Amazon." Everyone is hoarding at any cost...
The cost of cloud Mac hardware is quite frankly ludicrous across the board. Namespace's minimum for an M6 is, what, $0.06/minute for 6 vCPU? And that's on the cheap end - Semaphore wants $0.09/minute for 4 vCPU and GitHub Actions wants $0.062/minute for M1 hardware (forget AWS pricing even which is even more atrocious).
If I run just a UI test suite for an hour, that's $3.60/run, which is now higher than what I'm paying for AI inference that made the change that triggered the run. That's crazy! If I ran even just one run per night that would run me more than $100/month. A Mac Mini is $900 - I break even in nine months by buying the hardware outright. If I run multiple UI test suites per day, I break even much faster. These ARM chips are so power-light that the cost of electricity is practically irrelevant, and after all, these aren't rack-mounted monstrosities with 90 dB fans - they're meant to cool themselves sitting on your desk. Maybe it's different if you need like 60 of them?
Is anyone even running the numbers before they sign up? Or are companies really running that many Minis and are paying the premium to outsource the DCOps?
I looked at our GitHub Actions monthly expenses, then at the price of a M4 Mac mini (just when they were released, and still cheap), and decided to buy one as it would pay for itself in less than a year. There are definitely still cases when buying hardware still makes sense, over renting in the cloud.
Mac Minis are small and power efficient. Scaling questions are always a question of degree. Several Mac Minis, Ethernet cabling, a KVM switch, maybe a simple MDM to make your life a little easier, scales just fine.
I have a test suite that needs to run on a Mac in a project that merges < 20 PRs a year. No point in provisioning our own Mac and maintaining it for that.
We have all the hosts enrolled into MDM, so we have all sorts of access ready: SSH, KVM, VNC, MDM. With Macs in the DC we found having all these ways to access the machines is really essential. Things break in unexpected ways.
In terms of settings, besides the obvous (do not sleep, do not auto-update, etc), disabling USB restricted mode helps a lot (Privacy&Security > Allow accessories to connect) - no need to approve connected devices via UI.
Also: which solution you have chosen for the p2p deployment? I used to manage deployments on a large scale cluster and got it down from 2h to <1m using libtorrent.
Yeah, as alluded to in the post, we have a custom p2p distribution, very inspired by BitTorrent. So you're on the right track :). We'll probably share more on that topic at some point too.
But honestly before we had that, we started with a system where nodes pulled whole images from each other (a decentralized algorithm without central coordination, but no chunking). It worked surprisingly well considering how simple the implementation was.
That stack is based on xtool at its core (xtool.io). It's very cool, but still early for steady enterprise use IMO.
Also it doesn't have iOS Simulator, so automated testing is out of question. I doubt that porting the Simulator would be feasible, since it requires macOS kernel.
Well, maybe with XNU in a VM and reimplementing parts of macOS userland. Would love to see someone try.
I was hopeful that they might lift the limit with a refresh of Mac Pro (it even had a rack mount version). But with Mac Pro discontinued not keeping the hopes up anymore.
The registry distribution subject "for another blog post" is the most interesting part of this to me. I'm very interested in the efficiency/locality metrics that went into designing that. I look forward to that future post.
Apple bros, why don’t you just make servers? People would go nuts for a 1TB RAM Mac server. You’re a hardware company! Why are you throwing away one of the biggest hardware markets!?!?!
I am lost, doing it in a Hackintosh is feasible but against EULA terms? I remember having a Hackintosh around the office two years ago (yes, it was x64 based). BTW I just found "ARM is not a universal architecture, there will be no more Hackintosh" [1].
Many people/companies are willing renters today even when the resale value is very good with Mac’s in comparison to PCs. Never have trouble selling on decently equipped Mac computers. The PCs, aside from the graphics card, end up in a scrap heap. (computer graveyard).
That's not a useful watermark, most of Apple's own offerings are nowhere near the iPhone revenue level.
In which universe ? You generate a signing key, its password, configure the alias, the alias password and then just modify your gradle build script to say signingConfig.set(file("file/path.jks"))
Hell, if you want to go even further, give up on signing, use the debug signing certificate and let google sign for you with their own keys.
I've stopped counting how many issues you get on trying to build iOS apps because "whoops, sorry, xcodebuild just farted and can't sign"
I'm not building anything complicated, just a iPhone / watch app, but when I do get stuck with env its codex to the rescue. IMO relative to my experience many years ago its amazing.
You mean APIs that inconsiderate developers use to release a simple calculator in the form of a 300+MB binary that eats a quarter of my RAM?
So I get why they don't care.
Also you can pull one out and work on it with the integrated keyboard, trackpad, and screen if it has problems.
Obviously it’s super inefficient to need to do this, but there are some upsides.
I could see them flooding the used Macbook market and selling like hot cakes, but also giving them back "rack" space.
Plus, since the desperate are willing to do crazy-looking things like this, bulk-buying $6,000 premium laptops with batteries and screens, at enormous margins, Apple is even more incentivized to keep (not) doing exactly what it's (not) doing.
[1] https://developer.apple.com/xcode-cloud/
What would be more pertinent is changing the charging logic so the packs are held at 50% state of charge vs being always charged up to 100%, that will significantly decrease the chance of pack issues while greatly increasing the longevity of the battery itself. If you hold a lithium ion cell at 50% SOC for years without varying it and keep the temperatures flat, it can last for a very long time without capacity degredation. If you leave it at 100% it will be degrade in a few short years and can swell, at which point it has twice the chemical potential to flame out. Also if it's charged to 100% when the datacenter is at 20'C ambient and datacenter cooling fails and temps go up, having the pack warm up to 40'C can overvolt it and that is usually where they swell/fail.
Also fair to note here that Apple is probably at the frontier in terms of using the highest safety cells and battery management logic so I'd feel better about this than I would putting a lesser brand's lipo pouches in datacenter environment.
A lot of words, many of which might well be true. But it ignores the fundamental point.
I think we can safely assume its not their building, not their datacentre.
As a result, they need to play by the rules of the datacentre operator.
The datacentre operator has fire suppression systems build to a certain spec.
The datacentre has risk assessments and insurance covers to think about.
The datacentre has other customers to think about.
The datacentre has its own staff to think about.
Yes, datacentres have some of the most advanced multi-layer fire detection and suppression systems in the world.
But a fire caused by multiple racks stuffed full of LiPo batteries will likely result in very hard to control results, especially if it becomes a cascading event.
Yes, it is arguably highly unlikely to ever happen, especially with Apple hardware as you say.
However at the very least the guy should fess up and have a chat to the datacentre operator to at least give them a chance to say (preferably in writing) "ok, no problem, we'll just add it to our risk register".
Its The Right Thing To Do (TM).
Especially since they're not like a regular server, loud and power hungry. They're the definition of quiet and power sipping. I have one do all my background coding work when I want to keep my laptop clear (being headless helps with me moving around too).
12 MacBooks per shelf, if we say roughly 5U per shelf (21.12/4.445 rounded up), so roughly 9 shelves per 47U rack, so 100-ish MacBooks per rack.
That's a scary density of LiPo battery.
The author complains about the lack of redundant power in Mac Minis, but there are other ways to get redundant power into one AC input.
I mean, its basically the same thing in the MacBook, no ? The SMC ?
After all, there needs to be a magical chip inside to make sure you don't, for example, fry the laptop because it accepts charge via both USB-C and the MagSafe port.
Sure SMC failure is incredibly rare, but it doesn't mean the theoretical SPoF is not there.
Also when was the last time you had an internal PSU on a Mac fail ? Its been a very long time since I've ever seen one, and that's even thinking back to the old days with the old-school canister capacitors. Say what you like about Apple, but on the whole they do a decent job on their BOM selection.
Amazon solves this by disabling FileVault and warns against enabling it.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-...
Amazon also has much tighter integration and offers the Macs bare-metal. This has the downside that you can only gets Macs as dedicated hosts, so it's another layer to manage, and that it takes around an hour to wipe the Mac whenever you shut it down, even if you're not going to release the underlying dedicated host. Also for the last year, even grabbing a Mac dedicated hosts often runs into "None available" for hours at a time. Amazon tells you "try this other zone" but even that is no guarantee. I wasn't even able to get our TAMs to tell me whether trying other regions was likely to help. Besides that, it takes 5-10 minutes to launch a Mac, and once you acquire one, there's a 24 hour minimum billing period due to Apple licensing restrictions. Which is all to say, as a single customer, you can't really build an elastically scaled Mac CI/CD farm on top of AWS. You really need to size for your peak load. Which is expensive compared to just buying your own Macs, but $former_employer didn't want to be in the datacenter business.
FWIW, I believe MacStadium was the first to offer Macs in datacenters as a service:
https://9to5mac.com/2020/12/27/macstadium-mac-mini/
I.e. just because Apple requires you to do things in 24 hour increments does not mean your billing has to expose it in exactly the same way.
That doesn't mean the business has to charge the user for 24 hours. You could charge for 6 hours and then just let the Mac sit idle and unused the remaining 18.
You're eatting the cost of that unused time, but if its a differentiator that leads to more business than it might still be worth it.
EDIT: To be clear, imagine a case where the user has 24 hour sole access to the Mac. They could use up 24 hours if they wanted. But usage is tracked per hour, and they're only billed after the fact based on how much time they actually were running something during that 24 hour window.
Nobody cared.
Uptime.lab (see Insta) deshells M2sand runs them bare a bit like GitHub does
Yes.
Why else would you buy such an expensive machine with such a crappy operating system?
Not so much these days.[0]
[0] https://news.ycombinator.com/item?id=50031653
For real. Also disconnects if you wave a palm hair in its general direction. So much worse than screw-in DVI, mechanical design wise.
> But still, why laptops? Isn’t it a major waste of screen pixels and keyboard buttons and battery ions? Well yes, Mac Mini would be a better choice for a datacenter. Sure, it has very rudimentary remote management, no redundant power, and a beautiful case that ensures that cooling will be difficult regardless of which side you lay it on. But these are all known challenges that we have learned to manage.
> But have you actually tried to order a couple hundred Minis? We have. Ordered in January, delivered in July. Welcome to the world of 2026 when the supply chains are in disarray. Never mind that Apple doesn’t release updated CPU models to all devices at once. So Mac Mini didn’t get M5 cores for 10 months (and then suddenly got M6 before any other device model!). We promise the best and the newest to our customers so we had to be inventive. That’s how the MacBook-in-Datacenter project was born.
I'm not sure this is true anymore.
[0] https://xtool.sh
[1] https://news.ycombinator.com/item?id=44011515
If I run just a UI test suite for an hour, that's $3.60/run, which is now higher than what I'm paying for AI inference that made the change that triggered the run. That's crazy! If I ran even just one run per night that would run me more than $100/month. A Mac Mini is $900 - I break even in nine months by buying the hardware outright. If I run multiple UI test suites per day, I break even much faster. These ARM chips are so power-light that the cost of electricity is practically irrelevant, and after all, these aren't rack-mounted monstrosities with 90 dB fans - they're meant to cool themselves sitting on your desk. Maybe it's different if you need like 60 of them?
Is anyone even running the numbers before they sign up? Or are companies really running that many Minis and are paying the premium to outsource the DCOps?
When you take resale value into account, purchase makes even more sense than renting.
Used M1 Mac Minis from 2020 are currently selling for about $500 when they originally retailed for $699.
The clue to their decision may be right there, potentially going for an increase in sales.
But nowadays it seems they suffer from a lack of vision and drive; perhaps (hopefully) they’ll come around.
With up to three tiers of cores (Super/Performance/Efficiency), how do you handle getting them split evenly to customers?
Could you please share how you setup the host Mac? Which pmset settings, defaults, tips and tricks for it?
We have all the hosts enrolled into MDM, so we have all sorts of access ready: SSH, KVM, VNC, MDM. With Macs in the DC we found having all these ways to access the machines is really essential. Things break in unexpected ways.
In terms of settings, besides the obvous (do not sleep, do not auto-update, etc), disabling USB restricted mode helps a lot (Privacy&Security > Allow accessories to connect) - no need to approve connected devices via UI.
What are you building if it's not a secret?
But honestly before we had that, we started with a system where nodes pulled whole images from each other (a decentralized algorithm without central coordination, but no chunking). It worked surprisingly well considering how simple the implementation was.
Also it doesn't have iOS Simulator, so automated testing is out of question. I doubt that porting the Simulator would be feasible, since it requires macOS kernel.
Well, maybe with XNU in a VM and reimplementing parts of macOS userland. Would love to see someone try.
I'm hoping AI will change that soon.
https://github.com/kholia/OSX-KVM
[1] https://www.reddit.com/r/hackintosh/comments/1uboij0/arm_is_...
TL:DR; They couldn't buy enough minis or studios.
Did you literally open the link, see a picture of Macbooks, and come directly back here before reading a single word?
People are outraged landlords doing that with housing.
Yet someone such rent seeking behaviour is acceptable?
Get fucked.