Since October 2010, all Stack Change websites have run on bodily {hardware} in a datacenter in New York Metropolis (nicely, New Jersey). These have had a heat spot in our historical past and our hearts. After I first joined the corporate and labored out of the NYC workplace, I noticed the unique server mounted on a wall with a laudatory plaque like a beloved pet. Through the years, we’ve shared glamor shots of our server racks and information about updating them.
For nearly our total 16-year existence, the SRE staff has managed all datacenter operations, together with the bodily servers, cabling, racking, changing failed disks and the whole lot else in between. This work required somebody to bodily present up on the datacenter and poke the machines.
We’ve since moved all our websites to the cloud. Our servers at the moment are cattle, not pets. No person goes to need to drive to our New Jersey information heart and exchange or reboot {hardware}. Not after final week.
That’s as a result of on July 2nd, in anticipation of the datacenter’s closure, we unracked all of the servers, unplugged all of the cables, and gave these as soon as mighty machines their ultimate curtain name. For the previous couple of years, we have now been planning to embrace the cloud and wholly transfer our infrastructure there. We moved Stack Overflow for Groups to Azure in 2023 and proved we might do it. Now we simply needed to sort out the general public websites (Stack Overflow and the Stack Change community), which is hosted on Google Cloud. Early final yr, our datacenter vendor in NJ determined to close down that location, and we would have liked to be out by July 2025.
Our different datacenter—in Colorado—was decommissioned in June. It was primarily for catastrophe restoration, which we didn’t want any extra. Stack Overflow now not has any bodily datacenters or places of work; we’re totally within the cloud and distant!
Main kudos to the SRE staff, together with many other people who helped make this a actuality. We’ll have just a few blogs quickly to speak about migrating the Stack Change websites to the cloud, however for now, benefit from the footage.
We had about 50 servers all collectively on this location. Right here’s what the servers regarded like at first of the day:
Eight (or extra) cables per machine multiplied by over 50 machines is lots of cables! Within the above image you possibly can see the massive mass of cables. Although they’re neatly packaged in somewhat cage (referred to as an “arm”), one per server, it was lots of work to de-cable so many hosts.
Why so many cables per machine? Right here’s a staged photograph that exhibits the person cables individually:
- Blue: 1x 1G ethernet cable for the administration community (distant entry).
- Black: 1x cable that takes the VGA video and USB (keyboard and mouse) indicators to a “KVM swap.” From the KVM swap we might connect with the keyboard/video/mouse of any machine within the datacenter. It was costly however value it. In an emergency, we might all the time “be in entrance of the machine” with out leaving our house.
- Pink: 2x 10G ethernet cables to the principle community.
- Black: 2x extra 10G ethernet cables to the principle community (solely on machines that wanted further bandwidth, resembling our SQL servers).
- White+blue: 2x energy cables (every to a distinct circuit, for redundancy).
The {hardware} nerds on the market ought to recognize these. However then got here time to disassemble them. Josh Zhang, our Workers Website Reliability Engineer, received somewhat wistful. “I put in the brand new internet tier servers just a few years in the past as a part of deliberate upgrades,” he mentioned. “It is bittersweet that I am the one deracking them additionally.” It’s the IT model of Outdated Yeller.
We assume that almost all datacenter turn-downs contain preserving sure machines to maneuver them to the brand new datacenter. Nonetheless, in our case, the entire machines had been being disposed of. This gave us the freedom of having the ability to transfer quick and break issues. If it was in our cage, it was going to the disposal firm. For safety causes (and to guard the PII of all our customers and prospects), the whole lot was being shredded and/or destroyed. Nothing was being saved. As Ellora Praharaj, our Director of Reliability Engineering, mentioned, “No have to be mild anymore.”
Clearing out a rack has two steps: First we de-cable all of the machines, then we un-rack them. Here is some racks being de-cabled. Something salvageable had been eliminated. Thus, we did not need to be neat, nor cautious. Right here you see racks in numerous phases of being de-cabled. After that the mass of cables had been lobbed over to the massive pile.
Ever have problem disconnecting an RJ45 cable? Properly, right here was our alternative to only reduce the rattling issues off as a substitute of determining why the little tab would not launch the plug.
The junk pile. Our de-cabling course of concerned throwing the whole lot into the nook of the room till we realized we could also be blocking our solely exit. Then we piled larger, not wider.
All of the servers and community units had been put in piles on the ground. Seven piles in whole.
Is that this the “earlier than” image from 2015-ish after we constructed all this or the “after” image after we decommed all of it? We’ll allow you to guess!
Large due to Ellora Praharaj, Tom Limoncelli, and Josh Zhang for the images and information. And for doing the heavy lifting.
