Musk Hardcore Migration Server

There is a story in Musk’s biography. After Musk took over Twitter at the end of last year, in order to save money, he wanted to move the servers from Sacramento to Portland. The IT manager reported to Musk that it would take 9 months. Musk felt that this time was too long. , it could be done in a month, but the IT manager insisted on not compromising, so he personally went to the computer room to unplug the power and rent a truck to move the server.

"Elon Musk personally moves servers, demonstrating his "fanatical sense of urgency" at Company X (formerly Twitter)"

*This content is adapted from Walter Isaacson's biography "Elon Musk" published this month.

"Does this look like a timeline that I would consider marginally acceptable?" Musk asked. "Obviously not. It would be wrong to plan for a long time."

It was late at night on December 22, in the conference room on the 10th floor of Company X, and Musk was having a tense conversation with two infrastructure managers. They hadn't had much contact with him before, especially when he was in a bad mood.

One of them tried to explain the problem. A data services company in Sacramento allowed Company X to extend its server lease to allow for an orderly move out in 2023. “But this morning,” the nervous manager told Musk, “they came back and told us that this plan no longer works because they don’t think we’re financially sound anymore.”

This facility costs Company X over $100 million per year. Musk wanted to save this cost by moving the servers to Company X's other facilities in Portland, Oregon. Another manager said the work couldn't be done immediately. "We're going to need at least six to nine months because Sacramento still needs service traffic," she said quietly.

Over the years, Musk has often faced a choice between what he thinks is necessary and what others tell him is possible. The results are almost always the same. He was silent for a few seconds and then announced: "You have 90 days to complete this task. If you can't do it, you can resign."

The manager began to explain in detail the obstacles to migrating servers to Portland. "The rack density is different, the power density is different," she said. "So the computer room needs to be upgraded." She began to elaborate more, but was interrupted by Musk.

"It weighs on my brain," he said.

"I'm sorry, I didn't mean it," she replied evenly.

"You know the exploding head emoji?" he asked her. "That's what my head is feeling right now. What a load of bullshit. There's obviously tons of space in Portland, and moving from one place to another is a piece of cake."

When Company X managers tried to explain again, Musk interrupted them. "Can you go to our server center and send me a video?" he asked.

It was three days before Christmas and the manager promised the video would be available a week later. "No, tomorrow," Musk ordered. "I've built server centers myself, and I can tell if there's room for more servers. That's why I'm asking you if you've actually been to those facilities. If you haven't been, you're talking nonsense."

SpaceX and Tesla succeed because Musk relentlessly pushes his teams to be more agile, creative, and launch sprints to tackle any obstacles. That's how they quickly assembled a car production line in Fremont, set up test facilities in the Texas desert, and built a launch site at Cape Canaveral using old parts.

"All you need to do is move your servers to Portland," he said. "If it's more than 30 days, I'll be shocked." He paused, recalculating. "Get a moving company, it'll take a week to transport the computers, and then another week to hook them up. Two weeks. That's how it should be."

Everyone was silent. But Musk is still fuming. "If you rented a U-Haul (a car rental company), you could probably do it yourself." The two Company X managers looked at him, trying to decide if he was serious. Two of Musk's close aides, Steve Davis and Omead Afshar, were also present. They'd seen him like this enough times to know that he probably actually thought that.

"Why don't we do it now?" James Musk suggested.

James and his brother Andrew, Musk's younger cousin, flew with him from San Francisco to Austin on December 23, the day after that frustrating infrastructure meeting about how to move servers. They had planned to spend Christmas in Tahoe, but Elon invited them to Austin that day.

James was a bit hesitant, he was exhausted and didn't want to add to the pressure, but Andrew convinced them they should go. So they ended up sitting on a plane listening to Elon complain about servers.

As the plane flew over Las Vegas, James made a suggestion that they move the server now. This is the kind of impulsive, unrealistic advice Musk loves. Even though it was late at night, he turned the pilot around and they headed back toward Sacramento.

The only car they could rent after landing was a Toyota Corolla. They weren't sure how they got into the data center at night, but an X employee named Alex from Uzbekistan was still there. He readily let them in and gave them a tour.

The facility, which also provides server storage space for many other companies, is so secure that a retina scan is required to enter each vault. Alex from Uzbekistan helped them gain access to Company

"These things don't look hard to move," Elon declared. This is a distortion of reality, since each rack weighs approximately 2,500 pounds and is 8 feet tall.

“You’re going to need to hire a contractor to lift the floor panels,” Alex said. "They need to be lifted with suction cups." He said another team of contractors will be needed to remove the cables and seismic poles.

Musk turned to his security guard and borrowed a pocket knife. Using it, he was able to lift a vent in the floor, which allowed him to pry open the floor panel. Then he crawled under the server, used a knife to pry open an electrical box, unplugged the server, and waited to see what would happen. Nothing unusual happened. The server is ready for migration.

"It doesn't look that hard," Alex and the others stared at him, dumbfounded, when he said this. Musk was completely excited at this moment. He laughed and said it was like the Sacramento version of Mission: Impossible.

The next day, Christmas Eve, Musk called in reinforcements. Ross Nordeen, who drove from San Francisco with his friend James, who works at Tesla. He spent $2,000 at the Apple Store in Union Square to buy all the AirTags so the servers could be tracked during the migration. Then he went to Home Depot and spent $2,500 on wrenches, bolt cutters, a headlight and the tools he needed to unscrew the seismic bolts.

Steve Davis, Musk's loyal lieutenant, rented a semi and arranged for a moving truck. Other assistance teams from SpaceX have also arrived. The server racks all had wheels, so the team was able to disconnect four of them and roll them to a standby truck. This suggests that all 5,200+ servers could be moved within days. "Good job guys!" Musk said cheerfully.

Other staff at the facility looked on with a mixture of wonder and horror. Musk and his team pushed the servers without boxing them or wrapping them in protective material, then used store-bought straps to secure them to the truck. “I’ve never loaded a semi,” James admits. Ross called it "scary." It's like cleaning out a closet, "but what's inside is vital."

At 3 p.m., as they loaded the four servers onto the truck, the news reached executives at NTT, which owns and manages the data center. They ordered Musk's team to cease operations. Musk's expression was a mixture of joy and anger, emotions that often accompany his fervent aggressiveness. He called the CEO of the storage division, who told him that it was impossible to move the server racks without experts. "Bullshit," Musk countered. “We’ve already loaded four of them onto the semitrailer.”

The CEO then told him that some floors could not withstand more than 500 pounds of pressure, so rolling a 2,000-pound server would cause damage. Musk responded that the server has four wheels, so the pressure at any one point is only 500 pounds. “This guy is really bad at math,” Musk told his teammates.

After ruining NTT management's Christmas Eve and costing them potentially more than $100 million in revenue next year, Musk expressed sympathy and said he would pause server migrations for two days. But he warned they would continue after Christmas.

After Christmas, Andrew and James returned to Sacramento to see how many more servers they could move. They didn't bring enough clothes, so they went to Walmart and bought jeans and T-shirts.

NTT wants the moving contractor they use to charge $200 an hour. So James found a company on Yelp called Extra Care Movers that would cost a tenth of the price. This seemingly messy company takes the low-cost concept to the extreme. The boss had been living on the streets, had a child, and was trying to get back on his feet. He didn't have a bank account, so James paid him using PayPal.

The next day, the work team wanted to pay in cash, so James went to the bank and withdrew $13,000 from his personal account. Two team members had no identification, making it difficult for them to enter the facility. But they make up for it with diligence. "You can get a dollar tip for each additional server you move," James announced at one point. Since then, whenever they load new servers onto trucks, workers ask how many they have now.

There was user data on the server, and James didn't initially realize that for privacy reasons it should be wiped clean before migration. "By the time we knew this, the server had already been unplugged and pushed out, so there was no way we could push it back in, plug it in, and wipe it again," he said. Also, the erasing software doesn't work. "Fuck, what are we going to do?" he asked. Elon suggests they lock the truck and track it.

So James sent someone to Home Depot to buy some large padlocks, and they sent the combination code to Portland so the truck could be opened there. “I couldn’t believe it worked,” James said. "They all made it to Portland safely."

By the end of the week, they had used every available truck in Sacramento. Although the area was hit by rain, they moved more than 700 racks in three days. The facility's previous record was 30 moves a month. This still leaves a large number of servers in the facility, but this group has proven that they can be moved quickly. The remainder was handled by Company X's infrastructure team in January.

This all sounds exciting and inspiring, right? This is an example of Musk’s bold and creative approach! But like all things Musk, it's not that simple. It's also an example of his recklessness, his impatience with rebuttal and his ability to intimidate people. A week ago, Company He has a great track record of ignoring naysayers. But it's not perfect.

Over the next two months, Company X was unstable. The lack of servers led to multiple system crashes, including when Musk hosted Twitter Spaces for presidential candidate Ron DeSantis. By March 2023, Musk would admit: "Looking back, the whole Sacramento shutdown was a mistake. I was told we had redundancy in our data centers. What I didn't know was that we had 70,000 hard-coded references to Sacramento. The problems that resulted from that still exist today."

At Tesla and SpaceX, his most valuable assistants had learned how to ward off his bad ideas and trick him into gradually revealing unwelcome information, but Company X's older employees didn't know what to do with him. In other words, Company X is still alive. The incident in Sacramento showed Company X employees that he meant it when he talked about the need for a feverish sense of urgency.

Walter Isaacson is a CNBC contributor who has also written biographies of Elon Musk, Jennifer Doudna, Leonardo da Vinci, Steve Jobs, Albert Einstein, Benjamin Franklin and Henry Kissinger. He teaches history at Tulane University and was the editor of Time magazine and the CEO of CNN.

Related Links

Guess you like

Origin www.oschina.net/news/259436/elon-musk-moved-twitter-servers-himself