Hacking the Antarctic

 

Technology is a big part of modern scientific exploration, as one IT expert found out when he embarked on the adventure of a lifetime.

 

Carles Pina i Estany is not what comes to mind when you picture your typical Polar explorer. A native from sunny Barcelona, he works as a Software Engineer at Mendeley – a London-based technology company owned by science publishers Elsevier – and before this year, he had never even slept aboard a ship. Nevertheless, when the invitation came for him to embark on a 3-month expedition around the Antarctic, he jumped at the chance.

Carles might not be your typical polar explorer, but he played a key part in the trip Click To Tweet

It all happened rather quickly; his partner Jen Thomas – who had previously worked with the British Antarctic Survey – was engaged as Data Manager for a research trip led by the newly created Swiss Polar Institute which was sponsored by the billionaire adventurer Frederik Paulsen.

The Akademik Tryoshnikov research vessel undertook an ambitious Antarctic loop Click To Tweet

Having an IT person on board (in addition to the two maintenance and electronics engineers) was essential and so Carles found himself alongside Jen on board the Akademik Tryoshnikov research vessel undertaking an ambitious Antarctic loop via Cape Town in South Africa, Hobart in Tasmania and Punta Arenas in Chile.

Having an IT person on board is essential these days Click To Tweet

This was about the time when things started to go wrong, as he recalls. They had bad weather, the food was poor, and the telecommunications setup didn’t work properly. And while some troubleshooting is to be expected on any such trip, the fact that they’d had such little time to prepare the equipment before setting off presented Carles with a unique set of challenges, or – as he optimistically puts it – “more opportunities for me to solve things.”

“I am 35 now, and I have been working with computers for over 20 years – hack days, evenings, weekends, personal projects – I tell you, in this expedition I used everything I know, even little bits of knowledge I thought were pretty useless, it all becomes useful in the Antarctic!”

When Pina i Estany’s unusual office time wrapped up this summer and he returned to London in late July, he found time to meet with Ars and regale our inner networking nerd with tales of IT at the high seas. If there’s anything to take away from his experiences, it’s that tech support in the most extreme places in the world isn’t so different from tech support in more familiar settings—that is, except for the lack of reliable connectivity and ability to snag parts and equipment from Amazon or your local shop. But the need for help is constant, the emotions can run high, and requests run the gamut from simple e-mail server stuff to things no computer science curriculum will train you for.

The winch

One evening, about a month after setting sail, a researcher cheerfully approached Pina i Estany with what he called a “new challenge.” Up until then, he dealt with the sort of familiar equipment that was well within his comfort zone: computers, routers, hard drives, and Raspberry Pis.

But this support ticket fell into an entirely different league. A massive winch—which is a hauling mechanism made of a cable and a crank—had started to fail.

Until this point, Pina i Estany didn’t even know what the word “winch” meant (that was about to change, naturally). This particular winch was responsible for lowering the ship’s only CTD device, which collected and analyzed water. The CTD normally would be lowered to depths of up to 1,500 meters, and it proved crucial to most of the 22 separate research teams onboard.

The problem, as it turned out, was a software malfunction. And the error was interfering with the winch’s ability to lower the extensive cabling into the water smoothly. Pina i Estany attempted some debugging, but the manufacturer soon told him that inputting new parameters on the CTD winch computer remotely was impossible. Considering the Akademik Tryoshnikov was in the middle of the ocean, this presented a bit of a problem.

The solution, as it turned out, called for some “semi-hacking skills” in addition to willingness to brave the elements:

“The temperature was around 0 or -2, windy with lots of sea spray; the boat was rocking, and my hands were freezing. People were asking me what I was doing out there with a computer, but the CTD was connected with a very short network cable. So I had to work outside,” he tells me. “So I accessed the winch computer, which operated Windows CE, using my Linux machine. I discovered the IP from the boot screen; using nmap, I found that it had a remote desktop server. I had a moment of joy when I pressed enter and, Yes! I could change the parameters.”

Pina i Estany’s joy was short-lived, sadly; that didn’t quite solve the problem. After the winch manufacturers begrudgingly agreed to let him reinstall the software, he had to wait until they docked at Hobart for resupply in order to download it through the hotel’s Wi-Fi.

“So we reinstalled the software, and it still wouldn’t boot up,” Pina i Estany said. “That was one of the worst moments of the expedition IT-wise. The problem was that the CTD was one of the most essential bits of science equipment onboard, but you know, this is a big winch, and I really had no idea about it… ”

Luckily, this particular story had a happy ending. After reinstalling the update in a different way, the winch finally stopped acting up. Soon enough, all the scientists got their water samples safely collected, and Pina i Estany could finally take his computer back out of the cold.

Hacked networking

In another memorable help request, one day a scientist approached Pina i Estany with a machine she used to measure the reflection of light on the sea. She needed to retrieve data from the device but could only do that by connecting it to a remote access point.

“So I was like, ‘yeah, no problem, where’s the router?’” he recalled. “It wasn’t on the ship at all—it turned out to be somewhere in Australia.”

The extremes of this Cape Town to Punta Arenas loop are easily 6,500-miles-plus away from Australia, and Pina i Estany recalls being roughly 2,000 miles away from South Africa at this time. Not that this stopped him, of course, as he used his laptop as a remote access point to connect to the equipment’s FTP server to get the data. “But that was a bit of a faff, since every time they needed more data, they had to get me and my computer,” he said.

Instead, Pina i Estany looked to a simple yet favorite device for network hackers—a smartphone.

“I figured out a way to hack together a network for her by using an Android phone,” he explains. “With Android, you can set up a hotspot, even with no signal, so the device can connect to their own laptop via this phone. That way, the scientists could retrieve their data whenever they wanted without me getting involved.”

Mail system

Try and cast your mind back—if you’re old enough—to the days of dial-up. You’d spend eons watching that progress bar as it downloaded a single e-mail or page. What was annoying back then is now maddening for those habituated to modern broadband speeds. But when you have a lot of people relying on an unreliable satellite Internet connection for sending and receiving all their e-mails, it’s a recipe for disaster. Things can quickly become tense.

“I have never seen so many people hitting their computers,” recalls Pina i Estany. “It was very painful for me. I can’t stand watching a scientist standing in front of a computer waiting for things to happen. It breaks my heart, seriously.”

Apart from the widespread frustration, a mounting e-mail backlog meant they couldn’t access some crucial expedition permits and documents. At one point, about 100MB of e-mails were stuck in the system, so Pina i Estany set out to retrieve them.

“It would have been easier to sail back to port to get them than to retrieve them at sea via Outlook,” he explains.

Of course, that kind of rerouting wasn’t an option. Instead, Pina i Estany accessed a remote server, downloaded and compressed all the e-mails to it, and then sent those compressed files to the ship using a piece of software called Rsync, which deals very well with unstable connections. He also wrote a script that meant if the program stopped downloading at any point, it would start again from the same place once a connection was re-established.

“So I left this program running for eight or nine hours and then opened this huge file using Thunderbird,” he said. “With that, I was able to get all the wanted e-mails, including the permits we needed.”

Still, the quick fix didn’t provide a solution to the main problem—the ship may have had these e-mails, but the expedition didn’t have a reliable communications system.

“I knew how to fix this, but I needed a proper Internet connection for a few hours to do it,” Pina i Estany said. “During our three-day break in port after leg one, Jen and I went to a hotel, where I set up a new website domain and webmail server, then created users for everyone. It was very popular. Everywhere I went on the ship, people were using the webmail I had deployed. That was amazing to see.”

For things to continue to run smoothly, however, Pina i Estany needed to limit the size of e-mails that could be sent over this system to 200kb, which meant setting up an alternative solution for sending larger files.

“I designed a system where the e-mails were downloaded in chunks,” he recalled. “It was a bit of a hack since e-mail protocols generally do not allow e-mails to be split—they download it all or not. I got around that so that if the download timed out at 20 percent, it tried again to download the rest of the e-mail rather than starting from zero.”

Pina i Estany also managed a queuing system to deal with the slow connection. With this in place, even though it might take five or 10 minutes for someone to retrieve their e-mail, users got a confirmation message straight away and knew that things were in hand.

“I was obsessed with being fair, so I was downloading e-mails in the order they were received,” he said.

The ferry box

Pina i Estany’s final anecdote involved something virtually every IT pro deals with these days: data management. Managing the vast amounts of data the expedition collected in real time was probably the most consistent (and huge) challenge that he applied his IT skills to, and the best example of this was the Ferry Box. This Linux-based machine remained in regular use, constantly measuring properties such as salinity and temperature of the water from the sea’s surface.

Because it took several days for each batch of collected data to be made available, however, scientists were left with little scope for forward planning. For stretches of time, the researchers could initially be unaware of important happenings, like crossing into different water bodies with distinctly different characteristics.

“So [to solve this], I figured out a way to feed that data into a website that the scientists could access in real time,” Pina i Estany said. “That way, they could make timely decisions about where to stop and take samples, for example.”

This functionality effectively meant that the researchers could shift from a reactive to a proactive approach in data collection. It’s not difficult to see how that would impact the science itself. And after a few tales from Pina i Estany, it’s not difficult to see how even the most extreme science relies on a humble IT pro, either.

This article was originally published on Ars Technica

Tales of an IT professional sailing around the Antarctic loop https://t.co/k7VyECE1wb

— Ars Technica (@arstechnica) August 22, 2017

Alice Bonasio is a VR Consultant and Tech Trends’ Editor in Chief. She also regularly writes for Fast Company, Ars Technica, Quartz, Wired and others. Connect with her on LinkedIn and follow @alicebonasio and @techtrends_tech on Twitter.