… Deutsche Telekom technicians are awake, monitoring networks and services. In the Service Management Center, their "navigation bridge," they have a clear view of all systems. The company's technicians work around the clock to ensure that everything runs just the way it should. I had the opportunity to accompany a colleague on her night shift.
10 p.m. – it's nighttime in Bonn. Deutsche Telekom's facility in Bonn's Beuel district is mostly dark – but not completely dark. A few windows are brightly illuminated. It's shift-change time for technicians at the Service Management Center (SMC).
I move through empty hallways as I go to meet the staff for the next shift. During the day, things are always abuzz here. You always hear footsteps and all kinds of conversations. Now, the place is strangely quiet. Suddenly, I'm standing in front of a camera-monitored security gate. It opens onto a high-security area.
Julia (name has been changed), an SMC team member, meets me at the gate. A short while later, as she's offering me coffee in the staff break room, she explains that she doesn't want her real name and her picture to appear on the Internet. For this piece, therefore, I'm giving her a pseudonym (Julia). She shows me around, and then we reach the very heart of the Center. It looks kind of like the bridge on a big ship. One entire wall is taken up by a huge LED monitor. "Huge" really does mean huge – its screen area measures about 120 square meters, or about the size of a roomy apartment. The staff in this room face it. A closer look shows that it actually consists of about 20 smaller screens – each of which is displaying numbers or images. One monitor is showing "Heute-Journal," a nightly news program on German TV's ZDF channel. The others are all displaying such things as curves, diagrams and tables full of measurement readouts.
Phone use in Germany – right now
On one screen, I see a real-time snapshot of telecommunications in Germany. It tells the experts how many customers are using LTE wireless services, for example. It also shows how many are using voice services and how many are currently watching TV on Entertain, the company's IP TV service. They can also see how many machines and vehicles are currently engaged in machine-to-machine (M2M) communications. – And how well gateways into other operators' mobile networks are operating. – And whether any overloads or failures are appearing throughout the widely branching networks. – And much more.
I glance at the news screen. It's showing images of a peat fire in Germany's Emsland region that is spewing miles of thick smoke. This kind of information is relevant, because remote accidents and disasters can affect facilities such as mobile stations, telephone exchanges and street cabinets. Whenever such impacts occur, the Center receives all of the available information, just as it does in connection with possible problems in hardware and software systems. And then it responds, by bringing in the necessary specialists, planning and organizing service calls and setting up any quick workarounds that might be needed. All such actions are carried out with the aim of restoring network service as quickly and completely as possible. "Ideally, when a problem occurs, large or small, customers will either hardly notice it or not notice it at all. That's always what we want," Julia explains.
That's what Julia, 31 years of age, works for on every shift, in her role as part of the Center's 100-member team. Her special area of expertise is the IP telephony that takes place over Germany's millions of IP lines. Now we're reached her office. It looks kind of like the "bridge" next door, except that it's much smaller. Here as well, a wall monitor shows measurement readouts. We can see exactly how many customers are making calls at the moment. Julia keeps an especially close watch on quiz shows that use telephone voting. Such voting can generate usage spikes that can seem shocking, because they can resemble the initial moments of a network failure. Fortunately, Deutsche Telekom's network can handle more than a hundred thousand calls over and above such load spikes. "I'm very impressed by how Deutsche Telekom has upped its data capacities over the past few years," enthuses Julia, who has now been with the Center for seven years. "In the "old" days, managing peak loads was often a challenge."
40 telephones that constantly call each other
Midnight is just minutes away. Julia has opened a number of systems on her PC. She clicks around, moving from one to the other, always checking rows of numbers. During routine handover procedures, she learned that some of her colleagues would be working on the access network tonight. Again and again, updates and improvements have to be carried out. As a precaution, the technicians do as much of such work at night as they can. Julia stays in constant contact with any technicians working on updates, etc., and she keeps a close watch on her monitor data.
Along with major measuring instruments, a number of smaller sources contribute to her data feed. For example, Deutsche Telekom has programmed 40 fixed-line phones, located in various different German cities, to call each other constantly. "That enables us to see what kind of connection quality we're getting." Another example is "allestörungen.de," a program, written by the Center's developers, that provides a geographic overview of customers' messages. When multiple messages originate in a single city, a network issue, and not a problem in a customer's home, is probably to blame. Julia pulls out a pack of photos. One shows cellular antennas coated with ice. Another presents a church encased in metal scaffolding that disrupted transmissions to and from the mobile communications antenna mounted on the church roof. As I'm holding that picture, she tells me about an antenna that got inadvertently "jammed" by an aluminum foundry. In sum, the pictures show examples of the kinds of things that can create problems – problems that have to be pinpointed.
Working hand-in-hand with robots
Julia then shows me computers with bots that her colleagues have programmed. She's obviously proud of her colleagues' skills. Tables and forms on the screens fill in automatically, as if by magic. The bots free the technicians from a great deal of routine tasks. One bot, for example, checks temperatures in multifunctional cabinets. Those are the gray cabinets situated along the sides of many roads. If any cabinet overheats, a bot immediately informs the company's field service.
And what happens when major alerts come in? A lot – and very fast. When Julia's involved, she brings colleagues at the Center together with "Second Level" staff – i.e. specialists for the pertinent network elements or systems. Those specialists always have "hotline" access to the relevant network-equipment providers and/or manufacturers. Such teams get formed within 15 minutes. To get to the root of a problem, they apply a process of elimination. For instance: How many customers are being affected? What customers are not being affected? What hardware and software systems are involved? What systems are not being affected? And so on. Together, they pinpoint the problem, coordinate the necessary local response and find temporary workarounds.
"I want to understand each and every fault"
Julia explains how the Center's staff cover many different areas of expertise. "We complement each other." She has good English skills, and she provides German summaries, for the team's wiki, of English-language manuals on new network elements. That way, in emergency situations, all team members can always look up the information they need about any affected parts.
As she speaks, I begin to sense just how profound her knowledge about Deutsche Telekom's network systems is. To begin with, for example, she has committed the entire network plan to memory, in fulfillment of a requirement for her certification as a supervisor. "And yet I never feel as if I've learned everything there is to learn. I always want to keep improving, and to be able to understand every problem that occurs," she says. She greatly enjoys sharing information with her colleagues. And she loves the (almost) family-like atmosphere that prevails at the Center. She has spent many a holiday – including holidays such as Christmas Eve and New Year's – working away at the Center, with her colleagues. She gets together with many of them in her free time. Often, such meetings are devoted to online gaming, a passion for many Center team members.
A child who loved to take things apart
How did Julia get into her line of work? "I always wanted to be involved with technology. As a child, I was always taking things apart, to see how they worked, she relates. When she was eleven, she set up her family's telephone and Internet access. Later on, she trained as an IT system electronics technician, and that led her to Deutsche Telekom. She has no problem with night shifts. On the contrary – she loves the extra vacation they earn her. Julia definitely feels that she is in the right place. "I simply love technology," she exclaims.
I don't stay until the 6 a.m. handover to the next shift. I'm too tired for that. After all, my workday started much earlier today than the time at which Julia's shift began. At 2.30 a.m., I call it a night. Yes, I am tired – but Julia's enthusiasm for technology has inspired me. And I have the assurance that the company's customers are in best hands at night.
All good things come in threes
Deutsche Telekom has three centers in Germany that monitor its networks and systems. The Bonn-based staff focus especially on platforms that serve customers directly, including mobile and fixed networks (along with their core networks), TV platforms and online services. The teams at the Network Management Center in Bamberg concentrate on transmission paths and the company's "IP backbone," i.e. the core network of the larger network. The backbone connects a range of smaller platforms and networks. The International Network Management Center in Frankfurt monitors Deutsche Telekom's global telephone and data network.
