Jonathan Halsey (right), PD ALTESS Facilities Operations Engineer, checks the switchgear’s automatic load test function as Fred Porzio of PM DWTS looks on. (Photo by Stephen Larsen.)

A data center at the U.S. Army Product Director Acquisition, Logistics, and Technology Enterprise Systems and Services (PD ALTESS) facility that houses hundreds of servers hosting information systems supporting 155,000 users across the U.S. Army and DOD acquisition community now has highly reliable, redundant power that is fully automated and scalable for future growth. This is thanks to the state-of-the-art powerhouse plant provided by a team led by the Product Manager Defense Wide Transmission Systems (PM DWTS) and the U.S. Army Network Enterprise Technology Command/9th Army Signal Command (NETCOM/9th ASC).

According to Richard Eva, PD ALTESS, the upgrade was timely because the facility’s old power system, a mixture of manually operated 1950s vintage switchgear and three generators to which a substation and a fourth generator were added in the 1960s, was just about at the end of its life and had multiple single points of failure that could shut down the whole system. The system’s uninterrupted power supply (UPS) capacity was almost exhausted at 90 percent. “We had experienced 60 power outages in the last 3 years from our commercial utility power,” said Eva. David Wilson, PD ALTESS Chief of Operations and Security, added that a power failure occurred the day before they started the upgrade project. “We were sure that was the last time the old generator was going to start up again,” said Wilson.

Eva’s facilities engineers had to fabricate parts because the system was so old they couldn’t buy them any more. “The only reason it worked as long as it did,” said Eva, “was that Jonathan Halsey [PD ALTESS’ Facilities Operations Engineer] and his people had pushed the ‘Model T’ down the road as far as it would go.”

Add to all that the fact that PD ALTESS officials project their power requirements will grow about 100 percent in the next 5 years. “ALTESS is the information gateway for the Army and DOD acquisition community,” said Eva. “We have to keep the critical information systems we host up and running — period. We had to find a solution now.”

PD ALTESS officials project that their power requirements will grow about 100 percent in the next 5 years.

The solution presented itself when Halsey saw a story in Army AL&T Online(http://www.usaasc.info/alt_online/default.cfm?hdst=artl&iID=0605&aID=04&hsrc=toc) about how PM DWTS, part of Program Executive Office Enterprise Information Systems’ (PEO EIS’) Project Manager Defense Communications and Army Transmission Systems (PM DCATS), provided turnkey power solutions for two U.S. communications facilities in Japan. Halsey pointed this out to Eva, who directed the PD ALTESS staff to contact PM DWTS. They requested PM DWTS as the project manager to implement a comprehensive, multiphased power upgrade for the PD ALTESS facility, with phases implemented as funds became available.

Two Phases of Multiphase Project Completed
Fred Porzio, PM DWTS Project Leader, said that in phase one the team, which included American Power Conversion of West Kingston, RI, as the general contractor; Wells Global of Raleigh, NC, as installation subcontractor; and Randy White of NETCOM/9th ASC as the quality control engineer, provided a new 160 kilovolt-amp (kVA) UPS with a 1-hour battery backup, a maintenance bypass panel that allows for maintenance of the UPS while keeping the critical loads (such as servers and the Network Operations Security Center) online, and new critical distribution panels that meet National Electric Code standards and provide safety disconnects for the data room floor power distribution units. Phase one was completed in December 2006. “This [phase one] increased UPS capacity by 100 percent,” noted Halsey, “and created dual electrical paths to all critical information technology [IT] equipment, all the while improving the redundancy level of the data center.”

Shown left to right: Richard Eva, PD ALTESS, describes the operation of one of the new generators (in rear) in ALTESS’ new state-of-the-art powerhouse plant to Michael Bomba, NETCOM/9th ASC, as Fred Porzio, PM DWTS, and Jonathan Halsey, PD ALTESS Facilities Operations Engineer, stand by to provide support. (Photo by Stephen Larsen.)

In phase two, which started in September 2007 and was completed in February 2008, the team upgraded the facility’s power infrastructure. They replaced four antiquated generators — three 192 kilowatt (kW) and one 240 kW generators (a total of 816 kW) — with two new 500 kW generators (a total of 1,000 kW); replaced the existing 500 kVA and 750 kVA 208V substations (a total of 1,250 kVA) with one 1,500 kVA 480V substation; replaced manual switchgear with automated switchgear; and provided a new secondary distribution system, including 1,600A, 208V, and 480V secondary switchboard sections and a 500 kVA transformer.

The power infrastructure upgrade has provided sufficient capacity for 2,000 to 3,000 additional servers to be added within the ALTESS data center. “This new switchgear is completely automated and provides seamless transfer of power,” said Porzio. “It starts the engines, synchronizes the generators, and brings the data center online in 15 seconds.” White added that the project’s most significant aspects are the increase in power and the vast increase in reliability afforded by the modern equipment. Halsey is impressed with the switchgear’s automatic load test function, which allows PD ALTESS facilities engineers to test the generators under load without disrupting power to the facility.

Secure and Stable Environment for Critical Applications
PD ALTESS hosted a ribbon-cutting ceremony for the new power infrastructure equipment on Feb. 7, 2008, with the ribbon being cut by Lee Harvey, Deputy PEO EIS; COL Steve Bristow, Deputy PEO EIS; COL Quentin Peach, PEO EIS Project Director, Acquisition Domain; COL Gale Harrington, PM DCATS; and LTC Clyde Richards, PM DWTS.

At the ceremony, Eva thanked the team members for their work on the project. “Thanks to this project, PD ALTESS has taken a major step forward in its ability to support the U.S. Army and DOD,” he said. “Your efforts have given us the robust, resilient power we require to provide our clients a secure and stable environment for their critical applications.”

The project’s most significant aspects are the increase in power and the vast increase in reliability afforded by the modern equipment.

Eva presented certificates to members of the installation team and PD ALTESS’ emergency generator on-call team, who made themselves available nights and weekends to augment the facilities engineering staff in helping to restore power after failures. He also presented the Department of the Army Commander’s Award for Civilian Service to Porzio. “Mr. Porzio spent innumerable hours coordinating contractor manpower and equipment delivery, scheduling events, and reacting to short-notice and no-notice situations,” said Eva. “Even though he was frequently deployed to worldwide locations, he always made himself available, day or night, to answer our questions and address our concerns. Once the project started, I slept better more nights — because I knew Fred didn’t.” Richards echoed Eva’s thought, adding, “Fred is simply amazing. He has carved out a niche for DWTS from one critical power project to the next. We have experienced a growing demand for this type of work, which essentially has become a core competency.”

In addition to the critical power projects for PD ALTESS and in Japan, PM DWTS has also provided critical power infrastructure upgrades at command posts and camps in Korea to upgrade electrical distribution; heating, ventilating, and air conditioning (HVAC); and backup power systems. “Our power upgrades typically provide automation, redundancy, increased reliability, logistical supportability, and lower maintenance costs,” said Richards. “I’m happy that here at ALTESS we were able to help out a fellow PM within our PEO. It truly demonstrated the benefits of horizontal integration.”

Cutting the ribbon for the PD ALTESS new power infrastructure equipment on Feb. 7, 2008, are (left to right): COL Gale Harrington, PM DCATS; LTC Clyde Richards, PM DWTS; Lee Harvey, Deputy PEO EIS; COL Quentin Peach, PEO EIS Project Director, Acquisition Domain; and COL Steve Bristow, Deputy PEO EIS. (Photo by Stephen Larsen.)

The Future: Tier III Certification
ALTESS’ future growth could start very soon, as phase three of the project is well underway. Phase three started in January 2008 with an expected completion date of July 2008. This phase includes modernizing the HVAC system, installing chilled water towers, and starting the infrastructure and data rack build-out of rooms with Secret Internet Protocol Router Network (SIPRNET) equipment, including installing in-row cooling units between the data racks. This phase of the upgrade will provide ALTESS with the capability to house hundreds of additional SIPRNET servers.

Eva has plans for additional phases as funds become available. A possible phase four is slated to include completing the build-out of the SIPRNET rooms and installing in-row cooling units throughout the facility. A possible phase five could include filling out the PD ALTESS facility with 200 more data racks for servers and adding more generators, an additional 3,000 kVA electrical distribution, and an 850-ton chiller plant.

“Our own end-state goal is to move toward Tier III certification,” said Eva, referring to the 4-tier rating system of site infrastructure functionality that was developed by the Uptime Institute to rank data centers according to a common standard. Concurrent maintenance is a big part of the Tier III certification. For server hardware, it means power supplies or upload software can be replaced without turning it off. For infrastructure, it means any component of power distribution or cooling systems can shut down and have a redundant system working, uninterrupted by the maintenance, which ensures the availability of the IT systems to the end user.

Thanks to the highly reliable, redundant power infrastructure provided by PM DWTS, PD ALTESS is one step closer to that goal.

 

STEPHEN LARSEN is the PEO EIS Public Affairs Officer at Fort Monmouth, NJ. He has more than 20 years of experience writing about Army systems. He holds a B.A. in American studies from the College of Staten Island of the City University of New York.