Once an institution is connected to the California Research and Education Network (CalREN), CENIC’s Network Operations Center (NOC) is the first point of contact for any issues or questions that arise. NOC engineers monitor the network 24 hours a day, 7 days a week, 365 days a year. CENIC members who experience a problem notify the NOC and are immediately connected with an experienced, highly trained network engineer who records the issue in the NOC’s ticketing system and begins the troubleshooting process. Engineers resolve most issues, and fewer than 1% of incoming tickets are escalated to a higher level.
Ju Kim, senior manager of network operations at UCLA and chair of one of CENIC’s technical advisory councils, noted, “The NOC performs proactive analysis on maintaining the CalREN network around the clock, and is doing a good job of recognizing, isolating, and resolving faults. When a problem is identified, maintenance is initiated and communicated to its member institutions in a timely and appropriate manner, to ensure minimal interruption to production. I appreciate the NOC’s participation in TAC meetings and the way it provides the team with information on updates, progress, and changes to the CENIC network.”
The NOC operates in an open work space, and staff are often seen huddled around a whiteboard or overheard on a conference call, collaborating on potential solutions.
NOC engineers also diagnose and resolve problems identified by network monitoring tools, which generate alarms when issues are detected. Engineers often identify problems before the users themselves, and work to quickly resolve them.
The ticketing system tracks issues until they are resolved. Many can be resolved on the same day, while some are more complex and take longer. In 2016, the NOC processed 19,803 new tickets and assessed 85,410 alarms.
NOC engineers also coordinate and communicate with third-party maintenance and repair activities, perform equipment updates, replacements, and upgrades, decommission retired equipment, and handle tasks in support of circuit installations, service upgrades, and backbone upgrades. In 2016, the NOC performed 1,516 maintenance procedures, operationalized 196 new devices, and deployed and operationalized 289 new circuits.
Michael Kilpatrick, technology program director at the University of California Health System, said, “The CENIC network operations team has consistently been excellent for UC Health, delivering great value. Each of its engineers demonstrates a rapid response culture, an eagerness to engage, and a high level of professionalism and customer service.”
CENIC’s network engineers continuously strive to improve processes and documentation, and actively share technical knowledge and experience within the team. In addition to face-to-face interaction, internal mailing lists allow members to receive maintenance and outage announcements as well as resolution notices.
Network operations are core to CENIC’s mission to enable researchers and educators to leverage reliable, high-speed broadband in their work and maximize the best possible outcomes. Staff experts at the NOC are standing by to help network users whenever issues arise. CENIC members can expect a sympathetic and timely response to their concerns.
Engineers are available to assist CENIC members 24 hours per day, 7 days per week, including holidays. The CENIC NOC can be reached at: 714-220-3494 (714-220-FIXIT) and noc@cenic.org. Learn more.
Related blog posts
