{"id":5619,"date":"2019-02-19T13:57:55","date_gmt":"2019-02-19T18:57:55","guid":{"rendered":"https:\/\/blogs.swarthmore.edu\/its\/?p=5619"},"modified":"2019-02-19T14:12:32","modified_gmt":"2019-02-19T19:12:32","slug":"status-report-on-fridays-campus-network-outage","status":"publish","type":"post","link":"https:\/\/blogs.swarthmore.edu\/its\/2019\/02\/19\/status-report-on-fridays-campus-network-outage\/","title":{"rendered":"Status Report on Friday\u2019s Campus Network Outage"},"content":{"rendered":"

For much of this spring semester ITS staff have been chasing down several issues causing network instability. \u00a0One of the causes was failing fiber optic cable which has been located and replaced.<\/span><\/p>\n

We also identified a bug in the network operating system running on some of our core network switches. \u00a0We have been working with Juniper engineers (Juniper is the company that manufactures our networking equipment) for several weeks to identify and fix the problem. \u00a0This may sound like a lot of time but this is a complicated issue and had to be escalated through Juniper technical support.<\/span><\/p>\n

Last Friday evening, February 15, ITS staff performed emergency network maintenance. \u00a0The maintenance process was advised by Juniper. Specifically, the maintenance was an upgrade of the network operating system running on the core switches. \u00a0We expected a 10 to 20 minute loss of Internet connectivity during the process, nothing more. The work started at about 7:30 p.m. and ITS posted a message on the Dash in the afternoon announcing the necessary emergency maintenance. <\/span><\/p>\n

Unfortunately, the maintenance process didn\u2019t go as planned. \u00a0At about 8:30 p.m. we began experiencing unintended side effects of the upgrade. \u00a0From about 9:20 to 11:20 p.m. we lost the campus network completely:wired; eduroam; and telephone service. \u00a0Shortly after the start of the outage, Public Safety, in cooperation with ITS, sent out a message to the campus community via LiveSafe announcing that things were down. \u00a0ITS\/We also emailed faculty, staff, and students regarding the outage around 10 p.m. At about 11:30 p.m. Public Safety posted an “all clear” LiveSafe message when everything was operational again.<\/span><\/p>\n

Yesterday, Monday, February 18, ITS staff debriefed about Friday\u2019s outage and resumed working with Juniper technical support to identify and validate a reliable procedure to perform the upgrade as soon as possible and without another unplanned outage. \u00a0We also put in a request to have Juniper network engineer on site to work with ITS staff during the next upgrade process.<\/span><\/p>\n

Once we have resources lined up and have a clear idea of the procedure and how long it will take, ITS\/we will announce, in advance, any maintenance windows we need to do the upgrade. \u00a0We’ll make sure the schedule impacts services as little as possible (think: between midnight and 5 a.m.). By planning outages in this timeframe, we hope to avoid interrupting significant campus events.<\/span><\/p>\n

If you have any questions or concerns, please feel free to contact me.<\/span><\/p>\n

Joel Cooper<\/span><\/p>\n

Chief Information Technology Officer<\/span><\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

For much of this spring semester ITS staff have been chasing down several issues causing network instability. \u00a0One of the causes was failing fiber optic cable which has been located and replaced. We also identified a bug in the network operating system running on some of our core network switches. \u00a0We have been working with Juniper engineers (Juniper is the company that manufactures our networking equipment) for several weeks to identify and fix the problem. \u00a0This may sound like a lot of time but this is a complicated issue and had to be escalated through Juniper technical support. Last Friday … Continue reading Status Report on Friday\u2019s Campus Network Outage<\/span><\/a><\/p>\n","protected":false},"author":71,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[108,4,103],"tags":[90,157,257,164],"class_list":["post-5619","post","type-post","status-publish","format-standard","hentry","category-network","category-news","category-wireless","tag-featured","tag-network-2","tag-update","tag-upgrade"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/ph2nPL-1sD","_links":{"self":[{"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/posts\/5619","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/users\/71"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/comments?post=5619"}],"version-history":[{"count":2,"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/posts\/5619\/revisions"}],"predecessor-version":[{"id":5623,"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/posts\/5619\/revisions\/5623"}],"wp:attachment":[{"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/media?parent=5619"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/categories?post=5619"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.swarthmore.edu\/its\/wp-json\/wp\/v2\/tags?post=5619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}