Videoconference Outage Report
|Videoconference||09/13/18-9/19/18||Since last Thursday, September 13th, ITS Spokane and ITS Pullman video conferencing, network, and security engineers have been investigating significant outages to video conferencing services that have occurred at least once each day since Thursday.|
Yesterday, Tuesday, September 18th, video conferencing infrastructure experienced three such outages at approximately 9:20am, 2:50pm, and 5:20pm. It is believed the 9:20am and 5:20pm outages were triggered by Distributed Denial of Service (DDoS) attacks against the Pullman Domain Name Service (DNS) exacerbated by high-packet loss in critical core network and security infrastructure in Pullman. The 2:50pm outage, the most significant by far, was caused by accidental changes by network engineering staff as they attempted to bring up additional network monitoring to better help understand the various complex dimension of the outages.
Yesterday afternoon, network engineers and technicians did an exhaustive evaluation and reconfiguration of the parallel, redundant core network infrastructure in Pullman prior to failing over to this infrastructure at 8:40pm last night after all video conferencing classes had ended. Network and security engineers are reporting zero packet loss since the failover. While preventing DDoS attacks is not something we can easily prevent, last night’s and future planned changes will certainly help us weather future attacks much better.
Finally, network security engineers have put in place – effective immediately – process controls which will eliminate the probability of a recurrence of the accident that resulted in the 2:50pm outage and will significantly minimize the impact of more broad, general accidents in the future.
Please direct any questions to Tony Opheim, Associate Vice President and Deputy Chief Information Officer.