On Feb 15th, DoIT transferred to CALS IT control over delegation of the *.cals.wisc.edu domain used for websites, active directory, and other services. While doing so, they failed to set up the reverse lookup records, which is part of their anti-spoofing protocols. As a consequence, all requests for *.cals.wisc.edu information were ignored by the campus DNS servers. Local machines returning to CALS sites previously visited used locally cached information, so this problem didn’t really show up for 24 hr, the usual duration of the cache.
Once we notified DoIT, they elevated this to their highest priority, found the error, and fixed it within 90 minutes. Service resumed about 10:15 on Thursday morning.
Affected services included eCALS, CALS webpages, our electronic lab notebook service, our SigmaPlot and MatLab service, and the shared network drives inside Ag Hall.
Our servers and infrastructure never went down at any point and were accessible by anyone using something other than the campus DNS, as well through individual IP addresses.
We can’t say that we’ve never made a mistake or never will, but this service outage was not the result of anything we did, or didn’t do. Even so, we regret the inconvenience to all who use CALS IT services.
–Phillip Barak, PhD
Professor, Dept of Soil Science
Interim IT Director, CALS