 |
| Person | 9/15/2008 10:18 AM |
This morning, we were bombarded by a PayPal Phishing SPAM email attack (which is still continuing.) This caused a delay in incoming email delivery. The SPAM message appears to have penetrated the spam assassin filter, in addition to clogging up the incoming mail queues (because of invalid recipient addresses that were bouncing.)
The message payload includes the following text:
"If you received this notice and you are not the authorized account holder, please be aware that it is in violation of PayPal policy to represent oneself as an other PayPal user.Such action may also be in violation of local, national, and/or international law. PayPal is committed to assist law enforcement with any inquires related attempts to missapropriate personal information with the intent to commit fraud or theft. Information will be provided at the request or law enforcement agencies to ensure that perpetrators are prosecuted to the fullest extent of the law."
We're attempting to block this message on SPAM assassin, but like all such SPAM, there may be mutations of it that evade filtering for the next few days.
|
|
|
|
| Person | 9/2/2008 11:22 AM |
This morning, from about 7:00am to 10:30am, access to the Internet was broken. The visible symptoms included "page not found" for external websites, or extremely slow access at best. Incoming and outgoing email messages to non-Exchange users were affected as well. Access to internal web pages was not affected.
The problem was eventually isolated to an intermittent fault with the Fortinet content filtering engine. The appliance was reset, after which normal operation resumed at about 10:15am.
|
|
|
|
| System Account | 7/31/2008 10:03 AM |
Starting at about 4pm last evening, we experienced a problem with one of our upstream DNS servers located at AT&T. The result of the DNS failure was that (1) internet traffic appeared to be sluggish ( http://www.cnn.com took 10 seconds to display in a browser, for instance) but also, (2) delivery of email to and from the outside world was affected.
If you were expecting to see email from an outside sender, you should start to see it being delivered sometime before noon today. |
|
|
|
| Person | 5/21/2008 4:51 PM |
Exchange 2003 service unexpectedly interrupted
One of the storage volumes on the Exchange 2003 server filled up with runaway log files, causing email messages to queue or in some cases bounce. Normal operation was restored at approximately 4:45pm. If you received 'undeliverable' error messages for mail that you sent over the past few hours, please try resending them. |
|
|
|
| Person | 5/19/2008 2:45 PM |
Exchange 2007 Congestion Problem
Sometime during the past few hours, the routing connector between Exchange 2003 and Exchange 2007 became congested. This resulted in long delays for mail from inside and outside the district to reach email accounts on Exchange 2007 (there are only about 49 of us, and most people in this group are in Information Technology.)
The cause of the congestion was located and repaired (several files with large attachments were gumming up the connector.) Normal mail flow resumed at approximately 2:30 pm. |
|
|
|
| Gary S. Duran | 5/2/2008 8:24 AM |
GHRS IS Back Up as of 8:45am
Last night we experienced problems with a process for Human Resources that failed. We are in the process now of recovering the data for GHRS. We have closed the GHRS files during this recovery process, so that no user input can be made. The GHRS system should be back online by 9:15am. Any user input that was entered this morning will be overlaid and will have to be re-entered. Please inform you staff of the outage and the re-entering of data, once the system is back online. |
|
|
|
| Anthony Ramirez | 4/30/2008 9:29 AM |
Resolved
It appears as though the forwarders on DNS server 172.16.5.68 were pointing to an external DNS server which went offline. I updated the forwarders to point to our new DNS servers 10.223.231.12 and 10.223.241.12, as well as 63.202.63.72 (AT&T's external DNS server). The DNS server is now resolving properly for all clients pointing at 172.16.5.68. This problem would have only affected machines resolving against 172.16.5.68 and trying to get outside (ex: www.google.com), all internal DNS entries would have been accessible.
|
|
|
|
| Person | 4/28/2008 12:01 PM |
Unplanned Outage: Problems accessing certain FUSD sites: resolved at 10:30am
This morning, until 10:30am, certain users were not able to reach the following sites from inside the FUSD network:
rea.fresno.k12.ca.us
mail.fresnounified.org
*.powerschool.fresnounified.org
Users outside the FUSD network were able to reach these sites, and users inside the FUSD network were able to reach other Internet sites not belonging to FUSD.
Users affected: Anyone using statically assigned DNS server settings of 172.16.1.10. Those who have DHCP assigned DNS server addresses of 10.223.231.12 or 172.16.5.68 would have been able to reach all of the sites except for fresno.k12.ca.us addresses during the outage period.
The cause of the problem was a routing error on the Firewall Services Module. This error was introduced late Friday evening when we did work on the firewall. We did not become aware of the problem at that time, because we don't use the 172.16.1.10 DNS server and did not experience any of the problems accessing the above sites. |
|
|
|
| Person | 4/25/2008 7:54 PM |
Planned Service Outage Complete, Normal Internet Operation Resumes
The 4:45pm service outage lasted longer than anticipated due to some unexpected problems with the Firewall Services Module that occurred when its interfaces were reconfigured. Normal operation was restored at 7:30pm 4/25/2008. |
|
|
|
| Person | 4/25/2008 10:31 AM |
Service Outage Notification Friday, April 25th, 2008 4:45pm - 5:00pm
There will be a brief interruption in Internet acccess this afternoon beginning at around 4:45pm. Internal network access will continue to function normally; only access to external web resources, and access to FUSD from the outside, will be interrupted for about 15 minutes. We anticipate Internet access to be back up by 5pm. The reason for the interruption is that we will be modifying router configurations so that we have an alternate path to FCOE for Internet (fiber will be primary; wireless will be secondary.) |
|
|
|
| Person | 4/23/2008 6:04 PM |
Outlook, Sharepoint advisory
Access to Outlook email and Sharepoint was interrupted for 15 minutes around 5:45pm. We performed a firmware upgrade on the two switches that interconnect the blade servers running Exchange 2003 and Sharepoint to their storage servers, causing connections to drop during the time the switches were rebooting.
Normal operation has been restored as of 6:08pm. |
|
|
|
| Person | 4/23/2008 9:03 AM |
|
Wiki Content:
Empowernet is available
Tuesday 7:40pm Access to Empowernet was restored.
Empowernet is unavailable
Tuesday 7:20pm Empowernet was unavailable due to a change in security.
| |
|
|
|
| Person | 4/18/2008 10:06 AM |
Outlook Web access is online.
10:05 am Access to Outlook 2003 has been restored.
Outlook Web access is down.
9:15 am Web access to Outlook 2003 was lost unexpectedly due to a file space problem on the server that occurred early this morning. |
|
|
|
| Person | 4/10/2008 12:39 PM |
Internet Up
update: 1:16pm Internet access has been restored. FCOE reset the wireless link on their side.
Internet Down
Problem: Access to the Internet is down as of approximately 12:00pm on Thursday. Network and Engineering is aware of the problem and is currently researching the outage. Internal resources are available, only Internet access is affected.
update: 12:47pm The problem has been isolated to the FCOE wireless link. We are trying to switch over to the fiber feed to FCOE as an immediate backup. |
|
|
|
| Person | 4/9/2008 8:33 PM |
Description: The DHCP scopes for seven sites were discovered to be incorrectly configured on Wednesday, 04/09/2008 at about 6:00pm. The affected scopes had option 06 (DNS server) overrides that erroneously included an obsolete DNS server (10.223.248.210). The problem only appeared when users whose PC's had DHCP assigned IP addresses in one of the affected scopes attempted to authenticate to the STUDENTS domain, and then only if the secondary DNS (10.223.248.210) was used for name resolution; sometimes the problem would not occur because the first DNS server was used (172.16.5.68). Authentication to the FRESNOUNIFIED domain was not affected.
Symptom: Users were able to logon to resources in the fresnounified.org domain (e.g. http://www.fresnounified.org, using first.last@fresnounified.org format), but on the same PC, they could not login to http://www.fresnounified.org on the STUDENTS\ domain (for instance, using a student's account in the format STUDENTS\ab123456.) The authentication problem occurred because the obsolete DNS server that was specified in the scope did not have pointer records to the STUDENTS domain, with the result that no Domain Controller could be found to authenticate the user.
Solution: We have removed these DHCP scope DNS server overrides so that the seven affected scopes now inherit the correct global DNS server list (10.223.231.12, 10.223.241.12, and 172.16.5.68).
Sites Affected:
Centennial; Forkner; Gibson; Wolters; Yokomi; Ahwahnee; and Tech Services (but only on subnet 172.16.5.0 ) |
|
|
|
| Person | 4/9/2008 9:45 AM |
Outside Email Flowing Again
9:30am, Wednesday, April 9th
Email delivery from outside sources has been restored, after an interruption caused by yesterday afternoon's power outage at the Ed Center. It may take several hours for queued messages from external sources to reach your inbox this morning.
Internal email from one outlook address to another was not affected, but messages sent between Outlook and GroupWise were delayed. |
|
|
|
| Person | 4/6/2008 7:06 PM |
Over the last few days, spam messages that contain blogspot.com links have been leaking through the filters. The messages look something like this -- they have random senders, short subjects, and random bodies, but all contain the blogspot.com URI:
On Sunday evening, 4/6/2008, we added a custom rule to SpamAssassin's filter to block these messages. If you continue to see messages like these leak through, please send them to fightspam@fresnounified.org |
|
|
|
| Person | 4/8/2008 4:58 PM |
Email is Up
As of 5:50pm on Monday, Exchange 2003 was back online.
Email is Down
Around 4:30pm on Monday, April 8th, there was a momentary power outage in the Ed Center. We lost power to a handful of servers, including Exchange 2003 and Sharepoint. As of 5pm, we are still working on restoring Exchange 2003. |
|
|
|
| Person | 4/4/2008 4:19 PM |
What happened: Sequoia's Local Area network began failing Thursday afternoon and failed completely by Friday morning, with the office being the sole exception--network access continued to work there.
Resolution: Network and Engineering located a hardware failure with the core switch at Sequoia, and requested that James from Shawn Carter's group come out to replace the score switch with a spare that Shawn had on hand. As of 2:00pm Friday, Sequoia's LAN appeared to be operating normally again.
Details: Thursday afternoon, 4/3/2008, the local area network at Sequoia began exhibiting widespread outages. All classroom network access was affected, leaving only a few computers in the front office online. Network and Engineering attempted a repair* of the core switch (an older Cajun P550 six-slot chassis with a 12-port fiber tranceiver blade) early Friday morning, but within an hour after it appeared to be working, the LAN segments started failing again.
A team of us went out to diagnose the problem, and we eventually determined that the core switch had developed a hardware failure with its fiber optic blade. There are eight fiber optic feeds which originate from that core switch. These fibers connect to all of the IDF (Intermediate Distribution Frame) switches across campus. All eight ports were exhibiting failures, and our attempts to isolate the failure down to one particular port revealed instead that the entire switch module was defective. This particular model of switch has failed with similar symptoms in the past at other school sites (there are carcasses of Cajun P550s in several switch closets across the district.) These units are no longer under warranty; the cost of a replacement module would not be justified since the entire core switch at Sequoia is scheduled for replacement under our E-rate year 11 application. Our plan is therefore to rely on the spare switch for about 6-9 months until we can replace it with a brand new, high bandwidth core switch.
* As part of our first attempt to isolate the problems with Sequoia's LAN, we enabled spanning-tree protocol to prevent possible routing loops from affecting the network. This appeared to help, but later analysis showed that the real problem was faulty hardware. |
|
|
|
| Person | 4/4/2008 4:29 PM |
Local area network access was restored around 3:00pm on 4/4/2008 to the "N71" building at Bullard High. The Baystack 450 switch in the IDF at that location was still in a crashed state following the routing loop problem identified earlier in the week. The switch at that location simply needed to be power cycled to clear its MAC address table. |
|
|
|
| Person | 4/4/2008 4:24 PM |
Welcome to Opteman, Wawona!
Wawona today became the latest school to be connected to the AT&T Opteman fiber-optic backbone. The network went live around 2:00pm on Friday, 4/4/2008. Testing showed download speeds to internal servers reaching 74Mbits/sec from a previous limit of 1.4Mbit/sec before the change. Download speed to the Internet itself (using speedtest.net and Los Angeles as the target city) showed speeds as high as 40Mbits/sec. |
|
|
|
| Person | 3/29/2008 2:09 PM |
Problem: There is a known issue affecting Exchange 2007 web access from outside the firewall. When you try to access https://exchange01.fresnounified.org/exchange, you are presented with a logon screen as expected. However, once you sign in, your browser returns an HTTP 400 error page.
Resolution: None at the moment.
Workaround: Use terminal services ( Start -> run -> mstsc /v:termserv2.fresnounified.org or termserv5.fresnounified.org) and establish a desktop session. You will be able to access https://exchange01.fresnounified.org/exchange from your remote desktop.
Who is affected: Only those people who currently have mailboxes on Exchange 2007 (about 39 people, almost all of them IT staff members.) |
|
|