Issue #14 » Keira Server Offline

Project: Issues | Roadmaps | ChangelogsAventure Host Status Status: Resolved
Added On: 22/01/2009 13:55 Created By: Aventure
Last Updated: Issue History 22/01/2009 19:21 Assigned To: Aventure
Private: No Notifications: SubscribeNo Subscription

Description

Engineers have taken the Keira server offline for an FSCK due to concerns over the disks performance after recent datacenter power issues in December 2008. If the FSCK identifies issues, engineers may be choose to keep this server offline until the hard drives can be replaced. Further information will be made available when the FSCK completes which may take up to 1 hour.


Comments

Aventure: An FSCK is still running at this time. 22/01/2009 14:05
Aventure: Issues have been found with the drives.
Engineers are attempting to repair the problems before considering a hard drive replacement. Updates to follow.		 22/01/2009 14:35
Aventure: Engineers are progressing well, service due back shortly providing no other issues appear. 22/01/2009 14:56
Aventure: Engineers are taking an image of the hard drive prior to placing services back online.
This won't take long.		 22/01/2009 15:04
Aventure: Services are expected within 30 minutes. 22/01/2009 15:09
Aventure: All services except for the web server are online.
The web server will remain offline until a hard disk image backup has been taken.		 22/01/2009 15:27
Aventure: Disk image is 67% completed. 22/01/2009 15:41
Aventure: Disk image is still being taken. 22/01/2009 16:08
Aventure: Due to the suspected changes in the file system after the recent FSCK, taking the disk image is taking longer than usual due to the large amount of file changes.
For safety we would prefer to ensure an updated disk image is held on record with the file system corrections prior to enabling the final services.

As a bare metal image uses a high level of DISK IO, enabling these services now would create a huge load on the server resulting in a crash, so at this time we are going to wait until it's finished.
We appreciate your patience.		 22/01/2009 16:22
Aventure: All services are fully back online.
Updates will be posted once the issue can be monitored.		 22/01/2009 17:15
Aventure: Engineers are exceptionally pleased with disk performance after the work carried out this afternoon.
All services are still fully online.		 22/01/2009 19:21

History

Aventure: Issue Added 22/01/2009 13:55
Aventure: Status changed from New to Fault 22/01/2009 13:55
Aventure: Comment added 22/01/2009 14:05
Aventure: Comment added 22/01/2009 14:35
Aventure: Comment added 22/01/2009 14:56
Aventure: Comment added 22/01/2009 15:04
Aventure: Comment added 22/01/2009 15:09
Aventure: Comment added 22/01/2009 15:27
Aventure: Comment added 22/01/2009 15:41
Aventure: Comment added 22/01/2009 16:08
Aventure: Comment added 22/01/2009 16:22
Aventure: Comment edited 22/01/2009 16:22
Aventure: Comment added 22/01/2009 17:15
Aventure: Status changed from Fault to At Risk 22/01/2009 17:17
Aventure: Status changed from At Risk to Resolved 22/01/2009 19:21