NGDC fabric OS upgrade

No impact on service during period
Service affected: Data Centre
Start: 30 Apr 2017, 3:00 PM
End: 30 Apr 2017, 6:00 PM
Login to request notification
The NGDC ACI fabrics in Prentice and GPN DC are to be upgraded to a new version to solve a known issue happening on the fabric controllers.

Two brief connectivity dropouts to NGDC services are expected when the fabric edge nodes reboot to new version during this change window.

Medicine P: and S: Drive connection issues

Reduced service delivery during period
Service affected: nas02 Group & Home File Shares
Start: 5 Dec 2016, 2:39 PM
End: Unknown
Login to request notification
Some Windows 10 systems are not automatically mapping P: and S: Drives, or these drives are disconnecting throughout the day and require restart or re-mapping manually. Drive paths affected are nas02.storage.uq.edu.aushares and nas02.storage.uq.edu.auhomes%username%
Problem ticket 795741 created 10 Mar 2017

Organising vCenter folders and permissions - post restructure

Resolved
Service affected: VMware GPN DC
Start: 26 Apr 2017, 2:00 PM
End: 26 Apr 2017, 2:30 PM
Login to request notification
Post restructure, VM folders require adjustment to reflect the new structure. Over time as service ownership is further realised VMs can be moved into these new folders to provide access for the appropriate team.

Some of this is already known and this CR details these changes.

1. rename ITS-MS folder across all vCenters to ITS-AA (Application administration). this has been already done.
2. Create the following new folders, ITS-RI (research Infrastructure), ITS-IdM (identity management), ITS-IS (infrastructure services). already done.
3. obtains new AD group for each team from managers. (This groups will provide access to VMs)
4. move all FIT VMs to ITS-AA.
5. move science VMS to ITS-RI
Resolution: 26 Apr 2017, 2:30 PM
Automatic resolution.

Wireless Networks Connectivity Issues

Resolved
Service affected: Wireless Network
Start: 25 Apr 2017, 3:15 PM
End: 25 Apr 2017, 4:05 PM
Login to request notification
ITS engineers are investigating an issue with the wireless networks.

There may be connectivity issues in some areas.
Resolution: 25 Apr 2017, 4:05 PM
Connectivity should now be restored. There was a software fault, which caused APs to migrate between controllers. ITS engineers will continue to monitor for further faults.

SI-net production database unavailable

Resolved
Service affected: SI-net
Start: 25 Apr 2017, 6:10 AM
End: 25 Apr 2017, 6:25 AM
Login to request notification
The SI-net database was unavailable whilst the database was switched from Prentice to GPN as part of scheduled maintenance.

The root cause of the outage is to be determined but the outage was resolved by starting the database service.
Update: 26 Apr 2017, 5:28 PM
The database switchover was planned as part of CR#2723 http://serviceview.its.uq.edu.au/index.php?AP=change_request_ap&ACT=view_all&dialog=edit&change_request_id=2723 An incident report is being prepared for this outage.
Resolution: 26 Apr 2017, 6:25 AM
Automatic resolution.

Switchover Patched Databases Back to Crackle

Resolved
Service affected: Oracle Tier 1 DB Hosting
Start: 25 Apr 2017, 6:00 AM
End: 25 Apr 2017, 8:00 AM
Login to request notification
Switchover Patched Databases Back to Crackle

Resolution: 25 Apr 2017, 8:00 AM
Automatic resolution.

Hazardous Works: Blackboard Database Maintenance

Resolved
Service affected: Blackboard
Start: 23 Apr 2017, 6:00 AM
End: 23 Apr 2017, 6:30 AM
Login to request notification
The Learn.UQ Blackboard system will be undergoing maintenance operations on Sunday 23rd April between 6:00am and 6:30am. There may be minor service interruptions during this period. Maintenance is required to ensure reliable and secure operation of systems. We apologise for the inconvenience this may cause.
Resolution: 23 Apr 2017, 6:30 AM
Automatic resolution.

SI-net and UniFi Unavailable

Resolved
Service affected: UniFi & SI-net Application Servers Unix
Start: 23 Apr 2017, 6:00 AM
End: 23 Apr 2017, 6:30 AM
Login to request notification
Old application servers are being removed from the PeopleSoft application server platform. During this time web access to SI-net and UniFi will not be available.
Resolution: 23 Apr 2017, 6:30 AM
Automatic resolution.

Exadata Platinum patching - crackle-cluster

Resolved
Service affected: Oracle Tier 1 DB Hosting
Start: 23 Apr 2017, 6:00 AM
End: 23 Apr 2017, 7:00 PM
Login to request notification
Patch the GPN Exadata with the Jan 2017 Bundle Patch

Prior to the patching the following databases will be switched to the Prentice Exadata.

FS92P - UniFi
SA90P - SI-net
AUR11P - AURION
BBLRNP - Blackboard
PFABP - Archibus
SALAPD - Si-net legacy
CTMEMP - Control-M
CTMSVP - Control-M
ERAP - ERA
HUBP - Data Hub
ODIP - ODI
RMANP - Oracle Backup
TRIMP - Trim
TRIP - J D Edwards
ADVODP - Advantage Office
AWUP - Animal Welfare
BSHAP
COLLP - Collations
DSTATP - Stat
ECPSP - Electronic Course Profiles
ITSHSP - Headshots
ITSUTP - Various Web Databases
ITSWP - Various Web Databases
MYUQSP - My UQ
PLNAP - Academic Portal
RMASTP - Research Master
SASCMP - Committees
SASCRP - Credit Processing
SASELP - e-Elections
SPOKEP - SPOKE
SSSDMP - Disability Management
TRAFP - Pizza Traffic
UQORGP - UQORG
UQWEBP - Various Web Databases
OEMP - Oracle Enterprise Monitor

The following development databases will be unavailable during the patch period

SI-net Dev
===========
SA90DEST
SA90PERF
SA90PSUP
SA90TRNM
SA90TRNW
SA90TEST

Aurion Dev
===========
AUR11TR
AUR11UA
AURACPT
AURENQ
AURTRAIN
AURPP
Resolution: 23 Apr 2017, 7:00 PM
Automatic resolution.

Soefile migration change

Resolved
Service affected: File Serving - Unix
Start: 23 Apr 2017, 6:00 AM
End: 23 Apr 2017, 6:30 AM
Login to request notification
Soefile migration change
Resolution: 23 Apr 2017, 6:30 AM
Automatic resolution.

Upgrade Bluecat Address Manager and DNS/DHCP Servers

Resolved
Service affected: DDI orchestration
Start: 21 Apr 2017, 7:00 AM
End: 21 Apr 2017, 9:00 AM
Login to request notification
The BAM and BDDS will be upgraded to 8.2.0 as this update addresses an issue in DHCP affecting DNS/DHCP Server.
Resolution: 21 Apr 2017, 9:00 AM
Automatic resolution.

SharePoint Intranet Authentication Method Conversion

Resolved
Service affected: SharePoint
Start: 20 Apr 2017, 6:00 PM
End: 20 Apr 2017, 6:05 PM
Login to request notification
The SharePoint web instance https://intranet.sharepoint.uq.edu.au authentication method will be converted to Claims Based. Other SharePoint web instances will not be affected.
Resolution: 20 Apr 2017, 6:05 PM
Automatic resolution.

CIP database servers unavailable dur to UQ Direct Connect outage to Amazon

Resolved
Service affected: Central Integration Platform
Start: 20 Apr 2017, 10:11 AM
End: 21 Apr 2017, 8:23 AM
Login to request notification
Users of the Central Integration Platform will experience some service reduction and stale data due to a UQ link outage with the AWS backend databases.

This will effect all downstream consumers of CIP data.

Resolution: 21 Apr 2017, 8:23 AM

Some Amazon Web Services are not avaialbe

Resolved
Service affected: Networks
Start: 20 Apr 2017, 7:00 AM
End: 21 Apr 2017, 7:00 PM
Login to request notification
Network connectivity lost to Amazon Web Services - OLA. Currently investigating.
Resolution: 21 Apr 2017, 7:00 PM
AWS services have been restored. It is understood that a fault in the carrier network was to blame. A change in the network design was implemented during the outage in order to both resolve the issue and ensure that future faults of this kind in the carrier network will no longer affect the service.
Resolution: 24 Apr 2017, 12:43 PM
AWS services have been restored. It is understood that a fault in the carrier network was to blame. A change in the network design was implemented during the outage in order to both resolve the issue and ensure that future faults of this kind in the carrier network will no longer affect the service.
Update: 21 Apr 2017, 1:00 PM
Library services have been restored
Update: 21 Apr 2017, 9:45 AM
The remaining affected services are OLA and Library. A case has been logged with AWS to investigate the suspected issue.
Update: 20 Apr 2017, 7:17 PM
The root cause was not verified but the work around procedure has been taken and the most service has come back online. Network topology changed.
Update: 20 Apr 2017, 5:40 PM
The root fault has been logged with the carrier this morning and is currently still being investigated. ITS engineers are investigating / implementing workarounds to get services back online as soon as possible.

Direct Connect outage

Resolved
Service affected: Amazon Web Services
Start: 20 Apr 2017, 5:30 AM
End: 21 Apr 2017, 7:00 PM
Login to request notification
Network connectivity lost to Amazon Web Services via Direct Connect.

ITS is currently investigating.
Resolution: 21 Apr 2017, 7:00 PM
Network connectivity was restored on 21-Apr-2017 19:00
Resolution: 21 Apr 2017, 7:00 PM
Automatic resolution.

Upgrade PaperCut to 17.0

Resolved
Service affected: Library Printing System
Start: 19 Apr 2017, 6:00 AM
End: 19 Apr 2017, 7:00 AM
Login to request notification
Upgrade PaperCut installation on application server and all secondary servers.
Resolution: 19 Apr 2017, 7:00 AM
Automatic resolution.

Update Drupal CMS to release v1.47.1

Resolved
Service affected: CMS - ITS Drupal
Start: 18 Apr 2017, 8:00 AM
End: 18 Apr 2017, 4:00 PM
Login to request notification
The following updates will affect live sites, stage sites and sandbox sites using the Standard template.

Improvements:
* Added Eloqua tracking on all UQ Standard sites without secure content.
* Added new transition state in UQ Moderation: Strict to allow Approved to publish nodes to be changed back to Needs review.
* Added additional Views to UQ Moderation: Strict to see all Drafts, and all Approved to publish nodes.
* Added a shadow effect to white buttons when viewed inside a WYSIWYG editor field so they can be seen and edited.
* Added teaser image field to UQ Asset content type.
* Improved how front-end styles display within the wysiwyg editor so editors can manage content effectively.

Technical upgrades under the hood:
* Update Drupal core to 7.54.
* Updated jquery_update from 3.x to 3.0-alpha5.
* Patched xmlsitemap.
Resolution: 18 Apr 2017, 4:00 PM
Automatic resolution.

Blackboard reduced capactity

Resolved
Service affected: Blackboard
Start: 16 Apr 2017, 5:00 PM
End: 16 Apr 2017, 6:45 PM
Login to request notification
Learn.uq is suffering from reduced capacity. ITS technicians are investigating.

ITS apologises for any inconvenience caused.
Resolution: 16 Apr 2017, 6:45 PM
Learn.uq has returned to normal capacity. ITS apologises for any inconvenience caused

Exadata Platinum patching - snap-cluster

Resolved
Service affected: Oracle Tier 1 DB Hosting
Start: 16 Apr 2017, 6:00 AM
End: 16 Apr 2017, 11:30 PM
Login to request notification
Exadata Platinum patching - snap-cluster will take place on 16th April 2017. Development databases on the Snap-cluster will be unavailable for the duration. We apologise for any inconvenience this presents.
Resolution: 16 Apr 2017, 11:30 PM
Patching of the Prentice Exadata has completed successfully.
Update: 16 Apr 2017, 9:04 PM
Oracle patch engineers are still working on patching the databases and infiniband network Despite this, there is no expected impact to any production databases. Oracle patching engineers have hit an active bug when applying the patches to the database (Bug 25707907). A safe workaround will be enabled without any loss of service or data.
Update: 16 Apr 2017, 8:04 PM
Oracle patch engineers are still working on patching the databases and infiniband network Despite this, there is no expected impact to any production databases.
Resolution: 16 Apr 2017, 7:00 PM
Automatic resolution.
Update: 16 Apr 2017, 6:52 PM
Due to earlier issues in restarting the cluster, this outage will overrun. Despite the overrun, there is no impact to production services

Prentice Exadata Patching

Resolved
Service affected: Oracle Tier 1 DB Hosting
Start: 16 Apr 2017, 6:00 AM
End: 16 Apr 2017, 11:30 PM
Login to request notification
The Jan 2017 Patch Bundle will be applied to the to Prentice Exadata

This patching is required to ensure that the Exadata database machines and associated databases are protected from known security vulnerabilities and bugs.

Prior to the patching the following databases will be switched to the GPN Exadata.

Data Hub Production
=============
HUBP

Oracle Cloud Control
=============
OEMPDG

SI-net Legacy Archive Production
=====================
SALAPDDG

The following non-production databases will be unavailable for the duration of the outage

UniFi Development
===============
FS92FNTB
FS92LRN1
FS92REPT
FS92PREP

UniFi Production Support
===============
FS92PSUP

Blackboard Enquiry
===============
BBSTGP
BBENQDEV

SI-net Pre-production
================
SA90PREP

The following non-production environments databases will be patched once Oracle Platinum have completed the patch installation (ETA 6pm)

UniFi Pre-production
================
FS92PREP

UniFi Development
===============
FS92FNTB
FS92LRN1
FS92REPT
FS92PSUP

SI-net Pre-production
================
SA90PREP

Blackboard Development
===============
BBSTGP (staging-blackboard.elearning.uq.edu.au)
BBUPGRDV

Resolution: 16 Apr 2017, 11:30 PM
Patching of the Prentice Exadata has completed successfully with all database services restored.
Resolution: 16 Apr 2017, 7:00 PM
Automatic resolution.
Resolution: 16 Apr 2017, 7:00 PM
Please refer to Event# 16718: Oracle Tier 1 DB Hosting (Infrastructure)

Fix logout functionality in Unitask

Resolved
Service affected: Unitask
Start: 13 Apr 2017, 12:00 PM
End: 13 Apr 2017, 12:30 PM
Login to request notification
A fix will be released today, allowing users to logout from Unitask
Resolution: 13 Apr 2017, 12:30 PM
Automatic resolution.

Public Networks Unavailable

Resolved
Service affected: Internet Services UQ
Start: 12 Apr 2017, 10:00 PM
End: 13 Apr 2017, 11:46 AM
Login to request notification
ITS engineers are currently investigating an issue related to internet routing for a particular public subnet.

There have been reports some users are unable to access the internet at Gatton and Herston.

A more detailed list of affected networks are as follows, but may not be all inclusive.

Network, Description, Router
152.98.194.158/26 CBIT claw.router.uq.edu.au
152.98.194.222/26 PLPK in MacGregor / Skerman VTHRC/PLPK in Richie labs A,B,C lvl 2 Anatomy in Otto Hirschfeld Building eagle.router.uq.edu.au
152.98.195.130/26 Mol-Dynamics eagle.router.uq.edu.au
152.98.195.194/27 acts-a4 kite.router.uq.edu.au
152.98.195.226/27 BACS-SAN-mgt in Prentice MR1 eagle.router.uq.edu.au
152.98.195.30/25 CMM in Hawken,Otto Hisrchfd,Old Comp Sci eagle.router.uq.edu.au
152.98.196.30/23 PLPK in MacGregor / Skerman VTHRC/PLPK in Richie labs A,B,C lvl 2 Anatomy in Otto Hirschfeld Building eagle.router.uq.edu.au
152.98.2.30/24 No Description qimrcpe.router.uq.edu.au
152.98.200.31/22 LifeSci=(Botany in Hines,Zoology in Goddard) eagle.router.uq.edu.au
152.98.200.32/22 LifeSci (Botany in Hines,Zoology in Goddard) kite.router.uq.edu.au
152.98.204.31/23 MBBS Building 76 eagle.router.uq.edu.au
152.98.204.33/23 MBBS Building 76 kite.router.uq.edu.au
152.98.206.30/23 Chemistry eagle.router.uq.edu.au
152.98.216.129/25 CBIT claw.router.uq.edu.au
152.98.216.28/27 Faculty office for Social and Behavioural sciences kite.router.uq.edu.au
152.98.216.65/29 No Description kite.router.uq.edu.au
152.98.217.0/26 No Description kite.router.uq.edu.au
152.98.217.130/25 Faculty office for Social and Behavioural sciences kite.router.uq.edu.au
152.98.220.2/24 Anthropology ( ansoc ) Michie levels 1,2,7,8 claw.router.uq.edu.au
152.98.221.30/24 Social Work Chamberain kite.router.uq.edu.au
152.98.222.30/24 School of Journalism Scott Throssel claw.router.uq.edu.au
152.98.224.31/23 Clinical1 spleen.router.uq.edu.au
152.98.226.31/23 Clinical2 spleen.router.uq.edu.au
152.98.228.31/23 PopHealth spleen.router.uq.edu.au
152.98.230.159/26 Centre for On-line Health spleen.router.uq.edu.au
152.98.230.194/26 SASVRC spleen.router.uq.edu.au
152.98.230.30/25 Herston-Servers gibson.router.uq.edu.au
152.98.231.159/26 FacHealth spleen.router.uq.edu.au
152.98.231.222/26 Graduate Medical School Student Hosts charlie.router.uq.edu.au
152.98.231.28/27 Herston Library Private spleen.router.uq.edu.au
152.98.231.29/27 Herston Library Private liver.router.uq.edu.au
152.98.231.34/29 AV management and Video conf RM#2889 spleen.router.uq.edu.au
152.98.231.45/30 Graduate Medical School Student Hosts charlie.router.uq.edu.au
152.98.231.49/28 QUT staff at PCH charlie.router.uq.edu.au
152.98.231.65/27 SoM Vurtual machines gibson.router.uq.edu.au
152.98.231.97/27 Health Sciences Public Servers hercules.router.uq.edu.au
152.98.232.130/25 CCR client net spleen.router.uq.edu.au
152.98.232.31/26 CMR LAN Herston spleen.router.uq.edu.au
152.98.232.66/29 Student Union Hosts spleen.router.uq.edu.au
152.98.232.98/27 CVMH spleen.router.uq.edu.au
152.98.233.2/25 CCR printers spleen.router.uq.edu.au
152.98.234.31/24 SchoolMedicine spleen.router.uq.edu.au
152.98.238.1/26 Som-servers gibson.router.uq.edu.au
152.98.240.122/29 Vice-C via LRE chicken.router.uq.edu.au
152.98.240.159/25 UQG-ADMIN chicken.router.uq.edu.au
152.98.240.28/27 Gatton Server Subnet chicken.router.uq.edu.au
152.98.240.60/27 GAT-PRENTICE-02 chicken.router.uq.edu.au
152.98.240.64/27 No Description cow.router.uq.edu.au
152.98.240.83/28 Plant Nursery via LRE chicken.router.uq.edu.au
152.98.241.31/24 UQG-SHARE-02 chicken.router.uq.edu.au
152.98.243.31/26 UQG-LIB-Staff chicken.router.uq.edu.au
152.98.243.82/28 Plant Nursery via LRE cow.router.uq.edu.au
152.98.244.31/24 UQG-SHARE-03 chicken.router.uq.edu.au
152.98.246.31/24 UQG-SHARE-02 chicken.router.uq.edu.au
Resolution: 13 Apr 2017, 11:46 AM
A routing issue with an upstream provider was prohibiting traffic between a public /18 network and the Internet. The issue has been corrected and this traffic is now routing correctly.

Migrate VIP Task Manager to new server

Resolved
Service affected: Library VIP Task Manager
Start: 12 Apr 2017, 5:30 AM
End: 11 Apr 2017, 8:30 PM
Login to request notification
The server hosting VIP task manager is being decommissioned. This outage is to move the database to a new server.
Resolution: 11 Apr 2017, 8:30 PM
VIP Data migrated successfully.

Production Drupal web sites impacted by Database outage

Resolved
Service affected: CMS - ITS Drupal
Start: 7 Apr 2017, 2:00 PM
End: 7 Apr 2017, 2:10 PM
Login to request notification
The production Drupal CMS cluster is currently experiencing an outage due to technical issues with the database server cluster. Technicians are investigating and working to restore service
Resolution: 7 Apr 2017, 2:10 PM
The issue impacting the CMS Drupal Production cluster is now resolved.
Update: 7 Apr 2017, 2:14 PM
The issue impact the CMS Drupal production systems is now resolved.
Resolution: 7 Apr 2017, 2:10 PM
Automatic resolution.

Reduced availability of resolving DNS servers

Resolved
Service affected: DNS
Start: 7 Apr 2017, 8:00 AM
End: 7 Apr 2017, 12:10 PM
Login to request notification
Resolution: 7 Apr 2017, 12:10 PM
The resolving nameservers are now fully operational.

Telstra link maintenance

Resolved
Service affected: Internet Services - Commercial
Start: 7 Apr 2017, 1:00 AM
End: 7 Apr 2017, 3:15 AM
Login to request notification
Telstra is currently performing maintenance on Internet links out of St Lucia affecting a number of commercial and Schoolsnet sites. Due to the nature of the works a specific time of recovery cant be provided, however the maintenance window concludes at 6:00am.
Resolution: 7 Apr 2017, 3:15 AM
Telstra maintenance was unsuccessful; another maintenance window will be required at another time. All affected service have recovered

IPALOC database unavalable

Resolved
Service affected: IpAlloc VPN IP allocation
Start: 6 Apr 2017, 4:00 AM
End: 6 Apr 2017, 5:54 AM
Login to request notification
A host related incident at 4AM lead this morning has lead to an outage of the IPALOC database.

ITS staff will restart the service on the standby node.
Resolution: 6 Apr 2017, 5:54 AM
The IPALOC database service has been started on the standby node.

snapdev01 unavailable

Resolved
Service affected: Oracle Tier 3 DB Hosting
Start: 6 Apr 2017, 4:00 AM
End: 6 Apr 2017, 5:54 AM
Login to request notification
The host snapdev01 has been unavailable since 4AM.

The following development databases and environments are currently unavailable

#DB_NAME
#-------
BBARCHDV
BBEXPDEV
BBTESTDV
BSHADVDB
BSPRODDB
FS854SG1
FS854SG2
FS855SG1
FS92DEMO
FS92DEMP
FS92DEVN
FS92DEVT
FS92DVTS
FS92FNTS
FS92LRN0
FS92LRN2
FS92SBOX
FS92TECT
FS92TSTP
IPAPDG
PLNAPTR
SA854SG2
Resolution: 6 Apr 2017, 5:54 AM
The host snapdev01 has been restarted and the databases are available once more

Production CMS Outage

Resolved
Service affected: CMS - ITS Drupal
Start: 5 Apr 2017, 3:00 PM
End: 5 Apr 2017, 3:45 PM
Login to request notification
An issue with the Drupal CMS Database servers has resulted in an outage to service.
Resolution: 5 Apr 2017, 3:45 PM
Automatic resolution.

TRI Level 7 East side outage

Resolved
Service affected: Building Networks
Start: 3 Apr 2017, 8:57 AM
End: 3 Apr 2017, 9:45 AM
Login to request notification
Currently a hardware issue is impacting this level and side. ITS are on the way to replaced the failed equipment.
Resolution: 3 Apr 2017, 9:45 AM
ITS have replaced the failed hardware and confirmed with users that the original fault is no longer occurring.

Mater Hospital Whitty Building Network outage

Resolved
Service affected: Building Networks
Start: 30 Mar 2017, 9:00 AM
End: 31 Mar 2017, 10:00 AM
Login to request notification
ITS are investigating
Resolution: 31 Mar 2017, 10:00 AM
Mater property and facilities had restored power to comms room
Update: 31 Mar 2017, 9:30 AM
Mater has confirmed water damage to communication room 01-A005 and their property department have been notified. Power to the room has been isolated until electricians have deemed safe to restore power.

Deployment of new HR Process in Unitask

Resolved
Service affected: Unitask
Start: 29 Mar 2017, 1:00 PM
End: 29 Mar 2017, 1:30 PM
Login to request notification
A new version of Fixed Term Appointment Expiry (v1.13) is to be deployed.
No impact is expected to existing Unitask processes.
Resolution: 29 Mar 2017, 1:30 PM
Automatic resolution.

Exams Management System IP Address Change

Resolved
Service affected: Databee Exams Manager
Start: 28 Mar 2017, 4:00 PM
End: 28 Mar 2017, 4:15 PM
Login to request notification
The IP address for the Exams Management System server will be changed.
Resolution: 28 Mar 2017, 4:15 PM
Automatic resolution.

Routing issues occurring on eagle/kite

Resolved
Service affected: Networks
Start: 27 Mar 2017, 3:44 PM
End: 27 Mar 2017, 5:05 PM
Login to request notification
ITS engineers are currently investigating an issue with eagle/kite routing.

This is having an affect on data-center management networks and possibly user traffic within buildings connected to eagle/kite.

Updates will be provided once more information is known.
Resolution: 27 Mar 2017, 5:05 PM
ITS engineers have located the fault and connectivity should be stable.

EMS site unavailable

Resolved
Service affected: Databee Exams Manager
Start: 27 Mar 2017, 9:00 AM
End: 27 Mar 2017, 11:40 AM
Login to request notification
The Exams Management system is currently unavailable due to a configuration issue.
Resolution: 27 Mar 2017, 11:40 AM
DNS and firewall rules updated.
Resolution: 27 Mar 2017, 11:40 AM
DNS and firewall rules updated.
Resolution: 27 Mar 2017, 9:40 AM
DNS and firewall rules updated.

Update Drupal CMS to release v1.46.0

Resolved
Service affected: CMS - ITS Drupal
Start: 27 Mar 2017, 8:00 AM
End: 28 Mar 2017, 4:00 PM
Login to request notification
The following updates will affect live sites, stage sites and sandbox sites using the Standard and Legacy template.

Improvements:
* Improved wysiwyg editor so you can see the styles you are working with in the page while editing.
* Changed Video Pane component to behave as widescreen videos as 16:9 ratio is more common.
* Added option to hero banners to add spacing below them.

Technical upgrades under the hood:
* Added UQ Eloqua to legacy and standard distributions
* Update restws in leagcy to 2.7
* Disabled autocomplete in core Drupal login field.
Resolution: 28 Mar 2017, 4:00 PM
Automatic resolution.

Urgent Lync Service Maintenance

Resolved
Service affected: MS Lync 2010
Start: 24 Mar 2017, 3:00 PM
End: 24 Mar 2017, 4:00 PM
Login to request notification
ITS will be performing scheduled maintenance on the Lync service.
During this time, there may be interrupted/reduced service for Lync.
Resolution: 24 Mar 2017, 4:00 PM
Automatic resolution.

Unscheduled outage for Eduroam on the CityCats

Resolved
Service affected: Wireless Network
Start: 24 Mar 2017, 8:00 AM
End: 27 Mar 2017, 8:00 AM
Login to request notification
Telstra and UQ Mobiles are currently investigating accounting issues.
Resolution: 27 Mar 2017, 8:00 AM

Network issues in Goddard Building due to power outage

Resolved
Service affected: Building Networks
Start: 24 Mar 2017, 7:00 AM
End: 24 Mar 2017, 9:50 AM
Login to request notification
A communication cabinet in room 132 has no power effecting network connectivity to some rooms of level 1. P&F have been contacted and Archibus ticket 10132427 created. Resolution time is unknown at this stage.
Resolution: 24 Mar 2017, 9:50 AM
P&F Have restored power to the comms room and ITS has replaced a failed network switch.

Building Network Outage Due to Power Loss

Resolved
Service affected: Building Networks
Start: 21 Mar 2017, 10:44 AM
End: 21 Mar 2017, 11:26 AM
Login to request notification
Network connectivity to a number of buildings is currently affected due to a power outage to network devices.

The following buildings are affected, but may not include the full extent of outage:
- 01-0042
- 01-0065
- 01-0044
- 01-0043
- 01-0098d
- 01-0057
Resolution: 21 Mar 2017, 11:26 AM
Building access has been restored with power restoration.

Active Directory Org Unit Changes: Science

Resolved
Service affected: Active Directory
Start: 20 Mar 2017, 5:00 PM
End: 20 Mar 2017, 6:00 PM
Login to request notification
The following changes will be made to the Active Directory Organisational structure as part of this change.

Sub OUs, computer and user accounts will be moved at the discretion of ITS Microsoft Services to the most appropriate location if they will be affected by a deletion. If delegates want these objects moved to specific locations, they will need to arrange to move them in advance of the change.

###Deleted Org Units###
------------
Earth Sciences DELETED
Geography Planning and Environmental Management DELETED
Resolution: 20 Mar 2017, 6:00 PM
Automatic resolution.

Central Exchange Mail Services - OS Patching

Resolved
Service affected: Exchange Mail and Calendaring
Start: 18 Mar 2017, 9:00 AM
End: 18 Mar 2017, 1:00 PM
Login to request notification
As part of ongoing and proactive maintenance, some Exchange services will experience brief service interruptions.

Some Central Exchange users may experience small, intermittent interruptions to their mail and calendaring services during the scheduled maintenance window.

ITS apologises for any inconveniences that this may cause.

This continual, proactive work being undertaken is to provide an ongoing high level of Exchange Mail service availability to the University community.
Resolution: 18 Mar 2017, 1:00 PM
Automatic resolution.

SQL 2014 Clustered Database Services - OS Patching

Resolved
Service affected: SQL 2014 Clustered Database Hosting
Start: 18 Mar 2017, 9:00 AM
End: 18 Mar 2017, 1:00 PM
Login to request notification
This outage is to facilitate regular monthly operating system updates, security enhancements, and overall service improvements.

Clients and applications with databases hosted on the Microsoft SQL Server 2014 cluster, will experience a very brief service interruption whilst database instances are failed over between ITS data centres.

The service interruption duration is generally 60-120 seconds within the designated outage window. Affected applications and services typically reconnect automatically.
Resolution: 18 Mar 2017, 1:00 PM
Automatic resolution.

SQL 2008 Clustered Database Services - OS Patching

Resolved
Service affected: SQL 2008R2 Clustered Database Hosting
Start: 18 Mar 2017, 9:00 AM
End: 18 Mar 2017, 1:00 PM
Login to request notification
This outage is to facilitate regular monthly operating system updates, security enhancements, and overall service improvements.

Clients and applications with databases hosted on the legacy UQMSSQL SQL 2008 R2 cluster will experience a very brief service interruption whilst database instances are failed over between ITS data centres.

The service interruption duration is generally 60-120 seconds within the designated outage window. Affected applications and services typically reconnect automatically.
Resolution: 18 Mar 2017, 1:00 PM
Automatic resolution.

Remove S200 node network interfaces from nas02 pool

Resolved
Service affected: nas02 Group & Home File Shares
Start: 16 Mar 2017, 5:00 PM
End: 16 Mar 2017, 7:00 PM
Login to request notification
As a step toward retiring the older S200 performance nodes, we need to redirect clients to interfaces on alternative nodes within the cluster.
Resolution: 16 Mar 2017, 7:00 PM
Automatic resolution.

Update Drupal CMS to release v1.45.0

Resolved
Service affected: CMS - ITS Drupal
Start: 16 Mar 2017, 8:00 AM
End: 16 Mar 2017, 4:00 PM
Login to request notification
The following updates will affect live sites, stage sites and sandbox sites using the Standard template.

Improvements:
* Improved the print stylesheet of UQ Standard theme
Resolution: 16 Mar 2017, 4:00 PM
Automatic resolution.

Monthly Windows Patching - Faculty Servers

Resolved
Service affected: Faculty Server Infrastructure
Start: 15 Mar 2017, 6:00 PM
End: 15 Mar 2017, 10:00 PM
Login to request notification
Faculty servers will undergo monthly Windows patching and services will be offline while servers are rebooting during this process.
Resolution: 15 Mar 2017, 10:00 PM
Automatic resolution.

Computer Science Level 3 network outage

Resolved
Service affected: Building Networks
Start: 15 Mar 2017, 3:05 PM
End: 15 Mar 2017, 4:00 PM
Login to request notification
Power is down to the comms room on level 3, Building 69. This is impacting network services to many hosts on this level.
Resolution: 15 Mar 2017, 4:00 PM
power restored to comms room

JKMRC Network Outage

Resolved
Service affected: Networks
Start: 14 Mar 2017, 4:26 PM
End: 14 Mar 2017, 7:15 PM
Login to request notification
ITS engineers are investigating a network fault at JKMRCs site located at Indooroopilly.

Updates will be provided when more information is available.
Resolution: 14 Mar 2017, 7:15 PM
The onsite router was found in a faulty state due to defective software. The router had emergency maintenance performed to upgrade the software to a known stable version. Connectivity has been restored.

Tomcat and java update

Resolved
Service affected: ARCHIBUS Web Servers
Start: 11 Mar 2017, 7:00 AM
End: 11 Mar 2017, 9:00 AM
Login to request notification
This change is to update tomcat to version 7.x and java to version 1.7.x
Resolution: 11 Mar 2017, 9:00 AM
Automatic resolution.

Some Peer Networks went down

Resolved
Service affected: Networks
Start: 9 Mar 2017, 3:25 PM
End: 9 Mar 2017, 3:45 PM
Login to request notification
Unplanned outage occurred but resolved after the cause was found and eliminated.
Resolution: 9 Mar 2017, 3:45 PM
Automatic resolution.

Apply vCenter 6.0 update 3

Resolved
Service affected: VMware NextGEN PRN DC
Start: 9 Mar 2017, 2:00 PM
End: 9 Mar 2017, 3:00 PM
Login to request notification
Apply update 3 to the vcenter instance, itsprnpvc.compute.dc.uq.edu.au and PSC server, itsprnppsc.compute.dc.uq.edu.au
Resolution: 9 Mar 2017, 3:00 PM
Automatic resolution.

Update Drupal CMS to release v1.44.0

Resolved
Service affected: CMS - ITS Drupal
Start: 9 Mar 2017, 8:00 AM
End: 10 Mar 2017, 2:00 PM
Login to request notification
The following updates will affect live sites, stage sites and sandbox sites using the Standard and Legacy template.

Bug fixes:
* Prevented duplicate projects from listing on a profile page, where that person is tagged twice in a project.

Technical upgrades under the hood:
* Updated views from 3.14 to 3.15.
Resolution: 10 Mar 2017, 2:00 PM
Automatic resolution.

Apply ESXi patches to CCR Nutanix cluster

Resolved
Service affected: VMware CCR DC
Start: 8 Mar 2017, 9:00 AM
End: 8 Mar 2017, 3:30 PM
Login to request notification
Apply latest ESXi patches to Nutanix cluster.
Resolution: 8 Mar 2017, 3:30 PM
Automatic resolution.

Apply ESXi patches to GPN Nutanix cluster

Resolved
Service affected: VMware GPN DC
Start: 8 Mar 2017, 9:00 AM
End: 8 Mar 2017, 3:30 PM
Login to request notification
Apply latest ESXi patches to Nutanix cluster.
Resolution: 8 Mar 2017, 3:30 PM
Automatic resolution.

Apply ESXi patches to PRN Nutanix cluster

Resolved
Service affected: VMware PRN DC
Start: 8 Mar 2017, 9:00 AM
End: 8 Mar 2017, 3:30 PM
Login to request notification
Apply latest ESXi patches to Nutanix cluster.
Resolution: 8 Mar 2017, 3:30 PM
Automatic resolution.

Commercial Internet

Resolved
Service affected: UQSchoolsNet
Start: 8 Mar 2017, 8:30 AM
End: 9 Mar 2017, 9:00 AM
Login to request notification
Further issues with the Commercial Internet this morning that is service impacting.
Resolution: 9 Mar 2017, 9:00 AM

Update SiPass SiUQImport service

Resolved
Service affected: PFI SIPASS
Start: 6 Mar 2017, 7:30 PM
End: 6 Mar 2017, 9:30 PM
Login to request notification
SiPass SiUQImport service will be updated to a newer version that will allow common bulk access to be applied to people types (eg staff, student, visitor)
Resolution: 6 Mar 2017, 9:30 PM
Automatic resolution.

Unknown event affecting the OHS Scenario Based Learning Interactive

Resolved
Service affected: OHS Scenario Based Learning Interactive
Start: 6 Mar 2017, 1:56 PM
End: 6 Mar 2017, 1:56 PM
Login to request notification
An unknown problem is preventing logins to the OHS Scenario Based Learning interactive.
Resolution: 6 Mar 2017, 1:56 PM
The OHS Scenario Based Learning Interactive is again available for logins

UQ and eduroam wireless outage

Resolved
Service affected: Wireless Network
Start: 6 Mar 2017, 10:17 AM
End: 6 Mar 2017, 11:17 AM
Login to request notification
ITS are investigating the outage
Resolution: 6 Mar 2017, 11:17 AM
A hardware failure has occurred. The wireless system is currently running in a non redundant state until the new hardware arrives.
Resolution: 6 Mar 2017, 11:17 AM
Automatic resolution.
Resolution: 6 Mar 2017, 11:17 AM
Automatic resolution.

Add additional HDD raid set for golden Images. Expand VDIHVD05 capacity

Resolved
Service affected: VDI Servers
Start: 3 Mar 2017, 9:00 AM
End: 3 Mar 2017, 11:00 AM
Login to request notification
** Non-disruptive change**

This change will add capacity to the VDI hosts that stores the Golden images for the VDI solution. Once added we will re-locate the golden images to the new location.
Resolution: 3 Mar 2017, 11:00 AM
Automatic resolution.

Student Evaluation System Upgrade

Resolved
Service affected: EvaSys
Start: 2 Mar 2017, 8:00 AM
End: 9 Mar 2017, 6:00 PM
Login to request notification
A new version of the Student Evaluation System (EvaSys) will be installed.
Resolution: 9 Mar 2017, 6:00 PM
Automatic resolution.

Update Drupal CMS to release v1.43.0

Resolved
Service affected: CMS - ITS Drupal
Start: 2 Mar 2017, 8:00 AM
End: 3 Mar 2017, 4:00 PM
Login to request notification
The following updates will affect live sites, stage sites and sandbox sites using the Standard and Legacy template.

Improvements:
* Reduced the caching time of eSpace publication list on profile page to an hour
* Added Apple touch icon to UQ Standard.
* Changed feedback link in footer to point to new form hosted on ITS website.

Bug fixes:
* Fixed team directory so teams which are not assigned to a content queue will show at the bottom of the page

Technical upgrades under the hood:
* Updated Better Exposed Filters on legacy environment.
* Disabled and uninstalled dblog module on all sites.
Resolution: 3 Mar 2017, 4:00 PM
Automatic resolution.

Aurion Production App Zpool vdev replacement

Resolved
Service affected: Aurion Application Servers
Start: 2 Mar 2017, 7:00 AM
End: 2 Mar 2017, 12:00 PM
Login to request notification
Migrate Zpool vdevs to new storage array
Resolution: 2 Mar 2017, 12:00 PM
Automatic resolution.

Apply vCenter 6.0 update 3

Resolved
Service affected: VMware GPN DC
Start: 28 Feb 2017, 10:00 AM
End: 28 Feb 2017, 11:30 AM
Login to request notification
Apply update 3 to the vCenter instance, uqvcgpn01.soe.uq.edu.au

GPN vCenter will be unavailable during upgrade. VM access via SSH and RDP will not be affected.
Resolution: 28 Feb 2017, 11:30 AM
Automatic resolution.

Apply vCenter 6.0 update 3

Resolved
Service affected: VMware PRN DC
Start: 28 Feb 2017, 9:00 AM
End: 28 Feb 2017, 10:30 AM
Login to request notification
Apply update 3 to the vCenter instance, uqvcprn01.soe.uq.edu.au

PRN vCenter will be unavailable, SSH and RDP to VMs will not be affected.
Resolution: 28 Feb 2017, 10:30 AM
Automatic resolution.

SAN Director ISL Link Failure

Resolved
Service affected: SAN fabric
Start: 27 Feb 2017, 12:30 PM
End: 16 Mar 2017, 9:14 AM
Login to request notification
ISL Communication between two SAN Directors has failed. After rebooting Bacon for a separated issue, the ISL link between Bacon and Hearts has failed.

Servers connected to this Director have Reduced Redundancy.

They are still able to connect via their redundant link to the other Director Clubs.
Resolution: 16 Mar 2017, 9:14 AM
Director is stable. Case with HDS has been closed.

Commercial Internet

Resolved
Service affected: UQSchoolsNet
Start: 24 Feb 2017, 9:54 AM
End: 27 Feb 2017, 10:47 AM
Login to request notification
There is currently an issue on the Commercial Internet that is service impacting.
Resolution: 27 Feb 2017, 10:47 AM
Advanced monitoring features enabled on the gateway routers were found to be increasing traffic latency under certain conditions. Removing these features on affected interfaces has decreased latency to back to normal levels.
Resolution: 27 Feb 2017, 10:47 AM
Advanced monitoring features enabled on the gateway routers were found to be increasing traffic latency under certain conditions. Removing these features on affected interfaces has decreased latency to back to normal levels.

Casper Upgrade

Resolved
Service affected: ITS Staff Casper
Start: 23 Feb 2017, 4:00 PM
End: 23 Feb 2017, 5:00 PM
Login to request notification
Upgrade to 9.97
Resolution: 23 Feb 2017, 5:00 PM
Automatic resolution.

Reduced network services for UQSchoolsnet

Resolved
Service affected: UQSchoolsNet
Start: 23 Feb 2017, 8:30 AM
End: 23 Feb 2017, 11:00 AM
Login to request notification
We are currently investigating the cause of increased latency and loss to UQSchoolsnet and commercial network services.
Resolution: 23 Feb 2017, 11:00 AM
Automatic resolution.
Update: 23 Feb 2017, 2:17 PM
After investigations of the slow speeds has highlighted inadequacies of throughput of a core line card and is being upgraded during an emergency change tonight between 5pm and 8pm. No anticipated outage is expected during the change window.

Update Drupal CMS to release v1.42.0

Resolved
Service affected: CMS - ITS Drupal
Start: 23 Feb 2017, 8:00 AM
End: 24 Feb 2017, 4:00 PM
Login to request notification
The following updates will affect live sites, stage sites and sandbox sites using the Standard and Legacy template.

Improvements:
* Added a UQ Moderation: Strict sub-module that forces a stricter review and moderation workflow. This is an optional module for sites with many editors and rigorous approval processes
* Tweaked padding in accordions at small screen sizes to increase horizontal space for content.
* Stopped tabs from showing their url path after the tab title when print the page.

Bug fixes:
* Fixed a bug with secondary accordion style where accordion titles would indent strangely on small screens if they wrapped multiple lines.
* Fixed views using ajax pager to display properly inside QuickTab.
* Fixed issue where Articles, Projects, and Team content types did not display the node title in the browser title.
* Fixed secure content functions which result in feature override.
* Remove incorrect network error messaging when users device is not connected to internet

Technical upgrades under the hood:
* Upgrade Workbench in legacy environment.
* Remove OAuth module from legacy environment.
* Remove Twitter module from legacy environment.
Resolution: 24 Feb 2017, 4:00 PM
Automatic resolution.

TRI Voicemail

Resolved
Service affected: PABX Voice Services
Start: 21 Feb 2017, 11:00 AM
End: 21 Feb 2017, 9:00 PM
Login to request notification
The TRI Voicemail system (37799) is running in survivable mode due to a hardware failure. Technicians are investigating.

Callers are still able to leave messages at this time, however new announcements and password changes are not applied.

Note: TRI voicemail system only.
Resolution: 21 Feb 2017, 9:00 PM

WebDNS Hosting Migration

Resolved
Service affected: WebDNS
Start: 20 Feb 2017, 10:00 AM
End: 20 Feb 2017, 2:00 PM
Login to request notification
WebDNS will be migrating during this change window and will have some outage.
Resolution: 20 Feb 2017, 2:00 PM
Automatic resolution.
Resolution: 20 Feb 2017, 12:00 PM
Automatic resolution.

Cut over VM file system backups

Resolved
Service affected: Backup
Start: 20 Feb 2017, 9:00 AM
End: 24 Feb 2017, 4:00 PM
Login to request notification
Cut over VM file system backups to new Commvault environment
Resolution: 24 Feb 2017, 4:00 PM
Automatic resolution.

Prentice DC2 - Nexus 5K software upgrade

Resolved
Service affected: Data Centre Network
Start: 19 Feb 2017, 1:00 PM
End: 19 Feb 2017, 3:00 PM
Login to request notification
Software upgrades on the Prentice DC2 n5k pair.

Software upgrades on the following two n5ks:
01-0042-dc2-c2-ag01
01-0042-dc2-d8-ag01

This will also upgrade software on the following fexs:
01-0042-dc2-c3-as03
01-0042-dc2-a5-as01
01-0042-dc2-a6-as01
01-0042-dc2-a7-as01
01-0042-dc2-a7-as02
01-0042-dc2-c4-as01
01-0042-dc2-c4-as02
01-0042-dc2-c7-as01
01-0042-dc2-c7-as02
01-0042-dc2-c8-as01
01-0042-dc2-c8-as02
Resolution: 19 Feb 2017, 3:00 PM
Automatic resolution.

PACE VoIP

Resolved
Service affected: Voice Networks
Start: 19 Feb 2017, 11:30 AM
End: 20 Feb 2017, 11:10 AM
Login to request notification
Identified some VoIP handsets are not registering to the UQ network/PABX. Technicians investigating.
Resolution: 20 Feb 2017, 11:10 AM
Network restored by engineers onsite

PACE BMS systems

Resolved
Service affected: Building Networks
Start: 19 Feb 2017, 3:44 AM
End: 19 Feb 2017, 7:33 AM
Login to request notification
Security reported unreliable connectivity to the monitoring systems at PACE building.
Resolution: 19 Feb 2017, 7:33 AM
Automatic resolution.

PAH network

Resolved
Service affected: Building Networks
Start: 18 Feb 2017, 6:22 AM
End: 18 Feb 2017, 1:10 PM
Login to request notification
There has been a high volume of traffic within the PAH campus network since 6:22 this morning and theres report from P&F regarding connectivity issue to the network.
Resolution: 18 Feb 2017, 1:10 PM
High volume of traffic in the building network disrupted services. Changes were made to the network to limit impact of the traffic within the campus.

SQL 2008R2 Cluster Unavailable

Resolved
Service affected: SQL 2008R2 Clustered Database Hosting
Start: 17 Feb 2017, 7:17 AM
End: 17 Feb 2017, 8:49 AM
Login to request notification
An unknown issue is causing the SQL2008R2 Cluster to be inaccessible. Further updates will be made as information becomes available.
Resolution: 17 Feb 2017, 8:49 AM
Reduced temperature in data center

Raisers Edge Database Migration

Resolved
Service affected: The Raisers Edge
Start: 10 Feb 2017, 6:00 PM
End: 10 Feb 2017, 8:30 PM
Login to request notification
The Raisers Edge SQL database will be migrated to a different SQL server due to storage infrastructure changes.
Resolution: 10 Feb 2017, 8:30 PM
Automatic resolution.

Aurion and My Aurion not available

Resolved
Service affected: Aurion
Start: 10 Feb 2017, 12:00 PM
End: 13 Feb 2017, 10:00 AM
Login to request notification
My Aurion will be unavailable from 12:00PM on Friday, 10 February 2017 to 10:00AM on Monday, 13 February 2017 for system maintenance. We apologise for any inconvenience this may cause.

Aurion will be unavailable from 7:00PM on Friday, 10 February 2017 to 10:00AM on Monday, 13 February 2017 for system maintenance. We apologise for any inconvenience this may cause.
Resolution: 13 Feb 2017, 10:00 AM
The V11 upgrade completed, Aurion and My Aurion are now available.

Checkbox Survey System Upgrade

Resolved
Service affected: Checkbox
Start: 9 Feb 2017, 6:00 PM
End: 9 Feb 2017, 6:45 PM
Login to request notification
The Checkbox survey system will be upgraded to version Q4 2016.
Resolution: 9 Feb 2017, 6:45 PM
Automatic resolution.

Update Drupal CMS to release v1.40.0

Resolved
Service affected: CMS - ITS Drupal
Start: 9 Feb 2017, 8:00 AM
End: 9 Feb 2017, 4:00 PM
Login to request notification
The following updates will affect live sites, stage sites and sandbox sites using the Standard template.

Improvements:
* Show teams which are not assigned to a queue below the teams in queue
* Added an RSS feed view for all Articles on a site. Available at /article/feed
Resolution: 9 Feb 2017, 4:00 PM
Automatic resolution.

TRI network access interruption level 5

Resolved
Service affected: Building Networks
Start: 8 Feb 2017, 6:00 PM
End: 8 Feb 2017, 7:30 PM
Login to request notification
A problem with an access switch on level 5 is currently causing problems for users.
Resolution: 8 Feb 2017, 7:30 PM
The problem seems to have been a storm; the source port was shutdown and things are now back to normal.

SI-net load balancing re-config

Resolved
Service affected: SI-net
Start: 5 Feb 2017, 6:00 AM
End: 5 Feb 2017, 6:15 AM
Login to request notification
Update SI-net web to app server load balancing
Resolution: 5 Feb 2017, 6:15 AM
Automatic resolution.

SQL 2008 cluster databases unavailable

Resolved
Service affected: SQL 2008R2 Clustered Database Hosting
Start: 4 Feb 2017, 6:47 PM
End: 4 Feb 2017, 7:43 PM
Login to request notification
The server hosting the SQL 2008 cluster databases is currently unavailable.

ITS staff are investigating
Resolution: 4 Feb 2017, 7:43 PM
SQL services have been restarted.
Resolution: 4 Feb 2017, 7:43 PM

SAVEXDIR database offline

Resolved
Service affected: SQL 2014 Clustered Database Hosting
Start: 4 Feb 2017, 3:01 PM
End: 4 Feb 2017, 3:15 PM
Login to request notification
An emergency space issue on the SQL 2014 cluster requires ITS to take the SAVEXDIR database offline for 10 minutes.
Resolution: 4 Feb 2017, 3:15 PM
The SavexDir has been returned to online status

SQL 2014 Cluster Storage Issue

Resolved
Service affected: SQL 2014 Clustered Database Hosting
Start: 4 Feb 2017, 2:20 PM
End: 4 Feb 2017, 4:00 PM
Login to request notification
The SQL 2014 Production cluster is currently running very short on space for transaction logs. There is a high probability that the host will run out of log room and the service will stop
Resolution: 4 Feb 2017, 4:00 PM
The storage space issue has been resolved.

Updates to Staff Mail and Calendar (Exchange)

Resolved
Service affected: Exchange Mail and Calendaring
Start: 4 Feb 2017, 11:00 AM
End: 4 Feb 2017, 3:00 PM
Login to request notification
An Exchange product update will be applied to the servers providing the staff mail and calendaring service.

The product update includes security and other fixes. It also assists with maintaining support for the service. The update includes a notable functionality improvement which is to allow staff to view contact and calendar public folders in Outlook Web App (web mail) after making them a favourite in Outlook.

During the change an impact similar to applying the regular monthly security updates is expected. Staff may experience short service disruptions, usually less than 1 minute each, during the change window.
Resolution: 4 Feb 2017, 3:00 PM
Automatic resolution.

Blackboard unavailable

Resolved
Service affected: Blackboard
Start: 3 Feb 2017, 5:15 PM
End: 3 Feb 2017, 5:21 PM
Login to request notification
Blackboard is currently unavailable.
ITS technicians are investigating.
Resolution: 3 Feb 2017, 5:21 PM
Blackboard was temporarily unavailable due to database issues. These issues have now been resolved, and service has been restored.

Emergency database failover required due to ongoing issues in the Prentice Machine Room

Resolved
Service affected: Oracle Tier 1 DB Hosting
Start: 3 Feb 2017, 4:00 PM
End: 3 Feb 2017, 4:30 PM
Login to request notification
Due an active UPS incident in the Prentice Machine room, ITS wish to switch all production databases from Prentice to GPN.

Oracle production databases to be switched in this event include the following:

AURP
BBLRNP
CTMSVP
DSTATP
ECPSP
ERAP
ITSHSP
ITSUTP
ITSWP
MYUQSP
ODIP
OEMP
PFABP
PLNAP
RMANP
RMASTP
SA90P
SALAPD
SASCMP
SASCRP
SASELP
SPOKEP
SSSDMP
TRAFP
TRIMP
TRIP
UNIFI
UQORGP
UQWEBP
HUBP
Resolution: 3 Feb 2017, 4:30 PM
All the relevant databases have been successfully switched from Prentice to GPN
Resolution: 3 Feb 2017, 4:30 PM
Automatic resolution.
Resolution: 3 Feb 2017, 4:30 PM
Automatic resolution.
Update: 3 Feb 2017, 3:56 PM
The list of databases has been expanded to include: BBSTGP_2 CTMEMPDG_1 CTMSVPDG_1 FS92P_2 HUBP_1 AURPDG_1 FS92PP_1 SA90PDG_2 ADVODPDG AWUPDG BBLRNP BSHAP COLLPDG DSTATPDG SASELPDG

SQL 2014 failed over (short interruption)

Resolved
Service affected: SQL 2014 Clustered Database Hosting
Start: 3 Feb 2017, 3:05 PM
End: 3 Feb 2017, 3:25 PM
Login to request notification
Due to the DC issues, there was a short outage while the Ags were failed over. DBs are up and consistent.
Resolution: 3 Feb 2017, 3:25 PM
This was a pre-emptive failover of services to an alternative data centre.

Pre-emptive Move of Staff Mail and Calendar Databases

Resolved
Service affected: Exchange Mail and Calendaring
Start: 3 Feb 2017, 2:47 PM
End: 3 Feb 2017, 2:50 PM
Login to request notification
As a pre-emptive measure, staff mail and calendar (Exchange) databases were move to operate from an alternative data centre.

Staff may have noticed a momentary disruption in service. It is possible some staff may need to restart their mail client to reconnect.
Resolution: 3 Feb 2017, 2:50 PM
Automatic resolution.

SOEFILE Unavailable

Resolved
Service affected: File Serving - Unix
Start: 3 Feb 2017, 2:20 PM
End: 3 Feb 2017, 5:03 PM
Login to request notification
Server has been disconnected from the Disk Array.
Resolution: 3 Feb 2017, 5:03 PM
Failed over to backup node, but data is from the last backup at 11:30pm 02/02/2017.

SQL Server services unavailable

Resolved
Service affected: SQL 2008R2 Clustered Database Hosting
Start: 3 Feb 2017, 2:10 PM
End: 3 Feb 2017, 4:21 PM
Login to request notification
An unknown event is interrupting connections to several SQL 2008 instances. ITS Staff are investigating.
Resolution: 3 Feb 2017, 4:21 PM
SQL services have been brought back online. Databases for applications on the UQMSSQLAPPS instance were not affected by this outage.

Data Center Power Outage impacting UQ applications

Resolved
Service affected: Data Centre Network
Start: 3 Feb 2017, 1:55 PM
End: 14 Feb 2017, 8:42 AM
Login to request notification
A problem with the front-end serverfarm firewall has caused an outage to front-end services, the scope of which is not yet fully understood.
Resolution: 14 Feb 2017, 8:42 AM
Issues due to DC3 UPS.
Update: 3 Feb 2017, 2:48 PM
This outage has also impacted Tier 1 databases. The following databases have switched over from GPN to Prentice machine rooms. Aurion UniFi Control-M Myuq single signon
Resolution: 3 Feb 2017, 2:10 PM
Automatic resolution.
Update: 3 Feb 2017, 2:22 PM
It now appears a power fluctuation in DC3 may have been the underlying cause of the fault. The connection between this fluctuation and its ultimate affect on services is still under investigation.
Resolution: 3 Feb 2017, 2:10 PM
Automatic resolution.

Prentice VMWare web client service

Resolved
Service affected: VMware PRN DC
Start: 31 Jan 2017, 11:34 AM
End: 31 Jan 2017, 12:34 PM
Login to request notification
The Prentice VMWare web client service requires a restart as it is currently refusing connections.
VMs and access to them via SSH/RDP is unaffected and the GPNorth VMWare web client service is accessible
Resolution: 31 Jan 2017, 12:34 PM
Automatic resolution.

OLA Release 5.1.0 - Move OLA Production to Amazon Relational Database Service (RDS)

Resolved
Service affected: Online Applications - OLA
Start: 31 Jan 2017, 7:00 AM
End: 31 Jan 2017, 9:00 AM
Login to request notification
This outage will allow OLA Production and Staging to be moved to Amazon Relational Database Service (RDS). A small number of feature enhancements and bug fixes will be included in the release.
Resolution: 31 Jan 2017, 9:00 AM
Automatic resolution.

Server logins failing

Resolved
Service affected: SQL 2008R2 Clustered Database Hosting
Start: 30 Jan 2017, 10:20 AM
End: 30 Jan 2017, 12:51 PM
Login to request notification
A login trigger has malfunctioned causing logins to the database server to fail.

ITS staff are working to resolve the issue as soon as possible and apologise for the inconvenience.
Resolution: 30 Jan 2017, 12:51 PM
All SQL login services hosted on the legacy SQL 2008 failover cluster have now been returned to normal. A resolution was enacted via remote vendor support.
Update: 30 Jan 2017, 12:25 PM
ITS have engaged Microsoft Support engineers to assist with the issue. Investigations by the Microsoft engineers are currently underway.

UniFi outage for appserver refresh

Resolved
Service affected: UniFi
Start: 29 Jan 2017, 6:15 AM
End: 29 Jan 2017, 6:30 AM
Login to request notification
UniFi web servers need to be restarted to pick up new app server configuration. This will only impact user connections to UniFi, it will not impact batch processes
Resolution: 29 Jan 2017, 6:30 AM
Automatic resolution.

SI-net outage for appserver refresh

Resolved
Service affected: SI-net
Start: 29 Jan 2017, 6:00 AM
End: 29 Jan 2017, 6:15 AM
Login to request notification
SI-net web servers need to be restarted to pick up new app server configuration
Resolution: 29 Jan 2017, 6:15 AM
Automatic resolution.

Switchover IpAlloc Database

Resolved
Service affected: Oracle Tier 3 DB Hosting
Start: 28 Jan 2017, 5:45 AM
End: 28 Jan 2017, 6:00 AM
Login to request notification
Switchover IpAlloc Database from crackledev02 to snapdev01
Resolution: 28 Jan 2017, 6:00 AM
Automatic resolution.