Ever Walked in to a Data Centre and thought?
“Wow this looks impressive!”
– Data Centre Engineer
or
“How have they managed to hold that server in, with a piece of string?”
– Data Centre Engineer
or
“How does that cabinet stand up with all that hardware hanging out of it”?
– Data Centre Engineer
I know which feeling I prefer, when I walk in to a data centre…
Nobody likes to see messy patch panels, servers without labels, fibres stretched between cabinets, hard drives, memory, just laying around.
Maybe you have experienced this feeling too?
Is your data centre a chaotic entity spiralling out of control?
Or is it a finely tuned engine purring, ready for action at the heart of your organisation?
Either way, this post is intended to act as guidance for any new build, refurbishment or minor works that is carried out in a data centre.
It is impossible to provide for every situation in this post; but it should cover most frequently asked questions and related issues expected to arise in most data centre related project work.
Adherence to this post should ensure potential problems are minimised and handovers completed in a straightforward and timely fashion in any data centre environment.
This information can be used by all for the good of the data centre.
So lets get started…
Copper Data Cabling
- The internal copper cabling is to be a Category 6a U/FTP installation conforming to ISO/IEC.
- All installed cabling must be supplied with a minimum warranty of 15 years covering the cable work.
- The cable must be low smoke zero halogen sheathed.
- Cabling should adhere to the ACMTIX LTD colour code policy. BLUE = DATA / ORANGE = VOICE / PURPLE = ILO.
- Correct lengths of cable should be procured to keep communications cabinets tidy.
- Cables should be run carefully to the right or left of an installed switch /patch panel as per diagram below utilising cable management where possible.
- The components to be used must be complete end-to-end cabling systems with associated components.
- All cabling installation work must be carried out by qualified and OE Vendor approved/trained installation engineers.
- All internal cable routes must be agreed prior to commencing work with IT Services, and must only radiate from the agreed/specified Communications Rooms.
- The framework of the system must incorporate industry standard RJ45 plug and socket presentation and IDC cable terminations.
- The cable should be a U/FTP to a minimum performance level of Category 6a.
- The cable, patch panels, keystone jacks, modules and associated fixtures and fittings must be supplied by and/or approved by the Cable System OE Manufacturer thus ensuring that a full system guarantee can be issued.
- The length of cable from patch panel to room outlet will not exceed 90 metres.
- All cables shall be continuous from the user patch panel to the work area telecommunications outlet.
- Where a cable becomes damaged due to broken conductors or the sheath becomes torn or cut, the entire cable shall be removed and replaced with a known good one. The practice of jointing cables shall not be accepted.
- A plastic cable tie will secure the cable to the termination block.
- Cables will be secured together in groups of no more than 48 by Velcro straps to avoid cable damage.
- In special cases, the general cabling specification may be deviated from when agreed in writing from the Director of IT, Head of IT or Enterprise Services Manager:
- Where a small number of additional data points are required, and to enable integration with existing patch panels or installed infrastructure (for example, installing 2x category 5e data points in a part populated cat 5e panel)
Room Data Outlets
The data outlets will be unshielded RJ45 presentation and modular in design wired to 258A (T568B) presentation.
All Data Outlets must be fitted in pairs (Dual Outlets) on walls, not in the floor. Floor boxes are difficult to access and subject to risk of damage from water, insects and people. Wall mounted data outlets are much for durable and much more accessible.
A spring-loaded shutter must cover the entrance to the RJ45 jack.
The outlet printed circuit board (PCB) and the patch panel PCB must utilise PCBs from the OE Manufacturer to ensure complete system compatibility.
Port labelling and identification should be by way of a printed slide label running under a protective polycarbonate mask.
Room outlet accessories should match the general electrical installation.
Any work that involves the modification, re-installation or relocation of a room outlet requires retesting and re-certification to be completed.
Patch Panels
A 96 Port patch panel. The Patching Panels must be cable OE Manufacturer approved panels, managed in groups of 24 RJ45 ports in each 1U segment.
Each circuit should be individual modular PCB, wired to 258A (T568B) specification.
Label identification should be by way of a printed slide label running under a protective polycarbonate mask.
Cables should enter the patch panel from the side.
Cable Conduits and Containment
The cable containment system will have usable space (BS EN 50174-1:2000, Section 4.8.2) that allows for double the initial quantity of cables.
Labelling Data Outlets
Data Outlets will be labelled as follows (each part separated by a “/”):
[Part A]/[Part B]/[Part C]
Where:
- Part A Comprises the floor (G = Ground, 1 = 1st Floor etc.)
- Part B Comprises the terminating Cabinet Number (supplied by ACMTIX LTD)
- Part C Comprises a 3 digit serial number (001-999)
All Data Labels must comprise laser cut, black text on a white background, rectangles protected by a perspex window.
All Patch Panels must be labelled in a corresponding fashion in ascending numerical order.
Data points should be labelled in a consecutive and logical manner (e.g. data points 25-48 in Room ‘B’ should follow data points 1-24 in the adjacent Room ‘A’ etc.)
Testing
All test equipment used to ensure the performance of the installation shall be approved by the cabling system manufacturer, prior to system testing.
Prior to system hand-over, a 100% test shall have been conducted on the installed cabling.
The testing shall identify any faults due to open circuits, cross or split pairs as well as a series of performance faults.
The testing must satisfy the manufacturer’s installation guidelines in order to carry the manufacturer backed warranty. In some cases a representative of the manufacturer may be required to witness the testing.
ACMTIX LTD reserves the right to witness some of the testing.
All test results shall be recorded and certification handed to ACMTIX LTD on completion. Any cabling failures will require resolution of the fault and retesting before completion.
Change Approval will not be granted to connect untested Cable Installations to the network.
Connected Equipment
Unless otherwise agreed with IT Services, equipment shall not be connected to any part of the ACMTIX LTD network until it has been properly registered in the CMDB, an IP address and a hostname has been allocated, and the relevant switch port has been patched and configured.
Equipment shall be labelled with the hostname allocated by ACMTIX LTD IT Services, and this name must be included in any communication with ACMTIX LTD.
If any equipment is relocated or replaced, ACMTIX LTD IT Services must be informed of the change of location or MAC address and the new or relocated equipment shall not be reconnected until authorised.
Fibre Optic Data Cabling
ACMTIX LTD recommends that all buildings are to be connected to two other identified buildings by diverse cable routes.
Fibre optic cables that span outdoor spaces will follow a duct route to be identified by ACMTIX LTD.
All buildings require single mode OS2 (9/125) fibre cores. A minimum of twenty-four cores (12 pairs) of each type of fibre are required to each connected building.
Fibre routes should be kept separate as far as practically possible within and without the building as well as within the communications room.
ACMTIX LTD infrastructure utilises both traditional multi-core fibre as well as blown fibre varying by location. The project must identify with the aid of IT Services what sort of fibre installation will be suitable.
All fibres will be installed in one continuous length without any intermediary joints. Any cable faults found on installation will require replacement of the faulty cable.
Fibre Patch Panels
All fibre optic cores are to be terminated in 19” rack mount patch panels.
The fibre bundle will enter the patch panel through a cable gland at the rear of the panel.
The patch panel will be equipped with Duplex LC connectors. The LC connectors will be colour coded such that single mode connectors are white and multimode connectors are aqua.
Fibre Cable Containment
The most suitable cable containment method must be indicated before work is undertaken. Normally this would be basket, or tray work for large amounts of cable but pvc may be suitable for small amounts of cable.
Tray work or basket must be initially installed with no more than 75% of the capacity used to allow for expansion in future.
Existing tray work along the cable path can be reused. If the existing containment is full then IT Services should be consulted in order to determine whether or not the existing containment should be replaced or supplemented.
Fibre Labelling
All fibre tubes should be labelled at each building entry point with their fibre number and the names of the buildings at each termination point.
All Fibre Cables installed in underground service ducting must have clear labelling affixed to the cable at every inspection chamber.
All Fibre Panels must be labelled according to the following label scheme:
- [Part A]-[Part B]-[Part C]-[Part D]-[Part E] For example [DBY-C01-MAC-C01-SM01]
- Part A Comprising the ‘A’ End Terminus Building, Abbreviated as listed below
- Part B Comprising the ‘A’ End Cabinet/Rack Identifier
- Part C Comprising the ‘B’ End Terminus Building, Abbreviated as listed below
- Part D Comprising the ‘B’ End Cabinet/Rack Identifier
- Part E Comprising a unique cable run identifier combined with the cable type (SM = Single Mode, MM = Multi-Mode)
- Building Abbreviations (Depending upon project).
ACMTIX LTD may elect to rename buildings from time to time, as a consequence, some legacy labelling such as the list below may be in use:
Fibre Testing
Post installation a random selection of cable containment will be visually inspected to ensure that the provided cable containment meets any agreed standards and principles.
Fibre Patch Leads
All patch leads for the project will be supplied by ACMTIX LTD. Patch leads from any other supplier will not be used. Patch leads should adhere to the ACMTIX LTD colour code and a suitable length should be used.
Fibre Cable Containment
External ducts for Data services between buildings shall be twin walled rigid duct, minimum 100mm diameter, externally ribbed with a smooth interior.
Manhole covers over splicing or access chambers shall be robust and suitable for the chamber environment (e.g. appropriate load and security ratings where required)
Access chambers for data ducts shall be independent of access chambers for other services, such as power, heating, BMS systems, water, control cabling, etc.
All ducts shall be provided with draw ropes to permit pulling of cables at a later date.
Data Ducts must not be used for any other purpose (e.g. LV/HV Electrical or other services)
As-Built Drawings for ducts and Fibre routes must be provided to both ACMTIX LTD IT Services and Estates electronically in a suitable format.
BT Openreach Telco Ducts should be requested and provided by the Primary Contractor where the building use or future use may require the installation of POTS connected Fibre or Copper networks. Contractors are encouraged to read the “BT Openreach Developers Guide”
Communications Rooms
Cabinet Space
Where more than one cabinet is installed in a single comms room the cabinets can be joined together at the side to form a row.
There should always be one metre of clear space around each side and to the rear of each rack, or row of racks to allow for people and installation/maintenance of equipment. Any protrusions in the room should be deducted from this allowance. For smaller rooms please see the diagrams below in the cabinets section.
There should always be one point two metres of clear space to the front of each rack, or row of racks. Any protrusions in the room should be deducted from this allowance.
Lighting
Walls should be painted white for maximum light.
Lighting shall be five hundred lux in the horizontal plane and two hundred lux in the vertical plane, measured one metre (three ft) above the finished floor in the middle of all aisles between cabinets.
Lighting should be between 3000-5000 Kelvin to provide a clear lighting area around the data equipment.
Sufficient lighting must be present in the room to illuminate every side of the racks. If motion detection is used to control the lights the sensor must be able to detect a single person in any position in the room.
Electrical & Cooling
The electronic equipment in all communications rooms and cabinets should be provided with an in-line, pure sine wave uninterruptible power supply, rated to provide a minimum autonomy of thirty minutes. This may be achieved either through the use of a room based UPS or individual rack-mounted UPS Units.
Each cabinet within the comms room shall be provided with two dedicated 16A BS4343/IEC309 outlet. These need to be switched/interlocked type and a suitable plug supplied with each socket installed. Each socket must be on a dedicated radial socket. Also, where these socket outlets are installed beneath raised floors, the socket outlet must be fully accessible once the data cabinet is in place. Exceptions may be made in small installations to use one or more 13A BS1363 outlet instead under agreement with IT Services.
RCD/RCBO’s must be rated ‘Type C’ or higher, to tolerate high inrush current loads after any power outage.
Power outlets should either be mounted immediately above the rack in busbars supplied by two independent supplies at a height not exceeding 2500mm or on a nearby wall.
Supplies must range from 3KW to 20 KW for server and storage infrastructure. If the power outlet is wall mounted it must be possible to feed suitable cables from the rack mount PDU to this outlet in such a way that the cabinet can still be fully closed and the cable not present a trip hazard.
Each cabinet must be individually earth bonded in accordance with BS7671:1992.
Each cabinet must be fitted with a suitable 16A input APC switched PDU capable of being managed by Ethernet. In a 48U floor standing rack this should be a ZeroU PDU vertically mounted at the rear of the rack with C13 outlets. In a wall mounted rack this should be a 1U PDU mounted horizontally at the front of the rack with C13 outlets.
The communications room should be supplied with sufficient cooling to accommodate thermal loads of 1KW per rack, suitable for Industrial Use (capable of self-recovery after a power failure etc) and fault tolerant to N+1.
Communications rooms must be kept between 21-24 Celsius and 20-80% Relative Humidity. Temperatures must be kept stable at their set point throughout all seasons and all weather conditions.
Rooms should comply with EU code of conduct airflow bundled with air dams, in fill plates and 1U blanking panels as standard.
All cabinets and rooms should be fitted with SNMP temperature and humidity sensors connected to a suitable monitoring system (Solarwinds) to ensure that the environment does not exceed thresholds for data/comms rooms.
You don’t want to end up in The Upside Down…
Data Centre and Communications Rooms Access
These locations will not be shared with staff outside of IT.
With the exception of a small quantity of cold spares directly related to the operation of Network equipment in these rooms (such as for example spare patch leads) these locations must not be used for the general storage of materials.
Cabinets
The ‘Prism’ ranges of cabinets are preferred but others may be considered by approval of ACMTIX LTD IT services.
Floor standing 48U cabinets are much preferred however it is understood that this is not always possible or practical. In such situations a smaller floor standing cabinet is preferred to a wall mounted installation.
No more than three hundred and thirty six room outlets shall be terminated in a single 42U rack without explicit permission from IT Services.
The cabinet will have 19” rack mounting PDU strips to both the front and rear.
The front mounting strip will be recessed at least 150mm from the front door.
Where more than one cabinet is in any one location, all cabinets shall be connected together side by side using the manufacturers baying kit. All internal sides shall be removed.
Cable tray/basket will be mounted inside the rack and to the side for the full height of the rack. A clearance space in the centre of the rack of 450mm must be maintained to allow fitting of equipment.
Where a floor standing cabinet is used:
- The cabinet must be at least 800mm x 800mm.
- The room should be big enough to open the side panels on the cabinet and load equipment in and out of the cabinet with ease.
- The cabinet should ideally be at least 1000mm away from the wall to allow for wall mounted services but this isn’t always possible so make sure to leave enough space.
Where a wall mounted cabinet is used:
- The depth of the cabinet must be at least 800mm.
- The cabinet must either hinge away from the wall for access or allow the side panels to be removed.
- Removable side panels must be able to be locked.
- The cabinet must not be mounted with the top of the cabinet at a point higher than 2.1m above floor height.
- The cabinet must have adequate ventilation and cooling.
Typical Communications Cabinet Layout:
- 1 blank space for rack top fan
- 2 OS2 LC patch panel
- 3 OM3 LC patch panel
- 4 cable management bar
- 5 building aggregator switch
- 6 building aggregator switch
- 7 cable management bar
- 8 UTP patch panel
- 9 cable management bar
- 10 access switch
- 11 cable management bar
- 12 UTP patch panel
- 13 UTP patch panel
- 14 cable management bar
- 15 access switch
- 16 cable management bar
- 17 UTP patch panel
- 18 UTP patch panel
- 19 cable management bar
- 20 access switch
- 21 cable management bar
- 22 UTP patch panel
- 23 UTP patch panel
- 24 cable management bar
- 25 access switch
- 26 cable management bar
- 27 UTP patch panel
- 28 UTP patch panel
- 29 cable management bar
- 30 access switch
- 31 cable management bar
- 32 UTP patch panel
- 33 UTP patch panel
- 34 cable management bar
- 35 access switch
- 36 cable management bar
- 37 UTP patch panel
- 38 UTP patch panel
- 39 cable management bar
- 40 access switch
- 41 cable management bar
- 42 UTP patch panel
Data Centres
The definition of a data centre room is where data is stored and held in some form rather than a communications room where data is in transit.
If an IT room has a server in it should be considered a data centre.
Wireless
ACMTIX LTD strategy for wireless networking is to implement ubiquitous wireless access in all buildings. This means all buildings must have complete coverage in the interior and, where identified, at the exterior of the building.
IT Services maintains delegated responsibility for the use of 2.4 and 5 GHz radio spectrum on all ACMTIX LTD sites.
The Installation and Operation of RF emitting devices on ACMTIX LTD’s network is subject to standard ACMTIX LTD change control with the exception of standard consumer hardware, including:
Wi-Fi Client Devices
Mobile Phones operating on OFCOM approved frequencies;
The operation of Wireless Access Points, Routers and Premicell devices are expressly forbidden on the ACMTIX LTD, with the following exceptions:
In exceptional circumstances by prior agreement and on approval of a Change Request by ACMTIX LTD;
Where the device in question is broadcasting a Wi-Fi Personal Hotspot that is (i) not connected to any ACMTIX LTD network or equipment and clearly intended for personal use, (ii) does not broadcast an SSID in use by the ACMTIX LTD or one of its tenants, or an SSID constructed to deceive or otherwise fraudulently attract client connections and (iii) is operating for a short duration and on a temporary basis;
IT Services Monitors ACMTIX LTD Buildings for unauthorised (commonly referred to as ‘rogue’) Wireless Networks. Where these networks are found, The ACMTIX LTD reserves the right to undertake lawful measures necessary to block or remove the rogue device(s).
IT services are solely responsible for the procurement of the relevant wireless network equipment.
Third party supplier’s equipment must not be used. This will be at cost to the project.
All refurbishments and alterations affecting the layout or altering the use of an existing interior space must be subjected to a wireless survey by IT Services and, where required, wireless infrastructure alterations are provisioned for within the project. e.g.
The erection or demolition of partition walls of any material (including glass)
The change of use of a space.
The installation of substantial furniture or equipment (for example, bookshelves or racking) that may affect signal propagation
Project teams must supply a suitable floor layout plan to ACMTIX LTD. ACMTIX LTD will then return a labelled plan showing where data sockets should be installed to connect the wireless access point. We prefer the data point to be installed in a discrete/unseen location but must be accessible. The contractor must provide a wire-way (conduit) from this point to the location of the wireless access point. This will accommodate the patch lead to connect the wireless access point to the data socket. This process should be iterative between the project and IT Services to identify unsuitable or improved locations for the access point. This may be due to building materials, cable containment or similar. The fitting of the wireless equipment will be responsible by the project.
Access points should be installed on a horizontal plane for best performance. Installations on a vertical plane will only be considered in consultation with ACMTIX LTD IT Services. This may be because the ceiling is above fixed furniture or equipment which would hinder access for maintenance or installation, or it may be that the ceiling is too high for access with ladders.
Wireless access points can be installed on false ceiling grid systems, plasterboard or concrete using suitable fixings
Network Switches
ACMTIX LTD IT Services will be solely responsible for the procurement and fitting of network switches within ACMTIX LTD buildings. Network switches or any other active network equipment provided by third parties will not be used. ACMTIX LTD prefers CISCO equipment.
ACMTIX LTD does not permit the connection of unmanaged of Bridges/Range Extenders, or any other technology that forwards or retransmits network traffic.
Project Management
It is important to establish an early and clear dialogue with ACMTIX LTD on projects that have any sort of IT impact. Failure to do so may result in delays to the project or incorrect installation which will not be accepted. The standards below have been developed with reference to Sector Best Practices.
Communication
Larger projects should be planned and provisioned via a project engagement form.
Smaller projects should contact the ACMTIX IT Service team in order that the request is directed to the correct person.
All changes should be logged and agreed via change request forms and communicated via the 5W’s process to agreed communications group members on the project engagement form.
Documentation
IT Services should be issued with the following documentation:
- Floor layouts plans along with a general map of the site.
- Communication room locations and layouts.
- Cabling containment route plans.
- Number of data outlets terminated at each comms room.
- List of all building network connected services (e.g. BMS, intruder alarms etc).
Costs and Budgets
The project is responsible for funding all IT costs. This includes but is not limited to:
- Construction work to connect into the data duct system.
- Supporting Power Systems (UPS) and Cooling.
- Installation of specified fibre optic cables.
- Installation of internal building fibre and copper cabling.
- Purchase and installation of data cabinets.
- Purchase and installation of power distribution units within each rack Purchase of active networking equipment (switches, wifi and optics) Purchase of patch cabling.
- Purchase of network installation equipment.
- ACMTIX LTD must be consulted during any budget forecasting and costing exercises for IT Infrastructure requirements within any proposed projects. Any budget or forecast that does not have the express, written sign-off of the ACMTIX LTD should be considered invalid.
- ACMTIX LTD provides 3 levels of costing for the purposes of budget forecasting and financial planning; Preliminary, Budgetary and Firm costs. All Estimates and Costs are only valid for the agreed use-cases and must be read in conjunction with the assumptions supplied by ACMTIX LTD.
- Note that external IT specialists are generally required to provide costings and supporting elements on large or complex projects, such as High or Low Level Design, Architecture and Specifications. Consequently these have attendant costs that must be met by the project budget.
ACMTIX LTD produces 3 different types of cost:
Preliminary Estimates
An estimate based on high-level guideline industry parameters and/or cost modelling techniques. This value should only be used in circumstances where a project or proposal is being considered for financial viability. Information from the business on proposed use-cases and other low level information vital to establish accurate costs is not available at this stage, and this information should not be used to set the final budget for any IT Infrastructure spend. Margin of error should be considered +/-30% and valid for 12 months.
Budgetary Estimates
An estimate based on information supplied (or an agreed set of assumptions) by the business on proposed use-cases, used to determine budget levels. Some High-Level Design will have been completed to support this stage of the project, and budgetary quotes obtained from suppliers. Margin of error should be considered +/-15% and valid for 6 months.
Firm Costs
Firm costs are based on Low-Level Designs for IT Infrastructure, and supporting Plans or As-Built Documentation to determine accurate costings. Firm quotes will be obtained from suppliers and will be valid for 30 days. The margin of error is generally +/- 10%.
Thanks for reading… if you have any further questions or have a requirement that ACMTIX LTD can help you with in your data centre, get in touch in the comments below.
Leave A Comment