Musings of Rodos

There are a lot of things to write up from my UCS bootcamp last week. Over time I will post up many of the interesting bits and pieces. I thought I would start with something interesting, the indicator LEDs because everyone loves flashing lights.

So here you go, a short video of the chassis and blade locators turned on.



Now that we have your attention (sorry, that's a bad pun) a bit more details.

UCS has an indicator light on the top right of each chassis, shown below. Note the bit of masking tape which during shipping covers the button to protect it and prevent it snagging on things, very nice touch Cisco!

Very interesting to see Steve Herrod, CTO at VMware put out the announcement and details today of VMware's intention to acquire SpringSource.

You can read the details of SpringSource in the announcement as well as at the SpringSource website. Also some good cloud thinkers like Reuven have shared their thoughts on this acquisition.

Eight months ago I wrote about where VMware was playing in the various stacks of Cloud, Saas, Paas, and IaaS. I wrote how VMware fits into the SaaS through the virtual appliance market place and encouraging ISVs to create their wares as virtual machines for the Enterprise to run internally. The play for IaaS is obvious with what was then call the VDC-OS, now vSphere, being the OS for the Cloud along with vCloud.

PaaS was the interesting one. Here is what I wrote way back then.

The PaaS space is where VMware has the challenge. What VMware want here is everyone to build their systems on open standards. This allows them to be packaged into IaaS workloads that can be run anywhere (internally or externally) without lockin to propriety providers, such as Microsoft Azure. An example here is writing in Ruby and deploying in Ruby On Rails.

With the purchase of SpringSource VMware have jumped boots and all into the PaaS space, filling a hole in their amour. They have gone beyond my view of Open Standards and decided that they want to control the market more, drive investment to create competition to what the other PaaS players are providing. I guess you can do that when you have big pockets, forget waiting for the market, make the market.

Why was I so keen on PaaS being Open Source? Its the value of being able to execute a workload where you want it, choice. With SpringSource you can develop software and decide where you want to execute it. Internal or External, thats choice. With the PaaS stack being more open and available, even if it costs some form of licensing fee for a provider to offer the platform, means a breadth of available providers, each adding their own differentation. Compare that to Microsoft Azure, run it in a limited set of cloud locations with a single provider (who may just up and move all of your data). Microsoft have been consistant in their view the Azure is for Microsoft datacenters only (see my thoughts on this from March).

I think this is a really exciting development. Its going to take a lot of time, its not going to save the world. However over the longer term Enterprises do need to look at how they move above the IaaS and below the SaaS (the low hanging fruits) and look at how they build their applications. Yesterday a strong force in that space was Microsoft Visual Studio and Azure. Who knows, tomorrow it may be technologies from SpringSource, through VMware backing, that runs on hundreds of Cloud providers.

Exciting times and it will be great to see and discuss this more with people at VMworld. See you there.

Rodos

Well, last day of the UCS bootcamp today. Tomorrow is a day full of labs. Its been great going through it all with some smart people to bounce things off, thanks guys!

At the end of the day I redrew by own version of how UCS hangs together that summarises a lot of the diagrams in the course. It makes it all a lot simpler. The main area of simplifcation is for what occurs inside the IOM inside the Chassis. This layout is something you could easily remember and whiteboard in front of someone. I doubt you could remember the ones in the course notes.

Here is the sketch I whiteboarded.

When I get some time I will put it into Viso and annotate it. However everyone in the class grabbed their cameras so some will try and beat me to it. I hope they do, will save me the effort.

I would like to add a little table that shows the pinning of the IO ports from the MUX to the blades dependent on the number of uplinks (1,2,4).

Hope you find it helpful.



For those UCS architects out there, if you notice any mistakes (we went through it quite a bit to make sure it was right) post in the comments.

Rodos

[UPDATE
Our instructor David Chapman drew this up in Visio this afternoon. We have a few changes but this is a long way along.]

[UPDATE 2 22/Sept/09
Now has the blade details. Need to make it look nice for Networkers next week]

There are a lot of terms to come to grip with when learning and talking about UCS. So many new TLAs. Here is a Dictionary to fast track your learning and give you something to quickly lookup.

• AG – Application Gateway
• BMC – Baseboard Management controller (on blade)
• Chassis - The enclosure (5108) that contains the one or two IOMs and the blades.
• CMC – Chassis Management Controller (in the IOM)
• CMS - Chassis Management Switch (in the IOM)
• CNA - Converged Network Adapter
• CLP - Command Line Protocol
• CMP – Connectivity Management Processor
• DCOS - Data Center Operating Center
• DCOS System Manager - starts and manages the processes in the UCSM Controller on both primary and secondary.
• DME – Database Management Engine (Primary Active)
• Expansion Module - Provides connectivity into the Northbound Ethernet LAN and FC SAN networks. There is a FC only, Ethernet only or a combo module available.
• FEX - Fabric Extender (now referred to as the IO Module or IOM)
• Fabric A/B -
• Fabric Interconnect - The head unit or switch which the Chassis all connect into. The UCSM runs within the Fabric Interconnect.
• FSM - Finite State Machine, logical abstractions in the information model for hardware, software and workflows
• GEM - General Expansion Module
• Group - A set of two Fabric Interconnects. Individually called a node and each is identified by a unique ID.
• I2C - Inter-Integrated Circuit (BMC)
• IOM - I/O Module (FEX)
• IO Mux - A part of the IOM which multiplexes the IO ports and BMC from the blades (8) along with the the CMS and CMC from the IOM into the North bound ports (1,2 or 4) going to the Fabric Interconnect.
• LACP - Link Aggregation Protocol
• Menlo - Code name for one of the Mezz adapters. A CNA with 2 10GE and 2 FC ports.
• Mezz - Mezzanine card within a blade that provides the IO interface to the IOM
• Mid-Plane - The element within the Chassis that the blades connect into to obtain power and IO connectivity as well as access to the SEEPROM
• MO - Managed Object
• Node - A node is one of your two Fabric Interconnects. The set of two are called a group. Each node is identified by a unique ID.
• NXOS - Nexus Operating System
• Oplin - Code name for one of the Mezz adapters. 2 10GE ports, no failover functionality
• Palo - Code name for one of the Mezz adapters. Not released yet. Provides up to 128 Ethernet vNICs or FC vHBAs (limits apply)
• Pin Group - If you are doing manual pinning of Northbound traffic (rather than round robin) through uplinks you can create Pin Groups which you can apply to multiple service profiles. So you are chosing which uplink or port channel will be used for the selected interface (vNIC or pNIC).
• PNuOS – Processor Node Utility Operating System, serves as a "pre-OS configuration agent" for the blade, now named UCSuOS.
• SEEPROM - Serial EEPROM, contain in the mid-plane, half of which is maintained by each IOM in the Chassis.
• SEL - System Event Log (BMC)
• Service Profile - represents a logical server and will be associated with one blade at a time. Contains Identity/Personality, LAN/SAN config and various policies
• SoL - Serial Over LAN
• Stateless Computing -
• UCSM - UCS Manager, the management service for all of the UCS components which runs on the Fabric Interconnects.
• UCSM Controller - a distributed application running on both th primary and subordinate UCSM and decides which is primary and which is subordinate.
• UCSuOS - UCS Utility Operating System, serves as a "pre-OS configuration agent" for the blade, previously named PNuOS.
• UUID - A 128-bit number to uniquely identify a component worldwide.
• vNIC - a software configured NIC which is presented to the OS as a pNIC on the blade.
• VNTag - A virtual network tag.
• vHBA - a software configured HBA which is presented to the OS as a pHBA on the blade.

Like Scott Lowe I am doing the Cisco UCS bootcamp. Scott did a nice writeup of his notes. I will also write up my notes but here is a summary of something that has been a bit confusing, so I wanted to put out some clarity.

There are the three types of adapter cards for the UCS blade. A Oplin, Menlo or Palo. The Oplin has no FC ports, it only provides two 10GE.

The Menlo card provides two FC and two 10GE and is currently shipping. The details of the card are :

The Menlo adapter is a converged network adapter (CNA) with 2 host-side 10GE ports and 2 FC ports to the backplane. The 2 network ports can run either native Ethernet or FCoE protocols and can be configured for failover. This failover is performed by the Menlo ASIC and does not require multipathing sofrware on the host. The Menlo ASIC is a Cisco-designed multiplexor and FCoE protocol offload engine with a 350MHz 24K MIPS processor. There are 2 versions of this card: Menlo-E (with an Emulex chipset) and Menlo-Q (with a Qlogic chipset), thereby supporting existing proven driver stacks to the customer.

The interesting thing here is the statement about failover. On your first reading you would think that there is some great functionality here. People have been sprouting this failover without any diving into what it really means.

To cut to the chase, this is for the vNICs only and NOT for the vHBAs. A lot of the documentation is not that clear and its easy to just assume it works for all virtual interface types.

You can see it on the dialog for creating a vNIC for a server. Notice the "Enable Failover" check box. Its actually disabled in this shot because the lab environment I am connected to does not have redundant Fabric-Interconnects and IO Modules. If you enable this feature only one two vNICs will appear to the OS and the Menlo card will handle the failover internally, routing the failed vNIC via a path on the other IOM.



When you have a look at the dialog for creating a vHBA you can see there is no redundancy option.



For some further details the Frequently Asked Questions for UCS - Ciscowiki has an the following.

Q: If the fabric extender is connected to fabric interconnect using 4 links and one of the links fail, what will happen?
Ans: The server interfaces that are affected will either lose connectivity or fail over to another fabric extender, depending if interface is created as a HA interface. Menlo has the capability to fail over Ethernet interfaces if so configured. Oplin does not have this capability. Fibre Channel interfaces that are pinned to failing fabric extender link will just fail and their HA capability depends purely on host side multipathing driver. If HA/mutipathing is not configured for Ethernet/Fibre Channel then servers connected to the failed link will loose connectivity but the other three links will be working as usual. Remember that no automatic re-pinning will happen. You can manually re-pin the servers using two link topology, since three link topology is not supported.

Also note that to restore connectivity you need to re-pin the IO Module that had a failed link. To do this you need to reset it. From the CLI it would look like this.

carrot-A# scope chassis 1
carrot-A /chassis # scope iom 1
carrot-A /chassis/iom # where
Mode:   /chassis/iom
Mode Data:
       scope chassis 1
       scope iom 1
carrot-A /chassis/iom # reset
carrot-A /chassis/iom* # commit
carrot-A /chassis/iom #

However this is UCS and the GUI for UCSM is just as powerful as the CLI, so here is where you do it in the GUI.



Of course smart people like Brad Hedlund get this already, he wrote

The vHBA’s do not support the UCS fault tolerant feature and therefore a standard HBA multi-pathing configuration is still required in the operating system (ESX kernel) for Fibre Channel high availability.

Rodos

P.S. Thanks to everyone in the class today who put up with me asking endless questions about this, to Glenn from Cisco in Australia who answered some stuff over chat and our Instructor David. Any errors are all mine, post in the comments if you have any fixes or further insights.

[UPDATE : Brad Hedlund and I had a conversation over Twitter about how the redundant Ethernet is presented to the blade and how the fail over occurs. Turns out it is slightly different and better than everyone thought, but even Brad had to do some digging to be 100% sure. Thanks Brad!]