Monday, October 24, 2011

Cisco MDS switch interoperability with Brocade Access Gateway and Qlogic Adapters

During a recent SAN switch migration from McData directors to Cisco MDS 9509 switches, I came across two interoperability issues. The first issue involves certain Qlogic adapters connected to the Cisco MDS switches where the area portion of the Fiber Channel ID (FCID) is the same for the initiator and target. The second issue has to do with internal routing in the Brocade 4Gb switches running in "Access Gateway" mode. In my case the Brocades were embedded in blade chassis, but they could be stand alone. Note, the 8Gb version of the Brocade switches did not exhibit the routing issue.

Common to both issues are particular FCID combinations. The FCID is comprised of 3 bytes, 0xDDAAPP, where the letters stand for Domain, Area and Port numbers. The domain portion is unique between switches or VSANs in a fabric, but the same for all ports on a particular switch or VSAN. The area and port numbers are generated by the switch for each WWPN that logs into the switch. Each area can have 256 ports. Cisco begins with area 00 and sequentially assigns port numbers.

The first problem occurs with particular Qlogic adapters if the area portion is the same for the HBA and a zoned target. In this case, the adapter will not recognize the target. Cisco rectifies this issue by assigning a unique area for all Qlogic adapters that exhibit this issue. In our case, the WWPN begins with 21:00:00:1b:32… Each manufacturer device that is identified in the switches Organizationally Unique Identifier (OUI) table will be allocated a unique area for each WWPN. Each of the affected Qlogic adapters receives a unique area and the port number 0x00. So the area will never be the same as the target, but the port number portion will always be 0x00, which leads to the second issue.

First, the 4Gb Brocade switches in Access Gateway mode will not route to a WWPN with the port portion of the FCID being 0x00. Second it will not route to two or more ports with the same port number on the same switch. So Cisco's solution to the first issue is not compatible with routing on these switches. Brocade outlines 4 work arounds for the combined issue. Each recommendation depends on your configuration. The first two options are only available if you don't and will not have target devices connected to the Cisco MDS switch in question. If this is the case, you can undo Cisco's fix by deleting specific entries from the OUI table or putting the switch in flat-fcid allocation mode, which essentially ignores the table. It is probably best to avoid these two work a rounds in order to prevent future problems.

The third option is to basically swap the OUI entries of HBAs and targets. In other words, remove the HBA OUI IDs and add the target device OUI IDs. The HBAs will get sequential port numbers in the same area and the target devices will get unique area IDs resolving both issues. The overall problem with this solution is that the devices need to be connected after the OUI table manipulation. Devices connected before the table update will already have FCIDs assigned. If persistent FCID allocation is enabled, the FCIDs will not change without disconnecting the WWPN and manually deleting them. Otherwise, the VSAN needs to be restarted.

The last option is the option that we chose to implement. Basically, we manually assigned FCIDs to each of the affected HBAs connected via the Brocade switches in Access Gateway mode. We kept track of manually assigned FCIDs in a spreadsheet to ensure that no port numbers were duplicated on a switch. Actually, the switch will not allow you to assign duplicate FCIDs. An FCID is allocated the first time the HBA logs into the switch. The static FCID assignment can be performed in advance if the WWPNs for each HBA are known prior to connection. Otherwise, the host needs to be shutdown so that the FCID can be deleted and the static assignment performed. In my case, after the hosts were shutdown, I had to shut/no shut the interface to release the FCID. Prior, we did verify multipathing on other hosts accessing storage via the same FC port.

In the example below, first I list the free and allocated FCIDs in the VSAN in order to choose a range to assign. Next I check the FCIDs in use on the port that my Qlogic HBA is connected. If they are all the affected Qlogic HBAs, I can tell if they are in use by whether the FCID has been manually assigned. Otherwise, I need to consult the active zoneset. If another server is actively using SAN storage, I check the multipath software on that server. Now, I check to see if the initiator is logged into the switch. In my example, it is not logged in. Next, I proceed to shut the port as if the FCID was in use. Finally, I can delete the existing FCID and add a new one. Afterward, I bring the port back up, verify the new FCID and save the configuration. At this point, I am free to move onto creating aliases and fcdomain address-allocation vsan 5


Free FCIDs: 0xc90028 to 0xc9002a

0xc90066 to 0xc900e3

0xc900e5 to 0xc900ee

0xc900f0 to 0xc900ff

0xc90200 to 0xc902e3

0xc902e5 to 0xc902ee

0xc902f0 to 0xc903ee

0xc903f0 to 0xc904ee

0xc904f0 to 0xc905ee

0xc905f0 to 0xc905ff

0xc90e00 to 0xc90e01

0xc90e03 to 0xc90eee

0xc90ef0 to 0xc90eff

0xc91400 to 0xc914ee

0xc914f0 to 0xc915ee

0xc915f0 to 0xc916ee

0xc916f0 to 0xc919ff

0xc91d00 to 0xc91dee

0xc91df0 to 0xc91eee

0xc91ef0 to 0xc92000


0xc9201c to 0xc920ff

0xc92800 to 0xc929ff

0xc92b00 to 0xc92bff

0xc92d00 to 0xc92eff

0xc93000 to 0xc930ff

0xc93a00 to 0xc93aff

0xc93c00 to 0xc93cff

0xc93f00 to 0xc93fff

0xc94400 to 0xc944ff

0xc94b00 to 0xc9fffe


Assigned FCIDs: 0xc90000 to 0xc90027

0xc9002b to 0xc90065



0xc90100 to 0xc901ff






0xc90600 to 0xc90dff



0xc90f00 to 0xc913ff




0xc91a00 to 0xc91cff




0xc92003 to 0xc9201b

0xc92100 to 0xc927ff

0xc92a00 to 0xc92aff

0xc92c00 to 0xc92cff

0xc92f00 to 0xc92fff

0xc93100 to 0xc939ff

0xc93b00 to 0xc93bff

0xc93d00 to 0xc93eff

0xc94000 to 0xc943ff

0xc94500 to 0xc94aff



Reserved FCIDs: 0xc9ffff



Number of free FCIDs: 52852

Number of assigned FCIDs: 12684

Number of reserved FCIDs: 1

MDS# show flogi database interface fc1/3




fc1/3 5 0xc90100 21:00:00:1b:32:06:f8:e7 20:00:00:1b:32:06:f8:e7

fc1/3 5 0xc92c00 21:00:00:1b:32:94:99:da 20:00:00:1b:32:94:99:da

Total number of flogi = 2.


MDS# show fcdomain fcid persistent unused vsan 5

Total entries 1.


Persistent FCIDs table contents:

VSAN WWN FCID Mask Used Assignment

---- ----------------------- -------- ----------- ---- ----------

5 21:00:00:1b:32:06:f8:e7 0xc90100 ENTIRE AREA NO DYNAMIC

MDS# conf t

MDS(config)# interface fc1/3

MDS(config-if)# shut

(config-fcid-db)# fcdomain fcid database

(config-fcid-db)# no vsan 5 wwn 21:00:00:1b:32:06:f8:e7 fcid 0xc90100

(config-fcid-db)# vsan 5 wwn 21:00:00:1b:32:06:f8:e7 fcid 0xc92002

(config-fcid-db)# interface fc1/3

MDS(config-if)# no shut

MDS(config-if)# exit

MDS(config)# exit

MDS# show fcdomain fcid persistent vsan 5

Total entries 188.


Persistent FCIDs table contents:

VSAN WWN FCID Mask Used Assignment

---- ----------------------- -------- ----------- ---- ----------


5 21:00:00:1b:32:06:f8:e7 0xc92002 SINGLE FCID YES STATIC


MDS# show flogi database interface fc1/3




fc1/3 5 0xc92002 21:00:00:1b:32:06:f8:e7 20:00:00:1b:32:06:f8:e7

fc1/3 5 0xc92c00 21:00:00:1b:32:94:99:da 20:00:00:1b:32:94:99:da

Total number of flogi = 2.


MDS# copy running-config startup-config




  1. Thank you for blogging this, we hit the exact same problem and fixed with the method 3.