Monday, September 27, 2010

EX4200 Virtual Chassis Best Practice - Replacing the faulty member in VC

Back Ground:
Our customer was running EX4200 VC in their head office network. They had a planned power outage recently. After the power restored, they had a member switch failed. They replaced faulty switch followed the steps listed on Juniper web site. After that, they have two major outage with their VC. The member switches in VC were behaving strangely and crashed which cause "core-dump" on the RE. Juniper said this was cause by "memory-leaking" in the version they were running (10.0R3.10). The funny thing is they were running on this version for about 6 months without any issue. We believed the issue was cause by replacing the faulty switch. After reviewing their steps, we found the steps were not quite right even they were list on Juniper web site. Our customer re did the whole process again following my suggestions listed as per below. The problem with their virtual chassis was gone. 

Replacing the faulty member in EX4200 Virtual Chassis: 

1. Remove VC cables from the faulty member. 

2. Remove the faulty member. 

3. Check the firmware version on the new EX4200 switch. If the version is different, upgrade it to the version which VC is running.

* 4. If you are using a old switch which was used before . You need to 
   1. Revert the EX4200 to factory default first
   2. Upgrade the firmware version to the version which VC is running. Even the version on the old switch is same, you need to upgrade it again. 

* 5. Recycle the old member number on the master RE. After you finish this step, the next available member number should be the same with faulty one.

6. Patch the new switch into the chassis.

7. Powered on new member switch.

8. Confirmed that the replacement switch presented itself as same member with old one.

Example:


root> show virtual-chassis status

Virtual Chassis ID: ****.****.fedc
                                          Mastership            Neighbor List
Member ID  Status   Serial No    Model    priority    Role      ID  Interface
0 (FPC 0)  Prsnt    ********07283 ex4200-48p      255  Master*    3  vcp-0
                                                                 2  vcp-1
1 (FPC 1)  Prsnt    ********37531 ex4200-48p      128  Backup     3  vcp-0
                                                                 2  vcp-1
2 (FPC 2)  Prsnt    ********15701 ex4200-48p      128  Linecard   1  vcp-0
                                                                 0  vcp-1
3 (FPC 3)  Prsnt    ********17802 ex4200-48p      128  Linecard   0  vcp-0
                                                                 1  vcp-1

Member ID for next new member: 4 (FPC 4)

{master:0}




root> show virtual-chassis status

Virtual Chassis ID: ****.****.fedc
                                          Mastership            Neighbor List
Member ID  Status   Serial No    Model    priority    Role      ID  Interface
0 (FPC 0)  Prsnt    ********07283 ex4200-48p      255  Master*    3  vcp-0
1 (FPC 1)  Prsnt    ********37531 ex4200-48p      128  Backup     3  vcp-0
2 (FPC 2)  NotPrsnt ********15701 ex4200-48p
3 (FPC 3)  Prsnt    ********17802 ex4200-48p      128  Linecard   0  vcp-0
                                                                 1  vcp-1

Member ID for next new member: 4 (FPC 4)

{master:0}




root> request virtual-chassis recycle member-id 2

{master:0}
root> show virtual-chassis status

Virtual Chassis ID: ****.****.fedc
                                          Mastership            Neighbor List
Member ID  Status   Serial No    Model    priority    Role      ID  Interface
0 (FPC 0)  Prsnt    ********07283 ex4200-48p      255  Master*    3  vcp-0
1 (FPC 1)  Prsnt    ********37531 ex4200-48p      128  Backup     3  vcp-0
3 (FPC 3)  Prsnt    ********17802 ex4200-48p      128  Linecard   0  vcp-0
                                                                 1  vcp-1

Member ID for next new member: 2 (FPC 2)

{master:0}

root> show virtual-chassis status

Virtual Chassis ID: ****.****.fedc
                                          Mastership            Neighbor List
Member ID  Status   Serial No    Model    priority    Role      ID  Interface
0 (FPC 0)  Prsnt    ********07283 ex4200-48p      255  Master*    3  vcp-0
                                                                 2  vcp-1
1 (FPC 1)  Prsnt    ********37531 ex4200-48p      128  Backup     3  vcp-0
                                                                 2  vcp-1
2 (FPC 2)  Prsnt    ********17762 ex4200-48p      128  Linecard   1  vcp-0
                                                                 0  vcp-1
3 (FPC 3)  Prsnt    ********17802 ex4200-48p      128  Linecard   0  vcp-0
                                                                 1  vcp-1

Member ID for next new member: 4 (FPC 4)

{master:0}

-------------------------------------------------------------------------------------------------------------------------------------

Note:
1. Step 4 will make sure you have a clean "new" EX4200 switch for the VC.
2. If you use renumber instead of recycling in Step 5, VC may have issues after you install the new member switch.