openSolaris 2008 - Configuring IPMP Groups - System Administration Guide: IP Services

Configuring IPMP Groups

This section provides procedures for configuring IPMP groups. It also describes how to configure an interface as a standby.

Planning for an IPMP Group

Before you configure interfaces on a system as part of an IPMP group, you need to do some preconfiguration planning.

How to Plan for an IPMP Group

The following procedure includes the planning tasks and information to be gathered prior to configuring the IPMP group. The tasks do not have to be performed in sequence.

Decide which interfaces on the system are to be part of the IPMP group.
An IPMP group usually consists of at least two physical interfaces that are connected to the same IP link. However, you can configure a single interface IPMP group, if required. For an introduction to IPMP groups, refer to IPMP Interface Configurations. For example, you can configure the same Ethernet switch or the same IP subnet under the same IPMP group. You can configure any number of interfaces into the same IPMP group.
You cannot use the group parameter of the ifconfig command with logical interfaces. For example, you can use the group parameter with hme0, but not with hme0:1.
Verify that each interface in the group has a unique MAC address.
For instructions, refer to SPARC: How to Ensure That the MAC Address of an Interface Is Unique, in Solaris 10 3/05 ONLY or SPARC: How to Ensure That the MAC Address of an Interface Is Unique.
Choose a name for the IPMP group.
Any non-null name is appropriate for the group. You might want to use a name that identifies the IP link to which the interfaces are attached.
Ensure that the same set of STREAMS modules is pushed and configured on all interfaces in the IPMP group.
All interfaces in the same group must have the same STREAMS modules configured in the same order.
1. Check the order of STREAMS modules on all interfaces in the prospective IPMP group.
  You can print out a list of STREAMS modules by using the ifconfig interface modlist command. For example, here is the ifconfig output for an hme0 interface:
```
# ifconfig hme0 modlist
    0 arp
    1 ip
    2 hme
```
  Interfaces normally exist as network drivers directly below the IP module, as shown in the output from ifconfig hme0 modlist. They should not require additional configuration.
  However, certain technologies, such as NCA or IP Filter, insert themselves as STREAMS modules between the IP module and the network driver. Problems can result in the way interfaces of the same IPMP group behave.
  If a STREAMS module is stateful, then unexpected behavior can occur on failover, even if you push the same module onto all of the interfaces in a group. However, you can use stateless STREAMS modules, provided that you push them in the same order on all interfaces in the IPMP group.
2. Push the modules of an interface in the standard order for the IPMP group.
```
ifconfig interface modinsert module-name
```
```
ifconfig hme0 modinsert ip
```
Use the same IP addressing format on all interfaces of the IPMP group.
If one interface is configured for IPv4, then all interfaces of the group must be configured for IPv4. Suppose you have an IPMP group that is composed of interfaces from several NICs. If you add IPv6 addressing to the interfaces of one NIC, then all interfaces in the IPMP group must be configured for IPv6 support.
Check that all interfaces in the IPMP group are connected to the same IP link.
Verify that the IPMP group does not contain interfaces with different network media types.
The interfaces that are grouped together should be of the same interface type, as defined in /usr/include/net/if_types.h. For example, you cannot combine Ethernet and Token ring interfaces in an IPMP group. As another example, you cannot combine a Token bus interface with asynchronous transfer mode (ATM) interfaces in the same IPMP group.
For IPMP with ATM interfaces, configure the ATM interfaces in LAN emulation mode.
IPMP is not supported for interfaces using Classical IP over ATM.

SPARC: How to Ensure That the MAC Address of an Interface Is Unique, in Solaris 10 3/05 ONLY

Before you configure an IPMP group, you must verify that every interface in the prospective group has a unique MAC address. Almost all interfaces come configured with a factory-set unique MAC address. However, every SPARC-based system has a system-wide MAC address, which by default is used by all interfaces. In an IPMP group, each interface must have a unique MAC address. Therefore, you must ensure that the EEPROM parameter local-mac-address? is set to true so that the interfaces use their factory-set MAC addresses. You can use the eeprom command to check the current value of local-mac-address? and change it, if necessary.

On the system with the interfaces to be configured, assume the Primary Administrator role or become superuser.
The Primary Administrator role includes the Primary Administrator profile. To create the role and assign the role to a user, see Chapter 2, Working With the Solaris Management Console (Tasks), in System Administration Guide: Basic Administration.
Determine whether all interfaces on the system currently use the system-wide MAC address.
```
# eeprom local-mac-address?
local-mac-address?=false
```
In the example, the value of local-mac-address?=false indicates that all interfaces do use the system-wide MAC address. The value of local-mac-address?=false must be changed to true before the interfaces can become members of an IPMP group.
If necessary, change the value of local-mac-address? as follows:
```
# eeprom local-mac-address?=true
```
When you reboot the system, the interfaces with factory-set MAC addresses instead use these factory settings. Interfaces without factory-set MAC addresses continue to use the system-wide MAC address.

Check the MAC addresses of the interfaces on the system.

ifconfig -a
lo0: flags=1000849 <UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
     inet 127.0.0.1 netmask ff000000 
hme0: flags=1004843 <UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
     inet 10.0.0.112 netmask ffffff80 broadcast 10.0.0.127
     ether 8:0:20:0:0:1
hme1: flags=1004843 <UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
     inet 10.0.0.114 netmask ffffff80 broadcast 10.0.0.127
     ether 8:0:20:0:0:1
ge0: flags=1004843 <UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
     inet 10.0.0.118 netmask ffffff80 broadcast 10.0.0.127
     ether 8:0:20:1:1:1

Look for cases where multiple interfaces have the same MAC address. In the previous example, hme0 and hme1 both have the same MAC address.

Note - Continue to the next step only if more than one network interface still has the same MAC address.

If necessary, manually configure the remaining interfaces so that all interfaces have unique MAC addresses.
Place a unique MAC address in the /etc/hostname.interface for the particular interface.

Note - To prevent any risk of manually configured MAC addresses conflicting with other MAC addresses on your network, you must always configure locally administered MAC addresses, as defined by the IEEE 802.3 standard.

In the previous example, you must configure either hme0 or hme1 with a locally-administered MAC address. For example, to reconfigure hme1 with the locally-administered MAC address 06:05:04:03:02, you would add the following line to /etc/hostname.hme1:
```
ether 06:05:04:03:02 
```
You also can use the ifconfig ether command to configure an interface's MAC address for the current session. However, any changes made directly with ifconfig are not preserved across reboots. Refer to the ifconfig(1M) man page for details.
Reboot the system.

Configuring IPMP Groups

This section contains configuration tasks for a typical IPMP group with at least two physical interfaces.

For an introduction to multiple interface IPMP groups, refer to IPMP Group.
For planning tasks, refer to Planning for an IPMP Group.
To configure an IPMP group with only one physical interface, refer to Configuring IPMP Groups With a Single Physical Interface.

How to Configure an IPMP Group With Multiple Interfaces

Before You Begin

You need to have already configured the IPv4 addresses, and, if appropriate, the IPv6 addresses of all interfaces in the prospective IPMP group.

On the system with the interfaces to be configured, assume the Primary Administrator role, or become superuser.
The Primary Administrator role includes the Primary Administrator profile. To create the role and assign the role to a user, see Chapter 2, Working With the Solaris Management Console (Tasks), in System Administration Guide: Basic Administration.
Place each physical interface into an IPMP group.
```
# ifconfig interface group group-name
```
For example, to place hme0 and hme1 under group testgroup1, you would type the following commands:
```
# ifconfig hme0 group testgroup1
# ifconfig hme1 group testgroup1
```
Avoid using spaces in group names. The ifconfig status display does not show spaces. Consequently, do not create two similar group names where the only difference is that one name also contains a space. If one of the group names contains a space, these group names look the same in the status display.
In a dual-stack environment, placing the IPv4 instance of an interface under a particular group automatically places the IPv6 instance under the same group.
(Optional) Configure an IPv4 test address on one or more physical interfaces.
You need to configure a test address only if you want to use probe-based failure detection on a particular interface. Test addresses are configured as logical interfaces of the physical interface that you specify to the ifconfig command.
If one interface in the group is to become the standby interface, do not configure a test address for that interface at this time. You configure a test address for the standby interface as part of the task How to Configure a Standby Interface for an IPMP Group.
Use the following syntax of the ifconfig command for configuring a test address:
```
# ifconfig interface addif ip-address <parameters> -failover deprecated up
```
For example, you would create the following test address for the primary network interface hme0:
```
# ifconfig hme0 addif 192.168.85.21 netmask + broadcast + -failover deprecated up
```
This command sets the following parameters for the primary network interface hme0:
- Address set to 192.168.85.21
- Netmask and broadcast address set to the default value
- -failover and deprecated options set
  
  Note - You must mark an IPv4 test address as deprecated to prevent applications from using the test address.
Check the IPv4 configuration for a specific interface.
You can always view the current status of an interface by typing ifconfig interface. For more information on viewing an interface's status, refer to How to Get Information About a Specific Interface.
You can get information about test address configuration for a physical interface by specifying the logical interface that is assigned to the test address.
```
# ifconfig hme0:1
    hme0:1: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
    mtu 1500 index 2 
    inet 192.168.85.21 netmask ffffff00 broadcast 192.168.85.255
```
(Optional) If applicable, configure an IPv6 test address.
```
# ifconfig interface inet6 -failover
```
Physical interfaces with IPv6 addresses are placed into the same IPMP group as the interfaces' IPv4 addresses. This happens when you configure the physical interface with IPv4 addresses into an IPMP group. If you first place physical interfaces with IPv6 addresses into an IPMP group, physical interfaces with IPv4 addresses are also implicitly placed in the same IPMP group.
For example, to configure hme0 with an IPv6 test address, you would type the following:
```
# ifconfig hme0 inet6 -failover
```
You do not need to mark an IPv6 test address as deprecated to prevent applications from using the test address.

Check the IPv6 configuration.

# ifconfig hme0 inet6
    hme0: flags=a000841<UP,RUNNING,MULTICAST,IPv6,NOFAILOVER> mtu 1500 index 2
            inet6 fe80::a00:20ff:feb9:17fa/10 
            groupname test

The IPv6 test address is the link-local address of the interface.

(Optional) Preserve the IPMP group configuration across reboots.
- For IPv4, add the following line to the /etc/hostname.interface file:
```
interface-address <parameters> group group-name up \
    addif logical-interface -failover deprecated <parameters> up
```
  In this instance, the test IPv4 address is configured only on the next reboot. If you want the configuration to be invoked in the current session, do steps 1, 2, and, optionally 3.
- For IPv6, add the following line to the /etc/hostname6.interface file:
```
-failover group group-name up
```
  This test IPv6 address is configured only on the next reboot. If you want the configuration to be invoked in the current session, do steps 1, 2, and, optionally, 5.
(Optional) Add more interfaces to the IPMP group by repeating steps 1 through 6.
You can add new interfaces to an existing group on a live system. However, changes are lost across reboots.

Example 31-1 Configuring an IPMP Group With Two Interfaces

Suppose you want to do the following:

Have the netmask and broadcast address set to the default value.
Configure the interface with a test address 192.168.85.21.

You would type the following command:

# ifconfig hme0 addif 192.168.85.21 netmask + broadcast + -failover deprecated up

You must mark an IPv4 test address as deprecated to prevent applications from using the test address. See How to Configure an IPMP Group With Multiple Interfaces.

To turn on the failover attribute of the address, you would use the failover option without the dash

All test IP addresses in an IPMP group must use the same network prefix. The test IP addresses must belong to a single IP subnet.

Example 31-2 Preserving an IPv4 IPMP Group Configuration Across Reboots

Suppose you want to create an IPMP group called testgroup1 with the following configuration:

Physical interface hme0 with address 192.168.85.19
A logical interface address of 192.168.85.21
deprecated and -failover options set
Netmask and broadcast address set to the default value

You would add the following line to the /etc/hostname.hme0 file:

192.168.85.19 netmask + broadcast + group testgroup1 up \
    addif 192.168.85.21 deprecated -failover netmask + broadcast + up

Similarly, to place the second interface hme1 under the same group testgroup1 and to configure a test address, you would add the following line:

192.168.85.20 netmask + broadcast + group testgroup1 up \
    addif 192.168.85.22 deprecated -failover netmask + broadcast + up

Example 31-3 Preserving an IPv6 IPMP Group Configuration Across Reboots

To create a test group for interface hme0 with an IPv6 address, you would add the following line to the /etc/hostname6.hme0 file:

-failover group testgroup1 up

Similarly, to place the second interface hme1 in group testgroup1 and to configure a test address, you would add the following line to the /etc/hostname6.hme1 file:

-failover group testgroup1 up

Troubleshooting

During IPMP group configuration, in.mpathd outputs a number of messages to the system console or to the syslog file. These messages are informational in nature and indicate that the IPMP configuration functions correctly.

This message indicates that interface hme0 was added to IPMP group testgroup1. However, hme0 does not have a test address configured. To enable probe-based failure detection, you need to assign a test address to the interface.
```
May 24 14:09:57 host1 in.mpathd[101180]: No test address configured on interface hme0;
disabling probe-based failure detection on it.
testgroup1
```

This message appears for all interfaces with only IPv4 addresses that are added to an IPMP group.

May 24 14:10:42 host4 in.mpathd[101180]: NIC qfe0 of group testgroup1 is not 
plumbed for IPv6 and may affect failover capability

This message should appear when you have configured a test address for an interface.

Created new logical interface hme0:1
May 24 14:16:53 host1 in.mpathd[101180]: Test address now configured on interface hme0;
 enabling probe-based failure detection on it

Configuring Target Systems

Probe-based failure detection involves the use of target systems, as explained in Probe-Based Failure Detection. For some IPMP groups, the default targets used by in.mpathd is sufficient. However, for some IPMP groups, you might want to configure specific targets for probe-based failure detection. You accomplish probe-based failure detection by setting up host routes in the routing table as probe targets. Any host routes that are configured in the routing table are listed before the default router. Therefore, IPMP uses the explicitly defined host routes for target selection. You can use either of two methods for directly specifying targets: manually setting host routes or creating a shell script that can become a startup script.

Consider the following criteria when evaluating which hosts on your network might make good targets.

Make sure that the prospective targets are available and running. Make a list of their IP addresses.
Ensure that the target interfaces are on the same network as the IPMP group that you are configuring.
The netmask and broadcast address of the target systems must be the same as the addresses in the IPMP group.
The target host must be able to answer ICMP requests from the interface that is using probe-based failure detection.

How to Manually Specify Target Systems for Probe-Based Failure Detection

Log in with your user account to the system where you are configuring probe-based failure detection.
Add a route to a particular host to be used as a target in probe-based failure detection.
```
$ route add -host destination-IP gateway-IP -static
```
Replace the values of destination-IP and gateway-IP with the IPv4 address of the host to be used as a target. For example, you would type the following to specify the target system 192.168.85.137, which is on the same subnet as the interfaces in IPMP group testgroup1.
```
$ route add -host 192.168.85.137 192.168.85.137 -static 
```
Add routes to additional hosts on the network to be used as target systems.

How to Specify Target Systems in a Shell Script

On the system where you have configured an IPMP group, assume the Primary Administrator role or become superuser.
The Primary Administrator role includes the Primary Administrator profile. To create the role and assign the role to a user, see Chapter 2, Working With the Solaris Management Console (Tasks), in System Administration Guide: Basic Administration.

Create a shell script that sets up static routes to your proposed targets.

For example, you could create a shell script called ipmp.targets with the following contents:

TARGETS="192.168.85.117 192.168.85.127 192.168.85.137"

case "$1" in
        'start')
            /usr/bin/echo "Adding static routes for use as IPMP targets"
        for target in $TARGETS; do
      /usr/sbin/route add -host $target $target
        done
                  ;;
        'stop')
              /usr/bin/echo "Removing static routes for use as IPMP targets"
         for target in $TARGETS; do
        /usr/sbin/route delete -host $target $target
         done
                  ;;
  esac

Copy the shell script to the startup script directory.
```
 # cp ipmp.targets /etc/init.d  
```
Change the permissions on the new startup script.
```
# chmod 744 /etc/init.d/ipmp.targets
```
Change ownership of the new startup script.
```
# chown root:sys /etc/init.d/ipmp.targets
```
Create a link for the startup script in the /etc/init.d directory.
```
# ln /etc/init.d/ipmp.targets /etc/rc2.d/S70ipmp.targets
```
The S70 prefix in the file name S70ipmp.targets orders the new script properly with respect to other startup scripts.

Configuring Standby Interfaces

Use this procedure if you want the IPMP group to have an active-standby configuration. For more information on this type of configuration, refer to IPMP Interface Configurations.

How to Configure a Standby Interface for an IPMP Group

Before You Begin

You must have configured all interfaces as members of the IPMP group.
You should not have configured a test address on the interface to become the standby interface.

For information on configuring an IPMP group and assigning test addresses, refer to How to Configure an IPMP Group With Multiple Interfaces.

On the system with the standby interfaces to be configured, assume the Primary Administrator role or become superuser.
The Primary Administrator role includes the Primary Administrator profile. To create the role and assign the role to a user, see Chapter 2, Working With the Solaris Management Console (Tasks), in System Administration Guide: Basic Administration.
Configure an interface as a standby and assign the test address.
```
# ifconfig interface plumb ip-address <other-parameters> deprecated -failover standby up
```
A standby interface can have only one IP address, the test address. You must set the -failover option before you set the standby up option. For <other-parameters>, use the parameters that are required by your configuration, as described in the ifconfig(1M) man page.
- For example, to create an IPv4 test address, you would type the following command:
```
# ifconfig hme1 plumb 192.168.85.22 netmask + broadcast + deprecated -failover standby up
```
  hme1
  
  Defines hme1 as the physical interface to be configured as the standby interface.
  
  192.168.85.22
  
  Assigns this test address to the standby interface.
  
  deprecated
  
  Indicates that the test address is not used for outbound packets.
  
  -failover
  
  Indicates that the test address does not fail over if the interface fails.
  
  standby
  
  Marks the interface as a standby interface.
- For example, to create an IPv6 test address, you would type the following command:
```
# ifconfig hme1 plumb -failover standby up
```
Check the results of the standby interface configuration.
```
# ifconfig hme1
hme1: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,
      STANDBY,INACTIVE mtu 1500 
         index 4 inet 192.168.85.22 netmask ffffff00 broadcast 19.16.85.255
         groupname test
```
The INACTIVE flag indicates that this interface is not used for any outbound packets. When a failover occurs on this standby interface, the INACTIVE flag is cleared.

Note - You can always view the current status of an interface by typing the ifconfig interface command. For more information on viewing interface status, refer to How to Get Information About a Specific Interface.
(Optional) Preserve the IPv4 standby interface across reboots.
Assign the standby interface to the same IPMP group, and configure a test address for the standby interface.
For example, to configure hme1 as the standby interface, you would add the following line to the /etc/hostname.hme1 file:
```
192.168.85.22 netmask + broadcast + deprecated group test -failover standby up 
```
(Optional) Preserve the IPv6 standby interface across reboots.
Assign the standby interface to the same IPMP group, and configure a test address for the standby interface.
For example, to configure hme1 as the standby interface, add the following line to the /etc/hostname6.hme1 file:
```
-failover group test standby up
```

Example 31-4 Configuring a Standby Interface for an IPMP Group

Suppose you want to create a test address with the following configuration:

Physical interface hme2 as a standby interface
Test address of 192.168.85.22
deprecated and -failover options set
Netmask and broadcast address set to the default value

You would type the following:

# ifconfig hme2 plumb 192.168.85.22 netmask + broadcast + deprecated -failover standby up

The interface is marked as a standby interface only after the address is marked as a NOFAILOVER address.

You would remove the standby status of an interface by typing the following:

# ifconfig interface -standby

Configuring IPMP Groups With a Single Physical Interface

When you have only one interface in an IPMP group, failover is not possible. However, you can enable failure detection on that interface by assigning the interface to an IPMP group. You do not have to configure a dedicated test IP address to establish failure detection for a single interface IPMP group. You can use a single IP address for sending data and detecting failure.

How to Configure a Single Interface IPMP Group

On the system with the prospective single interface IPMP group, assume the Primary Administrator role or become superuser.
The Primary Administrator role includes the Primary Administrator profile. To create the role and assign the role to a user, see Chapter 2, Working With the Solaris Management Console (Tasks), in System Administration Guide: Basic Administration.
For IPv4, create the single interface IPMP group.
You can use either of the following methods:
- Use the following syntax to assign the single interface to an IPMP group.
```
# ifconfig interface -failover group group-name
```
  The following example assigns the interface hme0 into the IPMP group v4test:
```
# ifconfig hme0 -failover group v4test
```
  Unlike the multiple physical interface configuration, you would not mark a single physical interface as deprecated.
  This example includes the use of the -failover option of the ifconfig command to create an IFF_NOFAILOVER flag for the interface. Consider using -failover if you might later add more interfaces to the group. The in.mpathd daemon sends probe packets by using that address. Later, when you add more interfaces, the configuration should work properly.
- Alternatively, you can use the following syntax to add a single physical interface to an IPMP group:
```
# ifconfig interface group group-name
```
  When you use this configuration, in.mpathd chooses a data address to send probe packets.
For IPv6, create the single interface IPMP group.
Use either of the following two methods:
- Use the following syntax to assign the single interface to an IPMP group:
```
# ifconfig interface inet6 -failover group group-name
```
  For example, you would type the following to add the single interface hme0 into the IPMP group v6test:
```
# ifconfig hme0 inet6 -failover group v6test
```
- Use the following syntax if you do not want to set the NOFAILOVER flag:
```
# ifconfig interface inet6 group group-name
```
  When the in.mpathd daemon detects failures, the interface is marked and logged appropriately on the console.
In a single physical interface configuration, you cannot verify whether the target system that is being probed has failed or whether the interface has failed. The target system can be probed through only one physical interface. If only one default router is on the subnet, turn off IPMP if a single physical interface is in the group. If a separate IPv4 and IPv6 default router exists, or multiple default routers exist, more than one target system needs to be probed. Hence, you can safely turn on IPMP.

Configuring IPMP Groups

Planning for an IPMP Group

How to Plan for an IPMP Group

SPARC: How to Ensure That the MAC Address of an Interface Is Unique, in Solaris 10 3/05 ONLY

Configuring IPMP Groups

How to Configure an IPMP Group With Multiple Interfaces

Before You Begin

Troubleshooting

See Also

Configuring Target Systems

How to Manually Specify Target Systems for Probe-Based Failure Detection

How to Specify Target Systems in a Shell Script

Configuring Standby Interfaces

How to Configure a Standby Interface for an IPMP Group

Before You Begin

Configuring IPMP Groups With a Single Physical Interface

How to Configure a Single Interface IPMP Group