`MULTICAST TROUBLESHOOTING GUIDE`

last update 3/16/00 / mmcbride

There is a unique methodology to troubleshoot IP Multicast routing. For instance, a major difference between multicast and unicast routing is the reverse path forwarding involved with multicast. Unicast routing is primarily concerned with where to forward the packet based upon the destination address. Unicast routing is not much concerned with the source of the packet (unless Policy Routing). With multicast routing, on the other hand, the primary concern is the source address. Multicast routing will not forward a packet unless the source of the packet passes a reverse path forwarding check. Once a packet passes this RPF check, multicast routing will only then forward the packet based upon the destination address. When a multicast packet arrives on an interface, the RPF process checks to ensure that this incoming interface is the outgoing interface used by unicast routing to reach the source of the multicast packet. This RPF check process prevents loops.

Like unicast routing, multicast routing has several available protocols: PIM-DM, PIM-SM, DVMRP, MBGP, MSDP... but the case studies listed below are fairly generic across the protocols, except where otherwise noted.

Customers who need multicast support should open a case with the TAC (tac@cisco.com). Or you may send an email to cs-ipmulticast@cisco.com which is a more informal resource. Customer Support Engineers work closely with Multicast Development Engineers to resolve questions and problems.

The most commonly used multicast troubleshooting commands/debugs are:

show ip mroute <group>
show ip rpf <source>
show ip pim neighbor
show ip pim interface
show ip igmp group
debug ip pim <group>
debug ip igmp
debug ip mpacket <access-list>

There are a variety of resources available to assist with multicast troubleshooting:

ftp://ftpeng.cisco.com/ipmulticast/Multicast-Commands <--description of all IOS multicast commands

ftp://ftpeng.cisco.com/ipmulticast/mcast_config_guide.html <--multicast quickstart configuration guide

http://www.cisco.com/warp/public/732/Tech/multicast/ <--multicast resources

In the following real case studies, you will be walked through the process of troubleshooting these problems. You will see which commands can be used to quickly pinpoint the problem and learn what to do about it. These case studies are based upon the many customer issues which have been discussed on the multicast support alias.

`CASE STUDIES`

Reverse Path Forward (RPF)

Switch Flooding All Ports

Switch Flooding 224.0.0.X Packets

`RPF FAILURE`

+---+ +---+
video--->>>>e0/0|75a|e0/1-----e3/1|72a|e3/2---HostA
+---+ +---+

Multicast packets are coming into e0/0 of 75a from a server whose ip address is 1.1.1.1 and sending to group 224.1.1.1. This is know as an (S,G) or (1.1.1.1, 224.1.1.1).

Problem: Hosts who are directly connected to 75a are getting the multicast feed. But hosts (HostA), who are directly connected to 72a, are not getting the stream.

Let's see what's going on first with 75a:

ip22-75a#sh ip mroute 224.1.1.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned
       R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT
       M - MSDP created entry, X - Proxy Join Timer Running
       A - Advertised via MSDP
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.1.1.1), 00:01:23/00:02:59, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse-Dense, 00:01:23/00:00:00

(1.1.1.1, 224.1.1.1), 00:01:23/00:03:00, flags: TA
Incoming interface: Ethernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse-Dense, 00:01:23/00:00:00

Since it's dense mode we can basically ignore the *,G entry and focus on the S,G entry. It's telling us that the multicast packets are being sourced from a server whose address is 1.1.1.1 which is sending to a multicast group of 224.1.1.1. The packets are coming in the ethernet0/0 interface and being forwarded out the ethernet 0/1 interface. This is perfect. So, since we know that packets are being forwarded out ethernet 0/1 to the LAN which 72a is connected, let's take a look at what 72a is doing with the packets:

ip22-72a#sh ip mroute 224.1.1.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned
       R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT
       M - MSDP created entry, X - Proxy Join Timer Running
       A - Advertised via MSDP
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.1.1.1), 00:05:36/00:02:19, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet3/1, Forward/Sparse-Dense, 00:05:36/00:00:00
Ethernet3/2, Forward/Sparse-Dense, 00:05:37/00:00:00

The necessary S,G (1.1.1.1, 224.1.1.1) is not even listed. It's definitely not forwarding the packets. So what's going on. At least we found the trouble spot, now we'll have to probe deeper on this router.

Let's look to see if it's showing the upstream router (75a) as a pim neighbor:

ip22-72a#sh ip pim neigh
PIM Neighbor Table
Neighbor Address Interface Uptime Expires Ver Mode
2.1.1.1 Ethernet3/1 2d00h 00:01:15 v2

It sure is, no problems there. Let's see if we have a Reverse Path Forwarding (RPF) error:

ip22-72a#sh ip rpf 1.1.1.1
RPF information for ? (1.1.1.1)
RPF interface: Ethernet3/3
RPF neighbor: ? (4.1.1.2)
RPF route/mask: 1.1.1.1/32
RPF type: unicast (static)
RPF recursion count: 1
Doing distance-preferred lookups across tables

Wait a minute. It's showing the rpf interface being e3/3 but it should be e 3/1 as the incoming interface. Let's further confirm with some debug, although we'll need to be careful with this as it will be busy:

ip22-72a#debug ip mpacket
*Jan 14 09:45:32.972: IP: s=1.1.1.1 (Ethernet3/1) d=224.2.127.254 len 60, not RPF interface
*Jan 14 09:45:33.020: IP: s=1.1.1.1 (Ethernet3/1) d=224.2.127.254 len 60, not RPF interface
*Jan 14 09:45:33.072: IP: s=1.1.1.1 (Ethernet3/1) d=224.2.127.254 len 60, not RPF interface
*Jan 14 09:45:33.120: IP: s=1.1.1.1 (Ethernet3/1) d=224.2.127.254 len 60, not RPF interface

Sure enough, The packets are coming in on ethernet3/1 as we want but they are being dropped because that's not the interface the unicast routing table is using for rpf check. We can either change the unicast routing to satisfy this requirement or we can add a static mroute to force multicast to RPF out a particular interface regardless of what the unicast routing table states. We'll add a static mroute:

ip22-72a(config)#ip mroute 1.1.1.1 255.255.255.255 2.1.1.1

This static multicast route states that to get to the address 1.1.1.1, for rpf, use 2.1.1.1 as the next hop to get there, which is out e3/1.

ip22-72a#sh ip rpf 1.1.1.1
RPF information for ? (1.1.1.1)
RPF interface: Ethernet3/1
RPF neighbor: ? (2.1.1.1)
RPF route/mask: 1.1.1.1/32
RPF type: static mroute
RPF recursion count: 0
Doing distance-preferred lookups across tables

Now everything looks good:

(*, 224.1.1.1), 00:01:15/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet3/1, Forward/Sparse-Dense, 00:01:15/00:00:00
Ethernet3/2, Forward/Sparse-Dense, 00:00:58/00:00:00

(1.1.1.1, 224.1.1.1), 00:00:48/00:02:59, flags: CTA
Incoming interface: Ethernet3/1, RPF nbr 2.1.1.1, Mroute
Outgoing interface list:
Ethernet3/2, Forward/Sparse-Dense, 00:00:48/00:00:00

ip22-72a#debug ip mpacket
*Jan 14 10:18:29.951: IP: s=1.1.1.1 (Ethernet3/1) d=224.1.1.1 (Ethernet3/2) len 60, mforward
*Jan 14 10:18:29.999: IP: s=1.1.1.1 (Ethernet3/1) d=224.1.1.1 (Ethernet3/2) len 60, mforward
*Jan 14 10:18:30.051: IP: s=1.1.1.1 (Ethernet3/1) d=224.1.1.1 (Ethernet3/2) len 60, mforward

and HostA is happily getting the packets, problem solved.

`Time To Live (TTL)`

Problem 1: RouterA is not forwarding packets from source(S) to RouterB's directly connected receiver(R).

               +-------+              +-------+
Source----e0/0|ROUTERA|e0/1------e1/1|ROUTERB|e1/2------Receiver
               +-------+              +-------+
1.1.1.1      .2       2.1.1.1       .2       3.1.1.1     .2

sending to:
224.1.1.1

Troubleshooting:

Let's first look at 'sh ip mroute' on ROUTERA to see the multicast routing state:

ROUTERA#sh ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned
       R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT
       M - MSDP created entry, X - Proxy Join Timer Running
       A - Advertised via MSDP
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.0.1.40), 00:00:01/00:00:00, RP 0.0.0.0, flags: DJCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse-Dense, 00:00:01/00:00:00

(*, 224.1.1.1), 00:00:02/00:02:57, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse-Dense, 00:00:02/00:00:00

Well that's interesting. We can ignore the 224.0.1.40 since all routers will join this Auto-RP Discovery group. But we don't have a source listed for 224.1.1.1. In addition to "*, 224.1.1.1" we should be seeing "1.1.1.1, 224.1.1.1". We don't recognize the source as being valid for some reason.

Let's see if it's an RPF issue:

ROUTERA#sh ip rpf 1.1.1.1
RPF information for ? (1.1.1.1)
RPF interface: Ethernet0/0
RPF neighbor: ? (0.0.0.0) - directly connected
RPF route/mask: 1.1.1.0/24
RPF type: unicast (connected)
RPF recursion count: 0
Doing distance-preferred lookups across tables

Nope, the rpf check is correctly pointing out e0/0 to get to the source's ip address.

Let's see if pim is configured on the interfaces:

ROUTERA#sh ip pim int

Address          Interface          Version/Mode    Nbr   Query     DR
                                                    Count Intvl
1.1.1.2          Ethernet0/0        v2/Sparse-Dense 0    30     1.1.1.2
2.1.1.1          Ethernet0/1        v2/Sparse-Dense 2    30     2.1.1.2

Looks good, that's not the problem.

ROUTERA#sh ip mroute active
Active IP Multicast Sources - sending >= 4 kbps

So, the router doesn't recognize any active traffic.

ROUTERA#debug ip mpacket
IP multicast packets debugging is on

nothing...humm...

Perhaps the Receiver is not sending any igmp reports (joins) for group 224.1.1.1:

ROUTERB#sh ip igmp group
IGMP Connected Group Membership
Group Address    Interface            Uptime    Expires   Last Reporter
224.0.1.40       Ethernet1/1          00:34:34 never     2.1.1.2
224.1.1.1        Ethernet1/2          00:30:02 00:02:45 3.1.1.2

224.1.1.1 has been joined on e1/2 so that's fine. Perhaps ROUTERB is not sending PIM JOINS to ROUTERA informing ROUTERA that it needs to forward the traffic:

Actually, ROUTERB is not going to send pim joins to ROUTERA. Dense Mode is a flood and prune protocol, so there are no joins, but there are grafts. But since ROUTERB hasn't received any flood from RouterA, it doesn't know where to send a graft.

Perhaps it's a TTL (Time to Live) issue:

ROUTERA#sh ip traffic
IP statistics:
Rcvd: 248756 total, 1185 local destination
         0 format errors, 0 checksum errors, 63744 bad hop count
         0 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 0 with options

Yikes, 63744 bad hop counts, and each time I type this command, the bad hop counts increase. This is the TTL incrementing. We have found the problem. To solve the issue we need to increase the TTL on our Sender application. Once done, ROUTERA will look like this:

(*, 224.0.1.40), 01:16:32/00:00:00, RP 0.0.0.0, flags: DJCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse-Dense, 01:16:32/00:00:00

(*, 224.1.1.1), 00:28:42/00:02:59, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse-Dense, 00:28:42/00:00:00

(1.1.1.1, 224.1.1.1), 00:19:24/00:02:59, flags: TA
Incoming interface: Ethernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
    Ethernet0/1, Forward/Sparse-Dense, 00:19:24/00:00:00
                 ^^^^^^^^^^^^^^^^^^^^
                 this is what we want to see!

And on ROUTERB:

ROUTERB#sh ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned
       R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT
       M - MSDP created entry, X - Proxy Join Timer Running
       A - Advertised via MSDP
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.0.1.40), 01:23:57/00:00:00, RP 0.0.0.0, flags: DJCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet1/1, Forward/Sparse-Dense, 01:23:57/00:00:00

(*, 224.1.1.1), 01:19:26/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet1/1, Forward/Sparse-Dense, 01:19:26/00:00:00
Ethernet1/2, Forward/Sparse-Dense, 01:19:26/00:00:00

(1.1.1.1, 224.1.1.1), 00:17:46/00:02:59, flags: CTA
Incoming interface: Ethernet1/1, RPF nbr 2.1.1.1
Outgoing interface list:
Ethernet1/2, Forward/Sparse-Dense, 00:17:46/00:00:00

Beautifully done.

`TTL Threshold`

+---+
source----e0/0-|75a|-e0/1----receiver
1.1.1.1, +---+
224.1.1.1

Problem:

The receiver is not getting the multicast packets from the source.

Troubleshooting:

There may be several routers between the source and the 75a router, but let's first take a look at the 75a since it's directly connected to the receiver:

(*, 224.1.1.1), 00:32:05/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse-Dense, 00:08:17/00:00:00

(1.1.1.1, 224.1.1.1), 00:01:02/00:01:57, flags: CTA
Incoming interface: Ethernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/1, Forward/Sparse-Dense, 00:01:02/00:00:00

This is showing that 75a router is indeed forwarding the packets out ethernet0/1 so there is no need to work our way back towards the source. So, why isn't the receiver getting the packets?

Perhaps the receiver is messed up. That would be the obvious conclusion. What more can the router do then to forward the packets? But the customer is saying they put a sniffer on the line and they don't see any multicast packets. So we point the finger at the PC and they point the finger at the router.

We better ensure the router is forwarding the packets, just to give further ammo to the customer to take to his pc application vendor. We'll turn on debug just for this source and multicast group to help keep the chatter down a bit:

ip22-75a#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ip22-75a(config)#access-list 101 permit udp host 1.1.1.1 host 224.1.1.1
ip22-75a(config)#end
ip22-75a#debug ip mpacket 101
IP multicast packets debugging is on for access list 101
ip22-75a#
*Jan 17 09:04:08.714: IP: s=1.1.1.1 (Ethernet0/0) d=224.1.1.1 len 60, threshold denied
*Jan 17 09:04:08.762: IP: s=1.1.1.1 (Ethernet0/0) d=224.1.1.1 len 60, threshold denied
*Jan 17 09:04:08.814: IP: s=1.1.1.1 (Ethernet0/0) d=224.1.1.1 len 60, threshold denied

Wo! we need to eat our words, the router is not forwarding the packets because of a threshold being denied. We better look in the config and find out what this is all about, meanwhile eating crow...

And we find the culprit:

interface Ethernet0/1
ip address 2.1.1.1 255.255.255.0
ip pim sparse-dense-mode
ip multicast ttl-threshold 15

The customer configured a ttl threshold of 15, thinking that anything greater than a ttl of 15 will not be sent. Actually, just the opposite is configured. The application is being sent with a ttl of 15. By the time it gets to the 75a router, the multicast packets have a ttl less than 15. We better look at ftp://ftp-eng/ipmulticast/Multicast-Commands, and see what is going
on:

[no] ip multicast ttl-threshold <ttl-value>
   Configures a packet TTL threshold for traffic going out the interface.
   Any multicast packets with a TTL less than the threshold are not
   forwarded out the interface. The default value is 0 which means all
   multicast packets are forwarded out interface. [10.2]

So any packets with a ttl LESS than the threshold are not forwarded. This command is usually used to provide a border to keep internal multicast traffic from drifting out of the intranet.

Resolution: Either remove this command (and use 'ip pim border' instead) or lower the ttl threshold so that the traffic can pass.

Note:
Bug CSCdp69540
Headline : Display ttl-threshold in show ip mroute when configured on oif

This bug was created to help ease troubleshooting this situation. It will prevent needing to run debugs.

`Equal Cost Paths`

         +---+              +---+
         |   |-e0/1-----e1/1|   |
S---e0/0-|75a|-e0/2-----e1/2|75b|-e1/4---R
         |   |-e0/3-----e1/3|   |
         +---+              +---+

S=1.1.1.1, 224.1.1.1

Problem:

75b has 3 equal cost paths to chose from to get back to the source (1.1.1.1) and its chosing a link we don't want to be first choice as the rpf link.

Solution:

First, when faced with multiple equal cost routes to a source, multicast will chose one as the incoming interface. The interface chosen is the pim neighbor with the highest ip address. Prunes will be sent to pim neighbors on the other links.

So, your options are:

1)only configure pim on the interfaces of which you want multicast to traverse. You lose multicast redundancy.
or
2)change the subnets so as to make the highest ip address on the link you want to be the primary multicast link.
or
3)create a static mroute pointing out the preferred multicast interface. You lose multicast redundancy.

Using the static mroute as an example:

before the static mroute:

ip22-75b#sh ip route 1.1.1.1
Routing entry for 1.1.1.0/24
Known via "ospf 1", distance 110, metric 20, type intra area
Redistributing via ospf 1
Last update from 3.1.1.1 on Ethernet1/2, 00:26:21 ago
Routing Descriptor Blocks:
* 4.1.1.1, from 10.0.119.66, 00:26:21 ago, via Ethernet1/3
      Route metric is 20, traffic share count is 1
    2.1.1.1, from 10.0.119.66, 00:26:21 ago, via Ethernet1/1
      Route metric is 20, traffic share count is 1
    3.1.1.1, from 10.0.119.66, 00:26:21 ago, via Ethernet1/2
      Route metric is 20, traffic share count is 1

ip22-75b#sh ip rpf 1.1.1.1
RPF information for ? (1.1.1.1)
RPF interface: Ethernet1/3
RPF neighbor: ? (4.1.1.1)
RPF route/mask: 1.1.1.0/24
RPF type: unicast (ospf 1)
RPF recursion count: 0
Doing distance-preferred lookups across tables

ip22-75b#sh ip mroute 224.1.1.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned
       R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT
       M - MSDP created entry, X - Proxy Join Timer Running
       A - Advertised via MSDP
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 224.1.1.1), 01:30:34/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
    Ethernet1/4, Forward/Sparse-Dense, 01:30:34/00:00:00
    Ethernet1/1, Forward/Sparse-Dense, 01:30:35/00:00:00
    Ethernet1/2, Forward/Sparse-Dense, 01:30:35/00:00:00
    Ethernet1/3, Forward/Sparse-Dense, 00:24:22/00:00:00

(1.1.1.1, 224.1.1.1), 01:30:35/00:02:59, flags: CT
Incoming interface: Ethernet1/3, RPF nbr 4.1.1.1
Outgoing interface list:
    Ethernet1/1, Prune/Sparse-Dense, 01:30:35/00:02:32
    Ethernet1/4, Forward/Sparse-Dense, 01:30:35/00:00:00
    Ethernet1/2, Prune/Sparse-Dense, 00:24:22/00:02:42

after the static mroute:

ip22-75b#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ip22-75b(config)#ip mroute 0.0.0.0 0.0.0.0 2.1.1.1
ip22-75b(config)#end

ip22-75b#sh ip rpf 1.1.1.1
RPF information for ? (1.1.1.1)
RPF interface: Ethernet1/1
RPF neighbor: ? (2.1.1.1)
RPF route/mask: 1.1.1.1/0
RPF type: static mroute
RPF recursion count: 0
Doing distance-preferred lookups across tables

(*, 224.1.1.1), 01:31:29/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
    Ethernet1/4, Forward/Sparse-Dense, 01:31:29/00:00:00
    Ethernet1/1, Forward/Sparse-Dense, 01:31:30/00:00:00
    Ethernet1/2, Forward/Sparse-Dense, 01:31:30/00:00:00
    Ethernet1/3, Forward/Sparse-Dense, 00:25:17/00:00:00

(1.1.1.1, 224.1.1.1), 01:31:30/00:02:59, flags: CT
Incoming interface: Ethernet1/1, RPF nbr 2.1.1.1, Mroute
Outgoing interface list:
    Ethernet1/4, Forward/Sparse-Dense, 01:31:30/00:00:00
    Ethernet1/2, Prune/Sparse-Dense, 00:25:17/00:01:47
    Ethernet1/3, Prune/Sparse-Dense, 00:00:31/00:02:28

`Load Splitting`

         +---+              +---+
         |   |-e0/1-----e1/1|   |
S---e0/0-|75a|-e0/2-----e1/2|75b|-e1/4---R
         |   |-e0/3-----e1/3|   |
         +---+              +---+

S=1.1.1.1, 224.1.1.1

Problem:

75b has 3 equal cost paths to chose from to get back to the source (1.1.1.1). We want to load balance the multicast traffic across all three links.

Solution:

Since PIM will only chose one interface to rpf over, and prune the others, you need to remove pim from the redundant links and create a tunnel between 75a and 75b. You will then load balance at the link level and ip will run over the tunnel.

75a config:

interface Tunnel0
ip address 6.1.1.1 255.255.255.0
ip pim sparse-dense-mode
tunnel source Ethernet0/0
tunnel destination 5.1.1.1
!
interface Ethernet0/0
ip address 1.1.1.2 255.255.255.0
ip pim sparse-dense-mode
!
interface Ethernet0/1
ip address 2.1.1.1 255.255.255.0
!
interface Ethernet0/2
ip address 3.1.1.1 255.255.255.0
!
interface Ethernet0/3
ip address 4.1.1.1 255.255.255.0

75b config:

interface Tunnel0
ip address 6.1.1.2 255.255.255.0
ip pim sparse-dense-mode
tunnel source Ethernet1/4
tunnel destination 1.1.1.2
!
interface Ethernet1/1
ip address 2.1.1.2 255.255.255.0
!
interface Ethernet1/2
ip address 3.1.1.2 255.255.255.0
!
interface Ethernet1/3
ip address 4.1.1.2 255.255.255.0
!
interface Ethernet1/4
ip address 5.1.1.1 255.255.255.0
ip pim sparse-dense-mode
!
ip mroute 0.0.0.0 0.0.0.0 Tunnel0

Turning off multicast fast switching (no ip mroute-cache) will enable process level switching (per packet load splitting). Otherwise, when the default multicast fast switching is enabled, load balancing is performed on a per session basis.

(*, 224.1.1.1), 02:40:31/00:02:59, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense, 00:20:55/00:00:00

(1.1.1.1, 224.1.1.1), 02:40:32/00:03:29, flags: TA
Incoming interface: Ethernet0/0, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense, 00:20:55/00:00:00

(*, 224.1.1.1), 02:42:20/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel0, Forward/Sparse-Dense, 00:22:53/00:00:00
Ethernet1/4, Forward/Sparse-Dense, 02:42:20/00:00:00

(1.1.1.1, 224.1.1.1), 00:22:53/00:02:59, flags: CT
Incoming interface: Tunnel0, RPF nbr 6.1.1.1, Mroute
Outgoing interface list:
Ethernet1/4, Forward/Sparse-Dense, 00:22:53/00:00:00

To check that indeed the multicast data is being load split equally across the three links, you can look at the interface data of 75a:

incoming interface is 9000 bits/sec:

ip22-75a#sh int e 0/0
.
.
5 minute input rate 9000 bits/sec, 20 packets/sec
.

the 3 outgoing interfaces are each showing 3000 bits/sec:

ip22-75a#sh int e 0/1
.
.
5 minute output rate 3000 bits/sec, 7 packets/sec
.

ip22-75a#sh int e 0/2
.
.
5 minute output rate 3000 bits/sec, 7 packets/sec
.

ip22-75a#sh int e 0/3
.
.
5 minute output rate 3000 bits/sec, 7 packets/sec
.

`INVALID_RP_JOIN`

We are receiving the following error messages on the RP:

00:55:15: %PIM-6-INVALID_RP_JOIN: Received (*, 224.1.1.1) Join from 1.1.6.2 for invalid RP 1.1.5.4
00:56:15: %PIM-6-INVALID_RP_JOIN: Received (*, 224.1.1.1) Join from 1.1.6.2 for invalid RP 1.1.5.4

Let's look up the errors on CCO:

Error Message
%PIM-1-INVALID_RP_JOIN: Received (*, [int]) Join from [int] for invalid RP [int]

Explanation
A downstream PIM router sent a join message for the shared tree, which this router does not want to accept. This behavior indicates that this router will let only downstream routers join to a specific rendezvous point.

A related error:
Error Message
%PIM-1-INVALID_RP_REG: Received Register from [int] for [int], not willing to be RP

Explanation
A PIM router received a register message from another PIM router that thinks it is the rendezvous point. If the router is not configured for another rendezvous point, it will not accept the register message.

These descriptions are somewhat helpful but I'm still not really sure what to do. Apparently this router is using a filter to permit only certain RP's from being join so we better look at the config. We find the following:

interface Ethernet0/0
ip address 1.1.5.4 255.255.255.0
ip pim sparse-dense-mode
!
ip pim accept-rp 10.2.2.2 8
ip pim send-rp-announce Ethernet0/0 scope 15
ip pim send-rp-discovery scope 15
!
access-list 8 permit 224.0.0.0 0.255.255.255

What is this accept-rp statement supposed to accomplish? Let's look it up on ftp://ftp-eng/ipmulticast/Multicast-Commands

    [no] ip pim accept-rp {<address> | auto-rp} [<acl>]
        When this command is entered, the router will only accept (*,G) Joins
        with an RP address of <address> if G is in the group range specified
        by <acl>. When this command is entered with an address equal to one of
        the system's addresses, the system will be the RP only for the
        specified group range specified in <acl>. When not in the group range,
        the RP will not accept Joins or Register messages and will respond
        immediately to Registers with Register-Stop messages. There is no
        default setting for this command. [10.2]

When the keyword "auto-rp" is specified, Join and Register messages
will only be accepted for RPs that are in the Auto-RP cache. [11.1]

If <address> is 0.0.0.0, the filter will accept any RP for any
group accepted by <acl>, and deny any RP rejected by <acl>.

        When multiple "ip pim accept-rp" filters are configured, they must
        be configured in the following order (in 11.3(4.5), 12.0(1) and
        newer images the following order is guaranteed by the IOS,
        irrespective of the configuration sequence):

            ip pim accept-rp <specific RP address> <acl>
            ip pim accept-rp auto-rp
            ip pim accept-rp 0.0.0.0 <acl>

         The following example will accept 171.69.58.88 as the RP for groups
         in 239.0.0.0/8, and RPs for groups in the Auto-RP cache. If the
         RP and group don't match the first two filters, the 3rd filter is
         in effect, i.e. any RP is accepted for groups permitted by ACL 2,
         and no RP for 224.0.1.39 and 224.0.1.40 is accepted.

           ip pim accept-rp 171.69.58.88 1
           ip pim accept-rp auto-rp
           ip pim accept-rp 0.0.0.0 2

access-list 1 permit 239.0.0.0 0.255.255.255

          access-list 2 deny 224.0.1.39
          access-list 2 deny 224.0.1.40
          access-list 2 permit any

So we remove the 'ip pim accept-rp' command all together and the messages disappear. We are probably permiting the wrong RP address. Who is the RP supposed to be anyway?:

ip22-75a#sh ip pim rp map
PIM Group-to-RP Mappings
This system is an RP (Auto-RP)
This system is an RP-mapping agent

Group(s) 224.0.0.0/4
RP 1.1.5.4 (?), v2v1
Info source: 1.1.5.4 (?), via Auto-RP
Uptime: 01:00:48, expires: 00:02:07

So, this router is an RP, and the only RP learned via Auto-RP or otherwise. And the RP is only planning on being the RP for the groups 224.0.0.0/4. The pim accept-rp statement is wrong. We could either fix the filter to accept this RP or simply add the auto-rp keyword to the filter. We need to make certain that the access list (8 in this example) is permitting the necessary multicast group range also. If not, you will continue to see this error message.

change this: ip pim accept-rp 10.2.2.2 8 <--wrong rp
to this: ip pim accept-rp 1.1.5.4 8 <--correct rp
or to this: ip pim accept-rp auto-rp <--accept what's in the auto-rp cache

access-list 8 permit 224.0.0.0 0.255.255.255

Messages are now gone. well done.

`CGMP Not Querier`

Problem:

cgmp is not working. The static cam table in the catalyst switch is not showing any of the correct ports being populated. The cgmp configured router is not sending cgmp packets. The typology is:

         +---+           +-----+           +---+
Source---|75a|e0/2----2/3|CAT5K|2/4----e1/3|75b|---Host
1.1.1.1 +---+           +-----+           +---+
            2.1.1.2        2/6         2.1.1.1
                            |
                            |2.1.1.3
                           Host

You've correctly configured 'set cgmp enable' on the switch and you configured 'ip cgmp' on e0/2 of the 75a router. But on the switch, all you see is:

Console> (enable) sh multicast group
CGMP enabled
IGMP disabled
IGMP fastleave disabled
GMRP disabled

VLAN Dest MAC/Route Des [CoS] Destination Ports or VCs / [Protocol Type]
---- ------------------ ----- -------------------------------------------

Total Number of Entries = 0

Zero, zip, zilch. Why? Well that's the problem we need to solve. The output of this command should be listing each port which has a cgmp configured router on it (port 2/3) and each port which has an interested receiver on it (port 2/6). But nothing is listed right now so all ports are needlessly being flooded with this multicast traffic whether they want it or not. Let's take a look a few particulars:

ip22-75a#sh ip pim neigh
PIM Neighbor Table
Neighbor Address Interface Uptime Expires Ver Mode
2.1.1.1 Ethernet0/2 00:07:41 00:01:34 v2

So 75a is able to see 75b as a valid PIM neighbor through the switch, good.

(*, 224.1.1.1), 00:14:55/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/2, Forward/Sparse-Dense, 00:14:55/00:00:00

(1.1.1.1, 224.1.1.1), 00:14:56/00:02:59, flags: CTA
Incoming interface: Ethernet0/1, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet0/2, Forward/Sparse-Dense, 00:14:56/00:00:00

So, 75a is forwarding the multicast packets out e0/2 towards the switch. This also is good. And the following shows that the 75b is getting the multicast packets and forwarding them on correctly:

(*, 224.1.1.1), 00:17:57/00:02:59, RP 0.0.0.0, flags: DJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Ethernet1/3, Forward/Sparse-Dense, 00:17:57/00:00:00
Ethernet1/4, Forward/Sparse-Dense, 00:17:57/00:00:00

(1.1.1.1, 224.1.1.1), 00:00:35/00:02:59, flags: CTA
Incoming interface: Ethernet1/3, RPF nbr 2.1.1.2
Outgoing interface list:
Ethernet1/4, Forward/Sparse-Dense, 00:00:35/00:00:00

On the switch we see:

Console> (enable) sh multicast router
CGMP enabled
IGMP disabled
IGMP fastleave disabled

Port Vlan
--------- ----------------
2/3 6

Total Number of Entries = 1

which is good. Back on 75a we see:

ip22-75a#debug ip cgmp
CGMP debugging is on
*Feb 8 12:45:22.206: CGMP: Sending self Join on Ethernet0/2
*Feb 8 12:45:22.206:       GDA 0000.0000.0000, USA 00d0.ff2f.a002
*Feb 8 12:46:22.234: CGMP: Sending self Join on Ethernet0/2
*Feb 8 12:46:22.234:       GDA 0000.0000.0000, USA 00d0.ff2f.a002
*Feb 8 12:47:22.262: CGMP: Sending self Join on Ethernet0/2
*Feb 8 12:47:22.262:       GDA 0000.0000.0000, USA 00d0.ff2f.a002

0000.0000.0000 is the all-groups address and is used when routers send CGMP Join/Leave messages so switches can populate router ports. GDA=Group Destination Address in MAC level format.

USA=Unicast Source Address. This is the address of the host which originated the IGMP Report for which this CGMP message is being generated. It's the mac address for 75a e0/2:

ip22-75a#sh int e 0/2
Ethernet0/2 is up, line protocol is up
Hardware is cxBus Ethernet, address is 00d0.ff2f.a002 (bia 00d0.ff2f.a002)

We are periodically seeing:

*Feb 8 12:51:41.854: IGMP: Received v2 Report from 2.1.1.3 (Ethernet0/2) for 224.1.1.1

When 'debug ip igmp' is turned on, but we don't subsequently see a corresponding cgmp entry, so this router is receiving the igmp reports but not generating the necessary cgmp packets to help the switch know which ports to block. The following tells us why:

ip22-75a#sh ip igmp int e 0/2
Ethernet0/2 is up, line protocol is up
Internet address is 2.1.1.2/24
IGMP is enabled on interface
Current IGMP version is 2
CGMP is enabled
IGMP query interval is 60 seconds
IGMP querier timeout is 120 seconds
IGMP max query response time is 10 seconds
Last member query response interval is 1000 ms
Inbound IGMP access group is not set
IGMP activity: 3 joins, 1 leaves
Multicast routing is enabled on interface
Multicast TTL threshold is 0
Multicast designated router (DR) is 2.1.1.2 (this system)
IGMP querying router is 2.1.1.1
No multicast groups joined

When you have multiple routers on the same subnet, and both configured for cgmp, only one will be the sender of cgmp packets. The router that will send the cgmp will be the IGMP QUERYING ROUTER (lowest ip address). Since cgmp is only configured on 75a, but the 75a is not the igmp querier, no cgmp packets will be sent. If you then configure cgmp on the 75b, all is well. 75b is the one to watch the debug on:

ip22-75b#conf t
Enter configuration commands, one per line. End with CNTL/Z.
ip22-75b(config)#int e 1/3
ip22-75b(config-if)#ip cgmp
ip22-75b(config-if)#end
ip22-75b#debug ip cgmp
ip22-75b#debug ip igmp
*Feb 8 12:58:42.422: IGMP: Received v2 Report from 2.1.1.3 (Ethernet1/3) for 224.1.1.1
*Feb 8 12:58:42.422: CGMP: Received IGMP Report on Ethernet1/3
*Feb 8 12:58:42.422: from 2.1.1.3 for 224.1.1.1
*Feb 8 12:58:42.422: CGMP: Sending Join on Ethernet1/3
*Feb 8 12:58:42.422: GDA 0100.5e01.0101, USA 0030.b655.a859

75b receives an igmp report from 2.1.1.3 for group 224.1.1.1 and subsequently sends a cgmp join to the switch for the mac equivalent of 224.1.1.1 with the mac address (USA) of the 2.1.1.3 interested host. The switch knows which port that host is on so marks it as open and blocks the others.

Now things should be working on the switch:

Console> (enable) sh multicast group
CGMP enabled
IGMP disabled
IGMP fastleave disabled
GMRP disabled

VLAN Dest MAC/Route Des    [CoS] Destination Ports or VCs / [Protocol Type]
---- ------------------    ----- -------------------------------------------
6     01-00-5e-00-01-28             2/3-4
6     01-00-5e-01-01-01             2/3-4,2/6

Total Number of Entries = 2

Much better. The 224.1.1.1 (01-00-5e-01-01-01) packets are only being sent out switch ports 2/3, 2/4, and 2/6 as they should. Router ports (2/3-2/4) will always have the packets being forwarded. Flooding to all other ports has stopped. Just remember to have cgmp configured on
the igmp querying router.

`Switch Flooding All Ports`

         +---+           +-----+           +---+
Source---|75a|e0/2----2/3|CAT5K|2/4----e1/3|75b|---Host
1.1.1.1 +---+           +-----+           +---+
            2.1.1.2                    2.1.1.1

Problem:

Both routers and cat5k are properly configured for multicast/cgmp. There are no configuration issues. The host is properly getting the multicast traffic but so is every other host which is coming off of the cat5k. The cat5k is flooding the traffic out every port, cgmp is not working.

The switch is looking like this:

Console> (enable) sh multicast group
CGMP enabled
IGMP disabled
IGMP fastleave disabled
GMRP disabled

Total Number of Entries = 1

So, we don't have any groups. Well we do have the 224.0.1.40 group but we'll always have that one. So what could be the problem?

If the source was on the same subnet (vlan) as 75a and 75b, cgmp would be working perfectly. Or, if we had an interested host on this same vlan, cgmp would be working perfectly. The router would send a cgmp self join and the CAT5K would add the router ports and block all other uninterested ports.

The same is not true when the source is on another subnet. Multicast packets, coming from the source, hit the CAT5k and get flooded everywhere within the vlan. This continues and continues until some host in the vlan, coming off a port of the CAT5k, sends an igmp join. Only then, with the receipt of an igmp report, will the RSM send a cgmp packet which causes the CAT5k to add the appropriate host port as forwarding and all other ports blocked (aside from the router ports).

Since the source is not on the same subnet as the routers, the routers will not generate self joins for those groups and stop the flooding even when there are no active receivers. If the source were moved to the vlan, self joins (not just the all zero but the specific group self joins) would then be sent and ports would be pruned despite there being no igmp reports:

'debug ip cgmp' on 75b:

CGMP: Sending self Join on Ethernet1/3
GDA 0100.5e01.0101, USA 00d0.0695.1423

If you used igmp snooping, instead of cgmp, you do not have this issue. IGMP snooping can detect sources for both local and transit subnets.

So, for cgmp to work in this transit type typology, you could either add a host to the same vlan as 75a and 75b:

         +---+           +-----+           +---+
Source---|75a|e0/2----2/3|CAT5K|2/4----e1/3|75b|---HostA
1.1.1.1 +---+           +-----+           +---+
            2.1.1.2        2/6         2.1.1.1
                            |
                            |
                          HostB (2.1.1.3)

Or you could move the source to being on this same subnet:

         +---+           +-----+           +---+
         |75a|e0/2----2/3|CAT5K|2/4----e1/3|75b|---Host
         +---+           +-----+           +---+
            2.1.1.2        2/6         2.1.1.1
                            |
                            |
                          Source (2.1.1.3)

I've decided to move the source to the same subnet and this is what it now looks like on the switch:

Console> (enable) sh multicast router
CGMP enabled
IGMP disabled
IGMP fastleave disabled

Port       Vlan
--------- ----------------
2/3       6
2/4       6

Total Number of Entries = 2
'*' - Configured
Console> (enable)

Console> (enable) sh multicast group
CGMP enabled
IGMP disabled
IGMP fastleave disabled
GMRP disabled

Total Number of Entries = 2

Problem solved. 224.1.1.1 (01-00-5e-01-01-01) is only flooded to the router ports 2/3 and 2/4.

`Switch Flooding 224.0.0.X Packets`

Problem:

cgmp is not working, it is flooding the multicast stream out all switch ports and wasting bandwidth. I'm using multicast group address 225.0.0.1. If I change that to 225.1.1.1 cgmp works fine. What's wrong with using 225.0.0.1? It's not a registered address for a routing protocol or anything, so what's up?

Answer:

A catalyst switch floods multicast packets to 224.0.0.x because those addresses are link local and we want to make sure link local addresses get to all devices on the lan. Catalysts also flood multicast packets to other multicast addresses that are MAC level ambiguous with 224.0.0.x (for instance, 224.0.0.1 and 225.0.0.1 both map to 0100.5e00.0001).

So, we must avoid the use of class D addresses in the following ranges for multicast application data over switched networks.

224.0.0.x (x = 0 to 255)
225.0.0.x
.
239.0.0.x

and

224.128.0.x (x = 0 to 255)
225.128.0.x
.
239.128.0.x

These .128. addresses need to be avoided because of the same MAC address mapping result:

224.128.0.1= 11100000.10000000.00000000.00000001 in binary
^^^^ ^<0><0 > <0 ><0 > <0 ><1 >
lost 5

It becomes: 0100.5e00.0001 which is the same mac for 224.0.0.1, etc.

All IP multicast frames use MAC layer addresses beginning with the 24-bit prefix of 0x0100.5exx.xxxx. Since all layer 3 ip multicast addresses have the first 4 of the 32 bits set to 0x1110, this leaves 28 bits. These 28 bits must map into only 23 bits of multicast address information. We are left with 23 bits of MAC address space for mapping layer 3 multicast addresses into layer 2 mac addresses.