CCIE Challenge 4: Troubleshoot Ticket 2

May 12, 2019

This is the second troubleshooting challenge. I would rate this a 2 / 3p ticket.

You can find the configs of the routers here. For those of you using EVE I included the UNL file in there.

The topology is as follows:

Topology

R1 wants to be able to reach R6 via telnet. Match the output below.

R1#telnet 2002:C0A8:3806::6
Trying 2002:C0A8:3806::6 ... Open

R6>exit

[Connection to 2002:C0A8:3806::6 closed by foreign host]

You are not allowed to change anything on R4.

Solution

There are only two faults in this scenario. To solve this ticket, start by verifying it. After you’ve verified that telnet indeed isn’t working start verifying connectivity:

R1#ping 2002:C0A8:3806::6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2002:C0A8:3806::6, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R1#trace 2002:C0A8:3806::6
Type escape sequence to abort.
Tracing the route to 2002:C0A8:3806::6

  1  *  *  *

No connectivity, and the traceroute fails directly. Maybe it’s routing, so let’s check the routing table:

R1#show ipv6 route
IPv6 Routing Table - default - 4 entries
[SNIP]
S   ::/0 [1/0]
     via Tunnel0, directly connected

There is a static default, pointing towards the tunnel. If that’s the case, maybe there’s no return route. So let’s check R6:

R6#show ipv6 route
IPv6 Routing Table - default - 4 entries
[SNIP]
S   ::/0 [1/0]
     via Tunnel0, directly connected

Great, also a static pointing to the tunnel. So are these tunnels up and are they directly connected or is there something in between?

R1#show int tun0
Tunnel0 is up, line protocol is up
  Hardware is Tunnel
  MTU 17920 bytes, BW 100 Kbit/sec, DLY 50000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 192.168.12.1 (Ethernet0/0)
   Tunnel Subblocks:
      src-track:
         Tunnel0 source tracking subblock associated with Ethernet0/0
          Set of tunnels with source Ethernet0/0, 1 member (includes iterators), on interface <OK>
  Tunnel protocol/transport IPv6 6to4
[SNIP]

R6#show int tun0
Tunnel0 is up, line protocol is up
  Hardware is Tunnel
  MTU 17920 bytes, BW 100 Kbit/sec, DLY 50000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Keepalive not set
  Tunnel source 192.168.46.6 (Ethernet0/0)
   Tunnel Subblocks:
      src-track:
         Tunnel0 source tracking subblock associated with Ethernet0/0
          Set of tunnels with source Ethernet0/0, 1 member (includes iterators), on interface <OK>
  Tunnel protocol/transport IPv6 6to4
  [SNIP]

So, the tunnel is up. The source on both tunnels is eth 0/0, but no destination is specified. We so see someting else: Tunnel protocol/transport IPv6 6to4. What’s this? and how does it work? Apparently there’s something wrong with it, because no traffic traverses the tunnel.

The observant reader might have noticed directly at the start of this ticket that it would involve a 6to4 tunnel, since 6to4 tunnels use the 2002::/16 range.

To be able to solve this, you should know how a 6to4 tunnel determines the destination. The ipv6 address of the tunnel contains the IPv4 address of the outside interface. This is how a 6to4 tunnel knows to which IPv4 address to send the data. To get from an IPv4 to 6to4 IPv6 address some calculation is required. We need to check whether the IPv6 addresses on the tunnel interfaces are correct. Either one could be wrong.

So how do you get from IPv4 to IPv6 6to4?

the 6to4 address format is easy. It always starts with 2002. After that the IPv4 address is converted into hex and that’s it. So for R1’s IPv4 address of its tunnel0 interface:

IP address: 192.168.12.1
Hex: C0A8:0C01
IPv6 prefix: 2002:C0A8:0C01::/48

So, in the case of R1, the IPv6 address appears to be correct. You might be thinking, in the lab I don’t have time to convert IPv4 addresses to hex. There’s a sneaky shortcut for this. I’ll show you:

R1(config)#ipv6 general-prefix SHORTCUT 6to4 ethernet 0/0
R1(config)#do show ipv6 general-prefix
IPv6 Prefix SHORTCUT, acquired via 6to4
  2002:C0A8:C01::/48 Valid lifetime infinite, preferred lifetime infinite

The router has been so nice to translate the IPv4 address into a IPv6 prefix for us. This saves a lot of time, but don’t forget to remove the general prefix again. You would probably not lose points on it, but why take the risk.

Using the above command we check on R6 to see whether the IPv6 address is correct:

R6(config)#do sh ipv6 gene
IPv6 Prefix SHORTCUT, acquired via 6to4
  2002:C0A8:2E06::/48 Valid lifetime infinite, preferred lifetime infinite

No, it isn’t. But we need to match the output, so what could be the solution in this case? R6 has two uplinks and a loopback. Could it be that the source of the tunnel is wrong? Let’s check eth 0/1:

R6(config)#ipv6 general-prefix SHORTCUT 6to4 ethernet 0/1
R6(config)#do sh ipv6 gene                               
IPv6 Prefix SHORTCUT, acquired via 6to4
  2002:C0A8:3806::/48 Valid lifetime infinite, preferred lifetime infinite

That looks like the address we’re trying to reach. So let’s change the tunnel source on R6:

R6(config)#int tun0
R6(config-if)#tunn so eth 0/1

Now verify the ticket again:

R1#ping 2002:C0A8:3806::6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2002:C0A8:3806::6, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

Hmm, still not working. But we’re pretty sure the right tunnel interface has been configured now. So what else could go wrong here? The underlay transport of course. The tunnel must be built between two IPv4 addresses, so can those addresses reach each other?

Let’s verify:

R1#ping 192.168.56.6 so eth 0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.56.6, timeout is 2 seconds:
Packet sent with a source address of 192.168.12.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

They can. So routing is fine there. There must be something along the route preventing traffic in the tunnel. We were given a single restriction, do not change anything on R4. So far we haven’t even looked at R4, but I’m betting something is fishy on R4. But before we start spending time on R4, let’s check whether our traffic actually passes R4:

R1#trace 192.168.56.6 so eth 0/0
Type escape sequence to abort.
Tracing the route to 192.168.56.6
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.12.2 0 msec 0 msec 0 msec
  2 192.168.24.4 0 msec 0 msec 0 msec
  3 192.168.46.6 0 msec *  0 msec

It does. So let’s take a look on R4. But what are we looking for. We don’t actually know yet. So lets’s do a quick generic review of R4. We try to avoid looking at configuration as much as possible, but this reeks of some kind of filtering. So let’s take a shortcut here and use a very handy command to quickly check the config on filtering:

R4#show run | s access-list|prefix-list|route-map|policy-map
policy-map default
 class default
  police 8000 conform-action drop
ip access-list extended default
 permit 41 any any

Whoa, what do we have here? A policy map that is dropping all traffic in class default. And an access list that permits protocol 41. (any ideas what that may be?).

So Apparently R4 is dropping all IPv6 traffic that is encapsulated into IP. Our restriction states that we’re not allowed to change anything on R4, so we need to get our traffic around R4. So the traffic from R1 to R6 needs to go through R5, but not through R4. Any way to do this is fine, playing with delay, offset lists or summarization are your best bets. However, just implementing a summary might not help here, since the more specifics might be advertised to R4 by R5 so traffic might still traverse R4. We need to make R4 very unattractive.

The easiest way to do that is to increase the delay on R2’s eth 0/1 interface. Just keep in mind that increasing the delay influences all routes that might traverse that path. If you want to be more specific, use distribute lists or offset lists.

R2(config)#int eth 0/1
R2(config-if)#delay 9999

So let’s verify again. We’ll start with a ping:

R1#ping 2002:C0A8:3806::6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2002:C0A8:3806::6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

That looks great. Now for the real test:

R1#telnet 2002:C0A8:3806::6
Trying 2002:C0A8:3806::6 ... Open
R6>exit

[Connection to 2002:C0A8:3806::6 closed by foreign host]

Great! Another ticket fixed.

So just to summarize:

The wrong source interface was used on R6 for the 6to4 tunnel
The route traversed R4 which dropped the traffic and we needed to route around this.