Analyzing the Impact of Failures (and letting loose a Chaos Monkey)

Planned (maintenance) and unplanned failure of nodes and interfaces in the network is a frequent occurrence. While most networks are designed to be tolerant to such failures, gaining confidence that they are in fact tolerant is difficult. Network engineers often reason about network behavior under failures manually, which is a complex and error-prone task. Consequently, the network could be one link failure away from an outage that leads to a massive loss of revenue and reputation.

Fortunately, based just on device configurations, Batfish makes it easy to proactively analyze the network behavior under failures and offer guarantees on its tolerance to a range of failure scenarios.

In this notebook, we will show how to use Batfish to analyze network behavior under failures. Specifically, we will describe how to simulate a specific network failure scenario, how to check forwarding changes for all flows in that scenario, and finally how to identify vulnerabilities using Chaos Monkey style testing.

Check out a video demo of this notebook here.

Initialization

We will use the example network shown below with three autonomous systems (ASes) spread that connect via eBGP. Within each AS iBGP and OSPF is used. The configurations of these devices are available here.

example-network

[1]:

# Import packages
%run startup.py
bf = Session(host="localhost")

# Initialize the example network and snapshot
NETWORK_NAME = "example_network"
BASE_SNAPSHOT_NAME = "base"

SNAPSHOT_PATH = "networks/failure-analysis"

bf.set_network(NETWORK_NAME)
bf.init_snapshot(SNAPSHOT_PATH, name=BASE_SNAPSHOT_NAME, overwrite=True)

[1]:

'base'

bf.fork_snapshot: Simulating network failures

To simulate network failures, Batfish offers a simple API bf.fork_snapshot that clones the original snapshot to a new one with the specified failure scenarios.

Suppose we want to analyze the scenario where node London fails. We can use bf.fork_snapshot to simulate this failure as shown below.

[2]:

# Fork a new snapshot with London deactivated
FAIL_LONDON_SNAPSHOT_NAME = "fail_london"
bf.fork_snapshot(BASE_SNAPSHOT_NAME, FAIL_LONDON_SNAPSHOT_NAME, deactivate_nodes=["london"], overwrite=True)

[2]:

'fail_london'

In the code, bf.fork_snapshot accepts four parameters: BASE_SNAPSHOT_NAME indicates the original snapshot name, FAIL_LONDON_SNAPSHOT_NAME is the name of the new snapshot, deactivate_nodes is a list of nodes that we wish to fail, and overwrite=True indicates that we want to reinitialize the snapshot if it already exists.

In addition to deactivate_nodes, bf.fork_snapshot can also take deactivate_interfaces as a parameter to simulate interface failures. Combining these functions, Batfish allows us to simulate complicated failure scenarios involving interfaces and nodes, for example: bf.fork_snapshot(BASE_SNAPSHOT_NAME, FAIL_SNAPSHOT_NAME, deactivate_nodes=FAIL_NODES, deactivate_interfaces=FAIL_INTERFACES, overwrite=True)).

To understand network behavior under the simulated failure, we can run any Batfish question on the newly created snapshot. As an example, to ensure that the flows from Paris would still reach PoP even if London failed, we can run the traceroute question on the snapshot in which London has failed, as shown below. (See the Introduction to Forwarding Analysis using Batfish notebook for more forwarding analysis questions).

[3]:

# Get the answer of a traceroute question from Paris to the PoP's prefix
pop_prefix = "2.128.0.0/24"
tr_answer = bf.q.traceroute(
    startLocation="paris",
    headers=HeaderConstraints(dstIps=pop_prefix),
    maxTraces=1
).answer(FAIL_LONDON_SNAPSHOT_NAME)

# Display the result in a pretty form
show(tr_answer.frame())
  Flow Traces TraceCount
0 Start Location: paris
Src IP: 1.0.1.2
Src Port: 49152
Dst IP: 2.128.0.0
Dst Port: 33434
IP Protocol: UDP		 EXITS_NETWORK
1. node: paris
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 1.0.2.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
2. node: milan
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 10.13.22.3, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
3. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
6. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
7. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
8. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
9. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
1 Start Location: paris
Src IP: 1.0.2.2
Src Port: 49152
Dst IP: 2.128.0.0
Dst Port: 33434
IP Protocol: UDP		 EXITS_NETWORK
1. node: paris
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 1.0.2.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
2. node: milan
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 10.13.22.3, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
3. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
6. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
7. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
8. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
9. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
2 Start Location: paris
Src IP: 1.10.1.1
Src Port: 49152
Dst IP: 2.128.0.0
Dst Port: 33434
IP Protocol: UDP		 EXITS_NETWORK
1. node: paris
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 1.0.2.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
2. node: milan
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 10.13.22.3, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
3. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
6. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
7. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
8. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
9. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
3 Start Location: paris
Src IP: 90.90.90.2
Src Port: 49152
Dst IP: 2.128.0.0
Dst Port: 33434
IP Protocol: UDP		 EXITS_NETWORK
1. node: paris
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 1.0.2.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
2. node: milan
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 10.13.22.3, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
3. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
6. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
7. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
8. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
9. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4

Great! We have confirmed that Paris can still reach PoP via Asia even when London has failed.

differentialReachability: Checking changes of forwarding behavior for all flows

Above, we saw how Batfish can create new snapshots that simulate failure scenarios and run analysis on them. This capability is useful to test the forwarding behavior under interesting failure scenarios. In some cases, we may also want to verify that certain network failures have no impact to the network, i.e., the forwarding behavior of all flows would not be changed by those failures.

We now show a powerful question differentialReachability of Batfish, which allows us to analyze changes of any flow between two snapshots. This question will report any flow that was successfully delivered in the base snapshot but will not be delivered in failure snapshot or the other way around—not delivered in the base snapshot but delivered in the failure snapshot.

Let us revisit the scenario where London fails. To understand if this failure impacts any flow to the PoP in the US, we can run the differential reachability question as below, by scoping the search to flows destined to the PoP(from anywhere) and comparing FAIL_LONDON_SNAPSHOT_NAME with BASE_SNAPSHOT_NAME as the reference. Leaving the headers field unscoped would search across flows to all possible destinations.

[4]:

# Get the answer to the differential reachability question given two snapshots
diff_reachability_answer = bf.q.differentialReachability(
    headers=HeaderConstraints(dstIps=pop_prefix), maxTraces=1).answer(
    snapshot=FAIL_LONDON_SNAPSHOT_NAME,
    reference_snapshot=BASE_SNAPSHOT_NAME)

# Display the results
show(diff_reachability_answer.frame())
  Flow Snapshot_Traces Snapshot_TraceCount Reference_Traces Reference_TraceCount
0 Start Location: milan
Src IP: 1.0.2.1
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 EXITS_NETWORK
1. node: milan
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 10.13.22.3, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
2. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
6. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
8. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4 DENIED_IN
1. node: milan
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 1.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.12.11.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: paris
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 1.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.12.11.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: london
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.12.11.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.12.11.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: newyork
  RECEIVED(GigabitEthernet0/0)
  DENIED(AS1_TO_AS2 (INGRESS_FILTER))		 1
1 Start Location: milan interface=GigabitEthernet1/0
Src IP: 1.0.2.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 EXITS_NETWORK
1. node: milan
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 10.13.22.3, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
2. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
6. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
8. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4 DENIED_IN
1. node: milan
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 1.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.12.11.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: paris
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 1.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.12.11.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: london
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.12.11.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.12.11.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: newyork
  RECEIVED(GigabitEthernet0/0)
  DENIED(AS1_TO_AS2 (INGRESS_FILTER))		 1
2 Start Location: paris interface=GigabitEthernet0/0
Src IP: 1.0.2.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 EXITS_NETWORK
1. node: paris
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 1.0.2.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
2. node: milan
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet0/0 with resolved next-hop IP: 10.13.22.3, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.13.22.3)])
  TRANSMITTED(GigabitEthernet0/0)
3. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
6. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
7. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
8. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
9. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4 DENIED_IN
1. node: paris
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 1.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.12.11.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: london
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.12.11.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.12.11.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: newyork
  RECEIVED(GigabitEthernet0/0)
  DENIED(AS1_TO_AS2 (INGRESS_FILTER))		 1

We see from the result that the failures of London would in fact permit a flow that was originally being blocked by the AS1_TO_AS2 ACL on New York. This difference reveals a potential security vulnerability! Luckily, Batfish allows us to catch and fix it before something bad happens in production. Similarly, if there were flows that were carried in BASE_SNAPSHOT_NAME but dropped in FAIL_LONDON_SNAPSHOT_NAME (an availability issue), Batfish would have caught it.

Check out our Introduction to Forwarding Change Validation notebook for more use cases of differential reachability queries.

Chaos Monkey Testing

Chaos Monkey style testing is a common method to to build highly reliable software systems. In it, different components of the system are randomly failed to see what impact it has on the service performance. Such testing is known to be highly effective but is not possible to do in the networking context. Until now.

Batfish can easily enable Chaos Monkey testing for networks. Using the basic functions shown above, we can compose more complicated functions that randomly fail links and identify potential vulnerabilities in the network.

Suppose we wanted our network to be robust to any possible 2-link failures. The example below shows how to perform Chaos Monkey testing to identify 2-link-failures that can cause an outage. Specifically, we will fail a pair of links picked at random and check whether the forwarding behavior would be changed by the failure using the differentialReachability question.

Next, we run Chaos Monkey testing, shown as below.

[5]:

# Fix for demonstration purpose
random.seed(0)

max_iterations = 5

# Get all links in the network
links = bf.q.edges().answer(BASE_SNAPSHOT_NAME).frame()

for i in range(max_iterations):
    # Get two links at random
    failed_link1_index = random.randint(0, len(links) - 1)
    failed_link2_index = random.randint(0, len(links) - 1)

    # Fork a snapshot with the link failures
    FAIL_SNAPSHOT_NAME = "fail_snapshot"
    bf.fork_snapshot(
        BASE_SNAPSHOT_NAME,
        FAIL_SNAPSHOT_NAME,
        deactivate_interfaces=[links.loc[failed_link1_index].Interface,
                               links.loc[failed_link2_index].Interface],
        overwrite=True)

    # Run a differential reachability question
    answer = bf.q.differentialReachability(
        headers=HeaderConstraints(dstIps=pop_prefix)
    ).answer(
        snapshot=FAIL_SNAPSHOT_NAME,
        reference_snapshot=BASE_SNAPSHOT_NAME
    )

    # A non-empty returned answer means changed forwarding behavior
    # We print the bad failure scenario and exit
    if len(answer.frame()) > 0:
        show(links.iloc[[failed_link1_index, failed_link2_index]])
        break
  Interface IPs Remote_Interface Remote_IPs
32 seattle[GigabitEthernet2/0] 2.12.21.1 philadelphia[GigabitEthernet1/0] 2.12.21.2
31 seattle[GigabitEthernet1/0] 2.12.22.1 sanfrancisco[GigabitEthernet0/0] 2.12.22.2

We see that there is a failure scenario under to which the network is not robust, that is, the failure will lead to a change in the forwarding behavior of at least some flows. This scenario is the failure of two links that connect Seattle to Philadelphia and San Francisco. This is unexpected because Seattle has another link that connects it to the rest of the network and should generally be available for traffic.

Let us diagnose this situation to understand the problem. To begin, we first see which flows are impacted.

[6]:

show(answer.frame())
  Flow Snapshot_Traces Snapshot_TraceCount Reference_Traces Reference_TraceCount
0 Start Location: hongkong
Src IP: 3.0.1.2
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: hongkong
  ORIGINATED(default)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: hongkong
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
5. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
5. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
1 Start Location: hongkong interface=GigabitEthernet0/0
Src IP: 3.0.2.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
5. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
5. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
2 Start Location: hongkong interface=GigabitEthernet1/0
Src IP: 3.0.1.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: hongkong
  RECEIVED(GigabitEthernet1/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
5. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
5. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
3 Start Location: hongkong interface=GigabitEthernet2/0
Src IP: 90.90.90.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: hongkong
  RECEIVED(GigabitEthernet2/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
5. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: hongkong
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
5. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
4 Start Location: seattle
Src IP: 10.23.21.2
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: seattle
  ORIGINATED(default)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: seattle
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: seattle
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
3. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: seattle
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
2. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: seattle
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
2. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
3. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
5 Start Location: seattle interface=GigabitEthernet0/0
Src IP: 10.23.21.1
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: seattle
  RECEIVED(GigabitEthernet0/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
3. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
2. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
2. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
3. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
6 Start Location: singapore
Src IP: 10.13.22.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: singapore
  ORIGINATED(default)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: singapore
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
6. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
6. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
7 Start Location: singapore interface=GigabitEthernet0/0
Src IP: 10.13.22.2
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: singapore
  RECEIVED(GigabitEthernet0/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
6. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
6. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
8 Start Location: singapore interface=GigabitEthernet1/0
Src IP: 3.0.2.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: singapore
  RECEIVED(GigabitEthernet1/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
6. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
6. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
9 Start Location: singapore interface=GigabitEthernet2/0
Src IP: 30.0.2.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: singapore
  RECEIVED(GigabitEthernet2/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
5. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
6. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
6. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: singapore
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.2.2, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: hongkong
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 3.0.1.1, Routes: [ibgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
4. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
6. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
7. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
10 Start Location: tokyo
Src IP: 10.23.21.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: tokyo
  ORIGINATED(default)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: tokyo
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
4. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  ORIGINATED(default)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
4. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
11 Start Location: tokyo interface=GigabitEthernet0/0
Src IP: 3.0.1.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
4. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
4. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
12 Start Location: tokyo interface=GigabitEthernet1/0
Src IP: 10.23.21.1
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: tokyo
  RECEIVED(GigabitEthernet1/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
4. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
4. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4
13 Start Location: tokyo interface=GigabitEthernet2/0
Src IP: 30.0.2.3
Dst IP: 2.128.0.0
IP Protocol: ICMP (type=8, code=0)		 NO_ROUTE
1. node: tokyo
  RECEIVED(GigabitEthernet2/0)
  NO_ROUTE(Discarded)		 1 EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.22.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: losangeles
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 2.12.22.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet1/0)
3. node: sanfrancisco
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.21.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet3/0)
4. node: washingtondc
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.23.11.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
4. node: washingtondc
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.101.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)

EXITS_NETWORK
1. node: tokyo
  RECEIVED(GigabitEthernet2/0)
  FORWARDED(Forwarded out interface: GigabitEthernet1/0 with resolved next-hop IP: 10.23.21.2, Routes: [bgp (Network: 2.128.0.0/16, Next Hop: ip 10.23.21.2)])
  TRANSMITTED(GigabitEthernet1/0)
2. node: seattle
  RECEIVED(GigabitEthernet0/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.12.21.2, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.101.4),ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
3. node: philadelphia
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet3/0 with resolved next-hop IP: 2.23.12.3, Routes: [ibgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet3/0)
4. node: losangeles
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0 with resolved next-hop IP: 2.34.201.4, Routes: [bgp (Network: 2.128.0.0/24, Next Hop: ip 2.34.201.4)])
  TRANSMITTED(GigabitEthernet2/0)
5. node: houston
  RECEIVED(GigabitEthernet1/0)
  FORWARDED(Forwarded out interface: GigabitEthernet2/0, Routes: [connected (Network: 2.128.0.0/24, Next Hop: interface GigabitEthernet2/0)])
  TRANSMITTED(GigabitEthernet2/0)
  EXITS_NETWORK(Output Interface: GigabitEthernet2/0, Resolved Next Hop IP: 2.128.0.0)		 4

We see that when the links fail, if we ignore flows that end in Seattle (whose links have failed), a general pattern is that Asia loses connectivity to US. Given the network topology, this is quite surprising because after those failure we would have expected Asia to be able to reach US via Europe.

To investigate further into the root cause, we ask Batfish to show how the BGP RIB in the two cases differ. We do so using the bgpRib question and comparing the two snapshots as in the differential reachability question. We focus on the impacted destination prefix 2.128.0.0/16.

[7]:

diff_routes = bf.q.bgpRib(network="2.128.0.0/16").answer(snapshot=FAIL_SNAPSHOT_NAME,
                                                         reference_snapshot=BASE_SNAPSHOT_NAME)
diff_routes

[7]:
Node VRF Network Entry_Presence Snapshot_Status Reference_Status Snapshot_Next_Hop Reference_Next_Hop Snapshot_Next_Hop_IP Reference_Next_Hop_IP Snapshot_Protocol Reference_Protocol Snapshot_AS_Path Reference_AS_Path Snapshot_Metric Reference_Metric Snapshot_Local_Pref Reference_Local_Pref Snapshot_Communities Reference_Communities Snapshot_Origin_Protocol Reference_Origin_Protocol Snapshot_Origin_Type Reference_Origin_Type Snapshot_Originator_Id Reference_Originator_Id Snapshot_Received_From_IP Reference_Received_From_IP Snapshot_Received_Path_Id Reference_Received_Path_Id Snapshot_Cluster_List Reference_Cluster_List Snapshot_Tunnel_Encapsulation_Attribute Reference_Tunnel_Encapsulation_Attribute Snapshot_Weight Reference_Weight Snapshot_Tag Reference_Tag
0 hongkong default 2.128.0.0/16 Only in Reference None ['BEST'] None ip 10.23.21.2 None 10.23.21.2 None ibgp None 2 None 50 None 350 None ['2:3'] None ibgp None igp None 3.1.1.1 None 3.1.1.1 None 1 None [] None None None 0 None None
1 losangeles default 2.128.0.0/16 Only in Reference None ['BEST'] None ip 2.1.1.2 None 2.1.1.2 None ibgp None None 0 None 100 None [] None ibgp None igp None 2.1.1.2 None 2.1.2.1 None 2 None [33620481] None None None 0 None None
2 losangeles default 2.128.0.0/16 Unchanged ['BEST'] ['BEST'] ip 2.1.1.1 ip 2.1.1.1 2.1.1.1 2.1.1.1 ibgp ibgp 0 0 100 100 [] [] ibgp ibgp igp igp 2.1.1.1 2.1.1.1 2.1.2.1 2.1.2.1 1 1 [33620481] [33620481] None None 0 0 None None
3 philadelphia default 2.128.0.0/16 Only in Reference None ['BEST'] None ip 2.1.1.2 None 2.1.1.2 None ibgp None None 0 None 100 None [] None ibgp None igp None 2.1.1.2 None 2.1.1.2 None 1 None [] None None None 0 None None
4 philadelphia default 2.128.0.0/16 Unchanged ['BEST'] ['BEST'] ip 2.1.1.1 ip 2.1.1.1 2.1.1.1 2.1.1.1 ibgp ibgp 0 0 100 100 [] [] ibgp ibgp igp igp 2.1.1.1 2.1.1.1 2.1.1.1 2.1.1.1 1 1 [] [] None None 0 0 None None
5 sanfrancisco default 2.128.0.0/16 Only in Reference None ['BEST'] None ip 2.1.1.2 None 2.1.1.2 None ibgp None None 0 None 100 None [] None ibgp None igp None 2.1.1.2 None 2.1.1.2 None 1 None [] None None None 0 None None
6 sanfrancisco default 2.128.0.0/16 Unchanged ['BEST'] ['BEST'] ip 2.1.1.1 ip 2.1.1.1 2.1.1.1 2.1.1.1 ibgp ibgp 0 0 100 100 [] [] ibgp ibgp igp igp 2.1.1.1 2.1.1.1 2.1.1.1 2.1.1.1 1 1 [] [] None None 0 0 None None
7 seattle default 2.128.0.0/16 Only in Reference None ['BEST'] None discard None AUTO/NONE(-1l) None aggregate None None 0 None 100 None [] None aggregate None igp None 2.1.1.2 None 0.0.0.0 None None None [] None None None 32768 None None
8 singapore default 2.128.0.0/16 Only in Reference None ['BEST'] None ip 10.23.21.2 None 10.23.21.2 None ibgp None 2 None 50 None 350 None ['2:3'] None ibgp None igp None 3.1.1.1 None 3.10.1.1 None 1 None [50987265] None None None 0 None None
9 tokyo default 2.128.0.0/16 Only in Reference None ['BEST'] None ip 10.23.21.2 None 10.23.21.2 None bgp None 2 None 50 None 350 None ['2:3'] None bgp None igp None 2.1.1.2 None 10.23.21.2 None None None [] None None None 0 None None
10 washingtondc default 2.128.0.0/16 Only in Reference None ['BEST'] None ip 2.1.1.2 None 2.1.1.2 None ibgp None None 0 None 100 None [] None ibgp None igp None 2.1.1.2 None 2.1.2.1 None 2 None [33620481] None None None 0 None None
11 washingtondc default 2.128.0.0/16 Unchanged ['BEST'] ['BEST'] ip 2.1.1.1 ip 2.1.1.1 2.1.1.1 2.1.1.1 ibgp ibgp 0 0 100 100 [] [] ibgp ibgp igp igp 2.1.1.1 2.1.1.1 2.1.2.1 2.1.2.1 1 1 [33620481] [33620481] None None 0 0 None None

We see that routers in Asia (Hongkong, Singapore, and Tokyo) and Seattle do not have any BGP routes to the prefix in the failure snapshot, which they did in the reference snapshot. The missing route in Seattle can be explained via missing routes in Asia since Seattle depended on Asia after losing its two other links.

That Europe still has the routes after the failure alerts us to the possibility of improper filtering of incoming routes in Asia. So, we should check on that. There are many ways to analyze the incoming route filters; we’ll use the definedStructures question of Batfish to extract necessary definitions that we need to view.

[8]:

# View defined structures of type 'route-map' on 'hongkong'
bf.q.definedStructures(types='route-map', nodes="hongkong").answer()

[8]:
Structure_Type Structure_Name Source_Lines
0 route-map as1_to_as3 configs/hongkong.cfg:[119, 120]

We see the route map as1_to_as3 is defined on line 119 and 120. Now we can quickly navigate to the lines in the config file, as showing below.

[9]:

# See the config lines where the route map as1_to_as3 is defined
!cat networks/failure-analysis/configs/hongkong.cfg | head -121 | tail -4

!
route-map as1_to_as3 deny 100
 match ip address 102
!

We see that the route map is denying routes that match the access-list ‘102.’ Let’s look at the definition of this list, which is on lines 115-117 per the defined structures list above.

[10]:

# See the config lines where the access list '102' is defined
!cat networks/failure-analysis/configs/hongkong.cfg | head -118 | tail -5

!
access-list 102 permit ip host 1.0.1.0 host 255.255.255.0
access-list 102 permit ip host 1.0.2.0 host 255.255.255.0
access-list 102 permit ip host 2.128.0.0 host 255.255.0.0
!

We see that this list includes the prefix of interest, which is 2.128.0.0/16 on the last line. Thus, the route map inadvertently blocks the prefix, thus disconnecting Asia from US when Seattle or its links fail.

Without Batfish, it would have been hard to find this vulnerability, but the Chaos Monkey style testing enabled by Batfish makes it easy to find such vulnerabilities before they cause a service outage.

Summary

This notebook demonstrates how Batfish help analyze forwarding behavior in network failures. Specifically, 1. bf.fork_snapshot can clone a snapshot from another with deactivated interfaces and nodes; 2. differentialReachability can check all forwarding behavior changes for all flows between two snapshots; 3. We can build on top of the basic functions to create more involved analysis such as Chaos Monkey testing.

Get involved with the Batfish community

Join our community on Slack and GitHub.