Endpoint Security Groups Explored
In my post about the new features of ACI 5.0 (link) I explained the concept of the ESG shortly. This post explores the Endpoint Security Groups in more detail.
First we need to appreciate the fact that the introduction of the Endpoint Security Groups is the biggest change in the tenant policy model since ACI’s inception. The tenant policy model hasn’t changed in any major way since ACI 1.0 (as far as I know). The introduction of this feature now tells me that Cisco either had a change of heart as to how a tenant should be designed, or they had to based on customer feedback. I’ve built over 15 ACI networks now and one of the biggest challenges I encounter is the transition between the ‘old’ network centric paradigm to the new application centric paradigm.
Like I stated in my previously linked post. Many customers of mine have large vlans with multiple applications in them. Some customers have multiple of these vlans, and often servers from application A are in multiple vlans.
As I see it, ESGs have been created to fix two things:
- Large vlans with multiple applications which you can’t easily split into multiple EPGs due to vlan tags on physical equipment (especially when older switches are connected to ACI)
- Applications (or application tiers) that are spread across vlans which can’t easily be moved into their own EPGs.
So, looking at those two reasons why you want to use ESGs I can only conclude one thing: In a properly designed and implemented (ACI) network you don’t need ESGs. In my opinion ESGs are more of a bandaid to help implement good policy in imperfect networks. And, to be honest. I’ve never encountered a perfect network, without any legacy. So ESGs will be implemented. But, when implementing something it’s good to have an idea of how this works.
We already have EPGs with contracts. Now, with the introduction of ESGs, what changes in regards to these contracts? What happens if you have one contract allowing some kind of traffic, and one contract denying the same kind of traffic? Which contract will take preference?
To figure this out, I created four VMs and two EPGs. Two VMs in each EPG. Debian_1 and Debian_2 are in EPG1 and Debian_3 and Debian_4 are in EPG2. The image below shows the VMs in their EPGs and ESGs.
I’ll explore the following scenarios:
- Debian_2 & Debian_4 in an ESG, no contracts between the EPGs.
- Debian_2 & Debian_4 in an ESG, a contract allowing ICMP between the EPGs.
- Debian_2 & Debian_4 in an ESG, a contract allowing ICMP between the ESG and the EPGs.
- Debian_2 & Debian_4 in an ESG, a contract allowing all traffic with a Taboo contract prohibiting SSH between the EPGs.
- Debian1 & 3 in and ESG and Debian2 & 4 in another ESG.
Based on these scenarios I’ll test which VM can talk to which other VMs. This should teach me the order of contract application.
To quickly show you whether a contract is provided or consumed I’ll use moquery on the APIC. The following commands will be used to verify the contracts:
Consumed contracts within an EPG:
moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
Provided contracts within an EPG:
moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
Here’s an example of the result when no contracts are provided or consumed:
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
And the following shows a provided and consumed contract:
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1/rscons-Only_ICMP
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2/rsprov-Only_ICMP
Almost identical commands can be used to request the same information for an ESG:
Consumed contracts:
moquery -c fvESg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
Provided contracts:
moquery -c fvESg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
The results of the commands for ESGs is identical to the result of EPGs
Debian 2 and 4 in an ESG, no contracts.
Since there are no contracts traffic between EPGs should be prevented. However, I put Debian_2 and Debian_4 in an ESG as shown in the image. Using ICMP I’ll test which systems can reach each other.
So, first to verify there are no contracts present on either the EPGs or ESGs:
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
apic1# moquery -c fvESg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/esg-Debian-2_4
apic1# moquery -c fvESg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/esg-Debian-2_4
As you can see, there are no contracts. Furthermore, there’s just one ESG, called Debian-2_4.
Let’s connect to Debian4 and ping all other hosts.
michael@debian4:~$ ping 100.64.22.11
PING 100.64.22.11 (100.64.22.11) 56(84) bytes of data.
64 bytes from 100.64.22.11: icmp_seq=1 ttl=64 time=0.354 ms
64 bytes from 100.64.22.11: icmp_seq=2 ttl=64 time=0.192 ms
^C
--- 100.64.22.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 7ms
rtt min/avg/max/mdev = 0.192/0.273/0.354/0.081 ms
michael@debian4:~$ ping 100.64.21.11
PING 100.64.21.11 (100.64.21.11) 56(84) bytes of data.
^C
--- 100.64.21.11 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 81ms
michael@debian4:~$ ping 100.64.21.12
PING 100.64.21.12 (100.64.21.12) 56(84) bytes of data.
64 bytes from 100.64.21.12: icmp_seq=1 ttl=63 time=0.248 ms
64 bytes from 100.64.21.12: icmp_seq=2 ttl=63 time=0.273 ms
^C
--- 100.64.21.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 10ms
rtt min/avg/max/mdev = 0.248/0.260/0.273/0.020 ms
Ok, so what we can see here is that Debian4 can reach two other servers. Debian3 (100.64.22.11) and Debian2 (100.64.21.12). This makes sense. Debian2 is part of the same ESG and Debian3 is part of the same EPG. Debian1 is neither, so therefor not reachable.
Debian2 and 4 in an ESG, ICMP contract between EPGs.
On to the next test. In this test we will create a contract allowing ICMP and configure this between EPG1 and EPG2. After that we will do our ping test again.
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2/rscons-Only_ICMP
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1/rsprov-Only_ICMP
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
apic1# moquery -c fvESg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/esg-Debian-2_4
apic1# moquery -c fvESg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/esg-Debian-2_4
Ok, so EPG2 is consuming the Only_ICMP contract and EPG1 is providing this contract. There is only one ESG and this is not providing or consuming any contracts.
michael@debian4:~$ ping 100.64.22.11
PING 100.64.22.11 (100.64.22.11) 56(84) bytes of data.
64 bytes from 100.64.22.11: icmp_seq=1 ttl=64 time=0.136 ms
64 bytes from 100.64.22.11: icmp_seq=2 ttl=64 time=0.256 ms
^C
--- 100.64.22.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 0.136/0.196/0.256/0.060 ms
michael@debian4:~$ ping 100.64.21.11
PING 100.64.21.11 (100.64.21.11) 56(84) bytes of data.
^C
--- 100.64.21.11 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 32ms
michael@debian4:~$ ping 100.64.21.12
PING 100.64.21.12 (100.64.21.12) 56(84) bytes of data.
64 bytes from 100.64.21.12: icmp_seq=1 ttl=63 time=0.160 ms
64 bytes from 100.64.21.12: icmp_seq=2 ttl=63 time=0.157 ms
^C
--- 100.64.21.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 32ms
rtt min/avg/max/mdev = 0.157/0.158/0.160/0.012 ms
The result is exactly the same. We have Debian4 able to ping Debian2 and Debian3. So Debian4 is able to reach the systems within its own EPG and systems within its own ESG. Even though a contract exists between the two EPGs, traffic between the two is not possible for Debian4.
Traffic between Debian1 and Debian3 is possible as can be verified on Debian1:
michael@debian1:~$ ping 100.64.22.11
PING 100.64.22.11 (100.64.22.11) 56(84) bytes of data.
64 bytes from 100.64.22.11: icmp_seq=1 ttl=63 time=1.91 ms
64 bytes from 100.64.22.11: icmp_seq=2 ttl=63 time=0.313 ms
^C
--- 100.64.22.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 2ms
rtt min/avg/max/mdev = 0.313/1.112/1.911/0.799 ms
michael@debian1:~$ ping 100.64.22.12
PING 100.64.22.12 (100.64.22.12) 56(84) bytes of data.
^C
--- 100.64.22.12 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 17ms
So, the contract between the EPGs does work, but is not applicable on ESG to EPG communications. This also means you can’t use ESGs as a kind of special subgroup within an EPG, inheriting existing contracts. This is something to take into account when moving from EPGs to ESGs.
But this directly introduces the question. If I consume the contract on the ESG instead of the contract. Will I be able to ping between Debian1 and Debian4 in that case?
Debian2 and 4 in an ESG, ICMP contract between ESG and EPGs.
So, what happens when we try to apply a contract between an ESG and an EPG? This could be very useful when you have some services in an EPG that you don’t want to put into an ESG, but the systems in the ESG must be able to reach those services. As we’ve seen in the previous test you can’t connect to services when a system is part of an ESG and the EPG it belongs to has a contract with another EPG. That means that you would need a ESG to EPG contract to allow for such traffic.
Unfortunately, apparently that’s not possible:
I hope Cisco will make this a possibility in the near future as I can see its added benefit. Especially since adding systems to an ESG is based on its IP address. Maybe, when we can use other identifiers in the future this becomes less of an issue.
Debian2 and 4 in an ESG, Allow all contract and an ICMP Taboo contract between EPGs.
I think I can do a prediction on this one. We had a ICMP contract between the EPGs in an earlier test, which did not allow traffic between Debian1 and Debian4. What will happen when we put in a taboo contract between the EPGs, explicitly blocking ICMP? We already know that systems in an ESG will not be able to communicate with systems in an EPG, but will the taboo contract take precedence and block traffic within the ESG too?
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1/rscons-Permit_Any
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2/rsprov-Permit_Any
apic1# moquery -c fvESg -x "rsp-subtree=children rsp-subtree-class=fvRsCons" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/esg-Debian-2_4
apic1# moquery -c fvESg -x "rsp-subtree=children rsp-subtree-class=fvRsProv" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/esg-Debian-2_4
apic1# moquery -c fvAEPg -x "rsp-subtree=children rsp-subtree-class=fvRsProtBy" | grep dn | grep ap-ESG-Demo
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG1
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2
dn : uni/tn-Michael/ap-ESG-Demo/epg-EPG2/rsprotBy-NO-ICMP
You can see from the output that I’ve applied a Any traffic filter between the EPGs, but also put in a Taboo contract blocking ICMP. To verify this works we should be able to SSH from Debian1 to Debian3, but we should not be able to ping between Debian1 and Debian3.
michael@debian1:~$ ping 100.64.22.11
PING 100.64.22.11 (100.64.22.11) 56(84) bytes of data.
^C
--- 100.64.22.11 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 56ms
michael@debian1:~$ ssh 100.64.22.11
michael@100.64.22.11's password:
Linux debian3 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64
Last login: Tue Jun 2 15:54:08 2020 from 100.64.21.11
michael@debian3:~$
Ok, so that works as expected. Now, let’s see what happens between Debian2 and Debian4. Do they inherit the Taboo contract?
michael@debian4:~$ ping 100.64.21.12
PING 100.64.21.12 (100.64.21.12) 56(84) bytes of data.
64 bytes from 100.64.21.12: icmp_seq=1 ttl=63 time=0.229 ms
64 bytes from 100.64.21.12: icmp_seq=2 ttl=63 time=0.261 ms
64 bytes from 100.64.21.12: icmp_seq=3 ttl=63 time=0.226 ms
^C
--- 100.64.21.12 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 41ms
rtt min/avg/max/mdev = 0.226/0.238/0.261/0.023 ms
michael@debian4:~$ ssh 100.64.21.12
michael@100.64.21.12's password:
Linux debian2 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64
Last login: Tue Jun 2 13:55:29 2020 from 10.249.112.128
michael@debian2:~$ exit
Clearly not. So Taboo contracts applied on the EPG are not applied to the ESG level.
Debian1 & 3 in and ESG and Debian2 & 4 in another ESG
So, for the last test. We can already guess what will happen here. In one of the first tests we found out that if a system is part of an ESG it can still communicate with systems in the EPG. What then will happen when we put Debian1 in ESG Debian-1_3 and Debian2 in ESG Debian-2_4. Will Debian1 and 2 still be able to communicate with each other?
As this is one of the primary reasons ESGs exist, the answer should be no, they can’t communicate anymore. But let’s test this.
michael@debian1:~$ ping 100.64.21.12
PING 100.64.21.12 (100.64.21.12) 56(84) bytes of data.
64 bytes from 100.64.21.12: icmp_seq=1 ttl=64 time=0.166 ms
64 bytes from 100.64.21.12: icmp_seq=2 ttl=64 time=0.250 ms
^C
--- 100.64.21.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 7ms
rtt min/avg/max/mdev = 0.166/0.208/0.250/0.042 ms
michael@debian1:~$ ssh 100.64.21.12
michael@100.64.21.12's password:
Linux debian2 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64
Last login: Tue Jun 2 16:03:41 2020 from 100.64.21.11
michael@debian2:~$
Well. That means I either completely misunderstood the purpose of this, or I missed something configuring it. Traffic between ESG members that are also part of the same EPG appears to be possible.
It is possible that this behavior is caused by the fact that I’m running all these systems in a regular ESX vDS, so this might be caused by the vDS flooding the same-vlan traffic within the ESX host itself. I need to test this using physical equipment in the lab itself. Next time I’m in the office I’ll test this.
When that ESG member is part of another EPG, traffic isn’t possible, as can be seen here:
michael@debian1:~$ ping 100.64.22.12
PING 100.64.22.12 (100.64.22.12) 56(84) bytes of data.
^C
--- 100.64.22.12 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 67ms
Conclusion
ESGs are a nice new addition to the ACI featureset. It is important to look at your own business case for this and determine whether you need ESGs or not. Currently I still think that you wouldn’t need ESGs if the ACI network is designed from the start with the Application Centric model in mind in which you correctly use EPGs.
However, since that’s not always possible ESGs can be a solution. Based on my tests I can say the following:
- Traffic between an ESG and systems within the same EPG is possible. This should be taken into account (this might actually also be caused by the flooding behavior on the vDS)
- Traffic between ESGs and EPGs is not possible. You can’t define contracts between an ESG and an EPG (except for external EPGs and in-band EPGs). Putting systems in an ESG while using services in an EPG will not be possible.
So, what do you think about ESGs? Is it something you will use? If so, what’s your use case?
