Cloud1 doesn't allow nodes to communicate

[bingbucks]

Hey everyone,

I've got a baremetal EVE Community install on a SuperMicro (specs below) that from within a lab won't let 2 (Arista vEOS) nodes communicate through Cloud1. This should just be a simple L2 connection, but it seems that no unicast packets are going thru Cloud1, but oddly broadcast packets (ARPs) seem to make it. It's almost like a promiscuous mode issue.

Note there are 4 real NIC interfaces (eth0, eth1, eth2, eth3) on a single 4-port NIC. Only eth0 has something physically plugged into it and that interface works on the Cloud0 subnet of 10.81.45.0/24

I've got a few co-workers who also have EVE Community and we've compared everything that we can think of and one odd difference is that my "pnet1" (which is associated with Cloud1) is DOWN (see attached picture with the word DOWN circled in red).

See additional pictures also of simple lab, selection of Cloud1 for the cloud node called "internal", and more. Server specs are below also.

I assure you that the configurations on the Arista management interfaces are correct and have IP addresses on the same subnet (10.0.69.0/24). Like I said earlier, ARP request and replies are making it between the 2, but I can't ping or SSH to each other.

Any help would be appreciated. Otherwise, I'll try a reinstall of Ubuntu and EVE. I'd like to get this working before I buy the PRO license.

Thanks,
bingbucks

**** OUTPUT OF ifconfig *****
$ ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:28:96:33:e9
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth0 Link encap:Ethernet HWaddr ac:1f:6b:8a:81:f4
inet6 addr: fe80::ae1f:6bff:fe8a:81f4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:73831 errors:0 dropped:998 overruns:0 frame:0
TX packets:25311 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15021423 (15.0 MB) TX bytes:13535303 (13.5 MB)

eth1 Link encap:Ethernet HWaddr ac:1f:6b:8a:81:f5
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth2 Link encap:Ethernet HWaddr ac:1f:6b:1a:fb:10
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

eth3 Link encap:Ethernet HWaddr ac:1f:6b:1a:fb:11
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:1059481 errors:0 dropped:0 overruns:0 frame:0
TX packets:1059481 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:115076582 (115.0 MB) TX bytes:115076582 (115.0 MB)

pnet0 Link encap:Ethernet HWaddr ac:1f:6b:8a:81:f4
inet addr:10.81.45.161 Bcast:10.81.45.255 Mask:255.255.255.0
inet6 addr: fe80::ae1f:6bff:fe8a:81f4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:71988 errors:0 dropped:90 overruns:0 frame:0
TX packets:19642 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:13712448 (13.7 MB) TX bytes:13160973 (13.1 MB)

pnet1 Link encap:Ethernet HWaddr ac:1f:6b:8a:81:f5
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

pnet2 Link encap:Ethernet HWaddr ac:1f:6b:1a:fb:10
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

pnet3 Link encap:Ethernet HWaddr ac:1f:6b:1a:fb:11
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

pnet4 Link encap:Ethernet HWaddr 620f:e4:62:f4
inet6 addr: fe80::60de:fff:fee4:62f4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1628 (1.6 KB)

pnet5 Link encap:Ethernet HWaddr 2a:55:6a:8f:d9:3e
inet6 addr: fe80::2855:6aff:fe8f:d93e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1628 (1.6 KB)

pnet6 Link encap:Ethernet HWaddr c6:46:9e:29:1b:b8
inet6 addr: fe80::c446:9eff:fe29:1bb8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1628 (1.6 KB)

pnet7 Link encap:Ethernet HWaddr e2:8f:72:dc:e9:41
inet6 addr: fe80::e08f:72ff:fedc:e941/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1698 (1.6 KB)

pnet8 Link encap:Ethernet HWaddr 6aab:a0:ea:e6
inet6 addr: fe80::68ab:abff:fea0:eae6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1698 (1.6 KB)

pnet9 Link encap:Ethernet HWaddr de:22:3f:76:83:5e
inet6 addr: fe80::dc22:3fff:fe76:835e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1698 (1.6 KB)


****** OUTPUT of /etc/network/interfaces

$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
iface eth0 inet manual
auto pnet0
iface pnet0 inet static
address 10.81.45.161
netmask 255.255.255.0
gateway 10.81.45.1
dns-domain <REDACTED>
dns-nameservers <REDACTED>
bridge_ports eth0
bridge_stp off

# Cloud devices
iface eth1 inet manual
auto pnet1
iface pnet1 inet manual
bridge_ports eth1
bridge_stp off

iface eth2 inet manual
auto pnet2
iface pnet2 inet manual
bridge_ports eth2
bridge_stp off

iface eth3 inet manual
auto pnet3
iface pnet3 inet manual
bridge_ports eth3
bridge_stp off

iface eth4 inet manual
auto pnet4
iface pnet4 inet manual
bridge_ports eth4
bridge_stp off

iface eth5 inet manual
auto pnet5
iface pnet5 inet manual
bridge_ports eth5
bridge_stp off

iface eth6 inet manual
auto pnet6
iface pnet6 inet manual
bridge_ports eth6
bridge_stp off

iface eth7 inet manual
auto pnet7
iface pnet7 inet manual
bridge_ports eth7
bridge_stp off

iface eth8 inet manual
auto pnet8
iface pnet8 inet manual
bridge_ports eth8
bridge_stp off

iface eth9 inet manual
auto pnet9
iface pnet9 inet manual
bridge_ports eth9
bridge_stp off

***** OUTPUT of uname
$ uname -a
Linux <REDACTED> 4.9.40-eve-ng-ukms-2+ #4 SMP Fri Sep 15 02:07:02 CEST 2017 x86_64 x86_64 x86_64 GNU/Linux

***** OUTPUT of "ip a"

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master pnet0 state UP group default qlen 1000
link/ether ac:1f:6b:8a:81:f4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::ae1f:6bff:fe8a:81f4/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master pnet1 state DOWN group default qlen 1000
link/ether ac:1f:6b:8a:81:f5 brd ff:ff:ff:ff:ff:ff
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master pnet2 state DOWN group default qlen 1000
link/ether ac:1f:6b:1a:fb:10 brd ff:ff:ff:ff:ff:ff
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master pnet3 state DOWN group default qlen 1000
link/ether ac:1f:6b:1a:fb:11 brd ff:ff:ff:ff:ff:ff
6: pnet2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether ac:1f:6b:1a:fb:10 brd ff:ff:ff:ff:ff:ff
7: pnet1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether ac:1f:6b:8a:81:f5 brd ff:ff:ff:ff:ff:ff
8: pnet3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether ac:1f:6b:1a:fb:11 brd ff:ff:ff:ff:ff:ff
9: pnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ac:1f:6b:8a:81:f4 brd ff:ff:ff:ff:ff:ff
inet 10.81.45.161/24 brd 10.81.45.255 scope global pnet0
valid_lft forever preferred_lft forever
inet6 fe80::ae1f:6bff:fe8a:81f4/64 scope link
valid_lft forever preferred_lft forever
10: pnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 620f:e4:62:f4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::60de:fff:fee4:62f4/64 scope link
valid_lft forever preferred_lft forever
11: pnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 2a:55:6a:8f:d9:3e brd ff:ff:ff:ff:ff:ff
inet6 fe80::2855:6aff:fe8f:d93e/64 scope link
valid_lft forever preferred_lft forever
12: pnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether c6:46:9e:29:1b:b8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::c446:9eff:fe29:1bb8/64 scope link
valid_lft forever preferred_lft forever
13: pnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether e2:8f:72:dc:e9:41 brd ff:ff:ff:ff:ff:ff
inet6 fe80::e08f:72ff:fedc:e941/64 scope link
valid_lft forever preferred_lft forever
14: pnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 6aab:a0:ea:e6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::68ab:abff:fea0:eae6/64 scope link
valid_lft forever preferred_lft forever
15: pnet9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether de:22:3f:76:83:5e brd ff:ff:ff:ff:ff:ff
inet6 fe80::dc22:3fff:fe76:835e/64 scope link
valid_lft forever preferred_lft forever
16: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:28:96:33:e9 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever

***** OUTPUT OF "free -g"
$ free -g
total used free shared buff/cache available
Mem: 251 1 249 0 0 248
Swap: 0 0 0

****** OUTPUT OF CPU model
$ cat /proc/cpuinfo | grep 'model name' | uniq
model name : Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz

[Uldis (UD)]

BTW, nodes must be started before you start test, on your topo it is stopped !!
And I seeing docker interface on your community version !, remove docker from community then it will work. docker is supported on Pro only.

All works sir
Just connected Aristas to free Cloud2, but it doesnt matter Coud1 or Cloud2.
I configured IP addressess on mgmt1 interfaces, and here is result.
Cloud itself has no any IP address (pnet2)

# Cloud devices
iface eth1 inet manual
auto pnet1
iface pnet1 inet manual
bridge_ports eth1
bridge_stp off

iface eth2 inet manual
auto pnet2
iface pnet2 inet manual
bridge_ports eth2
bridge_stp off

root@eve-ng:~# ifconfig pnet2
pnet2 Link encap:Ethernet HWaddr 06:42:58:f6:13:98
inet6 addr: fe80::bc62:ddff:fec2:58ea/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:736 (736.0 B) TX bytes:4158 (4.1 KB)

6: pnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000

UD

[bingbucks]

Uldis,

Thank you for the reply. We're getting closer. Deleting docker0 (via "ip link del docker0") helped (see attachment) as "pnet1" now shows as "UP". But I can still not ping from vEOS-1 to vEOS-2 (over the ma1 interfaces with valid and same subnet IP addresses) - and, yes, the lab and the vEOS instances are running (that picture from the other day was taken after I shut things down).

Any additional thoughts?

Thanks,
bingbucks

**********************

sudo brctl show
bridge name bridge id STP enabled interfaces
pnet0 8000.ac1f6b8a81f4 no eth0
pnet1 8000.ac1f6b8a81f5 no eth1
pnet2 8000.ac1f6b1afb10 no eth2
pnet3 8000.ac1f6b1afb11 no eth3
pnet4 8000.000000000000 no
pnet5 8000.000000000000 no
pnet6 8000.000000000000 no
pnet7 8000.000000000000 no
pnet8 8000.000000000000 no
pnet9 8000.000000000000 no

[Uldis (UD)]

I guess you had issues, because tried install eve pro but after didnt roll back properly..
Try these steps below.

Rolling back to EVE-NG Community from EVE-Pro



1. To roll back from EVE-NG Pro to the Community Edition, issue the following commands in the CLI of EVE

apt-get install eve-ng eve-ng-guacamole

systemctl disable docker

systemctl disable docker.service

systemctl stop docker.service

systemctl disable udhcpd

2. Reboot EVE

WARNING: Labs built with features that are only available in EVE PRO will be limited to the features offered in EVE Community. Multi-configuration sets for labs will be lost, as only the initial default configuration will remain upon completion of the roll back. Please backup all device configs before rolling back.

[bingbucks]

Hi Uldis, thank you for the reply.

It looks like it worked! I can now ping between nodes on their ma1 interfaces.

Do note that your last command resulted in:

$ sudo systemctl disable udhcpd
Failed to execute operation: No such file or directory

$ ls /etc/u*
/etc/ucf.conf /etc/updatedb.conf

... but I rebooted anyways and pings started working.

I believe that you are correct about a failed upgrade to PRO. I believe that a co-worker of mine attempted the EVE-PRO upgrade but ran into problems. I know he ran into problems around not having the option to upgrade to PRO because of possible missing Intel VT-X/EPT support - but I'm not sure this is related. This co-worker is out of the country for a couple of weeks, so I'll have to wait to ask him where he left off when he returns from holiday.

Nonetheless, things are looking better. I'll keep you updated if I run into anything else.

Do you recommend that we goto PRO on this environment? Or is this environment possibility unstable?

Thanks,
Chris

[Uldis (UD)]

Looks like you had unsuccess EVE Pro update.
Actually if you want stay on EVE Community then my feeling that would be better reinstall your EVE Comm fully.
EVE PRO has huge advantage vs Community.
It is licensed (99 EUR a year).
Benefits you can find in our Cookbook.
http://www.eve-ng.net/images/EVE-COOK-BOOK-1.4.pdf

Uldis

[bingbucks]

Hi Uldis,

So I attempted the second upgrade from EVE community to EVE Pro and I'm not having great success.

There were some forced upgrades that I had to perform to move the ball forward because I think there were remnants of the failed initial PRO install (which I believe neglected to run the command "apt-get install eve-ng-pro" (hence some of the docker messup that you helped me with before).

Below are most of the commands that i ran in order to do the update - which I attempted a few times. I've also attached a few screenshots of some output. There's some Docker error ("Is Docker running?") in the logs (see attached) and most importantly I can't get the EVE GUI to come back - I assume it's a Tomcat issue (tried restarted Tomcat, but no luck)...

Any help would be great as I'd like to upgrade this environment. NOTE: I did Tar up everything in the unetlab/ folder in case I lost everything (as a backup).

151 apt-get update
152 apt-get install eve-ng-pro
153 reboot
154 apt-get update
155 apt-get install eve-ng-dockers-alt

161 dpkg -l eve-ng
162 apt-get update
163 apt-get install eve-ng-pro
164 apt autoremove
165 apt-get install eve-ng-pro
166 apt-get update
167 reboot
168 apt-get update
169 apt-get install eve-ng-dockers-alt
170 dpkg -l eve-ng

173 apt-get upgrade eve-ng-pro
174 dpkg -l eve-ng
175 reboot
176 dpkg -l eve-ng
177 apt-get update
178 apt-get upgrade
179 apt-get install eve-ng-dockers-alt

183 dpkg -l eve-ng-pro
184 tail /var/log/tomcat8/catalina.out

199 service docker restart

225 apt-get --with-new-pkgs upgrade eve-ng-pro
226 reboot
229 apt-get update
230 apt-get install eve-ng-dockers-alt

237 tail -f /var/log/tomcat8/catalina.out
238 tail -f /var/log/syslog

240 service docker restart

[Uldis (UD)]

simply take ready to go EVE Pro mate or install from ISO it.
if you are upgradeing from Community, then dockers needs to be installed extra.
btw show me output of your cli:

dc images

Uldis

[bingbucks]

Hi Uldis:

Interesting, nothing displays for this output (there are no entries):

root@eve-nashua-selab-1:~# dc images
REPOSITORY TAG IMAGE ID CREATED SIZE

Didn't this command in your instructions install Docker?

Code: Select all

apt-get install eve-ng-dockers-alt

I'd prefer to get to Pro via an upgrade from Community edition, and not have to reinstall anything.

What's missing? Or got messed up?

Thanks,
Chris

[bingbucks]

Uldis,

So I just realized that the upgrade instructions in the cookbook (http://www.eve-ng.net/images/EVE-COOK-BOOK-1.2.pdf - section 3.5.2) differ slightly from the upgrade instructions here: http://www.eve-ng.net/documentation/ins ... eve-ng-pro

I followed the latter, wish seems to be incorrect. Can you update the website instructions?

So after following the cookbook upgrade instructions, I now see:

root@eve-nashua-selab-1:~# dc images
REPOSITORY TAG IMAGE ID CREATED SIZE
eve-gui-server latest 820b4cb3a149 12 minutes ago 3.04GB
eve-wireshark latest aed0f4cda37a 32 minutes ago 772MB
eve-firefox latest b55a83fa0815 34 minutes ago 1.49GB
eve-desktop latest d6487cc2d119 38 minutes ago 2.79GB
dockergui-rdp latest 8762661280ca About an hour ago 553MB
phusion/baseimage 0.9.22 877509368a8d 18 months ago 225MB


and the EVE GUI is back up and running.

I will buy a license now. Hopefully we should be good.

Thanks for your guidance.
Chris