Before posting something, READ the changelog, WATCH the videos, howto and provide following:
Your install is: Bare metal, ESXi, what CPU model, RAM, HD, what EVE version you have, output of the uname -a and any other info that might help us faster.
Moderator: mike
-
Povelitel
- Posts: 3
- Joined: Thu Sep 28, 2017 5:36 am
Post
by Povelitel » Thu Sep 28, 2017 9:01 am
Hi Team!
I have an issue with my connections to console of my emulated devices. After some time, my telnet connection freezes. Here is an example:
Code: Select all
R3(config-router)#?
Router configuration commands:
address-family Enter Address Family command mode
auto-summary Enable automatic network number summarization
bfd BFD configuration commands
default Set a command to its defaults
default-information Control distribution of default information
default-metric Set metric of redistributed routes
distance Define an administrative distance
distribute-list Filter networks in routing updates
exit Exit from routing protocol configuration mode
flash-update-threshold Specify flash update threshold in second
help Description of the interactive help system
input-queue Specify input queue depth
maximum-paths Forward packets over multiple paths
neighbor Specify a neighbor router
network Enable rou
And that's all. Pressing any button don't get anything. When I try connecting one more time using same telnet port, I have successful connection, but have not any response from device.
On EVE server:
Code: Select all
root@eve-ngnetstat -anp | grep 32771
tcp6 0 0 :::32771 :::* LISTEN 5907/qemu_wrapper
tcp6 3 0 192.168.45.10:32771 192.168.54.111:61897 ESTABLISHED 5907/qemu_wrapper
tcp6 0 1149 192.168.45.10:32771 192.168.54.111:56516 ESTABLISHED 5907/qemu_wrapper
I think, that non zero numbers on Recv-Q/Send-Q is not good...
After, about 30 min. Telnet session continues to work:
Code: Select all
ng on an IP network
no Negate a command or set its defaults
offset-list Add or subtract offset from RIP metrics
output-delay Interpacket delay for RIP updates
passive-interface Suppress routing updates on an interface
redistribute Redistribute information from another routing
protocol
timers Adjust routing timers
R3(config-router)#
R3(config-router)#
R3(config-router)#
R3(config-router)#
R3(config-router)#
R3(config-router)#
On eve server:
Code: Select all
root@eve-ng:~# netstat -anp | grep 32771
tcp6 0 0 :::32771 :::* LISTEN 5907/qemu_wrapper
tcp6 0 0 192.168.45.10:32771 192.168.54.111:56516 ESTABLISHED 5907/qemu_wrapper
here the output of the evenginfi command:
Code: Select all
***** EVE-NG INFORMATION TOOL - CREATED BY R! ******
Thu Sep 28 08:41:45 EEST 2017
***** EVE-NG VERSION INSTALLED *****
eve-ng 2.0.3-79
***** HOSTNAME INFORMATION *****
Static hostname: eve-ng
Virtualization: vmware
Operating System: Ubuntu 16.04.3 LTS
Kernel: Linux 4.9.40-eve-ng-ukms-2+
Architecture: x86-64
***** FILE SYSTEM DISK SPACE USAGE *****
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 15M 3.2G 1% /run
/dev/mapper/eve--ng--vg-root 92G 13G 76G 14% /
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 472M 125M 323M 28% /boot
***** CPU INFORMATION *****
model name : Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
***** FREE AND USED MEMORY *****
total used free shared buff/cache available
Mem: 31G 8.8G 21G 35M 1.4G 21G
Swap: 6.0G 0B 6.0G
***** INTERFACE INFORMATION *****
pnet0 8000.005056b6debe no eth0
pnet1 8000.000000000000 no
pnet2 8000.000000000000 no
pnet3 8000.000000000000 no
pnet4 8000.000000000000 no
pnet5 8000.000000000000 no
pnet6 8000.000000000000 no
pnet7 8000.000000000000 no
pnet8 8000.000000000000 no
pnet9 8000.000000000000 no
***** KVM CHECK INFORMATION *****
INFO: /dev/kvm exists
KVM acceleration can be used
***** APACHE STATUS *****
* apache2.service - LSB: Apache2 web server
Active: active (running) since Wed 2017-09-27 18:33:11 EEST; 14h ago
***** CHECKING INERNET STATUS *****
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=52 time=25.6 ms
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 25.615/25.615/25.615/0.000 ms
***** COPY or SCREENSHOT THIS INFORMATION FOR EVE SUPPORT TO HELP/UNDERSTAND YOUR SYSTEM INFORMATION.******
***** http://www.eve-ng.net/forum/ ******
-
cianfa72
- Posts: 6
- Joined: Sun Oct 01, 2017 8:24 pm
Post
by cianfa72 » Tue Oct 03, 2017 6:25 am
Hi,
facing to the same problem, it seems we are solved just replacing EVE qemu_wrapper with the (old) version taken from Unetlab.
An issue with the tcp socket management I guess...can you confirm that ?
Regards.
-
Uldis (UD)
- Posts: 5180
- Joined: Wed Mar 15, 2017 4:44 pm
- Location: London
-
Contact:
Post
by Uldis (UD) » Tue Oct 03, 2017 7:37 am
replace wrapper from UNL is not right solution...
usually it is telnet client issue only !!!
-
Povelitel
- Posts: 3
- Joined: Thu Sep 28, 2017 5:36 am
Post
by Povelitel » Fri Oct 06, 2017 4:28 am
cianfa72 wrote: ↑Tue Oct 03, 2017 6:25 am
Hi,
facing to the same problem, it seems we are solved just replacing EVE qemu_wrapper with the (old) version taken from Unetlab.
An issue with the tcp socket management I guess...can you confirm that ?
Regards.
Hi,
I'am also think that it is some kind tcp socket management issue, because connections didn't work even localy (I tried telnetting using localhost and node port number).
cianfa72, can you share working qemu_wrapper package?
-
cianfa72
- Posts: 6
- Joined: Sun Oct 01, 2017 8:24 pm
Post
by cianfa72 » Mon Oct 09, 2017 8:22 am
executable file attachment is not supported...Anyway basically it is the qemu_wrapper of the old Unetlab
-
ecze
- Posts: 534
- Joined: Wed Mar 15, 2017 1:54 pm
Post
by ecze » Mon Oct 09, 2017 3:24 pm
This issue is pretty hard to analyze and to reproduce....
After a long wait, finally some evidences appears...
netstat -napto shows ACK retransmission.
Even if this issue is still under investigation, the first advice I can get is to disable sack for TCP stack
this could be done with:
sysctl -w net.ipv4.tcp_sack=0
To keep this permanent add also in /etc/sysctl.conf
net.ipv4.tcp_sack = 0
at the end of the file....
E.
-
Povelitel
- Posts: 3
- Joined: Thu Sep 28, 2017 5:36 am
Post
by Povelitel » Tue Oct 10, 2017 11:29 am
sysctl -w net.ipv4.tcp_sack=0
This didn't help much...
Here my netstat -napto output:
Code: Select all
tcp6 0 1128 192.168.45.10:32776 192.168.125.45:53769 ESTABLISHED 18291/qemu_wrapper on (51.96/8/0)
tcp6 0 0 192.168.45.10:80 192.168.125.45:54576 ESTABLISHED 2681/apache2 keepalive (6862.84/0/0)
tcp6 5 0 192.168.45.10:32778 192.168.125.45:54587 ESTABLISHED 17579/qemu_wrapper off (0.00/0/0)
tcp6 0 1128 192.168.45.10:32773 192.168.125.45:53766 ESTABLISHED 19359/qemu_wrapper on (56.05/8/0)
tcp6 0 1128 192.168.45.10:32771 192.168.125.45:53764 ESTABLISHED 20088/qemu_wrapper on (59.12/8/0)
tcp6 5 0 192.168.45.10:32773 192.168.125.45:54591 ESTABLISHED 19359/qemu_wrapper off (0.00/0/0)
tcp6 0 1128 192.168.45.10:32778 192.168.125.45:53762 ESTABLISHED 17579/qemu_wrapper on (51.96/8/0)
tcp6 0 1128 192.168.45.10:32770 192.168.125.45:53763 ESTABLISHED 20485/qemu_wrapper on (51.96/8/0)
tcp6 5 0 192.168.45.10:32772 192.168.125.45:54590 ESTABLISHED 19716/qemu_wrapper off (0.00/0/0)
tcp6 5 0 192.168.45.10:32775 192.168.125.45:54593 ESTABLISHED 18647/qemu_wrapper off (0.00/0/0)
tcp6 5 0 192.168.45.10:32777 192.168.125.45:54595 ESTABLISHED 17935/qemu_wrapper off (0.00/0/0)
tcp6 0 1131 192.168.45.10:32772 192.168.125.45:53765 ESTABLISHED 19716/qemu_wrapper on (56.05/8/0)
tcp6 0 1128 192.168.45.10:32769 192.168.125.45:53761 ESTABLISHED 20820/qemu_wrapper on (55.03/8/0)
tcp6 5 0 192.168.45.10:32771 192.168.125.45:54589 ESTABLISHED 20088/qemu_wrapper off (0.00/0/0)
tcp6 0 1128 192.168.45.10:32775 192.168.125.45:53768 ESTABLISHED 18647/qemu_wrapper on (63.22/8/0)
tcp6 5 0 192.168.45.10:32769 192.168.125.45:54586 ESTABLISHED 20820/qemu_wrapper off (0.00/0/0)
tcp6 0 1128 192.168.45.10:32777 192.168.125.45:53770 ESTABLISHED 17935/qemu_wrapper on (55.03/8/0)
tcp6 5 0 192.168.45.10:32776 192.168.125.45:54594 ESTABLISHED 18291/qemu_wrapper off (0.00/0/0)
tcp6 5 0 192.168.45.10:32770 192.168.125.45:54588 ESTABLISHED 20485/qemu_wrapper off (0.00/0/0)
tcp6 5 0 192.168.45.10:32774 192.168.125.45:54592 ESTABLISHED 19003/qemu_wrapper off (0.00/0/0)
tcp6 0 1128 192.168.45.10:32774 192.168.125.45:53767 ESTABLISHED 19003/qemu_wrapper on (59.12/8/0)
I have this problem very often. I can provide any information you requested..
-
ecze
- Posts: 534
- Joined: Wed Mar 15, 2017 1:54 pm
Post
by ecze » Tue Oct 10, 2017 11:43 am
Your screenshot shows a lot of re-transmission...
(51.96/8/0) means timer = 51 sec, 8 retransmit waiting for other end answer
Still under investigation.
Thanks for your report...
E.
PS: wrapper code Unetlab / EVE-NG is not different regarding FIFO, TCP Sockets.. So this issue is not related to the wrapper itself...
-
cianfa72
- Posts: 6
- Joined: Sun Oct 01, 2017 8:24 pm
Post
by cianfa72 » Thu Oct 12, 2017 12:58 pm
ecze wrote: ↑Tue Oct 10, 2017 11:43 am
PS: wrapper code Unetlab / EVE-NG is not different regarding FIFO, TCP Sockets.. So this issue is not related to the wrapper itself...
Sure, that is correct. Even using the old unetlab qemu_wrapper after some time we faced again the same issue...
-
ecze
- Posts: 534
- Joined: Wed Mar 15, 2017 1:54 pm
Post
by ecze » Thu Oct 12, 2017 1:23 pm
Legacy wrapper ( UNL ) suffer of same bug but it is real bug... found...
V84 will fix it.
This issue result of an old connection lost but not cleanly closed ( network outage, close lid of a laptop etc etc ) . subsequent console connection needs to wait the close of the ghost connection.(20 to 30 min )
Next release will not block anymore subsequent connection in this case.....
Available shortly....