Support and News Forum


https://www.eve-ng.net/forum/

Nokia VSR won't start on satellite member

https://www.eve-ng.net/forum/viewtopic.php?f=3&t=19659

Page 1 of 2

Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 4:32 pm
by mamakos
Hello,
With EVE-NG Pro, we have a cluster made of a master and a satellite node.
When I try to start a Nokia VSR on the master there is no issue.
But on a satellite, it fails.

At first I figured out that the symbolic links were involved as only the simlink was copied from the master to the satellite... So I made "hard copies" to get the regular files

I did the "/opt/unetlab/wrappers/unl_wrapper -a fixpermissions" stuff already .

On master /opt/unetlab/addons/qemu/

Code: Select all

drwxr-xr-x  2 root root 4096 Jan  6 17:24 timos-20.2.R1
lrwxrwxrwx  1 root root   38 Dec 20 18:51 timoscpm-20.2.R1 -> /opt/unetlab/addons/qemu/timos-20.2.R1
drwxr-xr-x  2 root root 4096 Jan  6 18:06 timosiom-20.10.R1
lrwxrwxrwx  1 root root   38 Dec 20 18:53 timosiom-20.2.R1 -> /opt/unetlab/addons/qemu/timos-20.2.R1

root@<MASTER>:/opt/unetlab# ls -al addons/qemu/timos-20.2.R1
total 432584
drwxr-xr-x  2 root root      4096 Jan  6 17:24 .
drwxr-xr-x 23 root root      4096 Jan  6 18:00 ..
-rw-r--r--  1 root root 442957824 Dec 20 18:49 hda.qcow2

root@<MASTER>:/opt/unetlab# ls -al addons/qemu/timoscpm-20.10.R1/
total 552524
drwxr-xr-x  2 root root      4096 Jan  6 18:06 .
drwxr-xr-x 23 root root      4096 Jan  6 18:00 ..
-rw-r--r--  1 root root 565772288 Jan  6 18:00 hda.qcow2

root@<MASTER>:/opt/unetlab# ls -al addons/qemu/timosiom-20.10.R1/
total 552524
drwxr-xr-x  2 root root      4096 Jan  6 18:06 .
drwxr-xr-x 23 root root      4096 Jan  6 18:00 ..
-rw-r--r--  1 root root 565772288 Jan  6 18:00 hda.qcow2
On the satellite after a try for each version (hard 20.10 and symblink 20.2) here is what I have on the satellite

Code: Select all

root@<satellite>9:/opt/unetlab/addons/qemu# ls -al
total 20
drwxr-xr-x 5 root root 4096 Jan  6 17:17 .
drwxr-xr-x 4 root root 4096 Dec 19 14:05 ..
drwxr-xr-x 2 root root 4096 Jan  6  2022 timoscpm-20.10.R1
lrwxrwxrwx 1 root root   38 Dec 20 18:51 timoscpm-20.2.R1 -> /opt/unetlab/addons/qemu/timos-20.2.R1
root@<satellite>:/opt/unetlab/addons/qemu# ls -al timoscpm-20.10.R1/
total 552524
drwxr-xr-x 2 root root      4096 Jan  6  2022 .
drwxr-xr-x 5 root root      4096 Jan  6 17:17 ..
-rw-r--r-- 1 root root 565772288 Jan  6  2022 hda.qcow2
Here are the log messages observed when trying to start a Nokia image on the satellite with 20.10 (so wich HARD files, no simlinks)

Code: Select all

Jan 06 18:23:11 Jan 06 18:23:11 Online Check state: Valid
Jan 06 18:23:11 Jan 06 18:23:11 ERROR: sat=1
Jan 06 18:23:11 Jan 06 18:23:11 LOG: will start node 8 1 300 3 49855
Jan 06 18:23:11 INFO: ip link add vun003000100800 type vxlan id 197120 dev wg0 dstport 0
Jan 06 18:23:11 INFO: bridge fdb append to 00:00:00:00:00:00 dst 172.29.130.1 dev vun003000100800
Jan 06 18:23:12 INFO: sysctl -w net.ipv6.conf.vun003000100800.disable_ipv6=1
Jan 06 18:23:12 INFO:  vun003000100800 -> 0,0,0,0
Jan 06 18:23:12 LOG: exec : brctl addif pnet1 vun003000100800 2>&1
sh: 1: ovs-vsctl: not found
Jan 06 18:23:12 INFO: ip link add vun003000100801 type vxlan id 197121 dev wg0 dstport 0
Jan 06 18:23:12 INFO: bridge fdb append to 00:00:00:00:00:00 dst 172.29.130.1 dev vun003000100801
Jan 06 18:23:12 INFO: sysctl -w net.ipv6.conf.vun003000100801.disable_ipv6=1
Jan 06 18:23:12 INFO:  vun003000100801 -> 0,0,0,0
Jan 06 18:23:12 LOG: exec : sysctl -w net.ipv6.conf.vnet00300010005.disable_ipv6=1
Jan 06 18:23:12 LOG: exec : sysctl -w net.ipv6.conf.vun003000100801.disable_ipv6=1
Jan 06 18:23:12 LOG: exec : brctl addif vnet00300010005 vun003000100801 2>&1
sh: 1: ovs-vsctl: not found
Jan 06 18:23:12 START RSYNC
time rsync  -e 'ssh -o ControlPath=/tmp/ssh-root@172.29.130.1:22 -o StrictHostKeyChecking=no' -azv /opt/unetlab/addons/qemu/timoscpm-20.10.R1 172.29.130.1:/opt/unetlab/addons/qemu/  >> /opt/unetlab/data/Logs/unl_wrapper.txt 2>&1
sending incremental file list
timoscpm-20.10.R1/
timoscpm-20.10.R1/hda.qcow2

sent 500,525,161 bytes  received 39 bytes  27,055,416.22 bytes/sec
total size is 565,772,288  speedup is 1.13
19.58user 0.90system 0:17.14elapsed 119%CPU (0avgtext+0avgdata 6376maxresident)k
0inputs+0outputs (0major+1153minor)pagefaults 0swaps
Jan 06 18:23:30 STOP RSYNC
 pgrep -f drop_cache || ( sync && echo 3 > /proc/sys/vm/drop_caches )
Jan 06 18:23:30 FLUSHING CACHE
Jan 06 18:23:30 ERROR: QEMU Arch is not set (80015).
Jan 06 18:23:30 INFO: CWD is /opt/unetlab/tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8
Jan 06 18:23:30 INFO: starting /opt/unetlab/wrappers/qemu_wrapper -C 49855 -T 1 -D 8 -t "7750SR_CPM" -F /opt/qemu-2.12.0/bin/qemu-system-x86_64 -d 0 --  -nographic -device e1000,netdev=net0,mac=50:01:00:08:00:00 -netdev tap,id=net0,ifname=vun003000100800,script=no -device e1000,netdev=net1,mac=50:01:00:08:00:01 -netdev tap,id=net1,ifname=vun003000100801,script=no -smp 1 -m 2048 -name 7750SR_CPM -uuid 81146517-d29e-439a-a88b-7391b0f3862a -qmp unix:./qmp-sock,server,nowait  -monitor unix:./mon-sock,server,nowait  -monitor unix:./mon2-sock,server,nowait  -hda hda.qcow2 -machine type=pc,accel=kvm -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc -smbios type=1,product=\"Timos:slot=A chassis=SR-12 card=cpm5 address=50.50.50.11/16@active license-file=ftp://anonymous:anonymous@50.50.0.254/licence_vSIM_R20_ICC.txt\"
Jan 06 18:23:30 INFO: array(3) {
  ["code"]=>
  int(200)
  ["pid"]=>
  string(3) "669"
  ["status"]=>
  string(2) "ok"
}

PHP Warning:  unlink(/opt/unetlab/tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8/l1down_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1171
Jan 06 18:23:30 INFO:  vun003000100800 -> 0,0,0,0
PHP Warning:  unlink(/opt/unetlab/tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8/l1down_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1171
Jan 06 18:23:30 INFO:  vun003000100801 -> 0,0,0,0
Jan 06 18:23:32 Jan 06 18:23:32 ERROR: node 8
And with Timos 20.2 which is made of symbolic links

Code: Select all

Jan 06 18:15:12 Jan 06 18:15:12 Online Check state: Valid
Jan 06 18:15:12 Jan 06 18:15:12 ERROR: sat=1
Jan 06 18:15:12 Jan 06 18:15:12 LOG: will start node 8 1 300 3 60857
Jan 06 18:15:12 INFO: ip link add vun003000100800 type vxlan id 197120 dev wg0 dstport 0
Jan 06 18:15:12 INFO: bridge fdb append to 00:00:00:00:00:00 dst 172.29.130.1 dev vun003000100800
Jan 06 18:15:12 INFO: sysctl -w net.ipv6.conf.vun003000100800.disable_ipv6=1
Jan 06 18:15:12 INFO:  vun003000100800 -> 0,0,0,0
Jan 06 18:15:13 LOG: exec : brctl addif pnet1 vun003000100800 2>&1
sh: 1: ovs-vsctl: not found
Jan 06 18:15:13 INFO: ip link add vun003000100801 type vxlan id 197121 dev wg0 dstport 0
Jan 06 18:15:13 INFO: bridge fdb append to 00:00:00:00:00:00 dst 172.29.130.1 dev vun003000100801
Jan 06 18:15:13 INFO: sysctl -w net.ipv6.conf.vun003000100801.disable_ipv6=1
Jan 06 18:15:13 INFO:  vun003000100801 -> 0,0,0,0
Jan 06 18:15:13 LOG: exec : sysctl -w net.ipv6.conf.vnet00300010005.disable_ipv6=1
Jan 06 18:15:13 LOG: exec : sysctl -w net.ipv6.conf.vun003000100801.disable_ipv6=1
Jan 06 18:15:13 LOG: exec : brctl addif vnet00300010005 vun003000100801 2>&1
sh: 1: ovs-vsctl: not found
Jan 06 18:15:13 START RSYNC
time rsync  -e 'ssh -o ControlPath=/tmp/ssh-root@172.29.130.1:22 -o StrictHostKeyChecking=no' -azv /opt/unetlab/addons/qemu/timoscpm-20.2.R1 172.29.130.1:/opt/unetlab/addons/qemu/  >> /opt/unetlab/data/Logs/unl_wrapper.txt 2>&1
sending incremental file list

sent 94 bytes  received 12 bytes  70.67 bytes/sec
total size is 38  speedup is 0.36
0.01user 0.00system 0:00.63elapsed 2%CPU (0avgtext+0avgdata 6264maxresident)k
0inputs+0outputs (0major+474minor)pagefaults 0swaps
Jan 06 18:15:14 STOP RSYNC
 pgrep -f drop_cache || ( sync && echo 3 > /proc/sys/vm/drop_caches )
Jan 06 18:15:14 FLUSHING CACHE
Jan 06 18:15:14 ERROR: QEMU Arch is not set (80015).
Jan 06 18:15:14 INFO: CWD is /opt/unetlab/tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8
Jan 06 18:15:14 INFO: starting /opt/unetlab/wrappers/qemu_wrapper -C 60857 -T 1 -D 8 -t "7750SR_CPM" -F /opt/qemu-2.12.0/bin/qemu-system-x86_64 -d 0 --  -nographic -device e1000,netdev=net0,mac=50:01:00:08:00:00 -netdev tap,id=net0,ifname=vun003000100800,script=no -device e1000,netdev=net1,mac=50:01:00:08:00:01 -netdev tap,id=net1,ifname=vun003000100801,script=no -smp 1 -m 2048 -name 7750SR_CPM -uuid 81146517-d29e-439a-a88b-7391b0f3862a -qmp unix:./qmp-sock,server,nowait  -monitor unix:./mon-sock,server,nowait  -monitor unix:./mon2-sock,server,nowait  -hda hda.qcow2 -machine type=pc,accel=kvm -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc -smbios type=1,product=\"Timos:slot=A chassis=SR-12 card=cpm5 address=50.50.50.11/16@active license-file=ftp://anonymous:anonymous@50.50.0.254/licence_vSIM_R20_ICC.txt\"
Jan 06 18:15:14 INFO: array(3) {
  ["code"]=>
  int(200)
  ["pid"]=>
  string(6) "106710"
  ["status"]=>
  string(2) "ok"
}

PHP Warning:  unlink(/opt/unetlab/tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8/l1down_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1171
Jan 06 18:15:14 INFO:  vun003000100800 -> 0,0,0,0
PHP Warning:  unlink(/opt/unetlab/tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8/l1down_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1171
Jan 06 18:15:15 INFO:  vun003000100801 -> 0,0,0,0
Jan 06 18:15:16 Jan 06 18:15:16 ERROR: node 8
When I start Juniper of Linux nodes on a satellite this is working fine.

Could there be something special for Nokia ? Thank you !

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 4:51 pm
by mamakos
Content of the VM directory on each node master/satellite

Code: Select all

root@<MASTER>:/opt/unetlab# ls -al  tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8
total 8
drwxrwsr-x  2 root unl 4096 Jan  6 18:29 .
drwxrwsr-x 11 root unl 4096 Jan  6 18:18 ..
-rw-rw-r--  1 root unl    0 Jan  6 18:29 l1up_0
-rw-rw-r--  1 root unl    0 Jan  6 18:29 l1up_1
-rw-rw-r--  1 root unl    0 Jan  6 18:18 .prepared


root@<SATELLITE>:/opt/unetlab# ls -al tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8
total 204
drwxrwsr-x 2 root unl    4096 Jan  6  2022 .
drwxr-xr-x 7 root root   4096 Jan  6 17:12 ..
-rw-r--r-- 1 root root 197120 Jan  6 17:12 hda.qcow2
-rw-rw-r-- 1 root unl       0 Jan  6  2022 l1up_0
-rw-rw-r-- 1 root unl       0 Jan  6  2022 l1up_1
-rw-r--r-- 1 root root      0 Jan  6 17:12 .prepared

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 6:49 pm
by Uldis (UD)
first thing, your sat servver must have enabled nasted virtualization.
to check it on sat
kvm-ok

and Timos works ok on sat

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 7:13 pm
by mamakos
well it is activated

Code: Select all

root@<satellite>:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
root@<satellite>:~#
And since it is working fine with JunOS I would have encountered a problem before, wouldn't I ?

I am on baremetal (if this helps)

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 7:25 pm
by Uldis (UD)
it does not make sense bare or VM
just be sure, clean out this timos image from sat server /opt/unerlab/addons/qemu/
then create new lab, set boot vsr from sat and start
sat will sysnc this ymage from master again

read eve pro cookbook chapter 15.8.4
https://www.eve-ng.net/index.php/docume ... -cookbook/

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 7:31 pm
by mamakos
well I did it already many times (removing the images from the satellite).
I'll do it again then
I'll try with a new lab, ok

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 7:45 pm
by mamakos
Same player shoots again

MUZTC629 = Satellite

BEFORE

Code: Select all

root@muztc629:/opt/unetlab/addons/qemu# rm -r timoscpm-20.2.R1

root@muztc629:/opt/unetlab/addons/qemu# rm -r timoscpm-20.10.R1
root@muztc629:/opt/unetlab/addons/qemu# ls -al
total 16
drwxr-xr-x 4 root root 4096 Jan  6 20:34 .
drwxr-xr-x 4 root root 4096 Dec 19 14:05 ..
drwxr-xr-x 2 root root 4096 Dec 20 18:50 vmxvcp-21.2R1-S1
drwxr-xr-x 2 root root 4096 Dec 20 18:50 vmxvfp-21.2R1-S1
AFTER using a new lab and trying both TimOS 20.2 an 20.10 releases

Code: Select all

root@muztc629:/opt/unetlab/addons/qemu# ls -al timoscpm-20.*
lrwxrwxrwx 1 root root   38 Dec 20 18:51 timoscpm-20.2.R1 -> /opt/unetlab/addons/qemu/timos-20.2.R1

timoscpm-20.10.R1:
total 552520
drwxr-xr-x 2 root root      4096 Jan  6 18:06 .
drwxr-xr-x 5 root root      4096 Jan  6 20:37 ..
-rw-r--r-- 1 root root 565772288 Jan  6 18:00 hda.qcow2
And in the temp directory of the lab

Code: Select all

root@muztc629:/opt/unetlab/addons/qemu# ls -al ../../tmp/1/e65f69f6-dcf6-4d80-aed0-b862def39e00/1/
total 204
drwxrwsr-x 2 root unl    4096 Jan  6  2022 .
drwxr-xr-x 3 root root   4096 Jan  6 20:37 ..
-rw-r--r-- 1 root root 197120 Jan  6 20:37 hda.qcow2
-rw-rw-r-- 1 root unl       0 Jan  6  2022 l1down_0
-rw-rw-r-- 1 root unl       0 Jan  6  2022 l1down_1
-rw-r--r-- 1 root root      0 Jan  6 20:37 .prepared
About the wrapper logs stating the the file could not be found (PHP Warning), could this point to any other issue on the satellite (or master) ?

Code: Select all

PHP Warning:  unlink(/opt/unetlab/tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8/l1down_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1171
Jan 06 18:15:14 INFO:  vun003000100800 -> 0,0,0,0
PHP Warning:  unlink(/opt/unetlab/tmp/1/6bc93f36-b1dd-4695-b124-8043b3665d18/8/l1down_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1171
Jan 06 18:15:15 INFO:  vun003000100801 -> 0,0,0,0
Jan 06 18:15:16 Jan 06 18:15:16 ERROR: node 8
Can the qemu version on the node definition be at stake ? It is using 2.12.0 H have many othe options in the GUI.
- 3.1.0
- 4.1.0
- 5.2.0
- 6.0.0
- tpl(2.12.0)

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 8:19 pm
by Uldis (UD)
tpl letters means what is set on template by default,

what CPU model has your Satellite?
Of course you can try other qemu versios also
but if you make changes before start nee wipe node to force new settings

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 9:08 pm
by mamakos
They are both Dell R640
Master : 2 times 338-BVKG Intel Xeon Gold 6258R 2.7G, 28C/56T, 10.4GT/s, 38.5M Cache, Turbo, HT (205W) DDR4-2933
Satellite : 2 times 338-BVKU Intel Xeon Gold 6238R 2.2G, 28C/56T, 10.4GT/s, 38.5 M Cache, Turbo, HT (165W) DDR4-2933
The slight difference is the 6238R vs 6258R model

Re: Nokia VSR won't start on satellite member

Posted: Thu Jan 06, 2022 9:19 pm
by mamakos
OK so QEMU 5.1.2 is the good one. None of the other works on my satellite
All times are UTC
Page 1 of 2
Powered by phpBB® Forum Software © phpBB Limited
https://www.phpbb.com/