QEMU Nodes start then stop

[carl.davis]

We have a large Bare Metal install of EVE-NG Professional with a Master and 2 satellite nodes. When we attempt to create a new lab or add a new node to an existing lab and attempt to start it, it will not remain started. I upgraded to the latest EVE-NG Version 5.0.1-20 and this made no difference. When I check the /opt/unetlab/data/Logs/unl_wrapper.txt log, I see messages like this:

Oct 28 12:08:53 Oct 28 12:08:53 Online Check state: Valid
Oct 28 12:08:53 Oct 28 12:08:53 ERROR: sat=0
Oct 28 12:08:53 Oct 28 12:08:53 LOG: will start node 4 4 300 1 52349
Oct 28 12:08:53 INFO: tunctl -u 33796 -g root -t vun001000400400 2>&1
Oct 28 12:08:53 INFO: sysctl -w net.ipv6.conf.vun001000400400.disable_ipv6=1
Oct 28 12:08:53 INFO: vun001000400400 -> 0,0,0,0
Oct 28 12:08:53 INFO: tunctl -u 33796 -g root -t vun001000400401 2>&1
Oct 28 12:08:53 INFO: sysctl -w net.ipv6.conf.vun001000400401.disable_ipv6=1
Oct 28 12:08:53 INFO: vun001000400401 -> 0,0,0,0
Oct 28 12:08:53 INFO: tunctl -u 33796 -g root -t vun001000400402 2>&1
Oct 28 12:08:53 INFO: sysctl -w net.ipv6.conf.vun001000400402.disable_ipv6=1
Oct 28 12:08:53 INFO: vun001000400402 -> 0,0,0,0
Oct 28 12:08:53 INFO: tunctl -u 33796 -g root -t vun001000400403 2>&1
Oct 28 12:08:53 INFO: sysctl -w net.ipv6.conf.vun001000400403.disable_ipv6=1
Oct 28 12:08:53 INFO: vun001000400403 -> 0,0,0,0
Oct 28 12:08:53 ERROR: QEMU Arch is not set (80015).
Oct 28 12:08:54 cmd = mountpoint -q /opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/jail || mount -B -o ro /opt/unetlab/jail /opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/jail
Oct 28 12:08:54 cmd = for i in /opt/qemu-*; do ( mountpoint -q /opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/$i || mount -B -o ro $i /opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/$i ) ; done
Oct 28 12:08:54 cmd = mountpoint -q /opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/opt/unetlab/addons/qemu || mount -B -o ro /opt/unetlab/addons/qemu /opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/opt/unetlab/addons/qemu
Oct 28 12:08:54 INFO: CWD is /opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4
Oct 28 12:08:54 INFO: starting /opt/unetlab/wrappers/qemu_wrapper -C 52349 -T 4 -D 4 -t "R4" -F /opt/qemu-2.12.0/bin/qemu-system-x86_64 -d 0 -- -nographic -device e1000,netdev=net0,mac=50:04:00:04:00:00 -netdev tap,id=net0,ifname=vun001000400400,script=no -device e1000,netdev=net1,mac=50:04:00:04:00:01 -netdev tap,id=net1,ifname=vun001000400401,script=no -device e1000,netdev=net2,mac=50:04:00:04:00:02 -netdev tap,id=net2,ifname=vun001000400402,script=no -device e1000,netdev=net3,mac=50:04:00:04:00:03 -netdev tap,id=net3,ifname=vun001000400403,script=no -smp 1 -m 1024 -name R4 -uuid f2d8f07e-d997-497d-8626-ae9111d20539 -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor unix:./mon2-sock,server,nowait -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc -cpu host
Oct 28 12:08:54 INFO: starting sudo systemd-run -G --no-block --property=ExecStopPost=/opt/unetlab/go/eve-clean --property=Environment=WDIR="/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4" --property=Environment=TYPE="qemu" --property=WorkingDirectory=/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4 --service-type=simple --uid=0 --gid=32768 --unit=eveng_4@9ab9da60-ce2c-47c2-8cbc-e335c9ff976c@4 /opt/unetlab/wrappers/qemu_wrapper -C 52349 -T 4 -D 4 -t "R4" -F /opt/qemu-2.12.0/bin/qemu-system-x86_64 -d 0 -- -nographic -device e1000,netdev=net0,mac=50:04:00:04:00:00 -netdev tap,id=net0,ifname=vun001000400400,script=no -device e1000,netdev=net1,mac=50:04:00:04:00:01 -netdev tap,id=net1,ifname=vun001000400401,script=no -device e1000,netdev=net2,mac=50:04:00:04:00:02 -netdev tap,id=net2,ifname=vun001000400402,script=no -device e1000,netdev=net3,mac=50:04:00:04:00:03 -netdev tap,id=net3,ifname=vun001000400403,script=no -smp 1 -m 1024 -name R4 -uuid f2d8f07e-d997-497d-8626-ae9111d20539 -qmp unix:./qmp-sock,server,nowait -monitor unix:./mon-sock,server,nowait -monitor unix:./mon2-sock,server,nowait -drive file=virtioa.qcow2,if=virtio,bus=0,unit=0,cache=none -machine type=pc,accel=kvm -serial mon:stdio -nographic -no-user-config -nodefaults -rtc base=utc -cpu host
Running as unit: eveng_4@9ab9da60-ce2c-47c2-8cbc-e335c9ff976c@4.service
Oct 28 12:08:54 INFO: starting systemctl show --property MainPID --value eveng_4@9ab9da60-ce2c-47c2-8cbc-e335c9ff976c@4
Oct 28 12:08:54 INFO: started process is 2314532
PHP Warning: unlink(/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/l1up_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1195
PHP Warning: unlink(/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/suspend_0): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1197
PHP Warning: unlink(/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/l1up_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1195
PHP Warning: unlink(/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/suspend_1): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1197
PHP Warning: unlink(/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/l1up_2): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1195
PHP Warning: unlink(/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/suspend_2): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1197
PHP Warning: unlink(/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/l1up_3): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1195
PHP Warning: unlink(/opt/unetlab/tmp/4/9ab9da60-ce2c-47c2-8cbc-e335c9ff976c/4/suspend_3): No such file or directory in /opt/unetlab/wrappers/unl_wrapper on line 1197
Oct 28 12:08:54 Oct 28 12:08:54 ERROR: node 4

Any ideas on how to fix this?

[Uldis (UD)]

Intel VT-X nested virtualization must be enabled on the Sat servers as well