Tuesday, 2020-06-02

« Monday, 2020-06-01 Index Wednesday, 2020-06-03 »
*** k_mouza has joined #openstack-containers01:04
*** k_mouza has quit IRC01:09
*** openstacking_123 has quit IRC01:21
*** threestrands has joined #openstack-containers01:21
*** born2bake has quit IRC01:34
*** rcernin has quit IRC02:01
*** rcernin has joined #openstack-containers02:03
*** rcernin has quit IRC02:20
*** rcernin has joined #openstack-containers02:52
*** dasp has quit IRC02:54
*** ricolin has joined #openstack-containers02:57
*** dasp has joined #openstack-containers03:00
*** sapd1 has joined #openstack-containers03:01
*** hongbin has joined #openstack-containers03:30
*** threestrands has quit IRC03:36
openstackgerritSimon Merrick proposed openstack/magnum-ui master: Fix formatting issue in workflow message  https://review.opendev.org/73247403:39
*** hongbin has quit IRC03:40
*** vishalmanchanda has joined #openstack-containers04:05
*** ykarel|away is now known as ykarel04:08
*** rcernin has quit IRC04:19
*** rcernin has joined #openstack-containers04:22
*** udesale has joined #openstack-containers04:41
*** k_mouza has joined #openstack-containers05:04
*** k_mouza has quit IRC05:09
openstackgerritSimon Merrick proposed openstack/magnum-ui master: Fix formatting issue in workflow message  https://review.opendev.org/73247405:19
*** threestrands has joined #openstack-containers05:26
openstackgerritSimon Merrick proposed openstack/magnum-ui master: Fix formatting issue in workflow message  https://review.opendev.org/73247405:42
*** nikparasyr has joined #openstack-containers05:47
*** faizy98 has quit IRC05:54
*** xinliang has joined #openstack-containers06:48
*** xinliang has quit IRC06:55
*** sapd1 has quit IRC06:57
*** ttsiouts has joined #openstack-containers07:12
*** born2bake has joined #openstack-containers07:25
*** rcernin has quit IRC07:48
*** ttsiouts has quit IRC08:16
*** threestrands has quit IRC08:32
*** strigazi has quit IRC08:34
*** strigazi has joined #openstack-containers08:35
*** ykarel is now known as ykarel|lunch08:43
*** k_mouza has joined #openstack-containers09:16
*** ttsiouts has joined #openstack-containers09:32
*** ykarel|lunch is now known as ykarel09:36
tobias-urdinim staring myself blind on issues with getting magnum 9.3.0 to properly deploy k8s on Fedora CoreOS 3109:45
tobias-urdinit stops here according to the heat-container-agent journal log09:45
tobias-urdinJun 02 07:57:37 tty-lxwwrmrpn7jt-master-0 podman[2647]: [2020-06-02 07:57:37,575] (heat-config) [DEBUG] Running /var/lib/heat-config/hooks/script < /var/lib/heat-config/deployed/5fc20446-73c6-489a-bbd2-62d45ab49114.json09:45
tobias-urdinJun 02 07:58:05 tty-lxwwrmrpn7jt-master-0 podman[2647]: Command failed, will not cache new data. Command 'os-refresh-config' died with <Signals.SIGTERM: 15>.09:45
tobias-urdinJun 02 07:58:05 tty-lxwwrmrpn7jt-master-0 systemd[1]: Stopping Run heat-container-agent...09:45
tobias-urdinall scripts are properly concatenated in /var/lib/heat-config/heat-config-script/5fc20446-73c6-489a-bbd2-62d45ab4911409:45
tobias-urdinso the data is properly read by heat-container-agent and placed in there09:46
tobias-urdinbut it stops on this line09:46
tobias-urdinhttps://github.com/openstack/magnum/blob/9.3.0/magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh#L17109:46
tobias-urdinbut the last curl request is successful https://github.com/openstack/magnum/blob/9.3.0/magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh#L13009:46
tobias-urdinand the /etc/kubernetes/certs/kubelet.crt file contains the certificate09:46
tobias-urdin[02/Jun/2020:07:57:51 +0200] "GET /v1/certificates/d3b3cd30-d34a-401b-ad9b-4bc2aba30fe5 HTTP/1.1" 200 1421 "-" "curl/7.70.0"09:46
tobias-urdin[02/Jun/2020:07:57:58 +0200] "POST /v1/certificates HTTP/1.1" 201 4200 "-" "curl/7.70.0"09:46
tobias-urdinUsing these versions; Fedora CoreOS 31.20200505.3.0   Magnum 9.3.0   Heat 13.0.109:47
tobias-urdinanybody has a clue? since it never continues to the echo commands after the function that calls curl I assume it thinks curl command failed09:47
tobias-urdinbut none of the logs refer to why the script exited/died so it's very hard to troubleshoot, copied the rest of the script from the /var/lib/heat-config/heat-config-script/5fc20446-73c6-489a-bbd2-62d45ab49114 file and executed it and k8s was properly deployed09:48
tobias-urdinthere also seems to be a race conditions sometimes when ssh on CoreOS is not ready causing the $ssh_cmd way to get a connection refused on port 22 but that happens more rarely09:49
tobias-urdinstrigazi: brtknr09:49
brtknrtobias-urdin: look inside /var/log/heat-config/heat-config-script09:51
tobias-urdinbrtknr: yeah, it stops here https://github.com/openstack/magnum/blob/9.3.0/magnum/drivers/common/templates/kubernetes/fragments/make-cert.sh#L17109:57
tobias-urdinsec i'll post a censored log from that line09:57
brtknrtobias-urdin: which version of heat container agent are you using?09:58
tobias-urdinthe default hardcoded in there heat-container-agent:ussuri-dev09:58
tobias-urdinhttp://paste.openstack.org/show/794231/ from the log09:58
*** udesale has quit IRC09:59
tobias-urdinthe log output doesn't help much, other than it stopping at the last curl request there, but api log says it's successful09:59
tobias-urdin"POST /v1/certificates HTTP/1.1" 201 4200 "-" "curl/7.70.0"09:59
tobias-urdinand /etc/kubernetes/certs/kubelet.crt file is populated09:59
*** k_mouza has quit IRC10:00
brtknrtobias-urdin: did you censor the api IP address? http://api:9511/v1/certificates10:00
tobias-urdinyeah, it's correct there10:00
brtknrI dont see any error messages?10:01
tobias-urdinme neither, that's why im thrown off, it just stops10:02
tobias-urdinheat stack says "timed out" on kube_masters10:03
tobias-urdinbut the /etc/kubernetes/certs/kubelet.crt is populated from that curl request, and api log (the POST line above) was 20110:03
tobias-urdindo you know if there is any recent fixes I might not have in 9.3.0? that's usually been the case that before, some missed backports, missing a new release to get the new changes out10:04
tobias-urdinbut i've never got fedora coreos to work, only the very old fedora atomic 2710:05
*** ramishra has quit IRC10:05
jakeyiphmm anyone's cluster at train yet?10:08
tobias-urdinbrtknr: tried another deploy now and noticed something10:15
tobias-urdinJun 02 10:10:07 newtest-z4kds5tnljh2-master-0 zincati[1018]: [INFO ] staged deployment '31.20200505.3.0' available, proceeding to finalize it10:15
tobias-urdinJun 02 10:10:10 newtest-z4kds5tnljh2-master-0 rpm-ostree[1027]: Initiated txn FinalizeDeployment for client(dbus:1.194 unit:zincati.service uid:980): /org/projectatomic/rpmostree1/fedora_coreos10:15
tobias-urdinJun 02 10:10:10 newtest-z4kds5tnljh2-master-0 rpm-ostree[1027]: Finalized deployment; rebooting into 01f074cc6cd88d8d2b43f821da692f2367c101eb4377802cb35092bde0ef02f710:15
tobias-urdinJun 02 10:10:10 newtest-z4kds5tnljh2-master-0 systemd-logind[1279]: System is rebooting.10:15
tobias-urdinthis commit "fcos: Disable zincati auto-updates" https://github.com/openstack/magnum/commit/56f3be8bcf42e7772385e843355a5963705e9f2b10:15
tobias-urdinis not released so the node is interrupted by auto updates on deploy10:15
tobias-urdinjakeyip: yes10:15
brtknrtobias-urdin: ah yes, you need that patch but we have yet to release 9.4.010:15
brtknrwaiting for some things to merge10:15
brtknrtobias-urdin: workaround is to use latest version of fcos10:16
brtknrtobias-urdin: workaround is to use latest stable version of10:16
tobias-urdinok, the above would explain why it stops working at different places all the time, sometimes almost a full deploy10:16
tobias-urdini'll upload new image10:16
tobias-urdinand try again10:16
*** k_mouza has joined #openstack-containers10:18
*** ramishra has joined #openstack-containers10:27
*** k_mouza has quit IRC10:36
*** ramishra has quit IRC10:36
*** k_mouza has joined #openstack-containers10:38
*** ramishra has joined #openstack-containers10:40
*** ramishra has quit IRC11:02
*** mgariepy has quit IRC11:03
*** ramishra has joined #openstack-containers11:03
*** ramishra has quit IRC11:08
*** ramishra has joined #openstack-containers11:08
openstackgerritBharat Kunwar proposed openstack/magnum master: Source /etc/bashrc where kubectl when kubectl is used  https://review.opendev.org/73252411:12
brtknrtobias-urdin: you might also need ^11:14
tobias-urdinbrtknr: ok, only 1.18 related thought right, i will only test with 1.17 as latest according to the compatibility matrix11:42
tobias-urdini will test everything later today and report back11:42
*** sapd1 has joined #openstack-containers11:43
*** ramishra has quit IRC11:52
*** k_mouza has quit IRC11:58
*** k_mouza has joined #openstack-containers11:58
born2bakeHi brtknr this can be closed - https://storyboard.openstack.org/#!/story/200774111:59
born2bakeerrors were due to cinder issues I had12:00
brtknrborn2bake: ok cool12:04
brtknrborn2bake: can you briefly explain in the story what the issue you had was?12:04
born2bakedone12:09
born2bakebrtknr I am just wondering, how magnum k8s cluster differ from kubespray-vanilla-kubeadm clusters?12:13
brtknrMagnum doesnt use kubeadm for a start born2bake12:14
*** mgariepy has joined #openstack-containers12:14
brtknrMagnum cluster comes bootstrapped with OCCM and allows you to use openstack credential for auth12:14
*** ramishra has joined #openstack-containers12:16
*** ttsiouts has quit IRC12:36
*** sapd1 has quit IRC12:42
*** ttsiouts has joined #openstack-containers12:45
*** ramishra has quit IRC12:54
*** ramishra has joined #openstack-containers12:55
*** munimeha1 has joined #openstack-containers13:25
*** openstacking_123 has joined #openstack-containers13:26
*** sapd1 has joined #openstack-containers13:46
tobias-urdinbrtknr: stuck on this now, I assume kubernetes v1.18 client is in ussuri-dev heat-container-agent?13:53
tobias-urdinsh-5.0# kubectl patch node ${INSTANCE_NAME} --patch '{"metadata": {"labels": {"node-role.kubernetes.io/master": ""}}}'13:53
tobias-urdinerror: no configuration has been provided, try setting KUBERNETES_MASTER environment variable13:53
tobias-urdinsetting KUBERNETES_MASTER=http://localhost:8080 fixes it13:53
tobias-urdinbecomes and infinite loop here: https://github.com/openstack/magnum/blob/stable/train/magnum/drivers/common/templates/kubernetes/fragments/enable-services-master.sh#L3313:54
tobias-urdinsame as KUBERNETES_MASTER being set for minion should be set for master https://github.com/openstack/magnum/blob/stable/train/magnum/drivers/common/templates/kubernetes/fragments/configure-kubernetes-minion.sh#L30513:54
*** nikparasyr has quit IRC14:11
*** jmlowe has quit IRC14:17
*** udesale has joined #openstack-containers14:18
*** jmlowe has joined #openstack-containers14:18
*** openstacking_123 has quit IRC14:40
*** belmoreira has joined #openstack-containers14:55
*** ttsiouts has quit IRC15:04
*** belmoreira has quit IRC15:07
*** ttsiouts has joined #openstack-containers15:12
brtknrtobias-urdin: this is a known issue15:12
brtknruse train-stable-2 tag for now15:13
brtknri am trying to fix it  this morning but needs to more: https://review.opendev.org/73252415:13
*** jmlowe has quit IRC15:13
*** jmlowe has joined #openstack-containers15:16
*** ttsiouts has quit IRC15:23
tobias-urdinbrtknr: thanks, it's working now when using train-stable-2, i patched scripts to set KUBERNETES_MASTER, will try removing it since might not be needed when using train-stable-215:42
*** ykarel is now known as ykarel|away15:43
tobias-urdinyeah worked without that workaround15:52
*** sapd1 has quit IRC15:59
*** udesale has quit IRC16:04
*** sapd1 has joined #openstack-containers16:37
*** mgariepy has quit IRC17:05
*** yolanda has quit IRC17:11
*** k_mouza has quit IRC17:48
*** sapd1 has quit IRC17:53
born2bakebrtknr I am wondering, is octavia supported? If I want to use LoadBalancer service types within k8s cluster, cloud provider does support it right? Is magnum able to deploy it automatically as well?18:06
*** mgariepy has joined #openstack-containers18:25
*** ttsiouts has joined #openstack-containers19:25
*** k_mouza has joined #openstack-containers19:48
*** ttsiouts has quit IRC19:52
*** k_mouza has quit IRC19:53
*** k_mouza has joined #openstack-containers19:57
*** k_mouza has quit IRC20:02
*** k_mouza has joined #openstack-containers20:03
*** vishalmanchanda has quit IRC20:05
*** ttsiouts has joined #openstack-containers20:06
*** ttsiouts has quit IRC20:10
*** ricolin has quit IRC20:38
*** k_mouza has quit IRC21:00
*** rcernin has joined #openstack-containers22:57
*** rcernin has quit IRC22:59
*** rcernin has joined #openstack-containers23:23
openstackgerritFeilong Wang proposed openstack/magnum master: [WIP]Fix proxy issue for etcd and kubelet  https://review.opendev.org/73302723:32
*** k_mouza has joined #openstack-containers23:52
*** k_mouza has quit IRC23:52
jakeyiptobias-urdin: how are you setting up your cloud? puppet / ansible / something else?23:54
*** born2bake has quit IRC23:56
« Monday, 2020-06-01 Index Wednesday, 2020-06-03 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!