| *** ricolin has joined #openstack-containers | 02:47 | |
| *** goldyfruit_ has quit IRC | 03:05 | |
| *** goldyfruit has joined #openstack-containers | 03:06 | |
| *** goldyfruit has quit IRC | 03:08 | |
| *** goldyfruit has joined #openstack-containers | 03:11 | |
| *** goldyfruit has quit IRC | 03:12 | |
| *** ykarel has joined #openstack-containers | 04:08 | |
| *** fungi has quit IRC | 04:21 | |
| *** fungi has joined #openstack-containers | 04:25 | |
| *** dave-mccowan has quit IRC | 04:41 | |
| *** ykarel is now known as ykarel|away | 05:16 | |
| brtknr | flwang: oh hey 0/ | 05:39 |
|---|---|---|
| brtknr | Are there any other changes you’d like to backports to train? | 05:40 |
| brtknr | Would be could if you could cherry pick them so we can wave them through tomorrow | 05:41 |
| *** sapd1_x has joined #openstack-containers | 06:19 | |
| *** trident has quit IRC | 06:46 | |
| *** trident has joined #openstack-containers | 06:49 | |
| *** rcernin has quit IRC | 07:05 | |
| *** ttsiouts has joined #openstack-containers | 07:10 | |
| *** ttsiouts has quit IRC | 07:26 | |
| *** ttsiouts has joined #openstack-containers | 07:27 | |
| *** ttsiouts has quit IRC | 07:31 | |
| *** lpetrut has joined #openstack-containers | 07:45 | |
| *** ttsiouts has joined #openstack-containers | 07:52 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: Pass ssh public key as string https://review.opendev.org/686986 | 07:56 |
| *** mgoddard has joined #openstack-containers | 08:10 | |
| *** sapd1_x has quit IRC | 08:27 | |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s_atomic: Run all syscontainer with podman https://review.opendev.org/685749 | 08:28 |
| strigazi | flwang: brtknr: fixed ^^ | 08:43 |
| *** pcaruana has joined #openstack-containers | 09:30 | |
| strigazi | flwang: brtknr we have a problem with ignition again | 09:57 |
| openstackgerrit | Merged openstack/magnum master: Pass ssh public key as string https://review.opendev.org/686986 | 10:07 |
| *** ykarel|away has quit IRC | 10:09 | |
| *** ttsiouts has quit IRC | 10:22 | |
| *** rcernin has joined #openstack-containers | 10:53 | |
| *** ttsiouts has joined #openstack-containers | 11:06 | |
| *** dave-mccowan has joined #openstack-containers | 11:27 | |
| brtknr | strigazi: what problem with ignition? | 12:06 |
| strigazi | brtknr: the patch in heat is wrong | 12:51 |
| brtknr | strigazi: Oh really? In what way? | 12:51 |
| strigazi | brtknr: we write this file | 12:53 |
| strigazi | https://review.opendev.org/#/c/683723/13/heat/engine/clients/os/nova.py@454 | 12:53 |
| strigazi | but os-collect-config thinks this is a directory: https://github.com/openstack/os-collect-config/blob/master/os_collect_config/local.py#L70 | 12:53 |
| strigazi | not thinks, expects | 12:53 |
| *** pcaruana has quit IRC | 12:53 | |
| strigazi | this is on problem | 12:53 |
| strigazi | this is one problem | 12:54 |
| brtknr | strigazi: that is because of steve's comment here: https://review.opendev.org/#/c/683723/7/heat/engine/clients/os/nova.py@446 | 12:55 |
| strigazi | yeap, but the code is clear /var/lib/os-collect-config/local-data should be directory | 12:56 |
| strigazi | LOCAL_DEFAULT_PATHS = ['/var/lib/os-collect-config/local-data'] | 12:56 |
| strigazi | for local_path in cfg.CONF.local.path: | 12:56 |
| strigazi | for data_file in os.listdir(local_path): | 12:56 |
| brtknr | strigazi: i understand what you mean | 12:58 |
| brtknr | if we had instead written a file to /var/lib/os-collect-config/local-data/ourfile, it would have been fine | 12:58 |
| brtknr | right? | 12:58 |
| strigazi | almost | 12:58 |
| brtknr | strigazi: why almost? | 12:59 |
| strigazi | because, our agent doesn't work still | 12:59 |
| strigazi | I try to figure out why | 13:00 |
| brtknr | Can we monkey patch the collector? | 13:00 |
| strigazi | I patched heat to test, | 13:01 |
| strigazi | it doesn't work. Not sure why | 13:02 |
| brtknr | strigazi: probaby obvious but did you restart heat? | 13:04 |
| brtknr | strigazi: also is this after starting? | 13:05 |
| strigazi | yes, I see the file in the ndoe | 13:06 |
| strigazi | was it so bad if we had cfn-init-data? | 13:07 |
| *** lpetrut has quit IRC | 13:08 | |
| strigazi | os-collect-config os-apply-config and os-refresh-config don't even have a docs dir | 13:08 |
| strigazi | I think nobody understands trully how they work | 13:08 |
| brtknr | i think they were worried about using a "legacy" feature | 13:13 |
| *** rcernin has quit IRC | 13:19 | |
| *** goldyfruit has joined #openstack-containers | 13:28 | |
| strigazi | I think I'll just copy that file to cfn-init-data and that's it | 13:30 |
| *** goldyfruit_ has joined #openstack-containers | 13:36 | |
| brtknr | strigazi: lol "workaround" | 13:38 |
| *** goldyfruit has quit IRC | 13:39 | |
| brtknr | may also want to delete it from /var/lib/os-collect-config/local-data | 13:39 |
| strigazi | I don't know, I'll take break because I'm extremely depressed with it | 13:43 |
| strigazi | it had taken me like two months to make it work properly. | 13:44 |
| *** ykarel|awat has joined #openstack-containers | 13:56 | |
| *** ykarel|awat is now known as ykarel|away | 13:56 | |
| *** ttsiouts has quit IRC | 13:58 | |
| *** ttsiouts has joined #openstack-containers | 13:59 | |
| *** ttsiouts has quit IRC | 14:00 | |
| *** ttsiouts has joined #openstack-containers | 14:00 | |
| brtknr | strigazi: don't worry, lets make it work with a workaround for now and work with the heat team to get to the root cause later | 14:03 |
| brtknr | strigazi: it is an amazing piece of work nonetheless... basically a seamless transition from fedora atomic to fedora coreos... | 14:04 |
| strigazi | brtknr: :) | 14:11 |
| strigazi | brtknr: thoughts on https://review.opendev.org/#/c/685749/ | 14:11 |
| strigazi | I think it is ready | 14:11 |
| brtknr | strigazi: just testing it now | 14:11 |
| brtknr | but visually looks good to me | 14:12 |
| *** ykarel|away is now known as ykarel | 14:15 | |
| brtknr | strigazi: where is your fedora coreos deployment failing exactly? | 14:35 |
| brtknr | it looked like everything was working yesterday until reboot at least | 14:39 |
| *** jmlowe has quit IRC | 14:42 | |
| strigazi | with the current patch it works, flannel fails cause selinux | 14:45 |
| strigazi | I don't want disable it, I'll give it one more go | 14:45 |
| *** pcaruana has joined #openstack-containers | 14:46 | |
| strigazi | brtknr: ^^ | 14:46 |
| strigazi | calico is fine | 14:46 |
| strigazi | brtknr: does it break for you? not sure what you are asking | 14:47 |
| brtknr | strigazi: I am trying to get as far as I did yesterday but not successfully | 14:48 |
| strigazi | really? | 14:48 |
| strigazi | brtknr: what breaks? | 14:48 |
| strigazi | CLUSTER_COMPLETE? | 14:48 |
| brtknr | NotADirectoryError: [Errno 20] Not a directory: '/var/lib/os-collect-config/local-data' | 14:49 |
| strigazi | heat master? | 14:49 |
| strigazi | probably you have another version of heat | 14:50 |
| strigazi | brtknr: one that was not using /var/lib/os-collect-config/local-data | 14:50 |
| strigazi | now I'm trying to copy that file | 14:51 |
| strigazi | brtknr: makes sense? | 14:52 |
| brtknr | I am using heat master | 14:53 |
| strigazi | where do you see NotADirectoryError: ? | 14:53 |
| brtknr | in the heat-container-agent log | 14:54 |
| strigazi | ls /var/lib/cloud/data/cloud-init-data | 14:54 |
| strigazi | no | 14:54 |
| strigazi | ls /var/lib/cloud/data/cfn-init-data | 14:54 |
| strigazi | OR | 14:55 |
| strigazi | ls /var/lib/heat-cfntools/cfn-init-data | 14:55 |
| brtknr | ls: cannot access '/var/lib/cloud/data/cfn-init-data': No such file or directory | 14:56 |
| brtknr | Are you also using this PS: https://review.opendev.org/#/c/678458/7 | 14:57 |
| brtknr | Would you mind pushing your local changes so we are on the same page? | 14:57 |
| strigazi | brtknr: I haven | 14:57 |
| brtknr | I rebase PS7 against the new podman changes | 14:58 |
| strigazi | brtknr: I haven't pushed :( I'm tired | 14:58 |
| strigazi | brtknr: this was with fcos right? | 14:59 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: k8s_atomic: Run all syscontainer with podman https://review.opendev.org/685749 | 14:59 |
| brtknr | strigazi: tahts right | 14:59 |
| openstackgerrit | Spyros Trigazis proposed openstack/magnum master: [WIP] Support Fedora CoreOS 30 https://review.opendev.org/678458 | 14:59 |
| brtknr | strigazi: fedora-atomic is working with podman :) | 15:00 |
| strigazi | \o/ | 15:00 |
| strigazi | brtknr: the last PS for FCOS works for calico | 15:00 |
| strigazi | with heat master | 15:01 |
| *** jmlowe has joined #openstack-containers | 15:01 | |
| brtknr | whats the alternative to disabling selinux? | 15:01 |
| *** ykarel is now known as ykarel|away | 15:01 | |
| strigazi | run as root | 15:02 |
| strigazi | run flannel as root | 15:02 |
| strigazi | same as calico | 15:02 |
| *** ivve has joined #openstack-containers | 15:03 | |
| strigazi | I'll ask in fedora-coreos, but I promised them to try something first | 15:03 |
| brtknr | strigazi: sounds good | 15:10 |
| brtknr | strigazi: hmm still both clusters in CREATE_IN_PROGRESS | 15:29 |
| brtknr | worker nodes aren't joining the master | 15:30 |
| *** ttsiouts has quit IRC | 15:34 | |
| *** ivve has quit IRC | 15:36 | |
| *** goldyfruit___ has joined #openstack-containers | 16:02 | |
| *** goldyfruit_ has quit IRC | 16:05 | |
| brtknr | + ssh -F /srv/magnum/.ssh/config root@localhost openssl genrsa -out /etc/kubernetes/certs/kubelet.key | 16:13 |
| brtknr | ssh: connect to host 127.0.0.1 port 22: Connection refused | 16:13 |
| brtknr | the workers fail to join master after this | 16:13 |
| *** sapd1_x has joined #openstack-containers | 16:14 | |
| *** sapd1_x has quit IRC | 16:50 | |
| *** ivve has joined #openstack-containers | 16:57 | |
| *** ricolin has quit IRC | 17:03 | |
| *** dave-mccowan has quit IRC | 17:28 | |
| *** dave-mccowan has joined #openstack-containers | 17:48 | |
| guilhermesp | has anyone tested k8s magnum clusters with v1.16.0 | 18:23 |
| guilhermesp | i see we only have images on docker hub for this tag ( no v1.16.1 and so forth ) | 18:24 |
| guilhermesp | https://www.irccloud.com/pastebin/Mbly7dgo/ | 18:34 |
| guilhermesp | kubelet seems to be failing to start, I think the only meaningful lines of logs for me is that one... I see a bunch of deprecation warnings too | 18:35 |
| brtknr | we havent built containers for v1.16.1 yet... | 18:36 |
| guilhermesp | but for v1.16.0? | 18:37 |
| guilhermesp | for example https://hub.docker.com/r/openstackmagnum/kubernetes-proxy/tags | 18:38 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Build k8s images v1.16.1 https://review.opendev.org/687385 | 18:42 |
| brtknr | guilhermesp: yes, v1.16.0 is available | 18:42 |
| guilhermesp | yeah, i have a failing cluster with part of the output above... seems that kubelet is constantly failing to start | 18:43 |
| brtknr | guilhermesp: which release are you using? master? | 18:43 |
| guilhermesp | yep, master | 18:44 |
| guilhermesp | v1.14.3, v1.15.1 works cleanly | 18:44 |
| brtknr | guilhermesp: latest master or few days old master? | 18:44 |
| guilhermesp | ops v1.15.3 | 18:44 |
| guilhermesp | let me give you the sha I'm using | 18:44 |
| guilhermesp | 919bf541c811229aa7f024180bdf33f15172a682 | 18:44 |
| brtknr | commit has? | 18:44 |
| guilhermesp | you mean this ? https://github.com/openstack/magnum/commit/919bf541c811229aa7f024180bdf33f15172a682 | 18:45 |
| brtknr | guilhermesp: im testing now | 18:47 |
| guilhermesp | thanks | 18:47 |
| brtknr | guilhermesp: looks like thats part of the change | 18:48 |
| guilhermesp | so maybe I'd need to pull up the latest commits ? | 18:48 |
| *** ykarel|away has quit IRC | 19:00 | |
| *** jmlowe has quit IRC | 19:03 | |
| *** dave-mccowan has quit IRC | 19:10 | |
| brtknr | guilhermesp: hmm no you shouldnt need to | 19:14 |
| brtknr | im still checking | 19:14 |
| *** jmlowe has joined #openstack-containers | 19:16 | |
| guilhermesp | cool brtknr thanks in advance | 19:20 |
| brtknr | guilhermesp: ah, i dont think it works | 19:34 |
| brtknr | guilhermesp: it will need to be deployed via podman: https://review.opendev.org/685749 | 19:35 |
| guilhermesp | oh, it makes sense accordingly to what I've seen brtknr | 19:38 |
| guilhermesp | thanks for the clarification | 19:38 |
| guilhermesp | so I think i might wait this to land and then I can push the latest commit to my deployment | 19:38 |
| brtknr | guilhermesp: thats right | 19:46 |
| brtknr | flwang needs to +2 +w it for it to land | 19:46 |
| *** trident has quit IRC | 19:51 | |
| guilhermesp | cool brtknr ! gonna be waiting for that then | 19:52 |
| *** trident has joined #openstack-containers | 19:54 | |
| *** dave-mccowan has joined #openstack-containers | 19:59 | |
| *** pcaruana has quit IRC | 20:35 | |
| brtknr | guilhermesp: feel free to test the patch with v1.16.0 before it even gets merged, would be of real help in case you manage to find other issues | 21:40 |
| *** ivve has quit IRC | 22:03 | |
| *** goldyfruit_ has joined #openstack-containers | 22:05 | |
| *** goldyfruit___ has quit IRC | 22:07 | |
| *** bline has joined #openstack-containers | 22:15 | |
| *** rcernin has joined #openstack-containers | 22:26 | |
| *** goldyfruit_ has quit IRC | 22:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!