| *** jmlowe has quit IRC | 00:00 | |
| *** threestrands has joined #openstack-containers | 00:24 | |
| *** hongbin has joined #openstack-containers | 00:37 | |
| *** rcernin has quit IRC | 01:13 | |
| *** rcernin has joined #openstack-containers | 02:13 | |
| *** ramishra has joined #openstack-containers | 03:36 | |
| *** udesale has joined #openstack-containers | 04:06 | |
| *** hongbin has quit IRC | 04:13 | |
| *** dave-mccowan has quit IRC | 04:36 | |
| *** lpetrut has joined #openstack-containers | 06:02 | |
| *** trident has quit IRC | 07:00 | |
| *** trident has joined #openstack-containers | 07:10 | |
| *** ivve has joined #openstack-containers | 07:17 | |
| *** sapd1_x has joined #openstack-containers | 07:25 | |
| *** threestrands has quit IRC | 07:32 | |
| *** rcernin has quit IRC | 07:40 | |
| *** mgoddard has joined #openstack-containers | 07:54 | |
| *** _nwonknu has quit IRC | 09:41 | |
| *** sapd1_x has quit IRC | 09:59 | |
| *** nwonknu has joined #openstack-containers | 10:00 | |
| *** udesale has quit IRC | 11:02 | |
| *** dave-mccowan has joined #openstack-containers | 11:34 | |
| *** danil has joined #openstack-containers | 12:19 | |
| *** jmlowe has joined #openstack-containers | 13:04 | |
| *** yolanda has quit IRC | 13:06 | |
| *** yolanda__ has joined #openstack-containers | 13:06 | |
| *** KeithMnemonic has joined #openstack-containers | 13:14 | |
| *** udesale has joined #openstack-containers | 13:32 | |
| *** udesale has quit IRC | 14:32 | |
| *** spsurya has joined #openstack-containers | 14:35 | |
| *** lpetrut has quit IRC | 14:44 | |
| *** jmlowe has quit IRC | 15:13 | |
| *** jmlowe has joined #openstack-containers | 15:14 | |
| *** itlinux has quit IRC | 15:18 | |
| openstackgerrit | Ricardo Rocha proposed openstack/magnum master: Drop deprecated APIs for kube v1.16 support https://review.opendev.org/678893 | 16:06 |
|---|---|---|
| *** ivve has quit IRC | 16:22 | |
| *** ivve has joined #openstack-containers | 17:25 | |
| *** ivve has quit IRC | 17:26 | |
| *** ivve has joined #openstack-containers | 17:27 | |
| NobodyCam | Morning Folks | 17:38 |
| NobodyCam | is there a way to force a minion node to re create networks | 17:38 |
| NobodyCam | I have one node that appears to not have setup correctly. I'm seeing: | 17:40 |
| NobodyCam | dial tcp 10.254.0.1:443: getsockopt: no route to host | 17:40 |
| *** itlinux has joined #openstack-containers | 17:40 | |
| *** itlinux has quit IRC | 17:47 | |
| *** itlinux has joined #openstack-containers | 17:50 | |
| *** itlinux has quit IRC | 17:55 | |
| *** itlinux has joined #openstack-containers | 17:55 | |
| brtknr | What version are you running NobodyCam? | 18:00 |
| NobodyCam | rocky.. | 18:01 |
| NobodyCam | looks like there is no kube-proxy.service on the minion | 18:01 |
| brtknr | You can delete CoreDNS pod in Kube-system namespace | 18:01 |
| brtknr | Can you try that | 18:02 |
| brtknr | It will get recreated and should start working again | 18:02 |
| NobodyCam | yep will do now | 18:02 |
| NobodyCam | delete both? | 18:03 |
| NobodyCam | `kube-system coredns-78df4bf8ff-cj9gq 1/1 Running 0 16h 192.168.54.66 os-ps-us-west-irvine02-tnvmbvbrlcks-minion-0 <none> | 18:04 |
| NobodyCam | kube-system coredns-78df4bf8ff-kwcm8 1/1 Running 0 16h 192.168.54.69 os-ps-us-west-irvine02-tnvmbvbrlcks-minion-0 <none>` | 18:04 |
| brtknr | Oh you don’t have it running... there’s your problem | 18:05 |
| brtknr | It’s possible your nodes are tainted | 18:06 |
| brtknr | What version of k8s are you trying to run? | 18:06 |
| brtknr | Rocky only supports up to 1.11 | 18:06 |
| NobodyCam | this is calico if that makes a difference. | 18:06 |
| NobodyCam | one minion out of 12 is having this issue | 18:07 |
| brtknr | So it’s working with flannel? | 18:07 |
| brtknr | I don’t understand your answer? Out of 12? | 18:07 |
| NobodyCam | if I delete the calico pod will it get recreated | 18:07 |
| NobodyCam | oh sorry I meant to say that the other 11 minions are working okay! | 18:08 |
| NobodyCam | I am seeing that that minion does not have a kibe-proxy service | 18:12 |
| NobodyCam | *kube-proxy | 18:12 |
| NobodyCam | fyi: atomic install --storage ostree --system --system-package=no --name=kube-proxy docker.io/openstackmagnum/kubernetes-proxy:v1.11.9 | 18:21 |
| NobodyCam | got it working | 18:21 |
| *** jmlowe has quit IRC | 18:39 | |
| *** ramishra has quit IRC | 18:43 | |
| NobodyCam | Thank you brtknr I didn't describe the situation well at all | 18:58 |
| NobodyCam | not sure why that one minion didn't get the kibe-proxy service | 18:59 |
| NobodyCam | but installing it did get things working | 18:59 |
| mnaser | jrosser: in your environment, does your control plane have the ability to talk to your VMs? | 19:19 |
| mnaser | jrosser: i.e. if magnum was running an ansible playbook against VMs, would magnum be able to reach said VMs? | 19:20 |
| *** itlinux has quit IRC | 19:33 | |
| *** itlinux has joined #openstack-containers | 19:33 | |
| *** itlinux has quit IRC | 19:42 | |
| jrosser | mnaser: I can make that happen for http/s, yes | 19:51 |
| jrosser | Oh ansible, hmm no not right now no | 19:52 |
| *** jmlowe has joined #openstack-containers | 19:56 | |
| brtknr | NobodyCam: strange! Glad u got it working! | 20:40 |
| *** lpetrut has joined #openstack-containers | 20:50 | |
| *** lpetrut has quit IRC | 20:51 | |
| *** lpetrut has joined #openstack-containers | 20:51 | |
| *** lpetrut has quit IRC | 20:58 | |
| flwang | strigazi: around? | 21:01 |
| strigazi | o/ | 21:02 |
| flwang | NobodyCam: as for your kube-proxy issue, it'd better to debug why there is no kube-proxy | 21:03 |
| flwang | #startmeeting | 21:04 |
| openstack | flwang: Error: A meeting name is required, e.g., '#startmeeting Marketing Committee' | 21:04 |
| flwang | #startmeeting magnum | 21:04 |
| openstack | Meeting started Tue Aug 27 21:04:15 2019 UTC and is due to finish in 60 minutes. The chair is flwang. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:04 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:04 |
| *** openstack changes topic to " (Meeting topic: magnum)" | 21:04 | |
| openstack | The meeting name has been set to 'magnum' | 21:04 |
| flwang | #topic roll call | 21:04 |
| *** openstack changes topic to "roll call (Meeting topic: magnum)" | 21:04 | |
| strigazi | o/ | 21:04 |
| flwang | brtknr: jakeyip: | 21:04 |
| flwang | anyone else online? | 21:04 |
| flwang | strigazi: ok, let's start first | 21:05 |
| flwang | #topic flannel conformance | 21:05 |
| *** openstack changes topic to "flannel conformance (Meeting topic: magnum)" | 21:05 | |
| flwang | strigazi: did you see my email? | 21:06 |
| strigazi | The nic patch is definitely an issue for master branch | 21:06 |
| strigazi | after that I get internal IPs | 21:06 |
| flwang | strigazi: yes | 21:06 |
| flwang | ok, cool | 21:06 |
| strigazi | I'm looking into sec groups now | 21:06 |
| flwang | have you completed another sonobuoy testing? | 21:07 |
| *** spsurya has quit IRC | 21:07 | |
| strigazi | and the iptables patch we dropped | 21:07 |
| strigazi | I'm just checking one test groups of tests regarding DNS | 21:07 |
| *** itlinux has joined #openstack-containers | 21:07 | |
| strigazi | sonobuoy run --e2e-focus "DNS" | 21:07 |
| flwang | strigazi: ok, did you see my last comment on https://review.opendev.org/#/c/668163/? | 21:07 |
| strigazi | this covers the network usually | 21:07 |
| strigazi | when this passes the rest should work | 21:08 |
| flwang | at least based on my testing, the iptable patch doesn't help | 21:08 |
| flwang | strigazi: so you also got 10 test cases failed, right? | 21:08 |
| strigazi | yes, do the same pass for calico? | 21:09 |
| flwang | yes | 21:09 |
| strigazi | for master branch? | 21:09 |
| flwang | yes | 21:09 |
| strigazi | so calico is the issue :) | 21:09 |
| flwang | :D | 21:09 |
| flwang | http://paste.openstack.org/show/763160/ | 21:09 |
| flwang | can you pls check if you got the same 10 cases? | 21:09 |
| strigazi | I haven't left one to finish | 21:10 |
| strigazi | but the DNS one I have them | 21:10 |
| strigazi | *ones | 21:10 |
| flwang | ok | 21:10 |
| flwang | you mean this one [Fail] [sig-network] DNS [It] should provide DNS for the cluster [Conformance] ? | 21:11 |
| strigazi | yes | 21:11 |
| strigazi | and for services | 21:11 |
| flwang | right | 21:12 |
| strigazi | anyway, tomorrow I guess I'll have it working. | 21:12 |
| *** itlinux has quit IRC | 21:13 | |
| strigazi | why is calico working? it is no affected by the NIC patch? | 21:13 |
| flwang | fantastic | 21:13 |
| flwang | strigazi: it's also blocked by the nic | 21:13 |
| strigazi | so it doesn't work for master | 21:13 |
| flwang | when i said calico working, i mean the test i did about several weeks ago | 21:14 |
| flwang | at that moment, the nic patch hasn't merged yet | 21:14 |
| strigazi | ok | 21:14 |
| flwang | strigazi: this patch should be able to fix the regression issue https://review.opendev.org/678067 | 21:15 |
| flwang | i will check with brtknr if it's ready for testing | 21:15 |
| flwang | strigazi: shall we move to next topic? | 21:16 |
| strigazi | ok | 21:16 |
| flwang | #topic fedora coreos 30 | 21:16 |
| *** openstack changes topic to "fedora coreos 30 (Meeting topic: magnum)" | 21:16 | |
| flwang | yesterday, i have managed to get the ssh key, hostname and openstack-ca working for the new fedora coreos 30 image | 21:17 |
| flwang | today i will work on the heat-container-agent part | 21:17 |
| strigazi | ok | 21:17 |
| brtknr | o/ | 21:18 |
| flwang | btw, i can't remember how the cfn-init-data is written into the instance, can you pls remind me? | 21:18 |
| flwang | brtknr: hey | 21:18 |
| strigazi | heat appends them in cloud-init user-data | 21:19 |
| brtknr | apologies, i was at the cinema | 21:19 |
| flwang | strigazi: ah, i see. so we may have to inject it by ignition "manually"? | 21:19 |
| strigazi | in our case this gile will need to be crafted and injected as user data | 21:19 |
| brtknr | flwang: its ready for testing | 21:19 |
| brtknr | flwang: https://review.opendev.org/#/c/678067/ this patch | 21:20 |
| flwang | strigazi: i see. i will try | 21:20 |
| flwang | brtknr: thanks for the confirmation | 21:20 |
| flwang | i will update the fedora coreos 30 work with you guys later when there is any progress | 21:22 |
| flwang | #topic rolling upgrade | 21:22 |
| *** openstack changes topic to "rolling upgrade (Meeting topic: magnum)" | 21:22 | |
| flwang | so far the rolling upgrade patch for node operating system has passed my testing, https://review.opendev.org/669593 it would be nice if you guys can start reviewing it | 21:23 |
| brtknr | flwang: I'll test it tomorrow | 21:23 |
| flwang | the other thing i'd like to test is, if it can support migrating from fedora atomic 29 to fedora coreos 30, given they're all based on (rpm-) ostree | 21:24 |
| flwang | strigazi: ^ any comments? | 21:24 |
| strigazi | flwang: I don't know if it is possible | 21:24 |
| strigazi | maybe it is | 21:25 |
| *** ivve has quit IRC | 21:25 | |
| flwang | strigazi: anyway, we still need this upgrade to support user upgrade for fedora atomic | 21:25 |
| brtknr | flwang: I remember seeing on #fedora-coreos channel that they recommend users to rebuild instances instead of trying to upgrade | 21:25 |
| flwang | no matter is fedora atomic 27- >29 or small upgrade based on fedora atomic 29 | 21:26 |
| flwang | brtknr: i understand that, just thinking aloud, i know it's not a recommended way :) | 21:26 |
| strigazi | rebuild is the best in all scenarios IMO | 21:26 |
| flwang | strigazi: but for rebuild, we can't resolve the downtime issue now | 21:27 |
| flwang | unless we have a better way to orchestrate the upgrade progress | 21:27 |
| strigazi | depends in the pattern of usage | 21:27 |
| flwang | yes, i know | 21:28 |
| strigazi | if the pattern is cloudy, rebuild works | 21:28 |
| flwang | assume the cluster is created in a private network, mangum controll plane can't reach the cluster, then there is no good way to control the rebuild process | 21:28 |
| strigazi | anyway, depending on flannl I'll test upgfdae | 21:29 |
| flwang | strigazi: thank you | 21:29 |
| flwang | strigazi: brtknr: anything else your want to discuss? | 21:31 |
| brtknr | yes, i wanted to talk about whther you guys have kube_tag=v1.15.x working? | 21:32 |
| brtknr | i see there are images but i can only get upto 1.14 working on master | 21:32 |
| strigazi | brtknr: for flannel we need to update the manifest and a pod security policy | 21:33 |
| strigazi | after that it works | 21:33 |
| brtknr | i see theres a patch for supporting 1.16 from Richardo | 21:33 |
| strigazi | this is for the apis | 21:33 |
| brtknr | we need a better debug output for heat-container-agent... its currently incomprehensible | 21:34 |
| strigazi | brtknr: we need set +x before every source of heat-params | 21:35 |
| strigazi | and before when we write files to disk | 21:35 |
| brtknr | i can see that heat-container-agent:stein-stable has a readable outout to debug log but since train, it is hard to see what is failing | 21:35 |
| flwang | brtknr: it's related to the py3 support i think | 21:36 |
| flwang | it's a formating issue i would say | 21:37 |
| flwang | in other words, we still get the same output, but current format is bad | 21:37 |
| brtknr | flwang: okay i'll create a story for this as a reminder to investigate | 21:38 |
| strigazi | we can use logging into a file? | 21:38 |
| strigazi | but journal is better IMO | 21:39 |
| brtknr | strigazi: thats also a good idea... like /var/log/heat-container-agent-output.log? | 21:39 |
| strigazi | yeap | 21:39 |
| strigazi | os-collect-config should have something | 21:39 |
| brtknr | if it is more readable that how it currently is, i'd like that.. but i also prefer journalctl | 21:40 |
| flwang | before we fix the formating issue, redirect to a file doesn't help, IMHO | 21:40 |
| brtknr | i think the entire debug is getting written to the journal at once upon failure at the moment: https://github.com/openstack/magnum/blob/master/dockerfiles/heat-container-agent/scripts/55-heat-config#L153 | 21:42 |
| flwang | brtknr: yes | 21:42 |
| brtknr | it needs to be written atomically | 21:42 |
| flwang | in pretty format :) | 21:43 |
| brtknr | i dont understand how it looked pretty before | 21:43 |
| flwang | strigazi: did cern do any security review for magnum deployed k8s ? | 21:43 |
| flwang | brtknr: basically convert \n to a real breakline | 21:44 |
| strigazi | flwang: only from the outside of the cluster. And it is fine | 21:45 |
| strigazi | we have also used kube-hunter | 21:45 |
| strigazi | shall we wrap? | 21:46 |
| flwang | strigazi: cool | 21:46 |
| strigazi | anything else to discuss? | 21:46 |
| flwang | i'm good | 21:46 |
| brtknr | 1 last question about nodegroups | 21:46 |
| flwang | brtknr: anything else? | 21:46 |
| brtknr | any progress? | 21:47 |
| flwang | brtknr: i asked yesterday :D | 21:47 |
| brtknr | or plans to? | 21:47 |
| strigazi | it is in good shape but the author had some family priorities :) | 21:47 |
| strigazi | next week he is back | 21:48 |
| flwang | ok, let's wrap this one | 21:49 |
| brtknr | ah yes I heard about the paternity :) please send him my congratulations! | 21:49 |
| flwang | thank you for joining, strigazi, brtknr | 21:49 |
| flwang | #endmeeting | 21:49 |
| *** openstack changes topic to "OpenStack Containers Team" | 21:49 | |
| openstack | Meeting ended Tue Aug 27 21:49:18 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-08-27-21.04.html | 21:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-08-27-21.04.txt | 21:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/magnum/2019/magnum.2019-08-27-21.04.log.html | 21:49 |
| strigazi | brtknr: I wiil | 21:49 |
| flwang | strigazi: have a good night | 21:49 |
| strigazi | see you guys, thanks | 21:49 |
| brtknr | nice speaking to you both | 21:51 |
| *** trident has quit IRC | 22:05 | |
| *** trident has joined #openstack-containers | 22:13 | |
| *** rcernin has joined #openstack-containers | 22:15 | |
| lxkong | flwang, strigazi, I found some issues related to the nginx ingress controller, i'd like to get your feedback before i am actually doing the fix. https://storyboard.openstack.org/#!/story/2006462 | 23:25 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!