| openstackgerrit | Feilong Wang proposed openstack/magnum master: [fedora-atomic][k8s][security] More strict security group rules https://review.opendev.org/675799 | 00:09 |
|---|---|---|
| *** yolanda has quit IRC | 00:12 | |
| *** yolanda has joined #openstack-containers | 00:13 | |
| *** yolanda has quit IRC | 02:32 | |
| *** yolanda has joined #openstack-containers | 02:37 | |
| *** yolanda has quit IRC | 02:43 | |
| *** yolanda has joined #openstack-containers | 02:44 | |
| *** yolanda has quit IRC | 03:17 | |
| *** yolanda has joined #openstack-containers | 03:22 | |
| *** ykarel|away has joined #openstack-containers | 03:35 | |
| *** udesale has joined #openstack-containers | 04:04 | |
| *** ykarel|away has quit IRC | 04:11 | |
| *** ykarel|away has joined #openstack-containers | 04:27 | |
| *** ykarel|away is now known as ykarel | 04:34 | |
| *** pcaruana has joined #openstack-containers | 05:29 | |
| *** pcaruana has quit IRC | 05:37 | |
| *** janki has joined #openstack-containers | 05:48 | |
| *** pcaruana has joined #openstack-containers | 05:50 | |
| *** ramishra has joined #openstack-containers | 06:28 | |
| *** ramishra_ has joined #openstack-containers | 06:57 | |
| *** ramishra has quit IRC | 06:59 | |
| *** ivve has joined #openstack-containers | 07:05 | |
| *** rcernin has quit IRC | 07:13 | |
| *** ykarel is now known as ykarel|lunch | 07:33 | |
| *** ramishra_ has quit IRC | 07:54 | |
| *** ramishra has joined #openstack-containers | 07:58 | |
| *** janki has quit IRC | 08:16 | |
| *** janki has joined #openstack-containers | 08:17 | |
| *** jchhatbar has joined #openstack-containers | 08:18 | |
| *** janki has quit IRC | 08:22 | |
| *** ramishra_ has joined #openstack-containers | 08:42 | |
| *** hogepodge has quit IRC | 08:42 | |
| *** jchhatbar has quit IRC | 08:44 | |
| *** ramishra has quit IRC | 08:45 | |
| *** openstackgerrit has quit IRC | 08:45 | |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| *** hogepodge has joined #openstack-containers | 08:47 | |
| *** hogepodge has quit IRC | 08:47 | |
| *** hogepodge has joined #openstack-containers | 08:47 | |
| *** irclogbot_3 has quit IRC | 08:49 | |
| *** irclogbot_1 has joined #openstack-containers | 08:52 | |
| *** flwang1 has joined #openstack-containers | 09:03 | |
| flwang1 | strigazi: around? | 09:03 |
| flwang1 | strigazi: seems we're have a new issue with the heat-container-agent | 09:03 |
| flwang1 | it's reporting a locale issue now | 09:03 |
| brtknr | flwang1: thats the same issue as what i had... had to change heat_container_agent_tag=stein-stable | 09:17 |
| flwang1 | brtknr: i had put some time on that, but haven't figured out the root cause yet | 09:17 |
| brtknr | flwang1: https://storyboard.openstack.org/#!/story/2006382 | 09:17 |
| brtknr | i repored the issue on storyboard on 8 august | 09:18 |
| flwang1 | brtknr: thanks | 09:19 |
| brtknr | flwang1: even tried changing the locale of the devstack machine but no help | 09:19 |
| flwang1 | brtknr: i will continue to debug, will update status on that story | 09:20 |
| brtknr | 09:20 | |
| brtknr | how do you approach debugging heat-container-agent container? | 09:21 |
| brtknr | flwang1: ^ | 09:21 |
| brtknr | flwang1: also could you please take a look at this urgently: https://review.opendev.org/#/c/675548/ | 09:22 |
| brtknr | the controller manager fails to deploy without this | 09:23 |
| flwang1 | brtknr: i'm going to enable more log for the launch script, and if that doesn't work, i will probably build a personal os-collect-config lib to debug | 09:23 |
| flwang1 | it's a big tricky, because the container can't start at this moment | 09:23 |
| flwang1 | a simpler way we can try is, don't run the os-collect-config script when start the container, then after start the container, login the bash of the container to run the command manually | 09:25 |
| flwang1 | and add pdb for the os-collect-config to get more information | 09:25 |
| flwang1 | without heat-container-agent, anything can't work | 09:25 |
| *** ArchiFleKs has quit IRC | 09:44 | |
| *** sapd1_x has joined #openstack-containers | 10:08 | |
| *** sapd1_x has quit IRC | 10:16 | |
| brtknr | flwang1: hmm yes | 10:29 |
| *** ykarel is now known as ykarel|afk | 10:36 | |
| *** pcaruana has quit IRC | 10:43 | |
| *** pcaruana has joined #openstack-containers | 10:43 | |
| *** ykarel|afk is now known as ykarel | 11:02 | |
| *** janki has joined #openstack-containers | 11:08 | |
| *** yankcrime has quit IRC | 11:11 | |
| *** yankcrime has joined #openstack-containers | 11:17 | |
| *** udesale has quit IRC | 11:18 | |
| *** yankcrime has quit IRC | 11:29 | |
| *** henriqueof has joined #openstack-containers | 11:42 | |
| *** yankcrime has joined #openstack-containers | 11:45 | |
| *** janki has quit IRC | 11:51 | |
| *** janki has joined #openstack-containers | 11:53 | |
| *** ykarel is now known as ykarel|afk | 12:16 | |
| *** ykarel|afk has quit IRC | 12:28 | |
| *** jchhatbar has joined #openstack-containers | 12:47 | |
| *** jchhatbar has quit IRC | 12:48 | |
| *** ykarel|afk has joined #openstack-containers | 12:48 | |
| *** jchhatbar has joined #openstack-containers | 12:48 | |
| *** janki has quit IRC | 12:50 | |
| *** jchhatbar has quit IRC | 12:52 | |
| *** ykarel|afk is now known as ykarel | 12:52 | |
| *** janki has joined #openstack-containers | 13:09 | |
| brtknr | goldyfruit_: thanks for the review | 13:29 |
| *** udesale has joined #openstack-containers | 13:30 | |
| goldyfruit_ | brtknr, :) | 13:33 |
| *** spiette has quit IRC | 13:51 | |
| *** spiette has joined #openstack-containers | 13:53 | |
| *** KeithMnemonic has joined #openstack-containers | 13:54 | |
| *** janki has quit IRC | 13:57 | |
| brtknr | goldyfruit_: btw have you seen this before: Aug 12 11:57:08 testk8s-zpmpusaolfdn-master-0.novalocal runc[3610]: error: SchemaError(io.k8s.api.apps.v1beta1.ControllerRevisionList): invalid object doesn't have additional properties | 14:03 |
| brtknr | this is when creating openstack-cloud-controller-manager pod in the kube-system namespace | 14:03 |
| brtknr | this is via the heat-container-agent | 14:03 |
| brtknr | if i apply the template manually, no complaints | 14:04 |
| brtknr | my suspicion is that the control plane is still getting ready | 14:04 |
| *** udesale has quit IRC | 14:06 | |
| *** udesale has joined #openstack-containers | 14:08 | |
| *** udesale has quit IRC | 14:08 | |
| *** udesale has joined #openstack-containers | 14:08 | |
| goldyfruit_ | brtknr, never get this one | 14:08 |
| *** ykarel is now known as ykarel|afk | 14:10 | |
| *** ykarel|afk has quit IRC | 14:43 | |
| *** ykarel|afk has joined #openstack-containers | 14:43 | |
| brtknr | goldyfruit_: i am trying to find out how the latest version of kubectl gets installed | 14:47 |
| brtknr | any idea? | 14:47 |
| brtknr | i cannot find the part of the softwarewaredeployment script that is responsible | 14:47 |
| brtknr | this is on the master node | 14:47 |
| brtknr | could it be one of the kube_* services' overlayfs? | 14:48 |
| goldyfruit_ | brtknr, https://github.com/openstack/magnum/blob/a70341e278a7592ef880076445d3f2a106ed6a23/dockerfiles/kubernetes-apiserver/Dockerfile | 14:48 |
| goldyfruit_ | It's part of kube-apiserver Docker image | 14:49 |
| brtknr | goldyfruit_: it looks like the reason I am hitting the issue I reported earlier is because, at the point that heat-container-agent gets fired, there is mismatch between the version of kubectl on the client and the server | 14:51 |
| brtknr | brtknr, https://github.com/openstack/magnum/blob/a70341e278a7592ef880076445d3f2a106ed6a23/dockerfiles/kub | 14:51 |
| brtknr | ernetes-apiserver/Dockerfile | 14:51 |
| brtknr | Writing File: /srv/magnum/kubernetes/openstack-cloud-controller-manager.yaml | 14:51 |
| brtknr | Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548b | 14:51 |
| brtknr | Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.5", GitCommit:"0e9fcb426b100a2aea5ed5c25b3 | 14:51 |
| brtknr | as so^ | 14:51 |
| brtknr | goldyfruit_: I found the problem! | 14:54 |
| goldyfruit_ | brtknr, Yeah! | 14:55 |
| brtknr | Because the script is running from heat-container-agent, it is using the kubectl client from the container | 14:55 |
| brtknr | heat-container-agent has the old kubectl client | 14:55 |
| brtknr | https://github.com/openstack/magnum/blob/master/dockerfiles/heat-container-agent/Dockerfile#L18 | 14:56 |
| goldyfruit_ | brtknr, but you are using rawhide tag which is from February | 14:57 |
| goldyfruit_ | So I guess this is normal to have an old client in there | 14:57 |
| *** ykarel|afk is now known as ykarel|away | 14:57 | |
| goldyfruit_ | If you want a newer client you will have to change your heat-container-agent tag | 14:57 |
| goldyfruit_ | but last time you tried stein-stable/train-dev you had troubles | 14:58 |
| brtknr | goldyfruit_: my heat_container_agent_tag is rocky-stable as I am using openstack rocky | 15:01 |
| brtknr | its not just an old client, its a really old client | 15:01 |
| goldyfruit_ | brtknr, rocky works until 1.11 I think | 15:02 |
| *** ykarel|away has quit IRC | 15:04 | |
| *** ykarel|away has joined #openstack-containers | 15:04 | |
| brtknr | goldyfruit_: ah i see | 15:05 |
| brtknr | goldyfruit_: i think it can work for other versions if we change heat_container_agent_tag to stein-stable | 15:07 |
| goldyfruit_ | brtknr, yes it could | 15:08 |
| *** dave-mccowan has joined #openstack-containers | 15:21 | |
| brtknr | goldyfruit_: looks like the heat_container_agent_tag doesnt get respected in rocky :( | 15:52 |
| *** udesale has quit IRC | 15:56 | |
| *** dave-mccowan has quit IRC | 15:56 | |
| brtknr | although i can see that the feature got backported to rocky | 15:59 |
| *** dtruong has quit IRC | 15:59 | |
| *** dtruong has joined #openstack-containers | 16:00 | |
| *** ykarel|away has quit IRC | 16:08 | |
| *** absubram has joined #openstack-containers | 17:11 | |
| *** ramishra_ has quit IRC | 17:26 | |
| *** ivve has quit IRC | 17:27 | |
| *** absubram has quit IRC | 17:47 | |
| *** ivve has joined #openstack-containers | 18:58 | |
| *** flwang1 has quit IRC | 19:09 | |
| *** dave-mccowan has joined #openstack-containers | 19:54 | |
| *** pcaruana has quit IRC | 20:56 | |
| *** absubram has joined #openstack-containers | 21:02 | |
| *** henriqueof has quit IRC | 22:04 | |
| *** lbragstad has joined #openstack-containers | 22:09 | |
| *** lbragstad has quit IRC | 22:17 | |
| *** ivve has quit IRC | 22:33 | |
| *** rcernin has joined #openstack-containers | 23:11 | |
| *** dave-mccowan has quit IRC | 23:17 | |
| *** absubram has quit IRC | 23:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!