| *** goldyfruit___ has joined #openstack-containers | 00:54 | |
| *** goldyfruit___ has quit IRC | 01:20 | |
| *** dasp has joined #openstack-containers | 02:26 | |
| *** dasp_ has quit IRC | 02:29 | |
| openstackgerrit | Merged openstack/magnum stable/stein: Set a fixed cipher suite set for Traefik https://review.opendev.org/696573 | 03:44 |
|---|---|---|
| *** ricolin has joined #openstack-containers | 03:49 | |
| *** ricolin has quit IRC | 03:51 | |
| *** ricolin has joined #openstack-containers | 04:09 | |
| *** udesale has joined #openstack-containers | 04:54 | |
| *** namrata has joined #openstack-containers | 05:48 | |
| *** elenalindq has joined #openstack-containers | 06:07 | |
| *** udesale has quit IRC | 06:07 | |
| *** xinliang has joined #openstack-containers | 06:27 | |
| *** udesale has joined #openstack-containers | 06:37 | |
| *** udesale has quit IRC | 06:39 | |
| *** udesale has joined #openstack-containers | 06:39 | |
| *** xinliang has quit IRC | 06:39 | |
| *** udesale has quit IRC | 06:42 | |
| *** jakeyip has joined #openstack-containers | 06:43 | |
| *** udesale has joined #openstack-containers | 06:52 | |
| *** namrata has quit IRC | 07:06 | |
| *** udesale has quit IRC | 07:13 | |
| *** udesale has joined #openstack-containers | 07:13 | |
| *** rcernin has quit IRC | 07:24 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: WIP: [k8s] Support docker storage driver for fedora coreos driver https://review.opendev.org/696256 | 07:27 |
| *** lpetrut has joined #openstack-containers | 07:28 | |
| *** namrata has joined #openstack-containers | 07:55 | |
| *** ramishra has joined #openstack-containers | 08:08 | |
| *** udesale has quit IRC | 08:28 | |
| *** udesale has joined #openstack-containers | 08:28 | |
| *** udesale has quit IRC | 08:29 | |
| *** trident has quit IRC | 08:55 | |
| *** trident has joined #openstack-containers | 08:56 | |
| *** ricolin has quit IRC | 09:18 | |
| andrein | Hello everyone | 09:55 |
| andrein | I've read the release notes for magnum 9.0.0 and noticed "When using a public cluster template, user still need the capability to reuse their existing network/subnet, and they also need to be able to turn of/off the floating IP to overwrite the setting in the public template. Now this is supported by adding those three items as parameters when creating cluster." | 09:55 |
| andrein | what are the label names I can use to do this? | 09:56 |
| *** rcernin has joined #openstack-containers | 10:32 | |
| *** pcaruana has joined #openstack-containers | 10:36 | |
| *** rcernin has quit IRC | 10:49 | |
| *** udesale has joined #openstack-containers | 10:53 | |
| andrein | I've tried adding fixed_network and fixed_subnet labels with the network/subnet ids I wanted, but it created a new network regardless | 10:53 |
| brtknr | andrein: fixed_network and fixed_subnet are cluster template parameters, not labels | 10:54 |
| andrein | ok, so I'm deffinitely missing something here | 10:55 |
| brtknr | you pass it using --fixed-network and --fixed-subnet args | 10:56 |
| brtknr | not --labels fixed_network | 10:56 |
| brtknr | not --labels fixed_network=id | 10:56 |
| brtknr | andrein: | 10:56 |
| andrein | ah, I see | 10:56 |
| namrata | brtknr we tried with stable/rocky and kube_tag 1.14.8 and the CLUSTER never gets completed as output of tail -f /var/log/cloud-init-output.log https://seashells.io/v/rwWjN799 | 11:03 |
| namrata | brtknr for reference output of cat /etc/sysconfig/heat-params https://seashells.io/v/5yPmJaqb | 11:04 |
| brtknr | namrata: tail is not that useful, could you show me journalctl -u heat-container-agent | 11:04 |
| namrata | brtknr https://seashells.io/v/URQGjerm | 11:06 |
| brtknr | are all the kube* services running? | 11:08 |
| brtknr | namrata: | 11:10 |
| namrata | brtknr https://seashells.io/v/6ncTaakt kubelet.service is in activating state | 11:11 |
| namrata | not active | 11:11 |
| brtknr | namrata: that looks bad | 11:12 |
| namrata | brtknr what we tested was adding a new script http://paste.openstack.org/show/786912/ `magnum/drivers/common/templates/kubernetes/fragments/patch-master.sh` and it worked fine and all the pods were running | 11:13 |
| namrata | does this help in any way where code is breaking | 11:13 |
| namrata | after this patch-master.sh script http://paste.openstack.org/show/786913/ all the pods were running | 11:14 |
| andrein | brtknr: I think I found a bug in the docs at https://docs.openstack.org/magnum/latest/user/#k8s-keystone-auth-tag | 11:16 |
| andrein | it links to https://hub.docker.com/r/k8scloudprovider/k8s-keystone-auth/tags/ | 11:16 |
| andrein | however, the cluster actually deploys https://hub.docker.com/r/openstackmagnum/k8s-keystone-auth/tags | 11:16 |
| andrein | is this a bug in the docs, or in the heat templates? | 11:17 |
| brtknr | andrein: most likely bug in the docs | 11:22 |
| andrein | brtknr: the one in the docs looks better maintained, there's a lot more labels available there | 11:23 |
| andrein | it has images for v1.13+ while the one under openstackmagnum only has v0.1.0 and v1.14.0 | 11:23 |
| trident | brtknr: So, it seems that at the point part-014 (enable-services-master.sh) is run the master node has not registered. They don't seem to register until after the last three parts (flannell-related) has been run. So, by making the patching the last step instead of i being done in part-014 everything works fine. | 11:26 |
| trident | brtknr: Could it be that this is done in a different order in later releases? | 11:27 |
| brtknr | commit d8df9d0c367943546e2f6498f7e3f5d1396126bc | 11:28 |
| brtknr | Author: Feilong Wang <flwang@catalyst.net.nz> | 11:28 |
| brtknr | Date: Thu Mar 14 16:49:37 2019 +1300 | 11:28 |
| brtknr | [fedora-atomic][k8s] Support default Keystone auth policy file | 11:28 |
| brtknr | With the new config option `keystone_auth_default_policy`, cloud admin | 11:28 |
| brtknr | can set a default keystone auth policy for k8s cluster when the | 11:28 |
| brtknr | keystone auth is enabled. As a result, user can use their current | 11:28 |
| brtknr | keystone user to access k8s cluster as long as they're assigned | 11:28 |
| brtknr | correct roles, and they will get the pre-defined permissions | 11:28 |
| brtknr | set by the cloud provider. | 11:28 |
| brtknr | The default policy now is based on the v2 format recently introduced | 11:28 |
| brtknr | in k8s-keystone-auth which is getting more useful now. For example, | 11:28 |
| brtknr | in v1 it doesn't support a policy for user to access resources from | 11:28 |
| brtknr | all namespaces but kube-system, but v2 can do that. | 11:28 |
| brtknr | NOTE: Now we're using openstackmagnum dockerhub repo until CPO | 11:28 |
| brtknr | team fixing their image release issue. | 11:28 |
| brtknr | Task: 30069 | 11:28 |
| brtknr | Story: 1755770 | 11:28 |
| brtknr | NOTE: Now we're using openstackmagnum dockerhub repo until CPO | 11:28 |
| brtknr | team fixing their image release issue. | 11:28 |
| brtknr | we can perhaps revert this if this issue has been fixed | 11:28 |
| *** rcernin has joined #openstack-containers | 12:31 | |
| *** goldyfruit___ has joined #openstack-containers | 12:43 | |
| andrein | what is the CPO team? | 13:01 |
| andrein | I'm guessing cloud-provider-openstack? | 13:18 |
| andrein | flwang: can we revert https://review.opendev.org/#/c/643225/22/magnum/drivers/common/templates/kubernetes/fragments/enable-keystone-auth.sh line 9? | 13:22 |
| andrein | looks like whatever was broken before was probably fixed, https://hub.docker.com/r/k8scloudprovider/k8s-keystone-auth/tags/ | 13:22 |
| brtknr | andrein: perhaps you could propose a patch | 13:39 |
| brtknr | and also update the default tag for Train release | 13:39 |
| andrein | Will do! | 13:39 |
| andrein | train's default is 1.14.x, right? | 13:39 |
| *** ramishra has quit IRC | 13:49 | |
| brtknr | andrein: 1.15.x i think | 13:51 |
| brtknr | train supports 1.16.x but requires podman | 13:51 |
| brtknr | so 1.15.x is better | 13:51 |
| andrein | https://review.opendev.org/#/c/685675/6/doc/source/user/index.rst << already outdated? :) | 13:51 |
| trident | brtknr: Would you think a patch proposal moving the node patching from enable-services-master.sh to a new separate patch-master.sh script that is run last would be a good idea? Or do you have any other suggestions? | 13:51 |
| trident | For stable/rocky that is. | 13:51 |
| brtknr | trident: sorry what is the context? | 13:52 |
| *** namrata has quit IRC | 13:53 | |
| trident | brtknr: The same issue as namrata and elenalindq have been discussing. I am with City Network as well. | 13:53 |
| trident | brtknr: So, it seems that at the point part-014 (enable-services-master.sh) is run the master node has not registered. They don't seem to register until after the last three parts (flannell-related) has been run. So, by making the patching the last step instead of i being done in part-014 everything works fine. So could it just be that things are done in a different order in newer releases than rocky so the patch that was cherry | 13:55 |
| trident | picked works there. | 13:55 |
| brtknr | trident: but it runs in a loop for that reason | 13:56 |
| brtknr | or is the loop blocking? | 13:56 |
| *** spsurya has joined #openstack-containers | 13:57 | |
| trident | brtknr: Yes, the loop is blocking. So it never gets to running part-015, part-016 and part-017. | 13:58 |
| brtknr | trident: interesting, wondering why its non blocking in stein | 13:58 |
| openstackgerrit | Andrei Nistor proposed openstack/magnum master: Change k8s-keystone-auth docker repo https://review.opendev.org/696706 | 13:59 |
| trident | brtknr: They do run them in order, right? And doesn't start running the next part until the previous one is finished. And part-014 never returns. | 13:59 |
| trident | brtknr: Ah, so that behaviour is different in >rocky... | 13:59 |
| brtknr | trident: seems that way | 14:00 |
| brtknr | trident: because it runs the script inside heat-container-agent rather than via cloud init | 14:01 |
| trident | brtknr: Ah, okay! | 14:02 |
| trident | brtknr: So, do you think moving that logic so it's run last by adding a new separate script to magnum/drivers/common/templates/kubernetes/fragments and modifying magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml to include it last in kube_master_init would be a good idea? | 14:06 |
| brtknr | trident: We dont normally modify previous branches, its normally make changes to master then backport | 14:07 |
| brtknr | it is possible a backport with the fix is missing | 14:07 |
| trident | brtknr: Hm, okay. I wonder which one that might be in that case. | 14:10 |
| *** dave-mccowan has joined #openstack-containers | 14:11 | |
| trident | brtknr: Are you sure that's in stein? And not in train by the way? That's when the enable-service-master.sh script start using ssh to start services for example. | 14:14 |
| trident | brtknr: https://github.com/openstack/magnum/commit/05c27f2d7399517c660ea233df816e74d8a75eae | 14:16 |
| trident | brtknr: Perhaps this could be related thoguh: https://github.com/openstack/magnum/commit/2ab874a5be951a6eba4f9d4f54c106bc0c53d9b1#diff-75661c66ec3a574f251543009c12d86f ? | 14:17 |
| trident | As that in fact makes enable_services be the last script run through cloud-init, right? | 14:18 |
| brtknr | trident: sorry for the slow response, i have lots of other things to do today but it sounds like you have a good sense of the problem | 14:28 |
| brtknr | i can take a look at this on monday | 14:28 |
| brtknr | this is why we were reluctant to propose fixes for rocky in the first place and now it seems like we may need to revert the changes that we merged yesterday and draw the line at support for v1.14.6 | 14:29 |
| trident | brtknr: Thanks! Yeah, I guess I have quite a good sense of the problem and can work around it locally. Just not sure what would be the right way to handle it in a more permanent manner. | 14:31 |
| trident | brtknr: I'll try applying https://github.com/openstack/magnum/commit/2ab874a5be951a6eba4f9d4f54c106bc0c53d9b1 locally as well and see if that helps. Feels like we easily can end up in dependency hell here. | 14:33 |
| brtknr | trident: if it doesn't apply cleanly, best to avoid it | 14:34 |
| brtknr | hence at StackHPC often tend to upgrade magnum before all our openstack services, Catalyst backport most things from master | 14:35 |
| brtknr | I think CERN do a lot of backports too | 14:35 |
| brtknr | if you want to run rocky... you couuld just loop and backport everything :D | 14:35 |
| brtknr | andrein: magnum-ui release is merged :D | 14:36 |
| brtknr | trident: basically, if you want to run the latest version of k8s, its important to run the latest version of magnum | 14:37 |
| brtknr | previous versions only guarentee support for older versions | 14:37 |
| andrein | andrein: Yay! thanks! | 14:38 |
| brtknr | we don't have the man power to make all previous openstack releases with all versions of k8s... hope that makes sense trident | 14:38 |
| brtknr | trident: this is because the kubernetes api changes in subtle ways over time and we try to adapt to those changes | 14:39 |
| *** udesale has quit IRC | 15:49 | |
| *** lpetrut has quit IRC | 15:52 | |
| *** dave-mccowan has quit IRC | 16:01 | |
| *** sapd1 has joined #openstack-containers | 16:11 | |
| *** lpetrut has joined #openstack-containers | 16:17 | |
| *** strobert1 has quit IRC | 16:21 | |
| openstackgerrit | Andrei Nistor proposed openstack/magnum master: Change k8s-keystone-auth docker repo https://review.opendev.org/696706 | 16:44 |
| brtknr | andrein: left another comment :) | 16:46 |
| andrein | Makes sense! | 16:47 |
| brtknr | andrein: we need to bump the versions in kube-cluster.yaml too | 16:48 |
| brtknr | i'd grep for k8s_keystone_auth_tag | 16:48 |
| andrein | brtknr: the version is already v1.14.0 in kubecluster.yaml as far as I see | 16:49 |
| andrein | did I miss something? | 16:50 |
| brtknr | andrein: ah so looks like the version got bumped up in Train | 16:52 |
| andrein | yep | 16:52 |
| brtknr | andrein: i g2g, have a good weekend! thanks for submitting those patches | 16:54 |
| andrein | brtknr: have a good weekend, it was a pleasure working with you this week! | 16:54 |
| brtknr | Likewise :) | 16:55 |
| openstackgerrit | Andrei Nistor proposed openstack/magnum master: Change k8s-keystone-auth docker repo https://review.opendev.org/696706 | 16:55 |
| brtknr | andrein: if you get a chance, please test with the new docker repo | 16:58 |
| andrein | I've manually edited the daemonset on my existing clusters and it worked (as in deployed) - i still haven't figured out how to use it yet | 16:59 |
| *** spsurya has quit IRC | 17:05 | |
| *** lpetrut has quit IRC | 17:16 | |
| *** lpetrut has joined #openstack-containers | 17:33 | |
| *** jmlowe has quit IRC | 19:56 | |
| *** jmlowe has joined #openstack-containers | 20:13 | |
| *** jmlowe has quit IRC | 20:53 | |
| *** elenalindq has quit IRC | 21:09 | |
| *** jmlowe has joined #openstack-containers | 21:50 | |
| *** pcaruana has quit IRC | 22:03 | |
| *** rcernin has quit IRC | 23:18 | |
| *** sapd1 has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!