| *** k_mouza has joined #openstack-containers | 01:12 | |
| *** k_mouza has quit IRC | 01:16 | |
| *** LuckyClover has joined #openstack-containers | 01:56 | |
| *** ricolin has joined #openstack-containers | 02:01 | |
| LuckyClover | has anyone had issues with swift where when a cluster is created and you include registry that it fails | 02:19 |
|---|---|---|
| flwang | LuckyClover: what's the error you got? | 02:34 |
| flwang | LuckyClover: we're not using this feature on our production yet, but it's a quite simple config. so shouldn't be hard to fix if there is any issue | 02:35 |
| LuckyClover | so the cluster creation fails | 02:38 |
| LuckyClover | if we manually do it from cli | 02:38 |
| LuckyClover | panic: Swift authentication failed: Operation forbidden | 02:38 |
| LuckyClover | When we try to use CURL to test the credentials on the cli to the swift API, it gets: | 02:39 |
| LuckyClover | tx000000000000000053f58-005f1704ac-dcba8-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 Date: Tue, 21 Jul | 02:39 |
| LuckyClover | tried adding swiftoperator to the account and it still doesnt work | 02:39 |
| LuckyClover | We are using Ceph RadiosGW in our setup as Ceph is acting as both our block and object storage. | 02:40 |
| *** sapd1 has joined #openstack-containers | 02:41 | |
| flwang | LuckyClover: right. I see. | 02:45 |
| flwang | is the cluster creation working without enabling the container registry? | 02:45 |
| flwang | and what's the magnum version you're using? | 02:46 |
| LuckyClover | Yes it is | 02:47 |
| LuckyClover | whats the best way I can check magnum version number? | 02:48 |
| LuckyClover | i feel like I do it wrong every time I try to look for versions | 02:48 |
| *** dave-mccowan has quit IRC | 02:51 | |
| flwang | how did you install it? | 02:55 |
| LuckyClover | ansible | 02:56 |
| *** LuckyClover has quit IRC | 03:49 | |
| *** ykarel has joined #openstack-containers | 04:30 | |
| *** udesale has joined #openstack-containers | 05:27 | |
| *** sapd1 has quit IRC | 05:32 | |
| *** tobias-urdin|pto has quit IRC | 05:39 | |
| *** sapd1 has joined #openstack-containers | 05:54 | |
| *** vishalmanchanda has joined #openstack-containers | 05:55 | |
| *** ykarel_ has joined #openstack-containers | 05:59 | |
| *** ykarel has quit IRC | 06:02 | |
| *** nikparasyr has joined #openstack-containers | 06:31 | |
| *** ykarel_ is now known as ykarel | 06:38 | |
| *** born2bake has joined #openstack-containers | 07:04 | |
| *** sapd1 has quit IRC | 07:10 | |
| *** sapd1 has joined #openstack-containers | 07:13 | |
| *** rcernin has quit IRC | 07:32 | |
| *** flwang1 has joined #openstack-containers | 08:04 | |
| *** sapd1 has quit IRC | 08:13 | |
| *** sapd1 has joined #openstack-containers | 08:22 | |
| flwang1 | strigazi: brtknr: meeting in 8 mins? | 08:52 |
| brtknr | flwang1: sure | 08:56 |
| flwang1 | brtknr: i don't have much topic for today, pls add yours on https://etherpad.opendev.org/p/magnum-weekly-meeting | 08:57 |
| flwang1 | #startmeeting magnum | 09:00 |
| openstack | Meeting started Wed Jul 22 09:00:27 2020 UTC and is due to finish in 60 minutes. The chair is flwang1. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:00 |
| *** openstack changes topic to " (Meeting topic: magnum)" | 09:00 | |
| openstack | The meeting name has been set to 'magnum' | 09:00 |
| flwang1 | #topic roll call | 09:00 |
| *** openstack changes topic to "roll call (Meeting topic: magnum)" | 09:00 | |
| flwang1 | o/ | 09:00 |
| strigazi | o/ | 09:00 |
| brtknr | O/ | 09:00 |
| flwang1 | strigazi: hey | 09:01 |
| flwang1 | stranger | 09:01 |
| strigazi | hello | 09:01 |
| flwang1 | we do have some patches waiting another +2 from you | 09:01 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/ussuri: [k8s] Use helm upgrade --install in deployment loop https://review.opendev.org/742374 | 09:01 |
| flwang1 | brtknr: strigazi: do you have any topic you'd like to discuss today? | 09:02 |
| strigazi | flwang1: I know, I'll take care of them | 09:02 |
| strigazi | flwang1: I have one, and a half | 09:02 |
| flwang1 | strigazi: brtknr: the only update from my side is i'm revisiting the /nodes API patch | 09:02 |
| brtknr | hi strigazi | 09:02 |
| flwang1 | will submit patchset soon | 09:03 |
| brtknr | flwang1: sounds good | 09:03 |
| flwang1 | strigazi: you first? | 09:03 |
| strigazi | ok | 09:03 |
| strigazi | 1. For hyperkube and 1.19 | 09:03 |
| flwang1 | oh, god | 09:04 |
| strigazi | For after ussuri (current master) | 09:04 |
| strigazi | I think we will be safer if we move to deploying the binary, or add an option to deploy the binary. | 09:05 |
| flwang1 | strigazi: i would try avoid that | 09:05 |
| strigazi | For ussuri and older releases, we can do a build | 09:05 |
| flwang1 | unless we have a good solution for upgrade | 09:05 |
| strigazi | flwang1: My argument is: With the binary we maintain nothing. With the build we chase kubernetes releases. | 09:06 |
| flwang1 | we have broken the upgrade from v1.15 to v1.16, and I don't want to do that again | 09:06 |
| strigazi | flwang1: if we do both, we won't have something broken, isn't it? | 09:06 |
| flwang1 | how to upgrade from container to binary? | 09:07 |
| strigazi | Also, wait | 09:07 |
| brtknr | strigazi: how does the binary get deployed? | 09:07 |
| brtknr | by build do you mean hyperkube build? | 09:08 |
| flwang1 | stop container and replace with binary in upgrade_kubernetes.sh? | 09:08 |
| strigazi | If we build our own image, people that don't use a mirror from both k8s.gcr.io and docker.io/openstackmagnum, they will be broken too | 09:08 |
| strigazi | Because in upgrade.sh you need to switch registries | 09:08 |
| flwang1 | they need to use openstackmagnum , just like for heat-container-agent | 09:09 |
| strigazi | yes, but upgrade.sh doesnt do that | 09:10 |
| brtknr | strigazi: makes sense, your argument is that since we are changing the mode anyway, lets switch to binary to reduce maintenance overhead | 09:10 |
| flwang1 | i don't like the idea of binary, we need more discussion about this | 09:10 |
| flwang1 | brtknr: what do you mean 'changing the mode anyway'? | 09:11 |
| *** rcernin has joined #openstack-containers | 09:11 | |
| strigazi | brtknr means: k8s.gcr.io -> docker.io/openstackmagnum | 09:13 |
| strigazi | flwang1: I propose the following: | 09:13 |
| flwang1 | if we go for binary, is there is a trust palace we can get the binary? | 09:13 |
| strigazi | yes, give me a sec | 09:14 |
| strigazi | Your concerns of breaking upgrade are very valid. But it will break no matter what we do because of upstream kubernetes. Do we agree on this? | 09:15 |
| strigazi | flwang1: ^^ | 09:15 |
| flwang1 | if we still use hyperkube, why it will break upgrade? | 09:15 |
| strigazi | Let me explain | 09:15 |
| brtknr | yes | 09:15 |
| strigazi | in upgrade_kubernetes.sh we just bump the version of the image. The registry is unchanged. At CERN git have a mirror so we are not affected, but stock magnum will fetch hyperkube from k8s.gcr.io/hyperkube | 09:16 |
| strigazi | in upgrade_kubernetes.sh we just bump the version of the image. The registry is unchanged. At CERN we have a mirror of the registry so we are not affected, but stock magnum will fetch hyperkube from k8s.gcr.io/hyperkube | 09:17 |
| flwang1 | stock magnum means? | 09:17 |
| brtknr | upstream magnum | 09:17 |
| flwang1 | i believe any user use magnum on production will have a mirror | 09:18 |
| strigazi | the default code, with the default cluster creation parameters/labels | 09:18 |
| flwang1 | that said, at least with hyperkube, it won't break prod level usage | 09:18 |
| strigazi | sure, this is still an assumption even with high probability. | 09:19 |
| strigazi | So what if we cover both cases? | 09:19 |
| strigazi | Regarding the safe place for the binary: | 09:19 |
| strigazi | https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#downloads-for-v1186 | 09:20 |
| flwang1 | strigazi: we do need to support both for sure | 09:20 |
| strigazi | https://dl.k8s.io/v1.18.6/kubernetes-client-darwin-amd64.tar.gz with a sha512sum | 09:20 |
| flwang1 | okay | 09:20 |
| strigazi | well not darwin, that would be crazy | 09:20 |
| flwang1 | we(catalystcloud) got a lot of pain because of the breakage from v1.15 -> v1.16 | 09:21 |
| brtknr | strigazi: would we use a container to deploy the binaries? | 09:21 |
| flwang1 | you mean crazy to support both? | 09:21 |
| strigazi | it was a joke, we use amd64 and a few cases arm AFAIK | 09:21 |
| flwang1 | ah, you mean macos | 09:22 |
| flwang1 | sorry, i misunderstood | 09:22 |
| strigazi | brtknr: like containerd's logic | 09:22 |
| strigazi | We fetch the binary, we write the systemd unit. zero things to maintain | 09:23 |
| brtknr | strigazi: makes sense | 09:23 |
| strigazi | well apart from magnum's code | 09:23 |
| strigazi | I work on this, this week | 09:24 |
| brtknr | strigazi: i think it would be diligent to ensure that there is a good upgrade path | 09:24 |
| flwang1 | strigazi: i can help for the upgrade part | 09:24 |
| strigazi | brtknr: we can create an upgrade path for switching to the binary. I don't know if we can do it with the magnum api. We can try | 09:25 |
| strigazi | flwang1: brtknr: Regarding supporting high profile users | 09:26 |
| strigazi | I guess catalyst have some important client | 09:26 |
| strigazi | we have some experiments | 09:26 |
| flwang1 | we do have some customers running clusters created 1year ago :( | 09:27 |
| flwang1 | though we tried to push them upgrade | 09:27 |
| strigazi | At CERN we dedicate time directly on them, we can't have a generic solution that works for those case and dev clusetrs | 09:27 |
| flwang1 | strigazi: i appreciate your understanding | 09:28 |
| strigazi | Also, we spend some time on serviceType LoadBalancer to create a path for moving to new cluster | 09:28 |
| strigazi | with this https://github.com/kubernetes/cloud-provider-openstack/pull/1118 | 09:29 |
| flwang1 | migrate a lb from cluster A to cluster B? | 09:30 |
| strigazi | Anyway, we move to solution to do both, hyperkube and binary. People with mirrors will have a transparent upgrade | 09:30 |
| brtknr | strigazi: i like the pool | 09:30 |
| strigazi | flwang1 no | 09:30 |
| strigazi | flwang1: one LB | 09:30 |
| strigazi | flwang1: one unmanaged LB | 09:30 |
| strigazi | flwang1: add members of both cluster to the LB | 09:30 |
| strigazi | gradually remove members and eventually delete the old cluster (or delete in one go when thet new cluster is added to the LB) | 09:31 |
| flwang1 | strigazi: ok, i will read it later | 09:31 |
| strigazi | move on? | 09:33 |
| flwang1 | strigazi: sure | 09:33 |
| flwang1 | i'm keen to review your patch | 09:33 |
| flwang1 | what's your half one? | 09:33 |
| strigazi | master resize, I work on dropping the discovery url | 09:34 |
| strigazi | the other half if supporting it in the API | 09:34 |
| flwang1 | support resizing master on api? | 09:35 |
| strigazi | yes | 09:35 |
| strigazi | will you do it? | 09:35 |
| flwang1 | i can do it | 09:35 |
| *** rcernin has quit IRC | 09:35 | |
| strigazi | that's it | 09:36 |
| flwang1 | as long as you submit the code, i can start a following patch to start on the api part | 09:36 |
| strigazi | ok | 09:37 |
| brtknr | Looks like this just got merged: https://github.com/kubernetes/autoscaler/pull/3155 | 09:37 |
| *** sapd1 has quit IRC | 09:37 | |
| flwang1 | brtknr: without the /nodes api support from magnum? | 09:38 |
| strigazi | it was blocked by the huawei provider | 09:38 |
| brtknr | flwang1: i think this extends the previous approach to update heat stack directly to support nodegroups | 09:39 |
| flwang1 | brtknr: right, so it still would be nice to have the /nodes api support in magnum, is it? | 09:40 |
| brtknr | flwang1: yes I believe so | 09:40 |
| strigazi | yes | 09:40 |
| strigazi | not only nice, it will be an improvement | 09:40 |
| flwang1 | strigazi: ok, just want to make sure if i should still put effort on that one | 09:41 |
| brtknr | flwang1: as you can see, this PR was open for a long time, long before /nodes api was proposed | 09:42 |
| flwang1 | brtknr: i see. | 09:43 |
| flwang1 | brtknr: strigazi: anything else you want to discuss? | 09:43 |
| strigazi | I'm good | 09:44 |
| flwang1 | strigazi: i need your bless on this https://review.opendev.org/#/c/726017/ so that i can start the work on dashboard side | 09:44 |
| flwang1 | i'm keen to reduce the templates | 09:45 |
| brtknr | we need a few more +2s for ussuri release: https://review.opendev.org/#/q/status:open+project:openstack/magnum+branch:stable/ussuri | 09:45 |
| flwang1 | brtknr: thanks for bring this | 09:46 |
| flwang1 | i had done a review for the ussuri release recently | 09:46 |
| brtknr | Also what do you guys think about this? https://review.opendev.org/#/c/740439/ | 09:46 |
| brtknr | I was tired of creating lots of similar templates with 1 small change | 09:47 |
| flwang1 | brtknr: i love it | 09:47 |
| brtknr | great! | 09:47 |
| flwang1 | i even would like to have a magic command to duplicate template across regions | 09:48 |
| flwang1 | so i'm thinking if we can "export" a template and "import" it | 09:48 |
| strigazi | I think that clone is good only for admins | 09:49 |
| flwang1 | strigazi: exactly | 09:49 |
| strigazi | I know users can clone things manually | 09:49 |
| flwang1 | end user doesn't really need it | 09:49 |
| strigazi | but if it is very easy and we serve it to them, we may promote a pattern that they don't use the public cluster templates | 09:50 |
| *** pcaruana has quit IRC | 09:50 | |
| flwang1 | for us, now we maintain v1.16, v1.17 and v1.18, the only difference for those templates are the kube_tag and the template name | 09:51 |
| strigazi | It would be greate to do it in the API but I inderstand that it is an overkill | 09:51 |
| flwang1 | strigazi: it's an overkill | 09:52 |
| flwang1 | we're using a pipeline to publish templates acutally | 09:52 |
| flwang1 | for public templates | 09:52 |
| flwang1 | but i still think a clone command maybe useful | 09:53 |
| *** k_mouza has joined #openstack-containers | 09:53 | |
| flwang1 | can we introduce it as an admin command? | 09:53 |
| flwang1 | before we're confident it's worth to open to all users | 09:54 |
| strigazi | we can add a fake validation in the client | 09:54 |
| flwang1 | strigazi: we should be able to check the user role in client, no? | 09:54 |
| strigazi | yes, but virtualenv, pip install, sed <the file with the validation>, profit | 09:55 |
| strigazi | I think it's ok, we can do it | 09:55 |
| flwang1 | ok, brtknr, happy with that? | 09:56 |
| flwang1 | we're running out time | 09:56 |
| strigazi | it's like cephfs that has quotas on the client xD | 09:56 |
| flwang1 | :D | 09:56 |
| flwang1 | strigazi: i'm keen to review your master resize code and the binary kube code | 09:57 |
| strigazi | +1 | 09:57 |
| strigazi | brtknr: where are you?/ | 09:57 |
| brtknr | ok | 09:57 |
| brtknr | sounds good | 09:57 |
| brtknr | not 100% clear what we mean by fake validation | 09:58 |
| strigazi | brtknr: you check if the user is admin on the client | 09:58 |
| brtknr | strigazi: ok i will look into it | 09:58 |
| strigazi | brtknr: this is fake in the sense that the user can modifythe client | 09:58 |
| *** udesale_ has joined #openstack-containers | 09:58 | |
| flwang1 | i'm going to close this meeting now | 09:59 |
| flwang1 | thank you joining | 09:59 |
| strigazi | thanks, I'm good | 09:59 |
| flwang1 | i hope you guys are doing well in the covid-19 world | 09:59 |
| flwang1 | #endmeeting | 10:00 |
| *** openstack changes topic to "OpenStack Containers Team | Meeting: every Wednesday @ 9AM UTC | Agenda: https://etherpad.openstack.org/p/magnum-weekly-meeting" | 10:00 | |
| openstack | Meeting ended Wed Jul 22 10:00:13 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 10:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-07-22-09.00.html | 10:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-07-22-09.00.txt | 10:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/magnum/2020/magnum.2020-07-22-09.00.log.html | 10:00 |
| strigazi | brtknr: did you watch the webinar? Was it any useful to you? | 10:00 |
| brtknr | sorry i have some people working on our kitchen and they keep asking questions | 10:01 |
| brtknr | strigazi: yes very useful! i hope to see more! | 10:01 |
| *** udesale has quit IRC | 10:01 | |
| brtknr | were you able to see I was present? | 10:01 |
| brtknr | Looks like ephemeral containers are very useful debugging tool | 10:02 |
| strigazi | only in the beginning, then i was screen sharing so I don't what was going after | 10:02 |
| strigazi | Ricardo can generate the report as the creator of the webinar from zoom, but I didn't bother him :) | 10:02 |
| *** pcaruana has joined #openstack-containers | 10:02 | |
| brtknr | can you enable ephemeral containers in existing clusters? i suppose you need to change the admission controller manually | 10:03 |
| brtknr | but it was cool to see all the debugging methods you have in your utility belt :) | 10:03 |
| brtknr | i use only the very basic kubernetes features usually | 10:04 |
| brtknr | our CEO advertised the CERN webinars independently internally too so he must have heard about it from a different source so seems to be gaining traction | 10:05 |
| *** sapd1 has joined #openstack-containers | 10:14 | |
| strigazi | Stig? | 10:16 |
| strigazi | yes, you need to enable the feature gate | 10:16 |
| strigazi | we will enabled it by default | 10:17 |
| strigazi | brtknr: I have a bit nastier hacks too but I don't want to give people ideas :) | 10:17 |
| *** yasemind has quit IRC | 10:22 | |
| openstackgerrit | Merged openstack/magnum master: Add master_lb_enabled to cluster https://review.opendev.org/726017 | 10:41 |
| *** ricolin has quit IRC | 10:48 | |
| *** pcaruana has quit IRC | 11:03 | |
| *** yasemind has joined #openstack-containers | 11:36 | |
| *** rcernin has joined #openstack-containers | 11:50 | |
| *** dave-mccowan has joined #openstack-containers | 12:10 | |
| *** ykarel has quit IRC | 12:21 | |
| *** yankcrime has quit IRC | 12:32 | |
| *** ykarel has joined #openstack-containers | 12:40 | |
| *** ioni has quit IRC | 12:40 | |
| *** yankcrime has joined #openstack-containers | 12:41 | |
| *** ioni has joined #openstack-containers | 12:48 | |
| *** pcaruana has joined #openstack-containers | 12:54 | |
| *** sapd1 has quit IRC | 12:56 | |
| *** rcernin has quit IRC | 13:32 | |
| *** ricolin has joined #openstack-containers | 13:43 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/ussuri: Add master_lb_enabled to cluster https://review.opendev.org/742443 | 14:06 |
| *** dave-mccowan has quit IRC | 14:07 | |
| *** yolanda has quit IRC | 14:10 | |
| *** yolanda has joined #openstack-containers | 14:12 | |
| *** dave-mccowan has joined #openstack-containers | 14:20 | |
| *** ramishra has quit IRC | 14:22 | |
| *** ramishra has joined #openstack-containers | 14:22 | |
| *** dave-mccowan has quit IRC | 14:26 | |
| *** sapd1 has joined #openstack-containers | 14:37 | |
| *** nikparasyr has left #openstack-containers | 14:54 | |
| *** KeithMnemonic has joined #openstack-containers | 14:59 | |
| *** k_mouza has quit IRC | 15:00 | |
| *** k_mouza has joined #openstack-containers | 15:01 | |
| *** k_mouza has quit IRC | 15:41 | |
| *** k_mouza has joined #openstack-containers | 15:44 | |
| *** k_mouza has quit IRC | 15:55 | |
| *** ykarel is now known as ykarel|away | 16:11 | |
| *** ykarel|away has quit IRC | 16:17 | |
| *** ricolin has quit IRC | 16:19 | |
| *** udesale_ has quit IRC | 16:47 | |
| *** sapd1 has quit IRC | 17:03 | |
| *** k_mouza has joined #openstack-containers | 17:55 | |
| *** k_mouza has quit IRC | 17:59 | |
| *** LuckyClover has joined #openstack-containers | 18:03 | |
| *** flwang1 has quit IRC | 19:09 | |
| *** LuckyClover has quit IRC | 19:34 | |
| *** yolanda has quit IRC | 20:05 | |
| *** yolanda has joined #openstack-containers | 20:08 | |
| *** vishalmanchanda has quit IRC | 22:01 | |
| *** rcernin has joined #openstack-containers | 22:53 | |
| *** rcernin has quit IRC | 22:58 | |
| *** rcernin has joined #openstack-containers | 23:02 | |
| *** rcernin has quit IRC | 23:04 | |
| *** rcernin has joined #openstack-containers | 23:05 | |
| *** yolanda has quit IRC | 23:23 | |
| *** yolanda has joined #openstack-containers | 23:24 | |
| *** born2bake has quit IRC | 23:30 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!