| *** salmankhan has quit IRC | 00:07 | |
| *** munimeha1 has quit IRC | 00:09 | |
| *** kevko_ has joined #openstack-containers | 00:37 | |
| kevko_ | flwang: thank you for +2 :) | 00:37 |
|---|---|---|
| flwang | kevko_: sure | 00:40 |
| kevko_ | flwang: i remember that i already fixed block of code with x509 keypair ..but i am not sure if included in this review ...will check tommorrow and if no ..i will post another one with py3 compatibility ...we in debian already using py3 ocata i think (maybe queens) | 00:41 |
| kevko_ | flwang: this was tested with barbican and worked .. | 00:41 |
| flwang | cool | 00:42 |
| kevko_ | flwang: hh, don't be happy ..i was still trying with rawhide image of heat-container | 00:42 |
| kevko_ | flwang: magnum 7.0.2 , heat-container-agent:rawhide, enable_cert_api true in template have worked ... | 00:42 |
| flwang | again | 00:43 |
| flwang | you can't use rawhide if you have multi regions | 00:43 |
| kevko_ | flwang: i don't have .. | 00:44 |
| kevko_ | flwang: you adviced me to try x509keypair and rocky-stable ..now i'm running it | 00:44 |
| kevko_ | flwang: i know it :) ..i am trying to have all functional ... all choices :) | 00:45 |
| kevko_ | flwang: we want to put it into production ..so want to have tested all combinations .. | 00:45 |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: wip: k8s: add boot volume support https://review.openstack.org/621734 | 00:45 |
| flwang | so does the rocky-stable work for you now? | 00:46 |
| flwang | mnaser: seems the minion yaml is missing something ? | 00:48 |
| mnaser | flwang: that was just a wip i threw up without much testing just to see how gates look, havent tested locally yet, let me see | 00:49 |
| mnaser | ah you're right | 00:49 |
| mnaser | didnt hit ctrl+s | 00:49 |
| mnaser | conditions will help clean up our code if we start using them, get rid of all that magnum::optional stuff that's loaded by env | 00:50 |
| flwang | mnaser: that's a nice feature, i discussed it with strigazi at Berlin | 00:52 |
| mnaser | flwang: its a bug in our case :) we do bfv only and we are more recently discovering how breaks :< | 00:53 |
| kevko_ | flwang: now rocky stable with x509 keypair master passed all 19 scripts ..waiting for minion :) | 00:53 |
| mnaser | so lets call it a bug so we can backport... :;p | 00:53 |
| mnaser | :P | 00:53 |
| kevko_ | flwang: but i have to test with barbican also | 00:53 |
| flwang | kevko_: pls open a bug if you found it works and barbican case not | 00:54 |
| flwang | mnaser: i don't think i understand the bug | 00:54 |
| mnaser | flwang: if your cloud uses boot from volume only, then magnum boots a vm in a flavor with root_gb=0 | 00:54 |
| mnaser | in that case, nova actually boots a vm with a local disk the size of the image.. so 5GB with atomic | 00:55 |
| mnaser | that's a security bug and in rocky it was patched to give a warning but in stein it is no longer possible to boot a vm with root_gb=0 and not provide a volume | 00:55 |
| flwang | mnaser: got it, thanks. it would be nice if you can file a bug/story to track it | 00:58 |
| mnaser | flwang: ack | 00:59 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Support Keystone AuthN and AuthZ for k8s https://review.openstack.org/561783 | 01:00 |
| kevko_ | flwang: hmm, do you really have working config with x509keypair ? my master node is completed , notified to heat , but i see that resource enable_cert_manager api is still create in progress... but i have cert_manager_api = false in template ...so why this resource is creating in deployment ? | 01:03 |
| flwang | kevko_: we're running magnum on our production | 01:04 |
| flwang | welcome to register Catalyst Cloud if you want to try | 01:05 |
| flwang | kevko_: what's your heat version? | 01:05 |
| openstackgerrit | Mohammed Naser proposed openstack/magnum master: wip: k8s: add boot volume support https://review.openstack.org/621734 | 01:06 |
| *** lbragstad has quit IRC | 01:07 | |
| kevko_ | flwang: http://paste.openstack.org/show/736604/ check | 01:11 |
| kevko_ | flwang: i have also password there ..but doesn't matter ... will destroy test env ..and create new ...maybe i'm missing something in configuration ? | 01:12 |
| kevko_ | flwang: interesting is end of paste .... publicURL in region null | 01:13 |
| flwang | did you check the log of heat-container-agent? | 01:16 |
| flwang | all good? | 01:16 |
| kevko_ | flwang: no error , everything goog | 01:16 |
| kevko_ | good | 01:16 |
| kevko_ | flwang: same symptoms always ..publicURL in region null .. if i switch to rawhide ..it just works | 01:17 |
| flwang | what do you mean publicURL in region null ? | 01:18 |
| kevko_ | flwang: oh, paste.openstack.org cut the end .. | 01:19 |
| kevko_ | flwang: wait | 01:19 |
| kevko_ | flwang: http://paste.openstack.org/show/736605/ | 01:21 |
| kevko_ | flwang: this | 01:21 |
| kevko_ | flwang: all other things looks good ...so, i'm wondering if i am not missing some config in magnum .. | 01:22 |
| flwang | i can see this 'cluster_user_trust = True' | 01:25 |
| flwang | kevko_: my conf http://paste.openstack.org/show/736606/ | 01:26 |
| *** jaewook_oh has joined #openstack-containers | 01:31 | |
| *** jaewook_oh has quit IRC | 01:31 | |
| kevko_ | flwang: hmm, ok ...i will try and let you know .. | 01:32 |
| kevko_ | flwang: probably tomorrow , now i'm going to sleep ..here is 2:33 am :) | 01:33 |
| openstackgerrit | Merged openstack/magnum master: Add Octavia python client for Magnum https://review.openstack.org/615591 | 01:34 |
| openstackgerrit | Lingxian Kong proposed openstack/magnum master: Support hook mechanism for cluster deletion https://review.openstack.org/497144 | 01:35 |
| openstackgerrit | Lingxian Kong proposed openstack/magnum master: Add load balancer hook for cluster pre-deletion https://review.openstack.org/620761 | 01:35 |
| *** hongbin has joined #openstack-containers | 01:43 | |
| kevko_ | bye bye guys | 01:52 |
| *** kevko_ has quit IRC | 01:52 | |
| *** dodo_o has quit IRC | 02:11 | |
| openstackgerrit | Jason SUN proposed openstack/magnum master: Change openstack-dev to openstack-discuss https://review.openstack.org/621836 | 02:14 |
| *** itlinux has joined #openstack-containers | 02:23 | |
| *** itlinux has quit IRC | 02:33 | |
| *** itlinux has joined #openstack-containers | 02:34 | |
| *** itlinux has quit IRC | 02:36 | |
| openstackgerrit | Jason SUN proposed openstack/python-magnumclient master: Change openstack-dev to openstack-discuss https://review.openstack.org/621887 | 02:55 |
| *** hongbin_ has joined #openstack-containers | 03:11 | |
| *** hongbin has quit IRC | 03:13 | |
| openstackgerrit | Guo Jingyu proposed openstack/magnum master: Change openstack-dev to openstack-discuss https://review.openstack.org/621922 | 03:17 |
| *** hongbin has joined #openstack-containers | 03:22 | |
| *** jmlowe has joined #openstack-containers | 03:22 | |
| *** hongbin_ has quit IRC | 03:23 | |
| *** ricolin has joined #openstack-containers | 03:44 | |
| *** ramishra has joined #openstack-containers | 03:46 | |
| *** dave-mccowan has quit IRC | 03:53 | |
| *** ykarel|away has joined #openstack-containers | 03:58 | |
| *** itlinux has joined #openstack-containers | 04:14 | |
| *** Nel1x has quit IRC | 04:15 | |
| *** ykarel|away has quit IRC | 04:24 | |
| *** ykarel|away has joined #openstack-containers | 04:25 | |
| *** ykarel|away is now known as ykarel | 04:27 | |
| *** janki has joined #openstack-containers | 05:22 | |
| openstackgerrit | Huang.Xiangdong proposed openstack/python-magnumclient master: Add "--labels-override" boolean option when creating a cluster https://review.openstack.org/621994 | 05:24 |
| *** hongbin has quit IRC | 05:39 | |
| *** dodo_o has joined #openstack-containers | 07:08 | |
| *** pcaruana has joined #openstack-containers | 07:10 | |
| *** ykarel_ has joined #openstack-containers | 07:29 | |
| *** ykarel has quit IRC | 07:31 | |
| *** rcernin has quit IRC | 07:38 | |
| *** ttsiouts has joined #openstack-containers | 07:40 | |
| *** ttsiouts_ has joined #openstack-containers | 07:58 | |
| *** ttsiouts has quit IRC | 08:01 | |
| *** belmoreira has joined #openstack-containers | 08:01 | |
| *** ttsiouts_ has quit IRC | 08:17 | |
| *** ttsiouts has joined #openstack-containers | 08:18 | |
| *** ttsiouts has quit IRC | 08:22 | |
| tobias-urdin | strigazi: will new clusters pull in the v1.11.5-1 image without any intervention, so i only have to worry about those currently running? | 08:23 |
| *** ykarel_ is now known as ykarel | 08:29 | |
| *** ianychoi has quit IRC | 08:30 | |
| *** ianychoi has joined #openstack-containers | 08:30 | |
| openstackgerrit | ShangXiao proposed openstack/magnum master: Add release notes link to README https://review.openstack.org/560822 | 08:50 |
| *** ykarel is now known as ykarel|lunch | 08:51 | |
| *** ttsiouts has joined #openstack-containers | 08:52 | |
| openstackgerrit | ShangXiao proposed openstack/magnum master: Change bugs link for README https://review.openstack.org/560822 | 08:54 |
| strigazi | tobias-urdin: yes | 09:04 |
| tobias-urdin | thank you | 09:20 |
| *** lbragstad has joined #openstack-containers | 09:24 | |
| *** ykarel|lunch is now known as ykarel | 09:26 | |
| *** salmankhan has joined #openstack-containers | 10:34 | |
| *** ttsiouts has quit IRC | 10:36 | |
| *** ttsiouts has joined #openstack-containers | 10:37 | |
| *** salmankhan1 has joined #openstack-containers | 10:38 | |
| *** ttsiouts_ has joined #openstack-containers | 10:38 | |
| *** salmankhan has quit IRC | 10:38 | |
| *** salmankhan1 is now known as salmankhan | 10:38 | |
| *** ttsiouts has quit IRC | 10:42 | |
| *** ttsiouts_ has quit IRC | 11:01 | |
| *** ttsiouts has joined #openstack-containers | 11:01 | |
| *** ttsiouts has quit IRC | 11:06 | |
| *** ttsiouts has joined #openstack-containers | 11:45 | |
| *** ttsiouts has quit IRC | 11:54 | |
| *** ttsiouts has joined #openstack-containers | 11:55 | |
| *** dodo_o has quit IRC | 11:58 | |
| *** ttsiouts has quit IRC | 11:59 | |
| *** vabada has joined #openstack-containers | 11:59 | |
| *** vabada has quit IRC | 11:59 | |
| *** vabada has joined #openstack-containers | 12:00 | |
| *** dave-mccowan has joined #openstack-containers | 12:12 | |
| *** lbragstad has quit IRC | 12:21 | |
| *** lbragstad has joined #openstack-containers | 12:23 | |
| *** lbragstad has quit IRC | 12:23 | |
| *** ttsiouts has joined #openstack-containers | 12:23 | |
| *** lbragstad has joined #openstack-containers | 12:24 | |
| *** lbragsta_ has joined #openstack-containers | 12:26 | |
| *** ykarel is now known as ykarel|afk | 12:29 | |
| *** shrasool has joined #openstack-containers | 12:29 | |
| *** lbragsta_ has quit IRC | 12:31 | |
| *** lbragstad has quit IRC | 12:31 | |
| *** lbragstad has joined #openstack-containers | 12:37 | |
| *** ykarel|afk is now known as ykarel | 12:43 | |
| *** shrasool has quit IRC | 12:47 | |
| *** ttsiouts has quit IRC | 12:49 | |
| *** ttsiouts has joined #openstack-containers | 12:50 | |
| *** ykarel is now known as ykarel|afk | 12:51 | |
| *** shrasool has joined #openstack-containers | 12:52 | |
| *** udesale has joined #openstack-containers | 13:00 | |
| *** mgariepy has quit IRC | 13:03 | |
| *** mgariepy has joined #openstack-containers | 13:08 | |
| *** ttsiouts has quit IRC | 13:24 | |
| *** ttsiouts has joined #openstack-containers | 13:25 | |
| *** ttsiouts_ has joined #openstack-containers | 13:29 | |
| *** ttsiouts has quit IRC | 13:30 | |
| *** shrasool has quit IRC | 13:31 | |
| *** munimeha1 has joined #openstack-containers | 13:45 | |
| *** janki has quit IRC | 13:50 | |
| *** udesale has quit IRC | 14:08 | |
| *** irclogbot_0 has quit IRC | 14:15 | |
| *** janki has joined #openstack-containers | 14:29 | |
| *** zul has quit IRC | 14:37 | |
| *** zul has joined #openstack-containers | 14:41 | |
| *** ttsiouts_ has quit IRC | 14:44 | |
| *** janki has quit IRC | 14:44 | |
| *** salmankhan has quit IRC | 14:44 | |
| *** ttsiouts has joined #openstack-containers | 14:44 | |
| *** ttsiouts_ has joined #openstack-containers | 14:46 | |
| openstackgerrit | Michal Arbet proposed openstack/magnum master: Fix python3 compatibility https://review.openstack.org/618756 | 14:47 |
| *** hongbin has joined #openstack-containers | 14:47 | |
| *** ttsiouts has quit IRC | 14:49 | |
| *** ttsiouts_ has quit IRC | 14:50 | |
| *** udesale has joined #openstack-containers | 14:53 | |
| *** irclogbot_0 has joined #openstack-containers | 14:55 | |
| *** zul has quit IRC | 14:58 | |
| *** salmankhan has joined #openstack-containers | 14:59 | |
| *** itlinux has quit IRC | 15:18 | |
| *** jmlowe has quit IRC | 15:26 | |
| *** zul has joined #openstack-containers | 15:41 | |
| *** jmlowe has joined #openstack-containers | 15:50 | |
| *** shrasool has joined #openstack-containers | 15:50 | |
| *** ykarel|afk is now known as ykarel | 15:53 | |
| *** jmlowe has quit IRC | 15:57 | |
| *** jmlowe has joined #openstack-containers | 15:57 | |
| *** shrasool has quit IRC | 16:02 | |
| *** hongbin has quit IRC | 16:06 | |
| *** pcaruana has quit IRC | 16:12 | |
| *** hongbin has joined #openstack-containers | 16:13 | |
| *** itlinux has joined #openstack-containers | 16:18 | |
| *** shrasool has joined #openstack-containers | 16:19 | |
| *** itlinux has quit IRC | 16:29 | |
| *** itlinux has joined #openstack-containers | 16:36 | |
| *** munimeha1 has quit IRC | 16:48 | |
| *** shrasool has quit IRC | 17:02 | |
| *** ykarel is now known as ykarel|away | 17:03 | |
| *** shrasool has joined #openstack-containers | 17:03 | |
| *** ramishra has quit IRC | 17:06 | |
| *** pcaruana has joined #openstack-containers | 17:12 | |
| *** shrasool has quit IRC | 17:17 | |
| *** ykarel|away has quit IRC | 17:22 | |
| *** munimeha1 has joined #openstack-containers | 17:24 | |
| *** shrasool has joined #openstack-containers | 17:32 | |
| *** jmlowe has quit IRC | 17:38 | |
| *** pcaruana has quit IRC | 17:56 | |
| *** salmankhan1 has joined #openstack-containers | 18:13 | |
| *** salmankhan has quit IRC | 18:14 | |
| *** salmankhan1 is now known as salmankhan | 18:14 | |
| *** shrasool has quit IRC | 18:26 | |
| *** shrasool has joined #openstack-containers | 18:27 | |
| openstackgerrit | melissaml proposed openstack/magnum-tempest-plugin master: Change openstack-dev to openstack-discuss https://review.openstack.org/622539 | 18:32 |
| *** salmankhan has quit IRC | 18:40 | |
| *** jmlowe has joined #openstack-containers | 18:50 | |
| *** itlinux has quit IRC | 19:07 | |
| *** itlinux has joined #openstack-containers | 19:10 | |
| openstackgerrit | Vieri proposed openstack/magnum-ui master: Change openstack-dev to openstack-discuss https://review.openstack.org/622568 | 19:15 |
| openstackgerrit | Vieri proposed openstack/magnum-specs master: Change openstack-dev to openstack-discuss https://review.openstack.org/622569 | 19:17 |
| *** itlinux has quit IRC | 19:23 | |
| *** jmlowe has quit IRC | 19:23 | |
| *** itlinux has joined #openstack-containers | 19:53 | |
| *** shrasool has quit IRC | 19:55 | |
| *** jmlowe has joined #openstack-containers | 20:23 | |
| *** jmlowe has quit IRC | 20:33 | |
| *** shrasool has joined #openstack-containers | 20:55 | |
| strigazi | Anyone here for meeting? | 20:56 |
| colin- | o/ | 21:00 |
| strigazi | #startmeeting containers | 21:01 |
| openstack | Meeting started Tue Dec 4 21:01:06 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:01 |
| *** openstack changes topic to " (Meeting topic: containers)" | 21:01 | |
| openstack | The meeting name has been set to 'containers' | 21:01 |
| strigazi | #topic Roll Call | 21:01 |
| *** openstack changes topic to "Roll Call (Meeting topic: containers)" | 21:01 | |
| strigazi | o/ | 21:01 |
| cbrumm_ | o/ | 21:02 |
| colin- | hello | 21:02 |
| strigazi | hello guys | 21:02 |
| strigazi | #topic Announcements | 21:03 |
| *** openstack changes topic to "Announcements (Meeting topic: containers)" | 21:03 | |
| strigazi | Following CVE-2018-1002105 https://github.com/kubernetes/kubernetes/issues/71411 | 21:03 |
| strigazi | I've pushed imaged for 1.10.11 and 1.11.5 | 21:03 |
| strigazi | #link http://lists.openstack.org/pipermail/openstack-discuss/2018-December/000501.html | 21:04 |
| cbrumm_ | thanks | 21:04 |
| strigazi | And some quick instructions for upgrading. | 21:04 |
| strigazi | After looking into the CVE, it seems that magnum clusters are suffering only from the anonymous-auth=true on the API server issue | 21:05 |
| strigazi | The default config with magnum is not using the kube aggregator API and in kubelet this option is to false. | 21:05 |
| colin- | understood | 21:06 |
| cbrumm_ | that's good, we've enabled the aggregator though :( | 21:06 |
| flwang | o/ | 21:07 |
| strigazi | We need to this anyway, by so far we haven't. | 21:07 |
| flwang | strigazi: does the v1.11.5 include the cloud provider support? | 21:07 |
| strigazi | flwang: v1.11.5-1, yes it does | 21:08 |
| flwang | strigazi: cool, thanks | 21:08 |
| strigazi | FYI, at today at cern I counted kubernetes 141 clusters, we plan to set anonymous-auth=false in the API and then advise users to upgrade manuallu or migrate to new clusters with v1.12.3 | 21:10 |
| cbrumm_ | 141, nice! | 21:11 |
| strigazi | All clusters are inside our private network, so only critical services are advised to take action. | 21:11 |
| strigazi | to be on the safe side | 21:12 |
| *** jmlowe has joined #openstack-containers | 21:12 | |
| strigazi | final comment about the CVE that monopolised by day and last night, | 21:12 |
| strigazi | multi-tenant clusters are also more vulnerable since non owners might run custom code in the cluster. | 21:13 |
| strigazi | #topic Stories/Tasks | 21:14 |
| *** openstack changes topic to "Stories/Tasks (Meeting topic: containers)" | 21:14 | |
| cbrumm_ | luckily we don't have any multi-tenant clusters | 21:14 |
| strigazi | last week, was too busy for me, I don't have any updates, I think I missed an issue with the heat-agent in queens or rocky, flwang ? | 21:15 |
| strigazi | cbrumm_: we all might have with keystone-auth? it is easier to give access | 21:16 |
| flwang | strigazi: for me, i worked on the keystone auth feature | 21:16 |
| flwang | and it's ready for testing | 21:16 |
| cbrumm_ | I haven't looked into that, great question | 21:16 |
| flwang | it works for me | 21:16 |
| flwang | and now i'm working on the client side to see if we can automatically generate the config | 21:17 |
| strigazi | flwang: shall we add one more label for authz? | 21:17 |
| flwang | strigazi: can you remind me the user case to split into authN and authZ? | 21:18 |
| strigazi | flwang: the user might want to manage RBAC only with k8s, with keystone authz you need to add the rules twice, one in the keystone policy one in k8s | 21:19 |
| flwang | but my point is if we need two labels here, because if user just want to manage RBAC with k8s, they can don't update the configmap, and leave what is where is | 21:20 |
| flwang | keep the default one | 21:20 |
| flwang | i'm just hesitate to introduce more labels here | 21:21 |
| strigazi | I'll check again if the policy is too restrictive, in general lgtm, thanks | 21:21 |
| flwang | strigazi: i'm trying to set a very general policy here, but i'm open for any comments | 21:23 |
| flwang | strigazi: me and lxkong are working on https://review.openstack.org/#/c/497144/ | 21:24 |
| strigazi | flwang: I'll leave in comment in gerrit if needed. Looks good as a first iteration, we can take it chnage smth if need it. I'll just need to test from the last time. | 21:24 |
| flwang | strigazi: cool, thanks | 21:24 |
| flwang | the delete resource feature is also an important one | 21:25 |
| strigazi | flwang: I'll review it | 21:25 |
| *** salmankhan has joined #openstack-containers | 21:25 | |
| flwang | now we're getting many of tickets saying can't delete cluster | 21:25 |
| strigazi | there is not hook that does smth yet, correct? | 21:27 |
| flwang | lxkong will submit a patch for LB soon | 21:27 |
| flwang | the current patch is just the framework | 21:27 |
| strigazi | I'm happy to include in this patch or merge the two together | 21:28 |
| lxkong | flwang: strigazi the patch was already there, working on fixing the CI https://review.openstack.org/#/c/620761/ | 21:28 |
| lxkong | i've already tested in the devstack environment | 21:28 |
| lxkong | but need to figure out the ut | 21:28 |
| lxkong | failure | 21:28 |
| strigazi | lxkong: that is the issue in the CI? | 21:28 |
| lxkong | strigazi: just unit test | 21:28 |
| strigazi | ok | 21:29 |
| lxkong | in the real functionality test, and lbs created by the services can be properly removed before the cluster deletion | 21:29 |
| strigazi | ok | 21:30 |
| strigazi | I'll test in devstack, we don't have octavia in our cloud so all my input will come from devstack. | 21:30 |
| lxkong | considering the differetnt k8s version and octavia/neutron-lbaas other people are using, that hook machinism is totally optional, it's up to the deployer to config it or not | 21:30 |
| lxkong | strigazi: yeah, that patch is for octavia | 21:31 |
| strigazi | got it | 21:31 |
| lxkong | strigazi: you also need to patch k8s with https://github.com/kubernetes/cloud-provider-openstack/pull/223 | 21:31 |
| lxkong | which will add the cluster uuid into the lb's description | 21:32 |
| strigazi | lxkong: does this work with the out-of-tree cloud-provider? | 21:32 |
| lxkong | we will include that PR in our magnum images | 21:32 |
| lxkong | strigazi: yeah, sure | 21:32 |
| lxkong | latest CCM already has that fix | 21:32 |
| strigazi | cool | 21:33 |
| strigazi | lxkong: kind of relevant question, when using the CCM you need the cloud config in the worker nodes too? | 21:35 |
| flwang | strigazi: btw, besides the keystone auth, i'm working on the ccm integration | 21:35 |
| lxkong | strigazi: no | 21:35 |
| lxkong | kubelet should have --cloud-provider=external | 21:35 |
| strigazi | only? | 21:35 |
| lxkong | yeah | 21:36 |
| strigazi | cool | 21:36 |
| flwang | lxkong: where does the pod read the cloud config? | 21:36 |
| lxkong | it doesn't talk to cloud stuff any more | 21:36 |
| lxkong | by com | 21:36 |
| flwang | talk to apiserver? | 21:36 |
| lxkong | cm | 21:36 |
| lxkong | configmap | 21:36 |
| lxkong | you need to create a cm with all the cloud config content | 21:37 |
| lxkong | and pass that cm to CCM | 21:37 |
| strigazi | it only makes sense. | 21:37 |
| flwang | ok | 21:37 |
| *** shrasool has quit IRC | 21:38 | |
| cbrumm_ | just make sure that if your cm has cloud credentials that you lock it down policies around accessing it. | 21:38 |
| strigazi | flwang: lxkong cbrumm_ for better security, if the ccm runs as a DS in the master nodes, it can mount the config from the node | 21:39 |
| flwang | lxkong: cbrumm_: does the cloud config cm need to be created manually? or it will be read by something and created on behalf? | 21:39 |
| strigazi | this way the creds are not accessible via any api | 21:39 |
| flwang | strigazi: i'm going to make the ds only running on master | 21:40 |
| strigazi | flwang: you can mount the config from the host then | 21:40 |
| strigazi | *cloud config | 21:40 |
| flwang | yes, but i'm not sure if ccm can still read the cloud config file or only happy with configmap now | 21:41 |
| cbrumm_ | strigazi: we do that too. Just saying that if creds are in cms that they must also be protected. | 21:41 |
| strigazi | cbrumm_: +1 | 21:41 |
| strigazi | flwang: it is the same from the ccm's point of view | 21:41 |
| strigazi | flwang: the pods will see file | 21:41 |
| strigazi | flwang: the pods will see a file that may come from the host or the config map | 21:42 |
| flwang | strigazi: cool, will double check | 21:42 |
| strigazi | flwang: it is better to not put passwords in config maps or even secrets without a KMS | 21:43 |
| flwang | strigazi: ack | 21:43 |
| strigazi | fyi, without the cloud provider that I tested, 1.13.0 works without issues with rocky. | 21:47 |
| flwang | strigazi: why don't test the cloud provider? ;) | 21:49 |
| strigazi | I have one last question for the cloud provider, is the external any better in terms of number of API calls? | 21:50 |
| flwang | strigazi: technically yes | 21:50 |
| flwang | because instead of sending api calls from each kubelet, there is only one caller, the ccm | 21:50 |
| flwang | lxkong: correct me if i'm wrong | 21:50 |
| strigazi | flwang: I tested in our production cloud, we don't have a use case for it there. | 21:50 |
| flwang | strigazi: fair enough | 21:50 |
| * lxkong is reading back the log | 21:51 | |
| strigazi | I'd like to have, but we don't :( not lbaas not cinder, only manila, cvmfs and our internal dns lbaas. | 21:51 |
| flwang | strigazi: right, make sense | 21:52 |
| lxkong | flwang: you are right | 21:53 |
| lxkong | this picture will help you understand better https://paste.pics/fe51956a0c2605edeaf2d42617fe108e | 21:53 |
| strigazi | Anything else for the meeting? | 21:55 |
| cbrumm_ | not today | 21:55 |
| flwang | lxkong: thanks for sharing, good diagram | 21:56 |
| flwang | strigazi: all good for me | 21:56 |
| flwang | strigazi: are you going to skip next meeting? | 21:56 |
| flwang | until the new year? | 21:56 |
| strigazi | no, I can do the next two | 21:57 |
| strigazi | 11 and 18 of Dec | 21:57 |
| cbrumm_ | me and my team will miss the 25th and 1st meetings | 21:57 |
| strigazi | me too | 21:57 |
| strigazi | I'll put it in the wiki | 21:58 |
| flwang | we won't have 25 and 1st meeting anyway :D | 21:59 |
| flwang | cool, i will work next week too | 21:59 |
| flwang | strigazi: thank you | 22:01 |
| *** shrasool has joined #openstack-containers | 22:02 | |
| strigazi | #link https://wiki.openstack.org/wiki/Meetings/Containers#Weekly_Magnum_Team_Meeting | 22:02 |
| strigazi | thanks for joining the meeting everyone | 22:03 |
| strigazi | #endmeeting | 22:03 |
| *** openstack changes topic to "OpenStack Containers Team" | 22:03 | |
| openstack | Meeting ended Tue Dec 4 22:03:52 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:03 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-04-21.01.html | 22:03 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-04-21.01.txt | 22:03 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-12-04-21.01.log.html | 22:03 |
| jakeyip | hi all, late to the party, didn't want to crash your meeting, just want to intro myself. I'm Jake from Nectar Research Cloud (Australia) | 22:04 |
| strigazi | hello jakeyip | 22:05 |
| cbrumm_ | Hi Jake | 22:05 |
| flwang | jakeyip: wave from NZ | 22:06 |
| jakeyip | We've deployed Magnum recently in prod. works well. Many thanks for the hard work you've put into it. | 22:06 |
| cbrumm_ | You're already in prod with users? Nice | 22:07 |
| strigazi | jakeyip: excellent, which release? | 22:07 |
| jakeyip | we are at queens. | 22:07 |
| flwang | jakeyip: you guys have only 1 region, right? | 22:08 |
| jakeyip | yes | 22:09 |
| brtknr | o/ hey all | 22:10 |
| brtknr | ive been following the autoscaling work under cernops... looking good | 22:11 |
| strigazi | brtknr :) | 22:11 |
| jakeyip | we have very limited 'pilot' users. need to work through some operational issues quickly before more people start jumping on | 22:12 |
| brtknr | looks like the meeting time has moved again for me... i thought was starting at 22:00 | 22:13 |
| strigazi | brtknr: winter time | 22:13 |
| strigazi | it is 2100 UTC | 22:14 |
| jakeyip | I was reading past logs, flwang: we are having cluster DELETE_FAIL issue too. I think this is due to LBaaS created by k8s external LB not being deleted before deleting cluster. looks like there's a patch for now | 22:14 |
| strigazi | brtknr: check the link in the wiki https://wiki.openstack.org/wiki/Meetings/Containers for the conversion | 22:15 |
| flwang | jakeyip: there are 2 cases | 22:15 |
| brtknr | strigazi: thanks :) | 22:15 |
| flwang | we can discuss offline if you want | 22:15 |
| flwang | jakeyip: lb of master nodes and lb running on top of k8s | 22:15 |
| strigazi | jakeyip: if this is your current issue you are in a pretty good state. | 22:17 |
| jakeyip | flwang: pretty good state is what I like to hear. :P | 22:18 |
| jakeyip | another thing I've found is that magnum-conductor periodically polls k8s API? I find that it generates lots of logs of ERROR level if someone mistakenly delete the instance. Filled up our logs one time. | 22:19 |
| jakeyip | is this a known issue, or should I file a bug? | 22:20 |
| strigazi | jakeyip: [drivers] send_cluster_metrics = False | 22:20 |
| strigazi | let's say it is false alarm. | 22:20 |
| flwang | jakeyip: just turn off the config mentioned by strigazi | 22:21 |
| jakeyip | thanks strigazi, flwang! | 22:22 |
| flwang | jakeyip: for lb deletion of master nodes, you also need to update you heat | 22:22 |
| flwang | jakeyip: you need this one https://review.openstack.org/#/c/619941/ | 22:22 |
| jakeyip | ok. I haven't got the master nodes lb config working yet though | 22:22 |
| strigazi | flwang: there is an issue with the LBs of the api too? | 22:22 |
| flwang | jakeyip: then for the lb on k8s, just monitor the progress we're doing | 22:22 |
| flwang | strigazi: sometimes, it's very hard to delete | 22:23 |
| flwang | have to delete the lb manually and then retry to delete the cluster | 22:23 |
| flwang | it's a heat problem, not magnum | 22:23 |
| jakeyip | flwang: for lb on k8s is this the right bug? https://review.openstack.org/#/c/620761/ | 22:23 |
| flwang | jakeyip: correct | 22:23 |
| flwang | jakeyip: and this https://review.openstack.org/#/c/620761/ | 22:24 |
| strigazi | flwang: jakeyip If I were you, in our internal docs I would put with the red letters, delete your Lbs first with kubeclt. I think it is a reasonable workaround, of course not ideal. | 22:24 |
| flwang | strigazi: we did, but customer won't read your docs | 22:24 |
| flwang | and when they see issues, they just raise ticket and complain your service | 22:25 |
| jakeyip | strigazi: yeah plan to do that. but depending on users to read docs... :P | 22:25 |
| strigazi | flwang: when the tickets arrives you can point them to the docs, at least you will have a point. | 22:25 |
| jakeyip | I've done some operator tooling to help me hunt down the offending resources to delete | 22:25 |
| strigazi | flwang: jakeyip of course they won't read the docs, at least the first time. | 22:26 |
| jakeyip | flwang: why do you say it's a heat issue? afaik the resources created by external lb in k8s doesn't show up in heat? | 22:27 |
| strigazi | jakeyip: I'm going to bed, flwang knows everything :), for real! If you need anything, feel free to ping me as well. Also, you are more than welcome to join the meeting next week. | 22:30 |
| strigazi | flwang: jakeyip have a nice day | 22:30 |
| jakeyip | sure, have a good night strigazi. thanks! | 22:30 |
| colin- | welcome jakeyip | 22:31 |
| jakeyip | flwang: also, change https://review.openstack.org/#/c/620761/ fixes it in magnum, so a bit confused on why you say it's a heat problem not magnum. | 22:32 |
| jakeyip | hi colin- :) | 22:32 |
| flwang | strigazi: night | 22:32 |
| openstackgerrit | Lingxian Kong proposed openstack/magnum master: Add load balancer hook for cluster pre-deletion https://review.openstack.org/620761 | 22:32 |
| flwang | jakeyip: sometimes, heat may have a race condition issue, or something like that | 22:33 |
| flwang | which cause heat try to delete the network/subnet before the lb is fully deleted | 22:33 |
| flwang | and then it will fail because there is port on that network/subnet | 22:33 |
| *** dave-mccowan has quit IRC | 22:33 | |
| *** rcernin has joined #openstack-containers | 22:34 | |
| jakeyip | but heat doesn't know that there's an lb on that network/subnet because it's not a resource created by heat? | 22:34 |
| flwang | no, i'm talking about the case of lb of master nodes | 22:35 |
| flwang | not lb on k8s | 22:35 |
| jakeyip | ah ok. I'm referring to lb on k8s | 22:35 |
| jakeyip | flwang: sorry sorry | 22:35 |
| jakeyip | flwang: hmm so I'm looking at this patch - https://review.openstack.org/620761 - for lb on k8s. | 22:36 |
| jakeyip | flwang: it seems like you are trying to find the octavian lb to delete it before deleting the cluster. | 22:37 |
| flwang | jakeyip: for lb on k8s, yes | 22:37 |
| jakeyip | flwang: I wonder if you have tried getting k8s to delete everything instead? | 22:38 |
| flwang | jakeyip: that's another option when your k8s api still works | 22:38 |
| flwang | for some cases, if the k8s api is down, you have trouble | 22:38 |
| jakeyip | you are right | 22:39 |
| jakeyip | flwang: this way sounds like a force-delete option. whereas if we can get k8s api to delete what it created it's much cleaner | 22:40 |
| flwang | agree | 22:41 |
| jakeyip | flwang: could be both are necessary | 22:42 |
| jakeyip | flwang: I am also wondering if this will mean more code in magnum; does magnum have to do this for each cloud provider resource? | 22:43 |
| jakeyip | we can bring this discussion offline if you want | 22:44 |
| flwang | that's alright | 22:44 |
| flwang | so far, i think only for resource LB | 22:45 |
| flwang | i don't think we need to do samethng for PV | 22:45 |
| flwang | but the hook can give you flexibility | 22:45 |
| *** itlinux has quit IRC | 22:45 | |
| *** rcernin_ has joined #openstack-containers | 22:45 | |
| *** rcernin has quit IRC | 22:45 | |
| *** shrasool has quit IRC | 22:46 | |
| *** jmlowe has quit IRC | 22:46 | |
| jakeyip | flwang: I agree. can do this as first pass to fix your problem. I can explore the k8s api route in my free time. :) | 22:48 |
| *** jmlowe has joined #openstack-containers | 22:48 | |
| jakeyip | flwang: it doesn't work for me cos it's Octavian :( | 22:48 |
| flwang | jakeyip: oops ;) | 22:48 |
| jakeyip | flwang: all good. we have to migrate anyway; it's just a long and hard road | 22:51 |
| flwang | jakeyip: good luck | 22:52 |
| jakeyip | thanks | 22:52 |
| *** salmankhan has quit IRC | 23:07 | |
| *** rcernin_ has quit IRC | 23:12 | |
| *** etp has quit IRC | 23:12 | |
| *** etp has joined #openstack-containers | 23:13 | |
| *** rcernin has joined #openstack-containers | 23:13 | |
| *** munimeha1 has quit IRC | 23:24 | |
| *** shrasool has joined #openstack-containers | 23:29 | |
| *** shrasool has quit IRC | 23:30 | |
| *** udesale has quit IRC | 23:40 | |
| *** hongbin has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!