Friday, 2018-06-15

*** dardelean_ has quit IRC00:15
*** vijaykc4 has joined #openstack-containers00:18
*** yamamoto has joined #openstack-containers00:23
*** yamamoto has quit IRC00:27
openstackgerritMerged openstack/magnum master: k8s_fedora: enable tls in traefik ingress  https://review.openstack.org/57537300:29
*** jmlowe has quit IRC00:40
*** hongbin has joined #openstack-containers00:42
*** vijaykc4 has quit IRC00:44
*** jmlowe has joined #openstack-containers00:45
*** vijaykc4 has joined #openstack-containers00:46
*** dardelean_ has joined #openstack-containers00:48
*** vijaykc4 has quit IRC00:52
*** dardelean_ has quit IRC00:53
*** hongbin has quit IRC00:54
*** Nel1x has joined #openstack-containers01:00
*** ricolin has joined #openstack-containers01:02
*** pengdake has joined #openstack-containers01:20
*** yamamoto has joined #openstack-containers01:24
*** markguz_ has joined #openstack-containers01:25
*** markguz_ has quit IRC01:26
*** markguz_ has joined #openstack-containers01:27
*** yamamoto has quit IRC01:29
*** markguz_ has quit IRC01:31
*** hongbin has joined #openstack-containers01:47
*** imdigitaljim2 has quit IRC01:50
*** Nel1x has quit IRC02:04
openstackgerritFeilong Wang proposed openstack/magnum master: Add health_status and health_status_reason to cluster  https://review.openstack.org/57081802:08
*** Nel1x has joined #openstack-containers02:22
*** yamamoto has joined #openstack-containers02:26
*** yamamoto has quit IRC02:31
*** gsimondon has joined #openstack-containers02:41
*** gsimondon has quit IRC02:45
*** ramishra has joined #openstack-containers03:13
*** yamamoto has joined #openstack-containers03:27
*** yamamoto has quit IRC03:33
*** gsimondon has joined #openstack-containers03:35
*** gsimondon has quit IRC03:39
*** lpetrut has joined #openstack-containers03:46
*** salmankhan has quit IRC03:49
*** pengdake has quit IRC03:58
*** ykarel_ has joined #openstack-containers04:08
*** lpetrut has quit IRC04:22
*** yamamoto has joined #openstack-containers04:29
*** gsimondon has joined #openstack-containers04:29
*** hongbin has quit IRC04:30
*** yamamoto has quit IRC04:33
*** yamamoto has joined #openstack-containers04:33
*** gsimondon has quit IRC04:34
*** pengdake has joined #openstack-containers04:37
*** flwang1 has quit IRC04:48
*** dardelean_ has joined #openstack-containers05:00
*** pengdake has quit IRC05:07
*** janki has joined #openstack-containers05:13
*** pcaruana has quit IRC05:18
*** dardelean_ has quit IRC05:22
*** gsimondon has joined #openstack-containers05:23
*** gsimondon has quit IRC05:28
*** mjura has joined #openstack-containers05:42
*** ykarel__ has joined #openstack-containers05:50
*** ykarel_ has quit IRC05:53
*** pengdake has joined #openstack-containers06:07
*** gsimondon has joined #openstack-containers06:17
*** gsimondon has quit IRC06:22
*** lpetrut has joined #openstack-containers06:22
*** pcaruana has joined #openstack-containers06:44
*** serlex has joined #openstack-containers06:47
*** ykarel_ has joined #openstack-containers06:52
*** ykarel__ has quit IRC06:55
*** zul has quit IRC06:56
*** zul has joined #openstack-containers06:56
*** pengdake has quit IRC07:01
*** gsimondon has joined #openstack-containers07:07
*** ykarel_ is now known as ykarel07:07
*** rcernin has quit IRC07:08
*** ramishra has quit IRC07:13
*** AlexeyAbashkin has joined #openstack-containers07:13
*** ramishra has joined #openstack-containers07:20
openstackgerritwei zhao proposed openstack/magnum master: Do not repeat pull images when container recreate in magnum cluster  https://review.openstack.org/57566307:23
*** mago_ has joined #openstack-containers07:25
*** AlexeyAbashkin has quit IRC07:29
*** AlexeyAbashkin has joined #openstack-containers07:29
*** AlexeyAbashkin has quit IRC07:39
*** AlexeyAbashkin has joined #openstack-containers07:39
*** ktibi has joined #openstack-containers07:48
*** ramishra has quit IRC07:49
*** ramishra has joined #openstack-containers07:51
*** AlexeyAbashkin has quit IRC08:00
*** AlexeyAbashkin has joined #openstack-containers08:01
*** olivenwk has joined #openstack-containers08:18
*** flwang1 has joined #openstack-containers08:18
flwang1strigazi: around? have a moment to discuss the service account key issue on multi masters?08:19
strigaziflwang1: yes08:20
flwang1strigazi: cool08:20
flwang1wait a moment08:20
flwang1let me show you something08:20
*** threestrands has quit IRC08:20
brtknrstrigazi: I also have a question regarding attaching special GPU nodes to an existing Swarm cluster - or creating a cluster with mixed flavour of nodes08:21
flwang1strigazi:  "You must pass a service account private key file to the token controller in the controller-manager by using the --service-account-private-key-file option. The private key will be used to sign generated service account tokens. Similarly, you must pass the corresponding public key to the kube-apiserver using the --service-account-key-file option. The public key will be used to verify the tokens during authentic08:21
strigaziflwang1: yes08:22
strigaziflwang1:  That sounds ok08:22
flwang1strigazi:  and based on this https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md#the-service-account-key-pair from KH, we probably need to sign the key pair by the cluster CA08:22
flwang1so you think we can still use the ca.key as the service account private key?08:23
flwang1btw, I'm 80% sure the sonobuoy failure on multi masters is caused by this issue08:23
strigaziflwang1 probably yes08:23
strigaziflwang1: the token are pretty failing to auth08:24
flwang1yep, for a 3 masters cluster, the failure rate is 66.6%08:24
strigaziflwang1 the ca.key I'm 100% sure it will work08:24
strigaziflwang1: Then, if it is the option or not, I would say no08:25
*** janki has quit IRC08:25
strigaziflwang1: IMO we have the following solutions:08:25
flwang1strigazi: ok, do you have a bug to track it? otherwise, i will use this story https://storyboard.openstack.org/#!/story/2002553 to track this issue08:25
strigaziwe habve08:26
strigaziwe have08:26
strigazithe rate of duplicates bug in launchpad was of the charts, I really couldn't keep up08:26
strigazihttps://storyboard.openstack.org/#!/story/176654608:27
strigaziflwang1: option A is to use the ca.key as a quick fix08:28
olivenwkHello, I remember there were issues to launch kubernetes cluster using coreos hosts on pike, It is working on Magnum queens?08:28
strigazioption B new keypair synced on all masters08:28
strigaziflwang1: it can be achieved in two ways08:29
strigaziflwang1: B1. magnum can create it and pass it in heat params08:29
flwang1but i assume option B will need new db schema08:29
strigaziflwang1: B2. We can put it in etcd and sync across the master nodes08:29
strigaziflwang1: master 0 can generate it and master > 0 can pick it up08:30
strigaziflwang1: B1 doesn't need necessarily a new schema, If it only exists in heat-params08:31
flwang1but to generate the public/private key, we still need the private key of the cluster CA, right?08:31
strigaziflwang1: no08:31
strigaziyou can ask magnum to sign it08:31
flwang1what happened if there is a scale up for B1?08:32
strigazithe key remains in the  heat db and it will be passed as is in the new nodes08:32
flwang1strigazi: true, i mean we still need to get it signed by the cluster CA08:32
flwang1i would prefer to go for B1 since it's a current way we're doing for other scenarios08:33
strigaziflwang1: for option B1 all tooling exists to sign a csr08:33
strigaziolivenwk: no, the issues remain, is there something missing from fedora?08:36
strigazibrtknr: tell me08:36
strigazibrtknr: what question08:36
brtknrstrigazi: is this something you've tried?08:36
strigazibrtknr: no08:36
brtknrclear08:36
flwang1strigazi: SFilatov said the public/private keys doesn't need to signed by the cluster CA, can you confirm?08:37
brtknrstrigazi: what about docker swarm federation?08:37
strigaziflwang1: kelsey hightower disagress08:37
strigaziflwang1: kelsey hightower disagrees08:37
strigaziflwang1: https://github.com/kelseyhightower/kubernetes-the-hard-way/blame/master/docs/04-certificate-authority.md#L37108:37
flwang1(20:22:51) flwang1: strigazi:  and based on this https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md#the-service-account-key-pair from KH, we probably need to sign the key pair by the cluster CA08:38
strigaziflwang1: it needs to08:38
flwang1that's the thing i'm confused a bit08:38
flwang1based on current design(we're starting master 0 first), so maybe we can go for the B208:39
flwang1how do you think?08:39
strigazibrtknr I don't know swarm federaion will be a thing08:40
strigazibrtknr I don't know if swarm federaion will be a thing08:40
strigaziflwang1: We can either one08:40
strigaziflwang1: We can do either one08:41
flwang1do you know if SFilatov is still working on this?08:42
flwang1almost 2 months passed and seems there is no commit yet?08:42
strigaziflwang1: No, I haven't seen him around08:42
flwang1strigazi: mind me picking it up?08:42
flwang1i will leave comments on that story08:43
flwang1and i'm happy to collaborate08:43
strigaziflwang1: go for it, but which option?08:43
flwang1i prefer B208:43
strigazietcd?08:43
flwang1yep, if i can't get it sorted quickly, i will try B108:44
flwang1the last sort is A08:44
flwang1but i don't mind if ask me go for B1 or even A08:44
flwang1if we go for plan A, i will definitely revisit this in next cycle08:45
*** AlexeyAbashkin has quit IRC08:45
*** yamamoto has quit IRC08:45
olivenwkstrigazi, Just the ostree system being too constraining for the Nvidia driver (no dkms). And according to the Atomic team there are huge chances that dkms will never be available. I applied some workarounds which would permit the end user to launch GPU clusters but I don't find that as satisfying as if the OS would permit easy nvidia installation08:45
*** AlexeyAbashkin has joined #openstack-containers08:45
strigaziolivenwk: can you create a story on storyboard for this? We have an intern working in GPUs as well we should sync08:46
strigaziolivenwk: and we won't change from fedora atomic anytime soon. Only when the coreos/RHEL/FedoraAtomic is crystal clear08:47
*** ykarel is now known as ykarel|lunch08:47
olivenwkstrigazi, what is storyboard?08:53
strigazistoryboard.openstack.org08:54
strigazihttps://docs.openstack.org/infra/storyboard/gui/tasks_stories_tags.html08:55
strigaziolivenwk: add a new story here: https://storyboard.openstack.org/#!/project/103208:59
strigaziolivenwk: makes sense? brtknr  is also intested and we (CERN) are interested09:01
*** AlexeyAbashkin has quit IRC09:03
*** AlexeyAbashkin has joined #openstack-containers09:03
*** yamamoto has joined #openstack-containers09:04
*** AlexeyAbashkin has quit IRC09:12
*** AlexeyAbashkin has joined #openstack-containers09:13
brtknrstrigazi: I'm thinking that I might create GPU nodes running Centos and manually add them to the swarm cluster09:15
brtknrolivenwk: ^09:15
strigazibrtknr: what is more in centos than fedora?09:16
*** janki has joined #openstack-containers09:16
brtknrstrigazi: Ah, we have Centos images with nvidia drivers preinstalled so makes things easier09:18
brtknrstrigazi: specially given the situation that it would not be currently possible to attach different flavour of nodes to the cluster using magnum09:19
brtknrunless masters were GPU nodes and workers were non-GPU nodes09:20
strigazior vice versa09:20
brtknryes, but we have many non-gpu nodes09:20
brtknrand at the moment, only 2 gpu nodes09:20
strigazibrtknr: you build centos with dib?09:21
strigazibrtknr: centos-atomic doesn't work?09:21
brtknri havent tried09:21
strigazibrtknr: what do you have in the image?\09:21
brtknryes i think it was built using dib, @mgoddaard did i think09:22
*** AlexeyAbashkin has quit IRC09:22
*** AlexeyAbashkin has joined #openstack-containers09:22
brtknrhttps://github.com/stackhpc/stackhpc-image-elements/tree/master/elements/nvidia-cuda09:24
*** ykarel|lunch is now known as ykarel09:25
brtknractually it was @oneswig who wrote the element09:25
brtknrit doesnt support atomic at the moment09:26
*** mvpnitesh has joined #openstack-containers09:27
strigazibrtknr: what is usually in this list? DIB_NVIDIA_CUDA_PKGLIST09:27
brtknrstrigazi: have you attempted installing nvidia drivers inside fedora atomic?09:27
strigazibrtknr: just cuda?09:27
strigazibrtknr: No but is seems I will09:27
strigazibrtknr: No but it seems I will09:27
brtknrDIB_NVIDIA_CUDA_PKGLIST=cuda09:27
strigaziI think if you don't have to change the kernel, it shoukd be possible09:28
strigaziI think if you don't have to change the kernel, it should be possible09:28
olivenwkstrigazi, yes of course I will publish it on the storyboard in the afternoon if I have manage to find time.09:36
olivenwkbrtknr, I did, it's not trivial and induces to override the read-only lock. Some others stuffs has been performed to make everything working with kubernetes as well09:37
olivenwkby now I could test it successfully on magnum pike and will be able to test fedora atomic 27/28 GPU on monday09:38
olivenwkon queens09:38
olivenwkAs soon as I wrote the story I link it to you09:38
brtknrstrigazi olivenwk, this solution for coreos installs the drivers inside a container: https://github.com/src-d/coreos-nvidia09:42
olivenwkbrtknr, yes that targets coreos, if we could do the same for fedora it would be great09:44
strigaziwe can09:45
olivenwkhonestly i didn't have time to invest more time on this topic09:45
brtknrolivenwk: strigazi: this issue on github explains issues for fa2609:45
brtknrhttps://github.com/NVIDIA/nvidia-docker/issues/64809:45
strigazifa26, so last year xD09:45
olivenwkhehe yes i posted that09:45
olivenwkolivier-dj is me x)09:45
brtknrolivenwk: haha great, small world09:46
strigaziconvergence man :) one nick to rule them all, but different passwords ;)09:46
olivenwki just don't know how fedora will behave for the provision of a kernel driver09:46
brtknrim going to create a fa27 gpu node and try replicating the issue today09:47
olivenwkbrtknr this has been solved09:55
olivenwkdon't remember exactly but i solved it nor disabling selinux or with special mounting in the system container09:56
brtknrolivenwk: sudo docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi?09:57
brtknrolivenwk: apparently it requires docker-CE09:57
brtknrhttps://github.com/NVIDIA/nvidia-docker/issues/63409:57
strigazibrtknr: olivenwk https://storyboard.openstack.org/#!/story/200257610:02
strigazibrtknr: olivenwk please add what you have tried already, what works, what doesn't, what you need. Just dump content so others don't need to go what you went through10:04
brtknrstrigazi: will do10:04
*** rcernin has joined #openstack-containers10:07
*** AlexeyAbashkin has quit IRC10:24
olivenwkstrigazi brtknr, sorry (lunch time), yes I will publish all of my advancement and what I see for the future improvement. My current set up allows me to launch GPU enabled pods everything looks functional (for f26/pike, testing it on f27/queens monday). Yes i had to exchange the docker version with a system container because kubernetes needs 17.03 for the ce edition, nvidia-docker needs ce for fedora, and fedora 26 supports only docker-ce10:42
olivenwk17.06+ and 17.09+ for fedora 27. )10:42
olivenwkI will have a meeting soon but I try to find time to publish this afterwards, otherwise in the worst case it would be done on monday10:45
*** AlexeyAbashkin has joined #openstack-containers10:57
*** serlex has quit IRC11:03
*** yolanda has quit IRC11:03
*** yolanda has joined #openstack-containers11:06
*** olivenwk has quit IRC11:06
*** serlex has joined #openstack-containers11:13
brtknrlooks like my nvidia/cuda image does not have nvidia-smi11:13
*** yamamoto has quit IRC11:13
*** olivenwk has joined #openstack-containers11:22
*** AlexeyAbashkin has quit IRC11:27
*** AlexeyAbashkin has joined #openstack-containers11:28
*** ricolin has quit IRC11:47
olivenwkstrigazi, do I write directly in the description?11:48
strigaziolivenwk, no, better add comment(s)11:48
olivenwkor in a comment11:48
olivenwkok11:48
*** yamamoto has joined #openstack-containers11:49
*** Nel1x has quit IRC11:55
*** mvpnitesh has quit IRC12:10
*** mjura has quit IRC12:17
*** chhagarw has joined #openstack-containers12:37
*** chhavi__ has joined #openstack-containers12:38
*** AlexeyAbashkin has quit IRC12:38
*** ykarel has quit IRC12:39
*** ykarel has joined #openstack-containers12:39
*** AlexeyAbashkin has joined #openstack-containers12:40
*** cbrumm__ has joined #openstack-containers12:45
*** yamamoto has quit IRC12:46
*** cbrumm_ has quit IRC12:49
*** AlexeyAbashkin has quit IRC12:50
*** AlexeyAbashkin has joined #openstack-containers12:50
*** yamamoto has joined #openstack-containers13:00
*** rcernin has quit IRC13:02
*** yamamoto has quit IRC13:04
*** yamamoto has joined #openstack-containers13:15
*** dave-mcc_ has joined #openstack-containers13:16
*** jmlowe has quit IRC13:18
*** yamamoto has quit IRC13:20
*** ykarel is now known as ykarel|away13:31
*** dave-mcc_ has quit IRC13:39
*** ykarel|away has quit IRC13:40
*** yamamoto has joined #openstack-containers13:45
*** yamamoto has quit IRC13:49
*** jmlowe has joined #openstack-containers13:57
*** yamamoto has joined #openstack-containers14:00
*** markguz_ has joined #openstack-containers14:02
olivenwkhttps://storyboard.openstack.org/#!/story/200257614:04
olivenwkstrigazi, brtknr14:04
*** yamamoto has quit IRC14:04
*** olivenwk has quit IRC14:05
strigazithanks olivenwk, that is pretty nice14:06
*** AlexeyAbashkin has quit IRC14:06
brtknrq14:07
*** spiette has quit IRC14:09
*** spiette has joined #openstack-containers14:11
*** yamamoto has joined #openstack-containers14:15
*** dave-mccowan has joined #openstack-containers14:16
*** yamamoto has quit IRC14:20
*** yamamoto has joined #openstack-containers14:30
*** yamamoto has quit IRC14:34
*** serlex has quit IRC14:36
*** hongbin has joined #openstack-containers14:41
*** ramishra has quit IRC14:41
*** olivenwk has joined #openstack-containers14:43
*** yamamoto has joined #openstack-containers14:45
*** gsimondon has quit IRC14:49
*** yamamoto has quit IRC14:49
*** ricolin has joined #openstack-containers14:53
*** ricolin has quit IRC14:57
*** AlexeyAbashkin has joined #openstack-containers14:58
*** chhavi__ has quit IRC14:59
*** yamamoto has joined #openstack-containers15:00
*** pcaruana has quit IRC15:01
*** canori01 has joined #openstack-containers15:04
*** yamamoto has quit IRC15:04
*** imdigitaljim has quit IRC15:13
*** yamamoto has joined #openstack-containers15:30
*** lpetrut has quit IRC15:30
*** yamamoto has quit IRC15:34
*** yamamoto has joined #openstack-containers15:45
*** armaan has joined #openstack-containers15:48
*** yamamoto has quit IRC15:50
*** ktibi has quit IRC15:51
*** markguz_ has quit IRC15:56
*** olivenwk has quit IRC15:58
*** imdigitaljim has joined #openstack-containers16:00
*** yamamoto has joined #openstack-containers16:00
*** yamamoto has quit IRC16:04
*** ykarel|away has joined #openstack-containers16:05
*** dave-mccowan has quit IRC16:06
imdigitaljimflwang1: I can also confirm the unauthorized problem with multimasters, leading to invalid tokens/unauthorized failures, making the control plane not working. I cannot get the fedora multimasters operational at all and the right certs in the right places seem to be the problem.16:15
*** yamamoto has joined #openstack-containers16:15
imdigitaljimill also look into the  B1. magnum can create it and pass it in heat params16:15
imdigitaljimive done "B2. We can put it in etcd and sync across the master nodes" in the past16:16
imdigitaljimand it worked seemingly well but I'm not sure the implication of new masters joining/recoverying16:16
imdigitaljimim might throw up some WIP code for this too16:19
*** yamamoto has quit IRC16:20
*** jmlowe has quit IRC16:23
*** armaan has quit IRC16:23
*** yamamoto has joined #openstack-containers16:30
*** jmlowe has joined #openstack-containers16:32
*** yamamoto has quit IRC16:35
*** ykarel|away has quit IRC16:43
*** yamamoto has joined #openstack-containers16:45
imdigitaljimany https://storyboard.openstack.org/#!/story/1766546 specifically16:45
*** yamamoto has quit IRC16:49
*** yamamoto has joined #openstack-containers16:55
*** yamamoto has quit IRC16:55
*** livelace has joined #openstack-containers17:15
*** chhagarw has quit IRC17:33
*** dave-mccowan has joined #openstack-containers17:33
*** janki has quit IRC17:34
*** ricolin has joined #openstack-containers17:36
*** pcichy has quit IRC17:46
*** markguz_ has joined #openstack-containers18:07
*** ricolin has quit IRC18:10
flwang1imdigitaljim: thanks for the confirmation18:52
*** kaiokmo has quit IRC18:55
*** livelace has quit IRC19:20
*** pcichy has joined #openstack-containers19:43
*** markguz_ has quit IRC20:11
*** markguz_ has joined #openstack-containers20:12
*** markguz_ has quit IRC20:22
*** flwang1 has quit IRC20:55
*** vijaykc4 has joined #openstack-containers20:56
*** markguz has quit IRC21:00
*** AlexeyAbashkin has quit IRC21:13
*** vijaykc4 has quit IRC21:25
*** vijaykc4 has joined #openstack-containers21:35
*** vijaykc4 has quit IRC21:41
*** vijaykc4 has joined #openstack-containers21:42
*** vijaykc4 has quit IRC21:47
*** vijaykc4 has joined #openstack-containers21:49
*** yamamoto has joined #openstack-containers21:58
*** yamamoto has quit IRC22:04
*** vijaykc4 has quit IRC22:17
*** hongbin has quit IRC22:43
*** dave-mccowan has quit IRC22:48
*** yasemin has quit IRC22:57
*** yamamoto has joined #openstack-containers23:00
*** yamamoto has quit IRC23:06
*** markguz has joined #openstack-containers23:22
*** markguz has quit IRC23:27

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!