Wednesday, 2018-12-12

*** shrasool has quit IRC00:00
openstackgerritMerged openstack/magnum master: functional: add body for delete_namespaced_service in k8s  https://review.openstack.org/62361200:06
openstackgerritMerged openstack/magnum master: functional: bump atomic version to latest  https://review.openstack.org/62362200:06
openstackgerritMerged openstack/magnum master: functional: use default admission_control_list values  https://review.openstack.org/62362800:06
openstackgerritMerged openstack/magnum master: functional: use vexxhost-specific nodes with nested virt  https://review.openstack.org/62360700:06
*** itlinux has joined #openstack-containers00:06
*** munimeha1 has quit IRC00:06
*** itlinux_ has joined #openstack-containers00:25
*** itlinux has quit IRC00:28
*** itlinux_ has quit IRC00:49
*** PagliaccisCloud has quit IRC00:53
*** dave-mccowan has joined #openstack-containers01:14
*** PagliaccisCloud has joined #openstack-containers01:31
*** ricolin has joined #openstack-containers02:02
*** hongbin has joined #openstack-containers02:44
*** ykarel|away has joined #openstack-containers02:58
openstackgerritFeilong Wang proposed openstack/magnum master: Fix grafana port access in prometeus monitoring  https://review.openstack.org/62455803:02
*** ykarel|away has quit IRC03:05
*** dave-mccowan has quit IRC03:52
*** ykarel|away has joined #openstack-containers04:08
*** udesale has joined #openstack-containers04:17
*** hongbin has quit IRC04:55
*** ykarel|away has quit IRC05:09
*** ricolin has quit IRC05:15
*** rtjure has quit IRC05:19
*** itlinux has joined #openstack-containers05:21
*** rtjure has joined #openstack-containers05:22
*** ykarel|away has joined #openstack-containers05:25
openstackgerritLingxian Kong proposed openstack/magnum master: [WIP] Delete Octavia loadbalancers for fedora atomic k8s driver  https://review.openstack.org/49714405:25
*** ykarel|away is now known as ykarel05:26
*** ricolin has joined #openstack-containers05:32
*** zufar has joined #openstack-containers05:39
*** itlinux has quit IRC05:51
*** lpetrut has joined #openstack-containers06:30
*** ramishra has quit IRC06:32
*** ramishra has joined #openstack-containers06:33
*** ricolin has quit IRC07:12
*** openstackgerrit has quit IRC07:29
*** rcernin has quit IRC07:30
*** ivve has quit IRC07:30
*** lpetrut has quit IRC07:46
*** openstackgerrit has joined #openstack-containers08:19
openstackgerritBharat Kunwar proposed openstack/magnum stable/queens: functional: use vexxhost-specific nodes with nested virt  https://review.openstack.org/62460808:20
lxkongstrigazi: hi could you please take brief look at https://review.openstack.org/497144? I wanan make sure i am doing right as discussed before spending time on the UT, thanks08:32
*** salmankhan has joined #openstack-containers09:00
*** ivve has joined #openstack-containers09:04
*** salmankhan has quit IRC09:24
*** salmankhan has joined #openstack-containers09:38
strigazilxkong: looking09:38
strigazilxkong: lgtm, can you add an debug log that it starts the pre_delete hook?09:40
*** lpetrut has joined #openstack-containers09:41
*** lpetrut has quit IRC09:45
openstackgerritSpyros Trigazis proposed openstack/magnum master: functional: stop using concurrency of 1 for api tests  https://review.openstack.org/62391709:45
*** ivve has quit IRC09:52
*** salmankhan has quit IRC10:14
*** salmankhan has joined #openstack-containers10:16
*** salmankhan has quit IRC10:21
*** salmankhan has joined #openstack-containers10:21
lxkongstrigazi: yeah, i've already added when i was testing just now10:59
*** udesale has quit IRC11:02
*** udesale has joined #openstack-containers11:03
*** lpetrut has joined #openstack-containers11:03
openstackgerritLingxian Kong proposed openstack/magnum master: Delete Octavia loadbalancers for fedora atomic k8s driver  https://review.openstack.org/49714411:04
openstackgerritMerged openstack/magnum master: functional: stop using concurrency of 1 for api tests  https://review.openstack.org/62391711:46
*** ttsiouts has joined #openstack-containers11:47
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: [WIP] Add nodegroup representation  https://review.openstack.org/60482311:48
openstackgerritTheodoros Tsioutsias proposed openstack/magnum master: [WIP] API changes for nodegroups  https://review.openstack.org/60482411:48
ttsioutsstrigazi: ^11:49
ttsioutsstrigazi: I'm preparing a new patch with the driver implementation11:49
strigazittsiouts: awesome11:50
ttsiouts:D11:50
*** salmankhan1 has joined #openstack-containers12:02
*** salmankhan has quit IRC12:03
*** salmankhan1 is now known as salmankhan12:03
*** ttsiouts has quit IRC12:07
*** ttsiouts has joined #openstack-containers12:07
*** salmankhan has quit IRC12:11
*** ttsiouts has quit IRC12:12
sayalilunkadhi! Anyone see this error when creating a magnum template? ERROR: No module named google (HTTP 500) (Request-ID: req-77ecd6c4-3fd1-4cc2-9443-69e970227c23)12:23
openstackgerritLingxian Kong proposed openstack/magnum master: Delete Octavia loadbalancers for fedora atomic k8s driver  https://review.openstack.org/49714412:34
*** dave-mccowan has joined #openstack-containers12:41
*** ttsiouts has joined #openstack-containers12:54
*** salmankhan has joined #openstack-containers13:14
*** salmankhan has quit IRC13:35
*** salmankhan has joined #openstack-containers13:35
*** salmankhan has quit IRC13:36
*** salmankhan has joined #openstack-containers13:37
*** roukoswarf has quit IRC14:12
*** irclogbot_0 has quit IRC14:36
mordredflwang: https://github.com/rook/rook/issues/2371 and https://github.com/rook/rook/issues/1921 are potentially worth being aware of and/or looking in to14:40
mordredflwang: tl;dr - using a magnum created k8s cluster using atomic nodes (this is on vexxhost public cloud) - trying to use rook to get a cephfs that can be used via flexvolume in pods14:41
mordredflwang: it almost works - except for when k8s thinks it has mounted the cephfs flexvolume, it really hasn't - and best we can tell it's because the mount is happening in the kubelet container and not propogating to the host14:42
mordredfrom what I can tell from the comments in 1921 and 1930, it seems specifically related to the containerized controlplane14:42
mordredfor now, I'm going to try spinning up a k8s by hand to verify all of the k8s yaml is doing what it's supposed to be doing14:43
*** ykarel has quit IRC14:47
mnasermordred: o/14:50
mordredbut it seems like a things that a) should be sorted out and b) is a potentially excellent end-to-end test case14:50
mordredmnaser: and I was actually just about to ping you about that too :)14:50
mnasermordred: i wonder if because we do a containerized deploy, the `/var/lib/kubelet/volumeplugins` path is actually empty inside the kubelet so it doesn't actually know what to do?14:51
*** irclogbot_0 has joined #openstack-containers14:51
mnasermordred: i think `atomic exec` is a thing.. so what if you did something like `atomic exec kubelet ls /var/lib/kubelet/volumeplugins` ?14:52
*** hongbin has joined #openstack-containers14:52
*** Miouge has quit IRC14:55
mordredmnaser: lemme try14:56
mordredmnaser: (although that said, we did verify that the cephfs is getting mounted into the kubelet container - so I think https://github.com/rook/rook/issues/1921#issuecomment-419518244 is likely to be the issue14:57
*** Miouge has joined #openstack-containers14:57
mnasermordred: lemme look at the second isuse, got distracted reading buffers in the morning :)14:57
mordredmnaser: which is that the flexvolume operation is happening in the container so it's not available on the host14:57
openstackgerritMohammed Naser proposed openstack/magnum stable/rocky: functional: retrieve cluster to get stack_id  https://review.openstack.org/62472314:58
*** ykarel has joined #openstack-containers14:58
mordredmnaser: that said:14:59
mordred# runc exec kubelet ls /var/lib/kubelet/volumeplugins14:59
mordredceph.rook.io~rook14:59
mordredceph.rook.io~rook-ceph-system14:59
mordredrook.io~rook14:59
mordredrook.io~rook-ceph-system14:59
mordredmnaser: (it's runc exec, not atomic exec, just for the record)14:59
*** hongbin has quit IRC15:00
mnasermordred: reading that second issue it looks like that's exactly what it is15:03
mordredyeah15:03
mordredany thoughts on workarounds - even hacky ones?15:03
mordredfwiw - I've also sent an internal rh email to travis, sage and fabian to see if they have any thoughts on workarounds15:04
mordredalthough with kubecon, I'm guessing it's unlikely for them to have much brainspace this week15:05
mordredmnaser: I'm guessing, because it's atomic, that running kubelet on the host without using a container is likely a difficult hack at this point, yeah?15:07
mordred(I say that - it's a static binary, so one could presumably just reach in to the container filesystem and execute the binary directly ... but that seems kind of like open heart surgery with a steak knife15:07
mordredmnaser: also - while I'm bothering you here ... once we get some of this figured out - it might not be a bad idea to submit a docs patch to rook for the flexvolume config section to add a magnum section15:08
mnasermordred: so the reason behind that architectural decision was an effort to allow us to deploy *any* version of k8s15:09
mnaserbut also because fedora atomic's model functions in a way that most everything runs in a container15:10
mordredyah15:10
mnaseri guess we COULD plop down binaries and systemd units and call it a day15:10
openstackgerritBharat Kunwar proposed openstack/magnum stable/rocky: functional: use vexxhost-specific nodes with nested virt  https://review.openstack.org/62472915:10
*** jonaspaulo has joined #openstack-containers15:10
jonaspaulohi all15:10
mordredmnaser: the atomic section just links to the openshift docs, and it's super hard to follow - the neat thing is that just setting FLEXVOLUME_DIR_PATH to /var/lib/kubelet/volumeplugins works because magnum has already done the work to pass the right flags to the kubelet15:10
mordredmnaser: well - no, I think the magnum architecture decision is the right one15:10
mordredand a solution should be found to this issue15:11
jonaspauloi am trying to deploy magnum and i get this error when launching a the cluster on the journalctl of the master node15:11
jonaspaulo runc[2915]: publicURL endpoint for orchestration service in null region not found  runc[2915]: Source [heat] Unavailable.  runc[2915]: /var/lib/os-collect-config/local-data not found. Skipping15:11
jonaspauloany1 has idea what could be wrong?15:11
mnasermordred: im really at a loss on a good next step15:11
mordredmnaser: but - while I'm just poking around for workarounds, do you think just switching the systemd unit to running the kubelet binary directly on the host would work?15:11
mordredmnaser: or would it just cause everythign to blow up?15:12
mnasermordred: is this an environment where i can have my ssh key in there to dig around logs a little bit today?15:12
mordredmnaser: yup. have a link to your key handy?15:12
mnaseri'd say it shouldn't but i have never tried it so i cant speak about it15:12
mordredkk15:12
mnaserhttps://github.com/mnaser.keys15:12
mnaser(i'm also at kubecon but i dont know enough people to know who to talk to about this :])15:13
mordredmnaser: I can send an email intro to some folks15:13
mnasermordred: sure!  be more than happy to chat, im here all day and tomorrow15:13
mnasermordred: also re documenting how to get this done, i think that's super valuable too15:14
mnaserthough in an ideal world, our k8s cluster "Just Work"™15:14
mordred++15:15
mordredmnaser: well - the main thing that needs documenting in the rook docs is that the default value for the rok config option FLEXVOLUME_DIR_PATH needs to be set differently from the defaults on atomic based hosts15:15
mordredso there is a REALLY easy answer that can be put into the docs15:15
mnasermordred: gotcha, well if this affects all flexvolume deploys then we should probably just set it to that path.. all the itme15:16
mordredactually, I Should just send in that pr right now, regardless of whether the thing works at the end of the day15:16
mordredyup15:16
mordredmnaser: and magnum DOES set the right values for the kubelet - so that part is great15:16
mnasersounds good, but yeah im for having that defauklt15:17
mnaserdefault15:17
openstackgerritBharat Kunwar proposed openstack/magnum stable/queens: functional: retrieve cluster to get stack_id  https://review.openstack.org/62473215:17
*** lpetrut has quit IRC15:22
*** zufar has quit IRC15:23
jonaspauloany help m8s?15:28
*** ricolin has joined #openstack-containers15:30
*** zufar has joined #openstack-containers15:35
zufarHi all, i want to install magnum into my openstack, are cinder is required?15:35
mordredmnaser: https://github.com/rook/rook/pull/2374 fwiw15:41
*** salmankhan has quit IRC15:42
*** ricolin has quit IRC15:42
*** ricolin has joined #openstack-containers15:43
jonaspaulozufar:  i dont think so15:48
*** lpetrut has joined #openstack-containers15:48
jonaspaulobut i am in the process of trying to make it work so.. lol15:49
zufarjonaspaulo: Hi thank you, do you have any example magnum configuration files? I want to create the ansible playbook and i need some example for magnum configuration files. its hard to read from official example.15:51
jonaspauloi am using kolla-ansible to deploy it automatically15:53
strigazimordred: mnaser either you need this patch https://github.com/openstack/magnum/commit/1b0fbc207484f4059b96446e1a270730454a49be OR15:55
strigazimordred: mnaser template create --labels kubelet-options="--volume-plugin-dir=/var/lib/kubelet/volumeplugins"15:56
*** lpetrut has quit IRC15:56
zufarjonaspaulo: yes, but i need to create custom ansible script for my client haha15:57
strigazimordred: mnaser template create --labels kubelet_options="--volume-plugin-dir=/var/lib/kubelet/volumeplugins" with _ not -16:01
jonaspaulohm ok :/16:03
*** ykarel is now known as ykarel|away16:10
*** ykarel|away has quit IRC16:19
*** udesale has quit IRC16:19
*** ttsiouts has quit IRC16:29
*** ttsiouts has joined #openstack-containers16:30
*** ttsiouts has quit IRC16:30
*** ttsiouts has joined #openstack-containers16:31
jonaspaulogotta go16:49
jonaspaulothanks anyway16:49
*** itlinux has joined #openstack-containers16:53
*** ricolin has quit IRC16:53
mordredstrigazi: yah - we've got that - kubelet is runnign with --volume-plugin-dir=/var/lib/kubelet/volumeplugins16:54
strigazimordred: /var/lib/kubelet is also rshared16:54
strigazimordred: Maybe rook should mount smth with rshared16:55
strigaziso that changes propagate16:55
mordredstrigazi: so - we're seeing the cephfs mounted in the kubelet container in what looks to be pod-specific storage location:16:56
mordred$ sudo runc exec -t kubelet mount | grep image-store16:56
mordred10.254.14.93:6790,10.254.11.181:6790,10.254.18.2:6790:/ on /var/lib/kubelet/pods/7db0357a-fda0-11e8-aeef-fa163eb20d90/volumes/ceph.rook.io~rook/image-store type ceph (rw,relatime,name=admin,secret=<hidden>,acl,mds_namespace=myfs,wsize=16777216)16:56
mordredstrigazi: and if I write a file to the theoretical mount location in the registry pod, the file shows up in that directory if I look at it on the minion - but each registry pod's files only show up in the mount-point directory on their own minion16:57
mordredstrigazi: so it's like the directory in the kubelet container is mapped into the pod's container - but it's not actually the cephfs even though mount shows type cephfs on that directory16:58
mordredstrigazi: does that make any sense? :)17:00
*** jonaspaulo has quit IRC17:02
*** ykarel has joined #openstack-containers17:03
mordredstrigazi: I just plopped some more debugging in to https://github.com/rook/rook/issues/2371 in case it's clearer in that context17:03
strigazimordred: reading17:03
strigazimordred: these are the rook pods or pods mounting cephfs volumes?17:06
mordredstrigazi: these are the pods mounting the cephfs volumes17:07
mordredstrigazi: basically, from https://rook.io/docs/rook/v0.9/ceph-filesystem.html the "Consume the Shared File System: K8s Registry Sample" section17:07
mordred(we figured we'd start with the example consumption as we'd figure that should work :) )17:08
mordredstrigazi: the ceph cluster and cephfs are created as per https://rook.io/docs/rook/v0.9/ceph-quickstart.html and https://rook.io/docs/rook/v0.9/ceph-filesystem.html  - with the only real change in teh k8s templates being to set FLEXVOLUME_DIR_PATH17:10
strigazioh, the registry pod tries to consume cephfs in the same cluster17:12
mordredyeah17:13
mordredso - I think I just saw something ...17:13
mordredon the minion, /var/lib/kubelet/pods/7db0357a-fda0-11e8-aeef-fa163eb20d90/volumes/ceph.rook.io~rook/image-store exists and the file I wrote to /var/lib/registry inside of the registry pod shows up there17:14
mordredbut on the minion (outside of the kubelet container) that directory is not a ceph mount17:15
strigazimordred: I'll save the link and ask my colleagues in the ceph team. I think they evalueted running rook in magnum clusters but only serving cephfs outside the cluster running rook17:15
mordredso basically this is just me being VERY verbosely back to the original thing - which is the mount of cephfs happened in the kubelet container and didn't propogate back out to the host17:17
strigazimaybe you need:17:17
mordredand so while there is a cephfs properly mounted on the correct dir IN the kubelet container, it's the dir on the host that is getting mounted in the pod17:17
strigazi--feature-gates=MountPropagation=true17:17
mordredooh. for kubelet?17:17
strigaziyes17:17
mordredlemme try. putting that in /etc/kubernetes/kubelet and then restarting the kubelet should pick that up right?17:18
strigaziyes17:19
mordredin fact, from the kubelet systemctl status right now:17:19
mordredDec 11 23:56:19 opendev-tbu64tqms7nw-minion-2.vexxhost.local runc[1673]: serviceaccount SubPath: MountPropagation:<nil>}17:19
strigaziafter using the feature gate?17:20
mordredthat was before17:21
mordredso now trying after having restarted kubelet with that gate17:21
strigazimordred: any luck?17:25
mordredstrigazi: not sure yet - I've got to do some manual cleanup first - things get weird when k8s thinks it mounted something but didn't when it tries to delete that thing :)17:27
mordredstrigazi: k. that didn't seem to help17:32
mordredstill have cephfs mounted on the pod volume dir in the kubelet dir but not on the corresponding dir on the host17:32
*** ttsiouts has quit IRC17:33
*** ttsiouts has joined #openstack-containers17:34
zufarHi, when try to use command `openstack coe service list` i get this error17:34
zufarmagnum-api[22822]: 2018-12-13 00:29:17.326 23411 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to https://127.0.0.1:35357: HTTPSConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f601cf4bc10>: Failed to establish a new connection: [Errno 111]17:34
zufarECONNREFUSED',)): ConnectFailure: Unable to establish connection to https://127.0.0.1:35357: HTTPSConnectionPool(host='127.0.0.1', port=35357): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f601cf4bc10>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',))17:34
zufari think this problem is in my magnum.conf, because I am creating from scratch the configuration. my magnum configuration : http://paste.opensuse.org/5198718217:36
strigazimordred Ι can't think of anything else, I can give it a try tmr, maybe I miss smth. /var/lib/registry shouldn't be changed in the host anyway17:37
strigazizufar: looks like your "vip" is localhost?17:38
*** ttsiouts has quit IRC17:38
* strigazi is leaving17:38
mordredstrigazi: cool. I wrote a quick followup comment on that gh issue - just to clarify17:38
*** ianychoi has joined #openstack-containers17:38
zufarstrigazi: no, my vip is generated well in all section17:41
zufarthank you, I am missing [keystone_authtoken] section to write17:46
*** mgoddard has joined #openstack-containers18:25
*** robertomls has joined #openstack-containers18:48
*** hongbin has joined #openstack-containers18:56
*** ykarel has quit IRC19:10
*** salmankhan has joined #openstack-containers19:20
*** itlinux_ has joined #openstack-containers20:14
*** itlinux has quit IRC20:15
*** salmankhan has quit IRC20:18
*** salmankhan has joined #openstack-containers20:31
*** PagliaccisCloud has quit IRC20:42
openstackgerritLingxian Kong proposed openstack/magnum master: Delete Octavia loadbalancers for fedora atomic k8s driver  https://review.openstack.org/49714420:56
*** rcernin has joined #openstack-containers21:06
*** jmlowe has quit IRC21:10
*** jmlowe has joined #openstack-containers21:11
*** jmlowe has quit IRC21:13
*** jmlowe has joined #openstack-containers21:14
*** PagliaccisCloud has joined #openstack-containers21:24
lxkongmnaser: are you around?21:31
lxkongmnaser: i've tested your patch https://review.openstack.org/#/c/623724, it failed in `configure-etcd` step21:32
lxkongmnaser: because the heat-container-agent can not access the file system, e.g. `ls /dev/disk/by-id`21:32
*** robertomls has quit IRC21:40
*** lpetrut has joined #openstack-containers21:44
*** itlinux_ has quit IRC22:00
*** itlinux has joined #openstack-containers22:00
*** salmankhan has quit IRC22:03
*** PagliaccisCloud has quit IRC22:20
*** mnasiadka_ has joined #openstack-containers22:23
*** mnasiadka has quit IRC22:29
*** andreykurilin has quit IRC22:29
*** logan- has quit IRC22:29
*** mnasiadka_ is now known as mnasiadka22:29
*** logan- has joined #openstack-containers22:31
*** lpetrut has quit IRC22:32
*** itlinux has quit IRC22:35
*** dave-mccowan has quit IRC23:19

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!