Thursday, 2018-06-07

« Wednesday, 2018-06-06 Index Friday, 2018-06-08 »
*** caoyuan has quit IRC00:39
*** felipemonteiro has joined #openstack-helm01:12
*** felipemonteiro_ has joined #openstack-helm01:14
*** felipemonteiro has quit IRC01:18
*** felipemonteiro_ has quit IRC01:24
*** caoyuan has joined #openstack-helm01:33
*** mdih has joined #openstack-helm01:52
*** jgu has quit IRC02:03
*** unicell has quit IRC02:08
*** roman_g has quit IRC02:49
*** zhangfei has joined #openstack-helm03:37
*** yamamoto has joined #openstack-helm04:00
openstackgerritPete Birley proposed openstack/openstack-helm-infra master: WIP: Dex: Chart  https://review.openstack.org/57200304:06
*** mdih has quit IRC04:40
*** p__arch has joined #openstack-helm05:08
*** p_arch has quit IRC05:12
*** mdih has joined #openstack-helm05:22
*** jgu has joined #openstack-helm05:29
*** sar has joined #openstack-helm06:10
*** unicell has joined #openstack-helm06:21
*** unicell has quit IRC07:02
*** gkadam has joined #openstack-helm07:07
*** cNilesh has joined #openstack-helm07:12
*** radek__ has joined #openstack-helm07:14
*** jistr|mtgs is now known as jistr07:22
*** jgu_ has joined #openstack-helm07:23
*** jgu has quit IRC07:26
*** sar has quit IRC07:28
*** jgu_ has quit IRC07:29
*** zhangfei has quit IRC07:51
*** zhangfei has joined #openstack-helm08:02
*** roman_g has joined #openstack-helm08:11
*** MarkBaker has quit IRC09:04
*** MarkBaker has joined #openstack-helm09:05
openstackgerritBertrand Lallau proposed openstack/openstack-helm master: Update developer references 'networking' section  https://review.openstack.org/57314409:11
openstackgerritBertrand Lallau proposed openstack/openstack-helm master: Update developer references 'networking' section  https://review.openstack.org/57314409:30
*** caoyuan_ has joined #openstack-helm09:33
*** caoyuan has quit IRC09:34
*** caoyuan_ has quit IRC10:07
*** MarkBaker has quit IRC10:43
*** julim has joined #openstack-helm10:58
*** caoyuan has joined #openstack-helm10:59
*** yamamoto has quit IRC11:14
*** radek__ has quit IRC11:28
*** MarkBaker has joined #openstack-helm11:29
*** zhangfei has quit IRC11:43
openstackgerritBertrand Lallau proposed openstack/openstack-helm master: Update developer references 'upgrade' section  https://review.openstack.org/57320511:56
*** yamamoto has joined #openstack-helm12:10
*** MarkBaker has quit IRC12:15
*** yamamoto has quit IRC12:21
*** yamamoto has joined #openstack-helm12:49
*** eeiden has joined #openstack-helm13:24
*** sthussey has joined #openstack-helm13:30
*** openstackgerrit has quit IRC13:34
*** zhangfei has joined #openstack-helm13:54
*** openstackgerrit has joined #openstack-helm13:57
openstackgerritSteve Wilkerson proposed openstack/openstack-helm-infra master: WIP: Use Elasticsearch to gather logs in post-run jobs  https://review.openstack.org/57300713:57
*** gkadam has quit IRC14:03
*** gkadam has joined #openstack-helm14:04
*** felipemonteiro has joined #openstack-helm14:15
*** felipemonteiro_ has joined #openstack-helm14:15
*** felipemonteiro has quit IRC14:20
*** radek__ has joined #openstack-helm14:20
*** cfriesen has quit IRC14:36
*** cfriesen has joined #openstack-helm14:36
*** roman_g has quit IRC14:52
*** zhangfei has quit IRC15:12
*** MarkBaker has joined #openstack-helm15:15
*** MarkBaker has quit IRC15:26
*** yamamoto has quit IRC16:14
*** MarkBaker has joined #openstack-helm16:18
*** yamamoto has joined #openstack-helm16:21
*** ericho has joined #openstack-helm16:24
*** yamamoto has quit IRC16:26
*** caoyuan has quit IRC16:31
*** felipemonteiro_ has quit IRC16:42
*** felipemonteiro_ has joined #openstack-helm16:42
*** caoyuan has joined #openstack-helm16:48
*** caoyuan has quit IRC16:53
*** MarkBaker has quit IRC16:54
*** radek__ has quit IRC16:55
*** cNilesh has quit IRC17:10
*** gkadam has quit IRC17:14
*** mdih has quit IRC17:16
*** yamamoto has joined #openstack-helm17:23
openstackgerritSteve Wilkerson proposed openstack/openstack-helm-infra master: WIP: Use Elasticsearch to gather logs in post-run jobs  https://review.openstack.org/57300717:30
*** yamamoto has quit IRC17:31
*** openstackgerrit has quit IRC17:34
*** patrickeast has joined #openstack-helm17:37
*** felipemonteiro__ has joined #openstack-helm17:48
*** unicell has joined #openstack-helm17:49
*** felipemonteiro_ has quit IRC17:52
*** MarkBaker has joined #openstack-helm18:34
*** openstackgerrit has joined #openstack-helm18:38
openstackgerritSteve Wilkerson proposed openstack/openstack-helm master: Ceph-config-helper: Add rgwadmin python package  https://review.openstack.org/57340518:38
openstackgerritSteve Wilkerson proposed openstack/openstack-helm master: Ceph-config-helper: Add rgwadmin python package  https://review.openstack.org/57340518:38
openstackgerritSteve Wilkerson proposed openstack/openstack-helm master: Ceph-config-helper: Add rgwadmin python package  https://review.openstack.org/57340518:43
*** gkadam has joined #openstack-helm18:51
*** ericho has quit IRC18:52
openstackgerritSteve Wilkerson proposed openstack/openstack-helm master: Ceph-config-helper: Add rgwadmin python package  https://review.openstack.org/57340518:53
*** ericho has joined #openstack-helm18:58
openstackgerritChris Wedgwood proposed openstack/openstack-helm-addons master: artifactory: Update to 5.11.1  https://review.openstack.org/57052619:08
*** felipemonteiro__ has quit IRC19:26
*** Talion has joined #openstack-helm19:36
TalionHello good evening.19:37
*** ericho has quit IRC19:38
TalionWhat is the avantage/disadvantage of installing openstack with kolla-kubernetes vs openstack-helm ? Can anyone explain it please? Exprienced will be appreciated.19:39
portdirectTalion: kolla-kubernetes was a great project, with some interesting design features19:41
*** ericho has joined #openstack-helm19:41
portdirectunfortunately it's not maintained anymore https://github.com/openstack/kolla-kubernetes19:41
*** gmmaha has quit IRC19:41
Talioni also notice it but confused when i see it in openstack documentation19:42
Talioncouldnt be sure19:42
portdirectyeah - i think there is some cleanup still ongoing19:42
Talionso installing openstack with kubernetes only possible with openstack-helm atm ?19:42
portdirectits the path I would recommend19:43
TalionOne thing i am curious about its management. Since we are using/adding many tool docker + kubernetes / helm guess also ansible etc....19:44
TalionCould we have hard times after we go production ?19:45
TalionWe are in mind "keep it simple" :)19:46
TalionOur team also considering openstack-ansible. But using containers comes better option for me.19:47
*** ericho has quit IRC19:47
*** ericho has joined #openstack-helm19:48
*** ericho has quit IRC19:52
*** ericho has joined #openstack-helm19:58
*** ericho has quit IRC20:02
*** ericho has joined #openstack-helm20:02
*** ericho has quit IRC20:04
*** ericho has joined #openstack-helm20:04
*** felipemonteiro has joined #openstack-helm20:09
p__archanticw: not sure if you can help with this but worth asking. After finally getting osh running, i rebooted one of the VMs and now ceph is continually going into error state. basicall ceph-mon is catching a signal and aborting. have you seen that before?20:10
p__archportdirect: feel free to chime in too :) basically everything else is breaking because ceph is down :(20:11
*** gkadam has quit IRC20:11
openstackgerritSteve Wilkerson proposed openstack/openstack-helm master: Add Elasticsearch image with s3 repository plugin  https://review.openstack.org/57342520:15
*** eeiden has quit IRC20:20
p__archapparently asserting in AuthMonitor::update_from_paxos() due to "int ret = get_version(keys_ver+1, bl);" returning 020:22
*** felipemonteiro_ has joined #openstack-helm20:22
p__archactually: "int ret = get_version(keys_ver+1, bl);" not returning 020:23
*** felipemonteiro has quit IRC20:25
*** julim has quit IRC20:40
anticwp__arch: i don't know if i've seen that exact error, is it possible the storage for ceph got corrupted/lost?20:42
p__archgreat question. not sure how I would figure that out. unfortunately ceph isn't really my area20:43
anticwi would expect if you are use block-logical it would create and mount an fs on the said devices20:43
anticwso you should check that is the case20:43
anticwp__arch: the mon's crashloop so it's difficult to exec in?20:45
p__archi can get into the pod for a while at least, but don't know what i'm looking for20:47
anticwtbqh i'm not 100% sure either, can you show (1) get pods (2) logs from said crashing pod ... pastebin somewhere perhaps?20:48
p__archsure. give me a sec20:54
p__archhttps://pastebin.com/dcFz42WY20:56
anticwis it possible this is slow to come up and the healthchecks are failing?21:00
anticwdescribe po ...21:00
anticwwould show if that's perhaps the case21:00
p__archall 5 ceph-mon pods are crashing btw. :(21:00
openstackgerritSteve Wilkerson proposed openstack/openstack-helm-infra master: Update start.sh to reflect osh-infra deployment changes  https://review.openstack.org/57344021:00
p__archyup one sec21:00
p__archthe last events are:21:01
p__arch  Normal   Created                46m (x3 over 47m)   kubelet, kube-5.novalocal  Created container21:01
p__arch  Warning  BackOff                2m (x188 over 46m)  kubelet, kube-5.novalocal  Back-off restarting failed container21:01
anticwyeah, i see line 31521:02
anticw/build/ceph-12.2.3/src/mon/AuthMonitor.cc: 157: FAILED assert(ret == 0)21:02
p__archyup, that's the one21:03
p__archget_version is failing apparently21:03
anticwgoogling for that i find the internet is awash with bug reports in that function21:04
p__archlatest code is at https://github.com/ceph/ceph/blob/master/src/mon/AuthMonitor.cc line 23321:05
anticwneed to look at the branch that represents the version you have there21:05
anticwi'm wondering if the auth files are missing/broken21:06
anticwhttps://github.com/ceph/ceph/blob/luminous/src/mon/AuthMonitor.cc#L15721:08
anticwp__arch: if you look inside sdb (from the host should be possible) ... what do you see?21:09
p__archhttps://github.com/ceph/ceph/blob/luminous/src/mon/AuthMonitor.cc21:09
p__archanticw: "look inside sdb" - clarify please?21:10
anticwyou mentioned the other day you were going to use 'sdb' for ceph on your nodes21:11
anticwif that's the case, on a node ... what does sdb have ... and if you mount it (probably already mounted); does it have a plausible on-disk structure for ceph?21:11
p__archnot me. i'm just using the default osh chart for helm21:12
anticwp__arch: it's using storage on the hosts then?21:12
p__archyes21:12
anticwso i think by default you have /var/lib/openstack-helm/ceph/ perhaps?21:12
p__archanticw: correct21:13
anticwinside that?21:14
anticwmaybe tree /var/lib/openstack-helm/ceph/ gives a suitable summary?21:15
*** gmmaha has joined #openstack-helm21:15
anticwsorry, my cluster right now isn't accessible or i would check21:15
p__archhttps://pastebin.com/gAHk03h421:18
*** Talion has quit IRC21:25
anticwp__arch: ok, plausible ... he's a thought ... did you install then reinstall without purging those directories?21:28
*** felipemonteiro_ has quit IRC21:28
*** felipemonteiro_ has joined #openstack-helm21:29
p__archnope - was a fresh set of VMs. I did have to hard-reboot 2 of them though.21:29
p__archanticw: so there's a definite chance of corruption21:29
p__archdo i really have to start again from scratch? grrrrr21:30
anticwp__arch: i rarely see issues with this, when i do i have a script which nukes and reinstalls since it's quick21:32
anticwthat's a terrible answer i'm sorry21:32
p__archnukes and re-installs the k8s cluster? or the openstack/ceph deployments?21:33
anticwjust ceph21:33
anticwk8s is pretty rock-solid for me21:33
anticwceph ... less so21:33
p__archit's possible to redo ceph without breaking the openstack stuff on top of it?21:33
anticwfor openstack testing/development i don't always use ceph21:33
anticwno, openstack puts much stuffs in the db which is on ceph21:34
p__archyeah, that's what i thought.21:34
anticwif you are quite sure it's not fully corrupted then ceph people can perhaps tell you how to fix this21:34
p__archthere's no multi-node guide for using nfs, just aio21:34
anticwbut when it exploded with stack traces ... i'm not a good resource, it takes me way too long to work things out21:34
p__archis nfs a possibility for multi-node or will i be fighting with it to get it running?21:35
anticwp__arch: nfs required you have robust nfs storage, but it's not hard otherwise21:35
anticwyou want to use nfs from inside the cluster or outside?21:35
p__archinside21:35
p__archactually, i'm really not sure21:36
anticwso ... aio will do that for you ... but it's not going to be resillient if that host serving nfs goes down21:36
anticwif you have an external nfs option that would be one approach21:36
anticwlike a netapp or similar21:36
p__archno - this is a development cloud, no good storage solution at all. which is why my deployments take 10+ hours I think21:37
portdirectnfs would be a good choice then21:37
p__archbtw, i tried setting up lvmcache, and almost got there, but the volume group the partition is on doesn't have enough extents remaining to add the ramdisk cache volume so I'm SOL there21:38
anticwagree, nfs often ends up being faster as well21:38
portdirectif you use the nfs provisioner from osh, for it to work you'll need to label a node and pin it to that21:38
anticwp__arch: do you have a machine with enough storage you could use as an nfs-sever?21:39
anticwserver even21:39
anticwsever is a bit dark21:39
p__archyes, i have an extra drive on each compute node that's dark right now. I would want to use just one of those right?21:40
p__archportdirect: thanks21:40
anticwp__arch: so long as it's large enough21:41
p__archyup it's 2 TB21:41
anticwyou could also possibly shuffle disks about so that you can raid/lvm/whatever spare disk21:41
anticwthen from that host export that result via nfs21:41
anticwand use the nfs-provisioner21:41
p__archphysical hosts are in a hosting env. so I'd need to plan that21:42
p__archbut I should be able to mount the second drive on one of the (4) compute hosts and export that via nfs21:42
anticwthat works21:43
p__archokay. i'll give that a go. thanks for your help.21:43
anticwfwiw, i do something a bit like that here most of the time, that is have a storage class on k8s that's nfs ... i then tell osh to use that sc21:44
anticwit also lets me snapshot the pvc's should i need to21:44
anticwPVs21:44
openstackgerritSteve Wilkerson proposed openstack/openstack-helm-infra master: WIP/DNM: Gate fix  https://review.openstack.org/57345222:06
openstackgerritSteve Wilkerson proposed openstack/openstack-helm-infra master: WIP/DNM: Gate fix  https://review.openstack.org/57345222:11
openstackgerritSteve Wilkerson proposed openstack/openstack-helm-infra master: WIP/DNM: Gate fix  https://review.openstack.org/57345222:11
openstackgerritSteve Wilkerson proposed openstack/openstack-helm-infra master: WIP/DNM: Gate fix  https://review.openstack.org/57345222:15
p__archanticw: as an alternative, would it makes sense for me to set up ceph on the underlying hosts to use the second physical disk on each compute node and then connect to that from inside the k8s VMs?22:17
*** felipemonteiro_ has quit IRC22:48
*** felipemonteiro has joined #openstack-helm22:49
anticwp__arch: in some sense that's what the ceph chart is doing ... it's just using containers to keep all the ceph stuffs together23:13
anticwif you can get ceph working robustly outside of k8s that's fine, but i'm not sure it's going to be much easier23:13
*** felipemonteiro_ has joined #openstack-helm23:21
openstackgerritSteve Wilkerson proposed openstack/openstack-helm-infra master: WIP/DNM: Gate fix  https://review.openstack.org/57345223:22
*** felipemonteiro has quit IRC23:22
*** ericho has quit IRC23:38
*** sthussey has quit IRC23:48
« Wednesday, 2018-06-06 Index Friday, 2018-06-08 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!