Thursday, 2016-08-11

*** rwsu has quit IRC00:08
openstackgerritMerged openstack/diskimage-builder: secure_path in sudoers: deal with possible quotes  https://review.openstack.org/35058200:21
*** abehl has quit IRC00:24
*** weshay has quit IRC00:36
*** limao has joined #tripleo00:39
*** Guest7941 is now known as saneax_AFK00:39
slagleEmilienM: i yamld again00:39
*** trozet has quit IRC00:54
*** rhallisey has quit IRC01:00
*** trozet has joined #tripleo01:06
*** penick has quit IRC01:07
*** ebalduf has quit IRC01:19
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: Expand templated bash to allow for doc generation  https://review.openstack.org/34759201:24
*** chlong|mtg has joined #tripleo01:56
*** jdob has quit IRC02:03
*** bfournie has joined #tripleo02:04
*** maeca1 has joined #tripleo02:12
*** jcoufal has joined #tripleo02:17
*** tzumainn has quit IRC02:21
dtrainortrying to debug a keystone issue here.  /etc/keystone/keystone.conf is now supposed to reference /etc/keystone/keystone-paste.ini for pasteconfig, correct?  trying to figure out what role /usr/share/keystone now plays, if any, in the runtime configuration of keystone under apache's wsgi02:24
dtrainorline 200 of major_upgrade_controller_pacemarker_1.sh provides the only real clue that I've seen so far02:25
dtrainorkeystone_spec.rb has an override_params value of paste_config to still be /usr/share/keystone/keystone-paste.ini02:27
*** morazi has quit IRC02:29
dtrainorI'm trying to configure keystone for cors and it's as if none of my changes are taking effect, that's why i ask02:34
*** penick has joined #tripleo02:35
*** penick_ has joined #tripleo02:40
*** penick has quit IRC02:41
*** penick_ is now known as penick02:41
*** jeckersb is now known as jeckersb_gone02:52
*** jcoufal has quit IRC02:55
*** yamahata has quit IRC02:58
*** krotscheck_ has joined #tripleo03:12
*** krotscheck has quit IRC03:13
*** krotscheck_ is now known as krotscheck03:14
*** julim has joined #tripleo03:26
*** limao has quit IRC03:26
*** trozet has quit IRC03:32
*** links has joined #tripleo03:37
*** limao has joined #tripleo03:38
*** dsariel has quit IRC03:41
*** dsariel has joined #tripleo03:43
*** sshnaidm|afk has quit IRC03:47
*** beagles has quit IRC03:47
*** wfoster has quit IRC03:47
*** jpeeler has quit IRC03:47
*** jtomasek has quit IRC03:48
*** bana_k has joined #tripleo03:49
*** wfoster has joined #tripleo03:49
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-quickstart: Add undercloud reboot  https://review.openstack.org/35048903:51
*** jeckersb_gone has quit IRC03:51
*** coolsvap has joined #tripleo03:51
*** b3nt_pin has joined #tripleo03:52
*** b3nt_pin is now known as Guest5456303:52
*** jpeeler has joined #tripleo03:53
*** jpeeler has quit IRC03:53
*** jpeeler has joined #tripleo03:54
*** jeckersb_gone has joined #tripleo03:55
*** fragatina has quit IRC03:57
*** sshnaidm|afk has joined #tripleo03:58
*** fragatina has joined #tripleo03:58
*** fragatina has quit IRC03:58
*** jtomasek has joined #tripleo03:59
*** dsariel has quit IRC04:08
*** rlandy has quit IRC04:13
*** dsariel has joined #tripleo04:22
*** julim has quit IRC04:27
*** maeca1 has quit IRC04:34
*** radeks has joined #tripleo04:46
*** dsariel has quit IRC04:49
*** numan_ has joined #tripleo04:56
*** masco has joined #tripleo04:56
*** bana_k has quit IRC05:03
*** skramaja has joined #tripleo05:19
*** jaosorior has joined #tripleo05:23
*** oshvartz has quit IRC05:23
*** fultonj has quit IRC05:25
*** rcernin has joined #tripleo05:29
*** dsariel has joined #tripleo05:30
jaosoriormarios: How's it going?05:33
jaosoriormarios: Got anything in need for review?05:33
*** liverpooler has joined #tripleo05:42
*** david-lyle_ has joined #tripleo05:43
*** liverpooler is now known as yrabl|WFH05:43
openstackgerritMerged openstack/tripleo-heat-templates: Remove Compute role parameters from overcloud.yaml  https://review.openstack.org/34792605:43
*** yrabl|WFH is now known as liverpooler05:43
*** liverpooler has quit IRC05:44
*** liverpooler has joined #tripleo05:44
*** beekneemech has joined #tripleo05:45
*** ramishra_ has joined #tripleo05:45
*** r2bit__ has joined #tripleo05:46
*** r-mibu_ has joined #tripleo05:46
*** StevenK_ has joined #tripleo05:46
*** ramishra has quit IRC05:48
*** david-lyle has quit IRC05:48
*** r2bit has quit IRC05:48
*** r-mibu has quit IRC05:48
*** yolanda has quit IRC05:48
*** karthiks has quit IRC05:48
*** bnemec has quit IRC05:48
*** StevenK has quit IRC05:48
*** yolanda has joined #tripleo05:49
mariossup jaosorior05:49
*** liverpoo1er has joined #tripleo05:49
*** karthiks has joined #tripleo05:50
*** liverpooler has quit IRC05:50
jaosoriormarios: Chilling, having a review round and getting coffee. How's it going over there?05:50
*** liverpooler has joined #tripleo05:51
jaosoriormarios: Anyway, so if you need some reviews, I can check stuff out. Currently I'm just going through the list in no particular order.05:57
*** cmyster has quit IRC05:59
mariosjaosorior: thanks man will do, theres a bunch of stuff i still need to revisit/rebase after long context switch - want to try and poke at the manila netap at some point06:00
jaosoriormarios: Talking about manila06:02
*** jprovazn has joined #tripleo06:02
jaosoriormarios: manila::keystone::auth assumes two versions of manila will always be deployed06:02
jaosoriorso with the switch to puppet handling the keystone endpoints, it creates an extra endpoint pointing to localhost06:02
jaosoriorso I ended up adding an endpoint for it in t-h-t to address that https://review.openstack.org/#/c/352353/06:03
jaosoriorif you can take a look, it's not a big CR06:03
jaosoriorI'm not sure if I needed to capitalize those v1 though06:04
mariosjaosorior: ack will look06:04
jaosoriormarios: Nevermind, I needed to capitalize them06:04
jaosoriorjust checked06:04
jaosoriorlol, sorry dude06:04
mariosjaosorior: np man added to list anyway06:04
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add hieradata for Manila v1 and v2 endpoints  https://review.openstack.org/35235306:07
jaosoriormarios: If you have time, there is a really similar patch addressing cinder v3 https://review.openstack.org/#/c/353364/306:07
jaosoriorthat one I did capitalize06:07
*** bana_k has joined #tripleo06:20
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Convert EndpointMap to not require per-service VIP parameters  https://review.openstack.org/34290006:23
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Convert ServiceNetMap to a nested template  https://review.openstack.org/35303206:23
*** florianf has joined #tripleo06:24
*** oshvartz has joined #tripleo06:30
*** dbecker has joined #tripleo06:41
*** pgadiya has joined #tripleo06:44
*** pcaruana has joined #tripleo06:46
*** dbecker has quit IRC06:46
*** tesseract- has joined #tripleo06:53
*** saneax_AFK is now known as saneax06:55
*** cmyster has joined #tripleo06:55
*** saneax is now known as Guest2003206:55
*** bana_k has quit IRC06:57
*** liverpoo1er has quit IRC06:58
*** limao has quit IRC07:00
*** Goneri has joined #tripleo07:01
*** limao has joined #tripleo07:01
*** ioggstream has joined #tripleo07:03
*** panda|zZ is now known as panda07:04
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: HAProxy should start before ironic and inspector  https://review.openstack.org/35386507:04
*** mcornea has joined #tripleo07:06
*** limao has quit IRC07:06
*** shardy has joined #tripleo07:10
*** milan has joined #tripleo07:11
jaosoriorshardy: Hey dude, just so you know, I rebased a couple of your patches07:11
shardyjaosorior: ack, thanks07:11
jaosoriorthis https://review.openstack.org/#/c/342900/ (also fixed the missing replacement for the nobrackets SQL)07:12
jaosoriorand this https://review.openstack.org/#/c/353032/307:12
jaosorior* no brackets mysql endpoint07:12
shardyjaosorior: thanks for that, I meant to do it yesterday but ran out of time07:12
jaosoriorshardy: Also, I think I found the issue that I had wrongly thought was a heat issue.07:12
shardyaha, what was it?07:12
jaosoriorin the nonha case07:13
jaosoriorhaproxy is started sometimes after some openstack services07:13
jaosoriorso, for instance, ironic inspector tries to ping ironic07:13
jaosoriorand it can't07:13
jaosoriorso one of the two (haven't figured out which) ends up in an error state because of that07:14
*** jprovazn has quit IRC07:14
jaosoriorthat's why those test runs always failed when trying to provision the nodes07:14
jaosoriorand somehow that ended up being reflected further in the overcloud deploy07:14
shardyjaosorior: interesting, so we can probably fix that with some puppet orchestration in the undercloud manifest?07:14
jaosoriorshardy: this is my attempt to do so https://review.openstack.org/35386507:14
*** jpena|off is now known as jpena07:15
jaosoriorbut yeah, it's quite an intermittent issue in the nonha case07:16
shardyjaosorior: ack, thanks for tracking this down, lets see what CI thinks but the patch lgtm07:16
jaosoriorcause of puppet's random ordering07:16
shardyit'd also be good to discuss with dtantsur|afk if this is an ironic/inspector bug (e.g should it retry instead of fail)07:16
shardyperhaps you can raise a bug with details so that can be discussed?07:16
jaosoriorshardy: checking the logs, it seems it retries around 7 times07:17
shardyah, perhaps that's configurable07:17
shardybut +1 on the puppet orchestration regardless07:17
jaosoriorshardy: Do we know of any cause of those: NodeLocked: Node a7dc9450-7a4c-4746-b000-827e50a45985 is locked by host localhost.localdomain, please retry after the the current operation is completed.07:20
*** akuznetsov has joined #tripleo07:22
*** ifarkas_afk is now known as ifarkas07:23
*** ebarrera has joined #tripleo07:28
*** mbound has joined #tripleo07:31
jaosoriorshardy: Hey dude, could you check this out? https://review.openstack.org/#/c/353655/ it's a commit bumping up the overcloud nodes' memory a bit07:32
*** ohamada has joined #tripleo07:33
mariosjaosorior: there goes my 32 gb minimum virt box setup07:33
mariosjaosorior: and that is my "good" box, the other one has 24 gb but it does nothing in its life cos 24gb07:34
*** aufi has joined #tripleo07:35
jaosoriormarios: FWIW it's for the ovb jobs07:35
mariosjaosorior: yah was just looking at that. was going to comment about why 5.5 not 6 but i saw it wasn't for instack-undercloud07:36
jaosoriormarios: I have a 32gb box and it works pretty good for a basic virtual HA setup07:41
mariosjaosorior: yeah me too i was thinking it was suddenly not enough on master07:42
mariosjaosorior: though, 32gb is just enough for 3control 1 compute 5gb nodes with a slightly bigger undercloud node07:43
*** jpich has joined #tripleo07:43
jaosorioryeah07:43
jaosoriorit's not optimal :(07:43
jaosoriorwish I could get a 64GB machine07:43
mariosjaosorior: yeah exactly i have heard rumours hewbrocca-afk was handing some out behind the bike shed yesterday but it could all be lies. srsly though there have been rumblings about a new round of h/w for devs so ask whoever approves those things for you07:45
*** ramishra_ has quit IRC07:45
mariosjaosorior: at least i've heard of beaker boxes getting bumped to 64 gb anyway so hoping mine will at some point too carlos was looking into that07:46
jaosoriornice!07:46
*** chlong|mtg has quit IRC07:46
jaosoriormarios: I actually have a physical machine bellow my desk :P07:46
*** ramishra has joined #tripleo07:46
jaosoriormight need to upgrade it myself07:46
jaosoriorused to have a node in beaker. But we don't have as much resources (different org)07:47
mariosjaosorior: yeah my main box is actually a machine here with 32gb ram but would be great to have another beaker box too.07:47
mariosjaosorior: like setup for >1 version cos upgrades testing/bz etc07:47
jaosorioryeah, every time I need to test a different version I need to whipe out my dev setup07:48
mariosjaosorior: indeed, been meaning to open this box up cos not sure what is happening with the ram slots... i suspect 4x807:48
*** ramishra has quit IRC07:50
*** ramishra has joined #tripleo07:50
*** jaosorior is now known as jaosorior_brb07:53
*** jprovazn has joined #tripleo07:56
*** limao has joined #tripleo07:57
*** akuznetsov has quit IRC07:58
*** jtomasek_ has joined #tripleo08:02
*** gfidente has joined #tripleo08:04
*** florianf has quit IRC08:05
*** TSCHAK has quit IRC08:07
*** florianf has joined #tripleo08:10
*** hewbrocca-afk is now known as hewbrocca08:11
*** amoralej|off is now known as amoralej08:12
hewbroccaMorning folks08:14
hewbroccaYes, ram is raining from the heavens08:14
shadower\o/08:15
mariosonto the golden flocks of roaming boxen08:16
*** lucas-dinner is now known as lucasagomes08:16
pandaif it is this kind of ram https://upload.wikimedia.org/wikipedia/commons/8/80/Bighorn_ram_animal_ovis_canadensis.jpg  we may be in trouble08:16
*** dbecker has joined #tripleo08:17
hewbroccaLOL08:19
*** dtantsur|afk is now known as dtantsur08:29
*** paramite has joined #tripleo08:30
*** dtantsur is now known as dtantsur|bbl08:32
openstackgerritMartin André proposed openstack/tripleo-common: Allow running validation against different plans  https://review.openstack.org/31819408:32
openstackgerritMartin André proposed openstack/tripleo-common: Disable retry files for ansible validations  https://review.openstack.org/32903908:32
openstackgerritMartin André proposed openstack/tripleo-common: Workflow to setup validations  https://review.openstack.org/31363208:32
openstackgerritMartin André proposed openstack/tripleo-common: Workflows to load validations  https://review.openstack.org/35389808:32
openstackgerritMartin André proposed openstack/tripleo-common: Workflows to run validations  https://review.openstack.org/35389908:32
*** electrofelix has joined #tripleo08:41
*** coolsvap is now known as _coolsvap_08:41
*** mbound has quit IRC08:43
*** jaosorior_brb is now known as jaosorior08:44
jaosoriorSimple commit removing a parameter that's no longer used and passing CI https://review.openstack.org/#/c/352907/08:45
jaosoriorBy the way, it appears that liberty is broken :/08:46
*** iogg has joined #tripleo08:48
ccamachojaosorior quick question im getting several OOM errors in CI for the Mistral service, the OOM fix re sahara that you pushed "SaharaWorkers = 0" worked?08:48
*** limao_ has joined #tripleo08:48
*** ioggstream has quit IRC08:48
jaosoriorccamacho: no08:48
ccamacho;(08:48
jaosoriorccamacho: Seems that sahara already had a default of 008:48
shardyFor most services workers=0 means use the number of CPUs08:48
shardyI would say try workers=108:48
shardywe already do that for heat and some other services in CI08:48
jaosoriorshardy: sahara's documentation claims that 0 is one thread for all sahara08:49
shardyyay for openstack consistency08:49
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Set Sahara workers to 1  https://review.openstack.org/35272908:50
*** aqkhan__ has quit IRC08:50
ccamachoIm adding the workers param to puppet-mistral and then to the submission and also to CI to see if this helps08:50
jaosoriorsahrdy, ccamacho: restored the commit, there it is ^^08:50
jaosoriorccamacho: does mistral take workers as a parameter? I thought it didn't08:50
*** aqkhan has joined #tripleo08:50
*** limao has quit IRC08:51
jaosorior* shardy08:51
*** aqkhan_ has joined #tripleo08:51
ccamachojaosorior not in puppet-mistral but is a parameter supported by mistral API, so first im adding it to puppet-mistral, then the THT and finally to CI08:51
ccamachojaosorior https://github.com/openstack/mistral/blob/dd195650792129b2f41c87c5160f6ff904ca31ca/mistral/api/service.py#L2908:51
jaosorioralright08:52
jaosoriorccamacho: Good idea!08:52
ccamachohopefully this helps a little bit there08:52
shardyccamacho: you could test it with the generic config interface without changing puppet-mistral08:53
shardyhttps://github.com/openstack/puppet-mistral/blob/master/manifests/config.pp08:53
shardyby passing that in ControllerExtraConfig08:53
shardyhttps://github.com/openstack-infra/tripleo-ci/blob/master/test-environments/worker-config-mitaka-and-below.yaml#L508:53
shardylike that but s/heat/mistral08:54
jaosoriorgfidente: hey dude, could you check this out? https://review.openstack.org/#/c/353364/08:54
ccamachoshardy =-O cool that at least will show me if it works08:54
gfidentejaosorior, morning08:54
shardyccamacho: Yeah, it's another approach if you don't want to wait on the puppet-mistral patch08:54
jaosoriorgfidente: morning, dude. How's it going?08:54
ccamachoshardy I'll test it then :P thanks!08:55
*** aqkhan has quit IRC08:55
shardyjaosorior: re https://review.openstack.org/#/c/353582/ I can't add a Depends-On or it will be circular08:55
shardyas I already commented08:55
shardysee https://review.openstack.org/#/c/353637/08:55
shardyI'd suggest we just land them together?08:56
jaosoriorI see08:56
jaosoriorshardy08:56
jaosoriorsure, lets do that08:56
jaosoriorshardy: done08:57
shardyjaosorior: thanks!08:57
Jokke_shardy: I think the worker value behavior really depends if that part of the project was implemented before or after Nova changed the behavior09:00
jaosoriorgfidente: thanks for checking it out dude09:00
*** bkopilov has joined #tripleo09:00
*** bkopilov has joined #tripleo09:01
*** iogg has quit IRC09:01
shardyJokke_: Yeah, but to be fair oslo.service (and before that processutils in oslo-incubator) have been around for years09:03
shardythere's been plenty of time for per-project interfaces to align with the patterns there09:03
shardyI'm sure we'll get more consistent eventually09:04
jaosoriorthen again, it's openstack09:04
Jokke_shardy: that, but again it's not that often people are willing to break established ops facing patterns just because someone merged different behavior in oslo09:04
shardyJokke_: I guess that's my point, options are deprecated all the time, and a really good reason to take that pain is to establish consistent conventions09:05
Jokke_like we deploy glance on single worker 64 threads just because 0 ment never "cpu count" for glance09:05
shardysome projects aren't willing to do it, which I accept but find frustrating09:05
*** abehl has joined #tripleo09:05
shardyin the long term, it's harmful to everyone involved with deploying openstack09:05
shardyJokke_: sure, so we have to stick with the status-quo which is all operators need to know the historical quirks of every single service09:06
shardythere are ways it could be done in a backwards compatible way, e.g a new common config section for all projects or something09:07
Jokke_shardy: I think the biggest problem there is that none of those are actually results of crossproject discussions. Someone implements A into nova dozen others copy it from there, someone else changes it in nova and next guy goes and implements it in oslo purely based on Nova behavior09:07
shardyJokke_: Yeah, I know, sad isn't it09:07
shardy:(09:08
jaosorioradding an endpoint for manila ready for merging https://review.openstack.org/#/c/352353/409:08
Jokke_shardy: yup and even more frustrating09:08
jaosoriorshardy, Jokke_ wasn't that what the cross-project meetings were supposed to solve?09:08
*** akrivoka has joined #tripleo09:08
jaosoriorwell, issues like that09:08
jaosoriorI think it'll get better09:09
shardygfidente: Hey, if you have time please can you revisit https://review.openstack.org/#/c/342900/ (and the subseqent patch in the series :)09:10
Jokke_jaosorior: problem with cross project meetings seems to be that they attract that exactly same group of people ;)09:11
shardygah09:11
jaosoriorshardy: I actually think we're gonna need to re-fresh that commit09:11
jaosoriorwith the merging of the cinder endpoint that was missing09:11
jaosoriorthe endpoint_map will change09:11
shardyjaosorior: yeah one of us is going to get hit with a rebase09:11
Jokke_jaosorior: and AFAIK that meeting got discontinued due to the fact that it was all bikeshedding and nothing got done :P09:11
shardyOk I guess lets land the manila one then I'll rebase it again09:12
jaosoriorshardy: I can do the rebase, it's not a big deal09:12
shardyanyone have anything else endpoint related we need to land before I rebase this?09:12
jaosoriorshardy: not that I know. But the commits providing a new service will need to be rebased too (mistral and barbican, that I know of) but those are easy09:13
jaosoriorshardy, lets land manila https://review.openstack.org/#/c/352353/ and I'll push the rebase for your patches09:13
gfidenteshardy, ack09:14
socialEmilienM: I already separated it to different class - it's https://review.openstack.org/#/c/350526 and https://review.openstack.org/#/c/33488109:14
socialEmilienM: only issue is that I thought it's mostly ironic inspector dependancy as that is taking care of putting pxe configs in place09:14
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: TEST DO NOT MERGE - Adding api_workers = 1 for composable mistral  https://review.openstack.org/35392209:15
shardyjaosorior: ack, done, thanks09:15
shardythat EndpointMap patch was always going to be tough to land but I'd like to try to get it done soon if possible09:16
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Do not merge, checking composable mistral.  https://review.openstack.org/35294709:17
jaosoriorshardy: I know the feeling. I'm really eager to get those patches for puppet managing the services/endpoints merged. Already have a special celebration beer and everything bought for when that merges09:18
jaosorior:P09:19
hewbroccaNot to repeat myself, but09:19
hewbroccaos-cloud-config.die.die.die09:19
hewbroccamight as well09:19
shardylol09:19
* shardy needs to get some special custom-roles beer in ;)09:19
mariosi totally missed this special beer thing for manila :/09:20
gfidenteshardy, thanks for host_nobrackets :)09:20
* d0ugal needs to get some Mistral deploy beers09:20
d0ugalactually09:20
d0ugals/beers/whisky/09:21
shardyd0ugal: Hey, good idea :)  What's the status of the tripleoclient deploy patch?09:22
shardyready for it to be pulled, tested, reviewed?09:22
* shardy hasn't tried if for a few days09:22
*** bootsha has joined #tripleo09:23
openstackgerritPaul Belanger proposed openstack/python-tripleoclient: Initial support for bindep  https://review.openstack.org/35392909:26
*** bootsha has quit IRC09:27
jaosoriorshardy, d0ugal by the way, what if the endpoint_map.yaml would be generated from a mistral workflow? That way we wouldn't need to maintain that file like we do now09:27
jaosoriorand adding endpoints to tripleo would be way easier09:27
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Add cinder-backup profiles  https://review.openstack.org/31121809:27
jaosorior* slightly easier09:27
shardyjaosorior: Yeah we could do but it would make things a bit more opaque for template authors09:28
*** psanchez has joined #tripleo09:29
Jokke_anyone here familiar enough with puppet that could have a look on my manila cephfs manifest and point me to right direction with it?09:30
Jokke_https://review.openstack.org/#/c/353928/109:30
d0ugalshardy: I am currently trying to figure out the best way to store the list of Heat environments. Just adding it to the Mistral env breaks the UI I think.09:31
d0ugalshardy: and I some rough path munging code that I need to create an obscure heat env to test with09:32
hewbroccasocial: ^^^ is familiar with puppet Jokke_09:32
shardyd0ugal: Hmm, I assumed it would be added to the mistral envioronment09:32
d0ugalshardy: Yeah, but I don't think that is enough for the UI09:32
shardyit will be very confusing if the UI stores the list there and the CLI stores it somewhere else09:32
d0ugalshardy: Indeed, they will both store it there - but I don't think it is that simple. jtomasek said something but I forget what it was09:33
shardyd0ugal: Ok, well if you want someone to try and break it I'm happy to try ;)09:33
* shardy has lots of obscure env stuff to try 09:33
d0ugalshardy: Do you have one you could send me?09:33
d0ugalshardy: I'm 99% sure it will break, so might be easier for me to itterate on it09:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Add constraints needed for ironic and inspector  https://review.openstack.org/35386509:34
socialJokke_: I don't think enabled_share_protocols belongs to cephfs backend09:36
socialJokke_: and you are missing spec test for the class09:36
socialJokke_: I'll comment on the review09:36
*** apetrich has joined #tripleo09:37
Jokke_social: thanks. Yes that spec testing is one thing I had no idea about. I was trying to look into them, but just couldn't figure it out what I'm supposed to be doing there :(09:38
*** _coolsvap_ is now known as coolsvap09:38
openstackgerritMerged openstack/tripleo-quickstart: Run dstat on the virthost and the undercloud  https://review.openstack.org/34959109:40
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validation to verify connectivity to overcloud  https://review.openstack.org/35393409:40
shardyd0ugal: http://paste.openstack.org/show/554103/ is a starting point09:40
shardyIf those all work the basics are probably OK09:41
shardyalso need to test combining multiple environments which are conflicting09:41
shardyto ensure the CLI ordering is persisted into the environment_files list09:41
d0ugaloh interesting, I wouldn't have thought of all of those. I'll let you know how I get on.09:42
shardyd0ugal: sounds good, thanks!09:43
shardyd0ugal: we need to think about that password generation stuff too, but that can be done after the basic flow is fixed09:43
d0ugalshardy: Yeah, at the moment I am leaving that in the CLI.09:44
shardyd0ugal: Yeah, I think we'll need to fix the bug I referenced, then move it to mistral afterwards09:45
shardyd0ugal: FYI I also had a discussion in the heat meeting, where we agreed I can probably remove the restriction about having one parameter_group per parameter09:45
shardywhich means we can probably add all *Password parameters to a special "generate_password" group09:46
d0ugalshardy: oh, that would be useful.09:46
shardyto identify those which need creating without the hard-coded list09:46
openstackgerritMartin André proposed openstack/instack-undercloud: Introduce 'enable_validations' option  https://review.openstack.org/32289309:47
d0ugalccccccfngfccikhvdktgnejrlndrnjhrhjltkibddrgb09:54
d0ugalgah09:54
* d0ugal needs to move his yubikey09:54
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validation to check ulimits on controller  https://review.openstack.org/35394309:54
pino|work(cat-on-keyboard?)09:54
shardygah, we made the multinode job voting in the gate pipeline didn't we09:58
shardywhich probably means https://review.openstack.org/#/c/353582/ will fail validation09:58
openstackgerritGiulio Fidente proposed openstack/tripleo-puppet-elements: Add thias/puppet-selinux module  https://review.openstack.org/35394509:59
gfidentesocial, ^^10:00
gfidenteis there a problem for opm?10:00
gfidenteI think puppet-ceph will have to set dircontext ...10:00
*** tosky has joined #tripleo10:01
*** bootsha has joined #tripleo10:01
Jokke_social: where am I supposed to hook into that enabled_share_protocols if I move it away from the backend manifest? Just trying to figure out the basic logic of these10:02
socialgfidente: not much, create new distgit and poke rdo people with it. but what we need it for?10:02
gfidentesocial, https://review.openstack.org/#/c/353673/1/manifests/osd.pp10:03
gfidentecheck the comments10:03
*** bootsha has quit IRC10:04
*** dtantsur|bbl is now known as dtantsur10:04
socialuuuh I need to read bit more about this10:04
gfidenteselinux::dircontext is going to run the same exec10:05
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate the IP range for the control plane  https://review.openstack.org/35395010:05
gfidentebut hey ... what can I tell10:05
socialwell we used to have rule to use defaults so it should land in location which already has ceph_var_lib_t and if user wants something else he should set it up sooner10:06
gfidenteright, we're in the scenario where 'user should set it up'10:06
* shardy shakes fist at gate jobs10:06
socialgfidente: this is ugly even with selinux module, we are just hiding the exec :)10:07
gfidentesocial, right selinux is doing the exec anyway10:07
gfidentebut at least if we manage it in ceph module we can test ! -b $data :)10:08
gfidentegoal seems good to me though, we just want the right selinux context applied to the data dir given10:08
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375010:08
socialgfidente: if we go down this path we should start setting up selinux context for ports on which we listen for keystone, nova...10:09
socialgfidente: eg we should use selinux module everywhere :)10:09
socialgfidente: which might help us getting rid of openstack-selinux package but I'm not sure if puppet upstream would like this10:10
gfidenteok I see what you mean10:10
*** sshnaidm|afk is now known as sshnaidm10:10
gfidenteso there are two problems10:10
gfidenteosd.pp expects you to give a path, not sure we can just default it to /var/lib/ceph10:11
gfidenteso it's slightly different than default binding ports10:11
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate haproxy configuration values  https://review.openstack.org/35395310:11
gfidenteand then there is the tripleo specific issue, we used to go in /srv/data10:11
socialbtw if you set selinux contexts before even creating path puppet will use the correct one and it'll work nicely so it is definetly matter for user to set up if he wants to override10:12
gfidenteso we need that dir to be labeled correctly, we can't "migrate" users10:12
socialyes10:12
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Align node_ips hiera keys with the service name.  https://review.openstack.org/35363710:12
socialgfidente: another one for openstack-selinux package :(10:12
gfidentesocial, yeah I was trying to avoid that, but maybe we can walk both paths in parallel10:12
openstackgerritSteven Hardy proposed openstack/puppet-tripleo: Align hiera keys with service names  https://review.openstack.org/35358210:12
socialgfidente: move the exex and selinux_type to THT10:15
gfidentesocial, huh how?10:15
*** bootsha has joined #tripleo10:17
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate the instackenv format  https://review.openstack.org/35395610:17
jaosoriorshardy: what had merged that ended up in a merge conflict in those commits?10:18
shardyjaosorior: nothing, we made the multinode jobs gating, so it's impossible to merge anything with red CI now10:18
jaosoriorI see10:19
jaosoriorshardy: should I workflow them again or do I wait for CI?10:21
shardyjaosorior: I guess we wait for CI again10:22
shardyit can't merge without CI passing anyway10:22
jaosorioralright10:23
jaosoriorwhen it passes CI I'll +A it10:23
*** bootsha has quit IRC10:23
*** bootsha has joined #tripleo10:24
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate the open-files-limit value in mysql  https://review.openstack.org/35396110:24
*** mgould|afk is now known as mgould10:26
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Move common includes to init.pp  https://review.openstack.org/35396210:26
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Include ::tripleo which imports the common manifests  https://review.openstack.org/35396410:29
*** bootsha has quit IRC10:31
*** fragatina has joined #tripleo10:31
sshnaidmdoes anybody deal with os_vif module problem in case?10:33
*** fragatina has quit IRC10:33
*** Goneri has quit IRC10:33
*** mbound has joined #tripleo10:33
*** fragatina has joined #tripleo10:34
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate ntp is running and synchronised  https://review.openstack.org/35396710:34
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate pacemaker status  https://review.openstack.org/35396910:38
*** mbound has quit IRC10:38
openstackgerritMerged openstack/tripleo-validations: Validation to verify undercloud disk size reqs  https://review.openstack.org/35122410:39
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate rabbitmq limits  https://review.openstack.org/35397010:40
gfidentesocial, those policies are executed at the time the package is installed and as such available from the first moment the node is booting right?10:41
*** Guest54563 is now known as beagles10:41
*** dbecker has quit IRC10:41
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Validate rogue DHCP servers  https://review.openstack.org/35397310:43
openstackgerritMerged openstack/tripleo-heat-templates: Add support for cinder v3 endpoint  https://review.openstack.org/35336410:43
yolandaquestion... i'm hitting an error with a baremetal deployment just on liberty, that complains about external network conflicting with another network10:44
yolandai've been checking and seems that neutron needs a flag allow_overlapping_ips set to true10:44
yolandawhat shall be the best way to set it? i tried on my env file, on parameter defaults, with ExtraConfig: neutron::allow_overlapping_ips: true, but no luck10:45
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Check for differences between introspected HW  https://review.openstack.org/35397810:47
*** Goneri has joined #tripleo10:47
jaosoriorshardy: Your patches for the endpoint map already went into merge conflict. I'm waiting for the manila patch to merge to push a rebase of them10:49
*** fragatin_ has joined #tripleo10:50
*** xuao has joined #tripleo10:52
*** paramite is now known as paramite|afk10:52
*** athomas has quit IRC10:53
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Align node_ips hiera keys with the service name.  https://review.openstack.org/35363710:53
*** fragatina has quit IRC10:54
*** athomas has joined #tripleo11:01
*** kullmar has joined #tripleo11:02
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services  https://review.openstack.org/32343611:03
*** xuao has quit IRC11:06
*** bkopilov has quit IRC11:11
*** kullmar has quit IRC11:19
openstackgerritSteven Hardy proposed openstack/puppet-tripleo: Add ironic inspector as a terminated HAProxy endpoint  https://review.openstack.org/34518311:19
jaosoriorshardy: thanks for updating the commit message there11:20
shardyjaosorior: np, when it passes CI I'll approve it11:20
jaosoriorgreat11:20
*** iogg has joined #tripleo11:23
openstackgerritMerged openstack/tripleo-heat-templates: Add hieradata for Manila v1 and v2 endpoints  https://review.openstack.org/35235311:24
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic ha test  https://review.openstack.org/34694911:25
*** jaosorior has quit IRC11:29
*** jaosorior has joined #tripleo11:30
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Allow to parameterize the number of kernels to be kept  https://review.openstack.org/35399411:30
openstackgerritmathieu bultel proposed openstack/tripleo-common: Do only a strick grep on nova show status  https://review.openstack.org/35399611:31
sshnaidmFYI, https://bugs.launchpad.net/tripleo/+bug/1612178 , handled by amoralej11:32
openstackLaunchpad bug 1612178 in tripleo "Nova compute fails because of "No module named os_vif"" [Critical,In progress]11:32
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Remove the incorrect warning caused by the boot_option local dict  https://review.openstack.org/35400111:33
shardysshnaidm: is there a patch fixing the spec file posted we can follow?11:34
sshnaidmshardy, not yet, I'll post it into bug when it will be ready11:34
shardysshnaidm: Ok, thanks!11:34
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: NO MERGE Test periodic  https://review.openstack.org/35400211:35
*** sshnaidm has quit IRC11:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Convert EndpointMap to not require per-service VIP parameters  https://review.openstack.org/34290011:36
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Convert ServiceNetMap to a nested template  https://review.openstack.org/35303211:36
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375011:36
shardyAnyone know why puppet fails to run like this: http://paste.openstack.org/show/554119/11:40
shardyError: Could not run: Could not retrieve facts for overcloud-controller-0.openstacklocal11:40
shardyAFAICS the facts are available via facter, but clearly I'm missing something11:41
slagleshardy: probably more hostname shenanigans11:41
gfidenteyeah needs a mapping in /etc/hosts11:42
slagleshardy: try export FACTER_fqdn=overcloud-controller-0 before the puppet apply11:42
slagleif that works, you know it's hostname related11:42
*** chlong|mtg has joined #tripleo11:43
shardyslagle, gfidente: Ah, yeah facter fqdn doesn't match hostname -f, something is broken in the hosts file11:44
shardythanks!11:44
gfidentethough it says no address so looks like it's the ip > name mapping missing11:45
shardyyeah the domain is missing from the hosts entry, not sure why, will debug11:45
slagleyou also need to check searchdomain in resolv.conf11:46
slagleas puppet might be pulling domain from there11:46
shardyslagle: Yeah that has search openstacklocal, but openstacklocal is missing from the hosts entry11:46
shardynot sure why that happens but I'm testing some local patches so will have to bisect and see what broke it11:47
shardythanks for the pointers :)11:47
*** cmyster has quit IRC11:48
*** dbecker has joined #tripleo11:49
openstackgerritJustin Kilpatrick proposed openstack/tripleo-quickstart: Add documentation for add-on roles  https://review.openstack.org/34673311:50
*** lucasagomes is now known as lucas-hungry11:52
pandapabelanger: do you know if there's any spell check non voting upstream test ? It's useful when reviewing documentation11:53
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate the deploy command to use the Mistral workflows  https://review.openstack.org/34226111:53
*** xuao has joined #tripleo11:54
d0ugalshardy: ^ Close, but still not there. The path munging is messy. I added a comment about the bit missing if you want to take a look. I gotta run now (see your emails if you've not yet)11:56
*** xuao has quit IRC11:56
*** xuao has joined #tripleo11:56
*** milan has quit IRC11:56
amoralejshardy, sshnaidm, i need to create a new package so first thing to do is to send a package review bz, then add the new package to rdo and finally add the dependency to nova spec11:58
shardyd0ugal: ack, thanks!11:58
*** alee has quit IRC11:58
*** rhallisey has joined #tripleo11:59
*** jpena is now known as jpena|lunch12:00
*** milan has joined #tripleo12:03
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Update Ironic composable services  https://review.openstack.org/34143312:11
*** weshay has joined #tripleo12:12
*** pradk has joined #tripleo12:16
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Add manila-netapp backend to manila class  https://review.openstack.org/35401412:16
*** xuao has quit IRC12:16
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud  https://review.openstack.org/35401612:17
*** xuao has joined #tripleo12:18
*** sshnaidm has joined #tripleo12:20
*** limao_ has quit IRC12:20
*** xuao has quit IRC12:20
*** xuao has joined #tripleo12:21
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add integration with NetApp Manila driver  https://review.openstack.org/35401912:21
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services  https://review.openstack.org/32343612:21
mariosrhefner: bswartz tbarron ^^^ i poposed a new review, bswartz was right in the end there wasn't much left in terms of files. there was a bit of copy/paste with the params, but it wasn't worth the rebase pain. I've proposed a new review https://review.openstack.org/354019 and the puppet-tripleo side in the depends on.12:22
*** jayg|g0n3 is now known as jayg12:23
mariosrhefner: are you ok with abandoning https://review.openstack.org/#/c/188138/40 in this case? i added you as co-author in the new one12:23
mariostbarron: hope that helps as a start regardless of how we proceed... i also want to revisit the manila generic puppet-tripleo side tomorrow since that needs a tidy up - we are duplicating config unnecessarily between tripleo-heat-templates and puppet-tripleo. but we don't need to block manila netap because of that.12:25
*** iogg has quit IRC12:26
*** paramite|afk is now known as paramite12:26
openstackgerritMerged openstack/tripleo-quickstart: remove hard setting of ssh.config.ansible in roles-deploy  https://review.openstack.org/35299012:26
*** dprince has joined #tripleo12:27
*** xuao has quit IRC12:27
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Do not merge, checking composable mistral.  https://review.openstack.org/35294712:27
*** chem has quit IRC12:28
*** radeks has quit IRC12:29
*** chem has joined #tripleo12:29
openstackgerritMerged openstack/os-net-config: Enable os-net-config to support and configure NFVSwitch  https://review.openstack.org/34559912:31
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add integration with NetApp Manila driver  https://review.openstack.org/35401912:32
openstackgerritMerged openstack/puppet-tripleo: Fix comparison to control_virtual_ip  https://review.openstack.org/30592812:33
*** jeckersb_gone is now known as jeckersb12:34
*** amoralej is now known as amoralej|lunch12:35
*** ohamada has quit IRC12:35
*** ohamada has joined #tripleo12:36
*** iogg has joined #tripleo12:37
rhefnermarios: yep, I'll abandon the old change now12:38
*** [1]cdearborn has joined #tripleo12:39
*** jdob has joined #tripleo12:39
mariosrhefner: appreciate thanks12:39
rhefnermarios: thank you for working on this!12:40
tbarronmarios: thanks, we'll take a look!12:40
honzafor tripleo-heat-templates, which environment files are commonly passed to "stack create" or is there one that imports everything?12:41
honzaah, the root environment file i guess :)12:47
mariosrhefner: tbarron ack thanks very much, please add yourselves as reviewers if you like I will revisit tomorrow for updates discussion etc. esp Q @ https://review.openstack.org/#/c/354019/212:47
*** rlandy has joined #tripleo12:47
jaosoriorpabelanger: Hey, I saw your commit about adding bindep to python-tripleo. Can you explain how that works? We might need that for a barbican job12:47
*** Ryjedo_ has joined #tripleo12:48
*** egafford has joined #tripleo12:48
*** Ryjedo has quit IRC12:48
*** Ryjedo_ is now known as Ryjedo12:48
tbarronmarios: sure thinng and thanks again12:48
shardyhonza: overcloud-resource-registry-puppet.yaml is always passed, most other environment files are optional12:54
honzashardy: thanks12:55
*** lucas-hungry is now known as lucasagomes12:55
mariostbarron: np, in the end wasn't as painful as expected... imo it shows the composable services is working very well. on the tht side, we have the env file for the netapp params, and then all the changes are in the manila-base file12:56
*** fultonj has joined #tripleo12:58
tbarronmarios: yeah, i'm thinking we want to move the ManilaGeneric parameters out of manila-base to a manila-generic env12:58
*** amoralej|lunch is now known as amoralej12:58
tbarronmarios: related to your Q in the review12:58
mariostbarron: ack thanks will look and fixup but context switch for now12:59
tbarronmarios: but we need to hash this out with bswartz (NetApp) and Jokke_ (cephfsnative plugin)12:59
mariostbarron: right OK, so give folks a chance to see it too12:59
tbarronmarios: yeah, we need to talk through the approach withh everyone first anyways12:59
mariostbarron: ok, today was just about getting a first pass out so we have something to actually point at.13:00
tbarronmarios: yep, perfect!13:01
*** lblanchard has joined #tripleo13:01
*** jpena|lunch is now known as jpena13:02
*** limao has joined #tripleo13:04
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Removing WARNING: line has more than 140 characters in puppet-tripleo profiles  https://review.openstack.org/35405413:06
*** morazi has joined #tripleo13:10
*** rodrigods has quit IRC13:10
*** rodrigods has joined #tripleo13:10
*** bootsha has joined #tripleo13:11
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Update Ironic composable services  https://review.openstack.org/34143313:11
*** cdearborn has joined #tripleo13:14
weshaysshnaidm, panda can one of you guys put in a review to update the tripleo periodic jobs to run every four hours vs. nightly13:15
rookEmilienM: can we get this pulled in? or get some comments on what needs to change? https://review.openstack.org/#/c/350219/13:16
yolandahi... so i'm doing a baremetal deployment on liberty. I see that the setting allow_overlapping_ips is hardcoded to false on neutron.conf, while on mitaka and master is just commented. Is there any way i can customize that with tripleo on liberty, to be set as true?13:16
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud  https://review.openstack.org/35401613:17
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Move Nova settings out of puppet/compute.yaml  https://review.openstack.org/35374813:17
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Mv Ceilometer settings out of puppet/compute.yaml  https://review.openstack.org/35406313:17
*** [1]cdearborn has quit IRC13:18
*** beagles is now known as beagles_brb13:18
jaosorioryolanda: I think you can do it by setting the appropriate hieradata in the controllerExtraConfig parameter, which you can set by creating an environment file and adding that to parameter_Defaults13:20
jaosorior*parameter_defaults13:20
jaosorioryolanda: however, you would need to investigate if that parameter setting is set, first, somewhere in tripleo-heat-templates; second, somewhere in puppet-neutron (both in the liberty branches)13:21
yolandajaosorior, i tried something like13:22
yolanda  ExtraConfig:13:22
yolanda    neutron::allow_overlapping_ips: true13:22
yolandabut it doesn't seem to be applied13:22
*** pgadiya has quit IRC13:22
*** julim has joined #tripleo13:22
*** jcoufal has joined #tripleo13:23
openstackgerritLucas Alvares Gomes proposed openstack/instack-undercloud: Ironic: Remove dnsmasq-ironic.conf  https://review.openstack.org/35406813:23
jaosorioryolanda: can you check if there's an extraconfig.yaml in /etc/puppet/hieradata ?13:25
weshaypanda, sshnaidm you there?13:25
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375013:25
jaosoriorshardy: is ExtraConfig supposed to still work in master?13:25
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Reinstate CloudDomain properties for Controller/Compute roles  https://review.openstack.org/35406913:25
shardyjaosorior: yes13:25
pandaweshay: how many hours ?13:25
*** tzumainn has joined #tripleo13:25
weshaypanda, 413:25
jaosoriorshardy: do we pass that through parameter_defaults nowadays?13:25
yolandajaosorior, in tripleo-heat-templates, i see puppet/hieradata/common.yaml, with neutorn::allow_overlapping_ips: true13:25
shardyslagle, gfidente: ^^ FYI that fixes the hostname issue I was seeing13:26
pandaweshay: oh, I read few, not four13:26
shardyturns out it was a custom-roles refactoring patch that broke the hosts file (without breaking CI..)13:26
*** bootsha has quit IRC13:26
shardyjaosorior: yup, everything should be passed via parameter_defaults13:26
*** rbrady has joined #tripleo13:27
*** beagles_brb is now known as beagles13:27
shardyyolanda: I'd double check you don't have ExtraConfig defined in more than one environment file13:27
shardyand grep in /etc/puppet/hieradata on the node for the key you're setting13:27
jaosoriorshardy: alright, just making sure. Cause it's been removed from the parameters that overcloud.yaml passes from the controller and compute, but not the rest of the node types13:27
yolandai don't have that setting on hieradat, looking more in depth13:27
shardyjaosorior: yup, by using parameter_defaults it should still be applied13:27
*** alee has joined #tripleo13:28
*** jistr has quit IRC13:28
dtantsurfolks, has anyone seen deployment step 3 failing because of missing sql_connection option in nova.conf?13:28
dtantsuror maybe at least has an idea why it could happen?13:28
jaosoriordtantsur: got a link to the logs?13:29
dtantsurnope, this is locally13:29
dtantsurI can dump the whole "stack failures list" if that helps13:30
dtantsurjaosorior, ^^13:30
*** jistr has joined #tripleo13:30
jaosoriordtantsur: haven't seen that, but it seems to me that it might be an ordering issue..13:31
dtantsurdo we have any places where ordering matters?13:31
jaosoriordtantsur: was it the db-sync failing?13:32
dtantsurjaosorior, yep. I'm testing https://review.openstack.org/#/c/341433/ locally13:32
jaosoriordtantsur: Let me check puppet-nova13:32
dtantsurbefore I introduced the environment file, everything was fine :( maybe I've messed some ordering when doing it13:32
jaosoriorwell... that's weird, nova's db-sync is supposed to be anchored13:33
jaosoriordtantsur: gotta go to a meeting :(13:33
dtantsuryeah.. so I'm getting a very partial nova.conf for some reason13:33
*** ohamada has quit IRC13:33
dtantsurjaosorior, np, thanks13:33
*** ohamada has joined #tripleo13:33
jaosoriordtantsur: but which db-sync was, nova's db-sync or ironic's?13:34
lucasagomeshi all, any chance someone can take a look at https://review.openstack.org/#/c/350627/ ? It has a +2 already :-)13:34
dtantsurjaosorior, /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync] and /Stage[main]/Nova::Db::Sync_api/Exec[nova-db-sync-api] both failed13:35
pandaweshay: this may probe difficult, there is a single periodic definition for all the projects in zuul layout. It may mean that periodicities below 1 day are not encouraged.13:35
pandaprove*13:35
weshayof course13:35
weshaypabelanger, is that the case re: periodic ^13:35
jaosoriordtantsur: :( I see... well, I think it's an ordering issue. and it's weird cause there is an anchor set for both of those https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync.pp#L2313:36
jaosorioranyway, I do have a meeting13:36
*** jaosorior is now known as jaosorior_mtg13:36
weshaypanda, go see if there is another trigger defined13:36
pandaweshay: I see only periodic and periodic-stable, both once a day13:36
weshaypanda, meh.. ok go put in a new definition and see what kind of reviews we get13:37
pandaweshay: ok13:37
dtantsurok, I'll try refreshing the environment, maybe old images or something like that13:38
pandaweshay: but it's better to explain exactly why we want to do this, if we'll have to defend the change.13:39
bandinigfidente: do we need to recheck https://review.openstack.org/#/c/254788/ due to the POST_FAILURE?13:39
gfidentebandini, the patch testing that passed but ...13:39
yolandajaosorior_mtg, shardy , i found that only on one place, on /home/stack/network-environment.yaml. Maybe is not a good place to define that?13:40
bandinigfidente: does it mean I should recheck?13:40
* bandini slightly confused13:41
gfidentebandini, yes13:42
bandinigfidente: thanks!13:42
shardyyolanda: provided it's not malformed and you're including /home/stack/network-environment.yaml that should work - if you'd like it checked can you paste it somewhere?13:43
shardyyolanda: you can also run the overcloud deploy with --debug and tee the results to a file, then check the ExtraConfig you expect is included in the call to heat13:43
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Update Ironic composable services  https://review.openstack.org/34143313:43
yolandashardy, here it is http://paste.openstack.org/show/554145/13:44
yolandathis file is being picked because it gets the network config from there13:45
yolandashardy, i wonder if i'm missing concepts again... because actually, that value is on neutron.conf of the undercloud13:46
shardyyolanda: Hmm, so you want that to be set for the undercloud neutron?13:46
shardyExtraConfig will only influence the overcloud services13:46
yolandathat was my suspect... so in undercloud for mitaka/master, that value is just commented13:47
yolandain liberty, is set to false, then network step of overcloud fails13:47
yolandawhere is that value set for the undercloud, you know?13:47
shardyyolanda: for recent undercloud versions you can pass a hieradata_override = option in the undercloud.conf13:48
shardyI'm not sure that got backported to mitaka tho13:48
yolandait is liberty so even worse13:48
yolandano hieradata_override for instack-undercloud on liberty13:50
*** ayoung has joined #tripleo13:51
*** myoung|afk is now known as myoung13:52
*** limao has quit IRC13:54
sshnaidmweshay, hi13:55
shardyEmilienM: Hey, testing your hiera/json patch, and I think there's a problem when a file doesn't exist that's defined in the hierarchy13:55
shardyhttp://paste.openstack.org/show/554148/13:55
shardythe deployment gets stuck in a loop doing that at step313:55
*** fultonj has quit IRC13:56
yolandashardy, maybe a way can be to have a undercloud-install-post.sh script that updates that setting on neutron?13:56
weshaysshnaidm, can you work w/ EmilienM so that we can add the puppet CI to the tripleo periodic job set that votes13:56
*** fultonj has joined #tripleo13:56
shardyyolanda: you could manually configure it every time the undercloud install is run as a workaround13:56
EmilienMshardy: will look in a few, in a mtg now13:57
yolandayes, something like that, oooq allows to have that custom post template, that could make the trick, although it looks a bit dirty13:57
shardyEmilienM: np, thanks - it may be a json/yaml issue13:57
shardyas it's looking for the yaml files as foo.json13:57
sshnaidmweshay, EmilienM, sure, which puppet CI job are we talking?13:58
slagledeep dive starting, https://bluejeans.com/176756457/13:58
*** maeca1 has joined #tripleo13:59
*** limao has joined #tripleo13:59
*** leanderthal|afk is now known as leanderthal13:59
hewbroccaslagle: I hope the pool for the deep dive isn't green...14:00
*** limao_ has joined #tripleo14:00
*** limao_ has quit IRC14:01
*** limao_ has joined #tripleo14:01
EmilienMshardy: I don't see how it's related to my patch in puppet-openstacklib, it must be something in tht, I can look14:01
shardyEmilienM: Ok, np I've got the related t-h-t patch applied so perhaps that's somehow broken14:02
shardyI don't see how either, will dig deeper14:02
mandreEmilienM: just to confirm I'm on the right track for https://review.openstack.org/#/c/344140/14:02
EmilienMshardy: my patch in puppet translate a string json into the right puppet type14:02
mandreEmilienM: I'll need a couple of patches like this one https://review.openstack.org/#/c/354087/, correct?14:03
shardyEmilienM: Hmm, running it again perhaps those warnings are spurious14:03
slagledprince: you dropped14:03
shardysorry, it may be getting stuck somewhere else14:03
shardywill try again, sorry for the noise14:03
*** limao has quit IRC14:03
*** dprince_ has joined #tripleo14:03
*** limao has joined #tripleo14:04
*** dprince has quit IRC14:04
EmilienMmandre: what is the context?14:05
mandreEmilienM: you asked me to submit patches to puppet-* to configure cors settings14:06
EmilienMah maybe14:06
mandreEmilienM: https://review.openstack.org/#/c/344140/2/elements/puppet-stack-config/puppet-stack-config.pp14:06
EmilienMmandre: good14:06
*** limao has quit IRC14:06
EmilienMmandre: patch needs tests & release note14:06
*** limao_ has quit IRC14:07
mandreEmilienM: so, yeah, I'm a puppet noob so I just wanted to confirm that https://review.openstack.org/#/c/354087/ was the way to go14:07
EmilienMmandre: it is14:07
mandreEmilienM: excellent! I'll add specs and release notes14:08
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port  https://review.openstack.org/33869414:08
pandaweshay: https://review.openstack.org/35409814:11
*** [1]cdearborn has joined #tripleo14:11
*** trozet has joined #tripleo14:13
*** panda is now known as panda|scuba14:13
*** masco has quit IRC14:13
trozetEmilienM: thanks for reviewing 335173, can you tell me more about what this "HA light" arch is?14:15
EmilienMtrozet: https://specs.openstack.org/openstack/tripleo-specs/specs/newton/pacemaker-next-generation-architecture.html14:16
*** jistr is now known as jistr|call14:16
*** oshvartz has quit IRC14:16
trozetEmilienM: thanks14:17
*** ebalduf has joined #tripleo14:18
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port  https://review.openstack.org/33869414:20
*** cdearborn has quit IRC14:22
*** beekneemech is now known as bnemec14:23
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bond  https://review.openstack.org/34918314:23
EmilienMshardy: re - t-i-e, do you want me to branch it and release it?14:24
*** jprovazn has quit IRC14:26
*** limao has joined #tripleo14:29
*** chlong|mtg has quit IRC14:33
shardyEmilienM: I think probably not now, lets decide if it should be branched when we branch everything for Newton?14:33
shardythanks for offering though14:33
EmilienMshardy: ack14:33
*** Vijayendra has quit IRC14:35
*** Vijayendra has joined #tripleo14:35
shardyEmilienM: so, testing further, it seems that the interpolated value looks correct (json string), but the value which we put into the service templates is not14:37
shardye.g passing a list into keystone::rabbit_hosts gives rabbit_hosts = 192.0.2.1014:37
trozetEmilienM: ok I like this new arch.  So as long as the service is under overcloud resource-registry, it should just work with HA?14:37
shardybut passing a json string gives rabbit_hosts = ["192.0.2.10"]14:37
shardyso it would seem that the json->list conversion isn't working?14:38
shardyEmilienM: any tips on how to debug the code you added?14:38
*** jtomasek_ has quit IRC14:40
EmilienMshardy: what happens if you don't use my code?14:40
*** limao_ has joined #tripleo14:42
shardyEmilienM: Not tried that yet as it's a combination I didn't expect to work, sec let me try14:42
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Removing WARNING: line has more than 140 characters in puppet-tripleo profiles  https://review.openstack.org/35405414:43
EmilienMshardy: I also found that my patch might be not finished14:44
EmilienMshardy: Ineed to patch another provider14:44
EmilienMlet me iterate on it14:44
*** limao has quit IRC14:45
*** chlong|mtg has joined #tripleo14:45
shardyEmilienM: ack, thanks!  I can re-test when you're ready14:45
*** numan_ has quit IRC14:45
EmilienMshardy: please test without my patch and see the result if it's different14:46
*** apetrich has quit IRC14:47
*** limao_ has quit IRC14:49
*** limao has joined #tripleo14:49
*** skramaja has quit IRC14:49
*** bana_k has joined #tripleo14:50
EmilienMshardy: I updated the patch14:52
mariosdprince_: thanks dan well done fwiw14:52
shardythanks dprince_!14:53
dprince_ack, thanks guys14:53
EmilienMdprince_: ++14:53
EmilienMdprince_: we want to see the switch now14:53
dprince_EmilienM: meh, it isn't that exciting I think14:54
*** limao has quit IRC14:56
*** limao has joined #tripleo14:56
*** rajinir has joined #tripleo14:58
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-quickstart: files on undercloud had wrong uid  https://review.openstack.org/35239814:59
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-quickstart: fix teardown (tagging and ansible_user)  https://review.openstack.org/35211214:59
jaosorior_mtglarsks: in quickstart is it possible to only run a specific task depending on what release we're deploying15:00
larsksjaosorior_mtg: you could add specific conditionals to a task to do that, yes.15:01
jaosorior_mtglarsks is there an example somewhere?15:01
larsksjaosorior_mtg: not on a task right now, but in some of the templated shell scripts.  On a task you would add something like "when: release in ['mitaka', 'liberty']"15:02
jaosorior_mtglarsks: thanks15:02
jristdprince_: thanks for the dd, can you add this to the etherpad? https://invis.io/UM87J4NBQ#/179046668_2016-8-1_TripleO_UI3715:07
*** limao has quit IRC15:07
dprince_jrist: sure15:08
jristthanks15:09
*** limao has joined #tripleo15:09
EmilienMI like how UI folks always have fancy tools :-)15:09
EmilienMjrist: I'm jealous !15:09
*** Guest20032 is now known as saneax_AFK15:09
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-quickstart: Persist ironic configuration changes  https://review.openstack.org/35168315:10
*** panda|scuba is now known as panda15:12
*** penick has quit IRC15:12
*** aufi has quit IRC15:14
jristEmilienM: hahaha15:17
jristEmilienM: like what? Invision?15:17
*** jaosorior_mtg is now known as jaosorior15:17
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Move Nova settings out of puppet/compute.yaml  https://review.openstack.org/35374815:17
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Mv Ceilometer settings out of puppet/compute.yaml  https://review.openstack.org/35406315:17
*** ebarrera has quit IRC15:18
*** limao has quit IRC15:19
EmilienMjrist: yeah15:19
jaosoriorEmilienM: couple of simple commits :D https://review.openstack.org/#/c/353962/  https://review.openstack.org/#/c/353964/15:19
EmilienMjaosorior: ack15:19
*** limao has joined #tripleo15:19
jristEmilienM: command line is so well done at this point15:19
jristwhat with emacs and all15:19
jrist:-P15:20
* jrist starts holy war15:20
jristtmux and vim15:20
jristwhat else do you need? :)15:20
dprince_jrist: nothing15:20
*** limao has quit IRC15:20
dtantsurso folks, I guess https://review.openstack.org/#/c/341433/14/puppet/services/haproxy.yaml is not enough for haproxy to start working with ironic, is it?15:21
dtantsurdo you know what I can be missing?15:21
*** ifarkas is now known as ifarkas_afk15:21
*** jistr|call is now known as jistr15:21
*** tesseract- has quit IRC15:23
shardyEmilienM: still not working - does the PSON::load work by reference, or should it be new_value = PSON::load(new_value) ?15:23
shardyseems like it's just passing the json string into the service config15:23
EmilienMshardy: so you tried without my patch and you got same results?15:23
shardyEmilienM: yup15:24
EmilienMyeah it's possible15:24
shardyEmilienM: I'll try adding the assignment15:24
EmilienMmhh15:24
*** mcornea has quit IRC15:25
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Removing WARNING: line has more than 140 characters in puppet-tripleo profiles  https://review.openstack.org/35405415:25
jaosoriorEmilienM: What do you mean they won't exist anymore? We still need to have common stuff for all the nodes15:25
jaosoriorwhat's gonna replce that?15:26
* shardy wonders if a unit test may be a quicker way to debug this15:26
bswartzjrist: why does everyone love tmux now?15:26
EmilienMjaosorior: the overcloud_*.pp files won't exist anymore15:26
dtantsurhmm, openstack-ironic-api fails with "address in use", so I guess I need to somehow tell it to move15:26
EmilienMshardy: I'll look in a few, I'm currently doing a CI thing for puppet/tripleo15:27
EmilienMshardy: I figured we don't run tripleo jobs in puppet stable/mitaka CI15:27
shardyEmilienM: Ok, no problem, thanks15:27
EmilienMI'm on it15:27
*** fultonj has quit IRC15:27
jaosoriorEmilienM: sure, so what will replace that? will we have a common role or something of the sort?15:28
jaosoriorwould be better to give an alternative than just say "I won't support it"15:28
*** fultonj has joined #tripleo15:28
sshnaidmEmilienM, hi, regarding puppet CI job as part of promotion jobs, are you familiar with this idea? just checking to be on the same page15:28
EmilienMI'm shutting down IRC, can't focus on my CI problem15:29
openstackgerritMerged openstack/puppet-tripleo: Handle redirects for keystone  https://review.openstack.org/35295615:29
*** zaneb has quit IRC15:30
rookhey shardy - is there somewhere I can track for custom roles?15:35
jristbswartz: no idea haha15:36
*** bana_k has quit IRC15:37
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Update Ironic composable services  https://review.openstack.org/34143315:39
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375015:39
*** jcoufal has quit IRC15:40
shardyjaosorior: Common stuff will have to be defined either via composable services (which can be deployed on all roles), or via the allNodesConfig for cluster-wide stuff (which I'm currently working on refactoring)15:42
shardythere will be no per-role manifests tho15:42
jaosoriorshardy: sure, that's no biggie. I'm just wondering where are ::tripleo::packages and ::tripleo::firewall gonna be included from?15:43
*** links has quit IRC15:43
shardyjaosorior: I think they'll need to move to a services template15:43
jaosoriorshardy: by the way... the keystone endpoints via puppet are being quite problematic. I'm running into a lot of OOM crashes there.15:43
jaosorioreither we bump up the memory for the nodes and hope it works there. Or I think we're gonna end up doing those in mistral15:44
shardyjaosorior: oh dear, that doesn't sound good15:44
shardywhy would puppet creating the endpoints use much more ram?15:44
shardyI assumed it probably just shells out to openstackclient (maybe openstackclient is the reason?)15:45
jaosoriorshardy: I have no ide dude15:45
jaosoriorbut it does use openstackclient in the background15:45
jaosoriormight be15:45
jaosoriorshardy: could also be that puppet does things quite concurrently. So doing several of those endpoints at the same time could be problematic due to the memory usage?15:46
shardyjaosorior: possibly, need to profile the memory usage while it's running really I think15:46
*** adarazs is now known as adarazs_brb15:47
openstackgerritJulie Pichon proposed openstack/tripleo-common: [WIP] Add Mistral actions for boot & root device configuration  https://review.openstack.org/35417415:47
openstackgerritPaul Belanger proposed openstack/python-tripleoclient: Initial support for bindep  https://review.openstack.org/35392915:47
jaosoriorshardy: somehow I don't get OOMs when running that locally though15:47
EmilienMjaosorior: we need to find a mechanism that include ::tripleo class all the time15:50
jaosoriorEmilienM: so, for now I think we can have that as a base.yaml or common.yaml15:50
jaosoriorin the form of a composable service15:50
jaosoriorbut we do need a way to make sure that all the node types always include it15:51
EmilienMjaosorior: yeah, I just don't want our users to forget addin git15:51
EmilienMright !15:51
*** milan has quit IRC15:51
jaosoriorelse we're gonna shoot ourselves in the foot and we're gonna have really funky bug reports15:51
EmilienMsshnaidm: yes i'm familiar, we need to run puppet-openstack-integration jobs on centos715:51
*** adarazs_brb is now known as adarazs15:52
EmilienMjaosorior: ok, let me think about it15:52
jaosoriorEmilienM: that's the reason why I was really stubbornly still using those templates15:52
jaosoriorI mean15:52
jaosoriormanifests15:52
EmilienMyeah but they will be gone very very soon15:53
EmilienMwe don't need it anymore15:53
EmilienMjaosorior: maybe in short term we can include ::tripleo in all base profiles in puppet-tripleo?15:54
mandreEmilienM: regarding https://review.openstack.org/#/c/354087/, the spec was failing in my environment, I'm not too sure why15:57
*** rhallisey has quit IRC15:57
mandreEmilienM: you might want to hold on before approving too quickly15:57
mandreEmilienM: how long before the test job returns in puppet-keystone usually?15:58
EmilienMmandre: ~30 min15:58
*** rhallisey has joined #tripleo15:58
EmilienMit depends if osinfra is busy15:59
EmilienMhttp://status.openstack.org/zuul/15:59
EmilienM250 jobs in queue, its busy15:59
EmilienMso maybe 45 min15:59
*** jprovazn has joined #tripleo16:00
mandreEmilienM: here is the exact failure in my environment, http://paste.openstack.org/show/554219/16:01
mandreEmilienM: I don't understand enough to debug why cors/allowed_origin defaults to nil16:01
*** bnemec has quit IRC16:04
*** rwsu has joined #tripleo16:09
*** bnemec has joined #tripleo16:12
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud  https://review.openstack.org/35401616:15
*** rcernin has quit IRC16:15
jaosoriorbnemec: Hey dude, what do we do with your commit that bumps the memory for the ovb nodes? https://review.openstack.org/#/c/353655/16:16
bnemecjaosorior: Really all we need to do is merge it.  I've already applied the change to rh1.16:16
jaosoriorshardy: could we merge this? https://review.openstack.org/#/c/353655/ it bumps up the memory for the ovb nodes, would surely be useful16:17
jaosorioror slagle, dprince_ ^^16:17
*** dtantsur is now known as dtantsur|afk16:18
*** iogg has quit IRC16:18
dprince_jaosorior: +A16:20
jaosoriorawww yeah16:20
*** zeroshft has joined #tripleo16:21
slaglejaosorior: they are already bumped up16:21
slaglejaosorior: bnemec did it manually yesterday16:21
slagleif we could find out how much memory we actually need...16:22
jaosorioroh16:22
*** egafford has quit IRC16:22
slagleif creating endpoints via puppet requires another 512mb, then i don't think we want to do it16:23
jaosoriorslagle: well... I use 6144 for the overcloud nodes :/16:23
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Move neutron settings out of puppet/compute.yaml  https://review.openstack.org/35420116:23
jaosoriorslagle: I'll see if it can pass with the current upgrade. and if it doesn't then we gotta do it in mistral then16:23
*** apetrich has joined #tripleo16:25
jaosorioranyway16:25
jaosoriorI'm off. Have a good day guys!16:25
*** dbecker has quit IRC16:25
*** jaosorior has quit IRC16:25
slaglettyl16:25
*** hewbrocca is now known as hewbrocca-afk16:26
*** yamahata has joined #tripleo16:27
*** ebarrera has joined #tripleo16:29
bnemecDB on rh1 crashed. :-(16:30
bnemecThings are going to be bad on the CI front for a while.16:30
gfidenteEmilienM, this is only adding files https://review.openstack.org/#/c/311218/16:32
gfidentethere is no way it can ever make CI fail and in fact16:32
gfidentehttps://review.openstack.org/#/c/304563/ which depends on it passed, can we merge them all?16:32
EmilienMgfidente: yes16:32
EmilienMgfidente: done16:33
gfidentethanks :016:33
gfidente^_^16:33
slaglebnemec: looks like you restarted mariadb?16:35
bnemecslagle: Yeah.16:36
bnemecLooks like maybe it ran out of fds.16:36
bnemecI also dropped the number of Heat engine workers.16:36
bnemecWhen it died, we had like 13 heat stacks creating.16:36
bnemecAnd the cpu was pegged.16:36
shardymichchap: Hi, I think https://blueprints.launchpad.net/tripleo/+spec/refactor-puppet-manifests can be marked implemented now, do you agree?16:37
*** jpich has quit IRC16:37
michchapshardy: I agree. Done.16:39
shardymichchap: thanks!16:39
michchapshardy: sorry for ending up the owner of that - it was other people's work!16:39
*** akrivoka has quit IRC16:39
EmilienMmichchap: lol16:40
EmilienMthanks for the 123023 patches !16:40
shardymichchap: hehe - it was a team effort :)16:40
michchapEmilienM: I submitted like 1 change at the start because I was impatient :D16:40
EmilienMahah16:40
slaglebnemec: i guess we never fixed the fd limit on the oc16:41
bnemecslagle: Yeah, I looked into it when this first happened and it looked like it was set to 4096 correctly.16:42
bnemecWhen I was watching the mysql connections they were hovering around 700, which seemed pretty safe.16:43
bnemecBut I haven't checked in a couple of days, so it's possible they had increased when we re-enabled jobs and increased the number of envs.16:43
*** florianf has quit IRC16:44
slaglebnemec: i dont think it's the connections, it's the open files limit16:46
slaglemaybe they're related16:46
bnemecslagle: Yeah, I think they are.  Rabbit has the same issue.16:48
*** zeroshft has quit IRC16:48
bnemecAnd I see it's limited to 1024: open files                      (-n) 102416:49
bnemecI thought we had fixed that. :-/16:49
*** zeroshft has joined #tripleo16:49
sshnaidmbnemec, does geard run therE?16:49
bnemecsshnaidm: No, that runs in a VM.  This is the host cloud controller.16:50
sshnaidmslagle, it's the same limit for both of them - files and connections16:50
shardyJokke_, gfidente: should we be either marking https://bugs.launchpad.net/tripleo/+bug/1589922 invalid or wontfix?16:50
openstackLaunchpad bug 1589922 in tripleo "OSD data dir expected in /var/lib/ceph/osd" [High,In progress] - Assigned to Erno Kuvaja (jokke)16:50
slaglebnemec: we fixed in the undercloud16:50
*** lucasagomes is now known as lucas-dinner16:50
pabelangerSo, regarding tripleo-test-cloud-rh1.  What are the changes of getting a new project called openstackzuul created?  Reason being, from nodepool we can see all the private infrastructure being run in the openstack-nodepool project which is starting to flood our nodepool debug logs16:50
sshnaidmbnemec, ok, because it was the same problem with geard before16:50
shardyThe discussion in the bug seems inconclusive16:50
*** tosky has quit IRC16:51
pabelangerwe'd still keep openstack-nodepool just add openstackzuul where we launch our VMs16:51
slaglebnemec: and also in my ha pacemaker env, the limit is 2048516:51
gfidenteshardy, was just working on that, let me add a comment16:51
slaglebnemec: so i think it's just not fixed for non-pacemaker envs16:51
shardygfidente: Ok, thanks, I'm just trying to groom the newton-3 bugs a bit16:51
slaglesince mariadb is started under systemd16:51
bnemecslagle: Oh /facepalm16:51
*** tosky has joined #tripleo16:51
*** fragatina has joined #tripleo16:51
slaglebnemec: quick fix would be to copy /etc/systemd/system/mariadb.service.d/limits.conf from uc to the oc, and restart mariadb16:52
*** rhallisey has quit IRC16:52
*** rhallisey has joined #tripleo16:54
*** sshnaidm has quit IRC16:54
*** fragatin_ has quit IRC16:54
bnemecslagle: Done16:54
bnemecWe need to add that to the deployment docs too so this doesn't happen again.16:55
*** yamahata has quit IRC16:55
*** yamahata has joined #tripleo16:55
slagleyea i'll add it to the etherpad16:56
slagleand file a bug to fix in tht16:56
pabelangerbnemec: slagle16:56
pabelangererr16:56
pabelangerbnemec: slagle: does openstack-infra have admin credentials on tripleo-test-cloud-rh1?16:57
pabelangershould they?16:57
pabelangerI would ask, because might be able to drive some of the configuration of the cloud via ansible and system-config16:57
pabelangerbut I'd need to confirm with openstack-infra if we wanted to do that16:58
bnemecpabelanger: It's a TripleO-deployed cloud, so we wouldn't want to do that.16:58
slaglepabelanger: afaik, openstack-nodepool is not admin16:58
bnemecIt's actually an excellent exercise in dogfooding.16:58
pabelangerRight16:58
slaglepabelanger: what type of configuration?16:58
*** julim has quit IRC16:58
bnemecBecause we probably have users who have hit this too.16:58
*** apetrich has quit IRC16:59
pabelangerslagle: basically, were using ansible now to provision our openstack clouds configuration, like projects, security groups, etc.  EG: https://review.openstack.org/#/c/353594/ Be relaunched blueblox yesterday and it worked well16:59
pabelangerwas going to talk with tripleo and see if you'd be interested in starting to do that too. From what I see, it is just bash scripts for the most part17:00
bnemecOh, so this is unrelated to the mariadb thing. :-)17:00
slagleyea :)17:00
slaglepabelanger: i guess this might work. you're just configuring things in keystone/neutron/etc?17:01
bnemecI'm fine with using standard infra tooling to configure things.17:01
slagleby talking to api's?17:01
pabelangerslagle: if you see scroll back about the openstackzuul project / user, I plan to use ansible to create our projects settings with it17:01
pabelangerslagle: creating of security groups for example, and flavors17:02
*** rcernin has joined #tripleo17:02
pabelangerthings we need to do to make nodepool work but now we are automating it with shade and ansible17:02
slagleok, i think it sounds fine17:03
slaglewhat do other clouds do?17:03
slagledo they create an admin user for infra to use?17:03
pabelangerusually not,17:03
pabelangerwe only have admin for bluebox17:03
pabelangerso, tripleo doesn't need to share admin creds with openstack-infra17:04
pabelangerbut maybe we start moving tripleo-ci away from bash scripts to ansible playbooks17:05
pabelanger_shurgs_17:05
pabelangeronly if people are interested17:05
*** rcernin has quit IRC17:05
*** rcernin has joined #tripleo17:05
*** fragatina has quit IRC17:06
*** fragatina has joined #tripleo17:07
slaglepabelanger: is prepare-ovb-cloud.sh what it replaces?17:07
slagle(from tripleo-ci)17:07
pabelangerslagle: Ya, that was the file I was looking at.17:08
slaglei think it makes sense. happy to use standard infra tooling if we can make it work17:08
bnemecNote that there's tripleo-specific stuff in that file too.17:08
slaglebnemec: yea, ovb stuff17:08
bnemecInfra's not going to configure the OVB flavors, for instance.17:09
bnemecRight17:09
slaglei noticed, we'd have to split that out17:09
*** apetrich has joined #tripleo17:09
openstackgerritMiles Gould proposed openstack/diskimage-builder: Allow installing proliant tools from an OS package  https://review.openstack.org/34421417:09
pabelangerLets do this, if you create the openstackzuul user / project (see backscroll for reason), I'll push a patch to review.o.o showing an example configuration17:09
bnemecBut yeah, total +1 to using infra tools to create infra bits.17:10
*** bana_k has joined #tripleo17:10
slaglepabelanger: ok. is there any special way I should create it?17:11
*** fragatina has quit IRC17:12
pabelangerslagle: I don't think so.  The openstackzuul user needs to be part of the openstackzuul project, everything else we should be able to automate for settings17:12
pabelangerslagle: we need to be on the same network as your private infra and OVB servers17:12
pabelangeras we are moving nodepool vms from openstack-nodepool to openstackzuul17:13
pabelangerwe won't make the cut over until we can manually launch a server and test networking bits17:13
*** numans has joined #tripleo17:15
bnemecThat shouldn't be a problem.  The private network is already shared.17:15
*** mgould is now known as mgould|afk17:17
*** jeckersb is now known as jeckersb_gone17:17
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bond  https://review.openstack.org/34918317:22
*** fultonj_ has joined #tripleo17:23
*** fragatina has joined #tripleo17:27
*** electrofelix has quit IRC17:28
*** Goneri has quit IRC17:28
*** apetrich has quit IRC17:35
*** egafford has joined #tripleo17:37
*** ebarrera has quit IRC17:38
*** abehl has quit IRC17:41
openstackgerritGiulio Fidente proposed openstack/tripleo-docs: Document Ceph repositories to be enabled  https://review.openstack.org/35291017:47
*** leanderthal is now known as leanderthal|afk17:50
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Move neutron settings out of puppet/compute.yaml  https://review.openstack.org/35420117:52
dprince_bnemec: hey, what do you have in mind for here https://review.openstack.org/#/c/300139/17:54
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Disable sahara services by default  https://review.openstack.org/35288617:54
dprince_bnemec: I would like to get that in. It has been broken for months now17:55
gfidenteEmilienM, slagle ^^ that sahara thing landed in mitaka17:56
gfidenteso I am not sure it's good to disable it in newton17:56
*** fragatina has quit IRC17:57
openstackgerritMerged openstack/tripleo-quickstart: files on undercloud had wrong uid  https://review.openstack.org/35239817:57
*** ohamada has quit IRC17:57
openstackgerritMerged openstack/tripleo-quickstart: fix teardown (tagging and ansible_user)  https://review.openstack.org/35211217:57
slaglegfidente: why? we dont need sahara by default17:57
gfidenteslagle, right I am just saying people upgrading have it enabled though17:58
slagleupgrading from mitaka?17:58
gfidentefrom mitaka yes17:58
openstackgerritGiulio Fidente proposed openstack/tripleo-docs: Document Ceph repositories to be enabled  https://review.openstack.org/35291018:00
slaglegfidente: ok, well darn18:00
*** jeckersb_gone is now known as jeckersb18:00
gfidenteslagle, yeah in general we don't have a 'ensure disabled' mechanism though it sounds like what we need really18:01
slaglegfidente: can u comment on the review?18:02
slaglegfidente: my motivation with disabling it is mainly for CI18:02
slaglebut i can do that via an environment18:02
gfidenteok18:02
slaglei'd like to optimize what we test in the gate to what is required18:02
*** tosky has quit IRC18:03
*** julim has joined #tripleo18:04
EmilienMgfidente: https://review.openstack.org/#/c/353673/2/manifests/osd.pp18:12
EmilienMoh dear...18:12
EmilienMagain an exec?18:12
gfidenteI think we'd need an exec to check if {data} is a dir or a block device anyway18:12
gfidentebut I have a local copy using selinux::dircontext18:12
gfidentemeantime as per comment in the BZ I think we can make '/srv/data' labeled correctly by ceph-selinux18:13
*** numans has quit IRC18:14
EmilienMgfidente: I -1 for stylistic things but I won't block the exec if we really need it18:15
gfidentethanks so I will fix those18:16
gfidentebut also let's wait if we can get ceph-selinux to adopt us18:16
gfidentethough in general the puppet-ceph feature looks sensible18:16
openstackgerritOpenStack Proposal Bot proposed openstack/os-cloud-config: Updated from global requirements  https://review.openstack.org/35067618:16
openstackgerritOpenStack Proposal Bot proposed openstack/os-collect-config: Updated from global requirements  https://review.openstack.org/35090518:16
gfidentewhat did you mean here "can't we do it in packaging side? do we really need this exec?" ?18:17
gfidentewhich alternative were you thinking about?18:17
*** ayoung has quit IRC18:24
EmilienMgfidente: in ceph-selinux?18:24
EmilienMI don't know..18:24
gfidenteah right18:24
EmilienMall those execs scare me18:24
EmilienMlunch now18:24
gfidenteyeah the probblem is if people passes any non-default path they will get denials18:24
gfidenteso we *can* fix the default path in ceph-selinux but in general the problem remains in the module18:24
EmilienMok let's ffix it in the module then18:25
gfidentewe're doing things in ceph-selinux as well, so we have less pressure on module part18:27
openstackgerritSarath Kumar Sankaran Kutty proposed openstack/os-net-config: Cleanups and new unit test for IVS  https://review.openstack.org/34616118:27
*** alee has quit IRC18:31
*** fragatina has joined #tripleo18:35
*** chlong|mtg has quit IRC18:37
*** chlong|mtg has joined #tripleo18:38
*** alee has joined #tripleo18:44
*** amoralej is now known as amoralej|off18:47
*** lblanchard has quit IRC18:54
*** apetrich has joined #tripleo18:54
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-quickstart: Add undercloud reboot  https://review.openstack.org/35048918:56
*** fragatin_ has joined #tripleo19:00
*** fragatina has quit IRC19:03
*** ebarrera has joined #tripleo19:04
jtomasekd0ugal, shardy: here is what the example mistral environment looks like: http://paste.openstack.org/show/554117/19:05
jtomasekd0ugal, shardy: that way is totally fine for GUI19:06
jtomasekd0ugal: GetCapabilitiesAction combines environments from here with the capabilities map structure19:07
*** athomas has quit IRC19:07
jtomasekd0ugal: IOW, GetParametersAction goes through the capabilities map and adds enabled: true to the environments there if they are present in mistral environment.19:08
*** dprince_ has quit IRC19:08
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Add missing deprecated param  https://review.openstack.org/35434519:10
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Removing WARNING: line has more than 140 characters in puppet-tripleo profiles  https://review.openstack.org/35405419:11
openstackgerritStephanie Miller proposed openstack/diskimage-builder: Spec for adding diskimage_builder module in-chroot  https://review.openstack.org/33666419:14
*** oshvartz has joined #tripleo19:23
*** sarath has joined #tripleo19:26
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699519:31
*** ayoung has joined #tripleo19:40
*** oshvartz has quit IRC19:43
*** jpena is now known as jpena|off19:45
*** jprovazn has quit IRC19:50
*** fultonj has quit IRC19:52
*** fultonj_ has quit IRC19:52
*** fultonj has joined #tripleo19:53
*** jprovazn has joined #tripleo20:02
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove deprecated net_ip_subnet_map output  https://review.openstack.org/34605520:05
*** fragatin_ has quit IRC20:15
*** fragatina has joined #tripleo20:16
rlandybnemec: if these are the values tripleo-quickstart uses for undercloud and overcloud vms, https://github.com/openstack/tripleo-quickstart/blob/master/roles/common/defaults/main.yml#L10, how are you getting away with using such small flavors for baremetal and undercloud in OVB testing?20:19
bnemecrlandy: I have nothing to do with quickstart.20:20
bnemecOur CI has always run on the smallest vms possible.20:20
rlandy that's admirable20:20
bnemecFor resource reasons, and to avoid us bloating the memory usage accidentally.20:20
rlandyI am not having a lot of success dropping the flavor requirements20:21
bnemecrlandy: undercloud, overcloud, or both?20:21
bnemecNote that we just bumped the overcloud vms to 5.5 GB because we were seeing intermittent ooms in jobs.20:22
*** Goneri has joined #tripleo20:22
rlandybnemec: I used smaller flavors for both undercloud and overcloud ...20:22
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: Only set  if it is not already defined  https://review.openstack.org/35436520:23
bnemecrlandy: How big?20:23
rlandyin the past, with a smaller undercloud, I was seeing issues with keystone dying - probably the same oom errors20:23
rlandynow, with a m1.medium undercloud, deploy is failing out20:23
bnemecrlandy: medium is only 4 GB.  That's not enough, unfortunately.20:24
bnemecWe run 6 in CI.20:24
rlandythe main issue is that I don't have access to create custom flavors (relic of the aim to optimize general cloud usage - not my personal quota)20:24
rlandyso I land up loosing 2,4,8, 1620:25
bnemec| 18e29a9b-8b32-4b61-bc76-6346dc7468f2 | undercloud | 6144      | 40   | 0         |      | 2     | 1.0         | True      |20:25
bnemec| 2aaef5e1-a585-4bb9-b1e8-134d82f55982 | bmc        | 512       | 20   | 0         |      | 1     | 1.0         | True      |20:25
bnemec| 49a7fbb0-2c0b-4441-a298-d69085760516 | baremetal  | 5632      | 41   | 0         |      | 1     | 1.0         | True      |20:25
bnemecrlandy: You're probably stuck with 8 then.20:25
rlandyI can accept that baremetal may be big enough (if I use your requirements) when we move to public cloud20:25
rlandybut the undercloud size surprises me20:26
bnemecIf they won't give you custom flavors, then the extra memory usage is the cost, I guess.20:26
bnemecThe other problem is that when you add vcpus to the flavor, it increases the memory usage too.20:26
bnemecSince I think m1.large has 4 vcpus, that might make 8 GB required anyway.20:27
rlandyyeah - I am probably doing a useless test here - trying to get minimum requirements for a public cloud using a ' no custom flavor' private one20:27
rlandystill - that aside, I had oom issue with a 8G undercloud20:28
bnemecStrange, I periodically use an 8 GB undercloud if I don't need to build images.20:30
rlandyI can get away with it for mitaka - not master20:32
*** [1]cdearborn has quit IRC20:36
*** ebalduf has quit IRC20:38
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Set coordination url for gnocchi storage  https://review.openstack.org/35437820:38
*** beagles is now known as beagles_biab20:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove deprecated net_ip_uri_map output  https://review.openstack.org/35438120:41
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: Only set  if it is not already defined  https://review.openstack.org/35436520:42
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove deprecated net_ip_uri_map output  https://review.openstack.org/35438120:43
*** ebarrera has quit IRC20:44
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Set the default gnocchi driver to swift  https://review.openstack.org/35438520:44
*** yamahata has quit IRC20:45
*** jprovazn has quit IRC20:49
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: Only set  if it is not already defined  https://review.openstack.org/35436520:51
*** pradk has quit IRC20:54
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove unneeded net_ip_map dump into hieradata  https://review.openstack.org/35439120:57
dtrainorHi.  I'm playing around with the UI and when I try to do node assignment, I see some failures that I don't recognize.  I think I'm missing an allowable method somewhere but I'm not certain where http://paste.openstack.org/show/554262/20:57
dtrainorif i had to guess, it's ironic-inspector not allowing a PATCH method I think...20:58
*** yamahata has joined #tripleo21:06
dtrainorahh, i see it.... "Client-side error" is a red herring.  The real problem is with the UI preforming introspection.21:07
*** rhallisey has quit IRC21:07
*** weshay has quit IRC21:09
stevebakerbnemec: does bmc-image.qcow2 get rebuilt?21:16
* stevebaker is *so* lazy21:16
bnemecstevebaker: No, it won't.  I uploaded it manually.21:17
bnemecI should upload a patched one though.21:17
*** ayoung has quit IRC21:18
*** zeroshft_ has joined #tripleo21:21
*** shardy has quit IRC21:21
*** alee has quit IRC21:21
*** coolsvap has quit IRC21:21
*** zeroshft has quit IRC21:23
*** weshay has joined #tripleo21:24
*** rlandy has quit IRC21:24
*** pradk has joined #tripleo21:26
*** Goneri has quit IRC21:28
stevebakeryeah, it would reduce problems for others21:29
*** ebalduf has joined #tripleo21:29
*** social has quit IRC21:30
*** zaneb has joined #tripleo21:30
*** zaneb has quit IRC21:30
*** zaneb has joined #tripleo21:31
*** jayg is now known as jayg|g0n321:31
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Pass ServiceNetMap to services  https://review.openstack.org/35441421:41
*** Goneri has joined #tripleo21:43
slaglehmm? we promoted. how?21:48
slagleand now the promoted repo is breaking the multinode job with the same cinder error21:48
*** maeca1 has quit IRC21:51
gfidenteslagle, cinder error was fixed in cinder21:51
slaglegfidente: https://bugs.launchpad.net/tripleo/+bug/161243421:52
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Undecided,New]21:52
slaglegfidente: maybe a repo got promoted that did not yet have the fix21:52
slagleb/c now i'm seeing the error in the check queue21:52
gfidenteis that possible at all?21:53
slaglei dont know it could have happened21:53
slaglei'm asking in rdo if it got done manually21:53
slagleperhaps accidentally21:53
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't deploy net-iso env when we won't use it  https://review.openstack.org/34750321:54
bnemecYeah, I'm seeing a bunch of cinder errors killing the ping test.21:55
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Now fly and set cinder::api::bind_host within service template  https://review.openstack.org/31834422:01
bnemeclol, nice commit message22:02
bnemecFly, you fools!22:02
*** egafford has quit IRC22:12
*** alee has joined #tripleo22:16
*** maeca1 has joined #tripleo22:19
*** tbarron is now known as tbarron|gone22:19
*** weshay has quit IRC22:20
*** Goneri has quit IRC22:21
*** morazi has quit IRC22:37
*** panda is now known as panda|zz22:41
*** saneax_AFK is now known as saneax22:46
*** saneax is now known as Guest9190922:46
openstackgerritGiulio Fidente proposed openstack/tripleo-docs: Document Ceph repositories to be enabled  https://review.openstack.org/35291023:00
*** maeca1 has quit IRC23:01
*** Guest91909 is now known as saneax_AFK23:02
*** thrash is now known as thrash|g0ne23:15
*** zeroshft_ has quit IRC23:30
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient: Use non-bracketed version of addresses for keystone.initialize  https://review.openstack.org/35444323:35
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient: Use non-bracketed version of addresses for keystone.initialize  https://review.openstack.org/35444323:37
*** StevenK_ is now known as StevenK23:38
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test  https://review.openstack.org/34694923:44
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test  https://review.openstack.org/34694923:47
*** beagles_biab is now known as beagles23:49

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!