Wednesday, 2016-08-31

*** mlupton has joined #tripleo00:00
*** mlupton has quit IRC00:05
*** vijaykc4 has quit IRC00:10
*** thrash is now known as thrash|g0ne00:13
*** yamahata has quit IRC00:14
*** chlong has quit IRC00:16
*** saneax is now known as saneax-_-|AFK00:42
*** yamahata has joined #tripleo00:47
*** Goneri has quit IRC00:56
*** fzdarsky_ has joined #tripleo01:25
*** akshai has quit IRC01:26
*** fzdarsky|afk has quit IRC01:29
*** vijaykc4 has joined #tripleo01:29
*** dmacpher has joined #tripleo01:36
*** chlong has joined #tripleo01:40
*** bana_k has quit IRC01:41
*** Goneri has joined #tripleo01:46
openstackgerritMerged openstack/tripleo-heat-templates: configure Heat with new authtoken parameters  https://review.openstack.org/36141201:47
*** Ryjedo_ has joined #tripleo01:52
*** Ryjedo has quit IRC01:53
*** Ryjedo_ is now known as Ryjedo01:53
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Enable modern authtoken for Neutron and Glance  https://review.openstack.org/36298001:54
*** bana_k has joined #tripleo01:57
*** limao has joined #tripleo02:03
*** bkopilov has quit IRC02:05
*** pblaho has quit IRC02:14
*** Goneri has quit IRC02:15
*** yamahata has quit IRC02:35
*** anbu has quit IRC02:36
openstackgerritRyan Brady proposed openstack/tripleo-common: Wire in jinja templating for custom roles  https://review.openstack.org/36246502:41
*** saneax-_-|AFK is now known as saneax02:43
saneaxhttps://blueprints.launchpad.net/tripleo/+spec/tripleo-ovs-dpdk closed02:43
larsksI have a review failing (quickly!) because "Exception registering nodes: Node 59a06f3a-41a9-4913-9a9e-85e3f157c420 is locked by host localhost.localdomain, please retry after the current operation is completed."  I am sure that has nothing to do with my review, but is it something that will eventually clear up on its own?  Or is something wedged that needs cleaning up?02:51
*** pblaho has joined #tripleo02:54
*** pblaho has quit IRC02:59
openstackgerritOpenStack Proposal Bot proposed openstack/os-collect-config: Updated from global requirements  https://review.openstack.org/35090503:08
*** vijaykc4 has quit IRC03:09
*** vijaykc4 has joined #tripleo03:12
*** bkopilov has joined #tripleo03:22
*** pblaho has joined #tripleo03:24
*** ayoung has quit IRC03:27
*** bana_k has quit IRC03:29
*** anbu has joined #tripleo03:30
*** jdob has quit IRC03:34
*** anbu_ has joined #tripleo03:38
*** saneax is now known as saneax-_-|AFK03:40
*** anbu__ has joined #tripleo03:41
*** anbu has quit IRC03:42
*** anbu_ has quit IRC03:44
*** padkrish has joined #tripleo04:00
*** bkopilov_ has joined #tripleo04:03
*** bkopilov_ has quit IRC04:03
*** bkopilov_ has joined #tripleo04:04
*** crinkle_ has joined #tripleo04:16
*** cschwede has quit IRC04:17
*** crinkle has quit IRC04:17
*** cschwede has joined #tripleo04:17
*** mpsairam has quit IRC04:18
*** tzumainn has quit IRC04:18
*** bana_k has joined #tripleo04:23
*** masco has joined #tripleo04:24
*** pmannidi has joined #tripleo04:26
*** pgadiya has joined #tripleo04:28
*** fultonj has quit IRC04:34
*** dtantsur|afk has quit IRC04:36
*** links has joined #tripleo04:36
*** coolsvap_ has joined #tripleo04:36
*** limao has quit IRC04:37
*** chlong has quit IRC04:38
*** chlong has joined #tripleo04:39
*** padkrish has quit IRC04:40
*** padkrish has joined #tripleo04:41
*** padkrish has quit IRC04:41
*** chlong has quit IRC04:45
*** limao has joined #tripleo04:53
*** chlong has joined #tripleo04:57
*** yamahata has joined #tripleo05:03
*** yamahata has quit IRC05:06
*** mcornea has joined #tripleo05:10
*** bana_k has quit IRC05:13
*** Ryjedo_ has joined #tripleo05:18
*** fragatina has quit IRC05:19
*** jaosorior has joined #tripleo05:21
*** Ryjedo has quit IRC05:22
*** Ryjedo_ is now known as Ryjedo05:22
jaosoriorCan someone review this? https://review.openstack.org/#/c/359038/05:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move service/endpoint configuration to keystone profile  https://review.openstack.org/35968105:43
*** dbecker has joined #tripleo05:46
*** ifarkas_afk has quit IRC05:50
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Remove the ha2 JOBTYPE  https://review.openstack.org/36341105:52
openstackgerritBabu Shanmugam proposed openstack/tripleo-heat-templates: OVN heat templates  https://review.openstack.org/30773405:58
*** fragatina has joined #tripleo06:02
*** bvandenh has quit IRC06:04
*** fragatina has quit IRC06:07
*** bana_k has joined #tripleo06:11
*** pcaruana has joined #tripleo06:15
skramajajaosorior:  CI passed fro the SR-IOV THT - https://review.openstack.org/#/c/361430/ - has 4 +2s, ptal06:18
jaosoriorskramaja: done06:19
skramajajaosorior: thanks.06:19
*** rcernin has joined #tripleo06:21
*** xinwu has joined #tripleo06:23
*** xinwu has left #tripleo06:27
*** bana_k has quit IRC06:29
*** abregman has quit IRC06:31
*** hewbrocca-afk is now known as hewbrocca06:37
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Refactor DeploymentPlan Component  https://review.openstack.org/35881406:39
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Mistral Actions error handling  https://review.openstack.org/35976406:39
hewbroccaGood morning everyone06:39
jaosoriorhewbrocca: sup dude, how's it going?06:39
hewbroccaVery well thanks06:40
hewbroccaI leave on PTO tomorrow!06:40
*** dsariel has joined #tripleo06:41
jaosoriornice!06:41
jaosoriorgonna travel anywhere?06:41
hewbroccaLa France!06:41
hewbroccaBTW06:41
hewbroccaif anyone has time to review these:06:41
jaosoriornice!06:41
hewbroccahttps://review.openstack.org/#/c/353506/ and06:42
hewbroccahttps://review.openstack.org/#/c/254788/06:42
hewbroccaops-tools reviews -- poor larsks keeps having to rebase them06:42
*** tobias_fiberdata has joined #tripleo06:43
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Roles Listing using Mistral Action  https://review.openstack.org/36056906:50
marioshewbrocca: "i don't often ask for reviews, but when I do they are >1K lines"06:54
*** bvandenh has joined #tripleo06:55
*** rasca has joined #tripleo06:55
*** liverpooler has quit IRC06:56
hewbroccaIKR06:57
hewbroccathis is why...06:57
mariosthe https://review.openstack.org/#/c/254788/ is the parent for the sensu one ... i've previously +2d it but staring at it a bit in gitk cos I was away last week06:58
marioserr sorry parent of the fluentd one06:58
hewbroccaYeah they are in the unfortunate situation where their reviews touch every damn thing06:59
jaosoriormarios: can you review this one? https://review.openstack.org/#/c/359038/ only 14 lines :D07:01
mariosjaosorior: added to queue07:01
jtomasekhm, the sensu patch still did not address comments I made - add environment to capabilities-map.yaml, parameters have no labels etc. defined ...07:02
jtomaseksame applies to the other one07:02
*** leanderthal|afk is now known as leanderthal07:05
hewbroccajtomasek: blearrgh, OM07:06
hewbroccaOK07:06
*** tesseract- has joined #tripleo07:08
*** karthiks has quit IRC07:09
*** florianf has joined #tripleo07:11
*** shardy has joined #tripleo07:12
*** jpena|off is now known as jpena07:13
*** sshnaidm|afk is now known as sshnaidm07:15
*** ifarkas has joined #tripleo07:18
*** dciabrin has joined #tripleo07:20
hewbroccajtomasek: can you make sure to poke mmagr / paramite when he gets on07:21
hewbroccaand tell him to address those issues immediately07:21
jaosoriormarios: can you check out my comment on this patch you just +2ed? https://review.openstack.org/#/c/254788/ What do you think?07:22
mariosjtomasek: please add comment on the review07:22
jtomasekok, I've just started relying to the email Lars sent07:22
mariosjaosorior: looking07:22
*** bvandenh has quit IRC07:23
mariosjaosorior: but each service has its own subscription though07:24
mariosjaosorior: i mean it is a separate string (default) for each service07:24
*** saneax-_-|AFK is now known as saneax07:24
mariosjaosorior: oh you mean pacemaker vs base not 'for each service' (i saw the top level comment)07:25
jaosoriormarios: is it? Seems to me that pacemaker/ceilometer-agent-central.yaml is different than ceilometer-agent-central.yaml07:25
jaosoriormarios: yes07:25
jaosoriorI mean07:25
jaosorioris the same07:25
jaosoriorthose are the same07:26
jaosoriorand so07:26
jaosorioreach service has the same string in the pacemaker and non-pacemaker version07:26
mariosjaosorior: yeah so i guess we could just remove the param from the pacemaker version and use only the base, but again, i personally wouldn't block on it ... i may change my vote though since jtomasek says the requests are ui critical07:27
*** zoli_gone-proxy is now known as zoliXXL07:27
jtomasekmarios: the changes to that patch are simple and bring great value to configuring those services via GUI07:28
*** jprovazn has joined #tripleo07:29
d0ugalshadower: Hey, can you re-review https://review.openstack.org/#/c/359784/?07:30
d0ugalAnyone else got a moment to review ^?07:30
hewbroccamarios: ISTR there were 4 or 5 reviews outstanding last week that are critical for 9-10 upgrades07:31
hewbroccayou happen to know if those landed or not?07:31
shadowerd0ugal: sure. Can you see if my reply is satisfactory here? https://review.openstack.org/#/c/329039/2007:31
marioshewbrocca: not not yet there are still some workarouds07:31
d0ugalshadower: Sure, sorry, I missed that reply.07:31
*** numans has joined #tripleo07:31
marioshewbrocca: sec07:31
*** mhenkel has joined #tripleo07:31
*** panda|bbl is now known as panda07:32
hewbroccamarios: do we need e.g. FFEs for these?07:32
marioshewbrocca: no upgrades always gets ffe07:32
marioshewbrocca: i mean, we always land things after ff for upgrades because of the nature of, we are going to upgrade the thing we are yet to release07:33
d0ugaljaosorior: This is the counterpart to the other command you reviewed if you have a moment: https://review.openstack.org/#/c/359784/07:33
hewbroccamarios: OK, I see07:33
d0ugalshadower: Sounds good to me +2/+w07:33
hewbroccaAny impediments to getting those fixes landed?07:33
shadowerthanks!07:33
d0ugal+all-the-things07:33
shadowerindeed07:34
d0ugalshadower: I'll take a look at the last one now07:34
*** ebalduf has quit IRC07:34
shadowerd0ugal: thanks :-)07:34
mariosjaosorior: so, i added a comment at https://review.openstack.org/#/c/254788/ - I agree with your point. I do not intend to diminish it, just saying that this is a huuge patch, i can imagine the author/interested parties have been running and testing locally, and as you well know, such testing is expensive, time consuming, tedious etc etc. So whilst i agree with your comment, i personally wouldn't block sin07:35
mariosce it isn't functionally borked? Like we could tidy up the params after. BUT07:35
jaosoriord0ugal: why do you have to create an object store client? Isn't that available through the client_manager interface of OpenStack client?07:35
mariosjaosorior: it depends on the author and how willing/easy it is for them to make that change and test etc etc and not make any typos07:35
mariosjaosorior: but that is just imo07:35
d0ugaljaosorior: It isn't swiftclient, it is a much lighter client that only does what they need.07:36
*** dmacpher has quit IRC07:36
d0ugaljaosorior: I added a wee comment about that here: https://review.openstack.org/#/c/359784/20/tripleoclient/plugin.py07:36
*** abregman has joined #tripleo07:36
jaosoriormarios: sure thing. Only thing is that I think this amount of repetition adds to the technical depth and is stuff we have to maintain07:36
jaosoriord0ugal: alright07:36
*** jpich has joined #tripleo07:37
*** abregman has quit IRC07:37
*** abregman has joined #tripleo07:38
*** shardy has quit IRC07:39
jaosoriormarios: but yeah, I know it's a tough commit07:39
jaosoriorI didn't -107:39
*** tremble has joined #tripleo07:39
shadowerd0ugal: btw love the "overcast" plan name07:40
mariosjaosorior: ack :) it isn't particularly complex i think it is just a lot of files. i mean the basics is it adds service/sensu and env/config file for it, and then for each service adds the subscription. just a lot of services07:40
*** shardy has joined #tripleo07:40
d0ugalshadower: haha, thanks. I was waiting for somebody to comment on that :)07:40
d0ugalshadower: I decided to stop using the default after I found it hardcoded somewhere any it was missed by tests.07:41
shadowerd0ugal: yeah it makes a lot of sense07:41
*** flepied has joined #tripleo07:43
*** fzdarsky_ is now known as fzdarsky07:43
pandain case someone missed, today is the last day to register for free ATC pass07:46
*** dmacpher has joined #tripleo07:49
*** bvandenh has joined #tripleo07:50
*** nyechiel has joined #tripleo07:50
*** xinwu has joined #tripleo07:52
*** dbecker has quit IRC07:53
*** bkopilov_ has quit IRC07:57
*** bkopilov has quit IRC07:57
*** liverpooler has joined #tripleo07:57
*** xinwu has quit IRC07:57
*** gfidente has joined #tripleo07:57
*** gfidente has quit IRC07:57
*** gfidente has joined #tripleo07:57
*** liverpooler has quit IRC07:58
*** liverpooler has joined #tripleo07:58
*** zoliXXL is now known as zoli|afk08:00
*** bkopilov has joined #tripleo08:01
*** r-mibu has quit IRC08:02
d0ugalhrm, I thought tripleo-common was going to be released at 5.0 yesterday?08:04
d0ugalI see 2.1.0 was released08:04
*** athomas has joined #tripleo08:04
d0ugalThis means PyPI has 5.0 beta releases and a new 2.1 release08:04
d0ugalPyPI versions: 0.0.2, 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.3.1, 1.0.0, 2.0.0, 2.1.0, 5.0.0.0b1, 5.0.0.0b208:05
d0ugalshardy: ^08:05
*** chlong has quit IRC08:06
d0ugal... but it looks like 2.1 isn't a recent version from git, so now I'm less sure :)08:07
honzai'm having a hard time running tripleo-quickstart: the issue seems to be regarding the creation of a virsh pool. If it's there, it complains it's there.  If I delete it, it complains it's not there.08:13
honzathe only arg i've specified is the release08:14
*** lucas-dinner is now known as lucasagomes08:20
*** ohamada has joined #tripleo08:23
jaosoriorhonza: you might want to ask in the #oooq channel08:24
openstackgerritJulie Pichon proposed openstack/tripleo-specs: Update bug tag policy to match actual tag names  https://review.openstack.org/36347708:27
*** milan has joined #tripleo08:29
shadowershardy: on top of what d0ugal said, tripleo-validations did not seem to get a new release08:30
d0ugalshadower: Maybe it hasn't happened yet?08:30
*** rhefner has quit IRC08:31
d0ugalshadower: I am only seeing Mitaka releases for everything08:31
*** derekh has joined #tripleo08:31
shadowerd0ugal: I thought so, too but then I checked https://github.com/openstack/tripleo-heat-templates/releases08:32
shadowerwhich lists 2.1.0 as created 15 hours ago08:32
d0ugalshadower: That is also 2.1, which I think is Mitaka08:32
shadowerso something does seem up08:32
shadowerah okay08:32
d0ugalshadower: 5.0 is Newton, there is only a beta for that08:32
shadowerright08:32
d0ugal... I think!08:32
shadoweri.e. we could still get some patches in if it weren't for the CI08:32
d0ugalAye08:33
shardyd0ugal: no, we released mitaka yesterday08:33
*** r-mibu has joined #tripleo08:33
shardywe'll release 5.0.0b3 today if all goes to plan08:33
d0ugalshardy: okay, cool - thanks :)08:33
shardyshadower: ack, we'll release it today as part of newton-308:33
derekhsshnaidm: so I spend some time yesterday trying to find out why we were sometimes getting timeouts in ipmitool, I noticed you were also looking at ipmitool timeouts last week08:33
shadowershardy: btw just realised -- since tripleo-validations is already 5.0.0 (at the request of the release people), what should we call the final Newton version?08:34
* shadower didn't think about that at the time08:34
shardyshadower: I'll have to check with dhellmann, sounds like it's been considered a library, so we may have to increment from 5.0.008:35
derekhsshnaidm: it turns out the ipmi library we use on our fake bmc ( pyghmi) had a bug in it and it didn't work if the client host used the same UDP port twice08:35
shadowerright08:35
honzawait, is oooq just "tripleo quickstart"?08:35
shardyshadower: the problem is, libraries were frozen recently, so I'm not sure we'll be allowed to release again :(08:35
shadowerhonza: afaik yeah08:35
honzaugh08:35
shardyshadower: I'll chat with dhellmann and see what can be arranged08:35
derekhsshnaidm: so that ended up getting fixed in this patch https://review.openstack.org/#/c/362944/08:35
shardyI had to fight to not release tripleo-common in the same way08:35
shadowerhonza: "openstack on openstack quickstart" I think? Because things weren't confusing before08:35
derekhsshnaidm: now we need to updated the template we use in CI to take advantage of it08:36
derekhsshnaidm:  do you want to do it? I can show you what needs to happen08:36
honzashadower: and i'm here running around with google in my hand trying to find this elusive oooq project08:36
shadowershardy: ah, that's not good :-(08:36
shardyshadower: worst case we'll have to branch the repo early and handle patches as "stable" backports08:36
shadowerok08:36
shardyhonza: it's https://github.com/openstack/tripleo-quickstart08:37
*** xinwu has joined #tripleo08:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902708:40
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: [NO MERGE] Test Ceph RadosGW as replacement for Swift  https://review.openstack.org/35718208:41
gfidenteshardy, fwiw, I am investigating status of RGW submissions to reply on the list08:42
shardygfidente: ack, thanks08:42
gfidenteI want to get a better handle myself of how much work is left08:42
gfidentety :)08:42
*** xinwu has quit IRC08:43
*** zoli|afk is now known as zoli08:45
*** zoli is now known as zoliXXL08:45
*** yolanda has joined #tripleo08:45
sshnaidmderekh, yeah, I'm here08:46
*** dbecker has joined #tripleo08:49
d0ugaljpich: Hey, did you ever hit this? http://paste.openstack.org/show/565102/ (note line 10 and all the lines above it)08:50
sshnaidmderekh, which template are we talking about?08:50
derekhwhen CI creates a testenv the bmc has some software installed on it, to speed this up we have a image template with the packages pre-installed, so you need to recreate the template08:51
sshnaidmderekh, is it bmc_template image?08:51
derekhsshnaidm: if you do a nova image-list as the openstack-nodepool user you should see an image called bmc-template08:51
derekhsshnaidm: yes08:52
sshnaidmderekh, ok08:52
sshnaidmderekh, where is it constructed?08:52
derekhsshnaidm: its a vanilla centos cloud image with this script run on it http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/bmc-template/bmc-template.sh08:52
derekhsshnaidm: its contructed as part of "scripts/prepare-ovb-cloud.sh"08:52
derekhsshnaidm: while I guess should be extracted out so it can be run independently08:53
derekhIt should be just a matter of logging onto the rh1 undercloud08:53
derekhand running the code inside the if statement (line 73 here http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/prepare-ovb-cloud.sh)08:53
derekhBut you'll also need to run this command "nova image-delete bmc-template"08:53
derekhbefor the "nova image-create ..."08:53
derekhsshnaidm: ^^08:53
jpichd0ugal: Yes, just a sec08:54
jpichd0ugal: https://bugs.launchpad.net/tripleo/+bug/161128408:54
openstackLaunchpad bug 1611284 in tripleo "[tripleoclient] Wrong message displayed when running introspect_manageable_nodes workflow" [Medium,Fix released] - Assigned to Julie Pichon (jpichon)08:54
d0ugaljpich: aha! Thanks :)08:55
derekhsshnaidm: brb08:55
sshnaidmderekh, if ! nova image-show bmc-template08:55
*** numans has quit IRC08:55
jpichd0ugal: You're welcome!08:55
sshnaidmderekh, ok08:55
*** oneswig has joined #tripleo08:57
d0ugaljpich: so it turns out I was only hitting it because I hadn't activated my virtualenv, oops.09:00
*** karthiks has joined #tripleo09:00
sshnaidmderekh, afaiu pyghmi should be released for installing it with pip09:00
jpichd0ugal: boo :)09:01
lucasagomesderekh, fyi, pyghmi 1.0.9 does fix the problem of ipmi clients trying to reuse the UDP ports09:03
lucasagomesderekh, I put a patch to requirements bumping the g-r in OS https://review.openstack.org/#/c/363497/09:03
derekhlucasagomes: ack, thanks09:04
derekhsshnaidm: don't worry about the "if ! nova image-show bmc-template"09:04
derekhsshnaidm: that there to prevent us having to run the code if the image already exists09:05
derekhsshnaidm: in other words its there to stop us doing exactly what we want to do09:05
sshnaidmderekh, yeah, exactly)09:05
*** nijaba has quit IRC09:05
sshnaidmderekh, we need to recreate the image, right?09:06
derekhsshnaidm: so we need to manually run the code inside that if statement09:06
derekhsshnaidm: with an extra image delete09:06
derekhsshnaidm: yes to recreate the image09:06
derekhsshnaidm: the the new version of pyghmi will be in the images09:06
*** nijaba has joined #tripleo09:07
*** nijaba has quit IRC09:07
*** nijaba has joined #tripleo09:07
*** electrofelix has joined #tripleo09:07
sshnaidmderekh, afaik it's in all tenants there09:08
sshnaidmderekh, is it same image in all tenants, or we need to do it for each one?09:09
*** numans has joined #tripleo09:10
derekhsshnaidm: yes it looks like it (I checked the uuid of the image), somebody must have made it public09:10
derekhit may even have been me..............09:11
sshnaidmderekh, so I start deleting it, ok?09:11
openstackgerritMerged openstack/tripleo-quickstart: pin the undercloud-post and overcloud roles  https://review.openstack.org/35666909:12
derekhsshnaidm: are you read to run the "nova image-create --poll bmc-template bmc-template" command yet?09:12
*** dtantsur has joined #tripleo09:13
derekhsshnaidm: wait until that point to delete the old one, then it wont be missing for as long09:13
dtantsurMorning folks. My IRC bouncer went down yesterday evening (as well as my lab environment, sigh), so please repeat any pings.09:14
derekhdtantsur: ping09:14
sshnaidmderekh, does it create snapshot?09:14
*** abregman has quit IRC09:15
derekhsshnaidm: yes09:15
sshnaidmderekh, ok, running09:15
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702909:15
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add profile to write overcloud VIPs into /etc/hosts  https://review.openstack.org/35776209:16
dtantsurderekh, From #rhos-mgt icmp_seq=1 Destination Host Unreachable09:18
sshnaidmderekh,  No server with a name or ID of 'bmc-template' exists09:18
derekhsshnaidm: what did you run?09:18
sshnaidmderekh, nova image-create --poll bmc-template bmc-template09:19
*** chem has joined #tripleo09:20
derekhsshnaidm:  did you run the other code in the if statment to create the bmc-template VM ?09:20
sshnaidmderekh, no09:20
sshnaidmderekh, sorry, I don't follow - what is the order?09:21
derekhsshnaidm: this is whats needed http://paste.openstack.org/show/565107/09:21
derekhsshnaidm: so it creates a VM, which installs some packages on it09:22
derekhsshnaidm: then deletes the old snapshot and creates a new one09:22
sshnaidmderekh, sorry, I thought you mean firstly to run snapshot creating09:22
*** limao has quit IRC09:22
openstackgerritMartin André proposed openstack-infra/tripleo-ci: Add quotes around systemctl command in test  https://review.openstack.org/36351809:22
*** akrivoka has joined #tripleo09:22
derekhsshnaidm: ahh ok, no prob09:22
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enabled_services moved from ironic to ironic::conductor manifest  https://review.openstack.org/36352009:23
openstackgerritMichael  Henkel proposed openstack/tripleo-heat-templates: add composable services for Contrail  https://review.openstack.org/34679909:24
sshnaidmderekh, when booting it which network to use? ERROR (Conflict): Multiple possible networks found,09:26
openstackgerritMichael  Henkel proposed openstack/puppet-tripleo: Added Contrail Composable Services  https://review.openstack.org/34890509:26
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enabled iPXE by default and make it configurable  https://review.openstack.org/36214809:26
derekhsshnaidm: private09:27
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Generate composable service node_names lists  https://review.openstack.org/35506809:27
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Generate composable service node_names lists  https://review.openstack.org/35506809:28
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enabled iPXE by default and make it configurable  https://review.openstack.org/36214809:28
*** abregman has joined #tripleo09:28
sshnaidmderekh, hmm.. and what is $EXTNET here? it doesn't appear in the script09:29
*** yolanda has quit IRC09:30
derekhsshnaidm: hmmm, It must have been something at some stage, weird09:30
derekhsshnaidm: leave it out, I don't think its needed09:30
sshnaidmderekh, not to make the float ip?09:31
*** aufi has joined #tripleo09:32
sshnaidmderekh, why do we make float ip at all? we need only image there, isn't it?09:32
derekhsshnaidm: it should just work without it I think09:33
*** _milan_ has joined #tripleo09:33
derekhsshnaidm: so we can ssh to the instance to check when the script is done09:33
derekhsshnaidm: thats what the while loop is doing09:34
openstackgerritDmitry Tantsur proposed openstack/puppet-tripleo: Ironic: manage PXE environment for conductors  https://review.openstack.org/36110909:34
sshnaidmderekh, ok, so to wait a little09:34
derekhsshnaidm: this script is what being run on the instance "http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/bmc-template/bmc-template.sh"09:35
*** milan has quit IRC09:35
derekhsshnaidm: exactly we're waiting on this line to be done "touch /var/tmp/ready"09:35
sshnaidmderekh, ok, done, snapshot is ready, to continue to deleting?09:39
derekhsshnaidm: go for it09:40
*** chem has quit IRC09:41
sshnaidmderekh, done09:41
*** chem has joined #tripleo09:42
sshnaidmderekh, need to make it public now?09:42
derekhsshnaidm: great ;-)09:42
derekhsshnaidm: last thing to do, is to make the image public09:42
derekhsshnaidm: its not needed for CI but just so it can be used for the dev testenvs09:43
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: Refactor upgrade checks.  https://review.openstack.org/35775009:43
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: M/M upgrade sahara-api fails to restart.  https://review.openstack.org/35802209:43
sshnaidmderekh, hmm..  glance image-update --visibility public 1aabe923-041d-46c2-b5bf-1bfa2831741609:44
sshnaidm403 Forbidden: You are not authorized to complete this action.09:44
derekhsshnaidm: do it as admin09:44
derekhsshnaidm: all done, other users can see it now09:46
sshnaidmderekh, yep09:46
*** gfidente has left #tripleo09:46
derekh;-)09:46
*** gfidente has joined #tripleo09:46
sshnaidmderekh, thanks :)09:46
sshnaidmderekh, so now the timeouts should be ok?09:47
derekhsshnaidm: yes, I believe so(or at the very least there'll be less of them)09:47
sshnaidmok, I'll keep eye on it..09:48
*** pgadiya has quit IRC09:49
openstackgerritMartin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses  https://review.openstack.org/36354509:50
hewbroccais this the image caching thing?09:52
openstackgerritMartin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses  https://review.openstack.org/36354509:52
hewbroccaoh, no, pxe09:52
openstackgerrityuyafei proposed openstack/tripleo-common: Replace raw_input with input to make PY3 compatible  https://review.openstack.org/33712909:57
openstackgerritMerged openstack/tripleo-specs: Spec: Centralized logging support  https://review.openstack.org/35669109:59
*** oneswig has quit IRC10:02
openstackgerritMerged openstack/tripleo-heat-templates: Fix Ironic service bind IPs  https://review.openstack.org/36145910:03
*** ifarkas_ has joined #tripleo10:05
*** pgadiya has joined #tripleo10:07
*** mburned is now known as mburned_out10:08
*** jubapa has joined #tripleo10:08
*** zoliXXL is now known as zoli|lunch10:11
*** adarazs is now known as adarazs_lunch10:12
jaosoriorjistr, marios: Hey guys, I'm trying to use mod_ssl from apache, but for some reason it doesn't seem to get installed when I enable it in puppet (in my case by including ::keystone::wsgi::apache and setting ssl to true) am Imissing something?10:13
*** jubapa has quit IRC10:13
EmilienMgood morning10:15
EmilienMjaosorior: yeah puppet doesn't install the package (on purpose in puppetlabs apache)10:16
EmilienMjaosorior: ayoung had the same thing with mellon10:16
EmilienMyou need to manage the package yourself, I sugest by using ensure_package10:16
shardysshnaidm: Hi, do you have any bug references re failing periodic jobs?10:16
shardyhttp://status-tripleoci.rhcloud.com/10:17
EmilienMperiodic still failing?10:17
shardyI see we've not had a clean run in a few days, wondering how close we are to fixing that, or if we need to derive the release SHA's from some non-periodic job run10:17
sshnaidmshardy, not yet, it's planned, for now it's tracked in: https://etherpad.openstack.org/p/tripleo-ci-status10:17
sshnaidmshardy, the main problem is timeouts10:18
sshnaidmshardy, as you see in http://status-tripleoci.rhcloud.com/ almost every ha job fails because of timeout10:18
EmilienMcan we change the pipeline?10:19
EmilienMand remove ha job10:19
EmilienMfrom promotion pipeline10:19
EmilienMtemporary10:19
shardyYeah we've clearly got performance problems with rh1 which I know bnemec is looking into10:19
sshnaidmshardy, I try to make a workaround here: https://review.openstack.org/#/c/362904/10:19
jaosoriorEmilienM: that makes sense now. thanks!10:19
jaosoriorEmilienM: what time is it there?10:20
shardygood morning EmilienM btw! :)10:20
EmilienMjaosorior: 6.20 am10:20
jaosoriorholy crap10:21
jaosoriorthat's early10:21
EmilienMI'm an early bird today10:21
sshnaidmEmilienM, we can, but it's too radical step imho, we can just a limit the feature tests there - without overcloud delete and undercloudidempotent10:21
shardyEmilienM: I'd suggest we push a WIP patch to t-h-t which makes a change e.g to a comment, then recheck until it's all green & use the SHA's from that run and the parent of the WIP patch for the release?10:21
EmilienMshardy: yes10:22
shardyEmilienM: ack, I'll push a patch now10:22
jaosoriorEmilienM: is there a difference between just using package and using ensure_package?10:23
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: WIP testing do not merge  https://review.openstack.org/36356010:23
shardyjtomasek, shadower: since they're not yet tested in CI, what SHA's do you want for tripleo-ui and tripleo-validations?10:25
EmilienMjaosorior: yes, wieh ensure_package you can declare it multiple time without resource duplication10:25
EmilienMjaosorior: while package can be used once for a resource10:25
shardyis the current master HEAD OK?10:25
EmilienMno10:26
shadowershardy: yeah. I.e.: 3c031c68e63d584079ed6195a8516d6b2fe5d9c710:26
EmilienMwe need latest consistent10:26
jaosoriorEmilienM: good to nkow. Will use that then, thanks a lot man10:26
EmilienM59/91/599171ade4475709ddd8544e7dfd7ee4ba4ba2a2_54a9900510:26
shadowerhey folks, this seems to fix a CI issue: https://review.openstack.org/#/c/363545/10:26
EmilienMthis is the latest consistent ^10:26
shadoweris there a way to have a patch try it out to verify?10:27
*** abehl has joined #tripleo10:27
shardyEmilienM: sure, we can compare with the versions in the delorean-current used for the CI run on the patch I just posted10:27
shardyI'm just making sure there aren't any release blocker pending patches10:27
jtomasekshardy: yes, tripleo-ui master is fine10:29
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fix an autoclass reference and add missing pages to the toctree  https://review.openstack.org/34274710:30
EmilienMshardy: ok I thought you wanted to test all latest openstack packages10:30
EmilienMsshnaidm: go for limit10:30
shardyEmilienM: I don't see how we can, given that current-tripleo hasn't promoted in 7 days10:30
shardyEmilienM: I just want to get a snapshot from a green CI run so we know a good point-in-time for the current tripleo repos10:32
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702910:32
shardyclearly it would be better if we had a newer current-tripleo pin, we'll have to fix that ASAP before the rc1 release10:32
EmilienMshardy: fair enough10:32
EmilienMI got one patch yesterday with all green10:33
shardyIt needs to be a patch which doesn't make any changes though, so we can derive that the parent SHA is OK?10:33
sshnaidmshardy, sorry for nit, but what do you think about this? https://review.openstack.org/#/c/362904/10:33
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enabled iPXE by default and make it configurable  https://review.openstack.org/36214810:34
EmilienMshardy: perfect10:34
shardyOk https://bugs.launchpad.net/tripleo/+milestone/newton-3 is all green, now we can wait for https://review.openstack.org/#/c/363560/ to (hopefully) pass and base the release on that10:35
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Last round of modern authtoken update  https://review.openstack.org/36318210:36
EmilienMshardy: can you approve https://review.openstack.org/#/c/362980/ please?10:36
EmilienMshardy: great news10:36
EmilienMand also can you guys review https://review.openstack.org/#/c/362504/ please? all green10:37
jtomasekflorianf: you can merge you'r GUI patch now (+ tiny fix the other one and we can merge that one too) before shardy takes GUI master and releases it:)10:38
shardyjtomasek: if that WIP patch passes CI it's too late, but I'm guessing we'll need to do rechecks10:39
florianfjtomasek: did you +2 yet?10:40
jtomasekflorianf: yes10:40
jtomasekflorianf: and added comment to this one https://review.openstack.org/#/c/361561/10:40
florianfjtomasek: yep, just saw it10:41
*** ansiwen has quit IRC10:42
EmilienMjaosorior: I suggest you to read https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/lib/puppet/parser/functions/ensure_packages.rb10:44
dtantsurfolks, do you have a quick hint where I screwed up syntax in https://review.openstack.org/#/c/362148/5/puppet/services/ironic-conductor.yaml ?10:46
EmilienMjaosorior: wait in puppet ci we include ::apache::mod::ssl and it works10:47
jaosoriorEmilienM: huh?10:48
jaosoriorEmilienM: yeah, that's the puzzling thing. When I had read the code for the ::apache::mod it seems to do the package install10:48
*** paramite has joined #tripleo10:48
EmilienMjaosorior: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/mod/ssl.pp#L70-L7210:48
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui: Port plan-edit file listing to use Swift  https://review.openstack.org/36156110:48
EmilienMbut tripleo overcloud prevents to install packages10:48
EmilienMyou need to patch puppet elements10:49
EmilienMdprince wrote a provider in puppet tripleo to noop any Package resource iirc10:49
jaosoriorEmilienM: I see10:49
jaosoriorso we need to do the same for mod_auth_mellon I guess10:49
EmilienMbecause we want everything managed in the image10:49
EmilienMyes10:49
EmilienMmaybe you can confirm with him10:49
jaosoriorwell, that makes sense10:49
EmilienMbut I'm almost sure now10:49
dtantsur"dprince wrote a provider in puppet tripleo to noop any Package resource" WUT?10:50
dtantsurso this is why PXE configuration does not work, right?10:50
jaosoriorEmilienM: by the way10:50
jaosoriorEmilienM: for some reason, puppet-barbican isn't getting included in the images10:50
EmilienMhttps://github.com/openstack/puppet-tripleo/blob/master/lib/puppet/provider/package/norpm.rb10:50
EmilienMdtantsur: ^10:50
dtantsurthis is insane10:51
EmilienMjaosorior: is it in puppet elements?10:51
EmilienMiirc yes10:51
jaosoriorEmilienM: AFAIK yes10:51
dtantsurthat means that random puppet elements we include will just silently not work10:51
dtantsursocial, ^^ we need fox fix dpri...ee I mean puppet elements10:51
dtantsurs/fox/to/ dunno how I typed that10:51
hewbroccawe definitely need to fix dprince10:51
hewbroccathat's well established :D10:52
EmilienMdtantsur: no10:52
EmilienMpackages need to be managed in puppet element10:52
EmilienMnot in puppet10:52
EmilienMie: puppet-tripleo, puppet-nova, etc10:52
EmilienMthey need to be managed in tripleo puppet elements10:52
dtantsurEmilienM, when do you plan to fix all puppet modules in the world to comply with it? >_<10:52
EmilienMbecause we think tripleo is better with a prepared image10:52
EmilienMdtantsur: tomorrow morning10:52
EmilienMnot today lol10:52
* dtantsur is close to rage quit10:52
shardydtantsur: you can re-enable package install by passing EnablePackageInstall: true10:53
openstackgerritBabu Shanmugam proposed openstack/tripleo-heat-templates: OVN heat templates  https://review.openstack.org/30773410:53
dtantsurwell, if you ask for it... now we'll have tftp-server and ipxe-bootimgs on the overcloud10:53
shardydtantsur: https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.yaml#L22710:53
dtantsurI hope nobody -2s this request10:53
EmilienMdtantsur: -310:53
shardydtantsur: so, the context is we need to support deployment with only images, and no external connectivity10:54
florianfjtomasek: I updated https://review.openstack.org/#/c/361561/10:54
shardyso any new packages must be able to be built into the overcloud-full image10:54
dtantsurshardy, then why don't we *fail* with a reasonable message?10:54
jaosoriorshardy: where do I add stuff to overcloud-full?10:54
*** thrash|g0ne is now known as thrash10:54
florianfjtomasek: we need another reviewer. honza? ^10:54
shardyhowever, folks can enable package install if they want and use "thin" images, it's just not our default10:54
dtantsurit's a pure luck I saw this conversation, otherwise social and me would get gray hair today10:54
shardydtantsur: agreed, if we dont then please raise a bug, it needs fixing10:54
dtantsurshardy, sure, against what?10:55
shardydtantsur: raise it on the main tripleo launchpad and we'll decide where it needs to get fixed10:55
jtomasekflorianf: you can +2 this one in the meantime so it runs the CI https://review.openstack.org/#/c/360550/10:55
shardywe should have the puppet provider interpret that boolean, and raise an error if any package install is attempted10:55
dtantsur"Sorry, something just went wrong in Launchpad. "10:57
* dtantsur quits for real10:57
jaosoriorshardy: which element in tripleo-puppet-elements is for overcloud-full?10:57
shardyjaosorior: we include a bunch of different elements, what do you need to add?10:58
jaosoriorshardy: I need to add some packages10:59
jaosoriorshardy: mod_ssl for the TLS work and mod_auth_mellon for fedeartion (to start with)10:59
*** mburned_out is now known as mburned11:01
thrashjaosorior: overcloud-full is actually in instack-undercloud11:01
jaosoriorthrash: excellent, thanks man11:02
*** rhallisey has joined #tripleo11:02
shardyhttps://github.com/openstack/tripleo-puppet-elements/blob/master/elements/overcloud-controller/install.d/package-installs-overcloud-controller11:02
jaosoriorthrash: uhm... there doesn't seem to be much there, only glance installs, what's up there?11:02
*** tobias-fiberdata has joined #tripleo11:03
shardyyeah but probably it's best to add them to the overcloud-controller element11:03
thrashjaosorior: I was just looking at that... you might want to use overcloud-base instead. i11:03
shardyIIRC overcloud-full is somewhat unused now11:03
openstackgerritDmitry Tantsur proposed openstack/tripleo-puppet-elements: Install ipxe-bootimgs and tftp-server on controller  https://review.openstack.org/36357611:03
dtantsurshardy, correct ^^^?11:04
dtantsuralso a bug: https://bugs.launchpad.net/tripleo/+bug/161883411:04
openstackLaunchpad bug 1618834 in tripleo "Fail with an explicit error when a puppet module attempts to install a package" [Medium,Triaged]11:04
thrashjaosorior: or overcloud-controller11:04
shardydtantsur: yup, +2, thanks11:05
dtantsurnow I need some help please figuring our a syntax problem in https://review.openstack.org/#/c/362148/5/puppet/services/ironic-conductor.yaml presumably in the list_join part.11:05
thrashjaosorior: overcloud-controller might be better for you as well.11:05
*** tobias_fiberdata has quit IRC11:05
*** pkovar has joined #tripleo11:06
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-puppet-elements: Add mod_ssl to overcloud-controller element  https://review.openstack.org/36357811:06
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-puppet-elements: Add mod_auth_mellon to overcloud-controller element  https://review.openstack.org/36357911:06
jaosoriorthrash, shardy ^^11:06
jaosoriorah11:07
jaosoriorI missed something I guess11:07
shardydtantsur: what error does it give?11:08
thrashjaosorior: follow the same pattern. append '_package', then map in the package map11:08
dtantsurshardy, puppet fails when it can't find files from the package it expects to be installed (precisely, /usr/share/ipxe/undionly.kpxe)11:08
thrashalthough I suppose that's not *totally* necessary11:08
thrashjaosorior: but better to stick with the existing pattern here. :)11:09
*** fzdarsky is now known as fzdarsky|lunch11:09
dtantsurthrash, jaosorior, one day we might support non-RH distributions, then it will be required..11:09
openstackgerritMichael  Henkel proposed openstack/puppet-tripleo: Added Contrail Composable Services  https://review.openstack.org/34890511:09
openstackgerritMerged openstack/tripleo-ui: Use Mistral/Zaqar for folder-based plan-creation  https://review.openstack.org/36055011:09
dtantsure.g. all openstack packages don't have "openstack" prefix on debian/ubuntu11:09
dtantsuripxe-bootimgs is just ipxe, etc11:10
jaosoriorthrash: is the _package suffix needed?11:10
thrashjaosorior: so yeah. use the pkg-map. :)11:10
shardydtantsur: I mean the template, why do you think the syntax is broken, what error does heat give?11:10
thrashjaosorior: I would stick with using it.11:10
jaosoriorthrash: I didn't use it when I introduced the certmonger package there11:10
thrashjaosorior: that part is completely arbitrary. But use of the pkg-map is not.11:11
dtantsurah11:11
dtantsurshardy, "The Referenced Attribute (ControllerServiceChain role_data) is incorrect"11:11
*** bkopilov has quit IRC11:11
jaosoriorthrash: ok, fixing the commits11:12
openstackgerritMichael  Henkel proposed openstack/tripleo-heat-templates: add composable services for Contrail  https://review.openstack.org/34679911:13
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-puppet-elements: Add mod_ssl to overcloud-controller element  https://review.openstack.org/36357811:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-puppet-elements: Add mod_auth_mellon to overcloud-controller element  https://review.openstack.org/36357911:14
jaosoriorthrash, shardy, EmilienM ^^11:14
*** paramite has quit IRC11:14
shardydtantsur: it's a problem with IronicIPXEPort, heat doesn't like trying to join an integer into a string11:15
dtantsurshardy, hmm, this was one of my suspects. do you think it's better to just make it a string or somehow convert number -> string?11:16
shardydtantsur: the easiest fix is s/number/string in the parameter type11:16
shardyI'd just change the parameter type11:16
dtantsurack, thanks a lot11:16
shardyit is a bug though, I'll raise a heat bug so we can cast it internally11:16
shardydtantsur: FYI this is how I debugged it - I hacked your template so it could be created standalone:11:18
shardyhttp://paste.fedoraproject.org/417847/42242147/11:18
shardyheat stack-create id1 -f ironic_debug.yaml11:18
shardyheat output-show id1 role_data11:18
shardythere is another heat bug related to output validation, when that is fixed we'll surface a better error when this happens11:18
dtantsurshardy, wow, cool! mind writing a blog post on it? or is it obvious for people who have a clue about heat? :)11:18
openstackgerritMichael  Henkel proposed openstack/tripleo-heat-templates: add composable services for Contrail  https://review.openstack.org/34679911:19
*** mkovacik__ has joined #tripleo11:19
shardydtantsur: possibly the second thing, but I'll definitely think about it if you think it would be helpful to folks :)11:19
shardyglad it was a simple fix11:20
*** maeca1 has joined #tripleo11:20
dtantsurshardy, maybe at least a couple of lines in the troubleshooting guide :) I guess with time we'll get more and more folks who don't know heat well enough11:20
dtantsur(I hope at least)11:20
*** ramishra has quit IRC11:21
shardydtantsur: Yeah, hopefully soon the error the user sees will be a lot clearer though11:21
shardyhttps://bugs.launchpad.net/heat/+bug/159911411:21
openstackLaunchpad bug 1599114 in heat "Outputs aren't correctly validated" [Medium,In progress] - Assigned to Oleksii Chuprykov (ochuprykov)11:21
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enabled iPXE by default and make it configurable  https://review.openstack.org/36214811:21
shardyThat is the reason we need the extra step of creating the standalone template, it should just fail and the error exposed to the user11:21
dtantsuryeah, makes sense11:22
*** _milan_ has quit IRC11:22
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902711:22
shardyWhen that is fixed, the error you saw would be like "Output error: Items to join must be string, map or list not 8088"11:22
shardywhich probably would have led you straight to the problem11:23
*** ramishra has joined #tripleo11:23
dtantsurshardy, yeah... now I wonder how to delete the failed stack11:23
dtantsuraha, the third delete attempt worked11:23
*** shardy is now known as shardy_lunch11:24
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Allow for building specific images  https://review.openstack.org/36309511:25
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702911:25
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui: Port plan-edit file listing to use Swift  https://review.openstack.org/36156111:26
*** adarazs_lunch is now known as adarazs11:27
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902711:28
dtantsurhmm, folks, have you seen http://logs.openstack.org/76/363576/1/check/gate-tripleo-ci-centos-7-nonha-multinode/8877c14/console.html#_2016-08-31_11_24_33_633896 before?11:29
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Only ask for the overcloud-full image  https://review.openstack.org/36359211:29
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add barbican profile  https://review.openstack.org/35244711:29
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add barbican profile  https://review.openstack.org/35244711:30
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Deprecate 'overcloud image build' command  https://review.openstack.org/33767611:31
*** jbadiapa has quit IRC11:32
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add Barbican to the overcloud - WIP  https://review.openstack.org/35245811:32
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE - TESTING BARBICAN  https://review.openstack.org/35326211:32
*** zoli|lunch is now known as zoli11:36
jaosoriordtantsur: there seems to be something quite screwed today in the CI11:36
*** zoli is now known as zoliXXL11:36
dtantsursigh...11:36
jaosoriorhave seen that and other really weird stuff today :/11:36
jaosoriorpatches are failing like there's no tomorrow11:36
dtantsurI hope you guys do give me feature freeze exception for ironic services :) seems like there is no chance of landing anything substantial today11:36
dtantsurquestions continue :) is it possible to use {get_param: ..} in dport fields for firewall_rules like https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/ironic-conductor.yaml#L56?11:41
dtantsurI've just realized I need to open this port as well11:41
dtantsurthis = iPXE one11:41
honzaflorianf: looking11:41
jaosoriordtantsur: which dport field?11:42
dtantsurjaosorior, see the link for an example11:42
*** lucasagomes is now known as lucas-hungry11:43
florianfhonza: thx!11:43
jaosoriordtantsur: you should be able to do get_param. Depending on where you want to get that value from though11:43
florianfakrivoka: thanks for the review!11:44
florianfhonza: ^11:44
dtantsurjaosorior, ok thanks, I'll just try it11:44
akrivokaflorianf: yw :)11:44
jaosoriorEmilienM: where are we getting the endpoint for the keystone puppet provider in tripleo? from the ini file?11:46
*** zoliXXL is now known as zoli|brb11:47
*** coolsvap_ is now known as coolsvap11:47
mcorneagfidente: hi! quick one re https://review.openstack.org/#/c/362816/ - how can I activate the Ceph RGW ?11:49
gfidentemcornea, I am testing that locally as well11:49
*** sshnaidm is now known as sshnaidm|afk11:50
gfidentemcornea, but you'd need quite a few patches11:50
gfidentemcornea, puppet-ceph: https://review.openstack.org/#/c/363164/ https://review.openstack.org/#/c/347956/11:51
gfidentemcornea, puppet-tripleo: https://review.openstack.org/#/c/334081/ (it depends on two more changes for puppet-tripleo, see the tree)11:51
gfidentemcornea, tht: https://review.openstack.org/#/c/289027/5211:51
hewbroccaoh boy11:52
openstackgerritMerged openstack/tripleo-ui: Port plan-edit file listing to use Swift  https://review.openstack.org/36156111:52
hewbroccasshnaidm|afk: and derekh is there any possibility that pxe change to CI this morning is causing CI failures?11:52
mcorneagfidente: ok, they're quite a lot,checking them11:53
colonwqIt is a long thread which pulling in a lot of code.11:53
gfidentemcornea, are you okay sharing the env where I am doing it otherwise?11:53
*** rhallisey has quit IRC11:54
mcorneagfidente: yeah, that works as well but I'd prefer to check them first so I get familiar with what's happening there11:55
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enabled iPXE by default and make it configurable  https://review.openstack.org/36214811:55
*** oshvartz has joined #tripleo11:56
*** mhenkel has quit IRC11:56
gfidentemcornea, ack thanks for helping11:57
jaosoriorgfidente: I had asked you this before but I can't find it again :( where did the replacement from network name to IP took place?11:58
*** rhallisey has joined #tripleo11:59
derekhhewbrocca: whats failing, readin back12:01
*** mkovacik__ is now known as milan12:01
jaosoriorderekh: everything12:01
*** panda is now known as panda|lunch12:01
jaosorioror at least it has been a while since I've seen the ovb patches or the multinode patches pass12:01
gfidentesshnaidm|afk, derekh, jaosorior the multinode job is failing with what seems to be an ipv6 issue in infra12:02
gfidentessh: connect to host 2001 port 22: Invalid argument12:02
gfidentethe command is actually an scp against 2001:4800:1ae1:18:f816:3eff:fe02:1d8912:03
gfidenteso this is "a problem of brackets" it seems12:03
hewbroccathat never happened before12:03
shadowergfidente: https://review.openstack.org/#/c/363545/12:03
gfidentehttp://logs.openstack.org/12/357312/10/check/gate-tripleo-ci-centos-7-nonha-multinode/0cbb93c/console.html#_2016-08-31_11_23_40_07966812:03
derekhhewbrocca: did you mean ipmi change this morning (the one I was talking sshnaidm|afk through?)12:04
shadowergfidente: that patch has a fix12:04
gfidenteshadower, ack +1 :)12:04
gfidentederekh, ^^12:04
shadowergfidente: it's mandre's work :-)12:04
*** morazi has joined #tripleo12:04
gfidentecan we merge it?12:04
gfidentelooks like it made multinode green so I wouldn't wait more12:04
*** jaosorior has quit IRC12:04
derekhgfidente: jaosorior: hewbrocca if the non ovb jobs are failing then there is no chance the update to pyghmi had anything to do with it12:04
hewbroccaOK12:05
mandregfidente: hold on, there is more scp invocations that I'm fixing right now12:05
hewbroccaI'm just automatically nervous about any CI change12:05
gfidentemandre :)12:05
shadowerhewbrocca: what's the worst that can happen12:05
hewbroccawait12:05
gfidentesorry you got into the brackets spirale as well12:05
hewbroccathese are real bugs?12:05
hewbroccain that case, hurray for CI actually working like it's supposed to12:05
*** jaosorior has joined #tripleo12:06
*** jayg|g0n3 is now known as jayg12:06
shadowerhewbrocca: nope, atleast the ipv6 thing is a CI bug I think12:06
hewbroccaOh12:07
*** rodrigods has quit IRC12:09
*** rodrigods has joined #tripleo12:09
*** zoli|brb is now known as zoli12:11
*** zoli is now known as zoliXXL12:11
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service  https://review.openstack.org/35350612:12
*** dtantsur is now known as dtantsur|bbl12:12
*** jpena is now known as jpena|lunch12:12
*** dprince has joined #tripleo12:13
*** shardy_lunch is now known as shardy12:15
*** tobias-fiberdata has quit IRC12:20
*** pradk has joined #tripleo12:20
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind to FQDNs instead of IPs  https://review.openstack.org/36361612:22
hewbroccafolks, special attention for 353506 ^^^ please?12:22
hewbroccamarios, jaosorior ^^^12:22
*** tobias-fiberdata has joined #tripleo12:23
socialow, step in right direction \o/12:23
derekhjaosorior: gfidente so there seems to also be another problem in addition to the non ovb jobs failing because of IPv6 problems12:24
derekhjaosorior: gfidente all of the OVB jobs seem to be timeing out after 3 hours12:24
openstackgerritKarthik S proposed openstack/tripleo-heat-templates: Added environment file for SR-IOV deployment  https://review.openstack.org/36361712:24
openstackgerritMartin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses  https://review.openstack.org/36354512:24
gfidentederekh, yeah I've seen local deployments take much longer when we added the part to create the endpoints in puppet12:25
derekhgfidente: jaosorior: this is different12:25
jaosoriorgfidente: this didn't start with the endpoints in puppet.12:25
derekhgfidente: they are all timing out before anything tripleo scripts run on this at all12:25
jaosorioractually stuff had been passing even with that12:26
derekhits during the zuul setup12:26
derekhe.g.12:26
derekhhttp://logs.openstack.org/81/359681/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/b9b1bd4/_zuul_ansible/ansible_log.txt12:26
jaosoriorderekh: yeah, that's what I've noticed, but I didn't know where in the steps did that happen12:26
derekh2016-08-31 07:02:35,056 p=25417 u=zuul |  TASK [zuul_runner with 10744 second timeout] ***********************************12:26
derekh2016-08-31 10:06:40,232 p=27103 u=zuul |  Using /tmp/tmpGBkMqI/ansible/ansible.cfg as config file12:26
derekhjaosorior: where did those 3 hours go12:26
openstackgerritRyan Brady proposed openstack/tripleo-common: Wire in jinja templating for custom roles  https://review.openstack.org/36246512:27
derekhseems to be during a rsync zuul_ansible does to set stuff up12:27
derekhpabelanger: ^^ any ideas ?12:27
derekhpabelanger: could the AFS server be banjaxed ?12:28
weshaysshnaidm|afk,  periodic tripleo is failing on http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/37fe6d7/logs/undercloud/var/log/undercloud_install.txt.gz12:28
*** sshnaidm|afk is now known as sshnaidm12:29
EmilienMjaosorior: I don't get the endpoint question12:29
jaosoriorEmilienM: was asking about the provider we use for keystone12:29
jaosoriorwhen provisioning12:29
jaosoriorbut it seems we get it from keystone.conf12:29
*** trown|outtypewww is now known as trown12:29
jaosorioras far as I can see12:29
sshnaidmweshay, it's from yesterday12:30
sshnaidmweshay, I don't see results of today still12:30
pabelangerderekh: looks like something in the job failed, and 3 hours later zuul killed the job.12:30
EmilienMjaosorior: sorry, I still dont get the question12:30
sshnaidmweshay, this problem was solved yesterday: https://bugs.launchpad.net/tripleo/+bug/161841812:30
openstackLaunchpad bug 1618418 in tripleo "CI: periodic jobs fail on undercloud install because "AttributeError: 'module' object has no attribute 'RegionInvalidationStrategy'"" [High,Fix released]12:30
EmilienMjaosorior: you want to know how the provider get credentials?12:31
jaosoriorEmilienM: yes12:31
EmilienMah, so ask it :)12:31
EmilienMcredentials are in /root/openrc12:31
jaosoriorEmilienM: that's what I thought I had asked. only thing that I don't call it credentials.12:31
EmilienMno prob12:31
jaosoriorEmilienM: but, we don't have an openrc12:31
weshaysshnaidm, thanks!12:31
jaosoriorin tripleo, or do we?12:31
EmilienMif you don't, puppet will find the resources in the catalog, with the admin resource12:32
derekhpabelanger: ahh ok, so that log file holds logs for setup and teardown?12:32
EmilienMjaosorior: keystone::roles::admin12:32
pabelangerderekh: not sure I follow12:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Bind to FQDNs instead of IPs  https://review.openstack.org/36361612:33
jaosoriorEmilienM: thanks man12:34
*** openstackgerrit has quit IRC12:34
derekhpabelanger: the entrys in the ansible log file for from 07:02 too 10:06 http://logs.openstack.org/81/359681/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/b9b1bd4/_zuul_ansible/ansible_log.txt12:34
*** openstackgerrit has joined #tripleo12:34
EmilienMjaosorior: admin token is disable12:34
EmilienMjaosorior: let me find the code12:34
derekhpabelanger: I origionaly thought it was stuck for 3 hours, but if the same log file is used for setup and teardown (at the start and then the end of the job) then that explains it12:35
pabelangerderekh: ya, the _zuul_ansible folder is logs from the zuul-launcher server, which invoked the job12:35
pabelangerderekh: zuul_runner is an ansible async task, which we wait until the timeout value before moving to the next task12:36
dprinceplumgrid will be broken in the release unless this lands https://review.openstack.org/#/c/338284/12:36
dprincehttps://review.openstack.org/#/c/338287/12:36
dprinceditto for contrail ^^12:36
EmilienMjaosorior: wait, i think we use admin token12:37
EmilienMjaosorior: sorry I was confused with puppet CI12:37
*** zoliXXL is now known as zoli|brb12:37
jaosoriorEmilienM: but where does the auth_url come from?12:37
pabelangerderekh: So, based on the log you posted. zuul-launcher ran http://logs.openstack.org/81/359681/6/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/b9b1bd4/_zuul_ansible/scripts/59964d4560064a7bbc1f61f170493006.sh async, then 3 hours later killed the job because of the timeout12:37
EmilienMjaosorior: keystone.conf12:37
jaosoriorI see12:37
EmilienMchem: should we use ::keystone::disable_admin_token_auth in tripleo?12:37
EmilienMI don't remember all the story but I thought the admin token was removed in keystone12:38
chemEmilienM: sorry, without any digging I can't reply to this one.12:38
chemEmilienM: I can dig though12:38
derekhpabelanger: so its not zuul-launcher itself that took 3 hours, ok so now I need to find out what else could be going on12:38
chemEmilienM: what's the matter ?12:38
pradkshardy, EmilienM, so the redis issue is fixed with dprince's patch merged https://review.openstack.org/#/c/357344/ .. the quickstart images were a few days old so it was still happening. But if i manually pull in this patch all is well with redis12:39
EmilienMchem: jaosorior is asking how credentials are given to our puppet provider to manage keystone resources12:39
pabelangerderekh: correct12:39
EmilienMchem: and it seems like in tripleo we don't use openrc file12:39
pradkshardy, confirmed locally12:39
*** liverpooler has quit IRC12:39
EmilienMchem: we seem to use admin token style12:39
chemEmilienM: oula, old stuff12:39
EmilienMpradk: excellent. Why do we have it in CI?12:40
EmilienMchem: yeah12:40
EmilienMchem: i'm going to fix that thing :)12:40
chemEmilienM: I didn't think it was still supported at all12:40
EmilienMme neither12:40
pradkEmilienM, not sure what you mean.. have what in CI?12:40
chemEmilienM: which part of tripleo, undercloud/overcloud ?12:40
shardypradk: ack, good news, thanks for confirming!12:41
EmilienMpradk: the error I showed you yesterday with redis / gnocchi was in our upstream CI12:41
EmilienMchem: overcloud12:41
EmilienMchem: but prob both12:41
EmilienMshardy: I'm still not sure gnocchi actually works in CI, I tested it with pingtest and it failed.12:42
EmilienMshardy: I'll work on it today to confirm.12:42
derekhpabelanger: the console is useless stdout isn't being synced so I can't see how far the script got, is it possible to add my key to a running VM so I can take a look?12:42
EmilienMjaosorior, chem: this thing is not urgent, i'll look a bit later, on my todo12:43
*** rlandy has joined #tripleo12:44
jaosoriorEmilienM: sure12:44
pradkEmilienM, may be your scenario was not initializing haproxy ? the ci on other patches passes fine12:44
EmilienMjaosorior: do you know if admin token is deprecated or removed in keystone?12:44
jaosoriorEmilienM: we shouldn't be using it12:44
EmilienMpradk: show me where it pass please.12:44
pradkEmilienM, initializing correctly i mean.. i donno why your scenario fails12:44
jaosoriorAFAIK12:44
EmilienMwhat do you mean by initializing12:45
pradkEmilienM, the key is tripleo::haproxy::redis_password should be set which is done in puppet/services/haproxy.yaml templates12:45
*** lucas-hungry is now known as lucasagomes12:45
pradkEmilienM, can you check if this is set locally for you when your scenario is deployed?12:46
*** ifarkas has quit IRC12:46
EmilienMpradk: I can check but tell me where you saw it working in CI12:46
*** bkopilov has joined #tripleo12:46
pradkEmilienM, any recently merged patch see the logs.. for example http://logs.openstack.org/62/361362/8/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/c07ec11/logs/overcloud-controller-2/var/log/gnocchi/metricd.txt.gz12:46
*** ifarkas_ is now known as ifarkas12:47
pradkEmilienM, see that gnocchi joins coordination as expected12:47
pradk2016-08-30 15:17:06.504 27404 INFO gnocchi.cli [-] Joined coordination group: gnocchi-scheduler12:47
hewbroccalarsks: are you guys ready to merge?12:47
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add NetApp Manila driver integration and tidy up generic  https://review.openstack.org/35401912:47
EmilienMpradk: ok thanks12:47
EmilienMpradk: i'll continue the tests today.12:47
*** jdob has joined #tripleo12:48
*** liverpooler has joined #tripleo12:49
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Default haproxy listeners activation on hiera service _enabled  https://review.openstack.org/36279812:49
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Add Ceph RGW profile  https://review.openstack.org/33408112:49
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Add Ceph RGW listener to HAProxy  https://review.openstack.org/36281612:49
*** zoli|brb is now known as zoli12:49
*** zoli is now known as zoliXXL12:49
pradkok12:50
*** abehl has quit IRC12:51
*** jefrite has quit IRC12:51
larskshewbrocca: well, *I* think so, but it looks like 254788 has developed a -1, and someone asked for changes on my review this morning. 353507 still seems to be in good shape.12:51
larskshewbrocca: the -1 is from dprince, asking that we drop the description on several of the parameters so that they don't show up in the ui.  I'll go ahead and make that change right now...12:52
shardylarsks: FYI I moved tripleo-opstools-centralized-logging to RC1 so we have a little more time to land this12:53
openstackgerritKarthik S proposed openstack/tripleo-heat-templates: Added environment file for SR-IOV deployment  https://review.openstack.org/36361712:53
shardylarsks: if you'd like to drop a mail to the FFE/newton-3 thread with a list of outstanding patches that would be good12:54
larsksshardy: Thanks, I'll do that.12:54
dprincelarsks: +2 on your fluent patch...12:54
larsksdprince: \o/12:54
dprincelarsks: and yeah, for sensu I'd just like the descriptions gone. Easy fix I think and it matches how we did Fluent12:55
EmilienMshardy: fyi https://review.openstack.org/36362912:55
larsksdprince: yeah, no problem. Tackling that right now.12:55
hewbroccajtomasek: did you have anything on the ops-tools patches12:55
hewbroccaw/r/t parameter description etc.12:56
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Install tripleo-admin ssh keys on CI nodes  https://review.openstack.org/36363012:56
*** jcoufal has joined #tripleo12:56
*** jefrite has joined #tripleo12:56
derekhOK, all tripleo OVB jobs are timing out after 3 hours I got now idea why until we can get onto a running ci instance and have a look12:56
jtomasekdprince: the parameters should have labels though12:57
jtomasekdprince: and added environments should get added in capabilities-map12:57
*** abehl has joined #tripleo12:57
hewbroccaderekh: what has changed?12:58
*** tzumainn has joined #tripleo12:58
dprincejtomasek: how would you like them then. Is a name a label?12:58
jtomasekdprince: name is like an ID, whereas label is what you'd expect to get in the form as a label12:59
*** fzdarsky|lunch is now known as fzdarsky12:59
jtomasekdprince: in general, I'd love to see all parameters having as much information provided as HOT spec allows (constraints etc.)13:00
*** egafford has joined #tripleo13:00
derekhhewbrocca: the image change we made this morning, and god know what else I havn't been following along13:00
dprincejtomasek: for "internal" parameters we started not including descriptions13:00
dprincejtomasek: that is a convention but it is all we've got ATM13:00
*** jbadiapa has joined #tripleo13:01
hewbroccadprince: we haven't changed anything else in CI since last week, have we?13:01
*** pradk has quit IRC13:01
jtomasekdprince: ok, that somewhat makes sense, hope there are not many deviations from that rule, so it is actually usable:)13:01
derekhhewbrocca: this appears to be related to something early in the process so could something out of our control completly13:01
hewbroccameh13:01
dprincejtomasek: if you want descriptions on everything. Then we could do that but there are quite a few missing descriptions at this point13:01
hewbroccapabelanger: you guys didn't make those shade changes on nodepool yet did you?13:01
dprincejtomasek: and labels I think are only for Heat parameter groups. Not for individual parameters13:01
derekhhewbrocca: dprince for example the image we use is rebuilt nightly, something my have gone into the zuul ansible scripts13:02
dprincehewbrocca: I have been trying to slow the rate of CI changes unless they are critical13:02
dprincehewbrocca: don't think we've changed a lot13:02
jtomasekdprince: hm, HOT spec defines label: http://docs.openstack.org/developer/heat/template_guide/hot_spec.html#parameters-section13:02
jtomasekdprince: and afaik the heat-validate output includes those labels for parameters which have them defined, so we use them in GUI13:03
hewbroccapabelanger: was talking about landing some shade patches to hopefully fix our FIP problem13:03
hewbroccaand doing a nodepool restart13:03
hewbroccaI don't know when that was going to happen though13:03
dprincejtomasek: cool. We aren't using them much13:03
dprincejtomasek: Heat may add them for you automatically based on the name13:04
dprincejtomasek: I guess my thought here is it is too late in the game to call out adding labels for this release13:04
derekhdprince: the console logs show nothing tripleo rated running at all, it seems to be stalling as part of the workspace setup so I don't think its something todo with the tripleo-ci  scripts (at least no likely)13:04
jtomasekdprince: I know, I am trying to get them in as much as possible because it is a big deal for GUI UX. And since those are new templates... it is better to add that now then never doing it later:)13:04
dprincejtomasek: we aren't using them in t-h-t. If you are getting them it is automatic13:04
jtomasekdprince: yeah, I am going to prepare the params update requirements for next summit discussion13:05
pabelangerhewbrocca: today we are planning on restarting nodepool13:05
jtomasekdprince: ok13:05
dprincejtomasek: larsks: so just to be clear. I'd recommend just dropping the descriptions for now... and we can have the debate about adding 'label's for the 'O' release13:06
larsksdprince: we're on the same page.13:06
shardydprince, jtomasek: If the label isn't specified it defaults to the parameter name13:06
*** zoliXXL is now known as zoli|brb13:07
jtomasekdprince: regarding the capabilities-map update, If the patch does not do it, I will have to do it myself later as it will be required for GUI - and it is much better when it is added by someone who actually understands the usage of that specific environment13:07
shardyso personally I don't see it as super critical unless we've got some unintuitive names?13:07
dprinceshardy: ack, I figured. sounds like jtomasek wants them everywhere or has UX ideas to improved them13:07
larsksjtomasek: the capabilities-map update is already there.13:08
jtomaseklarsks: oh, cool, I didn't check sorry13:08
* jtomasek takes a look13:08
shardySure, I'm not opposed to adding them, providing we're not just doing a hard-coded conversion to add spaces to the parameter name13:08
larsksjtomasek: but don't review, because I'm uploading a rebase in just a few minutes...13:08
jtomasekok13:09
hewbroccapabelanger: OK, that's not the problem then13:09
hewbroccablearrgh13:09
hewbroccapabelanger: thanks, sorry to bother you13:09
pabelangerhewbrocca: which problem, the job timeout?13:09
derekhpabelanger: can you ssh to a running ci node and see where its stuck?13:09
hewbroccapabelanger: yeah13:10
jtomaseklarsks: looks good, thanks for updating it13:10
openstackgerritMerged openstack/instack-undercloud: Enable modern authtoken for Neutron and Glance  https://review.openstack.org/36298013:11
pabelangerjenkins   9165  0.0  0.0 157976  3956 ?        S    11:29   0:05 git-remote-http http://zm01.openstack.org/p/openstack/horizon http://zm01.openstack.org/p/openstack/horizon13:11
pabelangerlooks like the job hung on git clone13:12
pabelangerswitching to #openstack-infra, there was some reports of git issue today13:13
jpichd0ugal: It looks like I may have broken something, would you know what the recommendation should be to resolve https://bugs.launchpad.net/tripleo/+bug/1618877 ? I guess it's because there hasn't been a promoted common yet, but there was a promoted client, or something like that?13:13
openstackLaunchpad bug 1618877 in tripleo "problem importing instackenv.json" [Undecided,New]13:13
*** maeca2 has joined #tripleo13:14
*** maeca1 has quit IRC13:15
gfidenteshardy, to do a release of tripleo-puppet-elements13:17
*** milan has quit IRC13:17
*** rhallisey has quit IRC13:18
*** jpena|lunch is now known as jpena13:18
gfidenteshardy, oh we need opentack/releases, fine will push it13:18
shardygfidente: we're planning to release everything for newton-3 imminently13:18
gfidenteshardy, ack holding on13:19
shardyas soon as we have a consistent set of SHA's from CI to use13:19
shardygfidente: what's the commit you need, to ensure we get it?13:19
gfidentehttps://github.com/openstack/tripleo-puppet-elements/commit/343f93f2a259b25ed5abe4b27bb3aa9070c92b0213:19
shardygfidente: ack, thanks13:20
gfidentety :)13:20
*** paramite has joined #tripleo13:21
*** lblanchard has joined #tripleo13:23
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service  https://review.openstack.org/35350613:23
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Availability monitoring agents support  https://review.openstack.org/25478813:23
*** coolsvap is now known as _coolsvap_13:23
larsksdprince: hewbrocca: jtomasek: rebased, with requested changes13:24
hewbroccawoowoo lars13:24
hewbroccawoowoo larsks13:24
*** oshvartz has quit IRC13:24
*** masco has quit IRC13:24
hewbroccadprince jtomasek can we get us some +2s on the ops 2ls13:24
larsksfor the record, there are three: https://review.openstack.org/#/c/353506/ https://review.openstack.org/#/c/254788/ https://review.openstack.org/#/c/353507/13:26
larsksThanks all!13:26
*** oshvartz has joined #tripleo13:26
*** Goneri has joined #tripleo13:27
*** flepied has quit IRC13:28
*** akuznetsov has joined #tripleo13:30
hewbroccadprince: ^^^13:31
dprincehewbrocca: ack13:32
*** panda|lunch is now known as panda13:33
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Document Third Party CI and developer usage  https://review.openstack.org/36000713:34
*** cdearborn has joined #tripleo13:34
*** ansiwen has joined #tripleo13:34
jristanyone comfortable +2 on this? it has two 2s and two 1s13:35
jristhttps://review.openstack.org/#/c/344140/13:35
openstackgerritHarry Rybacki proposed openstack/tripleo-docs: Fix minor doc errors  https://review.openstack.org/36366013:35
jristand some more eyes on this? https://review.openstack.org/#/c/361712/ it has one +2 and one +113:35
jrist1 +2 and 2 +1 https://review.openstack.org/#/c/360593/13:36
*** jeckersb_gone is now known as jeckersb13:36
jristthese are minor changes13:36
EmilienMjrist: both gate-tripleo-ci-centos-7-ovb-ha and gate-tripleo-ci-centos-7-ovb-nonha are red13:37
d0ugaljpich: Looking13:38
jristEmilienM: oh13:38
jristEmilienM: I'm sorry13:38
jristEmilienM: I didn't even notice13:39
d0ugaljpich: I'm not sure, I've never hit this situation before.13:39
jristEmilienM: what does POST_FAILURE mean?13:39
jristEmilienM: oh you ran some rechecks?13:39
EmilienMjrist: zuul failed to report the job status, which is because job timeouted13:39
jrist:(13:39
jpichd0ugal: Ok, thanks for looking. I'm going to create a new undercloud and see if I can reproduce, to begin with13:40
d0ugaljpich: Okay, sounds like a good start. Hopefully it is just unlucky timing - if package updates are available now they should be able to update with yum and re-run the undercloud install.13:40
rbradyshardy: the mistral custom roles patch is ready for another review when you get a chance: https://review.openstack.org/#/c/362465/13:41
shardyrbrady: awesome, thanks!13:41
shardyI'm planning to rebase the jinja2 patches today so will test them with the new action :)13:41
*** pkovar has quit IRC13:42
jpichd0ugal: Yeah, I'm hoping so! It sounds like something that should resolve itself shortly, but if it means no one can register nodes on a new undercloud right now it kinda sucks so I'll poke a little13:42
*** pkovar has joined #tripleo13:44
*** liverpooler has quit IRC13:44
*** wgd3 has joined #tripleo13:44
d0ugaljpich: Good luck!13:45
jpichd0ugal: Ta!13:45
EmilienMbnemec: can you +A or not https://review.openstack.org/#/c/362904/ please ?13:48
EmilienMbnemec: you had a comment13:48
EmilienMsshnaidm: once it's merged, any way to run periodic job? ^13:48
sshnaidmEmilienM, yeah, I'll trigger them13:49
*** zoli|brb is now known as zoli13:49
*** paramite has quit IRC13:49
hewbroccaEverything is timing out in CI right now13:49
*** zoli is now known as zoliXXL13:50
hewbroccadue to some infra git problem, pabelanger thinks13:50
EmilienMyeah I noticed last night13:50
EmilienMpuppet CI was all red13:50
pabelangerIf I had to guess, we are having networking issues in rackspace13:50
hewbroccadamn private companies13:51
*** dtantsur|bbl is now known as dtantsur13:51
sshnaidmpabelanger, is this related to fact we have periodic jobs queued for a 7 hr 51 min already?13:52
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types  https://review.openstack.org/36367413:54
*** ayoung has joined #tripleo13:54
*** anbu__ has quit IRC13:55
openstackgerritJiri Stransky proposed openstack/puppet-tripleo: Write restart flags to restart services only when necessary  https://review.openstack.org/35851013:55
EmilienMpanda: https://review.openstack.org/#/c/363674/1/toci_gate_test.sh13:55
EmilienMit won't work13:55
*** liverpooler has joined #tripleo13:55
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Restart only services that need it  https://review.openstack.org/35851113:55
EmilienMpanda: because the case will stop in ha)13:55
*** milan has joined #tripleo13:55
EmilienMthe jobname contains ha13:55
dtantsurshardy, may I use your brilliant brain again re https://review.openstack.org/#/c/362148/7/puppet/services/ironic-conductor.yaml ?13:56
pandaEmilienM: isn't there  a for loop for all the keywords in job type ?13:57
shardydtantsur: what's up?13:57
EmilienMpanda: mhh, maybe13:57
dtantsurshardy, I end up with http_url = http://ctlplane:8088 even though in other places the same {get_param: [ServiceNetMap, IronicNetwork]} results in a proper IP...13:57
EmilienMpanda: have you executed the experimental job?13:57
*** pgadiya has quit IRC13:57
shardydtantsur: I think that is because it should be IronicApiNetwork, sec13:58
shardydtantsur: Hmm, actually no, wait13:58
shardydtantsur: the problem is we do an exact map value substitution13:58
shardyand this is a substring13:59
shardyso foo: ctlplane would end up with foo: <ip>13:59
pandaEmilienM: not yet, spinning now13:59
shardydtantsur: it seems that http_url is the conductor endpoint?13:59
shardy(even if it's not something we put into keystone)14:00
*** adarazs is now known as adarazs_afk14:00
shardydtantsur: or, in other words, can we express this formatted URL as an endpoint?14:00
*** links has quit IRC14:00
*** ebalduf has joined #tripleo14:00
dtantsurshardy, yes, http_url is from conductor (not a VIP, just its IP). it should not be tagged if it matters14:01
EmilienMpanda: let's watch it in zuul14:01
shardydtantsur: Ok, I'm thinking it might be better to do something like:14:01
EmilienMpanda: I don't see it :(14:01
pandaEmilienM: so, the gerrit review will check itself with its own code ?14:01
shardy{get_param: [EndpointMap, IronicInspectorInternal, uri]}14:02
jaosoriorayoung: figured out that the issue you had with mod_auth_mellon was that tripleo overrides the package installs in puppet14:02
EmilienMpanda: so my patch was merged but zuul not restarted yet https://review.openstack.org/#/c/362966/14:02
dtantsurshardy, you mean IronicApiInternal?14:02
jaosoriorayoung: so we need to add mod_auth_mellon to the overcloud image.14:02
shardythen we can do the url formatting in the endpoint map, and stuff like changing the port or protocol can be handled in a standard way14:02
EmilienMpanda: I don't understand your question14:02
*** pradk has joined #tripleo14:02
shardydtantsur: No, I mean add an endpoint for the endpoint related to IronicIPXEPort14:02
jaosoriorayoung: submitted a fix to get it into the image https://review.openstack.org/#/c/363579/14:03
pandaEmilienM: add a post pipeline script to restart zuul :)14:03
dtantsurshardy, I'm a bit lost, we don't have an endpoint for IronicIPXEPort...14:03
shardybecause this isn't pointing at the Ironic API is it?14:03
jaosoriorayoung: and did the same for mod_ssl since I need it https://review.openstack.org/#/c/363578/214:03
ayoungjaosorior, excellent14:03
dtantsurshardy, no, it's not14:03
shardydtantsur: I'm saying should we add one instead of mangling an enpoint inside this template14:03
shardyseems like it might be cleaner14:03
shardywe already have endpoints for stuff like mysql in there that's internal14:03
dtantsurshardy, this is the problem. it's a standalone HTTP vhost used for iPXE. it does not match Ironic API14:03
pandaEmilienM: I commented "check experimental" in my own review. I wanted to be sure that the check will take code from my change to test my change itself14:04
EmilienMsshnaidm: I approved it14:04
dtantsurshardy, hmm, a new one. yes, maybe. do you have an example handy of how to introduce a new endpoint with a reasonable default?14:04
openstackgerritMerged openstack/tripleo-quickstart: Update section header indicators  https://review.openstack.org/36052214:04
EmilienMpanda: of course, that's how zuul works14:04
shardydtantsur: yes, sec14:04
sshnaidmEmilienM, thanks, but periodics look like stuck in zuul for 8 hours already14:05
pandaEmilienM: for normal changes yes, but this works also for the changes that affect the test launched by zuul ?14:05
sshnaidmEmilienM, I think it's similar issue that pabelanger told about14:05
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Remove repeated apache-related hieradata  https://review.openstack.org/36368714:06
*** saneax is now known as saneax-_-|AFK14:06
*** akshai has joined #tripleo14:07
shardydtantsur: https://review.openstack.org/#/c/305751/ shows how it was wired in for mysql14:08
shardythis case should be much simpler, I'd start by updating endpoint_data.yaml then re-running build_endpoint_map.py14:08
*** anbu has joined #tripleo14:08
shardythen check endpoint_data.yaml to see which output you need, reference it in your template14:09
dtantsurshardy, yeah.. is it fine to just skip enable_tls thing? I'm not sure we can even achieve SSL with iPXE..14:09
shardydtantsur: yep I think so14:10
hewbroccagood lord, ssl + ipxe?14:10
dtantsurthanks shardy14:10
dtantsurhewbrocca, no way14:10
dtantsurI mean, maybe there is a way... but it's going to be a lot of dirty fun14:11
hewbroccafor 0 reason14:11
jaosoriordtantsur: which enable_tls stuff?14:12
dtantsurjaosorior, I'm looking at https://review.openstack.org/#/c/305751/33/environments/enable-tls.yaml as an example14:13
*** matbu is now known as matbu|mtg14:14
jaosoriordtantsur: you need to add it there14:14
*** rhallisey has joined #tripleo14:14
jaosoriordtantsur: if ipxe can't do TLS, it doesn't matter, just put the protocol as http14:14
dtantsurjaosorior, aha, ok14:14
*** akuznetsov has quit IRC14:14
dtantsurwait, where is this file nowadays?14:14
*** mhenkel has joined #tripleo14:15
dtantsurenvironments/enable-tls.yaml no longer lists endpoints14:15
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902714:15
*** abregman has quit IRC14:16
mhenkelhey Steve, I pushed all the patches but tht failed because if nonha-multinode14:16
*** alee has joined #tripleo14:16
pabelangersshnaidm: it is not stuck, periodic queue has precedence low, so all other pipelines get assigned nodes first. Once check-tripleo clears out, the periodic jobs will get assigned nodes14:17
sshnaidmpabelanger, ok, let's call it "stuck in queue" then :)14:17
dtantsurshardy, I'm trying to figure our the relationship of this change with https://github.com/openstack/tripleo-heat-templates/commit/cb2570c237907da2b91d402fec19665c9e422adf that dprince did14:17
sshnaidmpabelanger, is there a way to schedule them a little bit earlier?14:18
aleehi all - I'm getting a failed undercloud install with something about "/Firewall[998 log all]: Dependency Augeas[lvm.conf] has failures: true"  .. any idea how to fix/ get around that?14:18
dtantsurshardy, I'm not sure how to ask the right question, but still: do need this endpoint to "belong" to this IronicNetwork?14:18
pabelangersshnaidm: yes, we can control when the pipeline starts periodic jobs but doubt it will be moved14:19
dprincedtantsur: which patch?14:19
*** jprovazn has quit IRC14:20
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enabled iPXE by default and make it configurable  https://review.openstack.org/36214814:20
pabelangersshnaidm: what is the periodic job doing?14:20
dtantsurdprince, ^^^14:20
dtantsurdprince, I seem to be highly confused by networks vs endpoints14:20
sshnaidmpabelanger, it's like regular gate job, but if it succeeds we update the current tripleo repository14:21
dprincedtantsur: there are VIPs and then there are local ports14:22
dprincedtantsur: the endpoint map primarily deals with the former (VIPs)14:22
dprincedtantsur: so long as what you are doing is meant to do something along those likes it should be fine14:22
pabelangersshnaidm: why don't you do that in the post pipeline?14:23
pabelangerI suspect since you don't have voting jobs in gate14:23
dtantsurdprince, I'm not sure, that's the problem. I need to figure our the endpoint for iPXE (which is the same as PXE, but over HTTP instead of TFTP)14:23
dtantsurdprince, you've changed the TFTP address to use IronicNetwork, now I'm changing HTTP address to a new endpoint.. the IP must be the same, also untagged, etc14:24
dtantsurI have a gut feeling something might be wrong in what I'm doing14:24
hewbroccadtantsur: I know what it is14:26
hewbroccadtantsur: OpenStack14:26
dtantsurlol14:26
dprincedtantsur: commented14:26
dprincedtantsur: your' patch is close...14:26
dtantsurdprince, so, net_param: Ironic, right?14:27
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement scenari001, 002 and 003  https://review.openstack.org/36250414:28
dprincedtantsur: basically, when you generate the endpoint map you should see your new iPXE entries using the same IronicNetwork which we are defining as the Ironic provisioning network14:28
dprincedtantsur: yes14:28
*** rajinir has joined #tripleo14:28
dtantsurdprince, oh, now I get it. thanks man :)14:28
*** zoliXXL is now known as zoli|brb14:28
dtantsurOpenStack Networking: magic and ponies :D14:28
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enabled iPXE by default and make it configurable  https://review.openstack.org/36214814:29
dtantsurdprince, makes sense ^^^?14:29
sshnaidmpabelanger, I'm not sure what post-pipeline is..14:29
hewbroccaathomas: you have a theory about networking and ponies, if I remember14:30
*** oshvartz has quit IRC14:30
athomashewbrocca, Very true.14:30
EmilienMsshnaidm: this pipeline is used for tags, releasenotes, etc14:30
pabelangersshnaidm: pipeline in zuul that runs after code has been merged14:30
dprincedtantsur: looks good to me14:31
*** abregman has joined #tripleo14:31
pradkis anyone else seeing this erro deploying undercloud  ---> /Firewall[998 log all]: Dependency Augeas[lvm.conf] has failures: true ?14:31
pradkalee is seeing this14:31
pradkseems related to https://review.openstack.org/#/c/360790/14:31
gfidenteathomas, what's the theory?14:31
* gfidente waiting for more rgw updates14:31
athomashewbrocca, By the time people are using network bridges, you might as well visualise it in terms of magical ponies, fying down, picking up the individual packets in their mouths, and flying them over to their destination. For all the sense any of it makes, that's as accessible a visualisation as any.14:32
sshnaidmpabelanger, ok, afaik in periodic jobs we use different repos than in patch gate jobs14:32
athomasgfidente, ^^ You're welcome. ;-)14:32
*** akshai has quit IRC14:32
aleepradk, I know jaosorior was seeing the same thing ..14:32
hewbroccadtantsur: there you go, magical network ponies14:33
pradkalee, yea seems like a regression .. my undercloud yesterday went fine, though quickstart image might have been older14:33
dtantsurathomas, ROFL14:33
pabelangersshnaidm: that is fine, just think of moving from periodic (once a day) to running the job everytime code merges. The jobs can be the same, but just run more often14:33
bnemecSo by "ponies" we really mean magical unicorn pegasus?14:33
*** akshai has joined #tripleo14:33
bnemec:-)14:33
hewbroccasure14:33
hewbroccaas long as it can pick up a packet14:33
gfidenteI thought that was OVB14:34
pabelangerthis is one of the reason why I think tripleo should stand up there own zuul instances, so we can provide better workflows in the form of pipelines14:34
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: Add env. setup for virt deployments doc  https://review.openstack.org/36312014:34
hewbroccapabelanger: makes sense to me14:34
dtantsurdprince, thanks! now I'll test it locally14:34
sshnaidmpabelanger, I'd say it would be great, the current tripleo ping would be updates much faster14:34
dprincedtantsur: I +2'd it. That means it works right :)14:34
pabelangerthe issue is, a 3+ hour job in post likely won't flight right now14:34
*** jlinkes has joined #tripleo14:35
dtantsurhmm, true, to hell with local testing, let's go to the beach :D14:35
pabelangerSo, you could do that process in rdoproject.org or work toward your own zuul server for the tripleo project14:35
dprincedtantsur: I'm there man14:35
dtantsurhaha14:35
EmilienMgfidente: nobody got your joke dude14:35
dtantsurdprince, one of the worst things about Czech republic is that it's equally far from all possible seas in Europe14:36
dprinceEmilienM: that is because gfidente is smarter than us, he is a level up man14:36
EmilienMhis joke was too technical for us :P14:36
*** ansiwen has quit IRC14:36
*** ansiwen has joined #tripleo14:37
dprincedtantsur: I drive 5-6 hours to get to a good beach14:37
dtantsurcool!14:37
* gfidente wonders where is marios when he needs him14:38
EmilienMat the beach14:38
EmilienMhe need to walk 2 min AFIK14:38
derekhI can't drive 4 hours without hitting a beach....14:38
jaosoriorpradk: so I consistently keep getting that augeas issue in oooq :/14:38
sshnaidmpabelanger, and how will help the own zuul server?14:38
jaosoriorhave been trying for days14:38
derekhor a cliff14:38
dtantsurderekh, lol14:38
EmilienMderekh: you win14:38
dprinceEmilienM: yeah, marios won the work on the beach lottery I think14:38
gfidenteEmilienM, I wasn't making any joke though14:39
EmilienMdprince: he works "remotely"14:39
gfidentedprince, and what bad did I do to you really14:39
*** ansiwen has quit IRC14:39
dprincegfidente: I'll never understand it anyways... I'm trying14:39
dprincegfidente: you have hair on your head? yep, thats it14:40
*** ansiwen has joined #tripleo14:40
*** tzumainn has quit IRC14:40
*** tzumainn has joined #tripleo14:41
lucasagomesderekh, I want to look at that NodeLocked in the tripleO ci, do you know off the top of your head a patch which failed with that ?14:41
* dtantsur got a strong feeling that it's Friday already14:41
lucasagomesotherwise I will start looking at random patches to find it14:41
dtantsurlucasagomes, I've attached one to the trello card14:41
lucasagomesdtantsur, a-ha14:41
lucasagomesthanks14:41
dtantsurlucasagomes, also this http://logs.openstack.org/25/354125/24/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/0c6890d/14:42
pabelangersshnaidm: greater flexibility in changing pipelines14:42
EmilienMdprince: please review https://review.openstack.org/#/c/363204/14:42
derekhlucasagomes: I think the pyghmi  may make the problem go away, but one sec I'll find you one14:42
EmilienMdprince: this is an ipv6 bug bnemec found14:42
derekhlucasagomes: *pyghmi fix14:42
bnemecThere are multiple ipv6 bugs14:42
lucasagomesderekh, found one already14:42
derekhlucasagomes: ok14:42
EmilienMbnemec: did you list them?14:42
lucasagomesderekh, cool, FWIW the g-r patch is now merged14:42
derekhg-r?14:43
derekhlucasagomes: requirments14:43
bnemecEmilienM: https://bugs.launchpad.net/tripleo/+bug/1618623 is the first.  There's one in cinder too.14:43
openstackLaunchpad bug 1618623 in tripleo "vncproxy config broken in ipv6 deployments" [Critical,In progress] - Assigned to Ben Nemec (bnemec)14:43
derekhlucasagomes: got it14:43
EmilienMbnemec: thanks14:43
*** wgd3 is now known as wgd3[away]14:43
lucasagomesderekh, yeah global-requirements (sorry for the shortcut)14:43
bnemecI'm still verifying the fix for the Cinder one and will push a patch shortly.14:43
EmilienMbnemec: panda and I are working on the experimental job to test ipv6/ha scenario14:43
EmilienMbnemec: do you test in CI?14:43
sshnaidmpabelanger, that's sure, although I suspect also additional efforts for its maintenance. Together with latest intentions to be similar as any other project in upstream CI, I'd say it has a little chance to be approved.. :(14:44
*** ansiwen has quit IRC14:44
EmilienMjaosorior: can you land https://review.openstack.org/363105 please ?14:44
* dtantsur can't resist imaging a panda testing something in CI...14:44
gfidentebnemec, I think there is a host_nobrackets output we can use in both14:44
bnemecEmilienM: No, locally, but I'm working on ipv6 support for OVB too, which you will also need.14:45
*** ansiwen has joined #tripleo14:45
EmilienMdtantsur: http://s.quickmeme.com/img/c4/c4196ca721c280fe5ba953c7d01949395aaf220d6ef3aadc30142b68756ac51f.jpg14:45
EmilienMbnemec: ack14:45
bnemecI actually think it's working fine, but I ran out of time last night.14:45
dtantsurTHIS14:45
pabelangersshnaidm: rdoproject.org has the infrastructure today, that would be 90% of the effort14:45
EmilienMbnemec: you can sync with panda on this thing14:45
dprincebnemec, EmilienM: I think we can just use host_nobrackets right?14:45
EmilienMdprince: also14:45
sshnaidmpabelanger, do they have zuul at all?14:45
pabelangersshnaidm: but yes, I believe this is the main reason it hasn't happend sooner. Lack of resources to manage it14:46
EmilienMsshnaidm: yes14:46
pabelangersshnaidm: yes, they are using Software Factory14:46
EmilienMsshnaidm: rdoproject.org is Software Factory14:46
EmilienMand it's a replica of OpenStack Infra14:46
bnemecdprince: gfidente: That's fine.  This is what we were doing in the vncproxy profile though, so I reused it.14:46
pandaEmilienM: dtantsur: I prefer this one https://media.tenor.co/images/60a9f8d2822a17e426ddaf001c1a7ab7/raw14:46
EmilienMsshnaidm: http://softwarefactory-project.io/docs/faqs.html#what-is-the-added-value-of-software-factory14:47
dprincebnemec: maybe change them both? I struck me as odd I think now that the endpoint map has _nobracket stuff14:47
dtantsurpanda, so-you-say-it-worked-on-devstack.gif14:47
EmilienMpanda: I don't see you doing that, you're too calm ;-)14:47
bnemecdprince: Yeah, that's fine.  I'll update it when I push the cinder patch.14:47
sshnaidmEmilienM, pabelanger cool, good to know..14:47
EmilienMjaosorior: thx14:47
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: Refactor upgrade checks.  https://review.openstack.org/35775014:50
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: M/M upgrade sahara-api fails to restart.  https://review.openstack.org/35802214:51
*** maeca2 has quit IRC14:51
sshnaidmEmilienM, pabelanger, and what if really to run additional gate job, looking exactly as periodic one, including only repos that periodic job uses, and use post-pipeline to update current tripleo pin in case it succeeds14:52
*** fragatina has joined #tripleo14:53
*** fragatina has quit IRC14:53
sshnaidmor just update the current pin within the job14:53
pabelangersshnaidm: can you rephrase your question, I didn't follow14:53
EmilienMme neither14:53
sshnaidmsorry, thinking loud :)14:53
*** fragatina has joined #tripleo14:54
sshnaidmI mean using gate job for updating current tripleo repo pin, as pabelanger said, each time patch is merged, just to create a special gate job for this14:54
EmilienMit will cost a lot of CI resources14:55
sshnaidmbut not to vote, so it won't block patches from being merged14:55
EmilienMcomparing to periodic jobs14:55
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: [WIP] Documentation for installing and using Ironic in overcloud  https://review.openstack.org/35401614:55
pabelangersshnaidm: right, we cannot do that today. Because we don't want per project pipelines in zuul.o.o however, it is possible to do that14:55
sshnaidmEmilienM, only when it's going to be merged, not each patch check14:55
EmilienMit would also break the job if something broke in OpenStack14:55
*** zoli|brb is now known as zoli14:56
*** zoli is now known as zoliXXL14:56
bnemecPfft, nothing every breaks in OpenStack. ;-)14:56
*** rbrady has quit IRC14:56
dtantsursshnaidm, post-jobs suffer from the same problem as periodic jobs: nobody see their results :(14:57
pabelangerEmilienM: after zuulv3 we'll also have the ability to "if existing job running in pipeline foo, don't add job to pipeline", to avoid waiting resources14:57
dtantsursshnaidm, we're using post jobs to build IPA images. sometimes we find that they haven't built for a month :D14:57
sshnaidmdtantsur, you just don't have a dashboard with red lights for monitoring this :)14:57
EmilienMpabelanger: awesome, so many things in zuul v3 that we're waiting for14:58
dtantsurthat's true, we don't :)14:58
EmilienMpabelanger: do you know when zuul v3 will be out?14:58
derekhdtantsur: that doesn't happen in tripleo, if the periodic job fails we know.....how long it takes to fix it is another thing entirely14:58
pabelangerEmilienM: a good example of wasted resources is the wheel-build jobs, that run in the release-post pipeline.  This is what tripleo could do, replacing the periodic job14:58
pabelangerEmilienM: soon(tm)14:58
dtantsurderekh, that's pretty cool actually14:58
EmilienMbnemec: never, it works in devstack14:58
pabelangerEmilienM: we're going to be hacking on it durning the mid-cycle in 2 weeks14:58
*** fragatin_ has joined #tripleo14:58
EmilienMplease don't sleep and don't go in bars14:58
EmilienMjust do zuul v314:59
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: remove the ci-scripts dir from tripleo-quickstart  https://review.openstack.org/36373214:59
*** milan has quit IRC14:59
*** bana_k has joined #tripleo15:01
*** fragatina has quit IRC15:02
*** milan has joined #tripleo15:02
*** bvandenh has quit IRC15:05
*** mwhahaha has joined #tripleo15:08
*** jaosorior has quit IRC15:08
*** _coolsvap_ is now known as coolsvap15:09
openstackgerritMerged openstack-infra/tripleo-ci: Split additional features across the periodic jobs  https://review.openstack.org/36290415:09
*** yamahata has joined #tripleo15:10
sshnaidmbnemec, I think there is an issue with infra, pabelanger was talking about it before15:10
bnemecsshnaidm: Okay, thanks.  I'll read the scrollback.15:10
*** matbu|mtg is now known as matbu15:10
*** rbrady has joined #tripleo15:11
*** mcornea has quit IRC15:11
*** dprince has quit IRC15:13
jpeelershardy: https://bugs.launchpad.net/tripleo-common/+bug/1612819 for rc115:14
openstackLaunchpad bug 1612819 in tripleo-common "GetCapabilitiesAction output incomplete" [Undecided,New]15:14
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart: WIP: Add directory to store documentation files  https://review.openstack.org/36148315:15
EmilienMjistr: ouch https://review.openstack.org/#/c/358510/15:15
mariosgfidente: sorry i was on a call15:15
marios(*at the beach obviosuly)15:15
EmilienMmarios: :-)15:16
*** panda is now known as panda|afk15:16
EmilienMjistr: couldn't you write something in puppet-pacemaker?15:17
dtantsurdpr<TAB> :(15:17
EmilienMjistr: it's a bit hackish, no?15:17
jistrEmilienM: heh yea i think it's our best shot though :) i'm a bit worried about the redis one, but the rest should be fairly ok i hope. I commented there that it's an "ouch", but probably the best we can do.15:17
jistrEmilienM: not in puppet-pacemaker unless we have the refactoring that chem has been working on15:18
EmilienMjistr: right15:18
jistrEmilienM: once we have it, all this may change, but i think it's fairly certain that the puppet-pacemaker refactoring will not make Newton15:18
EmilienMchem: what is the status of the refactoring btw?15:18
EmilienMjistr: right I'm ok with your change +215:18
EmilienMjistr: please refactor it asap though15:19
*** alee is now known as alee_afk15:19
*** egafford has quit IRC15:20
*** adarazs_afk is now known as adarazs15:20
jistrEmilienM: ack thanks15:21
*** leanderthal is now known as leanderthal|afk15:21
dtantsurfolks, could someone please help me with this iPXE network again? please see my latest comment at https://review.openstack.org/#/c/362148/15:22
*** pcaruana has quit IRC15:22
*** rcernin has quit IRC15:22
shardyjpeeler: So I think we agreed to retire the tripleo-common launchpad project, as I'm tracking everything in the main tripleo one for the release15:22
dtantsurI need the endpoint to contain the real IP, not the VIP...15:23
shardyjpeeler: can I ask you to mark the bug as affects tripleo, and target it to rc1?15:23
jpeeleri see, yes15:23
* shardy checks jpeeler is in the tripleo drivers group15:23
EmilienMshardy: a lot of people are missing15:23
EmilienMshardy: and I could not add them, I don't have permission15:23
EmilienMshardy: can you make me admin please?15:23
shardyjpeeler: added15:24
shardyyou should be able to adjust it so it's tracked in https://launchpad.net/tripleo/+milestone/newton-rc1 now15:24
shardyEmilienM: sure15:24
EmilienMthx15:24
dtantsureven shorter version of the same question: how do make an endpoint to use a real IP, not VIP?15:24
dtantsurshardy, maybe you know ^^^15:25
hewbroccadtantsur: dsneddon might know15:25
hewbroccaif not gfidente certainly will, if you don't mind him writing a few more bugs15:26
dtantsuryeah, I just did not want to drag more people in my patch :)15:26
dtantsurmaybe I have to though15:26
shardydtantsur: in a meeting, will take a look and let you know15:26
dtantsurcool, thnx15:27
shardymost of the services are mapped to a vip that's true15:27
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Expose parameter to enable combination alarms  https://review.openstack.org/36374815:27
EmilienMshardy: thx so much my dream came true15:28
EmilienMi can now add people to launchpad group15:28
dtantsurshardy, yeah, but iPXE is special.. it must be the local IP15:28
dtantsurin the same fashion as we already have tftp_server15:28
*** tremble has quit IRC15:29
jistrdtantsur: i'm not 100% sure about the inner workings there :) -- but could this help? https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-vncproxy.yaml#L5415:29
shardydtantsur: ack, I'll look at how we can wire it in15:29
gfidentedtantsur, so I might miss context but you need a local IP of the node where the service is running?15:29
jistri think that actually resolves to a local ip15:29
shardyand comment on the review15:29
dtantsurgfidente, correct15:30
dtantsurjistr, I tried using a similar thing, but I was unable to build a proper HTTP URL out of it15:30
gfidentedtantsur, right we build the URL in endpoint_map15:30
gfidentedtantsur, is shardy going to comment on same submission already?15:31
dtantsurhopefully15:31
gfidenteI think you can build it up15:31
*** bana_k has quit IRC15:31
dtantsurjistr, I've already tried: https://review.openstack.org/#/c/362148/7/puppet/services/ironic-conductor.yaml but it does work15:32
dtantsurlines 54-6115:32
gfidentedtantsur, yeah because map_replace doesn't do deep search15:32
*** jayg is now known as jayg|g0n315:32
jistrdtantsur: well, that's exactly what i was going to suggest :(15:32
gfidentethis guy {get_param: [ServiceNetMap, IronicNetwork]} is not replaced15:32
gfidentedtantsur, so you get http://ctlplane:blabla ?15:33
jistrright it's not str_replace, it only replaces if the match is exact, right?15:33
gfidenteyeah15:33
dtantsurgfidente, correct15:33
gfidentejistr, can we interpolate %{} within a string thouhg?15:33
jistrgfidente: jinx!!15:33
jistryea hack it via 2 hiera keys15:34
dtantsurso I ended up getting a proper endpoint, which looks cleaner. except for it gives my a VIP...15:34
*** aufi has quit IRC15:34
jistrugly but i'm not sure if we can do better ATM15:34
gfidenteugly yes15:34
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Set default for pingtest template  https://review.openstack.org/36375315:35
gfidentejistr, , dtantsur, so I am going to comment that unless shardy come up with nicer solution15:36
*** tesseract- has quit IRC15:36
dtantsurack thanks15:37
jistrgfidente, dtantsur: https://paste.fedoraproject.org/418392/72657814/raw/ ?15:37
gfidenteyes that15:37
gfidentepost it in gerrit15:37
jistrok will do15:37
jistrwill fix the indentation too :)15:38
gfidentewait15:38
gfidentedtantsur, why latest version doesn't work?15:38
dtantsurgfidente, ends up with VIP instead of a local IP15:38
dtantsurgfidente, see the last comment for details15:38
gfidenteah right right15:38
*** mlupton has joined #tripleo15:39
gfidentejistr, also not sure about the single quotes15:39
gfidenteand if those prevent {hiera()}15:39
jistrgfidente: well we have the bug that single line strings don't get quoted :)15:41
gfidentejistr, let's try locally, sec15:43
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement scenari001, 002 and 003  https://review.openstack.org/36250415:43
jistrgfidente, dtantsur: i expect that it should literally end up in hiera like this:15:43
jistr    ironic::conductor::http_url: http://%{hiera("ironic_conductor_http_host")}:808815:43
larskshewbrocca: do you know if http://logs.openstack.org/88/254788/51/check/gate-tripleo-ci-centos-7-nonha-multinode/0dd3a32/console.html#_2016-08-31_13_36_03_818182 is just ci being broken?15:43
gfidentejistr, that format works for me locally15:43
openstackgerritChristopher Brown proposed openstack/tripleo-docs: The documentation is missing the tenant network under the list of networks the controller should have. This adds it.  https://review.openstack.org/36376015:44
gfidentejistr, it also works with ironic::conductor::http_url: 'http://%{hiera("ironic_conductor_http_host")}:8088' though15:45
gfidentethough false alarm on that15:45
jistryea should work but to get the quotes there we'd have to do extra work :D15:45
jistrdtantsur, gfidente: FWIW https://paste.fedoraproject.org/418400/72658325/  so if i didn't muck up the list_join, we should be golden15:46
openstackgerritChristopher Brown proposed openstack/tripleo-docs: Fix missing tenant network item  https://review.openstack.org/36376015:46
jistr^^ that's just hacked directly into the file, i didn't do heat stack-update :)15:46
*** jayg|g0n3 is now known as jayg15:47
gfidenteweren't we talking about going to beach already before?15:47
dtantsurjistr, thanks! I'll try it until other folks explain how wrong we are in a few minute :)15:47
*** abregman has quit IRC15:47
gfidentedtantsur, yeah so you don't need the endpoint_map addition15:47
gfidentefor that15:48
gfidentewhich is probably good15:48
dtantsurI was just typing this question ;)15:48
openstackgerritMerged openstack/puppet-tripleo: Reload HAProxy on refresh  https://review.openstack.org/35903815:48
dtantsurjistr, I think the one-liner won't work though, as I also need to substitute a port15:48
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: Add env. setup for virt deployments doc  https://review.openstack.org/36312015:49
jistrdtantsur: yea that's just an example of how Heat should generate it into Hiera so that it works in the end. The raw form which should be in t-h-t is different, posted here: https://review.openstack.org/#/c/362148/9/puppet/services/ironic-conductor.yaml15:50
larsksjistr: do you know if http://logs.openstack.org/88/254788/51/check/gate-tripleo-ci-centos-7-nonha-multinode/0dd3a32/console.html#_2016-08-31_13_36_03_818182 is just ci being broken?15:50
jistrlarsks: hmm i think i saw it earlier today so now i think that yes, it's CI being broken15:51
larsksUgh. Okay.  Just recheck, or are we waiting for someone to fix things?15:51
bnemeclarsks: It's still broken, AFAIK.15:54
*** rcernin has joined #tripleo15:54
larsksbnemec: okay, thanks.15:54
bnemecFor some reason git clones from git.o.o are hanging for some people, including our CI cloud. :-/15:55
*** dsariel has quit IRC15:58
sshnaidmbnemec, afaiu from pabelanger it's rackspace issues15:59
*** ansiwen has quit IRC15:59
*** abregman has joined #tripleo16:00
*** alee_afk is now known as alee16:00
dtantsurdo I get it right that we also haven't had a promotion since more than a week?16:01
EmilienMdtantsur: right.16:01
EmilienMdtantsur: jobs are timeouting as hell16:01
dtantsursigh, got it16:02
*** weshay is now known as weshay_food16:02
*** trown is now known as trown|brb16:03
jpichd0ugal: FYI on that bug I mentioned earlier, things appear to work fine on a Brand New Undercloud16:04
*** ansiwen has joined #tripleo16:05
jristjtomasek: https://review.openstack.org/#/c/358814/3/src/less/components/DeploymentPlan.less why not just delete the code?16:06
*** hewbrocca is now known as hewbrocca-afk16:07
*** trown|brb is now known as trown16:08
*** jprovazn has joined #tripleo16:09
*** links has joined #tripleo16:10
*** ianw has quit IRC16:13
openstackgerritMerged openstack/tripleo-heat-templates: Skip warning for required parameters  https://review.openstack.org/36138416:13
*** abregman has quit IRC16:13
*** fragatin_ has quit IRC16:14
*** zoliXXL is now known as zoli|gone16:16
*** zoli|gone is now known as zoli_gone-proxy16:17
*** fragatina has joined #tripleo16:18
*** akshai has quit IRC16:18
*** jdob has quit IRC16:21
*** fragatina has quit IRC16:22
*** fragatina has joined #tripleo16:22
gfidentecan you guys imagine16:23
gfidenteI had sourced overcloudrc16:23
gfidenteand tried to deploy the overcloud in the overcloud?16:23
gfidenteI think bnemec can16:23
*** egafford has joined #tripleo16:24
pabelangersshnaidm: bnemec: we are still working on the issue16:24
bnemecgfidente: Heh.  That's what CI is doing.16:24
bnemecWell, it was when jobs were running. :-)16:24
bnemecpabelanger: Thanks for the update.  Let us know if there's anything we can do to help.16:25
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add NetApp Manila driver integration and tidy up generic  https://review.openstack.org/35401916:25
gfidentebnemec, I didn't do it on purpose, but it actually went up to fail for an actual error in the tht code16:25
dtantsurgfidente, why do you think I'm working on ironic support in overcloud?16:26
dtantsurspecifically to run TripleO there16:26
bnemecI suppose.  Creating servers, networks, etc. on the overcloud should work fine.16:26
bnemecNow I'm curious if you could run an entire deployment that way.16:27
gfidentedtantsur, aaaaaaaaaaah ahaaaaaaaaaaaa I knew there was something going on16:27
gfidentebnemec, was without netiso to be fair16:27
shardyIn theory you could do it without ironic, just deploying to virt OS::Nova::Server's16:28
dtantsurthis is not fun, c'mon16:28
shardyI tried a while ago but ran into some networking stuff I didn't find time to debug16:28
bnemecYeah.  That's kind of what the multinode job does.  Except it completely stubs out the Servers.16:29
*** jpich has quit IRC16:29
dtantsurshardy, will you have a second to check guys' suggestion on https://review.openstack.org/#/c/362148/9/puppet/services/ironic-conductor.yaml ?16:29
*** dprince has joined #tripleo16:30
*** weshay_food is now known as weshay16:30
shardydtantsur: Yeah, that's simpler, I think I missed the lack of VIP thing when we first discussed it16:31
dtantsurokie, lemme try16:31
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add NetApp Manila driver integration and tidy up generic  https://review.openstack.org/35401916:32
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: IGNORE - testing manila-generic deploy after tidyup  https://review.openstack.org/36286616:32
*** wgd3[away] has quit IRC16:34
*** nyechiel has quit IRC16:34
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates: Ironic: enable iPXE by default and make it configurable  https://review.openstack.org/36214816:37
dtantsurshardy, gfidente better ^^?16:37
gfidentedtantsur, I think it's fine16:39
gfidentewhy is it that we need it formatted as an url but points to local ip though?16:39
dtantsurgfidente, ironic uses it to serve images cached locally on the conductor16:40
dtantsurgfidente, this "locally on the conductor" means we can't load-balance it or anything like that16:40
gfidenteyeah16:40
gfidentegot it16:41
dtantsurwhat's wrong with gerrit and git today?? both work like if it's Friday evening or even night already...16:41
*** ianw has joined #tripleo16:43
*** lucasagomes has quit IRC16:44
*** lucasagomes has joined #tripleo16:46
*** akshai has joined #tripleo16:50
*** numans has quit IRC16:50
*** trown is now known as trown|lunch16:58
*** bana_k has joined #tripleo17:02
*** ohamada has quit IRC17:03
*** yamahata has quit IRC17:05
*** akshai has quit IRC17:07
*** derekh has quit IRC17:09
*** akshai has joined #tripleo17:12
*** dbecker has quit IRC17:13
*** akshai has quit IRC17:14
*** jpena is now known as jpena|off17:17
*** akshai has joined #tripleo17:18
*** lucasagomes is now known as lucas-dinner17:19
*** dsariel has joined #tripleo17:23
*** akshai has quit IRC17:23
*** akshai has joined #tripleo17:25
*** akshai has quit IRC17:26
*** akshai has joined #tripleo17:28
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: WIP testing do not merge  https://review.openstack.org/36356017:28
*** dsariel has quit IRC17:30
*** jdob has joined #tripleo17:31
EmilienMshardy: no release today i guess17:35
EmilienMshardy: I see less timeouts now17:35
EmilienMmaybe I'll submit it later if I see green jobs17:35
shardyEmilienM: No, I guess not, hopefully we'll get CI working and can release tomorrow17:35
*** akrivoka has quit IRC17:35
shardyEmilienM: ack, thanks, I'll do likewise if CI is working when I start tomorrow morning17:36
shardyEmilienM: note we need to release tripleo-ui and tripleo-validations too which I don't think we did for newton-217:36
*** dtantsur is now known as dtantsur|afk17:36
EmilienMshardy: right17:36
*** abregman has joined #tripleo17:37
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install  https://review.openstack.org/35891917:37
*** pcaruana has joined #tripleo17:39
*** athomas has quit IRC17:39
dhill_hi guys17:40
dhill_https://bugzilla.redhat.com/137168517:41
openstackbugzilla.redhat.com bug 1371685 in openstack-tripleo-heat-templates "When using HostnameMap with more than 9 computes, hostnames are not properly set" [Unspecified,New] - Assigned to jstransk17:41
dhill_Anyone knows a quick hack for this issue?17:41
*** yamahata has joined #tripleo17:42
*** electrofelix has quit IRC17:42
*** nyechiel has joined #tripleo17:44
*** rwsu has quit IRC17:44
*** mhenkel has quit IRC17:44
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove hard-coded per-service rabbitmq hieradata  https://review.openstack.org/36173117:52
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Create NetIpListMap for all roles  https://review.openstack.org/36177717:52
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Generate composable service node_names lists  https://review.openstack.org/35506817:52
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Composable per-service VIP hieradata  https://review.openstack.org/36177817:52
shardydhill_: yes, sec17:52
*** nyechiel has quit IRC17:53
shardydhill_: just commented, it's a duplicate of another bz related to https://bugs.launchpad.net/heat/+bug/160020917:56
openstackLaunchpad bug 1600209 in tripleo "HostnameMap can result in partial replacement" [High,Fix released] - Assigned to Steven Hardy (shardy)17:56
shardythere is a workaround of adding an end character to the strings, and also a heat fix which has landed17:56
*** akshai has quit IRC17:57
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart: WIP: Add directory to store documentation files  https://review.openstack.org/36148317:58
*** nyechiel has joined #tripleo17:58
*** trown|lunch is now known as trown18:02
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Composable per-service VIP hieradata  https://review.openstack.org/36177818:02
*** akshai has joined #tripleo18:06
*** abehl has quit IRC18:07
*** pkovar has quit IRC18:10
*** florianf has quit IRC18:16
*** shardy is now known as shardy_afk18:24
*** shardy_afk has quit IRC18:25
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: set up quickstart to replace instack-virt-setup  https://review.openstack.org/35808918:25
EmilienMtripleo newton-3 release patch is ready for review https://review.openstack.org/36386818:27
openstackgerritJeff Peeler proposed openstack/tripleo-common: Add List Roles Action  https://review.openstack.org/33028318:28
bnemecEmilienM: panda|afk: Okay, I give up.  I don't know how to make the cinder config right in ipv6.  https://bugs.launchpad.net/tripleo/+bug/161893018:31
openstackLaunchpad bug 1618930 in tripleo "cinder config broken in ipv6" [Critical,Triaged] - Assigned to Ben Nemec (bnemec)18:31
EmilienMbnemec: ok looking18:31
EmilienMdamn we had this problem before18:32
EmilienMI remember gfidente fixed that18:32
gfidenteoh a question of brackets again18:32
gfidenteit might have gone missed with the roles18:32
gfidentebnemec, right looks like we want the quotes around the brackets?18:33
bnemecgfidente: I _think_ so, but I can't figure out how to get the templates to write that to hiera.18:33
bnemecI tested adding the brackets manually to cinder.conf after deployment, but I don't know for sure what the hieradata needs to look like.18:34
*** dprince has quit IRC18:36
*** sshnaidm is now known as sshnaidm|afk18:36
pradkbnemec, hi, was looking at https://bugs.launchpad.net/tripleo/+bug/161892418:38
openstackLaunchpad bug 1618924 in tripleo "gnocchi-metricd fails to start in nonha" [High,Triaged]18:38
pradkbnemec, do you know which repo we use in ci? the issue is fixed in cotyledon 1.2.718:39
pradkbnemec, so we just need to pull in new version of cotyledon into deps repo and we should be good18:39
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Ensure cinder_iscsi_address is in quotes  https://review.openstack.org/36388218:39
gfidentebnemec, probably https://review.openstack.org/36388218:40
gfidentemy hopes are that heat will consider " as part of the template for the str_replace18:40
gfidentebnemec, I guess we can inspect the hieradata of the normal CI run to see if the IPv4 address goes in "18:40
bnemecpradk: Okay, we only have python-cotyledon-1.2.5-0.1.pre_review.el7.noarch in CI.18:41
bnemecIf that's coming from the deps repo then I guess we'll need to talk to RDO folks.18:41
pradkbnemec, yea thats the issue.. i'll chase it down. can you point me to the repo link we're using?18:41
bnemecgfidente: I can try it quick locally too.18:41
gfidentebnemec, ah nice18:41
*** _milan_ has joined #tripleo18:42
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Last round of modern authtoken update  https://review.openstack.org/36318218:42
*** milan has quit IRC18:43
bnemecpradk: It's whatever tripleo.sh --repo-setup downloads.18:43
*** chem has quit IRC18:43
pradkk i'll check18:43
*** chem has joined #tripleo18:44
EmilienMgfidente: nice :)18:45
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: midonet: update neutron parameter  https://review.openstack.org/36317818:47
bnemecSigh.  Stack updates are still broken?18:49
bnemecMine are anyway.18:49
*** jbadiapa has quit IRC18:49
bnemecSo much for a quick test.18:49
gfidenteEmilienM, if it works ...18:53
*** mwhahaha has quit IRC18:54
EmilienMbnemec: have you heat updated?18:54
bnemecEmilienM: I'm using whatever I got when I installed the undercloud yesterday.18:54
*** akshai has quit IRC18:55
*** akshai has joined #tripleo18:55
EmilienMthe problem is that promotion didn't happen in our CI for 7 days18:55
EmilienMand the heat fix was packaged in the meantime18:56
*** dprince has joined #tripleo18:59
*** mwhahaha has joined #tripleo19:00
*** hrybacki is now known as hrybacki|afk19:02
dhill_shardy: thank you sir... that'll help me a bit19:04
dhill_shardy: I'm wondering why I didn't find it19:04
bnemecGood grief.  A 12 GB undercloud is no longer large enough for a development environment.19:05
bnemecheat-engine just oom'd on just my third deployment today.19:05
chembnemec: time to add some swap file19:06
bnemecchem: No, this is why I just bought a box with 144 GB of RAM. ;-)19:07
*** kjw3 has joined #tripleo19:08
EmilienMyou can test all composable services with 144GB of RAM19:08
chembnemec: that will warmup you house by a few degree :)19:08
*** ifarkas is now known as ifarkas_afk19:11
bnemecchem: It does, but it's better than I thought it might be.  The box only pulls around 90 watts at idle.19:11
bnemecAnd in the winter I'll appreciate it. :-)19:11
bnemecEmilienM: Maybe...;-)19:11
*** links has quit IRC19:13
chembnemec: not that bad, what about noise, can we have nowadays a big  server at home without everybody going crazy19:14
gfidentebnemec, do you still buy these on ebay?19:15
EmilienMmy gf would kill me if I do that19:15
EmilienMor just leave19:15
dhill_bnemec: I have less memory than you19:16
dhill_bnemec and it's not ooming19:16
dhill_bnemec but I'm cheating with lots of swap on SSD :)19:16
bnemecchem: This one is not bad.  The Poweredge R710 is incredibly quiet by server standards.19:17
EmilienMjrist: fyi https://review.openstack.org/36389719:17
bnemecStill louder than my quiet desktops, but very tolerable.19:17
chembnemec: hum ... interesting.  Thanks for the info19:17
bnemecgfidente: Yeah, it's another EBay special.19:17
chemhehe19:17
bnemecdhill_: SSDs are nice for swap. :-)19:18
*** fragatin_ has joined #tripleo19:18
EmilienMjust make sure you keep enough space on your ssd disk not encrypted19:19
EmilienMso you win performance19:19
dhill_bnemec: on a 32gb laptop I can deploy 9 overcloud nodes and a director... running into all the possible race conditions possible ;)19:19
bnemecdhill_: I don't hate myself enough to do that. :-P19:20
*** coolsvap has quit IRC19:22
*** fragatina has quit IRC19:22
bnemecJust finished cleaning up all the leftover instances on rh1 from deleted heat stacks and the leftover instance_uuids in my undercloud ironic from deleted nova instances.19:23
bnemecOpenStack: Eventual consistency, for extremely large values of "eventual".19:23
bnemecOkay, ipv6 overcloud is finally deploying.  Time to eat before I pass out from low blood sugar.19:24
*** mlupton has quit IRC19:24
*** mlupton has joined #tripleo19:25
*** akshai has quit IRC19:25
*** akshai has joined #tripleo19:28
gfidentebnemec, oh the cinder thing worked?19:28
gfidentebnemec, I need guidance over those ebay buys19:28
openstackgerritJeff Peeler proposed openstack/tripleo-common: Include environments in capabilities output  https://review.openstack.org/35559819:29
*** mlupton has quit IRC19:29
openstackgerritMerged openstack/tripleo-heat-templates: apache: Remove product informations  https://review.openstack.org/36310519:30
bnemecgfidente: I'm just testing the cinder thing.  My undercloud oom'd the first time I tried it, then it was failing because of bogus node associations.19:30
*** akshai has quit IRC19:31
*** akshai has joined #tripleo19:32
*** nyechiel has quit IRC19:32
larsksHey folks; do we know if CI is feeling better at this point?19:32
*** jeckersb is now known as jeckersb_gone19:32
bnemecgfidente: http://www.ebay.com/itm/Dell-Poweredge-R710-2x-Xeon-X5670-2-93ghz-Hex-144gb-2x-2tb-SATA-Perc6i-2x-870w-/232063580171?hash=item360811380b:g:2o8AAOSwl8NVeaET19:33
bnemec:-)19:33
bnemeclarsks: Afraid not.  Still no jobs running.19:34
bnemecAt least not to the point where they would spin up a testenv.19:34
larsksWell okay then :)19:34
bnemecInfra is continuing to investigate, but from the scrollback it doesn't sound like they have a fix yet.19:35
pabelangerwaiting on rackspace now19:36
pabelangeris a support issue19:36
*** flepied has joined #tripleo19:37
*** mlupton has joined #tripleo19:37
EmilienMpabelanger: is it a bad time for asking a review ? https://review.openstack.org/#/c/363629/19:37
*** bfournie has left #tripleo19:38
pabelangerEmilienM: Ya, head down launching a new cloud. I'll review once finished19:38
EmilienMpabelanger: np19:38
*** mlupton has quit IRC19:40
EmilienMbnemec: what do you think if we add more debug during pingtest19:40
*** mlupton has joined #tripleo19:40
EmilienMit's hard to find out what was wrong during pingtest if it failed19:40
EmilienMpradk: you remember the error I had with gnocchi and pingtest?19:42
EmilienMpradk: I think it wasn't redis19:42
EmilienMpradk: http://logs.openstack.org/05/362505/2/experimental/gate-tripleo-ci-centos-7-scenario001-multinode-nv/2f6c560/logs/subnode-2/var/log/aodh/app.txt.gz19:42
EmilienMGnocchiUnavailable: Cannot use v2 authentication with domain scope19:42
EmilienMit seems like aodh is not able to talk with gnocchi19:42
*** mlupton has quit IRC19:43
EmilienMpradk: I don't even see gnocchi & aodh config in http://logs.openstack.org/05/362505/2/experimental/gate-tripleo-ci-centos-7-scenario001-multinode-nv/2f6c560/logs/subnode-2/etc/ wtf19:43
*** fragatin_ has quit IRC19:45
*** fragatina has joined #tripleo19:45
bnemecEmilienM: Such as?  We already dump out a bunch of stuff if it fails, but if there's more that would be useful there I'm fine with adding it.19:46
bnemecopenstack stack failures list tenant-stack would probably be good, I guess.  Now that that's a thing. :-)19:46
EmilienMbnemec: good idea19:46
pradkEmilienM, hmm i see.. need to double check service credentials . aodh should be able to find gnocchi service from keystone service catelog19:47
*** jeckersb_gone is now known as jeckersb19:47
bnemecgfidente: Bad news: tripleo::profile::base::cinder::volume::iscsi::cinder_iscsi_address: "storage_uri"19:47
bnemecEs no bueno.19:47
* bnemec needs to brush up on his Spanish before Barcelona19:47
*** mlupton has joined #tripleo19:48
*** mlupton has quit IRC19:49
*** akshai has quit IRC19:49
*** mlupton has joined #tripleo19:49
sshnaidm|afkis ipv6 ssh problem tracked anywhere?19:51
*** maeca1 has joined #tripleo19:55
*** mlupton has quit IRC19:56
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure  https://review.openstack.org/36391819:56
EmilienMbnemec: ^19:56
*** mlupton has joined #tripleo19:56
EmilienMsshnaidm|afk: not afik19:56
*** mburned has quit IRC20:01
*** liverpooler has quit IRC20:07
pradkEmilienM, we ran into similar issue with ceilo i think.. the issue is we append auth url with /v2 . But if you see the rest of the logs, i dont see any gnochi or aodh services come up due to redis not being up and also due to the bug https://bugs.launchpad.net/tripleo/+bug/161892420:08
openstackLaunchpad bug 1618924 in tripleo "gnocchi-metricd fails to start in nonha" [High,Triaged]20:08
chemgfidente: could you tell me what I should look to decipher this error: resources.Controller: resources[0]: NotFound_Remote: resources.UpdateDeployment: Software config with id efdb9688-0279-4173-854f-c2be1c83fe3e not found20:09
*** mburned has joined #tripleo20:09
pradkEmilienM, with metricd, statsd not up and evaluator not running.. aodh wont be able to much with gnocchi20:09
thrashEmilienM: I think it needs a stack name, right?20:09
thrashEmilienM: it does.20:10
chemgfidente: how can I find which conf trigger that20:10
thrashEmilienM: https://review.openstack.org/#/c/363918/1/scripts/tripleo.sh,unified20:10
*** jprovazn has quit IRC20:10
chemEmilienM: do you know who could answer this question ^20:11
EmilienMbrb20:11
thrashchem: Can you try running 'openstack stack failures list --long overcloud'?20:12
chemthrash: ack ... I have to wait a few minute before doing it, I have restarted my fail upgrade20:13
thrashchem: ack20:13
chemthrash: oki, it failed :)20:14
*** mlupton_ has joined #tripleo20:14
chemthrash: only one soft conf id was missing this time ... anyway running the command20:14
gfidentebnemec, right, the quotes are in there but the key is not replaceable anymore20:15
*** mlupton has quit IRC20:16
chemthrash: http://paste.openstack.org/show/565312/ , if you could help me debug this20:16
bnemecchem: Are you running with the current-tripleo Heat?  Apparently that one is broken for stack updates and you need to use the one from the latest dlrn repo.20:17
bnemecgfidente: Yeah, same problem I had in my test.  I have no idea how the replacement works, so I don't know how to fix it.20:17
gfidentebnemec, so I don't think we're going to get quotes after the key is replaced20:18
chembnemec: I have osp-d heat running20:18
chembnemec: version 1020:18
*** jayg is now known as jayg|g0n320:18
EmilienMthrash: ack, thanks20:18
EmilienMpradk: ok but why don't we see aodh/gnocchi etc?20:19
chembnemec: so not far from current-triple .. but don't know exactly whchi version I should look for20:19
*** mlupton has joined #tripleo20:19
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure  https://review.openstack.org/36391820:19
EmilienMthrash: ^20:19
bnemecchem: Probably the same issue.  The latest director builds probably have the same bug.20:20
*** mlupton has quit IRC20:20
chembnemec: do you have a link or something to the bug ?20:20
*** mlupton_ has quit IRC20:22
*** mlupton has joined #tripleo20:25
*** akshai has joined #tripleo20:27
*** nyechiel has joined #tripleo20:27
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Ensure cinder_iscsi_address is in quotes  https://review.openstack.org/36388220:29
gfidentebnemec, ^^ ugly but ...20:29
*** jpeeler has quit IRC20:30
EmilienMnice !20:30
*** Goneri has quit IRC20:30
gfidenteEmilienM, ahahah it's even worst than what it was20:30
gfidenteEmilienM, but I can't think of many other ways to make this work20:30
gfidenteif not *any* other way20:31
openstackgerritMerged openstack-infra/tripleo-ci: Implement scenari001, 002 and 003  https://review.openstack.org/36250420:32
gfidenteEmilienM, we can't work it out in puppet-tripleo for example because hiera fails parsing the value in brackets without inner quotes20:32
*** assassin has quit IRC20:33
*** ubijtsa has joined #tripleo20:33
EmilienMgfidente: right20:34
EmilienMgfidente: hopefully we can sort this out later with fqdn20:34
gfidenteyeah20:34
*** ubijtsa is now known as Guest4458820:34
thrashchem: I think what bnemec told you is best bet at this point.20:37
pblahohi guys... pls, where I can find how policy.json files are generated in RDO?20:38
*** nyechiel has quit IRC20:39
pblahob/c upstream one is empty - https://github.com/openstack/nova/blob/master/etc/nova/policy.json20:39
*** ansiwen has quit IRC20:41
bnemecgfidente: Heh, okay let me try that one.20:43
gfidentebnemec, :(20:43
chemthrash: ack20:45
chemthrash: bnemec thanks for looking20:45
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates: Add missing reference to ExternalPort for DVR  https://review.openstack.org/36318820:46
chembnemec: but ... if you have more info about this broken upgrade I would really apreciate :)20:47
bnemecchem: I'm on a call right now, but I assume a bug was opened in launchpad.20:47
bnemecI wasn't actually involved in it, I just heard about it in IRC.20:48
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Add Ceph RGW profile  https://review.openstack.org/33408120:49
*** _milan_ has quit IRC20:49
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902720:53
*** jdob has quit IRC20:58
*** rwsu has joined #tripleo21:00
*** rhallisey has quit IRC21:01
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902721:01
*** ansiwen has joined #tripleo21:03
pradkEmilienM, yea its weird you dont have those dir's in etc .. a good way to verify this would be to try creating gnocchi alarms from cli once overcloud comes up with master . I'll check tonight21:05
*** jlinkes has quit IRC21:08
pradk*this* == is aodh able to talk to gnocchi when overcloud comes up21:09
*** ayoung has quit IRC21:10
*** jcoufal has quit IRC21:12
*** jlinkes has joined #tripleo21:12
*** jlinkes has quit IRC21:16
chembnemec: ack thanks, will look for it21:17
*** matbu is now known as matbu|afk21:17
bnemecchem: https://bugs.launchpad.net/heat/+bug/161655021:19
openstackLaunchpad bug 1616550 in heat "Overcloud update is broken: UPDATE_FAILED NotFound_Remote: resources[0]: Software config with id not found" [High,Fix released] - Assigned to Zane Bitter (zaneb)21:19
*** dprince has quit IRC21:19
bnemecThe call is waiting for the user to configure their networking correctly. :-)21:19
*** lblanchard has quit IRC21:19
chembnemec: ah, thanks you.  I was about to look for it21:19
chembnemec: hum ... yep not easy :)21:19
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902721:20
*** ansiwen has quit IRC21:21
EmilienMthrash, bnemec: easy review https://review.openstack.org/#/c/363178/21:23
EmilienMand https://review.openstack.org/#/c/363182/21:23
*** yolanda has joined #tripleo21:26
*** egafford has quit IRC21:27
openstackgerritRyan Brady proposed openstack/tripleo-common: Decouple deployment plans from Heat stacks  https://review.openstack.org/36395021:29
*** fzdarsky has quit IRC21:29
*** jlinkes has joined #tripleo21:36
*** cdearborn has quit IRC21:38
*** rwsu has quit IRC21:38
*** rwsu has joined #tripleo21:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Use netapp_host_type instead of netapp_eseries_host_type  https://review.openstack.org/36395521:42
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install  https://review.openstack.org/35891921:43
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Use netapp_host_type instead of netapp_eseries_host_type  https://review.openstack.org/36395521:43
*** mburned is now known as mburned_out21:48
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Add Ceph RGW profile  https://review.openstack.org/33408121:50
*** maeca1 has quit IRC21:50
*** maeca1 has joined #tripleo21:51
*** maeca1 has quit IRC21:51
*** yolanda has quit IRC21:56
*** morazi has quit IRC21:59
*** hrybacki|afk is now known as hrybacki21:59
*** mlupton has quit IRC22:01
openstackgerritSteve Baker proposed openstack/tripleo-specs: Propose a config-agent bug tag  https://review.openstack.org/36396022:01
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs  https://review.openstack.org/35921522:01
*** mlupton has joined #tripleo22:02
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Ensure iscsi_ip_address is in brackets if IPv6  https://review.openstack.org/36396122:03
*** ansiwen has joined #tripleo22:05
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Pass cinder_iscsi_address without brackets  https://review.openstack.org/36388222:05
gfidentebnemec, EmilienM so this was my last attempt, I am sorry I wasted your time as well22:05
mwhahahadidn't a function get added in puppet to check if it's an ipv6 address and add brackets22:09
mwhahahacan't you just use that22:09
mwhahahachem wrote it i thought22:10
gfidentemwhahaha, enclose_ipv6 ?22:10
gfidenteI think it returns an array though22:11
gfidenteeven if you pass it a string22:11
bnemecBracket me maybe, to the tune of https://www.youtube.com/watch?v=fWNaR-rxAic22:11
bnemec:-)22:11
mwhahahano was something else22:11
mwhahahahttps://github.com/openstack/puppet-openstacklib/blob/master/lib/puppet/parser/functions/normalize_ip_for_uri.rb22:11
mwhahahanormalize_ip_for_uri22:11
mwhahahaaka we couldn't think of anything better22:12
gfidenteahahaha22:12
gfidentechecking22:12
gfidentethanks22:12
*** panda|afk has quit IRC22:12
*** akshai has quit IRC22:12
gfidentemwhahaha, welcome?22:12
mwhahahathanks? :D22:13
gfidente^_^22:13
chemgfidente: mwhahaha well everybody was against add_ipv6_bracket_maybe22:13
mwhahahais_this_an_ipv6_address_question_mark22:14
chemgfidente: it doesn't return an array, and it works with only one arg https://github.com/openstack/puppet-openstacklib/blob/master/spec/functions/normalize_ip_for_uri_spec.rb22:17
gfidentemwhahaha, chem yeah noticed22:17
gfidenteI need to include ::openstacklib22:19
gfidenteto call it right?22:19
mwhahahano22:19
mwhahahait's a parser function22:19
mwhahahaas long as openstacklib is in the module path it'll work22:19
mwhahahawhich it should be if you're using any openstack puppet modules :D22:20
gfidenteah I did not know those were global22:22
gfidenteor maybe never noticed22:22
gfidentethanks!22:22
mwhahahathey are pretty hidden22:23
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Ensure iscsi_ip_address is in brackets if IPv6  https://review.openstack.org/36396122:23
mwhahahaneed better docs around what's available22:23
gfidentemwhahaha, chem so that should just work?22:23
mwhahahayea22:24
*** panda|afk has joined #tripleo22:25
chemmwhahaha: yep, some entries in the readme.md would be nice.22:26
chemmwhahaha: will had bracket_me_maybe tomorrow22:26
mwhahaha:o22:27
gfidentedon't deprecate the old one though22:27
gfidentejust mv22:27
chemhehe22:27
chems/had/add/22:27
chemoki, I'm off22:28
*** chem is now known as chem|off22:28
bnemecgfidente: It's looking like that should work.22:29
bnemecI tested the original non-bracket me maybe version, but since they do the same thing I expect the results to match.22:30
gfidentebnemec, we'll have this _uri thing with all the outputs22:30
gfidentepotentially breaking hiera?22:30
*** akshai has joined #tripleo22:32
*** akshai has quit IRC22:34
bnemecgfidente: It's on step 5 and hasn't blown up on hiera errors, so I think we should be okay.22:35
bnemecOvercloud deploys take so damn long now though.22:35
bnemecI used to be able to do them in a little over 15 minutes.  Now it's close to 30. :-(22:36
*** akshai has joined #tripleo22:42
gfidentebnemec, good, thanks22:42
*** bana_k has quit IRC22:42
*** nyechiel has joined #tripleo22:43
bnemecgfidente: tripleo.sh -- Overcloud pingtest SUCCEEDED22:44
bnemecLooks like we're good.22:44
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Add Ceph RGW profile  https://review.openstack.org/33408122:45
gfidentebnemec, ehe22:46
*** nyechiel has quit IRC22:53
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902722:56
*** bana_k has joined #tripleo22:56
*** dtrainor has quit IRC22:58
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: [NO MERGE] Test Ceph RadosGW as replacement for Swift  https://review.openstack.org/35718222:59
*** pradk has quit IRC23:03
*** jlinkes has quit IRC23:03
*** dtrainor has joined #tripleo23:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install  https://review.openstack.org/35891923:11
*** akshai_ has joined #tripleo23:15
*** jlinkes has joined #tripleo23:16
*** akshai has quit IRC23:19
*** maeca1 has joined #tripleo23:23
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Pass cinder_iscsi_address without brackets  https://review.openstack.org/36388223:30
*** dtrainor has quit IRC23:31
*** dtrainor has joined #tripleo23:44
*** jlinkes has quit IRC23:57
*** rwsu has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!