Thursday, 2016-05-12

*** fragatina has quit IRC00:04
pabelangerany update on the hardware upgrade? I only ask because we disabled tripleo in nodepool for the moment00:07
*** saneax is now known as saneax_AFK00:09
*** fragatina has joined #tripleo00:09
*** thrash is now known as thrash|g0ne00:11
*** fragatina has quit IRC00:14
*** penick has joined #tripleo00:20
openstackgerritMerged openstack/diskimage-builder: Fix OpenSUSE support  https://review.openstack.org/31232500:36
*** lifeless has quit IRC00:46
crinkleis there another example of https://bugs.launchpad.net/tripleo/+bug/1580076 besides https://review.openstack.org/#/c/313650 ? i found https://review.openstack.org/#/c/312725/ but it doesn't seem to have overcloud logs00:46
openstackLaunchpad bug 1580076 in tripleo "Upgrades job failing pingtest with "Message: No valid host was found."" [Critical,Fix released] - Assigned to Dan Prince (dan-prince)00:46
*** lifeless has joined #tripleo00:47
*** derekh has joined #tripleo00:47
*** rcernin has quit IRC00:54
*** [1]cdearborn has joined #tripleo00:57
*** mbound has quit IRC00:59
*** absubram has joined #tripleo01:01
*** absubram_ has joined #tripleo01:02
*** absubram has quit IRC01:05
*** absubram_ is now known as absubram01:05
*** derekh has quit IRC01:09
*** trozet has quit IRC01:10
*** shakamunyi has joined #tripleo01:23
*** dmacpher-afk is now known as dmacpher01:23
*** penick has quit IRC01:27
openstackgerritBen Swartzlander proposed openstack/tripleo-heat-templates: Enable Manila integration  https://review.openstack.org/18813701:36
*** Goneri has quit IRC01:41
*** tzumainn has quit IRC01:42
*** fpan has joined #tripleo01:47
*** ebalduf has quit IRC01:48
*** fpan has quit IRC01:52
*** Goneri has joined #tripleo01:54
*** [1]cdearborn has quit IRC01:57
*** mbound has joined #tripleo02:00
*** mbound has quit IRC02:04
*** ccamacho has quit IRC02:14
*** ccamacho has joined #tripleo02:15
*** Goneri has quit IRC02:20
*** zaneb has quit IRC02:28
*** penick has joined #tripleo02:37
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/31372102:55
*** gbraad has quit IRC03:16
*** gbraad has joined #tripleo03:17
*** zaneb has joined #tripleo03:19
*** links has joined #tripleo03:34
*** ebalduf has joined #tripleo03:36
*** julim has joined #tripleo03:40
*** panda has quit IRC03:42
*** panda has joined #tripleo03:43
*** fragatina has joined #tripleo03:43
*** fragatina has quit IRC03:45
*** fragatina has joined #tripleo03:46
*** fragatina has quit IRC03:46
*** fragatina has joined #tripleo03:47
*** masco has joined #tripleo03:54
*** fpan has joined #tripleo04:01
*** shivrao has quit IRC04:04
*** julim has quit IRC04:07
*** fpan has quit IRC04:08
*** pcaruana has joined #tripleo04:25
*** pcaruana has quit IRC04:32
*** coolsvap has joined #tripleo04:39
openstackgerritBabu Shanmugam proposed openstack/puppet-tripleo: Add OVN support for neutron profile  https://review.openstack.org/31487504:47
*** coolsvap has quit IRC04:52
*** penick has quit IRC05:00
*** penick has joined #tripleo05:03
*** coolsvap has joined #tripleo05:08
*** shivrao has joined #tripleo05:17
*** stendulker has joined #tripleo05:19
*** apetrich has joined #tripleo05:37
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements  https://review.openstack.org/30004405:39
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/31135705:39
*** apetrich has quit IRC05:40
*** apetrich has joined #tripleo05:42
*** shivrao_ has joined #tripleo05:51
*** shivrao has quit IRC05:52
*** shivrao_ has quit IRC05:52
*** rcernin has joined #tripleo05:53
*** shivrao has joined #tripleo05:54
*** florianf has joined #tripleo05:55
*** tremble has joined #tripleo06:05
*** tremble has joined #tripleo06:05
*** saneax_AFK is now known as saneax06:05
*** cwolferh has quit IRC06:08
*** shivrao has quit IRC06:10
*** pcaruana has joined #tripleo06:12
*** jcoufal has joined #tripleo06:26
*** mcornea has joined #tripleo06:40
*** jprovazn has joined #tripleo06:40
*** shivrao has joined #tripleo06:41
*** devvesa has joined #tripleo06:43
*** paramite has joined #tripleo06:47
*** shivrao has quit IRC06:54
*** crinkle_ has joined #tripleo06:59
*** dmacpher has quit IRC07:00
*** crinkle has quit IRC07:00
*** StevenK has quit IRC07:00
*** crinkle_ is now known as crinkle07:00
*** StevenK has joined #tripleo07:01
*** jaosorior has joined #tripleo07:03
*** tesseract has joined #tripleo07:05
jaosoriorshardy hey dude07:05
*** bvandenh has joined #tripleo07:07
*** tesseract has quit IRC07:09
*** dmacpher has joined #tripleo07:12
*** shardy has joined #tripleo07:13
jaosoriorshardy: Hey dude07:16
jaosoriorregarding this commit https://review.openstack.org/#/c/304125/07:16
jaosoriorI'm a little confused07:16
jaosoriorHow would an endpoint-generation specific manifest help?07:16
shardyjaosorior: it would allow you to configure keystone, then configure the endpoints at a different step07:18
shardyI suppose we can do the same with a conditional in the profile07:18
jaosoriorshardy, but how, since there is no other step anymopre?07:18
jaosoriorstep 6 was removed07:18
jaosoriorwhich is the reasont hat commit is now failing07:18
jaosoriorwe have no way anymore to asure that keystone is up and running to be able to do that configuration07:18
jaosoriorwhich is why it fails in HA07:18
shardyOk, that is a problem07:19
jaosoriorshardy, which is what me and jistr were telling people before they erased that step07:19
jaosoriorI even -1ed and explained why07:19
shardyjaosorior: the main things I'm looking to improve are: (1) don't create endpoints for services we're not actually enabling07:20
*** zoli_gone-proxy is now known as zoliXXL07:20
jaosoriorshardy: I understand that issue07:20
shardy(2) stop the increase in monolithic complexity related to the endpoint_map07:20
shardye.g define default per-service endpoints alongside the service definition (if possible)07:20
jaosoriorand when I we talked about it (I think you were in the chat too) I was explaining that first I was trying to get it to work then we could iterate on that07:20
jaosoriorwhen it finally worked07:20
jaosoriorthen step 6 was deleted07:20
jaosoriorand now it's failing again07:20
shardyjaosorior: Ok, we'll have to discuss the step problem as a separate issue, but do you buy the general problems I'm referring to?07:21
jaosoriorshardy: I never argued against them07:21
shardycool07:21
zoliXXLgood morning07:21
*** ebalduf has quit IRC07:21
jaosoriorshardy: what I've been arguing for is solving problems step by step07:21
jaosoriorwhich is why I didn't do it in that commit07:21
shardyjaosorior: sure - but we've also got a need to not have every single commit hit the CI job timeout, which is why optimizations have been sought07:22
shardyI'm sure we can work out a compromise, but I've not got one to offer right this moment :)07:23
* shardy needs to finish his coffee first07:23
jaosoriorshardy: one option I was thinking about is a sepparate step in puppet... but not like a step 607:24
shardylets chat with EmilienM later when he wakes up07:24
jaosoriorbut a step that just runs those specific bits07:24
jaosoriorso basically the endpoint generation would be added in separate manifests for the services07:24
shardyjaosorior: Yeah, or we could even go non-puppet and decouple the service initialization from the configuration07:24
jaosoriorand that manifest would be specifically sought for07:24
shardye.g not in os-cloud-config, we could have a mistral workflow which configures things07:24
shardyAh, so you mean have an independent puppet apply which doesn't build the catalog for all previous steps?07:25
jaosoriorshardy: that is an option... but I would prefer it to be in puppet or t-h-t (something we have now and can effectively use in OSP10)07:25
jaosoriorshardy: pretty much07:25
jaosoriorshardy: That would speed up that step a lot07:26
*** mikelk has joined #tripleo07:28
*** yamahata has joined #tripleo07:31
*** jpena|off is now known as jpena07:34
*** aufi has joined #tripleo07:40
*** dmk0202 has joined #tripleo07:40
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Fix apt-sources configuration for debian-minimal  https://review.openstack.org/31526507:41
*** panda has quit IRC07:42
*** panda has joined #tripleo07:43
*** ebarrera has joined #tripleo07:49
*** fragatina has quit IRC07:56
*** devvesa has quit IRC07:59
*** devvesa has joined #tripleo08:00
*** Guest42459 has joined #tripleo08:00
*** veteran has joined #tripleo08:06
*** ifarkas has joined #tripleo08:08
*** chem has joined #tripleo08:09
*** veteran has quit IRC08:11
*** athomas has joined #tripleo08:15
*** dtantsur|afk is now known as dtantsur08:17
*** ohamada has joined #tripleo08:17
*** lucas-dinner is now known as lucasagomes08:17
*** florianf has quit IRC08:21
jidardoes Ironic-python-agent fix the old' my network interface names are not ordered properly out of os-config-net or whatever was doing it in OSP7?08:27
jidarhttps://bugzilla.redhat.com/show_bug.cgi?id=1296330 from this bug08:27
openstackbugzilla.redhat.com bug 1296330 in rhel-osp-director "Unable to consistently deploy overcloud nodes with OSP 7.2: Failed to mount root partition" [High,Closed: errata] - Assigned to mburns08:27
jidaror was this fixed in the overcloud-full image itself08:27
shardyjidar: it looks like it was fixed via moving to the IPA ramdisk08:31
*** florianf has joined #tripleo08:31
jidarfantastic08:31
jidarI've got a guy doing osp8 with some issues and wanted to make sure he wasn't running into anything with that08:32
*** penick has quit IRC08:33
*** penick has joined #tripleo08:34
*** jistr has joined #tripleo08:35
*** ohamada has quit IRC08:38
*** ohamada has joined #tripleo08:38
*** yamahata has quit IRC08:38
*** markmc has joined #tripleo08:42
d0ugalOn PyPI mistral had two naming conventions :/08:49
d0ugals/naming/version/08:49
d0ugal2.0.0 is the latest released, but pip thinks 2015.1.0 is the latest version08:50
d0ugalsigh08:50
d0ugalmarios: ^ if you wondered.08:50
*** electrofelix has joined #tripleo08:54
openstackgerritsandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI  https://review.openstack.org/28778408:54
*** gfidente has joined #tripleo08:55
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-pacemaker: WIP: integrate PCS provider in the merge.  https://review.openstack.org/31071308:55
*** jayg|g0n3 has quit IRC08:55
*** bvandenh has quit IRC09:00
*** radez has quit IRC09:00
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-pacemaker: WIP: integrate PCS provider in the merge.  https://review.openstack.org/31071309:00
*** bvandenh has joined #tripleo09:00
*** pcaruana is now known as pcaruana|afk|09:01
*** jtomasek_ has joined #tripleo09:04
*** openstackgerrit has quit IRC09:04
*** openstackgerrit has joined #tripleo09:04
mariosd0ugal: thanks. i scrolled down to see the version it was getting for msitral (i mean in the error output i linked on the review)09:14
mariosd0ugal: and saw that "2015.1.0" string but didn't give it any further thought09:14
* marios brb09:14
shardyAnyone have any idea why tripleo.sh --delorean-build openstack/heat has started trying to build puppet-heat?09:15
*** jtomasek has quit IRC09:22
mariosd0ugal: nice one on the !=2015.1 ... the error message from the py27 tests is not very helpful then for this root cause :/09:22
*** bvandenh has quit IRC09:23
shardyscripts/map-project-name no worky09:23
d0ugalYeah :/ good reason to not change how you version things. I just updated the review to hopefully get CI to pass.09:24
mariosd0ugal: yup thx revoted there too :)09:25
d0ugalThanks!09:25
*** florianf has quit IRC09:34
*** jtomasek_ is now known as jtomasek09:35
*** flfuchs has joined #tripleo09:35
*** bvandenh has joined #tripleo09:37
*** flfuchs has quit IRC09:41
*** flfuchs has joined #tripleo09:41
*** milan_ has quit IRC09:49
*** mgould has joined #tripleo09:55
*** zoliXXL is now known as zoli|brb09:57
*** tosky has joined #tripleo09:57
*** flfuchs has quit IRC10:00
hewbroccaHello all10:05
shardyhey hewbrocca10:05
hewbroccashardy: given the CI rack is down for maintenance indefinitely, we need to start working on an alternative strategy like ASAP10:06
*** florianf has joined #tripleo10:06
*** tosky has quit IRC10:06
*** tosky has joined #tripleo10:06
hewbroccashardy: slagle was talking about running ironic-less jobs directly on infra... do we think that's possible?10:07
shardyhewbrocca: derekh has been in discussions re running on rdocloud resources but last I heard there's not yet sufficient capacity10:07
shardyhewbrocca: when you say "indefinitely", I was expecting some delay while we rebuild the bastion host, but do you have some more dire news?10:08
*** dciabrin has joined #tripleo10:08
shardyhewbrocca: it probably is possible, but it's not going to be a quick thing to set up10:08
hewbroccaNothing beyond that, no10:09
hewbroccaIt's just that I have no idea how long rebuilding the bastion host is going to take10:09
*** zoli|brb is now known as zoli10:09
bandinijistr, gfidente, marios: do we have some docs around about how updates are supposed to be invoked and how they work? ( dciabrin and I need to investigate the update part of a BZ and would love to not harass you guys too much ;)10:09
*** zoli is now known as zoliXXL10:09
shardyhewbrocca: my assumption was that we should be able to do that today, if that proves unrealistic you're right we'll need a plan-b10:09
bandininote updates and not upgrades ;)10:09
shardyit's unfortunate derekh isn't around right now, but I'm planning to sync with dprince, slagle and bnemec re our options when they come online10:10
hewbroccashardy: yeah... I'm not filled with confidence that that will happen10:10
hewbroccaOK, good. Thanks.10:10
mariosbandini: sec i'll get the link i used yesterday10:10
bandinimarios: you are made of awesome sir. ANJ thanks you10:10
*** olap has quit IRC10:11
mariosbandini: this iswhat i had in mind https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Updating_the_Overcloud.html10:13
bandinimarios: merci!10:14
mariosbandini: should cover version specific stuff, np man, for ANJ, anytime10:15
bandinimarios: lol ack. We'll start digging. We will probably have a question or two (this is about https://bugzilla.redhat.com/show_bug.cgi?id=1323305#c10) but let us read through it first ;)10:15
openstackbugzilla.redhat.com bug 1323305 in rhel-osp-director "Password not required to login as root to MariaDB" [Medium,New] - Assigned to mbayer10:15
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds memcached as a composable service  https://review.openstack.org/31289910:16
*** chem has quit IRC10:16
*** zaneb has quit IRC10:35
*** zaneb has joined #tripleo10:35
*** dciabrin is now known as dciabrin|lunch10:39
*** zoliXXL is now known as zoli|lunch10:43
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Adds the base and pacemaker profile for the memcached service  https://review.openstack.org/31287710:43
*** mburned_out is now known as mburned10:49
*** bvandenh has quit IRC10:54
*** dciabrin|lunch is now known as dciabrin11:01
d0ugalIs it just me, or is this file now used or imported anywhere? https://github.com/openstack/tripleo-common/blob/master/tripleo_common/filters/list.py11:04
d0ugalbnemec: ^ :)11:04
*** olap has joined #tripleo11:08
*** mgould has quit IRC11:12
*** pkovar has joined #tripleo11:12
*** tosky has quit IRC11:14
*** ccamacho is now known as ccamacho|lunch11:15
*** zoli|lunch is now known as zoli11:16
*** hewbrocca is now known as hewbrocca-afk11:20
*** julim has joined #tripleo11:22
*** mgould has joined #tripleo11:22
*** rain has joined #tripleo11:25
*** rain is now known as leanderthal11:25
*** jaosorior has quit IRC11:28
*** jaosorior has joined #tripleo11:29
*** stendulker has quit IRC11:30
*** jayg|g0n3 has joined #tripleo11:31
*** thrash|g0ne is now known as thrash11:31
*** jayg|g0n3 is now known as jayg11:31
*** jprovazn has quit IRC11:34
*** milan_ has joined #tripleo11:38
*** coolsvap has quit IRC11:38
*** weshay has joined #tripleo11:43
shardyd0ugal: it is used https://github.com/openstack/instack-undercloud/commit/75660dbe5aceeaf1770f72772ec5d4d170f4b64311:50
shardywe need it for the predictable node placement via nova scheduler hints11:50
*** jprovazn has joined #tripleo11:54
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: composable heat services  https://review.openstack.org/31357711:56
thrashshardy: should we be doing the endpoint creation in puppet-tripleo? I think that's where we are headed...11:56
*** absubram has quit IRC11:56
thrashshardy: based on this: https://review.openstack.org/#/c/244162/4211:56
shardythrash: Yes, I didn't spot that that patch finally landed11:58
*** MaxPC has joined #tripleo11:59
*** ohamada has quit IRC11:59
shardyI guess we can add the CFN endpoint to the EndpointMap and wire it in the same way11:59
*** dprince has joined #tripleo11:59
*** ohamada has joined #tripleo11:59
shardyI was chatting with jaosorior earlier, and the next step will be figuring out how to wire in the endpoint creation data so we only create endpoints for services that are actually deployed11:59
thrashshardy: I'm thinking that's why we do it in puppet-tripleo. In each services manifest.12:00
shardythrash: there's been discussion on that here: https://review.openstack.org/#/c/304125/12:01
thrashshardy: ahh. right.12:01
shardyEmilienM said we can't do that, because we need the puppet stuff to happen where the keystone service is running12:01
thrashoic12:01
*** fragatina has joined #tripleo12:01
jaosoriorthrash, shardy: That patch only enables the data needed to create the endpoints12:03
jaosoriorit doesn't create them yet12:03
thrashshardy: in the meantime, I've added the outputs for heat::keystone::auth_cfn12:03
*** zoli is now known as zoli|relocating12:03
jaosoriorbut yeah, thrash, as I was discussing with shardy earlier. Right now a problem is that since step 6 doesn't exist anymore. We can't asure that keystone is up and running12:04
jaosoriorand thus the creation of endpoints fails sporadically in step 512:04
shardyjaosorior: right, so we can wire in the *url data via the per-service templaetes, but still need a suitable step to trigger creating the endpoints12:04
jaosorioryep12:04
*** zoli|relocating is now known as zoli_gone-proxy12:06
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: add heat-api-cfn to endpoint map  https://review.openstack.org/31550412:08
shardythrash: for now I think you can rebase on top of that, and make heat-api-cfn.yaml reference the HeatCfn* from the EndpointMap12:08
thrashshardy: excellent.12:09
shardyheat should still start OK without those removed config values, but we'll need to wire in the cfn endpoint before certain things will be fully working12:09
thrashI will work on that.12:09
shardythrash: cool - when you're happy that's working we'll have to figure out what to do re HeatAuthEncryptionKey12:10
thrashshardy: ack12:10
*** rodrigods has quit IRC12:15
openstackgerritLeif Madsen proposed openstack/tripleo-quickstart: Update contributing.md to match reality  https://review.openstack.org/31412712:15
*** rodrigods has joined #tripleo12:16
*** fpan has joined #tripleo12:17
*** pradk has joined #tripleo12:17
jtomasekrbrady: ping12:18
rbradyjtomasek: yo12:18
jtomasekrbrady: I am wondering what functionality I can count on in nodes registration. Is it going to be notifying via zaqar with every successful node registered?12:19
rbradyjtomasek: I haven't gotten that far yet, so I guess you have a little bit of room to define what it is you want it to do12:20
*** michchap has joined #tripleo12:20
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: add heat-api-cfn to endpoint map  https://review.openstack.org/31550412:21
jtomasekrbrady: I am going to send you an array of nodes as an input for the workflow. Then it would be nice if the workflow notified via zaqar that a certain node from the batch has been succesfully registered (or failed)12:22
rbradyjtomasek: ack12:23
*** michchap_ has quit IRC12:23
ansiwenfinally I've got my 64GB machine \o/12:23
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: composable heat services  https://review.openstack.org/31357712:23
thrashshardy: I rebased your patch, then rebased onto that.12:24
shardythrash: ack, thanks12:24
jtomasekrbrady: thx!12:25
EmilienMshardy: I finally found an alternative so we can run endpoint/services resources from another node12:27
EmilienMshardy: it will require a class, from a module we didn't use yet, openstack_extras, we have a class that manage /root/openrc file, that is read by Puppet catalog when creating endpoints/services12:27
jaosoriorEmilienM: Can you ellaborate?12:28
*** julim has quit IRC12:28
EmilienMjaosorior: ok, let me explain12:28
mandrerbrady: hi, will you have time to do a review of https://review.openstack.org/#/c/313632/ sometime soon? I'd appreciate your feedback. No rush, though12:28
rbradymandre: II read through it the other day and  I did follow along with dprinces comments there.  I'll try to look at it again today and see if there are any additional comments I can leave12:30
mandrerbrady: yeah, I'm going to address those12:31
*** morazi has joined #tripleo12:32
*** pblaho has quit IRC12:33
jaosoriorEmilienM: One possibility would be to enable step 6 again... but basically it would only run manifests dedicated to setting up service users and endpoints12:33
EmilienMso in Puppet OpenStack, we have some Ruby providers, that create some resource, like Keystone endpoints, Keystone services, Glance images, etc. They all use (or almost?) python-openstackclient, which means you need to auth against keystone first to get a token and then do your stuff. We have 2 methods of auth 1) read /etc/keystone/keystone.conf and get token value 2) create /root/openrc and read it. If you12:33
EmilienMwant to see this code, please look https://github.com/openstack/puppet-openstacklib/blob/master/lib/puppet/provider/openstack/auth.rb -- which means, if you wwant to use providers using 1) method, you need to run the classes where keystone server runs (keystone.conf). It's secure, because you don't need to store /root/openrc and everything is run from one node (most of our users in production do that). For12:33
EmilienM2), yes it's more flexible, because we have a class that can create /root/openrc for you, so Puppet catalog will be able to read the file and run the CLI. Yes it's flexible, so you can run the providers from anywhere (if you have openrc class), but less secure, because you distribute openrc on different nodes and expose it (even if it's in /root...)12:33
*** michchap_ has joined #tripleo12:33
EmilienMno: do not enable step 6 anymore. Stop thinking about more steps. We don't need that, we don't want that. Our deployments are already too long12:34
*** michchap has quit IRC12:34
*** rhallisey has joined #tripleo12:35
EmilienMnow, if you want my opinion, I would go 1). It's more secure, and that's how most of production deployments do.12:35
jaosoriorEmilienM: I'm not talking about running the whole catalog again. Just a manifest that does the keystone parts12:35
jaosoriorlets read12:36
EmilienMnope, we need to stop "packstack", we run a single catalog with all resources inside12:36
EmilienMthat's how Puppet needs to be used12:36
EmilienMstop thinking* I mean12:36
jaosoriorI didn't work with packstack dude, no idea how that works12:37
EmilienMwe don't run snippets of Puppet, because you expose the risk of missing dependency in catalog12:37
EmilienMwell, it works like this. and it's terrible12:37
jaosoriorSo, alright, I see your option. So you're talking about doing that after the deployment is done?12:37
*** rlandy has quit IRC12:38
*** pblaho has joined #tripleo12:38
*** ccamacho|lunch is now known as ccamacho12:38
jaosoriorwhere would it be called?12:38
EmilienMdoing what?12:38
*** rlandy has joined #tripleo12:38
jaosorioralright12:38
EmilienMwe have 5 steps now, and we won't have more12:38
jaosoriorso the issue is that we need to create the keystone endpoints somewhere else than in os-cloud-config (cause that's a hack)12:39
EmilienMif you search a place where endpoint resources should live, it's in keystone role12:39
jaosoriorthe proposal was to do that in puppet12:39
EmilienMright12:39
jaosoriorstep 5 doesn't asure us keystone being up12:39
EmilienMright, we did that a very long time ago and it was reverted.12:39
jaosoriorso now it's failing12:39
EmilienMmaybe you need orchestration12:40
EmilienMshow me the patch12:40
jaosoriorEmilienM: https://review.openstack.org/#/c/304125/12:40
jaosoriorI pointed to this patch when you were deleting step 612:40
jaosoriorjust to be taken into account12:40
EmilienMI've noticed a lot of patches sent in tripleo missed that Puppet is able to do some orchestration, and we ended up to 7 steps12:40
michchap_why isn't keystone assured of being up at the end of step 5?12:41
EmilienMmichchap_: right, we can have tags, resource ordering, etc12:41
jaosoriorwell, you guys know puppet better than I do. I'm open to ideas12:41
EmilienMmichchap_: see https://review.openstack.org/#/c/304125/12/manifests/profile/base/keystone.pp12:41
jaosoriorif you say we can do it in step 5, sure. Maybe I'm missing something (hopefully), and we can make it work12:42
michchap_EmilienM: yep this is the same conversation as gfidente and I had about DBs :)12:42
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove some unused parameters from overcloud.yaml  https://review.openstack.org/31552312:42
jaosoriorbut this is a blocker for TLS everywhere :/ so would sure like to fix this12:42
jaosoriormichchap_: Did you guys get to some conclusion regarding the DBs? Maybe a similar solution could work here12:43
michchap_EmilienM: yes, and shardy is right from a conceptual and extensibility standpoint12:44
michchap_jaosorior: ^12:44
gfidentemichchap_, for the database we use different step12:44
*** mburned is now known as mburned_out12:44
EmilienMI'm replying in Gerrit and proposing something12:44
gfidentesorry that was for jaosorior  :)12:44
michchap_gfidente: right. the problem in this case is that keystone might restart on another node while we try to make endpoints.12:44
jaosoriormichchap_, gfidente: That was what I was proposing to EmilienM, but he was quite adamant that it's not the way to go.12:44
gfidenteso the comment shardy left in that submission12:45
gfidenteis the same I had in patchset 612:45
gfidenteto create the endpoints from the role12:45
gfidenteso first I'd move those within the role12:46
gfidentethen, I know EmilienM will hate me, but I think using an additional step is totally fine12:46
michchap_jaosorior: which is why it would need a dedicated step if we did it that way. I think what would be better is a class with a bool param for each endpoint that defaults to false, included on keystone role, and each other service can enable their own bool as needed.12:46
gfidenteI was indeed of the opinion that they would come back ... maybe not 7, but 6 at least12:46
gfidentebecause steps are what give us some guarantee about the "global" state12:47
michchap_jaosorior: that way the implementation is centralised which is not great, but at least we know keystone won't go down during the request and also we don't need another step.12:47
gfidenteand that's what we need to create the endpoints IMHO12:47
michchap_gfidente: we don't even know that though - if keystone restarts during step N, you can't make any API calls to it from other services.12:47
jaosoriorexactly12:47
EmilienMguys, while you're talking I'm proposing something12:47
gfidentewe point to a VIP12:48
*** mburned_out is now known as mburned12:48
gfidentenot to the local keystone12:48
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: composable heat services  https://review.openstack.org/31357712:48
michchap_gfidente: but during the request it could go down, or the DB could go away, and we get a 500.12:48
*** jpena is now known as jpena|lunch12:48
gfidenteso the DB won't go away because it was setup in previous step12:48
jaosoriorgfidente: But the service could go down (which is what kept happening after step 6 was removed)12:49
michchap_gfidente: if there are any keystone restarts due to config changes, it can 500 on a request12:49
gfidenteyeah I am for keeping step 612:49
EmilienMno12:49
EmilienMno more steps, I'm super serious12:50
jaosoriorEmilienM: michchap_ raises some good points up there ^^12:50
EmilienMI didn't read, I'm still in Gerrit, making a proposal12:50
*** athomas has quit IRC12:50
jaosoriorok12:50
EmilienMbut no more steps12:50
EmilienMthat is a bad usage of Puppet to do that12:50
michchap_EmilienM: we can do this without more steps, just make a profile class for endpoints and allow services to pass params to it.12:50
gfidenteEmilienM, no it's not bad usage12:50
EmilienMcan you guys let me write my proposal in Gerrit?12:50
gfidentethey buy us different features12:50
gfidenteEmilienM, requires can orchestrate the order in which operations happen on a single node12:51
gfidentesteps orchestrate across nodes12:51
*** dixiaoli has quit IRC12:51
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Add gnocchi profiles  https://review.openstack.org/31552712:51
gfidenteI don't think that trimming on one step just to make CI a little faster is worth the cost12:51
gfidenteseriously12:52
shardygfidente: have you seen the number of jobs hitting the CI global timeout lately?!12:52
michchap_steps can't orchestrate individual resources though, so if a service restarts from a config change during the same step as that service is used on another profile, there will be timing based failures.12:52
gfidenteshardy, but deploying on baremetal takes 20mins12:52
gfidentenot 2h12:52
gfidenteand I never measured how long it takes to create the undercloud12:52
gfidentebut it's not to do with the steps12:52
gfidentethe undercloud installation12:52
shardygfidente: about 10mins12:52
gfidenteso insisting on the steps, a single one in this case12:52
michchap_I think we're setting up for very bad times if we have to document every service restart in every step and be aware of that when making requests from other profiles.12:52
gfidentealso, to be honest, steps did the job until yesterday12:53
gfidentewe'd have to redo the same job12:53
*** ohamada has quit IRC12:53
EmilienMshardy, jaosorior, gfidente, michchap_, please read my comment on https://review.openstack.org/#/c/304125/12/manifests/profile/base/keystone.pp12:54
michchap_EmilienM: good point about the auth required for cross service requests.12:55
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable roles within services - NTP  https://review.openstack.org/31042112:55
jaosoriorEmilienM: I gave it a read and I fail to see where that falls on the deployment process12:56
EmilienMgfidente: I'm not sure you realize all the CI failures because of timeouts we had lately. Deployments are just too long. We need to fix that another way than just adding more steps.12:56
EmilienMwe have the tools to make it12:56
gfidenteEmilienM, it's not steps which make it slow12:56
EmilienMit is man12:57
gfidenteseriously12:57
EmilienMit is12:57
*** julim has joined #tripleo12:57
gfidentethat is not going to improve image building or undercloud deployment12:57
EmilienMI dropped step7 that did NOTHING an we saved 14 min12:57
gfidenteit's just making overcloud faster12:57
gfidenteand overcloud takes 20mins on a real environment12:57
gfidentewith 7 steps12:57
EmilienMand when I dropped step 6 we also save ~20 min12:57
gfidenteso I agree with you, 7 needs to go away because we don't need12:57
gfidentebut, insisting on 6 which actually turns useful12:57
*** athomas has joined #tripleo12:57
gfidentelooks more an emergency measure12:58
michchap_I can write a patch now for the keystone profile that fixes this, as long as you're ok with doing it that way.12:58
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Configurable enabled plugins list for Sahara  https://review.openstack.org/29843012:58
gfidentefor example12:59
EmilienMmichchap_: fix what?12:59
gfidenteI think the endpoints should be created by the roles12:59
gfidenteso if to sacrifice step6 we need to use the booleans in the keystone class, like michchap_ suggested12:59
gfidenteI would rather not mess there, and use step613:00
gfidenteI totally understand that michchap_ is trying to find a solution13:00
*** [1]cdearborn has joined #tripleo13:00
*** morazi has quit IRC13:00
michchap_EmilienM: the patch you commented on just needs parameters to turn each endpoint on/off and then it's good.13:00
michchap_EmilienM: it's still composable and it will work with ordering because it's on the keystone profile.13:00
gfidenteso we'd have to understand based on the roles deployed13:00
gfidentewhich of those booleans needs to be enabled13:01
gfidenteand heat can't help us doing that13:01
gfidente*instead*13:01
*** pcaruana|afk| is now known as pcaruana13:01
*** chem has joined #tripleo13:01
jaosoriormichchap_: Actually that patch is failing with a 500 error while trying to setup an endpoint, because keystone restarted at some point13:01
michchap_gfidente: oh, it can't?13:01
gfidenteheat can do one more step to make sure each role can add its own endpoint after keystone gets ready13:01
gfidentemichchap_, no heat can't do conditionals13:01
gfidenteso to enable/disable a boolean we'd need to introspect the list of roles enabled13:01
gfidenteI mean13:01
gfidenteit looks to me totally not worth it13:02
*** [1]cdearborn has quit IRC13:02
pabelangerEmilienM: +1 for stop thinking about more steps13:02
michchap_jaosorior: oh, is that because it hits the VIP and the other controllers might be restarting keystone?13:02
jaosoriorI honestly don't have a strong opinion on how it should work. As long as we get this working. This is a pretty big blocker for me :/13:02
shardygfidente: the whole point of composable services is to enable conditional inclusionn of service specific data13:02
jaosoriormichchap_ That's what I could gather from the logs13:02
shardysurely we can output data from the per-service templates that is input to puppet in a way that it can be used for the conditionals?13:02
gfidenteshardy, yeah to enable or disable creation of a boolean in the keystone class, from the glance service13:03
*** morazi has joined #tripleo13:03
michchap_shardy: will the data for eg. heat role get onto a node with the keystone role?13:03
michchap_shardy: it has to cross node boundaries if that makes sense13:03
shardygfidente, michchap_: right now it doesn't. but we may need to wire in an interface to enable it13:04
*** absubram has joined #tripleo13:04
EmilienMmichchap_: I would try to follow my 2) proposal13:04
EmilienMmichchap_: using openrc and the right orchestration13:04
michchap_shardy: yeah, we might need a common dumping ground for cluster wide info.13:04
EmilienMand figure why we have a 50013:04
thrashEmilienM: +113:05
*** masco has quit IRC13:05
gfidenteok looks like I am the only one who is not scared by steps :)13:05
michchap_EmilienM: You want to run against the localhost keystone?13:05
EmilienMgfidente: I spent my whole week reading logs and catalogs13:05
EmilienMwe're basically wasting time13:05
gfidenteEmilienM, again it's buying us a feature13:06
gfidentewhich is to sync across nodes13:06
jaosoriorEmilienM: I'm not opposed to your proposal. But I would like to know where in the deployment process does that fall, and what do you mean by "right orchestration"13:06
EmilienMmichchap_: 2) is about creating /root/openrc and feed it via hieradata, so you can use providers13:06
gfidenteand it's not going to speed up anything *but* the overcloud deployment13:06
gfidenteand overcloud deployment takes 20mins with 7 steps13:06
EmilienMjaosorior: right, and we're going to help to figure13:06
michchap_EmilienM: right, and you'd point that at localhost keystone instead of the VIP?13:06
gfidenteso I understand in CI this makes an impact13:06
michchap_EmilienM: because that means the request can't go to a node that may be restarting keystone in this step?13:07
gfidentebut to me looks like not worth it as it only helps CI13:07
EmilienMmichchap_: it can't be localhost, because we're not sure heat-api will run on the same node as keystone13:07
gfidenteand makes our lives more difficult13:07
EmilienMmichchap_: we need to use the vip and find a way to determine if the vip is reachable13:07
EmilienMmichchap_: err, if keystone is reachable via the vip13:07
gfidenteand the code a bit less 'role-based'13:07
michchap_EmilienM: if it hits the VIP, we don't know if the underlying service might be restarting during this step, and if it does we can get an error13:07
EmilienMmichchap_: do you remember https://github.com/voxpupuli/puppet-healthcheck ?13:08
michchap_EmilienM: actually can we just retry?13:08
pabelangergfidente: as somebody who has tried for the past week to land a patch in tripleo-ci, it is super frustrating how long the CI jobs takes (and unstable because of timeouts). Anything that can be done to reduce the deployment time in the CI pipeline gets my vote, as I'll be a happier developer.13:09
michchap_EmilienM: yeah but if it passes the check and then restarts a second later when we're making the API call we still get an error.13:09
EmilienMright13:09
shardypabelanger: you're not alone ;)13:10
michchap_EmilienM: so that's why I say do it in the keystone profile and do it against the localhost bind on pacemaker_master13:10
rdopieranot a new idea, but how about keeping stats on which tests fail the most often, and running those tests first, so that it fails faster?13:10
shardywe need to optimize everywhere, and improving the overcloud deployment time is really important from a CI, operator experience and developer productivity perspective13:11
shardyIf we shave 5 mins off the overcloud deployment time I probably save an hour a day in my local deployment workflow, as will all other developers13:11
michchap_I think a more general comment as well is that this is going to be a continual problem. I wrote a similar system using consul notifies and I had 44 steps.13:11
EmilienMmichchap_: and we could enable/disable services in service yaml13:11
michchap_So just saying that we add a step whenever we need to do things is probably not going to scale as the # of services keeps growing13:12
EmilienMmichchap_: for example, we could include all <project>::keystone::auth in keystone profile, with booleans (false by default) and set true in puppet/services/glance-api.yaml for example13:12
jaosoriorEmilienM, michchap_: Why not have the service enablement flags retrieved from hieradata?13:12
EmilienMoh wait that does not work13:12
michchap_EmilienM: because the heat/glance/etc. service can't put data in the keystone profile?13:13
*** fultonj has joined #tripleo13:13
gfidentepabelanger, and you tried your submissions on your dev environment and they were faster after step6 was removed?13:13
EmilienMjaosorior: we need something composable, and hieradata is pretty common across all nodes13:13
shardywe could wire in an endpoint_data interface to all the service templates in addition to step_config perhaps13:13
shardyand combine that (for all roles) to provide an input to the keystone configuration step?13:14
shardydprince: ^^13:14
EmilienMmichchap_: if I feed glance-api profile data, no keystone role won't read it AFIK13:14
*** links has quit IRC13:14
dprinceshardy: agree, I thought of the same13:15
dprinceshardy: gradually we'll refine this into a really slick interface for each service I think13:15
pabelangergfidente: I don't have a local dev environment, nor have tested the removal of step6.  I'm simply trying to migrate the jenkins slave from fedora-22 to centos-7.  But, if there are logs posted in the some place, I am happy to review them13:15
gfidentepabelanger, the CI jobs are short13:16
shardydprince: +1 - it'd be nice to kill the endpoint_data.yaml in favor of defining the per-service endpoints in the service templates13:16
EmilienMgfidente: define short?13:16
gfidentesorry I meant shorte13:16
gfidente*shorter13:16
gfidentewithout the steps13:16
gfidenteand we all agreed on removing 713:17
gfidenteit's the tradeoff or removing 6 we were eventually discussing13:17
jistrshardy, dprince: on a related note -- what about service manifest https://review.openstack.org/#/c/313543/ https://review.openstack.org/#/c/313544/ -- it can be used to solve the endpoint problem, and other problems too13:17
pabelangerYa, anything to make it faster / more stable gets my +113:17
pabelangereven if it impacts CI13:17
jistrshardy, dprince: basically a list of services that are deployed, that can be queried either from the node, or could be queried from hiera too13:18
jaosoriorjistr: Actually that could actually help in this case13:18
gfidentepabelanger, sure, the problem is it impacts the way we need to do the endpoints as well13:18
*** lblanchard has joined #tripleo13:18
*** chem has quit IRC13:19
pabelangergfidente: Agreed, but it sounds like EmilienM and michchap_ have some thoughts about that.  Not saying it is an easy fix, but something that should be considered13:20
shardyjistr: that's interesting - we'd still have to figure out how that works when there are multiple roles (e.g multiple *ServiceChain resources)13:20
*** coolsvap has joined #tripleo13:21
shardyas the problem here is distributing data over potentially multiple roles (e.g how does "Controller" running keystone know heat-api is running on a custom "HeatController" role)13:21
*** chem has joined #tripleo13:21
gfidentepabelanger, yeah and I am certainly not going to -1 that13:21
michchap_jaosorior: I think one way to make your patch pass reliably is to make it run those commands against the local Keystone instead of the VIP using one of Emilien's auth methods, if you want to try for a quicker fix.13:22
jistrshardy: right, i see, that's not solved by the service manifest currently. Perhaps we need a per-role service manifest ("what do i need to upgrade on this node?") as well as global service manifst ("what endpoints should i create?")13:22
shardyjistr: Yeah, I'm basically proposing a global interface, which could be generic or specific to endpoints13:22
jaosoriormichchap_: Well, it is going to get -1ed to oblivion with the current proposal. So if people have ideas on how to make it more composable, they could work. Or would that be further patches?13:23
shardye.g we could just define an endpoint_data interface for all services13:23
shardyand put the data right into the mapping defined for each service13:23
shardyno puppet magic required ;)13:23
jaosoriormichchap_, shardy, EmilienM: It does sound to me like having flags that tell us which services are enabled could be done with hieradata though. Is that considered a dirty solution? Or what's the problem with that?13:24
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Write service manifest on the controllers  https://review.openstack.org/31354413:24
michchap_jaosorior: if puppet is going to be used to run the commands this issue won't go away regardless of composability.13:24
jaosoriormichchap_ I agree13:24
shardyjaosorior: the problem is the hieradata won't necessarily end up on the right nodes13:24
michchap_jaosorior: no that's a fine solution, but service X can't give data to Service Y (Keystone) to tell it to make the endpoint.13:24
jistrshardy, dprince, jaosorior: i'd lean towards a generic interface perhaps, where we could exchange more information between roles. I'd imagine it's not only about "what are the keystone endpoints (== haproxy binds)" but it would also about "what are the backend servers (== haproxy backends)"13:24
michchap_jaosorior: so the discussion is how to add that mechanism.13:25
shardye.g if there is a dedicated group of nodes running heat-api that hieradata won't end up on the node running keystone13:25
shardy(just using heat-api as an example)13:25
shardyjistr: sure - I'd personally prefer something with a defined structure tho vs a freeform interface13:26
shardywe want to make it really easy for service template authors to plug in their data in a common way13:26
jaosoriorshardy: why not just pass all the endpoint related data to the keystone profile and that way we make sure it's on the right place? it's pretty much what michchap_ had proposed. Makes a lot of sense13:26
shardyjaosorior: yeah, we just need to define a sane interface to enable that13:27
*** Goneri has joined #tripleo13:27
michchap_shardy: just a random comment, but we're going to have the same convo about haproxy listen/backends13:28
jaosoriortrue that13:28
michchap_and we dodged the DB case by using step3, which had nothing in it.13:28
shardymichchap_: ack, we'll have to do as jistr suggest then and define a data blob flexible enough to support both requirements13:28
gfidentemichchap_, well no the DB was a different problem, it was about orchestrating the process ... distributing the list of enabled backends in haproxy can't be done with steps, it's about knowing which roles are deployed and where13:30
gfidentefor the DB we just needed to make sure it was stable before moving forward13:30
michchap_the db case is the same as the endpoint one, though.13:31
gfidentesame as endpoint yes13:31
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add a service manifest  https://review.openstack.org/31354313:31
gfidentewhich goes back to step 6 ...13:31
*** tzumainn has joined #tripleo13:31
gfidente... the problem of distributing non-role-pertinent in keystone13:31
gfidenteis because we decided to remove step 613:31
jistrshardy, michchap_: we can always document some "well-known structure" of the data blob for the most usual cases (keystone endpoints, haproxy listens), and leave the rest for generic usage13:32
gfidentebut I don't want to start this conversation again, seriously :)13:32
*** fragatina has quit IRC13:32
jaosoriormichchap_, EmilienM: Is openstack_extras already included in the overcloud images?13:34
jistrshardy: only i wonder how do we make the merging happen sufficiently via map_merge, so that we can merge the sub-maps from the blob, rather than overwrite... we could use not just merging the whole blob, but perhaps something like deep-merge at least on one more level (think how merging environment files works wrt resource_registry)13:34
michchap_jaosorior: almost certainly13:34
EmilienMyes it's in OPM13:35
*** ramishra has quit IRC13:35
jaosoriorChecking how to use what13:36
shardyjistr: Yeah, we'll have to give that some thought - I was thinking mostly of the endpoint data where the keys should not overlap13:38
*** ramishra has joined #tripleo13:38
jistras a side note, i think gfidente has a point that we should be careful how far we optimize. Trading some time saving for increased complexity of the puppet manifest might not always be the best thing to do.13:40
jistrin theory we can deploy everything in a single step. Astapor did that.13:40
*** fragatina has joined #tripleo13:41
jistrbut see how the manifest looks like with arrows and anchor resources13:41
jistrhttps://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/pacemaker/galera.pp13:41
jistrit's hard to comprehend all the order dependencies there without knowing what's written in other manifests13:41
*** dtantsur is now known as dtantsur|bbl13:42
adarazslarsks: hi, when you have some time, can you take a look at this change and see if it looks okay (for quickstart being able to inject gating rpms to images)? https://review.openstack.org/31357013:42
gfidenteand also I think we couldn't do that unless we deploy all roles on same host13:42
gfidentebecause we won't necessarily have the depending resources in the same manifest13:43
larsksadarazs: absolutely.13:43
adarazslarsks: awesome. thanks. :)13:43
jistrgfidente: +1, good point. Puppet arrows only work on a single node. If we want to allow splitting the services to different nodes, and we need some ordering between those services, then steps are the only way to achieve that. Arrows won't help.13:44
EmilienMdtantsur|bbl: thx for your review!13:45
jistrEmilienM shardy ^^13:45
larsksadarazs: what calls the inject_gating_repo.sh script?13:45
EmilienMI nkow arrows won't help on multi node13:45
EmilienMbut I'm sure there is a way to tell the keystone node to execute the endpoints that we want13:46
adarazslarsks: https://review.openstack.org/#/c/313570/2/playbooks/roles/libvirt/setup/undercloud/tasks/inject_gating_repo.yml -- it's called on line 1113:46
EmilienMif 1) we manage to tell keystone profile which endpoints we want 2) keep managing endpoints at step5 on pacemaker_master, I think we're good13:47
EmilienMjistr: we're not talking about a single step13:47
larsksadarazs: oh, right, missed that the task was running a command in addition to install the tar.gz file. Thanks!13:47
EmilienMjistr: but 5 steps is good enough13:47
*** tosky has joined #tripleo13:49
openstackgerritMerged openstack/tripleo-quickstart: Add gating repo injecting for undercloud  https://review.openstack.org/31357013:50
adarazswoot \o/ :)13:50
jaosorioradarazs: Any way to get the delorean-current with the latest t-h-t and stuff with tripleo-quickstart? I think at the moment it's using stable/mitaka13:51
larsksadarazs: i'm a little suspicious of this model, as noted in my review comment, but it sounds like you need this right now to Actual Things...13:51
bnemecd0ugal: It isn't imported, it's returned from https://github.com/openstack/tripleo-common/blob/master/tripleo_common/filters/list.py13:52
bnemecWhich is configured for use in https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L32513:52
jaosorioror larsks ^^13:52
adarazslarsks: happy to hear improvements on the idea, but yeah, I'm trying to whip up some gating using quickstart.13:53
larsksjaosorior: you can certainly point the quickstart at an image from master, rather than using the mitaka or liberty undercloud images.13:53
larsksjaosorior: this is what we run in CI: https://gist.github.com/larsks/f3733820bac3ff4739b9baf65691b62113:53
dprincejistr: yes. the service manifest idea could help us I think13:54
jaosoriorlarsks: Thanks dude!13:55
adarazsjaosorior: so, I made this: https://github.com/redhat-openstack/ansible-role-tripleo-gate -- if you'd pass master as refspec to it, I think it would build the latest THT and would inject it into whatever image you're pointing it to with tripleo-quickstart.13:55
jistrdprince: cool13:55
EmilienMjistr: I'm currently reading again your patches and see how we could use it13:55
adarazsjaosorior: did this answer your question? :)13:55
*** zoli_gone-proxy is now known as zoliXXL13:55
*** jpena|lunch is now known as jpena13:56
adarazsjaosorior: oh you mean if it's possible to do all this on master? I think the tripleo-quickstart promote job is now passing on master, so it should work, but I only tested on stable for now.13:56
*** egafford has joined #tripleo13:56
EmilienMjistr: with your work on "service manifest", I don't see how the keystone profile can know which endpoints need to be created13:57
jaosorioradarazs: Lets see how it goes :O13:58
adarazsI'm going to test it soon.13:59
*** fragatina has quit IRC13:59
*** ebalduf has joined #tripleo13:59
d0ugalbnemec: aha, thanks :)14:00
jistrEmilienM: it will not know the endpoints, it will only say which services are deployed, and based on that you can select from endpoint info, which would be defined elsewhere. There's a persisting problem though, as shardy mentioned, that this will only know about the services deployed on the same node as keystone. If we want to create a different node role in the future, where we also deploy APIs, we'll have to collect the service map14:00
jistrglobally, or make the "information exchange blob" we talked about earlier.14:00
*** penick has quit IRC14:00
d0ugalbnemec: I was just looking at the code coverage and list.py looked like a candidate for deletion - but it seems not!14:00
EmilienMjistr: right14:01
bnemecd0ugal: :-)14:02
jistrEmilienM, dprince: we'll probably need the blob anyway to exchange other type of info than just endpoints, so i'd be +1 for that approach. We just didn't figure out how to make it happen in Heat yet i think, given that the tools we have are mostly list_join and map_merge, and we could use a map_deep_merge in fact...14:02
gfidentejistr, yeah the haproxy/backends is one of those cases14:07
gfidenteI think michchap_ raised it earlier14:07
shardyjistr: we could wire in an interface and prove it with just map_merge for endpoints, then address the deep_merge use-case afterwards14:09
jistrshardy: yea that sounds fine14:11
*** d0ugal has quit IRC14:14
*** d0ugal has joined #tripleo14:16
*** d0ugal has quit IRC14:23
*** bvandenh has joined #tripleo14:32
*** saneax is now known as saneax_AFK14:34
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-pacemaker: WIP: integrate PCS provider in the merge.  https://review.openstack.org/31071314:35
*** links has joined #tripleo14:36
jaosoriormichchap_ here's the log of the failure: http://logs.openstack.org/25/304125/12/check-tripleo/gate-tripleo-ci-f22-ha/02ef967/ from what I could gather it was failing with a 500 error when trying to access keystone14:37
*** mcornea has quit IRC14:38
*** thrash is now known as thrash|afk14:40
*** stendulker has joined #tripleo14:43
*** Guest42459 is now known as abehl14:44
*** abehl is now known as anshul14:45
*** anshul is now known as Guest9747814:46
*** Guest97478 is now known as abehl14:46
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Nodes Registration  https://review.openstack.org/31513614:53
*** hewbrocca-afk is now known as hewbrocca14:55
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Replace NodeConfigIdentifiers with DeployIdentifier  https://review.openstack.org/31561614:58
*** cwolferh has joined #tripleo14:59
*** coolsvap has quit IRC15:12
*** r-mibu has quit IRC15:14
*** r-mibu has joined #tripleo15:14
*** yamahata has joined #tripleo15:16
*** ohamada has joined #tripleo15:17
openstackgerritEthan Gafford proposed openstack/tripleo-heat-templates: Pass MysqlVirtualIP via EndpointMap  https://review.openstack.org/30575115:18
openstackgerritEthan Gafford proposed openstack/tripleo-heat-templates: Trove Integration  https://review.openstack.org/23324015:18
*** d0ugal has joined #tripleo15:18
*** shakamunyi has quit IRC15:19
*** leanderthal is now known as leanderthal|afk15:22
*** ebarrera has quit IRC15:22
openstackgerritMerged openstack/tripleo-quickstart: Correct a typo in post install template  https://review.openstack.org/31483015:27
*** coolsvap has joined #tripleo15:29
*** aufi has quit IRC15:31
*** trozet has joined #tripleo15:31
*** links has quit IRC15:32
*** dmk0202 has quit IRC15:35
*** dmk0202 has joined #tripleo15:36
*** mikelk has quit IRC15:37
*** tremble has quit IRC15:49
ifarkaslucasagomes, mgould, hi, I am trying to add ironic to the overcloud but I am hitting this error15:55
ifarkasdoes it look familiar to you?15:55
ifarkashttp://paste.openstack.org/show/496937/15:55
mgouldifarkas, looking now15:55
mgouldifarkas, I haven't seen that error before15:56
* mgould takes a look at the puppet-ironic source15:57
mgouldifarkas, you may have better luck asking in #puppet-openstack15:57
ifarkasmgould, ok, thanks15:57
jaosoriorlarsks: Well, that didn't work :/ with a recently built undercloud (with tripleo quickstart) I am not able to deploy an overcloud15:58
jaosoriorgetting no valid host found errors15:59
larsksjaosorior: I see that our ci jobs for https://ci.centos.org/view/rdo/view/promotion-pipeline/job/tripleo-quickstart-promote-master-delorean-minimal/ seem to have been failing since this morning.  I'll take a look in a bit.15:59
mgouldifarkas, it's complaining that /etc/ironic/ironic.conf isn't there16:00
*** jcoufal has quit IRC16:00
mgouldI don't know where that file is expected to come from16:00
ifarkasmgould, yeah, I would expect puppet to create it16:02
mgouldyep16:02
mgouldbut OTOH, I wouldn't be surprised if some other tool puts it there and puppet just checks for its presence16:02
larsksmgould: I think that's generally provided by the package, right?16:03
*** jtomasek has quit IRC16:03
mgouldlarsks, dunno16:03
* lucasagomes looks16:04
larsksmgould: let me rephrase; on my undercloud, that file is owned by the openstack-ironic-common package.16:04
mgouldlarsks, I'm looking at https://github.com/openstack/puppet-ironic/blob/master/manifests/init.pp#L27716:04
mgoulddoes that mean "ironic-common provides /etc/ironic/ironic.conf"?16:05
lucasagomesifarkas, odd, so ironic-dbsync is created when Ironic is installed (see setup.cfg)16:05
larsksmgould: It means it's *expected* to, it looks like.16:06
mgouldOK, thanks16:06
mgouldand that package is defined on line 282?16:06
larsksAssuming that $::ironic::params::common_package_name is correct, you should have openstack-ironic-common installed, and 'rpm -ql openstack-ironic-common' should show that file...16:06
*** jaosorior has quit IRC16:06
larsksYeah.16:06
jristEmilienM: that was super helpful. thanks for letting me lurk!16:07
mgouldifarkas, can you run that command on the machine you're deploying?16:07
ifarkaslars, mgould, right, so it's not installed16:08
*** pkovar has quit IRC16:08
mgouldoh, I see ironic-common is a deb or an rpm16:08
EmilienMjrist: you're welcome16:08
ifarkasI need to figure out how to add this dependency16:09
ifarkasthanks larsks and mgould!16:09
mgouldifarkas, looks like it's there already: https://github.com/openstack/puppet-ironic/blob/master/manifests/params.pp#L3116:09
*** egafford has quit IRC16:10
ifarkasmgould, oh, that is interesting...16:10
lucasagomesifarkas, mgould I think this is https://github.com/openstack/tripleo-puppet-elements/blob/master/elements/overcloud-controller/install.d/package-installs-overcloud-controller16:11
lucasagomesthis element is responsible for installing all the software needed in the controller node16:11
mgouldifarkas, https://docs.puppet.com/puppet/latest/reference/types/package.html looks relevant16:11
mgould"Some package types (e.g., yum and apt) can retrieve their own package files, while others (e.g., rpm and sun) cannot."16:11
ifarkaslucasagomes, right, I will give it a try16:11
* mgould thought yum packages were rpms, but what do I know...16:11
*** egafford has joined #tripleo16:11
lucasagomesmgould, aren't they!?16:12
mgouldlucasagomes, the authors of the puppet docs don't think so :-)16:12
ifarkasyum is a package manager using rpm, not?16:12
mgouldthat's what I thought, yes16:12
larsksifarkas: yes, it is.16:12
lucasagomesstrange... maybe that's because of the types of rpm?16:13
mgouldmaybe you can set provider => "yum" or provider => "rpm" in Puppet16:13
lucasagomese.g rpm5 is not used by red hat16:13
lucasagomesits used on mandriva/mageia16:13
lucasagomesred hat uses rpm416:13
mgouldand they're both using RPMs under the hood, but as far as Puppet's concerned only "yum" packages are fetchable16:13
lucasagomesbut not sure if that's what they meant there16:13
mgoulddunno16:14
larskslucasagomes: there are separate "rpm" and "yum" providers available.16:14
*** dmk0202 has quit IRC16:15
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Composable manila tht side WIP  https://review.openstack.org/31565816:16
*** oshvartz has joined #tripleo16:16
lucasagomeslarsks, ok, is it a puppet thing?16:16
lucasagomeswhether install a package using rpm or yum (for depenedncy mgmt)16:17
larsksRight.16:17
*** milan_ has quit IRC16:19
lucasagomesgotcha :-)16:20
lucasagomeslarsks, thanks16:20
*** devvesa has quit IRC16:20
*** d0ugal has quit IRC16:20
*** abehl has quit IRC16:22
*** d0ugal has joined #tripleo16:22
*** oshvartz has quit IRC16:25
*** bvandenh has quit IRC16:25
*** d0ugal has quit IRC16:27
*** fragatina has joined #tripleo16:27
*** d0ugal has joined #tripleo16:42
*** michchap has joined #tripleo16:47
*** michchap_ has quit IRC16:48
openstackgerritMiles Gould proposed openstack/tripleo-common: Import initial_state logic from tripleoclient  https://review.openstack.org/30952116:50
*** stendulker has quit IRC16:50
*** michchap_ has joined #tripleo16:51
*** michchap has quit IRC16:52
*** lucasagomes is now known as lucas-dinner16:54
*** sambetts is now known as sambetts|afk16:55
*** hewbrocca is now known as hewbrocca-afk16:58
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-pacemaker: WIP: integrate PCS provider in the merge.  https://review.openstack.org/31071317:00
*** michchap has joined #tripleo17:01
*** michchap_ has quit IRC17:02
*** thrash|afk is now known as thrash17:06
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: WIP prototyping fully composable/custom roles  https://review.openstack.org/31567917:07
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: WIP prototyping fully composable/custom roles  https://review.openstack.org/31567917:09
*** michchap has quit IRC17:09
*** jpena is now known as jpena|off17:10
*** michchap has joined #tripleo17:11
*** mgould has quit IRC17:14
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: WIP prototyping fully composable/custom roles  https://review.openstack.org/31567917:14
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: WIP prototyping fully composable/custom roles  https://review.openstack.org/31567917:15
*** mburned is now known as mburned_out17:15
dtrainornice shardy17:16
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: WIP prototyping fully composable/custom roles  https://review.openstack.org/31567917:17
shardygah, messed up pushing those fixes17:17
shardyanyway, appreciate any feedback on the general approach before I commit a lot more time to it17:17
shardyit's not functional yet17:17
dtrainorhehe.  doesnt make it any less exciting17:18
ayounglarsks, I'm trying to debug some libvirt setup issue in the undercloud.  I'm looking at the defaults which say the libvirt_uri: qemu:///session  but does not way what user.  I can't use virsh as the stack user:  Cannot create user runtime directory '/run/user/1002/libvirt': Permission denied17:19
ayoungbut the calls seem to be going through from ansible.  How are they being made, and how can I connect to see the state of the virsh subsystem?17:19
openstackgerritBen Nemec proposed openstack/tripleo-docs: Recommend stopping all services before undercloud upgrade  https://review.openstack.org/31568317:19
*** michchap_ has joined #tripleo17:20
larsksayoung: if became the stack user via 'su - stack', try ssh'ing in directly as that user (ssh -i id_rsa_virt_host stack@yourvirthost) and let me know if the behavior is different.17:21
*** michchap has quit IRC17:22
larsksThe 'id_rsa_virt_host' file is in your local working directory, which defaults to ~/.quickstart.17:22
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: WIP prototyping fully composable/custom roles  https://review.openstack.org/31567917:23
*** dtantsur|bbl is now known as dtantsur17:24
*** jistr has quit IRC17:24
ayounglarsks, OK, that works.  Why?17:26
*** tosky has quit IRC17:26
larsksBecause using 'su -' doesn't handle user sessions the same way as logins, although I'm not actually sure about the technical details.  Is selinux enabled on your system?  Does 'su - stack' work if you disable selinux?17:27
larsksPossibly it's just a simple policy issue.17:27
ayounglarsks, that would make sense.   getenforce is Enforcing17:27
*** shardy has quit IRC17:27
ayounganyway, this is enough for me to debug.  Thanks17:27
larsksSure.17:28
larsksAlthough if you can grab the audit messages that correspond to the failure when accessing stack via su, I would be curious...17:28
*** d0ugal has quit IRC17:34
*** julim has quit IRC17:35
*** rdo has quit IRC17:38
*** mburned_out is now known as mburned17:44
*** rdo has joined #tripleo17:46
*** saneax_AFK is now known as saneax17:49
*** shivrao has joined #tripleo17:50
Erming_Hi, how to correctly add nodes into registration and/or re-register nodes? I added a few nodes into instackenv.json, re-imported, and then the introspection always timed out. Can I delete all of them and then re-register the nodes?17:51
Erming_I mean introspection always timed out on the new nodes.17:51
Erming_How to clean node info in the local-cache? I mean I want to start from scratch.17:52
*** d0ugal has joined #tripleo17:53
*** coolsvap has quit IRC17:57
*** pcaruana has quit IRC17:58
*** ohamada has quit IRC18:01
larsksThe RDO CI overcloud deployment is failing; nova-compute reports "OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object".  Does that ring a bell with anybody?18:02
larsksOh, maybe this: https://bugs.launchpad.net/nova/+bug/158098718:02
openstackLaunchpad bug 1580987 in OpenStack Compute (nova) ""Cannot call obj_load_attr on orphaned Instance object" in baremetal_basic_ops" [High,In progress] - Assigned to Jim Rollenhagen (jim-rollenhagen)18:02
*** dciabrin is now known as dciabrin|away18:05
*** dtantsur is now known as dtantsur|afk18:07
*** d0ugal has quit IRC18:08
*** electrofelix has quit IRC18:09
*** rcernin has quit IRC18:09
*** jistr has joined #tripleo18:15
*** jistr is now known as jistr|afk18:15
*** julim has joined #tripleo18:17
*** openstackgerrit has quit IRC18:18
*** openstackgerrit has joined #tripleo18:18
*** jtomasek has joined #tripleo18:26
*** saneax is now known as saneax_AFK18:27
*** onovy has quit IRC18:28
*** trozet has quit IRC18:28
*** onovy has joined #tripleo18:28
*** chem has quit IRC18:30
*** trozet has joined #tripleo18:43
*** dmk0202 has joined #tripleo18:47
*** TSCHAK has quit IRC18:53
*** shivrao has quit IRC18:58
*** sambetts|afk has quit IRC18:59
*** sambetts has joined #tripleo19:03
*** d0ugal has joined #tripleo19:03
*** shivrao has joined #tripleo19:05
*** fragatina has quit IRC19:08
*** trozet has quit IRC19:14
jdobegafford: ping19:24
egaffordjdob: pong19:24
egaffordjdob: What's up?19:24
jdobegafford: hey, I'm trying to get a run of https://review.openstack.org/#/c/305751/ with a heat patch in CI, but you keep blowing away my Depends-On :)19:24
jdobcan you make sure you rebase before your next patch (I'm guessing it's a dep you are changing)19:25
*** trozet has joined #tripleo19:25
egaffordjdob: Oh, sorry. :/ I'll make certain not to touch that in the future (I am actually rebasing with each patch; not certain how I'm stepping on you. Regardless, will cease to step on you.)19:26
jdobto be honest, I'm not entirely sure how you're doing it either19:26
egaffordReally sorry about that; I know that's intensely frustrating.19:26
jdobthis is one of those weird ass git edge cases19:26
jdobthat you run into once a year19:26
egaffordWell, at least now I know that I am in fact doing it, and will cease to do it in the future.19:27
jdobawesome, thanks :)19:27
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-quickstart: work around ansible issue #15744  https://review.openstack.org/31573419:28
*** athomas has quit IRC19:28
thrashegafford: make sure you 'git review -d xxxxxxx' on jdob's patch before you rebase.19:29
openstackgerritJay Dobies proposed openstack/tripleo-heat-templates: [WIP] Pass MysqlVirtualIP via EndpointMap  https://review.openstack.org/30575119:30
egaffordthrash: Cool; makes sense.19:31
*** bswartz has quit IRC19:39
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-quickstart: Explicit Teardown  https://review.openstack.org/31312719:44
openstackgerritMerged openstack/tripleo-quickstart: work around ansible issue #15744  https://review.openstack.org/31573419:45
*** fragatina has joined #tripleo19:46
*** mbound has joined #tripleo19:49
*** jistr|afk has quit IRC19:52
*** yamahata has quit IRC19:52
*** myoung|lunch has joined #tripleo19:57
*** myoung|lunch has quit IRC19:57
*** myoung|lunch has joined #tripleo19:57
*** yamahata has joined #tripleo19:58
openstackgerritayoung proposed openstack/tripleo-quickstart: WIP Allow for multiple undercloud nodes  https://review.openstack.org/31574920:03
EmilienMayoung: see my latest thread on openstack-dev [tripleo]20:04
*** lblanchard has quit IRC20:05
openstackgerritayoung proposed openstack/tripleo-quickstart: WIP Allow for multiple undercloud nodes  https://review.openstack.org/31574920:05
ayoungEmilienM, policy file changes, too20:07
EmilienMoh yeah missed that one please add it20:07
*** ebalduf has quit IRC20:09
*** yamahata has quit IRC20:09
ayoungEmilienM, and here is where I realize I've been reading mail while the vpn is down and none of the messages I read are tagged as read.20:10
*** bvandenh has joined #tripleo20:19
*** ayoung has quit IRC20:20
*** rcernin has joined #tripleo20:24
*** yamahata has joined #tripleo20:33
*** cwolferh has quit IRC20:35
*** jprovazn has quit IRC20:37
*** dbecker has quit IRC20:41
*** Goneri has quit IRC20:50
*** dprince has quit IRC20:54
*** jayg is now known as jayg|g0n321:01
*** gfidente has quit IRC21:01
*** paramite has quit IRC21:06
*** bvandenh has quit IRC21:13
*** egafford has quit IRC21:15
*** cwolferh has joined #tripleo21:21
*** fpan has quit IRC21:24
*** fpan has joined #tripleo21:28
*** dtantsur|afk has quit IRC21:33
*** fpan has quit IRC21:34
*** pino|work has quit IRC21:34
*** pino|work has joined #tripleo21:35
*** dtantsur has joined #tripleo21:35
*** dmacpher has quit IRC21:43
*** bswartz has joined #tripleo21:47
*** dprince has joined #tripleo21:49
*** morazi has quit IRC21:53
*** dmk0202 has quit IRC22:00
*** ccamacho has quit IRC22:06
*** ccamacho has joined #tripleo22:09
*** MaxPC has quit IRC22:17
*** fultonj has quit IRC22:24
*** ayoung has joined #tripleo22:25
*** absubram has quit IRC22:28
*** apetrich has quit IRC22:29
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Remove Ceilometer Alarm from the overcloud  https://review.openstack.org/28812022:29
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm  https://review.openstack.org/31579622:29
*** weshay has quit IRC22:33
*** cwolferh has quit IRC22:35
*** TSCHAK has joined #tripleo22:45
*** dprince has quit IRC22:57
*** dprince has joined #tripleo22:57
*** ayoung has quit IRC23:03
*** saneax_AFK is now known as saneax23:08
*** dmacpher has joined #tripleo23:10
*** gbraad has quit IRC23:23
*** r-mibu has quit IRC23:23
*** gbraad has joined #tripleo23:27
*** zoliXXL is now known as zoli|gone23:33
*** zoli|gone is now known as zoli_gone-proxy23:34
*** chlong has quit IRC23:35
*** cwolferh has joined #tripleo23:35
*** trozet has quit IRC23:40
*** r-mibu has joined #tripleo23:45
*** dmacpher has quit IRC23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!