Wednesday, 2016-07-20

openstackgerritJames Slagle proposed openstack/puppet-tripleo: Only sync the gnocchi db on the pacemaker master
slaglelooks like the gnocchi db sync is being run from multiple controllers, potential fix ^00:09
*** mbound has joined #tripleo00:11
*** mbound has quit IRC00:16
*** dtrainor has joined #tripleo00:16
*** morazi has quit IRC00:18
*** dtrainor has quit IRC00:19
*** morazi has joined #tripleo00:30
*** limao has joined #tripleo01:02
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move elements into dib package
*** limao_ has joined #tripleo01:16
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: Do not exit prior to the inventory of the deployment.
*** limao has quit IRC01:20
*** limao_ has quit IRC01:22
*** limao has joined #tripleo01:22
*** rwsu has quit IRC01:34
EmilienMslagle: nice catch01:35
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move diskimage-image-create to an entry point
*** yamahata has quit IRC02:08
openstackgerritJames Slagle proposed openstack/puppet-tripleo: Only sync the gnocchi db on the pacemaker master
openstackgerritDan Prince proposed openstack/instack-undercloud: Increase the max zaqar messages size to 2^19
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: WIP: Test SoftwareConfigTransport: ZAQAR_MESSAGE
openstackgerritSanjay Upadhyay proposed openstack/tripleo-heat-templates: SR-IOV nic agent changes in line with composable roles.
*** saneax is now known as saneax_AFK02:36
*** lblanchard has quit IRC02:50
*** jdob has quit IRC03:05
*** ooolpbot has joined #tripleo03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,Triaged] - Assigned to James Slagle (james-slagle)03:10
*** vkmc has quit IRC03:15
*** vkmc has joined #tripleo03:21
*** numans has joined #tripleo03:26
*** jaosorior has joined #tripleo03:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move elements into dib package
openstackgerritIan Wienand proposed openstack/diskimage-builder: Move element-info a standard entry-point
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move diskimage-image-create to an entry point
jaosoriorbnemec still around?03:31
*** julim has quit IRC03:32
jaosoriord0ugal: Are you around?03:37
jaosorioror dprince03:37
*** shivrao has quit IRC03:39
*** padkrish has joined #tripleo04:00
*** shivrao has joined #tripleo04:03
*** ooolpbot has joined #tripleo04:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,Triaged] - Assigned to James Slagle (james-slagle)04:10
*** zaneb has quit IRC04:12
*** zaneb has joined #tripleo04:29
*** akshai has joined #tripleo04:31
*** akshai has quit IRC04:34
*** d0ugal has quit IRC04:46
*** padkrish has quit IRC04:47
*** d0ugal has joined #tripleo04:47
*** dmacpher has joined #tripleo04:49
*** zaneb has quit IRC04:51
*** zaneb has joined #tripleo04:51
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move elements into dib package
openstackgerritIan Wienand proposed openstack/diskimage-builder: Move element-info a standard entry-point
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move diskimage-image-create to an entry point
openstackgerritJason E. Rist proposed openstack/tripleo-ui: Check if parentPath exists
*** ooolpbot has joined #tripleo05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,Triaged] - Assigned to James Slagle (james-slagle)05:10
*** apetrich has quit IRC05:20
*** masco has joined #tripleo05:20
*** apetrich has joined #tripleo05:20
jaosoriorcoolsvap: are you around?05:24
coolsvapjaosorior: hey05:27
jaosoriorI remember you were online at this time more or less. Are you experiencing huge lags in gerrit also?05:28
openstackgerritJuan Antonio Osorio Robles proposed openstack/python-tripleoclient: Handle empty list of introspected nodes
*** jprovazn has joined #tripleo05:30
*** jprovazn has quit IRC05:31
*** jprovazn has joined #tripleo05:31
coolsvapjaosorior: no05:31
coolsvapjaosorior: i had this trouble some days ago and I was also asking around05:32
coolsvapbut after hour or two it again got to normal05:32
jaosoriorI see05:32
jaosoriorthen I guess it must be the same case here05:33
jaosoriorwill wait then05:33
coolsvapnot sure whats the reason behind it05:33
*** shivrao_ has joined #tripleo05:36
*** shivrao has quit IRC05:39
*** shivrao_ is now known as shivrao05:39
*** radek2 has joined #tripleo05:40
coolsvapEmilienM: around?05:41
*** rcernin has quit IRC05:44
*** yamahata has joined #tripleo05:48
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move elements into dib package
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move diskimage-image-create to an entry point
*** saneax_AFK is now known as saneax05:58
*** d0ugal has quit IRC05:58
*** aufi has joined #tripleo05:59
bandinijaosorior: I saw huge gerrit lags yesterday (haven't tried today yet)06:06
*** anshul has joined #tripleo06:06
*** rcernin has joined #tripleo06:07
*** leanderthal|afk is now known as leanderthal06:08
*** anshul has quit IRC06:09
*** anshul has joined #tripleo06:10
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services
*** ooolpbot has joined #tripleo06:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,Triaged] - Assigned to James Slagle (james-slagle)06:10
*** oshvartz has joined #tripleo06:13
*** rajinir has quit IRC06:15
*** radek__ has quit IRC06:19
*** jubapa has joined #tripleo06:19
*** jprovazn has quit IRC06:22
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add pacemaker mistral profile
*** pgadiya has joined #tripleo06:24
jaosoriorccamacho hey dude, is gerrit lagging a lot for you too?06:24
*** jubapa_ has joined #tripleo06:24
*** anshul has quit IRC06:27
*** afazekas is now known as afazekas|dentist06:27
ccamachojaosorior, man more than 2 hours and the jobs are giving timeouts ;(06:28
jaosoriorsame thing with me06:28
jaosoriorit's terribe06:28
ccamacholocally works, but in CI is giving issues06:29
*** jerrygb has quit IRC06:29
*** anshul has joined #tripleo06:29
ccamachojaosorior, the message you can see it on the postci log file, right?06:31
ccamachocan you point me to your submission to compare the error?06:31
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services
bandiniccamacho: I get some postci.txt error as well in all my rechecks, btw06:41
*** d0ugal has joined #tripleo06:42
*** anshul has quit IRC06:47
*** mcornea has joined #tripleo06:49
*** mcornea has quit IRC06:49
*** tesseract- has joined #tripleo06:50
*** mcornea has joined #tripleo06:50
*** d0ugal has quit IRC06:51
*** jprovazn has joined #tripleo06:52
*** anshul has joined #tripleo06:59
mariosanyone else having gerrit issues this morning?06:59
*** oshvartz has quit IRC07:00
*** oshvartz has joined #tripleo07:00
*** rdo has quit IRC07:05
*** d0ugal has joined #tripleo07:08
*** florianf has joined #tripleo07:08
*** ooolpbot has joined #tripleo07:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,Triaged] - Assigned to James Slagle (james-slagle)07:10
*** anshul has quit IRC07:11
*** anshul has joined #tripleo07:11
*** rdo has joined #tripleo07:13
ccamachomarios, timeouts?07:14
ccamachojaosorior and me at least07:14
*** athomas has joined #tripleo07:16
*** gchamoul has quit IRC07:16
mariosccamacho: yeah is flaky comes and goes the issue07:16
*** jmiu_ has quit IRC07:17
mariosccamacho: thanks. i was starting to questino my sanity. wget was retrieveing fine and firefox was hanging for same URL at one point...07:17
ccamachoyeahp, One thing the error reported  This additional option was configured here
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,Triaged] - Assigned to James Slagle (james-slagle)07:17
*** adarazs_pto has quit IRC07:17
*** jschlueter has quit IRC07:17
mariosccamacho: oh. i mean like gerrit code review ( in general is a bit flaky for me07:18
*** liverpooler has joined #tripleo07:18
*** ebarrera has joined #tripleo07:18
ccamachomarios, yeahp, the jobs are taking ~2hours and being killed but if you see this might be related  just to let you know07:19
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,Triaged] - Assigned to James Slagle (james-slagle)07:19
*** bootsha has joined #tripleo07:19
*** jschlueter has joined #tripleo07:20
*** jmiu_ has joined #tripleo07:22
*** adarazs has joined #tripleo07:22
mariosccamacho: tx07:22
*** gchamoul has joined #tripleo07:22
*** dsariel has joined #tripleo07:25
*** ifarkas_afk is now known as ifarkas07:27
*** shardy has joined #tripleo07:27
jaosoriorccamacho: There is a fix for that bug, and that fix itself keeps timing out because of review being so flaky07:28
*** jpich has joined #tripleo07:28
ccamachojaosorior passing CI07:29
jaosoriorhand't seen it passed CI07:29
jaosoriorI had rechecked it cause of a timeout07:30
jaosoriorshardy, marios: Can you check out this ? it's a fix for this bug
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,Triaged] - Assigned to James Slagle (james-slagle)07:30
*** jerrygb has joined #tripleo07:30
*** akuznetsov has joined #tripleo07:31
mariosjaosorior: ack07:32
shardyjaosorior: 'morning, done07:32
mariosjaosorior: though you realise i'm waiting for firefox to load it still...07:32
mariosthanks shardy07:33
shardyheh, I went back to chrome for a few days but the performance with lots of tabs is even worse07:33
*** jerrygb has quit IRC07:35
*** anshul has quit IRC07:35
*** anshul has joined #tripleo07:36
openstackgerritFlavio Percoco proposed openstack/tripleo-common: Don't clean after `yum update` in script
*** tremble has joined #tripleo07:40
*** zoli_gone-proxy is now known as zoliXXL07:40
openstackgerritMerged openstack/puppet-tripleo: Only sync the gnocchi db on the pacemaker master
*** zoliXXL is now known as zoli|WFH07:42
*** zoli|WFH is now known as zoli|wfh07:42
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add pacemaker mistral profile
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services
*** d0ugal has quit IRC07:46
*** devvesa has joined #tripleo07:49
flaper87jpeeler: around? Good morning07:49
flaper87jpeeler: do you happen to know why we need all these packages?
flaper87actually, like, all of them07:50
*** derekh has joined #tripleo07:50
flaper87we're installing nova-compute in the container07:50
flaper87I guess that's because we need to collect configs for those07:50
*** yamahata has quit IRC07:51
*** anshul has quit IRC08:01
*** shivrao has quit IRC08:05
*** d0ugal has joined #tripleo08:05
*** akuznetsov has quit IRC08:08
*** ooolpbot has joined #tripleo08:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1604624 in tripleo "CI: Error: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: gnocchi-upgrade --config-file /etc/gnocchi/gnocchi.conf --skip-storage --create-legacy-resource-types returned 1 instead of one of [0]\u001b[0m\n", " [Critical,In progress] - Assigned to James Slagle (james-slagle)08:10
*** anshul has joined #tripleo08:14
*** paramite has joined #tripleo08:16
*** dmacpher has quit IRC08:18
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates: Availability monitoring agents support
*** stendulker has joined #tripleo08:19
*** zoli|wfh is now known as zoli|brb08:20
shardyflaper87: 'morning, I'm guessing jpeeler is asleep, but yeah we need those to allow creation of an external config for the containers08:21
shardyflaper87: we use puppet to create the config for each service, then pass it into the container on launch08:21
shardyI agree it would be best if we could avoid all those packages in the long term08:21
*** anshul has quit IRC08:21
shardyas well as the potential coupling between the agent container and the service container versions08:22
*** derekjhyang has quit IRC08:22
jaosoriorccamacho hey dude, do you have a running undercloud?08:22
ccamachojust installing it, but yesterday night got an error with packaging08:23
ccamachojaosorior ^08:23
jaosoriorccamacho: Could you do me a favor testing a patch?08:23
jaosoriorccamacho: If you have some spare cycles, I would really appreciate you helping me test this out
ccamachojaosorio sure thing man08:25
jaosoriorccamacho: An easy way to enable TLS for the undercloud is to add this to your undercloud.conf08:25
jaosoriorgenerate_service_certificate = true08:25
*** limao has quit IRC08:25
ccamachooki, Ill do it and ask for some guidance if needed :D08:26
jaosoriorshardy: Have you run a deployment lately? Is introspection working for you?08:26
jaosoriornow I'm not sure if it's failing because of the TLS patch or if it's broken out of the box08:27
*** gfidente has joined #tripleo08:28
*** limao has joined #tripleo08:29
*** jerrygb has joined #tripleo08:31
*** devvesa has quit IRC08:33
*** anshul has joined #tripleo08:33
openstackgerritMerged openstack/puppet-tripleo: openstack-core resource does not have interleave=true
shardyjaosorior: did the mistral error get fixed yesterday?08:35
shardyI don't see an obvious fix to tripleo-common08:35
*** jerrygb has quit IRC08:35
jaosoriorshardy: Nothing happened regarding mistral. I'm trying to test it but am actually having mistral problems08:37
jaosoriornow I'm figuring out how to debug mistral and how to stop pending executions08:37
jaosoriorsince I have a lot of stacked baremetal instrospects08:37
jaosoriorthat didn't get stopped08:37
d0ugalWhat was the Mistral error?08:37
shardyd0ugal: Oh hey, you're back08:38
d0ugalshardy: ish, I am still at EuroPython08:38
shardyintrospection was breaking in CI after we tried to re-enable it08:38
d0ugalshardy: Sitting in a session at the moment08:38
shardylooked like an interface mismatch in the action/workflow08:38
d0ugalhmm, weird. I wonder how that happened08:39
*** tobias_fiberdata has joined #tripleo08:40
jaosoriord0ugal: I'm trying to reproduce it, but mistral is in an unusable state in my machine, so it seems08:40
jaosoriorgot a lot of pending executions for introspection08:41
*** zoli|brb is now known as zoli|wfh08:41
jaosoriorand it just gets stuck because of them08:41
shardyMistral was working for me, but I've just re-run openstack undercloud install against the latest delorean repos, and now I get connection refused talking to mistral-api08:41
shardyalthough strangely introspection has started08:42
d0ugalI'll see if I have an env that works08:42
jpichI couldn't reproduce it either, the introspection times out with no specific error on a newly set up env08:42
shardyHmm, Ok weird, it was running but temporarily DoSed by the introspection of 4 nodes at once08:42
jaosoriorshardy: is mistral still running on port
jaosoriorthat might be because recently mistral was moved to run on the undercloud's LOCAL_IP like the rest of the services08:43
jaosoriorI had to kill mistral in order to run the undercloud install again08:44
shardyso it's binding on br-ctlplane08:44
hewbroccablarrgh are we still using that non-private IP range?08:44
* hewbrocca hates it08:44
jaosoriorhewbrocca: me too :/08:45
shardyhewbrocca: Yeah, there is a patch up that fixes it tho08:45
tobias_fiberdatahow could that happen lol08:45
shardybit tricky from an upgrade standpoint tho :(08:45
shardytobias_fiberdata: historical mistake unfortunately08:46
tobias_fiberdatai guess i can adjust that with updating the subnet in neutron right?08:46
tobias_fiberdatain undercloud?08:46
shardytobias_fiberdata: you can just set network_cidr in your undercloud.conf before running the undercloud install08:47
shardythat patch deprecates the old default, but operators can choose whatever range they want08:47
tobias_fiberdatayea my collegue didnt think of that when he installed the undercloud node.08:47
jaosoriorshardy: so "netstat -tulpn | grep 8989" show's only ?08:48
shardytobias_fiberdata: have you deployed any overcloud nodes yet?08:48
tobias_fiberdatanot yet no08:48
*** devvesa has joined #tripleo08:48
*** anshul has quit IRC08:48
shardytobias_fiberdata: You can probably change it then re-run openstack undercloud install08:48
tobias_fiberdatai'm still confused of all the yaml-files :P08:48
tobias_fiberdatabut i'll figure it out08:48
shardytobias_fiberdata: for a simple test deployment, just run "openstack overcloud deploy --templates"08:49
*** anshul has joined #tripleo08:49
shardybut yeah, the more complex deployments take a while to get comforable with08:49
flaper87shardy: a-ha, thanks! That's what I thought.08:50
tobias_fiberdatashardy, i dont have my hardware either. the hardware i have is nothing like the ones i will get in a couple of weeks08:50
flaper87shardy: I'm trying to make the image thinner, it's quite huge right now08:50
flaper87at least to my taste08:50
tobias_fiberdatabut 2 days left to 4w of vacation08:50
tobias_fiberdataso taht will be my collegues headache lol08:50
shardyflaper87: Yeah, I'd maybe chat with dprince or Slower_ about it later08:50
shardythey may have ideas on how to improve it08:50
flaper87shardy: yup, planning to. So far I'm optimizing the things that can be optimized based on the current image08:51
* flaper87 just saved us 100mb08:51
flaper87not much but that's something08:51
tobias_fiberdatai'm so used to vmware and having something to look at instead of staring into config-files :p08:52
shardytobias_fiberdata: there is a tripleo-ui under development which should be ready for testing pretty soon08:52
tobias_fiberdatashardy, i know, but we are going to deploy this productionenvironment in a couple of weeks/months08:53
shardytobias_fiberdata: ack, CLI is the way to go then I guess08:53
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Stop Validation
tobias_fiberdatayea it is. i just have to figure out how to solve the small differences in our environment if you compare to a normal openstack-deployment. we wont deploy with ceph, we are going to be using 10G NFS instead for all our blockservices. how that is gonna work with a full HA-environment is hard to say though. but i hope it will work out okey.08:55
*** ohamada has joined #tripleo08:56
tobias_fiberdatawe'll figure it out08:56
tobias_fiberdatathat youtube-video made me see the pattern though08:56
shardyThe deep dive one from last week?08:57
shardyCool, good to hear that was useful :)08:57
shardyWe'll do more if folks have topics they want discussed08:57
gfidentetobias_fiberdata out of curiosity, why are you going nfs for cinder?08:57
tobias_fiberdatacause we dont have anything else to use atm.08:58
*** jaosorior has quit IRC08:58
gfidentethe nfs driver is still limited though afaik, for example you can't make volume snapshots I think08:58
*** jaosorior has joined #tripleo08:58
tobias_fiberdatagfidente, it's okey, we are running it right now in production08:58
tobias_fiberdataand if it's gonna be a problem we'll probably deploy a cephcluster and integrate that as a 2nd backend08:59
*** oneswig has joined #tripleo08:59
tobias_fiberdatabut for now08:59
tobias_fiberdatawe are going to use NFS08:59
tobias_fiberdatait's great to use tbh when it comes to live-migration and such08:59
*** jaosorior is now known as jaosorior_lunch09:00
*** bvandenh has joined #tripleo09:00
*** dtantsur|afk is now known as dtantsur09:01
tobias_fiberdataperhaps not the correct place to take it though, but there's a limitation when it comes to live-migrations, you need to have the full flavor space on the computenode to be able to live-migrate it. overcommit is fine aslong as it's not being moved. That was something that we got experience of during a maintenance which wasnt fun tbh :P09:01
tobias_fiberdatabut that's a limitation in nova i think09:01
ccamachojaosorior, I think the introspection is failing also for me Ill confirm you in a few mins.09:02
*** fzdarsky has joined #tripleo09:04
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Repurpose KeystoneWorkers add keystone::wsgi::apache::threads
*** jubapa has quit IRC09:08
*** devvesa has quit IRC09:10
mcorneagfidente: fyi, I hit this while testing hyperconverged ceph nodes09:13
openstackLaunchpad bug 1604728 in tripleo "Ceph osds get created only for the first deployment" [Undecided,New]09:13
*** anshul has quit IRC09:15
*** anshul has joined #tripleo09:17
*** electrofelix has joined #tripleo09:20
gfidentemcornea I think that'09:23
gfidentesorry, that's the expected behaviour09:23
gfidentewhen you delete the deployment the disks are not erased09:24
gfidenteso the second deployment will fail when running disk activate09:24
gfidenteto me the good news is you managed to get osd running on the compute node without dedicated nodes :)09:25
*** akrivoka has joined #tripleo09:25
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates: Added a new composable service for enabling DPDK on compute nodes.
mcorneagfidente: yeah, it looks good. so I guess for my testing I can use a firstboot script that erases the disks, right?09:26
gfidentemcornea ack09:27
gfidentewe need to get this going in upstream CI09:27
gfidentehave a few submissions up for review doing it with ceph2.009:27
mcorneagfidente: but shouldn't the deployment fail in this case? it completed ok even though the osds were not created09:27
gfidentemcornea right you mean puppet should exit with error if disk activate fails?09:28
mcorneagfidente: right09:28
gfidentemaybe we can turn the bug into that09:28
gfidenteand work in puppet side?09:28
*** radek2 has quit IRC09:28
mcorneagfidente: ok, which project is it, puppet-ceph?09:29
*** ifarkas has quit IRC09:30
*** karthiks has quit IRC09:31
*** yolanda has quit IRC09:31
*** radek__ has joined #tripleo09:31
*** jerrygb has joined #tripleo09:32
*** yolanda has joined #tripleo09:32
gfidentemcornea yep09:34
*** hanchao has quit IRC09:34
*** jerrygb has quit IRC09:36
*** devvesa has joined #tripleo09:37
*** tobias_fiberdata has quit IRC09:37
*** athomas has quit IRC09:39
*** abehl has joined #tripleo09:39
dtantsurhi folks! how to debug errors like:09:39
dtantsurError: /Stage[main]/Ironic::Db::Sync/Exec[ironic-dbsync]: ironic-dbsync --config-file /etc/ironic/ironic.conf  returned 1 instead of one of [0]09:39
dtantsurif it does not give any details, e.g. stderr output...09:39
dtantsuraha, answering my own question: stdout and stderr are displayed separately09:40
dtantsurnow I have to figure out why I get Access denied for user 'ironic'@'' (using password: YES)09:41
*** anshul has quit IRC09:41
*** radek2 has joined #tripleo09:41
*** ifarkas has joined #tripleo09:42
*** karthiks has joined #tripleo09:43
dtantsurany ideas on ^^^? full logs are at
*** radek2 has quit IRC09:48
*** devvesa has quit IRC09:52
ccamachojaosorior, yeahp, introspection is timing out09:53
*** tosky has joined #tripleo10:01
*** tobias_fiberdata has joined #tripleo10:02
*** radek2 has joined #tripleo10:04
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Rename introspect parameter
openstackgerritDmitry Tantsur proposed openstack/puppet-tripleo: Add base nova-ironic profile
openstackgerritDmitry Tantsur proposed openstack/puppet-tripleo: Fix Ironic dbsync ordering
dtantsurfolks, does the 2nd patch ^^ make any sense?10:08
dtantsurI'm totally lost in this database bootstrap ordering :(10:09
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add pacemaker mistral profile
*** devvesa has joined #tripleo10:14
*** limao has quit IRC10:16
*** jaosorior_lunch is now known as jaosorior10:19
jaosoriorccamacho: Are you getting that issue without TLS?10:19
*** jcoufal has joined #tripleo10:20
*** anshul has joined #tripleo10:20
*** abehl has quit IRC10:22
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services
ccamachojaosorio, without TLS, I wanted to check first if I was hitting the same issue you described10:24
jaosoriorccamacho: So the timeout you saw without TLS?10:24
ccamachojaosorior, yeahp the introspection failed without the TLS submission10:25
ccamachothe timeout error10:25
jaosorioraaah out10:25
jaosoriorwell, crap10:25
jaosoriornow we gotta figure out how to make it work again10:25
jaosoriordtantsur you could probably see the logs from /var/log/ironic no?10:26
gfidentesshnaidm you looking into the .spec issue for tht?10:26
*** jcoufal has quit IRC10:26
sshnaidmgfidente, yes10:26
gfidentecan I help anyhow as it's blocking a few things10:26
sshnaidmgfidente, trying to convince apevec it's specs problem, not :)10:27
jaosoriordtantsur: Nevermind, I went ahead and tried to check the logs for you, and they seem empty10:27
gfidentesshnaidm where?10:27
sshnaidmgfidente, I'm preparing a mail now to openstack list, please comment there10:28
gfidenteit's going to take a while then10:28
gfidentemaybe apevac knows something we can change in tripleo meantime?10:28
openstackgerritCarlos Camacho proposed openstack/puppet-tripleo: Add pacemaker mistral profile
*** karthiks has quit IRC10:30
*** mbound has joined #tripleo10:30
*** d0ugal has quit IRC10:31
ccamachoI'm still in (Waiting for introspection to finish...) Anyone interested in getting logs from that server? Just let me know10:32
*** jerrygb has joined #tripleo10:33
dtantsurjaosorior, yeah.. I hope will fix it10:35
*** jerrygb has quit IRC10:39
*** dhill_ has quit IRC10:40
shardyccamacho: I have the same problem, the issue is the nodes can't boot via ipxe because the ironic::drivers::deploy::http_port hiera variable isn't getting resolved correctly10:41
*** karthiks has joined #tripleo10:41
ccamachoshardy ack10:42
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Use certmonger for automatic cert generation
coolsvapshardy: ping  can you help me getting started with the release patches for n2?10:43
jaosoriorshardy: what value is that hiera variable giving you?10:44
jaosoriorseems to be giving me 8088 which looks ok10:44
openstackLaunchpad bug 1604770 in tripleo "introspection hangs due to broken ipxe config" [Undecided,New]10:44
shardyI get an unresolved hiera reference in the ipxe config10:44
shardycoolsvap: Sure, but I think we're still waiting on a clean stable periodic promotion?10:44
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo: WIP Next generation HA architecture work
shardycoolsvap: after that it's basically just a question of proposing a patch to openstack/releases with the SHAs tested in the periodic job10:45
jaosoriorshardy: Where is the ipxe config?10:45
jaosoriorI can check it in my machine10:45
shardyjaosorior: /httpboot10:45
tobias_fiberdatashardy, when running this: openstack overcloud deploy --templates does it take all templates in the originalfolder then or what is executed?10:46
coolsvapshardy: yes I checked it last time i want to do it once to be confident10:46
jaosoriorshardy, this is what I see
shardytobias_fiberdata: It deploys a default configuration (two nodes, nonha) from /usr/share/openstack-tripleo-heat-templates10:46
shardyyou can pass a path to --templates if you want to use a local git repo instead of the packaged templates10:47
tobias_fiberdatashardy, yea i know, but i'm more interested in knowing what the default configuration is.10:47
shardynot all of them are deployed, additional features are enabled by passing -e /usr/share/openstack-tripleo-heat-templates/environments/foo.yaml10:47
jaosoriorccamacho: Do you have the same issue with the hiera variable?10:47
ccamacho[root@instack ~]# hiera 'ironic::drivers::deploy::http_port'10:47
jaosoriorI have a working reference of that variable and still get the timeout10:48
shardytobias_fiberdata: two nodes, one controller one compute, nonHA and no network isolation (all networking via the provisioning network)10:48
tobias_fiberdatashardy, ah okey10:48
shardybasically the defaults are for dev, and there are additional environments you'll need to enable for production deployments (particularly customizing your network setup I expect)10:49
*** pkovar has joined #tripleo10:49
tobias_fiberdatashardy, yea, well i need the pacemaker template and stuff like isolation and also some differences in storage. as we are not gonna use ceph at all like i talked about earlier.10:50
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove the ControllerCount contraint
*** pkovar has quit IRC10:59
*** jcoufal has joined #tripleo11:00
*** jcoufal has quit IRC11:01
*** liverpooler has quit IRC11:02
*** liverpooler has joined #tripleo11:02
*** ccamacho is now known as ccamacho|lunch11:04
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates: Fix inconsistencies in Swift proxy server pipeline after an upgrade
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates: Enable Swift account_quotas middleware by default
jaosoriorshardy: I'm thinking of adding a hieradata variable to all nodes (all overcloud nodes and even the undercloud itself). To specify the CA that certmonger will use. is there any guideline I should follow to name it? Right now for the undercloud implementation I'm only calling it certmonger_ca11:10
gfidentejistr marios the processorcount thing is actually going literal for me11:11
*** jcoufal has joined #tripleo11:11
jaosoriorgfidente ^^11:11
gfidenteand I have no clue why in CI it isn't given in the hieradata files I see workers: 1 not workers: $::processorcount which probably explains why it passed11:12
*** tzumainn has quit IRC11:12
*** tzumainn has joined #tripleo11:12
gfidentejaosorior I was attempting something like that for overcloud in
*** jcoufal has quit IRC11:12
jaosoriorgfidente: I was just going to use all-nodes-config.yaml11:13
shardyjaosorior: that sounds fine for a first pass, but we may have to add a prefix when we establish a common pattern for deploying common values to all nodes11:13
tobias_fiberdataif the deployment stalls, how am i able to see where it stalls and on what?11:13
gfidentejaosorior yeah purpose of the global config was to emit the value from a service template11:14
shardyjaosorior: the problem with that is ideally you'll want it located with a certmonger service definition?11:14
derekhgfidente: havn't been following along but just noticed you mentioned workers in CI,11:14
derekhgfidente: if its the overcloud, we're using this
shardyjaosorior: again, fine for a first pass, but we've got some work remaining on allowing composable cluster-wide and all-nodes data11:14
gfidentederekh yeah that explains it11:14
gfidentederekh think we have to revert11:14
jaosoriorshardy: I don't see any case where only some nodes would be using certmonger11:15
jaosoriorthey all need to at least have the CA there11:15
shardyjaosorior: So it will always be deployed no matter what?11:15
mariosgfidente: :/11:15
shardyeven for nonha setups, and there's no chance folks will desire an alternative thing in future?11:15
jaosoriorshardy: if no internal TLS is enabled then it won't run the daemon11:15
gfidentewe've been gamed !11:15
*** pkovar has joined #tripleo11:16
shardyjaosorior: ack11:16
shardyjaosorior: Ideally, it'd be nice if the data was only deployed when a thing is enabled11:16
shardythat's kinda the whole point of where we've been headed with composable services11:16
jaosoriorshardy: that is the case, if internal TLS is enabled in the overcloud, then certmonger is started11:16
shardybut we can work that out afterwards if all-nodes gets things working for you11:16
jaosoriorfor the overcloud it will be for either internal TLS or auto-generated public certs11:17
jaosoriorI mean11:17
jaosoriorfor the undercloud11:17
jaosoriorthat last statement11:17
shardyjaosorior: got a link to the patch that deploys the certmonger service?11:17
jaosoriorfor the undercloud
shardyactually I have to go catch a train so we may have to pick this up later/tomorrow11:17
shardywill check out the patch, thanks :)11:18
*** shardy has quit IRC11:18
jaosoriorshardy: for the overcloud here is the keystone part
jaosoriorI need to update it now though11:18
jaosoriorshardy, would it be better then if I prefix the hieradata variable with tripleo ?11:18
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Revert "Repurpose KeystoneWorkers add keystone::wsgi::apache::threads"
jaosoriorah, he left11:19
jaosoriorwhat do you think gfidente? ^^11:19
*** lucasagomes is now known as lucas-hungry11:20
gfidentejaosorior if you need puppet code maybe you can create a class in puppet-tripleo and pass the parameter to it?11:21
jaosoriorgfidente: I tried actually11:21
gfidentenot sure if it's overkilling11:21
jaosoriorand that didn't work11:21
gfidenteit should11:21
jaosoriorbecause of the way puppet handles specifying default parameters11:21
gfidentejaosorior ah don't need I understand the problem11:22
gfidente*don't think11:22
jaosoriorgfidente: so this was my first attempt
jaosoriorI was trying to pass the CA via that class11:22
jaosoriorand specify the default parameters for the Certmonger_certificate resource11:23
jaosoriorso every cert I would issue via certmonger would use the same CA11:23
jaosoriorbuuut that didn't work11:23
jaosoriorbecause of how scoping works in puppet. A declaration of default values for a resource will only work for the same class and it's children11:23
jaosoriorand that leaf would have no effect11:24
jaosoriorso the only way I got it to work was via a hieradata variable11:24
jaosoriorwhich is what I'm figuring out what to name11:24
gfidentejaosorior yeah I think you want a define instead of a class11:25
gfidenteso you can call it multiple times11:25
jaosoriorgfidente: There I don't need to call it multiple times, as that just installs certmonger and tries to set default values11:25
gfidentejaosorior ok don't hate me11:26
gfidentegoing for lunch11:26
gfidentebe back soon11:26
gfidentebut don't need to explain me, go ahead :)11:26
*** rodrigods has quit IRC11:26
jaosoriorgfidente hahaha dude, no problems11:26
jaosoriorit's better to explain11:26
jaosoriorthat way I'm not gonna be the only one that understands this stuff11:26
jaosoriorsorry if it's a bit confusing11:26
gfidenteback soon11:27
jaosoriorgfidente: have a good one11:27
gfidenteyou can explain after11:27
jaosoriorgfidente: But anyway, I did get it to work finally, with FreeIPA and the local CA too
gfidenteinstead of before :)11:27
jaosoriorand there I use a hieradata variable11:27
jaosorioranyway, enjoy the lunch dude11:27
*** gfidente is now known as gfidente|lunch11:27
*** stendulker has quit IRC11:29
jistrgfidente|lunch, marios: damn, sorry. I checked the config file before merging but didn't realize it was only correct due to us overriding it in CI, and the default not getting utilized.11:33
jaosoriorjistr: Hey dude, have you tried doing introspection lately?11:35
*** jerrygb has joined #tripleo11:35
jistrnope... haven't even deployed upstream lately11:35
mariosjistr: ack nm we can fix it once we work out the proper way, i'll try poke at it in a bit11:36
jaosoriormarios: You were working with the Manila patch, right?11:38
*** milan has joined #tripleo11:39
*** jerrygb has quit IRC11:39
mariosjaosorior: i'm looking after it yeah, i made it into composable11:40
jaosoriormarios: it only has one endpoint, right?11:41
jaosoriorfor the API11:41
jaosorioror does it have any other sub-service somewhere?11:41
mariosjaosorior: there is api share scheduler lemme check11:41
mariosjaosorior: but really i'm 0% familiar with manila itself, as i said, this review adopted me11:42
jaosoriorlol, so am I, I'm trying to figure it out cause it needs TLS support as well11:42
mariosjaosorior: yah afaics is just one endpoint/'standard' endpoint map11:45
*** weshay has joined #tripleo11:46
jaosoriormarios: Cool11:46
jistrmarios: fwiw i'll try to deploy with a fix according to the doc i dug up earlier, if that goes well i'll post the patch11:46
mariosjistr: ack thanks11:47
*** pkovar has quit IRC11:51
jaosoriormarios: Sorry dude, VERY dumb question I think11:51
jaosoriorin the manila patch11:51
jaosoriorI actually don't see where we tell manila with what IP and port to listen to11:51
jaosoriorI see where we set the keystone endpoints and such11:51
jaosoriorbut no place where we set the service parameters11:52
jaosorioris it just hardcoded and there's no parameter to set that up? (for instance, the port)11:53
jaosoriormarios: NEvermind, just found that it's actually hardcoded11:54
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Fix delorean build for stable branches
*** gchamoul is now known as gchamoul|afk11:55
*** gchamoul|afk is now known as gchamoul11:56
dtantsurfolks, did anyone plan to work or does anyone have any thoughts about supporting multiple overclouds?11:56
*** gchamoul is now known as gchamoul|afk11:56
dtantsurit seems to become our team's headache now :)11:56
*** gchamoul|afk is now known as gchamoul11:56
jaosoriordtantsur: I think someone had gotten it working already... there was a document somewhere, I can try to find it11:56
hewbroccadtantsur: I know dsneddon has put a fair amount of thought into it11:56
dtantsurjaosorior, I know, I wonder how to bring it upstream11:57
dtantsurhewbrocca, aha, cool. I'll wait for him then.11:57
jaosoriorno clue11:57
dtantsurI see. And while I'm here, has anybody put any substantial amount of thought into undercloud HA?11:57
slagledtantsur: it's not really well represented on the tripleo backlog for newton11:58
slagledtantsur: that being said, i think we can, and ought to, fix any assumptions in our tooling that assume just one overcloud11:58
dtantsurslagle, I guess so, however it somehow showed up on our team's backlog :)11:58
slagleif we did that, it would mostly just work11:58
*** pkovar has joined #tripleo11:58
hewbroccaMy very dim memory is that dsneddon thought the only real difficulty was with Ironic and Inspector being able to differentiate DHCP requests from more than one set of overcloud machines11:59
hewbroccaBut I could be wildly off base11:59
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: Rename introspect parameter
hewbroccadtantsur: I think that's why it's on your backlog11:59
dtantsurhewbrocca, Ironic is not a problem here, Inspector is not HA ready indeed, and Milan is working on it11:59
hewbroccadtantsur: I see, would HA-ready cover the multiple-baremetal-app problem?12:00
dtantsurhewbrocca, I'm not sure I recognize the problem you're referring to12:00
dtantsurcoffee kicks in too slowly today :(12:00
hewbroccaLike I said I could be wildly off base12:00
dtantsuralso, do we have multitenant undercloud on our agenda?12:01
dtantsure.g. different overclouds belonging to different tenants?12:01
hewbroccaThat has not been given as a requirement12:02
hewbroccathank god12:02
dtantsuryeah, I'm glad too12:02
dtantsurthen we don't need ctlplane separation probably12:02
*** athomas has joined #tripleo12:03
*** d0ugal has joined #tripleo12:04
mariosjaosorior: which bit are you referring to (hardcoded) do you mean the extraconfig data (that will go away/move into service, but as an iteration)12:05
*** jayg|g0n3 is now known as jayg12:05
*** dprince has joined #tripleo12:07
jaosoriorI meant the port itself12:07
*** zoli|wfh is now known as zoli|lunch12:07
*** pkovar has quit IRC12:08
jaosoriormarios: I just noticed there is no way to set the port in puppet-manila12:08
jaosoriorit's not a big deal, I just wanted to know how it was set and if it was missing from the CR12:08
mariosjaosorior: ack, once could use hiera on the service to override the module default (there is nothing puppet-tripleo side for it which landed already just checked)12:09
openstackgerritFlavio Percoco proposed openstack/tripleo-common: Remove script
openstackgerritFlavio Percoco proposed openstack/tripleo-common: Remove unnecessary packages
hewbroccaHey hey12:10
hewbroccaremoving code!12:10
marios'unnecessary' tsk tsk tsk12:10
jistrmarios: looks like i have an undeletable overcloud in my env, i won't be able to test the fix soonish :( do you want me to submit the patch anyway? I'd mark it WIP until someone is able to test it.12:10
jistr(maybe me but at least in a few hours)12:11
mariosjistr: sure man, so plan is to include the mitaka version of it on my uprade converge run, so i can submit later anyway12:11
mariosjistr: np i was planning to do same as you (what you suggested in earlier comment)12:11
*** rodrigods has joined #tripleo12:12
mariosjistr: i mena mitaka version of the fixup for those workers, to test the syntax for the 'processorcount' thing12:12
*** sshnaidm is now known as sshnaidm|lnch12:13
*** pkovar has joined #tripleo12:20
dtantsurwe have a simple bug in I wonder how we missed it12:20
* dtantsur fixes now12:21
hewbroccaIs this what is breaking introspection for everybody?12:21
dtantsurI think so12:21
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud: Fix wrong template in puppet-stack-config.yaml.template
dtantsurhewbrocca, ^^^ now I need to figure our testing for it...12:23
jaosoriorpulling that patch to see if it fixes my environment12:24
slagledtantsur: you know how we missed that :)12:24
slaglewe weren't running introspection in CI12:24
dtantsuryeah :) lemme put a tripleo-ci patch removing the override for this value12:24
hewbroccaslagle: I think dtantsur was being Ironic12:24
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Fix KeystoneWorkers default value
*** pradk has joined #tripleo12:25
openstackgerritDmitry Tantsur proposed openstack-infra/tripleo-ci: Do not explicitly set inspector iPXE port
dtantsurslagle, ^^12:25
jistrmarios: -- WIP'd it for now, reprovisioning my upstream env12:26
mariosjistr: gfidente|lunch we should decide, revert or fix ?
*** gfidente|lunch is now known as gfidente12:26
slagledtantsur: lgtm12:26
*** rbrady has joined #tripleo12:27
gfidentemarios hey I was thinking about a fix but couldn't come up with a substitute of processorcount12:27
jistrmarios: if we're able to verify the fix soon enough, i'd fix. If we can't, then revert.12:27
gfidenteyeah same from me, I'd prefer fixing, but I failed to get ::processorcount parsed without the $ as well12:27
mariosjistr: ack i will also try a run of it assuming i get to converge (looking good, +2d on the keystone migration race patch)12:27
jistrgfidente: i think processorcount should be fine, maybe we just need a different syntax, see
pradkcan we get is in please
mariosjistr: gfidente ack, lets try get it fixed today then and then i can update the mitaka and land it fixed12:28
gfidentejistr ok didn't try that12:28
dtantsurfolks, in exchange I need your help... shows timeout and I see that nova scheduler didn't bother to start12:28
*** lucas-hungry is now known as lucasagomes12:28
dtantsurwhat could go wrong?12:28
*** limao has joined #tripleo12:29
*** limao_ has joined #tripleo12:31
*** julim has joined #tripleo12:31
jaosoriordtantsur: well... that's strange, in another controller nova-conducter never started either12:32
* dtantsur bangs his head on the table12:32
jaosoriordtantsur: Which CR is it?12:32
dtantsurhello EmilienM12:32
jaosoriorEmilienM: Hey dude, what's up?12:33
dtantsurjaosorior, it's a chain of 4 patches ending in
EmilienMjaosorior: nothing much, yet12:33
jaosoriorEmilienM: Hey, so I finally go the patch for certmonger working. I use a hieradata variable for setting the CA12:33
jaosoriorEmilienM: Can you check the patch out? also, should I prefix the CA hieradata variable with tripleo:: ? something like tripleo::certmonger_ca ?12:34
jaosoriorI plan on using that variable for the overcloud nodes also.12:34
jaosorioror is just certmonger_ca fine?12:34
*** limao has quit IRC12:34
dtantsurfolks, is it possible we at least merge ? currently I have to depend on it for my ironic work, it's pretty inconvenient...12:35
dtantsur1x +2, gate passed12:35
hewbroccaDo I hear 2x +2??112:35
*** jerrygb has joined #tripleo12:36
hewbroccaAnd we have got to get the introspection fix in, it is blocking RDO promotion for the last 15 days12:36
hewbrocca(not just it, but it's the one that's left)12:36
dtantsuryeah, let the gate pass on it12:36
slagledtantsur: for the nova-compute timeout, you probably need this:
slaglecompute wont start if conductor isnt started12:37
dtantsurslagle, awesome, is it possible we merge it?12:37
EmilienMjaosorior: ok I'll check that12:37
slagledtantsur: i did12:37
jaosoriordtantsur: Just test the fix you did for the undercloud. It fixed the timeout12:38
EmilienMjaosorior: don't prefix your hieradata if it's not in the scope of a class12:38
dtantsurfantastic! wanna land it without waiting for the tripleo-ci patch?12:38
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: WIP: Test SoftwareConfigTransport: ZAQAR_MESSAGE
dtantsurjaosorior, ^^12:38
jaosoriorEmilienM: Alright. I haven't prefixed it yet. So now I won't12:38
*** anshul has quit IRC12:39
jaosoriordtantsur: well, I rather wait for CI, so people have a clear mind about it merging12:40
dtantsurgot it12:40
jaosoriorunless more people test it locally12:41
*** jerrygb has quit IRC12:41
dtantsurjaosorior, mind taking a look at as well please? (I understand it's quite long)12:41
openstackgerritFlavio Percoco proposed openstack/tripleo-common: Build heat-docker-agents for specific versions
*** ccamacho|lunch is now known as ccamacho12:42
jaosoriordtantsur: checking it out12:42
gfidentejistr though doesn't work for me12:43
gchamoulGuys, someone has time to look at that puppet-tripleo patch, please?
*** rlandy has joined #tripleo12:45
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services
openstackgerritMerged openstack/puppet-tripleo: make sure we start nova-compute after nova-conductor
jaosoriorccamacho: Hey dude, if you have time, can you test the undercloud deployment with this CR?
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Fix KeystoneWorkers default value
jaosoriorccamacho, I mean, the deployment and then the introspection12:48
ccamachosure thing man, give me a few mins to deploy it.12:48
ccamachojaosorior ^12:48
jistrgfidente: ^^ updated the patch with one more try, i wonder if we need the nested quotes too12:49
* jistr 's upstream undercloud will be ready in an hour or two12:50
gfidentejistr it's one of the options I tested manually12:51
jistrmarios, gfidente: hmm then i'm out of ideas :( we might have to merge the revert first before we figure out how to approach this12:51
jistrmarios, gfidente: maybe only reference the ::processorcount from puppet code and not from hiera, which would require a combination of puppet-tripleo + t-h-t patches to fix this12:52
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: TEST IRONIC SSL - DO NOT MERGE
gfidentewhy it doesn't work it's unclear though12:53
*** jcoufal has joined #tripleo12:55
*** rhallisey has joined #tripleo12:56
marios jistr yeah going the route of doing this puppet-tripleo side allows us to be a bit more explicit about the logic there though requires more of a change... for what is seemingly a quick fix i think that was the hesitation... lets give it till eod and then we can merge the revert12:59
mariosjistr: gfidente i mean see what we come up withhh by then13:00
jistrmarios: ack that sounds good, thanks13:00
EmilienMslagle: how is multi node job going now?13:02
*** trown|outtypewww is now known as trown13:03
slagleEmilienM: multinode job passed13:03
slaglethe regular CI job failed due to the gnocchi db sync bug.13:03
slaglethat fix merged, so i've rechecked it13:03
EmilienMslagle: excellent news13:04
gfidentebandini did we actually test if it passes without pacemaker?13:04
bandinigfidente: yes I did (see last commit message)13:05
bandinigfidente: both with 1 controller and 3 controllers13:06
bandinigfidente: I did it because the experimental job seemed non functional so far13:06
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Add base constraint so gnocchi metricd is tied to core-clone
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Add base constraint so gnocchi metricd is tied to core-clone
*** limao_ has quit IRC13:07
*** tobias_fiberdata has quit IRC13:07
gfidentebandini so let's merge it13:07
bandinigfidente: ack from my side13:07
bandiniNg: ^13:07
*** limao has joined #tripleo13:08
Ngthanks gfidente :)13:08
*** numans has quit IRC13:08
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Enable SSL for ironic inspector
jaosoriorbnemec: I rebased the ironic inspector SSL patch on top of dtantsur's fix for the undercloud. It worked using that patch.13:09
gfidenteNg oh you're cmsj!13:10
*** limao_ has joined #tripleo13:10
Nggfidente: correct, sorry it's confusing. I picked this nick when I was a teenager and it's too late to change now ;)13:10
bandinigfidente: he lies. he chose the nick to align with the Ng HA arch13:10
* bandini ducks13:10
*** hewbrocca is now known as hewbrocca-afk13:11
*** limao has quit IRC13:13
*** lblanchard has joined #tripleo13:13
openstackgerritMerged openstack/puppet-tripleo: Make ::tripleo::profile::base classes work with multiple nodes
*** pgadiya has quit IRC13:13
*** d0ugal has quit IRC13:14
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add hyperconverged-ceph environment to include CephOSD on computes
*** hewbrocca-afk is now known as hewbrocca13:15
EmilienMtrown: hey, do you have progress on testing again?13:15
*** tobias_fiberdata has joined #tripleo13:16
*** limao has joined #tripleo13:16
*** [1]cdearborn has joined #tripleo13:16
*** jdob has joined #tripleo13:16
openstackgerritKarthik S proposed openstack/tripleo-heat-templates: Allow scheduler_available_filter to be customizable
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add hyperconverged Ceph to the OVB/HA job
trownEmilienM: I havent had any progress figuring out why CI passes but everything I try with that patch fails13:18
*** limao_ has quit IRC13:18
trownEmilienM: but I have tried lots of stuff which all failed :)13:18
*** bfournie has joined #tripleo13:19
trownI am 100% confident that patch will break RDO though13:19
jaosoriortrown: Got logs?13:19
trownjaosorior: the important part is that hiera is missing on the controllers when using that patch13:20
trownjaosorior: if I take the patch out, hiera is there13:20
trownoriginally I was testing with the RDO image, which is not an exact match in terms of what all of the packages are, but there are no dependencies listed in that patch13:21
trownI have since used the images from tripleo-ci with the same result though13:21
gfidentetrown so any of the hieradata moved with that patch is missing?13:22
*** akshai has joined #tripleo13:22
*** limao_ has joined #tripleo13:22
trowngfidente: well I did not do an exhaustive check, but puppet complains about rabbitmq hiera gone, and when I check on the controllers it is indeed not there13:23
jaosoriorEmilienM: Wasn't there a t-h-t patch that this patch was on top of that was needed for it to work?13:23
trownwho has tried that patch on an HA setup outside of CI?13:23
gfidentetrown I am trying it now and I see the hiera13:24
gfidenteincluding rabbitmq_environment13:24
*** limao has quit IRC13:25
trownk, then there is something that quickstart does differently that does not work with that patch13:26
trownnetwork isolation?13:26
gfidenteI am doing isolation as well and ceph13:26
jpeelerflaper87: if the containers were able to extract the config without the RPM, these lines could go away
gfidentetrown overall the patch seems to be moving stuff around but not changing anything of the workflow13:27
gfidenteso I can't think of a reason why only the moved data should be missing13:27
trownhmm... I will try with puppet modules from source later today, that is the only thing I can think of13:27
EmilienMjaosorior: not afik13:28
trowngfidente: right, but it is pretty obvious when you do a deploy from templates from master and it is CREATE_COMPLETE, then cherry-pick that patch and redeploy and hit the issue13:28
trownwhich I can do 100% of the time13:28
EmilienMjaosorior: I remember the gnocchi issue but it's merged now13:29
*** athomas has quit IRC13:29
trownit is just a matter of figuring out what is different in quickstart, but I would still ask we not merge that until we at least know... we totally screw RDO otherwise13:29
EmilienMtrown: no worries, we can wait13:29
*** yamahata has joined #tripleo13:31
*** zoli|lunch is now known as zoli|wfh13:35
*** jerrygb has joined #tripleo13:37
EmilienMjaosorior: I'm reviewing
EmilienMthis is a lot of logic, can't it live in puppet-tripleo?13:39
EmilienMin a profile or something13:39
jaosoriorEmilienM: I already moved parts of it13:39
jaosoriorbut what do you mean, the local CA parts?13:39
EmilienMjaosorior: I'm afraid to add more and more code in elements/puppet-stack-config/puppet-stack-config.pp13:39
EmilienMit increases our technical dept of adding more and more code in there, while we could have puppet profiles13:39
EmilienMthe more code we add in this file, more difficult will be to split undercloud manifest13:40
*** masco has quit IRC13:40
jaosoriorEmilienM: I have to get the variables from somewhere, I did move a lot of it to the stuff related to haproxy13:40
jaosoriorI could move the local CA code13:40
jaosoriorand just include it if needed13:40
EmilienMin this file, we should just "include ::my_class" and use hieradata to feed parameters13:40
jaosoriorEmilienM: The big problem is that I'll need to do something very similar for the internal TLS for the undercloud13:41
EmilienMI mean, the code is good, I can +2 it on my side. It's just it increase our code for undercloud manifest13:41
jaosoriorso there will be a second certificate requested for haproxy13:41
jaosoriorso I can't hardcode hieradata there13:41
*** jerrygb has quit IRC13:41
EmilienMimho, all this logic should reside in puppet-tripleo13:41
*** egafford has joined #tripleo13:41
hewbroccadtantsur: yer introspection patch pass CI yet?13:42
dtantsurlemme check, I suspect it does not run even13:42
jaosoriorEmilienM: so you want me to do a undercloud_tls.pp file pretty much?13:42
dtantsurhewbrocca, still queued :(13:43
hewbroccablearrgh :-)13:43
hewbroccawell it'll get there when it gets there13:43
*** d0ugal has joined #tripleo13:43
jaosorioronly issue I see with that is that it obscures the implementation even more and it will be very undercloud-specific13:45
EmilienMjaosorior: for ::tripleo::certmonger::haproxy { 'undercloud-haproxy-public': ... } - you could use create_resources()13:45
EmilienMand create the resource with Hiera and an hash13:45
EmilienMyou see what I mean?13:45
openstackgerritMerged openstack/tripleo-puppet-elements: Modify the midonet repositories to Mitaka
EmilienMlike this:
dtantsurFailed to connect to 2607:1680:0:1::2: Network is unreachable \o/13:45
dtantsurhewbrocca, ^^13:46
jaosoriorEmilienM: Will that work with the postsave_cmd there?13:46
hewbroccadtantsur: arrgh13:46
jaosoriorwhich now I figured out has a problem.13:47
EmilienMjaosorior: well, you'll need to adjust hieradata13:47
*** akrivoka has quit IRC13:47
*** rdo has quit IRC13:48
EmilienMjaosorior: I'm afraid we have built and are building a monolithic manifest like we did with THT (overcloud_*.pp)13:48
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add hyperconverged Ceph to the OVB/HA job
flaper87jpeeler: what do you mean w/ extract the config? You mean the config file from the rpm ?13:50
*** rdo has joined #tripleo13:50
jaosoriorEmilienM: Alright, let me iterate a little bit on it, and see what you think13:50
jpeelerflaper87: yes, exactly13:50
flaper87jpeeler: cool, thanks13:51
*** akrivoka has joined #tripleo13:51
*** jerrygb has joined #tripleo13:53
EmilienMjaosorior: nevermind, we can iterate later13:55
EmilienMjaosorior: I'll prototype something for you13:55
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add class to use certmonger's local CA
*** mgarciam has joined #tripleo13:56
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart: WIP gate upgrade
flaper87dprince: I think we're good here:
*** limao_ has quit IRC13:57
*** _milan_ has joined #tripleo13:57
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Use certmonger for automatic cert generation
*** athomas has joined #tripleo13:58
*** limao has joined #tripleo13:58
jaosoriorEmilienM: So I took away the CA part, which makes sense. Should I then move all that chunk that's there to puppet-tripleo?13:58
weshaysshnaidm|lnch, when you are back.. help me drive the CI on
jaosoriorI mean, the local CA part13:59
hewbroccadtantsur: what does that "network unreachable" error mean -- IPv6 problem?14:00
dtantsurhewbrocca, just failed centos mirror14:00
dtantsurnothing smart14:00
*** dtantsur is now known as dtantsur|mtg14:00
*** sshnaidm|lnch is now known as sshnaidm14:00
* hewbrocca facepalm14:00
weshaysshnaidm, greetings sir14:01
sshnaidmweshay, hi, I'm here14:01
*** milan has quit IRC14:01
*** shardy has joined #tripleo14:03
sshnaidmweshay, I see network issues there, right?14:04
weshayya.. that's what I see14:04
weshayhewbrocca, can we waive upstream ci?14:04
*** numans has joined #tripleo14:05
EmilienMjaosorior: yeah, you can even create a class that takes parameters and inside, use create_resources($hash) from the parameters, so create the local ca14:06
hewbroccaweshay: well, not really my call14:06
*** fultonj has joined #tripleo14:07
*** rhallisey has quit IRC14:07
weshayseems silly to block this particular change on it14:07
*** trozet has quit IRC14:07
hewbroccatrown, dtantsur|mtg, slagle EmilienM thoughts on this?14:08
*** zoli|wfh is now known as zoli|brb14:08
*** rhallisey has joined #tripleo14:08
hewbroccaWe propose to waive CI on dtantsur|mtg fix-introspection patch14:08
hewbroccabecause it is blocking (among other things) RDO trunk promote14:08
*** trozet has joined #tripleo14:08
weshaycurrently failing w/14:09
weshay2016-07-20 13:41:36.375700 | 2016-07-20 13:41:33.000 | [Errno 14] curl#7 - "Failed to connect to 2607:1680:0:1::2: Network is unreachable"14:09
openstackgerritDan Prince proposed openstack/puppet-tripleo: Add firewall::service_rules define
bandinigfidente: ever seen this one ?14:10
openstackLaunchpad bug 1604835 in tripleo "[Newton] hostname instability after fresh deployment" [Undecided,Confirmed]14:10
trownhmm, it would be nice to at least see the patch get through an undercloud install14:10
gchamoulGuys, someone has time to look at that puppet-tripleo patch, please?
EmilienMdprince: your approach is really complex compared to mine14:11
gchamoulbnemec, EmilienM, slagle ^^ , maybe? :S14:11
weshaytrown, just did that locally actually14:11
EmilienMgchamoul: does it pass CI?14:12
trownweshay: ya, locally != CI though14:12
slaglehewbrocca: and what are we doing to fix the CI?14:12
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Fix ceilometer auth url for compute
dprinceEmilienM: my approach doesn't duplicate an entire hiera tree into a different format14:12
dprinceEmilienM: and it works...14:12
EmilienMdprince: ok, shardy's approach does diplicate the hiera tree, right?14:13
dprinceEmilienM: do you not like it. I thought we agreed this was acceptable given the architecture today... noting that we do want to write a proper hiera hook in the future14:13
ccamachojaosorior, man the introspection still waiting to finish14:13
dprinceEmilienM: my approach will be 3 patches14:14
weshaysshnaidm, is that network issue hitting other jobs?14:14
dprinceEmilienM: 1 of which (the service_names) we want to do anyway14:14
EmilienMdprince: I didn't say I don't like it :)14:14
jaosoriorccamacho: with dtantsur's patch?14:14
trownccamacho: you probably need
EmilienMdprince: you should patch THT with an example of rules, like I did for keystone14:15
dprinceEmilienM: the only "complexity" is this and that is arguably the same level of complexity of all the other firewall stuff14:15
dprinceEmilienM: I am posting it now14:15
EmilienMdprince: ok, fair enough. wfm.14:15
gfidentebandini I bet cloud-init is doing some command at first boot which it doesn't do on reboot14:15
dprinceEmilienM: I would probably be done already.... :)14:15
hewbroccaslagle: that is a fair question -- I have the idea that the failure is transient network issue that is unrelated to the patch14:15
dprinceEmilienM: don't sweat me man14:15
EmilienMdprince: can you remove whitespaces in ?14:15
hewbroccabut I don't know if that is actually true. dtantsur|mtg ? weshay ?14:16
EmilienMdprince: lol14:16
sshnaidmweshay, nope, I ran recheck, let's see how is it going..14:16
weshaysshnaidm, I don't see downloading errors in your status page14:16
ccamachoyeahp jaosorior, trown changes were made here but still there waiting.14:17
sshnaidmweshay, it detects failure on building image stage14:17
bandinigfidente: ah that is a good clue14:17
sshnaidmweshay, I'll add this pattern14:18
bandiniNg: ^14:18
gfidentebandini once I pushed some tests for cloud-init around the hostname handling but it didn't get merged14:18
*** dhill_ has joined #tripleo14:18
weshaysshnaidm, ah ya.. | Failed to build dep image "overcloud-full.qcow2". |14:18
*** dhill_ has quit IRC14:19
weshaysshnaidm, anything to spot infra errors more specifically would help14:19
jpeelerflaper87: if you wanted to push this through too - are you just doing general clean ups?14:19
*** tobias_fiberdata has quit IRC14:19
*** dhill_ has joined #tripleo14:19
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Fix ceilometer auth url for compute
*** saneax is now known as saneax_AFK14:22
flaper87jpeeler: mostly improving the image size for now while I go through the dependency list14:23
flaper87jpeeler: I'll push that one through too14:23
openstackgerritGael Chamoulaud proposed openstack/puppet-tripleo: Ensure the Heat Domain parameters before starting heat-engine pcmk service
*** tosky has quit IRC14:24
*** tobias_fiberdata has joined #tripleo14:25
jaosoriorccamacho: what's the output of hiera ironic::inspector::http_port14:25
bnemecbandini: I pushed a long time ago, but haven't gotten around to addressing the comments on it.14:29
bandinibnemec: I see seems related to
openstackLaunchpad bug 1581472 in tripleo "CloudDomain doesn't correctly set hostname" [High,Triaged] - Assigned to Giulio Fidente (gfidente)14:31
jmiujaosorior: how long will you be around until?14:31
*** dtrainor has joined #tripleo14:31
*** julim has quit IRC14:32
pradkHi bandini, when you get a chance can you look at backport is
pradki ran into this when i manually tried to restart cluster14:32
gfidentebnemec bandini so I think the root cause is that nova doesn't give domainpart to the hosts because dhcp_domain is empty14:32
gfidenteso while at first boot cloud-init only has shortname14:32
*** d0ugal has quit IRC14:33
gfidenteon reboot it gets the domainpart from hosts file14:34
gfidentewhich makes me think the only real solution is to have dhcp_domain set in undercloud nova.conf14:34
gfidentecause anything else will happen only after the hostname is set14:34
jaosoriorjmiu: Not that long14:35
jaosoriorjmiu: what's up?14:35
ccamachojaosorior, crap, getting nil14:35
gfidente(including my patch setting hostname from hosts because hosts still isn't populated when cloud-init starts)14:35
jaosoriorccamacho: Getting nil? Got a log?14:35
*** shardy is now known as shardy_mtg14:35
jmiujaosorior: running into difficulties configuring freeipa server because of ip address + networking issues14:36
jaosoriorjmiu: so how does your setup look like?14:36
jmiujaosorior: i'm wondering that given enough time i will read enough to find out how to do this, but i remember you mentioning that the networking stuff can be really tricky14:36
bandinigfidente: I tend to agree yest. We must fix this disparity fully-qualified vs non-fully qualified, otherwise stuff will break (neutron agent-list is the first we noticed, but there might be others)14:36
bandinibnemec: what do you think about what gfidente is saying ^ ?14:36
bandinipradk: looking14:37
jmiujaosorior: right now, i just have a vm attached to ctlplane. nothing else14:37
jaosoriorjmiu: Well, so if you don't have network isolation and you're using the defaults that come with t-h-t it shouldn't be thaaat bad14:37
ccamachojaosorior, not sure were to check14:37
jaosoriorbut if you want to test net-iso, there will be pain14:37
jmiujaosorior: i don't think i'm using the defaults that come with t-h-t. i came up with this template from scratch14:38
jaosoriorjmiu: alright so the you already have the freeipa server installed, right?14:38
jmiujaosorior: i've been looking at overcloud.yaml. any templates you'd recommend i would look at?14:38
jmiulet's not worry about network isolation yet haha14:38
jmiuyes i do14:38
jaosoriorjmiu: And the vm is receiving an ip address via dhcp?14:39
*** zoli|brb is now known as zoli|wfh14:39
jmiuconfiguration fails because invalid ip address14:39
jaosorioror is it a static address?14:39
*** jistr is now known as jistr|mtg14:39
jmiuit's not static...i think14:39
ccamachojaosorior, hey it is trying to get the hiera file from another location
jmiui get an error messsage like so: 2016-07-20T14:23:12Z WARNING Invalid IP address for cannot use IANA reserved IP address14:39
ccamachothats why is returning nil14:40
jaosoriorccamacho: Well, that's messed up14:40
jaosoriorjmiu:  Yeah... that's a problem with the default addresses for tripleo. And it's messed up for historical reasons14:40
*** dhill_ has quit IRC14:40
jaosorioryou need to change it from the undercloud14:40
*** d0ugal has joined #tripleo14:41
jaosoriorjmiu: I actually do have a fix that I tried pushing to tripleo-quickstart
jmiujaosorior: what do you mean by 'historical reasons'? is this something i can do via heat template?14:41
jaosoriortrown: By the way,d o you have time to check again?14:41
bnemecbandini: I don't particularly care how it gets fixed.  I believe my patch fixed it for me, but if there's another way I'm fine with that.14:41
jaosoriorjmiu: historical reasons means that a long time ago that ip range which is not usable in many cases was introduced. And it just stuck there14:42
jaosoriorso now changing it is problematic14:42
jaosoriorhowever, you can configure it with a different range14:42
bandinibnemec: let me take a close look at your patch too, I can quickly test it here, just to be 100% sure it fixes my issue14:42
ccamachojaosorior, missing sudo.. getting 808814:42
jmiujaosorior: ah, i imagined some crazy political reason long ago :p14:42
jaosoriorccamacho: That looks correct14:43
jaosoriorbnemec: Which parameters did jmiu need to change from the undercloud.conf to use an alternative network than the default one?14:44
jaosoriorwas network_cidr enough?14:44
bnemecjaosorior: No, all the network settings need to line up.  I suggest just using
pradkbandini, thx14:45
* bnemec needs to add a link to that in the docs14:45
jaosoriorjmiu ^^14:45
jaosoriorbnemec: DUDE, that IS useful14:45
*** dhill_ has joined #tripleo14:46
jaosoriorjmiu: Anyway, use that (and I guess you need to change the hostname to match if you had changed it before) and then run the undercloud installation again14:46
jaosoriorjmiu: It's pretty much: openstack undercloud install14:46
trownjaosorior: I kicked the full deploy CI on it and will merge if it passes14:46
jaosoriortrown: excellent, thanks for taking a look14:47
*** jcoufal has quit IRC14:47
jaosoriorjmiu: Makes sense?14:47
*** jcoufal has joined #tripleo14:47
jmiujaosorior: whoa14:47
jmiuso it's an undercloud problem and not with the template i'm writing?14:48
jmiujaosorior: it feels super dubious for me to write the networking section in the template, because i know nothing of networking :|14:48
* bnemec has a tool for that too14:49
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: WIP: composable firewall rules
jaosoriorjmiu: pretty much. The undercloud uses an ip range that's only meant for documentation. freeipa (rightfully) doesn't accept ip's in that range. so you need to use a valid one14:49
jaosoriorthat's why you get those ip address errors14:50
*** tosky has joined #tripleo14:50
*** egafford has quit IRC14:50
tobias_fiberdataWhy isnt this on the documentation? :|14:50
jaosoriorbnemec: really?14:50
jmiujaosorior: i thought it was because i was neglecting to give the particular node an ip address in a different range14:50
jmiubnemec apparently has tools for everything14:50
*** rook has joined #tripleo14:50
tobias_fiberdataah oh14:51
tobias_fiberdatait's the undercloud configuration14:51
tobias_fiberdataofcourse :p14:51
jaosoriorbnemec: duuuude, that's awesome14:51
bnemectobias_fiberdata: I had been kind of holding off until I could find a more permanent location.  At some point I want it in the docs though.14:51
jaosoriorjmiu: nah, it's tripleo's fault for using an ip range we shouldn't be using14:51
bandinipradk: I commented on both. master one is problematic I think. Happy to chat with you more in detail about it14:52
tobias_fiberdatabnemec, aaah! I understand :P it isnt in the RH documentation either. that could be a great place to have it. but i guess you have regulations aswell14:52
jaosoriorbnemec: By the way, introspection was broken (with or without TLS). This attempts to fix it
bnemecjaosorior: I've been meaning to do another demo video of it (there's an ancient one that barely resembles the current tool), but...time.14:53
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: [WIP] Add 'openstack overcloud node import' command
openstackgerritJulie Pichon proposed openstack/python-tripleoclient: [WIP] Add 'openstack overcloud node configure' command
jmiujaosorior: i'm glad i asked you, then. otherwise i would have spent a lot of time researching this14:53
jaosoriorbnemec: So I put the inspector TLS patch on top and it seems to work for me. Just if you have time to check it out at some point.14:53
EmilienMdprince: ok, I like it :)14:53
jaosoriorjmiu: I stumbled upon the same thing14:53
dprinceEmilienM: testing that one locally. If it all seems great I will do the rest14:54
EmilienMdprince: but it won't work14:54
bnemecjaosorior: Weird, I thought it worked for me without the patch.14:54
EmilienMdprince: you needs to Depends-On the puppet-tripleo patch14:54
EmilienMdprince: not the THT14:54
pradkbandini, i see14:54
dprinceEmilienM: next up is I'm pushing the patches to get hiera out of compute.yaml and controller.yaml14:54
EmilienMdprince: ok14:55
pradkbandini, if we can get this landed for mitaka for now i think that would be helpful.. if we're anyway removing all these for newton we can hold off on master patch14:55
jaosoriorbnemec: maybe you had a version before it was broken?14:55
bnemectobias_fiberdata: Yeah, there's a link to it in an internal doc somewhere, but nothing official.14:55
pradkjistr|mtg, ^^ see bandini's comments14:55
bandinipradk: yeah mitaka is totally good14:55
bnemecjaosorior: Could be.  I'll try it again today.14:55
dprinceEmilienM: the static hiera files are of less importance to me I think because they don't block us from auto-generating (i.e. making jinja templates of) templates for composable roles14:55
ccamachojaosorior, removed and relaunched timeout went away14:55
jmiujaosorior: thank you so much :)14:55
pradkjistr|mtg, if we can land mitaka it would be very helpful14:55
bandinipradk: can we file an LP bug against gnocchi? so at least we have a reference?14:55
jaosoriorjmiu: no worries. You can always send me a mail with questions if I'm not around14:56
pradkbandini, yea sure i can open one now14:56
bandinipradk: thanks a bunch, much appreciated14:56
EmilienMdprince: ok14:56
dprinceEmilienM: when I test locally I don't use Depends-On so it is fine14:56
jaosoriorwell, now I gotta go14:56
dprinceEmilienM: I use upload-puppet-modules14:56
EmilienMdprince: please update it anyway14:57
jaosoriorhave a good day people!14:57
dprinceEmilienM: ack14:57
EmilienMdprince: to make it work in CI14:57
sshnaidmbnemec, hi, do you know where in our CI code the patch is applied for stable branch?14:57
EmilienMjaosorior: o/14:57
dprinceEmilienM: do we enable firewall rules in CI?14:57
EmilienMdprince: yes14:57
*** jaosorior has quit IRC14:57
tobias_fiberdatabnemec, ah, ah well! these kind of configurationsites are great. If there was one for all the templates for deploying overcloud i would be thrilled :P But i guess all those configurations would take quite some time to code aswell.14:57
dprinceEmilienM: okay, I thought we disabled them for the OVB stuff14:57
dprinceEmilienM: cool then14:57
tobias_fiberdatanow i'm going to back to updating backupclients on my server (fun job....not)14:57
tobias_fiberdatanow i'm going to back to updating backupclients on my servers (fun job....not)14:58
bnemecsshnaidm: You mean where we configure things to use stable branches?  There aren't any patches applied specifically for the stable branch.14:58
bnemectobias_fiberdata: The UI is coming.  This is standalone because it has to exist before any of the TripleO bits are installed.14:59
sshnaidmbnemec, I'm a bit confused, when we check patch for stable/liberty - where do we apply this patch actually? Or I don't understand something?14:59
pradkcan we please land this
sshnaidmbnemec, when we test patch in CI I mean15:00
*** kairat has left #tripleo15:00
*** sshnaidm is now known as sshnaidm|mtg15:00
bnemecsshnaidm: Yeah, that behavior isn't specific to stable branches.  I believe zuul just gives us the patched code in /opt/stack/new/$PROJECT and we do a delorean build of that.15:00
bnemecsshnaidm|mtg: derekh understands that better than I do though.15:00
sshnaidm|mtgderekh, ^^15:01
sshnaidm|mtgbnemec, yeah, I think the same, but can't find it in code..15:01
*** bootsha has quit IRC15:02
*** oneswig has quit IRC15:03
*** zeroshft has joined #tripleo15:03
derekhsshnaidm|mtg: ZUUL puts the relevant code (with patch applied) in /opt/stack/new/project15:05
derekhsshnaidm|mtg: we then use delorean to build a package with that source code (based on the projects listed in ZUUL_CHANGES)15:05
*** [1]cdearborn has quit IRC15:06
sshnaidm|mtgderekh, ok, so if I need to test the patch on devenv, I need to do it by myself, right?15:06
derekhsshnaidm|mtg: that package is then put into a ci yum repository (it usually only contains one package), and this repository is layered ontop of the others with a higher priority15:06
dtantsur|mtghewbrocca, sorry, what was the question?15:07
sshnaidm|mtgderekh, yeah, right, I do it so15:07
*** ebarrera has quit IRC15:07
derekhsshnaidm|mtg: yes, and set zuul_changes15:07
sshnaidm|mtgderekh, great, thanks15:07
derekhsshnaidm|mtg: and you may also need to set STABLE_RELEASE15:08
*** jprovazn has quit IRC15:08
derekh*will also15:08
sshnaidm|mtgderekh, yeah, sure15:08
*** rcernin has quit IRC15:09
sshnaidm|mtgderekh, I'll add this to etherpad15:09
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Add base constraint so gnocchi metricd is tied to core-clone
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Add base constraint so gnocchi metricd is tied to core-clone
derekhsshnaidm|mtg: ack, thanks15:11
*** oshvartz has quit IRC15:12
*** rcernin has joined #tripleo15:12
*** radek2 has quit IRC15:13
openstackgerritDan Prince proposed openstack/puppet-tripleo: Add firewall::service_rules define
*** [1]cdearborn has joined #tripleo15:17
hewbroccadtantsur|mtg: no worries15:18
hewbroccawe were debating waiving CI on your patch15:18
*** mgarciam has quit IRC15:18
hewbroccabut it looks like it's gonna pass this time anyway from what weshay said15:18
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart: WIP gate upgrade
*** trown is now known as trown|dr15:19
*** rcernin has quit IRC15:22
*** tesseract- has quit IRC15:25
*** aufi has quit IRC15:25
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart: WIP gate upgrade
*** dtantsur|mtg is now known as dtantsur15:27
jpichjtomasek: These are the 2 patches for which I'm wondering if additional Mistral workflows/work is needed, I added comments inline with links to where the functionality is currently implemented in Python - ,
*** krotscheck is now known as krotscheck_dcm15:30
jpichjtomasek, akrivoka: Is the goal to have the client do as little as possible in general, and everything somehow ends up in common? Maybe there're things that won't make sense/be needed for the UI so they might as well stay as they are in the client? Or better to migrate them as well just in case they do end up being needed?15:33
jtomasekjpich: thanks, so for the node import, there is a mistral workflow15:33
jtomasekjpich: yeah, keep the client to minimum - keep there just the client specific things15:34
jpichjtomasek: I did notice, though there are fewer options I think -- 2 switches are now "missing" as far as I can tell, related to instance boot options (local/netboot)15:34
akrivokajpich: I tend to say, as little actual business logic in the client as possible, we've been bitten by that in the past. but employ common sense, of course, if it's a one line api call or something along those lines, then I guess it's fine to keep it in the client.15:35
*** dsariel has quit IRC15:35
jtomasekjpich: can the boot option be specified in instackenv.json?15:35
jpichakrivoka: Right, that makes sense. I guess I'm asking for the opposite - the large functions that do something very specific for folks with a fair amount of domain knowledge (I assume)15:36
jtomasekjpich: it is deffinitely something that needs to be available for all clients, so it should be included in the workflow (tripleo-common) if it is not there15:36
bnemecjtomasek: You can set boot option in the json file:
jtomasekjpich: this has to do with the instackenv.json limitation imho, once we transition to using ironic nodes format in instackenv, we'll be able to just pass that through to ironic15:37
jpichjtomasek: The instance boot option logic wasn't migrated to use the workflow so I assume the workflow only handles the basic options (deploy kernel/ramdisk) for now15:37
jtomasekbnemec: thanks, ok, so if it is something that tripleo-common will catch and send to ironic, then  it is fine. if that part is not there, then we need to add it15:38
jpichjtomasek, bnemec: thank you for the information re:instackenv.json15:38
*** leanderthal is now known as leanderthal|afk15:38
jpichjtomasek: I guess I'll go and figure out how to check if it does then :)15:38
jpichGreat, that's a next step for one of the command. Only leaves all the stuff going on in ConfigureBaremetalBoot which as far as I can tell doesn't have on-going Mistral work, though I may have missed it15:39
*** dmsimard is now known as dmsimard|afk15:39
jtomasekjpich: here is the discussion to the baremetal workflows
jtomasekjpich: also here
sshnaidm|mtgwhat a nice message ERROR:dlrn:Known error building packages for openstack-tripleo-heat-templates, will retry later15:44
jtomasekjpich: I believe that nodes registered with the new workflow have no boot option set and it should default to 'local'15:44
*** mbound has quit IRC15:46
EmilienMsshnaidm|mtg: file a bug15:46
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: WIP: composable firewall rules
*** dprince has quit IRC15:47
jpichjtomasek: Thank you for all the links. I've been basing this work on the email by dtantsur linked in the commit message of the first URL - as far as I can tell, there are workflows for register/introspect/provide/set_node_state but nothing for configure so far - maybe rbrady can clarify if I missed it??15:47
jpichjtomasek: I'll dig down further about the boot option as well15:47
sshnaidm|mtgEmilienM, but it really retries! and fails after15:47
*** sshnaidm|mtg is now known as sshnaidm15:47
jtomasekjpich: yeah, I'd like to see what we need to update in GUI too in this regards15:47
EmilienMsshnaidm|mtg: mhh, not sure it's really the best way15:47
jtomasekjpich: in general, what stage is this happening at? after nodes are registered?15:48
*** jistr|mtg is now known as jistr15:48
jpichjtomasek: Yes, the idea seems to be "register the nodes" - "configure them some more if you don't like the default" then the usual introspect (or not)/make the nodes available15:49
dtantsurthis is correct ^^^15:49
bandinipradk: can you forward me the LP gnocchi bug whenever you get to it? (am trying to track all these issues that might affect us in the future)15:51
*** dprince has joined #tripleo15:51
jtomasekdtantsur, jpich: I wonder if simple update node call would be sufficient15:52
jtomasekif it is just a node update, nothing else, then I'd incline to not needing a workflow for that15:52
dtantsurjtomasek, jpich, the current configure command has some fancy logic to optionally detect the root device. I wonder if it should be several workflows15:52
dtantsurand yeah, maybe some bits do not need a workflow, dunno15:53
*** tremble has quit IRC15:54
*** mbound has joined #tripleo15:54
*** fzdarsky is now known as fzdarsky|afk15:54
jpichdtantsur, jtomasek: Yep - I think it looks complex enough to justify moving to common, based on what was said earlier15:55
pradkbandini, i liked it to the review15:56
openstackLaunchpad bug 1604860 in tripleo "Add a base constraint so gnocchi services should start after galera" [Undecided,New] - Assigned to Pradeep Kilambi (pkilambi)15:56
dtantsurjpich, jtomasek, right. everything else is just updating capabilities. it's still a bit trickier than just a node_update, but close enough. I suspect we might already have a workflow for updating capabilities15:56
bandinipradk: oh I was more thinking about a gnocchi bug, in order to make it resilient when the control plane is gone (argh sorry about the confusion there). I can file that one if you're too busy?15:58
pradkbandini, ?15:59
openstackLaunchpad bug 1604860 in tripleo "Add a base constraint so gnocchi services should start after galera" [Undecided,In progress] - Assigned to Pradeep Kilambi (pkilambi)15:59
pradkbandini, not that?15:59
*** egafford has joined #tripleo15:59
pradkbandini, thats the one you reviewed16:00
mariosjistr: gfidente am trying a run of  default: '\"%{::processorcount}\"' on a converge now... will take half hour ish16:00
mariosEmilienM: pehaps you know the answer... how can we make work16:00
bandinipradk: no, that's a tripleo bug. I was thinking more a pure gnocchi bug16:00
jtomasekdtantsur: that capabilities workflow is different stuff afaik, it is unfortunate naming16:01
EmilienMmarios: looking16:01
*** zoli|wfh is now known as zoli|brb16:01
dtantsurjtomasek, ah, I see. I was asking to make it generic, maybe it didn't work16:01
pradkbandini, you just want gnocchi statsd to retry untill success?16:01
pradkbandini, we already do that.. this tripleo bug is so we dont end up with traceback in the logs16:02
bandinipradk: I would like gnocchi statsd to have the same configuration knobs that the other services have to retry a certain amount of time to reconnect go galera/rabbit16:02
EmilienMmarios: I'm not sure we need %, a sec16:02
mariosEmilienM: appreciate16:02
bandinipradk: ah okay, this makes the issue somewhat minor. I thought the startup would fail16:02
EmilienMmarios: err, yes % is needed16:02
EmilienMmarios: %{::fact} should work16:03
EmilienMmarios: "%{::fact}" should also work16:03
EmilienMif it doesn't, I have no idea why16:03
mariosEmilienM: ok thanks for ansity check. i guess is because we are passing it as heat parameter default?16:03
EmilienMmarios: yes16:03
bandinipradk: let's say then that "it would be nice if gnocchi statsd had the max_retries tweak that other openstack services have, and not spew any traceback when backends are not there"16:04
jpichdtantsur: Do you mean ironic-specific mistral actions/workflows, for node capabilities update?16:04
EmilienMmarios: why not relying on the default upstream value?16:04
EmilienMmarios: in puppet-keyston16:04
bandinipradk: but yeah, if the service will do its job as soon as galera is back on track, then it is a very minor issue16:04
jpichdtantsur: The only workflow I could find related to update is "register_or_update" which takes in node_json rather than node_ids and forces a state change that isn't actually needed for configure (set to manageable), so it's not ideal16:04
pradkbandini, yea16:04
*** shivrao has joined #tripleo16:04
mariosEmilienM: upstream is 1 worker and processor threads i think so we override more at
openstackLaunchpad bug 1598092 in tripleo "Performance issues with keystone under wsgi apache on a multinode machine" [Medium,Fix released] - Assigned to Marios Andreou (marios-b)16:05
dtantsurjpich, hmm, maybe I'm wrong them16:05
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Don't fail if no zuul telnet is opened
*** d0ugal has quit IRC16:07
*** jubapa_ has quit IRC16:09
*** zoli|brb is now known as zoli|wfh16:10
*** beagles is now known as beagles_brb16:12
mariosjistr: gfidente EmilienM with default: '\"%{::processorcount}\"'  i have in /etc/httpd/conf.d/10-keystone_wsgi_admin.conf   WSGIDaemonProcess keystone_admin display-name=keystone-admin group=keystone processes=\"1\" threads=1 user=keystone16:12
marioslooks right no?^16:12
mariosi mean ignoring the \"1\" at least it is being set?16:12
EmilienMmarios: not sure apache will like it16:12
mariosEmilienM: right, but i mean it seems to be setting the conf as i passed it, that is a start16:13
*** rwsu has joined #tripleo16:13
mariosi did pass \" \"16:13
*** shivrao has quit IRC16:14
gfidentemarios so if you put it in single quotes it might wrk16:14
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Composable Mistral services
*** jprovazn has joined #tripleo16:15
gfidenteI suspect I get empty value in that case from cmdline because I don't load facts16:15
gfidentebut when doing apply it will load facts16:15
mariosgfidente: gimme a string and i will test it - one more run before hometime16:16
gfidente/etc/puppet/hieradata/service_configs.yaml:keystone::wsgi::apache::workers: '%{::processorcount}'16:16
gfidente# hiera keystone::wsgi::apache::workers16:16
mariosgfidente: yeah ok but i will use '%{::processorcount}' as the deafult for the KeystoneWorkers param16:17
mariosthat what you mean?16:17
marios v1 gfidente16:17
mariosi think jistr tried it16:17
gfidenteso the quotes have to get into the hieradata yaml file16:19
gfidentewhich is what jistr tried in patchset #216:19
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient: Update overcloud passwords on update command
*** electrofelix has quit IRC16:25
*** dprince has quit IRC16:27
*** shardy_mtg has quit IRC16:27
jistrmarios, gfidente: ack so i'm reverting it back to patch set 1, which should work, if i understand correctly what you said above16:29
gfidenteI think we need the quotes in the hiera yaml16:29
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Fix KeystoneWorkers default value
*** tobias_fiberdata has quit IRC16:30
*** _milan_ has quit IRC16:31
*** spredzy has quit IRC16:31
EmilienMit's green - does it mean we deploy jewel and it works?16:31
*** spredzy has joined #tripleo16:31
EmilienMgfidente: ah no, we need hypercongerged right?16:32
EmilienMgfidente: see my comment on
gfidenteyeah it's a few different submissions which we need to chain together to make this happen16:33
*** athomas has quit IRC16:33
jistrgfidente, marios: ah i see, so this will not work regardless i'm afraid. From hiera docs: "Hiera can interpolate values of any of Puppet’s data types, but the value will be converted to a string."16:33
jistri think what marios saw is that the value got passed in as a string, but puppet-keystone can work with only numbers for the workers param16:34
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add hyperconverged Ceph to the OVB/HA job
openstackgerritGiulio Fidente proposed openstack/tripleo-puppet-elements: Revert "bump puppet-ceph to stable/hammer"
*** jpich has quit IRC16:35
*** dtantsur is now known as dtantsur|afk16:35
gfidenteEmilienM let's see in this combination if we got the depends right16:35
hewbroccaweshay: we get dtantsur|afk patch yet?16:35
*** mcornea has quit IRC16:35
jistrgfidente, marios: so what we have in the nested quotes would only work if we had a fix in puppet-keystone that would handle string fine, but if we remove the nested quotes, it might not get inerpolated at all...16:35
jistror it would get interpolated but the result might be the same...16:36
gfidenteEmilienM though in this order we know will fail and only can pass16:36
weshaynot yet16:37
* weshay looking at 2016-07-20 16:14:45.749891 | ++ /opt/stack/new/tripleo-ci/scripts/ --overcloud-images16:37
*** jtomasek has quit IRC16:38
*** ayoung has joined #tripleo16:38
EmilienMgfidente: ok, will look after lunch16:39
openstackgerritMerged openstack/tripleo-heat-templates: Fix gnocchi::storage::ceph::ceph_keyring
EmilienMgfidente: you can't add 2 dependson on the same project16:40
gfidenteare there?16:40
EmilienMyou need to rebase the tripleo-ci patches each other and add one dependson16:40
EmilienMok I'm afk now16:40
gfidenteoh man16:40
gfidentefirstly you say you want to try the depends in the other order16:40
gfidentebut it was a trap16:40
gfidenteneed to revert as it was before16:41
*** zoli|wfh is now known as zoli|gone16:42
gfidentemerge things instead
derekhwherefore art thou dprince16:42
EmilienMgfidente: I just said your current patch won't be tested :)16:43
weshayhewbrocca, I apparently was looking at the wrong run.. afaict now the deployment is running.. so we're good.. unless I'm fos again of course16:43
EmilienMgfidente: I'll look after lunch16:43
gfidentethe revert was actually tested
*** zoli|gone is now known as zoli_gone-proxy16:45
*** trown|dr is now known as trown16:47
*** d0ugal has joined #tripleo16:48
*** lucasagomes is now known as lucas-afk16:51
flaper87jpeeler: do you remember what docker-compose is used for in that container ?16:51
hewbroccaweshay: OK -- let's get the damn promote job running if at all possible :)16:52
hewbroccaI'm heading out folks, see you all tomorrow16:52
weshayhewbrocca, aye.. have a good night16:52
trownto be fair it is running fine :P16:52
mariosjistr: gfidente guys i will have to revisit tomorrow morning on that keystone wsgi thing16:53
trownit is just not passing16:53
flaper87holy cow, I just saved us 500 MB on the heat-docker-agents container16:53
* flaper87 needs to test this16:53
flaper87this is too good to be true16:53
trowntripleo-ci would not be passing either if it tested intropsection16:53
*** derekh has quit IRC16:54
openstackgerritGiulio Fidente proposed openstack/tripleo-puppet-elements: Revert "bump puppet-ceph to stable/hammer"
*** radek2 has joined #tripleo16:54
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add hyperconverged Ceph to the OVB/HA job
*** limao has quit IRC16:55
openstackgerritBen Nemec proposed openstack/tripleo-docs: Update overcloud SSL docs
openstackgerritMartin Mágr proposed openstack/puppet-tripleo: New composable service: Sensu
*** limao has joined #tripleo16:56
openstackgerritHonza Pokorny proposed openstack/tripleo-ui: Add packaging spec and config files
*** shivrao has joined #tripleo16:58
jistrmarios, gfidente: hey patch set 2 worked for me as-is16:58
openstackgerritMartin Mágr proposed openstack/puppet-tripleo: New composable service: Sensu
weshaytrown, :)16:59
jistr[root@overcloud-controller-0 ~]# grep processes /etc/httpd/conf.d/10-keystone_wsgi_main.conf16:59
jistr  WSGIDaemonProcess keystone_main display-name=keystone-main group=keystone processes=2 threads=1 user=keystone16:59
mariosjistr: ok great then lets do that16:59
* jistr reverts the revert to patch set 1 so that we get patch set 2 back16:59
*** limao has quit IRC17:00
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Fix KeystoneWorkers default value
*** rajinir has joined #tripleo17:01
gfidentejistr ack17:03
*** jeckersb_gone is now known as jeckersb17:03
gfidentetty tomorrow17:03
*** pkovar has quit IRC17:04
gfidenteleft +217:04
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates: Availability monitoring agents support
*** gfidente has quit IRC17:04
*** paramite is now known as paramite|afk17:04
*** yamahata has quit IRC17:06
*** florianf has quit IRC17:12
*** mgould is now known as mgould|afk17:13
jpeelerflaper87: docker-compose needs to be switched out of the agent container for the new "hook-docker". Slower_ should be able to point you to where that lives if need be17:15
jpeelerwow it never made it in -
*** penick has joined #tripleo17:20
*** beagles_brb is now known as beagles17:21
pradkhas anyone seen this before locally?  Pacemaker::resource::Systemd is not a valid resource reference17:23
pradkError: No title provided and \" Pacemaker::resource::Systemd[]\" is not a valid resource reference17:24
*** jeckersb is now known as jeckersb_gone17:25
*** florianf has joined #tripleo17:26
*** pcaruana has quit IRC17:27
*** jeckersb_gone is now known as jeckersb17:28
*** jtomasek has joined #tripleo17:31
*** akrivoka has quit IRC17:34
slaglepradk: i'm hitting that right now :)17:38
slaglegoing to dig into in a bit17:38
trownEmilienM: what firefox extention do you use to browse telnet links?17:40
EmilienMcredits to pabelanger17:41
trownEmilienM: ah thanks17:41
openstackgerritJames Slagle proposed openstack/instack-undercloud: Remove ipxe selinux policy
*** ohamada has quit IRC17:45
*** ohamada has joined #tripleo17:47
*** ifarkas is now known as ifarkas_adk17:48
*** ifarkas_adk is now known as ifarkas_afk17:48
*** yamahata has joined #tripleo17:49
*** tosky has quit IRC17:51
*** devvesa has quit IRC17:51
egaffordHi all! Who should I ask about policy/process for installing Horizon plugins in TripleO?17:53
*** bootsha has joined #tripleo17:53
* egafford wants to install openstack-sahara-ui on Horizon nodes. Happily the resultant UI only appears if Sahara is actually running, so it *may* be okay to install in all cases, but naturally I want to follow whatever Horizon plugin guidance y'all have.17:54
weshayEmilienM, nice.. sshnaidm
EmilienMyes, it also works for chrome17:55
EmilienMbut soon, zuul will have an UI to see the terminal in live17:55
EmilienMit's WIP afik17:55
*** ohamada has quit IRC17:58
openstackgerritMerged openstack/instack-undercloud: Fix wrong template in puppet-stack-config.yaml.template
*** dprince has joined #tripleo18:02
*** akshai has quit IRC18:03
*** mbound has quit IRC18:04
*** numans has quit IRC18:07
*** akshai has joined #tripleo18:08
slaglepradk: do you happen to be using when getting that puppet error?18:10
slaglepradk: b/c that is what i'm testing atm, and hitting that18:10
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates: Fixes nova host configuration on control nodes
openstackgerritMerged openstack-infra/tripleo-ci: Add mulitnode CI job support to tripleo-ci
EmilienMslagle: BOOM ! ^18:10
EmilienMyou made it dude18:10
EmilienMpeople please note July 20th18:11
trownnot even 100 PS...18:11
EmilienMslagle: what are next steps? moving it to gate?18:12
slagleEmilienM: now i'd like to propose in project-config to move the job to the main check queue for tripleo projects, but be non-voting18:12
slaglethat way it's running on all our patches18:12
EmilienMslagle: ok, so you need to add -nv thing, you familiar with it?18:13
EmilienMI can do it if you want18:13
slagleEmilienM: sure if you already know how, go4it18:13
slagleand point me at it and i'll review18:13
slaglecool, thx18:22
openstackgerritElise Gafford proposed openstack/tripleo-puppet-elements: Installing openstack-sahara-ui
*** paramite|afk is now known as paramite18:23
*** bootsha has quit IRC18:23
*** bootsha has joined #tripleo18:25
*** dmsimard|afk is now known as dmsimard18:26
*** bootsha has quit IRC18:30
*** bootsha has joined #tripleo18:30
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Composable firewall rules
dprinceEmilienM: hey man, got em all except for aodh ^18:42
*** akshai has quit IRC18:42
EmilienMdprince: you're a machine18:43
EmilienMdprince: dude, don't pass CI18:43
EmilienMI'm reviewing the big patch now18:43
EmilienMdprince: shouldn't you indent the hash key?
EmilienMslagle: what are the next step about multinode job? HA ? ceph? etc18:46
dprinceEmilienM: looking18:48
*** pcaruana has joined #tripleo18:48
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion in periodic job
slagleEmilienM: i think next steps would be to go >2 nodes18:49
slagle1 controller, 1 compute is probably next logical move18:50
slaglealso, i was going to work on a job to just install the undercloud18:50
slaglesince that is already working to install the undercloud on the ci slave node18:50
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Composable firewall rules
slaglewe could break that out, and have a job that just stops after installing the undercloud, and perhaps a smoketest18:51
EmilienMslagle: what coverage does have current multinode job? ipv6? ssl? ceph? netiso? all of them?18:51
slagleto give quick feedback18:51
EmilienMslagle: yes, big +1 on undercloud job18:51
slagleEmilienM: it doesnt yet do any of those. it does not even use os-net-config yet18:51
EmilienMslagle: it would be so faster for us18:51
EmilienMand we would save resources18:51
slagleso if we want net-iso, we'd need to add the os-net-config support18:51
slaglei was going to work on that next too18:51
pradkslagle, yep thats the one im testing as well18:52
slaglei meant to start tracking some of these things in storyboard, but never got around to it18:52
EmilienMwould you need help?18:52
slaglei will try to set that up18:52
EmilienMyeah, tracking it would be helpful for contributors who want to help18:52
EmilienMslagle: once you set a list of features/tasks in storyboard, please let you know. I want to pick one and start working on it18:52
*** akshai has joined #tripleo18:52
EmilienMlet me know I mean18:52
slagleok, i will try and do that this week18:52
EmilienMdprince: foo foo !18:53
*** bootsha has quit IRC18:59
*** pabelanger has joined #tripleo19:03
openstackgerritDan Prince proposed openstack/puppet-tripleo: Add firewall::service_rules define
EmilienMpabelanger: yo19:04
*** mbound has joined #tripleo19:04
pabelangerSo, I have some bandwidth to work on tripleo-ci again.  I want to see how I can get rid of the local mirror server you are using, and replace it with AFS19:05
pabelangera quick look into your mirrorlist, show some github git repos19:05
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: Do not exit prior to the inventory of the deployment.
*** jcoufal_ has joined #tripleo19:09
*** mbound has quit IRC19:10
openstackgerritDan Prince proposed openstack/instack-undercloud: Increase the max zaqar messages size to 2^19
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Remove templates/database
*** r-mibu has quit IRC19:13
*** r-mibu has joined #tripleo19:13
*** jcoufal has quit IRC19:13
*** akshai has quit IRC19:14
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Composable firewall rules
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Add 'service_name' to composable services
slaglepradk: the problem seems to be that ::apache::params::service_name is undef19:17
*** weshay has quit IRC19:18
slaglepradk: i think it's b/c nothing includes ::apache in overcloud_controller_pacemaker.pp anymore19:20
openstackgerritBen Nemec proposed openstack/puppet-tripleo: Include gnocchi config on non-bootstrap nodes
bnemecslagle: pradk: ^19:25
bnemecgnocchi is broken on non-bootstrap controllers right now19:25
slaglebnemec: oops19:27
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Deploy MySQL using composable role
EmilienMslagle, bnemec: wdyt about ^ ?19:29
pradkslagle, ah i see.. i couldnt find it in the logs, but did you narrow down to the apache issue?19:30
slaglepradk: i had a warning in my output:19:30
slagleWarning: Scope(Class[main]): Could not look up qualified variable '::apache::params::service_name'; class ::apache::params has not been evaluated19:30
*** fzdarsky|afk has quit IRC19:30
*** akshai has joined #tripleo19:31
slagleyet we do pacemaker::resource::service { $::apache::params::service_name:19:31
EmilienMdprince: wdyt about
slaglei don't really know how puppet processes includes19:31
bnemecEmilienM: Assuming it works, I like it.19:31
bnemecLess undercloud-specific config is a Good Thing(tm)19:31
slagleEmilienM: yea it seems like a good path to me19:32
EmilienMI'll continue that way19:32
EmilienMso maybe one day we can make composable undercloud19:32
bnemecBeing able to scale out some of the services might quiet some of the cries for HA undercloud.19:33
* bnemec hopes anyway19:33
dprinceEmilienM: This was the core idea of tripleo. We used to deploy the undercloud with t-h-t. We need to get back to it19:33
pradkslagle, only apache specifc thing i remove is  include ::aodh::wsgi::apache ..
pradkso this is been broken already even before this patch?19:34
dprinceEmilienM: fine with the patch. I don't think it gives us a feature yet though so I personally wouldn't do the work on it19:34
pradkor including the aodh::wsgi::apache is some how pulling the apache params in?19:35
slaglepradk: right19:35
EmilienMdprince: right but it's a baby step19:35
dprinceEmilienM: like... why push 20-30 patches and waste CI on this ATM?19:35
slaglepradk: without that line, nothing pulls in apache19:35
*** ramishra has quit IRC19:35
EmilienMdprince: I'm giving a try dude. If I have something in my env, I'll push a bigger patch19:35
dprinceEmilienM: CI resources is a top concern. I'd let the queue run other things. Unless there is a feature hiding under the covers giving us true composability that is19:35
pradkslagle, so i guess i can just add include ::apache in there and remove the rest of aodh19:36
slaglepradk: with this:, keystone will pull in apache19:36
slaglebut I think the include still happens too late19:36
*** jprovazn has quit IRC19:36
*** trown is now known as trown|relocating19:36
*** ramishra has joined #tripleo19:36
slagleEmilienM: how do includes work with puppet, if you try to name a resource that is dependent on an include having happened?19:37
slagleEmilienM: look at line 146 here,
EmilienMdprince: gotcha.19:37
slaglewe try to define a new resource called pacemaker::resource::service { $::apache::params::service_name:19:37
slagleEmilienM: but what if ::apache has not yet been included?19:38
EmilienMyou need to include ::apache::params before the resource19:38
EmilienMand it should work19:38
slagleright, that's what I thought19:39
EmilienMand it doesnt work?19:39
slagleso this won't work b/c all the composable manifests get concattenated onto the end of the manifest19:39
slagleonto the end of overcloud_controller_pacemaker.pp19:39
EmilienMI think we need a new profile for Apache19:39
slagleso we can't define this resource at line 14619:39
EmilienMand this profile will manage pacemaker::resource::service { $::apache::params::service_name19:40
EmilienMslagle: for sure you need to include ::apache::params somewhere in THT until we move out the resource19:40
slaglepradk: so i would do that in the aodh patch ^19:40
pradkEmilienM, so reason it worked before the aodh change is because i had include aodh::wsgi::apache?19:41
pradkand that pulled in params?19:41
pradkslagle, k so just include ::apache::params somewhere above in _pacemaker.pp /.19:41
slaglepradk: aiui, yes :)19:41
EmilienMI'm working now on moving the apache resource into a composable service19:42
slaglethis is more about the puppet parser than I ever wanted to know :)19:42
*** akshai has quit IRC19:42
*** akshai has joined #tripleo19:43
*** fzdarsky|afk has joined #tripleo19:46
*** akshai has quit IRC19:46
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Add Aodh composable roles
pradkcan we get this in please
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Implement Apache pacemaker profile
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Refactor mirror logic
openstackgerritDan Sneddon proposed openstack/tripleo-heat-templates: Add environment to preselect only VIP IP addresses
openstackgerritMerged openstack/instack-undercloud: Fix ceilometer auth url on undercloud
*** weshay has joined #tripleo19:58
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Include new apache pacemaker profile
EmilienMdprince: I need your thought on  ^20:00
EmilienMdprince: for apache, do you think we need a composable service in puppet/services/*.yaml ? Or should we just include the apache/pacemaker profile in the profiles that require it (keystone, horizon, etc)20:00
EmilienMin a container world, I would say option #1 is best20:01
*** fzdarsky|afk has quit IRC20:01
dprinceEmilienM: I'd rather see us compose with the heat puppet/services20:01
dprinceEmilienM: so a separate puppet/services/apache.yaml I think20:02
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Implement Apache pacemaker profile
dprinceEmilienM: maybe not actually20:02
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Include new apache pacemaker profile
EmilienMdprince: ok good20:02
*** julim has joined #tripleo20:02
EmilienMI can hack it now then20:02
EmilienMdprince: yeah, I was hesitating20:02
dprinceEmilienM: with containers I think you'd have apache inside your keystone container for example20:03
EmilienMok so I can include the class directly in the profiles20:03
EmilienMand we don't need this new service20:03
*** jcoufal_ has quit IRC20:05
EmilienMok so we can run it in the container. Good.20:05
dprinceEmilienM: I would say this is probably the norm20:05
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Explicitly set nova and neutron host on controllers
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Deploy Keystone and Gnocchi API with new Apache profile
dprinceEmilienM: just noticed we spend over a minute setting up keystone endpoints in the undercloud via puppet20:09
EmilienMbnemec: please do it in the profiles directly
dprinceEmilienM: are you aware of any efforts to optimize this a bit?20:09
EmilienMdprince: we are using pure openstackclient20:09
EmilienMI don't see how faster we can go20:09
dprinceEmilienM: right, I mean like caching some of the calls within a puppet run20:10
EmilienMexcept rewrite everything in ruby but it's not an option, we already talked about it20:10
EmilienMthere  is already some caching20:10
dprinceEmilienM: My guess is multiple calls are doing the same thing20:10
EmilienMchem wrote it20:10
dprinceEmilienM: yeah, it may be as good as it gets. Just noticing that besides package installation it is the longest running set of resources on the undercloud20:11
*** dprince has quit IRC20:13
slaglethis saves about 30s :)
*** beagles is now known as beagles_l8r20:15
slagleif you're looking for optimizations :)20:15
*** radek__ has quit IRC20:15
* bnemec has been tempted to rewrite the Mistral population into Python20:17
bnemecDoing anything through clis is slow.20:18
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Deploy Horizon with composable apache service
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Implement aodh profiles
*** rasca_ has quit IRC20:19
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Use create-legacy-resource-types during gnocchi-upgrade
*** rasca_ has joined #tripleo20:25
openstackgerritMerged openstack/tripleo-heat-templates: Fix ceilometer auth url for compute
*** egafford has quit IRC20:27
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Add Aodh composable roles
*** trown|relocating is now known as trown20:29
bnemecEmilienM: So I just want to set host on every node where a nova or neutron service is deployed.  Is it acceptable to just put it in their -base.yaml or do I need to add it to every service individually?20:31
*** milan has joined #tripleo20:31
EmilienMlike I wrote20:31
bnemecEmilienM: Oh crap, sorry.  I just saw your IRC ping and not the review comments.20:33
EmilienMno worries20:33
*** egafford has joined #tripleo20:36
*** florianf has quit IRC20:38
*** lblanchard has quit IRC20:41
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Explicitly set nova and neutron host on controllers
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Use growroot for undercloud image
*** mbound has joined #tripleo20:51
*** jayg is now known as jayg|g0n320:51
*** pcaruana has quit IRC20:53
*** radek2 has quit IRC20:53
*** dsariel has joined #tripleo20:55
*** jerrygb has quit IRC20:56
pabelangerokay, 2 patches up. 1 switches tripleo-ci to start using AFS mirrors more20:58
pabelangeranother to reduce the size of undercloud image (by using growroot)20:59
pabelangerneed to talk with openstack-infra about adding git repos to AFS20:59
*** jerrygb has joined #tripleo21:02
openstackgerritFlavio Percoco proposed openstack/tripleo-common: Extract `etc` files from openstack packages
*** ebalduf has joined #tripleo21:16
*** egafford has quit IRC21:17
*** trown is now known as trown|outtypewww21:18
openstackgerritMerged openstack/diskimage-builder: Fix packaging problems for Debian
*** pradk has quit IRC21:25
*** paramite has quit IRC21:26
*** rhallisey has quit IRC21:32
*** jerrygb has quit IRC21:32
*** TSCHAK has quit IRC21:34
*** fultonj has quit IRC21:36
*** cdearborn has joined #tripleo21:42
*** lucas-afk has quit IRC21:42
*** greghaynes has quit IRC21:46
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Add option to be able to in parallel
*** lucasagomes has joined #tripleo21:50
*** [1]cdearborn has quit IRC21:56
*** milan has quit IRC21:57
*** greghaynes has joined #tripleo21:58
openstackgerritStephanie Miller proposed openstack/diskimage-builder: Fix proliant-tools dependencies
*** ccamacho has quit IRC22:02
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Workaround for building stable branches
*** rajinir has quit IRC22:10
*** rhallisey has joined #tripleo22:11
*** rajinir has joined #tripleo22:12
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move elements into dib package
openstackgerritIan Wienand proposed openstack/diskimage-builder: Move element-info a standard entry-point
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Move diskimage-image-create to an entry point
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Testing stable branch workaround
*** ebalduf has quit IRC22:30
*** cdearborn has quit IRC22:31
sshnaidmbnemec, I'd like to test it also with master branch, to see if it doesn't break something there..22:32
bnemecsshnaidm: Good idea22:32
*** jerrygb has joined #tripleo22:33
sshnaidmbnemec, the stable branch I tested locally on dev env, works ok22:33
*** jerrygb has quit IRC22:39
*** zeroshft has quit IRC22:41
*** rhallisey has quit IRC22:45
*** rhallisey has joined #tripleo22:57
*** yamahata has quit IRC22:58
*** yamahata has joined #tripleo22:58
*** bfournie has quit IRC22:59
*** sshnaidm is now known as sshnaidm|afk23:08
*** saneax_AFK is now known as saneax23:10
*** egafford has joined #tripleo23:17
*** rlandy has quit IRC23:19
*** panda is now known as panda|zZ23:34
*** jerrygb has joined #tripleo23:35
*** mbound has quit IRC23:37
*** ayoung has quit IRC23:39
*** dsariel has quit IRC23:40
*** jerrygb has quit IRC23:41
*** penick has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!