Tuesday, 2016-08-23

ayoungOK...quickstart with  --extra-vars @config/general_config/devmode.yml  lead to a undercloud install failuer...RPM versions.  What is the straightest path to get a Newton-based deployment setup?  I assume removing the installed RPMS and setting up the proper repos....any guidance?00:08
ayoungah...should have done00:09
ayoung--release master-tripleo \00:09
sshnaidm|afkayoung, https://github.com/openstack/tripleo-quickstart/blob/master/ci-scripts/devmode_tempest.sh#L19-L2700:11
*** maeca1 has quit IRC00:16
*** limao has joined #tripleo00:28
ayoungsshnaidm|afk, thanks, but I think the problem is with the yum repos00:30
ayoungI think I am getting a mismatched set.  I got this error again:00:30
ayoungError: Package: openstack-swift-proxy-2.7.1-0.20160613173619.d75e6d3.el7.centos.noarch (@delorean)00:31
ayoung           Requires: openstack-swift = 2.7.1-0.20160613173619.d75e6d3.el7.centos00:31
ayoung           Removing: openstack-swift-2.7.1-0.20160613173619.d75e6d3.el7.centos.noarch (@delorean)00:31
ayoung               openstack-swift = 2.7.1-0.20160613173619.d75e6d3.el7.centos00:31
ayoung           Obsoleted By: python-swift-2.9.1-0.20160819204243.29d13b7.el7.centos.noarch (delorean-tripleo)00:31
ayoung               openstack-swift = 2.9.1-0.20160819204243.29d13b7.el7.centos00:31
sshnaidm|afkhmm.. repos are constructed here: https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/undercloud/tasks/update_image.yml#L9-L1400:32
sshnaidm|afkcheck that you really have the right ones00:32
sshnaidm|afkayoung, oh, check if you have delorean.repo file, most likely it's from old image and should be deleted00:34
sshnaidm|afk /etc/yum.repos.d/delorean.repo00:34
ayoungsshnaidm|afk, yep, they are different.  Thanks00:38
*** mgarciam has left #tripleo00:39
*** mgarciam has quit IRC00:39
ayoungsshnaidm|afk, I think it was that a delorean.repo file was left behind.  I removed that.  It looks like that script only creates delorean-*.repo file names...00:44
ayoungand, now that I reread what you wrote, yes, that is what i saw00:44
sshnaidm|afkayoung, you can vote for the fix :) https://review.openstack.org/#/c/335953/00:46
ayoungsshnaidm|afk, does that leave the delorean.repo file name alone?00:48
sshnaidm|afkayoung, the fix will overwrite it00:49
*** bana_k has quit IRC00:51
sshnaidm|afkayoung, in undercloud?00:51
ayoungsshnaidm|afk, yep00:59
ayoungERROR: 2016-08-23 00:48:12,756 --     Hook FAILED.00:59
ayoungERROR: 2016-08-23 00:48:12,757 -- Failed running command ['dib-run-parts', u'/tmp/tmp4KY3dE/install.d']00:59
sshnaidm|afkayoung, can you paste all log?01:00
ayounghttp://paste.openstack.org/show/562321/ sshnaidm|afk01:00
sshnaidm|afkhmm.. is it from tripleo-quickstart run?01:02
*** saneax is now known as saneax-_-|AFK01:02
*** chem has quit IRC01:02
ayoungsshnaidm|afk, yes01:07
ayoungsshnaidm|afk, Certs makes me suspect jaosorio's stuff.01:08
sshnaidm|afkayoung, weird, just now ran it, was installing ok01:08
ayoungsshnaidm|afk, I wiped out the old VMs, so it was not a vestige.  But maybe I still have some RPMs from the early run.  Let me try killing it and applying your patch01:09
sshnaidm|afkayoung, afaik ssl on undercloud is not merged yet: https://review.openstack.org/#/c/356431/  shouldn't be a problem..01:10
sshnaidm|afkayoung, for using "from scratch" images, you can use --extra-vars force_cached_images=False01:12
*** chem has joined #tripleo01:12
sshnaidm|afkit will take them from server01:12
ayoungsshnaidm|afk, is that what you use?01:12
sshnaidm|afkayoung, usually after promotion01:12
ayoungsshnaidm|afk, Congrats, you are now a Colonel01:13
*** Ryjedo_ has joined #tripleo01:22
*** Ryjedo has quit IRC01:24
*** Ryjedo_ is now known as Ryjedo01:24
openstackgerritMerged openstack-infra/tripleo-ci: Add memory to overcloud vms up to 6144  https://review.openstack.org/35753201:25
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install  https://review.openstack.org/35891901:26
*** alee has joined #tripleo01:37
aleeayoung, hey -- so magic incantations for quickstart01:37
ayoungalee, here is what I have so far01:38
aleeayoung, I'm running whats on master in tripleo-quickstart and just pointing to my machine on which to install01:38
aleeits failing in "Get undercloud vm ip address"01:38
ayoung bash quickstart.sh   --release master-tripleo   --extra-vars @config/general_config/devmode.yml  --extra-vars force_cached_images=False  ayoung-dell-t1700.test01:39
ayoungalee, localhost install?01:39
ayoungThere was something special for that.01:39
aleeayoung, no remote ..01:39
ayoungOK, so, I have my remote machine's IP addres in /etc/hosts01:39
*** coolsvap has quit IRC01:39
ayoungthat is ayoung-dell-t1700.test01:39
aleeayoung, well I'm using a beaker lab machine ..01:39
ayoung --release master-tripleo  gets you working with master.  Otherwise defaults to Mitala01:40
aleebut I can put it in /etc/hosts to see if that makes a difference01:40
ayoungalee, nah, so long as it resolves01:40
ayoungI did it with a Beaker machine, too.  warp....01:40
aleeyeah - its resolving ..01:40
aleeayoung, what does "force_cached_images=False" do?01:41
ayoungalee, according to sshnaidm|afk who told me to do that:  "for using "from scratch" images, you can use --extra-vars force_cached_images=False.  it will take them from server.  "01:43
ayoungalee, and then I promoted him to Colonel01:44
aleeayoung, ok -trying with your magic incarnation01:45
ayoungalee, I would like to point out that my install is still running, and I have not gotten a successful run yet, either.  So you have now caught up to me01:45
*** dmacpher has quit IRC01:46
aleeayoung, :/01:49
ayoungalee, undercloud install underway...01:49
*** saneax-_-|AFK is now known as saneax01:51
*** dmacpher has joined #tripleo01:59
*** hk_ has joined #tripleo02:14
*** yamahata has quit IRC02:15
hk_hello all02:19
*** hk_ has quit IRC02:23
ayoungalee, I missed a big one02:26
ayoungI applied this patch02:27
ayoungwithout that, my undercloud install failed02:27
aleeayoung, ah02:27
aleeayoung, did it succeed this time?02:28
ayoungalee, I think so.02:28
ayoungI have flavors, anyway02:28
ayoungand it looks like the undercloud-post-install.sh ran02:28
aleemine hasn't failed yet -- but I'm sure it will ..02:28
ayoungalee, the issue was that there was two Yum repos, both with the sameish priorities02:29
ayoungone name delorean.repo, the other delorean-tripleo.repo02:29
ayoungbut the first one was an old one, and some old packages got installed.02:29
*** saneax is now known as saneax-_-|AFK02:30
ayoungalee, I think I am going to try an overcloud install, see what happens02:30
*** hk_ has joined #tripleo02:32
*** hk_ has quit IRC02:38
*** ramishra has quit IRC02:40
ayoungalee, OK,  so one other thing.  THe setup only creates 2 nodes.  I think in order to have an overcloud deploy I need to tell it to only try and user those two02:46
ayoungit was...02:47
ayoung   ControllerCount: 102:47
ayoungat the end of deploy-env.yml02:47
ayoungand I think I am just going to sneak that into the overcloud deploy script02:47
ayoungactually added an additional -e $PWD/deploy-env.yml param to the call and added ^^ to that (new) file02:49
*** jdob has quit IRC02:54
*** bana_k has joined #tripleo03:06
*** ayoung has quit IRC03:14
*** ebalduf has joined #tripleo03:20
*** tzumainn has quit IRC03:26
*** Goneri has quit IRC03:31
*** yamahata has joined #tripleo03:40
*** bana_k has quit IRC03:42
*** Jaison has joined #tripleo04:15
*** Jaison has quit IRC04:24
*** jraju has joined #tripleo04:26
*** jobewan has joined #tripleo04:26
*** assassin has quit IRC04:32
*** jobewan has quit IRC04:34
*** jtomasek has quit IRC04:42
*** fragatina has quit IRC04:59
*** jprovazn has joined #tripleo05:02
*** jprovazn has quit IRC05:02
*** jprovazn has joined #tripleo05:03
*** masco has joined #tripleo05:06
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776505:09
*** jaosorior has joined #tripleo05:10
*** aqkhan__ has joined #tripleo05:23
*** aqkhan_ has quit IRC05:24
*** aqkhan_ has joined #tripleo05:24
*** aqkhan__ has quit IRC05:29
*** bkopilov has joined #tripleo05:29
*** numans has joined #tripleo05:40
*** r-mibu has quit IRC05:52
*** bana_k has joined #tripleo05:53
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move resource registry override to enable-tls.yaml  https://review.openstack.org/35897105:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE - TESTING BARBICAN  https://review.openstack.org/35326206:01
*** fragatina has joined #tripleo06:04
*** fragatina has quit IRC06:05
*** fragatina has joined #tripleo06:07
*** fragatina has quit IRC06:07
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE - TESTING BARBICAN  https://review.openstack.org/35326206:08
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add Barbican to the overcloud - WIP  https://review.openstack.org/35245806:08
*** fragatina has joined #tripleo06:09
*** numans has quit IRC06:11
*** r-mibu has joined #tripleo06:12
*** fragatina has quit IRC06:14
*** pcaruana has joined #tripleo06:14
*** tobias_fiberdata has joined #tripleo06:20
jaosoriorccamacho: ping06:22
*** jbadiapa has joined #tripleo06:24
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port  https://review.openstack.org/33869406:32
*** mcornea has joined #tripleo06:32
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643006:33
*** abregman has joined #tripleo06:37
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643006:39
*** ubijtsa has joined #tripleo06:40
*** tvignaud has quit IRC06:40
*** ubijtsa is now known as Guest1118206:40
*** tvignaud has joined #tripleo06:40
*** Guest11182 has quit IRC06:41
*** Guest11182 has joined #tripleo06:41
*** Guest11182 is now known as assassin06:42
*** saneax-_-|AFK is now known as saneax06:45
*** mhenkel has joined #tripleo06:49
*** florianf has joined #tripleo06:55
*** yolanda has quit IRC06:56
*** tremble has joined #tripleo06:58
*** tremble has joined #tripleo06:58
jaosoriorbandini: it seems that when the haproxy configuration is updated in an HA deployment. HAProxy is not reloading the configuration :/07:00
bandinijaosorior: how do you update the configuration? i.e. is pacemaker_resource_restart not triggering or something else?07:02
*** TSCHAK has joined #tripleo07:02
jaosoriorbandini: so, from a non-ssl I updated to an SSL configuration07:03
jaosoriorand haproxy is still serving the non-ssl configuration. So Step 3 of the post-deploy (puppet) fails07:03
*** yolanda has joined #tripleo07:04
bandinihow do you update a non-ssl conf to an ssl config? just change some environment variables and launch overcloud redeploy?07:04
jaosoriorbandini: yep07:04
*** dsariel has joined #tripleo07:04
*** bana_k has quit IRC07:05
bandinijaosorior: and pacemaker_resource_restart.sh is only triggered if we do a stack update, not when we redeploy against an existing overcloud, correct?07:05
*** vijaykc4 has joined #tripleo07:06
jaosoriorbandini: I think it's also triggered on stack update07:06
jaosoriorbandini: but I think it might be triggered too late07:07
bandiniah I see07:07
jaosoriorbandini: indeed07:08
jaosoriorbandini: it's ran after all the puppet stuff is ran07:08
jaosoriorbandini: so in my case it doesn't get triggered cause puppet fails itself07:09
bandinijaosorior: what are our options? move it earlier on? add another script that driggers early on?07:09
jaosoriorbandini: I'm not sure. I tried to manually run pcs resource restart haproxy-clone07:10
jaosoriorand that didn't do anything07:10
bandinimmh that does not sound right07:11
jaosoriorbandini: however, doing manually systemd restart helped07:12
jaosoriorpcs resource restart just had a success but didn't do crap :(07:12
*** bootsha has joined #tripleo07:12
*** milan has joined #tripleo07:12
jaosoriorbandini: do we disable refreshes to haproxy in puppet? Can we set it do systemctl reloads instead?07:13
jaosoriorjistr: are youa round?07:15
bandinijaosorior: so pcs resource restart haproxy-clone works fine here on a newton system07:15
jaosoriorbandini: I'm on a newton system07:16
bandinijaosorior: this is what I get in the logs after restarting via pcs : https://paste.fedoraproject.org/412735/36562147/07:16
bandinijaosorior: do you have some logs, more info?07:16
jaosoriorbandini: no logs07:16
jaosoriorbandini: nothing went on the logs07:16
bandiniyes we do inhibit the restart of pacemaker resources via puppet07:16
jaosoriorbandini: it just printed success and that's it07:17
bandinijaosorior: so you launch "pcs resource restart haproxy-clone" via shell and it does nothing?07:17
jaosoriorno restarting of haprox07:17
jaosorioras fucked up as that sounds07:17
bandinican you get me the "pcs cluster cib /tmp/cib.xml" (I need /tmp/cib.xml)07:17
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bond  https://review.openstack.org/34918307:18
jaosoriorbandini: will do07:18
*** tesseract- has joined #tripleo07:18
bandinijaosorior: thanks, happy to take a look there07:18
jaosoriorbandini: re-running the deploy now (after manually restarting haproxy with systemd)07:18
openstackgerritMerged openstack/tripleo-ui: Move plan upload errors from notifications to formerrors  https://review.openstack.org/33218107:19
bandinijaosorior: the only quick explanation I can think of is that "pcs property |grep maintenance-mode" returns "true"07:21
bandinianything else is rather weird07:21
jaosoriorbandini: I see07:22
jaosoriorbandini: didn't we set maintenance-mode for updates?07:23
bandinijaosorior: I don't think we do. (we will do for the migration from Full HA to NG HA)07:23
bandinijaosorior: do you have it set to true?07:23
jaosoriorbandini: deploy is still on-going07:24
jaosoriordon't wanna mess much with it cause I constantly get OOMs if I do small extra things...07:24
jaosoriormy deployment is fragile as fuck07:24
bandinisi que horror ;)07:25
*** bana_k has joined #tripleo07:25
*** ohamada has joined #tripleo07:26
*** jpich has joined #tripleo07:27
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776507:29
jaosoriorbandini: it was indeed in maintenance mode07:31
bandinijaosorior: ah okay at least we have a sensible explanation. any ideas why?07:33
jaosoriorbandini: I'm not entirely sure man07:33
bandinilet me dig around07:34
jaosoriorbandini: we do set it to maintenance mode07:35
jaosoriorin the standard pacemaker deployment07:35
bandiniin extraconfig/tasks/pacemaker_maintenance_mode.sh you mean?07:35
jaosoriorbandini: that's included in extraconfig/tasks/pre_puppet_pacemaker.yaml07:36
jaosoriorwhich is included in environments/puppet-pacemaker.yaml07:36
bandiniI see, yeah. Well then during that phase systemd restart/reload is the only way that will work07:36
jaosoriorbandini: that seems to be the case... now we need to figure out how to do that07:37
ccamachojaosorior good morning man, just waking up, Im a little feberish today :(07:39
ccamachotell me :)07:39
jaosoriorccamacho: haha I don't remember what the deal was07:40
*** hewbrocca-afk is now known as hewbrocca07:41
*** ifarkas_afk is now known as ifarkas07:42
jaosoriorbandini: I guess setting service_manage for haproxy when we're in maintenance mode wouldn't be too bad07:43
bandinijaosorior: that is one way to do it I guess, if it overrides the restart disabling that we do somewhere else07:44
jaosoriorbandini: which restart desabling?07:44
jaosoriorbandini: I guess the restart-disabling (that I mean) IS the service_manage flag that we give to puppet07:45
*** dbecker has joined #tripleo07:45
hewbroccaMorning folks07:47
bandinijaosorior: we used to have a bunch of these https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/pacemaker/neutron.pp#L5507:48
*** mpsairam has quit IRC07:48
bandiniI can't see it for haproxy though, maybe I misremembered07:48
bandinimorning hewbrocca07:48
*** sshnaidm|afk is now known as sshnaidm07:48
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui: Use Mistral API for plan deletion  https://review.openstack.org/35645807:48
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install  https://review.openstack.org/35891907:48
jaosoriorccamacho: now I remember. Did you dig further into reducing the number of workers in the undercloud>?07:49
jaosoriorbandini: for haproxy we have https://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy.pp#L46007:50
*** zoli_gone-proxy is now known as zoliXXL07:50
bandinijaosorior: ah righto07:50
ccamachoyes and no, :) I didnt realized than in my virtual environment there was no swap, so, what I did yesterday was to set the workers to 4 and add a 2GB swap file to the undercloud. And worked fine07:51
ccamachojaosorior ^07:51
ccamachoso, Im doing more deployments to see how it goes07:51
*** zoliXXL is now known as zoli|trng07:51
ccamachobut for now its working fine07:51
jaosoriorbandini: and this is where it ends https://github.com/puppetlabs/puppetlabs-haproxy/blob/master/manifests/service.pp#L1407:51
jaosoriorccamacho: I see.07:51
ccamachoI want to propose a small 1GB swap file for UC07:51
ccamachojust testing it now07:52
*** ebalduf has quit IRC07:54
*** rcernin has quit IRC07:56
*** nyechiel has joined #tripleo07:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE - TESTING BARBICAN  https://review.openstack.org/35326208:02
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add Barbican to the overcloud - WIP  https://review.openstack.org/35245808:02
*** bana_k has quit IRC08:02
*** openstackgerrit has quit IRC08:03
*** openstackgerrit has joined #tripleo08:04
*** skramaja has quit IRC08:07
*** jtomasek has joined #tripleo08:07
*** Goneri has joined #tripleo08:13
*** athomas has joined #tripleo08:14
*** lucas-dinner is now known as lucasagomes08:15
*** derekh has joined #tripleo08:18
*** apetrich has quit IRC08:19
*** dsariel has quit IRC08:20
*** ohamada_ has joined #tripleo08:20
*** ohamada has quit IRC08:20
*** apetrich has joined #tripleo08:21
*** pgadiya has joined #tripleo08:22
*** fragatina has joined #tripleo08:22
*** vijaykc4 has quit IRC08:22
*** vijaykc4 has joined #tripleo08:23
openstackgerritsandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI  https://review.openstack.org/28778408:24
*** coolsvap has joined #tripleo08:25
*** dsariel has joined #tripleo08:26
jaosorioreasy commit https://review.openstack.org/#/c/358971/ :D08:31
*** maeca1 has joined #tripleo08:31
*** vijaykc4 has quit IRC08:31
*** adarazs has quit IRC08:32
*** adarazs has joined #tripleo08:32
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Adding a 1GB swap file to the undercloud.  https://review.openstack.org/35903508:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Reload HAProxy on refresh  https://review.openstack.org/35903808:40
jaosoriorbandini: ^^08:40
*** ifarkas has quit IRC08:41
*** ifarkas has joined #tripleo08:42
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart: Reproducing specific CI run  https://review.openstack.org/34015608:45
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart: Devmode: rename delorean current so any existing repo is overwritten  https://review.openstack.org/33595308:45
*** akrivoka has joined #tripleo08:45
*** TSCHAK has quit IRC08:47
jaosoriorbandini: you think https://review.openstack.org/359038 will be alright?08:48
bandinijaosorior: just added a comment08:48
*** TSCHAK has joined #tripleo08:48
jaosoriorbandini: aha! one thing I could do is add an onlyif08:50
jaosoriorand check for08:50
jaosoriorpcs property | grep -q maintenance-mode08:50
bandinijaosorior: ah yes!08:50
jaosoriorthen we need to know if pacemaker is enabled08:50
jaosoriorcause that runs even on nonha08:50
bandiniin the short term yes, in the mid-term I hope we can just kill nonha now that the new HA stuff has landed08:50
jaosoriorbandini: it also runs in the undercloud08:51
bandinijaosorior: I guess a if [[ $(systemctl is-active pacemaker) = "active" ]] should do it though?08:53
jaosoriorbandini: I have an easier way08:53
jaosoriorjsut figured it out :P08:53
bandiniah cool08:57
openstackgerritmathieu bultel proposed openstack/tripleo-common: Use Openstack cli instead in order to get the instance status  https://review.openstack.org/35399608:57
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Reload HAProxy on refresh  https://review.openstack.org/35903808:58
*** electrofelix has joined #tripleo09:00
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: [NO MERGE] Test write performances  https://review.openstack.org/35905409:01
*** gfidente has joined #tripleo09:02
*** gfidente has quit IRC09:02
*** gfidente has joined #tripleo09:02
*** ifarkas_ has joined #tripleo09:02
jaosoriorgfidente: let me know when you're up for a cal09:03
*** mcornea has quit IRC09:04
*** mcornea has joined #tripleo09:04
jaosoriorbandini: uploaded another version. What do you think?09:04
jaosoriorjistr: are you around?09:05
jistrjaosorior: hi, yes but i have 0 extra bandwidth :/09:05
jaosoriorjistr: no problem. Was gonna have a call with gfidente to explain the certmonger stuff. But if you don't have time it's fine09:06
jistrjaosorior, gfidente: ok guys please ping me when you are starting and i'll decide then. I need to get to test one fix as soon as possible, perhaps i could be on a call in parallel.09:09
openstackgerritMerged openstack-infra/tripleo-ci: Use separated SSL endpoint environment file  https://review.openstack.org/35648809:09
*** abehl has joined #tripleo09:11
gfidentejistr can you ping us when it's best for you?09:11
*** abehl has quit IRC09:11
*** abehl has joined #tripleo09:11
jaosoriorccamacho: are you interested in joining also? Would sure use reviewers.09:12
ccamachohey! thanks sure :)09:12
jistrgfidente, jaosorior: ack thanks :) i'm hoping in 20 minutes or so, i just need to start the upgrade run09:12
jaosoriorjistr: it can be in 20 minutes. It's nothing with a hard-schedule09:13
jistrjaosorior: cool, thanks :)09:13
jaosoriorjistr: gives me time to make more coffee :P09:13
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: WIP DO NOT MERGE - Enable the HA deployment by default  https://review.openstack.org/35906009:13
gfidentejistr ccamacho cool guys :)09:14
*** d0ugal has quit IRC09:15
*** skramaja has joined #tripleo09:15
*** zoli|trng is now known as zoli_gone-proxy09:15
*** akuznetsov has joined #tripleo09:16
*** d0ugal has joined #tripleo09:16
*** milan has quit IRC09:17
*** ebarrera has joined #tripleo09:17
ccamachoQuick question, do you remember which is the file from CI logs that shows the node status (RAM, disk usage, etc) ?09:21
openstackgerritAna Krivokapic proposed openstack/instack-undercloud: Add default plan creation  https://review.openstack.org/34953209:22
*** ebarrera has quit IRC09:23
jaosoriordon't remember :/09:23
ccamachoajajaj I know its there but cant remember neither09:25
ccamachojaosorior, found it http://logs.openstack.org/94/355394/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/0abf721/logs/undercloud/var/log/host_info.txt.gz09:26
*** Goneri has quit IRC09:30
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: Refactor upgrade checks.  https://review.openstack.org/35775009:32
*** pgadiya_ has joined #tripleo09:33
*** Goneri has joined #tripleo09:33
*** spredzy has quit IRC09:33
*** maeca1 has quit IRC09:33
*** asbishop has joined #tripleo09:34
*** pgadiya has quit IRC09:34
*** athomas has quit IRC09:34
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Upgrade ceph-mon  https://review.openstack.org/35713009:36
jistrjaosorior, gfidente, ccamacho: upgrade running, i'm ready for a call if you are :)09:36
jistrthanks for waiting :)09:36
gfidentehttps://bluejeans.com/u/gfidente/ ?09:38
gfidentejaosorior ^^ ?09:38
jaosoriorcould be google hangouts if people prefer that09:38
jaosoriorI'm in gfidente's bluejeans09:39
jaosoriorso yeah09:39
jaosoriorhttps://bluejeans.com/u/gfidente/ is cool09:39
gfidenteexcept I don't hear you guys09:39
jistrgfidente: it says wait for moderator to start the meeting. Are you the moderator? :)09:39
*** lucasagomes is now known as lucas-afk09:40
*** athomas has joined #tripleo09:40
*** zoli_gone-proxy is now known as zoliXXL09:40
*** zoliXXL is now known as zoli|trng09:41
jaosoriorthere are three instances of gfidente in that meeting09:41
*** tosky has joined #tripleo09:41
gfidentelet's use something else09:41
gfidenteI have no clue guys :)09:41
jaosoriorwhat about this? https://bluejeans.com/252741440909:42
jaosoriorgfidente: that seems to work ^^09:42
*** dtantsur|afk is now known as dtantsur09:42
gfidentejaosorior much better thanks09:42
*** spredzy has joined #tripleo09:45
*** ebarrera has joined #tripleo09:46
Jokke_is this known issue or did I hit something new http://logs.openstack.org/94/355394/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/0abf721/console.html#_2016-08-22_14_57_37_19296409:51
*** bootsha has quit IRC09:53
Jokke_oh darn it failed here http://logs.openstack.org/94/355394/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/0abf721/console.html#_2016-08-22_14_47_49_233311 still have no idea09:53
*** ifarkas has quit IRC09:54
*** ifarkas_ is now known as ifarkas09:54
*** bootsha has joined #tripleo09:56
bandiniJokke_: I have seen that one too09:59
*** ebarrera has quit IRC10:01
*** asbishop has quit IRC10:01
Jokke_bandini: should I just recheck again or is there something else I actually can do about it?10:03
bandiniJokke_: I haven't investigated much but a recheck seemed to fix it10:04
Jokke_thnx, trying again10:05
*** limao has quit IRC10:10
*** ccamacho is now known as ccamacho|afk10:15
gfidentejaosorior topic is bp/tls-via-certmonger ?10:15
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add HAProxy TLS handled by certmonger as composable service  https://review.openstack.org/35643010:18
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776510:18
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable usage of FQDNs for the endpoints  https://review.openstack.org/35641610:18
jaosoriorgfidente: yeah. I'm updating the reviews10:19
gfidentejaosorior ah nice10:20
d0ugaljpich: Hey, do you have plans to write an openstack overcloud plan create?10:21
jpichd0ugal: No plans10:22
d0ugaljpich: nice pun!10:22
jpichI totally did that on purpose10:22
jpich(I didn't)10:22
jaosoriorgfidente, ccamacho|afk, jistr so it's updated now https://review.openstack.org/#/q/status:open+topic:bp/tls-via-certmonger10:22
d0ugaljpich: Okay, in the process of documenting stuff, I realise we really need a plan create and a way to deploy the plan.10:22
d0ugaljpich: openstack overcloud deploy --plan $NAME10:23
* d0ugal will create bugs10:23
shadowerd0ugal: why would you create bugs???10:24
d0ugalshadower: I don't know how to do anything else :(10:25
d0ugalshadower: I don't always report them tho'10:25
jpichd0ugal: From your experience in previous releases, what's the deadline for getting stuff into the client? I see the Newton schedule says Feature Freeze is also the final release for client libraries but.... :/10:25
d0ugaljpich: clients are usually a bit more relaxed10:25
jpichd0ugal: I thought the plan deploy stuff was what you were working on? Is that different?10:25
d0ugaljpich: Yeah, it's different - subtly.10:26
d0ugaljpich: I am making the existing command use mistral.10:26
d0ugaljpich: it creates a plan on the fly and deploys it10:26
jpichI see10:26
d0ugaljpich: we need to break that out so users can do the individual steps10:27
d0ugaljpich: for example, you added a delete plan - but you can't create a plan without deploying it yet.10:27
d0ugaljpich: all the code is in my patch, so it should be quite easy - in theory10:27
jpichd0ugal: Right... I've been calling the create default plan action but I don't think there's a CLI command for it either yet10:28
jpichd0ugal: "In theory" :P10:28
d0ugaljpich: True, the create plan command should probably allow users to select the default or provide their own templates.10:28
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776510:28
d0ugaljpich: the version I am working on always provides templates, because that is the current behaviour - often they will be the same as the default templates, but I don't think there is a good way to check.10:29
derekhdtantsur: did that tftp/iptables thing get unstuck yesterday?10:31
derekh*get you unstuck10:31
jpichd0ugal: Ok, thank you for the explanations! I'll try to give a hand with the missing plan commands10:31
d0ugaljpich: https://bugs.launchpad.net/tripleo/+bug/161601410:32
openstackLaunchpad bug 1616014 in tripleo "Provide an `openstack overcloud plan create` command" [High,Confirmed]10:32
d0ugaljpich: k, I can help with it too probably10:32
dtantsurderekh, yes, you're amazing (with bnemec who fixed my subnets misunderstanding)10:32
dtantsurderekh, now I got as far as passing out-of-band cleaning, so we're ready for deploy10:32
jpichd0ugal: Excellent :D10:32
d0ugaljpich: One more incoming.10:33
dtantsurderekh, do you mind reviewing https://review.openstack.org/356539 please? just adds a couple of missing bits?10:33
* jpich dodges10:33
derekhdtantsur: will do10:33
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Upgrade ceph-osd  https://review.openstack.org/35731210:34
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Upgrade ceph-mon  https://review.openstack.org/35713010:34
openstackgerritMerged openstack-infra/tripleo-ci: Pass TRIPLEO_ROOT directory to heat_deploy_times.sh  https://review.openstack.org/35694610:35
*** bootsha has quit IRC10:35
jpichd0ugal: I'm spending some time on reviews/testing patches at the moment, if you were about to write these don't mind me and go ahead - I'll give a hand reviewing them instead :)10:36
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Upgrade ceph-osd  https://review.openstack.org/35731210:36
d0ugaljpich: Sure, I'll do the create plan command. FYI, https://bugs.launchpad.net/tripleo/+bug/161601510:37
openstackLaunchpad bug 1616015 in tripleo "Provide a way to deploy a plan" [High,Confirmed]10:37
openstackgerritDougal Matthews proposed openstack/tripleo-docs: [WIP] Mistral API Documentation  https://review.openstack.org/35868510:38
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Reload HAProxy on refresh  https://review.openstack.org/35903810:38
jpichd0ugal: Roger. I'll look at whatever's left unassigned later on then10:38
jaosoriorbandini: thanks for the review10:38
*** ramishra has joined #tripleo10:39
openstackgerritDougal Matthews proposed openstack/tripleo-docs: [WIP] Mistral API Documentation  https://review.openstack.org/35868510:41
bandinijaosorior: no problem dude ;)10:44
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Move out of roles any Ceph setting  https://review.openstack.org/35665710:45
d0ugalhttps://review.openstack.org/#/c/298732/ <- needs some attention if anyone has time for reviews :)10:48
d0ugaljaosorior: Thanks!10:50
*** coolsvap is now known as coolsvap_10:53
*** TSCHAK has quit IRC10:56
*** TSCHAK has joined #tripleo10:58
*** rodrigods has quit IRC10:59
*** rodrigods has joined #tripleo10:59
*** pkovar has joined #tripleo11:03
*** milan has joined #tripleo11:04
*** jpena is now known as jpena|lunch11:06
*** bootsha has joined #tripleo11:07
*** ccamacho|afk is now known as ccamacho11:09
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Enable SSL for ironic inspector  https://review.openstack.org/33264111:10
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Refactor endpoint env variable population  https://review.openstack.org/35039611:10
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Enable TLS for Zaqar API  https://review.openstack.org/35157611:10
*** vijaykc4 has joined #tripleo11:12
openstackgerritAna Krivokapic proposed openstack/instack-undercloud: Add default plan creation  https://review.openstack.org/34953211:13
jaosoriorgfidente: hey dude, could you help me out a bit figuring out what's missing here? https://review.openstack.org/#/c/352458/1211:14
jaosoriorgfidente: also, if you jistr and ccamacho have some time to check out again the submission for FQDNs for the overcloud VIPs https://review.openstack.org/#/c/356416/ would appreciate it11:15
*** pkovar has quit IRC11:19
*** bkopilov has quit IRC11:19
*** ramishra has quit IRC11:20
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Enable SSL for undercloud-only job  https://review.openstack.org/35913111:20
*** ramishra has joined #tripleo11:22
Jokke_jaosorior: looks like the common error from the different runs is http://logs.openstack.org/58/352458/12/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/ebf551e/console.html#_2016-08-23_10_31_36_03455711:22
jaosoriorJokke_: which commit is that11:22
Jokke_jaosorior: 1211:23
jaosoriorJokke_: that's the error I'm seeing in the barbican-for-the-overcloud submissions. But I haven't figured out what's missing there exactly11:23
Jokke_the one you mentioned above11:23
jaosoriorI know what the error is11:23
jaosoriorbut it is not entirely apparent to me what did I actually miss11:23
jaosoriorJokke_: since  I'm not changing the VipDeployment's input. Just adding barbican there11:24
*** pkovar has joined #tripleo11:26
*** mburned_out is now known as mburned11:28
*** dprince has joined #tripleo11:29
*** adarazs is now known as adarazs_lunc11:29
*** adarazs_lunc is now known as adarazs_lunch11:29
*** numans has joined #tripleo11:30
*** limao has joined #tripleo11:31
*** limao_ has joined #tripleo11:32
*** rbowen has joined #tripleo11:33
*** rbowen is now known as DrBacchus11:34
d0ugalslagle: Thanks for the deploy workflow review!11:35
*** limao has quit IRC11:35
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate the deploy command to use the Mistral workflows  https://review.openstack.org/34226111:36
jaosoriord0ugal: wow11:36
jaosoriorlets see how that goes :O11:36
jaosoriorcool stuff11:36
d0ugaljaosorior: It wont actually be used anywhere until the CLI patch lands - but at least that can test against CI soon.11:37
*** jtomasek has quit IRC11:37
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install  https://review.openstack.org/35891911:37
d0ugal... and I need to do a few updates to the CLI deploy patch still, should be easier now.11:37
thrashd0ugal: awesome. I'm still trying to track down exactly what was failing for me yesterday.11:38
d0ugalthrash: Yeah, did you send me your envs?11:38
d0ugalI kinda hope that CI will fail and reveal issues...11:39
d0ugalbut I also kinda hope CI just works ;)11:39
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job  https://review.openstack.org/35914611:39
*** bfournie has quit IRC11:40
*** vipulnayyar has joined #tripleo11:40
*** ccamacho is now known as ccamacho|lunch11:41
vipulnayyarHey there, while setting up packages in undercloud, I'm getting following yum dependency issues:11:42
vipulnayyarError: Package: python-tripleoclient-5.0.0-0.20160819210400.f666339.el7.centos.noarch (delorean-current)11:42
vipulnayyar           Requires: python-osc-lib >= 0.3.011:42
Jokke_jaosorior: could it be because you did not add it to overcloud-without-mergepy.yaml?11:42
vipulnayyarApparently, python-osc-lib rpm is not present in delorean repo only src.rpm11:42
vipulnayyarCan someone guide me what would be the workaround for this?11:43
dtantsurvipulnayyar, please raise it on #rdo11:44
*** rcernin has joined #tripleo11:44
sshnaidmvipulnayyar, which repos do you have?11:47
jaosoriorJokke_: we're not using overcloud-without-mergepy.yaml. Actually I'm not sure if we can remove that file already :D that would be cool11:47
jaosoriorslagle: ?? ^^11:47
vipulnayyarsshnaidm: This one: http://trunk.rdoproject.org/centos7/aa/bf/aabf94f4cf5d5d3973e76b4b1b274ccfcdd29300_586add6611:48
*** jtomasek has joined #tripleo11:49
EmilienMgood morning folks11:49
Jokke_jaosorior: ok, just thought as VipDeployment was defined there as well, but noticed that you're not touching it ;)11:50
Jokke_hi EmilienM11:50
*** jaosorior has quit IRC11:50
thrashd0ugal: I sent you my list. They're all readily available. :)11:50
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Enable SSL for ironic inspector  https://review.openstack.org/33264111:51
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Refactor endpoint env variable population  https://review.openstack.org/35039611:51
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Enable TLS for Zaqar API  https://review.openstack.org/35157611:51
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Mistral db-populate depends on inspector's endpoint  https://review.openstack.org/35915311:51
*** jaosorior has joined #tripleo11:51
thrashd0ugal: puppet-pacemaker.yaml, tripleo-ci/test-environments/workers.yaml, and a puppet-modules.yaml from using the latest puppet modules.11:51
jaosoriorEmilienM: sup dude11:51
*** abehl has quit IRC11:52
thrashd0ugal: so yeah... just enabling puppet-pacemaker.yaml on master is doing it.11:52
*** limao_ has quit IRC11:52
thrashd0ugal: it didn't trigger on a packaged version of tht (I don't think I have the latest promote installed)11:52
thrashlet me try that.11:52
*** limao has joined #tripleo11:52
sshnaidmvipulnayyar, weird, it has such: https://trunk.rdoproject.org/centos7/aa/bf/aabf94f4cf5d5d3973e76b4b1b274ccfcdd29300_586add66/python2-osc-lib-0.4.1-0.20160802185554.ef4a4f1.el7.centos.noarch.rpm11:54
thrashd0ugal: works with [stack@instack ~]$ rpm -q openstack-tripleo-heat-templates11:54
thrashd0ugal: does not work with master.11:54
vipulnayyarsshnaidm: I tried manually installing that package, but it further brings up another dependency issue with python-oslo-utils11:55
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Permit usage of raw disks as backends for Swift storage  https://review.openstack.org/35084111:55
d0ugalthrash: k, thanks - just going for lunch but I'll take a look when I get back. Must have missed the list yesterday, sorry :)11:55
sshnaidmvipulnayyar, did you try yum install python2-osc-lib?11:55
thrashd0ugal: no worries. I'm troubleshooting on my end.11:55
sshnaidmvipulnayyar, if it's reproducible, like dtantsur said it's better to raise it in #rdo11:56
*** Goneri has quit IRC11:57
vipulnayyarsshnaidm: All this was working fine for me a couple weeks ago, by following the TripleO docs. Suddenly, it seems to be broken11:57
vipulnayyarsshnaidm: the package python2-osc-lib is somehow not available in any repo provided by TripleO11:57
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: WIP DO NOT MERGE - Enable the HA deployment by default  https://review.openstack.org/35906011:57
vipulnayyarsshnaidm: Also, I have raised this in #rdo11:57
*** dsariel has quit IRC11:58
sshnaidmvipulnayyar, I'm not sure, but it seems like yum issue, because python2-osc-lib is definitely in http://trunk.rdoproject.org/centos7/aa/bf/aabf94f4cf5d5d3973e76b4b1b274ccfcdd29300_586add66 . Maybe try "yum clean ll", rebuilding rpmdb and again11:58
*** adarazs_lunch is now known as adarazs12:00
dtantsurI've left my overcloud to work for some time, and now all my API services fail with broken pipe12:00
dtantsurI wonder if something is wrong around haproxy...12:00
*** yamahata has quit IRC12:00
openstackgerritMerged openstack/python-tripleoclient: Fix unbracket_ipv6()  https://review.openstack.org/35872912:01
EmilienMcan anyone summarize 1) how ovb jobs are back to green (specially HA) and 2) how promotion did happen?12:01
jaosoriorgfidente: I guess we need to backport this too https://review.openstack.org/#/c/358729/12:02
*** pgadiya_ is now known as pgadiya12:03
gfidentejaosorior yeah I don't know if akrivoka is pushing it maybe?12:03
jaosoriorakrivoka: can you backport https://review.openstack.org/#/c/358729/ ?12:03
akrivokajaosorior: sure12:03
*** Goneri has joined #tripleo12:04
sshnaidmslagle, hi12:04
jaosoriorakrivoka: thanks12:04
vipulnayyarsshnaidm: That worked for me :-) Thanks a lot12:04
slaglesshnaidm: hello12:05
sshnaidmvipulnayyar, np12:05
sshnaidmslagle, can I run periodic jobs again today, based on your patch? https://review.openstack.org/#/c/346949/12:05
sshnaidmslagle, of course setting cashupload to 1 and everything that needs..12:06
slaglesshnaidm: did the jobs pass last night?12:06
sshnaidmslagle, ha passed, nonha failed, I suspect it's casual fail12:07
slagleand if not, why. and have we fixed that problem yet12:07
*** abehl has joined #tripleo12:07
slaglesshnaidm: i think we should start filing some bugs for all these casual failures12:07
slagleand get some eyes on them12:08
dtantsurfor the record: I had to restart httpd to bring my overcloud make to life12:08
slaglesshnaidm: but if you want to rerun the periodic test, you can. we got a promote yesterday though12:08
sshnaidmslagle, ok, sure12:08
*** limao has quit IRC12:08
slaglesshnaidm: the other thing that needs doing is reporting on the status of the periodic jobs on tripleo.org12:09
sshnaidmslagle, yes, the second question is the mechanism to run it on demand, not waiting a day, is this patch is acceptable way to do it?12:09
sshnaidmslagle, we have this reporting in my page: http://status-tripleoci.rhcloud.com/12:10
slaglesshnaidm: yes, but we need something on tripleo.org12:10
slaglesshnaidm: it looks like both the jobs failed due to timeout this morning?12:11
slaglesshnaidm: we need to figure out why jobs are running so slowly in CI12:12
*** vipulnayyar has quit IRC12:12
slaglei'd focus on that instead of trying to get another promote12:12
sshnaidmslagle, ha job succeeded to upload images and report its success to mirror server before it was cut by timeout12:12
slagleit will probably just time out again though12:12
slagleb/c regular HA jobs are taking more than 2h40m12:13
sshnaidmslagle, right, it's a big problem now12:13
sshnaidmslagle, I thought maybe return to use cached images, but I'm not sure about status of puppet modules in overcloud images: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/common_functions.sh#L12812:15
slaglesshnaidm: we should have that now if a new image got uploaded on monday12:16
slaglesince the promotes passed on monday12:16
sshnaidmslagle, ok, then I'll try to use cache12:16
slagleok, great. you can confirm if the new image is there on the mirror server12:16
sshnaidmI'll check it\12:17
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port  https://review.openstack.org/33869412:18
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bond  https://review.openstack.org/34918312:20
*** DrBacchus has quit IRC12:22
*** rcernin has quit IRC12:23
*** asbishop has joined #tripleo12:23
*** jdob has joined #tripleo12:27
*** rhallisey_ has joined #tripleo12:28
*** morazi has joined #tripleo12:28
*** abregman has quit IRC12:28
*** rcernin has joined #tripleo12:29
*** cdearborn has joined #tripleo12:29
*** masco has quit IRC12:30
openstackgerritMerged openstack/tripleo-common: Adds Deployment Action & Workflow  https://review.openstack.org/29873212:30
hewbroccaslagle: if we're not using a cached image then CI is certainly going to be show...12:32
hewbroccaer slow12:32
pradkcan i get some reviews on these please - https://review.openstack.org/#/c/352476/ & https://review.openstack.org/#/c/293642/12:32
*** jcoufal has joined #tripleo12:32
*** abehl has quit IRC12:32
slaglehewbrocca: relative to using a cached image, of course it will be slower12:35
*** coolsvap_ is now known as coolsvap12:36
slaglebut it's not just image building12:36
hewbroccaOK, I see12:36
slaglejobs are slow across the board12:36
slagleha is taking 30-40 mins longer than it was just a month or 2 ago12:37
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-quickstart: DO NOT MERGE Add undercloud reboot - Testing  https://review.openstack.org/35048912:37
slagleanecdotally it seems it's just b/c we keep making everything slower12:37
slagleand not an actual infrastructure issue12:37
hewbroccaslagle: don't talk about gfidente that way12:38
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate the deploy command to use the Mistral workflows  https://review.openstack.org/34226112:38
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Add a big HA deployment configuration  https://review.openstack.org/35917312:38
*** bfournie has joined #tripleo12:39
*** ebarrera has joined #tripleo12:39
hewbroccaAll BS aside... did we make any progress not activating some of the new overcloud services?12:39
*** alee has quit IRC12:39
hewbroccaOr is that not even the issue?12:39
slagleyes, sahara is now off12:39
slagleas is cinder backup12:39
slagleso now we are just squeaking through on ram and timeout12:40
*** jpena|lunch is now known as jpena12:40
hewbroccaAnything else we can easily turn off?12:40
gfidenteI noticed the nonha job is also taking 2h40min12:40
gfidenteso it doesn't look like pcmk scenario is any better or worse12:41
*** vijaykc4 has quit IRC12:41
gfidentehere for example nonha is slower https://review.openstack.org/#/c/354613/12:41
akrivokajaosorior: https://review.openstack.org/#/c/358729/ backport needs to go into mitaka and liberty, right?12:41
slaglegfidente: yep12:42
openstackgerritAna Krivokapic proposed openstack/python-tripleoclient: Fix unbracket_ipv6()  https://review.openstack.org/35917412:42
gfidenteakrivoka both yes but liberty version needs to be on top of https://review.openstack.org/#/c/355882/12:42
slaglegfidente: the overcloud deploy took an hour in that job12:42
akrivokagfidente: ok, there's the mitaka version ^ and I'll wait for your patch to land into liberty first12:43
*** vijaykc4 has joined #tripleo12:43
jaosoriorakrivoka: yes12:43
slaglegfidente: might want to check the logs to see if maybe the nova instances got rescheduled a bunch of times, or what caused it to take so long12:43
*** vijaykc4 has quit IRC12:43
*** ebarrera has quit IRC12:43
*** abregman has joined #tripleo12:44
*** rlandy has joined #tripleo12:45
*** abehl has joined #tripleo12:45
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add manifests to inject and trust CA certificates  https://review.openstack.org/35302112:45
*** bootsha has quit IRC12:47
*** jubapa has joined #tripleo12:48
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: Documentation for installing and using Ironic in overcloud  https://review.openstack.org/35401612:49
*** dtantsur is now known as dtantsur|mtg12:50
jaosoriorEmilienM: Hey dude, before you went on PTO you mentioned that there might be an issue with deploying the keystone endpoints via puppet. Did you get time to ellaborate on what's up?12:51
EmilienMjaosorior: yes, chem and I commented on one of your patches12:51
EmilienMI -2'ed it12:51
*** lblanchard has joined #tripleo12:52
jaosoriorEmilienM: ah12:54
jaosoriornow I see12:54
jaosoriorEmilienM: so we need to make sure that all the *::keystone::auth parameters get into the hieradata in a node where keystone is running12:55
jaosoriorEmilienM: how do we do it for the firewall rules?12:55
EmilienMjaosorior: yes12:55
EmilienMfirewall rules are declared where api services are running12:55
jaosoriorEmilienM: seems to me that those are only executed in the specific nodes where those services are running12:56
*** morazi has quit IRC12:56
jaosoriorso that approach doesn't address this case12:56
*** chlong has quit IRC12:56
*** mburned is now known as mburned_out12:56
*** tzumainn has joined #tripleo12:56
*** tzumainn has quit IRC12:57
*** tzumainn has joined #tripleo12:57
*** tobias_fiberdata has quit IRC13:00
*** coolsvap is now known as coolsvap_13:00
*** rhallisey_ is now known as rhallisey13:03
*** noslzzp has joined #tripleo13:03
*** rcernin has quit IRC13:03
sshnaidmslagle, images weren't uploaded, because mirrors server is 100% used space, arrgh13:04
gfidenteslagle have any clue why I can't find this output in the console https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/get_host_info.sh#L37 ?13:07
gfidentethough reality is that in https://review.openstack.org/#/c/354613/ same code took 2h45m on ovb and 1h on multinode job13:08
sshnaidmgfidente, it should be in separate file host_info.txt13:08
gfidentesshnaidm yeah I can't find it there13:09
gfidentealso that command is not given as part of host_info so I think it should be in the console log no?13:09
tzumainnd0ugal, your mistral documentation is really great so far, big thumbs up from me13:09
*** akuznetsov has quit IRC13:10
*** akuznetsov has joined #tripleo13:10
*** akuznetsov has quit IRC13:11
gfidentesshnaidm looks like it's looking for stackrc in wrong location?13:11
slaglegfidente: in that nonha job, ControllerOvercloudServicesDeployment_Step4 took 18 minutes13:12
slaglegfidente: and ControllerOvercloudServicesDeployment_Step5 took 15 minutes13:12
slaglethat is insane13:12
*** chlong has joined #tripleo13:12
gfidenteslagle right I was looking for the event-list output as well, where is it?13:12
sshnaidmgfidente, it seems right location13:12
*** lucas-afk is now known as lucas-hungry13:12
slaglegfidente: it's possibly in the postci log13:12
sshnaidmgfidente, lemme see where this output is going to..13:12
gfidenteslagle++ postci :)13:13
gfidentesshnaidm thanks for checking as well13:13
slaglegfidente: http://logs.openstack.org/13/354613/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/8ed156c/logs/postci.txt.gz13:13
sshnaidmslagle, can I delete some folder in mirror server then?13:13
sshnaidmalthough 40G is not too much for server that hosting images..13:14
slaglesshnaidm: yes, you can delete some of the older uploads under /var/www/html/builds13:18
sshnaidmslagle, I think we need to consider to extend volume size there for long range usage..13:19
EmilienMjaosorior: can you land https://review.openstack.org/#/c/356754/ please?13:19
*** julim has joined #tripleo13:21
*** ccamacho|lunch is now known as ccamacho13:21
*** egafford has joined #tripleo13:21
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Add a big HA deployment configuration  https://review.openstack.org/35917313:21
jaosoriorEmilienM: do we have in hiera which services are enabled globally?13:21
jaosoriorgfidente: ^^13:21
EmilienMjaosorior: I don't recall13:21
*** ioggstream has joined #tripleo13:22
jaosoriordprince: ^^13:22
ioggstreamgfidente: hi13:22
*** pgadiya has quit IRC13:23
gfidenteslagle EmilienM I collected some data from the logs13:27
gfidenteI do not think this is a problem with the overcloud per-se13:27
gfidentenor with any of the steps in particular13:27
gfidenteslagle EmilienM look at these numbers https://paste.fedoraproject.org/412868/13:29
EmilienMgfidente: nonha is longer than ha?13:31
gfidenteEmilienM exactly, that reinforces the point13:31
gfidenteand multinode nonha takes 1h3m *less* than ovb nonha13:31
gfidentebut in general, looking at the ovb ha13:31
EmilienMgfidente: do you want me to +2 the patch?13:31
*** bkopilov has joined #tripleo13:32
gfidenteEmilienM no the patch is not actually needed, I used those CI logs to collect the numbers13:32
gfidente*yes* overcloud deploy runs for 50mins (out of 2h25mins)13:34
yolandahi, i'm on the need of overriding ipxe template, with pxe_append_params in the undercloud. I see that this is possible on newton, i look at puppet-stack-config and i see support for drivers::pxe there, but not on liberty or mitaka, how can i handle this?13:34
gfidenteit takes *10mins* to go from *nova please give me 4vms* to *here are your 4vms*13:34
*** tobias_fiberdata has joined #tripleo13:34
hewbroccaI guess the cloud is busy...13:35
gfidenteand 40mins to do nodes configuration, maybe we can speed that up a little13:35
gfidentebut undercloud configuration takes ~25mins13:35
jaosoriorgfidente: well, doesn't nonha do introspection also?13:35
slaglegfidente: i just don't understand how steps 4 and 5 can take almost 35-40 minutes13:35
gfidenteslagle right it's still a bit slower than undercloud in doing the puppet part13:37
gfidentebut some nodes depend on others, if you take controllers only13:37
gfidentenodes are ready at 8.45 and deployed at 9.3313:37
*** jcoufal_ has joined #tripleo13:38
gfidenteif you remove validations and networking13:38
gfidenteit goes from 8.56 to 9.3313:39
gfidenteand it makes 5 puppet steps13:39
slaglei wonder if it's all those openstackclient calls13:40
gfidentewe could make it a bit faster I suppose but it's not really the 2h25mins13:40
*** [1]cdearborn has joined #tripleo13:40
*** dprince has quit IRC13:40
*** jraju has quit IRC13:40
*** jcoufal has quit IRC13:40
slagleyea, there is no one single thing13:40
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-quickstart: Add undercloud reboot  https://review.openstack.org/35048913:40
*** dprince has joined #tripleo13:41
slaglegfidente: we've just made everything slower13:41
*** ayoung has joined #tripleo13:42
hewbroccahow excellent of us13:42
gfidentefor example, it takes 40mins to build the image13:42
hewbroccaare we taking advantage of the SSDs at all, or is that no longer possible with OVB13:42
gfidentebut I am unsure how we could speed that up13:43
*** akuznetsov has joined #tripleo13:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776513:43
ayoungjaosorior, ok...so got through some of the Newton based issues last night, ut can't deploy overcloud.  Some reason it could not find enoughhosts.13:43
*** abregman has quit IRC13:44
gfidenteif not by going with cached images and skip build entirely13:44
jaosoriorayoung: what steps did you take?13:44
jaosoriorayoung: and, are you using the latest quickstart?13:44
ayoungjaosorior, so ran quickstart like this...13:44
ayoungactually, starte by applying a patch13:45
ayoungjaosorior,  this one https://review.openstack.org/#/c/335953/413:45
ayoungpretty sure it is the latest plus that13:45
jaosoriorayoung: Do you have this patch included? https://review.openstack.org/#/c/351683/13:46
jaosoriorayoung: and, did you update the undercloud?13:46
ayoungjaosorior, OK, I don't think I have that one.  Let me rebase and try again13:47
*** akuznetsov has quit IRC13:47
openstackgerritMerged openstack/tripleo-heat-templates: Don't trigger mariadb upgrade dump/restore when not needed  https://review.openstack.org/35875513:48
EmilienMtripleo meeting in 10 min13:48
ayoungjaosorior, I've been tearing down the undercloud manually...doing so now13:48
EmilienMplease add off items here: https://etherpad.openstack.org/p/tripleo-meeting-items13:48
ayoungjaosorior, then13:49
ayoung bash quickstart.sh   --release master-tripleo   --extra-vars @config/general_config/devmode.yml  --extra-vars force_cached_images=False  ayoung-dell-t1700.test13:49
saneaxayoung, https://bugzilla.redhat.com/show_bug.cgi?id=122886213:49
openstackbugzilla.redhat.com bug 1228862 in instack-undercloud "Can `openstack undercloud install` have a --force-clean option so an error doesn't require restarting?" [Urgent,Closed: errata] - Assigned to jslagle13:49
*** mburned_out is now known as mburned13:49
jaosoriorayoung: alright, that makes sense. I use master (not master-tripleo) but I guess that shouldn't make much difference13:51
ccamachohey jaosorior, re the OOM issues, at the end by adding some swap (not done by default in undercloud or overcloud) the OOM issues went away was able even to deploy mistral and sahara... This is awesome!   This is how im doing it.. http://www.anstack.com/blog/2016/08/23/oom-swap-fix-in-tripleo.html   now im trying to get different CI jobs as some of them dont have the host_info.txt.gz to see if they are using swap13:52
gfidentebnemec curiosity, do you have iowait of the ovb hoss?13:53
*** cdearborn has quit IRC13:53
hewbroccaand also the ovb hoss13:53
hewbroccab/c it makes me think of Bonanza13:53
jaosoriorccamacho: nice!13:54
jaosoriorayoung: This is exactly what I do http://jaormx.github.io/2016/deploying-master-tripleo-quickstart/13:54
ccamachoits a simple fix but makes the deployments work better, im even using now 6GB overcloud nodes!!! :P13:54
ayoungjaosorior, thanks.  need to pass that on to alee and jdennis13:54
jaosoriorayoung: And what I do for FreeIPA http://jaormx.github.io/2016/freeipa-tripleo-integration-notes/13:55
ayoungjaosorior, need to get you your own TLD.  github should not get credit for your work.13:56
ayoungBoth look like they are spectacularly useful.  Thanks13:56
slaglederekh: which are the sdd drives in rh1? /dev/sd<what>13:56
jaosoriorayoung: adarazs mentioned that they're thinking on deprecating the master-tripleo option13:57
ayoungjaosorior, ok...so long as it works this time, I'll be happy13:57
*** dprince has quit IRC13:58
adarazsyeah, not sure if it makes sense to use the upstream images vs. building it with the image building role in centos ci.13:58
ayoungbut I'll move over to master next.  It seems righter.13:58
jaosoriorayoung: I use just master, and then go ahead and rebuild the images13:58
adarazsas we're not testing it extensively.13:58
derekhslagle: it depends on the host your on, iirc some have 2 spinning hd's and others have 4, but its usually the last one on the list13:58
derekhslagle: iirc I think its also the smallest one13:58
*** dprince has joined #tripleo13:58
EmilienMmeeting is about to start in a minute, get ready!13:58
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Set the default gnocchi driver to swift  https://review.openstack.org/35438513:59
*** zaneb has quit IRC13:59
derekhslagle: when it was deployed the compute nodes just started using them for / (at least the ones I checked)13:59
slaglederekh: ok, i was wondering b/c lsblk is saying all the disks are rotational14:00
slaglethe but last disk is mounted at /14:00
slagleso, i guess14:00
*** zaneb has joined #tripleo14:01
*** fragatina has quit IRC14:01
gfidentederekh do we have iowait from the hosts?14:02
*** pcaruana has quit IRC14:02
derekhgfidente: do you mean are we monitoring it?14:02
gfidenteno just sample data14:02
gfidentelike a 1minute sample14:03
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs  https://review.openstack.org/35921514:03
derekhgfidente: I'm sure we could get it,14:03
gfidentelonger better, ideally on a loaded host but I supposed load is spread on all of them in a similar way14:03
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Don't trigger mariadb upgrade dump/restore when not needed  https://review.openstack.org/35921814:04
*** tbarron|gone is now known as tbarron14:05
*** coolsvap_ is now known as coolsvap14:05
socialwhat ironic devs do we have here?14:05
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: WIP Add a function to upgrade from full HA to NG HA  https://review.openstack.org/35862614:06
derekhslagle: I'm gonna guess its reported as rotational because we added it as a volume in PERC14:07
slaglederekh: yea, agreed14:07
slaglethe speed seems about right for an ssd14:07
slaglei'm getting around 400MB/s14:08
slaglewith my scientific dd tests14:08
slaglei don't actually think there's anything going on here other than we just keep making shit slower14:08
slaglewe lost 10 minutes on the undercloud install alone by just moving it all to puppet14:09
* bnemec wonders if we're still taking a hit on node registration and introspection too14:10
*** lucas-hungry is now known as lucasagomes14:10
bnemecAlso worth noting that I found a few compute hosts in rh1 that don't appear to have ssds.14:10
slaglebnemec: what's the node registration problem?14:11
slaglebesides just the general mistral slowness14:11
bnemecslagle: When we went to the Mistral version of node registration it added 2-3 minutes to that step.14:12
*** ebalduf has joined #tripleo14:12
bnemecI was told that it was because we were setting the provisioning state on the nodes unnecessarily for some reason.14:12
d0ugalbnemec: Really? That's odd :/14:12
slaglebnemec: k, yea, that's the same thing i was thinking14:13
d0ugalah :/14:13
d0ugalIs there a bug for this? I could look into it14:13
bnemecI mean, it doesn't explain why our jobs went from taking 1:40 to 2:30 alone.14:13
bnemecBut enough of those sorts of things and you might get there.14:13
bnemecd0ugal: I don't know.  I believe I heard this in an IRC conversation when I brought up the sudden slowness in that step.14:14
gfidentefwiw, in https://review.openstack.org/#/c/354613/ the --register-nodes took only 2mins14:15
*** hewbrocca is now known as hewbrocca-afk14:15
d0ugalbnemec: k, thanks - I'll take a look later anyway when I get a moment.14:16
socialdtantsur|mtg: https://bugs.launchpad.net/ironic/+bug/1616092 but if someone from ironic pokes me I think I should be able to track it down14:16
openstackLaunchpad bug 1616092 in Ironic "code path where capabilities aren't converted" [Undecided,New]14:16
slaglegfidente: yea i see that14:16
*** pcaruana has joined #tripleo14:17
gfidenteslagle I am trying to figure how much it took to puppet to compile the overcloud manifest, which should be cpu-bound14:17
gfidentedon't ask me why but it's only loggin up to step 3 ;P14:19
slaglegfidente: 4, 7, 18, 56, 12014:19
*** rajinir has joined #tripleo14:19
slaglein seconds, for each step14:19
slaglei did: journalctl --file ./journal/b7177a4eb7d9df34f0b10265f669e835/system.journal -u os-collect-config | grep -i 'compiled catalog'14:20
gfidenteslagle cool thanks ;)14:20
slagleso 2 minutes just to compile the catalog in step 514:20
derekhgfidente: looks ok on a random compute node http://paste.openstack.org/show/562479/14:21
gfidenteslagle right and on my VMs it takes 40secs14:21
gfidentewhich means cpu is probably about three times slower14:21
slaglegfidente: what do your vm's have for cpu/ram?14:21
gfidenteqemu virtual cpu14:22
gfidenteon a xeon14:22
gfidente1 vcpu14:22
gfidentelet's check multinode14:22
slagleok, same as CI then14:22
gfidenteas it only takes 1h !14:22
dtantsur|mtgsocial, moving to tripleo, this has nothing really to do with ironic14:22
slaglemultinode vm's have 4 vcpu and 8GB ram14:23
slaglei wonder if we shouldnt just make ovb match that14:23
slagleit would at least match what infra uses14:23
dtantsur|mtgjpich, I think you've faced https://bugs.launchpad.net/tripleo/+bug/1616092 already, haven't you?14:23
openstackLaunchpad bug 1616092 in tripleo "code path where capabilities aren't converted" [Undecided,New]14:23
*** dsariel has joined #tripleo14:23
gfidenteslagle multinode makes step5 in 27secs14:24
jpichdtantsur|mtg: I have14:24
sshnaidmslagle, bnemec do you know what is cache mode on virtual disks on ci?14:24
jpichdtantsur|mtg: But it was using my Brand New Code paths I think. Maybe it's better to fix everywhere though14:24
gfidentederekh nice so iowait is pretty much 0 so it's not disk issues it seems14:25
jpichdtantsur|mtg: I'll add a comment with what I saw and how I fixed it, in case it turns out handy14:25
gfidenteslagle bnemec ^^ instead looks like CPU has much bigger impact14:25
slaglelet me see if i can run a multinode job in rh114:25
slaglei'll need to set it up manually, but that will give us something to compare to14:26
bnemecsshnaidm: I believe we're using unsafe caching in CI, so it should be the fast, but you lose everything if a system goes down setting.14:26
*** dtantsur|mtg is now known as dtantsur14:26
derekhgfidente: at least on that random host over a minute14:26
bnemecWe were having CPU throttling issues back before the move to OVB.  I wonder if those could be back.14:27
gfidentederekh yeah are there reasons to think it changes much on a different node?14:27
gfidentefirst output from iowait are stats from boot, and it still says 0.01, 95% idel14:27
* bnemec needs to get his rh1 monitoring set up on his new laptop again14:27
derekhgfidente: well bnemec mentioned that a couple of the compute nodes don't have the SSD (we may not have configure them in perc properly)14:28
bnemecYeah, the list I have is .53, .55, .5714:28
*** maeca1 has joined #tripleo14:28
bnemecI think those are the provisioning addresses for the compute nodes that appeared to have rotational disks.14:29
*** adam_g has quit IRC14:29
bnemecI only checked ~half of them though.14:29
asbishopbnemec gfidente slagle: hoping you can look at https://review.openstack.org/#/c/354839/14:29
*** dhill_ has joined #tripleo14:31
openstackgerritMerged openstack/puppet-tripleo: Clean up unused params from aodh profiles  https://review.openstack.org/35652314:31
openstackgerritMerged openstack/instack-undercloud: (mitaka only) Fix sysctl_settings configuration  https://review.openstack.org/35675414:31
*** jaosorior is now known as jaosorior_away14:32
jpeelerrbrady: i see your review is also having difficulty with the nonha gate14:33
jpeelerslagle: after rechecking yesterday the nonha gate still didn't pass for me either https://review.openstack.org/#/c/348537/14:34
*** adam_g has joined #tripleo14:34
*** adam_g has quit IRC14:34
*** adam_g has joined #tripleo14:34
rbradyjpeeler: neither of our patches are being run in CI that I know of, so these are unrelated failures14:35
jpeelerexactly, but nobody wants to approve them until they pass14:35
slaglejpeeler: exactly, so work on making them pass14:35
slagleor don't, and they won't get approved14:36
*** davidlenwell has quit IRC14:36
openstackgerritMerged openstack/puppet-tripleo: Move ceilometer api to run under apache wsgi  https://review.openstack.org/35247614:36
openstackgerritJiri Stransky proposed openstack/puppet-tripleo: Add manifests to inject and trust CA certificates  https://review.openstack.org/35302114:36
*** davidlenwell has joined #tripleo14:37
*** ChanServ sets mode: +v davidlenwell14:37
jpeelerslagle: is there not anybody else to work with on this? i'm not asking you to do the work for me14:39
jpeeleri don't think i have the necessary knowledge of tripleo to debug these failures14:41
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: DO NOT MERGE TESTING CI JOBS  https://review.openstack.org/35923714:43
*** fragatina has joined #tripleo14:43
*** akuznetsov has joined #tripleo14:43
slaglejpeeler: i dont know who can debug them for you14:43
*** zoli|trng is now known as zoli14:44
*** zoli is now known as zoliXXL14:44
rbradyslagle: do we still have a CI team?14:45
slaglerbrady: to do what exactly?14:46
rbradyslagle: I thought we used to have a group of people who had knowledge/experience and access to CI to fix when issues arose.14:47
rbradyslagle: not sure if that group still exists14:47
slaglewe have such a group of people14:48
*** akuznetsov has quit IRC14:48
slaglei'm not hearing about an issue for that group to fix (i don't think)14:48
jpeelerit was just briefly mentioned in the meeting14:48
rbradyslagle: the heat stack create is failing, but the patches in question aren't tested in CI - they shouldn't be the cause of the failure14:49
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: Documentation for installing and using Ironic in overcloud  https://review.openstack.org/35401614:49
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: [WIP] Document hybrid virtual and bare metal overcloud  https://review.openstack.org/35924014:49
gfidentebnemec if we can install sysstat14:49
gfidenteit'll start collecting data with sar14:49
rbradyslagle, jpeeler: I guess we need to file a bug first?14:50
gfidentemaybe we can check the iowait in a longer timeframe then14:50
slaglerbrady: jpeeler : to be clear on my stance, it is everyone's job to work on CI14:50
gfidenteincluding CPU as things seem cpu-limited given puppet compile times14:50
slaglerbrady: jpeeler : yes, i'd file a bug and help on finding the fix. the maybe the job will pass on your patch14:50
ccamachothere was a repo promote, is the submission rebased? http://logs.openstack.org/37/348537/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/c2b5efe/logs/postci.txt.gz14:51
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: [WIP] Add fluentd client service  https://review.openstack.org/35350614:51
openstackgerritLars Kellogg-Stedman proposed openstack/puppet-tripleo: [WIP] updates to fluentd support  https://review.openstack.org/35350714:51
rbradyslagle: I get that we have to be good citizens and pitch in.  It's just inefficient to tell someone who works in a small subset of tripleo to go tackle debugging a CI job alone14:51
ayoungjaosorior_away, that looked better.  When I ran in the past I didn't get the Virtual Environment Setup Complete screen.  This time I did.14:51
slaglerbrady: jpeeler : but i dont feel that there is "another group" to assign it to14:51
ccamachosometimes I have these issues and rebasing all as there were jobs passing CI i.e. https://review.openstack.org/#/c/357312/14:52
slagleif people get stuck, we can try and get help or experts to help get them unstuck14:52
*** Goneri has quit IRC14:52
slaglerbrady: i agree that would be inefficient14:53
slaglebut no one has told anyone that14:53
slaglerbrady: to be clear, this convo started with "CI job is failing, it's not my patch"14:53
slaglei would not expect someone else to take over at that point :)14:54
jpeelerapologies for that, that's my fault14:54
slaglelet's get a bug (if there isn't one), some triage done, some sufficient log diving, etc to see if anyone knows what's going on14:55
jpeelerpart of the problem is it hasn't been failing consistently14:56
jpeeleri'll do one more recheck to see if yesterday's problem is confusing things14:56
ccamachojpeeler, not recheck, mind to rebase if not rebased?14:57
ccamachojust to be sure14:57
colonwqjpeeler, slagle Rabbitmq-server failed to start on controller 0.14:57
colonwqI have this happen 1/5 of the times I test locally on my laptop.14:58
*** maeca1 has quit IRC14:58
*** weshay is now known as weshay_afk14:58
jpeelerccamacho: force a rebase? i can try that.14:59
openstackgerritJeff Peeler proposed openstack/tripleo-common: Change output for GetCapabilitiesAction  https://review.openstack.org/34853714:59
colonwqentry in Aug 22 23:21:12 of http://logs.openstack.org/37/348537/4/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/c2b5efe/logs/overcloud-controller-0/var/log/messages15:00
sshnaidmzoliXXL, hi15:01
zoliXXLhi sshnaidm - I am dialing in15:01
jpeelercolonwq: thanks, rabbit is failing on mine as well. did you already file a bug?15:02
zoliXXLsshnaidm, just need to struggle with BJN15:02
ccamachojpeeler, nice lets wait for a fresh CI15:02
colonwqjpeeler, I was thinking it was my environment since I bootlegged it locally. I run the integration stack but my VM and network config may be different.15:03
jpeelerccamacho: to be clear though, wouldn't a rebase only change the reviewed code. that is a recheck wouldn't use anything the latest CI correct?15:03
colonwqrabbitmq connects to epmd but failes some times. epmd is running by the time I look at it. Rerunning the puppet config a second time does not help.15:04
*** Goneri has joined #tripleo15:07
*** weshay_afk is now known as weshay15:07
ccamachoyou are rebasing your patchset to latest master, right? then will run latest tht master (with your changes) against all the other latest master repos, right? I had sometimes issues that having old puppet-tripleo submissions using latest tht in which case.. Had some module dependencies issues, and rebasing everything up to HEAD made work correctly.15:08
ccamachojpeeler ^15:08
jpeelerah i see what you're saying15:08
*** hewbrocca-afk is now known as hewbrocca15:08
*** jtomasek has quit IRC15:08
ccamachojpeeler I'm saying what I usually do :P "But it's not in any case a must"15:09
gfidentechem do you have submissions still up for review to enable the updates/upgrades jobs?15:11
chemgfidente: you mean gerrit review for the upgrade ?15:13
gfidentechem yep, upgrade and update15:13
gfidenteI remember we moved update in the experimental queue, correct?15:13
chemgfidente: yep a lot :)15:14
chemgfidente:  https://review.openstack.org/358022 , https://review.openstack.org/358006 , https://review.openstack.org/358005 , https://review.openstack.org/357022 , https://review.openstack.org/354713 , https://review.openstack.org/35775015:14
dtantsurfolks, do THT allow to create some overcloud objects as part of the installation? e.g. networks, host aggregates, etc?15:14
openstackgerritLars Kellogg-Stedman proposed openstack/puppet-tripleo: [WIP] updates to fluentd support  https://review.openstack.org/35350715:14
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: [WIP] Add fluentd client service  https://review.openstack.org/35350615:15
chemgfidente: maybe you know the answer to this one.  Is there a way to send back and forth information from the overcloud during deployment to the undercloud like writing/reading in the undercloud swift storage  ?15:16
gfidentechem no, we thought about adding a tempurl in undercloud swift but never built on it15:17
hewbrocca dtantsur: we create keystone endpoints15:17
hewbroccaand I think that's done by puppet execing a script that talks to keystone (if I'm not mistaken)15:18
dtantsurhewbrocca, IIRC it's not in THT, rather some specific python code in os-cloud-config, no?15:18
dtantsurhmm, maybe15:18
hewbroccano b/c os-cloud-config.die.die.die15:18
hewbroccathe *new* way IIUC is do it with puppet15:18
gfidentedtantsur it's not in tht no, you'd have to do it in puppet15:18
EmilienMgfidente, dprince, slagle: I would be super happy to have feedback on https://review.openstack.org/#/c/356675/15:19
dtantsuranyway, we have an unpleasant chicken-and-egg problem with ironic requiring a network UUID to *start* in case of multi-tenant15:19
chemgfidente: I asked as it could be a easy way to sync the overcloud node during upgrade, but maybe there is another way ?15:19
dtantsurgfidente, so e.g. for networks I need support in puppet-neutron, right?15:19
gfidentechem you mean steps syncing?15:20
chemgfidente: no within script, like non boostrap node waiting for success on the boostrap node or the like15:20
gfidentedtantsur yeah, or you can pull in puppet-neutron in puppet-ironic and make it create the network you need if you know the other parameters15:20
*** ifarkas is now known as ifarkas_afk15:21
chemgfidente: a node pooling a undercloud swift file like a lock and waiting for the right output with a curl in a script15:21
gfidentechem right this is what we usually need the steps for15:22
gfidentebootstrp does it in step-115:22
chemgfidente: ack, oki, will look closer into it, thanks15:23
gfidentechem point is that heat gives guarantee about the steps15:23
gfidentechem so you can make bootstrap do things before the others15:23
thrashd0ugal: ok. it was user error/misunderstanding. Once the correct templates were uploaded to swift, it worked.15:25
*** abregman has joined #tripleo15:25
hewbroccagfidente: dtantsur wants to create an *overcloud* neutron network though15:27
*** davidlenwell has quit IRC15:27
hewbroccagfidente: not an undercloud neutron network...15:27
gfidentehewbrocca right if it was undercloud we could have used THT15:27
hewbroccajust making sure that part was clear15:28
*** larstobi has quit IRC15:28
*** yamahata has joined #tripleo15:29
gfidentedtantsur so this guy https://github.com/openstack/puppet-neutron/blob/master/lib/puppet/provider/neutron_network/neutron.rb from puppet-neutron can create neutron networks15:29
gfidentethough we don't create any network in overcloud neutron15:30
d0ugalthrash: phew :)15:30
*** dprince has quit IRC15:30
d0ugalthrash: it will be easier when we have a "create plan" command that does the container create and upload.15:30
dtantsurgfidente, interesting, thanks!15:30
*** abregman is now known as abregman|mtg15:31
*** dprince has joined #tripleo15:32
gfidentedtantsur it will get keuystone credentials from neutron.conf https://github.com/openstack/puppet-neutron/blob/master/lib/puppet/provider/neutron.rb#L31 it seems15:33
gfidentedtantsur so if you try a neutron_network after overcloud keystone is ready it might get you somewhere15:33
dtantsuryeah, probably I'll have to pick one node to run on it... (or will it run correctly on several nodes in parallel?)15:34
*** larstobi has joined #tripleo15:34
gfidenteright, bootstrap_nodeid that is15:34
gfidenteyou can pick that in puppet-tripleo ironic role15:34
dtantsurokie. I'm still trying to solve it on ironic side, but will fallback to this approach if I fail there. thanks!15:37
thrashd0ugal: gotcha15:37
*** davidlenwell has joined #tripleo15:40
*** ChanServ sets mode: +v davidlenwell15:40
*** abehl has quit IRC15:42
*** tremble has quit IRC15:42
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate the deploy command to use the Mistral workflows  https://review.openstack.org/34226115:43
gfidentechem EmilienM but is 'updates' meant to become 'experimental' ?15:44
gfidenteor should it be gating?15:44
EmilienMupdates should be gating probably15:44
EmilienMbut upgrades are really exp15:44
gfidenteslagle can we do netiso in multinode?15:45
slaglegfidente: yes, i think we could15:45
slaglegfidente: the job does not yet use os-net-config to configure networking though15:45
slaglewould need to do add that first15:45
chemgfidente: not sure what you mean here ?15:46
*** tesseract- has quit IRC15:46
gfidentecause I am trying to bring back ipv6 in ci15:46
gfidenteideally together with tls15:47
openstackgerritDougal Matthews proposed openstack/tripleo-common: Better namespace the list of TripleO actions  https://review.openstack.org/35768215:47
gfidenteand nonha looked a good fit15:47
gfidentebut previously we had it for updates job15:47
gfidentewhich will be based on multinode though15:47
*** saneax is now known as saneax-_-|AFK15:51
slaglegfidente: i think we just need to get it switched over to use os-net-config, and we could do that15:51
openstackgerritDougal Matthews proposed openstack/tripleo-common: Remove "type: direct" from workflows as it is the default  https://review.openstack.org/34161715:51
openstackgerritFlorian Fuchs proposed openstack/tripleo-common: Exclude more unneeded files from default plan  https://review.openstack.org/35547215:52
*** pcaruana has quit IRC15:56
*** [1]cdearborn has quit IRC15:58
*** maeca1 has joined #tripleo15:58
*** [1]cdearborn has joined #tripleo15:59
gfidenteslagle like this guy? single-nic-linux-bridge-vlans/ceph-storage.yaml15:59
gfidentethis one actually net-single-nic-linux-bridge-with-vlans.yaml16:00
*** jaosorior_away is now known as jaosorior16:00
jaosoriorayoung: nice!16:00
slaglegfidente: it's not going to be that simple16:00
*** ohamada_ has quit IRC16:01
slaglegfidente: you'll have to dig into how the vxlan tunnels are configured among the subnodes16:01
jaosoriorjistr: ping16:01
jistrjaosorior: pong16:01
slaglegfidente: and come up with an os-net-config template that configures the tunnels the same way the function does from devstack-gate16:01
slaglegfidente: and we'll also need a step to bootstrap the initial networking via os-net-config so that we can start the polling from heat16:02
jaosoriorjistr: by the way, I'm testing the CA profile stuff in CI here https://review.openstack.org/#/c/357791/316:02
jistrjaosorior: ah cool thanks for the info. Btw not sure if you noticed yet, i did a mini tweak to https://review.openstack.org/#/c/353021/ as i noticed a typo.16:03
jistr+2'd also16:03
jistrthanks for updating that one16:03
gfidenteslagle so the bridge I see in net-config-multinode.yaml I can't just add vlans in there?16:03
jaosoriorjistr: ah damn, where was the typo?16:03
jaosoriorjistr: you mean the lack of .pem?16:04
jistrjaosorior: yea16:04
jaosoriorjistr: thanks for updating that, dude16:04
jistrhehe sure thing16:04
hewbroccadtantsur: do you have outstanding reviews you need for Ironic-in-overcloud etc.?16:04
jistrjaosorior: and btw kudos on your blog posts, just read 2 so far but helpful things, more in the queue after i get unleashed from downstream a bit more16:05
dtantsurhewbrocca, yes, but it's blocked by gate issues, I guess16:05
dtantsurhewbrocca, https://review.openstack.org/35653916:05
hewbroccathat the only one?16:05
jaosoriorjistr: if you find any info on the blog posts you can submit a bug report on github haha. Glad they're useful16:05
dtantsurhewbrocca, we've landed the majority :) there are some smaller improvements here and there, but this is the most critical. also docs https://review.openstack.org/354016, but here I use newer ironicclient commands, so I put it on hold. still worth reviewing.16:07
ayoungjaosorior,  success16:07
hewbroccadtantsur: thank you sir16:07
ayoungOK...now I am up to speed with Newton development....16:07
ayounglet me make sure I can tear down and redeploy16:08
*** pcaruana has joined #tripleo16:09
dsneddonDoes anyone know if the job "gate-tripleo-ci-centos-7-ovb-nonha" is having issues? I have been seeing patches fail with this error message: "Unable to establish connection to"16:09
dsneddonSame failure for 4 or 5 days.16:10
dtantsurfolks, what to do if stack deletion hangs?16:10
dsneddondtantsur, Sometimes you need to restart the Ironic conductor and issue the delete command again.16:12
dtantsurdsneddon, well, it hasn't even started deleting nova instances16:12
dtantsurI've interrupted stack creation in the middle, that's why.. still, I don't think it should just hang16:12
dsneddondtantsur, Yeah, I've had that happen, too. I think it happens when the actual state of the overcloud does not match what Heat expects.16:13
dsneddondtantsur, Sometimes I have had luck by manually deleting nova instances, ports, etc. Sometimes I've even had to go into the DB to delete ports/routers/etc.16:13
tzumainndtantsur, deleting the nova instances from the CLI works for me sometimes, I dunno if there's some reason I shouldn't do that though16:13
dtantsurlemme check16:14
jaosoriorayoung: aawww yeah!16:15
*** jobewan has joined #tripleo16:16
dsneddondtantsur, When I've had to delete Neutron ports from the DB it is because the manual Nova delete didn't work, just FYI.16:16
*** jpich has quit IRC16:16
dtantsurnova delete worked for me fwiw16:16
dtantsurstack is still stuck, I guess it's easier to kill the whole env with fire16:17
ebaldufI'm trying to get a development environment setup.  What am I missing here with TripleO-Quickstart?  I can send it off with ./quickstart.sh -T all tripleo-testing.pm.solidfire.net and get a working version based on the RDO images, but when I send it off with ./quickstart.sh -T all --config config/general_config/devmode.yml --release master-tripleo tripleo-testing.pm.solidfire.net I keep getting dependency problems with the16:18
ebalduf undercloud deployment. I have tried to wipe out all caches and update everything with no luck.  What am I missing here?16:18
*** jaosorior has quit IRC16:18
*** zoliXXL is now known as zoli|gone16:19
*** morazi has joined #tripleo16:19
*** zoli|gone is now known as zoli_gone-proxy16:20
*** numans has quit IRC16:22
*** jbadiapa has quit IRC16:30
*** jubapa has quit IRC16:30
*** florianf has quit IRC16:32
*** jraju has joined #tripleo16:38
*** jraju has quit IRC16:39
*** bana_k has joined #tripleo16:40
*** ioggstream has quit IRC16:40
*** maeca1 has quit IRC16:42
dtantsurhewbrocca, forgot to mention https://review.openstack.org/354125 (in progress by social)16:43
dtantsurit's optional, but we have a few manual steps without it16:43
openstackgerritRichard Su proposed openstack/tripleo-common: Add List Roles Action  https://review.openstack.org/33028316:43
dtantsurnot sure if we'll have enough time to wire it in before feature freeze though16:43
hewbroccaFFEs are going to get very expensive :)16:44
*** nyechiel has quit IRC16:46
dtantsurwell, everything, except for the HAProxy patch, can be done manually, I even have it on my docs patch16:46
slaglesshnaidm: did you end up freeing up some space on the mirror server?16:47
sshnaidmslagle, yes16:47
slaglesshnaidm: thanks16:47
sshnaidmslagle, 13G is available now\16:47
*** yamahata has quit IRC16:48
derekhslagle: once this merges, freeing up space wont be manual https://review.openstack.org/#/c/348806/16:49
*** awayne has quit IRC16:49
hewbroccawe still don't know why CI is slow though, right?16:49
slaglehopefully we'll get a pass tonight and will have some images16:50
hewbroccaother than we keep making it Slower ?16:50
slaglenothing definitive yet16:50
slaglebut yes, we do keep making it slower16:51
slagledprince: i think this is the most frequent erorr happening in nonha: http://logs.openstack.org/53/359153/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/4dd74c5/console.html16:52
dtantsuris there anyone here who knows a thing or two about host aggregates?16:53
dtantsurI'd like to check if https://review.openstack.org/#/c/359240/1/doc/source/advanced_deployment/baremetal_overcloud.rst makes any sense whatsoever16:53
dprinceslagle: looking16:53
gfidentethough I am suspicious about cpu being 3/5 times slower than I see in local test env16:54
gfidenteslagle hewbrocca ^^16:54
ayoungsshnaidm, thanks for your help last night.  Got Overcloud deployment working this morning.16:55
slagledprince: https://bugs.launchpad.net/tripleo/+bug/161614416:55
openstackLaunchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to" [Critical,Triaged]16:55
sshnaidmayoung, great!16:55
sshnaidmayoung, using devmode?16:55
ayoung Next up is to figure out how to add an additional RPM/Apache module prior to brining up the HTTPD server16:55
slaglethere is a connection reset by peer error in the heat api log16:55
ayoungsshnaidm, yeah16:55
*** penick has joined #tripleo16:55
sshnaidmayoung, cool, so it works16:56
ayoungsshnaidm, I was a bit out of date, and had to rebase.16:56
sshnaidmweshay, ^^16:56
sshnaidmayoung, yeah, it's fast changing16:56
weshaysshnaidm, k cool.. I still think we need to evolve devmode.16:57
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient: WIP wire in jinja templating for custom roles  https://review.openstack.org/33662516:57
sshnaidmweshay, maybe to arrange some mtg or ml about it? if removing it, then we need to think what to use instead16:58
ayoungsshnaidm, So, there is a feature we want to use a Heat Artifact to deploy, but it adds in a config file for Apache (something htat Puppet does not know about yet) and I need to have apache run with the module right up front.  Is there some sort of pre-deploy hook I can use to do that?16:58
weshaysshnaidm, agree. thanks for keeping me in the loop w/ some examples16:58
*** derekh has quit IRC16:59
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating  https://review.openstack.org/31567916:59
sshnaidmayoung, sorry.. I don't think I can help you with this, not heat expert yet :)17:00
dprinceslagle: there is a heat-api failure there http://logs.openstack.org/53/359153/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/4dd74c5/logs/undercloud/var/log/heat/heat-api.txt.gz17:00
slagledprince: yea, i added that in the bug17:00
slagledprince: i wonder if that just due to the client closing the connection though?17:00
dprinceslagle: yep, could be unrelated17:01
*** dtantsur is now known as dtantsur|afk17:01
slaglei wonder if it is load related, and the client attempts to connect and times out or something17:01
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install  https://review.openstack.org/35891917:04
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating  https://review.openstack.org/31567917:05
*** anshul has joined #tripleo17:06
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop  https://review.openstack.org/33726717:06
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop  https://review.openstack.org/33758717:07
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: DONT MERGE Testin OOOQ job  https://review.openstack.org/35914617:07
*** fragatina has quit IRC17:08
*** lucasagomes is now known as lucas-afk17:08
*** bana_k has quit IRC17:09
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Create per-role node_names in hieradata for all nodes  https://review.openstack.org/34897317:09
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Create composable mapping between enabled services and role ips  https://review.openstack.org/34897417:09
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Add service_ips in csv format to allNodes hieradata  https://review.openstack.org/35506617:10
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Move ceph cluster config out of allNodesConfig  https://review.openstack.org/35506717:10
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Generate composable service node_names lists  https://review.openstack.org/35506817:10
*** ooolpbot has joined #tripleo17:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to" [Critical,Triaged]17:10
*** abregman|mtg is now known as abregman17:10
ayoungEmilienM, is there a way to tell the Apache Puppet module to add in mod_auth_mellon?17:10
EmilienMayoung: yes, https://github.com/puppetlabs/puppetlabs-apache#installing-apache-modules17:12
*** abregman has quit IRC17:12
ayoungEmilienM, I see in /usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml  there is an apache::mod17:12
ayoung# Horizon17:13
ayoung                apache::mod::remoteip::proxy_ips: {get_input: horizon_subnet}17:13
*** dsariel has quit IRC17:13
ayoungEmilienM, does the puppet module need to know about the specific Apache module?  If Puppet does not know about mod_auth_mellon, is there still a way to get it to install?17:13
ayoungah   apache::mod { 'mod_authn_mellon': }17:15
ayoungSo I want OS::TripleO::ControllerExtraConfigPre17:16
ayoungand that should have17:16
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service  https://review.openstack.org/35350617:17
openstackgerritLars Kellogg-Stedman proposed openstack/puppet-tripleo: updates to fluentd support  https://review.openstack.org/35350717:18
*** jrist has quit IRC17:18
ayoungEmilienM, (or anyone else)  is it possible to add additional information on to: ControllerConfig: properties.config.hiera.controller.mapped_data  without editing /usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml17:19
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-specs: Spec: Centralized logging support  https://review.openstack.org/35669117:20
*** bana_k has joined #tripleo17:24
*** fragatina has joined #tripleo17:24
*** dsariel has joined #tripleo17:27
*** yamahata has joined #tripleo17:28
*** Ryjedo has quit IRC17:29
*** Ryjedo has joined #tripleo17:31
*** mcornea has quit IRC17:32
*** sshnaidm is now known as sshnaidm|afk17:34
*** maeca1 has joined #tripleo17:36
*** jprovazn has quit IRC17:36
*** cdearborn has joined #tripleo17:38
*** milan has quit IRC17:41
*** tosky has quit IRC17:42
*** jcoufal_ has quit IRC17:45
*** jrist has joined #tripleo17:46
*** jmiu_ has quit IRC17:49
*** [1]cdearborn has quit IRC17:52
*** gfidente has quit IRC17:52
*** fragatina has quit IRC17:53
*** fragatina has joined #tripleo17:53
jobewanCurious, should nova->ironic "node:" scheduler hints should work in OSP-7?  I've updated my ironic capabilities and added the hostnamemap, and the hostname mapping is correct - although the ironic capability "node:" doesn't seem to be used in regard to the scheduler_hints .  Any info as to where I can debug that?17:58
tzumainnhi!  I have a dumb question - does tripleo support using glance v2 on the overcloud?18:02
*** paramite has quit IRC18:03
*** akrivoka has quit IRC18:04
bnemecjobewan: The predictable placement/hostname functionality wasn't in OSP 7.18:04
*** jcoufal has joined #tripleo18:06
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test.  https://review.openstack.org/34694918:08
*** ooolpbot has joined #tripleo18:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to" [Critical,Triaged]18:10
jobewanwell, part of it is though right...  So, to get predictive IPs, I added IPPool and NodeIndex to my compute templates, and created the ips-from-poo-all.yaml to use the existing "from_pool" yaml files in the network/ports - which works fine.  The predictive naming is the hiccup really... Due to the str_replace functionality in 2015-04-30, I added the "node: hostname" params directly to the template - and "that" seems to work as well, to a degree...  It's18:12
jobewan just randomly grabbing a hostname and not considering the hints.18:12
*** cdearborn has quit IRC18:12
*** cdearborn has joined #tripleo18:13
jobewanhints in ironic that is18:13
*** hewbrocca is now known as hewbrocca-afk18:13
*** athomas has quit IRC18:14
*** ioggstream has joined #tripleo18:14
*** jcoufal has quit IRC18:17
*** jbadiapa has joined #tripleo18:17
*** jubapa has joined #tripleo18:17
*** bana_k has quit IRC18:19
*** electrofelix has quit IRC18:21
bnemecjobewan: That's because there's a scheduler filter needed in order for the node hints to take effect.  It didn't exist until Liberty (OSP 8).18:25
*** jcoufal has joined #tripleo18:26
jobewanbnemec: yea, that's what I'm looking for really.   Is it simply a matter of enabling a filter in nova.conf, or rather actual coding differences?18:29
*** pkovar has quit IRC18:30
bnemecjobewan: It's new code18:30
*** bana_k has joined #tripleo18:31
jobewanSo, is it safe to assume that I could use the augmented osp7 templates on an osp8 UC and achieve that functionality?18:31
bnemecjobewan: I would not call it "safe", no. :-)18:34
bnemecPossible?  Sure.18:34
bnemecBut at that point, why not just use OSP 8 templates too?18:34
bnemec*Liberty, since this is an upstream channel :-)18:34
jobewanosp8 templates req liberty right?, I can't upgrade from kilo yet18:34
*** bana_k has quit IRC18:36
bnemecWell, backporting Liberty features to Kilo is going to be unsupported anyway, but yes you could theoretically do an older deployment from a newer undercloud.18:38
bnemecIt is a thing we support, but only for the features in the older release.18:38
*** dtantsur|afk has quit IRC18:41
*** mhenkel has quit IRC18:46
*** bana_k has joined #tripleo18:55
*** maeca1 has quit IRC18:59
*** dsariel has quit IRC19:02
*** milan has joined #tripleo19:03
*** penick has quit IRC19:05
*** anshul has quit IRC19:05
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Implement non-ovb overcloud update job - Newton -> Newton  https://review.openstack.org/35133019:06
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement undercloud upgrade job - Mitaka -> Newton  https://review.openstack.org/34699519:06
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton  https://review.openstack.org/32375019:06
*** nyechiel has joined #tripleo19:09
*** ooolpbot has joined #tripleo19:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to" [Critical,Triaged]19:10
*** trozet has quit IRC19:11
*** trozet has joined #tripleo19:12
*** ioggstream has quit IRC19:14
*** jtomasek has joined #tripleo19:18
*** jtomasek has quit IRC19:20
*** jtomasek_ has joined #tripleo19:20
*** pcaruana has quit IRC19:20
*** nyechiel has quit IRC19:24
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Expose enable_combination_alarm setting  https://review.openstack.org/35938719:31
jobewanso, is there a way to specify ctlplane IP addressing per node?19:33
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service  https://review.openstack.org/35350619:34
jobewanerr rather a "preferred method"...19:34
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: ensure the OPT_WORKDIR is set appropriately when executed from any directory  https://review.openstack.org/33364319:35
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (local loop)  https://review.openstack.org/31959119:45
*** dhill_ has quit IRC19:47
*** dhill_ has joined #tripleo19:47
*** dhill_ has quit IRC19:49
*** dhill_ has joined #tripleo19:49
openstackgerritimain proposed openstack/tripleo-heat-templates: WIP: Containerized Services for Composable Roles  https://review.openstack.org/33065919:53
openstackgerritimain proposed openstack/tripleo-heat-templates: Bind mount files to run DiD in latest atomic host  https://review.openstack.org/34721819:53
*** rasca has quit IRC20:02
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Update authtoken parameters to match recent changes  https://review.openstack.org/35940020:08
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Update authtoken parameters to match recent changes  https://review.openstack.org/35940120:09
*** ooolpbot has joined #tripleo20:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to" [Critical,Triaged]20:10
*** anshul has joined #tripleo20:12
EmilienMok tripleo CI just broke20:15
EmilienMI'm working on  afix20:15
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: Pin puppetlabs_spec_helper to 1.1.1  https://review.openstack.org/35940620:19
openstackgerritEmilien Macchi proposed openstack/instack-undercloud: Pin puppetlabs_spec_helper to 1.1.1  https://review.openstack.org/35940720:20
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Pin puppetlabs_spec_helper to 1.1.1  https://review.openstack.org/35940820:21
EmilienMdprince, slagle: we'll need that ^ asap20:21
*** penick has joined #tripleo20:25
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Ceph cluster health validation on upgrade  https://review.openstack.org/35941020:27
*** [1]cdearborn has joined #tripleo20:29
*** NobodyCam has quit IRC20:29
*** NobodyCam has joined #tripleo20:30
*** rhefner has quit IRC20:31
bnemecEmilienM: Is this going to be a problem going forward?  Are we going to start seeing more and more puppet modules requiring newer Ruby?20:31
*** coolsvap has quit IRC20:31
EmilienMbnemec: it has been a problem since we have lint syntax and unit test jobs20:32
EmilienMfwiw, most of the problems are also affecting puppet CI so we detect & fix them early20:32
*** rhefner has joined #tripleo20:33
bnemecEmilienM: I guess I'm just wondering if pinning is a sustainable fix.  Do we need to look into getting newer Ruby into CentOS so we can continue moving forward with the puppet modules?20:35
EmilienMbnemec: it's clearly a bug in the gem20:35
EmilienMbnemec: puppet community is working on it20:35
EmilienMbut yeah having ruby 2.2 would be awesome20:35
bnemecEmilienM: Oh, so this is a temporary thing.20:35
EmilienMbnemec: what would be the process if we want to update ruby on centos?20:36
EmilienMI guess we need to contact the maintainer?20:36
bnemecEmilienM: I have no idea.  I don't even know if we can bump a version like that.  I suppose it's possible there's a Ruby SIG that maintains newer versions.20:37
EmilienMbnemec: I'll look into it20:37
EmilienMbnemec: thx for the +2, it helps!20:39
*** cdearborn has quit IRC20:42
*** rhefner has quit IRC20:43
*** ioggstream has joined #tripleo20:43
*** oneswig has joined #tripleo20:44
*** rhefner has joined #tripleo20:48
*** jcoufal has quit IRC21:03
*** nyechiel has joined #tripleo21:05
*** ooolpbot has joined #tripleo21:10
openstackLaunchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to" [Critical,Triaged]21:10
*** ooolpbot has quit IRC21:10
*** julim has quit IRC21:12
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Add fluentd client service  https://review.openstack.org/35350621:13
*** lblanchard has quit IRC21:13
*** bfournie has quit IRC21:21
*** pradk has quit IRC21:24
*** ioggstream has quit IRC21:24
*** dprince has quit IRC21:26
*** nyechiel has quit IRC21:28
*** mburned is now known as mburned_out21:35
*** trozet has quit IRC21:36
*** asbishop has left #tripleo21:36
*** anshul has quit IRC21:40
*** Goneri has quit IRC21:43
*** egafford has quit IRC21:53
*** mburned_out is now known as mburned21:54
*** oneswig has quit IRC22:08
*** ooolpbot has joined #tripleo22:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to" [Critical,Triaged]22:10
openstackLaunchpad bug 1616219 in tripleo "latest puppetlabs_spec_helper now requires ruby 2.2" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)22:10
*** ayoung has quit IRC22:27
*** ccamacho has quit IRC22:42
*** morazi has quit IRC22:44
*** trozet has joined #tripleo22:45
*** yamahata has quit IRC22:47
*** mburned is now known as mburned_out22:54
*** chlong has quit IRC23:03
*** yamahata has joined #tripleo23:09
*** ooolpbot has joined #tripleo23:10
openstackLaunchpad bug 1616144 in tripleo "tripleo-ci: nonha jobs failing with Unable to establish connection to" [Critical,Triaged]23:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1616219 in tripleo "latest puppetlabs_spec_helper now requires ruby 2.2" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)23:10
*** rhallisey has quit IRC23:11
*** padkrish has joined #tripleo23:14
*** mpsairam has joined #tripleo23:15
*** jobewan has quit IRC23:18
*** saneax-_-|AFK is now known as saneax23:22
openstackgerritMerged openstack/python-tripleoclient: Fix unbracket_ipv6()  https://review.openstack.org/35917423:26
openstackgerritMerged openstack-infra/tripleo-ci: Remve old cache files on the mirror server  https://review.openstack.org/34880623:26
*** mlupton has joined #tripleo23:26
openstackgerritMerged openstack/tripleo-common: Add Plan Deletion  https://review.openstack.org/29993723:26
mluptonI had a failed undercloud installation and I'm trying to completely remove RabbitMQ so I can start fresh, how can I go about doing this?23:29
mluptonI'm running on RHEL and have tried doing yum remove rabbitmq, yum remove rabbitmq-server, but everything seems like it's still there23:30
openstackgerritMerged openstack/instack-undercloud: Pin puppetlabs_spec_helper to 1.1.1  https://review.openstack.org/35940723:32
openstackgerritMerged openstack/tripleo-heat-templates: Pin puppetlabs_spec_helper to 1.1.1  https://review.openstack.org/35940823:39
openstackgerritMerged openstack/puppet-tripleo: Pin puppetlabs_spec_helper to 1.1.1  https://review.openstack.org/35940623:40
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Use cached images  https://review.openstack.org/35948123:40
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/34869523:42
*** mburned_out is now known as mburned23:43
*** padkrish has quit IRC23:46
*** padkrish has joined #tripleo23:47
openstackgerritMerged openstack/tripleo-heat-templates: Set cinder::api::bind_host within service template  https://review.openstack.org/31834423:53
*** padkrish has quit IRC23:53
*** rlandy has quit IRC23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!