Tuesday, 2016-11-08

*** fragatin_ has quit IRC00:05
*** dciabrin has joined #tripleo00:06
*** fragatina has joined #tripleo00:08
*** Viswanath has joined #tripleo00:10
*** Viswanath has quit IRC00:16
*** jefrite has joined #tripleo00:19
openstackgerritAlex Schultz proposed openstack/puppet-tripleo: Standardize httpchk for http services  https://review.openstack.org/39468600:23
*** numans has quit IRC00:38
*** limao has joined #tripleo00:57
*** fultonj has quit IRC01:09
openstackgerritMerged openstack/python-tripleoclient: Pass clients to get the get_password function  https://review.openstack.org/39457301:14
openstackgerritMerged openstack/tripleo-common: Power off new nodes when making them available, not right after enrolling  https://review.openstack.org/39454101:17
openstackgerritMerged openstack/python-tripleoclient: Fix handling of missing environment files  https://review.openstack.org/39463001:19
*** ayoung has joined #tripleo01:20
*** mhenkel has quit IRC01:22
*** saneax is now known as saneax-_-|AFK01:25
*** kjw3 has quit IRC01:26
*** mhenkel has joined #tripleo01:38
*** kjw3 has joined #tripleo01:38
*** dmacpher has quit IRC01:47
*** thrash is now known as thrash|g0ne01:52
*** ayoung has quit IRC02:02
*** fzdarsky|afk has quit IRC02:13
*** tiswanso has joined #tripleo02:17
*** tiswanso has quit IRC02:17
*** tiswanso has joined #tripleo02:18
*** cwolferh has quit IRC02:19
*** michapma_alt has joined #tripleo02:23
*** maticue has quit IRC02:32
*** bana_k has quit IRC02:32
*** tzumainn has quit IRC02:40
*** yamahata has joined #tripleo02:50
*** fragatina has quit IRC02:50
*** fragatina has joined #tripleo02:55
*** fragatina has quit IRC02:56
*** mhenkel has quit IRC02:56
*** dmacpher has joined #tripleo02:56
*** bana_k has joined #tripleo02:59
*** dmacpher has quit IRC03:07
*** cwolferh has joined #tripleo03:17
*** bana_k has quit IRC03:25
*** kjw3 has quit IRC03:30
*** ayoung has joined #tripleo03:37
*** cmyster has quit IRC03:46
*** cmyster has joined #tripleo03:47
*** cmyster has quit IRC03:47
*** cmyster has joined #tripleo03:47
*** limao has quit IRC03:48
*** ayoung has quit IRC03:54
*** ayoung has joined #tripleo03:56
*** ebarrera has quit IRC03:57
*** links has joined #tripleo04:00
*** noslzzp has quit IRC04:02
*** noslzzp has joined #tripleo04:03
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore  https://review.openstack.org/36061804:06
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore  https://review.openstack.org/36544904:06
*** ayoung has quit IRC04:06
openstackgerritBrent Eagles proposed openstack/puppet-tripleo: WIP: Call VF configuration from udev rules  https://review.openstack.org/39460404:09
*** fzdarsky|afk has joined #tripleo04:11
*** ccamacho has quit IRC04:13
*** ccamacho has joined #tripleo04:13
*** sudipto_ has joined #tripleo04:17
*** sudipto has joined #tripleo04:17
*** sudipto has quit IRC04:21
*** sudipto_ has quit IRC04:21
*** chandankumar has joined #tripleo04:34
*** dmacpher has joined #tripleo04:34
*** tiswanso has quit IRC04:35
*** cwolferh has quit IRC04:39
openstackgerritBrent Eagles proposed openstack/puppet-tripleo: Add proper handling of IPv6 addresses for rabbit host/port handling  https://review.openstack.org/39474204:39
*** sudipto has joined #tripleo04:59
*** sudipto_ has joined #tripleo05:00
*** limao has joined #tripleo05:05
*** bana_k has joined #tripleo05:16
*** dsavineau has quit IRC05:17
*** dsavineau has joined #tripleo05:18
openstackgerritBrent Eagles proposed openstack/puppet-tripleo: Add proper handling of IPv6 addresses for rabbit host/port handling  https://review.openstack.org/39474205:21
*** masco has joined #tripleo05:27
cmysterramishra: speaking of which, did you ever try to install with net-multiple-nics by any chance?05:28
*** ramishra has quit IRC05:30
*** ramishra has joined #tripleo05:31
*** pmannidi has quit IRC05:38
*** bana_k has quit IRC05:40
*** numans has joined #tripleo05:42
*** pmannidi has joined #tripleo05:42
*** fragatina has joined #tripleo05:43
*** fragatina has quit IRC05:47
*** rcernin has joined #tripleo05:54
*** limao has quit IRC06:00
*** limao has joined #tripleo06:02
*** abregman has joined #tripleo06:06
*** cwolferh has joined #tripleo06:08
*** saneax-_-|AFK is now known as saneax06:19
*** jaosorior has joined #tripleo06:20
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job  https://review.openstack.org/39173806:30
*** jaosorior has quit IRC06:32
*** jaosorior has joined #tripleo06:33
dtrainorhowdy, jaosorior.  how are you feeling?06:35
jaosoriordtrainor: still flu-ish but better than yesterday06:35
jaosoriordtrainor: how's it going over there?06:35
dtrainorahh, that's no good.  glad you're starting to get over it though.06:36
dtrainorOooh, you know.  Breaking code, making problems for people.  The usual.06:36
jaosoriordtrainor: haha such is life in software06:36
jaosoriordtrainor: looking into the bug you sent me yesterday06:36
jaosoriorbluejeans?06:36
dtrainorworks for me06:37
*** anshul has joined #tripleo06:37
jaosoriordtrainor: PMed you the link06:38
dtrainori'm in06:38
*** saneax is now known as saneax-_-|AFK06:45
*** pcaruana has joined #tripleo06:53
*** anshul has quit IRC06:55
*** anshul has joined #tripleo06:55
*** ealcaniz has joined #tripleo06:57
*** saneax-_-|AFK is now known as saneax06:59
*** florianf has joined #tripleo07:00
*** tesseract has joined #tripleo07:03
*** oshvartz has joined #tripleo07:03
*** tesseract is now known as Guest657007:03
*** david-lyle has quit IRC07:03
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Include local CA in haproxy PEM  https://review.openstack.org/39479307:06
openstackgerritThomas Herve proposed openstack/python-tripleoclient: Use environmen fixture in tests  https://review.openstack.org/39479407:09
openstackgerritThomas Herve proposed openstack/python-tripleoclient: Use environment fixture in tests  https://review.openstack.org/39479407:09
*** iranzo has joined #tripleo07:09
*** iranzo has joined #tripleo07:09
*** dsariel has joined #tripleo07:13
*** dciabrin has quit IRC07:15
*** oshvartz has quit IRC07:15
*** oshvartz has joined #tripleo07:16
*** sudipto has quit IRC07:16
*** sudipto_ has quit IRC07:16
*** yamahata has quit IRC07:18
jaosoriormarios: hey dude, if you have time could you check these two? https://review.openstack.org/#/c/377645/  https://review.openstack.org/#/c/377650/07:21
mariosjaosorior: added to list07:22
*** ealcaniz__ has joined #tripleo07:23
*** ebalduf has quit IRC07:23
*** ealcaniz has quit IRC07:24
*** dmacpher has quit IRC07:28
*** yamahata has joined #tripleo07:32
*** milan has joined #tripleo07:36
*** lmiccini has quit IRC07:39
*** lmiccini has joined #tripleo07:41
*** dsariel has quit IRC07:44
*** hjensas has joined #tripleo07:47
*** hjensas has joined #tripleo07:47
*** jprovazn has joined #tripleo07:47
openstackgerritNoam Angel proposed openstack/diskimage-builder: add option to configure cloud-init to allow password authentication  https://review.openstack.org/39176507:48
*** panda|zZ is now known as panda|ko07:50
jaosoriorpanda|ko: hey dude, how are you doing?07:52
*** numans has quit IRC07:53
panda|kojaosorior: I feel a lot better after vomiting everything of the past 24h, but can't eat, and can't sleep07:54
panda|kojaosorior: you ?07:54
*** jlinkes has joined #tripleo07:55
jaosoriorpanda|ko: well, no fever for me today, so it's all good. Was gonna ping you about work, but you should probably rest dude, hope you get better soon.07:55
panda|komaybe I can start to reason again in the afternoon07:56
jaosoriorpanda|ko: no worries man, health comes first :)07:56
panda|koanyway, the most outstanding bug for CI is https://bugs.launchpad.net/tripleo/+bug/1639970, seems real, can't find anything strange about the processes, no particular performance issue07:58
openstackLaunchpad bug 1639970 in tripleo "CI: cinder fails to allocate memory while creating volume for ping test tenant" [Critical,Confirmed]07:58
*** tremble has joined #tripleo07:58
*** abregman has quit IRC07:59
jaosoriorpanda|ko: is it so that httpd is taking too much memory then?08:00
*** shardy has joined #tripleo08:01
panda|kojaosorior: nothing is taking too much memory, but I think that's just one of the easy places from were we can scrub forks08:03
*** abregman has joined #tripleo08:03
jaosoriorpanda|ko: I see. Thanks for the info08:03
*** numans has joined #tripleo08:07
*** chem has quit IRC08:12
*** chem has joined #tripleo08:12
*** zoli|gone is now known as zoli08:16
*** zoli is now known as zoliXXL08:16
*** bana_k has joined #tripleo08:16
*** paramite has joined #tripleo08:22
*** abregman has quit IRC08:27
*** dsariel has joined #tripleo08:27
*** ebarrera has joined #tripleo08:27
*** links has quit IRC08:29
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Reload haproxy configuration as a post-deployment step  https://review.openstack.org/39364408:29
*** cylopez has joined #tripleo08:30
ccamachomorning!08:31
*** aufi has joined #tripleo08:32
jaosoriorccamacho: sup dude08:33
ccamachojaosorior, man the haproxy reload indeed wasnt executed at all... :P08:34
*** jpich has joined #tripleo08:34
jaosoriorccamacho: yep. But I think this is a bigger issue, cause this means that pacemaker is not being restarted, did you talk to jistr or bandini about this?08:35
*** abregman has joined #tripleo08:36
ccamachoyeap yesterday night, I added the post restart to the post.j2 (Was removed) and now testing the deployment08:36
*** abregman is now known as abregman|mtg08:36
ccamachojistr gave feedback there08:36
jaosoriorccamacho: excellent08:37
ccamachoI want to make it work and then to ask for some reviews..08:37
*** fzdarsky|afk is now known as fzdarsky08:38
*** jpena|off is now known as jpena08:38
jaosoriorccamacho: let me know when you get it working, I can review it.08:39
*** dciabrin has joined #tripleo08:39
openstackgerritMartin Mágr proposed openstack/python-tripleoclient: Use correct region value  https://review.openstack.org/39451508:39
jaosoriorccamacho: by the way, got time for a quick review? https://review.openstack.org/#/c/392473/08:39
jaosoriorshadower: ^^08:39
*** amoralej|off is now known as amoralej08:39
jaosoriorshadower: nevermind, you had reviewed it already, sorry for the spam08:40
ccamachojistr around? you were right there, I added  ControllerPostPuppetRestart to post.j2.yaml and now testing, Ill add the other missing bits once this starts to work.08:40
*** bana_k has quit IRC08:41
*** mcornea has joined #tripleo08:43
shadowerjaosorior: no problem, but it's still got my +1 from earlier08:43
*** links has joined #tripleo08:44
shadowerd0ugal: can you think of anyone online who could review the ssh keys tripleo-common patch?08:44
shadowerhttps://review.openstack.org/#/c/391093/08:44
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates: Fix typo in Keystone Sensu subscription  https://review.openstack.org/39482408:52
*** rasca has joined #tripleo08:53
*** sudipto has joined #tripleo08:54
*** sudipto_ has joined #tripleo08:54
*** athomas has joined #tripleo08:56
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates: Fix typo in Keystone Sensu subscription  https://review.openstack.org/39482408:57
*** numans has quit IRC09:02
skramajashardy: hello. is there any plan for the THT repo to host multiple roles_data.yaml file.. for example, for DPDK, an environement file already exist, similary a roles_data.yaml will ease the usage for composable role..09:02
*** abregman_ has joined #tripleo09:08
*** anton has quit IRC09:09
*** links has quit IRC09:10
d0ugalshadower: good question :/09:11
*** abregman|mtg has quit IRC09:11
d0ugalshadower: What eu tripleo cores do we have?09:11
Ngfwiw https://review.openstack.org/#/admin/groups/190,members is the list of cores09:12
*** anton has joined #tripleo09:12
NgI have it open because I am hunting for +2s on https://review.openstack.org/#/c/393673 ;)09:12
*** ohamada has joined #tripleo09:13
*** lucas-afk is now known as lucasagomes09:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Deploy heat APIs over httpd  https://review.openstack.org/39483709:16
social/win 1909:17
d0ugalshadower: shardy maybe?09:19
*** dciabrin has quit IRC09:22
*** links has joined #tripleo09:23
*** dciabrin has joined #tripleo09:23
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Specify tripleo::haproxy for keepalived dependency.  https://review.openstack.org/39484409:25
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates: Use default Sensu redact  https://review.openstack.org/39484709:26
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Update git commit sources for Midonet element  https://review.openstack.org/39484809:27
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates: Use default Sensu redact  https://review.openstack.org/39484709:27
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart: Install gating repo when you have undercloud  https://review.openstack.org/39484909:30
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Installs more packages with the Midonet element  https://review.openstack.org/39485109:32
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Separate Datastax repository from the Midonet one  https://review.openstack.org/39485209:32
*** b00tcat has quit IRC09:35
*** cylopez has quit IRC09:37
jaosoriord0ugal, shadower question about https://review.openstack.org/#/c/391093/4/workbooks/validations.yaml wouldn't one need to specify that unset_validation_key_parameter should on-success call set_validation_key_parameter? It would then establish that set_validation_key_parameter depends on both get_pubkey and unset_validation_key_parameter right?09:38
d0ugaljaosorior: I think unset is only called if there is an error, see line 19209:39
d0ugaljaosorior: oh09:40
d0ugaljaosorior: I see what you mean, sorry09:40
d0ugaljaosorior: I'd guess that unset_validation_key_parameter should call get_pubkey on-success but I'm not sure now09:41
d0ugalshadower: ^09:42
jaosoriord0ugal: it's not specified anywhere on the workflow. So unset_validation_key_parameter might be called in the beginning (as far as I can understand), but I'm not sure if we can actually asure this, since we now depend on mistral's scheduler09:42
*** b00tcat has joined #tripleo09:43
*** b00tcat has joined #tripleo09:43
jaosoriord0ugal: ah damn09:43
jaosoriornow I see09:43
jaosoriorI didn't see the on-error09:43
jaosoriorwell that makes more sense now haha, had missed that line09:44
d0ugal:)09:45
*** hewbrocca_afk is now known as hewbrocca09:45
d0ugalbut if the on-error is called, then unset_validation_key_parameter is called - but then what?09:45
jaosoriord0ugal: nothing09:45
jaosoriord0ugal: that's the end of it.09:45
d0ugalso the workflow is like a toggle on/off?09:45
d0ugalit seems like the name is confusing because it doesn't always add09:45
d0ugalbut I guess that is just cleaning up09:46
d0ugalso that's fine09:46
d0ugalI think I am over thinking it now :)09:46
* d0ugal drinks more coffee09:46
jaosoriord0ugal: well, if validations aren't enabled, then it will just remove that parameter. Which is a bit unexpected cause of the workflow name09:46
jaosoriorbut I guess it's fine for a cleanup09:47
d0ugalyeah, but fine I think09:47
d0ugalagreed09:47
shadoweryeah that's it09:47
jaosoriord0ugal: is it cool if I merge it?09:47
shadowersorry, was afk for a bit09:47
d0ugaljaosorior: sure09:47
jaosoriorshadower: +Aed09:47
d0ugalshadower: back just in time for the good bit :)09:47
shadowerhaha yay!09:48
shadowerthakns09:48
shadowerd0ugal: basically, when I did a deploy with validations enabled, then deleted the stack, then disabled the validations and then did a new deploy, it would set up the key. Which was unexpected I think09:48
jaosoriorccamacho: could you review https://review.openstack.org/#/c/388495/ ?09:48
*** derekh has joined #tripleo09:49
ccamachojaosorior, sure man, just let me finish some work, Ill add it to my queue09:49
d0ugalshadower: makes sense, thanks09:50
*** lmiccini has quit IRC09:50
Ngccamacho: could I add https://review.openstack.org/#/c/393673 to your queue please? :)09:50
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Reload haproxy configuration as a post-deployment step  https://review.openstack.org/39364409:50
ccamachoNg sure man09:50
Ngthanks :D09:50
*** pcaruana has quit IRC09:51
jaosoriorNg: why are the base member options removed https://review.openstack.org/#/c/393673/1/manifests/haproxy.pp ?09:51
*** lmiccini has joined #tripleo09:51
Ngjaosorior: they clash with the settings I'm proposing, the union() would produce an inconsistent set, e.g. with both "inter 2000" and "inter 1s"09:53
Ngjaosorior: and the only one of the defaults that remain, is "check"09:53
*** ohamada has quit IRC09:53
*** ohamada has joined #tripleo09:54
*** cylopez has joined #tripleo09:54
*** ohamada has quit IRC09:54
*** ohamada has joined #tripleo09:55
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/38995709:55
jaosoriorNg: commented on the patch09:55
jistrccamacho: sorry i started going through e-mail and forgot to log into IRC :D gonna look at the haproxy patch update right now09:55
ccamachojistr no worries man, once spotting the issue its way easier to fix it. :) thanks!09:57
*** limao has quit IRC09:57
openstackgerritMerged openstack/tripleo-ui: Use mistral action to create new containers  https://review.openstack.org/39218909:59
*** limao has joined #tripleo09:59
openstackgerritJulie Pichon proposed openstack/puppet-tripleo: Enable communication between UI and the Undercloud by making HAProxy proxy for the UI  https://review.openstack.org/39339010:01
jaosoriorjpich: pinged you in the wrong channel.10:07
jpichOk :)10:07
jaosoriorjpich: anyway, the policy is that we need an LP bug for backports :/10:07
jpichjaosorior: The patch is small and whether this is a RFE or a bug can be argued, but yeah there is only a bp open for this at the moment...10:08
jaosoriorjpich: can you link to the LP bug # or bp in the commit message?10:08
jpichjaosorior: The name's in there, just not linked properly. I guess I can fix that part of the commit message?10:09
jaosoriorjpich: sure10:09
* jpich not sure how much moving things around is acceptable with backports :)10:09
*** milan has quit IRC10:10
openstackgerritJulie Pichon proposed openstack/puppet-tripleo: Enable communication between UI and the Undercloud by making HAProxy proxy for the UI  https://review.openstack.org/39339010:10
openstackgerritMarios Andreou proposed openstack/puppet-tripleo: Specify tripleo::haproxy for keepalived dependency.  https://review.openstack.org/39484410:10
jaosoriormarios, chem what was the issue with the previous fix? In the end, we do use the class ::haproxy, it's just inside ::tripleo::haproxy, but the dependency should have been fine anyway10:12
*** Jokke_ has quit IRC10:12
mariosjaosorior: seems like chem hit an issue testing the fix this morning hence the new patch10:13
chemjaosorior: it doesn't appears in the manifest, we use define in haproxy, but not class (or not during the upgrade)10:13
jistrccamacho: i wonder if we should bring back the PrePuppet / PostPuppet resources rather than just the restart...10:13
chemjaosorior: not the class directly, I mean10:13
Ngjaosorior: replied to your comment :)10:13
jistrccamacho: if we do what's proposed, we'll still probably have a problem between puppet and pacemaker whenever config changes10:13
chemjaosorior: all in all, the if(defined(class[::haproxy])) was false10:14
chemjaosorior: working version was if(defined(class[haproxy]))10:14
chemjaosorior: notice the relative path10:14
mariosjaosorior: yeah i see what you mean, tripleo::haproxy is a wrapper for config/running haproxy but still it seems there was a dependency issue as the path wasn't resolved10:15
jistrccamacho: it used to be that these were the important resources that we set for the pacemaker scenario, because they handle both restarts and pacemaker maintenance mode https://github.com/openstack/tripleo-heat-templates/blob/master/environments/puppet-pacemaker.yaml#L5-L610:15
jistrccamacho: the 3rd entry in resource registry there, the actual restart here https://github.com/openstack/tripleo-heat-templates/blob/master/environments/puppet-pacemaker.yaml#L7 is actually referenced from the 2nd entry's template https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/tasks/post_puppet_pacemaker.yaml#L3310:16
chemjaosorior: well, I see too, the class is indeed loaded in tripleo::haproxy.  I'll log some paste later10:16
jistrIOW10:17
jistrif we do what we have now, we get the restarts but not maintenance mode10:17
jaosoriorchem, marios https://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy.pp#L67210:17
ccamachojistr, I see.. Currently the ControllerPostPuppetRestart its being executed, let me add the other resources and then run some tests10:17
mariosjaosorior: ack10:18
jaosoriorjistr: well, for pacemaker cases we should be doing systemctl haproxy reload in case some configurations change.10:18
jistrccamacho: ok. If you add PrePuppet and PostPuppet, you can remove PostPuppetRestart from post.j2.yaml, because PostPuppetRestart will get executed automatically from PostPuppet10:18
ccamachojistr ack10:18
*** sudipto_ has quit IRC10:19
*** sudipto_ has joined #tripleo10:19
jaosoriorjistr, ccamacho shouldn't reloading the configuration be taken care of by https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/haproxy.pp#L91 ?10:19
jistrjaosorior: right, i just wonder what will happen with e.g. cinder-volume. Previously we had issues that when puppet tried to restart services which were under pacemaker's control, it failed, and the whole stack-update failed because of that too... That's why we added the mainenance mode. We removed majority of the services out of pacemaker, but a few A/P ones still remain.10:20
jaosoriorjistr: yeah, that might be an issue with the restart. Which is why I advocated for using that reload instead.10:21
*** gfidente has joined #tripleo10:21
*** gfidente has quit IRC10:21
*** gfidente has joined #tripleo10:21
jistryea definitely for services which support reload it's better to reload. I don't think e.g. cinder-volume or manila-share support that though...10:21
chemjaosorior: anyway, this is not working currently, looking into it...10:21
jaosoriorjistr: well, crap10:22
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Deploy heat APIs over httpd  https://review.openstack.org/39483710:24
*** nyechiel has joined #tripleo10:24
jpichjaosorior: Thanks a lot for looking into https://review.openstack.org/#/c/394793/ - let me know if there's any way I can help, with testing the patch or something else10:24
hewbroccapanda|ko, bandini were you guys able to get redis/ceilo/swift straightened out yesterday10:25
jaosoriorjpich: well, if you can apply that patch to your newton deployment and test it out it would be great help. Once we make sure that works, then I still need to make changes to instack-undercloud to make renewals work too.10:26
jpichjaosorior: Ok, great! I wasn't sure if things were testable yet without the instack changes based on your comment. I'll give it a try and report back on the review10:27
bandinihewbrocca: not me no. I got pulled into an upgrade escalation and have not looked at anything else since :/10:28
*** dtantsur|afk is now known as dtantsur10:31
*** sshnaidm|away is now known as sshnaidm10:32
*** tosky has joined #tripleo10:34
*** limao has quit IRC10:34
jaosoriorwhat's the right procedure if someone wants to update a network driver in "bm-deploy-ramdisk" and "overcloud-full-initrd" ?10:35
jaosoriordtrainor: do you happen to know? ^^10:36
jaosoriorI mean dtantsur ^^10:36
dtantsur?10:36
dtantsurelaborate on "update a network driver" please10:36
openstackgerritMerged openstack/tripleo-common: Fix the validation ssh keys workflow  https://review.openstack.org/39109310:37
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Enable MidoNet element to install MEM  https://review.openstack.org/39488910:37
jaosoriordtantsur: the nvme0n1 driver that comes in the kernel for those images seems to be too old and is causing people trouble10:37
*** akrivoka has joined #tripleo10:38
dtantsurjaosorior, well, it's a question to the distribution used, right? ;) otherwise, if you have a newer RPM you can try injecting it into DIB, but I don't know the new image building procedure any well, ask thrash|g0ne10:39
*** ealcaniz__ has quit IRC10:40
*** numans has joined #tripleo10:40
jaosoriordtantsur: thanks.10:40
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/38995710:43
openstackgerritChris Jones proposed openstack/puppet-tripleo: Improve failed mysql node removal time in HA deploys.  https://review.openstack.org/39367310:45
openstackgerritTomas Sedovic proposed openstack/tripleo-common: Fix the validation ssh keys workflow  https://review.openstack.org/39297110:46
shadowerd0ugal: that's a backport to stable/newton ^10:47
*** milan has joined #tripleo10:50
Ngjaosorior: would you be able to have another look? (393673) - I think I've addressed your request :)10:50
*** mgould|afk is now known as mgould10:52
d0ugalshadower: thanks10:52
jaosoriorNg: will do10:53
Ngthanks!10:53
*** ealcaniz__ has joined #tripleo10:55
*** zoliXXL is now known as zoli|brb10:58
jaosoriorNg: done.11:03
Ngjaosorior: thanks. I'll reply on the review for the sake of history, but my thinking with the default setup like that was to not disturb the current defaults where $mysql_clustercheck is false, since that didn't have things like on-marked-down shutdown-sessions before.11:05
*** cylopez has quit IRC11:06
Ngjaosorior: (if clustercheck is false, it doesn't really matter what is set, because haproxy isn't going to take any action anyway, afaics)11:06
dtantsurmorning! could you please approve two more easy bug fixes with passed CI: https://review.openstack.org/#/c/392653/ and https://review.openstack.org/#/c/392181/11:09
dtantsurthanks!11:09
shardyskramaja: Hi, sorry missed your ping earlier11:13
*** jokke_ has joined #tripleo11:13
shardyre multiple roles_data - yes we could add them to tht, but my main concern is we may end up with a lot of duplication11:13
shardye.g if we have 10 roles_data.yaml examples then we'll end up with inconsistencies and/or will have to update every file whenever a new service is added11:14
shardyis there any reason we can't just document the services which need to be added in tripleo-docs?11:14
shardythat would be my preference I think, but open to other ideas11:14
arxcruzquestion, if a service is in ServicesDefault in roles_data.yaml on tripleo-heat-templates, and the same service is in overcloud-resource-registry-puppet set as OS::Heat::None it means, this service will not be installed / executed right ?11:14
shardyarxcruz: yes, exactly11:14
shardyarxcruz: in the future we plan to remove the mappings to OS::Heat::None because there's a new heat feature (environment merging) which means we can assemble the list from fragments merged together11:15
shardybut right now we "stub out" disabled resources with the None resource11:15
shardyhttps://github.com/openstack/tripleo-heat-templates/blob/master/environments/hyperconverged-ceph.yaml#L1011:16
shardyarxcruz: example of the new pattern we expect to adopt in due course11:17
shardywhich will hopefully be a little cleaner and less confusing11:17
arxcruzshardy: so, is there a reason for by default   OS::TripleO::Services::CinderBackup: is set to OS::Heat::None in overcloud-resource-registry-puppet ?11:17
shardyat some point someone decided it was a non-essential service so we disabled it by default - I can't provide the rationale for it tbh11:18
arxcruzjust wondering, because some tempest tests fails because cinder-backup isn't started11:18
skramajashardy: documentation is good enough to deploy it.. adding it part of tht will have simpler steps for documentation.. thats all the reason i was thinking about it. as you said, adding a new service and modified in all roles is a pain..11:18
openstackgerrityolanda.robla proposed openstack/python-tripleoclient: WIP: Support full disk images in TripleO  https://review.openstack.org/39442611:18
shardyarxcruz: yeah, it's a balance because we're resource constrained in CI, so in some cases we've turned stuff off to save resources11:18
arxcruzshardy: I see11:19
shardyhttps://review.openstack.org/#/c/357729/11:19
shardythat's the commit which disabled it - seems it was primarily to align with the previous releases behavior11:20
shardyyou can pass -e environments/cinder-backup.yaml to enable it11:20
shardyarxcruz: we could look at enabling it for one of the CI test scenarios11:20
shardyhttps://github.com/openstack-infra/tripleo-ci#service-testing-matrix11:21
*** fzdarsky is now known as fzdarsky|lunch11:21
openstackgerritMerged openstack/diskimage-builder: dhcp-all-interfaces: support Centos/RHEL 6  https://review.openstack.org/39255511:21
arxcruzshardy: that would be nice, I'm in an effort to run full tempest in ooo and I have these backup tests in my plate to make it pass :)11:22
arxcruzshardy: what would be the easy way to enable this service? I mean, as a parameter11:22
*** saneax is now known as saneax-_-|AFK11:23
*** zoli|brb is now known as zoli11:23
shardyarxcruz: you can just include that environment file in the test where you want to run tempest11:23
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Pass X-Forwarded-Proto for missing services  https://review.openstack.org/39491011:23
shardyI assume that's the periodic job, so you'd add it here:11:23
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Reload haproxy configuration as a post-deployment step  https://review.openstack.org/39364411:23
*** dprince has joined #tripleo11:24
jaosoriormcornea: if you have time, can you verify the bug with this patch? https://review.openstack.org/394910 I had forgotten about that.11:24
shardyhttps://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L19811:24
jaosoriorsshnaidm: hey, are you around?11:24
shardyarxcruz: I think you can add an OVERCLOUD_DEPLOY_ARGS similar to other tests?11:24
sshnaidmjaosorior, mostly :)11:25
mcorneajaosorior: yep, adding it to the list11:25
openstackgerrityolanda.robla proposed openstack/python-tripleoclient: WIP: Support full disk images in TripleO  https://review.openstack.org/39442611:25
shardyor perhaps the tempest entry, I'm not that familiar with how the periodic jobs are wired up11:25
jaosoriormcornea: thanks dude11:25
*** zoli is now known as zoli|lunch11:25
jaosoriorsshnaidm: so... regarding the TLS-everywhere job I was working on https://review.openstack.org/#/c/391738/ I'm currently blocked cause of the default range we choose. FreeIPA will refuse to deploy using that range, since it's not valid. So I need to get a different range set up (which I was meaning to do for a long time), do you have some time to help me out with that?11:26
shardyd0ugal: congrats on the nomination to mistral core! :)11:27
d0ugalshardy: Thanks :)11:28
jaosoriord0ugal: hey! I didn't know, congrats dude :D\11:28
d0ugaljaosorior: just this morning, thanks \o/11:28
openstackgerrityolanda.robla proposed openstack/python-tripleoclient: Support full disk images in TripleO  https://review.openstack.org/39442611:29
Ngjaosorior: thanks! much appreciated :)11:30
jaosoriorNg: no problem, thanks for the patience11:32
sshnaidmjaosorior, yeah, in 20 mins, ok?11:32
*** jkilpatr has quit IRC11:32
jaosoriorsshnaidm: sure, no worries11:33
arxcruzshardy: hmmm do you have an example?11:34
shardyarxcruz: https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L18511:34
shardyit would be e.g -e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml in the OVERCLOUD_DEPLOY_ARGS11:35
arxcruzshardy: awesome, thanks :D11:39
*** adarazs is now known as adarazs_lunch11:41
owalshshardy: skramaja: I thinking about duplication in roles_data & roles too. Would jinja2 inheritance help in this case?11:42
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Use --globoff when downloading artifacts  https://review.openstack.org/39491411:42
shardyowalsh: I'm not sure it would, because we use the roles_data.yaml to do the j2 templating11:42
shardyowalsh: I have been thinking we should provide a way to override the services only for one role though11:42
shardyso I'm sure there are ways we can make this easier in due course11:43
shardye.g allow a user to provide only a subset of roles_data.yaml (an additional or overridden role)11:43
gfidenteshardy, could that be just the merge_strategies option ?11:45
*** sudipto has quit IRC11:45
*** sudipto_ has quit IRC11:45
shardyI suppose j2 inheritance might be one way to implement that, but it seems fairly heavyweight compared to just merging/appending the lists in the mistral action11:45
shardygfidente: we could do something similar, but it's not a file that's parsed by heat, so we'll need to implement something (either in tripleoclient or ideally tripleo-common mistral actions)11:46
d0ugalshardy: if you have a minute, I've got a question related to mistral/jinja2 etc. :)11:48
shardyd0ugal: sure11:48
d0ugalshardy: cool, I actually planned to ask you at summit but forgot. At the moment (AFAIK) we don't store the jinja2 templates, just the built templates.11:49
shardyd0ugal: in swift we store both11:49
d0ugalshardy: I wonder if we should store the source, because otherwise there isn't a good way to export your templates without loosing that flexability11:49
d0ugaloh, we do!11:49
d0ugalThen, I need to look at this again.11:49
shardyIIRC we simply write the rendered foo.yaml back to the swift container11:50
shardywe don't remove the source foo.j2.yaml11:50
d0ugaloh, I see.11:50
d0ugalThat sounds correct.11:50
shardymkdir tmp && pushd tmp && swift download overcloud should confirm :)11:50
d0ugalIndeed, looking now11:50
d0ugalshardy: would you expect a "plan export" to include both?11:51
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable internal TLS for Cinder API  https://review.openstack.org/39209511:51
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add preload wsgi script option for cinder  https://review.openstack.org/39379011:51
shardyd0ugal: it could, or just the .j2.yaml11:52
shardyI'd say just export both, even though the rendered file will be overwritten on any import of the plan11:52
d0ugalYeah, that is certainly easier - and might be useful for debugging11:52
jaosoriorAny +As for these? https://review.openstack.org/#/c/388503/ https://review.openstack.org/#/c/388495/ :D11:52
*** ebarrera has quit IRC11:55
openstackgerrityolanda.robla proposed openstack/python-tripleoclient: WIP: Support full disk images in TripleO  https://review.openstack.org/39442611:56
jpichjaosorior: What kind of changes should I look for after applying https://review.openstack.org/#/c/394793/ ? My results are not looking very good so far but it's possible I applied the patch wrong, is there a way I can double-check?11:57
jaosoriorjpich: right, so that is supposed to include the self-signed CA cert in the PEM file served by haproxy, with the intent that it will properly hint the browser which is the certificate that it should query the user to trust (or not)11:58
jaosoriorjpich: what are you seeing?11:59
*** jpena is now known as jpena|lunch11:59
d0ugalshardy: You are of course correct, I can confirm those files are there :)11:59
*** ebarrera has joined #tripleo12:00
jaosoriorjpich: first off, you could check the contents of /etc/pki/ca-trust/source/anchors/cm-local-ca.pem12:00
*** amoralej is now known as amoralej|lunch12:00
jaosorioruuuh12:01
jaosoriord'oh12:01
jaosoriorI mean12:01
jpichjaosorior: I'm getting SEC_ERROR_EXTENSION_VALUE_INVALID at the moment. Looking at /etc/pki/ca-trust/source/anchors/cm-local-ca.pem I did not notice much difference with the version before applying the patch (only one cert). The cert at /etc/pki/tls/certs/undercloud-192.0.2.2.pem seems to have more in it, but I don't have the version from before to compare12:02
jaosoriorjpich: /etc/pki/tls/certs/undercloud-$UNDERCLOUD_VIP.pem12:03
*** Vijayendra has quit IRC12:03
jaosoriorjpich: right, so /etc/pki/tls/certs/undercloud-192.0.2.2.pem includes the extra CA cert then?12:04
jpichjaosorior: There are 2 "CERTIFICATE" stanzas in it, yes12:05
jaosoriorjpich: do you see the same issue using another browser?12:06
*** paramite has quit IRC12:08
jpichjaosorior: A NET::ERR_CERT_AUTHORITY_INVALID from Chrome, though it's easier to accept the exception than Firefox12:08
jpichjaosorior: Is this the correct order for the certs to be in, in the undercloud-vip.pem? http://paste.openstack.org/show/588362/12:09
jaosoriorjpich: verifying12:10
jaosoriorjpich: don't find much info about the ordering here http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#5.1-crt12:12
* jpich looking12:13
openstackgerritJames Slagle proposed openstack/python-tripleoclient: Don't assume CountDefault set in roles_data.yaml  https://review.openstack.org/39492412:16
slaglemcornea: there is the patch ^12:17
mcorneaslagle: thanks, will try it on my env12:18
openstackgerritJames Slagle proposed openstack/python-tripleoclient: Don't assume CountDefault set in roles_data.yaml  https://review.openstack.org/39492412:19
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Use --globoff when downloading artifacts  https://review.openstack.org/39491412:19
*** maticue has joined #tripleo12:19
sshnaidmjaosorior, which network range doesn't IPA like?12:20
jaosoriorsshnaidm: 192.0.2.X12:20
jaosoriorsshnaidm: that's cause it's meant only for documentation and we shouldn't have been using in the first place. But yeah, can't deploy FreeIPA on an IP on such a range :/12:21
sshnaidmjaosorior, but there are other networks there, internal, storage etc12:21
sshnaidmjaosorior, could it work in 172.16.x.x?12:21
jaosoriorsshnaidm: sure it would12:22
jaosoriorsshnaidm: however, those are not yet created at the time I'm deploying FreeIPA12:22
*** thrash|g0ne is now known as thrash12:23
*** zoli|lunch is now known as zoli12:23
*** zoli is now known as zoliXXL12:23
sshnaidmjaosorior, iirc there was a proposition to remove this docs range and switch to regular one, maybe there is even patch for it..12:23
jaosoriorsshnaidm: I was the one that had the patch on it, however, it was too messy to do so at the time and people told me to hold it off12:24
sshnaidmjaosorior, oh, right, it was your patch :)12:24
jaosoriorsshnaidm: so.. that patch is quite outdated, which is why I ended up asking for help12:24
hewbroccasshnaidm: hey, did you guys get the ceilometer thing sorted yesterday12:25
sshnaidmjaosorior, I see12:25
*** rodrigods has quit IRC12:25
*** rodrigods has joined #tripleo12:25
*** rhallisey has joined #tripleo12:26
*** dmacpher has joined #tripleo12:26
sshnaidmhewbrocca, it seemed like ceilometer problem wasn't the main cause for our problem, but it's gonna be solved anyway in the patch12:26
*** paramite has joined #tripleo12:27
sshnaidmhewbrocca, actually it was: https://review.openstack.org/#/c/394548/12:27
openstackgerritMerged openstack/python-tripleoclient: Do not try to "provide" updated nodes not in "manageable" state  https://review.openstack.org/39265312:27
hewbroccaoh, wow12:28
hewbroccawell, that's good12:28
sshnaidmhewbrocca, the redis problem still exists and requires additional investigation, it's tracked in https://bugs.launchpad.net/tripleo/+bug/163988512:28
openstackLaunchpad bug 1639885 in tripleo "CI: pingtest timeouts cause by performance issues (redis, swift, ceiliometer)" [High,Triaged]12:28
openstackgerritDmitry Tantsur proposed openstack/python-tripleoclient: Do not try to "provide" updated nodes not in "manageable" state  https://review.openstack.org/39493012:29
sshnaidmhewbrocca, another problem was found, with memory outage: https://bugs.launchpad.net/tripleo/+bug/163997012:29
openstackLaunchpad bug 1639970 in tripleo "CI: cinder fails to allocate memory while creating volume for ping test tenant" [Critical,Confirmed]12:29
sshnaidmjaosorior, let me check something..12:29
mcorneaslagle: it worked on my env12:33
*** dtantsur is now known as dtantsur|brb12:34
*** david-lyle has joined #tripleo12:37
*** pkovar has joined #tripleo12:37
*** adarazs_lunch is now known as adarazs12:39
jpichjaosorior: I have to go for an appointment, but I'll keep poking at this after lunch12:44
jaosoriorjpich: my deployment is pretty messed up at the moment, will dig better into this when I get to fix it12:45
sshnaidmjaosorior, do you have dev env setup with your patch?12:45
jaosoriorsshnaidm: not at the moment.12:45
*** egafford has quit IRC12:46
sshnaidmjaosorior, ok, I'll set it up.. I'd like to see it live12:46
jaosoriorsshnaidm: see what live?12:46
jaosoriorsshnaidm: the error I'm talking about?12:47
sshnaidmjaosorior,  all networks setup12:47
jpichjaosorior: That sounds great, thank you12:47
jaosoriorsshnaidm: ah, right, so as I said, the network range change patch is quite outdated, so I'm not even sure if it's working anymore. probably it isn't.12:47
jaosoriorsshnaidm: I did manage to change the defaults for quickstart, but that's it.12:48
jaosoriorsshnaidm: I was trying it out around May :/12:48
*** lucasagomes is now known as lucas-hungry12:49
*** bfournie has quit IRC12:51
*** fzdarsky|lunch is now known as fzdarsky12:52
sshnaidmjaosorior, are you sure this error is related to 192.0.0.x subnet?12:53
jaosoriorsshnaidm: it's a documented issue, yes12:53
sshnaidmjaosorior, maybe it means not resolved to host?12:53
jaosoriorsshnaidm: that is actually the whole reason why I started the discussion on changing that range12:54
jaosoriorsshnaidm: cause FreeIPA (due to it strictly conforming to standards) refuses to use that range at all12:54
*** rhallisey has quit IRC12:54
openstackgerritAna Krivokapic proposed openstack/tripleo-ui: Allow user to specify FQDN instead of IP address when registering nodes  https://review.openstack.org/39494712:55
jaosoriorsshnaidm: http://logs.openstack.org/38/391738/29/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/0a3cc30/console.html#_2016-11-08_07_17_38_36847312:55
jaosoriorsshnaidm: this is the entry it sets http://logs.openstack.org/38/391738/29/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-fakeha-caserver/0a3cc30/console.html#_2016-11-08_07_17_38_36750912:56
*** maticue has quit IRC12:56
weshayjaosorior, when you have a sec can you point me at a review or email that goes into a little more depth re: the ip range and FreeIPA12:56
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Specify tripleo::haproxy for keepalived dependency.  https://review.openstack.org/39484412:57
jaosoriorweshay: could try to fetch it, at some point ayoung tried to make FreeIPA less strict about the ranges it accepts (unsuccessfuly). But that mails is from the beginning of this year12:57
weshayjaosorior, k.. from you?12:57
jaosoriorweshay: sent12:58
weshaythanks man12:58
jaosoriorweshay: at least I sent you the last reply from it12:58
*** masco has quit IRC13:00
*** fultonj has joined #tripleo13:00
*** trown|outtypewww is now known as trown13:01
*** sudipto_ has joined #tripleo13:02
*** sudipto has joined #tripleo13:03
*** zoliXXL is now known as zoli|brb13:03
sshnaidmjaosorior, which network your ipaserver connects to?13:03
*** pkovar has quit IRC13:04
jaosoriorsshnaidm: ctlplane, which is the only network existing at the time I launch that stack.13:04
sshnaidmjaosorior, I think there is provision network too, isn't it?13:05
*** pradk has joined #tripleo13:05
*** jayg|g0n3 is now known as jayg13:05
sshnaidmjaosorior, ok, I'll set up dev env for this patch, it'll be much easier to see there13:05
jaosoriorsshnaidm: thought the ctlplane and provision network had the same range in CI13:06
adarazsshardy: hello Steve! Can you please ack adding the tripleo-quickstart-extras repo please? https://review.openstack.org/39120913:06
*** rhallisey has joined #tripleo13:07
openstackgerritMerged openstack/tripleo-specs: Single process Heat driven Undercloud installer  https://review.openstack.org/35211313:07
sshnaidmjaosorior, you're right, I meant private network with 192.168.100.x range13:08
shardyadarazs: done!13:08
adarazsshardy: thank you! \o/13:08
*** abregman_ is now known as abregman|mtg13:08
weshay:)13:09
*** jkilpatr has joined #tripleo13:12
*** rhallisey has quit IRC13:13
*** rhallisey has joined #tripleo13:13
*** pkovar has joined #tripleo13:17
trownshardy++ thanks13:17
sshnaidmshardy, can you please ack this patch too? :) https://review.openstack.org/#/c/394394/13:20
sshnaidmweshay, ^^13:21
*** bfournie has joined #tripleo13:22
*** jpena|lunch is now known as jpena13:22
jaosoriorshardy: hey dude, if you get some time could you help me debug this? https://review.openstack.org/#/c/394837/ trying to get heat to run over httpd. However, it seems to time out on creating the nova server resources. The cfn httpd access logs seem quite suspicious though13:23
jaosoriorhttp://logs.openstack.org/37/394837/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/c77957e/logs/undercloud/var/log/httpd/heat_api_cfn_wsgi_access.txt.gz since it seems all actions result in a 404 error :/13:23
*** rlandy has joined #tripleo13:23
*** abregman|mtg is now known as abregman13:24
*** jcoufal has joined #tripleo13:24
hewbroccasshnaidm: so, cool -- what about the memory issue, then?13:27
shardyjaosorior: not sure tbh - is heat-api-cfn even starting?  I don't see /var/log/heat/heat-api-cfn.log but I'm not sure if that's just because of the httpd thing13:27
hewbroccasshnaidm: do we know how to fix it? Have we fixed it?13:28
shardyit'll be kind of inconvenient if the normal API log files suddenly dissappear13:28
*** david-lyle has quit IRC13:29
*** fultonj_ has joined #tripleo13:29
*** fultonj_ has quit IRC13:29
jaosoriorshardy: interesting, let me verify... it might be that I messed up the wsgi script path for cfn13:29
shardyI see the httpd access/error log for the service, but it doesn't appear to contain the same information as I'd normally expect13:30
shardyI'm also not sure how the ec2token paste middleware works in this case?13:30
shardyhonestly, I've never run heat set up like this, but folks in #heat may have some ideas13:30
sshnaidmhewbrocca, I don't know yet. If it continue to happen, maybe it's a time to increase memory for overcloud nodes too, now they have 6GB13:30
hewbroccaheh13:32
hewbroccathe memory vacuum revs up13:32
shardy:(13:32
*** maticue has joined #tripleo13:32
shardywhy don't we figure out why it's using more memory instead?13:32
hewbroccathat's an excellent idea13:32
*** links has quit IRC13:33
*** amoralej|lunch is now known as amoralej13:34
openstackgerritJames Slagle proposed openstack/python-tripleoclient: Don't assume CountDefault set in roles_data.yaml  https://review.openstack.org/39492413:35
openstackgerritSven Anderson proposed openstack/puppet-tripleo: Implement Nova ec2api profile  https://review.openstack.org/33656213:36
*** zoli|brb is now known as zoli13:39
shadowerd0ugal: the ssh keys backport passed CI. Can I have a +A please? https://review.openstack.org/#/c/392971/13:39
d0ugalshadower: done13:39
shadowerd0ugal: thanks!13:40
panda|koshardy: hewbrocca sshnaidm, yesterday I spent some times searching for memory eaters. There is no outstanding process that is taking a lot of memory. Mysqld is using ~6%, then httpd is at ~3% then a long list of 0-2%. unless I'm missing something, this seems real. Not ewasy to understand if something is taking more memory than it should, or there are more processes than we need. Unfortunately today I'm13:41
panda|kounable to work.13:42
hewbroccablearrgh13:42
hewbroccapanda|ko: go back to resting...13:42
hewbroccabut -- good information13:42
openstackgerritTomas Sedovic proposed openstack/tripleo-specs: Allow specifying multiple sources of validations  https://review.openstack.org/39377513:42
*** lblanchard has joined #tripleo13:45
fultonjwould someone please review https://review.openstack.org/#/c/394442/ ?13:46
*** rbowen has quit IRC13:46
openstackgerritkrogon-intel proposed openstack/tripleo-heat-templates: Temporary UCSM mapping files should be opened with write mode  https://review.openstack.org/39059113:47
*** morazi has joined #tripleo13:48
*** openstackgerrit has quit IRC13:48
*** openstackgerrit has joined #tripleo13:48
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Enable MidoNet element to install MEM  https://review.openstack.org/39488913:48
*** pkovar has quit IRC13:49
*** jkilpatr has quit IRC13:51
*** tzumainn has joined #tripleo13:55
weshaypanda|ko, ya.. thanks man.. sleep please :)13:57
*** fragatina has joined #tripleo13:57
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates: Reload haproxy configuration as a post-deployment step  https://review.openstack.org/39364413:58
shardyTripleo meeting in 2mins in #openstack-meeting-alt13:58
shardyhttps://wiki.openstack.org/wiki/Meetings/TripleO13:58
shardyhttps://etherpad.openstack.org/p/tripleo-meeting-items13:59
*** lucas-hungry is now known as lucasagomes13:59
*** pkovar has joined #tripleo13:59
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: ceilometer compute agent needs restart on compute upgrade  https://review.openstack.org/39496814:00
slagledprince: can you review https://review.openstack.org/#/c/394914/14:00
*** rbowen has joined #tripleo14:01
dprinceslagle: looking14:01
*** fragatina has quit IRC14:02
*** cdearborn has joined #tripleo14:02
dprinceslagle: +214:02
slaglethx14:02
*** ebalduf has joined #tripleo14:03
*** egafford has joined #tripleo14:04
*** jkilpatr has joined #tripleo14:05
hewbroccashardy: so does the info from Panda above mean that CI is at least mostly functioning14:06
hewbroccacaveat, our overcloud nodes sometimes seem to run out of memory?14:06
shardyhewbrocca: I think so, we need to figure out if specific scenarios are triggering the OOM or if some other regression is causing that14:07
*** egafford has quit IRC14:07
beaglesbnemec: I fired off a patch for the ipv6 rabbit hosts thing, but don't have a current IPv6 environment to test.14:07
beaglesbnemec: is there a "check expermimental" or something I can do?14:08
bnemecbeagles: I already did. :-)14:08
beaglesbnemec: awesome, thanks!14:08
bnemecBut yes, there are a couple of experimental jobs that use ipv6 now.14:08
hewbroccaWell, at least we can land stuff, I guess14:08
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Update git commit sources for Midonet element  https://review.openstack.org/39497514:08
hewbroccashardy: excellent14:08
* beagles needs to buy a physical IPv6 book - a heavy one - to smack himself with whenever he breaks something IPv6'ish14:09
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Installs more packages with the Midonet element  https://review.openstack.org/39497614:09
*** egafford has joined #tripleo14:09
bnemecbeagles: I don't even understand what happened here.  A month ago it was working, and now it's broken for every service.14:09
bnemecDid we change how we are generating the rabbit_node_ips or something?14:09
* bnemec shrugs14:10
beaglesbnemec: previously ports were not being factored into the the host list... I fixed that, but didn't take into account IPv6 addresses14:10
beaglesbnemec: so yeah, I broke it14:10
bnemecbeagles: Ah, okay.  So there was a reason for this.  That makes me feel better. :-)14:10
bnemecWe need to get an ipv6 job back in the gate for this stuff.14:11
jaosoriorshardy: is CFN on the undercloud terminated by haproxy?14:11
*** tiswanso has joined #tripleo14:14
*** Goneri has joined #tripleo14:15
openstackgerritAlejandro Andreu proposed openstack/tripleo-puppet-elements: Separate Datastax repository from the Midonet one  https://review.openstack.org/39497814:15
*** egafford has quit IRC14:16
*** egafford has joined #tripleo14:16
*** rbowen has quit IRC14:17
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Mitaka-Newton upgrade fix network disruption during convergence  https://review.openstack.org/39498014:17
gfidenteany core on https://review.openstack.org/#/c/392558/ ?14:17
gfidenteand https://review.openstack.org/#/c/391892/ ?14:17
*** cylopez has joined #tripleo14:18
*** rbowen has joined #tripleo14:19
*** jkilpatr has quit IRC14:21
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002  https://review.openstack.org/36502614:22
*** jayg is now known as jayg|g0n314:23
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Mistral to scenario003  https://review.openstack.org/36880514:23
chemEmilienM: you're on vacation right ?14:23
jtomasekEmilienM: do all the working items of a spec be finished in ocata? I mean, there is a chance that not all dependencies will be in place for all the work described in https://review.openstack.org/#/c/393365/114:24
jtomasekEmilienM: should I remove those?14:24
hewbroccaYes EmilienM is on PTO14:25
hewbroccahe is unlikely to answer you :)14:25
*** jayg|g0n3 is now known as jayg14:25
cmysteris anyone here useing net-multiple-nics by any chance ?14:25
chemhewbrocca: ack, thanks14:25
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Use --globoff when downloading artifacts  https://review.openstack.org/39498714:26
*** dtantsur|brb is now known as dtantsur14:27
*** jaosorior has quit IRC14:27
jtomasekoh, I should have realized that, sorry14:29
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: ceilometer compute agent needs restart on compute upgrade  https://review.openstack.org/39496814:29
mariosgfidente: revote please i wiped your +2 ^^14:30
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add SNMP role to the CephStorage nodes  https://review.openstack.org/39499014:32
*** abregman is now known as abregman|brb14:32
*** weshay is now known as weshay_vote14:34
*** jkilpatr has joined #tripleo14:35
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add SNMP role to the CephStorage nodes  https://review.openstack.org/39499014:35
openstackgerritMerged openstack/tripleo-common: Fix the validation ssh keys workflow  https://review.openstack.org/39297114:37
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Do not reference CephBase from CephExternal service  https://review.openstack.org/39499314:37
*** absubram has joined #tripleo14:39
*** absubram_ has joined #tripleo14:41
*** jayg is now known as jayg|g0n314:42
openstackgerritMerged openstack/tripleo-heat-templates: Move per role Services defaults into environment file  https://review.openstack.org/39444214:43
*** absubram has quit IRC14:44
*** absubram_ is now known as absubram14:44
*** abregman|brb is now known as abregman14:45
*** kjw3 has joined #tripleo14:46
*** iranzo has quit IRC14:48
*** maticue has quit IRC14:48
*** kaslcrof has joined #tripleo14:49
openstackgerrityolanda.robla proposed openstack/python-tripleoclient: Support full disk images in TripleO  https://review.openstack.org/39442614:49
yolandahi dtantsur , i addressed your comments on https://review.openstack.org/#/c/394426/, thanks for review14:50
yolandai also added some unit testing14:50
*** gfidente has quit IRC14:50
dtantsurcool! I'll try to get back to it today14:51
yolandadtantsur, i see release notes for python-tripleoclient created, but not really used. Is there a will to start using those?14:51
dtantsuryolanda, I'm not the best person to ask, but I'm always for using release notes for everything14:52
*** maticue has joined #tripleo14:52
yolandayep, i'm used to. But when i was going to add it, i saw this shall be the first reno for that project14:53
bnemecYeah, we need to get reno into all things tripleo.14:54
*** maticue_ has joined #tripleo14:55
*** fultonj has quit IRC14:56
*** maticue has quit IRC14:56
*** penick has joined #tripleo14:57
openstackgerrityolanda.robla proposed openstack/python-tripleoclient: Support full disk images in TripleO  https://review.openstack.org/39442614:58
owalshskramaja: we appear to have some common requirements for DPDK and realtime compute nodes14:58
owalshskramaja: tuned, huge pages, CPU pinning etc...14:58
* skramaja listening owalsh14:58
*** penick has quit IRC14:58
owalshskramaja: would be good to coordinate to avoid duplicating the effort14:59
*** gfidente has joined #tripleo14:59
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Change the qemu-img bindep to be qemu-img-ev  https://review.openstack.org/39500414:59
skramajaowalsh: good. that would be great.14:59
thrashtrown: ^^^ might fix the images ci14:59
skramajaowalsh: FYI - raised a BP for tuned - https://blueprints.launchpad.net/tripleo/+spec/tuned-nfv-dpdk14:59
trownthrash: nice, that is the first I have seen that bindep.txt :)15:00
thrashtrown: tbh... me too. :D15:00
skramajai am going to create a docs WIP patch for steps to install DPDK.. i will forward it once uploaded..15:01
shardyWhat do folks think about merging https://review.openstack.org/#/c/394378/15:01
openstackgerritJulie Pichon proposed openstack/puppet-tripleo: Persist the UI config file  https://review.openstack.org/38859215:01
shardyit's alleged to help with bug #1638908 but I've not been able to confirm it15:02
openstackbug 1638908 in tripleo-quickstart "Overcloud deployment fails in minimal configuration with ('Connection aborted.', BadStatusLine("''",))" [Undecided,In progress] https://launchpad.net/bugs/1638908 - Assigned to Alfredo Moralejo (amoralej)15:02
*** oshvartz has quit IRC15:02
shardylanding it to find out should be safe tho I think15:02
thrashtrown: this http://logs.openstack.org/30/389830/1/check/gate-tripleo-buildimage-overcloud-full-centos-7-nv/974e8fa/console.html#_2016-11-08_13_58_39_584135 and this http://logs.openstack.org/30/389830/1/check/gate-tripleo-buildimage-overcloud-full-centos-7-nv/974e8fa/console.html#_2016-11-08_13_58_47_898319 tipped me off...15:02
*** jayg|g0n3 is now known as jayg15:03
thrashtrown: bindep could probably be improved to take into account obsoletions (is that a word??)15:03
owalshskramaja: great, thanks. I'm working out the details for a realtime compute spec if you're interested - https://review.openstack.org/#/c/388162/15:04
skramajathanks owalsh. will go through.15:04
*** snecklifter has joined #tripleo15:05
jpichtzumainn: Sorry I didn't follow up yesterday, were you able to find out about the parameters you needed? In *theory* the absolutely-required-ones should be set up by tripleo-common...15:05
snecklifterI have two sets of compute nodes with different nic naming and the abstraction doesn't work, how is this handled in tripleo?15:06
tzumainnjpich, no problem at all!  I'm currently re-testing, I think I may have mutilated my nova configuration somewhere down the line so I decided to start fresh15:07
tzumainnthanks for checking!15:07
openstackgerritJiri Tomasek proposed openstack/tripleo-specs: GUI Deployment Configuration improvements  https://review.openstack.org/39336515:07
jpichtzumainn: Cool! I think if you hit issues, saving your "mistral environment-get overcloud" output would be helpful for debugging15:08
trownthrash: ah right, that is pretty odd, packaging successfully handles the obsoletes and gets the right package, but then bindep is specificially fixated on getting qemu-img15:08
bnemecshardy: Agreed.  2 minutes seems excessively long for any OpenStack request, but it shouldn't hurt anything to try it.15:08
jpichtzumainn: Good luck!15:08
*** weshay_vote is now known as weshay15:08
tzumainnjpich, lol, thanks!15:08
trownthrash: I think your change should fix it though15:08
openstackgerritMerged openstack/tripleo-heat-templates: Add SNMP role to the CephStorage nodes  https://review.openstack.org/39189215:08
snecklifterusing Mitaka, in case anyone can answer the above15:08
*** jprovazn has quit IRC15:09
jtomasekshardy: how to proceed with the blueprints that will need to be created based on spec work items? Is it ok for those to be created after ocata blueprints/specs deadline?15:09
shardyjtomasek: I think if possible then create them now and reference them from the spec, e.g if the work items are known15:10
thrashtrown: I'll depends-on the image patch so it picks it up.15:10
jtomasekshardy: ok15:10
trownthrash++ will be very nice to get the YAML based image building into tripleoclient15:11
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Switch overcloud build images to use ImageBuildManager  https://review.openstack.org/38553015:12
thrashtrown: better yet, I rebased on top15:12
*** liverpooler has joined #tripleo15:12
Nghttps://review.openstack.org/#/c/393673/ got green CI and a +2. Can I tempt anyone to consider a +A? :)15:15
*** kjw3 has quit IRC15:16
*** kjw3 has joined #tripleo15:17
*** tosky has quit IRC15:18
openstackgerritMerged openstack/tripleo-heat-templates: Use --globoff when downloading artifacts  https://review.openstack.org/39491415:19
Ngbnemec: thanks :D15:21
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Use --globoff when downloading artifacts  https://review.openstack.org/39498715:21
*** absubram has quit IRC15:22
bnemecnp15:24
thrashtrown: it definitely fixes the job :)15:27
trownnice15:27
*** yamahata has quit IRC15:28
*** fultonj has joined #tripleo15:28
openstackgerritMerged openstack/tripleo-heat-templates: Do not reference CephBase from CephExternal service  https://review.openstack.org/39255815:29
*** michapma_alt has quit IRC15:29
*** kaslcrof has quit IRC15:29
openstackgerritMichael Glaser proposed openstack/puppet-tripleo: Wrong default in docstring  https://review.openstack.org/39504015:35
openstackgerritMichael Glaser proposed openstack/puppet-tripleo: Wrong default in docstring  https://review.openstack.org/39504015:39
*** numans has quit IRC15:41
dtantsur(sorry, forgot about the meeting), if you plan on a newton release tomorrow, could you please merge this backport also: https://review.openstack.org/#/c/394930/ ?15:48
*** snecklifter has quit IRC15:48
*** florianf has quit IRC15:49
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Ensure keepalived is restarted when necessary.  https://review.openstack.org/39505315:50
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: ceilometer compute agent needs restart on compute upgrade  https://review.openstack.org/39496815:52
*** ayoung has joined #tripleo15:52
openstackgerritHonza Pokorny proposed openstack/tripleo-ui: Redirect user to login page when token expires  https://review.openstack.org/39258915:53
*** ealcaniz__ has quit IRC15:56
slaglecan i get another review on https://review.openstack.org/#/c/394924/ ?15:58
slagletrown: bnemec ^ perhaps15:58
trownlooking15:58
slaglesorry, you get bugged b/c i see you revewing recently :)15:58
trown:)15:58
openstackgerritMerged openstack/puppet-tripleo: Increase haproxy timeouts  https://review.openstack.org/39437816:00
*** Guest6570 has quit IRC16:01
*** florianf has joined #tripleo16:04
openstackgerritDmitry Tantsur proposed openstack/tripleo-common: Remove unused bits from node management code in utils/node.py  https://review.openstack.org/39506016:04
openstackgerritAlex Schultz proposed openstack/puppet-pacemaker: Prep 0.4.0 tag  https://review.openstack.org/39506116:05
*** jlinkes has quit IRC16:08
*** tiswanso has quit IRC16:08
*** oshvartz has joined #tripleo16:10
openstackgerritMerged openstack/puppet-tripleo: Improve failed mysql node removal time in HA deploys.  https://review.openstack.org/39367316:12
*** ebarrera has quit IRC16:13
*** ramishra has quit IRC16:17
*** ohamada has quit IRC16:17
*** tremble has quit IRC16:17
*** ohamada_ has joined #tripleo16:18
*** ramishra has joined #tripleo16:19
openstackgerritMerged openstack/puppet-tripleo: Enable TLS in the internal network for gnocchi  https://review.openstack.org/37764516:19
*** rcernin has quit IRC16:20
*** tiswanso has joined #tripleo16:21
openstackgerritMerged openstack/tripleo-heat-templates: Enable internal TLS for gnocchi  https://review.openstack.org/37765016:22
thrashtrown: ultimately, this should have the full image build with yaml https://review.openstack.org/#/c/387405/16:25
thrashtrown: so, if that job passes, we should be good16:25
dmsimardslagle: hey, the multinode gate job stuff you're working on with EmilienM, it's not running on rh1/rh2, right ? It's pure openstack-infra gate ?16:26
trowndmsimard: ya, it is nodepool16:27
openstackgerritMerged openstack/puppet-pacemaker: Prep 0.4.0 tag  https://review.openstack.org/39506116:28
*** zoli is now known as zoli|brb16:29
*** b00tcat has quit IRC16:32
thrashneed one more review on https://review.openstack.org/#/c/389830/16:32
thrashthen needs to be backported to stable/newton for https://bugs.launchpad.net/tripleo/+bug/163530116:33
openstackLaunchpad bug 1635301 in tripleo "openstack overcloud update stack does not update plan witch changed THT" [Wishlist,In progress] - Assigned to Brad P. Crochet (brad-9)16:33
*** aufi has quit IRC16:33
ayoungany plan to release a new stable mitaka anytime soon?16:33
openstackgerritDmitry Tantsur proposed openstack/tripleo-common: Remove unused bits from node management code in utils/node.py  https://review.openstack.org/39506016:33
*** thrash is now known as thrash|biab16:33
d0ugaltzumainn: How did the deploy go?16:35
tzumainnd0ugal, going better this time - last time it failed to find a valid host, this time it moved past that and hasn't failed yet!16:36
tzumainnd0ugal, I forgot that in a previous iteration of the code I had modified some nova config that probably screwed up the nova scheduling this time around16:37
d0ugaltzumainn: aha, cool, glad you are moving forward at least :)16:37
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Containerized Services for Composable Roles  https://review.openstack.org/33065916:37
d0ugaltzumainn: I would be very interested in feedback as you are the first "external" user of it as an API.16:37
tzumainnd0ugal, omg so honored16:38
d0ugalhaha16:38
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Ensure keepalived is restarted when necessary.  https://review.openstack.org/39505316:38
tzumainnd0ugal, but in all seriousness, the workflows are making things much easier for me, so thanks to you guys for implementing them!16:39
d0ugaltzumainn: you are welcome, do let us know where and how we can make it even easier16:40
tzumainnd0ugal, sure, can it clean my toilet too?16:40
d0ugaltzumainn: If it part of the IoT, yes.16:41
*** mcornea has quit IRC16:41
tzumainn\o/16:41
tzumainnI don't know what an IoT is  :(16:41
d0ugaltzumainn: https://en.wikipedia.org/wiki/Internet_of_things16:41
d0ugalanyway...16:42
tzumainnlol16:42
*** kjw3 has quit IRC16:42
*** nyechiel has quit IRC16:45
*** abregman has quit IRC16:45
openstackgerritMerged openstack/python-tripleoclient: Do not try to "provide" updated nodes not in "manageable" state  https://review.openstack.org/39493016:46
*** anshul has quit IRC16:47
*** yamahata has joined #tripleo16:48
*** yamahata has quit IRC16:53
*** rcernin has joined #tripleo16:56
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Better way to ensure keepalived before haproxy.  https://review.openstack.org/39484416:57
*** kjw3 has joined #tripleo16:57
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Ensure keepalived is restarted when necessary.  https://review.openstack.org/39505316:57
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Ensure keepalived is restarted when necessary.  https://review.openstack.org/39505317:00
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Format the nodes list in openestack overcloud delete node  https://review.openstack.org/39508317:01
d0ugallol, "# TODO(someone)" - https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/tests/v1/overcloud_node/test_overcloud_node.py#L5317:02
*** anshul has joined #tripleo17:02
*** bana_k has joined #tripleo17:04
*** paramite has quit IRC17:04
bnemecNice try, but git blame is always watching. ;-)17:06
d0ugalhaha, indeed - I looked at that too :)17:07
*** zoli|brb is now known as zoli17:07
tzumainnd0ugal, overcloud created successfully \o/17:07
d0ugaltzumainn: Great!17:08
tzumainnd0ugal, now I just need to test it with a non-default plan, but if the default plan works17:08
tzumainnI can be sure that any failure is my fault17:08
*** fultonj has quit IRC17:11
*** zoli is now known as zoli|gone17:12
*** lmiccini has quit IRC17:15
*** ohamada_ has quit IRC17:15
*** lmiccini has joined #tripleo17:15
*** dtantsur is now known as dtantsur|afk17:16
*** anshul has quit IRC17:20
*** paramite has joined #tripleo17:20
*** thrash|biab is now known as thrash17:28
thrashslagle: going to go forward with https://review.openstack.org/#/c/389830/17:28
*** rcernin has quit IRC17:29
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart: Add support of deploy on a VM w/o nested virt  https://review.openstack.org/39372017:30
*** sudipto has quit IRC17:30
*** sudipto_ has quit IRC17:30
*** snecklifter has joined #tripleo17:32
*** snecklifter has quit IRC17:32
slaglethrash: ok, thx for the update17:33
*** bana_k has quit IRC17:33
*** bogdando has joined #tripleo17:33
bogdandofolks, please take a look this minor UX improvement for quickstart https://review.openstack.org/#/c/39372017:33
*** jpich has quit IRC17:34
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Better way to ensure keepalived before haproxy.  https://review.openstack.org/39484417:36
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Ensure keepalived is restarted when necessary.  https://review.openstack.org/39505317:36
bogdandowould like to know if failures of the "Get undercloud vm ip address" task related to the "localhost" case I'm trying...17:37
*** hewbrocca is now known as hewbrocca_afk17:37
trownbogdando: likely17:38
* bogdando sigh17:38
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: Ensure keepalived is restarted when necessary.  https://review.openstack.org/39505317:38
trownbogdando: you can use 127.0.0.x (ie 127.0.0.2) as long as you can `ssh root@127.0.0.2`17:38
bogdandotrown, intersting, thanks. will try17:39
*** fzdarsky is now known as fzdarsky|afk17:40
*** jprovazn has joined #tripleo17:40
tzumainnd0ugal, question if you have a moment - you mentioned that the default deployment plan comes from /usr/share/openstack-tripleo-heat-templates, right?17:41
*** trown is now known as trown|lunchnvote17:43
*** rajinir has joined #tripleo17:45
shardytzumainn: yes it does17:48
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Set the (new) SELinux label for OSD data directories on upgrade  https://review.openstack.org/39509717:48
*** athomas has quit IRC17:48
tzumainnshardy, I'm seeing some files from the overcloud plan that aren't in /usr/share/openstack-tripleo-heat-templates17:49
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Set the (new) SELinux label for OSD data directories on upgrade  https://review.openstack.org/39509717:49
tzumainnshardy, https://paste.fedoraproject.org/475792/14786273/17:49
tzumainnI was wondering where those come from?17:49
shardytzumainn: do they have coresponding *.j2.yaml files in /usr/share/openstack-tripleo-heat-templates ?17:49
tzumainnshardy, some of them do17:49
shardytzumainn: we now do a jinja2 templating step inside the TripleO API (e.g inside a mistral action) when the plan is created17:50
tzumainnshardy, ah, okay17:50
shardyit allows folks to specify arbitrary custom roles17:50
tzumainnshardy, so it's likely that the extra files come from part of the plan create workflow?17:50
shardytzumainn: see /usr/share/openstack-tripleo-heat-templates/roles_data.yaml which is the default input to jinja217:50
shardy(can be overridden with -r some_different_data.yaml on the CLI)17:51
*** mgould is now known as mgould|afk17:51
shardytzumainn: I'm certain that they do, yes17:51
*** derekh has quit IRC17:51
tzumainnshardy, great, thanks!17:51
*** bana_k has joined #tripleo17:52
shardyhttps://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/templates.py#L7117:52
shardytzumainn: ^^ FYI that's where it's done17:52
shardyI wrote a blog post with some details, tripleo-docs still todo17:52
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Set the (new) SELinux label for OSD data directories on upgrade  https://review.openstack.org/39509717:52
*** shardy has quit IRC17:54
*** chandankumar has quit IRC17:55
openstackgerritMerged openstack/tripleo-heat-templates: Mitaka-Newton upgrade fix network disruption during convergence  https://review.openstack.org/39498018:00
openstackgerritMerged openstack/puppet-tripleo: Add proper handling of IPv6 addresses for rabbit host/port handling  https://review.openstack.org/39474218:00
*** lucasagomes is now known as lucas-afk18:07
*** fragatina has joined #tripleo18:08
gfidentettyl18:09
*** gfidente has quit IRC18:09
openstackgerritBrent Eagles proposed openstack/puppet-tripleo: Add proper handling of IPv6 addresses for rabbit host/port handling  https://review.openstack.org/39510418:11
*** pkovar has quit IRC18:12
openstackgerritMerged openstack/python-tripleoclient: Change the qemu-img bindep to be qemu-img-ev  https://review.openstack.org/39500418:16
*** abregman has joined #tripleo18:22
*** abregman is now known as abregman|nb18:30
*** jpena is now known as jpena|off18:31
*** myoung is now known as myoung|afk18:32
bogdandotrown|lunchnvote, not sure if localhost / 127.x is related to my case. I can see the undercloud VM just can't obtain IPs via dhcp, and "ip neigh" shows some IP matching the external net's DHCP range as "dev brext  FAILED"...18:34
bogdandoand I can't enable root login w/o passwords to my gce instance for sanity reasons :)18:35
bogdandoit is centos7 instance btw18:35
bogdandoI expected things to go a little bit more smooth :(18:36
*** fultonj has joined #tripleo18:40
*** abregman|nb has quit IRC18:41
*** anshul has joined #tripleo18:41
*** rasca has quit IRC18:43
*** fragatina has quit IRC18:46
*** fragatina has joined #tripleo18:46
*** david-lyle_ has joined #tripleo18:47
*** jkilpatr has quit IRC18:52
*** david-lyle_ has quit IRC18:53
*** lmiccini has quit IRC18:54
*** ebalduf has quit IRC18:55
openstackgerritMerged openstack/tripleo-heat-templates: Do not reference CephBase from CephExternal service  https://review.openstack.org/39499318:59
*** fragatin_ has joined #tripleo19:00
*** tiswanso has quit IRC19:00
*** fragatina has quit IRC19:03
sshnaidmwho is familiar with gnocchi or its people?19:03
sshnaidmI'd like to get attention to this issue https://bugs.launchpad.net/gnocchi/+bug/164022519:04
openstackLaunchpad bug 1640225 in tripleo "Gnocchi is bombing logs with error" [Undecided,New]19:04
*** tiswanso has joined #tripleo19:05
*** dsariel has quit IRC19:06
sshnaidmweshay, ^^19:07
*** liverpooler has quit IRC19:07
weshaysshnaidm, pradk can point you in the right direction19:08
*** ccamacho has quit IRC19:08
pradksshnaidm, seems like endpoint is not reachable? either redis is down or swift is down .. also check if gnocchi end point is reachable19:10
pradklooks like a configuration issue19:10
sshnaidmpradk, redis might be not reachable, but the issue is with logging which sticks the host and takes all space there19:10
sshnaidmpradk, it logs a few lines every millisecond19:11
sshnaidmpradk, the load average is about 2019:11
*** amoralej is now known as amoralej|off19:11
pradksshnaidm, you dont want it to report errors in the logs?19:11
sshnaidmpradk, definitely not every millisecond :)19:12
pradkwe do log rotate ever 10M i believe.. but i think we need to first address underlying issue why the error is occuring19:12
sshnaidmpradk, it's not even about rotating, it should have some considerable timeout for tries and logs.19:13
pradksshnaidm, understand, in this case i think its just retrying and logging the trace .. we can look into it on gnocchi side19:13
sshnaidmpradk, I suppose it's logging (crazy writing to disk) which causes load average to be about 2019:14
*** jobewan has joined #tripleo19:14
sshnaidmpradk, because this pingtest timeouts on tripleo-ci and breaks jobs, so I'd like you'll give it some good priority19:15
beaglesif it is the logging, I wonder if it could keep a timestamp and after the first few retries starts loging every 10 seconds that it hasn't been successfuly... or maybe a counter since it is pretty quick on the retry...19:16
*** ayoung has quit IRC19:16
beaglesWARNING: last 30 attempts have failed....19:16
beagles*shrug*19:16
pradki think we already do that with tenacity.. we might just need to increase timeout19:16
*** ayoung has joined #tripleo19:17
sshnaidmbeagles, pradk I suppose it logs when tries to connect, right? So connection timeout would be useful too..19:17
*** nyechiel has joined #tripleo19:18
*** numans has joined #tripleo19:19
*** jkilpatr has joined #tripleo19:21
openstackgerritAlex Schultz proposed openstack/puppet-tripleo: Standardize httpchk for http services  https://review.openstack.org/39468619:23
*** anshul has quit IRC19:28
openstackgerritAlex Schultz proposed openstack/puppet-tripleo: Standardize httpchk for http services  https://review.openstack.org/39468619:30
openstackgerritMerged openstack/python-tripleoclient: Don't assume CountDefault set in roles_data.yaml  https://review.openstack.org/39492419:30
*** paramite has quit IRC19:30
*** trown|lunchnvote is now known as trown19:35
openstackgerritJames Slagle proposed openstack/python-tripleoclient: Don't assume CountDefault set in roles_data.yaml  https://review.openstack.org/39513119:35
thrashanyone got some time to review https://review.openstack.org/#/c/389830/ ? Need this for major newton bug...19:44
slaglethrash: was just reviewing actually19:45
slaglei'll approve it19:45
thrashslagle: woot :)19:46
slaglebut can you fix bnemec's nit in a followup?19:46
thrashslagle: Yes. I was really trying to drive home that this is extremely temporary. :)19:47
*** numans has quit IRC19:48
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Change wording of help for --update-plan-only  https://review.openstack.org/39513519:49
thrashslagle: bnemec ^^^19:49
slaglethanks19:50
openstackgerritHonza Pokorny proposed openstack/tripleo-specs: Add gui logging spec  https://review.openstack.org/39513819:50
*** milan has quit IRC19:51
honzajrist: who should i add as reviewers for an ocata spec?  i added some but not sure if more needed, see ^^^19:51
*** abregman has joined #tripleo19:55
*** rhallisey has quit IRC19:56
*** fragatin_ has quit IRC19:58
*** fragatina has joined #tripleo19:58
*** ebalduf has joined #tripleo20:01
*** kjw3 has quit IRC20:01
*** fragatina has quit IRC20:02
*** dbecker has quit IRC20:07
*** jkilpatr has quit IRC20:08
*** dsariel has joined #tripleo20:09
openstackgerritMerged openstack/tripleo-heat-templates: Use --globoff when downloading artifacts  https://review.openstack.org/39498720:13
*** fragatina has joined #tripleo20:14
*** jcoufal has quit IRC20:16
*** nyechiel has quit IRC20:17
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: Move ENV var setting to a common function  https://review.openstack.org/37703520:20
*** Goneri has quit IRC20:21
*** jkilpatr has joined #tripleo20:21
*** fragatina has quit IRC20:26
*** kjw3 has joined #tripleo20:26
*** fragatina has joined #tripleo20:26
*** Guest64 has joined #tripleo20:33
Guest64dsneddon20:34
*** Guest64 is now known as gbarros20:34
gbarrosdsneddon20:34
dsneddongbarros and I have been trying to troubleshoot this error, which doesn't make sense since we have no Managment network or ports enabled. There was a previous (deleted) stack that had the management network defined in the NIC configs, but currently it is commented out: [overcloud-Controller-rdmbclvm5yqb-1-yx3bqifarmx5]: CREATE_FAILED  Resource CREATE failed: The Referenced Attribute (ManagementPort ip_subnet) is incorrect.20:36
dsneddonslagle, bnemec, dprince: Help with above would be appreciated. Ideas for troubleshooting? ^^^20:36
bnemecCrap, we broke the scenario002 job with the client change.20:38
bnemechttp://logs.openstack.org/04/395104/1/check/gate-tripleo-ci-centos-7-scenario002-multinode/9322cfb/console.html20:39
bnemecThat's blocking the ipv6 backport.20:39
openstackgerritMerged openstack/python-tripleoclient: Add a flag to allow the plan to be updated without deploying  https://review.openstack.org/38983020:39
bnemecdsneddon: My first thoughts would be a heat bug (I think we got those errors a bunch at one point) or that a reference to the management network was left in a template somewhere.20:41
dsneddonbnemec, Is there any way that a template named compute.yaml.back lying around could be automatically included?20:42
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Add a flag to allow the plan to be updated without deploying  https://review.openstack.org/39515820:42
dsneddonbnemec, I've never seen Heat include entire directories of YAML files, so I assume not.20:42
bnemecdsneddon: I wouldn't think so.  Those files have to be explicitly referenced from the resource registry AFAIK.20:43
*** ayoung has quit IRC20:44
*** tiswanso has quit IRC20:46
*** myoung|afk is now known as myoung20:46
*** abregman has quit IRC20:49
gbarrosbnemec, dsneddon, can we instrument the deploy somehow to see where that is getting called in?20:50
gbarrosmorazi, do you know if thats possible?20:51
gbarrosdsneddon, is that you driving?20:51
dsneddongbarros, Was, I'm done20:53
gbarrosdsneddon, any ideas on how we can track down whats needing the managmentip bits?20:54
*** cdearborn has quit IRC20:55
bnemecgbarros: dsneddon: You might be able to look at the resource list and see what the parent of the failed resource is.20:56
dsneddongbarros, No, I don't see anything that is non-standard about your deployment.20:56
*** liverpooler has joined #tripleo20:56
dsneddonbnemec, How do you identify the parent of a failed rsource?20:56
*** rcernin has joined #tripleo20:59
bnemecdsneddon: Weird, I thought there was a parent resource id in the resource-list output, but I don't see it now.21:00
bnemecI'm wondering where I saw that.21:00
thervebnemec, You have the stack name, that's the parent21:05
*** jayg is now known as jayg|g0n321:09
*** cdearborn has joined #tripleo21:16
*** fzdarsky|afk has quit IRC21:17
*** Guest24494 is now known as melwitt21:18
*** dprince has quit IRC21:18
*** lblanchard has quit IRC21:20
*** ppradhan has joined #tripleo21:21
*** jprovazn has quit IRC21:33
openstackgerritThiago da Silva proposed openstack/tripleo-heat-templates: set url_base option in static web middleware  https://review.openstack.org/39291821:38
*** chem has quit IRC21:44
*** tbonds has quit IRC21:46
*** jkilpatr has quit IRC21:51
morazigbarros, what exact managementip key/values are you passing in via the template?21:53
*** florianf has quit IRC21:54
*** trown is now known as trown|outtypewww21:59
*** akrivoka has quit IRC22:00
jtomasekhonza: still around?22:04
jtomasekhonza: why this change? https://review.openstack.org/#/c/392589/5/src/js/actions/ParametersActions.js22:05
*** tiswanso has joined #tripleo22:06
*** jcoufal has joined #tripleo22:06
*** jcoufal has quit IRC22:07
*** jcoufal has joined #tripleo22:09
slaglebnemec: perhaps we should also post a revert of https://review.openstack.org/#/c/389057/ ?22:09
slaglethe project-config thing might take a bit to merge22:09
slaglewhereas we could unblock ourselves with a revert22:09
*** jcoufal has quit IRC22:10
bnemecslagle: That works for me too.  I agree it's likely to unblock the ipv6 backport faster than fixing project-config.22:10
*** tiswanso has quit IRC22:10
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Revert "Add Barbican key order to scenario002"  https://review.openstack.org/39521722:12
slagleboom22:12
bnemecHeh, beat me to it.22:13
*** jkilpatr has joined #tripleo22:15
bnemecI opened a separate bug for the bogus error message from the client too: https://bugs.launchpad.net/tripleo/+bug/164032422:16
openstackLaunchpad bug 1640324 in tripleo "Client reporting wrong missing file" [High,Triaged]22:16
openstackgerritMerged openstack/tripleo-image-elements: Remove galera package from being explicitly installed with mariadb  https://review.openstack.org/35695222:17
*** jcoufal has joined #tripleo22:18
*** bfournie has quit IRC22:19
bnemecStill disappointed that we didn't go with multinod3s for the 3node job. ;-)22:19
*** jobewan has quit IRC22:31
openstackgerritBen Nemec proposed openstack/python-tripleoclient: Mock yaml.safe_load safely  https://review.openstack.org/39523122:37
*** pradk has quit IRC22:40
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates: Add ceph to the hieradata file for compute  https://review.openstack.org/39523422:42
*** ppradhan has left #tripleo22:47
*** david-lyle_ has joined #tripleo22:50
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates: Add ceph to the hieradata file for compute  https://review.openstack.org/39523422:52
*** david-lyle_ has quit IRC22:54
*** liverpooler has quit IRC22:58
*** bfournie has joined #tripleo22:58
*** dmarlin has joined #tripleo23:02
*** rajinir has quit IRC23:06
*** rcernin has quit IRC23:18
*** absubram has joined #tripleo23:21
*** egafford has quit IRC23:31
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Change wording of help for --update-plan-only  https://review.openstack.org/39513523:33
*** michapma_alt has joined #tripleo23:34
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates: Add ceph to the hieradata file for compute  https://review.openstack.org/39523423:42
*** ayoung has joined #tripleo23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!