Wednesday, 2016-10-19

*** yamahata has joined #tripleo00:00
*** ramishra has joined #tripleo00:01
*** ramishra has quit IRC00:16
*** ramishra has joined #tripleo00:18
*** limao has joined #tripleo00:19
*** limao has quit IRC00:20
*** limao has joined #tripleo00:20
openstackgerritMerged openstack/puppet-tripleo: Fix broken rabbitmqctl commands when using ipv6  https://review.openstack.org/38692100:21
*** cdearborn has joined #tripleo00:26
*** ctayal has quit IRC00:30
*** ramishra has quit IRC00:30
*** ramishra has joined #tripleo00:32
*** x220_ has joined #tripleo00:35
*** newmember has quit IRC00:39
*** akuznetsov has joined #tripleo00:39
*** akuznetsov has quit IRC00:44
*** ctayal has joined #tripleo00:54
*** ramishra has quit IRC00:59
*** ramishra has joined #tripleo01:00
*** xuao has joined #tripleo01:01
*** ctayal has quit IRC01:06
*** xuao has quit IRC01:11
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: Fixed NoneType issue when monitoring-environment.yaml  https://review.openstack.org/38827201:11
*** bana_k has quit IRC01:13
*** xuao has joined #tripleo01:14
*** ramishra has quit IRC01:28
*** ramishra has joined #tripleo01:28
*** fzdarsky_ has joined #tripleo01:30
*** fzdarsky|afk has quit IRC01:33
*** cdearborn has quit IRC01:34
*** dmacpher has joined #tripleo01:36
*** sudipto_ has joined #tripleo01:38
*** sudipto has joined #tripleo01:38
*** skramaja has joined #tripleo01:39
*** akuznetsov has joined #tripleo01:40
*** akuznetsov has quit IRC01:44
*** sudipto_ has quit IRC01:59
*** sudipto has quit IRC01:59
*** padkrish has joined #tripleo02:00
*** padkrish has quit IRC02:06
*** sudipto_ has joined #tripleo02:12
*** sudipto has joined #tripleo02:12
*** chlong has joined #tripleo02:16
openstackgerritRyan Brady proposed openstack/tripleo-common: [WIP]Adds retry to plan workflows  https://review.openstack.org/38828502:19
*** yamahata has quit IRC02:20
*** ibravo has joined #tripleo02:20
*** ibravo has left #tripleo02:21
*** sudipto_ has quit IRC02:21
*** sudipto has quit IRC02:21
*** ramishra has quit IRC02:21
*** rbrady is now known as rbrady-afk02:21
*** ramishra has joined #tripleo02:23
*** bana_k has joined #tripleo02:26
*** bana_k has quit IRC02:38
*** akuznetsov has joined #tripleo02:41
*** limao has quit IRC02:44
*** limao has joined #tripleo02:45
*** akuznetsov has quit IRC02:45
*** ibravo has joined #tripleo02:56
*** x220_ has quit IRC03:03
*** limao has quit IRC03:23
*** limao has joined #tripleo03:25
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates: GATE TEST, please ignore  https://review.openstack.org/36544903:30
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements  https://review.openstack.org/38575403:33
*** akuznetsov has joined #tripleo03:41
*** links has joined #tripleo03:46
*** akuznetsov has quit IRC03:46
*** bana_k has joined #tripleo03:51
*** yamahata has joined #tripleo03:51
*** akuznetsov has joined #tripleo03:55
openstackgerritOpenStack Proposal Bot proposed openstack/os-cloud-config: Updated from global requirements  https://review.openstack.org/35067603:57
openstackgerritOpenStack Proposal Bot proposed openstack/os-collect-config: Updated from global requirements  https://review.openstack.org/35090503:57
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-validations: Updated from global requirements  https://review.openstack.org/38833604:00
*** akuznetsov has quit IRC04:00
*** akuznetsov has joined #tripleo04:02
*** ibravo has quit IRC04:02
*** ibravo has joined #tripleo04:03
*** padkrish has joined #tripleo04:05
*** akuznetsov has quit IRC04:06
*** padkrish_ has joined #tripleo04:10
*** padkrish has quit IRC04:10
*** limao has quit IRC04:12
*** numans has joined #tripleo04:15
*** padkrish_ has quit IRC04:21
*** padkrish has joined #tripleo04:21
*** ctayal has joined #tripleo04:23
*** ctayal has quit IRC04:23
*** ctayal has joined #tripleo04:23
*** bana_k has quit IRC04:25
*** ibravo has quit IRC04:26
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements  https://review.openstack.org/38575404:30
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/38114804:41
*** padkrish has quit IRC04:46
*** coolsvap has joined #tripleo04:47
*** dsariel has joined #tripleo04:48
*** skramaja has quit IRC04:49
*** skramaja has joined #tripleo04:54
*** akuznetsov has joined #tripleo04:56
*** nyechiel has joined #tripleo04:58
*** dsneddon_ has joined #tripleo04:58
*** akuznetsov has quit IRC05:00
*** anbu_ has joined #tripleo05:04
*** anbu has quit IRC05:05
*** ekoorjr has quit IRC05:05
*** itlinux has quit IRC05:09
*** ctayal has quit IRC05:21
*** jaosorior has joined #tripleo05:21
*** ctayal has joined #tripleo05:22
*** tzumainn has quit IRC05:25
jaosoriordtrainor: are you still around?05:27
dtrainorsure am05:30
*** akuznetsov has joined #tripleo05:30
jaosoriordtrainor: so, I tested your patch. I was a bit confused initially since all I saw was a blank page05:32
jaosoriorbut then realized that the UI relies very heavily on javascript05:32
dtrainoryep sure does05:32
jaosoriorand I was using lynx (which doesn't have support for it)05:33
dtrainoroh haha05:33
jaosoriorbut the haproxy configuration looked just fine05:33
dtrainordon't scare me liek that!05:33
dtrainorexcellent05:33
jaosoriorhahaha, so yeah, +2ed your patch05:34
jaosoriornow, I think the SSL by default, while it's something I would really like, I suggest you bring it up in the summit05:35
jaosoriorto see what people think05:35
dtrainorah you think so?  i won't be at summit unfortunately.05:36
jaosoriordamn05:36
jaosoriorI can bring it up05:36
jaosoriordtrainor: if you won't be there, maybe it would be good to ping more people about it. At least to get their attention for reviews05:36
dtrainorfair enough.  openstack-dev mailing list?05:37
jaosoriordtrainor: good idea!05:38
dtrainorcool, just making sure05:38
dtrainori brought it up during the TripleO Team Meeting this morning and it went over well05:39
*** limao has joined #tripleo05:39
jaosoriordtrainor: ah, you brought it up already?05:39
jaosoriorThen that's great!05:39
dtrainoryep sure did.  trivial concerns that warped in to a discussion about something else but it went over well, that's when i decided to do the work05:39
jaosoriordtrainor: I didn't know. Anyway, if that's the case. It's a matter of fixing the issues in the unit tests and you got a +2 from my side.05:41
dtrainorexcellent, i'll start on that in the A.M.05:42
*** nyechiel has quit IRC05:42
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add barbican profile rspec testing  https://review.openstack.org/38809905:46
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add barbican profile  https://review.openstack.org/35244705:46
*** nyechiel has joined #tripleo05:52
*** apetrich has quit IRC05:54
*** saneax-_-|AFK is now known as saneax05:55
dtrainorhmm, running in to a failed deployment from the controller, I see:  galera(galera)[89879]: INFO: Waiting on node <overcloud-controller-0.internalapi.localdomain> to report database status before Master instances can start.05:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for gnocchi  https://review.openstack.org/37764505:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for ceilometer  https://review.openstack.org/37764405:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for aodh  https://review.openstack.org/36545705:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702905:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable internal TLS for MySQL  https://review.openstack.org/37847105:56
*** pkovar has joined #tripleo05:56
dtrainorin ControllerDeployment_Step205:56
jaosoriordtrainor: that's the galera nodes taking time to report to pacemaker05:56
jaosoriordtrainor: is the deployment taking unusually long?05:57
dtrainorit is, it feels like a timeout05:57
*** nyechiel has quit IRC05:57
*** apetrich has joined #tripleo05:58
jaosoriordtrainor: do you have this commit in your overcloud nodes? https://github.com/openstack/puppet-tripleo/commit/39d88a49bf83a7a3437edc82f42986596356d33105:59
jaosoriordtrainor: you can check if the code is there in /etc/puppet/modules/tripleo/05:59
dciabrindtrainor: that's because of the value passed to gcomm://, it should read overcloud-controller-0. You  probably miss a recent puppet-tripleo06:00
dtrainorjaosorior, sure do.06:01
jaosoriordtrainor: so you have that in the puppet manifest?06:01
dtrainori do, jaosorior06:01
jaosoriorok06:02
jaosoriorso, what about t-h-t06:02
jaosoriorlet me fetch the commit for you06:02
dtrainordciabrin, i see waht you mean:06:02
*** aufi has joined #tripleo06:02
dtrainor(well, if my paste buffer was working....?!?!?!)06:02
dciabrin:)06:03
jaosoriordtrainor: do you have this one https://github.com/openstack/tripleo-heat-templates/commit/6a408ff27e6fd6bff0a822f225c77a76690f805a ?06:03
jaosoriorin t-h-t06:03
dtrainorFailed to open channel 'galera_cluster' at 'gcomm://overcloud-controller-0.internalapi.localdomain': -110 (Connection timed out)06:03
dciabrinhttps://www.redhat.com/archives/rdo-list/2016-October/msg00080.html <- someone on RDO had the same issue a few weeks ago06:03
dtrainori saw that06:03
jaosoriordtrainor: yep, that means that you're missing one of those two.06:03
dtrainorgot it.  thank you06:03
jaosoriordciabrin: will you get some time to do the RA fix so we can stop using those short-names?06:04
dtrainor(RA?)06:05
dciabrinjaosorior, it's the 3rd thing on my todo-list, for real :)06:05
jaosoriordtrainor: Resource Agent? I think... dciabrin knows more specifics about this one. He's the Galera expert :D06:06
dciabrindtrainor, resource agent06:06
jaosoriordciabrin: duuude nice :D06:06
dtrainorah ha.06:06
*** dsneddon_ has quit IRC06:08
*** rcernin has joined #tripleo06:11
*** akuznetsov has quit IRC06:17
*** lmiccini has joined #tripleo06:18
*** lmiccini has quit IRC06:18
*** akuznetsov has joined #tripleo06:19
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Create directories with root  https://review.openstack.org/38489206:21
*** rasca has joined #tripleo06:21
*** chandankumar has joined #tripleo06:23
*** tobias_fiberdata has joined #tripleo06:24
*** jprovazn has joined #tripleo06:26
*** ealcaniz has joined #tripleo06:27
*** newmember has joined #tripleo06:28
*** apetrich has quit IRC06:29
bandinimorning *06:29
*** ealcaniz__ has joined #tripleo06:31
*** dsariel has quit IRC06:31
*** ealcaniz has quit IRC06:32
*** apetrich has joined #tripleo06:32
*** xuao has quit IRC06:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for gnocchi  https://review.openstack.org/37764506:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for ceilometer  https://review.openstack.org/37764406:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for aodh  https://review.openstack.org/36545706:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702906:33
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable internal TLS for MySQL  https://review.openstack.org/37847106:33
*** cylopez has joined #tripleo06:35
*** tremble has joined #tripleo06:37
*** lmiccini has joined #tripleo06:37
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add Barbican to the overcloud  https://review.openstack.org/35245806:40
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE testing barbican  https://review.openstack.org/38573706:40
*** akuznetsov has quit IRC06:41
*** nyechiel has joined #tripleo06:41
*** jaosorior has quit IRC06:47
*** hjensas has joined #tripleo06:48
*** hjensas has joined #tripleo06:48
*** lazy_prince has quit IRC06:51
*** lazy_prince has joined #tripleo06:51
openstackgerritIan Wienand proposed openstack/diskimage-builder: Revert "Fix grub installation for RHEL"  https://review.openstack.org/38846806:52
*** hjensas has quit IRC06:54
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Use Password Generation provided by the Mistral workflows  https://review.openstack.org/37554406:55
*** flepied1 has quit IRC06:56
*** nyechiel has quit IRC06:59
*** jaosorior has joined #tripleo07:00
*** mgrohar has joined #tripleo07:03
jaosoriorHey guys, could anybody check these two commits out https://review.openstack.org/#/c/387202/  https://review.openstack.org/#/c/387203/ ??07:03
*** bana_k has joined #tripleo07:06
*** tesseract has joined #tripleo07:08
*** mcornea has joined #tripleo07:09
*** tesseract is now known as Guest9021107:09
*** mgrohar has quit IRC07:09
*** jlinkes has joined #tripleo07:10
*** tosky has joined #tripleo07:10
d0ugalWhat is the etiquette when backporting two changes to Newton? The first one has an issue that the second resolved. Should they be squashed?07:11
*** akuznetsov has joined #tripleo07:11
*** snecklifter has joined #tripleo07:12
*** akuznetsov has quit IRC07:13
*** akuznetsov has joined #tripleo07:14
*** jlinkes has quit IRC07:15
*** zoli_gone-proxy is now known as zoliXXL07:18
*** zoliXXL is now known as zoli|wfh07:19
*** b00tcat has joined #tripleo07:20
*** jlinkes has joined #tripleo07:23
b00tcathi, can someone give me +workflow on these patches? https://review.openstack.org/#/c/383057/ https://review.openstack.org/#/c/383056/07:25
d0ugaljtomasek: FWIW, this has landed in the CLI now - https://review.openstack.org/#/c/382277/07:27
d0ugaljtomasek: so should be safe07:27
jtomasekd0ugal: cool, I'll test the GUI patch on new setup I am going to do today07:28
d0ugaljtomasek: cool, obviously still not urgent but thought I'd let you know07:28
jtomasekd0ugal: yep, thanks07:29
*** ctayal has quit IRC07:29
snecklifterAny reason why diskimage-builder removes the repo files at the end of it's build?07:29
*** zoli|wfh is now known as zoli_gone-proxy07:29
*** bana_k has quit IRC07:29
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for Nova API  https://review.openstack.org/38849507:31
*** skramaja_ has joined #tripleo07:32
*** skramaja has quit IRC07:32
b00tcatsnecklifter: where?07:33
snecklifterb00tcat: on the built overcloud image07:33
snecklifterthis is only when I follow the mitaka stable instructions and just use the two export lines07:34
snecklifterit basically calls 99-remove-yum-repo-conf but I'm not sure the logic behind this07:35
snecklifteras you cannot update packages on the overcloud when this happens07:35
b00tcatah right, afaik you install the packages at image creation time for security/time optimization07:35
b00tcatfor instance I have puppet code that recreates them afterwards07:36
b00tcatat provisioning time07:36
snecklifterhmmm, weird07:36
*** zoli_gone-proxy is now known as zoliXXL07:36
snecklifterb00tcat: but when using trunk instructions the repo files remain in the finished image07:37
*** fzdarsky_ is now known as fzdarsky07:37
b00tcathm, I use Mitaka for the time being too07:38
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Generate internal TLS hieradata for apache services  https://review.openstack.org/36607507:39
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable TLS in the internal networkf or Mysql  https://review.openstack.org/37853707:39
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable internal TLS for ceilometer  https://review.openstack.org/37764807:39
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable internal TLS for aodh  https://review.openstack.org/37764907:39
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable internal TLS for gnocchi  https://review.openstack.org/37765007:39
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Enable internal TLS for Nova API  https://review.openstack.org/38850307:39
snecklifterb00tcat: ok, so you just pull down the repo files in firstboot.yaml?07:40
*** ebarrera has joined #tripleo07:40
b00tcatsnecklifter: in theory all repo files that you have defined in $DIB_ variables should be removed https://github.com/openstack/diskimage-builder/blob/master/elements/yum/cleanup.d/99-remove-yum-repo-conf#L1607:41
b00tcatsnecklifter: I do it in the .pp files actually that are ran when spawning the overcloud07:41
*** masco has joined #tripleo07:42
snecklifterb00tcat: ok, its another thing to do but I suppose there must be some logic to it, would just like to understand why they are removed07:43
*** akuznetsov has quit IRC07:45
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud: Always deploy HAProxy  https://review.openstack.org/36036807:48
*** jpich has joined #tripleo07:50
*** akuznetsov has joined #tripleo07:51
*** amoralej|off is now known as amoralej07:52
*** abregman|afk is now known as abregman07:59
d0ugalTRIPLEO_ROOT is used in tripleo.sh here: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L98-L100 but it isn't set until... ttps://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L15408:01
d0ugalMy bash is terrible, but isn't that a problem?08:01
d0ugalshould we move L154 up, or the other block down?08:01
*** jpena|off is now known as jpena08:02
d0ugalThis just happened recently. https://github.com/openstack-infra/tripleo-ci/commit/67deb95ed15870113ebb856629cbb0a1bd4eee5508:02
*** ccamacho|out is now known as ccamacho08:04
*** nyechiel has joined #tripleo08:05
EmilienMmorning08:07
jaosoriorEmilienM: wow, you're early08:07
jaosoriorEmilienM: morning dude, what's up?08:08
EmilienMjaosorior: i'm arrived in Spain08:08
jaosoriorEmilienM: nice! how's things over there? Are you in BCN already?08:08
EmilienMyes but moving out of the city for PTOs08:08
EmilienMI just came here and say hi08:09
EmilienMjaosorior: I need to know which puppet modules did you backport the proxy/tls stuff08:10
jaosoriorEmilienM: have fun dude!08:10
EmilienMaodh, ceilo, gnocchi and more?08:10
jaosorioraodh, ceilo and gnocchi, that's all. Neutron hasn't approved my backport so I can't backport to puppet-neutron yet.08:10
EmilienMok. I'll release them today08:11
EmilienMplease let me know if you backport more. I need to know08:11
jaosoriorEmilienM: you can't +A this one yet https://review.openstack.org/#/c/388099/ since it depends on this https://review.openstack.org/#/c/352447/08:11
jaosoriorEmilienM: well, you can, but it won't merge it.08:11
b00tcatEmilienM: if you like beer, there's the clone of oktoberfest in BCN at the moment08:11
EmilienMjaosorior: right, it won't merge. I dropped +A08:11
*** flepied has joined #tripleo08:12
EmilienMb00tcat: thanks, but not staying in town this week. I'm with family until Sunday, out of town08:12
jaosorioroktoberfest clone? Damn! should have travelled earlier :(08:12
EmilienMand nothing can clone real oktoberfest :P08:12
EmilienMI did it one time, it's not clonable !08:12
b00tcatlol08:12
EmilienMI actually don't remember everything08:13
jaosoriorbeer is good anyway, doesn't have to be the same08:13
EmilienMwell, that's another story08:13
jaosoriorlol08:13
d0ugalEmilienM: hi!08:13
EmilienMd0ugal: hey oh08:13
jaosoriorEmilienM: anyway, enjoy your PTO dude!08:13
EmilienMthanks08:13
jaosoriorEmilienM: is there any patches you need me to take over?08:13
d0ugalYeah, well deserved PTO :)08:13
* d0ugal was just saying hi back08:13
EmilienMplease watch d0ugal so he doesn't break CI08:13
d0ugallol08:13
EmilienM /joke08:13
d0ugalI'm gonna stop fixing things now :P08:14
*** ohamada has joined #tripleo08:14
EmilienMd0ugal: no please :P08:14
EmilienMcan you guys approve this backport ? https://review.openstack.org/#/c/388272/08:15
*** ohamada has quit IRC08:15
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo: Fix broken rabbitmqctl commands when using ipv6  https://review.openstack.org/38852908:15
*** rcernin has quit IRC08:16
jaosoriorEmilienM: done08:16
*** rcernin has joined #tripleo08:16
*** ohamada has joined #tripleo08:17
EmilienMjaosorior: thx08:17
d0ugalEmilienM: What is the etiquette when backporting two changes to Newton? The first one has an issue that the second resolved. Should they be squashed?08:17
*** rcernin has quit IRC08:18
EmilienMd0ugal: good idea08:18
EmilienMd0ugal: I'm ok with squashing, as long as it's cleanly done08:18
d0ugalEmilienM: okay, I'll include both commit messages and add a note to say why. thanks08:18
*** rcernin has joined #tripleo08:19
EmilienMd0ugal: excellent08:19
*** rcernin has quit IRC08:19
d0ugalLooks like I have finally got this to go green - https://review.openstack.org/#/c/375544/08:19
*** rcernin has joined #tripleo08:19
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4  https://review.openstack.org/37120908:19
d0ugalIt has only taken me about two weeks of trying different approaches and breaking and fixing things :P08:20
EmilienMd0ugal: I saw, but it depends on some patches that fail. I guess you want them squashed?08:20
* jaosorior off for food08:20
*** jaosorior is now known as jaosorior_lunch08:20
d0ugalEmilienM: I need to go through them and figure out the best approach.08:21
d0ugalEmilienM: all the depends-on have merged in master, but not Newton08:21
d0ugalEmilienM: so yeah, I think I might need to squash for newton, I'll figure it out now.08:21
*** milan has joined #tripleo08:22
EmilienMd0ugal: ack08:22
d0ugalEmilienM: How is your bash? Shouldn't this be after line 154? https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L9808:24
d0ugal(or 154 should be before that)08:24
*** akuznetsov has quit IRC08:24
openstackgerritDougal Matthews proposed openstack-infra/tripleo-ci: Set TRIPLEO_ROOT before it is first used  https://review.openstack.org/38854208:25
d0ugali.e. ^08:25
EmilienMd0ugal: mhh good question, let me look08:25
EmilienMd0ugal: have you seen problems in CI?08:25
d0ugalEmilienM: I got an error on my local scripts08:26
EmilienMd0ugal: but your patch makes sense08:26
d0ugalEmilienM: I think I might have a stricter bash flag that doesn't allow unbound variables?08:26
EmilienMmaybe? it could have been exported in another script also08:27
EmilienMI'm +2ing your patch as it makes sense to set it before using it08:27
d0ugalEmilienM: Thanks08:27
openstackgerritDougal Matthews proposed openstack-infra/tripleo-ci: Set TRIPLEO_ROOT before it is first used  https://review.openstack.org/38854208:34
d0ugalEmilienM: ^ add some more info to the commit message.08:34
EmilienMok08:35
*** nyechiel has quit IRC08:35
*** florianf has joined #tripleo08:36
*** derekh has joined #tripleo08:39
*** anbu has joined #tripleo08:42
*** anbu_ has quit IRC08:43
*** anbu_ has joined #tripleo08:45
*** dtantsur|afk is now known as dtantsur|sick08:46
*** anbu has quit IRC08:46
*** anbu__ has joined #tripleo08:47
*** egafford has joined #tripleo08:49
*** anbu_ has quit IRC08:50
*** anbu_ has joined #tripleo08:52
*** anbu__ has quit IRC08:54
*** anbu has joined #tripleo08:55
*** anbu_ has quit IRC08:57
openstackgerritMerged openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/38114808:57
*** social has joined #tripleo09:00
openstackgerritMerged openstack/tripleo-heat-templates: Fixed NoneType issue when monitoring-environment.yaml  https://review.openstack.org/38827209:02
*** abregman is now known as abregman|afk09:05
* EmilienM off for pto09:05
*** jaosorior_lunch is now known as jaosorior09:05
jaosoriorhey guys, I would REALLY appreciate reviews for these commits https://review.openstack.org/#/q/topic:bp/tls-via-certmonger+status:open they've been there for a while and they're stacking up09:07
*** chandankumar has quit IRC09:08
*** tobias_fiberdata has quit IRC09:08
*** chandankumar has joined #tripleo09:08
jaosoriorccamacho: hey dude, how's it going?09:08
jaosoriorccamacho: feeling better from the flu?09:09
*** links has quit IRC09:09
*** chandankumar has quit IRC09:09
d0ugaljaosorior: wow, that is quite a few reviews :)09:09
*** chandankumar has joined #tripleo09:09
openstackgerritJiri Stransky proposed openstack/diskimage-builder: RH grub: don't fail if grub is already installed  https://review.openstack.org/38856709:09
jaosoriord0ugal: yeah dude, I REALLY need reviews, people don't like reviewing TLS stuff so it seems :(09:10
ccamachojaosorior, hey man! well... not really, now Im getting keystone issues as the apache is not working, the thing is that im like a potato in front of the laptop (cant focus because the flu)..09:11
shadowerjaosorior: I'll have some time in a bit. Will have a look09:11
ccamachobetween the headache the fever and the nose leaking.. got me covered..09:11
*** anbu_ has joined #tripleo09:11
jaosoriorccamacho: apache is not working? did you enable Gnocchi?09:12
ccamachopeople dont like to review the TLS patches :P  BTW the issues im getting are not related with the submissions09:12
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Fix broken rabbitmqctl commands when using ipv6  https://review.openstack.org/38856809:12
ccamachoIts just I dont know what im doing wrong..09:12
ccamachohave you recently tried to deploy locally master?09:13
jaosoriorccamacho: if you enable Gnocchi, it might break, because we need some selinux rules update that already merged in RHEL09:13
*** anbu has quit IRC09:13
pandaCI is a colander .. I cork a failure, another shows up immediately after :(09:13
jaosoriorccamacho: alright, is that issue in the overcloud or undercloud?09:13
jaosoriorpanda: what happened now? :O09:14
jaosoriorccamacho: if it's undercloud, that issue should be fixed with a recent commit. It was an issue with ceilometer.09:15
jaosoriorccamacho: if it's overcloud, it might be a selinux issue, and you might need to disable it for now. We need an update in CentOS09:15
ccamachojaosorior, I though t that dude, so I said f#*k it.. and re-installed everything.. now the endpoints are just not starting (httpd in the OC controller)09:15
jaosoriorif it's none of the above, I am yet to see it09:15
ccamachoso now im reading logs in the controller and I need to records some screencasts for the summit...09:16
ccamachoso..09:16
pandajaosorior: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-newton/250707b/logs/postci.txt.gz#_2016-10-19_08_15_38_00009:16
jaosoriorccamacho: can I get access to the node? I could check it out09:16
pandajaosorior: I think ceilometer patch caused some other issues09:16
jaosoriorpanda: the ceilometer patch was for the undercloud, not the overcloud09:17
pandaalso, I have liberty and tempest timing out, so no logs09:17
ccamachosure, just give me some mins (call with my manager)09:17
ccamachojaosorior ^09:17
jaosoriorccamacho: sure09:17
pandajaosorior: ah, yay! a new whole problem then09:17
*** zoliXXL is now known as zoli|wfh09:17
jaosoriorpanda: that might be the same issue ccamacho is seeing. issues with httpd09:17
ccamachojaosorior panda, yeahp same error09:19
jaosoriorccamacho: so you're seeing an upstream issue.09:19
jaosoriorcrap09:20
jaosoriorok, lets see what's up09:20
d0ugalSilly question: If I build and install a package locally, how do I have yum install the latest version from delorean or whever is setup?09:20
ccamachojaosorior yeah because I said, let me re-install all from scratch and then buuum... facepalm... another (new) issue...09:21
openstackgerritafazekas proposed openstack/python-tripleoclient: Disable encrypted volume in the tempest-deployer-input.conf  https://review.openstack.org/38486809:22
*** stendulker has joined #tripleo09:22
pandad0ugal: priority ?09:22
d0ugalpanda: ?09:22
jaosoriorany idea what this means?09:22
jaosoriorOct 19 11:04:50 overcloud-controller-1 systemd[1]: httpd.service start-pre operation timed out. Terminating.09:22
*** anbu_ has quit IRC09:25
d0ugalpanda: Yeah, I am hitting something with priorities that I don't understand :)09:25
*** anbu_ has joined #tripleo09:25
jaosoriorpanda: I can see that at some point the httpd services go up and haproxy marks them as such. But at some point httpd is killed or something, still not sure what happened there.09:25
*** anbu__ has joined #tripleo09:29
jaosoriorpanda: only log I see from httpd is that there is a SIGWINCH that shut down apache at some point... nothing else looks suspicious that I could find :/09:31
*** anbu_ has quit IRC09:32
pandaSIGWINCH ? ... signal on window size change ? not the first time I see it, and it makes it even more weird09:32
*** anbu__ has quit IRC09:34
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] remove rh grub workaround stuff  https://review.openstack.org/38858009:34
*** anbu has joined #tripleo09:34
jaosoriorpanda: the only suspicious thing is that I found these ones: Oct 19 10:43:19 overcloud-controller-1 python[15373]: ERROR:scss.ast:Function not found: twbs-font-path:109:35
jaosoriorwhich I think come from horizon09:35
jaosoriorcould it be that the horizon vhost is having issues and thus preventing the other vhosts to start?09:35
pandaI think I've seen scss.ast errors other times, an they were not fatal09:36
jaosoriorpanda: ok... will look at the audit log, maybe there's something there09:38
*** akrivoka has joined #tripleo09:40
jaosoriorpanda: interesting:  type=SERVICE_START msg=audit(1476864299.785:1925): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=httpd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'09:40
bandinigo 2409:40
bandiniops09:40
jaosoriorah09:40
jaosoriornevermind09:40
jaosoriorbandini: O_O09:41
bandini:)09:41
*** ohamada has quit IRC09:42
*** ohamada has joined #tripleo09:42
*** yamahata has quit IRC09:42
*** jaosorior has quit IRC09:43
*** jaosorior has joined #tripleo09:43
*** anbu has quit IRC09:44
*** anbu has joined #tripleo09:44
*** tobias_fiberdata has joined #tripleo09:46
*** hjensas has joined #tripleo09:46
*** hjensas has joined #tripleo09:46
akrivokajtomasek: so your deployment worked without post config?09:47
*** anbu has quit IRC09:47
jaosoriorpanda: I'm really suspicious of horizon though09:49
jaosoriorpanda: in system.journal I see a LOT of Oct 19 11:03:35 overcloud-controller-1 python[31338]: ERROR:scss.ast:Function not found: function-exists:109:49
jtomasekakrivoka: not entirely, I am trying to debug it but I am getting http 504 error both when trying to access horizon and when running any openstack cli command09:49
jaosoriorlike09:49
jaosoriora disturbing lot09:50
jtomasekakrivoka: but I think the endpoints are setup09:50
jtomasekakrivoka: at least openstack stack show has that info09:50
*** limao has quit IRC09:50
jtomasekakrivoka: I am trying to run another deployment09:50
akrivokajtomasek: and is it a newton installation? or latest?09:51
pandajaosorior: I'm prettty sure those errors can be ignored09:51
jtomasekakrivoka: latest09:51
akrivokajtomasek: another question, were you using DIB_INSTALLTYPE_puppet_modules=source when installing undercloud?09:52
jtomasekakrivoka: yes09:52
akrivokajtomasek: oh and how did you work around the syntax errors from yesterday?09:53
jtomasekakrivoka: that setup where I managed to deploy is from Friday last week09:53
jtomasekakrivoka: the other one I was still hitting the syntax error. I am doing another install today to see if it still occurs09:53
akrivokaah, I see09:54
flepiedhi derekh, do you know who has worked on building packages on the fly for tripleo CI jobs?09:54
jaosoriorpanda: alright.09:55
jaosoriorccamacho: let me know if I can poke on your deployment, if you have reproduced the issue would be good to take a look there.09:55
derekhflepied: it was me who originally wrote it but that was will over a year ago, you should find a few people in here familiar with it, shardy sshnaidm|away panda slagle bnemec would all be familiar with it09:58
*** andrey-mp has joined #tripleo09:58
andrey-mpHi! Does this bug https://bugzilla.redhat.com/show_bug.cgi?id=1221776 have related bug in launchpad? is it fixed somewhere?09:59
openstackbugzilla.redhat.com bug 1221776 in rhel-osp-director "nova migrate fails with ssh command failure" [High,Assigned] - Assigned to emacchi09:59
pandaflepied: what do you mean with "on the fly" ?09:59
pandaccamacho: you crated a bug somewhere for the errors on httpd ?10:00
ccamachohey panda, not yet, hitting another issue with tripleo.sh10:00
openstackgerritJulie Pichon proposed openstack/puppet-tripleo: Persist the UI config file  https://review.openstack.org/38859210:03
flepiedpanda: building packages from reviews10:04
*** hewbrocca-afk is now known as hewbrocca10:05
*** mgould|afk is now known as mgould10:08
pandaflepied: we are calling delorean to create packages on the fly in tripleO CI, setting up a local repo, and giving it an high priority10:09
flepiedpanda: yes i'm looking for this code10:13
pandaflepied: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L34710:14
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Unbound variable when executing tripleo.sh  https://review.openstack.org/38859410:14
jaosoriorderekh: can you check this out? ^^10:19
ccamachojaosorior, panda. trying to reproduce the httpd error ill let you know asap.. /me hoping not to hit more things :P10:20
*** pblaho has joined #tripleo10:20
jaosoriorccamacho: good luck10:20
ccamachojaja10:20
jaosoriorccamacho: I'm re-deploying everything from scratch, see if I can reproduce it10:22
ccamachojaosorior, let the force be with you10:23
jaosoriorccamacho: seems d0ugal had proposed a commit for the unbound variable https://review.openstack.org/#/c/388542/210:23
d0ugal:)10:23
ccamachod0ugal jaosorior, ACK :)10:24
jtomasekakrivoka: I am still getting the same error http://paste.openstack.org/show/586346/10:26
jtomasekakrivoka, jaosorior^ you've discussed this yesterday, what was the patch which was supposed to fix it?10:27
jaosoriorjtomasek: we are pinning puppetlabs-ntp IIRC. Since they no longer support puppet 310:27
jaosoriorjtomasek: that was another bug :D10:27
jtomasekjaosorior: ok, how do I fix that?10:27
jaosoriorjtomasek: good question. I don't know.10:27
*** cdearborn has joined #tripleo10:28
jtomasekjaosorior: I am going to file a bug for it, ok?10:28
jaosoriorjtomasek: sure10:28
derekhjaosorior: looking10:29
*** tobias-fiberdata has joined #tripleo10:31
*** tobias_fiberdata has quit IRC10:35
jaosoriormarios: hey dude, could you check this out https://review.openstack.org/#/c/327029/ ?10:39
*** fzdarsky is now known as fzdarsky|lunch10:40
mariosjaosorior: k10:42
*** phpcodemonkey has joined #tripleo10:44
*** zoli|wfh is now known as zoli|lunch10:46
*** abregman|afk is now known as abregman10:48
bandinimeh launchpad timing out here10:50
openstackgerritDougal Matthews proposed openstack/instack-undercloud: Only create the Mistral config env if it doesn't exist  https://review.openstack.org/38783210:50
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fetch the snmpd_readonly_user_password from the Mistral environment  https://review.openstack.org/38532310:52
*** phpcodemonkey has quit IRC10:52
openstackgerritDougal Matthews proposed openstack/tripleo-common: Fetch the snmpd_readonly_user_password from the Mistral environment  https://review.openstack.org/38532310:53
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Use Password Generation provided by the Mistral workflows  https://review.openstack.org/37554410:54
openstackgerritRyan Brady proposed openstack/tripleo-common: [WIP]Adds retry to plan workflows  https://review.openstack.org/38828510:56
jtomasekakrivoka, jaosorior: it should have been fixed: https://review.openstack.org/#/c/387098/ https://review.rdoproject.org/r/gitweb?p=rdoinfo.git;a=commitdiff;h=ecf4cecc0087b1595243af19d049783005f033ba10:59
jtomasekakrivoka, jaosorior: the pupetlabs-ntp package version I have on my undercloud is 4.2.0-0.20161017093645.93da3bd.el7.centos11:00
jtomasekakrivoka, jaosorior: I don't understand why I get the error11:00
*** tobias_fiberdata has joined #tripleo11:01
openstackgerritMerged openstack/puppet-tripleo: Include ::swift::config in Swift API and Storage roles  https://review.openstack.org/38372111:02
*** tobias-fiberdata has quit IRC11:04
akrivokajtomasek: I have puppet-ntp-4.2.0-0.20161017093645.93da3bd.el7.centos.noarch on a fresh install11:04
akrivokastill haven't tried a deploy tho11:04
jtomasekjaosorior, akrivoka: the code in /usr/share/openstack-puppet/modules/ntp/manifests/init.pp is correct, the error is in overcloud controller11:04
jtomasekakrivoka, jaosorior: so I assume it is a same problem but in overcloud image?11:05
*** limao has joined #tripleo11:06
*** limao_ has joined #tripleo11:08
*** thrash|g0ne is now known as thrash11:10
*** pkovar has quit IRC11:10
*** tobias-fiberdata has joined #tripleo11:10
*** limao has quit IRC11:11
*** stendulker has quit IRC11:14
*** tobias_fiberdata has quit IRC11:14
*** jcoufal has joined #tripleo11:26
*** snecklifter has quit IRC11:30
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Fix api_extensions_path in neutron-opencontrail environment  https://review.openstack.org/38862911:32
pandajaosorior: apparently apache2 officially uses SIGWINCH as graceful stop, because all the other signals are taken. For for esample you're running in foreground, and you resize the terminal, apache shuts down.11:32
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Fix api_extensions_path in neutron-opencontrail environment  https://review.openstack.org/38863111:33
jaosoriorpanda: lol, wtf11:33
jaosoriormarios: here's the t-h-t change https://review.openstack.org/#/c/366075/11:34
jistrjaosorior: thanks for the backport review, can you please look at the newton and liberty ones too https://review.openstack.org/388047  https://review.openstack.org/38863111:35
jistrthe newton one is ready to land11:35
mariosjaosorior: thanks added for tomorrow11:35
jaosoriormarios: thanks dude, appreciate it11:36
*** zoli|lunch is now known as zoli_gone-proxy11:36
jaosoriormarios: if you have extra time, this is where the tls-via-certmonger stuff is being tracked https://review.openstack.org/#/q/status:open+branch:master+topic:bp/tls-via-certmonger11:37
jaosoriormarios: but those two are the most crucial commits. The rest is a bit reppetitive stuff for enabling specific services.11:37
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-quickstart: Stop using deprecated network range  https://review.openstack.org/34344311:38
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/38863511:38
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart: make upgrade use the extras dependencies  https://review.openstack.org/38550311:43
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: TEST - DO NOT MERGE  https://review.openstack.org/38740511:43
mariosjaosorior: noted11:43
*** snecklifter has joined #tripleo11:44
jistrthanks jaosorior11:45
jaosoriorjistr: no biggie11:46
*** lucas-afk is now known as lucas-sick11:47
*** zoli_gone-proxy is now known as zoliXXL11:47
*** zoliXXL is now known as zoli|lunch11:47
*** saneax is now known as saneax-_-|AFK11:48
openstackgerritSven Anderson proposed openstack/puppet-tripleo: Implement Nova ec2api profile  https://review.openstack.org/33656211:48
*** rbrady-afk is now known as rbrady11:49
akrivokajtomasek: yeah I'm hitting the same issue again11:49
*** pkovar has joined #tripleo11:49
*** [1]cdearborn has joined #tripleo11:50
openstackgerritsebastien aime proposed openstack/tripleo-image-elements: fixed an error in os-collect-config if /{a,b} files exist  https://review.openstack.org/38864411:51
*** bfournie has quit IRC11:51
*** athomas has quit IRC11:51
*** abregman is now known as abregman|afk11:52
afazekaslooking for reviewers for this backport: https://review.openstack.org/#/c/387299/11:53
pandawat ? Oct 19 08:04:56 localhost python: ERROR:scss.compiler:Maximum number of supported selectors in Internet Explorer (4095) exceeded!11:56
openstackgerritMerged openstack/tripleo-quickstart: make upgrade use the extras dependencies  https://review.openstack.org/38550311:57
*** fzdarsky|lunch is now known as fzdarsky11:57
*** limao_ has quit IRC11:58
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart: Upgrade job dont need the overcloud workers settings  https://review.openstack.org/38821111:58
*** athomas has joined #tripleo12:00
*** noslzzp has quit IRC12:02
*** noslzzp has joined #tripleo12:03
*** abehl has joined #tripleo12:03
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Dont run gnocchi upgrade before converge  https://review.openstack.org/38864912:05
*** cdearborn has quit IRC12:05
*** dprince has joined #tripleo12:06
*** gfidente has joined #tripleo12:06
openstackgerritMerged openstack/tripleo-quickstart: Upgrade job dont need the overcloud workers settings  https://review.openstack.org/38821112:08
*** sudipto has joined #tripleo12:11
*** sudipto_ has joined #tripleo12:11
*** maticue has joined #tripleo12:11
*** skramaja_ is now known as skramaja12:12
*** abehl has quit IRC12:16
*** morazi has joined #tripleo12:16
*** zoli|lunch is now known as zoli|wfh12:16
*** rhallisey has joined #tripleo12:19
*** links has joined #tripleo12:20
openstackgerritMerged openstack/tripleo-heat-templates: Fix api_extensions_path in neutron-opencontrail environment  https://review.openstack.org/38804712:21
*** jayg|g0n3 is now known as jayg12:24
*** morazi has quit IRC12:24
*** [1]cdearborn has quit IRC12:26
*** trown|outtypewww is now known as trown12:26
bandinipanda: I have seen that one too12:26
*** hewbrocca is now known as hewbrocca-afk12:29
openstackgerritMartin André proposed openstack/tripleo-heat-templates: [WIP] Testing adding metadata to roles  https://review.openstack.org/38647212:32
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Have docker start script honor configuration  https://review.openstack.org/36613812:32
openstackgerritMartin André proposed openstack/tripleo-heat-templates: WIP: Containerized Services for Composable Roles  https://review.openstack.org/33065912:32
openstackgerritMartin André proposed openstack/tripleo-heat-templates: Bind mount files to run DiD in latest atomic host  https://review.openstack.org/34721812:32
openstackgerritMerged openstack-infra/tripleo-ci: Set TRIPLEO_ROOT before it is first used  https://review.openstack.org/38854212:33
*** abregman|afk is now known as abregman12:33
*** bfournie has joined #tripleo12:36
*** morazi has joined #tripleo12:37
*** rlandy has joined #tripleo12:38
pandabandini: which one ?12:39
akrivokajtomasek: jaosorior: here's the bug for the puppet syntax error issue https://bugs.launchpad.net/tripleo/+bug/163371312:40
openstackLaunchpad bug 1633713 in tripleo "puppet-ntp is breaking ci" [Critical,Fix released]12:40
*** afazekas has quit IRC12:41
akrivokajtomasek: jaosorior the fix is merged but the problem still remains, I guess I should reopen the bug12:41
*** afazekas has joined #tripleo12:42
shadowerd0ugal: any idea how to move this forward? https://review.openstack.org/#/c/379348/12:42
bandinipanda: Maximum number of supported selectors in Internet Explorer ...12:43
shadowerd0ugal: it got +A before its dependency merged but that has happened now and this one is still waiting12:43
*** myoung|alt is now known as myoung12:43
d0ugalshadower: Yeah, I think that was the problem. Removing the +A and re-adding usually solves it.12:44
*** jaosorior has quit IRC12:44
shadowerd0ugal: ah, thanks!12:44
d0ugalshadower: I think "reverify" might also do the trick if that doesn't12:44
*** jaosorior has joined #tripleo12:44
d0ugalshadower: but it seems jenkins is already on it12:44
shadoweryeah12:44
*** dciabrin has quit IRC12:44
openstackgerritDougal Matthews proposed openstack/tripleo-docs: Remove the replace controller docs  https://review.openstack.org/38146912:46
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Fix the rabbitmq/redis pacemaker resource timeouts on updates  https://review.openstack.org/38867612:46
honzawhen using the cli to deploy, it's complaining about a missing tripleo.undercloud-config --- is that new?  am i missing some config?12:47
d0ugalhonza: What version are you running?12:47
honzad0ugal: latest master12:47
d0ugalhonza: This is a new thing, it had a slightly rough landing12:47
honzad0ugal: of tripleo-common --- is that what you mean?12:47
jpichhonza: Of aaall the things :-)12:47
d0ugalhonza: Yeah, but also instack-undercloud is impacted12:48
honzad0ugal: jpich: YISS I'm not crazy12:48
honzad0ugal: so it's been resolved, and i just need all the new things?12:48
*** jpena is now known as jpena|lunch12:48
d0ugalhonza: Yeah, only just solved, so it might depend on package promotion?12:49
d0ugalhonza: https://review.openstack.org/#/q/I94428d1deb000c65a1c0266d01f660b76d4a3ee5,n,z12:49
d0ugalhonza: That resolves it, it isn't resolved in Newton yet12:49
honzad0ugal: thanks12:49
d0ugalhonza: instack-undercloud is a depends-on which is why that is also impacted, and then you need to run the undercloud install again if you are going to update12:50
honzad0ugal: hm, that commit is in my tree12:50
d0ugalhonza: Can you show me `mistral environment-list`?12:51
*** amoralej is now known as amoralej|lunch12:52
d0ugalhonza: it might be that deleting the default plan and re-creating it works.12:52
d0ugalhonza: it depends how far you got really...12:52
*** Goneri has joined #tripleo12:53
honzad0ugal: http://paste.openstack.org/show/586372/12:53
d0ugalhonza: lol @ ohhai12:54
honza:D12:54
d0ugalhonza: Okay, so, do you have this commit? https://review.openstack.org/#/q/I186217fd0e1125519149763e610d3efdff583388,n,z12:54
honzamakes it easy to grep for :)12:54
*** egafford has quit IRC12:55
d0ugalhonza: I use "overcast" which is just confusing because at a glance it looks like the default12:55
*** tzumainn has joined #tripleo12:56
d0ugalhonza: it seems like your tripleo-common is newer than your instack-undercloud12:56
d0ugaland that would cause an issue12:56
honzad0ugal: not sure where to look for the instack repo12:56
honzad0ugal: sweet, thanks --- upgrading now12:56
d0ugalhonza: >>> import instack_undercloud12:57
d0ugal>>> instack_undercloud.__file__12:57
d0ugal'/usr/lib/python2.7/site-packages/instack_undercloud/__init__.pyc'12:57
d0ugalYou could just look at the files in there I guess12:57
d0ugalhonza: after you update instack-undercloud, (yum would probably do it?) you will need to run `openstack undercloud install` again.12:58
d0ugalhonza: or, if you want a hacky workaround I can show you how to do that too :)12:58
*** egafford has joined #tripleo12:58
honzad0ugal: yep12:58
jpichYou might need to update your delorean repo to point to the latest12:58
honzayes, using the upstream docs12:59
*** links has quit IRC12:59
*** sudipto_ has quit IRC13:01
*** sudipto has quit IRC13:01
*** remix_tj has quit IRC13:01
*** remix_tj has joined #tripleo13:01
d0ugalhonza: the hacky fix: http://paste.openstack.org/show/586375/13:02
d0ugalhonza: you need to change the password to whatever you get from: sudo hiera snmpd_readonly_user_password13:02
d0ugalhonza: that is all the instack-undercloud code does.13:03
*** snecklifter has quit IRC13:05
*** jprovazn has quit IRC13:06
openstackgerritMerged openstack/tripleo-quickstart: Remove images/publish role  https://review.openstack.org/38818913:06
*** dciabrin has joined #tripleo13:06
*** cylopez has quit IRC13:08
*** jcoufal_ has joined #tripleo13:09
*** StevenK has quit IRC13:09
rbradyjtomasek: ping13:09
*** florianf has quit IRC13:10
rbradyjtomasek: just stepped through a deployment with a debugger and confirmed the _deploy_postconfig method didn't call _keystone_init at all13:10
*** fultonj has joined #tripleo13:11
*** cylopez has joined #tripleo13:11
jtomasekrbrady: nice13:11
openstackgerritMerged openstack/tripleo-quickstart: Generalize check for release name for OVB stack clean up  https://review.openstack.org/38662413:11
*** jcoufal has quit IRC13:12
*** lblanchard has joined #tripleo13:12
jtomasekakrivoka: the bug occurs in overcloud controller, so I wonder that maybe we just need latest images. I am not sure how recent images are used when following upstream docs13:14
akrivokajtomasek: I built images following the docs13:15
*** StevenK has joined #tripleo13:15
rbradyjtomasek: is there something similar to the overcloudrc file the UI needs?13:16
jtomasekakrivoka: hm, me too, I assume that repo used is the latest we can get http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-master-tripleo/13:16
*** jaosorior is now known as jaosorior_brb13:17
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Remove double tcp_listen_options entries for rabbit  https://review.openstack.org/38869513:18
jtomasekrbrady: we're providing similar information via GUI, but I think it would be good to make sure overcloudrc is created regardless of what is used to deploy, so if user swiches to CLI he has it available13:18
jtomasekrbrady, d0ugal: what is this? https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L123213:19
d0ugaljtomasek: a comment? :) it seems to just be the logic which controls if the postconfig code is ran13:20
jaosorior_brbIf someone has time, could you check out these two commits https://review.openstack.org/#/c/366075/  https://review.openstack.org/#/c/327029/  ??13:20
*** social has quit IRC13:20
jtomasekd0ugal: oops, this one https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L123013:21
*** dsariel has joined #tripleo13:21
d0ugaljtomasek: oh, I think you can skip that.13:21
jtomasekrbrady, d0ugal: that is another thing we need to create post-deployment imo13:21
jtomasekd0ugal: ok:)13:21
*** snecklifter has joined #tripleo13:21
d0ugaljtomasek: I mean, it might be nice to do later, but isn't required AFAIK13:21
d0ugaljtomasek: it is only useful if you are going to start triggering tempest13:22
jtomasekd0ugal: it is some config file for tempest overcloud testing, tight?13:22
jtomasek*right13:22
d0ugaljtomasek: Yeah, exactly13:22
d0ugaljtomasek: and I'd guess that is out of scope for the GUI at the moment13:22
*** florianf has joined #tripleo13:23
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: move gnocchi upgrade to controller upgrade step 3  https://review.openstack.org/38864913:23
jtomasekd0ugal: well, since we have some post-deployment validations, I could see tempest being one of them probably. But it is deffinitely not anything we'd need in newton13:23
d0ugaljtomasek: yeah, that's what I was thinking.13:23
jtomasekrbrady: same as overcloudrc13:23
jtomasekI'll note it to summit session etherpad13:24
openstackgerritMerged openstack/puppet-tripleo: Fix broken rabbitmqctl commands when using ipv6  https://review.openstack.org/38852913:25
*** rlandy has quit IRC13:27
*** rlandy has joined #tripleo13:28
*** rlandy has quit IRC13:29
*** rlandy has joined #tripleo13:29
*** cylopez has quit IRC13:30
*** jeckersb_gone is now known as jeckersb13:31
*** cylopez has joined #tripleo13:33
*** nyechiel has joined #tripleo13:34
*** ohamada_ has joined #tripleo13:35
*** ohamada has quit IRC13:35
*** cylopez has quit IRC13:36
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Create directories with root  https://review.openstack.org/38489213:37
*** ohamada__ has joined #tripleo13:39
*** ohamada_ has quit IRC13:39
*** snecklifter has quit IRC13:42
*** zoli|wfh is now known as zoli|brb13:43
*** abregman is now known as abregman|afk13:44
*** akuznetsov has joined #tripleo13:46
*** shardy has quit IRC13:48
trozetshardy: hi13:48
trozetEmilienM: do you mind setting workflow on https://review.openstack.org/#/c/386737/ ?13:49
pandatrozet: IIUC they're both on PTO13:51
trozetpanda: ah, how about you dsneddon ? :)13:52
* panda sighs13:53
*** limao has joined #tripleo13:53
pandaI'm unable to reproduce failure in periodic newton13:53
trozetpanda: why the sigh?13:53
pandadeploy works locally13:53
trozetpanda: what is the failure?13:54
d0ugalLooking for reviews here if anyone has time: https://review.openstack.org/#/c/375544/13:55
d0ugalFinally kills the password generation in tripleoclient! ^^13:55
d0ugalAll green and all the depends-on have merged in master.13:55
openstackgerritMerged openstack/puppet-tripleo: Add port to rabbitmq node ip list  https://review.openstack.org/38676113:55
openstackgerritMerged openstack/tripleo-heat-templates: glance_multiple_locations when NovaEnableRbdBackend=true  https://review.openstack.org/38729913:55
openstackgerritMerged openstack/tripleo-common: Configure run-validation to use the custom output  https://review.openstack.org/37934813:55
pandatrozet: http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-newton/250707b/logs/postci.txt.gz#_2016-10-19_08_15_38_00013:55
pandatrozet: but apparently is transient13:56
d0ugalthrash, EmilienM: can you re-review https://review.openstack.org/#/c/375544/ ?13:56
*** amoralej|lunch is now known as amoralej13:57
trozetpanda: I haven't seen this particular error before.  But if it is reproducible I'm sure we will hit it in OPNFV fairly soon.  Will ping you if I see it13:58
*** flepied has quit IRC14:00
*** flepied has joined #tripleo14:01
thrashd0ugal: ack14:02
thrashd0ugal: EmilienM is on PTO. Not sure if he's actually here.14:02
*** jpena|lunch is now known as jpena14:02
trozetd0ugal: in your patch tripleoclient still configuring keystone?14:03
*** [1]cdearborn has joined #tripleo14:03
d0ugalthrash: he was earlier, but I guess he has left now.14:03
d0ugaltrozet: the password removal? I think the keystone configuration is deprecated, but the code is still there.14:04
*** limao has quit IRC14:04
*** limao has joined #tripleo14:05
trozetd0ugal: ya. The configuration of keystone endpoints was done post heat stack deployment by tripleoclient over public VIP, i was just wondering if that has been moved into the heat side yet14:05
d0ugaltrozet: I thought it was done in puppet, but I am not sure. I think rbrady and akrivoka were looking into this today tho'14:06
pandatrozet: mh, things are actually worse on local build ... heat says deployment complete, but cluster failed on a redis problem. So now I have two problems to solve instead of one. yay!14:07
rbradytrozet: the code is still in tripleoclient but deprecated and skipped.  message printed says it happens in puppet, but I didn't go looking in puppet for where it happens14:08
trozetpanda: we went through a month of fixing transient errors in stable/mitaka to get HA deployments to pass 100% of the time - i feel your pain14:08
trozetpanda: one thing we found is bringing up all the DBs in the same step caused tons of cpu resource contention14:09
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements  https://review.openstack.org/38863514:09
trozetpanda: so we moved mongo, and sql around in different steps14:09
pandaI think I found a source of problems, again gnocchi is eating all the CPU and causing various timeout issues14:09
trozetcould be the same thing with redis14:09
*** ramishra has quit IRC14:10
panda2016-10-19 13:41:03.766 10873 ERROR cotyledon ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)14:10
pandathis is gnocchi metricd log14:10
trozetpanda: yeah that is usually the problem very high cpu contention, we used to see mysql client failures in puppet cause sql coudlnt respond14:10
trozetrbrady: thanks14:11
pandatrozet: mh, this is actually an effect not the cause, tooz is trying to access redis that is down.14:11
pandatrozet: anything you remember I could look at for the problem you mentioned ?14:11
pandai see this in 22475:M 19 Oct 13:04:56.776 # Opening Unix socket: bind: Permission denied14:13
pandaredis.log ..14:13
trozetpanda: https://bugs.launchpad.net/tripleo/+bug/158527514:14
openstackLaunchpad bug 1585275 in tripleo "SQL errors on undercloud/overcloud cause HA deployments to fail" [High,Triaged]14:14
*** nyechiel has quit IRC14:14
trozetpanda: im pretty sure we will see the same failures in newton, but we haven't gotten the chance to test it yet14:14
trozetpanda: but ive already seen failures in tripleoci logs pointing to what looks like the same cause14:15
*** akuznetsov has quit IRC14:15
*** ayoung has quit IRC14:15
*** limao_ has joined #tripleo14:16
*** ramishra has joined #tripleo14:17
*** oshvartz has joined #tripleo14:19
*** andrey-mp has left #tripleo14:20
*** limao has quit IRC14:20
*** hjensas has quit IRC14:23
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for gnocchi  https://review.openstack.org/37764514:24
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for ceilometer  https://review.openstack.org/37764414:24
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for aodh  https://review.openstack.org/36545714:24
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702914:24
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable internal TLS for MySQL  https://review.openstack.org/37847114:24
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for Nova API  https://review.openstack.org/38849514:24
jaosorior_brbhey guys, if someone has time to review that ^^ I would really appreciate it. It's getting pretty hard to maintain all those patches and the keystone one is already in patchset 5414:25
*** jaosorior_brb is now known as jaosorior14:25
*** zoli|brb is now known as zoli|wfh14:25
*** limao_ has quit IRC14:27
*** limao has joined #tripleo14:27
*** jtomasek_ has joined #tripleo14:28
*** jistr is now known as jistr|call14:28
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: move gnocchi upgrade to controller upgrade step 3  https://review.openstack.org/38864914:29
*** hjensas has joined #tripleo14:29
*** hjensas has joined #tripleo14:29
* mwhahaha waves to trozet14:31
trozetah!14:31
*** saneax-_-|AFK is now known as saneax14:36
*** jprovazn has joined #tripleo14:38
*** limao has quit IRC14:38
*** limao has joined #tripleo14:39
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for gnocchi  https://review.openstack.org/37764514:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for ceilometer  https://review.openstack.org/37764414:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for aodh  https://review.openstack.org/36545714:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702914:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable internal TLS for MySQL  https://review.openstack.org/37847114:43
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for Nova API  https://review.openstack.org/38849514:43
jaosoriormwhahaha: thanks for the feedback dude!14:44
mwhahahanp14:45
*** hjensas has quit IRC14:45
openstackgerritRyan Brady proposed openstack/python-tripleoclient: Use workflow for overcloud node delete  https://review.openstack.org/38373114:47
mwhahahajaosorior: you missed one :o14:47
*** yamahata has joined #tripleo14:49
tdasilvatrown: hello, ran into a different problem with running quickstart to install with master image: http://paste.fedoraproject.org/455702/14768885/14:50
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable internal TLS for MySQL  https://review.openstack.org/37847114:50
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for Nova API  https://review.openstack.org/38849514:50
jaosoriormwhahaha: done ^^14:50
*** limao has quit IRC14:50
*** jaosorior has quit IRC14:50
*** jtomasek_ has quit IRC14:50
*** limao has joined #tripleo14:51
trowntdasilva: so second attempt worked? (second deploy in that paste)14:51
tdasilvatrown: no, it just stops there14:51
trowntdasilva: I wonder if we are seeing odd behavior because of previous runs on the same virthost... there is a "--teardown all" option for quickstart that will destroy everything before starting to build it up that may be helpful14:53
*** skramaja has quit IRC14:53
tdasilvatrown: i always run with '--teardown all'  :/14:53
*** yamahata has quit IRC14:54
tdasilvatrown: my last run was: ./quickstart.sh --no-clone --release master --teardown all $VIRTHOST14:54
*** yamahata has joined #tripleo14:54
trowntdasilva: what kind of hardware are you working with for the virthost?14:54
tdasilvatrown: it's a pretty beefy hardware with intel xeon and 32G ram14:56
tdasilvatrown: is it possible that 32G is not enough? i'm only deploying two overcloud nodes (1 controller, 1 compute)14:57
trownhmm, that is pretty much what I run on14:57
tdasilvatrown: so what I thought was strange this time around was that message about the overcloud container in swift?14:58
*** dsariel has quit IRC14:58
tdasilvatrown: basically i followed the steps here: http://tripleo.org/basic_deployment/basic_deployment_cli.html14:59
trowntdasilva: ya, that is what made me ask about hardware... there is an open bug about that, but I think it only reproduced under load14:59
tdasilvaand when I got to overcloud deploy, it stopped with that error, but that was a first time i saw that14:59
tdasilvaoh, ok14:59
trowntdasilva: oh... we need to document the low-memory env heat environment14:59
trowntdasilva: I bet that is your problem15:00
*** ealcaniz__ has quit IRC15:01
tdasilvatrown: this is currently on the VIRTHOST: http://paste.fedoraproject.org/455710/68892801/15:01
trownin CI, we use this https://github.com/redhat-openstack/ansible-role-tripleo-overcloud/blob/fdaeeedb1cc54122eed5fe3adc82d86ab911f0dc/tasks/pre-deploy.yml#L22-L2615:01
*** limao has quit IRC15:02
trowntdasilva: ya,  passing in that heat environment to the overcloud deploy will lower the number of service workers15:02
*** limao has joined #tripleo15:02
trown-e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml15:03
tdasilvatrying now15:03
trownhttps://github.com/openstack/tripleo-heat-templates/blob/master/environments/low-memory-usage.yaml15:04
*** hjensas has joined #tripleo15:05
*** hjensas has joined #tripleo15:05
d0ugalgfidente: Hey, you around?15:05
d0ugalgfidente: actually, nevermind - I'll catch you about it tomorrow probably. I needed to do a recheck.15:06
tdasilvatrown: no luck, same issue: http://paste.fedoraproject.org/455712/76889615/15:07
trowntdasilva: does `openstack stack list` show an overcloud stack?15:08
* trown thinks we may have cruft we need to clean up first15:08
tdasilvanothing15:08
*** b00tcat has quit IRC15:11
*** pradk has quit IRC15:12
*** jistr|call is now known as jistr15:12
*** tremble has quit IRC15:13
*** limao has quit IRC15:13
*** Guest90211 has quit IRC15:13
*** limao has joined #tripleo15:13
*** pcaruana has quit IRC15:14
*** ayoung has joined #tripleo15:15
*** rcernin has quit IRC15:16
*** pradk has joined #tripleo15:17
*** limao has quit IRC15:19
*** hjensas has quit IRC15:20
*** bana_k has joined #tripleo15:24
mandreEmilienM: I'm actually surprised to see the experimental CI job for containers was green in https://review.openstack.org/#/c/288915/59, was it really tested?15:25
*** ebarrera has quit IRC15:25
openstackgerritDeepak proposed openstack/python-tripleoclient: Fix the author and author-email in setup.cfg file  https://review.openstack.org/38876315:26
jristanyone else good with puppet?15:34
*** hjensas has joined #tripleo15:35
*** hjensas has joined #tripleo15:35
mwhahahajrist: i have been known to write puppet, what's up?15:37
jristakrivoka: ^15:37
akrivokamwhahaha: hi, wondering if you might have a clue about https://bugs.launchpad.net/tripleo/+bug/163371315:38
openstackLaunchpad bug 1633713 in tripleo "puppet-ntp is breaking ci" [Critical,Triaged]15:38
mwhahahaakrivoka: yea the ntp module updated to puppet4 only15:38
akrivokamwhahaha: I'm still hitting that bug even though I have the correct version of the ntp package15:38
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Add bindep support  https://review.openstack.org/34911415:38
mwhahahaso you need an older version of ntp15:38
akrivokaon my undercloud15:38
mwhahahai just hit it as well so i'm looking at it15:39
mwhahahamy undercloud was fine but my overcloud had the newer package15:39
mwhahahawhat version do  you have?15:39
akrivokamwhahaha: I have 4.2.0, is that not the correct one?15:39
akrivokamwhahaha: that's on the undercloud15:39
*** jistr is now known as jistr|biab15:39
openstackgerritMerged openstack/tripleo-heat-templates: Fix api_extensions_path in neutron-opencontrail environment  https://review.openstack.org/38862915:39
akrivokamwhahaha: yes, the overcloud seems to be the problem, but I don't know how to debug that15:39
*** bana_k has quit IRC15:40
mwhahahaakrivoka: i'm looking into it now, not exactly sure either15:41
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Don't hide output from get-node.sh  https://review.openstack.org/35022915:44
*** rbrady is now known as rbrady-run15:45
*** links has joined #tripleo15:46
trownakrivoka: mwhahaha is overcloud using puppet from source instead of packages?15:46
mwhahahathat's a possibility15:47
mwhahahawhich if that's the case we'd need to pin to the 4.2.x branch15:47
trownI know we have done that before, but looking for where15:47
akrivokatrown: I think so,  I did export DIB_INSTALLTYPE_puppet_modules=source before building overcloud images15:48
trownakrivoka: ah ya that would do it15:48
trownthere is another DIB var that can be used to pin15:48
trownCI is using packages I guess then?15:49
mwhahahayea i think it is15:49
mwhahahawhich is why the rdo pin solved ci15:49
*** hjensas has quit IRC15:50
trownakrivoka: any reason to use puppet modules from source? packages are pretty much equivalent these days since they are built by DLRN15:50
akrivokatrown: I did that because the undercloud installation was failing for me with packages15:51
akrivokaI guess I could try rebuilding images with packages instead of source and retrying the deploy that way15:51
mwhahahayou could  try export DIB_REPOREF_puppetlabs_ntp=4.2.x15:52
*** panda is now known as panda|bbl15:52
mwhahahaat least that appears to be the syntax, https://review.openstack.org/#/c/184844/3/elements/stable-puppet-dependencies/environment.d/01-puppet-stable-deps.sh15:52
trownakrivoka: ya, or if you dont need anything custom on the images you could get the tripleo-ci produced ones http://66.187.229.139/builds/current-tripleo/15:52
trownthats the var I was thinking of15:53
mwhahahathough we probably should do that until we switch to puppet415:53
mwhahahai'll test it out and submit a patch if it works :D15:53
trownyep, given that is broken otherwise15:53
akrivokatrown: ok, let me try with the ci images first15:53
*** rwsu has quit IRC15:56
*** kjw3 has joined #tripleo15:56
*** hjensas has joined #tripleo15:56
*** hjensas has joined #tripleo15:56
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: growroot: Add SUSE package mappings  https://review.openstack.org/38878015:57
*** ccamacho is now known as ccamacho|afk15:57
*** penick has joined #tripleo16:00
*** hjensas has quit IRC16:03
*** hjensas has joined #tripleo16:04
*** hjensas has quit IRC16:04
*** hjensas has joined #tripleo16:04
*** jistr|biab is now known as jistr16:05
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf  https://review.openstack.org/37708816:06
*** ayoung has quit IRC16:06
*** ebarrera has joined #tripleo16:10
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf  https://review.openstack.org/37708816:16
*** jlinkes has quit IRC16:17
*** hjensas has quit IRC16:19
openstackgerritFrederic Lepied proposed openstack-infra/tripleo-ci: Allow to use an RDO review to fix a packaging issue  https://review.openstack.org/38879116:20
*** rasca has quit IRC16:20
*** mcornea has quit IRC16:23
*** links has quit IRC16:31
dtrainordciabrin, re: galera chat last night, thanks for the background info in https://bugzilla.redhat.com/show_bug.cgi?id=138183616:32
openstackbugzilla.redhat.com bug 1381836 in resource-agents "galera resource agent cannot bootstrap cluster when galera and pacemaker nodes' name differ" [Medium,New] - Assigned to dciabrin16:32
*** jpich has quit IRC16:34
dciabrindtrainor, hey glad you found the bz, forgot to tell you about that16:35
openstackgerritMerged openstack/tripleo-heat-templates: Fix api_extensions_path in neutron-opencontrail environment  https://review.openstack.org/38863116:35
dtrainorno worries!16:35
*** dmacpher is now known as dmacpher-afk16:37
*** fzdarsky is now known as fzdarsky|afk16:38
*** numans has quit IRC16:40
*** jdob has joined #tripleo16:41
*** jdob has left #tripleo16:41
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart: Remove publish from build-images playbook  https://review.openstack.org/38880416:43
*** ohamada__ has quit IRC16:43
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart: Remove publish from build-images playbook  https://review.openstack.org/38880416:44
*** saneax is now known as saneax-_-|AFK16:44
*** masco has quit IRC16:47
*** pcaruana has joined #tripleo16:51
*** flepied has quit IRC16:53
*** lmiccini has quit IRC16:54
*** derekh has quit IRC16:54
*** ebarrera has quit IRC16:56
openstackgerritDan Trainor proposed openstack/tripleo-ui: Make sample file use %(tenant_id)s macro  https://review.openstack.org/38810116:57
*** yamahata has quit IRC16:59
*** hjensas has joined #tripleo17:00
*** hjensas has joined #tripleo17:00
*** zoli|wfh is now known as zoli|gone17:01
*** [1]cdearborn is now known as cdearborn17:01
*** zoli|gone is now known as zoli_gone-proxy17:02
*** amoralej is now known as amoralej|off17:04
*** owalsh has quit IRC17:09
akrivokamany thanks mwhahaha and trown - using the pre built images did the trick17:10
*** athomas has quit IRC17:10
trownnice!17:10
jtomasekakrivoka: I am at 99%:)17:10
akrivokajtomasek: yeah kinda takes forever but it gets there eventually :)17:10
jtomasekakrivoka: I rebuilt images with export DIB_INSTALLTYPE_puppet_modules=package17:11
*** owalsh has joined #tripleo17:11
akrivokaI'll put the workaround in the bug17:11
jtomasekakrivoka: can you login to controller and run keystone endpoint list?17:11
jtomasekakrivoka: err ssh into controller...17:12
akrivokasure, just a sec17:12
*** michapma_alt has quit IRC17:13
*** bana_k has joined #tripleo17:13
akrivokajtomasek: Missing value auth-url required for auth plugin password17:13
akrivokajtomasek: how do I auth against overcloud?17:15
*** owalsh_ has joined #tripleo17:15
jtomasekakrivoka: I did openstack --os-username admin --os-password <the password you see at deployment success screen> --os-auth-url http://<ip from success screen>:5000/v2.017:16
*** pkovar has quit IRC17:18
*** owalsh has quit IRC17:18
*** jcoufal_ is now known as jcoufal17:18
akrivokajtomasek: https://paste.fedoraproject.org/455783/76897552/17:19
jtomasekakrivoka: beautiful! can you try to curl horizon? http://192.0.2.13:80/dashboard17:20
*** owalsh_ is now known as owalsh17:20
*** yamahata has joined #tripleo17:20
jtomasekakrivoka: argh, I am still at 99% hah17:20
trozetdsneddon, dprince: do you mind setting workflow +1 on https://review.openstack.org/#/c/386737/ please17:21
akrivokajtomasek: curl http://192.0.2.13:80/dashboard is not responding17:22
*** trown is now known as trown|lunch17:22
dsneddontrozet, Done17:22
jtomasekakrivoka: ok, 'openstack stack show overcloud' will give you a bunch of outputs, you can find horizon url there. But I am not sure what is the state of Horizon in overcloud tbh17:23
akrivokajtomasek: actually it's there :) (it was just redirect to login page)17:26
trozetdsneddon: thanks17:26
jtomasekakrivoka: nice17:26
jtomasekakrivoka: btw. how did the post-deployment validations end up?17:26
jtomasekakrivoka: succeeded?17:26
akrivokajtomasek: most failed17:27
akrivokajtomasek: ansible returned "Failed to connect to the host via ssh."17:27
jtomasekprobably valuable info for shadower ^17:28
shadoweroooh17:29
shadowersadface17:29
* jtomasek senses something wrong with ssh keys workflow17:29
shadowerakrivoka: can you try to SSH to one of the overcloud nodes directly?17:30
akrivokashadower: yeah, I can do that17:30
shadowerakrivoka: also, can you try some of the undercloud validations? E.g. undercloud-ram? That should not rely on SSH keys17:31
shadowerand would help us determine whether it's really an SSH issue or something with ansible/mistral17:32
jtomasekshadower: only post-deployment ones are failing with that error17:33
jtomasekshadower: e.g. Check the number of OpenStack processes on undercloud succeeds17:33
akrivokashadower: The RAM on the undercloud node is 16048 MB, the minimal recommended value is 16384 MB17:33
akrivoka:)17:34
shadoweroh, right, because the other ones were  run earlier17:34
jtomasekshadower: we can re-run17:34
shadowerakrivoka: lol, I'm really sorry17:34
akrivokaboohoo17:34
*** tiswanso has joined #tripleo17:34
jtomasekakrivoka, shadower: haha problem is that even though you set the variable to 16384, the node ends up with less:)17:35
akrivokathat is... not very useful.17:36
shadowerjtomasek: really? That's a little strange17:36
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf  https://review.openstack.org/37708817:36
jtomasekit is17:36
shadowerjtomasek: btw do you see the same problem in your deployment?17:37
jtomasekshadower: yes17:37
shadowerk17:37
*** mgould is now known as mgould|afk17:37
openstackgerritBen Nemec proposed openstack/python-tripleoclient: Remove osc boilerplate from the readme  https://review.openstack.org/38882217:38
jtomasekshadower: I re-run the pre-deployment one which succeeded previously and it still spins so I expect that it will end up with the same error17:38
*** pblaho has quit IRC17:38
jtomasekshadower: no, it succeeded again17:38
jtomasekshadower: just my 6gb controller is maxed out and totally slow:D17:39
shadowerright17:39
akrivokaI have no flavors or images on overcloud17:39
akrivokais that normal?17:39
akrivokahow can I launch an instance?17:39
*** rbrady-run is now known as rbrady17:40
jtomasekthat is normal imo17:40
shadowerjtomasek, akrivoka: just on a hunch: can you run this on the undercloud? mistral execution-create tripleo.validations.v1.copy_ssh_key17:40
shadowerwith source stackrc obviously17:40
openstackgerritBen Nemec proposed openstack/instack-undercloud: Remove instack-create-overcloudrc  https://review.openstack.org/38882317:40
shadowerthat's what should set up the SSH keys17:40
jtomasekshadower: I'll try in a moment17:41
shadowercheers17:41
shadowerin the meantime, I'll try to run the validations via mistral on my deployment (haven't tried that yet on this new one)17:41
*** jpena is now known as jpena|off17:42
shadowerjtomasek, akrivoka: okay I've got the same issue in my deployment17:44
shadowerwhich is good -- will be easier to debug17:44
jtomasekakrivoka: my service catalog is empty, noooo :D17:44
*** ebarrera has joined #tripleo17:45
akrivokajtomasek: did you include --tenant-name=admin in your command17:45
*** owalsh has quit IRC17:45
akrivokajtomasek: it was empty for me when I missed that part17:45
jtomasekakrivoka: that must be it17:45
*** paramite has quit IRC17:45
jtomasekakrivoka: bingo, thanks17:46
*** newmember has quit IRC17:47
akrivokajtomasek: what would be the thing we need to check to verify that all the post-deployment config is in place?17:47
akrivokajtomasek: apart from keystone endpoint being there17:48
jtomasekakrivoka: endpoint list :)17:48
akrivokajtomasek: that's it?17:48
jtomasekakrivoka: that should be all, yeah17:48
akrivoka\o/17:48
* akrivoka dropmic17:48
jtomasekakrivoka: in addition, CLI creates overcloudrc and prepares tempest config, but those are bonus things IMHO. If we want to get those done too we'll need to add additional post-deployment workflow. I've added it to summit session etherpad17:49
*** chandankumar has quit IRC17:50
*** yamahata has quit IRC17:50
shadowerjtomasek, akrivoka: okay I have a local repro. Are you fine if I have a look at it tomorrow? I'm pretty braindead at this point17:51
shadowerI expect you two must be too17:51
*** ebarrera has quit IRC17:51
shadower(for the vorecloud validations)17:51
akrivokashadower: fine with me, of course17:51
shadowerokay cool, see ya tomorrow17:51
jtomasekshadower: yeah, sure... I tried to run that workflow and it seems to run for 5 minutes already17:51
shadowerjtomasek: yep, same here.  I'll investigate in the morning17:51
jtomasekshadower: great, thanks all17:52
trozetdsneddon: how about this one https://review.openstack.org/#/c/386820/17:54
shadowerjtomasek: hm, the copy_ssh_keys workflow fails. Which would certainly produce this behaviour17:55
bnemecOMG, I just googled vore.17:55
bnemecDon't do that.17:55
dsneddontrozet, I'll give it a review, but even if I +2 it we'll need some more cores to review and give at least one more +2.17:55
bnemec(I was looking for https://en.wikipedia.org/wiki/Vore_Buffalo_Jump )17:55
*** owalsh has joined #tripleo17:56
dsneddontrozet, Gave it my +2, but I do want someone else to sanity check17:56
*** ebarrera has joined #tripleo18:01
*** flepied has joined #tripleo18:01
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf  https://review.openstack.org/37708818:03
*** florianf has quit IRC18:08
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: source-repositories: Add git package mapping for SUSE  https://review.openstack.org/38884018:14
gfidentebnemec you sound hungry18:16
*** rhallisey has quit IRC18:16
*** rhallisey has joined #tripleo18:17
bnemecgfidente: I was confused how you knew I was eating lunch, then remembered my last IRC messages. ;-)18:17
*** dsariel has joined #tripleo18:20
*** yamahata has joined #tripleo18:20
*** ebarrera has quit IRC18:20
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: zypper: Drop not pull recommended packages  https://review.openstack.org/38884718:20
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf  https://review.openstack.org/37708818:20
*** cwolferh has quit IRC18:21
openstackgerritMarkos Chandras proposed openstack/diskimage-builder: elements: zypper: Do not pull recommended packages  https://review.openstack.org/38884718:21
*** pradk has quit IRC18:21
*** pradk has joined #tripleo18:23
*** rajinir has joined #tripleo18:24
openstackgerritMerged openstack/tripleo-heat-templates: Fixes missing provider mappings for OpenDaylight  https://review.openstack.org/38673718:26
*** cwolferh has joined #tripleo18:30
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf  https://review.openstack.org/37708818:33
*** trown|lunch is now known as trown18:38
*** abregman|afk has quit IRC18:39
*** rbowen has quit IRC18:39
*** abregman|afk has joined #tripleo18:39
*** abregman|afk has quit IRC18:39
*** rbowen has joined #tripleo18:43
*** aufi has quit IRC18:45
*** jtomasek_ has joined #tripleo18:51
*** fzdarsky|afk is now known as fzdarsky18:59
*** abregman has joined #tripleo19:09
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: tripleo-routed-networks-deployment - Routed Spine and Leaf  https://review.openstack.org/37708819:09
openstackgerritMerged openstack/tripleo-quickstart: Remove publish from build-images playbook  https://review.openstack.org/38880419:12
*** panda|bbl is now known as panda19:20
*** jtomasek_ has quit IRC19:22
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Hiera optimization: use a new hiera hook  https://review.openstack.org/38475719:23
trownEmilienM: do you know if we are allowed to propose different hash in https://review.openstack.org/388799 or it has to be RC3 hash?19:31
trownlooked and instack-undercloud has 2 commits on newton since hash in that review... didnt check other projects yet19:32
trowntripleo-heat-templates has quite a few19:33
trown10ish19:34
*** gfidente has quit IRC19:35
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Create directories with root  https://review.openstack.org/38489219:39
EmilienMtrown: I'm not sure19:42
EmilienMtrown: I think rc3 will be our GA and we'll propose a new tag right after summit19:42
EmilienMthat contains our ongoing work (and things recently merged)19:42
EmilienMtrown: wdyt?19:42
trownEmilienM: fine by me...19:42
trownEmilienM: though if we are allowed to pick a different hash then RC3 for GA, it is probably worth getting these other fixes in19:43
EmilienMtrown: I'm not sure about that, I would ask Doug19:44
trownEmilienM: k, I will just ask on the review ;)19:44
EmilienMtrown: ok, let me know19:46
*** ayoung has joined #tripleo19:47
dhill_EmilienM, https://bugs.launchpad.net/tripleo/+bug/163420219:48
openstackLaunchpad bug 1634202 in tripleo "Failure to import new nodes using RDO Newton on RHEL" [Medium,New] - Assigned to wes hayutin (weshayutin)19:48
*** pcaruana has quit IRC19:50
stevebakermorning19:50
*** tzumainn has quit IRC19:52
*** tzumainn has joined #tripleo19:53
*** tiswanso has quit IRC19:53
*** dbecker has quit IRC19:54
*** dprince has quit IRC19:55
*** abregman has quit IRC19:57
stevebakerEmilienM: hey, are you aware of any plans to make undercloud endpoints https by default?20:08
EmilienMstevebaker: dtrainor has a task to make https default20:08
EmilienMand a patch too iirc20:08
stevebakerok, cool20:09
stevebakerI was wondering how the cert would be propagated to the overcloud20:09
*** akrivoka has quit IRC20:12
*** saneax-_-|AFK is now known as saneax20:12
trownhmm so heat could call back?20:15
openstackgerritJames Slagle proposed openstack/instack-undercloud: Make module name and file name equivalent  https://review.openstack.org/38888920:16
*** coolsvap has quit IRC20:16
ayoungShort of rerunning openstack undercloud deploy again, is there something I can do that will effectively be puppet apply but use all the hiera data to update an undercloud?20:21
ayoungstevebaker, cert should not need to be propagated if it uses the same CA....20:22
ayoungI know Iknow...we are going to do self signed by default because we hate real security and want to hasten the downfall of western civilization....20:23
stevebakerayoung: heh, indeed20:25
openstackgerritBen Nemec proposed openstack/tripleo-specs: Area Experts Policy  https://review.openstack.org/38889120:25
ayoungstevebaker, out of curiousity, why that particular concern?20:26
trownayoung: I have never done it mind you, but I don't see what would be wrong with `puppet apply --detailed-exitcodes /etc/puppet/manifests/puppet-stack-config.pp`20:26
stevebakerayoung: https on the undercloud?20:27
trownayoung: that is what is being done in the element: https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/os-refresh-config/configure.d/50-puppet-stack-config#L1920:27
EmilienMbnemec: nice proposal20:27
EmilienMbnemec: just saw your comment on my specs20:28
bnemecEmilienM: I've been meaning to write that up for a while, but summit was finally the deadline that forced me to sit down and actually do it. :-)20:28
*** jprovazn has quit IRC20:28
bnemecI don't have any idea how receptive people will be, but it should make for some interesting conversation.20:29
EmilienMbnemec: I hope you'll be around during the "Grow the team" session that trown is leading20:30
EmilienMyour and my specs are going to be discussed there I think20:31
bnemecEmilienM: Yeah, I'll make sure to be there.20:31
EmilienMexcellent20:31
trownI am thinking of attending that one20:31
bnemecI would hope so...<EmilienM> bnemec: I hope you'll be around during the "Grow the team" session that ***trown is leading***20:32
bnemec;-)20:32
EmilienMtrown: LOL20:32
EmilienMtrown: did I tell you that you don't have the choice?20:32
*** milan has quit IRC20:33
* bnemec will be on the beach growing his tan20:33
*** milan has joined #tripleo20:33
stevebakerayoung: is letsencrypt in the default trusted CAs?20:35
ayoungstevebaker, Um...let's find out.  I know it works for my blog....20:35
trownall of rdoproject https is letsencrypt... so probably20:36
ayoungstevebaker, curl https://adam.younglogic.com/ seems to work20:36
ayoungon a brand new Quickstart installed undercloud too20:36
stevebakeroh yeah, thats encouraging20:37
stevebakerayoung: anyway, I was looking at docker tooling like docker-registry, and its strong preference for properly encrypted endpoints, which yak-shaved into undercloud endpoints in general20:38
EmilienMbnemec: I'm on pto all the week but i'll try to get a first read and review asap20:40
trownayoung: btw just tried `sudo puppet apply --detailed-exitcodes /etc/puppet/manifests/puppet-stack-config.pp` on a fresh undercloud... seems to work20:42
ayoungtrown, thanks.  key piece of knowledge there20:43
*** saneax is now known as saneax-_-|AFK20:43
*** jcoufal has quit IRC20:48
*** egafford has quit IRC20:48
*** rbrady is now known as rbrady-afk20:49
*** saneax-_-|AFK is now known as saneax20:50
EmilienMayoung: FYI, rdo reported https://bugs.launchpad.net/tripleo/+bug/1634828 and mwhahaha commented20:53
openstackLaunchpad bug 1634828 in tripleo "Specified fernet token keys are not utilised" [Low,Incomplete]20:53
mwhahahaayoung: EmilienM: I was unable to repro the bug cause it worked correctly for me20:53
EmilienMayoung: where are we with the feature in tripleo?20:53
*** lblanchard has quit IRC20:53
mwhahahaso it might have been an outdated setup or something cause i thought we fixed it recently20:53
EmilienMmwhahaha: right, we fixed it in Newton iirc20:54
EmilienMor at least start actually testing it in Newton20:54
ayoungEmilienM, its on hold until after the summit20:55
EmilienMayoung: ok, cool20:55
ayoungEmilienM, the approach we are doing in Newton is  based on20:56
ayounghttp://adam.younglogic.com/2016/09/fernet-overcloud/20:56
EmilienMayoung: I think it's a first good iteration and we are open to change it in the future with something more dynamic20:56
ayoungand that uses the keys from the tarball20:56
*** noslzzp has quit IRC20:56
stevebakerayoung: certmonger doesn't support ACME yet, thats a shame https://fedorahosted.org/certmonger/ticket/3720:57
ayoungah well20:58
stevebakercertbot is in epel though20:58
openstackgerritLokesh Jain proposed openstack/tripleo-heat-templates: Updates for configuring Nuage in Mitaka release  https://review.openstack.org/37274920:59
*** noslzzp has joined #tripleo21:00
*** maticue has quit IRC21:01
*** Goneri has quit IRC21:03
*** jayg is now known as jayg|g0n321:14
openstackgerritFrederic Lepied proposed openstack-infra/tripleo-ci: Allow to use an RDO review to fix a packaging issue  https://review.openstack.org/38879121:18
*** r-mibu has quit IRC21:19
*** r-mibu has joined #tripleo21:19
*** rhallisey has quit IRC21:19
*** cdearborn has quit IRC21:32
*** jeckersb is now known as jeckersb_gone21:33
*** bfournie has quit IRC21:33
*** noslzzp has quit IRC21:34
*** kjw3 has quit IRC21:40
*** Goneri has joined #tripleo22:01
dtrainorhi EmilienM, stevebaker22:05
dtrainorgive me a sec to get caught up22:05
stevebakerdtrainor: hey22:05
dtrainoralrighty.  are we all good?22:06
stevebakerdtrainor: just that I'm interested in undercloud https by default, because then docker-registry can use the same cert22:07
dtrainoroh i see22:09
stevebakerdtrainor: and if we persist with self-signed by default then we need to figure out how to propagate the CA to the overclouds, or we need to come up with a real CA default like letsencrypt22:09
*** morazi has quit IRC22:10
dtrainorhmmmmmm22:15
*** jeckersb_gone is now known as jeckersb22:20
dtrainorpardon my ignorance for a minute... why does docker-registry need to use that cert?22:21
dtrainoraren't undercloud and overcloud ssl different animals at that point?22:22
*** Goneri has quit IRC22:22
stevebakerdtrainor: sorry, I'm talking about the docker-registry that is currently running on the undercloud - at some point in the future overcloud nodes may pull images from that registry22:23
stevebakerdtrainor: on a containerised overcloud22:23
dtrainoroh i see.  i wasn't familiar with that effort, good point.22:24
dtrainorwhere would the ca cert live?  as a swift object?22:24
stevebakerdtrainor: I think firstboot config would be the best way to get it to the overcloud - leaving aside upgrade issues for now22:25
openstackgerritMatthew Flusche proposed openstack/os-net-config: Add route_options parameter  https://review.openstack.org/38891622:26
*** bfournie has joined #tripleo22:30
*** dciabrin has quit IRC22:33
*** trown is now known as trown|outtypewww22:38
*** panda is now known as panda|Zz22:39
*** dciabrin has joined #tripleo22:40
openstackgerritDavid Hill proposed openstack/tripleo-common: Adding retry in set_node_state in baremetal.yaml  https://review.openstack.org/38892022:50
*** dciabrin has quit IRC22:53
openstackgerritSteve Baker proposed openstack/instack-undercloud: Bind docker-registry to controller_host  https://review.openstack.org/38892323:02
*** yamahata has quit IRC23:07
dtrainoranyone use any early indicators that a deployment is going to fail?23:07
dtrainorsomething simple even, like not being able to ssh heat-admin@ or something23:07
openstackgerritDavid Hill proposed openstack/tripleo-common: Adding retry in set_node_state in baremetal.yaml  https://review.openstack.org/38892023:07
dtrainordeployments take a long time.... i want to know when to just call it and start over23:17
stevebakerdtrainor: that depends entirely on where it stalls23:18
dtrainoryep23:19
dtrainorany indicator, really.  especially since I'm using fake_pxe, I think that hides a lot of information23:19
*** rlandy has quit IRC23:19
dtrainorthe heat logs look pretty busy. other than that, i just see a lot of progress states in the heat resource list23:20
*** yamahata has joined #tripleo23:20
dtrainor(i think i messed it up by rebooting the overcloud nodes at the wrong time / in the wrong order)23:21
*** fragatina has joined #tripleo23:24
stevebakerdtrainor: if the nova servers don't boot, bail. If they do but the network config resources don't go to complete, you could try ssh heat_admin then bail. From then on its software configuration23:26
dtrainoryea, i think i exploded the deployment by rebooting at the wrong time23:29
dtrainori should manually power cycle overcloud nodes when the ironic state changes to 'wait call-back', right?23:30
*** dciabrin has joined #tripleo23:32
dtrainordoes it matter when I run 'openstack baremetal configure boot'  ?  before/after i assign nodes?23:39
*** thrash is now known as thrash|g0ne23:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!