Wednesday, 2017-08-23

*** achadha_ has joined #tripleo00:02
*** achadha has quit IRC00:05
*** achadha_ has quit IRC00:06
pabelangerstevebaker: mwhahaha: ya, check-tripleo-jobs template needs to be add for tripleo-quickstart-extras00:07
*** hexo_ is now known as kokot00:08
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140300:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Ian Main (imain)00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295500:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209800:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243600:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]00:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)00:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]00:10
openstackgerritMerged openstack/tripleo-heat-templates master: Add Ceilometer API and Collector service to roles_data  https://review.openstack.org/49458900:10
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Make rolling update interactive.  https://review.openstack.org/49644900:16
*** thrash is now known as thrash|g0ne00:40
*** achadha has joined #tripleo00:40
*** limao has joined #tripleo00:44
*** yamahata has quit IRC00:47
*** achadha has quit IRC00:50
*** achadha has joined #tripleo00:51
*** itlinux has joined #tripleo00:51
*** achadha has quit IRC00:52
*** achadha has joined #tripleo00:53
*** achadha has quit IRC00:54
*** achadha has joined #tripleo00:55
*** fpan has quit IRC00:55
*** cshastri has joined #tripleo00:57
*** dixiaoli has joined #tripleo00:59
*** achadha has quit IRC00:59
*** funzo has joined #tripleo01:04
*** itlinux has quit IRC01:04
openstackgerritPaul Belanger proposed openstack/tripleo-quickstart master: Clean up repo_cmd_before command for setup-repo  https://review.openstack.org/49608501:05
*** fzdarsky_ has joined #tripleo01:06
*** funzo has quit IRC01:08
*** fzdarsky has quit IRC01:09
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140301:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295501:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209801:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243601:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Ian Main (imain)01:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]01:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)01:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]01:10
*** achadha has joined #tripleo01:10
*** achadha has quit IRC01:11
*** achadha has joined #tripleo01:12
*** achadha has quit IRC01:16
openstackgerritMerged openstack/tripleo-heat-templates master: Accept multiple registries in DockerInsecureRegistryAddress  https://review.openstack.org/49261301:20
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates master: WIP container ovs-agent, ensure br-ex exists  https://review.openstack.org/49369501:28
*** itlinux has joined #tripleo01:29
openstackgerritMerged openstack/tripleo-common master: Add Barbican Config Image to overcloud containers  https://review.openstack.org/49349301:32
openstackgerritMerged openstack/tripleo-common master: Configure mistral event engine container  https://review.openstack.org/49596601:35
openstackgerritMerged openstack/tripleo-common master: Add Qdrouterd to the overcloud containers  https://review.openstack.org/49188901:35
*** itlinux has quit IRC01:35
*** achadha has joined #tripleo01:40
*** itlinux has joined #tripleo01:43
*** achadha_ has joined #tripleo01:43
*** achadha has quit IRC01:44
*** ecerquei has quit IRC01:47
*** achadha_ has quit IRC01:48
*** bkopilov has quit IRC01:48
*** ramishra has joined #tripleo01:48
*** achadha has joined #tripleo01:53
*** rbowen has quit IRC01:54
*** itlinux has quit IRC01:56
*** ramishra has quit IRC02:02
*** ramishra has joined #tripleo02:05
*** itlinux has joined #tripleo02:06
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140302:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295502:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209802:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243602:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]02:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)02:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]02:10
*** dixiaoli has quit IRC02:15
*** tzumainn has joined #tripleo02:15
*** dixiaoli has joined #tripleo02:19
*** cdearborn_ has quit IRC02:19
*** mdnadeem has joined #tripleo02:21
*** itlinux has quit IRC02:21
*** achadha has quit IRC02:28
*** weshay is now known as weshay_PTO_AM02:37
openstackgerritSteve Baker proposed openstack/tripleo-heat-templates master: WIP container ovs-agent, ensure br-ex exists  https://review.openstack.org/49369502:48
*** itlinux has joined #tripleo02:50
*** itlinux has quit IRC02:57
*** bkopilov has joined #tripleo03:04
*** funzo has joined #tripleo03:04
*** funzo has quit IRC03:09
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140303:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295503:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209803:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243603:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]03:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)03:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]03:10
*** itlinux has joined #tripleo03:12
*** itlinux has quit IRC03:14
*** itlinux has joined #tripleo03:17
*** itlinux has quit IRC03:30
*** tzumainn has quit IRC03:30
*** gbarros has joined #tripleo03:31
*** gkadam has joined #tripleo03:40
*** limao has quit IRC03:42
*** udesale has joined #tripleo03:43
*** limao has joined #tripleo03:43
*** rlandy has quit IRC03:45
*** psahoo has joined #tripleo03:47
*** leitan has joined #tripleo03:47
*** jobcespedes has quit IRC03:57
*** links has joined #tripleo03:57
*** funzo has joined #tripleo04:05
*** funzo has quit IRC04:10
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140304:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295504:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209804:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243604:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]04:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)04:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]04:10
*** shreshtha has joined #tripleo04:17
openstackgerritMerged openstack/tripleo-quickstart-extras master: Revert "Filter image list by deployed services"  https://review.openstack.org/49644504:17
*** ykarel has joined #tripleo04:17
*** shreshtha_ has joined #tripleo04:20
*** shreshtha has quit IRC04:22
*** achadha has joined #tripleo04:23
*** mdnadeem has quit IRC04:28
*** itlinux has joined #tripleo04:30
*** itlinux has quit IRC04:34
*** fpan has joined #tripleo04:43
*** nyechiel has joined #tripleo04:44
*** dsariel has joined #tripleo04:47
*** pgadiya has joined #tripleo04:50
*** mdnadeem has joined #tripleo04:52
*** nyechiel has quit IRC04:56
*** pgadiya has quit IRC04:58
*** leitan has quit IRC05:01
*** jtomasek_ has joined #tripleo05:01
*** jtomasek has quit IRC05:02
*** sdake has quit IRC05:02
*** jpena|off has quit IRC05:02
*** afazekas has quit IRC05:02
*** sdoran has quit IRC05:02
*** kokot has quit IRC05:02
*** zul has quit IRC05:02
*** Hazelesque has quit IRC05:02
*** hexo_ has joined #tripleo05:02
*** Hazelesque has joined #tripleo05:02
*** hexo_ is now known as hexo05:02
*** afazekas has joined #tripleo05:02
*** sdake has joined #tripleo05:03
*** sdake has quit IRC05:03
*** sdake has joined #tripleo05:03
*** rbowen has joined #tripleo05:03
*** jpena|off has joined #tripleo05:04
*** radez has quit IRC05:06
*** trown|outtypewww has quit IRC05:06
*** rook has quit IRC05:06
*** trown has joined #tripleo05:06
*** itlinux has joined #tripleo05:07
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140305:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295505:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209805:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243605:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]05:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)05:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]05:10
*** tvignaud has quit IRC05:10
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Docker: Enable TLS in the internal network for libvirt  https://review.openstack.org/49635105:11
*** radez has joined #tripleo05:11
*** rook has joined #tripleo05:12
*** rook is now known as Guest9146405:12
*** itlinux has quit IRC05:19
*** yprokule has joined #tripleo05:19
*** dpawar has joined #tripleo05:20
*** aditya_r has joined #tripleo05:22
*** tvignaud has joined #tripleo05:22
*** ratailor has joined #tripleo05:25
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Configure listen_address for libvirtd when TLS is enabled  https://review.openstack.org/49649205:25
*** tvignaud has quit IRC05:26
*** tvignaud has joined #tripleo05:28
*** aufi has joined #tripleo05:30
*** masco has joined #tripleo05:32
*** fpan has quit IRC05:33
*** skramaja has joined #tripleo05:35
*** chem has quit IRC05:40
*** aditya_ra has joined #tripleo05:43
*** aditya_r has quit IRC05:44
*** iranzo has joined #tripleo05:47
*** iranzo has quit IRC05:47
*** iranzo has joined #tripleo05:47
*** pdeore has joined #tripleo05:48
*** jfrancoa has joined #tripleo05:52
*** dmacpher has joined #tripleo05:55
*** achadha has quit IRC05:58
*** achadha has joined #tripleo05:58
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260706:00
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Certmonger: Only attempt to reload haproxy is it's active  https://review.openstack.org/49650106:03
kristaps_morning guys06:05
*** funzo has joined #tripleo06:06
kristaps_i have issue with glance images, when i upload it size is 0. i show logs to #glance folks and they said " it looks like in the log that the image is being uploaded to swift, but rbd:// is ceph" but actually there is no ceph06:08
kristaps_any ideas?06:08
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140306:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295506:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209806:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243606:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]06:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)06:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]06:10
*** funzo has quit IRC06:10
*** prateek has joined #tripleo06:12
*** prateek has left #tripleo06:12
jaosoriorkristaps_: I suggest you wait a couple of hours for more folks to log in. And a dude whose handle is gfidente might be able to help with that.06:12
*** shardy has joined #tripleo06:13
kristaps_jaosorior: okey thank you06:13
kristaps_jaosorior: maybe any quick ideas?06:15
*** shrjoshi has joined #tripleo06:15
*** shreshtha_ has quit IRC06:16
*** rcernin has joined #tripleo06:16
jaosoriorkristaps_: I'm not very acquainted with that area to be honest. Sorry.06:16
kristaps_jaosorior: but there is no special settings whoich i could configure before undercloud deployment for glance?06:17
*** dparkes has joined #tripleo06:19
openstackgerritSteven Hardy proposed openstack/os-net-config master: This patch adds initial support for the Contrail vRouter interface  https://review.openstack.org/49249206:20
*** agurenko has joined #tripleo06:23
jaosoriorkristaps_: the undercloud shouldn't use ceph at all, which is why it's pretty confusing that you're getting that.06:27
*** ffiore has joined #tripleo06:30
*** ykarel_ has joined #tripleo06:30
jaosoriorshardy, bandini: Could you check these out if you ahve time https://review.openstack.org/#/c/496345/ https://review.openstack.org/#/c/496349/06:30
kristaps_jaosorior: can i just re-run undercloud installation if there no existing overcloud?06:32
bandinijaosorior: on it06:32
jaosoriorkristaps_: yep06:32
*** ykarel has quit IRC06:33
*** ramishra has quit IRC06:35
*** ramishra has joined #tripleo06:37
*** anshul has joined #tripleo06:38
jaosoriorbandini: thanks!06:38
bandini;)06:38
jaosoriormandre: let me know when you're around06:43
*** bogdando has joined #tripleo06:45
*** achadha_ has joined #tripleo06:45
mandrejaosorior: good morning06:46
jaosoriormandre: sup dude! how's it going?06:47
*** achadha has quit IRC06:48
jaosoriormandre: have you gone forward with the redis SSL work? I just finished the libvirtd TLS bits. So we could pair that if you want.06:48
mandrejaosorior: so the deployment i kicked off last night failed on config file generation which is weird because my patch didn't touch anything around that area, trying to debug now06:49
*** limao has quit IRC06:49
jaosoriormandre: regarding the haproxy check. Apparently we can enable ssl for the tcp-check https://www.haproxy.com/doc/aloha/7.0/haproxy/healthchecks.html#checking-any-service   so we're covered on that area.06:49
jaosoriorjust a matter of enabling the config option "tcp-check connect ssl"   if TLS everywhere is used.06:50
mandrejaosorior: sure, do you want me to post the WIP patch so you can have a peek already?06:50
jaosoriormandre: sure! that would be great06:50
jaosoriormandre: I'm gonna go for a quick jog. But if you want, we can starting pairing this after that.06:51
*** dmacpher has quit IRC06:51
openstackgerritMartin André proposed openstack/tripleo-common master: Add apache to redis image for tls  https://review.openstack.org/49651906:54
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: TLS proxy for redis  https://review.openstack.org/49652006:54
*** cschwede_ has joined #tripleo06:55
*** limao has joined #tripleo06:55
jaosoriormandre: we would need to add the TLS proxy code in puppet-tripleo. but that should be fairly fast.06:56
mandrejaosorior: yep, let's sync after you're back and I had my breakfast06:56
jaosoriormandre: ack06:57
*** shardy has quit IRC06:58
*** shardy has joined #tripleo06:59
*** sdoran has joined #tripleo07:00
bandinianyone around for a one-liner review https://review.openstack.org/#/c/495965/ ?07:00
*** zul has joined #tripleo07:00
*** jlinkes has joined #tripleo07:02
*** jpich has joined #tripleo07:04
*** ebarrera has joined #tripleo07:04
skramajashardy: even after taking the reduced set of images, a big list of DockerXXXImages are not used - http://logs.openstack.org/00/495300/3/check/gate-tripleo-ci-centos-7-containers-multinode/1604f4f/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-08-21_13_36_0407:05
*** achadha_ has quit IRC07:05
skramajashardy: i still feel that filterint out these parameters will enhance the useability. WDYT07:05
shardyskramaja: Hmm, why are those parameters defined if the services aren't enabled in the job?07:08
shardyI thought stevebaker's patch would stop that - are we still waiting on a quickstart patch to use it perhaps?07:08
* shardy looks07:08
*** dtrainor has quit IRC07:09
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140307:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295507:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209807:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243607:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]07:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)07:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]07:10
shardyskramaja: it was reverted see https://review.openstack.org/#/c/496445/07:10
kristaps_guys, anyone has any idea, why when upload images to glance i got size 0, and in log see Cannot determine image size because it is either not provided in the request or chunked-transfer encoding is used. Adding image as a segmented object to Swift07:10
*** florianf has joined #tripleo07:10
openstackgerritMerged openstack/tripleo-validations master: Do not output empty variables in inventory  https://review.openstack.org/49596507:10
openstackgerritMerged openstack/tripleo-validations master: Make sure to keep case when for host definition in host inventory.  https://review.openstack.org/49597507:10
skramajashardy: ohh.. ok.07:10
skramajashardy: probably i will run it locally and see it once before we push it.07:11
shardyskramaja: if possible, I'd like to leave the warnings for all parameters, at least for now, as we know the cost of having many unused images is quite high07:11
*** ykarel__ has joined #tripleo07:11
shardye.g even if we don't pull them on deploy, we pull them during the prepare/upload step which wastes a bunch of time and disk space07:11
*** pcaruana has joined #tripleo07:11
shardyso it's probably good to warn the user about it?07:12
*** ffiore has quit IRC07:12
skramajashardy: ok. if we have a way to avoid it, the it is OK to warn, but docker.yaml environment is default and we dont want user to override (mostly)07:13
skramajashardy: let me try it locally and will provide feedback.07:13
*** ykarel_ has quit IRC07:13
shardyskramaja: ack sounds good, thanks!07:14
*** jprovazn has joined #tripleo07:15
*** tesseract-RH has joined #tripleo07:17
*** oidgar has joined #tripleo07:20
*** dixiaoli_ has joined #tripleo07:29
*** jpena|off is now known as jpena07:29
shardyskramaja: hey https://review.openstack.org/#/c/450264 passed CI now, thanks for the review - are you happy to approve it?07:29
shardyI've rechecked the next two patches in the series so hopefully we can land them later07:29
skramajasure07:29
shardythanks!07:29
*** oidgar has quit IRC07:30
kristaps_ guys, anyone has any idea, why when upload images to glance i got size 0, and in log see Cannot determine image size because it is either not provided in the request or chunked-transfer encoding is used. Adding image as a segmented object to Swift07:31
*** hjensas has joined #tripleo07:31
*** hjensas has quit IRC07:31
*** hjensas has joined #tripleo07:31
*** daidv has joined #tripleo07:32
*** dixiaoli has quit IRC07:32
*** dtrainor has joined #tripleo07:32
shardykristaps_: Hi, not seen that error myself, but providing a paste of the logs, and the full command used to deploy (is this overcloud or undercloud) may help figure it out07:33
shardysounds like something is misconfigured, but it's hard to say exactly what without more info07:33
*** ykarel__ is now known as ykarel07:34
*** hjensas has quit IRC07:39
kristaps_shardy: im talking about overcloud, after openstack overcloud image upload i see in glance that are images are size 0 http://paste.openstack.org/raw/619072/ , if i grep one of the images in glance logs i get http://paste.openstack.org/raw/619132/07:39
kristaps_ shardy: sorry not overcloud but undercloud07:39
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy OpenShift using OOO on the overcloud  https://review.openstack.org/49447007:40
kristaps_shardy: before uploading images not empty ofcourse ))07:41
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Move deprecated SchedulerHints logic to overcloud.j2.yaml  https://review.openstack.org/49532407:41
shardykristaps_: Ok, have you made any changes to the default undercloud.conf?07:42
openstackgerritMartin Kopec proposed openstack/tripleo-quickstart-extras master: Allow removing of options from tempest conf  https://review.openstack.org/47707907:42
shardyand are the swift services all running OK?07:42
kristaps_shardy: yesterday in glance irc i get answer : that looks a bit odd ... it looks like in the log that the image is being uploaded to swift, but rbd:// is ceph07:42
shardykristaps_: can you paste your glance.conf please?07:43
kristaps_shardy: only enable ssl, add my NIC and hostname07:43
shardysorry glance-api.conf and glance-swift.conf07:44
shardywhere do you see the rbd:// ?07:45
*** suuuper has joined #tripleo07:46
*** amoralej|off is now known as amoralej07:46
*** hjensas has joined #tripleo07:46
*** hjensas has quit IRC07:46
*** hjensas has joined #tripleo07:46
*** agurenko has quit IRC07:47
*** gbarros has quit IRC07:48
*** athomas has joined #tripleo07:49
*** cylopez has joined #tripleo07:53
kristaps_shardy: glancec-swift.conf http://paste.openstack.org/raw/619134/07:53
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder  https://review.openstack.org/49400807:57
kristaps_shardy: can i send you glance-api.cong as i file?07:59
*** egonzalez has joined #tripleo08:00
*** agurenko has joined #tripleo08:00
shardykristaps_: yes, or just tell me where you see rbd:// ?08:00
shardythe glance-api.conf should have a line like stores=glance.store.filesystem.Store,glance.store.swift.Store08:01
shardyand default_store=swift08:01
shardykristaps_: I assume you've got plenty of disk space?08:01
*** openstackgerrit has quit IRC08:03
kristaps_shardy: it was glance folk comment about one one the log, i can try to find it08:03
*** suuuper has quit IRC08:04
kristaps_shardy: there is no prbolems with space08:05
*** suuuper has joined #tripleo08:05
shardykristaps_: does swift work if you do "openstack container create test" ?08:05
*** funzo has joined #tripleo08:07
kristaps_shardy: stack@undercloud glance]$ openstack container create test08:07
kristaps_+---------------------------------------+-----------+------------------------------------+08:07
kristaps_| account                               | container | x-trans-id                         |08:07
kristaps_+---------------------------------------+-----------+------------------------------------+08:07
kristaps_| AUTH_e1270b1e77f34774935ab237b134e47a | test      | tx3d7fdf4048594abab55e9-00599d378e |08:07
shardyyou could then do e.g openstack object create test overcloud-full.qcow208:07
kristaps_+---------------------------------------+-----------+------------------------------------+08:07
*** derekh has joined #tripleo08:07
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140308:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295508:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209808:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243608:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]08:10
kristaps_shardy: done http://paste.openstack.org/raw/619139/08:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)08:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]08:10
shardyOk so swift is working fine, so the problem must be the glance config08:10
*** funzo has quit IRC08:11
*** hjensas has quit IRC08:11
*** suuuper has quit IRC08:11
*** hjensas has joined #tripleo08:14
*** hjensas has quit IRC08:14
*** hjensas has joined #tripleo08:14
*** suuuper has joined #tripleo08:15
kristaps_shardy: share to you glance-api.conf?08:18
*** lucas-afk is now known as lucasagomes08:18
*** paramite has joined #tripleo08:20
* shardy cries08:22
shardykristaps_: yes, please08:24
*** openstackgerrit has joined #tripleo08:24
openstackgerritMerged openstack/puppet-tripleo master: Add -s (silent) to curl command for CRL refresh  https://review.openstack.org/49634508:24
kristaps_shardy: did you receive?08:25
*** kristaps_ has quit IRC08:25
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates master: Specify the start count to 0 for the update step loop  https://review.openstack.org/49639708:26
*** lvdombrkr has joined #tripleo08:27
lvdombrkrshardy: did you receive it i send you via irc client08:28
shardylvdombrkr: you PM'd me a checksum08:28
*** mdnadeem has quit IRC08:32
*** mdnadeem has joined #tripleo08:32
*** ffiore has joined #tripleo08:35
lvdombrkrshardy: https://files.fm/u/83b3sp8y08:37
*** mcornea has joined #tripleo08:38
lvdombrkrshardy: get it?08:44
shardylvdombrkr: thanks, sorry I can't see any problem with the file, I'm still confused about the rbd:// comment08:44
* shardy looks around for a glance expert08:45
shardyflaper87: ^^ maybe you could sanity check the glance config above?08:45
flaper87shardy: checking08:46
shardythanks!08:46
*** hjensas has quit IRC08:47
*** aramteke has joined #tripleo08:48
flaper87shardy: lvdombrkr looks correct08:50
* flaper87 reads the backlog to get more context08:51
*** oidgar has joined #tripleo08:51
*** rwsu has joined #tripleo08:51
lvdombrkrflaper87: in few words issue is that i have 0 size images in glance08:51
*** aditya_ra has quit IRC08:51
shardyhttp://paste.openstack.org/raw/619132/ is the glance log08:52
* flaper87 checks glance's logs08:52
flaper872017-08-22 21:10:27.945 5176 DEBUG glance_store._drivers.swift.store [req-6150da2d-a2aa-4e83-8919-41e693992fe5 3646e98ad7c5458bac33d44c10950fe9 e1270b1e77f34774935ab237b134e47a - default default] Not writing zero-length chunk. add /usr/lib/python2.7/site-packages/glance_store/_drivers/swift/store.py:91108:53
flaper87it looks like something is failing on glance side. I would first check if the data is actually getting to glance08:54
flaper87Make sure the upload request is actually sending data08:54
lvdombrkrflaper87: how can i check it?08:54
flaper87I used to do this putting a breakpoint somewhere in glance and verifying the data was being sent correctly08:54
flaper87From the logs, it looks like the request is being sent chunked08:55
lvdombrkrflaper87: can you give more instructions how to check it, its no clear for me08:57
*** salmankhan has joined #tripleo08:57
lvdombrkrflaper87: any ideas why its hapening?08:58
lvdombrkrflaper87: what i know that finally this upladed image is corrupt cause i cant boot from it.. thats was reason why i start investigation08:59
*** hjensas has joined #tripleo09:00
*** hjensas has quit IRC09:00
*** hjensas has joined #tripleo09:00
flaper87lvdombrkr: have you tried uploading new images manually?09:01
*** hjensas has quit IRC09:01
lvdombrkrflaper87: yes, same result09:02
flaper87you could start from there. IF that results in 0-sized images, then you can proceed to debug using breakpoints09:02
flaper87mmh09:02
flaper87ok09:02
flaper87is that using v2 or v1?09:02
flaper87it seemed v2 from the logs09:02
lvdombrkrflaper87: yes v2 i suggest09:03
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud master: Undercloud/Certmonger: Only attempt to reload haproxy is it's active  https://review.openstack.org/49656409:04
*** ebarrera has quit IRC09:04
*** tosky has joined #tripleo09:04
openstackgerritDavid Sariel proposed openstack/tripleo-validations master: Add containers_sanity_check  https://review.openstack.org/48340309:06
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud master: Undercloud/Certmonger: Only attempt to reload haproxy is it's active  https://review.openstack.org/49656409:06
*** ebarrera has joined #tripleo09:06
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140309:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295509:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209809:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243609:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]09:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)09:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]09:10
openstackgerritAlfredo Moralejo proposed openstack/tripleo-heat-templates stable/newton: Make get-occ-config.sh support custom roles  https://review.openstack.org/49656609:10
openstackgerritMartin André proposed openstack/tripleo-common master: Add apache to redis image for tls  https://review.openstack.org/49651909:13
*** sdake has quit IRC09:15
*** sdake has joined #tripleo09:16
*** bkopilov has quit IRC09:16
*** sdake is now known as Guest2894309:16
openstackgerritMerged openstack/tripleo-common master: Download and run upgrade/deploy_steps_playbooks for upgrade  https://review.openstack.org/49084709:17
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata  https://review.openstack.org/49656809:17
openstackgerritMerged openstack/python-tripleoclient master: Write user parameters environment to swift  https://review.openstack.org/45026409:21
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: HAProxy: Make certmonger bundle the cert and key on renewal  https://review.openstack.org/49657209:22
*** aramteke has quit IRC09:23
oidgarguys, remind me, what is the new way to create the docker local registry for overcloud images?09:23
jaosorioradarazs: any idea what's wrong with the ovb jobs today? They've been failing a lot today09:23
adarazsjaosorior: no, let me take a look09:24
*** limao has quit IRC09:26
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: [WIP] TLS proxy for redis  https://review.openstack.org/49652009:26
shardyoidgar: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L18 shows the commands used09:32
oidgarshardy: thanks!09:33
adarazsjaosorior: looks like the "no more mirrors to try" -- rpm download issues :/09:37
adarazssshnaidm|off: do you know what's going on?09:38
*** ffiore has quit IRC09:41
*** ffiore_ has joined #tripleo09:41
*** sshnaidm|off is now known as sshnaidm09:42
*** adarazs is now known as adarazs_lunch09:42
sshnaidmadarazs_lunch, looking..09:42
jaosoriormandre: hey dude09:46
*** gfidente has joined #tripleo09:48
*** gfidente has quit IRC09:48
*** gfidente has joined #tripleo09:48
lvdombrkrshardy: i checked right now my first undercloud installation (on other HW) and there is also 0 size images09:50
lvdombrkrthat means, i do wrong steps09:50
lvdombrkrin installation09:50
*** dixiaoli_ has quit IRC09:52
mandrehey jaosorior, I'm looking at the puppet-tripleo part now09:53
jaosoriorbandini:09:54
mandrejaosorior: I'm trying to replicate the changes made to other services, not exactly sure what I'm doing09:54
jaosoriorbandini: when redis is managed by pacemaker, is it binding to the VIP or the regular IP ?09:54
jaosoriormandre: wanna pair it in bluejeans?09:54
lvdombrkrshardy: im deploying undercloud by this documentation : https://docs.openstack.org/tripleo-docs/latest/install/installation/installation.html09:55
*** dpawar has quit IRC09:55
lvdombrkrversion Ocata09:56
lvdombrkrwihout CEPH09:56
lvdombrkrwithout puppet09:56
mandrejaosorior: maybe a bit later, it's about lunch time here09:57
jaosoriormandre: ok, let me know when you're back09:57
mandrejaosorior: what port should the tls proxy be listening on for redis?09:57
lvdombrkrshardy: can you confirm that i can falow this documetation?09:57
lvdombrkrshardy: i try to understood what goes wrong09:58
*** ykarel_ has joined #tripleo09:58
jaosoriormandre: the same as redis. then the actual redis server will listen on localhost instead.09:59
bandinijaosorior: haproxy binds to the redis_vip and then forwards the requests to haproxy which binds to the host ip (hope I got the question right?)09:59
shardylvdombrkr: yes the documentation should work - do you have selinux enforcing?09:59
lvdombrkrshardy: its permessive10:00
*** ykarel has quit IRC10:01
jaosoriorbandini: you did. Thanks!10:01
bandinijaosorior: the second 'haproxy' in my sentence above should read 'redis' (sorry about that)10:02
jaosoriorbandini: haha I understood :P10:02
bandini:)10:03
jaosoriorbandini: are you acquainted with the redis heartbeat resource agent?10:03
bandinijaosorior: very little. but I have to start looking at it because I might need to change some stuff around. what's up?10:03
jaosoriorbandini: I'm taking a read. researching how to do TLS on redis. And I was wondering if there are some changes needed on how to access it if it's behind a TLS proxy.10:05
jaosoriorbandini: now, from the code https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/redis I can see that there are some commands ran with the redis cli client... I don't know yet if we'll need some changes there or not y et10:06
bandinijaosorior: ack yeah, I presume the redis stanza in haproxy.cfg will need a bit of love as well10:07
shardyrajinir: Hi, can we mark https://blueprints.launchpad.net/tripleo/+spec/dellemc-unity-cinder implemented now the patches on the whiteboard merged?10:07
*** funzo has joined #tripleo10:07
shardypike-rc1 is planned for tomorrow, so we need to update the status of all the blueprints10:08
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Set DNS for rh1 cloud before installation  https://review.openstack.org/49659610:08
jaosoriorbandini: it does. I think I got the answer for that one though10:08
bandinioh nice10:08
jaosoriorbandini: there is are tcp-check statements in the haproxy config, that send commands to redis. it would be a matter of adding 'tcp-check connect ssl' and it would do the same via TLS10:09
bandinijaosorior: ah simple enough, nice10:09
*** ykarel__ has joined #tripleo10:10
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140310:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295510:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209810:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243610:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)10:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]10:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)10:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]10:10
jaosoriorbandini: problem is, the blog posts that I find, while they require a TLS proxy to be in front of redis (which is not a problem) they also require a TLS proxy in front of the client, which is strange :/10:11
*** funzo has quit IRC10:12
sshnaidmjaosorior, adarazs_lunch, seems like problem with DNS in infra causes OVB failures, trying to find somebody to check it..10:12
bandinijaosorior: oh wow yeah that sounds odd10:12
*** ykarel_ has quit IRC10:12
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Move deprecated SchedulerHints logic to overcloud.j2.yaml  https://review.openstack.org/49532410:14
jaosoriorbandini: do you know how the redis client reads the redis connection?10:15
*** pblaho has joined #tripleo10:16
bandininope never looked at that stuff close enough10:17
*** hjensas has joined #tripleo10:19
*** hjensas has quit IRC10:19
*** hjensas has joined #tripleo10:19
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Move deprecated SchedulerHints logic to overcloud.j2.yaml  https://review.openstack.org/49532410:24
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Fix hardcoded references to deprecated *ExtraConfig parameters  https://review.openstack.org/49660410:24
*** milan has joined #tripleo10:25
lvdombrkrshardy: that glance issue is can be related that i enable wrong repositories?10:26
jaosoriorbandini: this doesn't look good :( https://github.com/antirez/redis/issues/217810:28
jaosoriormandre: ^^10:28
shardylvdombrkr: it could be, hard to be sure without figuring out the root cause of your problem10:29
shardylvdombrkr: you could remove the repo files you added to /etc/yum.repos.d, then run this script:10:29
shardyhttps://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh10:29
*** ykarel__ is now known as ykarel10:29
shardyexport STABLE_RELEASE=ocata10:29
shardytripleo.sh --repo-setup10:29
shardyand compare with the repos you configured manually10:30
shardyIf they differ we may have an error in the docs10:30
*** dtantsur|afk is now known as dtantsur10:30
jprovaznhi, I'm hitting an ssl verification issue - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) when deploying overcloud with ceph-ansible, does this sound familiar? http://paste.openstack.org/show/619155/10:30
*** udesale has quit IRC10:31
*** skramaja has quit IRC10:33
jaosoriorbandini: is the pacemaker agent running in the same node as the redis instance?10:37
lvdombrkrshardy: this script will configure correct repositories and after i can continue configuration step by step?10:37
jaosoriorjprovazn: it sure does. do the overcloud nodes trust the certifiacte?10:38
jaosorior*certificate10:38
jprovaznjaosorior: from the error message I would say that not10:38
jaosoriorjprovazn: that message could be many things (openssl doesn't give the best error messages)10:39
*** Guest91464 is now known as rook10:39
jaosoriorjprovazn: 1) the node doesn't trust the cert   2) the node was accessed with a wrong hostname (needs to match the cert)  3) the cert is malformed10:40
jaosoriorjprovazn: so lets go for #1 first. Did you pass the undercloud's CA certificate to the overcloud so it could trust it?10:40
*** jkilpatr has quit IRC10:42
jaosoriorjprovazn: that would be done by passing the undercloud's CA certificate (not the undercloud's haproxy server cert) via the SSLRootCertificate parameter as described in SSLRootCertificate10:42
*** cshastri has quit IRC10:42
jaosorior* as described in http://tripleo.org/install/advanced_deployment/ssl.html#certificate-details10:43
jprovaznjaosorior: OC was deployed with the following env files (should be mostly what oooq uses), let me check SSLRootCertificate10:43
jprovaznhttp://paste.openstack.org/show/619156/ - is the deployment command10:43
jaosoriorjprovazn: check /home/stack/enable-tls.yaml10:44
jaosoriorjprovazn: sorry, wrong file. Check /home/stack/inject-trust-anchor.yaml10:44
jprovaznjaosorior: regarding certificate - I can't check now - I deleted the stack before your feedback, but iirc yesterday the issue was that CA authority was not trusted by the controller node10:44
* jprovazn checks10:44
amoralejsshnaidm, we are hitting dns issues for tripleo jobs in review.r.o10:45
amoralejit's using an internal dns, right?10:45
jprovaznjaosorior: in the /home/stack/inject-trust-anchor.yaml is SSLRootCertificate included10:45
jaosoriorjprovazn: gimme a minute10:45
*** _jbadiapa has joined #tripleo10:46
sshnaidmamoralej, we have problems now, yes, which job are you talking?10:46
*** adarazs_lunch is now known as adarazs10:46
jaosoriorsshnaidm: all ovb jobs are having dns issues it seems10:46
*** akrivoka has joined #tripleo10:47
sshnaidmjaosorior, yeah10:47
amoralejsshnaidm, https://logs.rdoproject.org/03/8303/3/check/rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset005-nv/Zb4e636fb44cb40b0bf261191b3ed9dba/devstack-gate-setup-host.txt10:47
amoralejwe are having issues with google dns in other tenants in rdo-cloud, is it the same issue?10:48
*** jbadiapa_ has quit IRC10:48
jaosoriorshardy: is heat-config-notify supposed to be hitting a public endpoint for swift? http://paste.openstack.org/show/619155/10:49
sshnaidmamoralej, infra image has 127.0.0.1 as dns and unbound server witch pre-cached data10:49
bandinijaosorior: oh gasp ouch10:49
sshnaidmamoralej, and this seems like not working.. no idea what happened there10:49
bandinijaosorior: not necessarily, depends how you deployed it (composable HA, etc)10:49
jaosoriorjprovazn: ok, it seems that quickstart using different CA files, thus it doesn't trust the one in the undercloud. For one, I don't think it should be hitting swift's public endpoint. but given that that's the situation now, lets get the overcloud to trust the undercloud's CA: First, lets copy /usr/share/openstack-tripleo-heat-templates/environments/inject-trust-anchor-hiera.yaml    and lets fill that up with10:52
jaosoriorthe contents of '/etc/pki/ca-trust/source/anchors/cm-local-ca.pem'  and use it in10:52
jaosorior* and we'll use it as described here http://tripleo.org/install/advanced_deployment/ssl.html#getting-the-overcloud-to-trust-cas10:52
jaosoriorjprovazn: ^^10:52
jprovaznjaosorior: ack, sec10:53
dtantsurmwhahaha: morning! do you think https://github.com/openstack/tripleo-common/commit/53b48a303053fbe5c7308ca2623db4d2c2156be5 is a backportable change?10:55
*** cylopez has quit IRC10:55
dtantsurthe reason I ask is because 'openstack baremetal instackenv validate' is semi-broken now, and it should use these new workflows instead10:55
dtantsurI've got https://bugzilla.redhat.com/show_bug.cgi?id=1480699 about it not working with vbmc in ocata10:56
openstackbugzilla.redhat.com bug 1480699 in python-ironicclient "instackenv validate not supporting custom ports" [Unspecified,New] - Assigned to rhos-maint10:56
jaosoriormandre, bandini: Could you check this out https://review.openstack.org/#/c/496492/ ?10:56
jaosoriormandre, bandini: And this one https://review.openstack.org/#/c/496351/10:57
shardyjaosorior: Hmm, I thought we defaulted to CFN_SIGNAL, so we should poll swift and signal heat-api-cfn10:57
jprovaznjaosorior: so should I copy key from /etc/pki/ca-trust/source/anchors/cm-local-ca.pem under first-ca-name's content? and leave second-ca-name there?10:58
jaosoriorjprovazn: so, those values are dynamic, you could remove second-ca-name (since you're not using it)10:58
jaosoriorjprovazn: and you could rename first-ca-name to undercloud-ca  to make the name more descriptive10:59
jaosoriorshardy: right, so from that paste it seems it's calling swift. However, it seems to be using swift's public endpoint for that.10:59
jaosoriorshardy: thought it should be using the internal one.10:59
shardyjaosorior: No I think the public endpoint is expected, as heat doesn't expect instances to have access to internal networks11:00
shardyit's configurable in heat.conf tho if we need to use internal for TripleO reasons11:00
*** atoth has quit IRC11:00
jprovaznjaosorior: does this look sane? http://paste.openstack.org/show/619161/11:00
shardygot to drop for an hour, bbiab11:00
*** shardy has quit IRC11:01
jaosoriorjprovazn: oh, I see you used inject-trust-anchor.yaml . if you're planning to use that file for both the undercloud and the overcloud's CA files (which could in theory be the same, but it's not that way in quickstart)  then you need to add the overcloud CA there too11:02
*** dsariel is now known as dsariel|afk11:02
jaosoriorjprovazn: you'll find the overcloud's CA cert in /etc/pki/ca-trust/source/anchors/overcloud-cacert.pem11:03
mhenkelHi All. After updating a 7.3/OSP11 undercloud to 7.4/OSP11 I get this:11:03
mhenkelRemoving the current plan files11:03
mhenkelUploading new plan files11:03
mhenkelStarted Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: c04784c4-c22e-48e8-84eb-9d9cf528185d11:03
mhenkelPlan updated11:03
mhenkelDeploying templates in the directory /tmp/tripleoclient-C16WLW/tripleo-heat-templates11:03
mhenkelInvalid input [name=tripleo.deployment.v1.deploy_plan, class=DirectWorkflowSpec, unexpected=[u'skip_deploy_identifier']11:03
mhenkelany idea? I checked recent bugs but couldn't find anything11:03
*** Tengu has joined #tripleo11:04
openstackgerritMartin André proposed openstack/puppet-tripleo master: Use TLS proxy for Redis' internal TLS  https://review.openstack.org/49661611:04
jprovaznjaosorior: aha, so I should have used both, e.g. like this? http://paste.openstack.org/show/619163/11:04
Tenguhello!11:04
jaosoriorjprovazn: that looks good11:05
TenguI'm wanting to uupgrade a test infra from ocata to spike. Any advice? We're using a standard tripleO deploy, without QuickStart.11:05
TenguI was pointed to http://tripleo.org/install/post_deployment/upgrade.html - but it concerns earlier releases, I don't know if the newton->ocata process may apply for ocata->pike.11:06
*** florianf has quit IRC11:06
*** pgadiya has joined #tripleo11:06
jprovaznjaosorior: ok, thanks, I added the second one into deploy command and redeployed11:06
jaosoriorjprovazn: you could also move the contents of SSLRootCertificate  and add a second field to the CAMap in inject-trust-anchor-hiera.yaml11:06
jaosoriorjprovazn: that way you would keep track of only one file.11:06
Tengualso, I stumbled on https://github.com/openstack/tripleo-upgrade - but that seems to be redhat oriented (and won't work on centos)11:06
jprovaznjaosorior: I can try that on next re-deploy11:06
jaosoriorjprovazn: sure11:07
*** pgadiya has quit IRC11:07
*** jprovazn has quit IRC11:08
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140311:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295511:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171209811:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243611:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]11:10
openstackLaunchpad bug 1712098 in tripleo "Periodic master tempest job with 107 failures" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)11:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]11:10
*** ffiore_ has quit IRC11:10
openstackgerritMerged openstack/os-net-config master: Delete controller for standalone OVS bridges  https://review.openstack.org/49591911:11
*** salmankhan1 has joined #tripleo11:11
*** salmankhan has quit IRC11:11
*** salmankhan1 is now known as salmankhan11:11
openstackgerritMerged openstack/tripleo-ui master: Imported Translations from Zanata  https://review.openstack.org/49656811:13
*** salmankhan has quit IRC11:13
openstackgerritArundhati Surpur proposed openstack/tripleo-docs master: Update .gitignore  https://review.openstack.org/49662311:16
*** salmankhan has joined #tripleo11:17
*** pdeore has quit IRC11:18
*** jkilpatr has joined #tripleo11:19
*** limao has joined #tripleo11:20
*** florianf has joined #tripleo11:21
*** pkovar has joined #tripleo11:22
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-quickstart-extras master: Make overcloud trust the undercloud's CA certificate  https://review.openstack.org/49663911:22
*** limao_ has joined #tripleo11:27
*** limao has quit IRC11:27
*** bfournie has quit IRC11:30
*** salmankhan1 has joined #tripleo11:32
*** salmankhan1 has quit IRC11:32
*** salmankhan has quit IRC11:34
*** salmankhan has joined #tripleo11:34
chandankumarsshnaidm: please have a look on this https://review.openstack.org/#/c/493030/11:35
*** salmankhan has quit IRC11:37
*** salmankhan has joined #tripleo11:37
*** salmankhan has quit IRC11:38
*** salmankhan has joined #tripleo11:38
*** salmankhan has quit IRC11:40
*** salmankhan has joined #tripleo11:40
*** salmankhan has quit IRC11:41
*** salmankhan1 has joined #tripleo11:41
*** dpawar has joined #tripleo11:42
*** salmankhan1 has quit IRC11:43
openstackgerritMartin André proposed openstack/puppet-tripleo master: [WIP] Use TLS proxy for Redis' internal TLS  https://review.openstack.org/49661611:44
*** thrash|g0ne is now known as thrash11:50
*** ratailor has quit IRC11:51
*** _jbadiapa has quit IRC11:54
mandrejaosorior: wdyt of https://review.openstack.org/496616? we still need to hook the needed hieradata in t-h-t for puppet/services/database/redis11:54
jaosorioruhm11:56
jaosoriormandre: so this is a tricky one.11:56
mwhahahadtantsur: backport to what? newton?11:56
*** shrjoshi has quit IRC11:57
jaosoriormandre:11:57
jaosoriormandre: The clients are expecting to access Redis on the VIP11:57
*** brault has quit IRC11:57
jaosoriormandre: so we have two options, 1) we make haproxy serve TLS and use a certificate for the VIP    2) we get the redis' TLS proxy to serve the cert for the VIP11:58
jaosoriormandre: httpd usually serves certs with the fqdn for the node no the specific interface, not the VIP. So we would need to request a cert specifically for Redis.11:59
*** atoth has joined #tripleo11:59
*** pradk has joined #tripleo12:00
mandrejaosorior: ack, this is a bit more complex than my single line modification to haproxy.conf12:00
jaosoriormandre: so the haproxy configuration is more or less what you did, I commented on that. The issue is the cert (that will be used by the TLS proxy)12:01
*** lucasagomes is now known as lucas-hungry12:01
*** ansmith_ has quit IRC12:01
jaosoriormandre: And there's another tricky issue in this. And that's pacemaker. It uses redis-cli to access redis and check the status of the service. However, I'm not sure if the cli client detects TLS, and I think it doesn't, cause folks recommend setting up stunnel on the client side12:02
*** bfournie has joined #tripleo12:02
jaosoriormandre: so... we'll need to set stunnel (or a different apache-based TLS proxy for clients), on the pacemaker nodes as well12:02
jaosoriorthen we need to change all the references to redis (the URLs) to use the rediss:// protocol instead of redis://12:02
jaosoriormandre: soo... yeah, giving a bit more research to this, it's definitely not straight forward as other services :/12:03
*** dprince has joined #tripleo12:04
*** jpena is now known as jpena|lunch12:04
* jaosorior curses redis12:04
pabelangermwhahaha: weshay_PTO_AM: EmilienM okay, OVB job seems to be hitting mirror.centos.org directly: http://logs.openstack.org/28/495328/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024/c7a9a98/console.html#_2017-08-23_11_24_22_406999 This looks like mirrors haven't been setup just yet12:05
*** funzo has joined #tripleo12:05
mwhahahapabelanger: yea that's from the builder12:06
mwhahahapabelanger: actually isnt' that the logs fetching, weird12:06
*** limao_ has quit IRC12:07
*** limao has joined #tripleo12:07
pabelangermwhahaha: I think the issue is, toci_gate_test.sh is installing stuff before repo files have been rewritten12:07
mwhahahapabelanger: actually that's the end of the run, something else went wrong in that job12:08
mwhahahai think12:08
mwhahahai'll take a look at it more indeptb in a bit12:08
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Move multinode overcloud user to multinode settings  https://review.openstack.org/49667512:08
*** morazi has joined #tripleo12:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Release extra-args from multinode environment settings  https://review.openstack.org/49667612:09
sshnaidmnumans, ^^12:09
*** funzo has quit IRC12:09
*** hewbrocca_afk is now known as hewbrocca12:10
sshnaidmnumans, I release extra-args for you, please depend on my changes12:10
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140312:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295512:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243612:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]12:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]12:10
*** _jbadiapa has joined #tripleo12:10
openstackgerritOpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements  https://review.openstack.org/49276812:11
*** pkovar has quit IRC12:11
*** sbrzozow has joined #tripleo12:11
*** pblaho1 has joined #tripleo12:12
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: HAProxy: Make certmonger bundle the cert and key on renewal  https://review.openstack.org/49657212:13
*** ansmith has quit IRC12:13
*** ffiore_ has joined #tripleo12:16
*** brault has joined #tripleo12:17
*** abishop has joined #tripleo12:17
*** pblaho has quit IRC12:18
*** pblaho1 has quit IRC12:18
*** pblaho has joined #tripleo12:18
*** jcoufal has joined #tripleo12:19
d0ugalmhenkel: DId you find a solution?12:21
d0ugalmhenkel: that error suggests your python-tripleoclient is newer than your tripleo-common12:22
*** brault has quit IRC12:22
d0ugal(or, maybe tripleo-common is updated, but the workflow have not been reloaded into Mistral)12:22
*** brault has joined #tripleo12:22
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements  https://review.openstack.org/48739812:23
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements  https://review.openstack.org/48814812:23
mhenkeld0ugal: python-tripleoclient-6.2.0-1.el7ost.noarch  openstack-tripleo-common-6.1.0-2.el7ost.noarch12:23
mhenkelso, yes, there is a version difference12:23
d0ugalmhenkel: They wouldn't have the same version numbers...12:24
mhenkelhow can I tell?12:24
mhenkelbooth come from the same repo12:24
d0ugalInteresting12:25
mhenkelrhel-7-server-openstack-11-rpms12:25
d0ugalBasically, you need to make sure you have both of these changes...12:25
d0ugalhttps://github.com/openstack/python-tripleoclient/commit/f3acb69ffce3c630df5b3051779a7af21b96af4712:25
d0ugalhttps://github.com/openstack/tripleo-common/commit/9d158874301eaa9444811f2c21cc9d86c2eaad7a12:25
d0ugalThe error suggests your have the first one, as it is passing the parameter, but the workflow doesn't understand it so you get an error12:26
*** cylopez has joined #tripleo12:26
*** pkovar has joined #tripleo12:29
*** rlandy has joined #tripleo12:31
openstackgerritJulien Danjou proposed openstack/tripleo-heat-templates master: Fix Panko public port  https://review.openstack.org/49668912:32
numanssshnaidm, ack. i will rebase the patch accordingly12:32
numansthanks12:32
mhenkeld0ugal: you were right. I downgraded the python-tripleoclient to 6.1.0.6 and deployment started12:33
sshnaidmnumans, you can just set "Depends-On" on my two patches, no need to rebase12:33
d0ugalmhenkel: great12:33
numanssshnaidm, ack. i guess i don't need this patch then - https://review.openstack.org/#/c/495258/12:34
mhenkelso, it looks like that either the python-tripleoclient rpm was pushed too early to the rhel-7-server-openstack-11-rpms or somebody missed to push the tripleo-common rpm12:35
*** ecerquei has joined #tripleo12:36
sshnaidmnumans, yep, just try to use extra args12:36
*** psahoo has quit IRC12:37
mhenkeld0ugal: I checked the repos online and it is indeed the case12:38
dtantsurmwhahaha: (sorry, was having coffee) Ocata. this is where we introduces vbmc, which requires custom ports12:39
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy OpenShift using OOO on the overcloud  https://review.openstack.org/49447012:39
mwhahahadtantsur: it's a good ux improvement but not sure if it's backportable. We would need a bug to backport it as well. Maybe file a bug and explain a case for a backport12:42
dtantsurmwhahaha: yeah, I'm going to fix it for master first anyway12:42
*** rbrady has joined #tripleo12:43
*** rbrady has quit IRC12:43
*** rbrady has joined #tripleo12:43
*** gkadam has quit IRC12:43
*** jlabarre has joined #tripleo12:44
sshnaidmderekh, do you know whom to talk when there are network issues in rh1 cloud?12:46
openstackgerritNuman Siddique proposed openstack/tripleo-quickstart master: Add fset030 for OVN with containerized oc  https://review.openstack.org/49062412:46
mhenkeld0ugal: https://bugzilla.redhat.com/show_bug.cgi?id=148439612:47
openstackbugzilla.redhat.com bug 1484396 in rhosp-director "the rhel-7-server-openstack-11-rpms repo python-tripleoclient and openstack-tripleo-common are incompatible" [Unspecified,New] - Assigned to athomas12:47
*** dmacpher has joined #tripleo12:47
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo master: Fix panko port to match tht  https://review.openstack.org/49669412:48
*** udesale has joined #tripleo12:48
d0ugalmhenkel: thanks!12:48
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates master: Support deploying OVN as container services  https://review.openstack.org/46886012:49
*** atoth has quit IRC12:50
*** jlinkes has quit IRC12:50
jaosoriorpradk: is that something we need to backport? can you file a launchpad bug too?12:50
*** dpawar has quit IRC12:51
*** pchavva has joined #tripleo12:51
pradki think we changed this recently.. lemme double check12:51
pradkyea looks like we need to backport.. i'll file a bug12:52
pradkjaosorior, i guess this proves no one is using panko .. lol12:52
jaosoriorpradk: well, most deployers want to stick with the long term supported releases :/ that might be a reason12:53
pradkyea12:53
pradkk bug on the way12:53
jaosoriorpradk: thanks12:53
*** ffiore_ has quit IRC12:54
*** shardy has joined #tripleo12:55
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo master: Fix panko port to match tht  https://review.openstack.org/49669412:55
openstackgerritRaoul Scarazzini proposed openstack/tripleo-quickstart-extras master: Restore tags inside the main playbook  https://review.openstack.org/47349112:56
*** tzumainn has joined #tripleo12:56
jaosoriorpradk: don't we test panko in some scenario?12:57
pradkjaosorior, we do, perhaps not with ssl12:57
jaosorioraah that would make sense12:58
jaosoriorhaha well shit12:58
pradkand panko is deprecated in pike .. ;)12:59
pradkhopefully it will die in queen, lets see12:59
*** cdearborn has joined #tripleo12:59
egonzaleztrozet, ping13:00
trozetegonzalez: i got to go afk for a bit, will ping you when i get back13:01
*** oidgar has quit IRC13:01
egonzaleztrozet, roger13:01
*** gbarros has joined #tripleo13:02
*** lucas-hungry is now known as lucasagomes13:03
*** atoth has joined #tripleo13:03
*** fzdarsky_ has quit IRC13:08
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140313:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295513:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243613:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)13:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]13:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]13:10
*** jpena|lunch is now known as jpena13:10
*** florianf has quit IRC13:16
*** dsariel|afk has quit IRC13:16
*** oidgar has joined #tripleo13:18
*** dparkes has quit IRC13:20
*** florianf has joined #tripleo13:21
*** snecklifter has joined #tripleo13:21
openstackgerritJakub Libosvar proposed openstack/os-net-config stable/pike: Delete controller for standalone OVS bridges  https://review.openstack.org/49670713:23
pabelangermwhahaha: weshay_PTO_AM: EmilienM: could somebody please add the missing container job scenarios to quickstact-extras? That seemed to be the reason from stevebaker last night for the gate breakage13:24
*** bkopilov has joined #tripleo13:26
*** bnemec has joined #tripleo13:26
*** noslzzp has quit IRC13:27
*** amoralej is now known as amoralej|lunch13:27
shardydtantsur: Hey can we mark https://blueprints.launchpad.net/tripleo/+spec/node-discovery-by-range implemented?13:27
shardylooks like everything except the UI patch landed now?13:27
*** noslzzp has joined #tripleo13:28
*** ansmith has joined #tripleo13:28
dtantsurshardy: I'm fine with that, dunno about UI folks13:30
dtantsurbut yes, the UI patch was the last13:30
shardyhonza: ^^ can we declare the BP complete and raise a bug to track adding UI support?13:31
*** masco has quit IRC13:31
*** hewbrocca is now known as hewbrocca_afk13:33
*** oidgar has quit IRC13:34
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404313:35
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404313:37
*** fzdarsky has joined #tripleo13:37
*** links has quit IRC13:38
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404313:38
trownsshnaidm: could I get a review on https://review.openstack.org/475747 ? would be nice for the periodic job to work for container builds instead of needing to run it via "check manual"13:39
abishopshardy: hoping you can take another look at https://review.openstack.org/#/c/480587/ (commit message has been updated)13:42
lvdombrkrshardy: just to let you know for future it was my mistake i was added needless repo13:42
lvdombrkr shardy: now glance works fine13:42
*** agurenko has quit IRC13:43
lvdombrkr shardy: thank you in assist13:43
*** ansmith_ has joined #tripleo13:43
shardylvdombrkr: ack, well that's good news, glad you got it working :)13:43
shardyabishop: will do13:43
*** eck`gone is now known as eck`13:45
openstackgerritJakub Libosvar proposed openstack/os-net-config stable/pike: Delete controller for standalone OVS bridges  https://review.openstack.org/49670713:46
*** fpan has joined #tripleo13:46
*** noslzzp has quit IRC13:48
*** noslzzp has joined #tripleo13:49
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add ironic-inspector to the undercloud-containers job  https://review.openstack.org/47613013:51
*** Goneri has joined #tripleo13:57
*** limao has quit IRC13:58
abishopshardy: thank you! once that one merges, my own https://review.openstack.org/#/c/489239/ will needs reviews13:58
shardyabishop: ack - it'd be good if we could land those before the RC causes tripleo-common to branch tomorrow13:59
abishopshardy: whoa, yeah! In that case, can you please review https://review.openstack.org/#/c/489239/ ?14:00
*** oidgar has joined #tripleo14:01
abishopd0ugal: please revisit https://review.openstack.org/#/c/480587/; you approved once before, but commit msg was updated14:02
d0ugalabishop: will do14:02
shardyI just rechecked due to the ovb job failures, but the following patch in the series passed them, so perhaps we should just approve them14:03
abishopd0ugal, shardy, jaosorior: thank you for the fast response!14:04
*** hamzy has quit IRC14:05
*** florianf has quit IRC14:06
*** udesale has quit IRC14:08
*** florianf has joined #tripleo14:08
*** cylopez has quit IRC14:08
*** florianf has quit IRC14:08
pabelangertrown: where will 475747 run, on rdocloud?14:09
*** sshnaidm is now known as sshnaidm|afk14:09
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140314:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295514:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243614:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)14:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]14:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]14:10
openstackgerritMerged openstack/tripleo-common master: Add ironic-api healthcheck  https://review.openstack.org/49083614:12
*** amoralej|lunch is now known as amoralej14:12
*** bkopilov has quit IRC14:15
*** udesale has joined #tripleo14:15
*** ykarel has quit IRC14:17
*** florianf has joined #tripleo14:20
mwhahahagfidente: does ansible-ceph try and install epel?14:20
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Add panko publisher in the event pipeline  https://review.openstack.org/49672314:22
mwhahahagfidente: http://logs.openstack.org/74/495374/1/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/cf08cc9/logs/undercloud/var/log/mistral/executor.log.txt.gz#_2017-08-23_11_51_24_377 seems to have failed while trying to reach out to epel or something14:23
mwhahahagfidente: changed: [192.168.24.11] => (item=epel-release)\nfailed: [192.168.24.11] (item=python-setuptools) => {"failed": true, "item": "python-setuptools", "msg": "Failure talking to yum: Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again"}\nok: [192.168.24.11] => (item=libselinux-python)\n\nPLAY RECAP14:23
trownpabelanger: ya that runs on rdo cloud14:23
openstackgerritAdriano Petrich proposed openstack/instack-undercloud master: Add an hourly cron trigger for tripleo-ui logging  https://review.openstack.org/46960814:24
pabelangertrown: thanks14:25
apetrichhonza, I added the changes to delete cron-triggers on https://review.openstack.org/#/c/469608/ is that alright? it was removed between 9..1014:26
*** itlinux has joined #tripleo14:28
numanssshnaidm|afk, can you please see this review https://review.openstack.org/#/c/468860/ and the job http://logs.openstack.org/60/468860/25/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/e4d85bd/logs/undercloud/home/jenkins/overcloud-deploy.sh.txt.gz14:29
numanssshnaidm|afk, something seems to be wrong in the way overcloud-deploy.sh is generated14:30
*** udesale has quit IRC14:31
*** udesale__ has joined #tripleo14:31
*** itlinux has quit IRC14:34
lvdombrkrguys my Stack CREATE completed successfully but there is no overcloudrc generated in /home/stack14:36
lvdombrkrwhat does thats mean?14:36
*** mdnadeem has quit IRC14:37
shardylvdombrkr: did you ctrl-C from the client before the stack create completed?14:38
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo stable/ocata: Move gnocchi::api resource to run with wsgi setup  https://review.openstack.org/48943714:38
lvdombrkrshardy: what you mean?14:39
*** trown is now known as trown|brb14:41
*** vpickard_ is now known as vpickard14:41
*** hamzy has joined #tripleo14:43
*** ykarel has joined #tripleo14:47
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: WIP: Deploy OpenShift using OOO on the overcloud  https://review.openstack.org/49447014:47
shardylvdombrkr: If you exit the client early the overcloudrc isnt written, also check which directory you ran the overcloud deploy command in, as IIRC it writes to $PWD/overcloudrc14:47
gfidentemwhahaha yeah I noticed that error in a few runs14:47
gfidenteit looks like it tries yes14:48
mwhahahagfidente: can we supress that?14:48
gfidentewe had to allow it do install packages because on deployed-server there isn't everything we need14:48
mwhahahagfidente: cause that's going to be a problem14:48
gfidentebut yes looking into it14:48
*** ebarrera has quit IRC14:49
*** ebarrera has joined #tripleo14:49
*** anshul has quit IRC14:49
lvdombrkrshardy: how can i chack that stack was deployed reallu succees?14:50
*** trown|brb is now known as trown14:50
*** oanson has quit IRC14:53
*** oanson has joined #tripleo14:54
*** leitan has joined #tripleo14:56
*** yprokule has quit IRC14:57
*** fpan has quit IRC14:58
jaosoriorshardy: is there an easy way to tell if a deployment is containerized? from a container point of view15:00
jaosoriordprince: ^^15:02
shardyjaosorior: you mean if any services are running in containers, or a specific service?15:03
jaosoriorany service15:04
jaosoriordo we have a hiera key that tells that docker is being deployed? or something like that?15:04
dprincejaosorior: I think there is probably something in /proc15:05
dprincejaosorior: but we aren't doing any fancy tricks for that yet15:05
*** sshnaidm|afk is now known as sshnaidm15:06
jpenadprince: about the openstackclient command, I've just tested it on a local container and "openstack overcloud container image prepare" is exposed15:06
dprincejaosorior: as far as hiera goes I don't think we write anything yet to signal that a container is used. This would need to be per-service at this point as our architecture is currently hybrid15:06
*** ebarrera has quit IRC15:06
jpenaversions: python2-openstackclient-3.12.0-0.20170818132251.f67ebce.el7.centos.noarch and python-tripleoclient-7.2.1-0.20170823093447.4ae96dd.el7.centos.noarch (from the "/current" directory)15:06
dprincejpena: really. I'm perplex cause it was working for me too15:07
openstackgerritMichael Bayer proposed openstack/puppet-tripleo stable/ocata: Add option for innodb_flush_log_at_trx_commit = 2 for Galera only  https://review.openstack.org/49679015:07
dprincejpena: I'm building a container with just python-tripleoclient in it15:07
shardydprince: inside the container can you see the bind mounts?15:08
* shardy doesn't have a containerized deployment running atm to check15:08
amoralejdprince, i see the endpoint "openstack overcloud container image prepare" available in master repo15:08
dprinceshardy: if they are exposed. But they just show up as normal dirs15:08
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Release extra-args from multinode environment settings  https://review.openstack.org/49667615:09
dprincejpena, amoralej: http://paste.openstack.org/show/619186/15:09
amoralejdprince, what repo are you using15:09
amoralej?15:09
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Add panko publisher in the event pipeline  https://review.openstack.org/49672315:10
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140315:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295515:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243615:10
*** ooolpbot has quit IRC15:10
dprinceamoralej: https://trunk.rdoproject.org/centos7/75/a2/75a2e3bde576a18219eddedba9dcc796673f1d4b_a2b1e6ac15:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]15:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]15:10
dprinceamoralej: but fwiw I tried current-tripleo too and got similar15:10
sshnaidmnumans, please try again, I hope fixed it15:10
numanssshnaidm, sure. thanks.15:10
amoralejdprince, using current-passed-ci http://paste.openstack.org/show/619188/ let me check what's happening with current15:11
dprinceamoralej: cool15:11
mwhahahapradk: https://review.openstack.org/#/c/496751/ do we need this in pike rc1 for puppet-ceilometer?15:13
mwhahahapradk: nm just saw your update, i'll get it in15:13
pradkmwhahaha, yea just replied15:13
pradksorry for last minute15:13
jpenaamoralej: I used current in my test15:14
openstackgerritMichael Bayer proposed openstack/puppet-tripleo stable/newton: Add option for innodb_flush_log_at_trx_commit = 2 for Galera only  https://review.openstack.org/49679415:14
pabelangerany suggestions to see why this gate job failed: http://logs.openstack.org/00/495900/3/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/b09cf60/logs/undercloud/home/jenkins/failed_deployment_list.log.txt.gz15:15
*** udesale__ has quit IRC15:15
pabelangerresources.WorkflowTasks_Step2_Execution: ERROR I do not know where to look for that15:16
mwhahahapabelanger: see my comment about epel to gfidente15:16
pabelangermwhahaha: thanks, looking15:16
pabelangeroh noes, epel15:16
mwhahahapabelanger: yea15:17
mwhahahaceph-ansible is needing to install stuff which is problematic15:17
mwhahahaor ansible-ceph whichever it is15:17
pabelangerk, I don't yet understand all the bit, but is that happening in a docker container?15:19
trozetegonzalez: hey15:19
pabelangeror on the overcloud15:19
egonzaleztrozet, is tacker master tested in tripleo or apex?15:19
lvdombrkrshardy: any ideas why overcloudrc no generated?15:19
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Make archive policy configurable  https://review.openstack.org/49679615:20
pabelangerhttp://logs.openstack.org/00/495900/3/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/b09cf60/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz is a much nicer log15:20
trozetegonzalez: i'm going to be testing it soon with ooo/apex on ocata15:20
mwhahahapabelanger: i'm not sure where that occurs but yea that log is much nicer :D15:20
trozetegonzalez: why? what's up?15:20
egonzaleztrozet, saying this, because tacker requires mistral for vim monitoring, which requires tacker-conductor that as far i can find is not implemented15:21
pabelangermwhahaha: gfidente: do you happen to know where the epel install happens? In container or on overcloud node?15:21
pabelangermwhahaha: gfidente: hopefully overcloud15:21
mwhahahapabelanger: i'd assume overcloud15:21
pabelangermwhahaha: ya, I am hoping that too15:21
gfidentepabelanger mwhahaha sorry guys haven't really started as I am pulled into an admin issue :D15:21
pabelangerbecause, if overcloud. python-setuptools is already installed15:22
gfidentebut yeah it's installing some packageson the baremetal15:22
*** fpan has joined #tripleo15:22
*** pkovar has quit IRC15:23
pabelangerhttps://github.com/ceph/ceph-ansible ?15:23
pabelangeris that the playbook it is using15:23
mwhahahayes15:23
pabelangerhttps://github.com/ceph/ceph-ansible/blob/master/roles/ceph-defaults/defaults/main.yml#L5915:23
pabelangerokay, at least they expost it15:24
*** jlinkes has joined #tripleo15:24
pabelangerso, can pass in the package dependencies with epel15:24
mwhahahai like how epel-release is a dependency15:24
trozetegonzalez: afaik tacker is monolithic and does not have separated services15:24
pabelangerin fact, I would unset both15:24
pabelangerand just have tripleo manage them15:24
*** pkovar has joined #tripleo15:24
mwhahahaindeed15:24
egonzaleztrozet, changed in pike https://docs.openstack.org/tacker/latest/contributor/tacker_conductor.html15:25
trozetegonzalez: maybe that changed, I see https://github.com/openstack/tacker/tree/master/tacker/conductor15:25
trozetegonzalez: looks like we have some work to do then in ooo and puppet-tacker15:25
egonzaleztrozet, conductor is meant for RPC calls with mistral15:25
*** catintheroof has joined #tripleo15:25
egonzaleztrozet, yep. there is not that much config to do, just start a new service with same config file15:27
pabelangermwhahaha: gfidente: can we create a bug in launchpad to triage? Since it is breaking the gate15:27
trozetegonzalez: https://github.com/openstack/tacker/blob/master/tacker/conductor/conductorrpc/vim_monitor_rpc.py#L3015:27
mwhahahapabelanger: gfidente yea let me file a bug15:28
trozetegonzalez: are you going to handle doing those updates?15:28
pabelangermwhahaha: tyty15:28
egonzaleztrozet, -20% puppet skills15:28
trozetegonzalez: haha ok, do you want to create a LP and then assign to me and I will get to it?15:29
egonzaleztrozet, sure, blueprint or bug for backport?15:29
mwhahahapabelanger: gfidente https://bugs.launchpad.net/tripleo/+bug/171259915:31
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,Triaged]15:31
egonzaleztrozet, i can target this kolla bug to tripleo too https://bugs.launchpad.net/kolla-ansible/+bug/171087415:31
openstackLaunchpad bug 1710874 in kolla-ansible "tacker broken in pike, need conductor service" [Undecided,Fix released] - Assigned to Eduardo Gonzalez (egonzalez90)15:31
*** dsariel has joined #tripleo15:31
gfidentemwhahaha thanks guys15:31
gfidentemwhahaha pabelanger though some runs passed15:34
gfidenteso it's unclear when causes the issue with the epel repo itself15:34
gfidente*what15:34
gfidentebut I agree we should try not to use it anyway15:34
mwhahahagfidente: network because it's probably pulling remote15:34
mwhahahagfidente: we're trying to reduce our reliance on external stuff15:35
mwhahahaplus we shouldn't be allowing it to pull in packages anyway15:35
gfidenteso we can disable some tagfs15:35
gfidenteand that prevents installation of packages15:35
gfidentebut we need to install some stuff on the infra nodes15:35
pabelangergfidente: mwhahaha: yup, it is hitting epel directly, and not using our AFS mirrors. That is why it is intermittent. But bigger thing, depending on epel is bad now. I know a lot of effort has gone into remove that15:36
gfidenteright15:36
gfidentewe don't want epel anyway15:36
pabelangeryou shouldn't need python-setuptool, it should be there15:36
*** oidgar has quit IRC15:36
pabelangeralso, I would see why you need python-setuptools15:36
pabelangeris that building a package locally?15:36
mwhahahahttps://github.com/ceph/ceph-ansible/blob/master/roles/ceph-defaults/defaults/main.yml#L8015:37
mwhahahaprobably if ceph_origin is upstream15:38
mwhahahaor something15:38
amoralejdprince, it's working to me with current, do you have some log of failure?15:38
gfidentetrown so would https://bugs.launchpad.net/tripleo/+bug/1712432/comments/4 unblock us?15:38
openstackLaunchpad bug 1712432 in tripleo "mistral fails to start the ceph container" [High,Triaged] - Assigned to John Trowbridge (trown)15:38
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Set gnocchi archivepolicy in scenario001 job to high  https://review.openstack.org/49680015:39
pabelangermwhahaha: okay, I think I see a bigger issue: https://review.openstack.org/#/c/495900/15:40
pabelangermwhahaha: Why didn't gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container run in check15:40
pabelangerand only in gate15:40
pabelangersame with gate-tripleo-ci-centos-7-scenario001-multinode-oooq15:41
mwhahahapabelanger: we added that after i think15:41
mwhahahapabelanger: ie added yesterday afternoon15:41
pabelangerAh15:41
pabelangerokay15:41
mwhahahaspecifically for this :D15:41
pabelangerya15:41
pabelangercool15:41
mwhahahahrm seems to be happening a bunch more so i'm going to throw an alert on that bug15:43
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder  https://review.openstack.org/49400815:43
pabelangermwhahaha: what ansible-ceph recently added?15:45
mwhahahano15:46
mwhahahawell sort of15:46
*** bnemec has quit IRC15:46
egonzaleztrozet, cannot find/assign you (probably not having permissions in tripleo-drivers LP group) This is bug, https://bugs.launchpad.net/tripleo/+bug/171087415:46
openstackLaunchpad bug 1710874 in tripleo "tacker requires conductor service in pike" [Undecided,New]15:46
pabelangerthere is also the issue that mistral logs are not getting indexed by logstash15:46
pabelangerso, elastic-recheck didn't catch this15:46
*** bnemec has joined #tripleo15:47
vkmchey guys, I'm seeing some errors when trying to get container images on the compute node... http://paste.openstack.org/show/619195/15:47
*** itlinux has joined #tripleo15:47
pabelangervkmc: is that local or in the gate?15:47
trowngfidente: ya... not really blocked but ya that will fix it15:47
vkmcis there some known problem with the repo or is something on my env?15:47
vkmcpabelanger, local15:48
gfidentetrown you have a change to implement that or shall I?15:48
pabelangervkmc: ya, looks like networking issue to docker.io. If it was the gate, we have a reverse proxy to docker.io15:48
gfidentemwhahaha pabelanger fultonj I might have a fix for the scenarios15:49
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP create default settings for environment types  https://review.openstack.org/49540715:49
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default  https://review.openstack.org/47241215:50
Slowermandre: were you going to build new tripleoupstream images?15:50
* Slower checks to see if you did :)15:50
vkmcpabelanger, k, so... how should I be deploying the overcloud for testing purposes...? should I be using an additional environment file?15:51
pabelangergfidente: I couldn't mind why python-setuptools is needed. If you have a sucessful run of ansible-cept log, it would be helpful15:51
vkmcpabelanger, this is the overcloud deploy I'm performing "openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e $HOME/docker_registry.yaml"15:51
pabelangervkmc: when you say local, are you just using your local environment or is this in rdocloud for example15:52
*** gbarros has quit IRC15:52
larsksSo...it looks like newer versions of hiera (I've got 3.x locally) will break on many of the variables we're using in tripleo (it looks like "." becomes a hierarchy separator, so all the "tripleo.x.y.z" vars stop working).  Is that something for which there is/should an open issue?15:52
vkmcpabelanger, local environment15:52
pabelangervkmc: best you could do in that case would be setup https://docs.docker.com/registry/recipes/mirror/ then configure your docker pulls to hit your local registry first.15:53
trowngfidente: oh... didnt see comment 4 yet... thought we were just missing commits in tripleo-ci15:53
pabelangervkmc: you'd start caching the requests over going to docker.io everytime15:53
*** dixiaoli has joined #tripleo15:53
vkmcpabelanger, so... building images locally15:53
vkmcright?15:53
trowngfidente: ya I can put up patch for that15:54
vkmcpabelanger, does this always happen?15:54
pabelangervkmc: that is an option too. The proxy cache does the same thing, without you building15:54
gfidentetrown cool, ty! :D15:54
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Do not install epel-release with ceph-ansible  https://review.openstack.org/49680615:54
pabelangervkmc: in the gate it happens a lot. That is why we setup a reverse proxy cache to docker. Now our regional mirrors cache all images from docker.io15:54
gfidentemwhahaha pabelanger fultonj ^^15:54
vkmcpabelanger, can you point me to some doc explaining on how to configure my docker pulls to hit the local registry first?15:55
vkmcall right15:55
pabelangervkmc: mirror.dfw.rax.openstack.org:8081/registry-1.docker for exmaple15:55
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: Use resource collector for the fencing -> stonith ordering  https://review.openstack.org/49680715:55
pabelangervkmc: 1 sec, I can find the changes we made for the gate15:55
openstackgerritSteven Hardy proposed openstack/python-tripleoclient master: Consume new RoleConfig data in config download  https://review.openstack.org/48573215:55
vkmcpabelanger, k, thanks15:55
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Do not install epel-release with ceph-ansible  https://review.openstack.org/49680615:56
fultonjgfidente: i see, so you override the deps list with a list not containing epel15:57
gfidentefultonj yeah15:57
pabelangervkmc: https://review.openstack.org/491923/ seeing appears to be undercloud_docker_registry_mirror15:57
gfidenteonly for scenario jobs15:57
pabelangersetting*15:57
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Telemetry specific config for scenario001  https://review.openstack.org/49680015:57
*** dixiaoli has quit IRC15:57
vkmcpabelanger, thanks15:58
pabelangervkmc: np15:59
*** gbarros has joined #tripleo15:59
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Separate config_volume for ringbuilder  https://review.openstack.org/49400815:59
*** tongl has joined #tripleo15:59
*** Guest28943 is now known as sdake16:00
*** sdake has joined #tripleo16:00
*** bogdando has quit IRC16:00
*** florianf_ has joined #tripleo16:01
*** florianf has quit IRC16:02
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Telemetry specific config for scenario001  https://review.openstack.org/49680016:03
trozetegonzalez: thanks assigned it to myself16:05
*** bnemec has quit IRC16:06
*** funzo has joined #tripleo16:07
*** jkilpatr has quit IRC16:08
*** milan has quit IRC16:08
*** suuuper has quit IRC16:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140316:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295516:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243616:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171259916:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]16:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]16:10
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)16:10
*** funzo has quit IRC16:11
*** iranzo has quit IRC16:11
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart master: Add missing ceph-ansible install to appropriate releases  https://review.openstack.org/49682216:15
*** florianf_ is now known as florianf16:16
*** shardy has quit IRC16:16
*** jfrancoa has quit IRC16:19
*** jfrancoa has joined #tripleo16:20
*** snecklifter has quit IRC16:20
mandreSlower: yeah! I pushed new images yesterday16:22
mandreyou probably figured this by now :)16:22
*** weshay_PTO_AM is now known as weshay16:25
*** ykarel has quit IRC16:26
*** colonwq has quit IRC16:26
*** colonwq has joined #tripleo16:27
weshaypabelanger, did you get help re: missing container jobs? not sure what we're talking about yet to be honest16:27
mwhahahaweshay: i think it was having the scenario jobs on oooq-extras16:28
weshayto check gate tqe?16:28
mwhahahayea16:28
weshayk16:28
mwhahahacause it broke stuff last night16:28
*** dtantsur is now known as dtantsur|afk16:29
*** rcernin has quit IRC16:29
tonglmwhahaha: can you point to some docs that talks about more details about how to deploy Red Hat OpenStack with master code? I checked tripleo.org yesterday and have no clue where to start. Thanks!16:31
pabelangerweshay: mwhahaha: ya, thanks16:32
mwhahahatongl: not sure what more you want because those docks are the ones to deploy master packages from rdo16:32
weshaymwhahaha, pabelanger k.. thanks for letting me know, doesn't look like there are open reviews16:32
weshayso I'll go take care of it16:32
pabelangerweshay: ya, I haven't proposed anything16:32
tonglmwhahaha: Am I supposed to use instack-undercloud to pull master? http://tripleo.org/install/introduction/components.html#instack-undercloud16:34
mwhahahatongl: so i'm not sure what you're doing. to use master you would just use the master repos from rdo. then deploy it as needed.16:35
tonglmwhahaha: we added our networking driver to tripleo master/pike. currently the release version of OpenStack Platform 10 is newton and 11 is ocata, which don't have our code. I want to test our neutron driver on master/pike.16:37
mwhahahatongl: then you would just follow the dev instructions from tripleo.org16:37
*** ykarel has joined #tripleo16:38
*** jlinkes has quit IRC16:38
openstackgerritJuan Badia Payno proposed openstack/tripleo-heat-templates master: Containerized Fluentd client  https://review.openstack.org/46707216:41
*** mdnadeem has joined #tripleo16:42
openstackgerritJuan Badia Payno proposed openstack/tripleo-common master: Run fluentd container  https://review.openstack.org/48703816:42
openstackgerritDavid Sariel proposed openstack/tripleo-validations master: Add containers_sanity_check  https://review.openstack.org/48340316:42
openstackgerritDavid Sariel proposed openstack/tripleo-validations master: Add containers_sanity_check  https://review.openstack.org/48340316:43
*** itlinux has quit IRC16:44
*** aditya_r has joined #tripleo16:45
jristalmost ready. please review https://review.openstack.org/#/c/469608/16:47
jristjpich, EmilienM, d0ugal16:47
jristplz16:47
*** jpich has quit IRC16:48
*** mcornea has quit IRC16:48
*** tongl has quit IRC16:48
*** yprokule has joined #tripleo16:50
weshaypabelanger, https://review.openstack.org/#/c/496830/16:50
*** jkilpatr has joined #tripleo16:53
mwhahahauh oh16:53
mwhahahai think ovb just got broken16:53
mwhahaha2017-08-23 16:35:05 | Error: Package: rubygem-fluent-plugin-secure-forward-0.4.5-1.el7.noarch (quickstart-centos-opstools)16:54
mwhahaha2017-08-23 16:35:05 |            Requires: rubygem(proxifier)16:54
mwhahahahttp://logs.openstack.org/00/495900/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq/d94fe0b/logs/undercloud/home/jenkins/overcloud_image_build.log.txt.gz#_2017-08-23_16_35_0916:54
*** egonzalez has quit IRC16:54
*** itlinux has joined #tripleo16:55
*** jcoufal has quit IRC16:56
* mwhahaha files a bug and sighs16:56
*** trown is now known as trown|lunch16:56
mwhahahahttps://bugs.launchpad.net/tripleo/+bug/171262116:57
openstackLaunchpad bug 1712621 in tripleo "ovb overcloud image build broken due to rubygem-fluent-plugin-secure-forward" [Critical,Triaged]16:57
*** gbarros has quit IRC17:03
*** fzdarsky is now known as fzdarsky|afk17:03
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: allow tqe roles to work in sync with ARA  https://review.openstack.org/48223717:05
rookmwhahaha dprince what is the best way to restart a container?17:06
*** thrash is now known as thrash|biab17:07
*** lucasagomes is now known as lucas-afk17:07
*** pcaruana has quit IRC17:08
*** yprokule has quit IRC17:08
*** derekh has quit IRC17:10
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140317:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295517:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243617:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171259917:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171262117:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]17:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]17:10
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)17:10
openstackLaunchpad bug 1712621 in tripleo "ovb overcloud image build broken due to rubygem-fluent-plugin-secure-forward" [Critical,Triaged]17:10
mwhahaharook: can't you just do docker restart <container>17:11
pabelangerweshay: mwhahaha: gfidente: left a -1 on 496806. I think the patch, we should apply it to all scenories by default17:11
rookmwhahaha: i dunno... i haven't tried.. didn't know if that would break things17:12
rookill try it17:12
gfidentepabelanger well, that is the only scenario testing ceph-ansible for now17:12
rooki want to kick the haproxy container17:12
mwhahaharook: pretty sure that's the way to do it17:12
*** tesseract-RH has quit IRC17:12
pabelangergfidente: will others in the future?17:13
mwhahahagfidente: pabelanger: so it doesn't need to be in all scenarios (they don't all deploy ceph) but I'm still concerned why we have any packages being installed by this process17:13
pabelangergfidente: my fear, is it will get overlooked and we hit the issue again17:13
pabelangermwhahaha: yes, better understanding that too is helpful17:13
pabelangerI think these are dependencies for ansible17:13
mwhahahawhich don't we already have in our images?17:14
pabelangerbut really need to see a successful deploy log17:14
*** gbarros has joined #tripleo17:14
pabelangermwhahaha: I'd expect them to already be in place17:14
*** pkovar has quit IRC17:14
gfidentepabelanger mwhahaha so the default tht templates skip these two tags https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceph-ansible/ceph-base.yaml#L6017:14
pabelangerI am guessing the ceph-ansible playbooks is trying to manage all the things17:14
openstackgerritMerged openstack/tripleo-heat-templates master: docker: Stop all active ceilometer services during compute upgrade  https://review.openstack.org/49382517:15
gfidentepabelanger mwhahaha so the entire task is skipped in regular deployments https://github.com/ceph/ceph-ansible/blob/9da848abaca0798fa627dcb09aefed6636da76ea/roles/ceph-common/tasks/main.yml#L4417:16
mwhahahagfidente: so why are we enabling that here?17:16
gfidentefultonj ^^ you know which package in particular was missing on pre-deployed servers which required us to unset skiptags?17:17
*** achadha has joined #tripleo17:17
*** gbarros has quit IRC17:18
fultonjgfidente: yes, it was the ceph-jewel repo17:18
mwhahahawe turned on all package installs because the ceph repo wasn't there?17:19
gfidentemwhahaha fultonj the ceph repo is already there17:19
fultonjmwhahaha: gfidente let's shorten the list to only what we need17:19
*** achadha_ has joined #tripleo17:19
*** achadha has quit IRC17:19
*** gbarros has joined #tripleo17:19
gfidentefultonj no my point is why can't we skip the tags entirely in upstream CI?17:19
gfidentedid you find a particular package we needed to install which was not?17:20
gfidentebecause if not, I'd leave SkipTags to the default instead of overriding the list indeed17:20
fultonjgfidente: yes, i think it was the ceph-ansible pacakge. let me dig up what i found.17:20
fultonjgfidente: agreed17:20
gfidentewell we can tentatively clear the parameter and see what fails17:21
fultonjsure17:21
fultonjgfidente: so you're going to update https://review.openstack.org/#/c/496806 to temporarily have the original skip tags and we'll see how it does?17:22
gfidenteyeah17:22
*** achadha has joined #tripleo17:22
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Do not install epel-release with ceph-ansible  https://review.openstack.org/49680617:22
gfidentemwhahaha pabelanger problem is the infra nodes are not pre-installed with tripleo baremetal images17:23
gfidentebut let's see if anything is actually needed17:23
fultonjwe could have a document to support ceph in a deployed-server scenario17:24
mwhahahawe should for split stack17:24
gfidentefultonj I think the problem is just figuring what package we eventually need17:24
*** achadha_ has quit IRC17:24
mwhahahawe need to know what packages we need to know17:24
mwhahahabecause we need to be able to provide a list, we can't rely on ceph-ansible to just do magic17:25
fultonjgfidente: yes, then we put that shortend minimal list in the scenario's test and document it17:25
*** achadha_ has joined #tripleo17:25
fultonjmwhahaha: and we can get that done by passing it the list from tripleo17:25
gfidentefultonj but are you saying it's possible that ceph-mon/-osd wasn't installed at all?17:25
*** aufi has quit IRC17:25
fultonjgfidente: yes, this was deployed-servers IIRC17:26
*** achadha_ has quit IRC17:26
fultonjclarification: i know it's deployed servers, what ISTR is that those ceph packages were not there17:26
weshayany cores avail for an easy review? https://review.openstack.org/#/c/496822/17:26
pabelangerI'm still confused what is coming from EPEL in this case17:26
*** achadha_ has joined #tripleo17:26
gfidenteso that poses an additional problem, we need it to use the centos/sig repo eventually, still not epel17:26
gfidentepabelanger the RPMs from ceph.com require EPEL17:26
pabelangergfidente: Ya17:26
fultonjpabelanger: agreed, i don't think we need EPEL or setuptools and we'll get them out17:27
gfidenteso we need to have that as an alternative to ceph.com and not install epel-release in that case17:27
mwhahahabut we don't suppose the ceph.com packages17:27
pabelangergfidente: right, but which dependency is getting pulled in? Doesn't RDO have a ceph package?17:27
mwhahahas/suppose/support17:27
*** achadha_ has quit IRC17:27
*** achadha has quit IRC17:27
fultonjand we need it working with CentOS Storage sig, not ceph.com17:27
pabelangerya, that is what I'd expect17:27
gfidentepabelanger not RDO explicitly, but in centos we maintain the storage sig repo17:27
*** achadha has joined #tripleo17:27
pabelangerk17:27
pabelangersounds like it needs to be using http://mirror.dfw.rax.openstack.org/centos/7/storage/x86_64/ceph-jewel/ or http://mirror.dfw.rax.openstack.org/centos/7/storage/x86_64/ceph-hammer/17:28
*** achadha has quit IRC17:28
gfidenteone more thing is17:28
fultonjpabelanger: yes, jewel17:28
fultonji thought when scneario001 was working it was using centos sig, i probably just failed to do this clean up17:28
gfidentethose steps could be made entirely optional on containerized17:28
fultonjgfidente: that's what i thought too, but ceph-ansible needed to run ceph -s at the end that that command failed17:29
gfidenteahahahaa17:29
*** aditya_r has quit IRC17:29
gfidenteno thing is simple here17:29
fultonjgfidente: thus, i enabled the repos early on, and then ceph-ansible 'yum installed' it so it could run that command17:29
fultonjceph-ansible installed it because i dropped skip tags17:30
*** jokke_ has joined #tripleo17:30
fultonjwe made 001 work, but we need to clean up the mess now17:30
fultonjthe epel mess17:30
gfidenteright so I'd say17:30
*** aditya_r has joined #tripleo17:30
pabelangerfultonj: it is much work for epel mess?17:31
gfidentewe might actually *need* epel to install some deps for ceph.com17:31
gfidenteuntil it supports centos/sig repo17:31
fultonjyes, but we're not using ceph.com17:31
fultonjit will work with centos sig repos17:31
gfidentewell, but it's unsupported for now17:31
gfidentein ceph-ansible17:31
pabelangerYa, I think the fix is use centos sig repo, which we mirror17:31
gfidentepabelanger do you have any clue if that repo is pre-installed on the infra nodes?17:32
gfidenteor if oooq installs it?17:32
mwhahahaoooq installs it17:32
pabelangerya, somebody installs it17:32
fultonjbut this is deployed servers17:32
*** dpawar has joined #tripleo17:32
*** mdnadeem has quit IRC17:32
*** itlinux has quit IRC17:32
pabelangerhttps://review.openstack.org/496085/ cleans it up a bit17:32
pabelangerfultonj: what is deployed servers in this context? sorry I don't know much about tripleo testing17:33
mwhahahapabelanger: multinode17:33
mwhahahaso we use split stack to deploy on the provisioned servers17:33
fultonjpabelanger: aka splitstack17:33
*** jpena is now known as jpena|off17:33
fultonjthus the overcloud image, with the packages preinstalled, isn't used so the packages are not there17:34
pabelangerall I know, is non OVB jobs, should have it setup17:34
mwhahahapabelanger: when using that, tripleo installs the packages it needs where as in ovb we're using images so we skip installing packages17:34
fultonj+117:34
*** rcernin has joined #tripleo17:34
*** dpawar has quit IRC17:34
*** achadha has joined #tripleo17:34
pabelangermwhahaha: k, so that process would need to also setup mirrors for AFS. I am hoping it does17:34
fultonj"tripleo installs the packages it needs" i thought i made a change so it would include ceph-jewel for scenario001, i will try to find it17:35
mwhahahapabelanger: we do, oooq/tripleo-ci does it17:35
pabelangerk17:35
*** achadha has quit IRC17:35
*** tongl has joined #tripleo17:35
mwhahahawe just need to make sure the ceph repo gets installed (which it should)17:35
*** achadha has joined #tripleo17:36
pabelangerhttp://logs.openstack.org/82/490082/1/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/0e3413d/logs/undercloud/etc/yum.repos.d/quickstart-centos-ceph-jewel.repo.txt.gz17:36
pabelangerhttp://logs.openstack.org/82/490082/1/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/0e3413d/logs/subnode-2/etc/yum.repos.d/quickstart-centos-ceph-jewel.repo.txt.gz17:36
pabelangerboth nodes have it setup17:36
*** ykarel has quit IRC17:37
*** tosky has quit IRC17:37
gfidentemwhahaha pabelanger yeah and we set ceph_origin: distro in tht17:37
gfidenteso ceph-ansible isn't adding the ceph.com repo anyway https://github.com/ceph/ceph-ansible/blob/23f551a05fa948c0c7e291f863ebae2075661f40/roles/ceph-common/tasks/installs/install_on_redhat.yml#L1617:37
pabelangerokay, cool17:38
pabelangerso dropping epel-release will stop yum.repo for epel getting added to the node17:38
gfidenteyeah it brings us back to the initial patchset which overrides the list of packages17:38
pabelangerwhich, then breaks because yum client tries to get indexes17:38
gfidentewe can probably trim down that list further17:38
fultonj+117:38
pabelangerbut, I think mwhahaha has a good point. Why do we even need any of those dependencies now17:38
pabelanger++17:39
*** gbarros has quit IRC17:39
gfidentebut we do have to set skiptags empty or it won't install the necessary ceph packages17:40
mwhahahaok well let's just make sure we don't need extra deps17:40
gfidentealso, confirmed the ceph.com repo is not there as intended http://logs.openstack.org/00/495900/3/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/b09cf60/logs/undercloud/etc/yum.repos.d/17:40
mwhahahait's better to know what we actually need than just guessing17:40
mwhahahai have a feeling we shouldn't need any of those deps17:41
gfidentemwhahaha right I assumed someone did that in ceph-ansible for a reason :D17:41
*** dpawar has joined #tripleo17:41
mwhahahai don't :D17:41
gfidentebut we can try a blank list17:41
gfidenteack17:41
*** ebarrera has joined #tripleo17:42
fultonjif we're using deployed-servers, then we expect puppet-nova and similar to install the nova packages17:42
mwhahahalet's try a blank list and add to it17:42
mwhahaharight i get that concept but these packages aren't related to ceph because those should be handled when we yum install the ceph packages17:42
fultonjblank_list++17:42
mwhahahathe deps seems to be for the dev workflow17:42
mwhahahanot prod17:42
fultonjmwhahaha: yes, i agree17:43
mwhahahaso lets go with a blank list for now and if we need something just document it in the scenario17:43
fultonjmy point was that we can't tell ceph-ansible to skip all installs17:43
fultonjsounds good to me17:43
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: WIP create default settings for environment types  https://review.openstack.org/49540717:43
fultonjright, we exclude non-ceph installs with a blank list and grow it minimally if at all17:44
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default  https://review.openstack.org/47241217:44
*** jcoufal has joined #tripleo17:46
*** trown|lunch is now known as trown17:46
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Do not install additional CentOS packages via ceph-ansible  https://review.openstack.org/49680617:48
gfidenteok this should do it then17:49
gfidenteI think I am going AFK for today17:49
gfidenteplease pick it up should it fail!17:49
mwhahahak17:49
fultonjgfidente: ok17:49
*** gbarros has joined #tripleo17:49
gfidentethanks for helping17:50
gfidentewe can probably "port" some of it in ceph-ansible17:50
gfidenteitself17:50
*** itlinux has joined #tripleo17:50
pabelangerokay, Hmm17:54
pabelangernm, was looking at wrong location17:55
*** thrash|biab is now known as thrash17:56
*** social has quit IRC17:56
pabelangerweshay: https://review.openstack.org/496830/ is that the review to catch the regression from last night?17:57
*** pblaho has quit IRC17:57
*** tosky has joined #tripleo17:58
trowni missed it... what was the regression last night?18:00
pabelangertrown: https://review.openstack.org/496445/18:01
pabelangerthat broke the gate18:01
pabelangerspecifically, scenario container jobs18:02
*** gfidente has quit IRC18:04
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP create default settings for environment types  https://review.openstack.org/49540718:04
trownpabelanger: ah thanks18:04
openstackgerritMerged openstack-infra/tripleo-ci master: Add containers periodic test  https://review.openstack.org/47574718:04
trownpabelanger: then ya weshay's project-config patch should prevent that in the future18:05
pabelangertrown: ya, basically, I am in favor of insta-reverts for things that break gate :) Then figure out why it was allowed to pass check.18:05
pabelangertrown: ++18:05
pabelangerso far, things have been looking really good this weel18:06
trownya18:06
*** pkovar has joined #tripleo18:06
pabelangerweek*18:06
trownknock on wood just to be safe18:06
pabelangerYa, I think most of the hard stuff is done18:06
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Change devmode to deploy containerized services by default  https://review.openstack.org/47241218:06
*** catintheroof has quit IRC18:07
*** funzo has joined #tripleo18:07
*** pkovar has quit IRC18:08
*** gbarros_ has joined #tripleo18:09
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140318:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295518:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243618:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171259918:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171262118:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]18:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]18:10
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)18:10
openstackLaunchpad bug 1712621 in tripleo "ovb overcloud image build broken due to rubygem-fluent-plugin-secure-forward" [Critical,Triaged]18:10
*** gbarros has quit IRC18:11
*** funzo has quit IRC18:12
*** jfrancoa has quit IRC18:12
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add port security option for OVB with no network isolation  https://review.openstack.org/49683818:14
*** yamahata has joined #tripleo18:16
*** apetrich has quit IRC18:18
*** ansmith__ has joined #tripleo18:19
itlinuxhi cschwede_: are you around?18:19
*** apetrich has joined #tripleo18:20
mwhahahapabelanger: did you ever figure out the cause of http://logs.openstack.org/82/490082/1/gate/gate-tripleo-ci-centos-7-scenario001-multinode-oooq-container/0e3413d/logs/undercloud/home/jenkins/repo_setup.log.txt.gz#_2017-08-23_16_17_0518:21
*** aditya_r has quit IRC18:21
pabelangermwhahaha: ya, my patch will fix that.18:22
mwhahahapabelanger: ok18:22
pabelangermwhahaha: I think the curl request is 40418:22
*** aditya_r has joined #tripleo18:22
pabelangerand we get empty data18:22
pabelangermy patch will now validate and error if empty18:22
*** fpan has quit IRC18:23
*** ebarrera has quit IRC18:23
*** dixiaoli has joined #tripleo18:24
*** dixiaoli has quit IRC18:28
*** oidgar has joined #tripleo18:29
*** ansmith has quit IRC18:31
*** ansmith_ has quit IRC18:32
*** aditya_r has quit IRC18:34
*** gbarros_ has quit IRC18:35
*** jcoufal has quit IRC18:36
*** jcoufal has joined #tripleo18:38
*** cschwede_ has quit IRC18:40
*** brault has quit IRC18:44
*** rcernin has quit IRC18:47
*** ansmith_ has joined #tripleo18:49
*** ansmith has joined #tripleo18:49
mwhahahaowalsh: you around?18:49
pabelangerweshay: trown: do you mind looking at 496085 simplifies mirror setup18:51
pabelangeralso adds some error handling18:51
*** jtomasek_ has quit IRC18:52
trownpabelanger: sure18:53
*** akrivoka has quit IRC18:54
*** fpan has joined #tripleo18:55
*** shardy has joined #tripleo18:57
tonglif I want to use upstream master, is "master" the right --release option for quickstart.sh?19:05
*** dpawar has quit IRC19:05
tonglalso, if anything goes wrong for quickstart, where can I look at the relevant logs?19:06
openstackgerritAlex Schultz proposed openstack/instack-undercloud master: Disable compute auto disabling  https://review.openstack.org/49685119:07
*** ansmith has quit IRC19:09
*** itlinux has quit IRC19:10
*** ansmith has joined #tripleo19:10
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140319:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295519:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243619:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171259919:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171262119:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]19:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]19:10
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)19:10
openstackLaunchpad bug 1712621 in tripleo "ovb overcloud image build broken due to rubygem-fluent-plugin-secure-forward" [Critical,Triaged]19:10
*** oidgar has quit IRC19:11
*** ansmith_ has quit IRC19:13
*** eck` is now known as eck`gone19:16
pabelangerweshay: 496830 merged, but no containers-scenario003 jobs being run. I believe that is because you need to update the files regex in zuul/layouts.yaml19:25
weshayhrm.19:26
* weshay looks19:26
weshayin 1-1, so slowly looking19:26
*** ansmith_ has joined #tripleo19:26
pabelangerweshay: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n1799 will need to atleast match quickstart-extras files19:27
pabelangerlikely something like roles/.*19:27
*** ebarrera has joined #tripleo19:28
*** paramite has quit IRC19:30
weshaypabelanger, juse roles/.* would work?19:32
pabelangerweshay: likely roles and playbooks directories19:34
pabelangerweshay: same for ci-scripts and config, what ever files apply to scenerio3 need to be listed19:35
pabelangeryou can see why I am not a fan, you end up building a large static list19:35
weshayk19:36
weshaylooks a bit more19:36
weshaythanks for the hand19:36
mwhahaharoles/ would also trigger on THT which might not be ideal19:37
pabelangerya, you'd have to make the regex smarter to skip tripleo-heat-templates19:38
weshaypabelanger, w/ can you use a skip-if w/ project and files just from that project?19:39
pabelangerweshay: you'd need a unique job name for aside project19:41
pabelangersaid*19:41
*** pchavva has quit IRC19:41
weshaytrown, do you have any suggestions here?19:41
pabelangerzuulv3 changes is actually. Really not sure how much still applies19:41
weshayan alternative is to list out each tqe role/foo role/bar19:42
*** itlinux has joined #tripleo19:43
weshayseems sloppy, but it's the only thing that is clear to me atm19:43
pabelangerk, looks like we do still support it19:43
weshaypabelanger, and btw.. I'd need to do the same for https://review.openstack.org/#/c/496831/19:44
pabelangeryou'd need to match the / in the sub directory name for roles19:44
pabelangertripleo-heat-templates doesn't have any sub directories19:45
weshayah cool19:45
mwhahahayea19:45
weshaypretty lucky19:46
pabelangerroles\/(.*\/)19:46
pabelangeror something like that19:46
*** itlinux has quit IRC19:47
weshayya.. playing w/ it now.. /me tests that19:49
weshaypabelanger, that passes19:50
*** eck`gone is now known as eck`19:51
*** sbrzozow has quit IRC19:55
*** shardy has quit IRC20:00
*** itlinux has joined #tripleo20:01
*** tongl has quit IRC20:01
*** funzo has joined #tripleo20:05
owalshmwhahaha, what's up?20:08
mwhahahaowalsh: nm, was going to ask about a bug but I found consecutive_build_service_disable_threshold20:09
*** funzo has quit IRC20:10
mwhahahaowalsh: just fyi, https://bugs.launchpad.net/tripleo/+bug/1712632 which explains why i've randomly seen the undercloud's nova-compute disabled. I've proposed disabling that for our undercloud20:10
openstackLaunchpad bug 1712632 in tripleo "Check that enough nodes are in "available" state with maintenance mode off error" [High,In progress] - Assigned to Alex Schultz (alex-schultz)20:10
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140320:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295520:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243620:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171259920:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171262120:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]20:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]20:10
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)20:10
openstackLaunchpad bug 1712621 in tripleo "ovb overcloud image build broken due to rubygem-fluent-plugin-secure-forward" [Critical,Triaged]20:10
owalshmwhahaha, yes, vkmc hit that the other day when the undercloud ran out of disk space20:11
mwhahahait's a really bad UX when that happens on the undercloud :/20:12
owalshmwhahaha, I think it would be better if the validation reported that the compute service had been auto-disabled20:12
mwhahahaowalsh: we should have both but I don't think we should auto disable in the undercloud case20:12
mwhahahaaka ComputeFilter failures are really crappy20:13
weshaypabelanger, mwhahaha https://review.openstack.org/49686220:13
weshayand btw.. this is a cool site http://regexr.com/20:13
owalshmwhahaha, that was my initial reaction but if we get 10 consecutive build failures then something ain't right20:14
*** itlinux has quit IRC20:14
mwhahahaowalsh: yea but i've had it happen without build filures20:14
owalshmwhahaha, then I think it's broken20:14
mwhahahaowalsh: it's possible, so given current cycle i'd like to disable it for now. so i've proposed patches for that now to puppet-nova/instack-undercloud20:15
*** Goneri has quit IRC20:15
*** ansmith has quit IRC20:17
*** ansmith_ has quit IRC20:17
*** itlinux has joined #tripleo20:17
owalshmwhahaha, ah, that bug came from vkmc... it think we are doomed when the undercloud runs out of space so it's not going to help much but I agree the usability sucks and it took us quite a while to figure that one out20:19
mwhahahayea it took me a few hours the first time it happened too20:19
mwhahahaso i'd like not to subject end users or some poor support folks to that20:19
owalshmwhahaha, ok with me if it's disabled, deploy will fail eventually in any case.. but maybe a tech debt bug to catch this condition in the pre deploy validation for queens?20:20
mwhahahaowalsh: yea that would make sense20:20
*** jkilpatr has quit IRC20:22
*** toure is now known as toure_biab20:26
*** dprince has quit IRC20:26
*** rcernin has joined #tripleo20:36
*** achadha_ has joined #tripleo20:38
*** achadha has quit IRC20:38
rcerninshadower: got minute?20:39
rcerninshadower: about ooo-validations20:39
openstackgerritMerged openstack/tripleo-quickstart master: Add missing ceph-ansible install to appropriate releases  https://review.openstack.org/49682220:40
openstackgerritMerged openstack/tripleo-heat-templates master: Configure listen_address for libvirtd when TLS is enabled  https://review.openstack.org/49649220:40
*** itlinux has quit IRC20:41
*** ecerquei has quit IRC20:41
*** achadha_ has quit IRC20:42
*** jcoufal has quit IRC20:43
*** achadha has joined #tripleo20:47
*** ffiore has joined #tripleo20:57
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add option to build $RELEASE version of containers images  https://review.openstack.org/49692020:58
*** trown is now known as trown|outtypewww21:01
*** ffiore has quit IRC21:06
*** fpan has quit IRC21:08
radezmwhahaha: is the control vip on a single control node deploy brought up by os-net-config?21:08
radezmy more recent issue is the ip on the port labeled control vip doesn't come up. the db is listening there so nothing can connect to the db21:09
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates master: WIP: Resolve dependency issues prior to doing yum update  https://review.openstack.org/49692121:09
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140321:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295521:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243621:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171259921:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171262121:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]21:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]21:10
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)21:10
openstackLaunchpad bug 1712621 in tripleo "ovb overcloud image build broken due to rubygem-fluent-plugin-secure-forward" [Critical,Triaged]21:10
mwhahaharadez: i don't know21:11
radezkk21:12
*** SlickNik has left #tripleo21:17
*** SlickNik has joined #tripleo21:18
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart master: Enable opstools-testing repo  https://review.openstack.org/49692321:20
mwhahahalarsks, dmsimard -^21:21
*** leitan has quit IRC21:22
*** itlinux has joined #tripleo21:25
*** ffiore has joined #tripleo21:25
*** bfournie has quit IRC21:25
itlinuxis the tripleo default selinux now permissive? I thought it was set to enforced21:28
*** cdearborn has quit IRC21:28
mwhahahaupstream has always been permissive, downstream is enforced21:28
*** atoth has quit IRC21:29
mwhahahabecause openstack-selinux isn't always up to date with the upstream changes21:29
*** jkilpatr has joined #tripleo21:29
itlinuxahh ok.. thanks mwhahaha.:. so this can be changed once deployed TY21:33
*** abishop has quit IRC21:37
*** fpan has joined #tripleo21:37
*** morazi has quit IRC21:42
*** tongl has joined #tripleo21:43
*** ffiore_ has joined #tripleo21:51
*** ffiore has quit IRC21:52
*** jlabarre has quit IRC21:53
*** ffiore_ has quit IRC21:54
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates master: Add support for Dell EMC Isilon Manila  backend  https://review.openstack.org/48889622:04
openstackgerritSteve Baker proposed openstack/tripleo-quickstart-extras master: Filter image list by deployed services  https://review.openstack.org/49695022:04
*** ebarrera has quit IRC22:10
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140322:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295522:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243622:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171259922:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171262122:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]22:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]22:10
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)22:10
openstackLaunchpad bug 1712621 in tripleo "ovb overcloud image build broken due to rubygem-fluent-plugin-secure-forward" [Critical,Triaged]22:10
openstackgerritLukas Bezdicka proposed openstack/puppet-tripleo master: HAproxy should get full response from ironic-inspector  https://review.openstack.org/49695422:17
openstackgerritRajini Karthik proposed openstack/puppet-tripleo master: Support for Dell EMC VNX Manila Driver  https://review.openstack.org/49064222:19
*** achadha_ has joined #tripleo22:26
*** limao has joined #tripleo22:27
*** achadha__ has joined #tripleo22:28
*** rlandy is now known as rlandy|bbl22:28
*** achadha has quit IRC22:29
*** achadha__ has quit IRC22:29
*** itlinux has quit IRC22:29
*** achadha has joined #tripleo22:30
*** achadha_ has quit IRC22:30
*** leitan has joined #tripleo22:32
openstackgerritMerged openstack/tripleo-quickstart master: Clean up repo_cmd_before command for setup-repo  https://review.openstack.org/49608522:33
*** leitan has quit IRC22:36
*** dparkes has joined #tripleo22:37
*** dparkes has quit IRC22:45
*** ebarrera has joined #tripleo22:49
*** achadha_ has joined #tripleo22:51
*** achadha_ has quit IRC22:53
*** achadha_ has joined #tripleo22:53
*** achadha has quit IRC22:54
*** achadha has joined #tripleo22:55
*** achadha_ has quit IRC22:57
*** achadha has quit IRC23:00
*** achadha has joined #tripleo23:00
*** achadha_ has joined #tripleo23:06
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/169140323:10
openstackLaunchpad bug 1691403 in tripleo "containerized overcloud - neutron-openvswitchagent fails to start" [Critical,In progress] - Assigned to Steve Baker (steve-stevebaker)23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/170295523:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171243623:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171259923:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/171262123:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1702955 in tripleo "tripleo upgrade jobs timeout on stable/ocata" [Critical,Triaged]23:10
openstackLaunchpad bug 1712436 in tripleo "stable/newton gate-tripleo-ci-centos-7-nonha-multinode-oooq is broken due to Controller create timeout" [Critical,Triaged]23:10
openstackLaunchpad bug 1712599 in tripleo "scenario001 fails because ceph-ansible is trying to install epel-release" [Critical,In progress] - Assigned to Giulio Fidente (gfidente)23:10
openstackLaunchpad bug 1712621 in tripleo "ovb overcloud image build broken due to rubygem-fluent-plugin-secure-forward" [Critical,Triaged]23:10
*** achadha has quit IRC23:10
*** achadha_ has quit IRC23:11
*** athomas has quit IRC23:11
*** limao_ has joined #tripleo23:16
*** limao has quit IRC23:17
openstackgerritMerged openstack/tripleo-heat-templates master: More fixes for the Ceph docker images url parsing  https://review.openstack.org/49537423:24
*** tosky has quit IRC23:25
*** rbowen has quit IRC23:28
*** tzumainn has quit IRC23:32
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common master: Updated from global requirements  https://review.openstack.org/48739823:37
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements  https://review.openstack.org/48814823:37
*** rbowen has joined #tripleo23:45
*** tongl has quit IRC23:52
*** bnemec has joined #tripleo23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!