Monday, 2018-05-14

openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044500:00
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086000:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097200:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]00:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)00:10
*** slaweq has joined #tripleo00:11
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add instance reboot ssh test during upgrade  https://review.openstack.org/55385800:11
*** slaweq has quit IRC00:16
*** mburned has joined #tripleo00:29
*** etingof has quit IRC00:32
*** gbarros has joined #tripleo00:44
*** dmacpher has joined #tripleo00:45
*** etingof has joined #tripleo00:45
*** etingof has quit IRC00:53
*** etingof has joined #tripleo01:07
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086001:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097201:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]01:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)01:10
*** psahoo has joined #tripleo01:12
*** louis_ has joined #tripleo01:21
*** gbarros has quit IRC01:25
*** dprince has joined #tripleo01:27
*** gsdgsdg has joined #tripleo01:27
*** gsdgsdg changes topic to "/!\ THIS CHANNEL HAS MOVED TO #krustykrab /!\"01:28
*** ykarel has joined #tripleo01:30
*** louis_ has quit IRC01:31
*** louis_ has joined #tripleo01:35
*** louis_ has left #tripleo01:36
openstackgerritMerged openstack/python-tripleoclient master: Add --stack-only cli arg  https://review.openstack.org/56571701:42
*** gsdgsdg has quit IRC01:44
*** bkopilov has quit IRC01:56
*** dprince has quit IRC02:10
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086002:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097202:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]02:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)02:10
*** elgxl has joined #tripleo02:13
*** ykarel has quit IRC02:17
*** elgxl has quit IRC02:18
openstackgerritMerged openstack/tripleo-quickstart-extras master: Align UC net/env-files override with instack  https://review.openstack.org/56363602:29
*** etingof has quit IRC02:42
*** dbecker has quit IRC02:47
*** psachin has joined #tripleo02:53
*** etingof has joined #tripleo02:56
*** gbarros has joined #tripleo02:57
*** dbecker has joined #tripleo03:00
*** gbarros has quit IRC03:01
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086003:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097203:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]03:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)03:10
sl0wsshnaidm|bbl: see also https://review.openstack.org/#/c/565556/03:13
sl0wsshnaidm|bbl: I'll build a new pip pkg in the morning03:14
sl0wsshnaidm|bbl: unless you tell me it's super urgent in the next 2 hrs :)03:14
*** etingof has quit IRC03:22
openstackgerritTristan Cacqueray proposed openstack/diskimage-builder master: rpm-distro: set the contentdir yum var  https://review.openstack.org/56818003:31
*** bkopilov has joined #tripleo03:31
*** etingof has joined #tripleo03:35
*** etingof has quit IRC03:41
EmilienMmwhahaha, weshay: should we approve https://review.openstack.org/#/c/567636/ without OVB?03:42
mwhahahaYea03:42
mwhahahaThat change is for the promotion process03:42
mwhahahaWe don't rebuild containers03:43
*** udesale has joined #tripleo03:44
EmilienMmwhahaha: done03:48
*** shreshtha has joined #tripleo03:50
*** agopi has quit IRC03:52
*** agopi has joined #tripleo03:52
*** etingof has joined #tripleo03:54
*** radeks_ has joined #tripleo03:58
*** karthiks has quit IRC03:58
*** threestrands has joined #tripleo03:59
*** threestrands has quit IRC03:59
*** threestrands has joined #tripleo03:59
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy Docker via Ansible and not Puppet  https://review.openstack.org/56137704:00
Tenguhello there04:00
*** threestrands_ has joined #tripleo04:00
*** threestrands_ has quit IRC04:01
*** ykarel has joined #tripleo04:01
*** threestrands_ has joined #tripleo04:02
*** ykarel has quit IRC04:03
*** threestrands has quit IRC04:04
*** ykarel has joined #tripleo04:05
*** radeks_ has quit IRC04:08
openstackgerritEmilien Macchi proposed openstack/ansible-role-container-registry master: Add validation for docker registry v2  https://review.openstack.org/56818504:10
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086004:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097204:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)04:10
TenguEmilienM: heya! is the channel topic right?!04:10
openstackgerritEmilien Macchi proposed openstack/ansible-role-container-registry master: Add validation for docker registry v2  https://review.openstack.org/56818504:10
*** slaweq has joined #tripleo04:11
Tenguguess nope:  [03:27] -!- gsdgsdg changed the topic of #tripleo to: /!\ THIS CHANNEL HAS MOVED TO #krustykrab /!\   smells fishy.04:11
*** gvrangan has joined #tripleo04:13
*** ramishra has joined #tripleo04:14
*** slaweq has quit IRC04:16
*** gvrangan has quit IRC04:19
*** gvrangan has joined #tripleo04:21
*** gvrangan has quit IRC04:21
*** gvrangan has joined #tripleo04:22
*** gvrangan has quit IRC04:26
EmilienMTengu: damn indeed04:29
EmilienMthanks04:29
Tengunp :). You might want to add some privy configuration to the channel I guess04:29
*** EmilienM changes topic to "Welcome to Rocky. CI status: YELLOW (OVB failing on master) | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest"04:30
EmilienMTengu: not sure how04:33
EmilienMTengu: can you change the topic?04:33
TenguEmilienM: just checked the chanserv help04:33
*** Tengu changes topic to "foo"04:33
Tenguapparently..04:33
EmilienMholy04:33
Tengu:)04:33
TenguEmilienM: TOPICLOCK and SECURE and KEEPTOPIC would be interesting I guess :)04:34
Tenguthe first allows topic change only for users with +t flag, meaning some kind of list has to be created/maintained.04:34
Tengubut at least.. if a not-authorized-user tries to change the topic, it will be reverted by chanserv04:34
*** dxiri has joined #tripleo04:35
Tenguthis obviously wouldn't be a bad idea.04:35
*** Tengu changes topic to "Welcome to Rocky. CI status: YELLOW (OVB failing on master) | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest"04:35
Tengu(back to correct topic now ;))04:35
EmilienMI'm not op04:36
EmilienMI can't make changes on chanels04:36
Tenguduh04:36
EmilienMoh well, I'm off anyway.. we'll look at that later04:36
Tengu:)04:36
* Tengu waiting for info for his first day04:37
TenguEmilienM: have a good night then :). I'll make some coffee/tea.04:37
Tenguguess Alex would be the right guy to ping later for the channel issues.04:37
*** threestrands_ has quit IRC04:39
*** links has joined #tripleo04:41
*** pdeore has joined #tripleo04:49
*** udesale has quit IRC04:51
*** udesale has joined #tripleo04:56
*** fragatina has quit IRC04:57
*** cshastri has joined #tripleo05:04
*** threestrands has joined #tripleo05:04
*** fragatina has joined #tripleo05:07
*** threestrands has quit IRC05:07
*** yprokule has joined #tripleo05:08
*** skramaja has joined #tripleo05:10
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086005:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097205:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]05:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)05:10
*** gvrangan has joined #tripleo05:14
*** ratailor has joined #tripleo05:17
*** gvrangan has quit IRC05:18
*** skramaja_ has joined #tripleo05:18
*** skramaja has quit IRC05:18
*** fragatina has quit IRC05:22
*** fragatina has joined #tripleo05:22
*** udesale_ has joined #tripleo05:23
*** agurenko has joined #tripleo05:24
*** udesale has quit IRC05:25
*** jfrancoa has joined #tripleo05:31
*** chkumar|off has quit IRC05:32
*** dxiri has quit IRC05:33
*** chandankumar has joined #tripleo05:33
*** radeks_ has joined #tripleo05:34
*** radek__ has joined #tripleo05:35
*** quiquell|off is now known as quiquell05:37
openstackgerritTristan Cacqueray proposed openstack/diskimage-builder master: rpm-distro: set the contentdir yum var  https://review.openstack.org/56818005:39
*** paramite has joined #tripleo05:39
*** fragatina has quit IRC05:43
*** marios has joined #tripleo05:46
*** dciabrin has quit IRC05:47
*** dciabrin has joined #tripleo05:47
*** radeks_ has quit IRC05:50
*** zshi has quit IRC05:52
*** zshi has joined #tripleo05:53
*** assassin has quit IRC05:59
*** skramaja_ is now known as skramaja06:03
*** karthiks has joined #tripleo06:04
openstackgerritMerged openstack/tripleo-common master: Refactor functions out of ProcessTemplateAction  https://review.openstack.org/56246006:07
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086006:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097206:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)06:10
*** slaweq has joined #tripleo06:11
*** ramishra has quit IRC06:12
*** adrianreza has quit IRC06:12
*** gfidente has joined #tripleo06:16
*** gfidente has quit IRC06:16
*** gfidente has joined #tripleo06:16
*** slaweq has quit IRC06:16
*** kopecmartin has joined #tripleo06:23
*** paramite has quit IRC06:26
*** pcaruana has joined #tripleo06:31
*** jtomasek has joined #tripleo06:33
*** moshele has joined #tripleo06:33
*** dparkes has joined #tripleo06:37
*** paramite has joined #tripleo06:41
*** holser__ has joined #tripleo06:44
*** pblaho has joined #tripleo06:45
*** slaweq has joined #tripleo06:49
*** paramite has quit IRC06:49
*** jbadiapa has quit IRC06:49
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Add new nova-event-callback validation  https://review.openstack.org/51333306:51
*** quiquell is now known as quiquell|afk07:00
*** ffiore has joined #tripleo07:04
*** florianf has joined #tripleo07:07
chandankumarsshnaidm|bbl: Hello07:08
chandankumarsshnaidm|bbl: For collecting /var/log/container/tempest directory in ci, do I need to add some changes on collect-logs played to pull that directory in log server?07:09
*** tesseract has joined #tripleo07:10
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086007:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097207:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)07:10
*** paramite has joined #tripleo07:11
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient stable/queens: Add exclude-path to undercloud backups  https://review.openstack.org/56821007:13
*** jbadiapa has joined #tripleo07:14
*** masco has joined #tripleo07:15
*** elgxl has joined #tripleo07:15
*** jaganathan has joined #tripleo07:17
*** amoralej|off is now known as amoralej07:19
*** leanderthal has joined #tripleo07:20
openstackgerritNir Magnezi proposed openstack/tripleo-common master: Make lb-mgmt-subnet a class B subnet  https://review.openstack.org/56808907:26
*** jbadiapa has quit IRC07:27
*** cylopez has joined #tripleo07:28
*** bogdando has joined #tripleo07:30
*** jaganathan has quit IRC07:31
*** jaganathan has joined #tripleo07:31
*** cylopez has left #tripleo07:33
*** sshnaidm|bbl is now known as sshnaidm07:33
sshnaidmsl0w, thanks, I added this to tripleo-common: https://review.openstack.org/#/c/568118/07:33
sshnaidmchandankumar, you need to add only /var/log/tempest/ afaik07:34
*** fragatina has joined #tripleo07:35
*** tosky has joined #tripleo07:37
*** rcernin has quit IRC07:39
*** janki has joined #tripleo07:40
sshnaidmsl0w, ping me when you build pip package, or just +w on https://review.openstack.org/#/c/567550/07:41
*** pblaho has quit IRC07:41
*** saneax is now known as saneax-_-|AFK07:42
chandankumarsshnaidm: but in collect-log https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/defaults/main.yml#L1107:42
chandankumarsshnaidm: does it will not help?07:42
*** pblaho has joined #tripleo07:43
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Consume deployed t-h-t from the working output dir  https://review.openstack.org/56523307:45
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Use predictable name for generated hiera overrides  https://review.openstack.org/56633307:45
*** elgxl has quit IRC07:45
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Persist generated undercloud parameters t-h-t  https://review.openstack.org/56576407:46
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Store UC hosts/portmaps heat params in output_dir  https://review.openstack.org/56413507:46
*** quiquell|afk is now known as quiquell07:46
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart master: Run ovb (fs035) with a containerized undercloud  https://review.openstack.org/56691607:46
sshnaidmchandankumar, for ci you need to change here: https://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/config/collect-logs.yml#L18107:46
chandankumarsshnaidm: sure, thanks , i will update07:47
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Fix path and wire-in UC deploy role data file  https://review.openstack.org/56398807:47
*** elgxl has joined #tripleo07:49
*** ykarel is now known as ykarel|lunch07:49
*** saneax-_-|AFK is now known as saneax07:50
openstackgerritChandan Kumar proposed openstack-infra/tripleo-ci master: Collect tempest logs in CI from /var/log/tempest  https://review.openstack.org/56822007:53
*** jpena|off is now known as jpena07:53
*** fragatina has quit IRC07:54
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart master: Enable undercloud_enable_tempest to true  https://review.openstack.org/56633207:54
*** ssbarnea_ has joined #tripleo07:55
*** olap has joined #tripleo07:57
*** ubijtsa has joined #tripleo07:58
*** ubijtsa is now known as assassin07:58
*** assassin has quit IRC07:59
*** lucas-afk is now known as lucasagomes07:59
openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044508:00
*** akrivoka has joined #tripleo08:03
openstackgerritCarlos Camacho proposed openstack/tripleo-upgrade stable/queens: Allow running update/upgrade converge with --update-plan-only  https://review.openstack.org/56822208:04
*** jbadiapa has joined #tripleo08:05
openstackgerritShangXiao proposed openstack/tripleo-common master: Add release notes link to README  https://review.openstack.org/56822308:05
*** ssbarnea_ has quit IRC08:06
openstackgerritMerged openstack/diskimage-builder master: Trivial: update url to new url  https://review.openstack.org/56811708:06
*** athomas has joined #tripleo08:06
*** ukalifon has joined #tripleo08:08
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Help generate playbook commnands  https://review.openstack.org/56574008:09
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086008:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097208:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]08:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)08:10
*** ubijtsa has joined #tripleo08:11
*** ubijtsa is now known as assassin08:11
*** holser___ has joined #tripleo08:12
*** salmankhan has joined #tripleo08:12
*** holser__ has quit IRC08:13
*** numans_ has joined #tripleo08:14
*** skramaja_ has joined #tripleo08:14
*** jaganathan_ has joined #tripleo08:14
*** karthiks has quit IRC08:14
*** jaganathan has quit IRC08:14
*** janki has quit IRC08:14
*** skramaja has quit IRC08:14
*** janki has joined #tripleo08:14
openstackgerritShangXiao proposed openstack/tripleo-validations master: Add documentation links to README  https://review.openstack.org/56822608:15
*** suuuper has joined #tripleo08:16
*** mdnadeem has joined #tripleo08:16
*** numans has quit IRC08:17
*** dtantsur|afk is now known as dtantsur08:17
*** skramaja_ has quit IRC08:18
*** skramaja_ has joined #tripleo08:18
*** jaganathan_ has quit IRC08:18
*** numans_ has quit IRC08:18
dtantsursshnaidm: please don't silently assign bugs to people, it is pure luck that I noticed 177097208:19
sshnaidmdtantsur, I don't think I could find you here yesterday :)08:20
dtantsursshnaidm: then please drop a email :)08:21
sshnaidmdtantsur, ok08:21
dtantsursshnaidm: now a practical question: where do I find node's console log?08:21
*** skramaja has joined #tripleo08:21
sshnaidmdtantsur, seem like on te-broker only08:21
sshnaidmif it exists, it's there08:21
dtantsursshnaidm: fine, where is that? :)08:21
*** janki has quit IRC08:22
*** masco_ has joined #tripleo08:22
sshnaidmdtantsur, it has restricted access, I'll try to pull it from there08:22
*** janki has joined #tripleo08:22
dtantsursshnaidm: please do. actually, we have to have it published. without the console logs it may be plainly impossible to debug introspection.08:23
*** numans has joined #tripleo08:24
*** skramaja_ has quit IRC08:25
*** masco has quit IRC08:25
*** skramaja_ has joined #tripleo08:25
*** skramaja has quit IRC08:25
openstackgerritJose Luis Franco proposed openstack/tripleo-quickstart master: Remove ceph from major upgrade jobs.  https://review.openstack.org/56822908:26
dtantsursshnaidm: I'm stuck otherwise. everything I can check seems fine at first glance. Have we changed anything around containers configuration recently? selinux, firewall?08:27
dtantsurhjensas: ^^^ re https://bugs.launchpad.net/tripleo/+bug/177097208:27
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)08:27
sshnaidmdtantsur, maybe, I can't say exactly08:27
*** derekh has joined #tripleo08:29
*** karthiks has joined #tripleo08:30
*** jaganathan_ has joined #tripleo08:30
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade stable/pike: Add Neutron L3 agent failover check during upgrade  https://review.openstack.org/56823108:33
hjensasdtantsur: looking.08:33
sshnaidmderekh, hi, do you remember where can I find node's console logs on te-broker ^^ ?08:35
dtantsursshnaidm: also do you know when the failure started?08:35
sshnaidmdtantsur, form which day you mean?08:36
dtantsuryep, ideally time08:36
*** elgxl has quit IRC08:36
sshnaidmdtantsur, nope, we had containers problem that hided all bugs08:36
dtantsurOH08:36
*** elgxl has joined #tripleo08:37
sshnaidmdtantsur, well, it was ok before 7.5 was released in Thu08:37
mandre_offdtantsur: could it be caused by new centos release maybe? https://lists.centos.org/pipermail/centos-announce/2018-May/022829.html08:37
dtantsurhjensas: it feels like we can rule out the dnsmasq filter, since the DHCP looks correct08:37
dtantsurmandre_off: easily08:37
derekhsshnaidm: I got no scrollback, just logged in, which logs are you looking for the console for the te-broker itself?08:37
*** mandre_off is now known as mandre08:37
dtantsurderekh: we need console logs from OVB nodes during introspection08:37
sshnaidmderekh, for nodes console, in introspection.. I found bmc-console-logs, not sure it is..08:38
dtantsurthey fail to post back results for some reason, we cannot understand why08:38
hjensasdtantsur: ack, and the files transfer OK it looks like. We need consol logs.08:39
dtantsuryep08:39
pcaruanaping Tengu08:40
*** udesale_ has quit IRC08:41
Tengupong pcaruana :)08:41
derekhdtantsur: sshnaidm I don't think you can (or at least you couldn't back when I was doing this stuff), best you could do is try and watch the console in the cloud which its being inspected, if you have the creds08:41
derekhdtantsur: sshnaidm or we could try running devmode.sh --ovb08:41
derekhto see if that fails in the same way08:42
*** suuuper1 has joined #tripleo08:42
sshnaidmdtantsur, I pulled bmc-console-logs, not sure it helps though, please take a look08:42
*** ssbarnea_ has joined #tripleo08:42
*** udesale has joined #tripleo08:43
dtantsursshnaidm: no, it's not that. it's the logs of the fake BMC service08:43
sshnaidmdtantsur, I see08:44
sshnaidmdtantsur, so it's impossible atm to have logs from nodes08:44
sshnaidmderekh, dtantsur you can use any reproducer script from jobs instead of devmode also08:45
*** suuuper has quit IRC08:45
dtantsur.. which also won't store logs08:45
dtantsurthen I cannot help much, I'm afraid08:45
sshnaidmdtantsur, but you'll have full access to nodes then, it's your tenant08:45
dtantsurcan I please use it as a motivation for someone to urgently fix the CI to publish the badly needed logs? I've been complaining about that for years.08:46
*** moshele has quit IRC08:47
sshnaidmdtantsur, if it was so easy as it seems, we'd do it much earlier. We never had it before, it's not something new08:47
dtantsurthen should we switch the CI to devstack? ;)08:47
dtantsurseriously though, we need to pipe output of 'nova console-log <node>' to some file during introspection08:47
*** moshele has joined #tripleo08:48
dtantsurI can try hacking something together, but I don't know the tripleo CI any well08:48
sshnaidmdtantsur, unfortunately the problem is much more complex08:48
sshnaidmdtantsur, as I said, everything seems very trivial until you know how it works08:49
dtantsursorry, is it supposed to be a good excuse? there must be logs in the CI runs, you cannot ask derekh or me to run it locally every time something breaks08:50
sshnaidmdtantsur, no, it's not excuse, it's priorities of ci team. Although I don't think reproducing jobs is something bad to do, we put a lot of effort to make it easy..08:52
openstackgerritMerged openstack/diskimage-builder master: Fixes add-apt-keys in dpkg element  https://review.openstack.org/56559908:52
dtantsursshnaidm: it's still a lot of time wasted. and if basic debugability of the CI is not a priority for the CI team, why should this bug be a priority for me?08:53
derekhWe could try hacking in a console-log somewhere, but would that work through a virtual bmc? I'm going to try devmode anyways to see how far that gets us,08:56
sshnaidmdtantsur, I understood your attitude, unfortunately it wasn't highest priority till now and because of current CI implementation it's not basic and trivial to do it in OVB, you're welcome to talk about it in tripleo-ci meeting tomorrow right after tripleo meeting.08:56
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates stable/queens: FFU Set NetworkDeploymentActions CREATE,UPDATE for ffwd-upgrade prepare  https://review.openstack.org/56823608:56
dtantsurderekh: no, not with vbmc. but the CI uses OVB, which is normal Nova.08:56
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Separate scripts creation tasks for under/overcloud.  https://review.openstack.org/56629108:56
dtantsurI'm trying reproducer.sh as well, will see where it brings us08:57
dtantsursshnaidm: I08:57
dtantsurI'll file a High bug for that08:57
*** assassin has quit IRC08:57
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Separate scripts creation tasks for under/overcloud.  https://review.openstack.org/56629108:58
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Move SSL undercloud validation out of UC script create tasks.  https://review.openstack.org/56719008:58
derekhdtantsur: correct but the OVB bmc would have to support retreiving the logs, I'm just saying we can't do it now to debug this08:58
sshnaidmderekh, you can use one this script for example in rdo cloud tenant: https://logs.rdoproject.org/50/567550/7/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Zce161d1851514638b555adae340341dc/reproducer-quickstart.sh08:59
dtantsurderekh: it's unrelated to the bmc at all08:59
dtantsurwe can just talk to Nova to get logs, directly, without involving OVB BMC and Ironic08:59
dtantsursshnaidm: https://bugs.launchpad.net/tripleo/+bug/177108208:59
openstackLaunchpad bug 1771082 in tripleo "TripleO CI must collect and publish console logs from fake overcloud nodes during introspection and deployment" [High,Triaged]08:59
sshnaidmdtantsur, great, thanks08:59
derekhdtantsur: but the CI node doesn't know about the cloud, nor have credentials for it,09:00
*** yprokule has quit IRC09:00
derekhdtantsur:  as a once off for this bug somebody with the credentials could just run the command to see the log while its introspecting09:00
*** yprokule has joined #tripleo09:01
dtantsurderekh: it's not only for this bug, we have to do it to avoid these situations in the future..09:01
dtantsurseriously, if I had the logs, I could have maybe proposed a fix already...09:01
derekhdtantsur: yes, I agree all I was saying is that for this bug right now we can't "hack it in"09:01
dtantsuryep, trying locally thus09:02
openstackgerritMerged openstack/diskimage-builder master: Add pip cache cleanup to pip-and-virtualenv  https://review.openstack.org/56205509:03
*** ykarel|lunch is now known as ykarel09:07
*** dtantsur is now known as dtantsur|brb09:08
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086009:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097209:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]09:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)09:10
*** sshnaidm is now known as sshnaidm|rover09:12
dtantsur|brbaaaand the reproducer failed :( will try again later09:17
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: DO NOT MERGE: Test boot params in the config-download  https://review.openstack.org/56624109:20
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates stable/queens: FFU Set NetworkDeploymentActions CREATE,UPDATE for ffwd-upgrade prepare  https://review.openstack.org/56823609:21
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable default polling interval override  https://review.openstack.org/56824109:22
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates stable/pike: Switch remaining env files to containers defaults  https://review.openstack.org/56781709:23
*** aufi has joined #tripleo09:23
*** salmankhan has quit IRC09:25
openstackgerritCarlos Camacho proposed openstack/tripleo-common master: Add validation for empty file system backup on Undercloud  https://review.openstack.org/56824509:28
*** aufi has quit IRC09:28
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates stable/queens: Switch remaining env files to containers defaults  https://review.openstack.org/56781309:28
*** suuuper has joined #tripleo09:30
*** salmankhan has joined #tripleo09:31
*** suuuper1 has quit IRC09:31
chandankumarsshnaidm|rover: I am not sure where I am doing wrong, tempest container log is coming09:32
chandankumarsshnaidm|rover: https://review.openstack.org/#/q/topic:tempest_log+(status:open+OR+status:merged)09:32
openstackgerritMerged openstack/tripleo-common master: Add yum update to base  https://review.openstack.org/56763609:32
chandankumarsshnaidm|rover: please have a look09:32
sshnaidm|roverchandankumar, will look later, sorry09:33
chandankumarsshnaidm|rover: np :-)09:33
*** stendulker has joined #tripleo09:37
*** moshele has quit IRC09:37
*** moshele has joined #tripleo09:38
openstackgerritCarlos Goncalves proposed openstack/tripleo-common master: Check pub key file perms and default to pub key data  https://review.openstack.org/56720509:38
sshnaidm|roverdtantsur|brb, derekh, interesting fact, ykarel discovered that built images did introspection successfully: https://logs.rdoproject.org/89/568089/3/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/Zd6da518dd7d347818c6fab2b51156204/09:39
openstackgerritMerged openstack/tripleo-upgrade master: Convert nic templates to new format before starting fast forward  https://review.openstack.org/56742809:40
sshnaidm|roversshnaidm|rover, but those we download them from promotion server fail09:40
derekhsshnaidm|rover: ack09:40
derekhsshnaidm|rover: btw, does deploy work? in jobs that skip introspection?09:40
sshnaidm|roverderekh, yep09:42
derekhok09:43
sshnaidm|roverderekh, oops, sorry, didn't try it09:43
sshnaidm|roverderekh, confused with downstream09:43
derekhsshnaidm|rover: ok09:43
*** abishop has joined #tripleo09:45
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates stable/queens: Switch remaining env files to containers defaults  https://review.openstack.org/56781309:45
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: DNM: test without introspection  https://review.openstack.org/56824609:45
sshnaidm|roverderekh, let's see ^^09:46
*** masco_ has quit IRC09:47
*** karthiks has quit IRC09:51
*** karthiks has joined #tripleo09:51
openstackgerritCarlos Camacho proposed openstack/tripleo-docs master: Change, 'openstack install undercloud' by 'openstack undercloud install'  https://review.openstack.org/56824709:55
*** jaosorior has joined #tripleo09:56
*** panda|off is now known as panda09:56
*** masco_ has joined #tripleo09:59
*** chem has joined #tripleo09:59
*** chem has quit IRC10:04
openstackgerritCarlos Camacho proposed openstack/tripleo-common stable/queens: Add validation for empty file system backup on Undercloud  https://review.openstack.org/56825010:04
openstackgerritCarlos Camacho proposed openstack/tripleo-common stable/queens: Add validation for empty file system backup on Undercloud  https://review.openstack.org/56825010:05
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086010:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097210:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109010:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]10:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)10:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)10:10
*** psahoo has quit IRC10:11
*** bkopilov has quit IRC10:13
*** chem has joined #tripleo10:15
openstackgerritRaoul Scarazzini proposed openstack/tripleo-ha-utils master: Add pre checks before installing instance HA  https://review.openstack.org/56789210:28
*** saneax is now known as saneax-_-|AFK10:35
*** dtantsur|brb is now known as dtantsur10:38
dtantsursshnaidm|rover: curious indeed... broken images?10:39
openstackgerritRaoul Scarazzini proposed openstack/tripleo-ha-utils master: Add pre checks before installing instance HA  https://review.openstack.org/56789210:39
sshnaidm|roverdtantsur, maybe kernel.. idk10:40
dtantsursshnaidm|rover: I cannot run reproducer, it fails on installing epel-release...10:40
dtantsur(which is one of the reasons we cannot rely on devs running them all the time)10:40
sshnaidm|roverdtantsur, I didn't hear about such problem before, can you paste a log please?10:40
*** abishop has quit IRC10:42
sshnaidm|roverDIB cores, please review/merge: https://review.openstack.org/#/c/568180/10:42
sshnaidm|roverderekh, are you from those? :) ^^10:42
derekhsshnaidm|rover: afraid not10:44
*** apetrich has quit IRC10:46
derekhsshnaidm|rover: dtantsur http://goodsquishy.com/downloads/ipa_1.png10:46
derekhsshnaidm|rover: dtantsur  console isn't getting any further then that10:47
*** saneax-_-|AFK is now known as saneax10:48
*** atoth has quit IRC10:48
*** dprince has joined #tripleo10:49
*** apetrich has joined #tripleo10:49
ykareldtantsur, sshnaidm|rover i too faced the issue in reproducer, i workaround that updating nodepool-setup role by doing s/$contentdir/centos/ /etc/yum.repos.d/Centos-QEMU-EV.repo10:50
dtantsursigh..10:50
dtantsurderekh: hmm, selinux then?10:51
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: build image in every OVB job  https://review.openstack.org/56825810:51
sshnaidm|roverderekh, dtantsur checking if it helps ^^10:51
dtantsurcan we rebuild the promoted images maybe?10:51
dtantsurcool10:51
derekhdtantsur: maybe, I'm going to openthic ramdisk and see if its enforcing10:51
derekhdtantsur: sshnaidm|rover also I now got a partial tracback http://goodsquishy.com/downloads/ipa_2.png10:52
sshnaidm|roverdtantsur, promotions are blocked by other 7.5 issues, so I'm not sure it's possible today..10:52
dtantsurderekh: seems like networks has not started at all..10:52
dtantsur* networking10:52
dtantsurderekh: thanks for investigation, mind updating https://bugs.launchpad.net/tripleo/+bug/1770972 ?10:52
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Dmitry Tantsur (divius)10:52
derekhdtantsur: but, but that could be down to selinux, will see what else I can find10:52
dtantsurfeel free to assign to yourself if you're going to look more10:53
dtantsurI still cannot reproduce it...10:53
derekhdtantsur: ack, will do10:53
*** numans_ has joined #tripleo10:53
derekhdtantsur: SELINUX=permissive10:55
*** dprince has quit IRC10:55
*** dprince has joined #tripleo10:56
openstackgerritRajesh Tailor proposed openstack/tripleo-heat-templates master: Allow configuration of NFS backend for Nova  https://review.openstack.org/56417910:56
*** janki has quit IRC10:57
*** aufi has joined #tripleo11:00
openstackgerritCarlos Camacho proposed openstack/tripleo-common master: Add validation for empty file system backup on Undercloud  https://review.openstack.org/56824511:00
*** shreshtha has quit IRC11:01
jaosoriord0ugal: could you check this out https://review.openstack.org/#/c/554926/ ?11:01
openstackgerritCarlos Camacho proposed openstack/tripleo-common stable/queens: Add validation for empty file system backup on Undercloud  https://review.openstack.org/56825011:01
*** stendulker_ has joined #tripleo11:02
*** quiquell is now known as quiquell|lunch11:03
Tengujaosorior: hello sir :)11:03
*** mdnadeem has quit IRC11:04
*** links has quit IRC11:05
*** stendulker has quit IRC11:05
*** numans has quit IRC11:06
*** stendulker_ has quit IRC11:06
*** numans_ has quit IRC11:07
jaosoriorTengu: hey! how's it going?11:08
jaosoriorTengu: first day?11:08
*** numans has joined #tripleo11:09
Tengujaosorior: running after my credentials and the doc, yep :)11:09
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086011:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097211:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109011:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]11:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)11:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)11:10
jaosoriorTengu: welcome!11:10
Tenguthank you :).11:10
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Enable tempest.log in /var/log/tempest for container  https://review.openstack.org/56652611:11
*** apetrich has quit IRC11:11
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart master: Enable undercloud_enable_tempest to true  https://review.openstack.org/56633211:11
*** dprince has quit IRC11:11
ianwif my comments are about right on "rpm-distro: set the contentdir yum var" (lohttps://review.openstack.org/#/c/568180/), and it's holding things up, i'm ok to quick merge it11:14
*** mdnadeem has joined #tripleo11:16
*** apetrich has joined #tripleo11:16
*** links has joined #tripleo11:17
*** abishop has joined #tripleo11:18
*** ykarel is now known as ykarel|afk11:21
sshnaidm|roverykarel|afk, fyi, https://bugs.centos.org/view.php?id=1478611:21
openstackgerritChuck Short proposed openstack/tripleo-quickstart master: Fix roles/environment/vars file  https://review.openstack.org/56826411:27
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add health check for OVN containers  https://review.openstack.org/56826511:29
*** jpena is now known as jpena|lunch11:31
*** bkopilov has joined #tripleo11:34
sshnaidm|roversl0w, ping me please when you build the new pip package for container-check11:34
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add health check for OVN containers  https://review.openstack.org/56826511:35
*** pkovar has joined #tripleo11:36
*** janki has joined #tripleo11:37
derekhsshnaidm|rover: dtantsur the IPA image appears to have been currupted (or not package correctly)11:38
dtantsurwow11:38
dtantsurwhat's wrong?11:38
derekhsshnaidm|rover: dtantsur  I opened it up, set the root password so I could ssh to it, the new image works, nothing else has changed11:38
openstackgerritMartin Mágr proposed openstack/tripleo-common master: [WIP] Activate remaining set of healthchecks  https://review.openstack.org/56387411:39
derekhsshnaidm|rover: dtantsur I've had 4 or 5 successfull inspections with this image, switched back to the origional and FAILED11:39
dtantsurhmmmm.. DIB bug?11:40
derekhOr I could be using a different command to package the ramdisk "sudo find . | sudo cpio -H newc -o | gzip -9 > ../ramdisk"11:42
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable health check for OVN containers  https://review.openstack.org/56826711:43
*** wolverineav has joined #tripleo11:44
*** apetrich has quit IRC11:44
*** atoth has joined #tripleo11:44
*** apetrich has joined #tripleo11:44
*** amoralej is now known as amoralej|lunch11:45
sshnaidm|roverderekh, promotion jobs that build images just started, if everything will be ok we'll have images for test in 2h maybe, even if it's not promoted.11:45
beaglesjaosorior: quick sanity check as I start my monday - is CI open for rechecking yet?11:46
derekhsshnaidm|rover: ack11:46
* beagles picks jaosorior at random ;)11:46
sshnaidm|roverderekh, can you please update the lp bug?11:47
jaosoriorbeagles: so far so good11:47
beaglesjaosorior: ack thanks11:47
derekhsshnaidm|rover: I have added what I said above, I don't have much more that would be useful11:48
sshnaidm|roverderekh, also if you can build this image and share it somewhere would be nice.. we don't have a direct access to images server, but maybe can hack and upload it instead of broken11:48
*** tcw has joined #tripleo11:48
*** dprince has joined #tripleo11:49
sshnaidm|roverderekh, this is for worst case we don't have neither images nor promotion11:49
sshnaidm|roverderekh, and worst case is usual in CI :D11:49
*** ansmith has quit IRC11:51
*** links has quit IRC11:52
*** links has joined #tripleo11:52
*** rh-jelabarre has joined #tripleo11:53
*** elgxl has quit IRC11:53
*** ykarel|afk is now known as ykarel11:57
*** jcoufal has joined #tripleo11:57
ykarelsshnaidm|rover, ack, sbonazzo shared this bug with me11:58
openstackgerritSagi Shnaidman proposed openstack/tripleo-common stable/queens: Add yum update to base  https://review.openstack.org/56827011:59
*** weshay is now known as weshay_interview11:59
openstackgerritMerged openstack/tripleo-upgrade master: Convert CLI options to variables for updates/upgrades/ffwd.  https://review.openstack.org/56724612:00
sshnaidm|rovermwhahaha, EmilienM please look at backport: https://review.openstack.org/#/c/568270/12:00
*** dprince has quit IRC12:00
*** raildo has joined #tripleo12:02
openstackgerritMartin André proposed openstack/tripleo-common master: Run yum clean to reduce size of docker image layer  https://review.openstack.org/56827212:03
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient stable/pike: Add --stack to update, upgrade and ffwd-upgrade 'run' CLI.  https://review.openstack.org/56827312:03
hjensasbeekneemech: https://bugs.launchpad.net/tripleo/+bug/1771082 wdyt, plausible?12:05
openstackLaunchpad bug 1771082 in tripleo "TripleO CI must collect and publish console logs from fake overcloud nodes during introspection and deployment" [High,Triaged]12:05
openstackgerritBogdan Dobrelya proposed openstack-infra/tripleo-ci master: Optimize CI triggers for UC containers upgrades  https://review.openstack.org/56827512:10
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086012:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097212:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109012:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]12:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)12:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)12:10
*** paramite has quit IRC12:10
*** gyankum has joined #tripleo12:11
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade stable/queens: Convert CLI options to variables for updates/upgrades/ffwd.  https://review.openstack.org/56827712:14
*** numans_ has joined #tripleo12:19
*** karts has joined #tripleo12:19
*** karthiks has quit IRC12:19
*** jaganathan_ has quit IRC12:19
*** skramaja_ has quit IRC12:19
*** jaganathan_ has joined #tripleo12:19
*** masco has joined #tripleo12:20
openstackgerritBogdan Dobrelya proposed openstack-infra/tripleo-ci master: Optimize CI triggers for instack-undercloud upgrades  https://review.openstack.org/56827812:21
*** dougbtv has joined #tripleo12:22
*** masco_ has quit IRC12:23
*** skramaja_ has joined #tripleo12:23
*** numans has quit IRC12:23
*** karts has quit IRC12:24
*** numans_ has quit IRC12:24
*** karts has joined #tripleo12:24
*** jpena|lunch is now known as jpena12:25
*** trown|outtypewww is now known as trown12:25
Tenguheya! small question with heat: is there a way to use a parameter value in another parameter default value?12:29
*** dprince has joined #tripleo12:29
*** radek__ has quit IRC12:29
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add fast_forward_upgrade_tasks in ironic services.  https://review.openstack.org/56784912:29
*** numans has joined #tripleo12:30
*** radeks has joined #tripleo12:30
*** karts is now known as karthiks12:31
*** eck`gone is now known as eck`12:31
jaosoriorTengu: not AFAIK12:31
Tenguhumpf.12:31
*** skramaja_ is now known as skramaja12:32
jaosoriorHey folks, I would really use reviews here https://review.openstack.org/#/c/554926/12:32
*** haleyb has joined #tripleo12:32
*** rlandy has joined #tripleo12:33
*** fultonj has joined #tripleo12:34
Tengujaosorior: on it :). Although I'm only +112:34
*** fultonj has quit IRC12:35
openstackgerritMerged openstack/diskimage-builder master: rpm-distro: set the contentdir yum var  https://review.openstack.org/56818012:36
*** fultonj has joined #tripleo12:37
*** amoralej|lunch is now known as amoralej12:39
*** elgxl has joined #tripleo12:39
*** moshele has quit IRC12:40
*** gyankum has quit IRC12:42
*** bfournie has joined #tripleo12:43
*** pchavva has joined #tripleo12:43
*** drbacchus has quit IRC12:50
*** rbowen has joined #tripleo12:50
openstackgerritRajesh Tailor proposed openstack/tripleo-heat-templates master: Allow configuration of NFS backend for Nova  https://review.openstack.org/56417912:51
*** vpickard_ is now known as vpickard12:51
openstackgerritRajesh Tailor proposed openstack/tripleo-heat-templates master: Allow configuration of NFS backend for Nova  https://review.openstack.org/56417912:53
jaosoriorjistr: hey dude, could you check this out https://review.openstack.org/#/c/554926/ ?12:56
*** wolverineav has quit IRC12:57
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: [WIP] Implement --output-file to write the bash script  https://review.openstack.org/56828512:57
*** dmsimard is now known as dmsimard|off12:58
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Run undeploy_plan workflow to delete deployment  https://review.openstack.org/56636612:58
*** jcoufal_ has joined #tripleo13:01
*** weshay_interview is now known as weshay13:01
sshnaidm|rovermwhahaha, can't we backport "yum update" to pike? https://review.openstack.org/#/c/568270/13:03
*** agurenko has quit IRC13:03
*** jcoufal has quit IRC13:03
*** radek__ has joined #tripleo13:04
*** agurenko has joined #tripleo13:05
*** toure|gone is now known as toure13:05
*** ansmith has joined #tripleo13:06
zolifolks, is that "offical" mailing list: Welcome to the Redhat-triple0@mail.shadowserver.org mailing list!13:07
zoli?13:07
* zoli is not sure whether it is not some phishing attempt13:07
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add releases script pytest tests to tox.ini  https://review.openstack.org/56764913:09
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: Clear up Upgrade params on converge  https://review.openstack.org/56726513:09
*** zaneb has joined #tripleo13:09
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086013:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097213:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109013:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]13:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)13:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)13:10
jaosoriorzoli: that looks quite fishy13:10
*** tzumainn has joined #tripleo13:11
*** pdeore has quit IRC13:12
TenguPTAL https://review.openstack.org/#/c/566276/  ? :) would be lovely to get it backported in Queens.13:13
derekhsshnaidm|rover: http://38.145.33.212/ironic-python-agent.tar13:14
*** Goneri has joined #tripleo13:15
Tenguthank you jaosorior :)13:15
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add linter jobs  https://review.openstack.org/56828713:16
*** cshastri has quit IRC13:17
*** trozet has joined #tripleo13:20
jistrjaosorior: +2. Btw very nice code @ https://review.openstack.org/#/c/554926/31/tripleo_common/actions/deployment.py13:20
jaosoriorjistr: thanks!13:21
Tengujistr: ah, good catch for the way to enable the +e flag :).13:22
Tenguwell. -e, sorry.13:22
Tenguanyway. weird, but at least this would enable a bullet-proof script.13:22
jistrwill re+2 if we decide to go for -e13:22
Tengu:)13:22
sshnaidm|roverderekh, thanks!13:23
jistrjaosorior, Tengu: yea. i think the RC of the script would be a RC of the if (last statement), which would be a RC of the last command there (calls to getcert). So it should already be somewhat sane w/r/t return codes, but -e is best.13:24
Tengu:)13:25
jaosoriorTengu, jistr: Well, I'm just doing a new deployment. Something I could do is test it out with -e and the additions that jistr suggested, and add a patch on top of this one.13:25
jistrjaosorior: sounds good13:26
jaosoriorso, let this patch as-is, and then have a follow-up that adds the -e13:26
jistrack13:26
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient stable/pike: Add --stack to update, upgrade and ffwd-upgrade 'run' CLI.  https://review.openstack.org/56827313:30
*** mcornea has joined #tripleo13:30
*** superdan is now known as dansmith13:33
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Add deployment status tracking infrastructure  https://review.openstack.org/55902113:33
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Enable config-download deployment tracking  https://review.openstack.org/55902213:33
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Chain ansible messages in deployment progress  https://review.openstack.org/56473313:33
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Run undeploy_plan workflow to delete deployment  https://review.openstack.org/56636613:33
*** masco has quit IRC13:36
*** cshastri has joined #tripleo13:36
*** alee has joined #tripleo13:39
weshaymwhahaha, morning13:40
*** karthiks has quit IRC13:40
*** shreshtha has joined #tripleo13:42
*** lblanchard has joined #tripleo13:43
EmilienMhello13:45
* mwhahaha hides from weshay 13:47
*** zoli is now known as zoli|afk13:47
mwhahahasshnaidm|rover: yes we can backport to pike13:47
cgoncalvesbogdando, hi. could you please revisit https://review.openstack.org/#/c/567205/ and https://review.openstack.org/#/c/568022/ ? thanks!13:48
*** Guest1701 has quit IRC13:48
cgoncalvesmwhahaha, hi. also if you have some cycles to review ^. thank you!13:49
*** pkovar has quit IRC13:50
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add linter jobs  https://review.openstack.org/56828713:50
toskyuhm, do you know of any issue with the "Send tempest results by mail" step? I've noticed a failure connecting to tempest-sendmail.tripleo.org13:51
openstackgerritSagi Shnaidman proposed openstack/tripleo-common stable/pike: Add yum update to base  https://review.openstack.org/56829213:51
*** pblaho has quit IRC13:51
sshnaidm|rovermwhahaha, I hope did it right ^^13:52
*** pkovar has joined #tripleo13:53
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade stable/queens: Convert nic templates to new format before starting fast forward  https://review.openstack.org/56829313:53
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart-extras master: Add opendaylight to collect-logs  https://review.openstack.org/49404313:53
mwhahahasshnaidm|rover: yea undercloud on stable/pike is broken tho. there's another pending change that needs to be on top of. i'll fix13:54
sshnaidm|rovermwhahaha, ack13:54
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/pike: Add yum update to base  https://review.openstack.org/56829213:54
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Enable tempest.log in /var/log/tempest for container  https://review.openstack.org/56652613:54
*** jgwentworth is now known as melwitt13:57
*** dsavineau has quit IRC13:58
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add linter jobs  https://review.openstack.org/56828713:59
*** dxiri has joined #tripleo13:59
*** beekneemech is now known as bnemec13:59
weshaymwhahaha, we myoung|ruck sshnaidm|rover have some concerns w/ the yum update patch to tripleo-common14:01
mwhahahaweshay: ok14:01
weshaywe should get updated base-os packages in containers via https://review.rdoproject.org/r/#/c/13761/1/ci-scripts/container-push/container-push.yml14:02
weshaymwhahaha, if we add yum udpate note each container would run this individually and we may get different package versions in a set of containers14:02
mwhahahaweshay: no14:02
weshayam I making sense w/ that14:03
weshay?14:03
mwhahahathe yum update doesn't get added to all the containers14:03
mwhahahait only gets added to the base container14:03
weshayk /me looks14:03
*** links has quit IRC14:03
openstackgerritMerged openstack/tripleo-docs master: Change, 'openstack install undercloud' by 'openstack undercloud install'  https://review.openstack.org/56824714:04
weshayblock_base_footer14:04
weshayI guess that is why14:04
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: NFV: Support for config-download to deploy node with kernel args  https://review.openstack.org/56076714:05
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: DO NOT MERGE: Test boot params in the config-download  https://review.openstack.org/56624114:05
mwhahahaweshay: yes14:06
weshayroger that14:06
*** dsavineau has joined #tripleo14:06
openstackgerritRaoul Scarazzini proposed openstack/tripleo-ha-utils master: Make instance HA uninstallation idempotent  https://review.openstack.org/56829614:06
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart master: Updates OpenDaylight feature set 31 to use tempest  https://review.openstack.org/50087214:07
openstackgerritRaoul Scarazzini proposed openstack/tripleo-ha-utils master: Make instance HA uninstallation idempotent  https://review.openstack.org/56829614:08
openstackgerritMichael Chapman proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment  https://review.openstack.org/48690514:08
*** dsavineau1 has joined #tripleo14:08
*** lblanchard1 has joined #tripleo14:08
jistrweshay, trown, panda: i could draft a suggestion on handling the zuul repo insertion points (maybe an etherpad so that we can co-edit, with a link sent to the list for whom-it-may-concern). WDYT?  I imagine it will not be implemented soon but it might be just useful to have a mental picture of where we'd like to go with that, when we're making decisions about other things.14:09
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086014:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097214:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109014:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]14:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)14:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)14:10
jistrweshay, trown, panda: also i just selfishly want to dump the info somewhere so that i can let it go from my active memory :D14:10
trownjistr: ya that sounds great14:10
trownjistr: at the very least, it will help inform the epic for a future CI sprint14:11
*** dsavineau has quit IRC14:11
jistrcool :) yea it's totally fine if we prefer a different solution in the end14:12
*** lblanchard has quit IRC14:12
beaglesmcornea: question regarding ffu - when during a FFU does a reboot of controllers happen? Is it part of the upgrade process or is it something that is to be initiated by the operator14:13
*** pkovar has quit IRC14:13
*** udesale has quit IRC14:14
mcorneabeagles: the mandatory one is after the pre-ffu minor update and the reboots are done by the operator14:15
*** liverpooler has joined #tripleo14:16
mcorneabeagles: but the bug we were checking on Friday involves rebooting the nodes post upgrade, not related strictly to the upgrade procedure but to what happens after upgrade14:16
beaglesmcornea: okay so in terms of control/dataplane failovers, one might say that's pretty much in the hands of the operator?14:16
beaglesmcornea: yup understood.. we were discussing this bug in a meeting today and I was asked about when nodes were rebooted14:17
beaglesmcornea: I couldn't think of when that would happen in my understanding of how this works14:17
*** rpioso|afk is now known as rpioso14:18
beaglesmcornea: that's because it "sort of doesn't" :)14:18
beaglesnot automatically anyways14:18
beagleswhich makes sense - even if we wanted to automate that it would make more sense to have it as a separate mechanism14:19
mcorneabeagles: so yeah, the reboots are done manually by the operator after the minor update so in this case the failovers are manually controlled but when it comes to the fast forward procedure there shouldn't be any more reboots required so the assumption is that during ffu control plane is unavailable but dataplane remains available14:19
beaglesmcornea: right14:19
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Use i18n for undercloud installer and utils messages  https://review.openstack.org/56829814:20
*** pkovar has joined #tripleo14:21
*** skramaja has quit IRC14:22
*** gbarros has joined #tripleo14:24
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade master: [WIP] Fix validation for A/P services  https://review.openstack.org/56830114:26
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade master: Pass overcloud stack name to upgrade run.  https://review.openstack.org/56830214:30
toskyarxcruz: are you still in charge for tempestmail, or should I nag someone else? :)14:32
arxcruztosky: yup, it's me14:32
toskyarxcruz: I noticed a failure; the tempest tests failed, but then the job failed trying to contact the email server, let me find the log14:33
toskyarxcruz: this one: http://logs.openstack.org/54/509554/12/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/70015bd/job-output.txt.gz#_2018-05-14_00_05_44_97946614:33
*** gyankum has joined #tripleo14:34
*** gyankum has quit IRC14:34
arxcruztosky: server is up again, should not have this problem :)14:37
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: DNM. Use only switch-from-non-containerized-to-containerized-ceph-daemons  https://review.openstack.org/56830514:37
toskyarxcruz: can I try a recheck?14:37
*** wolverineav has joined #tripleo14:37
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Store UC hosts/portmaps heat params in output_dir  https://review.openstack.org/56413514:38
arxcruztosky: sure, but give some time, for some reason, the server is not responding properly, probably rdocloud is overloaded14:38
sshnaidm|roverderekh, can you please try this image? https://images.rdoproject.org/master/rdo_trunk/tripleo-ci-testing/  it will be promoted shortly, interesting if it will solve issue14:38
toskyarxcruz: the email server? So it's not up again :P14:38
derekhsshnaidm|rover: will do14:38
arxcruztosky: didn't return yet from the start command14:38
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: DNM. Use only switch-from-non-containerized-to-containerized-ceph-daemons  https://review.openstack.org/56830514:38
*** wolverineav has quit IRC14:39
*** wolverineav has joined #tripleo14:39
jaosoriormwhahaha: if you have some time could you check this out https://review.openstack.org/#/c/554926/ ?14:39
*** rajinir has joined #tripleo14:40
mwhahahayou and your tls14:40
Tengu:]14:40
arxcruztosky: it's up :)14:41
toskyarxcruz: thanks, let me recheck14:42
pandajistr: yes, thanks. Let me know where you put it, I will create an EPIC card based on those informations14:44
derekhsshnaidm|rover: introspection works with that ramdisk14:44
sshnaidm|roverderekh, \o/14:45
sshnaidm|roverweshay, myoung|ruck ^^14:45
weshaynice14:45
openstackgerritMichael Chapman proposed openstack/tripleo-heat-templates master: Add OPNFV scenario environment  https://review.openstack.org/48690514:45
weshayderekh thanks man14:45
sshnaidm|roverderekh, dtantsur than I'll wait for promotion and will close the bug14:45
*** noslzzp has joined #tripleo14:45
dtantsur\o/14:46
* dtantsur likes when problems solve themselves14:46
derekhweshay: no prob, fingers crossed I had the correct problem reproduced ;-)14:46
*** gbarros has quit IRC14:47
jistrpanda: thanks :)14:47
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Persist generated undercloud parameters t-h-t  https://review.openstack.org/56576414:48
bogdandofolks, PTAL https://review.openstack.org/#/q/topic:undercloud_ci_pipelines+(status:open+OR+status:merged)14:48
*** stendulker has joined #tripleo14:53
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade master: [WIP] Fix validation for A/P services  https://review.openstack.org/56830114:53
EmilienMbogdando: tbh I don't know about these changes14:53
EmilienMwe're not saving any time14:53
*** zoli|afk is now known as zoli14:54
EmilienMwe might save resources though but not time14:54
*** quiquell|lunch is now known as quiquell|off14:54
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add releases script pytest tests to tox.ini  https://review.openstack.org/56764914:55
bogdandoEmilienM: that reduces zuul queue and we can not be so sure it wouldn't save some time in the end, as jobs will be done faster and w/o wainting for the upgrade jobs doomed14:55
bogdandowill be started faster*14:55
bogdandogiven that upgrades take the most time...14:55
*** zoli is now known as zoli|wfh14:56
bogdandowell, OVB is not upstream any more so, it is14:56
*** zoli|wfh is now known as zoli14:56
bogdandoso that trades some time from a developer to wait for his pipeline for all waiting their patches to pass faster :)14:56
*** ratailor has quit IRC14:57
bogdandoEmilienM: so I really think it saves average wait-for-ci time for all14:58
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable default polling interval override  https://review.openstack.org/56824114:58
*** dsavineau1 has left #tripleo15:00
*** paramite has joined #tripleo15:01
*** dsavineau has joined #tripleo15:02
bogdandoEmilienM: replied in the patch as well15:02
*** pcaruana has quit IRC15:02
trozetgfidente: hey, can you help me with something?15:03
*** ykarel is now known as ykarel|away15:06
*** fultonj has quit IRC15:07
*** olap has quit IRC15:09
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177086015:10
openstackLaunchpad bug 1770860 in tripleo "tracker-bug: network lag in tripleo-infra tenant prevents container promotions" [Critical,Triaged]15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097215:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109015:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)15:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)15:10
gfidentetrown what is it?15:11
bogdandoEmilienM: I'm also thinking if we have a CI job might predict a 100% failure for http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/zuul.d/layout.yaml#n9 ?..15:12
*** ykarel|away is now known as ykarel15:15
*** derekh has quit IRC15:17
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable default polling interval override  https://review.openstack.org/56824115:18
*** gbarros has joined #tripleo15:19
*** derekh has joined #tripleo15:20
*** derekh has quit IRC15:20
aleeweshay, trown - where are the default tags that are run in the CI jobs set?15:20
*** derekh has joined #tripleo15:21
aleeweshay, trown I created a new tag overcloud-redeploy and overcloud-generate-passwords - and tried to run them in a ci job, but they were not executed.  I'm guessing I need to define them somewhere to run15:22
openstackgerritMerged openstack/tripleo-upgrade stable/queens: Include nova check script during update/upgrade/ffwd  https://review.openstack.org/56786615:23
openstackgerritMichael Chapman proposed openstack/tripleo-quickstart master: Updates OpenDaylight feature set 31 to use tempest  https://review.openstack.org/50087215:24
aleemcornea, ^^15:25
*** psachin has quit IRC15:26
*** tiswanso has joined #tripleo15:27
*** yprokule has quit IRC15:27
*** ukalifon has quit IRC15:27
sshnaidm|roversl0w, ping15:35
jistrweshay, trown, panda: here it is https://etherpad.openstack.org/p/tripleo-ci-zuul-repo-insertion15:36
weshaythanks jistr :)15:37
*** karthiks has joined #tripleo15:39
*** jpena is now known as jpena|brb15:43
*** thrash is now known as thrash|biab15:45
aleeweshay, trown , mcornea ?15:50
pandajistr: https://trello.com/c/vQZW8lEE/754-add-the-ability-to-inject-zuul-changes-and-corresponding-package-repos-in-various-stages-of-the-upgrade-workflow15:50
*** marios has quit IRC15:51
weshayalee, CI is on fire.. we're a little swamped atm15:51
weshaysorry for the delay in response15:51
aleeweshay, gotcha15:52
*** ayoung has joined #tripleo15:53
jistrpanda: perfect, thanks :) BTW in case the etherpad gets lost, i also have a local version in txt file. (I thought about adding the whole context into the e-mail as a backup solution, but figured it would just be annoying :D )15:53
pandanever seen an etherpad lost, I usually have the opposite problem15:53
*** ykarel has quit IRC15:54
weshaythanks panda15:58
*** radek__ has quit IRC15:58
*** stendulker has quit IRC15:59
*** dparkes has quit IRC15:59
jistrpanda: haha yes true :D i just tend to be overly cautions with all kinds of information backup16:00
*** wolverineav has quit IRC16:00
*** ffiore has quit IRC16:01
*** trown is now known as trown|lunch16:01
*** leanderthal has quit IRC16:02
*** ccamacho has quit IRC16:05
*** rmascena has joined #tripleo16:06
*** janki has quit IRC16:08
*** raildo has quit IRC16:08
openstackgerritBogdan Dobrelya proposed openstack-infra/tripleo-ci master: Optimize CI triggers for some of OC deploy jobs  https://review.openstack.org/56832616:09
weshaymwhahaha, EmilienM 3rd party ovb jobs for master look like we'll start passing again, at least fs00116:09
EmilienMgood16:10
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097216:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109016:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)16:10
EmilienMI saw fs035 also this morning16:10
*** lucasagomes is now known as lucas-afk16:10
*** kopecmartin has quit IRC16:11
*** numans has quit IRC16:13
bogdandoJFYI http://lists.openstack.org/pipermail/openstack-dev/2018-May/130459.html16:15
weshayfs035 looks good too16:16
bogdandoEmilienM: ^^16:18
EmilienMbogdando: ack16:18
*** numans has joined #tripleo16:18
bogdandoweshay: oh, nice! rechecking my https://review.openstack.org/#/c/566916/ then )16:19
*** pcaruana has joined #tripleo16:20
*** jpena|brb is now known as jpena16:24
*** panda is now known as panda|bbl16:27
*** rbrady has joined #tripleo16:27
*** rbrady is now known as Guest2305616:28
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable default polling interval override  https://review.openstack.org/56824116:30
*** salmankhan has quit IRC16:31
*** wolverineav has joined #tripleo16:31
*** bogdando has quit IRC16:36
*** holser___ has quit IRC16:38
*** aufi has quit IRC16:39
*** wolverineav has quit IRC16:40
EmilienMI don't know if someone noticed but the deploy_steps_tasks are now ignored and I suspect the last upgrade of Ansible related to it16:41
EmilienMwe now have Ansible 2.5.2 and I've reproduce the bug locally, but the bug goes away when downgrading to ansible-2.4.4.0-116:42
openstackgerritAde Lee proposed openstack/tripleo-upgrade master: Add redeploy role  https://review.openstack.org/56730016:43
*** dprince has quit IRC16:44
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add dry run option to toci_quickstart  https://review.openstack.org/56706016:44
*** shreshtha_ has joined #tripleo16:48
*** shreshtha has quit IRC16:50
*** wolverineav has joined #tripleo16:50
*** gfidente has quit IRC16:52
openstackgerritJames Slagle proposed openstack/python-tripleoclient master: Convert enable-ssh-admin.sh to python  https://review.openstack.org/56612916:54
*** toure is now known as toure|biab16:55
*** itlinux has joined #tripleo16:57
*** pchavva has quit IRC16:58
*** derekh has quit IRC17:00
*** trown|lunch is now known as trown17:00
*** suuuper has quit IRC17:01
*** pkovar has quit IRC17:02
*** mugsie has quit IRC17:06
*** mugsie has joined #tripleo17:06
*** mugsie has quit IRC17:06
*** mugsie has joined #tripleo17:06
*** dprince has joined #tripleo17:06
*** mdnadeem has quit IRC17:07
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: Add acl to paths that are shared among related neutron processes  https://review.openstack.org/56765517:07
*** Guest23056 is now known as rbrady_17:07
*** rbrady_ has quit IRC17:07
*** rbrady_ has joined #tripleo17:07
*** mugsie has quit IRC17:08
*** radeks has quit IRC17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097217:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109017:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)17:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)17:10
*** jfrancoa has quit IRC17:10
*** tesseract has quit IRC17:11
*** mugsie has joined #tripleo17:12
*** mugsie has joined #tripleo17:12
*** jfrancoa has joined #tripleo17:13
*** dtantsur is now known as dtantsur|afk17:14
*** jpena is now known as jpena|off17:19
*** etingof is now known as etingof|afk17:21
*** rbrady__ has joined #tripleo17:24
*** rbrady_ has quit IRC17:24
*** rbrady__ has quit IRC17:24
*** agopi has quit IRC17:25
*** agopi has joined #tripleo17:25
*** SamYaple has joined #tripleo17:25
*** rbrady has joined #tripleo17:27
*** rbrady is now known as Guest1078417:27
*** Guest10784 has quit IRC17:28
*** dciabrin has quit IRC17:28
*** dciabrin has joined #tripleo17:29
*** ykarel has joined #tripleo17:30
*** tosky has quit IRC17:30
sl0wsshnaidm|rover: https://pypi.org/project/container-check/#history17:32
*** radeks has joined #tripleo17:33
*** tosky has joined #tripleo17:34
mwhahahaso looks like the CI might be red again?17:36
mwhahahapacemaker failures17:36
mwhahahahttp://logs.openstack.org/92/568292/2/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/59611ec/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-05-14_16_56_1617:36
*** toure|biab is now known as toure17:39
*** jfrancoa has quit IRC17:41
*** fragatina has joined #tripleo17:45
*** fragatina has quit IRC17:46
EmilienMmwhahaha, sdoran, slagle : FYI https://bugs.launchpad.net/tripleo/+bug/177117117:46
openstackLaunchpad bug 1771171 in tripleo "ansible 2.5 takes internal role_name var seriously" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)17:46
*** fragatina has joined #tripleo17:47
EmilienMwho upgraded ansible btw? have we tested it?17:47
*** fragatina has quit IRC17:47
*** fragatina has joined #tripleo17:48
*** agurenko has quit IRC17:50
EmilienMit's going to be tricky to fix this one, we used "role_name" in tripleoclient, tripleo-common and tht17:50
* EmilienM cries17:53
mwhahahawe really need a policy for ansible var names17:53
mwhahahasince namespaces aren't a thing17:53
*** pchavva has joined #tripleo17:57
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add dry run option to toci_quickstart  https://review.openstack.org/56706017:58
*** ykarel has quit IRC17:59
*** amoralej is now known as amoralej|off17:59
slagleEmilienM: are you working on a fix?18:03
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Run containers update only for required packages  https://review.openstack.org/56755018:03
EmilienMslagle: I'm on it18:03
EmilienMit'll take several patches18:03
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Include 'tripleo_role_name' in the inventory  https://review.openstack.org/56834018:05
*** zoli is now known as zoli|gone18:05
*** zoli|gone is now known as zoli18:05
*** marrusl has joined #tripleo18:06
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Revert "temp workaround to bring ci gates back online"  https://review.openstack.org/56834118:07
*** sshnaidm|rover is now known as sshnaidm|off18:07
sshnaidm|offweshay, ^^18:08
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: deploy/upgrade: include tripleo_role_name  https://review.openstack.org/56834218:08
weshaysshnaidm|off, nice work18:09
*** lvdombrkr has joined #tripleo18:09
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097218:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109018:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)18:10
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: deploy-steps: switch to tripleo_role_name  https://review.openstack.org/56834318:16
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: (cleanup) remove usage of role_name  https://review.openstack.org/56834618:19
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: (cleanup) remove usage of role_name  https://review.openstack.org/56834718:21
EmilienMmwhahaha, slagle, sdoran : when you have time https://review.openstack.org/#/q/topic:bug/177117118:21
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy Docker via Ansible and not Puppet  https://review.openstack.org/56137718:22
slagleEmilienM: why do we need the separate cleanup patches?18:22
*** agopi has quit IRC18:24
EmilienMslagle: THT changes needs tripleo-common & tripleoclient patches to merge first18:24
EmilienMslagle: we can't just rename the var in the repos18:24
EmilienMjob would never pass, as the var is used cross repos18:25
*** akrivoka has quit IRC18:25
slagleright, ok18:25
*** akrivoka has joined #tripleo18:26
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates master: Fix for the manila-vnx configuration error  https://review.openstack.org/56834818:27
*** akrivoka has quit IRC18:27
*** cshastri has quit IRC18:28
slagleEmilienM: check https://review.openstack.org/#/c/56834318:29
slaglethe rest lgtm18:29
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates master: Fix for the manila-vnx configuration error  https://review.openstack.org/56834818:30
*** hamzy has quit IRC18:30
EmilienMslagle: ack, thx for quick look18:31
EmilienMmr typo it's me18:31
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: deploy-steps: switch to tripleo_role_name  https://review.openstack.org/56834318:31
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy Docker via Ansible and not Puppet  https://review.openstack.org/56137718:31
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates master: Fix for the manila-vnx configuration error  https://review.openstack.org/56834818:32
*** salmankhan has joined #tripleo18:33
*** hamzy has joined #tripleo18:33
*** wolverineav has quit IRC18:36
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates master: Fix for the manila-vmax configuration error  https://review.openstack.org/56834918:40
*** etingof|afk is now known as etingof18:40
*** Guest16323 is now known as mgagne18:40
*** mgagne has joined #tripleo18:40
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add linter jobs  https://review.openstack.org/56828718:42
*** dparkes has joined #tripleo18:43
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates master: Fix for the manila backend configuration errors  https://review.openstack.org/56834818:45
*** thrash|biab is now known as thrash18:47
*** radeks has quit IRC18:48
openstackgerritHarald JensÃ¥s proposed openstack/puppet-tripleo master: Fix duplicate entries in /etc/sysconfig/iptables  https://review.openstack.org/56835118:49
openstackgerritHarald JensÃ¥s proposed openstack/instack-undercloud master: Fix duplicate entries in /etc/sysconfig/iptables  https://review.openstack.org/56835218:54
*** trown is now known as trown|brb18:54
*** atoth has quit IRC18:55
*** ssbarnea_ has quit IRC18:56
*** ssbarnea_ has joined #tripleo18:58
*** holser__ has joined #tripleo19:05
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177097219:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/177109019:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1770972 in tripleo "CI: Images introspection fails in OVB jobs" [Critical,Triaged] - Assigned to Derek Higgins (derekh)19:10
openstackLaunchpad bug 1771090 in tripleo "Kolla fails to build containers for tripleo in Ci build-container job" [Critical,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)19:10
*** agopi has joined #tripleo19:10
*** cgoncalves has quit IRC19:13
*** cgoncalves has joined #tripleo19:14
*** lvdombrkr89 has joined #tripleo19:21
*** mrunge has quit IRC19:21
*** Goneri has quit IRC19:21
*** dprince has quit IRC19:23
*** lvdombrkr has quit IRC19:24
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add linter jobs  https://review.openstack.org/56828719:24
dpeacockAnyone seen an overcloud complete successfully but with enable-ssh-admin failed?19:24
dpeacockInitializing overcloud plan deployment19:24
dpeacockCreating overcloud Heat stack19:24
dpeacock Stack overcloud/fae7ab9b-eab2-469d-89a3-79ab97ac3e3c CREATE_COMPLETE19:24
dpeacockDeploying overcloud configuration19:24
dpeacock/usr/share/openstack-tripleo-heat-templates/deployed-server/scripts/enable-ssh-admin.sh failed.19:24
*** cgoncalves has quit IRC19:27
*** cgoncalves has joined #tripleo19:29
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Add python script to dynamically compose releases  https://review.openstack.org/56752119:29
*** trown|brb is now known as trown19:31
*** radeks has joined #tripleo19:32
mwhahahadpeacock: slagle is working on a patch to address that it hink19:33
*** gbarros has quit IRC19:33
*** radeks_ has joined #tripleo19:34
dpeacockmwhahaha: ack - thanks - slagle ^^ I hit this.19:34
*** gbarros has joined #tripleo19:34
mwhahahahttps://review.openstack.org/#/c/566129/19:34
mwhahahanot sure if the root cause was ever really determined19:35
*** radeks has quit IRC19:37
*** dprince has joined #tripleo19:39
*** itlinux has quit IRC19:41
*** mrunge has joined #tripleo19:41
*** salmankhan has quit IRC19:48
*** fultonj has joined #tripleo19:50
*** radeks__ has joined #tripleo19:51
aleetrown, mcornea , weshay -- I changed my tripleo-upgrade tasks to use overcloud-deploy as a tag so that they'd be executed but it seems now that they're being skipped.  http://logs.openstack.org/97/567897/3/experimental/tripleo-ci-centos-7-scenario000-multinode-oooq-container-password-changes/248d45a/job-output.txt.gz#_2018-05-14_18_44_26_37421919:51
aleeI must be missing something obvious -- could do with a little guidance when you get a chance.19:52
*** radeks_ has quit IRC19:53
mcorneaalee: from what I can tell the tasks in tripleo-upgrade in upstream jobs are controlled by using vars - for example the ffu job https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset036.yml#L40-L5219:55
aleemcornea, yeah -- but I think I did that ..19:58
aleemcornea, see https://review.openstack.org/#/c/567888/1/config/general_config/featureset053.yml19:59
*** athomas has quit IRC20:00
aleemcornea, I define redeploy_noop and overcloud_generate_passwords20:00
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: (cleanup) remove usage of vars when calling ansible  https://review.openstack.org/56834620:00
mcorneaalee: do you have the output of the job so I can check the tripleo-upgrade tasks?20:01
aleemcornea, which is used https://review.openstack.org/#/c/567300/3/tasks/main.yml and https://review.openstack.org/#/c/567300/3/tasks/redeploy/main.yml20:01
slagledpeacock: test out my patch and let me know if it helps you at all: https://review.openstack.org/#/c/566129/20:01
aleemcornea, http://logs.openstack.org/97/567897/3/experimental/tripleo-ci-centos-7-scenario000-multinode-oooq-container-password-changes/248d45a/20:02
*** ansmith has quit IRC20:04
EmilienMmwhahaha: I see https://review.openstack.org/#/c/561377/ passing now with my fixes applied, so I confirm it was this var20:05
EmilienM(kudos to sdoran to find it)20:05
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Add method for getting dlrn_hash from release and hash_name  https://review.openstack.org/56732020:06
*** radeks_ has joined #tripleo20:06
*** radeks__ has quit IRC20:08
*** dprince has quit IRC20:09
*** liverpooler has quit IRC20:10
mwhahahaEmilienM: so did you propose a patch somewhere to drop the role name vars20:11
EmilienMmwhahaha: this one? https://review.openstack.org/56834620:11
mwhahahayes that one20:11
mwhahahathanks20:11
*** tiswanso_ has joined #tripleo20:11
*** itlinux has joined #tripleo20:12
*** holser__ has quit IRC20:13
*** gbarros has quit IRC20:14
*** tiswanso has quit IRC20:14
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Add method for getting dlrn_hash from release and hash_name  https://review.openstack.org/56732020:14
trozetmwhahaha, EmilienM: are you guys aware of any bug like this: https://paste.fedoraproject.org/paste/KdLMuu6nm5TnJ4CLLPqhUQ/raw ?20:15
EmilienMouch20:15
mwhahahaso i saw something similar in ci20:15
trozetim using tripleo master20:15
mwhahahanot sure the current state of things20:15
trozethit it 2x now20:15
EmilienMmaster?20:15
EmilienMah yeah20:15
mwhahahahttp://logs.openstack.org/92/568292/2/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/59611ec/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-05-14_16_56_1620:15
mwhahahai posted that ~3 hours ago20:16
*** tiswanso_ has quit IRC20:16
trozetmwhahaha, EmilienM: it looks like that file /var/lib/pacemaker/cib/puppet-cib-backup20180514-7-1jfj932 is missing on the node20:16
trozetneed to start digging through the puppet20:17
mwhahahathat's a question for bandini & dciabrin20:17
mwhahahait's likely a pacemaker issue20:17
*** alee is now known as alee_afk20:17
mwhahahatrozet: are your hosts 7.4 or 7.520:18
trozetmwhahaha: yeah im also deploying a noha one at the same time just to see if that one finishes20:18
mwhahahasame with container bits as well20:18
trozetmwhahaha: 7.520:18
dciabrintrozet, containerized deployment? you may want to docker logs the failed docker_puppet_haproxy and look for the first failing puppet dependency20:19
mcorneaalee: ok, I see comparing to the ffu job that the tags in your job do not include overcloud-upgrade tag but I am not sure what uses that tag because it's not from tripleo-upgrade20:20
trozetdciabrin: yeah its a container deployment. Did you look at my log or the one mwhahaha posted?20:20
trozetdciabrin: because in mine it shows the exact error20:20
weshayanybody see what just reset the tripleo gate?20:21
*** fultonj has quit IRC20:22
*** fultonj has joined #tripleo20:22
trozetdciabrin, mwhahaha: heres more output from the docker logs that shows the puppet failure https://paste.fedoraproject.org/paste/On8lki1Gc0JOjxCqbTEw8w/raw20:22
*** rlandy is now known as rlandy|brb20:23
trozetError: unable to get crm_config20:23
trozetcibadmin: Connection to local file '/var/lib/pacemaker/cib/puppet-cib-backup20180514-8-b9gazj' failed: Update does not conform to the configured schema20:23
trozetSignon to CIB failed: Update does not conform to the configured schema20:23
dciabrintrozet, oh...20:23
dciabrintrozet,  that's the infamous pacemaker version mismatch between the host and the container20:23
trozetdciabrin: oh?20:23
dciabrinpacemaker is more recent on the host than in the container image that are being used... unfortunately there's not much we can do right now except making sure that the pacemaker version in container matches the one from the host :/20:24
*** itlinux has quit IRC20:24
trozet[root@overcloud-controller-0 ~]# rpm -q pacemaker20:24
trozetpacemaker-1.1.18-11.el7.x86_6420:24
trozet[root@overcloud-controller-0 ~]# docker ps | grep a140904d8fb020:24
trozeta140904d8fb0        192.0.2.1:8787/master/centos-binary-haproxy:current-tripleo-rdo     "/docker_puppet_ap..."   10 minutes ago      Up 10 minutes                                      haproxy_init_bundle20:24
trozet[root@overcloud-controller-0 ~]# docker exec -it a140904d8fb0 /bin/bash20:24
trozet()[root@overcloud-controller-0 /]# rpm -q pacemaker20:24
trozetpacemaker-1.1.16-12.el7_4.8.x86_6420:24
trozetdciabrin:^ yeah they are different20:24
dciabrinwe're working with bandini to come up with a nice way to fix that, but it's non trivial :(20:25
*** itlinux has joined #tripleo20:25
trozetdciabrin: so how is pacemaker running here?  I'm trying to understand why there is interaction between the host version and the one in the container20:26
trozetdciabrin: also is there an LP filed for this already?20:27
dciabrintozet ok so at install time every pacemaker resource setup is run in transient containers.20:28
dciabrintrozet, yeah i'm sure there is I don't reember it out of the top of my head20:28
dciabrintrozet, accessing the cluster from the container is done via the container's pacemaker binary20:28
trozetdciabrin: ok with you so far20:29
dciabrintrozet, so if pacemaker is too old in the container, it may not be able to validate the xml data coming from the cluster20:29
trozetdciabrin: so pacemaker actually runs on the host and not int he container?20:30
dciabrintrozet, exactly20:30
trozetdciabrin: but hte config is generated in the container?20:30
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates master: Add ability to control Glance's enabled_import_methods  https://review.openstack.org/56766720:30
dciabrintrozet, correct20:30
dciabrintrozet, the cluster itself, and the vip resources run on the host20:30
eric-youngI know that CI is yellow right now but I've a minor patch that could use some review love... https://review.openstack.org/#/c/563914/20:30
trozetdciabrin: and we generate the config in the container because the services are containerized that need to generate the resource configs?20:30
dciabrintrozet, we create the pacemaker resource from a container because in tripleo, a containerized service can only run containers. so any initialization step has to be run from a container20:31
mcorneaalee_afk: ok, I see the overcloud-upgrade tag comes from here https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/multinode-overcloud-upgrade.yml#L3820:31
dciabrintrozet, I'm simplifying a bit, but if you want to run puppet at specific steps, you need to do so in a container20:32
trozetdciabrin: you can still run regular puppet at specific steps that are non-container, we do that for example in step 5 for configuring OVS to connect to opendaylight20:32
trozetdciabrin: cause ovs is not containerized, but ODL is20:33
mcorneaalee_afk: and from here https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test-oooq.sh#L144-L14920:33
dciabrintrozet, let me look before answering too fast :)20:34
dciabrintrozet, can you point me to the specific file you're mentioning?20:37
trozetdciabrin: ok, if you could also find that bug id that would help me out.  I can write a workaround in the installer for now to downgrae the pacemaker version to the one in the redis container20:37
trozetdciabrin: or haproxy container i guess i shoudl say20:37
trozetdciabrin: sure, all the old puppet steps still execute for non container, here is an example:20:39
trozetdciabrin, this service is not containerized: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/services/neutron-opendaylight.yaml#L820:39
*** lblanchard1 has quit IRC20:40
trozetdciabrin: and it is executed in step 4 https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/neutron/plugins/ovs/opendaylight.pp#L10520:40
dciabrintrozet, ok right, I wasn't sure to get your point. so as long as your service is entirely containered (e.g. galera, rabbit, haproxy) the only thing that you can run in steps are containers20:41
*** rlandy|brb is now known as rlandy20:41
dciabrintrozet, which is our problem here :/20:41
dciabrintrozet, I can't seem to find the lp for that bug, let me check with bandini tomorrow and I'll get back to you20:42
trozetdciabrin: yeah, unless you split the service into non-container and container20:42
trozetdciabrin: and only run the pacemaker puppet stuff in hte non-container version20:42
dciabrintrozet, i'd better fix the pcmk issue that split the services :)20:42
trozetdciabrin: yeah20:42
*** pcaruana has quit IRC20:43
*** jcoufal_ has quit IRC20:43
trozetdciabrin: i'll put a workaround in for now to download the container image, run it to check the version, then downgrade hte version on the overcloud-full image before i deploy20:43
trozetdciabrin: thanks for the help and let me know on that LP when you find it20:44
*** lvdombrkr89 has quit IRC20:45
myoung|rucktrozet, dciabrin, mwhahaha, the LP for pacemaker version mismatch: https://bugs.launchpad.net/tripleo/+bug/1770692 - it needs a better writeup20:47
openstackLaunchpad bug 1770692 in tripleo "pacemaker version mismatch (BM vs.container) causing OC deploy failures in gates" [Critical,Fix released]20:47
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add releases script pytest tests to tox.ini  https://review.openstack.org/56764920:47
trozetmyoung|ruck: but thats just a fix for CI20:48
*** tiswanso has joined #tripleo20:48
myoung|rucktrozet: yes i just updated to that effect20:48
trozetmyoung|ruck: oh ok, thanks20:49
dciabrintrozet, i sent a mail to bandini. we have a tracker internally but I can't get my hand on any lp. I'll create one tomorrow if needed20:49
trozetdciabrin: we can use that one myoung|ruck provided20:49
myoung|rucktrozet: or perhaps we should leave that one for just the temporary issue and create a new one with a more nuanced writeup, where we can discuss proposed solutions20:49
myoung|ruck(I would prefer to just use the existing one for the history / less bug sprawl)20:49
* myoung|ruck flipped it back to Triaged20:50
openstackgerritSteve Baker proposed openstack/ansible-role-container-registry master: Don't install kolla  https://review.openstack.org/56837520:57
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add linter jobs  https://review.openstack.org/56828720:58
*** vpickard is now known as vpickard_20:58
*** trown is now known as trown|outtypewww21:00
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add linter jobs  https://review.openstack.org/56828721:00
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: WIP: Add linter jobs  https://review.openstack.org/56828721:00
openstackgerritSteve Baker proposed openstack/puppet-tripleo master: Don't install kolla by default  https://review.openstack.org/56837621:04
myoung|rucktrozet, dciabrin fyi updated https://bugs.launchpad.net/tripleo/+bug/1770692/comments/3...make sense?21:05
openstackLaunchpad bug 1770692 in tripleo "pacemaker version mismatch (BM vs.container) causing OC deploy failures in gates" [Critical,Triaged]21:05
* myoung|ruck thinks https://bugs.launchpad.net/tripleo/+bug/1770692 is a better subject21:06
openstackLaunchpad bug 1770692 in tripleo "potential for pacemaker version mismatch (BM vs.container) is a risk for OC deploy failures in gates" [Critical,Triaged]21:06
trozetmyoung|ruck: yeah i mean tripleo promotion images should be the same version as the containers21:06
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Make standalone role name configurable  https://review.openstack.org/56837821:07
dciabrinmyoung|ruck, thx for the update21:07
trozetmyoung|ruck: looks like this version of pacemaker 1.1.16-12 is in centos 7.4 updates repo21:07
myoung|rucksorry for delay...things have been...lively21:07
*** rmascena has quit IRC21:07
*** ansmith has joined #tripleo21:08
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Make standalone role name configurable  https://review.openstack.org/56837821:10
*** panda|bbl is now known as panda21:10
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Make standalone role name configurable  https://review.openstack.org/56837821:14
*** sshnaidm|off has quit IRC21:16
*** bnemec has quit IRC21:17
*** bnemec has joined #tripleo21:18
myoung|ruckweshay, mwhahaha: boom https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/226/consoleFull21:19
*** abishop has quit IRC21:20
myoung|ruck^^ reviewing and confirming results etc21:20
*** pchavva has quit IRC21:21
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Remove unused option enable_container_images_build  https://review.openstack.org/56838421:25
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Drop default inclusion of undercloud.yaml  https://review.openstack.org/56838621:29
*** paramite has quit IRC21:30
*** itlinux has quit IRC21:34
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Parameterized deployment hosts  https://review.openstack.org/56675221:38
EmilienMweshay: is tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates supposed to be stable?21:43
EmilienMI see a bunch of failures today21:43
*** florianf has quit IRC21:44
SamYapleme too. but i have alot of mirrors around my house21:45
myoung|ruckweshay, mwhahaha: https://review.openstack.org/#/c/568341 (revert container update disable) is thru check...21:49
*** rcernin has joined #tripleo21:53
openstackgerritwes hayutin proposed openstack/tripleo-common master: DNM: test ccheck patch  https://review.openstack.org/56759621:54
myoung|ruckweshay, mwhahaha: 2018-05-14 21:24:24,089 18184 INFO     promoter Promoting the container images for dlrn hash c06e315e87d6326f984bbbb038f7cdec9d1e187f on master to current-tripleo21:54
*** ssbarnea_ has quit IRC21:58
openstackgerritwes hayutin proposed openstack/tripleo-common master: DNM, TESTING 568341  https://review.openstack.org/56838922:02
*** rh-jelabarre has quit IRC22:03
*** bfournie has quit IRC22:06
*** jistr has quit IRC22:07
*** etingof has quit IRC22:08
*** jruzicka has quit IRC22:08
*** jruzicka has joined #tripleo22:08
*** jistr has joined #tripleo22:10
*** threestrands has joined #tripleo22:16
*** threestrands has quit IRC22:16
*** threestrands has joined #tripleo22:16
*** radeks_ has quit IRC22:17
*** slaweq has quit IRC22:20
*** slaweq has joined #tripleo22:21
*** etingof has joined #tripleo22:21
openstackgerritSteve Baker proposed openstack/tripleo-quickstart-extras master: Run "overcloud container image prepare" only once  https://review.openstack.org/52862122:22
weshaygo steve baker go!22:24
*** abishop has joined #tripleo22:29
mwhahahaovb just erroed on getting envs22:29
mwhahahamyoung|ruck, weshay: rdocloud problems? https://logs.rdoproject.org/51/567951/1/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z271f8a1868a44b34a2eccd6d5b3b92fd/console.txt.gz#_2018-05-14_22_23_54_91522:30
* weshay looks22:30
* mwhahaha successfully installed memcache in an all in one22:31
* mwhahaha gives up for monday22:31
* myoung|ruck mutters curses22:32
*** dxiri_ has joined #tripleo22:34
weshaymwhahaha, ya that looks like tebroker22:35
weshayis borked22:35
weshayor the cloud.. would need to poke at which22:35
mwhahahahappy monday22:35
weshaymwhahaha, there really wasn't a weekend.. so22:35
weshayit's more like wednesday22:35
myoung|ruckweshay: do you have a few to chat BJ?  My eyeballs are getting dry and would like to sync briefly before I run out of brain juice22:36
weshaymyoung|ruck, k going to blue22:36
weshaymy channel22:36
myoung|ruckditto22:36
*** dxiri has quit IRC22:37
*** mcornea has quit IRC22:44
*** bfournie has joined #tripleo22:46
*** alee_afk is now known as alee22:52
*** dougbtv_ has joined #tripleo22:56
*** agopi has quit IRC22:57
openstackgerritMerged openstack/tripleo-quickstart-extras master: Include extra_params option in reproducer-quickstart.sh.  https://review.openstack.org/54693223:00
openstackgerritMerged openstack/tripleo-quickstart-extras master: Replace the invalid links with the right ones for the document  https://review.openstack.org/53007123:10
openstackgerritMerged openstack/tripleo-upgrade stable/queens: Allow running update/upgrade converge with --update-plan-only  https://review.openstack.org/56822223:10
openstackgerritMerged openstack/tripleo-common master: Incorrect plan name in send_message task  https://review.openstack.org/56713023:10
openstackgerritMerged openstack/tripleo-heat-templates master: Clean up Cinder backends in capabilities map  https://review.openstack.org/56656623:10
openstackgerritMerged openstack/tripleo-heat-templates master: yaml-validate: sets isn't python3  https://review.openstack.org/56712523:16
openstackgerritMerged openstack/tripleo-upgrade stable/queens: Convert CLI options to variables for updates/upgrades/ffwd.  https://review.openstack.org/56827723:16
openstackgerritMerged openstack/tripleo-heat-templates master: Default Octavia SSH pub key to UC default keypair  https://review.openstack.org/56802223:17
openstackgerritMerged openstack/tripleo-common master: Check pub key file perms and default to pub key data  https://review.openstack.org/56720523:17
openstackgerritCarlos Goncalves proposed openstack/tripleo-common stable/queens: Check pub key file perms and default to pub key data  https://review.openstack.org/56839423:18
openstackgerritCarlos Goncalves proposed openstack/tripleo-heat-templates stable/queens: Default Octavia SSH pub key to UC default keypair  https://review.openstack.org/56839523:18
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Run tempest containerized  https://review.openstack.org/56192023:19
*** rajinir has quit IRC23:19
*** rlandy has quit IRC23:20
*** saneax is now known as saneax-_-|AFK23:21
*** rpioso is now known as rpioso|afk23:26
openstackgerritMerged openstack/tripleo-heat-templates master: Switch public endpoints to use FQDNs by default  https://review.openstack.org/55992623:27
openstackgerritMerged openstack/tripleo-quickstart master: Remove explicit --config-download  https://review.openstack.org/56585623:27
*** fultonj has quit IRC23:28
weshaymwhahaha, ok.. I'll take a bit of a merge party23:35
openstackgerritSteve Baker proposed openstack/tripleo-quickstart-extras master: Move BUILD_ID setting logic out of template  https://review.openstack.org/56839623:35
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates stable/queens: Clean up Cinder backends in capabilities map  https://review.openstack.org/56839723:35
openstackgerritMerged openstack/tripleo-common master: Run yum clean to reduce size of docker image layer  https://review.openstack.org/56827223:37
openstackgerritMerged openstack/tripleo-heat-templates master: Add compute node L3 agent container for DVR  https://review.openstack.org/53793123:37
openstackgerritMerged openstack/tripleo-heat-templates master: Make the KeystoneURL stack output versionless  https://review.openstack.org/56680823:37
openstackgerritMerged openstack/tripleo-upgrade master: Pass overcloud stack name to upgrade run.  https://review.openstack.org/56830223:37
*** agopi has joined #tripleo23:37
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Include 'tripleo_role_name' in the inventory  https://review.openstack.org/56834023:39
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: deploy-steps: switch to tripleo_role_name  https://review.openstack.org/56834323:42
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deploy Docker via Ansible and not Puppet  https://review.openstack.org/56137723:43
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: (cleanup) remove usage of role_name  https://review.openstack.org/56834723:44
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/pike: Add in a STOPSIGNAL configuration  https://review.openstack.org/56555223:51
*** tosky has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!