Thursday, 2020-03-12

openstackgerritJohn Fulton proposed openstack/tripleo-ansible master: Ensure enabled_services is a list, not a string  https://review.opendev.org/71258900:01
openstackgerritJohn Fulton proposed openstack/tripleo-ansible master: Ensure enabled_services is a list, not a string  https://review.opendev.org/71258900:02
EmilienMdciabrin_: not afik00:44
dciabrin_ack thx00:44
EmilienMdciabrin_: no sleep? :D00:44
*** openstackstatus has joined #tripleo00:45
*** ChanServ sets mode: +v openstackstatus00:45
dciabrin_EmilienM, that's the next task on my todo list :P00:45
EmilienMlol00:45
*** ayoung has quit IRC01:15
*** ayoung has joined #tripleo01:15
weshay|ruckEmilienM, GO TO SLEEP01:53
EmilienMok01:54
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Remove duplicated topic parts from default value  https://review.opendev.org/71216901:57
openstackgerritMerged openstack/tripleo-heat-templates master: Add NeutronDhcpAgentDnsmasqEnableAddr6List param  https://review.opendev.org/71034201:57
openstackgerritMerged openstack/tripleo-heat-templates master: Create ResellerAdmin role when deploying Ceph RGW  https://review.opendev.org/71233801:57
openstackgerritMerged openstack/puppet-tripleo master: Remove duplicate entry of collectd-python package  https://review.opendev.org/71228901:57
openstackgerritMerged openstack/tripleo-quickstart master: Pinning pytest-html to <= 2.0.1  https://review.opendev.org/71252701:57
openstackgerritMerged openstack/tripleo-quickstart-extras master: Pinning pytest-html to <= 2.0.1  https://review.opendev.org/71252601:57
openstackgerritMerged openstack/tripleo-heat-templates master: rabbitmq: Open ports 25673-25683 for CLI tools  https://review.opendev.org/71197501:57
openstackgerritMerged openstack/tripleo-heat-templates stable/train: [TRAIN-ONLY] Add systemd-wrapper update tasks  https://review.opendev.org/71220001:57
weshay|ruckbam01:58
openstackgerritRyan McCabe proposed openstack/tripleo-heat-templates master: Add STF environment  https://review.opendev.org/69462302:12
*** rlandy|bbl is now known as rlandy02:15
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Move PrepareContainerImageEnv functionality to utils  https://review.opendev.org/71259802:15
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Move UploadTemplatesAction functionality to utils  https://review.opendev.org/71259902:15
*** ayoung has quit IRC02:18
*** ayoung has joined #tripleo02:19
EmilienMcloudnull: https://417134255a541bdb4972-2f7fc4cc5295aef11715d5c702aba78c.ssl.cf1.rackcdn.com/712464/7/check/tripleo-ci-centos-8-containers-multinode/4bf537d/logs/undercloud/home/zuul/overcloud_deploy.log02:21
EmilienMpublicURL endpoint for workflowv2 service not found02:21
EmilienMd0ugal: ^ in tripleoclient/v1/overcloud_deploy.py we call self.clients.workflow_engine02:21
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Use jinja templating when setting facts  https://review.opendev.org/71187202:21
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Fix error message from tripleo_passwords_rotate module  https://review.opendev.org/71260102:21
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Add tripleo_image_params_prepare ansible module  https://review.opendev.org/71260202:21
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Add tripleo_templates_upload module  https://review.opendev.org/71260302:21
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Add tripleo_plan_parameters_update module  https://review.opendev.org/71260402:21
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Add playbook for create deployment plan  https://review.opendev.org/71260502:21
openstackgerritwes hayutin proposed openstack/tripleo-ci master: make scenario001 voting/gating  https://review.opendev.org/71198802:39
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Update overcloud image with component repo  https://review.opendev.org/71252402:53
*** ayoung has quit IRC02:58
*** ayoung has joined #tripleo02:59
*** ayoung has quit IRC03:08
*** ayoung has joined #tripleo03:09
*** surajpatil1 has joined #tripleo03:17
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Update overcloud image with component repo  https://review.opendev.org/71252403:19
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Update overcloud image with component repo  https://review.opendev.org/71252403:23
openstackgerritwes hayutin proposed openstack/tripleo-upgrade master: move update to centos-8  https://review.opendev.org/71198503:25
*** psachin has joined #tripleo03:37
openstackgerrityatin proposed openstack/tripleo-quickstart-extras master: Add gating repo to update repos only if it's setup  https://review.opendev.org/71162703:41
*** ayoung has quit IRC03:41
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Update overcloud image with component repo  https://review.opendev.org/71252403:43
*** rlandy has quit IRC03:44
*** ayoung has joined #tripleo03:45
*** ykarel|away is now known as ykarel03:48
*** apetrich has quit IRC04:00
*** bdodd has joined #tripleo04:00
*** udesale has joined #tripleo04:17
*** ayoung has quit IRC04:19
*** ayoung has joined #tripleo04:22
openstackgerritMerged openstack/tripleo-heat-templates stable/train: [update/upgrade] Use include_tasks instead of import_tasks  https://review.opendev.org/71193904:47
*** jmasud has quit IRC04:53
*** skramaja has joined #tripleo04:54
*** jmasud has joined #tripleo04:55
*** raukadah is now known as chandankumar05:16
*** ayoung has quit IRC05:24
*** ayoung has joined #tripleo05:26
*** lbragstad__ has joined #tripleo05:33
*** lbragstad_ has quit IRC05:35
*** evrardjp has quit IRC05:35
*** evrardjp has joined #tripleo05:36
*** ayoung has quit IRC05:43
*** ayoung has joined #tripleo05:44
openstackgerritMerged openstack/tripleo-ci master: make centos-8-containers-multinode voting  https://review.opendev.org/71257305:49
*** sri_ has joined #tripleo05:52
*** surajpatil1 has quit IRC05:53
*** surajpatil1 has joined #tripleo05:58
openstackgerrityatin proposed openstack/tripleo-quickstart-extras master: Fix installation of contextlib2  https://review.opendev.org/71250106:02
openstackgerrityatin proposed openstack/tripleo-heat-templates stable/queens: Unnessary deploying of bootstrap-osd ceph auth entry  https://review.opendev.org/71234806:03
*** Damjanek has joined #tripleo06:07
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Set overcloud_container_cli to podman for c8  https://review.opendev.org/71201306:11
*** abdysn has joined #tripleo06:14
*** waleedm has joined #tripleo06:18
*** saneax has joined #tripleo06:27
*** surajpatil1 has quit IRC06:40
*** marios has joined #tripleo06:48
ykarelchandankumar, can u review https://review.opendev.org/#/c/711627/, just rebase on molecule fix06:52
*** lmiccini has joined #tripleo06:53
*** bandini has joined #tripleo07:01
abdysnTengu, good morning, it seams this bug https://bugs.launchpad.net/tripleo/+bug/1861782?comments=all started to show on train.07:02
openstackLaunchpad bug 1861782 in tripleo "Master TLS job failed while deploying overcloud with Could not find versioned identity endpoints when attempting to authenticate" [High,Fix released] - Assigned to Cédric Jeanneret (cjeanner)07:02
Tenguabdysn: heya! soooo we might want to backport the patch I guess?07:03
abdysnTengu, yes would be great07:03
Tenguok, gimme a minute to set the proper flags on the LP, and start the backport process.07:04
abdysnTengu, also this problem arised http://paste.openstack.org/show/790572/07:04
*** dpawlik has joined #tripleo07:04
Tenguabdysn: any log showing the issue on train?07:04
abdysnlet me check07:04
Tenguabdysn: i.e. did you get it on CI, or on a custom lab, or... ?07:04
abdysncutome07:04
Tenguah, for keystone, you probably want to wait for EmilienM07:04
abdysnTengu, i tried to deploy an overcloud with tlseverywhere enabled again from scratch, and encountered these issues07:05
Tenguabdysn: care to comment the LP?07:06
Tengumeh.... cherry-pick conflict.....07:06
abdysnsorry, what does LP stand for?07:06
Tenguabdysn: launchpad07:06
abdysnah, for the pld bug ?07:06
abdysnold*07:07
Tenguyep07:07
abdysnok07:07
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-ansible stable/train: Also use enabled_networks  https://review.opendev.org/71261707:08
Tenguabdysn: -^^07:08
abdysngreat07:08
Tengulet's wait for zuul state, but it should be pretty easy to merge.07:10
abdysndon't we also need the other patch07:10
abdysnhttps://review.opendev.org/#/c/705634/207:10
Tenguoh. good point. yes.07:10
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates stable/train: Generate /etc/hosts early on both under and overcloud  https://review.opendev.org/71261907:12
Tenguabdysn: -^07:12
Tenguoh. f...07:13
Tengudepends-on07:13
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates stable/train: Generate /etc/hosts early on both under and overcloud  https://review.opendev.org/71261907:14
Tengubetter.07:14
abdysnTengu, thanks07:14
*** rpittau|afk is now known as rpittau07:17
abdysnTengu, here is the log http://paste.openstack.org/show/790574/07:18
Tenguabdysn: ah, that's the one for keystone. unrelated, but ok.07:20
Tenguabdysn: you probably want to open a new bug on launchpad07:20
abdysnit is the same for the old bug07:20
Tenguor... meh, on one line, it's just unreadable.07:20
abdysnTengu, sorry about that, would separate them next time.07:22
Tenguah, yes, that's the same one then.07:22
Tenguthanks vim and :s :)07:22
Tenguso yeah - both backported patches should correct this situation on train.07:23
*** avivgt has quit IRC07:57
*** bogdando has joined #tripleo08:00
*** slaweq has joined #tripleo08:02
*** jfrancoa has joined #tripleo08:03
*** apetrich has joined #tripleo08:03
*** amoralej|off is now known as amoralej08:06
openstackgerritRabi Mishra proposed openstack/python-tripleoclient master: WIP Use ansible playbook for plan creation  https://review.opendev.org/71262108:07
*** tesseract has joined #tripleo08:08
*** jcapitao has joined #tripleo08:15
*** tosky has joined #tripleo08:18
abdysnTengu, hi so tried to apply the patches manually, but got this error http://paste.openstack.org/show/790576/, have any ideas?08:24
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Convert all of the fedora28 container usage to CentOS8  https://review.opendev.org/71250908:25
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: order pcmk_remote after Exec['wait-for-settle']  https://review.opendev.org/71262308:25
Tenguoh. tripleo-ansible wasn't renamed in train? lemme check08:25
openstackgerritDaniel Bengtsson proposed openstack/python-tripleoclient master: Stop configuring install_command in tox.  https://review.opendev.org/69350208:26
Tengumeh.08:26
*** gchamoul is now known as strider|afk08:27
Tenguabdysn: you didn't apply the patch, did you? because the patch seems to have the right value, i.e.: name: tripleo-hosts-entries08:27
*** strider|afk is now known as gchamoul08:27
Tenguwait.. what?!08:27
Tengu#facepalm... ok, got it. nice catch abdysn08:28
*** matbu has quit IRC08:29
abdysnTengu, ???08:29
*** matbu has joined #tripleo08:29
abdysnTengu, i cant see a problem! did you get it ?08:29
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-heat-templates stable/train: Generate /etc/hosts early on both under and overcloud  https://review.opendev.org/71261908:29
Tenguabdysn: I did, yeah -^^  :)08:30
Tenguthe cherry-pick wasn't clean, and, in train, tripleo-ansible still has the hyphens instead of underscores for the roles.08:30
abdysnahhhh08:30
abdysnTengu, thanks :08:30
abdysnTengu, i would have wasted my day and wouldn't have found the issue08:31
Tenguhehe08:31
*** ayoung has quit IRC08:33
*** ayoung has joined #tripleo08:36
*** jbadiapa has joined #tripleo08:36
*** ramishra has quit IRC08:37
*** ramishra has joined #tripleo08:39
*** jpena|off is now known as jpena08:45
*** jpich has joined #tripleo08:47
slaweqchandankumar: hi, can You help me (again) with this tripleo-multinode job for tobiko?08:47
slaweqchandankumar: patch is still https://review.opendev.org/#/c/683126/08:47
slaweqI proposed some changes which I thought will be needed (see depends-on patches)08:47
slaweqbut I stil have issues with it https://8e338d999459fea4c1be-a137dd3672ee6ad9ee2c9c4f84c4aa5c.ssl.cf2.rackcdn.com/683126/28/check/tobiko-tripleo-multinode-containers/ead56c4/logs/undercloud/home/zuul/overcloud_deploy.log08:47
slaweqcan You help me with that maybe?08:48
slaweqor maybe I should use some other featureset (last time I was trying 006)08:48
chandankumarslaweq, ah this is new error08:49
chandankumarslaweq, why fs006?08:50
chandankumarslaweq, earlier we were using fs016 na?08:50
*** florianf has joined #tripleo08:50
*** pkopec has joined #tripleo08:51
*** rpittau is now known as rpittau|bbl08:52
*** mmagr has joined #tripleo08:53
chandankumarslaweq, if we are using fs06 then we need to do this https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset001.yml#L7808:53
chandankumarmay be we also need overcloud_container_cli here also08:53
*** jmasud has quit IRC08:54
*** pcaruana has joined #tripleo08:54
*** jmasud has joined #tripleo08:56
openstackgerritMerged openstack/tripleo-ci master: make scenario001 voting/gating  https://review.opendev.org/71198808:56
slaweqchandankumar: I remember that with 016 I had some other issues also08:58
*** xek has joined #tripleo08:58
slaweqso I was trying other which has got heat installed in oc08:58
slaweqI can try again fs016 is You think is the best one to use08:59
slaweqchandankumar: I think that I can also use fs01, right?08:59
chandankumarslaweq, last time after setting overcloud container cli, it failed at different location08:59
chandankumarslaweq, yes fs01 takes more time, i think more than 3 hours08:59
chandankumarfs01 is most trusted fs runned in rdo third party job09:00
ykarelfs001 works with ovb, iirc it will not work with multinode as it is09:06
ykarelor it works?09:07
*** marios has quit IRC09:11
*** lucasagomes has joined #tripleo09:12
*** alexmcleod has joined #tripleo09:13
*** dtantsur|afk is now known as dtantsur09:13
slaweqchandankumar: ok, so I will try fs016 again and will ping You when it will fail :)09:13
chandankumarslaweq, yes sure09:14
*** ykarel is now known as ykarel|lunch09:19
chandankumarslaweq, https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml#L355 was job defined in toci related to fs01609:21
slaweqchandankumar: thx09:28
slaweqI will check it09:28
*** holser has joined #tripleo09:28
*** Damjanek has left #tripleo09:30
openstackgerritMerged openstack/puppet-pacemaker master: order pcmk_remote after Exec['wait-for-settle']  https://review.opendev.org/71262309:33
openstackgerritMerged openstack/tripleo-ansible master: Trigger molecule jobs once a week  https://review.opendev.org/71247709:33
*** avivgt has joined #tripleo09:37
*** sshnaidm|afk is now known as sshnaidm09:39
*** derekh has joined #tripleo09:41
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Added tripleo-ci-centos-8-scenario007-multinode-oooq-container  https://review.opendev.org/71229409:43
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Set overcloud_container_cli to podman for c8  https://review.opendev.org/71201309:44
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: [WIP] Start to remove the last uses of Mistral in tripleocliento  https://review.opendev.org/71263109:44
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: [WIP] Start to remove the last uses of Mistral in tripleoclient  https://review.opendev.org/71263109:45
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates master: [Q->T] Introduce Queens to Train Upgrade  https://review.opendev.org/71145909:50
*** ayoung has quit IRC09:55
openstackgerritGabriele Cerami proposed openstack/tripleo-common master: Remove skydive support  https://review.opendev.org/71251709:55
*** ayoung has joined #tripleo09:55
*** ansmith_ has joined #tripleo09:57
*** ansmith has quit IRC10:00
openstackgerritFrancesco Pantano proposed openstack/tripleo-ansible master: Ensure enabled_services is a list, not a string  https://review.opendev.org/71258910:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-heat-templates master: Use exists filter instead of stat where possible  https://review.opendev.org/71248310:05
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-heat-templates master: Use jinja raw tag instead of quoted concatenation  https://review.opendev.org/71250410:05
openstackgerritMerged openstack/tripleo-quickstart-extras master: Add gating repo to update repos only if it's setup  https://review.opendev.org/71162710:09
*** marios has joined #tripleo10:16
*** ykarel|lunch is now known as ykarel10:19
*** dmacpher has joined #tripleo10:37
*** dmacpher__ has quit IRC10:38
*** pcaruana has quit IRC10:42
*** jmasud has quit IRC10:52
*** jmasud has joined #tripleo10:53
*** pcaruana has joined #tripleo10:55
Tengumeh.... "Failed to download metadata for repo 'delorean-component-baremetal'"  weird.11:00
Tenguwhile trying master on c811:00
*** jamesdenton has joined #tripleo11:01
*** jcapitao is now known as jcapitao_afk11:02
Tenguoh. damned. my fault. my network..11:05
openstackgerritSergii Golovatiuk proposed openstack/tripleo-heat-templates stable/train: Use a jinja block to reduce str_replace repetition  https://review.opendev.org/71194011:08
*** derekh has quit IRC11:10
*** sshnaidm has quit IRC11:18
*** sshnaidm has joined #tripleo11:19
*** derekh has joined #tripleo11:20
*** sshnaidm has quit IRC11:22
*** sshnaidm has joined #tripleo11:30
*** sshnaidm has quit IRC11:30
abdysnTengu, so just wanted to inform you that the deployment succeeded. it seams that the keystone issue was not related.11:44
*** avivgt has quit IRC11:46
*** jpena is now known as jpena|lunch11:49
openstackgerritFrancesco Pantano proposed openstack/tripleo-ansible master: Revert "Add a new set of tasks to build the openstack pool and key list"  https://review.opendev.org/71266111:50
openstackgerritFrancesco Pantano proposed openstack/tripleo-heat-templates master: Revert "Add CephBasePoolVars and CephKeyVars structures"  https://review.opendev.org/71266211:50
*** sshnaidm has joined #tripleo11:52
*** sshnaidm has quit IRC11:53
*** sshnaidm has joined #tripleo11:54
*** ansmith_ has quit IRC11:54
*** holser has quit IRC11:54
*** toure has joined #tripleo11:58
*** rlandy has joined #tripleo12:01
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-heat-templates stable/train: Use a common playbook import for common plays  https://review.opendev.org/71210212:03
openstackgerritFrancesco Pantano proposed openstack/tripleo-ansible master: Revert "Add a new set of tasks to build the openstack pool and key list"  https://review.opendev.org/71266112:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-heat-templates stable/train: Remove unneccessary indentation from common tasks  https://review.opendev.org/71234612:05
Tenguabdysn: told ya ;)12:11
abdysnTengu, :D12:12
openstackgerritArx Cruz proposed openstack/tripleo-ci master: Make centos-8 scenario007 and scenario010 voting  https://review.opendev.org/71266512:12
weshay|ruckTengu, you have a sec? wondering what vars to update to enable basic validations that the DF would want to maintain as a first example12:14
weshay|ruckhttps://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L795-L79912:14
*** udesale_ has joined #tripleo12:14
weshay|ruckthanks artom12:14
weshay|ruckthanks arxcruz|rover12:14
Tenguweshay|ruck: heya - lemme check that12:15
Tenguweshay|ruck: btw, saw your ping from yesteday - I understand mwhahaha answered, and that we can drop this service container? at least I had to drop it from tripleo-common in order to deploy a standalone on c8 against master, with 3621159be13b41f8ead2e873b357f4a5 as container tag value12:16
weshay|ruckarxcruz|rover, comments on it12:16
*** udesale has quit IRC12:17
weshay|ruckTengu, oh panda put up a review12:17
* weshay|ruck checks it12:17
Tenguweshay|ruck: ah, good! it just wasn't available in packages, so I could ensure my lab IS supporting package building on centos-8 with the new thing :).12:17
Tenguandreaf: it does.12:17
Tenguerrr... what?12:17
weshay|ruckhttps://review.opendev.org/#/c/712517/12:18
*** social has joined #tripleo12:18
Tenguoh... *and:     my finger sliped on tab, sorry andreaf ^^'12:18
openstackgerritMerged openstack/tripleo-validations stable/train: Skeleton role, get with the decade  https://review.opendev.org/71228612:18
Tenguweshay|ruck: fun fact: that's *exactly* what I've done in my env.12:18
Tenguso. back to validation.12:19
weshay|ruckTengu, you should have put up a patch dude12:19
Tenguweshay|ruck: I did it this morning and was able to validate it just now :)12:19
weshay|ruckdon't make me act like EmilienM :)12:19
weshay|ruckcool12:19
TenguI'd rather avoid pushing things that aren't tested.12:19
Tenguanyway, patch is up, I'll watch it.12:19
weshay|ruckTengu, looks like it's going to pass :)12:20
openstackgerritArx Cruz proposed openstack/tripleo-ci master: Make centos-8 scenario007 and scenario010 voting  https://review.opendev.org/71266512:20
Tenguweshay|ruck: I'd love to +A but.... I'm still no core (*hint-hint*)12:20
Tengu:)12:20
*** gfidente has joined #tripleo12:21
*** rh-jelabarre has joined #tripleo12:24
*** rcernin has quit IRC12:25
fultonjgfidente fmount fyi https://bugs.launchpad.net/tripleo/+bug/186701212:26
openstackLaunchpad bug 1867012 in tripleo "tripleo-ansible broken scenario tripleo_ceph_run_ansible " [High,Confirmed]12:26
*** ysandeep is now known as ysandeep|away12:26
*** tkajinam has quit IRC12:27
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Enable objects versioning for Ceph RadosGW  https://review.opendev.org/71020212:28
gfidentefultonj ah nice12:29
fultonjgfidente: also https://review.opendev.org/#/c/712661 + https://review.opendev.org/#/c/71266212:30
fultonjgfidente: i think we need a new approach12:30
fultonji'll try to reproduce the Launchpad bug 1867012 on my own machine w/ molecule12:31
openstackLaunchpad bug 1867012 in tripleo "tripleo-ansible broken scenario tripleo_ceph_run_ansible " [High,Confirmed] https://launchpad.net/bugs/186701212:31
*** jpena|lunch is now known as jpena12:32
fultonjgfidente: disambiguation("new appraoch") --> on building pools/keys with ansible; there's a flaw in hiera appraoch but we can revert for now to fix bug and come up with a new way later12:32
gfidentefultonj fmount ok I see it didn't go in train yet12:34
cgoncalvesfultonj: hi. a couple of days ago gfidente and I had a chat about a CI issue involving Ceph in the upgrade phase. the issue is still unresolved. would it be possible if you could help us?12:35
gfidentehaven't looked much into the problem though12:35
gfidenteis it necessary to revert the changes in master?12:35
gfidentevs fixing it?12:35
fultonjunless you have a fast fix12:36
fultonjgfidente: given the circumstances fmount and i thought yes12:36
gfidenteah because ci failure12:36
gfidenteok so let's merge the revert12:37
fultonjnot exactly but yes, i think we should merge the revert12:37
fultonjcgoncalves: which CI upgrade job?12:37
cgoncalvesfultonj, https://review.opendev.org/#/c/710473/12:37
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/train: Create ResellerAdmin role when deploying Ceph RGW  https://review.opendev.org/71267012:37
cgoncalvesceph issue: https://zuul.opendev.org/t/openstack/build/125e42ce57724b67b368a3f8f15f4c21/log/logs/undercloud/home/zuul/standalone-ansible-T9r6F_/ceph-ansible/ceph_ansible_command.log#559712:38
fultonjcgoncalves: what is the name of the job that produed that log?12:39
*** ansmith_ has joined #tripleo12:39
dciabrin_abishop, hey o/ I opened https://bugs.launchpad.net/tripleo/+bug/1867135 to fix upload-swift-artifacts in train and I believe your two patches in master are sufficient to fix it? am i correct?12:39
openstackLaunchpad bug 1867135 in tripleo "Train: upload-puppet-modules fails to store data in swift when called multiple times" [Undecided,In progress] - Assigned to Damien Ciabrini (dciabrin)12:39
cgoncalvesfultonj, tripleo-ci-centos-7-scenario010-standalone-upgrade. job definition not merged yet (WIP patch)12:39
abishopdciabrin_: yes! I was planning to propose both to train but got tied up yesterday12:40
abishopas I suspect you noticed, you need two patches there12:41
dciabrin_abishop, right. want me to merge the two and propose a review while i'm at it?12:41
abishopdciabrin_: sure, thx!12:41
gfidentecgoncalves fultonj so let me add some comments12:42
fultonjthanks gfidente12:42
gfidenteas far as I can tell, the job is triggering site-docker twice, it's not triggering openstack overcloud upgrade or update12:43
gfidenteI was looking into the upgrade command .sh to make sure12:43
weshay|ruckEmilienM, you good w/ https://review.opendev.org/#/c/712665/ ?12:44
cgoncalveshttps://zuul.opendev.org/t/openstack/build/125e42ce57724b67b368a3f8f15f4c21/log/logs/undercloud/home/zuul/standalone_upgrade.log12:44
*** ahosam has joined #tripleo12:44
cgoncalves^ is this not the standalone upgrade log?12:44
abishopdciabrin_: one sec. I forgot I have https://review.opendev.org/71079312:44
dciabrin_oh12:44
gfidentecgoncalves this guy https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_125/710473/2/check/tripleo-ci-centos-7-scenario010-standalone-upgrade/125e42c/logs/undercloud/home/zuul/standalone-upgrade.sh12:45
abishoplet me clean it up now, and I'll have both patches proposed on train in a few minutes12:45
gfidenteso it's running tripleo upgrade command12:45
dciabrin_abishop, ok awesome!12:45
*** dciabrin_ is now known as dciabrin12:45
gfidentewhich is at some point rerunning site-docker.yaml and while running that, timing out when the monitor is brought back up after update (because there aren't enough to preserve quorum)12:46
EmilienMweshay|ruck: why do we have 2 patches for enabling voting on scenario1012:46
gfidenteso I think there are two problems12:46
EmilienMhttps://review.opendev.org/#/c/712189/ was my patch, in the gate now12:46
*** jcapitao_afk is now known as jcapitao12:46
weshay|ruckAH12:46
EmilienMand now arxcruz|rover has another one https://review.opendev.org/#/c/712665/12:47
*** psachin has quit IRC12:47
EmilienMfor 7 as well12:47
weshay|ruckEmilienM, because you didn't use the topic we were12:47
weshay|ruckmeh12:47
EmilienMwhat topic?12:47
gfidente1) site-docker is failing to bring back up the pre-existing mon (this might be by design, when there is a single mon)12:47
cloudnullmornings12:47
weshay|ruckEmilienM, https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged)12:47
arxcruz|roverEmilienM: weshay|ruck i'll update mine to set only scen007 ok ?12:48
weshay|ruckarxcruz|rover++12:48
gfidente2) tripleo upgrade comman is re-running site-docker but it should be running rolling_update (which *will* fail with 1 mon anyway)12:48
weshay|ruckgfidente, arxcruz|rover you guys work out the queens thang12:48
gfidentecgoncalves I think we need to check with someone more familiar with "upgrade" command why it is re-running site-docker12:48
gfidenteweshay|ruck I just added the depends-on; will report if it works yes, thanks!12:49
gfidentearxcruz|rover++12:49
weshay|ruckreviews on https://review.opendev.org/#/c/712517/12:49
EmilienMarxcruz|rover: sure of feel free to abandon mine, I don't care12:49
cgoncalvesgfidente, sure. should I file bug?12:49
gfidentecgoncalves I think jfrancoa or jistr can help understand if that is the expected behavior to start with12:50
fultonjcgoncalves: perhaps as workaround you can override the playbook12:50
arxcruz|roverEmilienM: you'rs already on gate, and I already make the change on mine, let's keep it as it is12:50
EmilienMk12:50
openstackgerritArx Cruz proposed openstack/tripleo-ci master: Make centos-8 scenario007 voting  https://review.opendev.org/71266512:50
fultonjas workaround to bug to open12:50
cgoncalvesfultonj, which playbook and how?12:50
openstackgerritAlan Bishop proposed openstack/tripleo-common stable/train: Improve upload-swift-artifacts handling of json data  https://review.opendev.org/71079312:50
openstackgerritAlan Bishop proposed openstack/tripleo-common stable/train: Fix upload-swift-artifacts handling of json data  https://review.opendev.org/71267212:50
abishopdciabrin: ^^12:51
fultonjhttps://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/ceph-ansible/ceph-base.yaml#L6012:51
gfidentefultonj cgoncalves note that rolling_update with fail by design with 1 mon12:51
fultonjgfidente: ahh12:51
gfidenteso even if we solve the tripleo upgrade problem, triggering wrong playbook (maybe it's set via tht somewhere in the job), we won't see it passing12:51
cgoncalvesfultonj, hmm. I am no expert in tripleo-ceph. I'm afraid I wouldn't be able to come up with a workaround12:51
fultonjcgoncalves: given what gfidente just pointed out and you ahve 1 mon, i don't see how you can trigger12:52
fultonjcgoncalves: so the wrokaround won't work12:52
fultonjhttps://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/ceph-ansible/ceph-base.yaml#L675-L68112:53
cgoncalvesjfrancoa, jistr: thoughts? ^12:53
gfidentejistr jfrancoa if you guys have a few, just checking if what we're seeing is a bug or not12:53
fultonjcgoncalves: i was going to say you can override CephAnsiblePlaybook with /usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml12:53
gfidentejistr jfrancoa there is this job https://review.opendev.org/#/c/710473/ where after initial deploy (with ceph) is starting upgrade using "tripleo upgrade" command12:54
gfidentejistr jfrancoa see https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_125/710473/2/check/tripleo-ci-centos-7-scenario010-standalone-upgrade/125e42c/logs/undercloud/home/zuul/standalone-upgrade.sh12:54
fultonjcgoncalves: but gfidente is right, that won't work with one mon. which makes me question the upgrade in the fist place12:54
gfidentejistr jfrancoa but I see ceph-ansible being re-run with default playbook instead of rolling_update (like it should be for update and upgrade command), is this an issue with "upgrade" command ?12:55
cgoncalvesfultonj, anything we should change in the featureset/scenario to make it work?12:55
fultonjgfidente: yes, that might be a bug for upgrade team to comment on, but even if it triggers the right playbook he'll still have the 1-mon problem. Should cgoncalves change direction? ^12:56
*** mmethot has joined #tripleo12:56
gfidentefultonj I think it might be worth seeing if we get the right playbook triggered without hacking it in the scenario12:56
gfidentefultonj but yeah we won't see upgrade passing anyway12:56
fultonjgfidente: i agree, but that won't help cgoncalves ;)12:56
gfidenteright12:56
dciabrinabishop, don't we need Ife347f754ca9129580c092bb271bacdc032ae14b as well on top? i.e Temp-Url-Key instead of Temp-URL-Key?12:57
*** florianf has quit IRC12:57
beaglesgfidente, fultonj : is this fundamentally that ceph shouldn't be use in a standalone based upgrade job?12:57
gfidentecgoncalves maybe that wasn't obvious, ceph-ansible does not support upgrading a live cluster with a single mon or single osd by design12:57
gfidentebeagles yeah ceph-ansible just won't do it because it is designed around the idea that it needs to keep the cluster up at all times12:57
gfidentebeagles and with 1 instance of mon or osd it can't12:58
abishopdciabrin: just replied to your review comment12:58
gfidentebut what I am saying is that in this specific job, we're not even getting to the point where it will stop and refuse to do the upgrade12:58
beaglesgfidente, ack - so octavia standalone update will have to be cephless12:58
gfidentebecause we're triggering the wrong ceph-ansible playbook12:58
fultonjbeagles: it deploys great though so don't get any fancy ideas12:58
fultonjbut yeah, no single node upgrades12:58
beaglesfultonj, lol12:58
beaglesfultonj, maybe we base our upgrades job on ovn provider job12:59
beaglesfultonj, and make it cephless12:59
gfidentebeagles cgoncalves so I was trying to get the job to the point where the upgrade fails in a controlled manner12:59
gfidentebut we won't see it passing anyway12:59
beaglesack12:59
cgoncalvesbeagles, octavia upgrades + ovn provider? no, please13:01
gfidentefultonj btw we set rolling_update explicitly for external_upgrade_tasks and external_update_tasks in tht13:01
gfidenteso tripleo upgrade not using that seems a bug13:01
fultonjgfidente: seems like13:01
beaglescgoncalves, so yet another scenario that is cephless so we can upgrade?13:02
fultonjcgoncalves: are you going to file that bug anyway13:02
fultonj?13:02
*** rpittau|bbl is now known as rpittau13:02
beaglescgoncalves, the alternative would be to make the non-ovn provider version cephless13:02
fultonjgfidente: yeah, we should probably have a bug13:02
cgoncalvesfultonj, I was told to hold on. I can file one, sure13:03
beagles(so three choices)13:03
cgoncalvesbeagles, if that is what it takes to workaround ceph issues...13:03
gfidentebeagles cgoncalves so any fresh deployment with ceph should work, but upgrade won't13:04
gfidentebeagles cgoncalves and it's not a ceph "issue" per se but a decision made in ceph-ansible13:04
cgoncalvesright. matches with what we are seeing in CI13:04
fultonj"if that is what it takes to workaround ceph issues..." :(13:04
beaglesunderstood13:04
fultonjnot exactaly a ceph issue, it's a 1-node upgrade issue13:05
beaglesyup13:05
gfidentecgoncalves I am probably not best person to discuss why that decision but I believe it is essential from ceph perspective to make sure upgrade does not break the running cluster and it makes little sense support any scenario where they break that promise13:06
jfrancoagfidente: we don't have any ceph enabled upstream job mostly because we couldn't have 3 mon, so we decided to remove ceph from the equation13:07
cgoncalvesagreed. all I am saying is if it won't be fixed or will take time, the workaround on the octavia upgrade job is to disable ceph13:07
gfidentejfrancoa ack but I was thinking it'd still try to trigger the correct playbook ... and fail with 1 node13:07
jfrancoagfidente: plus the standaone-upgrade scenario does not run any ceph related upgrade commands in the tripleo-upgrade role13:07
jfrancoagfidente: that would explain why you don't see any change in the ceph_ansible_playbook ansible parameter13:07
gfidentejfrancoa but is it possible it's still running the standard external_deploy commands?13:08
*** avivgt has joined #tripleo13:08
openstackgerritMerged openstack/tripleo-validations stable/stein: Add a validation to fail if a Ceph dependency is not installed  https://review.opendev.org/70393113:08
jfrancoagfidente: I need to see the standlone-upgrade playbook but I'm quite sure it's not triggering the ceph-upgrade part13:09
gfidentejfrancoa ack and that makes sense13:09
jfrancoagfidente: and the second question is, we can add the step...but having a single mon, won't htat fail??13:09
gfidentejfrancoa but I still see it running the external_deploy* tasks though13:09
gfidentejfrancoa no indeed, I don't think we want to add the external upgrade; I am just asking if "upgrade" is still supposed to trigger the standard external_deploy* tasdks13:10
jfrancoagfidente: external_deploy steps are part of the deployment steps..but the external_upgrade are only triggered when running the openstack overcloud external upgrade command13:11
jfrancoagfidente: and that's what we aren't calling, for sure13:11
cgoncalveshttps://bugs.launchpad.net/tripleo/+bug/186714413:11
openstackLaunchpad bug 1867144 in tripleo "TripleO standalone upgrade job fails when Ceph enabled" [Undecided,New]13:11
fultonjthanks cgoncalves13:11
gfidentejfrancoa ok sorry let me ask with pointers to actual code13:11
*** raildo has joined #tripleo13:11
gfidentejfrancoa we have https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L675 and https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L68213:12
gfidenteas external_update and external_upgrade13:12
fultonjso cgoncalves is going to modify the job to not upgrade ceph and gfidente will talk to upgrades team about bug 186714413:12
openstackbug 1867144 in tripleo "TripleO standalone upgrade job fails when Ceph enabled" [Undecided,New] https://launchpad.net/bugs/186714413:12
dciabrinabishop, oh ok :) do we need to reference the bug in the commit of the two reviews?13:12
gfidentewhat I am asking is, will "upgrade" command also run https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L579 ?13:12
*** psachin has joined #tripleo13:13
yoctozeptohey tripleo friends, kolla train stuck on your build job: https://zuul.opendev.org/t/openstack/build/bf343e1ad1684bfb902fb0703085e92213:13
yoctozeptolooks like this https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L43-L49 is breaking due to lack of python3-devel (while pip3 seemingly present)13:14
jfrancoagfidente: https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L579 will run as part of the deploy steps if I'm not mistaken, I belive deployment_tasks run external_deploy_steps as part of the deployment13:14
yoctozeptothat's how far we got with debugging13:14
jfrancoagfidente: however, the upgrade doesn't trigger external_upgrade_tasks unless you say it explicitly via command13:15
*** amoralej is now known as amoralej|lunch13:15
gfidentejfrancoa yeah on deploy they work, I am not sure if they also run on "upgrade"13:15
abishopdciabrin: I did reference it in the 2nd review13:15
gfidentejfrancoa tripleo upgrade command I mean13:15
dciabrinabishop, ok awesome13:15
jfrancoagfidente: let me have a look at the job logs, do you have it in hand?13:16
gfidentejfrancoa this is the upgrade .sh https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_125/710473/2/check/tripleo-ci-centos-7-scenario010-standalone-upgrade/125e42c/logs/undercloud/home/zuul/standalone-upgrade.sh13:16
gfidentejfrancoa++ thanks for checking1213:17
*** avivgt has quit IRC13:20
gfidentecgoncalves++ thanks for pushing on it13:25
cgoncalvesthanks for the help troubleshooting and reaching out to the Upgrades team!13:28
*** derekh has quit IRC13:29
*** jpich has quit IRC13:29
cgoncalvesas per fultonj's comment on bug #1867144, Ceph will have to be disabled in the CI job. this will require a new scenario file, I guess.13:29
openstackbug 1867144 in tripleo "TripleO standalone upgrade job fails when Ceph enabled" [Undecided,New] https://launchpad.net/bugs/186714413:29
*** jpich has joined #tripleo13:30
*** ayoung has quit IRC13:34
*** ayoung has joined #tripleo13:35
cloudnullweshay|ruck you around? mind joining us in the #openstack-kolla channel? https://zuul.opendev.org/t/openstack/build/bf343e1ad1684bfb902fb0703085e922 has been failing and the kolla folks are considering making our job there non-voting because of it .13:36
*** dtantsur is now known as dtantsur|brb13:36
jfrancoagfidente: cgoncalves: fultonj: commented on the lp bug https://bugs.launchpad.net/tripleo/+bug/1867144/comments/213:38
openstackLaunchpad bug 1867144 in tripleo "TripleO standalone upgrade job fails when Ceph enabled" [Undecided,New]13:38
* fultonj reads13:39
*** avivgt has joined #tripleo13:39
*** skramaja has quit IRC13:40
fultonjthanks jfrancoa . i think gfidente fmount and i need to comment next on ceph's place in standlone_upgrade13:40
* fmount reading/catching up13:41
fmountcloudnull: o/ quick question: I have my deployment (standalone or multinode), I need to check the enabled_services in my overcloud13:43
fmountI see each deployment has global_vars.yaml13:43
fmount(e.g. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d7c/711942/7/check/tripleo-ci-centos-8-containers-multinode/d7c96ed/logs/undercloud/home/zuul/undercloud-ansible-ccgdj54d/global_vars.yaml)13:43
fmountso I think I can use {{ enabled_services }} in tripleo-ansible everywhere, right?13:43
fmountbecause it's already set ^13:44
fmountfultonj: gfidente  ^13:45
cloudnullfmount yes. you should be able to check that global var in your playbook/role within tripleo-ansible13:46
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Pinning pytest-html to <= 2.0.1  https://review.opendev.org/71267913:46
*** amoralej|lunch is now known as amoralej13:46
fmountok great, thanks cloudnull, since they're available, no reason to play w/ /bin/hiera -c ..13:46
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud cell export  https://review.opendev.org/71218113:46
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud config download  https://review.opendev.org/71257813:47
cloudnullfmount ++13:47
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud netenv validate  https://review.opendev.org/71258113:47
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud netenv validate  https://review.opendev.org/71258113:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-heat-templates master: Use jinja raw tag instead of quoted concatenation  https://review.opendev.org/71250413:49
cloudnulli'd probably set that to a default in your role/playbook, using a namespaced var: something like `tripleo_enabled_services` which has a default of  `tripleo_enabled_services: "{{ enabled_services | default([]) }}"` that way you can consume the global and ensure that your role always uses namespaced options.13:49
cloudnull-cc fmount13:49
fultonjcloudnull: those are the enabled_services of the overcloud right?13:50
cloudnullyup, or I think they are ?13:50
cloudnull-cc EmilienM13:50
fultonjcloudnull: because these roles are executed as part of external_deploy_steps they only run on the undercloud13:50
fultonjcloudnull: so hiera was returning what was enabled on the undercloud but we wanted to know what was enabled on the undercloud13:50
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud container prepare  https://review.opendev.org/71198913:51
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud container image tag discover  https://review.opendev.org/71210913:51
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud container image upload  https://review.opendev.org/71212813:51
fultonjit makes sense to me that {{ playbook_dir }} / globals.yml would desribe OC, and not the UC however13:51
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add note about Queens only roles  https://review.opendev.org/71213013:51
fmountcloudnull: ok great, noted it!13:51
fmountfultonj: right, that's the quick fix we were looking for13:52
yoctozeptoweshay|ruck: in case you come, we are now on #openstack-infra with cloudnull and fungi13:52
*** holser has joined #tripleo13:52
fultonjfmount: hopefully13:52
fmounthiera -c of course returned the enabled_services in the UC13:52
mwhahahaweshay|ruck: did https://review.opendev.org/#/c/712571/ work?13:52
fultonjfmount: want to put up an alternative fix for bug/186700413:52
fultonj?13:52
fmountas cloudnull suggested, using something like: tripleo_enabled_services: "{{ enabled_services | default([]) }} can help us solve the issue13:53
fmountyes fultonj, I'll do as soon as possible13:53
*** jdandrea has joined #tripleo13:54
weshay|ruckmwhahaha, /me looks13:54
weshay|ruckmwhahaha, don't think so.. that element is not in the build log13:55
mwhahaha2020-03-11 23:17:28.138 | Copying hooks /home/zuul/src/opendev.org/openstack/tripleo-image-elements/elements/interface-names/install.d/70-clear-net-ifnames13:56
mwhahahayes it is13:56
mwhahaha2020-03-11 23:25:44.368 | dib-run-parts Running /tmp/in_target.d/install.d/70-clear-net-ifnames13:56
mwhahahai guess i'll spin up an env my self later to test13:57
ykarelbut i think ^^ is not yet tested with ovb yet13:57
ykarellast test was with openstack-base element13:57
ykarelhttps://review.opendev.org/#/c/712521/ was tested with ovb patch, and it worked13:58
*** waleedm has quit IRC13:58
mwhahahaso if that works, it's likely the other will work13:59
ykarelsame can be done for https://review.opendev.org/#/c/712571/13:59
mwhahahabut is more correct13:59
ykarelmwhahaha, ok let's test that, so 71- not needed in this?13:59
ykareli mean ur other patch in tripleo-image-element13:59
mwhahahayea13:59
mwhahahait already existed for rhel8 we just need to apply it for centos813:59
*** jistr is now known as jistr|mtg13:59
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/stein: Use config for layers  https://review.opendev.org/71070514:00
ykarelmwhahaha, but u abandoned https://review.opendev.org/#/c/712487/ image-element14:01
* ykarel confused14:01
mwhahahawe don't need that14:01
mwhahahabecause the real solution is to clean net.ifnames14:01
mwhahahawhich is https://review.opendev.org/#/c/712571/14:01
ykarelmwhahaha, ok /me not sure if garbage ens3 will be cleaned up with this14:02
mwhahahait's never created14:02
mwhahahathats why14:02
ykarelmwhahaha, ens3 is in centos8 cloud image14:02
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: WIP use exec when spawning dnsmasq inside sidecar container  https://review.opendev.org/71268514:02
ykarelso at some place it's needs cleanup14:02
mwhahaharight thats why i wanted to know if it worked14:02
*** derekh has joined #tripleo14:03
mwhahahaso if i need both ok, i'll test later today14:03
mwhahahaand i'll restore 71248714:03
ykarelmwhahaha, ack, periodic pipeline not running currently, so we can test that now, ok?14:03
weshay|ruckmwhahaha, I built images w/ the unmerged patch.. and have them in current-tripleo14:04
*** ayoung has quit IRC14:08
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Added tripleo-ci-centos-8-scenario007-multinode-oooq-container  https://review.opendev.org/71229414:10
*** ayoung has joined #tripleo14:12
weshay|ruckmwhahaha, so go w/ https://review.opendev.org/#/c/712521/ ?14:15
mwhahahano14:15
mwhahahahttps://review.opendev.org/#/c/712487/ w/ https://review.opendev.org/#/c/712571/14:16
openstackgerritAlex Schultz proposed openstack/tripleo-common master: Add interface-names to centos8 images  https://review.opendev.org/71257114:16
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Update overcloud image with component repo  https://review.opendev.org/71252414:18
openstackgerritLance Bragstad proposed openstack/tripleo-heat-templates master: Add code to register nodes as ipa-clients  https://review.opendev.org/69829214:21
openstackgerritLance Bragstad proposed openstack/tripleo-heat-templates master: Create a new parameter for the HAProxy external network  https://review.opendev.org/69598814:21
openstackgerritLance Bragstad proposed openstack/tripleo-heat-templates master: WIP Added scale_tasks to handle cleanup on scale down of nodes  https://review.opendev.org/70656414:21
weshay|ruckcloudnull, latest news is that we're not loosing 3rd party now.. they changed the migration plan14:26
*** jfrancoa has quit IRC14:29
*** pkopec has quit IRC14:29
openstackgerritMerged openstack/tripleo-operator-ansible master: Pinning pytest-html to <= 2.0.1  https://review.opendev.org/71267914:29
cloudnullweshay|ruck welp we'll be ready when they change it again :D14:30
weshay|ruckcloudnull, :)14:30
*** abdysn has quit IRC14:31
d0ugalcloudnull: do you know if there are patches for any of the 5 things I commented on here? https://review.opendev.org/#/c/712631/114:36
d0ugalOtherwise we still have 5 mistral uses in tripleoclient14:36
*** holser has quit IRC14:37
mwhahahaweshay|ruck: the centos8 hardened job is bad btw (it's pulling py2 deps)14:39
weshay|ruckmwhahaha, ya.. I'm ignoring that atm14:40
mwhahahak i'll poke at it14:40
*** holser has joined #tripleo14:42
*** jfrancoa has joined #tripleo14:44
*** beagles has quit IRC14:49
openstackgerritFrancesco Pantano proposed openstack/tripleo-ansible master: Get the list of the enabled_services from the overcloud  https://review.opendev.org/71270114:51
*** psachin has quit IRC14:52
cloudnulld0ugal nope14:53
cloudnullmind adding those to our worklist14:54
d0ugalcloudnull: sure, I'll do that14:54
cloudnullthanks!14:54
*** leanderthal has joined #tripleo14:55
cloudnulld0ugal looks like you had a review for undeploy_plan ?14:56
d0ugalcloudnull: oh, was that one of them. I'll recheck them all now.14:57
cloudnulland it seems stevebaker was working on deploy_roles14:58
d0ugalcool14:59
*** sri_ has quit IRC15:08
*** pkopec has joined #tripleo15:10
openstackgerritSorin Sbarnea proposed openstack/tripleo-ci master: Install python headers  https://review.opendev.org/71270615:14
*** holser has quit IRC15:14
*** holser has joined #tripleo15:15
openstackgerritLuke Short proposed openstack/tripleo-common master: Expose --limit, --skip-tags, and --tags in config-download.  https://review.opendev.org/69242015:17
openstackgerritLuke Short proposed openstack/python-tripleoclient master: Expose --limit, --skip-tags, and --tags on the CLI.  https://review.opendev.org/69242115:17
*** Goneri has joined #tripleo15:18
*** rainleander has joined #tripleo15:20
*** beagles has joined #tripleo15:21
Tenguekultails: -^ that's great!15:21
ekultailsTengu: In theory, yes. :-) Per EmilienM 's suggestion, I'm making sure that scale up and scale downs work correctly.15:22
Tenguekultails: might be good to ensure it yeah :)15:23
*** leanderthal has quit IRC15:23
d0ugalcloudnull: so I am feeling stupid, I can't figure out how to create anything in storyboard15:24
d0ugalcloudnull: got it15:25
d0ugalThis UI is basically abusive :)15:25
weshay|rucklolz15:26
weshay|ruck++15:26
ramishrajobs are all red,15:28
ramishrahttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bbd/711872/2/gate/tripleo-ci-centos-8-undercloud-containershttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bbd/711872/2/gate/tripleo-ci-centos-8-undercloud-containers/bbdf18f/logs/undercloud/https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuu15:28
ramishral_opendev_logs_bbd/711872/2/gate/tripleo-ci-centos-8-undercloud-containers/bbdf18f/logs/undercloud/home/zuul/install_packages.sh.loghome/zuul/install_packages.sh.log/bbdf18f/logs/undercloud/home/zuul/install_packages.sh.log15:28
*** holser has quit IRC15:28
ramishrasorry wrong link15:28
ramishrahttps://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bbd/711872/2/gate/tripleo-ci-centos-8-undercloud-containers/bbdf18f/logs/undercloud/home/zuul/install_packages.sh.log15:29
*** holser has joined #tripleo15:29
ykarelamoralej, hit upstream as well ^^15:29
openstackgerritSorin Sbarnea proposed openstack/tripleo-ci master: Consolidate how we determine python virtualenv  https://review.opendev.org/71270615:29
amoralejyeah15:29
amoralejexoected, until mirrors refresh content15:30
amoralejshould be short15:30
*** holser has quit IRC15:30
amoralejlet me check that mirror15:31
*** jistr|mtg is now known as jistr15:33
*** ykarel is now known as ykarel|away15:34
*** holser has joined #tripleo15:35
weshay|ruckseeing this fail in a few jobs FYI15:37
weshay|ruck2020-03-12 15:15:16 |   msg: 'Failed to download packages: Cannot download noarch/python3-cssselect-0.9.2-13.el8.noarch.rpm: All mirrors were tried'15:37
weshay|ruck2020-03-12 15:15:16 |   results: []15:37
*** weshay|ruck changes topic to "CI status: GREENISH | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs"15:37
*** TrevorV has joined #tripleo15:39
weshay|ruckcloudnull, EmilienM some success on centos-8 https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001&result=success15:42
weshay|ruckmwhahaha, ^15:42
* weshay|ruck notes.. I'm carrying a patch in the image used there15:42
*** ed_b has quit IRC15:42
weshay|ruckto nuke ens315:43
EmilienMwow nice15:44
* EmilienM in mtg15:44
*** rlandy is now known as rlandy|afk15:46
*** rainleander is now known as leanderthal15:46
openstackgerritMerged openstack/tripleo-ci master: Make scenario010 voting  https://review.opendev.org/71218915:48
openstackgerritMerged openstack/tripleo-quickstart-extras master: Bumps up default Ceph container images  https://review.opendev.org/70665115:48
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: Check Ceph*Key value format and halt on error  https://review.opendev.org/71230515:48
openstackgerritMerged openstack/tripleo-ansible master: Fix os_baremetal_node_info module documentation  https://review.opendev.org/71220615:53
openstackgerritJohn Eckersberg proposed openstack/tripleo-heat-templates stable/train: rabbitmq: Open ports 25673-25683 for CLI tools  https://review.opendev.org/71208915:53
*** dtantsur|brb is now known as dtantsur15:56
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Check Ceph*Key value format and halt on error  https://review.opendev.org/71272115:56
*** bogdando has quit IRC15:57
d0ugalcloudnull: so I added to the huge epic, but that might have been a mistake16:00
d0ugalcloudnull: I am struggling to load that page because it feezes my browser. it so huge16:01
d0ugalI think I might have added one twice but I can't delete it now16:01
cloudnullYou can improve the performance of the page by turning off all those comments16:02
cloudnullunder your profile > preferences16:03
d0ugalaha16:03
*** ed_b has joined #tripleo16:03
cloudnullI just load the tasks.16:03
d0ugalcloudnull: yeah, that is much better16:04
cloudnullIt seems storyboard is constantly trying to load more comments.16:04
*** cylopez has quit IRC16:05
d0ugalit knows how much everyone loves comments16:05
openstackgerritRajini Karthik proposed openstack/puppet-tripleo master: Support for PowerMax Volume Config  https://review.opendev.org/71218416:07
openstackgerritDamien Ciabrini proposed openstack/tripleo-heat-templates master: HA: drop spurious mysql user on stack deploy  https://review.opendev.org/71272616:08
*** leanderthal has quit IRC16:09
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud container image upload  https://review.opendev.org/71212816:19
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add note about Queens only roles  https://review.opendev.org/71213016:19
openstackgerritFrancesco Pantano proposed openstack/tripleo-ansible master: Get the list of enabled_services on the overcloud  https://review.opendev.org/71270116:22
weshay|ruckgate keeps failing on 2020-03-12 15:15:16 |   msg: 'Failed to download packages: Cannot download noarch/python3-cssselect-0.9.2-13.el8.noarch.rpm: All mirrors were tried'16:22
mwhahahasee discussion around this in #rdo i think16:23
mwhahahaor was that in here earlier16:24
*** TrevorV has quit IRC16:27
*** TrevorV has joined #tripleo16:28
*** dpawlik has quit IRC16:28
*** dpawlik has joined #tripleo16:28
*** Blinkiz has quit IRC16:30
*** Blinkiz has joined #tripleo16:31
*** TrevorV has quit IRC16:31
*** TrevorV has joined #tripleo16:32
*** jcapitao is now known as jcapitao_afk16:40
*** tesseract has quit IRC16:40
*** fultonj has quit IRC16:44
*** udesale_ has quit IRC16:49
*** fultonj has joined #tripleo16:52
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates master: Support for PowerMax Cinder Backend  https://review.opendev.org/71253316:53
*** lucasagomes has quit IRC16:56
*** sshnaidm is now known as sshnaidm|afk17:07
*** jcapitao_afk is now known as jcapitao17:12
*** rlandy|afk is now known as rlandy17:19
openstackgerritSorin Sbarnea proposed openstack/ansible-role-collect-logs master: Make collection commands configurable  https://review.opendev.org/70562317:20
*** marios is now known as marios|out17:27
*** ayoung has quit IRC17:27
*** rpittau is now known as rpittau|afk17:27
*** ahosam has quit IRC17:28
openstackgerritLance Bragstad proposed openstack/tripleo-ansible master: Add role to add ipa services for tls-everywhere  https://review.opendev.org/69190617:28
openstackgerritLance Bragstad proposed openstack/tripleo-ansible master: Add code to deregister nodes and services from freeipa  https://review.opendev.org/70022617:28
openstackgerritLance Bragstad proposed openstack/tripleo-ansible master: Add undercloud-ipa-install to playbooks  https://review.opendev.org/70958917:28
*** jpich has quit IRC17:28
*** ahosam has joined #tripleo17:29
*** weshay|ruck changes topic to "CI status: RED, 1867177 | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs"17:29
*** ayoung has joined #tripleo17:30
*** dtantsur is now known as dtantsur|afk17:34
*** evrardjp has quit IRC17:35
*** evrardjp has joined #tripleo17:36
*** marios|out has quit IRC17:41
*** kozhukalov has joined #tripleo17:46
openstackgerritMerged openstack/tripleo-ansible master: Fix name of baremetal node info module in docs  https://review.opendev.org/71229117:50
*** derekh has quit IRC18:01
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Add needed parameters in containers-prepare-parameters for FFWD2.  https://review.opendev.org/71275418:07
*** Trevor_V has joined #tripleo18:07
*** TrevorV has quit IRC18:10
openstackgerritDamien Ciabrini proposed openstack/puppet-tripleo master: Make all mysql root users managed during stack creation/update  https://review.opendev.org/71275718:11
fultonjhas anyone seen container-puppet-zaqar fail an undercloud deploy on cent8 ? http://paste.openstack.org/show/790623/18:14
fultonjEmilienM: ^ ?18:14
mwhahahayes18:14
mwhahahaif you have py2 containers on centos818:14
mwhahahait blows up18:14
mwhahahadon't use py2 containers18:14
mwhahahawe had a conversation around this yesterday afternoon18:14
*** holser has quit IRC18:15
mwhahahaconversation starting around http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2020-03-11.log.html#t2020-03-11T16:04:2718:16
fultonjmwhahaha: ah, ok. i'll read that. thank you18:16
mwhahahathe tl;dr is for py3 containers use the md5 as the tag from https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md518:16
* weshay|ruck flips tables18:16
* mwhahaha blames weshay|ruck 18:16
* weshay|ruck blames new DLRN18:17
* weshay|ruck flips cup18:17
weshay|ruckoh fun18:17
fultonjso in containers-prepare-parameter.yaml ContainerImagePrepare / tag: 3621159be13b41f8ead2e873b357f4a5 ?18:17
weshay|ruckbfournie, howdy..18:17
weshay|ruckbfournie, holding two new nodes for introspection / node import issues.. not sure yet which18:18
weshay|ruckmwhahaha, didn't realize so many folks are doing this by hand18:18
weshay|ruckfultonj, look at a c8 job and just steal the file from that18:18
mwhahahayou mean trying to actually devel on the thing we're moving to?18:18
weshay|ruckor at least the hash18:18
mwhahahaA NOVEL CONCEPT18:18
weshay|ruckmeh18:19
mwhahahaupstream, pffffft18:19
weshay|ruckmwhahaha, ok.. let me restate that.. I didn't realize one couldn't look at ci logs18:19
weshay|ruckpfffft18:19
mwhahahawhy would you look at the ci logs18:19
weshay|ruckto get the hash18:19
mwhahahawhy would you do this18:19
weshay|rucklolz18:19
mwhahahano seriously18:19
weshay|ruckso you're shit would work18:20
mwhahahawhy would anyone think of CI as a source of info18:20
mwhahahait shouldn't require CI18:20
*** jcapitao has quit IRC18:20
weshay|ruckI use the term shit in a loving way18:20
weshay|ruckok.. so look at DLNR18:20
weshay|ruckDLRN18:20
weshay|ruckor look at the recent tags in docker.io18:20
mwhahahalike i get your point, but i think it's an incorrect assumption to think folks will look to CI for these types of things18:20
weshay|ruckok..18:20
weshay|ruckfair18:21
mwhahaha we spent so much effort so that folks shouldn't have to do this18:21
mwhahahabut c8 doesn't use any of those patterns yet18:21
weshay|ruckI shall send an email three people will read18:21
mwhahahapffft 318:21
mwhahahayou're overestimating18:21
weshay|ruckwhat patterns18:21
openstackgerritDamien Ciabrini proposed openstack/puppet-tripleo master: Make all mysql root users managed during stack creation/update  https://review.opendev.org/71275718:21
weshay|ruckthere's the disconnect18:21
mwhahahatags are autodiscovered18:21
mwhahahatripleo-repos magically finds the right content18:21
weshay|ruckah ya.. so if we can merge a patch.. we'll be there.. or close18:21
mwhahahathe problem is that we mix py2 and py318:21
*** pkopec has quit IRC18:21
weshay|ruckactually.. that needs love18:22
weshay|ruckk18:22
mwhahahawe might have thought about not sharing tripleomaster namespace for centos818:22
mwhahahaor using centos8- int he container names for now18:22
weshay|ruckmwhahaha, WE DID18:22
weshay|ruckand it was a bad idea18:22
weshay|ruck:)18:23
* mwhahaha would like to see the receipts on that one18:23
weshay|rucklolz18:23
weshay|ruckmwhahaha, moonlights as a tax accountant18:23
mwhahahaANYWAY, i'm off to mess with centos8 overcloud image builds18:23
weshay|ruckcurrent-tripleo tags are coming18:24
mwhahahaand will also hit this issue shortly i'm certain18:24
weshay|ruckit's not my fault... that we're18:24
weshay|ruck1. changing how dlrn works18:24
weshay|ruck2. moving infra18:24
weshay|ruck3. need to be on centos-818:24
weshay|ruck4. and unpin 2-3 months of patches18:24
weshay|ruckat the same time18:24
weshay|ruckthank you very much :)18:24
mwhahahayou need more work18:25
weshay|rucklolz18:25
weshay|ruckmwhahaha, apparently we'll have plenty18:25
weshay|ruckthank you dbecker18:25
*** ahosam has quit IRC18:26
weshay|ruckmwhahaha, retagging the current set w/ current-tripleo18:34
weshay|ruckmy bad18:34
weshay|ruckwhy didn't just ask for that.. knuckle head18:35
weshay|ruck:)18:35
mwhahahai did?18:35
mwhahahayesterday?18:35
weshay|ruckmaybe I'm just slightly stooopid18:36
weshay|ruck8-ball says.. likely18:36
weshay|ruckrlandy, bfournie avail to look18:37
weshay|ruckrlandy, let us know when infra holds a node18:37
weshay|ruckbfournie, we'll need ur pubkey again18:37
rlandyweshay|ruck: yeah - we don't have the nodes yet18:37
rlandybut we may get logs soon18:37
bfournierlandy, weshay|ruck: k, thanks18:38
rlandybfournie: will ping you when we have something to look at - thanks18:38
bfourniestevebaker: ^ maybe related to that provision delay18:38
openstackgerritSorin Sbarnea proposed openstack/tripleo-ci master: Consolidate how we determine python virtualenv  https://review.opendev.org/71270618:39
* fultonj reads weshay|ruck's email18:39
*** gfidente is now known as gfidente|afk18:41
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Drop resource registry override in ip-from-pool  https://review.opendev.org/71153718:44
openstackgerritBob Fournier proposed openstack/os-net-config stable/train: Enable new routing rules on the fly without restarting interfaces  https://review.opendev.org/71276718:50
*** jpena is now known as jpena|off18:51
*** Trevor_V has quit IRC18:51
openstackgerritDamien Ciabrini proposed openstack/puppet-tripleo master: Support for mariadb's ed25519 authentication  https://review.opendev.org/71276919:00
*** ekultails has quit IRC19:01
*** dsneddon has joined #tripleo19:02
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Use exec when spawning dnsmasq inside sidecar container  https://review.opendev.org/71268519:05
*** gfidente|afk has quit IRC19:05
*** kozhukalov has quit IRC19:13
*** bandini has quit IRC19:14
*** amoralej is now known as amoralej|off19:24
openstackgerritMerged openstack/python-tripleoclient master: Add option to control ironic network interfaces  https://review.opendev.org/70010319:28
*** jfrancoa has quit IRC19:36
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Add role_specific tag to missing role-specific parameters  https://review.opendev.org/71202719:38
openstackgerritMerged openstack/tripleo-ci master: Remove rhel_image_source workaround  https://review.opendev.org/71224819:38
openstackgerritMonty Taylor proposed openstack/diskimage-builder master: Remove hacking from requirements  https://review.opendev.org/71277819:45
openstackgerritMonty Taylor proposed openstack/diskimage-builder master: Remove hacking from requirements  https://review.opendev.org/71277819:46
*** ayoung has quit IRC19:48
*** jfrancoa has joined #tripleo19:51
*** ayoung has joined #tripleo19:51
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud container image prepare  https://review.opendev.org/71198919:52
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud container image tag discover  https://review.opendev.org/71210919:52
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud container image upload  https://review.opendev.org/71212819:52
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add note about Queens only roles  https://review.opendev.org/71213019:52
openstackgerritMonty Taylor proposed openstack/diskimage-builder master: Remove hacking from requirements  https://review.opendev.org/71277819:54
*** ekultails has joined #tripleo19:58
EmilienMweshay|ruck: nice to see tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 green19:59
EmilienMwe're close to being able to switch to podman by default19:59
weshay|ruckEmilienM, where? it's consistent :)19:59
weshay|ruck:)19:59
EmilienMand remove any docker19:59
weshay|ruckit's inconsistent19:59
EmilienMoh it's inconsistent now?19:59
EmilienMwe're close anyway19:59
weshay|rucksorry.. still hitting some issues20:00
EmilienMI'll list and rebase all the patches which remove docker finally20:00
EmilienMthere are a few20:00
openstackgerritMerged openstack/os-net-config stable/stein: We need to use the MAC instead of PCI Address for Mellanox cards  https://review.opendev.org/71122320:02
*** jamesdenton has quit IRC20:03
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Remove skydive  https://review.opendev.org/71278320:03
mwhahahaweshay|ruck: -^ that's the rest of the cleanup for it20:03
*** jamesdenton has joined #tripleo20:04
openstackgerritEmilien Macchi proposed openstack/tripleo-puppet-elements master: Drop docker packages from overcloud-base  https://review.opendev.org/63644420:04
EmilienMdciabrin: do you think https://review.opendev.org/#/c/359060/ is something we'll still consider?20:05
dciabrinEmilienM, i think is has value yes, although i don't see it happening with podman on centos. banidni probably has a stronger opinion20:07
dciabrin(on centos 7 i mean)20:07
mwhahahawe're murdering centos7 as fast as we can20:08
weshay|ruckmwhahaha, thanks20:08
EmilienMdciabrin: yes what alex said20:10
EmilienMwhich is why i'ma sking20:10
EmilienMi'm trying to reduce the gap between downstream & upstream20:11
dciabrinEmilienM, then yes i think this makes even more sense20:11
EmilienMand HA is a gap20:11
dciabrin+1 to that20:11
EmilienMk20:11
stevebakermorning20:12
weshay|ruckrlandy, bfournie stevebaker let's see if we can hold a node again20:16
stevebakerweshay|ruck: I'm ready20:16
weshay|ruck:)20:16
weshay|ruckwas going to appologize for greeting you w/ work20:17
weshay|ruckgood morning!20:17
rlandyweshay|ruck: ok - we can abandon and rekick https://review.rdoproject.org/r/#/c/25865/20:17
stevebakerweshay|ruck: as an aside, how do you hold a node? In my personal zuul I do a server lock, but it still un-attaches the floating-ip20:17
*** Goneri has quit IRC20:18
weshay|ruckmeh.. missed the window.. we'll need to rekick20:19
*** pcaruana has quit IRC20:20
openstackgerritMerged openstack/tripleo-heat-templates master: Don't add IpList for disabled networks  https://review.opendev.org/67935420:21
*** rh-jelabarre has quit IRC20:22
*** rh-jelabarre has joined #tripleo20:29
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Fix TenantInterfaceDefaultRoute in net-env-v6  https://review.opendev.org/71278920:29
EmilienMweshay|ruck: do you know which ci/environments/scenario***-multinode-containers.yaml are still being used?20:32
EmilienMcheck/gate/promotion anywhere20:32
EmilienMI thought we could remove some of them which are unused20:32
weshay|rucksec20:32
EmilienM(since the transition to standalone)20:32
*** alexmcleod has quit IRC20:33
*** ansmith_ has quit IRC20:33
weshay|ruckEmilienM, I asked carlos and he still wanted them20:33
weshay|ruckfor 7 and 1020:33
EmilienMjust 7 and 10?20:34
EmilienMwe can trash the others?20:34
weshay|ruckya..20:35
weshay|ruckbut they should be gone20:35
weshay|ruckwhat do you see?20:35
mwhahahaclean all the things20:35
EmilienMweshay|ruck: they aren't gone20:35
EmilienMlet me propose a patch20:35
cgoncalvesbeagles said scenario010 multinode would still be useful20:36
weshay|ruckaye20:36
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Switch to Podman by default  https://review.opendev.org/63644120:38
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Switch to Podman by default  https://review.opendev.org/63644120:39
EmilienMweshay|ruck: wake me up when there is NO MORE centos7 job deploying tripleo20:40
EmilienMwe'll land https://review.opendev.org/#/c/636441/20:40
EmilienMand move on20:40
mwhahahawe were only keeping it for the ovb jobs20:40
mwhahahaif there are no more centos7 ovb jobs we can switch20:41
EmilienMyes20:41
mwhahahaoh there was 1 scenario that was still using pacemaker tho20:41
EmilienMyeah which is why I asked wes how close are we to have consistent centos8 ovb jobs passing20:41
mwhahahaso we need that one too20:41
EmilienMwasn't it 4?20:41
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: ci: remove unused multinode scenarios  https://review.opendev.org/71279120:42
EmilienMweshay|ruck: ^20:42
mwhahahai don't recall20:42
mwhahahagrep for pacemaker i guess20:42
EmilienMscenario004-standalone.yaml:  ContainerCli: docker20:42
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: ci: remove unused multinode scenarios  https://review.opendev.org/71279120:43
weshay|ruckOH20:43
EmilienMcgoncalves: we can keep scenario010 multinode but it'll break. It's using docker and we are switching to podman byd efault20:43
weshay|ruckI thought you meant in zuul20:43
EmilienMyeah i'm cleaning up stuff people forgot to clean :)20:43
EmilienMless yamls to maintain20:44
cgoncalvesEmilienM, we could either fix it soon or delete it now and re-add it later. just hoping it does not impact stable branches where scn010 multinode is the only option we have20:46
EmilienMcgoncalves: I'm not touching stable branches atm20:46
EmilienMI'm cleaning master20:46
EmilienMand won't backport20:46
cgoncalvesok, that's good.20:46
EmilienMcloudnull: we'll need https://review.opendev.org/#/c/712523/ I believe20:47
cloudnullI agree20:47
cgoncalvesI don't intend to rely on multinode on master. I guess it is up to beagles20:47
cgoncalvess/rely/work/20:48
EmilienMweshay|ruck: mh I could even remove the 3node templates20:48
cloudnullEmilienM these are the server side calls we know about https://storyboard.openstack.org/#!/worklist/845 if there are any more it'd be great to get those on that worklist20:48
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: ci: remove unused multinode scenarios  https://review.opendev.org/71279120:48
weshay|ruckEmilienM, aye.. 3 node was a silly idea20:50
EmilienMweshay|ruck: sorry for the thousands of pings but :20:50
EmilienMlook at zuul / 71279120:50
weshay|ruckyes.. I'll marry you20:50
weshay|ruck\0/20:50
EmilienMi only remove ci/envs/ stuffs and see all the jobs that are running20:50
EmilienMwe should use find a way to skip these jobs if files not touched in ci/environments etc20:50
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Add needed parameters in containers-prepare-parameters for FFWD2.  https://review.opendev.org/71275420:51
weshay|ruckEmilienM, agree20:51
weshay|ruckI have to run to the bank20:51
weshay|ruckbrb20:51
*** jfrancoa has quit IRC20:51
EmilienMI see tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 is running on check still, i guess we're waiting for tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 to stabilize?20:52
EmilienMweshay|ruck: ^ when you're done with your bank20:52
EmilienMmwhahaha: tripleo-ci-centos-7-containers-undercloud-minion needs to be converted as well20:53
EmilienMis there a patch already?20:53
EmilienMah yeah20:53
EmilienMmwhahaha: I don't remember, is there a reason not to run the minion job on train?20:56
openstackgerritEmilien Macchi proposed openstack/tripleo-ci master: Remove tripleo-ci-centos-7-containers-undercloud-minion  https://review.opendev.org/71279320:57
openstackgerritEmilien Macchi proposed openstack/tripleo-operator-ansible master: Convert jobs to centos8  https://review.opendev.org/71279520:58
mwhahahaNo reason not to run it. Just don't think it was updated20:58
EmilienMmwhahaha: i'm removing the job. If you think we could run it on stable/train (which isn't the case now); feel free to -1 and we'll run the job in that version20:59
EmilienMoh it seems tripleo-ci-centos-7-containers-undercloud-minion is supposed to run on tripleoclient train21:00
mwhahahaI can run on train now21:00
EmilienMok21:00
EmilienMlet me undo my change21:00
openstackgerritEmilien Macchi proposed openstack/tripleo-ci master: Only run tripleo-ci-centos-7-containers-undercloud-minion on train  https://review.opendev.org/71279321:02
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/train: DNM Test tripleo-ci-centos-7-containers-undercloud-minion on train  https://review.opendev.org/71279621:03
*** ansmith_ has joined #tripleo21:06
*** ayoung has quit IRC21:07
openstackgerritEmilien Macchi proposed openstack/tripleo-ci master: Only run tripleo-ci-centos-7-containers-undercloud-minion on train  https://review.opendev.org/71279321:07
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/train: DNM Test tripleo-ci-centos-7-containers-undercloud-minion on train  https://review.opendev.org/71279621:08
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/train: DNM Test tripleo-ci-centos-7-containers-undercloud-minion on train  https://review.opendev.org/71279621:10
*** ayoung has joined #tripleo21:10
openstackgerritMonty Taylor proposed openstack/diskimage-builder master: Remove hacking from requirements  https://review.opendev.org/71277821:12
weshay|ruckEmilienM, why is minion train only?21:19
weshay|ruckah I see nvermind21:19
openstackgerritMonty Taylor proposed openstack/diskimage-builder master: Remove hacking from requirements  https://review.opendev.org/71277821:21
*** rh-jelabarre has quit IRC21:26
*** rh-jelabarre has joined #tripleo21:27
mwhahahaseems to be a bug in diskimage-builder trying to pull in PyYAML21:28
mwhahahaweshay|ruck: were we pinning diskimage builder with everything else?21:31
weshay|ruckmwhahaha, GO ASK IN RDO :) /me flips tables21:32
*** rlandy is now known as rlandy|afk21:32
mwhahahai'm asking you because image building is broken on centos8 which seems odd since it was passing in ci?21:32
weshay|ruckmwhahaha, https://review.rdoproject.org/r/#/c/25612/19/tags/ussuri-uc.yml21:32
mwhahahathat would bea  no21:33
weshay|ruckyou see it broken locally21:33
weshay|ruck?21:33
cloudnullI ran into broken image building today on c8 as well.21:33
weshay|ruckhttp://zuul.openstack.org/builds?job_name=tripleo-buildimage-overcloud-full-centos-821:34
cloudnullI ended up downloading images from elsewhere, get my env up.21:34
weshay|ruckcloudnull, two diff things21:34
weshay|ruckis the image build broken? or you couldn't pull an image21:34
cloudnullImage build.21:34
cloudnullOvercloud image build21:35
weshay|ruckhttp://zuul.openstack.org/builds?job_name=tripleo-buildimage-overcloud-full-centos-821:35
mwhahahayea it's broken21:35
mwhahahai'm trying to test my build package21:35
mwhahahai must be missing something21:35
cloudnull^ was doing something similar. Dib debug didn’t provide much useful either.21:36
mwhahahaah i'm not including the python3 yaml21:36
mwhahahabut when i did that ig ot a different failure21:36
mwhahahathough at the same time we should probably combine the centos8/py3 files21:36
mwhahahabecause we don't need a seperate one anymore21:36
cloudnull++21:36
cloudnullit took me longer than I’d like to admit to realize I needed both files.21:37
mwhahaha2020-03-12 21:37:30.821 | diskimage_builder.element_dependencies.AlreadyProvidedException: pip-and-virtualenv: already provided by ['override-pip-and-virtualenv']21:37
mwhahahais what i get if i include both21:37
mwhahahameh we're papering over this in CI again21:38
mwhahahayou use overcloud-images.yaml OR overcloud-images-python3.yaml (which seems wrong)21:38
* mwhahaha will fix later21:39
mwhahahahttps://github.com/openstack/tripleo-ci/blob/master/roles/oooci-build-images/templates/build-images.sh.j2#L20-L2121:39
* mwhahaha eyeballs people21:39
cloudnull?!?21:39
*** alexmcleod has joined #tripleo21:39
mwhahahaif you look in the previous incarnation of the roles we didn't do this21:40
mwhahahaANYWAY21:41
* mwhahaha goes back to pushing bits around21:41
weshay|ruckmwhahaha, don't think we should have hardened in check anyway21:42
mwhahahathat's for the harden job which doesn't work at the moment21:42
mwhahahaprobably for a similar issue21:42
*** abishop has quit IRC21:42
weshay|ruckya21:42
cloudnullI tried copying that from a successful job. But I could make it go https://logserver.rdoproject.org/48/712248/1/openstack-check/tripleo-rhel-8-buildimage-overcloud-full/237ba64/build_images.sh21:43
weshay|ruckcloudnull, what's the issue?21:43
mwhahahathat's curently running for me, so we'll see if it fails21:43
weshay|ruckcloudnull, why not use the already built images21:43
weshay|ruckk21:44
cloudnullThat’s what I did to get past.21:44
mwhahahabecause sometimes we don't like your artifacts21:44
mwhahahaSO TAKE THAT21:44
weshay|ruckthat's it.. let's ristle21:44
mwhahaha2020-03-12 21:44:30.046 | Error: Unable to find a match: pacemaker-remote pacemaker pcs21:44
weshay|ruckthat's my best western accent on irc21:44
cloudnullI can rebuild and post the logs.21:44
*** xek has quit IRC21:45
weshay|ruckhttp://images.rdoproject.org/centos8/master/rdo_trunk/21:45
cloudnullmwhahaha yup I saw that.21:45
cloudnullAnd lots of other one off Error issues.21:45
mwhahahasince that's shipped in a centos repo it hink you have to include /etc/yum.repos.d/* for DIB_YUM_REPO_CONF21:46
cloudnullThough I’m not sure if any of those error messages are fatal ?21:46
mwhahahathe errors are always misleading21:46
mwhahahawhich is the fun part of DIB21:46
weshay|ruckyup21:46
cloudnullI was including all my repos.21:46
* cloudnull goes to recreate the failure. 21:47
*** raildo_ has joined #tripleo21:49
*** raildo has quit IRC21:49
openstackgerritHarald Jensås proposed openstack/python-tripleoclient stable/train: Add option to control ironic network interfaces  https://review.opendev.org/71280621:54
*** amoralej|off is now known as amoralej22:00
*** rcernin has joined #tripleo22:04
*** rcernin has quit IRC22:04
*** rcernin has joined #tripleo22:05
*** jbadiapa has quit IRC22:06
mwhahahalooks like i was able to build22:08
mwhahahathe ironic bits are building now22:09
*** jbadiapa has joined #tripleo22:09
*** ekultails has quit IRC22:13
*** abishop has joined #tripleo22:21
mwhahahaweshay|ruck: i tested https://review.opendev.org/#/c/712571/ and https://review.opendev.org/#/c/712487/ and they are ok22:22
*** abishop has quit IRC22:23
*** abishop has joined #tripleo22:23
weshay|ruckmwhahaha, aye.. ur job failed on a known bug.. caused by the rdo team 2020-03-12 14:53:02.630 |   Cannot download noarch/python3-cssselect-0.9.2-13.el8.noarch.rpm: All mirrors were tried22:25
weshay|ruck+2 on both.. recheck22:25
mwhahahayea22:25
*** abishop_ has joined #tripleo22:26
weshay|ruckboth jobs did22:26
weshay|ruckcloudnull, is that what you hit? 2020-03-12 15:23:51.354 |   Cannot download noarch/python3-cssselect-0.9.2-13.el8.noarch.rpm: All mirrors were tried22:26
mwhahahathat was a while ago so hopefully recheck comes back22:26
cloudnullI dont have the old log to look at, but I will in just a few22:27
* cloudnull kicking a new env 22:27
*** abishop has quit IRC22:27
weshay|ruckI cix'd the issue22:28
*** slaweq has quit IRC22:28
openstackgerritSteve Baker proposed openstack/tripleo-quickstart-extras master: Make reproducer script centos-8 friendly  https://review.opendev.org/70697722:32
*** jbadiapa has quit IRC22:37
openstackgerritSteve Baker proposed openstack/tripleo-quickstart master: Handle pip being installed as /usr/bin/pip3  https://review.opendev.org/70697922:40
ianw2020-03-12 21:43:34.774 |  python3-cssselect                 noarch0.9.2-13.el8                                   delorean-master-build-deps      40 k22:46
ianwdo we know about this being corrupt, seemingly?22:46
ianwok ... yeah scrollback sorry22:47
*** rlandy|afk is now known as rlandy22:55
*** tkajinam has joined #tripleo22:56
*** ayoung has quit IRC23:07
*** Goneri has joined #tripleo23:08
*** ayoung has joined #tripleo23:09
EmilienMmwhahaha: tripleo-ci-centos-7-containers-undercloud-minion passed on train23:20
EmilienMgood23:20
EmilienMmwhahaha: https://review.opendev.org/#/c/712793/ plz23:20
EmilienMcloudnull: 2020-03-12 22:28:35 | Error: The following files were not found: /home/zuul/overcloud-baremetal-deployed.yaml23:22
EmilienMyou saw that?23:23
EmilienMhttps://logserver.rdoproject.org/41/636441/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/3201294/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz23:23
EmilienMwondering if I missed something23:23
cloudnullyup i have seen that23:23
weshay|ruckYUP23:23
weshay|ruckYUP23:23
weshay|ruckYUP23:23
weshay|ruckYUP23:23
weshay|ruckYUP23:23
cloudnullseems intermittent , though i have no idea why23:23
EmilienMweshay|ruck: unping23:23
weshay|ruckYUP23:23
weshay|ruckYUP23:23
weshay|ruckYUP23:23
cloudnullhahahaha23:23
EmilienMcan someone mute him23:23
weshay|ruckbut I talked to stevebaker23:23
weshay|ruckand there's a patch out23:23
EmilienMso we're safe23:23
weshay|ruckEmilienM, cloudnull follow the deps here https://review.opendev.org/#/c/711364/23:24
EmilienMweshay|ruck: follow the dep yourself23:24
EmilienMboom23:24
EmilienMYUP i'm gone now baie23:24
openstackgerritMerged openstack/puppet-tripleo stable/stein: Make sure we create stonith resources before stonith levels  https://review.opendev.org/71256823:26
*** EmilienM was kicked by weshay|ruck (EmilienM)23:26
weshay|ruck:)23:26
weshay|rucklolz23:26
*** EmilienM has joined #tripleo23:26
*** ChanServ sets mode: +v EmilienM23:26
* weshay|ruck runs23:26
weshay|ruckcloudnull, ping23:27
cloudnullpong23:27
weshay|ruckcan join a quick chat or way too late23:27
weshay|ruckcloudnull, the logging here23:28
weshay|ruckon 2020-03-12 05:02:35,391 p=28455 u=root |  TASK [tripleo-modify-image : Run yum_update.sh] ********************************23:28
weshay|ruckseems to be boosted23:28
weshay|ruckhttps://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-cloudops-master/144cb40/logs/undercloud/home/zuul/ansible.log.txt.gz23:28
weshay|ruckcheck that line..23:28
weshay|ruckwe should see a yum transaction there I think23:28
weshay|ruckcan go through it tomorrow23:28
weshay|ruck  stdout_lines: <omitted>23:28
weshay|ruckwhy is that ommitted23:29
*** amoralej is now known as amoralej|off23:29
cloudnull¯\_(ツ)_/¯23:29
* cloudnull helper 23:29
weshay|rucklolz23:29
*** ayoung has quit IRC23:43
*** ayoung has joined #tripleo23:43
*** threestrands has joined #tripleo23:50
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Trigger ceph-ansible on ceph_systemd tag too  https://review.opendev.org/71191323:56
*** ayoung has quit IRC23:57
*** ayoung has joined #tripleo23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!