Tuesday, 2020-03-03

weshay|ruckrlandy, think that may have worked00:08
rlandyweshay|ruck: maybe .., it's still going00:09
rlandyweshay|ruck: I think the update job may die on undercloud install though :(00:10
weshay|ruckya00:10
rlandywe can update the hackmd to say updates is started though00:10
weshay|ruckrlandy, fyi.. fix for panda's change to emit https://review.opendev.org/#/c/710869/00:11
rlandyah00:11
*** jmasud has quit IRC00:53
*** owalsh has quit IRC00:54
*** jmasud has joined #oooq00:55
*** owalsh has joined #oooq01:00
*** yolanda has quit IRC01:15
*** Goneri has quit IRC02:41
rlandy2020-03-03 02:05:48.262888 | primary | async task did not complete within the requested time - 7200s02:57
rlandyjust hit OVB02:57
rlandyweshay|ruck: still around?03:03
*** dsneddon has quit IRC03:15
*** rlandy has quit IRC03:15
*** dsneddon has joined #oooq03:16
*** apetrich has quit IRC03:17
*** skramaja has joined #oooq03:57
*** raukadah is now known as chandankumar04:04
*** skramaja has quit IRC04:29
*** skramaja has joined #oooq04:29
*** jmasud has quit IRC04:53
*** jmasud has joined #oooq04:54
*** ratailor has joined #oooq04:57
*** ykarel|away is now known as ykarel04:58
*** ratailor has quit IRC04:59
*** ratailor has joined #oooq04:59
*** saneax has joined #oooq05:35
*** marios|ruck has joined #oooq06:25
*** sanjayu_ has joined #oooq06:29
*** saneax has quit IRC06:31
*** dsneddon has quit IRC06:33
*** jfrancoa has joined #oooq06:51
*** sanjayu__ has joined #oooq06:53
*** chem has quit IRC06:55
*** sanjayu_ has quit IRC06:55
*** yolanda has joined #oooq07:21
*** ykarel is now known as ykarel|lunch07:39
*** tosky has joined #oooq08:20
*** jbadiapa has joined #oooq08:21
*** sanjayu_ has joined #oooq08:30
*** tesseract has joined #oooq08:30
*** sanjayu__ has quit IRC08:32
*** jpena|off is now known as jpena08:41
*** sshnaidm|pto has quit IRC08:45
*** jmasud has quit IRC08:55
*** jmasud has joined #oooq08:56
*** sshnaidm|pto has joined #oooq08:59
*** apetrich has joined #oooq09:00
*** ccamacho has quit IRC09:02
*** udesale has joined #oooq09:14
*** sshnaidm|pto is now known as sshnaidm09:17
*** tesseract has quit IRC09:25
*** tesseract has joined #oooq09:26
*** bogdando has joined #oooq09:38
*** holser has joined #oooq09:39
*** holser has quit IRC09:40
*** holser has joined #oooq09:41
*** derekh has joined #oooq09:43
*** dtantsur|afk is now known as dtantsur10:03
*** ChanServ sets mode: +o panda10:10
pandaDocs: https://docs.openstack.org/tripleo-quickstart/latest/ || 3nd Mar promoter patch to review https://review.rdoproject.org/r/24715 || next promoter patch in the chain https://review.rdoproject.org/r/2471610:11
*** panda changes topic to "Docs: https://docs.openstack.org/tripleo-quickstart/latest/ || 3nd Mar promoter patch to review https://review.rdoproject.org/r/24715 || next promoter patch in the chain https://review.rdoproject.org/r/24716"10:11
pandagrrrr10:11
*** panda sets mode: -o panda10:11
*** ykarel|lunch is now known as ykarel10:40
*** chem has joined #oooq10:50
*** ccamacho has joined #oooq10:52
*** ccamacho has quit IRC10:53
*** ccamacho has joined #oooq10:54
*** openstackstatus has quit IRC10:59
*** sshnaidm has quit IRC11:07
*** sshnaidm has joined #oooq11:12
*** sshnaidm has quit IRC11:13
*** sshnaidm has joined #oooq11:13
chandankumarmarios|ruck, \o/11:22
chandankumarmarios|ruck, need some guidence there https://review.opendev.org/#/c/710980/1 to set python_intepreter value in env file11:23
marios|ruckchandankumar: ack reviewed but not sure what your question is? you need to pass that paramter PythonInterpreter into the deployment?11:27
chandankumarmarios|ruck, yes we need to pass python_intepreter to the deployment11:28
marios|ruckchandankumar: just set it in featureset ? which one is it?11:36
marios|ruckchandankumar: you don't need that i think https://review.opendev.org/#/c/710980/1/deployment/neutron/neutron-ovs-agent-container-puppet.yaml you can just use the existing PythonInterpreter vars11:36
weshay|rucksshnaidm, ping12:01
sshnaidmweshay|ruck, joining12:01
weshay|ruckpanda, can you see why the promoter is not promoting train https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-train-current-tripleo/12:01
pandaweshay|ruck: looking12:03
pandaweshay|ruck: do the weirdo job usually vote on dlrnapi ? I don't see any weird job success in detail page at https://trunk.rdoproject.org/api-centos-train/api/civotes_detail.html?commit_hash=67a865f1234718e712cd280c943a21317e7fe289&distro_hash=046903a2c5f8eba0b8ecd022e4ed8a2195e2361712:08
pandaweshay|ruck: how do I get the hash taht passed all the jobs ?12:09
* marios|ruck food biab12:16
*** rlandy has joined #oooq12:24
chandankumarmarios|ruck, https://review.rdoproject.org/r/#/c/25618/ it worked here12:27
weshay|rucksshnaidm, https://hackmd.io/HrQd03c9SxOMtFPFrq50tg#Known-Blocking-Bugs12:29
chandankumarrlandy, weshay|ruck https://review.opendev.org/#/c/710980/1/deployment/neutron/neutron-ovs-agent-container-puppet.yaml12:32
*** jpena is now known as jpena|lunch12:32
marios|ruckchandankumar: loo12:33
marios|ruckchandankumar: (looking)12:33
marios|ruckchandankumar: sorry don't follow what worked there12:34
marios|ruckchandankumar: i don't see python_interpreter override there or you talking about something else12:34
chandankumarweshay|ruck, https://review.rdoproject.org/r/#/c/25612/12:36
*** ratailor has quit IRC12:39
chandankumarweshay|ruck, https://review.rdoproject.org/r/#/c/25612/12:45
chandankumarweshay|ruck, https://review.rdoproject.org/r/#/c/24683/12:48
*** jmasud has quit IRC12:53
*** jmasud has joined #oooq12:56
chandankumarmarios|ruck, https://logserver.rdoproject.org/18/25618/3/check/periodic-tripleo-ci-centos-8-scenario007-standalone-master/478f2b1/logs/undercloud/var/log/extra/podman/containers/neutron_ovs_agent/stdout.log.txt.gz13:05
chandankumar python3 -m neutron.cmd.destroy_patch_ports13:05
chandankumarearlier it was /usr/bin/python (which does not exists on c8)13:05
*** jaosorior has joined #oooq13:11
* chandankumar headed home13:15
*** udesale_ has joined #oooq13:17
*** udesale has quit IRC13:18
marios|ruckchandankumar: where is scen7 stanadlone i can't find it is it defined in that review13:21
marios|ruckchandankumar: but not seen that one yet13:21
marios|ruckchandankumar: so you mean         * https://review.opendev.org/#/c/710980/1  works?13:22
marios|ruckchandankumar: ok , but i still think that is wrong approach as it breaks the current interface, better to pass it with the parameter ^^ as commented13:22
*** udesale_ has quit IRC13:24
*** udesale has joined #oooq13:24
rlandychandankumar: weshay|ruck: arxcruz: sshnaidm: panda: community call meet.google.com/bqx-xwht-wky13:30
weshay|ruckpanda, https://review.opendev.org/#/c/710869/13:30
weshay|ruckrlandy, chandankumar https://review.opendev.org/#/c/710441/13:31
*** udesale has quit IRC13:31
*** udesale has joined #oooq13:31
*** jpena|lunch is now known as jpena13:32
rlandychandankumar: weshay|ruck: ping community call13:32
*** marios|ruck is now known as marios|ruck|call13:32
weshay|ruckrlandy, in another mtg13:32
pandaweshay|ruck: yep, all passed it seems13:32
*** ykarel is now known as ykarel|afk13:35
*** skramaja has quit IRC13:35
*** marios|ruck|call is now known as marios|ruck13:42
sshnaidmrlandy, where was that mail from Alan?13:42
pandaarxcruz: were you calling from under a steam train with rheumatism ?13:42
rlandysshnaidm: marios|ruck: search for mail "rdocloud update / PHX2 future"13:42
rlandymaybe you guys are not on that list?13:43
rlandylet me know - will forward13:43
arxcruzpanda: now that you mention, there's something on here in the office doing a sound13:43
sshnaidmrlandy, I don't see, which list is it?13:43
arxcruzpanda: i was with noise cancelling, so i wasn't hearing13:43
rlandyrhos-role-ppc13:43
rlandysshnaidm: will forward13:44
sshnaidmoook, I'm definitely not on this list :)13:44
pandaarxcruz: too bad the noise canceling is not working in both directions13:44
arxcruzpanda: sorry, i keept on mute mostly of the time13:46
arxcruzbut yeah, now that i took the phones off, it's pretty annoying13:46
marios|ruckrlandy: thanks i don't have it either sshnaidm13:48
rlandymarios|ruck: forwarding13:48
marios|ruck'sorry sir you're not on the list'13:48
marios|ruck;)13:49
marios|ruckthanks rlandy13:49
*** ratailor has joined #oooq13:50
*** udesale has quit IRC13:58
*** udesale has joined #oooq13:58
*** ykarel|afk is now known as ykarel13:59
chandankumarmarios|ruck, rlandy but a bit away14:00
chandankumarmarios|ruck, https://review.rdoproject.org/r/#/c/25618/ here is the review14:00
weshay|ruckfolks.. please follow the #tripleo mtg..14:01
weshay|rucka lot of it will be about centos-814:01
chandankumarrlandy, it is good to land https://review.opendev.org/#/c/708599/ image build14:16
rlandychandankumar: ok - pls +2 and I'll w+14:17
chandankumarrlandy, done14:17
chandankumarrlandy, one big thing down14:18
rlandychandankumar: it has to get through gates :)14:18
rlandychandankumar: when it does, I'll w+  the rdo imagebuild patches14:18
rlandyand add to pipeline14:18
chandankumarrlandy, aye14:19
rlandytripleo-buildimage-overcloud-full-centos-8SUCCESS in 24m 42s14:19
rlandypassed - yay14:19
rlandyhttp://zuul.openstack.org/builds?job_name=tripleo-buildimage-overcloud-full-centos-814:20
ykarelchandankumar, rlandy can u review https://review.opendev.org/#/c/649418/14:21
*** Goneri has joined #oooq14:21
rlandyhttps://review.opendev.org/#/c/710179/ - what do we want to do about the multinode job upstream?14:23
rlandyit's non-voting14:23
rlandyhold it until we fix the hang?14:23
chandankumarrlandy, I think yes14:24
chandankumarrlandy,otherwise we will waste the upstream resources14:24
rlandyack14:24
chandankumartill we fix the hang14:24
rlandyhttps://review.opendev.org/#/c/710856/14:26
rlandy^^ voting14:26
rlandycould also get hit by the hang14:26
rlandymaybe we should adjust that14:26
weshay|ruckchandankumar, rlandy please review https://review.opendev.org/#/c/710441/14:45
rlandyweshay|ruck: I did :)14:45
rlandyweshay|ruck: https://review.opendev.org/#/c/710856/ that is w+'ed but it might get caught by the hang ...14:46
rlandygoing to put in a follow up patch to make that job non-voting in case we need it14:46
weshay|ruckrlandy, the hang is only undercloud on centos-814:46
*** ratailor has quit IRC14:46
rlandytripleo-ci-centos-8-undercloud-containers14:47
chandankumarrlandy, weshay|ruck keeping -1 for emit release till I revert my patch and test it as discussed14:47
rlandyweshay|ruck: so you are saying the hang will not hit tripleo-ci-centos-8-undercloud-containers14:48
rlandywhatever - if it blocks we can make it non-voting14:48
chandankumarrlandy, weshay|ruck ykarel and apevec were downgrading the package to see whether it fix or not14:50
chandankumardepends on lan work14:50
chandankumari mean hang14:51
rlandychandankumar: they are downgrading which package? to see if it fixes the hang?14:52
chandankumarrlandy, secretstorage14:52
rlandyoh I see on #rdo14:52
chandankumarrlandy, https://github.com/rdo-common/python-SecretStorage/blob/ussuri-c8-rdo/python-SecretStorage.spec14:53
rlandychandankumar: thanks14:53
chandankumarrlandy, I discussed during today's alan meeting, then we planned based on rhel and downstream job14:54
rlandychandankumar: cool - when will we get the downgraded package to test?14:54
chandankumarrlandy, it will be automatically available in c8 deps repo once built on cbs14:54
weshay|ruckrlandy, is tripleo-ci-centos-8-undercloud-containers merged upstream?14:57
rlandyweshay|ruck: not yet - it failed gates14:57
rlandyI rechecked14:57
weshay|ruckrlandy, ya.. anything undercloud w/ the known bug.. we may want to mark nv14:57
weshay|ruckat this time14:57
weshay|ruckeven though we've seen some success14:58
chandankumarweshay|ruck, it will waste resources, so better we can fix it and then add back14:58
chandankumarweshay|ruck, undercloud deploy success was 2h 30 mins last time14:59
*** ykarel is now known as ykarel|away14:59
rlandyweshay|ruck: chandankumar: so we have two choice here ...14:59
rlandystop the patch from merging - I vote no on that one15:00
rlandylet it go and make the job non-voting15:00
rlandyit's one undercloud job15:00
rlandyit passes sometimes15:00
rlandylet's get the feed back we can15:00
rlandywe are no merging the multinode15:00
chandankumarrlandy, ok, then make it nv and merge it15:02
rlandychandankumar: updating patch15:02
chandankumarrlandy, weshay|ruck tomorrow's goal to clear scenario7 and 10 and make it part of pipeline15:04
rlandyhttps://review.opendev.org/#/c/710856/15:04
rlandyvoting: false now - pls review again15:04
rlandychandankumar: weshay|ruck: ^^15:04
rlandyshould save us some temp headaches15:04
rlandychandankumar: today's goal is to get build-images into the pipeline15:05
rlandyand maybe standalone tempest if it runs enough15:05
chandankumarrlandy, aye15:05
rlandyhttps://review.rdoproject.org/r/#/c/25325/15:08
weshay|ruckrlandy++15:08
rlandyrunning again with 4 hr timeout15:08
rlandyweshay|ruck: I am starting now on the component pipeline reviews - to add the jobs discussed15:08
chandankumarWe are Building the C8 pipeline just like nOrmal building constructions15:08
marios|ruckweshay|ruck: do we have a way to rekick that https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/15:09
rlandyyou mean with no real plan??15:09
weshay|ruckmarios|ruck, don't worry about it15:09
marios|ruckweshay|ruck: seems to run every 3/days? last run yesterday failed for i think unrelated15:09
marios|ruckweshay|ruck: ah... was trying to get phase 115:09
weshay|ruckmarios|ruck, it will flip in it's time.. it's only required for rdo release at the end of cycle15:09
chandankumarrlandy, weshay|ruck for component pipeline jobs, I will coming with tempest tests tomorrow via hackmd15:09
ykarel|awaychandankumar, fyi build containers with depends-on didn't worked somehow, seems issue with the our job https://review.rdoproject.org/r/#/c/25636/15:10
weshay|ruckmarios|ruck, focus on something w/ higher value15:10
rlandychandankumar: yep - going to start laying out reviews you can edit15:10
marios|ruckweshay|ruck: k was just following up on https://bugs.launchpad.net/tripleo/+bug/1863632/comments/1115:10
openstackLaunchpad bug 1863632 in tripleo "lastest virtenv workarounds broke native tripleo-quickstart install" [Critical,Triaged]15:10
ykarel|awaychandankumar, but kolla patch is approved, so should be live in next run15:10
rlandyfirst one - to make promote pipeline run hourlu15:10
rlandyhourly15:10
chandankumarrlandy, what about c8 nodes/15:10
chandankumarsince we are going to kick hourly we need nodes15:11
weshay|ruckmarios|ruck, moved to fixed_released15:11
marios|ruckweshay|ruck: ok...15:11
rlandychandankumar: going to chat with ifra about that as soon as the review is up15:14
rlandyon my list15:14
chandankumarrlandy, aye15:14
*** sanjayu_ has quit IRC15:14
weshay|ruckmarios|ruck, chandankumar https://review.rdoproject.org/r/#/c/24716/15:14
marios|ruckweshay|ruck: looking15:15
marios|ruckweshay|ruck: well there's on on top of that first15:15
marios|ruck        * https://review.rdoproject.org/r/#/c/24715/15:15
marios|ruckweshay|ruck: ^15:15
chandankumarJust a general question #oooq, how many uses gertty?15:15
marios|ruckweshay|ruck: panda puts in in the topic (the next one in the chain)15:16
weshay|ruckI know.. there are two there.. just pinged on the wrong one15:16
chandankumarrlandy, weshay|ruck logging out now, will deal with few stuff tomorrow, see ya15:18
*** chandankumar is now known as raukadah15:18
rlandychandankumar: ack - have a good night15:18
rlandyhttps://softwarefactory-project.io/r/#/c/17091/3/nodepool/rdo-cloud.yaml15:20
rlandyraukadah: weshay|ruck: ^^ so I think this will be ok15:20
rlandythe nodes are not distro specific15:20
rlandyI asked on rhos-ops - but the resources should be per tenant15:21
weshay|ruckraukadah, thanks15:21
rlandynot per distro15:21
raukadahweshay|ruck, rlandy sshnaidm does not some magic build-images : run the image build script (direct ovb comes here15:21
raukadahovb results15:22
raukadah*does15:22
raukadahhttps://logserver.rdoproject.org/32/25332/25/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/d785687/job-output.txt15:22
raukadahhttps://logserver.rdoproject.org/32/25332/25/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/d785687/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz15:22
raukadahsudo: yum: command not found15:22
weshay|ruckoh wow15:22
weshay|rucknice15:22
raukadahsshnaidm, we love you :-)15:22
weshay|ruckone got through15:22
weshay|ruckso.. raukadah rlandy the question atm.. is whether or not.. now is the time to kill tqe/role/build-images and replace w/ tripleo-ci/roles/image-build15:23
weshay|ruckor not15:23
weshay|ruckprobably not.. but we DO need to get there15:23
raukadahweshay|ruck, yes kill it15:23
weshay|rucknow?15:23
rlandymight as well15:24
raukadahweshay|ruck, yes15:24
rlandyotherwise we will need to fix two places15:24
weshay|ruckok.. let's look at that work for 24hrs.. and report back15:24
weshay|ruckrlandy, aye15:24
weshay|ruckrlandy, should be fixed already in tripleo-ci15:24
raukadahweshay|ruck, rlandy image build fixed today it is in gates15:24
rlandyweshay|ruck: ack - we already did that work in one place15:24
rlandyraukadah: ack - watching it - we need to get that through15:25
rlandyweshay|ruck: raukadah: reviewing the promote pipeline schedule ... we sure we want that EVERY hour?15:25
* rlandy making the change now15:26
rlandy15 jobs15:26
weshay|ruckrlandy, for promote.. yes15:26
rlandyok - let's see how that goes15:26
raukadahrlandy, yes15:26
weshay|ruckthey are small.. and quick15:26
weshay|ruckrlandy, raukadah we can launch containers now.. I think15:26
rlandy<nhicher> rlandy: there is no allocation per label, only max-servers for a tenant15:27
weshay|ruckinstead of nodepool images15:27
rlandyso we're ok on node llocation15:27
rlandyallocation15:27
weshay|ruckthat kind of job is a good target for that15:27
weshay|ruckrlandy, I guess every two hours is fine15:27
rlandyfine - coming up15:27
weshay|ruckrlandy, it's easy to adjust if it kills us15:28
weshay|ruckbut I think containers would be better15:28
weshay|ruckrlandy, michal has shown that working..15:28
raukadahjpena, is it possible to run a complete pipeline in check?15:28
weshay|ruckit's a new feature that tristanC got in15:29
raukadahweshay|ruck, you mean pipeline?15:29
weshay|ruckraukadah, no.. I'm referring to s/nodepool_images/nodepool_containers for small jobs... like tox, lint.. or our promotion routine15:29
raukadahweshay|ruck, ah ok15:30
weshay|ruckand it's just a nodepool image change in zuul15:30
weshay|ruckjpena, tristanC is the ocp integration for nodepool in rdo software factory or just internal?15:30
raukadahrlandy, weshay|ruck standalone fs changes coming https://review.rdoproject.org/r/#/c/25618/3/zuul.d/standalone-jobs.yaml15:30
raukadah52 to 6415:31
raukadahwe have a fork of 5215:31
weshay|ruckheh15:31
weshay|ruckya.. raukadah I'll look at that today15:32
*** udesale_ has joined #oooq15:35
rlandyhttps://review.rdoproject.org/r/25642 Run component promote pipeline every two hours15:36
*** Goneri has quit IRC15:36
rlandy^^ I think that should work :)15:37
rlandyraukadah: ^^ review pls15:37
*** udesale has quit IRC15:37
rlandyhttps://review.rdoproject.org/r/#/c/25642/ merging this unless objections ... weshay|ruck15:42
jpenaraukadah: about the complete pipeline in check... I guess we could, if we defined the jobs correctly. Not sure about OCP, though15:43
raukadahweshay|ruck, rlandy verified heere https://crontab.guru/#0_*/2_*_*_*15:43
weshay|ruckmigi, ping15:43
rlandythat's cool15:43
rlandyok15:43
weshay|ruckmigi, re: your container trick for tox15:43
migiweshay|ruck: yeah?15:43
migiweshay|ruck: bj ?15:43
weshay|ruckmigi, you were working w/ tristian15:43
weshay|ruckmigi, can't in a manager mtg15:43
migiweshay|ruck: yep15:43
rlandyhow does that work?15:43
*** tesseract has quit IRC15:43
weshay|ruckmigi, any idea if they deployed the same in rdo?15:44
rlandyyou can run multiple jobs15:44
weshay|ruckrlandy, it's a node swap15:44
weshay|ruckvm -> container15:44
migiweshay|ruck: let me check15:44
migiweshay|ruck: should be there, see https://review.rdoproject.org/zuul/nodes15:47
migiweshay|ruck: managed-runc-provider15:47
rlandymigi: so how do you get a job to run on a container as opposed to a vm?15:48
migiweshay|ruck: https://review.rdoproject.org/r/#/q/status:open+project:config+branch:master+topic:add-container-nodeset15:48
rlandywhat does that buy you?15:48
migirlandy: change label: https://review.rdoproject.org/r/#/c/18931/1/zuul.d/projects.yaml15:48
rlandyso I am still not sure what that buys you15:48
migirlandy: time, flexibility, coolness factor x100015:48
rlandyuse of less resource?15:48
migirlandy: yep you can have one VM and concurrently run multiple jobs15:49
rlandyI'm very uncool  - I am only interested in time and resources15:49
rlandyok - so that may work15:49
migirlandy: so in upshift instead of 1 VM for 1 job switching to it it's 10 jobs per 1 vm15:49
rlandyhow many containers do you run on one vm?15:49
rlandy1015:49
rlandythat could work15:49
migirlandy: currently it's up to 10, but easy to configure15:49
weshay|ruckthe containers run on a vm?15:49
weshay|ruckno15:50
weshay|ruckmigi, /me thought it's just a nodeset change from a vm based node to a container15:50
migiplus it's direct path to use ocp/k8s at some point15:50
weshay|ruckno?15:50
migiweshay|ruck: from job perspective yes15:50
weshay|ruckrlandy, this is not about running 10 jobs on a single vm15:50
rlandywell that's what I want to do :)15:51
weshay|ruckheh15:51
rlandyotherwise why do we care?15:51
rlandyless resources is a goal here15:51
weshay|ruckright.. and a container would be much less15:51
rlandyless what though?15:52
migiit will cosume less resources, think of container can be set on VM or bare metal15:52
weshay|ruckthe promotion-jobs.. that check/promote.. can run like tox15:52
migihttps://softwarefactory-project.io/cgit/config/tree/containers15:52
migithis is how containers are specified15:52
rlandyif we have to spawn a node, we're in the same boat15:52
weshay|ruckno15:52
weshay|ruckthe node is a container15:52
weshay|rucknot a vm15:52
rlandyso we stil have the same node limit15:52
migirlandy: we spawned a persistent VM to have dynamic containers, because we had to have the containers somewhere15:52
rlandyno matter how we spawn it15:53
weshay|ruckrlandy, look at https://review.rdoproject.org/r/#/c/18933/2/zuul.d/projects.yaml15:53
*** tesseract has joined #oooq15:53
migirlandy: so in d/s we are using rhel-8 container on top of persistent VM that is running fedora15:53
migifedora runs for multiple days and containers lifetime is per job15:53
migi1 persistent OS (in our case fedora ) = 10 containers (dynamic) = 10 jobs in parallel, to start the job it takes 5 seconds because it's how much it takes to spawn container15:54
rlandymigi: so the win here is in the time it takes to spawn the nodeset>15:54
migirlandy: plus reproducibility I can create same container on my box using: podman build && podman run15:55
migirlandy: no15:55
migirlandy: bj? I will show you :)15:55
rlandyok15:56
migirlandy: bluejeans.com/mpryc15:57
rlandymigi: thanks for the demo16:15
rlandyweshay|ruck: ^^ this is a good idea - it does allow you to have multiple containers on one vm ...16:15
rlandybut the promote jobs just run16:15
rlandyaround 150 s per job16:15
rlandyso not urgent atm16:16
weshay|ruckaye16:16
weshay|ruckseed planted16:16
weshay|ruckthoughts will grow :)16:16
rlandysee - sometimes I do get what I want16:16
rlandymultiple jobs on one vm :)16:16
rlandyhappiness is ...16:16
marios|ruckrlandy: end of ruck|rover sprint ? ^16:22
*** udesale_ has quit IRC16:29
*** jmasud has quit IRC16:29
rlandymarios|ruck: lol - wait until we rope you into centos-8!!16:30
weshay|ruckdang.. that was a lot of mtgs16:33
weshay|ruck4.5 hrs16:33
*** holser has quit IRC16:37
marios|ruckweshay|ruck: sshnaidm: rlandy: is anyone already digging at centos8 jobs & grafana?16:41
marios|rucki can dig there tomorrow if no-one is already doing that16:41
sshnaidmmarios|ruck, I don't16:41
marios|rucki mean promotions especially16:41
marios|rucksshnaidm: ack16:41
marios|rucksshnaidm: maybe i can copy/paste the right things from the centos7 promotions ;)16:42
sshnaidmmarios|ruck, yeah, mostly it should work :)16:42
sshnaidmmarios|ruck, but don't forget telegraf changes16:42
marios|rucksshnaidm: k lets see how far i get and assuming thre are no new explosions between now and then16:43
rlandymarios|ruck: no, would be great if you can pick that up16:43
raukadahrlandy, marios|ruck approved cleared !16:44
raukadahmarios++16:44
marios|ruckraukadah: which bit? :)16:45
raukadahmarios|ruck, pythoninterpreter16:45
*** jpena is now known as jpena|brb16:46
marios|ruckraukadah: ack just clicked a button but thanks raukadah ;)16:48
rlandycool16:48
rlandyraukadah; does that spring scenario7?16:48
rlandyperiodic-tripleo-ci-centos-8-scenario004-standalone-master success16:49
rlandywow - when did that start passing?16:49
raukadahrlandy, s10 cleared16:50
rlandynice!16:50
rlandymarios|ruck: just fyi ... https://review.rdoproject.org/r/#/c/25549/1/zuul.d/tripleo.yaml merged16:50
rlandyso fs010 and fs027 will be centos-8 next time the pipeline runs16:51
rlandyalso ...16:51
rlandyif the upstream image build job passes gates,16:51
rlandywe will move to centos-8 image jobs in the pipeline16:51
marios|ruckrlandy: thanks16:52
sshnaidmrlandy, raukadah did we run centos8 jobs for images build?16:53
rlandysshnaidm: we are waiting to merge those jobs16:54
rlandywe have run them in testproject16:54
rlandyyesterday I ran the periodic jobs that should have pushed images16:54
rlandyuploaded16:54
rlandychecking what happened there16:54
rlandysshnaidm: once https://review.opendev.org/#/c/708599/ merges, periodic image build will run in master16:55
sshnaidmrlandy, marios|ruck weshay|ruck please vote https://review.opendev.org/#/c/710982/16:55
sshnaidmnice16:56
rlandyhttps://review.rdoproject.org/r/#/c/24995/16:57
rlandy^^ that should have uploaded images16:57
rlandylet's check16:58
*** tesseract has quit IRC16:58
rlandy2020-03-02 18:15:24.005923 | primary | + ls '*.tar'16:58
rlandy2020-03-02 18:15:24.007137 | primary | ls: cannot access '*.tar': No such file or directory16:58
rlandy2020-03-02 18:15:24.009838 | primary | cp: cannot stat '/home/zuul/*repo_setup*log': No such file or directory16:58
rlandy2020-03-02 18:15:24.220727 | primary | ok: Runtime: 0:00:00.07291216:58
marios|rucksshnaidm: ack but does it need to be conditional for centos8?16:59
rlandyplaybooks/tripleo-ci-periodic-base-upload/post.yaml17:00
rlandyugh - one more fix to make17:00
sshnaidmmarios|ruck, tbh I think it's not needed at all17:01
* weshay|ruck wonders how the rhel8 jobs are still triggering17:07
* rlandy cries17:08
rlandyyet ANOTHER get-hash line that needs modifying17:09
weshay|ruckBAH17:10
rlandyraukadah: you still around?17:12
rlandycould use some help with where the built images are stored17:12
rlandyweshay|ruck: shoot me ... https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/upload-cloud-images.sh#L317:12
rlandyhow do we do this for rhel??17:13
*** dtantsur is now known as dtantsur|afk17:13
marios|ruckweshay|ruck: getting ready for eod need something in particular?17:13
weshay|ruckrlandy, rhel has /etc/yum.repos.d/delorean.repo17:14
weshay|ruckrlandy, not sure I understand17:14
rlandywe use the same role for rhel so this must work17:15
weshay|ruckmarios|ruck, please vote on https://review.rdoproject.org/r/#/c/24716/17:15
weshay|ruckmarios|ruck, thanks :)17:15
weshay|ruckrlandy, what's broken?17:16
marios|ruckweshay|ruck: but its parent still neds merging first (we already had this conversation) https://review.rdoproject.org/r/#/c/24715/3117:16
rlandyweshay|ruck: chat - will explain17:16
marios|ruckweshay|ruck: i +2 there already17:16
rlandytripleo-ci-periodic-base-upload/post.yaml17:17
weshay|ruckmarios|ruck, the parent doesn't stop us from reviewing https://review.rdoproject.org/r/#/c/24716/17:17
weshay|ruckright?17:17
weshay|ruckyou have a +2 on the parent.. I know17:17
weshay|ruckrlandy, https://meet.google.com/kkd-qihx-hxu?authuser=117:18
marios|ruckweshay|ruck: ack well its on my queue... it wont merge until https://review.rdoproject.org/r/#/c/24715/31 unless panda rebases it.... i can review it more properly tomorrow ? it isn't blocked on my review right now...17:18
weshay|ruckrlandy, wooops17:18
weshay|ruckhttps://meet.google.com/djj-evih-fkh?authuser=117:18
weshay|ruckrlandy, someone is still in my asdf meeting17:18
weshay|ruckmarios|ruck, ya.. fine17:19
*** jpena|brb is now known as jpena17:25
*** marios|ruck is now known as marios|out17:26
raukadahrlandy, yes say?17:27
raukadahwhat got broke17:28
raukadahsshnaidm, yes we run image build for c817:28
sshnaidmraukadah, ack17:28
*** derekh has quit IRC17:28
raukadahsshnaidm, https://review.opendev.org/#/c/708599/ it will trigger in check and gate, rlandy has patches for make it in periodic and it is passing17:29
raukadahplease donot -1, there are some hack which works17:29
rlandyraukadah: we have an issue on periodic17:30
rlandydebugging that17:30
raukadahrlandy, and what is that?17:30
weshay|ruckrlandy, https://codesearch.rdoproject.org/?q=tripleo_image_source&i=nope&files=&repos=17:30
raukadahrlandy, weshay|ruck it was rhel_image_source17:31
raukadahI made it to tripleo image source and it works with rhel-817:31
raukadahif not defined, for c8, it will find required repos automatically17:31
weshay|ruckraukadah, patch needs to be reworked for centos-817:32
raukadahweshay|ruck, what is the error?17:33
raukadahI might help17:33
weshay|ruckwe're in https://meet.google.com/djj-evih-fkh?authuser=117:33
weshay|ruckif you want.. or go to sleep17:33
*** marios|out has quit IRC17:33
raukadahtripleo_image_source is undefined that it will pull images from cloud c8 url17:33
weshay|ruckrlandy, https://codesearch.rdoproject.org/?q=tripleo_image_source&i=nope&files=&repos=17:34
weshay|ruckhttps://opendev.org/openstack/tripleo-ci/src/branch/master/roles/oooci-build-images/tasks/main.yaml#L2017:37
rlandyraukadah: weshay|ruck: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/convert-upload-undercloud.sh17:39
weshay|ruckrlandy, https://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/convert-image/defaults/main.yml17:42
raukadahrlandy, weshay|ruck https://review.opendev.org/71098017:43
weshay|ruckrlandy, can you pass the log link to your last run w/ images17:58
* weshay|ruck needs to check how it's invoked17:58
rlandyweshay|ruck: https://review.rdoproject.org/r/#/c/24995/17:58
rlandyyou can edit that17:58
weshay|ruckthanks17:59
rlandyjust remove the two non-periodic17:59
rlandythey don't exist anymore17:59
*** bogdando has quit IRC18:00
weshay|ruckrlandy, https://review.opendev.org/711078 adding to ur job18:09
*** dsneddon has joined #oooq18:17
rlandyweshay|ruck: ^^ will run testproject again once we merge the r.r.o config review18:19
weshay|ruckk18:20
rlandylunch quickly ...18:20
rlandyweshay|ruck: wrt https://review.opendev.org/#/c/711078/ - how was it working with rhel before then?18:26
* rlandy include patch in testproject18:26
weshay|ruckrlandy, probably wasnt18:27
weshay|ruckdoesn't matter18:27
rlandyI know - just curious18:34
*** jpena is now known as jpena|off18:41
rlandyweshay|ruck: better ... https://review.rdoproject.org/zuul/stream/da6cc2077462416389b4c1ac976c57d5?logfile=console.log18:55
rlandybut we need a centos8 dir on eth image server18:55
rlandy2020-03-03 18:53:46.549481 | primary | rsync: mkdir "/var/www/html/images/centos8/master/rdo_trunk" failed: No such file or directory (2)18:55
weshay|ruckrlandy, bah did it fail on that18:56
rlandygetting18:57
rlandyhttps://logserver.rdoproject.org/95/24995/24/check/periodic-tripleo-centos-8-buildimage-overcloud-full-master/da6cc20/job-output.txt18:57
weshay|ruckhrm.. rlandy we should have access but rhos-ops I guess18:57
rlandyweshay|ruck: ^^18:57
rlandyweshay|ruck: is the image server still images.rdoproject.org?18:58
rlandyit just moved18:58
rlandyI think we can get on to that server from the promoter18:58
rlandypanda: ping ^^19:04
weshay|ruckrlandy, yes19:04
rlandydo we still have access to the image server from the promoter?19:04
weshay|ruckbut same url19:04
weshay|ruckwould have to try19:04
rlandyweshay|ruck: I'm on the promoter now19:05
rlandyssh'ed to images.rdoproject.org - no dice19:05
weshay|ruckrlandy, sftp19:05
weshay|ruckonly19:06
weshay|rucklet's chat19:06
rlandyk19:06
weshay|ruckhttps://meet.google.com/zse-wcdu-txc?authuser=119:06
rlandytps://logserver.rdoproject.org/95/24995/24/check/periodic-tripleo-centos-8-buildimage-overcloud-full-master/da6cc20/job-output.txt19:10
weshay|ruckrlandy, raukadah updated rpm for undercloud hang should be in tomorrow19:33
rlandyweshay|ruck: cool19:33
rlandyweshay|ruck: https://logserver.rdoproject.org/95/24995/24/check/periodic-tripleo-centos-8-buildimage-ironic-python-agent-master/21a337e/job-output.txt19:33
rlandyuploaded an image ... but19:33
rlandyhttps://images.rdoproject.org/centos8/master/rdo_trunk/19:33
rlandynot permissioned to view19:33
raukadahaye sir19:34
rlandyhttps://images.rdoproject.org/centos7/master/rdo_trunk/ we are19:34
rlandychecking permissions there19:34
raukadahdpawlik, will put the index.html might help to view it19:34
weshay|ruckseemless!19:35
dpawlikraukadah,  https://review.rdoproject.org/r/#/c/25393/ hope it will be merged soon :)19:36
raukadah:-)19:38
raukadahrlandy, weshay|ruck ^^ please have alook19:38
rlandyimages are there in the dir though19:38
weshay|ruckrlandy, double check me on https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?both#CentOS-8-OverCloud-Images19:39
rlandylooking19:40
rlandyI think we're good with periodic image build/push now19:40
weshay|ruckcool19:41
rlandyweshay|ruck: Depends-On: https://review.rdoproject.org/r/#/c/24997/ not needed19:41
rlandycheck is upstream19:41
weshay|ruckthanks19:42
rlandyjust the two upstream reviews19:42
weshay|ruckrlandy, ok.. let's look at undercloud jobs19:43
weshay|rucksince that should free up tomorrow19:43
rlandyweshay|ruck: ok19:44
weshay|ruckwe have https://review.opendev.org/#/c/710876/19:45
rlandythe update job19:46
rlandyhaven't had a chance to look at it19:46
weshay|ruckrlandy, np19:46
weshay|ruckrlandy, we need the containers-undercloud as well19:47
weshay|rucktripleo-ci-centos-7-undercloud-containers19:47
* weshay|ruck taking notes19:47
rlandyweshay|ruck: chat? might be quicker?19:47
rlandyalso want to confirm component pipeline19:47
rlandyso I can complete that work tonight419:48
weshay|ruckrlandy, ah.. give me 10 min19:48
weshay|rucklet me get my thoughts down in hackme19:48
weshay|ruckmd19:48
rlandysure19:48
weshay|ruckrlandy, meh.. I'm ready I guess19:51
weshay|ruckwhen ever you are19:51
weshay|ruckhttps://meet.google.com/uye-bfid-gkz?authuser=119:51
rlandyjoining19:51
rlandyweshay|ruck: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml#L50019:55
weshay|ruckrlandy,  https://github.com/openstack/tripleo-heat-templates/blob/master/README.rst#service-testing-matrix20:02
*** jfrancoa has quit IRC20:17
*** tosky has quit IRC20:22
weshay|ruckrlandy, https://review.rdoproject.org/r/#/c/25644/20:39
*** Goneri has joined #oooq20:47
*** jmasud has joined #oooq20:57
weshay|ruckrlandy, is this right? https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-component-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-baremetal-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-cinder-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-clients-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-cloudops-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-common-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-compute-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-glance-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-manila-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-network-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-octavia-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-security-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-swift-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-tempest-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-tripleo-master21:08
weshay|ruckperiodic-tripleo-ci-centos-8-standalone-ui-master21:08
rlandyyeah21:09
weshay|ruckrlandy, /me updating the cockpit21:09
weshay|ruckbut don't see a build histroy on that component21:09
rlandyweshay|ruck: on which component?21:10
rlandyperiodic-tripleo-ci-centos-8-standalone-component-master - that is a parent job21:10
weshay|ruckwait.. I think I filtered wrong21:10
weshay|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-compute-master21:10
weshay|ruckthat's better :)21:11
weshay|rucksorry!21:11
rlandyok21:11
rlandyweshay|ruck: ha ... this is a little more complicated than I thought  - each job will need to add the built repo ...21:21
rlandyexample review coming up21:21
weshay|ruckk21:21
*** jmasud has quit IRC21:53
weshay|ruckrlandy, http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=122:00
*** dpawlik has quit IRC22:00
*** jmasud has joined #oooq22:02
*** owalsh has quit IRC22:03
*** owalsh has joined #oooq22:03
*** jmasud has quit IRC22:11
*** jmasud has joined #oooq22:12
*** jbadiapa has quit IRC22:12
*** jmasud_ has joined #oooq22:14
*** jmasud has quit IRC22:17
rlandyweshay|ruck: cool22:20
rlandyweshay|ruck: stempest reported no logs again22:20
rlandywrt component jobs - we need to double inherit like: https://review.rdoproject.org/r/#/c/25649/2/zuul.d/standalone-jobs.yaml22:21
rlandyif that works, we'll be ok22:21
*** jmasud_ has quit IRC22:24
*** jmasud has joined #oooq22:24
*** jmasud has quit IRC22:24
weshay|ruckrlandy, k.. I have kid duty bbl22:30
rlandyweshay|ruck: no worries - we'll touch base tomorrow22:31
*** jmasud has joined #oooq22:31
*** jmasud has quit IRC22:39
*** jmasud has joined #oooq22:58
*** jmasud has quit IRC23:17
*** owalsh_ has joined #oooq23:25
*** owalsh has quit IRC23:28
*** jmasud has joined #oooq23:33
*** jmasud has quit IRC23:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!