Thursday, 2020-03-12

*** openstackstatus has joined #oooq		00:43
*** ChanServ sets mode: +v openstackstatus		00:43
*** rlandy\|bbl is now known as rlandy		02:15
rlandy	weshay\|ruck: hi - back	02:16
weshay\|ruck	rlandy, 0/	02:29
rlandy	weshay\|ruck: hey	02:30
weshay\|ruck	how's the other gig	02:30
rlandy	weshay\|ruck: tiring	02:30
rlandy	very few volunteers	02:30
rlandy	everyone is scared of the corona virus	02:30
weshay\|ruck	oh man.. this is how it ends.. it was nice working w/ you!!	02:31
rlandy	weshay\|ruck: in other news ... https://review.rdoproject.org/r/#/q/topic:add-minion-master+(status:open+OR+status:merged)	02:31
rlandy	and need to fix the criteria .. missing -master	02:31
weshay\|ruck	k	02:31
rlandy	coming up	02:32
rlandy	https://review.rdoproject.org/r/#/q/topic:add-minion-master+(status:open+OR+status:merged)	02:34
rlandy	has all three now	02:34
weshay\|ruck	boom boom boom	02:36
rlandy	ok - let's see what's going on with virt-customize	02:36
rlandy	weshay\|ruck: did you figure out the fs001 failure - baremetal/nova issue?	02:37
rlandy	cockpit components look good - thanks	02:38
weshay\|ruck	cool	02:40
weshay\|ruck	rlandy, https://review.opendev.org/#/c/711988/	02:40
rlandy	oh gee - travel from europe suspended	02:40
weshay\|ruck	https://bugs.launchpad.net/tripleo/+bug/1866965	02:40
openstack	Launchpad bug 1866965 in tripleo "Duplicate declaration: Package[collectd-python] is already declared at puppet-collectd and puppet-tripleo" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)	02:40
weshay\|ruck	rlandy, fixed	02:40
weshay\|ruck	rlandy, yup	02:40
weshay\|ruck	nba season suspended	02:40
rlandy	wow	02:40
weshay\|ruck	this will be a good lesson for the world	02:40
weshay\|ruck	WASH YOUR HANDS and DON'T EAT BATS	02:41
weshay\|ruck	eom	02:41
rlandy	lol	02:41
rlandy	hmmm ... https://logserver.rdoproject.org/95/24995/35/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/08efecd/logs/undercloud/home/zuul/overcloud-prep-images.sh.txt.gz	02:45
rlandy	the whole thing didn't even run	02:45
rlandy	ha	02:50
weshay\|ruck	rlandy, WHAAAA https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/988d2e0/	02:55
weshay\|ruck	rlandy, that's a pass	02:56
rlandy	what??	02:58
rlandy	weshay\|ruck: how is that possible?	02:59
weshay\|ruck	rlandy, looking through it.. looks legit so far	03:00
rlandy	weshay\|ruck: also looking - don't believe this	03:00
rlandy	https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/988d2e0/logs/undercloud/var/log/tempest/stestr_results.html.gz	03:01
rlandy	no way!!!	03:01
rlandy	weshay\|ruck: lol - that looks pretty legit	03:02
rlandy	periodic-tripleo-ci-centos-8-scenario001-standalone-cinder-master still failed	03:03
weshay\|ruck	https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/988d2e0/logs/undercloud/home/zuul/ansible.log.txt.gz	03:04
rlandy	https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/988d2e0/logs/overcloud-controller-0/etc/yum.repos.d/baremetal-component.repo.txt.gz	03:04
weshay\|ruck	rlandy, ya.. it just merged	03:04
rlandy	weshay\|ruck: the component repo is on the overcloud	03:04
weshay\|ruck	name: Import role tripleo-modify-image	03:04
weshay\|ruck	nothing updated	03:04
rlandy	well that could be true	03:04
weshay\|ruck	yes	03:05
rlandy	'update_repo': 'gating-repo,delorean-current,baremetal '	03:05
rlandy	idk	03:05
weshay\|ruck	rlandy, we should trigger that in a test project.. a few times	03:05
weshay\|ruck	so when we unpin tomorrow we know how consistent it is	03:05
rlandy	https://review.rdoproject.org/r/#/c/24995/	03:06
rlandy	running with repo change	03:06
rlandy	periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master	03:07
rlandy	weshay\|ruck: ^^ running in integration line now	03:07
weshay\|ruck	rlandy, I think this has to be one line --run-command '{{ ansible_pkg_mgr }} clean all'	03:07
weshay\|ruck	--run-command '{{ ansible_pkg_mgr }} update -y'	03:07
rlandy	but that is tripleo-ci-testing	03:07
weshay\|ruck	better the first way you had it	03:07
weshay\|ruck	rlandy, unless you saw --run-command used more than once	03:08
weshay\|ruck	I don't think I have	03:08
weshay\|ruck	could be wrong though	03:08
rlandy	let me check	03:08
rlandy	weshay\|ruck: do I need to sudo there?	03:10
weshay\|ruck	think so /me looks again	03:11
weshay\|ruck	https://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/libvirt/setup/undercloud/tasks/update_image.yml https://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/libvirt/setup/undercloud/templates/update_image.sh.j2	03:13
weshay\|ruck	not saying you should do it that way	03:13
weshay\|ruck	just posting as another example	03:13
weshay\|ruck	rlandy, hrm.. I don't think you need sudo	03:14
weshay\|ruck	rlandy, https://gist.github.com/goneri/961e34956171a5d35c50d440054eba96	03:15
*** surajpatil1 has joined #oooq		03:16
weshay\|ruck	rlandy, from goneri.. trusted source	03:16
rlandy	weshay\|ruck: yeah ...	03:16
rlandy	--update	03:16
weshay\|ruck	remove the subscription manager bits	03:16
weshay\|ruck	and looks like --update does what you want	03:17
rlandy	--upload /etc/yum.repos.d/dci_repo.repo:/etc/yum.repos.d/dci_repo.repo \	03:17
rlandy	--update \	03:17
rlandy	let me try that	03:17
weshay\|ruck	grafana]$ virt-customize --help \| grep update	03:17
weshay\|ruck	--update Update packages	03:17
rlandy	sec - updating review	03:18
rlandy	export LIBGUESTFS_BACKEND=direct	03:18
rlandy	virt-customize -a {{ overcloud_full_tar_name }}	03:18
rlandy	--upload /etc/yum.repos.d/{{ job.component }}-component.repo:/etc/yum.repos.d/{{ job.component }}-component.repo	03:18
rlandy	--update	03:18
rlandy	rechecking ...	03:20
rlandy	ok - let's see what that does	03:21
weshay\|ruck	rlandy, you need trailing \	03:21
weshay\|ruck	line 32, 33	03:22
rlandy	k - updating	03:22
rlandy	well the component pipeline is clearly kicking now	03:30
rlandy	weshay\|ruck: anything else needed before the great event tomorrow?	03:32
rlandy	europe day time starting shortly	03:32
weshay\|ruck	rlandy, 0/ go to sleep	03:36
weshay\|ruck	see u tomorrow :)	03:36
rlandy	hmm ... one more mistake	03:36
rlandy	correcting that then I am out	03:36
rlandy	overcloud-full.tar	03:37
weshay\|ruck	k	03:37
weshay\|ruck	ah	03:37
rlandy	the image is called what at this point??	03:37
rlandy	overcloud_full_tar_name: overcloud-full.tar	03:37
weshay\|ruck	OH	03:37
* weshay\|ruck looks		03:37
rlandy	overcloud-full.qcow2?	03:37
rlandy	I'm not sure	03:37
weshay\|ruck	yes	03:37
weshay\|ruck	it's untarred	03:38
weshay\|ruck	curl an image line 26	03:38
weshay\|ruck	untar it .. line 27	03:38
rlandy	2020-03-11 20:09:53 \| + openstack overcloud image upload --http-boot=/var/lib/ironic/httpboot	03:38
rlandy	2020-03-11 20:10:08 \| Image "overcloud-full-vmlinuz" was uploaded.	03:38
rlandy	https://logserver.rdoproject.org/95/24995/35/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/08efecd/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz	03:38
rlandy	weshay\|ruck: so what is the name of the image we are customizing?	03:39
rlandy	it doesn't look like it get downloaded	03:39
weshay\|ruck	just overcloud-full	03:39
* weshay\|ruck downloading		03:39
rlandy	it's already on the undercloud	03:39
rlandy	see the log	03:39
rlandy	leo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/08efecd/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz	03:39
rlandy	ugh	03:39
rlandy	https://logserver.rdoproject.org/95/24995/35/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/08efecd/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz	03:40
rlandy	there is no download there	03:40
rlandy	https://logserver.rdoproject.org/95/24995/35/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/08efecd/logs/undercloud/home/zuul/overcloud-prep-images.sh.txt.gz	03:40
weshay\|ruck	Image "overcloud-full" was uploaded.	03:42
weshay\|ruck	so.. overcloud-full.qcow2	03:42
rlandy	right	03:42
rlandy	let me change that in the review	03:42
weshay\|ruck	2020-03-11 20:10:08 \| \| 0e419b30-967c-4551-b261-59d23b78c5d3 \| overcloud-full \| qcow2 \| 1002619392 \| active \|	03:42
weshay\|ruck	rlandy, ya.. that can be hardcoded	03:43
rlandy	ok - rechecked	03:44
rlandy	weshay\|ruck: gn - hope tomorrow surprises us for the good!	03:44
weshay\|ruck	:)	03:44
*** rlandy has quit IRC		03:44
*** ykarel\|away is now known as ykarel		03:48
*** apetrich has quit IRC		04:00
*** udesale has joined #oooq		04:17
*** jmasud has quit IRC		04:53
*** skramaja has joined #oooq		04:54
*** jmasud has joined #oooq		04:55
*** raukadah is now known as chandankumar		05:16
*** surajpatil1 has quit IRC		05:53
*** surajpatil1 has joined #oooq		05:58
*** saneax has joined #oooq		06:27
*** surajpatil1 has quit IRC		06:40
*** marios has joined #oooq		06:48
*** dpawlik has joined #oooq		07:04
*** bogdando has joined #oooq		08:00
*** jfrancoa has joined #oooq		08:03
*** apetrich has joined #oooq		08:03
*** amoralej\|off is now known as amoralej		08:06
*** tesseract has joined #oooq		08:08
*** tosky has joined #oooq		08:18
*** gchamoul is now known as strider\|afk		08:27
*** strider\|afk is now known as gchamoul		08:27
*** matbu has quit IRC		08:29
*** matbu has joined #oooq		08:29
*** jbadiapa has joined #oooq		08:36
*** jpena\|off is now known as jpena		08:45
*** jmasud has quit IRC		08:54
*** jmasud has joined #oooq		08:56
zbr	panda: help me with this 2 liner: https://review.opendev.org/#/c/712477/	09:08
panda	zbr: define "help"	09:10
zbr	review.	09:11
marios	biab	09:11
*** marios has quit IRC		09:11
panda	zbr: a week ? wouldn't the jobs run before ?	09:11
*** dtantsur\|afk is now known as dtantsur		09:13
*** ykarel is now known as ykarel\|lunch		09:19
zbr	we discovered that several of the 78 scenarios there got rotten in the last 10 days as only some jobs are triggered on changes.	09:21
zbr	so yesterday, after few talks with infra, I added a "periodic-weekly" pipeline near the daily one.	09:22
zbr	and we got ok to add some jobs to it, specially for that, to prevent them from not running for long periods of time.	09:22
zbr	is quite similar with what we have on rdo	09:22
zbr	panda: here is one example: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-7-molecule-tripleo_ceph_run_ansible&project=openstack/tripleo-ansible	09:24
zbr	you see the gap of 10 days?	09:24
zbr	from next week we will also have weekly threre, making easier to spot issues.	09:25
*** holser has joined #oooq		09:28
dpawlik	Hi, do you still need fedora-29 host ? (https://github.com/rdo-infra/ansible-role-tripleo-ci-zuul-runner/blob/master/zuul.yaml#L32)	09:29
dpawlik	sshnaidm\|afk, weshay\|ruck ^^ (or can I update that to 31)	09:30
dpawlik	and one more question: do you need jobs running with fedora 30 or it can be also updated to 31 (soon 30 will be eol)	09:32
dpawlik	and I guess fedora 28 is sitll required	09:35
sshnaidm\|afk	dpawlik, no, we can change it to 30	09:38
*** sshnaidm\|afk is now known as sshnaidm		09:39
sshnaidm	dpawlik, jobs can be updated to 31 only when we'll have repos for 31 http://mirror.regionone.rdo-cloud.rdoproject.org/fedora/releases/	09:41
*** derekh has joined #oooq		09:41
dpawlik	sshnaidm, oh	09:42
* dpawlik what is this		09:42
sshnaidm	dpawlik, I talked with tristanC and others to add repos, it requires an image of f31 in upstream, and I don't know where it stopped	09:46
dpawlik	sshnaidm, ack. Will check that today	09:46
*** marios has joined #oooq		10:16
*** ykarel\|lunch is now known as ykarel		10:19
zbr	what f28 needed, where?	10:20
marios	panda: what's the story on https://review.rdoproject.org/r/#/c/24774/ why don't you rebase	10:23
marios	panda: 'cannot merge'	10:24
marios	panda: so all green \o/ and please review that too https://review.rdoproject.org/r/#/c/25782/ py tests green	10:24
dpawlik	zbr, just "loose question". If its not required, lucky I	10:31
marios	chandankumar: so once the rdo folks are done merging the distgits, we merge the unpin	10:40
chandankumar	marios, yes	10:40
marios	chandankumar: and then? wait for pipeline. should we just start posting testproject i wonder	10:40
chandankumar	marios, once unpinning merges, 12:00 utc component pipeline will pikc	10:41
marios	chandankumar: lets see what time it merges	10:41
marios	chandankumar: it might be close to pipeline	10:41
marios	chandankumar: dont want to have it duplicate	10:41
chandankumar	marios, ok	10:42
chandankumar	marios, i am mostly worried with this review https://review.rdoproject.org/r/#/c/23793/	10:42
chandankumar	in our testing it does not break anything up still we can see in pipeline	10:43
marios	chandankumar: there was discussion about that on rdo just now	10:43
marios	that review i mean	10:43
ykarel	marios, chandankumar unpin patch will take couple of hours after +W	10:48
ykarel	unless and until we force merge it	10:48
ykarel	once it merges, it will take couple of more hours for DLRN to build all these projects	10:49
chandankumar	ykarel, marios we need to merge before 12:00 utc	10:49
marios	chandankumar: sounds like it doesn't matter see what ykarel said	10:49
marios	chandankumar: so we have to wait anyway	10:49
ykarel	yes jsut merging will not help	10:49
ykarel	and anyway i think component pipeline is 00:00 not 12:00	10:50
ykarel	and that is too far	10:50
ykarel	we will be all set till than	10:50
ykarel	can u cross check this?	10:50
chandankumar	wait checking	10:51
marios	pipe times all in there https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/upstream.yaml	10:51
marios	ykarel: chandankumar: ^	10:51
ykarel	yup i checked that already	10:51
marios	ykarel: looks like each component has different time	10:51
ykarel	marios, yes	10:51
ykarel	check the starting one	10:51
ykarel	first component	10:51
marios	yah midnight	10:51
marios	ykarel: ack so once things go through we can post a testproject with the pipeline	10:52
chandankumar	2020-03-13 00:00:00	10:52
*** jmasud has quit IRC		10:52
ykarel	yeah	10:52
ykarel	marios, yes u can prepond testing	10:52
ykarel	once all set	10:52
chandankumar	marios, ykarel each component starts after another by one and half hr difference	10:53
*** jmasud has joined #oooq		10:53
ykarel	yes	10:53
ykarel	chandankumar, do u know what's the reason for this?	10:53
ykarel	1.5 hr difference	10:53
panda	marios: reviewsing, in the meantime I'm adding the final unit tests for the promoter. Maybe a couple functional tests too. Just discovered a bug in sanity checks for the configuration ...	10:54
panda	marios: also I have this that will need some more +2 later https://review.opendev.org/712517	10:54
panda	marios: officially skydive is not supported, so we can remove it.	10:55
chandankumar	ykarel, due to limited resources i think forgot about the exact reason there was some issue with dlrn md5 creation, will wait for rlandy to come and answer	10:55
marios	panda: k	10:56
ykarel	chandankumar, ack	10:56
ykarel	hmm but looks like very limited jobs run in those pipeline	10:56
*** derekh has quit IRC		11:10
*** sshnaidm has quit IRC		11:18
*** sshnaidm has joined #oooq		11:19
*** derekh has joined #oooq		11:20
*** sshnaidm has quit IRC		11:22
*** sshnaidm has joined #oooq		11:30
*** sshnaidm has quit IRC		11:30
chandankumar	brb for an hour	11:37
weshay\|ruck	arxcruz\|rover, 0/	11:43
arxcruz\|rover	weshay\|ruck: o/	11:43
*** jpena is now known as jpena\|lunch		11:49
weshay\|ruck	arxcruz\|rover, got the board updated for master http://rhos-release.virt.bos.redhat.com:3030/rhosp	11:51
*** sshnaidm has joined #oooq		11:52
arxcruz\|rover	weshay\|ruck: cool, the failures from today, already have bugs assigned, and it was because it wasn't merged yet thee patches	11:52
*** sshnaidm has quit IRC		11:53
weshay\|ruck	arxcruz\|rover, rock on	11:53
*** sshnaidm has joined #oooq		11:54
*** holser has quit IRC		11:54
*** rlandy has joined #oooq		12:01
weshay\|ruck	arxcruz\|rover, can you please make http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario007-standalone and http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-standalone voting	12:04
weshay\|ruck	and add to https://review.opendev.org/#/q/topic:centos-8-jobs+status:open	12:05
weshay\|ruck	chandankumar, is soniya around?	12:06
chandankumar	weshay\|ruck, noone is in office, everyone is work from home	12:09
arxcruz\|rover	weshay\|ruck: https://review.opendev.org/712665	12:12
*** udesale_ has joined #oooq		12:14
*** udesale has quit IRC		12:17
weshay\|ruck	arxcruz\|rover, both scenarios?	12:20
arxcruz\|rover	weshay\|ruck: yes, you want me to split the patches ?	12:20
weshay\|ruck	arxcruz\|rover, no	12:24
weshay\|ruck	should be in the same patch	12:24
arxcruz\|rover	weshay\|ruck: ok, already update adding the jobs to the gate	12:25
*** ysandeep is now known as ysandeep\|away		12:26
rlandy	ugh - seriously? /home/zuul/overcloud-prep-images.sh: line 20: virt-customize: command not found	12:27
weshay\|ruck	heh	12:28
*** jpena\|lunch is now known as jpena		12:32
chandankumar	marios, https://trunk.rdoproject.org/centos8-master/queue.html	12:33
chandankumar	rlandy, weshay\|ruck add it to review queue https://review.rdoproject.org/r/25726	12:45
weshay\|ruck	chandankumar, are they ok w/ the expanded coverage?	12:45
chandankumar	weshay\|ruck, yes	12:45
chandankumar	rlandy, since I am done with c8 works, next item you want to me deal image build or any other stuff?	12:46
chandankumar	weshay\|ruck, https://review.rdoproject.org/r/25726 it is not going to run automatically someone needs to call that job	12:47
chandankumar	we added it there so that anyone can pick the job in .zuul.yaml and run it	12:48
rlandy	looking	12:49
rlandy	chandankumar: I'm working on the image update for ovb	12:49
chandankumar	rlandy, you mean injecting repos patch?	12:49
rlandy	chandankumar: it would be good to work on the image build for to_Build	12:49
rlandy	yeah - got somewhere yesterday	12:49
chandankumar	rlandy, ok	12:50
rlandy	also checking fs030	12:50
marios	rlandy: i was looking at that	12:50
marios	rlandy: trying to see why it failed to pull from the local registry	12:50
marios	rlandy: * 2020-03-10 05:32:05 \| Error: error pulling image "192.168.24.1:8787/tripleomaster/centos-binary-mariadb:af182654cc32d30ea7f4774eb06ed9fd-updated-20200310044415": unable to	12:50
marios	* https://logserver.rdoproject.org/50/25550/6/check/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master/eb360b6/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz	12:50
rlandy	marios: we should rerun that	12:50
chandankumar	anyway I need to update puppet jobs to standalone c8	12:50
marios	rlandy: has that been a consistent error or is it a new thing?	12:50
rlandy	weshay was updating the containers	12:51
panda	sorry I'm having some difficulties. Ireland annouced a semi shutdown, and I'm providing with medical supply from my parents through amazon. I'll be delayednow and then	12:51
rlandy	maybe some where missing	12:51
marios	rlandy: ah ack , ok then lets recheck	12:51
rlandy	chandankumar: wrt https://review.rdoproject.org/r/#/c/25726/13/zuul.d/distgit-jobs.yaml	12:51
rlandy	how often should that run?	12:51
rlandy	you want me to put it in a pipeline?	12:51
chandankumar	rlandy, never unless we call it something like this https://review.rdoproject.org/r/#/c/25727/14/.zuul.yaml	12:52
chandankumar	it was added for testing	12:52
rlandy	chandankumar: ok - so what should I do?	12:52
rlandy	kick it a few times today?	12:52
chandankumar	rlandy, needs to merge that that's it	12:52
rlandy	oh ok	12:53
rlandy	ok - I'll take care of that	12:53
chandankumar	rlandy, cool, thanks!	12:53
rlandy	chandankumar: anything else?	12:53
chandankumar	rlandy, nope	12:53
rlandy	chandankumar: get some rest then !!!	12:53
chandankumar	weshay\|ruck, Can I update podman third party jobs to c8?	12:54
weshay\|ruck	chandankumar, YES!	12:54
weshay\|ruck	chandankumar, thank you	12:54
chandankumar	weshay\|ruck, ok	12:54
rlandy	we merged a bunch of stuff last night	12:54
chandankumar	weshay\|ruck, you are suppose to arrange a Virtual Pizza party for us by End of sprint	12:54
weshay\|ruck	chandankumar, are these in an experimental queue or something https://review.rdoproject.org/r/#/c/25726/13/zuul.d/distgit-jobs.yaml ?	12:54
weshay\|ruck	rlandy, queue was good	12:55
rlandy	chandankumar: lastly - wrt tempest failures - they were not consistent - so I didn't add anything to the skip list yet	12:55
rlandy	weshay\|ruck: which one?	12:55
rlandy	component? integration?	12:55
chandankumar	weshay\|ruck, it is not in any queue, it will added on demand when we update a rdoinfo tag	12:56
chandankumar	by an user	12:56
rlandy	periodic-tripleo-ci-centos-8-standalone-full-tempest-masteropenstack/tripleo-cimasteropenstack-periodic-mastermaster100012020-03-12T09:05:06SUCCESS	12:56
chandankumar	to give more confidence if updating a tag can break it ro not	12:56
rlandy	OMG	12:56
rlandy	https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master - very pretty	12:56
weshay\|ruck	chandankumar, ya.. imho it's too much	12:57
weshay\|ruck	at least from the feedback I've received in the past	12:57
chandankumar	weshay\|ruck, but we need to enforce the people to test the new tag before merging it	12:58
weshay\|ruck	chandankumar, rlandy scrum	13:01
weshay\|ruck	https://hackmd.io/7MBqFHurTA2e5H8kYRwgag?view#UNPINNING-NOTES--Component-pipeline	13:04
*** amoralej is now known as amoralej\|lunch		13:15
chandankumar	rlandy, https://review.rdoproject.org/r/#/c/25727/14/.zuul.yaml	13:16
chandankumar	- tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 https://review.rdoproject.org/zuul/build/008de10c274d43539d08ba114fddc74f : SUCCESS in 2h 29m 32s	13:16
marios	panda: let me update mine too sec thn you can pull it for test on promoter	13:18
marios	panda: done v11 https://review.rdoproject.org/r/#/c/25782/ thanks#	13:20
weshay\|ruck	https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged)	13:23
chandankumar	rlandy, tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 https://review.rdoproject.org/zuul/build/008de10c274d43539d08ba114fddc74f : SUCCESS in 2h 29m 32s	13:26
*** derekh has quit IRC		13:29
chandankumar	amoralej\|lunch, rlandy, ykarel weshay\|ruck marios https://trunk.rdoproject.org/centos8-master/queue.html queue is cleared now merging ironic now	13:30
*** dtantsur is now known as dtantsur\|brb		13:36
rlandy	marios: could you forward me that email	13:37
marios	rlandy: sure fetching	13:38
marios	rlandy: (done)	13:39
rlandy	thanks	13:39
*** skramaja has quit IRC		13:40
rlandy	marios: thanks - ha - I really wasn't sent that email.	13:42
rlandy	kind of insulated :(	13:42
rlandy	chandankumar: http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/17.0-RHEL-8/	13:44
*** amoralej\|lunch is now known as amoralej		13:46
chandankumar	ykarel, let me know when it is up	13:46
ykarel	chandankumar, yes will do	13:47
rlandy	creating testproject for when we are ready to go there	13:47
chandankumar	marios, https://aa8dd8dcd025e9c032ca-2413b0a350e1f4a4b77815cdd4c6afd0.ssl.cf2.rackcdn.com/712013/5/check/tripleo-ci-centos-8-scenario007-multinode-oooq-container/65703ab/logs/undercloud/home/zuul/overcloud_deploy.log	13:48
chandankumar	seen mariadb container issue	13:48
rlandy	periodic-tripleo-ci-centos-8-containers-undercloud-minion-master	13:50
rlandy	weshay\|ruck: ^^ in the pipeline now	13:51
weshay\|ruck	rlandy, rock	13:51
chandankumar	rlandy, https://trunk.rdoproject.org/centos8-master/queue.html queue is up once get empty send the test project for baremetal and tripleo	13:51
*** holser has joined #oooq		13:52
rlandy	chandankumar: sec - testprojects coming up	13:52
rlandy	chandankumar: https://review.rdoproject.org/r/#/c/25865/	13:55
rlandy	^^ abandoned the review so it doesn't run	13:56
chandankumar	rlandy, aye	13:56
rlandy	restore it when you want to run	13:56
rlandy	chandankumar: tripleo will run later today - you want to run it now as well?	13:56
rlandy	fine by me - either way	13:56
chandankumar	rlandy, right now not	13:56
chandankumar	watching the queue currently	13:57
rlandy	ok	13:57
rlandy	chandankumar: k - just ping me if you need other component stuff	13:57
chandankumar	rlandy, sure	13:57
marios	chandankumar: looking	13:57
marios	chandankumar: ack not to do with the container itself i think i.e .mariadb just there is issue reaching local registry	13:58
chandankumar	rlandy, https://review.opendev.org/712248 please +w	13:58
marios	chandankumar: same as in fs30	13:58
chandankumar	marios, https://review.opendev.org/712013 https://review.opendev.org/712294 are good to go then	13:58
chandankumar	5 more to go	13:59
chandankumar	rlandy, , https://review.opendev.org/712013 https://review.opendev.org/712294 two more	14:02
*** derekh has joined #oooq		14:03
rlandy	chandankumar: all good except ... https://review.opendev.org/#/c/712294	14:06
chandankumar	rlandy, since there will be no patches coming for rocky it will not harm	14:07
rlandy	really?	14:07
chandankumar	rlandy, I think so	14:07
rlandy	let's be sure	14:08
chandankumar	rlandy, ok removing	14:08
chandankumar	rlandy, done	14:10
marios	needs reviews please scen10-ovn @ https://review.opendev.org/711507 https://review.rdoproject.org/r/25745 https://review.rdoproject.org/r/25746 https://review.rdoproject.org/r/25747 ( there https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?both#Standalone )	14:12
rlandy	marios: ^^ will do	14:14
marios	rlandy: thanks not urgent just please add to your reviews queue	14:16
chandankumar	rlandy, light the candle now, queue processed	14:18
rlandy	chandankumar: baremetal?	14:18
chandankumar	rlandy, yes	14:18
rlandy	k - here we go	14:19
chandankumar	rlandy, all ftbfs also merger	14:19
chandankumar	*merged	14:19
chandankumar	real fun will come now	14:19
chandankumar	rlandy, I am confused here https://review.opendev.org/712524 what are your plans on fixing image build	14:21
rlandy	chandankumar: it updates	14:25
chandankumar	rlandy, only the yum repos na?	14:28
*** jfrancoa has quit IRC		14:29
chandankumar	rlandy, what about dib, image elements and tripleo client?	14:29
weshay\|ruck	zbr, let's chat	14:29
rlandy	chandankumar: so we need that for image build	14:31
weshay\|ruck	zbr, :) :) :)	14:31
rlandy	this is just to get the updated component in the image	14:32
chandankumar	rlandy, yes, if we update the packages it does not make any sense, unless we also update the dib related stuff	14:32
rlandy	it doesn't build the image from scratch	14:32
rlandy	chandankumar: for component this is what we need	14:32
rlandy	when we run to_build, we need more	14:32
chandankumar	rlandy, ok	14:32
rlandy	chandankumar: make sense?	14:33
chandankumar	rlandy, I will look into other aspects which I am thinking	14:33
chandankumar	rlandy, yes	14:33
rlandy	cool	14:33
chandankumar	rlandy, How is the situation in your area related to covid-19? I need to stop ordering food from tomorrow and learn cooking due to covid	14:34
rlandy	chandankumar: crazy here	14:34
chandankumar	everyone comes with covered faces, it makes me too afraid	14:34
rlandy	my immediate are is not so bad but westchester which is not far is totally shut down	14:35
rlandy	schools are closed	14:35
chandankumar	rlandy, same here	14:35
chandankumar	schools closed	14:35
rlandy	just saw they canceled Denver	14:36
marios	here schools closed since yesterday	14:36
marios	if for nothing else this is scary just for how unprecedented it is for e.g. schools to be closed in so many places :/	14:37
*** holser has quit IRC		14:37
chandankumar	in pune, weworks office was closed for two days	14:37
weshay\|ruck	zbr, https://review.opendev.org/#/c/712422/	14:38
weshay\|ruck	tripleo-build-containers-centos-7SUCCESS in 55m 01s	14:38
weshay\|ruck	zbr, ^	14:38
chandankumar	better I can go to home town and work from there, but hard to take flights also, no case in my state till now	14:39
*** holser has joined #oooq		14:42
*** jfrancoa has joined #oooq		14:44
weshay\|ruck	[08:47:17] <yoctozepto> weshay\|ruck: let me summarize this for you	14:48
weshay\|ruck	[08:47:25] <weshay\|ruck> k	14:48
weshay\|ruck	[08:47:30] <yoctozepto> weshay\|ruck: infra installed python3+pip3 in centos7 images	14:48
weshay\|ruck	[08:47:34] <yoctozepto> (well, all for that matter)	14:48
weshay\|ruck	[08:47:47] * zbr (sid419200@gateway/web/irccloud.com/x-lsvwlbjijmfrciut) has joined	14:48
weshay\|ruck	[08:47:48] <yoctozepto> weshay\|ruck: and base job does detection of py version based on pip3 availability	14:48
weshay\|ruck	[08:47:53] <weshay\|ruck> ah	14:48
weshay\|ruck	[08:48:01] <yoctozepto> weshay\|ruck: and replaced virtualenv happily	14:48
weshay\|ruck	[08:48:03] <yoctozepto> weshay\|ruck: ta-da	14:48
weshay\|ruck	zbr, ^	14:48
*** beagles has quit IRC		14:49
weshay\|ruck	arxcruz\|rover, fyi.. keep an eye on centos-7 jobs for python issues..	14:49
arxcruz\|rover	weshay\|ruck: ok	14:49
weshay\|ruck	arxcruz\|rover, infra is changing some things here..	14:49
weshay\|ruck	[08:47:17] <yoctozepto> weshay\|ruck: let me summarize this for you	14:49
weshay\|ruck	[08:47:25] <weshay\|ruck> k	14:49
weshay\|ruck	[08:47:30] <yoctozepto> weshay\|ruck: infra installed python3+pip3 in centos7 images	14:49
weshay\|ruck	[08:47:34] <yoctozepto> (well, all for that matter)	14:49
weshay\|ruck	[08:47:47] * zbr (sid419200@gateway/web/irccloud.com/x-lsvwlbjijmfrciut) has joined	14:50
weshay\|ruck	[08:47:48] <yoctozepto> weshay\|ruck: and base job does detection of py version based on pip3 availability	14:50
weshay\|ruck	[08:47:53] <weshay\|ruck> ah	14:50
weshay\|ruck	[08:48:01] <yoctozepto> weshay\|ruck: and replaced virtualenv happily	14:50
weshay\|ruck	[08:48:03] <yoctozepto> weshay\|ruck: ta-da	14:50
weshay\|ruck	[08:48:08] * ricolin_ has quit (Quit: Leaving)	14:50
weshay\|ruck	[08:48:10] <yoctozepto> weshay\|ruck: working with infra on this	14:50
weshay\|ruck	[08:48:37] <weshay\|ruck> k.. thank you.. /me catching up in there as well	14:50
weshay\|ruck	arxcruz\|rover, https://meet.google.com/pqf-dxyz-djo?authuser=1	14:50
weshay\|ruck	arxcruz\|rover, zbr https://review.opendev.org/#/c/710698/	14:52
weshay\|ruck	arxcruz\|rover, zbr 2020-03-12 01:51:36.567684 \| LOOP [build-containers : Install deps from upper-constraints]	14:53
weshay\|ruck	https://c492bb739031187edac3-364fef5d0294fafc2826805b949e4cc6.ssl.cf2.rackcdn.com/710698/2/check/tripleo-build-containers-centos-7/bc297a1/job-output.txt	14:53
zbr	https://621ce5ea301cd3e5f94e-66098508dda66f6b765978d818669c61.ssl.cf2.rackcdn.com/712422/1/check/kolla-build-centos8-source/aaa2a1c/kolla/build/neutron-server-opendaylight.log	14:53
rlandy	we got failures ... https://review.rdoproject.org/r/#/c/25865/	14:57
rlandy	periodic-tripleo-ci-centos-8-standalone-baremetal-masterFAILURE in 12m 12s	14:57
rlandy	periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-masterFAILURE in 19m 08s	14:57
rlandy	https://logserver.rdoproject.org/65/25865/1/check/periodic-tripleo-ci-centos-8-standalone-baremetal-master/8e0f88f/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz	14:59
rlandy	chandankumar: ^^	14:59
chandankumar	rlandy, awesome	15:00
chandankumar	amoralej, ykarel https://logserver.rdoproject.org/65/25865/1/check/periodic-tripleo-ci-centos-8-standalone-baremetal-master/8e0f88f/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz	15:00
chandankumar	recent results	15:01
chandankumar	after upinning	15:01
ykarel	should not be related to unpinning though	15:03
ykarel	but let's check	15:03
chandankumar	cssselect- coming from deps	15:03
weshay\|ruck	https://6dce6a2bde778b7bfc11-e183e0d9eb75cb729bae1aa0be881d78.ssl.cf5.rackcdn.com/712644/1/check/tripleo-build-containers-centos-7/bf343e1/job-output.txt	15:03
weshay\|ruck	zbr, arxcruz\|rover ^	15:03
amoralej	chandankumar, unrelated to unpin	15:03
amoralej	recheck it	15:03
chandankumar	rlandy, ^^	15:04
rlandy	ok	15:04
amoralej	it's related to build-deps movement	15:04
chandankumar	amoralej, did we changed something there?	15:04
amoralej	btw, if it's needed to install tripleoclient that shouldn't be in build-deps	15:04
amoralej	chandankumar, yes i did	15:04
chandankumar	amoralej, do we need to update release files?	15:05
ykarel	some week deps	15:05
amoralej	nop	15:05
ykarel	python3-pygraphviz also	15:05
amoralej	no need to touch nothing in your side	15:05
chandankumar	amoralej, aye	15:06
amoralej	my only doubt is wrt infra mirrors	15:06
amoralej	if they may have some content pending to sync	15:06
amoralej	mmm	15:06
amoralej	weak deps	15:06
weshay\|ruck	zbr, arxcruz\|rover https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/venv_setup.yml	15:06
chandankumar	amoralej, do we track somewhere what got synced or not synced?	15:06
amoralej	anyway we should move it to -testing instead of build-el8	15:06
chandankumar	it would be a useful metric for us	15:06
amoralej	no, mirror sync should be automatic	15:07
chandankumar	amoralej, build-le8 where?	15:07
amoralej	rdoinfo	15:07
amoralej	https://github.com/redhat-openstack/rdoinfo/blob/master/buildsys-tags/cloud8-openstack-ussuri-el8-build.yml#L169	15:08
chandankumar	amoralej, ah ok	15:08
ykarel	alteast why build-deps enabled in periodic jobs?	15:08
ykarel	ideally shouldn't be needed though	15:09
ykarel	unless and until packages are build	15:09
weshay\|ruck	[09:06:14] <mgoddard> weshay\|ruck: FWIW, I think we have two candidate solutions for kayobe at least	15:09
weshay\|ruck	[09:06:54] <mgoddard> weshay\|ruck: bodge: uninstall virtualenv3, reinstall virtualenv2 https://review.opendev.org/712673	15:09
weshay\|ruck	[09:06:55] <patchbot> patch 712673 - kayobe (stable/train) - WIP: Force python 2 for virtualenvs - 4 patch sets	15:09
weshay\|ruck	[09:07:36] <mgoddard> weshay\|ruck: better: fix in base zuul job: https://review.opendev.org/712694	15:09
weshay\|ruck	[09:07:36] <patchbot> patch 712694 - kayobe (stable/train) - DNM: use base-test for kayobe-base to test drop of... - 1 patch set	15:09
weshay\|ruck	[09:08:59] * sri_ has quit (Ping timeout: 260 seconds)	15:09
weshay\|ruck	arxcruz\|rover, zbr ^	15:09
amoralej	yea	15:09
amoralej	ykarel, i'll send a patch to move those two packages to -teting	15:09
amoralej	-testing	15:10
chandankumar	rlandy, weshay\|ruck https://post-office.corp.redhat.com/mailman/listinfo/container-teams for cashing podman people , one more list to follow	15:10
amoralej	weak deps are tricky to catch	15:10
ykarel	amoralej, ack but from where those comings?	15:10
amoralej	https://logserver.rdoproject.org/65/25865/1/check/periodic-tripleo-ci-centos-8-standalone-baremetal-master/8e0f88f/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz	15:10
weshay\|ruck	zbr, arxcruz\|rover <yoctozepto> and mark goddard	15:10
amoralej	installing tripleoclient and ceph-ansible	15:10
amoralej	it's pulling them	15:10
ykarel	ohhk in c7 they didn't enabled it	15:11
ykarel	it's got enabled in c8, and thus the issue got visible	15:12
rlandy	done	15:12
ykarel	https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L64	15:12
*** holser has quit IRC		15:14
*** holser has joined #oooq		15:15
zbr	arxcruz\|rover: weshay\|ruck : kolla-python-devel approach no1 https://review.opendev.org/#/c/712706/	15:15
chandankumar	ykarel, you are proposing a patch to enable that?	15:16
*** Goneri has joined #oooq		15:18
*** beagles has joined #oooq		15:21
amoralej	chandankumar, i checked that the mirror that failed in that job is updated now	15:25
amoralej	i'd expect recheck to pass	15:25
weshay\|ruck	amoralej, no idea why that would have not been enabled	15:26
rlandy	weshay\|ruck: are there any more changes coming into rocky now?	15:26
rlandy	ie: how often would rocky branch jobs kick?	15:27
weshay\|ruck	rlandy, yes.. i can be moved to the weekender	15:27
rlandy	weshay\|ruck: no for that reason ... https://review.opendev.org/#/c/712294/4/zuul.d/multinode-jobs.yaml	15:28
*** holser has quit IRC		15:28
marios	weshay\|ruck: rlandy: i thought rocky is already in wednesday weekend no?	15:28
marios	rocky and queens	15:28
rlandy	not sure if my comment here is needed	15:28
rlandy	marios: ack - was asking for branch excludes	15:28
*** holser has joined #oooq		15:29
marios	rlandy: so have any of our pipelines kicked since the unpin i don't see one running in https://review.rdoproject.org/zuul/status	15:29
rlandy	marios: look at the component pipelines	15:30
weshay\|ruck	marios, security and octavia aare running	15:30
rlandy	we have to wait on the zuul queues	15:30
marios	rlandy: ah network	15:30
weshay\|ruck	and network	15:30
*** holser has quit IRC		15:30
marios	component-network	15:30
rlandy	sec and octavia might not have new code yet	15:30
marios	NICE so it was green ?	15:30
rlandy	just rekicked baremetal that passed delorean builds	15:31
weshay\|ruck	marios, I don't think we have new packages yet	15:31
rlandy	marios: in progress now	15:31
marios	periodic-tripleo-ci-centos-8-scenario007-standalone-network-mastersuccess	15:31
weshay\|ruck	can look	15:31
rlandy	weshay\|ruck: we have for baremetal	15:31
marios	rlandy: yeah they are all success though	15:31
rlandy	see the testproject	15:31
weshay\|ruck	but yes.. all green so far	15:31
marios	weshay\|ruck: ah i see	15:31
marios	k	15:31
marios	weshay\|ruck: re 17:31 < weshay\|ruck> marios, I don't think we have new packages yet	15:31
rlandy	weshay\|ruck: we will have for tripleo	15:31
rlandy	when that kicks	15:31
rlandy	25865	15:31
weshay\|ruck	meh.. tripleo	15:31
rlandy	weshay\|ruck: marios: ^^	15:31
rlandy	baremetal pipeline rekicked through testproject	15:32
rlandy	testing unpinned	15:32
rlandy	https://trunk.rdoproject.org/centos8-master/queue.html	15:32
rlandy	that queue is cleared now	15:33
weshay\|ruck	ah cool	15:33
weshay\|ruck	heh.. https://review.rdoproject.org/r/#/c/25863/	15:33
weshay\|ruck	skip unit tests	15:33
rlandy	chandankumar: ^^ are all components now rebuilt with dlrn?	15:33
rlandy	queue is clear	15:33
chandankumar	rlandy, yes	15:34
*** ykarel is now known as ykarel\|away		15:34
rlandy	chandankumar: so all component pipelines will pick up unpinned content?	15:35
*** holser has joined #oooq		15:35
chandankumar	rlandy, yes	15:35
rlandy	weshay\|ruck: marios: ^^ things are getting real now	15:35
marios	/me prepares the hazmat suit	15:36
rlandy	so swift will be a real unpinned run	15:36
rlandy	marios: you mean you aren't wearing your hazmat suit anyways due to corona?	15:36
marios	rlandy: yes of course typing is a little tricky but still safety first	15:37
rlandy	marios: always	15:37
marios	keyboard wrapped in celphane	15:37
rlandy	lol	15:37
rlandy	sanitize your cell phone?	15:37
rlandy	New York subway is a ghost town	15:37
marios	honestly i actually sanitized it like ~1 hour ago for the 2nd time today (cellphone)	15:38
chandankumar	rlandy, first sanitize hand	15:38
*** TrevorV has joined #oooq		15:39
rlandy	they posted a picture of grand central station in Manhattan - nobody there ... so crazy	15:39
zbr	arxcruz\|rover: weshay\|ruck MUST READ (also check time) https://review.opendev.org/#/c/712107/	15:40
weshay\|ruck	lolz	15:41
marios	rlandy: what time swift runs?	15:41
* marios fetches definitions		15:41
rlandy	shortly	15:41
marios	in 20 mins	15:41
marios	https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/upstream.yaml#L371	15:42
marios	1600 utc	15:42
rlandy	- time: '30 16 * * *'	15:42
weshay\|ruck	rlandy, chandankumar marios https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001&result=success	15:42
weshay\|ruck	:)	15:42
rlandy	lovely	15:42
rlandy	weshay\|ruck: chandankumar: marios: my nephew school closed and they sent back all the guys from America - so he is hanging out with me today - going to take him to the lunch and then to the airport	15:43
rlandy	back in about 1.5 hours	15:44
marios	ah 1630 not 1600 still in a bit anyway	15:44
rlandy	hoping for no explosions	15:44
rlandy	marios: we can make those more frequent if we need to	15:44
rlandy	security might be a real run	15:44
rlandy	25865,1	15:45
rlandy	^^ pls keep an eye on that	15:46
*** rlandy is now known as rlandy\|afk		15:46
marios	rlandy\|afk: ack https://review.rdoproject.org/r/#/c/25865/	15:46
rlandy\|afk	marios: yep it's the rerun of the baremetal pipeline after unpinning	15:47
rlandy\|afk	running now	15:47
weshay\|ruck	marios, panda fyi.. skydive patch should merge with in the hour	15:48
panda	weshay\|ruck: ok, the change as been checked out in promoter server	15:51
panda	weshay\|ruck: currently the master.ini is modified for the tests	15:51
weshay\|ruck	panda, rock on	15:51
*** dtantsur\|brb is now known as dtantsur		15:56
*** bogdando has quit IRC		15:57
*** TrevorV has quit IRC		16:27
*** TrevorV has joined #oooq		16:28
*** dpawlik has quit IRC		16:28
*** dpawlik has joined #oooq		16:28
panda	weshay\|ruck: gate queue restarted	16:29
weshay\|ruck	doh.. there is some missing package	16:29
weshay\|ruck	just bitched about it	16:29
weshay\|ruck	[10:22:40] <weshay\|ruck> gate keeps failing on 2020-03-12 15:15:16 \| msg: 'Failed to download packages: Cannot download noarch/python3-cssselect-0.9.2-13.el8.noarch.rpm: All mirrors were tried'	16:29
weshay\|ruck	meh	16:29
marios	component swift just started	16:30
*** TrevorV has quit IRC		16:31
*** TrevorV has joined #oooq		16:32
*** tesseract has quit IRC		16:40
chandankumar	weshay\|ruck, regrarding csselect amoralej is working on that	16:42
weshay\|ruck	chandankumar, is there a bug?	16:42
chandankumar	weshay\|ruck, I think it was mirror issue rlandy\|afk recheck will solve the issue	16:43
weshay\|ruck	https://bugs.launchpad.net/tripleo/+bug/1867177	16:44
openstack	Launchpad bug 1867177 in tripleo "Cannot download noarch/python3-cssselect-0.9.2-13.el8.noarch.rpm" [Critical,Triaged]	16:44
*** udesale_ has quit IRC		16:49
amoralej	chandankumar, weshay\|ruck still hitting it?	16:59
amoralej	chandankumar, weshay\|ruck could i find all occurrences of this issue in some logstash or something like that?	17:05
*** sshnaidm is now known as sshnaidm\|afk		17:07
marios	rlandy\|afk: weshay\|ruck: getting ready to take off in a short while.. been watching component-swift swift scen2 standalone deploying @ https://review.rdoproject.org/zuul/stream/cb5fe0a3cf6d4ac392201c4c659b134f?logfile=console.log & * https://review.rdoproject.org/r/#/c/25865/ovb fs1 runnint @ https://review.rdoproject.org/zuul/stream/409dadc8ba30468ea1e3257980381926?logfile=console.log	17:12
marios	both ongoing	17:12
chandankumar	amoralej, i will wait for rlandy results	17:13
marios	we'll get some more runs while the different pipes kick and we can see where we're at tomorrow	17:13
chandankumar	let me check	17:13
marios	weshay\|ruck: panda: so we gonna merge that then ? https://review.rdoproject.org/r/#/c/24774/	17:16
marios	weshay\|ruck: panda: or run some more on the old promoter first. did you get the container push today panda?	17:16
marios	panda: lets talk in our morning tomorrow and take from there	17:16
panda	marios: skydive patch has not merged yet, and without it I can't test the full run	17:17
panda	marios: sure	17:17
marios	panda: needs votes or gate issue	17:17
panda	marios: gates delays	17:17
marios	panda: while you're waiting for the aircraft to arrive would you mind looking at v11 https://review.rdoproject.org/r/25782	17:17
marios	thanks	17:18
pojadhav	weshay\|ruck, today finally openstack exam cleared :-)	17:18
panda	marios: gate is broken	17:18
panda	marios: correction, gate is broken not delayed	17:18
panda	marios: all standalone jobs failing	17:18
marios	panda: ack thanks i see your review will address tomorrow	17:19
panda	marios: yep, ping me tomorrow if you have any doubt	17:19
weshay\|ruck	pojadhav, !!	17:19
weshay\|ruck	NICE	17:19
weshay\|ruck	CONGRATS	17:19
weshay\|ruck	pojadhav++	17:19
*** rlandy\|afk is now known as rlandy		17:19
pojadhav	weshay\|ruck, thanks :)	17:19
rlandy	back	17:19
rlandy	chandankumar: hey - what did I miss?	17:20
marios	rlandy: your test patch still ongoing * https://review.rdoproject.org/r/#/c/25865/ovb fs1 runnint @ https://review.rdoproject.org/zuul/stream/409dadc8ba30468ea1e3257980381926?logfile=console.log	17:20
marios	rlandy: but am going to leave was watching that ...	17:20
marios	rlandy: also swift component	17:20
marios	* component-swift swift scen2 standalone deploying @ https://review.rdoproject.org/zuul/stream/cb5fe0a3cf6d4ac392201c4c659b134f?logfile=console.log	17:20
rlandy	marios: ack thanks	17:20
chandankumar	rlandy, sorry what the testproject link?	17:21
rlandy	chandankumar: 25865,1	17:21
rlandy	still running	17:21
chandankumar	amoralej, https://review.rdoproject.org/r/#/c/25865/	17:22
rlandy	chandankumar: but all the component kicking now should give us feedback	17:23
chandankumar	rlandy, I will keep an eye on that tomorrow mornoing	17:23
rlandy	https://review.rdoproject.org/zuul/stream/7f62ff96102e44298b14cc95b3422569?logfile=console.log	17:23
rlandy	ugh - this keeps hanging on introspection	17:24
amoralej	chandankumar, it seems jobs are passing this run	17:24
rlandy	that might be a legit problem	17:24
rlandy	updates the images	17:24
weshay\|ruck	rlandy, yes.. it's a real issue has a bug	17:25
rlandy	openstack-component-tripleo should kick shorty	17:25
weshay\|ruck	rlandy, we should hold a node	17:25
rlandy	weshay\|ruck: but we will be able to see the virt-customize	17:25
weshay\|ruck	because there's been a lot of confusion around that bit	17:26
rlandy	weshay\|ruck: so I am not sure why it's failing	17:26
rlandy	weshay\|ruck: I think we will have to do this on the next run - so I can set stack delete to false	17:26
rlandy	let's see what this set of logs shows	17:26
*** marios is now known as marios\|out		17:27
weshay\|ruck	amoralej, chandankumar that issue w/ cssselect is blowing out the gate	17:27
weshay\|ruck	https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_714/712517/2/gate/tripleo-ci-centos-8-scenario001-standalone/714c1ab/logs/undercloud/home/zuul/install_packages.sh.log	17:27
rlandy	https://logserver.rdoproject.org/50/25550/7/check/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master/4642770/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz	17:28
rlandy	hmmm .... marios: fs030 results	17:28
rlandy	^^	17:28
rlandy	same deal - can't pull local containers	17:28
rlandy	weirdness	17:28
*** dtantsur is now known as dtantsur\|afk		17:34
weshay\|ruck	rlandy, can't pull containers?	17:37
weshay\|ruck	rlandy, which one?	17:37
rlandy	unable to pull 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:8ad99c83b890c0b8b265888a7b774c4e-updated-20200312145841: unable to pull image: Error initializing source docker://192.168.24.1:8787/tripleomaster/centos-binary-mariadb:8ad99c83b890c0b8b265888a7b774c4e-updated-20200312145841: error pinging docker registry 192.168.24.1:8787: Get https://192.168.24.1:8787/v2/: http: server gave HTTP response to HTTPS client	17:38
rlandy	I feel like we have seen this error before	17:38
rlandy	maybe it's using the wrong hash	17:39
amoralej	weshay\|ruck, i just refreshed that cache	17:40
weshay\|ruck	rlandy, http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario007-multinode-oooq-container&branch=master	17:40
weshay\|ruck	amoralej, thanks	17:40
amoralej	more failures?	17:40
weshay\|ruck	ya.. there's a bunch	17:40
weshay\|ruck	same package	17:40
amoralej	where can i see it?	17:40
weshay\|ruck	bunch of jobs	17:40
weshay\|ruck	zuul.openstack.org	17:40
amoralej	i want to check the failing mirrors	17:40
weshay\|ruck	queue: tripleo	17:40
weshay\|ruck	amoralej, this may help understanding which mirrors http://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=61	17:41
*** marios\|out has quit IRC		17:41
rlandy	we have a lot of success there	17:41
weshay\|ruck	but it takes a few to refresh and pull info from zuul so not 100% up 2 date	17:41
weshay\|ruck	rlandy, I have yet to see any container updates	17:41
amoralej	ok	17:41
rlandy	so it's legit?	17:42
weshay\|ruck	so it's not legit	17:43
weshay\|ruck	https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-common-master/6b98bbf/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz	17:43
weshay\|ruck	for example	17:43
rlandy	let's try that one again	17:45
weshay\|ruck	rlandy, should see something like https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e2a/712571/2/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/e2ab845/logs/undercloud/var/log/tripleo-container-image-prepare-ansible.log	17:45
ykarel\|away	rlandy, that https error happens when registries.conf is not updated	17:45
ykarel\|away	rlandy, /etc/containers/registries.conf	17:46
ykarel\|away	registeries.inecure sectoin	17:46
rlandy	checking	17:46
ykarel\|away	section	17:46
weshay\|ruck	no new rpms yet	17:46
rlandy	[registries.insecure]	17:48
rlandy	registries = ['undercloud.ctlplane.localdomain', '192.168.24.1', '192.168.24.3']	17:48
ykarel\|away	rlandy, ^^ on overcloud?	17:48
ykarel\|away	see overcloud	17:48
rlandy	[registries.insecure]	17:48
rlandy	registries = []	17:48
ykarel\|away	as the failure is during overcloud deploy	17:48
rlandy	yeah - missing	17:49
ykarel\|away	ok that's the issue	17:49
ykarel\|away	confirm with other passing multinode job	17:49
rlandy	will have to see how this passed before	17:49
rlandy	periodic-tripleo-ci-centos-8-standalone-baremetal-master success	17:50
ykarel\|away	yes /me checked in fs010 it's there https://44edc35488597c9e08e1-f58e754018cda62a73be8e2f51d66f4e.ssl.cf5.rackcdn.com/712294/4/check/tripleo-ci-centos-8-containers-multinode/162576f/logs/subnode-1/etc/containers/registries.conf	17:50
rlandy	2020-03-12 16:10:05.861078 \| primary \| TASK [overcloud-prep-images : Prepare the overcloud images for deploy] *********	17:50
rlandy	2020-03-12 16:10:05.879712 \| primary \| Thursday 12 March 2020 16:10:05 +0000 (0:00:01.796) 0:00:17.406 ********	17:50
ykarel\|away	so something is wrong with this specific job	17:50
rlandy	^^ hanging there as well	17:50
rlandy	probably definition - look at it	17:50
*** derekh has quit IRC		18:01
*** Trevor_V has joined #oooq		18:07
*** TrevorV has quit IRC		18:10
rlandy	weshay\|ruck: need anything from me? otherwise going to rekick bm and hold nodes	18:14
weshay\|ruck	rlandy, which jobs are you going to hold?	18:15
*** holser has quit IRC		18:15
weshay\|ruck	don't worry about 39	18:15
weshay\|ruck	f.. it for now	18:15
rlandy	25865 and 24995	18:15
* weshay\|ruck looks		18:15
rlandy	both are hanging on prep-images	18:15
rlandy	see testproject on both	18:15
weshay\|ruck	rlandy, +1	18:15
weshay\|ruck	rlandy, and let's get on it and see if the containers are updated	18:16
rlandy	weshay\|ruck: you mean from the undercloud box?	18:17
weshay\|ruck	aye	18:17
rlandy	k - going to get my keys on that box	18:19
rlandy	tristanC: hi - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master is running in job 25865 - could you put my or weshay\|ruck's keys on that box so we can see if conatiners updated?	18:20
rlandy	also 24995 is that's possible	18:21
weshay\|ruck	github.com/weshayutin.keys	18:22
weshay\|ruck	rlandy, microsoft nuked my key	18:22
rlandy	weshay\|ruck: lol - it's started	18:23
rlandy	I think we must have hit the update	18:23
rlandy	if they are both hanging	18:23
rlandy	weshay\|ruck: octavia and security will hit the integration pipeline	18:24
rlandy	worth it to rerun failing tests in other components?	18:24
rlandy	periodic-tripleo-ci-centos-8-standalone-network-master	18:25
weshay\|ruck	rlandy, no..	18:26
weshay\|ruck	jobs were hanging inconsistently.. bad assumption	18:26
rlandy	https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-master/e35bc8f/job-output.txt - tempest	18:29
amoralej	weshay\|ruck, i've checked the failed jobs i've found in http://zuul.openstack.org	18:32
amoralej	one of the rax mirrors were still out of sync	18:33
amoralej	i've synced	18:33
rlandy	weshay\|ruck: http://tripleo-cockpit.usersys.redhat.com/d/2tivP9BWz/component-pipeline?orgId=1 missing swift	18:33
amoralej	i'll keep check jobs starting recently	18:35
rlandy	amoralej: is it possible that swift had no new content?	18:36
amoralej	let me check	18:37
amoralej	you mean after unpin?	18:37
amoralej	it was not pinned probably	18:38
amoralej	swift is still supporting py2, iirc	18:38
amoralej	i'm double checking	18:38
amoralej	yeah, unpinned	18:39
rlandy	amoralej: yep - because the latest run didn't pick up any new content	18:39
rlandy	ok - so that's possible then	18:39
amoralej	yes, it has not been pinned	18:39
rlandy	I didn't realize anything was not pinned - ok	18:39
amoralej	so easily no new commits	18:39
* rlandy breathes again		18:40
amoralej	in fact swift has been the only project that keeps some py2 support iirc	18:40
rlandy	amoralej: http://tripleo-cockpit.usersys.redhat.com/d/2tivP9BWz/component-pipeline?orgId=1 tracks promotions	18:40
rlandy	so we can see if new content was picked up	18:40
rlandy	if the tests pass, we expect a promotion per component	18:41
rlandy	the promote jobs run every two hours	18:41
amoralej	that's pretty cool	18:41
rlandy	looks like octavia passed and promoted	18:41
rlandy	swift is actually missing there	18:41
rlandy	just told weshay\|ruck	18:41
rlandy	security still running - expect that to promote	18:42
amoralej	rlandy, last merge in swift is from 5 days ago https://review.opendev.org/#/q/project:openstack/swift+status:merged	18:43
amoralej	rlandy, that means pipeline for swift was never triggered?	18:43
amoralej	last build was promoted before creating pipelies or something like that?	18:43
rlandy	tripleo-ci-testing/2020-03-07 01:37	18:43
rlandy	that's right ...	18:44
rlandy	swift did trigger ... it's just missing from the dashboard ... getting	18:44
rlandy	https://review.rdoproject.org/zuul/builds?pipeline=openstack-component-swift	18:44
rlandy	amoralej: ^^	18:44
rlandy	last time it promoted was ...	18:44
rlandy	https://logserver.rdoproject.org/openstack-promote-component/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-component-swift-promote-to-promoted-components/6569146/job-output.txt	18:47
*** jpena is now known as jpena\|off		18:51
*** Trevor_V has quit IRC		18:51
*** dsneddon has joined #oooq		19:02
weshay\|ruck	rlandy, ocatavia did?	19:05
* weshay\|ruck looks		19:05
rlandy	weshay\|ruck: looks like it	19:05
weshay\|ruck	rlandy, we should create previous-promoted-components	19:05
rlandy	weshay\|ruck: what would we do with that?	19:05
weshay\|ruck	rlandy, those rpms are el.8	19:06
rlandy	swift wasn't pinned so that's two out of 15 in	19:06
weshay\|ruck	oh wait	19:07
* weshay\|ruck looks at commit hash		19:07
weshay\|ruck	oh ya https://opendev.org/openstack/octavia/commit/801b99ed0069f438646c106389bb9b84f37fb619	19:07
weshay\|ruck	it's good.. it's updated	19:07
rlandy	shehecheyanu - our first component	19:07
weshay\|ruck	https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-octavia-master/5675ab6/logs/undercloud/var/log/tripleo-container-image-prepare-ansible.log.txt.gz	19:08
weshay\|ruck	IT WORKED	19:08
weshay\|ruck	puppet-octavia noarch16.1.1-0.20200310145254.32bb808.el8 delorean-current 61 k	19:08
rlandy	ugh - security hit the timeout hang	19:10
rlandy	that should have promoted	19:10
weshay\|ruck	rlandy, but	19:10
weshay\|ruck	chat w/ me	19:10
weshay\|ruck	https://meet.google.com/dtb-oexi-uyj?authuser=1	19:10
amoralej	i don't see any new cssselect issue in jobs starting recently	19:23
amoralej	i'm leaving	19:23
amoralej	leave a comment here to my nick if you find more	19:23
amoralej	i'll check later	19:23
amoralej	weshay\|ruck, ^	19:23
*** amoralej is now known as amoralej\|off		19:24
weshay\|ruck	amoralej\|off, thanks	19:27
weshay\|ruck	rlandy, https://trunk.rdoproject.org/centos8-master/component/octavia/component-ci-testing/delorean.repo	19:33
weshay\|ruck	rlandy, https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-octavia-master/5675ab6/logs/subnode-1/etc/yum.repos.d/quickstart-centos-base.repo.txt.gz	19:34
*** jfrancoa has quit IRC		19:36
*** jfrancoa has joined #oooq		19:51
*** Goneri has quit IRC		20:18
*** jfrancoa has quit IRC		20:51
zbr	ansible team has humous, codename for 2.10: https://github.com/ansible/ansible/blob/devel/lib/ansible/release.py#L24	20:58
zbr	with openstack we lost the opportunity to name it Vendetta :p	20:59
rlandy	weshay\|ruck: https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tripleo-master/c9ca410/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz	21:08
rlandy	https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tripleo-master/c9ca410/logs/undercloud/var/log/extra/package-list-installed.txt.gz	21:08
rlandy	^^ you can see the updates there	21:08
rlandy	see the @tripleo repos	21:08
weshay\|ruck	rlandy, hey	21:12
weshay\|ruck	sorry had to run to the bank	21:12
rlandy	weshay\|ruck: you didn't miss much	21:13
rlandy	weshay\|ruck: I just compared the tripleo pipeline for updates	21:13
rlandy	weshay\|ruck: if you look at the /package-list-installed - you will see @tripleo	21:14
rlandy	that is the repo the component provides	21:14
weshay\|ruck	k.. did tristan hold that new node?	21:15
* weshay\|ruck looks		21:15
rlandy	idk	21:15
weshay\|ruck	rlandy, ok.. cloudops is in tempest	21:26
rlandy	https://logserver.rdoproject.org/71/25871/1/check/periodic-tripleo-ci-centos-8-scenario002-standalone-cloudops-master/b11a098/logs/undercloud/var/log/extra/package-list-installed.txt.gz	21:29
rlandy	weshay\|ruck: ^^	21:30
rlandy	no cloudops	21:30
weshay\|ruck	?	21:30
rlandy	source_image: 192.168.24.1:8787/tripleomaster/centos-binary-neutron-metadata-agent-ovn:3621159be13b41f8ead2e873b357f4a5	21:31
rlandy	target_image: 192.168.24.1:8787/tripleomaster/centos-binary-neutron-metadata-agent-ovn:3621159be13b41f8ead2e873b357f4a5	21:31
weshay\|ruck	not sure what we're talking about	21:31
rlandy	weshay\|ruck: git to go to dentist	21:31
rlandy	got	21:32
weshay\|ruck	k	21:32
rlandy	back in hour or so	21:32
*** rlandy is now known as rlandy\|afk		21:32
*** amoralej\|off is now known as amoralej		22:00
*** jbadiapa has quit IRC		22:06
*** jbadiapa has joined #oooq		22:09
*** jbadiapa has quit IRC		22:37
*** rlandy\|afk is now known as rlandy		22:55
rlandy	weshay\|ruck: back - how are things looking?	22:56
weshay\|ruck	rlandy, the node we held.. actually worked	22:56
weshay\|ruck	meh	22:56
rlandy	good	22:56
rlandy	checking cloudops	22:56
weshay\|ruck	rlandy, I did.. let's chat	22:56
weshay\|ruck	https://meet.google.com/tve-rfyk-fnc?authuser=1	22:57
rlandy	weshay\|ruck: https://logserver.rdoproject.org/71/25871/1/check/periodic-tripleo-ci-centos-8-scenario001-standalone-cloudops-master/ef0ce8c/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz	23:07
*** Goneri has joined #oooq		23:08
rlandy	weshay\|ruck: https://logserver.rdoproject.org/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-cloudops-master/144cb40/logs/undercloud/home/zuul/ansible.log.txt.gz	23:22
*** amoralej is now known as amoralej\|off		23:29
rlandy	weshay\|ruck: https://logserver.rdoproject.org/71/25871/1/check/periodic-tripleo-ci-centos-8-scenario001-standalone-cloudops-master/ef0ce8c/logs/undercloud/var/log/	23:34
weshay\|ruck	rlandy, https://hackmd.io/twHmV2f2SGCMvfHjLMoGoA	23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!