Friday, 2020-02-14

*** jmasud has quit IRC00:18
*** tosky has quit IRC00:51
*** rfolco has quit IRC01:32
*** jmasud has joined #oooq02:21
*** apetrich has quit IRC03:13
*** rlandy|bbl is now known as rlandy03:57
*** jmasud has quit IRC04:13
*** jmasud has joined #oooq04:14
*** ykarel|away is now known as ykarel04:49
*** udesale has joined #oooq05:04
*** raukadah is now known as chandankumar05:36
*** skramaja has joined #oooq05:52
*** ratailor has joined #oooq05:53
*** marios has joined #oooq05:58
*** marios is now known as marios|ruck05:58
*** ykarel is now known as ykarel|afk06:48
*** saneax has joined #oooq06:53
chandankumarsshnaidm|afk, Hello07:56
chandankumarsshnaidm|afk, https://review.rdoproject.org/r/24929 vexxhost image server push and pull is working fine for rhel807:56
chandankumarI will prepre the final patch07:56
chandankumardpawlik_off, ^^07:57
chandankumarjpena|off, ^^07:57
*** dpawlik_off is now known as dpawlik07:58
dpawlikchandankumar, cool! THanks for info :)07:58
chandankumardpawlik, I will prepare final patches and we can merge on monday in sequence07:58
dpawlikack08:02
*** amoralej|off is now known as amoralej08:10
*** holser has joined #oooq08:18
*** holser has quit IRC08:19
*** holser has joined #oooq08:20
*** ykarel|afk is now known as ykarel|lunch08:20
*** tesseract has joined #oooq08:21
*** tosky has joined #oooq08:36
*** jpena|off is now known as jpena08:46
marios|ruckanyone know what is the max we can set for timeout: in rdo-jobs? ykarel|lunch or jpena maybe or chandankumar (sorry for noise if you don't know just ignore we have fs1 issue...)09:05
marios|ruckis it 5 hours? (18000) or can we increase it. i can't see something with greater in rdo-jobs currently09:06
jpenamarios|ruck: I don't see any upper limit in the zuul documentation09:07
*** jfrancoa has joined #oooq09:08
marios|ruckjpena: ack thanks i recall someone mentioning an 'upper' in rdo specifically but maybe am wrong thanks09:08
*** bogdando has joined #oooq09:21
*** derekh has joined #oooq09:32
*** apetrich has joined #oooq09:37
chandankumarmarios|ruck, fs01 issue is fixed now09:43
chandankumarmarios|ruck, is it still coming?09:44
marios|ruckchandankumar: timeout?09:45
marios|ruckchandankumar:         * https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/2269731/job-output.txt09:45
marios|ruck            2020-02-14 06:07:36.772572 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run09:45
chandankumarmarios|ruck,yes fs01 centos7 timeout got fixed wit09:45
chandankumarlet me check09:45
marios|ruckchandankumar: do we have bug for it? i didn't see anything on the etherpad today09:46
chandankumarmarios|ruck, I have fixed for check job09:46
chandankumarmarios|ruck, nope09:46
marios|ruckchandankumar: where is the fix please09:46
marios|ruckchandankumar: i am trying that tempest_run_concurrency posted https://review.opendev.org/707791 & test https://review.rdoproject.org/r/2494309:46
marios|ruckchandankumar: it is *just* timing out looks like. so test are green and it hits 5hrs basically just needed to collect_logs09:47
chandankumarmarios|ruck, https://review.opendev.org/#/c/707147/09:47
marios|ruckchandankumar: no this is timeout issue i think different? ^09:47
marios|ruckchandankumar: in *periodic*09:48
marios|ruckchandankumar: i am going to file new bug for it09:48
marios|ruckchandankumar: but we need to either bump timeout a bit more or hope tempest_run_concurrency or something similar might help instead09:48
chandankumarmarios|ruck, yes, it is different, TASK [os_tempest : Execute tempest tests] got time out but https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/2269731/logs/undercloud/var/log/tempest/stestr_results.html.gz there is tempest result file09:48
chandankumarI am not sure how09:48
marios|ruckchandankumar: ack right the jobs are green09:49
marios|ruckchandankumar: so it either needs to complete faster (with concurrency if it helps() or we need to extend for half hour09:49
chandankumarmarios|ruck, yes I would say to go with concurrency09:49
marios|ruckchandankumar: ok thanks confirm this is new issue am looking into that will file bug lets see how concurrency goes09:49
chandankumarmarios|ruck, but in fs01, controller are more there, only one compute there09:51
marios|ruckchandankumar: yeah 3 ctrl 1 comp09:51
chandankumarlet's see increasing the concurrency fixes it or not09:52
marios|ruckchandankumar: k09:52
marios|ruckchandankumar: i used same magic number as arxcruz09:52
*** ykarel|lunch is now known as ykarel09:52
ykarelmarios|ruck, jpena hmm there is a param iirc job-max-timeout09:53
ykarelbut let me check09:53
marios|ruckykarel: ack thanks, i want to say 6 hours but not sure. hopefully we can avoid it but good to know. don't dig too much there if it isn't obvious thanks ykarel09:53
marios|ruckykarel: no actually that is nonsense... am pretty sure we've had jobs running more than 6 hours so can't be that.09:54
ykarelmarios|ruck, it's max-job-timeout: 1800009:54
*** beagles has quit IRC09:54
ykarelalso we should avoid running longer jobs if possible09:55
ykarelhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul/rdo.yaml#L409:55
chandankumarmarios|ruck, jpena https://bugs.launchpad.net/tripleo/+bug/1863224 please have a look09:59
openstackLaunchpad bug 1863224 in tripleo "podman-package-rhel-8 job is giving post_failure after RHEL-8.1 movement" [Critical,Confirmed]09:59
marios|ruckykarel: ah ok thanks... that is good to know.. we already bumped it to 18000 for fs110:00
ykarelmarios|ruck, ack10:01
chandankumarmarios|ruck, weshay|ruck and I were discussing last night about podman package failure, we are filing the bug so that prodchain team can look into that10:01
jpenaamoralej: about https://bugs.launchpad.net/tripleo/+bug/1863224, some packages are missing from RHUI, it seems (device-mapper-devel glibc-static go-compilers-golang-compiler gpgme-devel libassuan-devel) after moving to RHEL 8.1. Does that ring any bell?10:02
openstackLaunchpad bug 1863224 in tripleo "podman-package-rhel-8 job is giving post_failure after RHEL-8.1 movement" [Critical,Confirmed]10:02
amoralejjpena, it was working before?10:03
jpenaamoralej: yep, but after switching to rhel 8.1 it started to fail10:03
amoralejcould i see logs of working jobs?10:03
chandankumaramoralej, https://logserver.rdoproject.org/openstack-regular/github.com/containers/libpod/master/podman-package-rhel-8/5665b5b/job-output.txt10:05
chandankumaramoralej, last passing was 6 Feb10:06
chandankumaramoralej, https://review.rdoproject.org/zuul/builds?job_name=podman-package-rhel-8&result=SUCCESS10:07
amoralejoh, they were pulled from codeready10:07
amoralejlet me check10:07
chandankumaramoralej, Do you mean after enabling rhui, we need to enable codeready?10:08
amoralejcodeready is a rhui repo10:09
amoralejlet me check10:09
chandankumaramoralej, codeready got missing in rhui10:14
chandankumari have added info from passed and failed in bug itself10:15
amoralejyes, it's what i'm checking10:16
amoralejit seems it dissapeared10:16
*** ccamacho has quit IRC10:40
*** matbu has joined #oooq11:00
*** beagles has joined #oooq11:12
*** ykarel is now known as ykarel|afk11:37
*** udesale_ has joined #oooq11:56
*** ccamacho has joined #oooq11:57
*** udesale has quit IRC11:59
weshay|ruckchandankumar, yup.. perfect thank you12:31
weshay|ruckchandankumar, dpawlik re: the image server... is david simard still around? the purge scrijpt is already written.. any purge script can not remove any real FILE that has a soft-link attached...12:32
weshay|ruckto prevent removing images in use..12:32
weshay|ruckif you look at that dir structure and files in the server you'll see what I'm talking about12:33
weshay|rucksoniya29, hello12:33
soniya29weshay|ruck: meeting12:33
soniya29weshay|ruck: hello12:33
weshay|rucksoniya29, I think we have network issue12:36
marios|ruckweshay|ruck: o/ let me know when you want to sync12:37
marios|ruckweshay|ruck: adding stuff to https://etherpad.openstack.org/p/ruckroversprint2212:38
*** jpena is now known as jpena|lunch12:40
*** ykarel|afk is now known as ykarel12:47
*** rfolco has joined #oooq12:47
weshay|ruckmarios|ruck, something very odd is happening in ci.centos12:49
marios|ruck        * Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was12:49
marios|ruck12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')12:50
marios|ruckweshay|ruck: that has been affecting buildah container builds for couple days but intermittent ^^ filing it now12:50
*** ratailor has quit IRC12:50
marios|ruckweshay|ruck: you seeing something else?12:50
*** rlandy has joined #oooq12:50
weshay|ruckmarios|ruck, containers.. yes.. see that.. also someting in ci.centos12:51
weshay|rucksec.. in 1-112:51
marios|ruckweshay|ruck: k np12:51
rlandychandankumar: hey - aren't you out today?12:52
chandankumarrlandy, I am less available on IRC today from right now onwards12:53
chandankumarrlandy, I got the right project for podman dm https://src.osci.redhat.com/rpms/podman/branches?branchname=master12:54
rlandychandankumar: ok - I have the work we set up a while back12:54
rlandyI need to add that repo to the tenant12:54
rlandyrfolco: hi ...13:00
rfolcorlandy, o/13:00
rlandyrfolco: please see questions in https://tree.taiga.io/project/tripleo-ci-board/task/151813:01
rlandyI am nit sure where to go next here ...13:01
rlandybuild out all the components13:01
rlandyor build the integration pipeline jobs13:01
rlandyare we blocking the integration jobs on having all the components running?13:02
rfolcorlandy, we'll need at least a manual component promotion to run the 1st integration job13:03
rlandyrfolco: so we will not have periodic-tripleo-centos-8-master-promote-consistent-to-tripleo-ci-testing EVER built13:04
rlandythe pipeline will only ever take promoted components?13:05
rfolcothis has changed over time13:05
rfolcoso thats why I never merged https://review.rdoproject.org/r/#/c/24511/13:05
*** fmount- has joined #oooq13:05
*** fmount has quit IRC13:06
rfolcowe need that job yes, promote from promoted-components to tripleo-ci-testing13:06
rlandyso what's the ask now13:06
rlandyit wasn't clear to me from what weshay|ruck said yesterday whether we have an old-style integration or not13:07
rfolcorlandy, IMO build the component pipeline, and at the same time run the container build job pushing the containers13:07
rlandyalso there was a mention of image build jobs13:07
rfolcowe'd have in place all components jobs13:08
rlandyrfolco: do you know what the deal is with image build jobs13:08
rlandyI think weshay|ruck said zbr was doing something?13:08
rfolcorlandy, not sure, he said we need it, but I don't know all the requirements13:08
rfolcochandankumar, ^13:09
* rlandy is trying not to ping chandankumar13:09
rlandyhe's attempting a day off13:09
chandankumarrlandy, rfolco no one has started the image build13:09
rlandychandankumar: k - now go :)13:10
chandankumarok13:10
rfolcorlandy, it's a chicken and egg problem. We need component pipeline, which needs container build job from integration pipeline....13:10
rfolcorlandy, chandankumar I am changing https://review.rdoproject.org/r/#/c/24511/ to reflect the new labeling: promoted-components to tripleo-ci-testing13:12
rlandyrfolco: if there is a manual first push of containers the component pipeline will be fine13:12
rlandyugh13:13
rlandyI though promoted components was the new name13:13
rfolcorlandy, we need to run the periodic job and do a push. It was depending on get-hash. Now get-hash is ready, just need to make the switch here https://review.rdoproject.org/r/2484713:13
rfolcorlandy, yes its the new name13:14
rfolcorlandy, the patch is old and needs to be updated... thats why I put it on hold13:14
rlandyrfolco: no worries - we'll redo that work13:15
*** skramaja has quit IRC13:16
*** ysandeep is now known as ysandeep|brb13:16
rfolcorlandy, so now I am enabling the new get-hash to be able to run the periodic version of container build, so we can push to registry13:17
rfolcorlandy, and updating the promoted-components label13:17
rfolcorlandy, after that I can help w/ the other component jobs13:17
rlandyrfolco: it's ok ... jobs are under control for the moment13:19
dpawlikweshay|ruck, hi, David will be absent for some time. I look around on both image server and what I get is just this script: https://review.rdoproject.org/r/gitweb?p=rdo-infra/rdo-infra-playbooks.git;a=blob;f=roles/rdo-infra/image-server/files/prune_tripleo_images.sh;h=5a0d070a1d1b9151713e50285636bd6cafd7e332;hb=HEAD13:19
dpawlikand13:19
dpawlikhttps://review.rdoproject.org/r/gitweb?p=rdo-infra/rdo-infra-playbooks.git;a=blob;f=roles/rdo-infra/image-server/files/prune_octavia_images.sh;h=173c125b796b078bc01d999e71b9f6fe57a2bd86;hb=HEAD13:20
rlandyrfolco: no worries to update https://review.rdoproject.org/r/#/c/24511/13:20
rlandyI'll do it as part of adding all the jobs13:20
dpawlikweshay|ruck, so lets say that I will monitoring free space on the server and if it will be not enough, I will add new volume. And when David will be available I will talk with him about the script13:21
rlandybecause we need a whole centos 8 set13:21
marios|ruckweshay|ruck: arxcruz: chandankumar: periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-masterSUCCESS in 3h 50m 28s  please vote @ https://review.opendev.org/70779113:21
marios|ruckrlandy: sorry can you vote again :D13:22
marios|ruckrlandy: updated the commit message thank you13:22
*** fmount- is now known as fmount13:23
rlandymarios|ruck: done - how13:23
rlandy's the rucking going?13:23
marios|ruckrlandy: ? how which bit13:23
marios|ruckrlandy: ah ok ... i mean everything is broken so you know usual stuff :D13:23
rlandymarios|ruck: business as usual - fun13:24
*** ysandeep|brb is now known as ysandeep13:24
marios|ruckrfolco: panda: can you please vote there https://review.opendev.org/#/c/707791/ we need it for fs1 periodic13:25
marios|ruckrlandy: sorry i updated the commit message again :/ added the bug ^ please don't hit me13:26
rlandymarios|ruck: third time lucky13:27
marios|ruckrlandy: ;) thanks sorry for noise13:27
rlandymarios|ruck: no worries13:29
*** jpena|lunch is now known as jpena13:36
marios|ruckpanda: zbr: damn ERROR! the playbook: /home/zuul/src/review.rdoproject.org/rdo-infra/ci-config/ci-scripts/infra-setup/roles/promoter/molecule/container-push/playbook.yml could not be found https://logserver.rdoproject.org/45/24945/1/check/molecule-container-push/8cd15b0/tox/reports.html13:40
marios|ruckblocking https://review.rdoproject.org/r/#/c/24945/13:40
zbras I said, pin it to 3.0a5 last known good.13:42
zbri plan to fix it this weekend as I am unable to find time during the day, I am still negociating with virtualenv maintainer the fixes and preventive measures.13:42
marios|ruckzbr: ack ok thanks i recall now the conversation from yesterday13:43
zbreverywhere you see a molecule job failing, check the version.13:43
zbri would have removed the newer ones from pypi but i have no rights.13:44
zbrthat is one place where using openstack constraints could have helped13:44
*** ykarel is now known as ykarel|away13:48
rlandymarios|ruck: https://review.rdoproject.org/r/#/c/24512/ - updated and responded to your comments  - thanks for the review13:49
*** sshnaidm|afk has quit IRC13:50
*** sshnaidm has joined #oooq13:50
marios|ruckrlandy: ack thanks for update will check again13:51
*** amoralej is now known as amoralej|lunch13:59
weshay|ruckysandeep, https://17c6cb2f9fa2917c93e9-c8c2a5181a911daab043eb1d43163b4b.ssl.cf2.rackcdn.com/707469/3/check/tripleo-ci-centos-7-containers-multinode/5ffb145/logs/undercloud/home/zuul/containers-prepare-parameter.yaml14:03
*** sshnaidm is now known as sshnaidm|off14:05
weshay|ruckysandeep, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L15-L2514:07
marios|ruckzbr: it already is https://github.com/rdo-infra/ci-config/blob/master/tox.ini#L78-L95      molecule==3.0a514:10
marios|ruckzbr: is there a better one to use ?14:10
zbrno, in that case there is something wrong there, does the playbook really exist?14:11
marios|ruckzbr: well its called converge.yml not playbook.yml @ https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/promoter/molecule/container-push14:12
marios|ruckERROR! the playbook: /home/zuul/src/review.rdoproject.org/rdo-infra/ci-config/ci-scripts/infra-setup/roles/promoter/molecule/container-push/playbook.yml could not be found14:12
marios|ruckzbr: so it should be looking for converge.yml but it is trying to find playbook.yml14:13
zbrno! converga.yml was introduced in newer molecule versions.14:13
zbrit never existed for older ones.14:13
marios|ruckzbr: wtf14:14
marios|ruckzbr: so if we don't pin we have to use converge.yml14:14
zbrin fact the coverage.yml is the reason why I am telling people to avoid newer verions.14:14
marios|ruckzbr: but if we pin we use playbook.yml14:14
weshay|ruckmarios|ruck, want to sync real quick?14:14
marios|ruckzbr: so the how the F***k was that merged https://review.rdoproject.org/r/#/c/24807/ green scenarios, 3 hours ago14:15
marios|ruckzbr:14:15
marios|ruck:/14:15
weshay|ruckhttps://meet.google.com/dkj-vuij-yup?authuser=114:15
marios|ruckweshay|ruck: sure14:15
zbrprobably innovative conditions regarding when to trigger a job, this always caused problems like this. premature optimization of what job is triggered.14:15
zbrlet me know if I need to join too14:16
weshay|ruckmarios|ruck, coming?14:16
marios|ruckweshay|ruck: yeah sorry14:16
*** TrevorV has joined #oooq14:18
*** marios|ruck is now known as marios|ruck|call14:25
*** marios|ruck|call is now known as marios|ruck14:26
*** amoralej|lunch is now known as amoralej14:30
dpawlikchandankumar, so switching image server will be transparent for TripleO team or?14:48
dpawlikchandankumar, you have change the script, playbooks jobs14:49
dpawlikchandankumar, or we still need to announce that?14:49
*** saneax has quit IRC15:03
weshay|ruckok.. out of mtgs15:05
weshay|ruckzbr, hey man.. qq.. so when molecule is reporting it can't find a particular playbook.. that is just a sympton of the current molecule build right?15:09
zbrif is complaining about not finding converge.yml, it means you have wrong version.15:10
zbrit it complains aboutmissing playbook.yml, it means that either file is really missing.15:11
weshay|ruckk15:11
weshay|ruck[WARNING]: log file at /home/zuul/zuul-output/logs/ansible-execution.log is not writeable and we cannot create it, aborting15:11
weshay|ruck    Using /home/zuul/.cache/molecule/promoter/container-push/ansible.cfg as config file15:11
weshay|ruckERROR! the playbook: /home/zuul/src/review.rdoproject.org/rdo-infra/ci-config/ci-scripts/infra-setup/roles/promoter/molecule/container-push/playbook.yml could not be found15:11
weshay|ruck15:11
weshay|ruckso that is the latter15:11
marios|ruckzbr: its the same thing i was askinga bout earlier15:11
marios|ruckweshay|ruck: not clear what the issue is there... we are already pinning to 3.0.5 there15:12
weshay|ruckso if that playbook never existed15:12
marios|ruckweshay|ruck: so apparently we should have playbook.yml not converge.yml (it DID exist.. its just that playbook all got renamed to converge, cos molecule)15:12
weshay|ruckzbr, would it hurt to just write a dummy playbook for that bit?15:12
weshay|ruckzbr, to fix this yesterday I did the following15:12
zbrnope. you can easily put a placeholder one.15:12
weshay|ruckhttps://review.rdoproject.org/r/#/c/24762/29/ci-scripts/infra-setup/roles/promoter/molecule/default/playbook.yml15:13
zbrbut it needs to be real playbook, empty file will not work.15:13
weshay|ruckzbr, right.. k15:13
weshay|ruckwill do the same15:13
weshay|ruckzbr, thanks15:13
weshay|ruckmarios|ruck, updated https://review.rdoproject.org/r/#/c/24945/15:14
weshay|ruckzbr, last question.. is that a bug or just how it works and we should expect to need a dummy playbook.yml in all molecule tests?15:15
zbra role without a playbook makes no sense to me. is like writing a test where you don't want to test anything.15:16
zbrso i think that a playbook is needed all the time.15:16
zbrfor other steps the answer is more nuanced.15:17
weshay|ruckzbr, k.. I guess it's just the expectation of hte name of the playbook15:17
weshay|ruckzbr, is there a molucule command to create the expected directory structure for molecule.. like there is for roles?15:17
weshay|ruckguess I should rtfm15:18
zbr"molecule init role foo", or "molecule init scenario foo", if you are already inside a role.15:18
weshay|ruckah.. cool15:18
weshay|ruckall ^^^15:18
zbrit can also receive few arguments, like which driver to use15:18
zbrsyntax changed a little bit between versions, but "init" is your friend.15:19
zbris only cookiecutter behind15:19
zbrbut very useful to start15:19
*** apetrich has quit IRC15:20
marios|ruckzbr: weshay|ruck: this is a bug15:22
weshay|ruckhow so?15:22
marios|ruckzbr: weshay|ruck: the plays were renamed from playbook to converge.yml15:22
marios|ruckzbr: weshay|ruck: depending on the version of molecule we have15:22
marios|ruckweshay|ruck: currently we have 3.0,5 which *should* be playbook.yml (right zbr?)15:23
zbryes15:23
marios|ruckweshay|ruck: but then how did that merge with green scenario https://review.rdoproject.org/r/24807 3 hours ago15:23
marios|ruckweshay|ruck: i.e. the things we had in molecule scenarios were in files previously called playbook.yml. after molecule version upgraded, they got renamed to converge.yml15:24
marios|ruckweshay|ruck: so its not that the files didn't exist15:24
marios|ruckweshay|ruck: its just that different version of molecule uses different name for the scenario file15:24
zbrthe only job that could have prevented it was NV15:24
zbrcan't we rename it back to playbook.yml?15:25
dpawlikweshay|ruck, chandankumar, raukadah: I prepared a PoC of the email for the community, it is available on the bottom of https://softwarefactory-project.io/etherpad/p/images_migration (backup http://paste.openstack.org/show/789587/).15:26
marios|ruckzbr: weshay|ruck: i think i know why sec (we bumped molecule so everything got renamed, then we downgraded it again)15:27
weshay|ruckdpawlik, perfect... to: openstack-discuss[tripleo]15:27
weshay|ruckmarios|ruck, ya15:27
marios|ruckweshay|ruck: zbr: bumped molecule there https://github.com/rdo-infra/ci-config/commit/9451a994f686e8dc97194c5addf3e3aa66c97de9 with renames15:27
marios|ruckand then rfolco patch pinned it back to 3.0.5 https://github.com/rdo-infra/ci-config/commit/d1662dfc9f4a6635f939d07be69988f2f9f4b87615:28
marios|ruckzbr: so maybe solution is to make it 3.0.6 again?15:29
marios|ruckzbr: weshay|ruck: trying15:30
rlandyweshay|ruck: please ping me when you have a few minutes for a design question - centos 8 periodic15:30
zbri said NO. whatever you do not not use any version newer than 3.0a6, they are all bugged. and I hope i will have enough time to fix during that weekend.15:30
marios|ruckzbr: ok 3.0a6 i think works for us15:30
zbrno, ***3.0a5** is last15:31
marios|ruckzbr: but lets find out15:31
marios|ruckzbr: but you made it 3.06 https://github.com/rdo-infra/ci-config/commit/9451a994f686e8dc97194c5addf3e3aa66c97de915:31
zbri am getting offline because nobody listen to me, you know better feel free to discover what is broken about new releases yourself.15:31
marios|ruckzbr: and that brought the renames from playbook to molecule15:31
marios|ruckzbr: ok then we need to renamed them all back/?15:32
marios|ruckzbr: that is the solution?15:32
zbryes15:32
marios|ruckzbr: ok ...15:32
zbror wait till monday15:32
weshay|ruckmarios|ruck, zbr softlinks would solve this both ways?15:32
zbryeah15:32
weshay|rucklet's do that15:32
zbrsure symlink should do the trick and ease migration.15:33
zbrbe sure is relative15:33
marios|ruckweshay|ruck: k doing15:33
weshay|ruckmarios|ruck, done15:34
marios|ruckweshay|ruck: ah. but does it need to be done for all the scenarios zbr then15:34
weshay|ruckaye15:34
marios|ruckweshay|ruck: but why were the other ones green15:34
* marios|ruck so confused15:34
weshay|ruckmarios|ruck, if you got that that bit.. awesome15:34
zbryep: find . -name converge.yml15:34
marios|ruckzbr: molecule-tripleo-common SUCCESS in 6m 19s15:35
marios|ruckmolecule-delegated-promote-images SUCCESS in 2m 39s15:35
marios|ruckhow15:35
zbrand again, sorry for the trouble caused by that broken release.15:35
marios|ruckzbr: sorry for being grumpy but you're 'the molecule guy'15:35
zbrit went at worst time possible: the same moment virtualenv 20 was released.15:35
marios|ruckso obviously its your fault15:35
marios|ruckbut how are the other scenarios OK with out rename?15:35
marios|ruckbrb15:40
*** jfrancoa has quit IRC15:52
*** dpawlik has quit IRC15:53
*** bogdando has quit IRC15:55
weshay|ruckpanda, you need to rebase https://review.rdoproject.org/r/#/c/24676/15:57
*** Goneri has joined #oooq15:57
weshay|ruckmarios|ruck, pass what ever you have cooked up w/ regards to molecule before you take off15:57
weshay|ruckrlandy, can you please take a shot at a periodic centos-8 containers build job today that uses rfolco new get_hash role15:58
marios|ruckweshay|ruck: haven't cooked anything :) i was just going to update that https://review.rdoproject.org/r/#/c/24945 but you did it15:58
marios|ruckweshay|ruck: i won't rename the others cos they are green15:58
marios|ruckweshay|ruck: didn't understand why that one scenario is special15:58
marios|ruckweshay|ruck: e.g. molecule-container-pushSUCCESS in 8m 31s is green now cos you did a symlink15:59
marios|ruckweshay|ruck: but then why is molecule-tripleo-common and the other also green even though they don't have a converge. i don't know15:59
pandaweshay|ruck: mmmhh, again ? weird. doing16:00
weshay|ruckzbr, marios|ruck please review https://review.rdoproject.org/r/#/c/24676/ before your day ends16:00
weshay|ruckI think that's the next one16:00
marios|ruckweshay|ruck: even though they don't have a *playbook.yml* i mean, they do have converge.yml https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/promoter/molecule/tripleo-common-integration16:00
weshay|ruckmarios|ruck, my thought is this..16:00
weshay|ruckif we're going to bounce around molecule versions...16:01
weshay|rucklet's make the code compat w/ both..16:01
* weshay|ruck assumes at some point zbr will stick his foot up someone's ass in molecule land or fix it himself16:01
marios|ruckweshay|ruck: ok sure i can add the symlink in all16:03
rlandyweshay|ruck: ok16:06
*** TrevorV has quit IRC16:09
marios|ruckpanda: https://review.rdoproject.org/r/#/c/24676/ needs rebase but otherwise lgtm16:11
chandankumarrlandy, I am back for a while16:13
chandankumarrlandy, anything can help.16:13
rlandychandankumar: let's tal parenting16:14
rlandytalk16:14
chandankumarrlandy, what was the issue there?16:15
rlandychandankumar: no issue ... question of design16:15
rlandychandankumar: if centos-7 master is going to be defunct ... is that the right base to parent off?16:16
* rlandy thinks not16:16
chandankumarrlandy, I think we need to create seperate parent of centos-816:16
rlandychandankumar: yea - but with how much duplication?16:17
rlandychandankumar: I guess this is something I'll raise to the group on monday16:17
rlandyit's a question for all the jobs16:17
rlandythink about it :)16:17
*** udesale_ has quit IRC16:17
chandankumarrlandy, what about creating another parent from centos-7 and change the nodeset only -> mark them as a base16:18
* rlandy tries container image builds16:18
chandankumarso we have less duplication16:18
rlandychandankumar: I want to switch the parent where most of the settings are linked16:18
chandankumarrlandy, ok, then let me think on that16:18
rlandyand the add a job that swiches the 7 nodeset so we don't break anything16:19
rlandychandankumar: notes are on your card16:19
rlandychandankumar: lastly16:19
rlandyhttps://review.rdoproject.org/r/#/c/24512/ - Change component promotion labels16:20
rlandyhttps://review.rdoproject.org/r/#/c/24938/ - Add centos 8 standalone component jobs16:20
rlandy^^ please review on monday16:20
rlandythink they are ok to go16:20
rlandylet me know16:20
marios|ruckzbr: weshay|ruck: ok split into two https://review.rdoproject.org/r/24957 for symlinks then https://review.rdoproject.org/r/24945 criteria ontop16:34
*** amoralej is now known as amoralej|off16:37
marios|ruckweshay|ruck: did youw ant to sync again before I go or are you ok with https://etherpad.openstack.org/p/ruckroversprint22 & sync so far16:39
marios|ruckweshay|ruck: starting shutdown sequence16:40
weshay|ruckmarios|ruck, etherpad is fine for today.. thanks16:40
weshay|ruckmarios|ruck, fyi.. chandankumar is going to be migrating image servers on monday16:40
weshay|ruckfrom rdo -> vexx16:40
marios|ruckweshay|ruck: k16:40
weshay|ruckemail w/ warning will be sent16:40
weshay|ruckmarios|ruck, have a good weekend16:40
marios|ruckweshay|ruck: thanks you too & hope its a nice quiet rest of friday ;)16:41
weshay|ruck:)16:45
rlandyrebooting ...16:45
*** rlandy has quit IRC16:45
*** rlandy has joined #oooq16:48
pandamarios|ruck: weshay|ruck rebased and added an extra check to verify that the --force-legacy cmd argument works correctly. In case of problems, you can always launch the dlrnapi_promoter --force-legacy and the old workflow will be used.16:52
weshay|ruckk16:52
weshay|ruckthank you16:52
weshay|ruckmarios|ruck, I may bend the criteria on master depending on what I find there16:53
weshay|ruckit's mostly good16:53
weshay|ruckhrm.. def.. not ok16:53
weshay|ruck2020-02-14 04:35:31,743 p=8305 u=root |  TASK [Generate container puppet configs for step 1] ****************************16:53
weshay|ruck2020-02-14 04:35:31,744 p=8305 u=root |  Friday 14 February 2020  04:35:31 -0500 (0:00:00.316)       0:10:12.728 *******16:53
weshay|ruck2020-02-14 04:35:32,204 p=8305 u=root |  An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: write() argument must be str, not bytes16:53
weshay|ruck2020-02-14 04:35:32,207 p=8305 u=root |  fatal: [undercloud]: FAILED! => changed=false16:53
marios|ruckweshay|ruck: k, i think fs1 and standalone-upgrade should do it there (fs1 fixed with concurrency and remove standalone-upgrade)16:54
* marios|ruck hope16:54
weshay|ruckaye16:54
*** marios|ruck is now known as marios|out17:06
pandazbr: https://logserver.rdoproject.org/76/24676/8/check/molecule-container-push/75bc6f5/tox/reports.html ??17:08
marios|outpanda: see https://review.rdoproject.org/r/2495717:08
* marios|out actually going now bai17:08
*** marios|out has quit IRC17:09
pandaok, merging17:10
pandaI'll have to rebase again.17:10
zbrpanda: read [WARNING]: log file at /home/zuul/zuul-output/logs/ansible-execution.log is not writeable and we cannot create it, aborting17:20
zbrthat counts as critical in my view17:20
pandazbr: yep, merged https://review.rdoproject.org/r/24957 , then I'll rebase17:20
*** rlandy is now known as rlandy|brb17:20
*** tesseract has quit IRC17:29
*** jmasud has quit IRC17:45
*** jmasud_ has joined #oooq17:45
*** rlandy|brb is now known as rlandy17:46
*** jpena is now known as jpena|off17:50
*** holser has quit IRC17:58
*** holser has joined #oooq17:58
rlandyrfolco: ping re: get-hash in build containers18:01
rfolcoo/18:01
rlandyrfolco: afaict, the only place get hash comes into play is in :18:01
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/pre.yaml18:01
*** holser has quit IRC18:01
rlandy^^ the same periodic pre all josb use18:01
rlandyrfolco: ^^ correct?18:02
rfolcorlandy, yes, we need a patch to call get_hash role instead of that bash task18:02
rfolcorlandy, I can do it if you like18:02
rlandyrfolco: no - I can do it18:02
rfolcorlandy, ok :)18:02
rlandybut it will impact ALL the jobs18:02
rlandythe playbook is in config18:03
rlandyso no testing18:03
rlandyOR ...18:03
rlandyI can switch the job to run a different playbook18:03
*** derekh has quit IRC18:03
rfolcorlandy, to be safe you can create a condition.... if its centos8, do the new way18:03
rlandyyeah - probably easiest18:06
rlandyweshay|ruck: rfolco: pls review https://review.rdoproject.org/r/#/c/24959/ WIP: Use get-hash role in centos-8 periodic jobs18:18
rlandythis is config though :(18:20
rlandynot speculative testing18:20
rlandyhold on - they would include rhel 818:22
weshay|ruckrlandy, feel free to merge config18:39
rlandyweshay|ruck: they will impact all periodic18:39
weshay|ruckya.. but       when: ansible_distribution == "CentOS" and ansible_distribution_major_version >= 818:39
weshay|ruckprotects it18:39
weshay|ruckI would think18:39
rlandybrave man18:40
rlandyok - here we go18:40
rlandyweshay|ruck: then I will testproject it with podman push set to false18:40
weshay|ruckk18:40
weshay|ruck:)18:40
rlandyrevert is your friend18:40
rlandyand savior18:40
ykarel|awayrlandy, need |int18:44
ykarel|awayi see that was missed18:44
rlandyah18:44
rlandyykarel|away: thanks - updating18:45
ykarel|awayack18:45
weshay|ruckrlandy, bbiab.. getting tires18:46
rlandyok18:46
weshay|ruckah ya. .|int18:46
weshay|ruckfor sure18:47
rlandyykarel|away, saves us again18:47
weshay|ruckykarel|away, should be getting some kudo's from pweeks hopefully18:47
rlandyhttps://review.rdoproject.org/r/#/c/24959/18:48
rlandyykarel|away: weshay|ruck: ^^ updated18:48
ykarel|awayack18:51
ykarel|awayrlandy, wrt include zuul role: rdo-infra/ci-config, i guess u will be adding to other jobs later?18:54
ykarel|awaywhen other jobs will added for centos818:54
rlandyykarel|away: we'll have to18:55
rlandyotherwise include_role won't work18:55
ykarel|awayhmm i guess u would have to add to some base job,18:55
ykarel|awayotherwise it will be hard to remember to add to which job18:56
rlandyykarel|away: ack in the process of all that work now18:56
ykarel|awayrlandy, ack18:56
rlandyin truth18:56
rlandyall jobs will use the new get-hash once it's stable so this can go into base periodic18:56
rlandyand we'll be done18:56
ykarel|awayrlandy, ack18:56
rlandyhttps://tree.taiga.io/project/tripleo-ci-board/us/143618:57
rlandy^^ bunch of tasks there to deal with centos 8 a18:57
ykarel|awayrlandy, ack18:58
ykarel|awayrlandy, i also realized today that centos 8 container job using last component hash in delorean.repo18:58
ykarel|awayi see some work going on for aggregate hash18:58
ykarel|awayso that need to be adapted for container build job, may be u already have plan for it18:59
rlandyykarel|away: rfolco probably does ... I just stepped into the container work now18:59
ykarel|awayrlandy, ack for reference https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/main.yaml#L32-L3919:00
rlandybut good to note19:00
ykarel|awayi noticed today, so sharing19:00
rlandyykarel|away: thanks - will raise this19:00
rlandyok - let's try this get-hash change19:01
rfolcoso the story is: we have added support to aggregated hash to get-hash, and converted get-hash into a separate role. But still working on https://review.rdoproject.org/r/#/c/24847/ to switch to the new get_hash for periodic jobs.19:01
rlandyrevert is always an option, I guess19:01
rfolcoykarel|away, rlandy ^19:02
rlandyrfolco: k - as long as you're aware19:02
ykarel|awayrfolco, ack for ^^, my point was for calculating hash for centos8 container build https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/main.yaml#L32-L3919:06
ykarel|awaycurrently container are bulid with hash of last component in delorean.repo19:06
ykarel|awaywould be needed to switch that too to aggregated hash thing19:07
ykarel|awayrlandy, ack voting19:08
ykarel|away+W, you can revert if something gets wrong19:08
* ykarel|away out now19:08
rlandycool ... will see what breaks :(19:10
rlandyrfolco: here we go ... https://review.rdoproject.org/zuul/stream/5be9097798d846efb4019a3567dae133?logfile=console.log19:12
rlandyrfolco: broke something19:21
rlandyoh nvm - it was me19:21
rlandyreverting - will fix19:22
rlandyERROR! the role 'get_hash' was not found19:23
rfolcohttps://review.rdoproject.org/r/#/c/24847/19:28
rfolcorlandy, you need to add roles from ci-config19:28
rlandyrfolco: I added them to the wrong parent19:29
rlandyfixing19:29
ykarel|awayhmm wrong job19:29
rfolcorlandy, k here anyway https://review.rdoproject.org/r/#/c/24847/8/zuul.d/tripleo-rdo-base.yaml19:29
ykarel|awayneed to add to job where pre.yml is mentioned19:29
rlandyykarel|away: mentioned in two parents19:30
rlandyfixing19:30
weshay|ruckhey19:31
*** tosky has quit IRC19:36
rlandyhttps://review.rdoproject.org/r/#/c/24962/19:36
rlandyykarel|away: rfolco: ^^19:37
rlandyweshay|ruck: don't look - I broke it19:37
rlandyfix ^^19:37
weshay|ruckwell.. to make an omelette19:37
rlandyyeah - but I rack the pan as well19:37
rlandycrack19:37
rfolcohttps://review.rdoproject.org/r/#/c/24847/19:38
rlandyI place the role on the wrong job19:38
rfolcowhy don't we merge my patch then?19:38
rfolcoanyway, do whatever you need, I'll rebase and adjust19:40
ykarel|awayrlandy, commented19:40
ykarel|awayneed at other place also19:40
rlandyk19:41
* rlandy does a search for that pre now19:41
rlandyit's everywhere19:41
ykarel|awayrlandy, ohhk i see u also added tripleo-ci-base-singlenode-rdo-containers-build-push19:42
ykarel|awayso not need in childs19:42
ykarel|awaybut ok for now19:42
ykarel|awaytaking back my -119:42
rlandyykarel|away: that pre is added in two places19:42
rlandythe parent and the child job19:43
rlandysee ...19:43
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml#L10619:44
rlandyand19:44
ykarel|awaymmm, re looking, looks wrong now19:44
rlandyykarel|away: so really19:44
ykarel|awaytripleo-ci-periodic-base vs tripleo-rdo-base19:44
rlandylooking at the job output  - we have two19:44
rlandyits periodic only19:45
rlandyyour -1 is right19:45
ykarel|awayanyway since you added to parent tripleo-ci-base-singlenode-rdo-containers-build-push, for child it should not be needed19:46
weshay|ruckneed eyes?19:46
ykarel|awayyes please19:46
ykarel|awaymore eyes more power :)19:47
weshay|ruckwe have a tmate?19:47
weshay|ruckykarel|away, it's late for you..  :)19:47
weshay|ruckrlandy,  https://meet.google.com/_meet?authuser=119:47
weshay|ruckykarel|away, welcome to join.. but no pressure..19:47
ykarel|awayhmm almost out19:47
rlandyjoining19:48
rlandyweshay|ruck: do you have meeting started?19:49
weshay|ruckI'm in https://meet.google.com/auo-oxmr-hpw?authuser=119:49
weshay|ruckoops19:49
weshay|rucksorry it didn't paste correctly19:49
rlandyrfolco: just waiting for zuul to be done20:04
weshay|ruckrlandy,20:25
weshay|ruck2020-02-14 20:25:10.147116 | primary |   "msg": "The task includes an option with an undefined variable. The error was: 'promote_source' is undefined\n\nThe error appears to be in '/var/lib/zuul/builds/6d694e01d24b46b5b95035c999b82a76/trusted/project_6/review.rdoproject.org/rdo-infra/ci-config/ci-scripts/infra-setup/roles/get_hash/tasks/get_hash.yaml': line 43, column 3, but may\nbe elsewhere in the file depending on the exact syntax20:25
weshay|ruckproblem.\n\nThe offending line appears to be:\n\n# integration (centos8/rhel8)\n- name: discover the hashes url (integration)\n  ^ here\n"20:25
rlandyI see20:25
rlandyidk what that var should be20:26
rlandythat's in get-hash20:27
weshay|ruckya.. in the role20:27
rlandywhat would I call that now?20:28
weshay|ruckrlandy, so atm..20:28
weshay|ruckthat would tripleo-ci-testing20:28
rlandyyeah20:28
rlandyok - let me define that in the job20:28
weshay|ruckactually it will continue to be that for this job20:28
rlandyweshay|ruck: I ma going to try define it on testproject to start20:29
rlandyso we don;t need another merge20:29
rlandyto test20:29
rlandybuild containers are not run anywhere. right?20:30
weshay|ruckrlandy, atm.. all we have are non-push container builds for centos-820:31
rlandyweshay|ruck: ok -then I better fix it20:32
rlandyreview coming up20:32
rlandy2020-02-14 20:37:44.680265 | TASK [get_hash : get md5 file]20:38
rlandy2020-02-14 20:37:45.814140 | primary | ERROR20:38
rlandyis that me or legit?20:38
weshay|ruckrlandy, that's not you20:38
weshay|ruckthat's how it's going to be .. hrm.. we're going to have to fake that out to get this going20:38
rlandyok - so new patch coming20:39
weshay|ruckso we need to look at that20:39
weshay|ruckbit20:39
weshay|ruckwhere is that?20:40
rlandyhttps://review.rdoproject.org/r/#/c/24966/1/zuul.d/tripleoci.yaml20:41
rlandyweshay|ruck: ^^ pls see where I have added the var20:41
rlandywe might want to change that20:41
rlandyfor test run, I just put it in testproject def20:41
weshay|ruckmd5_url20:41
weshay|ruckactually20:42
* weshay|ruck looks20:42
weshay|ruckrlandy, https://review.rdoproject.org/r/#/c/24762/29/ci-scripts/infra-setup/roles/get_hash/tasks/get_hash.yaml20:42
weshay|ruckline 7920:42
weshay|ruckdang20:43
weshay|ruckrlandy, push this bit until monday20:44
rlandyyeah doesn;t exist20:44
weshay|rucklet's pick it up then20:44
weshay|ruckya20:44
rlandyhttps://review.rdoproject.org/r/#/c/24966/ - what do you want to do with this?20:44
rlandythose jobs will fail w/o it20:45
rlandyno default for promote_source20:45
weshay|ruckrlandy, punt until monday20:46
rlandyack - ok20:46
* rlandy goes back to image builds20:49
weshay|ruckrlandy, PING PING private20:53
rlandylooking21:01
*** holser has joined #oooq21:33
*** jmasud_ has quit IRC21:53
*** rlandy has quit IRC21:59
*** rfolco has quit IRC22:02
*** holser has quit IRC22:56
*** holser has joined #oooq23:42

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!