Thursday, 2021-04-22

« Wednesday, 2021-04-21 Index Friday, 2021-04-23 »
*** anbanerj|ruck has quit IRC00:07
*** anbanerj|ruck has joined #oooq00:08
*** Tengu_ has joined #oooq00:26
*** Tengu_ has quit IRC00:28
*** Tengu has quit IRC00:29
*** Tengu has joined #oooq00:31
*** jmasud has joined #oooq00:31
*** jmasud has quit IRC00:34
*** jmasud has joined #oooq00:35
*** jmasud has quit IRC00:38
*** jmasud has joined #oooq00:41
*** jmasud has quit IRC00:55
*** jmasud has joined #oooq00:58
*** jmasud has quit IRC01:02
*** rlandy|rover|bbl has quit IRC01:53
*** apetrich has quit IRC02:09
*** ysandeep|away is now known as ysandeep04:14
*** ratailor has joined #oooq04:19
*** skramaja has joined #oooq04:26
*** jmasud has joined #oooq04:35
*** ykarel has joined #oooq04:45
*** saneax has joined #oooq04:56
*** zbr9 has joined #oooq05:04
*** jfrancoa has joined #oooq05:04
*** zbr has quit IRC05:06
*** zbr9 is now known as zbr05:06
*** saneax has quit IRC05:10
*** marios has joined #oooq05:13
*** jfrancoa has quit IRC05:27
*** jfrancoa has joined #oooq05:30
*** saneax has joined #oooq05:36
*** jfrancoa has quit IRC05:38
*** jfrancoa has joined #oooq05:39
*** udesale has joined #oooq05:59
*** jpodivin has joined #oooq06:04
*** ykarel_ has joined #oooq06:08
*** ykarel has quit IRC06:11
*** ykarel__ has joined #oooq06:11
*** slaweq has joined #oooq06:12
*** ykarel_ has quit IRC06:14
*** ykarel__ is now known as ykarel06:17
*** ykarel_ has joined #oooq06:32
*** ykarel has quit IRC06:35
*** saneax has quit IRC06:40
*** jpena|off is now known as jpena06:50
*** ysandeep is now known as ysandeep|lunch06:58
*** amoralej|off is now known as amoralej07:13
*** jmasud has quit IRC07:18
jbadiapaHi guys!07:24
jbadiapathe nodes on the tripleo-ansible@stable/train are centos-8 or centos-8-stream?07:25
jbadiapait seems to me that they are still centos-807:25
jbadiapahttps://eb10f7108f52b9b0c724-cbd455b36f4013168ee28d0c0ff6b698.ssl.cf1.rackcdn.com/786408/14/check/tripleo-ansible-centos-8-molecule-backup-and-restore/ae74b68/reports.html07:25
mariosjbadiapa: see patch from sagi at https://review.opendev.org/c/openstack/tripleo-ansible/+/787423 for molecule jobs07:26
mariosjbadiapa: i think related to what you're asking07:26
jbadiapayeah, but that's master branch07:27
jbadiapaI'm talking about train branch07:27
jbadiapamarios, On train I tried to add that nodeset https://review.opendev.org/c/openstack/tripleo-ansible/+/786408/12 but I got a weird error07:29
jbadiapamarios: do you know if sagi's patch is going to be backported to train?07:30
mariosjbadiapa: i would imagine so... seems the molecule jobs are using their own nodeset and this is what sagi is addressing there07:30
mariossshnaidm: ^^07:31
jbadiapamarios, ok, thx07:32
mariosjbadiapa: for the "general case jobs" train was done with https://review.opendev.org/c/openstack/tripleo-quickstart/+/779803/07:34
mariosjbadiapa: so yeah we likely need Sagi patch for fixing the molecule jobs seems they arent done even for master yet07:35
marioscoffee brb07:35
*** jfrancoa has quit IRC07:41
*** jfrancoa has joined #oooq07:42
*** apetrich has joined #oooq07:45
*** tosky has joined #oooq07:49
*** derekh has joined #oooq08:03
*** ysandeep|lunch is now known as ysandeep08:50
sshnaidmjbadiapa, ack, we are going to convert to stream in all branches I think08:56
*** saneax has joined #oooq09:11
*** ykarel_ has quit IRC09:18
*** chem has joined #oooq09:43
ysandeepanbanerj|ruck, hey validations jobs are failing https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master-validation/db5d018/job-output.txt10:04
ysandeepI think https://review.opendev.org/c/openstack/tripleo-quickstart/+/787443 ^^ broke it10:04
ysandeepweshay|ruck, ^^ fyi..10:06
anbanerj|ruckysandeep, the patch says validation should be available from rpms or zuul. Lemme ask Wes.10:20
ysandeepanbanerj|ruck, thanks! we will need a bug as well10:20
ysandeepanbanerj|ruck, are you aware of bug opening process, otherwise I can open a bug for this.10:22
anbanerj|ruckysandeep, lemme try10:27
ysandeepanbanerj|ruck, you want me to shadow you how to open a bug.. I can show you over gmeet10:27
anbanerj|ruckyes please, I thought it would be too dumb of me to ask :D https://meet.google.com/kin-ptxv-mgd10:28
ysandeepnot at all.. there are two flags which we need to set carefully.. otherwise cix card will not be generated..10:29
ysandeepjoining10:29
*** ratailor_ has joined #oooq10:30
*** ratailor has quit IRC10:31
ysandeephttps://bugs.launchpad.net/tripleo/+bug/192547010:42
openstackLaunchpad bug 1925470 in tripleo "Validation jobs in master component line is failing with error with "ERROR! the role 'tripleo_validations' was not found"" [Critical,Triaged]10:42
ysandeepanbanerj|ruck, ^^10:42
*** ykarel has joined #oooq11:29
*** jpena is now known as jpena|lunch11:34
*** ysandeep is now known as ysandeep|afk11:39
*** rlandy has joined #oooq11:47
*** rlandy is now known as rlandy|rover11:50
sshnaidmweshay|ruck, rlandy|rover do you remember what was the problem with ubi and stream? Which ubi image should we use..?11:52
rlandy|roversshnaidm: initially there was no stream ubi11:53
rlandy|roverso we just carried on with the one we always had11:53
rlandy|roverthere were no failures we picked up at the time we converted to stream11:53
rlandy|roverlately we saw openstack tox failures11:54
rlandy|roverzbr fixed those for us yesterday11:54
sshnaidmI'm looking at this one https://review.opendev.org/c/openstack/python-tripleoclient/+/76170211:54
*** ykarel has quit IRC11:55
sshnaidmwonder is we should do this..11:55
rlandy|rovermerged?11:55
rlandy|roveras long as you are bot dealing with any c7 distros11:57
sshnaidmrlandy|rover, I have a problem now with molecule jobs on tripleo-ansible, they had still centos8 non-stream node11:57
*** ykarel has joined #oooq11:57
sshnaidmand with stream it fails with "'Failed to download metadata for repo ''appstream'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried"11:58
rlandy|roversshnaidm: thought you changed the nodeset11:58
* rlandy|rover has join recruiting call11:58
sshnaidmnope, just yesterday started to fail, when tripleo-repos set centos-stream by default11:59
sshnaidmrlandy|rover, np, will look further today11:59
rlandy|roversshnaidm: call should be short11:59
*** ykarel has quit IRC12:00
*** rlandy|rover is now known as rlandy|rover|cal12:00
zbrmarios: what is you take on the use of review priority? does it work or not? are we really using it?12:00
*** rlandy|rover|cal is now known as rlandy|rvr|call12:00
zbrasking because i see [nova][placement] trying to do the same experiment :D12:00
*** amoralej is now known as amoralej|lunch12:00
*** ykarel has joined #oooq12:00
marioszbr: are we using that? to be honest I haven't been checking it12:00
marioszbr: probably worth discussing again during one of next scrums next week for example12:01
marioszbr: i am a bit skeptical about how well it could work though to be honest12:02
zbrmy impression is that is not working, use of hashtags for one-off prio looks more plausible.12:02
zbrthe review prio is a technical measure aiming to sort a human one...12:03
zbri seen very few such attempts succeeding, wondering if I should reply to nova with our team "outcomes"12:04
zbrthe only thing I would find working is to have "review-swat" rolling role where one or two people goal is to perform reviews, nothing else.12:05
zbrkinda similar to ruck/rovering but aimed at nurturing any reviews, prio them, abandon them, whatever it takes to lower their number.12:06
zbrany stalling open review is wasted effort, primary human one, and secondary CI one (with side effects on everyone, as it adds noise)12:07
* marios tired12:15
rlandy|rvr|calljpodivin: hi ... https://review.rdoproject.org/r/c/openstack/tripleo-validations-distgit/+/33454 https://review.opendev.org/c/openstack/tripleo-validations/+/78741612:17
rlandy|rvr|call^^ can we get W+ on those?12:17
*** rlandy|rvr|call is now known as rlandy|rover12:18
jpodivinrlandy|rover: would if I could.12:19
jpodivinrlandy|rover: but they are my patches so I can't do that. Not to mention the fact that I'm not core so...12:19
jpodivinrlandy|rover: I can try to get dpeacock, cjeanner, matbu or strider.12:20
rlandy|roverjpodivin: would appreciate that12:20
gchamouljpodivin, rlandy|rover: just did it right now12:20
rlandy|rovergchamoul: thank you12:21
Tenguthanks gchamoul for jumping in that fast :)12:24
jpodivingchamoul++12:28
rlandy|roverperiodic-tripleo-ci-centos-8-scenario004-standalone-wallaby success12:32
rlandy|roverwoohoo12:32
rlandy|rovergouthamr: tbarron: ^^ thanks for sticking with this one - closing the bug out12:32
rlandy|rovermanila_tempest_tests.tests.api.test_shares.SharesNFSTest.test_create_get_delete_share[backend,id-21ad41fb-04cf-493c-bc2f-66c80220898b,positive]12:33
tbarronrlandy|rover: thank you!!12:33
rlandy|roverhmmm ... we have a new ovn failure ... on deploy 2021-04-22 11:08:45.162083 | primary | async task did not complete within the requested time - 5700s12:35
rlandy|rover2021-04-22 11:08:45.164644 | primary |12:35
*** jpena|lunch is now known as jpena12:35
*** ysandeep|afk is now known as ysandeep12:39
gouthamrrlandy|rover ++12:39
ysandeeprlandy|rover, weshay|ruck https://bugs.launchpad.net/tripleo/+bug/1925470 .. I think spill over from https://review.opendev.org/c/openstack/tripleo-quickstart/+/78744312:41
openstackLaunchpad bug 1925470 in tripleo "Validation jobs in master component line is failing with error with "ERROR! the role 'tripleo_validations' was not found"" [Critical,Triaged]12:41
rlandy|roverysandeep: thanks12:42
rlandy|roverthere are a couple other patches in play12:43
weshay|ruckysandeep, just remove them from criteria12:43
weshay|ruckI want to nuke as much validations as possible12:43
rlandy|roverhttps://review.rdoproject.org/r/c/openstack/tripleo-validations-distgit/+/33454 https://review.opendev.org/c/openstack/tripleo-validations/+/78741612:43
weshay|ruckand stategically put it back12:43
rlandy|rover^^ got w+'ed today12:43
ysandeepweshay|ruck, **validation jobs** in component line are failing12:44
weshay|ruckysandeep, yes.. you just said that12:44
weshay|ruckysandeep, take them out of criteria12:45
weshay|ruckthank you12:45
ysandeepThey are not in component criteria already: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-8/component/master.yaml12:46
jpodivinIs there anything I can do to help with that?12:48
weshay|ruckjpodivin, we got it..12:50
jpodivinif the issue really is in https://review.opendev.org/c/openstack/tripleo-quickstart/+/787443 we can reinstate the repo downloads in a while.12:50
jpodivinthe fixes removing osc-lib are about to get merged.12:51
weshay|ruckjpodivin, that's it12:51
weshay|ruckbut it's still not a good idea to have validations included in that way12:51
weshay|ruckjpodivin, what's the ansible role path used by validations after rpm install?12:52
weshay|ruckjpodivin, ?12:54
jpodivinysandeep: there is the     - periodic-tripleo-ci-centos-8-standalone-validation-master12:55
gchamoulweshay|ruck: /usr/share/ansible/roles/12:55
weshay|ruckgchamoul, what do you think about a bindep install of tripleo-validations vs.. https://review.opendev.org/c/openstack/tripleo-quickstart/+/787443/1/quickstart-extras-requirements.txt12:56
weshay|ruckwe should already have /usr/share/ansible/roles in the path12:56
weshay|ruckand if we don't use rpms... then your changes will not come in through dlrn12:56
weshay|ruckgchamoul, rlandy|rover ysandeep ok.. I think I have what we need now12:57
weshay|ruckgive me a few12:57
gchamoulweshay|ruck: bindep is ok to me12:57
gchamoulweshay|ruck: tripleo-validations rpm is handling the path too12:57
gchamoulweshay|ruck: it requires ansible rpm, so yes /usr/share/ansible should already be there13:00
weshay|ruckgchamoul, and just to be 100% if I install tripleo-validations the rest gets pulled in via spec right?13:01
gchamoulweshay|ruck: what is the rest for you? :D13:01
gchamoulvalidations-common? YES13:02
weshay|ruckvalidations-common and lib gets pulled in via the13:02
weshay|ruckya13:02
weshay|ruckk13:02
weshay|ruckgchamoul, 2 seconds13:02
*** ratailor_ has quit IRC13:03
gchamoulthe chain is TV requires validations-common which requires validations-libs13:03
*** chem has quit IRC13:04
*** saneax has quit IRC13:04
*** apetrich has quit IRC13:04
*** jpodivin has quit IRC13:04
*** marios has quit IRC13:04
*** soniya29 has quit IRC13:04
rlandy|roverbhagyashris: weshay|ruck: akahat: we have a fix for fs002 under test13:04
rlandy|roverif you need to promote master, we can remove that for criteria13:04
bhagyashrisrlandy|rover, ack13:04
rlandy|roverotherwise it should be fixed with https://review.opendev.org/c/openstack/tripleo-quickstart/+/78746913:04
rlandy|roverlooking into the OVB failure13:05
*** chem has joined #oooq13:05
*** saneax has joined #oooq13:05
*** apetrich has joined #oooq13:05
*** jpodivin has joined #oooq13:05
*** marios has joined #oooq13:05
*** soniya29 has joined #oooq13:05
rlandy|roverfs020 seems sporadic13:05
rlandy|roverwallaby looking good13:05
gchamoulweshay|ruck: so yes, if you install tripleo-validations via rpm, VC and VL will get pulled in13:05
weshay|ruckgchamoul, matbu so it should be just as easy as .. https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/78755113:06
gchamoulweshay|ruck: package name is openstack-tripleo-validations13:07
*** jpodivin has left #oooq13:08
*** jpodivin has joined #oooq13:10
jpodivinI know very little about the way CI works. I gather that the right ansible role is missing, but I don't understand how it happened.13:11
gchamoulweshay|ruck: hmm ... it should actually be installed during the standalone undercloud deployment at the previous task, shouldn't it?13:14
ysandeepits installed, i just checked13:14
ysandeephttps://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master-validation/db5d018/logs/undercloud/var/log/extra/package-list-installed.txt.gz13:14
ysandeep~~~13:14
ysandeepopenstack-tripleo-validations.noarch       14.0.1-0.20210416070243.06d7104.el8       @delorean-component-validation13:14
ysandeep~~~13:14
zbrkopecmartin: do you have a minute to look at https://zuul.opendev.org/t/openstack/build/661cc10989284a4c81914a0c4979861b ?13:15
gchamoulweshay|ruck, ysandeep: so it won't fix it ... it's matter of ANSIBLE_ROLES_PATH here13:15
*** amoralej|lunch is now known as amoralej13:15
zbrinfrared fails to "ir plugin add /root/artcl-src --src-path infrared_plugin" due to some recursive path13:15
ysandeepgchamoul, +1 agree with you13:15
gchamoulysandeep, weshay|ruck: what is the ANSIBLE_* configuration used by oooq?13:16
zbri know we have some symlinks around but what ir does is clearly not ok.13:16
jpodivingchamoul: path is defined in a few places, including in extras13:18
ysandeepgchamoul, not 100% sure on this though.. https://github.com/openstack/tripleo-quickstart/blob/master/ansible.cfg#L2013:20
matbuysandeep: weshay|ruck: gchamoul: i didnt take a look closely on that, but the issue is because the validations role is not installed on the node where oooq kicks the playbook13:20
kopecmartinzbr: my first guess would be that infrared got into a never ending loop trying to add (install) the plugin .. any chance that the change caused that infrared is called from a different dir or any scope change like that?13:20
matbuth role are on the undercloud only13:20
rlandy|roverysandeep: are we safe to just promote up baremetal, tripleo and network?13:20
rlandy|rover16.213:20
rlandy|roverthey are behind but the testing history is not good13:21
gchamoulmatbu: so that's why you pulled TV from extra-requirements ...13:21
matbugchamoul: yes i think13:21
zbrkopecmartin: i found a symlink inside the infrared_plugin folder that makes no sense, probably before infrared supported the roles_path in config, removing it and trying to see what happens.13:21
zbri am wondering why i see that only now13:22
*** ratailor has joined #oooq13:22
kopecmartinzbr: hmm, you added the symlink there, i guess it was about the time the role was converted into a collection? role_path option in infrared_plugin/plugin.spec is used to tell infrared where to find the role13:24
gchamoulmatbu, ysandeep, weshay|ruck: tripleo-validations role should be in $VIRTUAL_ENV/usr/share/ansible/roles  https://github.com/openstack/tripleo-quickstart/blob/master/ansible.cfg#L20 (link from ysandeep)13:24
gchamoulback to square one!13:24
ysandeeprlandy|rover, no, some of them got new hash(not tested), I would weigh on removing ovb from criteria and let those component promote themselves .. https://code.engineering.redhat.com/gerrit/#/c/237631/13:25
kopecmartinzbr: i don't understand why the symlink was needed :/ and yeah, it's interesting that only now we see the recursive loop13:25
matbugchamoul: ysandeep: weshay|ruck: there is two ways: 1/ installed tripleo-validation on the node-1 where tripleo-quickstart is launching the playbooks, 2/ installed from source like i  did previoulsy13:25
kopecmartinzbr: i experimented with the symlinks too but it lead to recursive issues like this one, therefore i added the roles_path option to infrared13:27
rlandy|roverysandeep: ^^ ok - let's try that13:28
rlandy|roverif it's just OVB blocking13:28
rlandy|roverweshay|ruck: ^^ fyi - merging  https://code.engineering.redhat.com/gerrit/#/c/237631/13:28
rlandy|roverlet's see if we get the component to clear13:29
matbujpodivin: not related but can you remove your -1 on https://review.rdoproject.org/r/c/openstack/validations-libs-distgit/+/3343613:29
matbujpodivin: i mean workflow13:29
*** skramaja has quit IRC13:29
ysandeeprlandy|rover, sc10 is also failing for network.. but issue is cleared for train.. I need to traceback what solved the issue..13:29
jpodivinmatbu: done13:29
rlandy|roverok13:30
matbujpodivin: thx :)13:32
weshay|ruckthanks gchamoul13:39
gchamoulweshay|ruck: but it won't fix it ...13:40
gchamoulweshay|ruck: it is already installed at the previous, during the deployment of the undercloud and as matbu said it should be installed and in the ROLE_PATH of tripleo-quickstart, where it runs13:41
gchamoulargh .. forgot to type half of my words...13:42
gchamoulweshay|ruck: it is already installed at the previous task during the deployment of the undercloud and as matbu said it should be installed where oooq runs and part of the $VIRTUAL_ENV/usr/share/ansible/roles13:43
gchamoulbetter for the repetition!13:43
* ysandeep thinks.. with rpm based openstack-tripleo-validation.. /usr/share/ansible/roles/ should be in path , I don't think $VIRTUAL_ENV/usr/share/ansible/roles will have that role13:45
* ysandeep testing with a dnm patch 13:46
gchamoulysandeep: actually the TV rpm is installed in undercloud only but not where oooq is running13:47
matbuysandeep: the role is not in the oooq virtualenv, only on the UC13:48
zbrrlandy|rover: weshay|ruck: a quick review on https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/787538 just to avoid self merge ;)13:48
weshay|rucklooking13:49
rlandy|roverchecking13:51
zbrthe queries repo did not had our python linters, we are gradually adding them now.13:51
weshay|ruckzbr, does this replace the patch I was looking at from anbanerj|ruck ?13:51
anbanerj|rucknope, this is additional one to add linters. The previous one was for the converters13:52
gchamoulso /go 2513:54
weshay|ruckgchamoul, jpodivin rlandy|rover, ysandeep the other thing we need to do .. is add the new validations job to check changes to tq and tqe13:54
weshay|ruckI'll get that up later today13:54
* ysandeep wondering.. if we can add "/usr/share/ansible/roles" at the end of roles_path, like: https://review.opendev.org/c/openstack/tripleo-quickstart/+/787567/1/ansible.cfg13:55
weshay|ruckysandeep, we can.. and we should see if we can prune some of that out as well14:01
weshay|ruckit's old14:01
sshnaidmysandeep, then also /usr/share/ansible/collections14:01
ysandeepweshay|ruck, true some are obselete.. i just login to one of the node.. some dir.. doesn't exists14:02
sshnaidmand $VIRTUAL_ENV/share/ansible/collections/ansible_collections doesn't make much sense14:02
ysandeepsshnaidm, ack o/14:03
ysandeepsshnaidm, Just to confirm suggestion is to replace $VIRTUAL_ENV/share/ansible/collections/ansible_collections with /usr/share/ansible/collections here: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/ansible.cfg#L2114:05
sshnaidmysandeep, yep14:06
ysandeepaye o/14:06
zbrdo we have any place we ansible-role-collect-logs is installed as a python package?14:11
zbri removed pbr usage from it and i was expecting to break something but apparently i did see no problems so far.14:12
anbanerj|ruckzbr, weshay|ruck rlandy|rover https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/787569 when you get some time - Adding new query from a current bug to see what ER shows on the dashboard14:34
zbrarxcruz: now I realise why you wanted to use tee, it can be really useful for a CR like this14:35
zbri will look into an approach that display the output only for changed queries, if we do for all is close to spam14:35
zbranbanerj|ruck: ^ that was for you, bad tab complete.14:36
arxcruzzbr: ?14:36
zbrarxcruz: ignore.14:37
anbanerj|ruckzbr, yes, we'll add the fancy stuff later14:37
zbrexactly, small changes at a time (later = sometimes in the future)14:38
*** ratailor has quit IRC14:39
jbadiapasshnaidm, https://review.opendev.org/c/openstack/tripleo-ansible/+/787423 it's failing https://zuul.openstack.org/status/change/787423,6 . There are still some molecule failing for example https://db3cff29bdb713e861e1-7db0c1fa1bd98a0adf758f7c1d49f672.ssl.cf5.rackcdn.com/787423/6/check/tripleo-ansible-centos-8-molecule-tripleo_cephadm/2785361/reports.html14:39
jbadiapatripleo_repos.main.InvalidArguments: --stream provided, but OS is not the Stream version. Please ensure the host is Stream. << not sure how it's possible TBH14:40
sshnaidmjbadiapa, yeah, working on that14:40
*** saneax has quit IRC14:52
*** amoralej is now known as amoralej|afk15:01
bhagyashrisweshay|ruck, rlandy|rover for manifest push testing i am going to use same namespace "tripleomaster" and removing the fs002 from master the criteria15:04
weshay|ruckbhagyashris, k.. w/ the dlrn and image modules turned off right????15:08
bhagyashrisweshay|ruck, yes15:08
weshay|ruck++15:08
rlandy|roverysandeep: following up with 16.2 components ...15:34
rlandy|roverperiodic-tripleo-ci-rhel-8-scenario010-standalone-network-rhos-16.2node_failure (non-voting)15:35
rlandy|rover^^ are we tracking any other failures?15:35
rlandy|roverhttp://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp16-2/component/baremetal/promoted-components/commit.yaml15:35
rlandy|rover03/3015:36
*** ykarel is now known as ykarel|away15:36
rlandy|rover[DIR]promoted-components/2021-03-24 12:0815:36
weshay|ruckrlandy|rover, ysandeep keep spending time in 16.2, upstream is fine.. I'll be poking at 17 today15:40
*** ykarel|away has quit IRC15:41
ysandeeprlandy|rover, nope.. sc010 is the remaining one15:41
ysandeeprlandy|rover, i have a patch up to sync the job def of sc10 with rdo/upstream.. https://code.engineering.redhat.com/gerrit/#/c/233534/ but testproject is failing on one of a tempest test15:43
ysandeephttps://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/39/check/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2/32c76dc/logs/undercloud/var/log/tempest/stestr_results.html15:43
*** dviroel is now known as dviroel|lunch15:46
ysandeeprlandy|rover, ignore ^^.. just noticed.. chandhan reverted https://github.com/rdo-infra/rdo-jobs/commit/21a3dd9d84b08bc6fd994c9dcc45b7cb4f5875a615:47
rlandy|roveryep15:50
rlandy|roverwas working15:50
rlandy|rovermissing backport?15:51
ysandeeptrain is green.. 16.2 is not.. i don't think its about backport15:51
ysandeeperrr.. why are we zipping stestr_results.html.gz15:53
weshay|ruckrlandy|rover, do you know what this card is for? https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=4285&projectKey=TRIPLEOCI&view=detail&selectedIssue=TRIPLEOCI-38715:56
* weshay|ruck needs a little more context to recall15:57
rlandy|roverdependency wallaby jobs15:58
* rlandy|rover joins meeting15:58
weshay|rucksshnaidm, fyi http://rdo-ci-fx2-02-s2.v101.rdoci.lab.eng.rdu2.redhat.com:3000/ansible-collections-openstack15:59
rlandy|roverweshay|ruck: to add dependency jobs for container-tools and the like16:00
*** jpena is now known as jpena|away16:00
weshay|ruckpojadhav, can you pick that up and work w/ rlandy|rover ?16:01
pojadhavweshay|ruck, sure16:01
weshay|ruckand check w/ chandan when he's back about picking up https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=4285&projectKey=TRIPLEOCI&view=detail&selectedIssue=TRIPLEOCI-37416:02
pojadhavyup16:02
weshay|ruckpojadhav, also start reviewing others patches please16:04
weshay|ruckyou'll find things that way as well16:04
pojadhavweshay|ruck, ack16:04
*** jmasud has joined #oooq16:17
sshnaidmweshay|ruck, nice16:18
*** slaweq_ has joined #oooq16:20
mariosrlandy|rover: thank you very much for agreeign to swap with me i think i may have quit if i had to also ruck|rove this week16:22
marios;)16:22
*** marios is now known as marios|out16:22
*** amoralej|afk is now known as amoralej16:26
*** jpodivin has quit IRC16:28
*** sshnaidm is now known as sshnaidm|afk16:28
*** ysandeep is now known as ysandeep|away16:30
ysandeep|awayrlandy|rover, i will keep looking at sc010 in downstream tomorrow16:31
rlandy|roversure16:31
*** marios|out has quit IRC16:47
*** udesale has quit IRC16:55
*** dviroel|lunch is now known as dviroel16:58
*** derekh has quit IRC17:02
weshay|ruckrlandy|rover, http://pastebin.test.redhat.com/95810017:13
rlandy|roverweshay|ruck: ok - looking into that17:14
*** jmasud has quit IRC17:21
*** jmasud has joined #oooq17:24
*** rlandy|rover is now known as rlandy|rover|mtg17:27
*** jmasud has quit IRC17:33
*** kopecmartin has quit IRC17:48
weshay|ruckrlandy|rover|mtg, can I borrow your eyes when you have a sec18:02
* weshay|ruck looking for the source code for:18:03
weshay|ruck2021-04-21T21:59:29+0000 DDEBUG Command: dnf repoquery --quiet --provides --disablerepo=* --enablerepo=delorean-current -a18:03
weshay|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario002-standalone-rhos-17/57ee55e/logs/undercloud/var/log/extra/podman/containers/swift_container_updater/log/dnf.log18:03
weshay|ruckI don't see where current is defined any where in tc/container-builds18:03
weshay|ruckmeh.. looks like it happens in a job that passes too18:06
weshay|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario002-standalone-rhos-17/0f1cf6d/logs/undercloud/var/log/extra/podman/containers/swift_container_updater/log/dnf.log18:06
*** jpena|away is now known as jpena|off18:07
*** rlandy|rover|mtg is now known as rlandy|rover18:07
rlandy|roverweshay|ruck: sorry - grocery delivery18:08
rlandy|roverlooking18:08
weshay|rucknp18:08
weshay|ruckrlandy|rover, I think it's the container-update code18:08
weshay|ruckbut it's not fatal18:08
weshay|ruckdisregard18:08
rlandy|roverdnf.exceptions.RepoError: Unknown repo: 'delorean-current'18:09
rlandy|roverweshay|ruck: ^^18:09
rlandy|rover2021-04-21T21:59:29+0000 CRITICAL Error: Unknown repo: 'delorean-current'18:09
rlandy|rovercritical error18:09
weshay|ruckya.. but I see that in a working job too18:10
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario002-standalone-rhos-17/57ee55e/logs/undercloud/etc/yum.repos.d/18:10
weshay|ruckit's not fatal18:10
weshay|ruckya.. and we don't have current18:10
rlandy|roverso something is expecting current18:10
weshay|ruckrlandy|rover, ya.. works here https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario002-standalone-rhos-17/0f1cf6d/logs/undercloud/var/log/extra/podman/containers/swift_container_updater/log/dnf.log18:10
rlandy|roverok then18:10
weshay|rucksorry18:11
rlandy|roverwe should try look for that specifcation18:11
rlandy|roverprobably some default value18:11
*** amoralej is now known as amoralej|off18:33
rlandy|roverweshay|ruck: where are we with validations - https://review.opendev.org/c/openstack/tripleo-validations/+/787416 still needed?18:37
rlandy|roverhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/78746918:37
rlandy|roverbhagyashris: weshay|ruck: ^^ fs002 should be passing now18:38
weshay|rucknice18:38
weshay|ruckrlandy|rover, have to merge the role path18:38
rlandy|roverok18:39
weshay|ruckrlandy|rover, in check https://review.opendev.org/c/openstack/tripleo-quickstart/+/78756718:39
rlandy|roverk18:39
rlandy|roverso many patches flying around :)18:39
rlandy|roverweshay|ruck: wow https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-promote-component 17 components are promoting well19:15
weshay|ruckrlandy|rover, I forced promoted but yes they are passing well now19:28
rlandy|roverweshay|ruck: ok - debugging OVB ... https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/find_undercloud_uuid.yml#L1319:28
rlandy|roverfailing out there19:28
rlandy|roverhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/find_undercloud_uuid.yml#L4 resolves in rdo19:29
rlandy|roverhasn't passed since 04/0219:30
weshay|ruckrlandy|rover, the curl or the next line?19:30
rlandy|roverthe next line19:30
weshay|ruckya.. probably the curl19:30
*** slaweq_ has quit IRC19:30
rlandy|roverthe first will only resolve locally19:30
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-16.2/5cf072b/logs/undercloud/19:31
rlandy|rovershoot we don't gather /mnt19:31
rlandy|roverprobably upgrade to ovb 2.019:31
rlandy|rover2021-03-29 01:13:17.976698 | TASK [ovb-manage : Find out UUID of instance with metadata URL]19:33
rlandy|rover2021-03-29 01:13:18.167651 | primary | 64564cc2-bfd9-48bb-b3c4-506608d06e5c19:33
rlandy|rover2021-03-29 01:13:19.064031 | primary | ok: Runtime: 0:00:00.35442719:33
rlandy|rover2021-03-29 01:13:19.164064 |19:33
rlandy|roveractually that resolves internally at some point19:33
rlandy|rover2021-03-29 01:13:17.976698 | TASK [ovb-manage : Find out UUID of instance with metadata URL]19:33
rlandy|rover2021-03-29 01:13:18.167651 | primary | 64564cc2-bfd9-48bb-b3c4-506608d06e5c19:33
rlandy|rover2021-03-29 01:13:19.064031 | primary | ok: Runtime: 0:00:00.35442719:33
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-16.2/5cf072b/job-output.txt19:34
rlandy|rover169.254.169.254 - what is that?19:34
rlandy|roverlink-local address19:34
weshay|ruckrlandy|rover, https://docs.openstack.org/nova/latest/user/metadata.html19:35
rlandy|roverso that doesn't work anymore19:35
weshay|ruckprobably a cloud error19:35
weshay|ruckwait.. is the successful one old?19:35
rlandy|roverall the time?19:36
rlandy|rover04/0219:36
weshay|ruckcan we log into a node?19:36
rlandy|roveryeah - if the job is running19:36
weshay|ruckya.. I bet the metadata service is not running on the cloud now19:37
weshay|ruckcommand works fine19:38
weshay|ruckcat /tmp/foo | python3 -c 'import json, sys; print(json.load(sys.stdin)["uuid"])'19:38
weshay|ruckfoobar19:38
weshay|ruckcreated a fake json19:38
rlandy|roverasking rhos-ops19:38
weshay|ruckrlandy|rover, just open a cix :)19:40
rlandy|roverokie dokie19:40
weshay|ruckcurrently 3 fails list in the cockpit gate failures.. will be interesting to see how many we have at the end of the day w/ this long queue19:43
weshay|ruckmerged 34 patches in 24 hours19:44
weshay|ruckgoing to hit 50+ again19:44
rlandy|roverlovely19:44
rlandy|roverdefinitely more failures with more load19:45
weshay|ruckrlandy|rover, can we start running 17 twice a day?19:54
rlandy|roverweshay|ruck: sure  - will set that up19:54
weshay|ruckthank you19:54
weshay|ruckbbl19:57
rlandy|roverk20:00
*** jfrancoa has quit IRC20:11
*** jfrancoa has joined #oooq20:11
*** jmasud has joined #oooq20:37
*** slaweq has quit IRC20:44
*** jfrancoa has quit IRC20:51
rlandy|rover169.254.169.254 192.168.100.1220:58
rlandy|rovergateway unreachable20:58
*** jbadiapa has quit IRC21:28
*** irclogbot_0 has quit IRC21:50
*** irclogbot_1 has joined #oooq21:53
rlandy|roverperiodic-tripleo-ci-rhel-8-multinode-multiple-overcloud-2ctlr-featureset010-tripleo-rhos-16.2 has been failing forever22:05
rlandy|roverpromote w/o it?22:05
*** rlandy|rover is now known as rlandy|rover|bbl22:17
weshay|ruckrlandy|rover, that's containers-multinode.. did the bm deployment pass?22:17
*** kopecmartin has joined #oooq22:28
*** tosky has quit IRC23:17
*** anbanerj|ruck has quit IRC23:40
*** anbanerj|ruck has joined #oooq23:41
« Wednesday, 2021-04-21 Index Friday, 2021-04-23 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!