| *** jmasud has joined #oooq | 00:08 | |
| *** jmasud has quit IRC | 00:09 | |
| *** tosky has quit IRC | 00:36 | |
| *** jmasud has joined #oooq | 01:26 | |
| *** jmasud has quit IRC | 01:37 | |
| *** jmasud has joined #oooq | 01:45 | |
| *** jmasud has quit IRC | 01:52 | |
| *** jmasud has joined #oooq | 04:00 | |
| *** udesale has joined #oooq | 04:19 | |
| *** saneax has joined #oooq | 04:32 | |
| *** jmasud has quit IRC | 04:47 | |
| *** jmasud has joined #oooq | 04:57 | |
| *** ykarel has joined #oooq | 05:22 | |
| *** ykarel has quit IRC | 05:39 | |
| *** ykarel has joined #oooq | 05:41 | |
| *** ysandeep|out is now known as ysandeep|afk | 05:43 | |
| *** ykarel_ has joined #oooq | 05:54 | |
| *** ratailor has joined #oooq | 05:55 | |
| *** ykarel has quit IRC | 05:57 | |
| *** ykarel__ has joined #oooq | 06:05 | |
| *** marios has joined #oooq | 06:07 | |
| *** ykarel__ is now known as ykarel | 06:07 | |
| *** ykarel_ has quit IRC | 06:08 | |
| *** udesale_ has joined #oooq | 06:09 | |
| *** udesale has quit IRC | 06:12 | |
| *** jmasud has quit IRC | 06:22 | |
| pojadhav | chandankumar, 0/ | 06:24 |
|---|---|---|
| pojadhav | as per https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both#2021-01-07-Unified-Sprint-38-Planning , frenzy_friday also interested in the promoter work. I think she is missing in invite. Idk about her time zone. | 06:25 |
| *** jfrancoa has joined #oooq | 06:26 | |
| chandankumar | pojadhav: ah my bad, we can sync once again in evening if ok | 06:27 |
| pojadhav | chandankumar, its totally fine :) | 06:27 |
| akahat|rover | ykarel, o/ | 06:51 |
| akahat|rover | ykarel, i need to hold one node for testing purpose. | 06:51 |
| akahat|rover | this is review link: https://review.rdoproject.org/r/#/c/28014/ and job is: tripleo-ci-promotion-staging-single-pipeline-centos-8 | 06:51 |
| ykarel | akahat|rover, hi | 06:52 |
| ykarel | ok putting up hold request | 06:52 |
| akahat|rover | ykarel, thank you :) | 06:52 |
| ykarel | akahat|rover, your pub key | 06:55 |
| ykarel | https://github.com/amolkahat.keys ? | 06:55 |
| ykarel | added these, try ssh zuul@38.102.83.46 | 06:55 |
| akahat|rover | ykarel, yes.. add any one. | 07:03 |
| akahat|rover | ykarel, ok | 07:03 |
| akahat|rover | ykarel, i'm in thanks :D | 07:04 |
| ykarel | ack | 07:05 |
| akahat|rover | chandankumar, directory location is /home/zuul/src/review.rdoproject.org/rdo-infra/ci-config/ | 07:06 |
| akahat|rover | andi t's there. | 07:06 |
| chandankumar | akahat|rover: not this path | 07:16 |
| chandankumar | akahat|rover: look for /home/promoter | 07:16 |
| akahat|rover | chandankumar, yes. this is also there. | 07:17 |
| chandankumar | akahat|rover: then why it is not able to access it | 07:18 |
| chandankumar | akahat|rover: can you try running the playbook locally itself there | 07:18 |
| akahat|rover | chandankumar, ok | 07:18 |
| *** ysandeep|afk is now known as ysandeep | 07:21 | |
| marios | arxcruz: chandankumar: what is the status on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770188 | 07:37 |
| *** udesale has joined #oooq | 07:37 | |
| marios | arxcruz: chandankumar: it was merging yesterday and it still blocks ussuri (i also saw it onvictoria today ) | 07:37 |
| marios | arxcruz: chandankumar: was it fixed some other way? | 07:37 |
| chandankumar | marios: arxcruz was working on some other fixes instead of revert | 07:37 |
| marios | chandankumar: why workflow -1/ | 07:37 |
| marios | chandankumar: arxcruz: but those can come after the revert? it was basically in the gate?! | 07:38 |
| marios | chandankumar: thanks | 07:38 |
| marios | arxcruz: any update on that please | 07:38 |
| marios | arxcruz: do you need reviews on the other fixes? | 07:38 |
| marios | arxcruz: can you please put the other fixes on the bug and add some words about them https://bugs.launchpad.net/tripleo/+bug/1911020 | 07:39 |
| openstack | Launchpad bug 1911020 in tripleo "Ugrades ussuri jobs fail in CI" [Critical,Triaged] | 07:39 |
| chandankumar | marios: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770357 was one of the fix | 07:39 |
| *** udesale_ has quit IRC | 07:39 | |
| *** akahat|rover is now known as akahat|lunch | 07:45 | |
| *** udesale has quit IRC | 07:48 | |
| marios | chandankumar: thanks | 07:49 |
| marios | chandankumar: arxcruz: but that one is from alex, were there others arxcruz ? | 07:50 |
| *** jpena|off is now known as jpena | 07:52 | |
| chandankumar | marios: https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359 | 07:52 |
| marios | chandankumar: thanks | 07:52 |
| marios | arxcruz: https://bugs.launchpad.net/tripleo/+bug/1911020/comments/6 https://bugs.launchpad.net/tripleo/+bug/1911020/comments/7 add if there are more than those | 07:54 |
| openstack | Launchpad bug 1911020 in tripleo "Ugrades ussuri jobs fail in CI" [Critical,Triaged] | 07:54 |
| *** apetrich has joined #oooq | 08:02 | |
| *** slaweq has joined #oooq | 08:03 | |
| *** amoralej|off is now known as amoralej | 08:09 | |
| *** matbu has quit IRC | 08:29 | |
| *** matbu has joined #oooq | 08:31 | |
| zbr | good read: https://clig.dev/ -- Command Line Interface Guidelines | 08:35 |
| *** tosky has joined #oooq | 08:49 | |
| *** slaweq has quit IRC | 08:55 | |
| *** slaweq has joined #oooq | 09:00 | |
| marios | arxcruz: are you around today? | 09:01 |
| *** udesale has joined #oooq | 09:12 | |
| *** ykarel_ has joined #oooq | 09:13 | |
| *** ykarel has quit IRC | 09:16 | |
| arxcruz | marios: yes i am | 09:25 |
| marios | arxcruz: hi can you please update the bug 09:54 < marios> arxcruz: https://bugs.launchpad.net/tripleo/+bug/1911020/comments/6 | 09:26 |
| openstack | Launchpad bug 1911020 in tripleo "Ugrades ussuri jobs fail in CI" [Critical,Triaged] | 09:26 |
| openstack | bug 9 in Launchpad itself "Rosetta's po parser is too strict" [Medium,Fix released] https://launchpad.net/bugs/9 - Assigned to Carlos Perelló Marín (carlos) | 09:26 |
| marios | arxcruz: and comment 7 | 09:26 |
| marios | arxcruz: in particular what else are we waiting for please that is blocking ussuri patches | 09:26 |
| marios | arxcruz: are there more patches that need workflow | 09:26 |
| arxcruz | marios: sure, tive me a second | 09:27 |
| arxcruz | marios: https://bugs.launchpad.net/tripleo/+bug/1911020/comments/8 i hope i made myself clear | 09:31 |
| openstack | Launchpad bug 1911020 in tripleo "Ugrades ussuri jobs fail in CI" [Critical,Triaged] | 09:31 |
| marios | arxcruz: did you test that https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359 fixed the upgrade job? | 09:34 |
| marios | arxcruz: is there a testproject somewhere | 09:34 |
| arxcruz | marios: the upgrade job is passing on the patch | 09:35 |
| arxcruz | is there a reason to do a testproject ? | 09:35 |
| marios | arxcruz: 11:34 < marios> arxcruz: did you test that https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359 fixed the upgrade job? | 09:35 |
| marios | arxcruz: for that reason ^ | 09:35 |
| marios | arxcruz: ? | 09:36 |
| arxcruz | marios: tripleo-ci-centos-8-standalone-upgrade https://zuul.opendev.org/t/openstack/build/11e906f36c87494a81b68ffc04c6f9a8 : SUCCESS in 2h 33m 33s (non-voting) | 09:36 |
| marios | arxcruz: the failure is on undercloud-upgrade-ussuri | 09:36 |
| arxcruz | tripleo-ci-centos-8-undercloud-upgrade https://zuul.opendev.org/t/openstack/build/76026dfaff8e4d93ae73901008ea088c : SUCCESS in 1h 49m 27s (non-voting) | 09:36 |
| marios | arxcruz: https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-undercloud-upgrade-ussuri | 09:36 |
| marios | arxcruz: it blocks ussuri gate | 09:37 |
| marios | arxcruz: for a few days now | 09:37 |
| marios | arxcruz: master upgrade jobs are non voting anyway | 09:37 |
| arxcruz | marios: the issue was tempest running on upgrade jobs, which we don't do | 09:37 |
| arxcruz | updating the featureset to not run tempest, it will fix it | 09:38 |
| marios | arxcruz: we do run tempest on upgrade jobs | 09:38 |
| arxcruz | but i can of course create a testproject | 09:38 |
| marios | arxcruz: e.g. on standalone-upgrade | 09:38 |
| marios | arxcruz: never mind man... my objection is that you blocked the revert after it was already in the gate. so i was hoping you had tested the thing you proposed instead of the revert. | 09:38 |
| marios | arxcruz: lets hope if all merges today | 09:39 |
| marios | arxcruz: no point doing a testproject now | 09:39 |
| arxcruz | marios: come on man, i got the wrong information, if you check my conversation with alex, he told we don't run tempest on upgrade | 09:39 |
| marios | arxcruz: k thanks | 09:40 |
| arxcruz | he asked for revert, and i said, let's not revert, let's set the variable to false, since the issue is with tempest | 09:40 |
| marios | arxcruz: ack ok it's ok i am grumpy cos it's always my fault when upgrade jobs are borked and i am blocked on ussuri there ussuri https://review.opendev.org/c/openstack/tripleo-heat-templates/+/761412 https://review.opendev.org/c/openstack/tripleo-common/+/769166 https://review.opendev.org/c/openstack/python-tripleoclient/+/769336 https://review.opendev.org/c/openstack/puppet-tripleo/+/769340 | 09:41 |
| marios | https://review.opendev.org/c/openstack/os-net-config/+/769493 | 09:41 |
| marios | arxcruz: i mainly objecting to blocking the revert after it hit the gates. i think you could have just let it go through, take off any pressure from yourself and then fix it in your own peace after | 09:42 |
| arxcruz | marios: the latest passing ussuri upgrade doesn't run tempest | 09:42 |
| arxcruz | 2021-01-09 08:02:50.549224 | primary | TASK [Run os_tempest role] ***************************************************** | 09:42 |
| arxcruz | 2021-01-09 08:02:50.549282 | primary | Saturday 09 January 2021 08:02:50 +0000 (0:00:00.206) 0:00:43.723 ****** | 09:42 |
| arxcruz | 2021-01-09 08:02:50.585544 | primary | skipping: [undercloud] | 09:42 |
| marios | arxcruz: cos you removed it | 09:42 |
| arxcruz | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5aa/769510/2/gate/tripleo-ci-centos-8-undercloud-upgrade-ussuri/5aadfa6/job-output.txt | 09:42 |
| arxcruz | marios: no, i haven't | 09:42 |
| arxcruz | this is from a few days ago | 09:43 |
| arxcruz | on january 9 | 09:43 |
| arxcruz | before my os_tempest everywhere patch | 09:43 |
| marios | arxcruz: you posted a patch to remove the tempest execution from fs50 didn't you? | 09:43 |
| arxcruz | marios: can we chat? | 09:43 |
| marios | arxcruz: ah you abandoned that https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770351 | 09:45 |
| arxcruz | marios: https://meet.google.com/ymr-bcig-egh | 09:46 |
| *** derekh has joined #oooq | 09:50 | |
| chandankumar | stepping out for a bit | 09:55 |
| marios | arxcruz: https://bugs.launchpad.net/tripleo/+bug/1911194 | 09:59 |
| openstack | Launchpad bug 1911020 in tripleo "duplicate for #1911194 Ugrades ussuri jobs fail in CI" [Critical,Triaged] | 09:59 |
| arxcruz | marios: http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2021-01-12.log.html#t2021-01-12T14:49:10 | 10:03 |
| *** ykarel_ is now known as ykarel | 10:06 | |
| marios | arxcruz: ack | 10:07 |
| arxcruz | :( | 10:07 |
| marios | arxcruz: thanks for chatting ... hopefully it gets resolved today | 10:07 |
| zbr | that day started just right: my lenovo (f33) failed to boot, grub stuff related to the 5.10 kernel. | 10:30 |
| marios | zbr: oh really? is it a known issue? I'm on 5.9.16 right now | 10:34 |
| bhagyashris | pojadhav, sshnaidm|afk zbr hi, could you please help me to complete the sprint report. please reply to an email 'sprint report' | 10:38 |
| bhagyashris | thank you :) | 10:38 |
| bhagyashris | chandankumar, ^^ | 10:39 |
| pojadhav | bhagyashris, ack | 10:39 |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 10:40 | |
| bhagyashris | pojadhav, thanks! | 10:54 |
| *** dtantsur|afk is now known as dtantsur | 10:56 | |
| zbr | marios: that was the one still working.... this is how I manager to boot again. | 10:59 |
| zbr | anyway i am going to do a full reinstall with formatting, i have nothing valuable on it. | 11:00 |
| marios | zbr: ouch thanks for the heads up | 11:02 |
| zbr | last time i got a surprise like this was with fedora 6/7, and switching to something else. Now I cannot afford that luxury. | 11:08 |
| zbr | usually i would have tried to fix dig more but i observed that the installation was made using classic BIOS, and this prevented fwupdate from running, and there is no way to convert to UEFI. Full reinstall needed. | 11:10 |
| zbr | marios: if you have "quiet" or "rhgb" on grub conf, i would worry. https://bugzilla.redhat.com/show_bug.cgi?id=1903332 | 11:13 |
| openstack | bugzilla.redhat.com bug 1903332 in kernel "Can't boot with kernel-5.9.10-200.fc33.x86_64 on Asus UX305CA/UX305CA" [Urgent,New] - Assigned to kernel-maint | 11:13 |
| marios | zbr: thx | 11:16 |
| *** ysandeep is now known as ysandeep|afk | 11:17 | |
| marios | zbr: looks like i do (must be a default i haven't touched that or changed anything here from vanilla 33 install) | 11:18 |
| marios | zbr: both quiet and rhgb | 11:18 |
| *** chem has quit IRC | 11:18 | |
| *** chem has joined #oooq | 11:20 | |
| ykarel | is it just me seeing errors in http://dashboard-ci.tripleo.org/d/_ZOYIidMk/vexxhost?orgId=1 or it's a known issue? | 12:04 |
| bhagyashris | arxcruz, hi, Bugs related to os_tempest that is affecting upgrade jobs -> do you have bug link ? it would be great if you share the bug link with me . thank you :) | 12:05 |
| arxcruz | bhagyashris: sorry my lack of information https://bugs.launchpad.net/tripleo/+bug/1911020 | 12:06 |
| openstack | Launchpad bug 1911020 in tripleo "Ugrades ussuri jobs fail in CI" [Critical,Triaged] | 12:06 |
| bhagyashris | arxcruz, np thank you :) | 12:06 |
| arxcruz | bhagyashris: would you like me to send a followup email with this info, or is it fine? | 12:06 |
| bhagyashris | it's fine | 12:06 |
| bhagyashris | :) | 12:06 |
| bhagyashris | just one more thing : Add the add test command on tempest-skiplist and documentation -> is there any WIP review link ? | 12:07 |
| arxcruz | bhagyashris: yes https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/754994 | 12:10 |
| arxcruz | i'll update in a few, just finishing writing the documentation | 12:10 |
| bhagyashris | ok . thank you arxcruz :) | 12:10 |
| marios | sshnaidm|ruck: thanks for comments but i don't understand can you check my reply at https://review.opendev.org/c/openstack/tripleo-ci/+/770766 when you have time thank you | 12:21 |
| sshnaidm|ruck | marios, solution you pointed is not the best also, if we fix it - ovb can return to use multiple playbooks. The point was to have playbooks so independent, so one can run one of them and it works | 12:24 |
| sshnaidm|ruck | marios, mostly for devs that need to rerun various parts of deploy | 12:25 |
| sshnaidm|ruck | I'm not sure it's the case now, but it shouldn't be removed just so | 12:25 |
| marios | sshnaidm|ruck: maybe easier to discuss in scrum but, i don't see what the difference is from the ovb case 14:24 < sshnaidm|ruck> marios, solution you pointed is not the best also, if we fix it - ovb can return to use | 12:26 |
| marios | sshnaidm|ruck: we can also do the same here... | 12:26 |
| marios | sshnaidm|ruck: return to use the multiple if we fix it? i don't see the difference | 12:26 |
| sshnaidm|ruck | marios, difference where? between multiple playbooks and single? | 12:26 |
| marios | sshnaidm|ruck: no you're saying in the ovb case https://review.opendev.org/c/openstack/tripleo-ci/+/764657 "ovb can return to use multiple playbooks" | 12:27 |
| marios | sshnaidm|ruck: so what's the difference between ovb case and this one | 12:27 |
| sshnaidm|ruck | marios, yes, ovb patch is not the solution, it's a workaround and should be reverted when we fix the bug | 12:27 |
| sshnaidm|ruck | I don't see a point to make another workaround | 12:27 |
| marios | sshnaidm|ruck: k, well if we don't have a fix now then what do we do, besides apply the workaround? | 12:27 |
| sshnaidm|ruck | marios, we can make it conditional and not use in CI as we discussed before | 12:28 |
| marios | sshnaidm|ruck: for the record, i am not sure it is OK yet, i have workflow -1 it until i test with the testproject reviews as i commented there | 12:28 |
| marios | sshnaidm|ruck: https://review.rdoproject.org/r/31555 for the train update https://review.rdoproject.org/r/31556 for victoria upgrade | 12:28 |
| sshnaidm|ruck | this part is mostly for quickstart.sh runs from devs hosts, we don't need it in ci | 12:28 |
| *** dsneddon has quit IRC | 12:31 | |
| *** ratailor has quit IRC | 12:31 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** akahat|lunch is now known as akahat|rover | 12:36 | |
| bhagyashris | hi all, do we need to keep the scrum today as we just finished the planning meeting two days before ? needs vote accordingly will decide | 12:51 |
| bhagyashris | akahat|rover, frenzy_friday arxcruz chandankumar marios pojadhav sshnaidm|ruck zbr soniya29 ^^ | 12:52 |
| bhagyashris | ysandeep|afk, ^ | 12:52 |
| *** ysandeep|afk is now known as ysandeep | 12:52 | |
| soniya29 | bhagyashris, i think we don't need scrum today since planning meeting has happened just two days before | 12:53 |
| ysandeep | bhagyashris, I am okay to skip it, if everyone agrees.. | 12:55 |
| akahat|rover | bhagyashris, i'm with soniya29 ysandeep !! | 12:56 |
| bhagyashris | ysandeep, soniya29 ok , others plz let me know ... | 12:56 |
| zbr | ysandeep: "if nobody comments" ;) sure. | 12:56 |
| pojadhav | bhagyashris, we can skip the scrum for today :) | 12:57 |
| *** rlandy has joined #oooq | 12:58 | |
| marios | bhagyashris: seems a bit late to be asking the question though, e.g. US folks are just waking up now | 12:58 |
| bhagyashris | rlandy, do we need to keep the scrum today as we just finished the planning meeting two days before ? needs vote accordingly will decide | 12:58 |
| marios | bhagyashris: imo we should have it since scrum != planning meeting | 12:59 |
| rlandy | bhagyashris: marios: I'd say yes | 12:59 |
| bhagyashris | marios, ok | 12:59 |
| *** amoralej is now known as amoralej|lunch | 12:59 | |
| rlandy | let's look at the boards | 12:59 |
| rlandy | which I don't think I can access | 12:59 |
| rlandy | if cards are available | 12:59 |
| marios | rlandy: right and figure out who/what is doing what with who ;) | 12:59 |
| rlandy | what's listed | 12:59 |
| rlandy | marios: ack | 12:59 |
| bhagyashris | rlandy, marios ok np :) | 12:59 |
| rlandy | so from tomorrow/monday people can get going | 13:00 |
| ysandeep | folks, could i please get some eyes on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/762350/ whenever time permits. | 13:00 |
| marios | rfolco: are you around? | 13:03 |
| rfolco | marios, o/ | 13:03 |
| marios | rfolco: we need someone to start the sprint for us (apparently we don't have permissions to do that) | 13:03 |
| marios | rfolco: never mind bhagyashris just told me she already asked you | 13:04 |
| rfolco | marios, something happened in jira, I can't do it anymopre, I don't have permissions either | 13:04 |
| marios | rfolco: thanks | 13:04 |
| rfolco | marios, she has the ticket number that weshay|ruck opened | 13:05 |
| marios | rfolco: ack thx | 13:05 |
| rfolco | yw | 13:05 |
| chandankumar | sshnaidm|ruck: hello | 13:08 |
| chandankumar | sshnaidm|ruck: need some help here https://logserver.rdoproject.org/14/28014/84/check/tripleo-ci-promotion-staging-single-pipeline-centos-8/96266e4/job-output.txt | 13:08 |
| chandankumar | sshnaidm|ruck: https://review.rdoproject.org/r/#/c/28014/85/ci-scripts/infra-setup/roles/promoter/tasks/promotion_run.yml@16 this part tries to copy the file to the zuul execute then and try to load the vars, but message": "Could not find or access '/home/promoter/ci-config/ci-scripts/dlrnapi_promoter/config_environments/staging/defaults.yaml' on the Ansible Controller.\nIf you are using a module and | 13:12 |
| chandankumar | expect the file to exist on the remote, see the remote_src option" | 13:12 |
| marios | rlandy: https://projects.engineering.redhat.com/browse/TRIPLEOCI-197 | 13:13 |
| sshnaidm|ruck | chandankumar, I'm not familiar with it, do you have a playbook that fails there? | 13:18 |
| akahat|rover | sshnaidm|ruck, this is the playbook: https://review.rdoproject.org/r/#/c/28014/84..85/ci-scripts/infra-setup/roles/promoter/tasks/promotion_run.yml | 13:22 |
| *** ksambor has quit IRC | 13:23 | |
| rlandy | zbr: you around? | 13:27 |
| rlandy | scrum | 13:27 |
| ysandeep | rlandy, If you don't get better slot, I am okay to join the mtg for first half an hour and then i will drop for another mtg | 13:27 |
| *** jpena|lunch is now known as jpena | 13:29 | |
| marios | bhagyashris: https://projects.engineering.redhat.com/browse/TRIPLEOCI-197 | 13:30 |
| marios | bhagyashris: https://projects.engineering.redhat.com/browse/TRIPLEOCI-249 | 13:31 |
| bhagyashris | chandankumar, akahat|rover frenzy_friday pojadhav can we continue the promoter sync? | 13:37 |
| frenzy_friday | yep | 13:37 |
| zbr | rlandy: now i am | 13:44 |
| rlandy | zbr: no worries ... we just wanted to go through the elastic recheck epic at sync | 13:45 |
| rlandy | and you're the main contact on that epic | 13:45 |
| *** pojadhav is now known as pojadhav|afk | 13:45 | |
| rlandy | can do it on monday's call | 13:45 |
| akahat|rover | bhagyashris, yes | 13:46 |
| zbr | rlandy: link to the epic? | 13:47 |
| zbr | somehow jira board seams empty https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=4285 | 13:48 |
| rlandy | zbr: ack - none of us can get to the board - jira issue | 13:49 |
| rlandy | you can view the epics from backlog | 13:49 |
| rlandy | bhagyashris: ^^ can you point zbr to the elastc recheck epic - where you accessed it? | 13:49 |
| bhagyashris | rlandy, sure | 13:57 |
| bhagyashris | zbr, https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=4285&projectKey=TRIPLEOCI&view=planning.nodetail&selectedIssue=TRIPLEOCI-58&epics=visible&issueLimit=100&selectedEpic=TRIPLEOCI-129 | 13:58 |
| bhagyashris | zbr, https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=4285&projectKey=TRIPLEOCI&view=planning&selectedIssue=TRIPLEOCI-177&epics=visible&issueLimit=100&selectedEpic=TRIPLEOCI-176 | 13:58 |
| zbr | so our sprint didnt even started because we have no issues in it, and is ending tomrorow. | 13:59 |
| *** dsneddon has joined #oooq | 14:01 | |
| *** ykarel has quit IRC | 14:02 | |
| *** amoralej|lunch is now known as amoralej | 14:03 | |
| rlandy | ysandeep: attila has passing job on the current 16.s hash | 14:06 |
| rlandy | 16.2 | 14:06 |
| rlandy | we have a failure on scenario010 | 14:06 |
| ysandeep | looking | 14:07 |
| rlandy | rerunning scenario010 | 14:07 |
| ysandeep | ack | 14:07 |
| rlandy | ysandeep: ^^ timeout | 14:07 |
| rlandy | also fs001 timeout | 14:07 |
| rlandy | fs035 passed | 14:08 |
| ysandeep | yes timedout on tempest , but i don't see any failures in tempest https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2/44dc062/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz | 14:13 |
| *** ysandeep is now known as ysandeep|cinder_ | 14:14 | |
| *** ysandeep|cinder_ is now known as ysandeep|session | 14:14 | |
| rlandy | marios: did we stop queens promotions? | 14:19 |
| rlandy | sshnaidm|ruck: akahat|rover: re we still watching/trying to promote queens? https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable6 | 14:25 |
| rlandy | we start the 13z15 import on the 25th | 14:25 |
| rlandy | ^^ per rel del | 14:25 |
| sshnaidm|ruck | rlandy, hmm.. looks bad | 14:26 |
| sshnaidm|ruck | rlandy, will look into | 14:26 |
| rlandy | sshnaidm|ruck: yeah - also looking into it | 14:26 |
| rlandy | last promotion was 11/18 | 14:26 |
| rlandy | 2021-01-13 12:50:18.853913 | primary | TASK [Create clouds.yaml if it doesn't exist] ********************************** | 14:27 |
| marios | rlandy: not that am aware of | 14:29 |
| marios | rlandy: we do still need them (osp import) | 14:29 |
| marios | rlandy: afaik | 14:29 |
| sshnaidm|ruck | rlandy, related to last tempest changes | 14:33 |
| rlandy | marios: apparently so | 14:33 |
| sshnaidm|ruck | rlandy, when we set os_tempest to run | 14:33 |
| rlandy | sorry - half listening on osp meeting | 14:33 |
| rlandy | sshnaidm|ruck: you working on the fix there? | 14:40 |
| rlandy | going to create a job to rekick the line | 14:40 |
| sshnaidm|ruck | rlandy, yeah, found where it is | 14:40 |
| rlandy | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tasks/tempest.yml#L58 | 14:40 |
| rlandy | cool thanks | 14:40 |
| rlandy | ysandeep|session: k - so if scenario010 passes rerun, we promote | 14:50 |
| ysandeep|session | rlandy, yes o/ | 14:51 |
| sshnaidm|ruck | rlandy, https://bugs.launchpad.net/tripleo/+bug/1911696 | 14:51 |
| openstack | Launchpad bug 1911696 in tripleo "Tempest tries to run on undercloud containers queens job" [Critical,Triaged] | 14:51 |
| sshnaidm|ruck | rlandy, probably should be solved by patches in gates | 14:52 |
| sshnaidm|ruck | arxcruz, can you please take a look, promotion blocker: https://bugs.launchpad.net/tripleo/+bug/1911696 if it will be solved by current patches? | 14:52 |
| sshnaidm|ruck | akahat|rover, fyi ^ | 14:52 |
| rlandy | sshnaidm|ruck: arxcruz: akahat|rover: adding a a testproject with those jobs | 14:53 |
| rlandy | let's see if it works | 14:53 |
| sshnaidm|ruck | rlandy, cool | 14:53 |
| *** TrevorV has joined #oooq | 14:55 | |
| rlandy | https://review.rdoproject.org/r/#/c/25325/ | 14:56 |
| rlandy | sshnaidm|ruck: akahat|rover: ^^ k- let's see what this does | 14:56 |
| *** ykarel has joined #oooq | 15:02 | |
| arxcruz | rlandy: sshnaidm|ruck https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359 fix the problem since it set featureset023 to not run tempest | 15:03 |
| zbr | team: i need some comments on https://github.com/rdo-infra/queries/pull/3#discussion_r557425557 -- naming challenge, do not lose the chance! ;) | 15:06 |
| rlandy | arxcruz: ^^ to confirm - it will fix the problem? | 15:06 |
| marios | rlandy: sshnaidm|ruck: need a vote on that when you have a sec https://review.opendev.org/c/openstack/tripleo-heat-templates/+/770160 (rebased weshay|ruck patch for merge conflict) | 15:09 |
| mjturek | marios: Could you take a look here? We're hitting this error in our container build job. Have you seen anything like it? Maybe we simply need to touch the file? http://paste.openstack.org/show/801625/ | 15:20 |
| marios | mjturek: looking | 15:21 |
| mjturek | if looking for context, see here https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/3033/consoleFull | 15:21 |
| sshnaidm|ruck | akahat|rover, rerunning train c8 job that failed: https://review.rdoproject.org/r/#/c/23626/ | 15:22 |
| marios | mjturek: don't think the missing file is the root cause trying to find what it is (that is just the log file of the build/error) | 15:24 |
| sshnaidm|ruck | akahat|rover, only 1 test failed ther last time: TestVolumeBootPattern.test_volume_boot_pattern | 15:24 |
| marios | mjturek: possibly "root" vs "jenkins" user is the problem | 15:24 |
| sshnaidm|ruck | akahat|rover, if it fails again in https://review.rdoproject.org/r/#/c/23626/ need to look for a fix.. | 15:25 |
| marios | mjturek: can we access any more files on this or only the console? | 15:28 |
| mjturek | marios: let me grab the link for the collected logs | 15:29 |
| mjturek | marios https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/3033/logs/ | 15:29 |
| marios | mjturek: thx | 15:30 |
| rlandy | marios: looking | 15:32 |
| marios | mjturek: so i suspect it is because jenkins user can't access /root/workspace/build.log | 15:32 |
| marios | mjturek: https://opendev.org/openstack/tripleo-ci/src/commit/15023d0e98265570547ffd11132608f7045f6c74/roles/build-containers/tasks/main.yaml#L212-L225 | 15:33 |
| rlandy | arxcruz: still have a failure on the testproject job ... https://review.rdoproject.org/r/#/c/25325/ | 15:33 |
| marios | mjturek: the tasks aren't executed with become there ... not sure why it is running as root user in https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/3033/consoleFull | 15:34 |
| mjturek | marios: is that a recent change?? | 15:34 |
| mjturek | because this used to work | 15:35 |
| rlandy | 2021-01-14 15:02:06.829348 | primary | + export TOCI_JOBTYPE=singlenode-featureset023 | 15:35 |
| rlandy | 2021-01-14 15:02:06.829441 | primary | + TOCI_JOBTYPE=singlenode-featureset023 | 15:35 |
| marios | mjturek: alternatively, you could try using root (see the next task below the one i pointed to) | 15:35 |
| marios | mjturek: i mean by adding a become there https://opendev.org/openstack/tripleo-ci/src/commit/15023d0e98265570547ffd11132608f7045f6c74/roles/build-containers/tasks/main.yaml#L220-L233 | 15:35 |
| marios | mjturek: compare those two ^^ | 15:35 |
| rlandy | use_os_tempest: false | 15:35 |
| rlandy | is that correct? | 15:35 |
| akahat|rover | sshnaidm|ruck, TestVolumeBootPattern.. i've seen it earlier today.. it shows ssh issue for cirros. I've checked history of job.. found it is very unpredictable.. :| | 15:36 |
| akahat|rover | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train | 15:36 |
| marios | mjturek: don't think it is a new change git blame doesn't think so at least https://opendev.org/openstack/tripleo-ci/blame/commit/15023d0e98265570547ffd11132608f7045f6c74/roles/build-containers/tasks/main.yaml | 15:37 |
| mjturek | marios: so that would require us to force ansible distribution to redhat, which is inaccurate | 15:37 |
| mjturek | we always used the root user as the ansible user | 15:38 |
| mjturek | maybe something changed in centos-ci then | 15:38 |
| marios | mjturek: no i meant rather, if you have to run this as root, then you might add become on the https://opendev.org/openstack/tripleo-ci/src/commit/15023d0e98265570547ffd11132608f7045f6c74/roles/build-containers/tasks/main.yaml#L220 | 15:38 |
| marios | mjturek: but this is just a guess i don't have much to go on here. but it would explain the 'no such file' thing | 15:38 |
| mjturek | that's fair marios - definitely on the right track, I'm going to ask if something changes in centos-ci | 15:39 |
| marios | mjturek: in our jobs th ansible user is zuul and we have all the things in /home/zuul/... | 15:39 |
| mjturek | that's fair | 15:39 |
| mjturek | thanks marios! | 15:41 |
| marios | mjturek: ack hope it helps anyway | 15:41 |
| ykarel | mjturek, marios seems https://opendev.org/openstack/tripleo-ci/commit/d227115b1dc26a65598c5935fba7522ad9aad0d3 caused the issue in ci.centos jobs | 15:43 |
| ykarel | likely we create the logs directory in zuul at some place so working there and in ci.centos jobs it missing | 15:43 |
| marios | ykarel: yeah could be | 15:44 |
| mjturek | ahh | 15:44 |
| marios | ykarel: but it didn't get to that point yet i mean the build report | 15:45 |
| ykarel | marios, that patch changed log path {{ workspace }}/build.log --> {{ workspace }}/logs/build.log | 15:45 |
| ykarel | and for that to work logs directory should exist | 15:46 |
| mjturek | ykarel: think creating a /root/logs/ dir is an appropriate fix? | 15:46 |
| ykarel | mjturek, i think it should be fixed upstream as it's regression as part of that patch, but for now you can create /root/workspace/logs | 15:47 |
| ykarel | in ci.centos job | 15:48 |
| ykarel | i think u must be creating /root/workspace somewhere already | 15:48 |
| mjturek | ykarel: yeah I believe so | 15:48 |
| rlandy | sshnaidm|ruck: arxcruz: k - so https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359/4/config/general_config/featureset023.yml will not fix the problem ... | 15:50 |
| rlandy | the featureset passed to the tempest run is different | 15:50 |
| rlandy | actually scratch that | 15:50 |
| mjturek | ykarel: I'll also take a quick look and see if I can find where that dir is created upstream, it might be as simple as removing a hardcoded "zuul" | 15:51 |
| mjturek | thanks ykarel and marios for the help | 15:51 |
| rlandy | --extra-vars @/home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/general_config/featureset023.yml | 15:51 |
| rlandy | it is passed | 15:51 |
| ykarel | mjturek, where u see zuul is hardcoded? | 15:52 |
| mjturek | ykarel nowhere, sorry just saying it could be something like that | 15:53 |
| ykarel | ack that task depend on workspace var so hardcoding shouldn't be there but good to check | 15:53 |
| marios | np mjturek | 15:54 |
| sshnaidm|ruck | arxcruz, something is wrong there with tempest_cloud_name maybe: https://logserver.rdoproject.org/25/25325/73/check/periodic-tripleo-centos-7-queens-containers-build/2022671/job-output.txt | 15:54 |
| sshnaidm|ruck | arxcruz, it shouldn't be overcloud.. | 15:54 |
| rlandy | 'Create clouds.yaml if it doesn't exist' is executing before the switch | 15:56 |
| rlandy | of whether or not to run os_tempest | 15:56 |
| *** ysandeep|session is now known as ysandeep | 15:58 | |
| ykarel | mjturek, so get_hash : Ensure legacy workspace directory is creating /root/workspace | 16:00 |
| mjturek | ykarel: right and prepare_node is makes the logs dir it seems | 16:00 |
| ykarel | so you need to add additional task to create /root/workspace/logs | 16:01 |
| sshnaidm|ruck | chandankumar, do you know where is tempest_cloud_name defined before gets there: https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/tasks/tempest.yml#L79 | 16:03 |
| rlandy | home/zuul/workspace/.quickstart/playbooks/multinode-validate.yml | 16:03 |
| rlandy | sshnaidm|ruck: arxcruz: ^^ https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/multinode-validate.yml#L29 | 16:04 |
| rlandy | tempest_cloud_name: 'overcloud' | 16:04 |
| sshnaidm|ruck | ok, so it should be a different condition there | 16:05 |
| sshnaidm|ruck | instead of "not tempest_cloud_name in ['undercloud', 'standalone'] | 16:06 |
| sshnaidm|ruck | " | 16:06 |
| mjturek | ykarel yep seems so!! Thanks a ton! | 16:06 |
| *** udesale has quit IRC | 16:08 | |
| rlandy | problem is the reuse of multinode playbook | 16:08 |
| *** jmasud has joined #oooq | 16:20 | |
| arxcruz | sshnaidm|ruck: once featureset023 use_os_tempest pass on gate, this will be fixed because it will not call tempest.yml playbook | 16:24 |
| *** ykarel is now known as ykarel|away | 16:24 | |
| sshnaidm|ruck | arxcruz, we ran job with these changes and it still failed, please read back | 16:26 |
| arxcruz | sshnaidm|ruck: sorry, let me check | 16:26 |
| arxcruz | sshnaidm|ruck: yeah, you're right, i'll submit a fix, tempest.yml should only be called when use_os_tempest is set to true | 16:28 |
| *** saneax has quit IRC | 16:33 | |
| arxcruz | sshnaidm|ruck: do you have the bug quickly? | 16:37 |
| rlandy | arxcruz: sshnaidm|ruck put in another change - under test now | 16:37 |
| sshnaidm|ruck | arxcruz, https://bugs.launchpad.net/tripleo/+bug/1911696 | 16:37 |
| openstack | Launchpad bug 1911696 in tripleo "Tempest tries to run on undercloud containers queens job" [Critical,Triaged] | 16:37 |
| rlandy | and correct - the switch on os_tempest was after this task ran | 16:38 |
| rlandy | hence the issue | 16:38 |
| sshnaidm|ruck | arxcruz, trying https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770830 now, but feel free to hijack it | 16:38 |
| *** zbr3 has joined #oooq | 16:38 | |
| arxcruz | sshnaidm|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770837 it's a better approach and save more time, since all the tasks under tempest.yml only matter if we actually run tempest | 16:39 |
| arxcruz | rlandy: ^ | 16:39 |
| *** zbr3 has quit IRC | 16:39 | |
| sshnaidm|ruck | arxcruz, thanks | 16:39 |
| arxcruz | np, it is my mess anyway :) | 16:40 |
| *** zbr9 has joined #oooq | 16:40 | |
| *** zbr has quit IRC | 16:40 | |
| *** zbr9 is now known as zbr | 16:40 | |
| rlandy | arxcruz: k - pls confirm which set of patches I shoudl test with and I will rerun - thanks | 16:40 |
| *** ykarel|away has quit IRC | 16:46 | |
| rlandy | running second test | 16:47 |
| *** amoralej is now known as amoralej|off | 16:57 | |
| *** marios has quit IRC | 17:03 | |
| *** jpena is now known as jpena|off | 17:07 | |
| arxcruz | rlandy: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770837 and https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359 should do it | 17:18 |
| rlandy | arxcruz: already under test here: https://review.rdoproject.org/r/#/c/29969/ thanks | 17:19 |
| *** ysandeep is now known as ysandeep|out | 17:35 | |
| sshnaidm|ruck | rlandy, running arxcruz patch now: https://review.rdoproject.org/r/#/c/25325/ | 17:51 |
| *** derekh has quit IRC | 18:04 | |
| *** dtantsur is now known as dtantsur|afk | 18:10 | |
| rlandy | sshnaidm|ruck: arxcruz: https://review.rdoproject.org/r/#/c/29969/ just passed | 18:28 |
| rlandy | with | 18:28 |
| rlandy | Depends-On: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770837 | 18:28 |
| rlandy | Depends-On: https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359 | 18:28 |
| rlandy | sshnaidm|ruck: pls vote on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770837 as well | 18:30 |
| rlandy | we can get this through gate today if lucky | 18:30 |
| *** apetrich has quit IRC | 18:31 | |
| rlandy | chandankumar: ^^ if you are still around ... pls vote | 18:31 |
| *** slaweq has quit IRC | 18:38 | |
| *** apetrich has joined #oooq | 18:58 | |
| weshay|ruck | rlandy, https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0&fvid=1778061576 | 19:33 |
| *** slaweq has joined #oooq | 20:19 | |
| *** slaweq has quit IRC | 20:27 | |
| *** jmasud has quit IRC | 20:30 | |
| *** jmasud has joined #oooq | 20:40 | |
| rlandy | weshay|ruck: you ok with our promoting 16.2? scenario010 juts passed | 20:41 |
| rlandy | fs035 passed in the run | 20:41 |
| weshay|ruck | aye | 20:41 |
| rlandy | fs001 timeout out running now | 20:41 |
| rlandy | fs020 had one tempest failure | 20:42 |
| rlandy | weshay|ruck: k- will promote ... since we have a passing test from the jenkins side | 20:42 |
| weshay|ruck | rlandy, 020 just had one tempest error https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-16.2/76a2c49/logs/undercloud/var/log/tempest/stestr_results.html.gz | 20:43 |
| weshay|ruck | so fs001 should have passed | 20:43 |
| rlandy | timedout | 20:43 |
| rlandy | rerunning now | 20:43 |
| rlandy | but soon the next hash will kick | 20:43 |
| rlandy | so promoting this one | 20:44 |
| weshay|ruck | rlandy, what about 010 | 20:44 |
| weshay|ruck | er.. | 20:44 |
| rlandy | just passed | 20:44 |
| weshay|ruck | scenaro 010 | 20:44 |
| weshay|ruck | rlandy, k.. promote it | 20:44 |
| rlandy | see testproject rerun | 20:44 |
| rlandy | on it | 20:44 |
| rlandy | and we're rolling | 20:45 |
| rlandy | weshay|ruck: not to jinx anything but possible gate queue will clear up quite a bit | 20:46 |
| weshay|ruck | rlandy, ya.. things are merging | 20:47 |
| *** jmasud has quit IRC | 20:53 | |
| rlandy | shoot - gate failure | 21:04 |
| rlandy | so close | 21:04 |
| *** TrevorV has quit IRC | 21:34 | |
| sshnaidm|ruck | train c8 should be promoted soon | 21:41 |
| rlandy | great | 22:22 |
| *** jmasud has joined #oooq | 23:13 | |
| *** rlandy is now known as rlandy|bbl | 23:28 | |
| *** jmasud has quit IRC | 23:33 | |
| *** jmasud has joined #oooq | 23:34 | |
| *** rfolco has quit IRC | 23:35 | |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!