| *** rlandy is now known as rlandy|out | 00:41 | |
| *** ysandeep|out is now known as ysandeep | 05:52 | |
| marios | good day oooci o/ | 06:02 |
|---|---|---|
| ysandeep | good morning o/ marios and everyone | 06:17 |
| ysandeep | folks o/ anyone know from where we get secondary node entry in primary node's /etc/hosts? https://f0b14fac337d4721647f-43001c5c460bd4e8e3c2be96e8c7a69c.ssl.cf2.rackcdn.com/852808/12/check/tripleo-ci-centos-9-containers-multinode/005472b/logs/undercloud/etc/hosts | 06:19 |
| *** amoralej|off is now known as amoralej | 06:19 | |
| ysandeep | maybe marios, ykarel ^^ when you have time. | 06:19 |
| ykarel | ysandeep, i think https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-hosts-file/tasks/main.yaml#L14 | 06:21 |
| ysandeep | ykarel, thanks okay so when we request more than 1 node in zuul, this role add the entries of other nodes | 06:22 |
| marios | ysandeep: not sure .. i found where that file is written but not where it gets those from - from zuul/nodepool sounds right though what ykarel pointed to | 06:26 |
| ysandeep | ykarel, marios ++ thanks! | 06:27 |
| marios | ysandeep: found that fwiw https://opendev.org/openstack/tripleo-ansible/src/commit/eb55be2320e200924763b5a3569efbc8119ab42a/tripleo_ansible/roles/tripleo_hosts_entries/tasks/main.yml#L50 | 06:28 |
| marios | ysandeep: for mixed os i had to tweak the tripleo-inventory to grab the extra node/make appropriate groups, anyway there fwiw/if useful https://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/tripleo-inventory/tasks/inventory.yml#L51 | 06:28 |
| ysandeep | marios: ++ thanks, I was not aware we had this info in /etc/nodepool/sub_nodes as well | 06:51 |
| dpawlik | hello folks. What promotion has been moved to IBM BM02 host ? | 06:58 |
| marios | cool glad helpful ysandeep | 06:59 |
| chandankumar | dpawlik: currently all the jobs are running on https://baremetal02.rdoproject.org | 06:59 |
| chandankumar | I mean all cs8 jobs | 07:00 |
| marios | dpawlik: c8/train chandankumar what else please? maybe components | 07:00 |
| marios | ah thanks chandankumar | 07:00 |
| marios | so train and wallaby 8 is on ibm? | 07:00 |
| chandankumar | wallaby and train integration and component lines | 07:00 |
| marios | dpawlik: ^^ | 07:00 |
| chandankumar | dpawlik: we are still running cs8 rdo check job on vexxhost cloud | 07:01 |
| dpawlik | aha | 07:01 |
| marios | bhagyashris|ruck: o/ are you OK for updating the doc for program call? | 07:01 |
| marios | bhagyashris|ruck: at least for the rhel lines and jm1 can update the centos ones when around | 07:01 |
| chandankumar | dpawlik: are we getting additional openstack clouds nodes on ibm cloud? | 07:02 |
| dpawlik | chandankumar: FYI https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/26050 | 07:02 |
| dpawlik | chandankumar: not that I'm aware of | 07:02 |
| marios | bhagyashris|ruck: links/info in private please check when you have time let me know if you need help on that | 07:02 |
| dpawlik | one more think, https://softwarefactory-project.io/r/c/config/+/26030 has been merged. I'm building the image right now | 07:04 |
| dpawlik | if you can switch some job just to test the image, would be awesome | 07:04 |
| dpawlik | and why did you switch all the jobs without cs8 to the IBM02 ? | 07:06 |
| *** jm1|ruck is now known as jm1|rover | 07:12 | |
| jm1 | good rainy morning folks 🌧 | 07:14 |
| chandankumar | dpawlik: we wanted to decrease the load on vexxhost cloud, we moved older and stable branches to IBM cloud | 07:15 |
| chandankumar | dpawlik: regarding testing cs9 dib image, let me check the promotion and move few jobs on that | 07:15 |
| chandankumar | dpawlik: left a comment https://softwarefactory-project.io/r/c/config/+/26030/10#message-5f75bb6f238d854e5952f3349e6b27c57687486b | 07:26 |
| marios | jm1: o/ morning | 07:32 |
| marios | jm1: are you ok with updating the program call doc? i asked bhagyashris|ruck to add the downstream stuff there so can you please add the upstream info as soon as you get a chance | 07:32 |
| marios | jm1: sending link in pvt but i think you have it already | 07:32 |
| jm1 | marios: o/ mirror issues solved? | 07:33 |
| marios | bhagyashris|ruck: lets talk on phone maybe easier? | 07:34 |
| marios | jm1: maybe that happened after i left? didn't see anything about mirror issues | 07:35 |
| bhagyashris|ruck | marios, ok | 07:35 |
| jm1 | marios: i am talking about the bug you posted yesterday... XD | 07:35 |
| marios | jm1: want to join us? we'll be editing the doc so prevent overwrite/we can just update it quickly now | 07:35 |
| marios | jm1: ah the RETRY you mean | 07:36 |
| marios | k didn't check that | 07:36 |
| marios | jm1: we can schedule it in 25 mins if you prefer bhagyashris|ruck instead of right.now() | 07:36 |
| jm1 | marios: let me first see what our current status for upstream is. you can go ahead with bhagyashris|ruck and fill in that doc, i will add my stuff later | 07:37 |
| marios | jm1: bhagyashris|ruck: my concern is we do that asap this morning because as soon as someone is on from prod chain (eg jjoyce or mburns they will take that doc and update th program call doc) | 07:37 |
| *** jpena|off is now known as jpena | 07:37 | |
| marios | jm1: bhagyashris|ruck: k sending invite. jm1 i plan to update the upstream info there during the call you can edit if you want later | 07:37 |
| bhagyashris|ruck | marios, ack so should we meet now or after 25 mins? | 07:38 |
| marios | bhagyashris|ruck: sent you invite 20 mins from now | 07:38 |
| dpawlik | chandankumar: so the image is available | 07:38 |
| dpawlik | chandankumar: will change name in next PS | 07:38 |
| marios | bhagyashris|ruck: should be quick 10 mins | 07:38 |
| bhagyashris|ruck | marios, ok | 07:38 |
| chandankumar | dpawlik: ok, sending a tp in few mins | 07:39 |
| jm1 | bhagyashris|ruck: yesterday you had ssl errors with rr script. did you get this error? "Max retries exceeded with url: /gerrit/gitweb?p=tripleo-environments.git;a=blob_plain;f=ci-scripts/dlrnapi_promoter/config_environments/rdo/RedHat-9/rhos-17.yaml;hb=HEAD (Caused by SSLError(CertificateError("hostname 'code.engineering.redhat.com' doesn't match either of 'gerrit.hosts.prod.psi.rdu2.redhat.com', 'gerrit2.hosts.prod.psi.rdu2.redhat.com'")))" | 07:41 |
| bhagyashris|ruck | jm1, not sure but it resolved automatically | 07:43 |
| bhagyashris|ruck | not it's working fine | 07:44 |
| jm1 | bhagyashris|ruck: ok, hm.. i get an outage error | 07:46 |
| jm1 | marios, ysandeep, bhagyashris|ruck: internal code.engineering... is down? | 07:54 |
| jm1 | marios, ysandeep, bhagyashris|ruck: ah its planned outage | 07:56 |
| chandankumar | dpawlik: https://review.rdoproject.org/r/c/testproject/+/44982 let's see how it goes | 07:57 |
| marios | jm1: was around earlier | 07:57 |
| dpawlik | thanks chandankumar | 07:57 |
| marios | jm1: but yeah down now here too | 07:58 |
| jm1 | marios: its planned outage, they say on google chat | 08:00 |
| marios | k thanks | 08:00 |
| marios | jm1: good job keeping promotions green we are pretty good for prog call | 08:05 |
| marios | jm1: we updated the doc please check when you have a chance | 08:07 |
| jm1 | marios: looks like you already entered all info into that doc. is there anything missing? | 08:09 |
| chandankumar | dpawlik: https://review.rdoproject.org/r/c/testproject/+/44982 seeing node failure | 08:09 |
| dpawlik | yup | 08:09 |
| dpawlik | checking that | 08:09 |
| marios | jm1: no we should be good to go | 08:11 |
| jm1 | marios: thank you man! | 08:11 |
| marios | welcome :) | 08:11 |
| jm1 | ysandeep: o/ do you know whether amol is doing well? only know of some flood and he is out for days.. | 08:18 |
| ysandeep | jm1, hello o/ | 08:19 |
| dpawlik | will work on your testproject chandankumar | 08:19 |
| chandankumar | dpawlik: ack | 08:19 |
| ysandeep | jm1, sry I have the same info as you have, maybe chandankumar knows more | 08:20 |
| chandankumar | jm1: ysandeep yes he is doing fine, the front side of his house got damaged due to rain. He is busy with repairing it | 08:21 |
| chandankumar | I spoke with him last night | 08:21 |
| chandankumar | jm1: ysandeep thank you for checking in :-) | 08:21 |
| ysandeep | good to hear Amol is doing fine :D | 08:22 |
| jm1 | chandankumar, ysandeep: oha 😱 hope its not too bad 😬 but good to know he is fine | 08:24 |
| * bhagyashris|ruck lunch | 08:39 | |
| rlandy | jm1: marios: bhagyashris|ruck: thanks for updating the proram doc notes | 10:38 |
| rlandy | just adding 17 | 10:38 |
| marios | rlandy: ah wasnt sure if we should have that or not | 10:41 |
| marios | rlandy: last week's notes had a pipelines: section for it but this week's did not | 10:41 |
| rlandy | marios: yeah - don't know - asking mburns | 10:41 |
| rlandy | jm1: bhagyashris|ruck: marios: so we need to sync on anything? | 10:42 |
| rlandy | jm1: bhagyashris|ruck: marios: otherwise just want to run through CIX | 10:42 |
| bhagyashris|ruck | rlandy, yeah we can run through cix | 10:43 |
| marios | rlandy: not aware of something - am tracking the mixed-rhel stuff (updated the card/bug for that https://trello.com/c/e2us0FZu#comment-63219160c4d253021d6175bd ) | 10:43 |
| rlandy | marios: ack - thanks | 10:44 |
| rlandy | bhagyashris|ruck: sure - just waiting for jm1 | 10:44 |
| rlandy | bhagyashris|ruck: all good in downstream? | 10:44 |
| rlandy | ysandeep: https://trello.com/c/R6SuOv6E/2661-cixlp1983718tripleociproa-periodic-master-scen1-standalone-fails-timeout-manage-firewall-rules - alan asked about this card on monday | 10:44 |
| rlandy | is that still in the right status? | 10:45 |
| bhagyashris|ruck | rlandy, yeah re-running th failing jobs | 10:45 |
| ysandeep | rlandy, afaik.. dpawlik created the c9 upstream image today and chandankumar posted a patch to testing that | 10:45 |
| rlandy | ysandeep: thanks - will update that card | 10:46 |
| jm1 | rlandy: sync? | 10:47 |
| rlandy | jm1: bhagyashris|ruck: yeah - I think we just need to cover CIX https://meet.google.com/wen-msen-rrk?pli=1&authuser=0 | 10:47 |
| rlandy | marios: ^^ can join if you want to - should be quick | 10:48 |
| marios | rlandy: ack | 10:52 |
| marios | rlandy: fyi bug for the RETRY jobs at https://bugs.launchpad.net/tripleo/+bug/1989452 but i removed the cix flags yesterday we may want it on the board | 11:01 |
| rlandy | bhagyashris|ruck, https://review.rdoproject.org/r/c/testproject/+/31165/90/.zuul.yaml | 11:03 |
| * jm1 lunch | 11:06 | |
| ysandeep | rlandy, good news https://github.com/Sandeepyadav93/keystone-operator/actions/runs/3052022550 , operator image build, push to quay and e2e test using image we build. | 11:08 |
| rlandy | ysandeep++++++++ | 11:11 |
| rlandy | very cool | 11:11 |
| arxcruz | pojadhav please join review meeting today :) | 11:14 |
| pojadhav | arxcruz, joining | 11:15 |
| *** dviroel|brb is now known as dviroel | 11:37 | |
| rlandy | anyone else having login issues on code.eng? | 11:46 |
| marios | rlandy: there was an outage earlier but it seemed to be back ~1 hour ago | 11:49 |
| rlandy | marios: hmm ... I can't login to anything code.eng related | 11:51 |
| rlandy | and I have a lot of reviews to vote on :( | 11:51 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 11:51 |
| pojadhav | rlandy, its working fine for me | 11:51 |
| ysandeep | chandankumar, rlandy on vexx cloud, do we have nested virt enabled on all our compute nodes? | 11:54 |
| rlandy | ysandeep: we have a nodeset that can use nested virt | 11:54 |
| rlandy | marios: ugh - still not | 11:54 |
| chandankumar | rlandy: ysandeep I think it is enabled on all the nodes | 11:56 |
| chandankumar | for vexxhost | 11:56 |
| rlandy | it should be | 11:58 |
| *** amoralej is now known as amoralej|lunch | 11:58 | |
| ysandeep | chandankumar, Actually intermittently sometime CRC(nested) fails to start on my VM and I suspect if its due to my node ending up on compute node which don't have nested virt enabled. | 11:58 |
| ysandeep | chandankumar, but i can see svm flag at vm level | 12:00 |
| ysandeep | for failed case as well | 12:00 |
| chandankumar | ok | 12:01 |
| chandankumar | can we open an issue on crc side? may be they can help | 12:01 |
| ysandeep | chandankumar, http://pastebin.test.redhat.com/1073604 coreos can't boot, softlock in the failure case | 12:03 |
| rlandy | I can log in to everything except code eng | 12:04 |
| ysandeep | dviroel, chandankumar, if l0 don't support nested, we should not see vmx/svm flag at l1, right ? | 12:05 |
| marios | rlandy: so is kinit working for you? are you getting 503/something else on code.eng? | 12:06 |
| marios | rlandy: did you try clear cache etc for code.eng? | 12:06 |
| dviroel | ysandeep: yeah, i think so | 12:07 |
| dviroel | rlandy: i just did a "kdestroy -A" and a "kinit", and now it is working | 12:08 |
| rlandy | kinit works | 12:09 |
| rlandy | klist | 12:09 |
| rlandy | dviroel: did you have an issue before? | 12:09 |
| chandankumar | ysandeep: if nested virt enabled, we can see svm flag | 12:09 |
| chandankumar | ysandeep: give some time, dpawlik is enabling nested virt on ibm cloud | 12:09 |
| dviroel | rlandy: a few minutes ago | 12:09 |
| chandankumar | you can check it there | 12:10 |
| * rlandy reboots | 12:10 | |
| chandankumar | there = crc setup on ibm cloud | 12:10 |
| ysandeep | yes, that will be great | 12:11 |
| chandankumar | brb | 12:17 |
| rlandy | woohoo finally | 12:24 |
| bhagyashris|ruck | brb | 12:28 |
| rlandy | that took half my morning | 12:28 |
| * pojadhav brb | 12:28 | |
| rlandy | marios; ok - I'll go back now and fix the base- thingy | 12:28 |
| rlandy | ysandeep: ^^ | 12:28 |
| ysandeep | rlandy, szymon did already | 12:29 |
| ysandeep | in his patches, and I have +wed the patches | 12:29 |
| ysandeep | rlandy, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428034/7/zuul.d/component-jobs-rhos-17.1-rhel-9.yaml#562 | 12:30 |
| ysandeep | and https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428188/5/zuul.d/component-jobs-rhos-17.1-rhel-9.yaml#550 | 12:30 |
| rlandy | nice | 12:31 |
| rlandy | I should have login problems more often | 12:31 |
| ysandeep | hahaha, lol! | 12:31 |
| ysandeep | Today we were bombarded with reviews :D I need to find some time tomorrow to review dasm rr patches. | 12:32 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:32 |
| rlandy | bhagyashris|ruck: hey - trigger jobs .. did they work? | 12:33 |
| rlandy | https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428003 | 12:33 |
| rlandy | bhagyashris|ruck: found the test project ... merging https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428003 | 12:36 |
| rlandy | marios: what happened with dasm's patches from yesterday? | 12:41 |
| rlandy | did you guys cover them in review | 12:41 |
| rlandy | I crossed out all the merged patches | 12:41 |
| marios | rlandy: we looked at them and ysandeep added them to check when he next reviews | 12:42 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:42 |
| ysandeep | >> <ysandeep> Today we were bombarded with reviews :D I need to find some time tomorrow to review dasm rr patches. | 12:43 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 12:43 |
| rlandy | ok | 12:44 |
| rlandy | n - just catching up after code.eng login fun this morning | 12:44 |
| rlandy | jm1: hey - so I've been getting better results doing reruns on IBM cloud | 12:47 |
| jm1 | rlandy: good good | 12:48 |
| bhagyashris|ruck | rlandy, yes | 12:52 |
| bhagyashris|ruck | thanks | 12:52 |
| marios | jm1: o/ around? question about sdk in #rdo | 12:54 |
| jm1 | marios: yeap, answered. thanks! | 12:55 |
| *** jpena is now known as jpena|off | 13:03 | |
| *** ysandeep is now known as ysandeep|afk | 13:14 | |
| jm1 | rlandy: rhos17 is zed, isnt it? | 13:15 |
| jm1 | rlandy: oh its wallaby | 13:15 |
| jm1 | rlandy: zed is 17.1? | 13:15 |
| rlandy | jm1: 17 and 7.1 are wallaby | 13:16 |
| rlandy | 18 will be zed | 13:16 |
| rlandy | 17.1 is a minor release off major 17 | 13:16 |
| rlandy | per the RHOS release schedule | 13:16 |
| jm1 | rlandy: ok great, then we have time | 13:16 |
| rlandy | like 16.2 is still train | 13:16 |
| rlandy | jm1: well yes and no | 13:17 |
| rlandy | when zed is cut, the sprint after we will need an 18 line | 13:17 |
| jm1 | rlandy: "time" > we dont have to raise a 17 blocker | 13:17 |
| rlandy | the fun keeps rolling | 13:17 |
| *** amoralej|lunch is now known as amoralej | 13:28 | |
| rlandy | jm1: bhagyashris|ruck: nedd help with anything? if not, pls ping when you are EoD with what to watch | 13:30 |
| jm1 | rlandy: nope, ack, thanks :) | 13:30 |
| rlandy | arxcruz: sorry - had to drop review call (kinit issues) - are you all set with the allow list reviews? | 13:31 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 13:31 |
| arxcruz | rlandy yes | 13:31 |
| arxcruz | rlandy need more cores though to merge | 13:31 |
| * rlandy looks | 13:32 | |
| rlandy | arxcruz: https://review.opendev.org/c/openstack/tripleo-quickstart/+/839725 merged | 13:33 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-quickstart/+/851963 - mreging that | 13:33 |
| marios | ysandeep|afk: o/ hey i am looking at the toolbox nodes on irc - maybe we can talk about that briefly tomorrow morning? I am lookign for somewhere to migrate our trello card creation script to | 13:34 |
| marios | ysandeep|afk: toolbox nodes on *psi* sorry not irc ;) | 13:35 |
| rlandy | marios: ack - just noting the trello keys | 13:38 |
| rlandy | it should be part of the infra tasks | 13:39 |
| rlandy | marios: although you'll have a lot to do next sprint with zed | 13:39 |
| rlandy | so maybe we punt on that task | 13:39 |
| *** ysandeep|afk is now known as ysandeep | 13:41 | |
| ysandeep | marios, ack lets chat tomorrow o/ | 13:42 |
| ykarel | Hi is the issue already known, i have seen it twice recently | 13:42 |
| rlandy | arxcruz: for fs035 https://logserver.rdoproject.org/93/855493/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035/48aabde/job-output.txt | 13:42 |
| ykarel | stderr: Error: error creating container storage: the container name "container-puppet-rabbitmq" is already in use by | 13:42 |
| ykarel | bhagyashris|ruck, jm1|rover ^ | 13:42 |
| rlandy | can you see the error on that log | 13:42 |
| ykarel | https://logserver.rdoproject.org/82/857182/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/7ea04b3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 13:42 |
| marios | rlandy: sure was just exploring should not be a hard thing to do so lets see | 13:42 |
| rlandy | An unhandled exception occurred while templating '{% if ('periodic' in zuul.pipeline and not job.force_non_periodic|default(false)|bool) or (job.force_periodic|default(false)|bool) -%} {{ tempest_allowed_group_periodic }} | 13:42 |
| *** ysandeep is now known as ysandeep|out | 13:43 | |
| rlandy | jm1: ^^ you familiar with ykarel's error? don;t see it listed in https://hackmd.io/dKeK6zo9R66heikGyCb4NA? | 13:44 |
| rlandy | centos8 | 13:44 |
| ykarel | rlandy, jm1 seen in c9 too https://logserver.rdoproject.org/03/853503/3/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/51d6860/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 13:45 |
| marios | rlandy: ysandeep|out: chandankumar: please add to reviews when you next have time https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/428027/ | 13:49 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 13:49 |
| rlandy | ykarel: k - will add to list to check/bug | 13:51 |
| ykarel | Thanks rlandy | 13:51 |
| bhagyashris|ruck | rlandy, nope thanks | 13:56 |
| *** Guest305 is now known as dasm | 13:57 | |
| dasm | o/ | 13:57 |
| rlandy | marios: jm1: added some comments to https://bugs.launchpad.net/tripleo/+bug/1989452 | 14:03 |
| rlandy | will discuss at CIX | 14:03 |
| rlandy | coming in before we lay down repos | 14:03 |
| dasm | marios: o/ i see you're in the osp retro meeting. are you gonna speak up? i think i'm missing the purpose of that meeting | 14:04 |
| dasm | oh, rlandy is there too ^ | 14:04 |
| marios | dasm: no i don't plan to.. rlandy asked me to join to see if we can do a better job in the next cycle with linking/reporting our stuff up to OSP | 14:05 |
| dasm | ack | 14:05 |
| marios | dasm: in fact, i see we aren't even in that dashboard as far as i can see | 14:05 |
| marios | dasm: i checked dfg a b c and i dont see us? or missed it? | 14:05 |
| dasm | marios: i found tripleo in dfg c group. but there is no tripleo ci | 14:05 |
| marios | dasm: right, so i don't know if we are even going to be asked for this bit | 14:06 |
| dasm | ack | 14:06 |
| marios | dasm: hmm we might be 'tripleo' but not sure | 14:07 |
| marios | anyway lets see :) | 14:07 |
| dasm | it's testing rbac policies. but it does not seem to be anyhow affecting us. | 14:09 |
| dviroel | jm1: fyi "Configured CPU model: qemu64 is not compatible with host CPU" | 14:09 |
| rlandy | marios: we're a dependent team | 14:09 |
| dasm | actually, it's tempest plugin for rbac testing | 14:09 |
| *** frenzyfriday is now known as frenzyfriday|lunch | 14:11 | |
| jm1 | dviroel: ok great. so the workaround to change cpu model in l1 (for l2) does not work | 14:16 |
| dasm | jm1: if the workaround does not work, is it great or not so great? :) | 14:17 |
| jm1 | dasm: it means we cannot do anything, we need help from psi | 14:17 |
| jm1 | dviroel: can you put your findings into cix card? really laborious investigation | 14:18 |
| dviroel | jm1: testing another nodeset, it seems that only master branch is failing, and wallaby job has a different nodeset - maybe there was a change in those flavors | 14:19 |
| dasm | jm1: from that PoV, you're right. but it still means we're gonna have issus. which is not good :/ | 14:20 |
| jm1 | dasm: yeah i should have put sarkasm tags on that comment ;) | 14:20 |
| dasm | jm1: /s ;) | 14:21 |
| jm1 | dasm: [Error] s: Unknown command. XD | 14:21 |
| rlandy | ugh - how is it a long da already at 10:22 am? | 14:22 |
| dasm | jm1: i knew it! you're a robot! Android spotted! We don't need Voight-Kampff test anymore :) | 14:22 |
| dasm | (Blade Runner any1?) | 14:22 |
| marios | dasm: rlandy: so confirmed looks like we are 'tripleo' on that dashboard and we have only 1 story in that cycle mixed rhel ci so i can speak to that | 14:22 |
| bhagyashris|ruck | rlandy, fs020 wallaby passed on ibm cloud https://review.rdoproject.org/r/c/testproject/+/44874/4#message-2a666f10647f375dcce104df36ab8ff76ccecf82 | 14:22 |
| bhagyashris|ruck | may be we can merge tempest skiplist patch https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/856886 | 14:23 |
| dasm | marios: ack. | 14:23 |
| rlandy | bhagyashris|ruck: ok so ... https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/856886/3/roles/validate-tempest/vars/tempest_skip.yml | 14:23 |
| rlandy | ok for me to merge that? | 14:23 |
| rlandy | then we can close out the bug and card | 14:23 |
| bhagyashris|ruck | yeah | 14:23 |
| rlandy | merging | 14:24 |
| bhagyashris|ruck | ok | 14:24 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1980255 is closed | 14:24 |
| rlandy | I guess that card just moves to done | 14:25 |
| jm1 | rlandy, marios: little present for friday. hopefully this helps a bit. autogenerated rr notes https://hackmd.io/8MEZ5R-USXGyRTDiamNW2Q?view | 14:25 |
| rlandy | thanks | 14:26 |
| jm1 | rlandy, marios: one thing is missing though, the autogenerated rerun testproject's. only locally available for now | 14:26 |
| marios | jm1: :) nice | 14:26 |
| jm1 | dasm, frenzyfriday|lunch: ^ | 14:26 |
| jm1 | marios, rlandy: it has to be triggered manually, its ultra hacky for now and only works on my machine. but at least it works ^^ | 14:27 |
| dasm | jm1: wow. nice! | 14:28 |
| jm1 | marios, rlandy: it also gives you a first idea of what the failure reason was for each job | 14:28 |
| dasm | show us the code, mate! :) | 14:28 |
| dasm | jm1: nice job! i was slowly moving towards something like that. I'm glad someone took that over from me :) | 14:29 |
| jm1 | dasm: i will, for sure. in its current state i cannot publish it without jeopardizing my dev career. gimme some time to clean it up please | 14:29 |
| dasm | jm1: haha | 14:29 |
| rlandy | CIX time | 14:29 |
| rlandy | marios: can you confirm: https://review.rdoproject.org/r/c/rdoinfo/+/44884 | 14:46 |
| rlandy | that this is the right version | 14:46 |
| rlandy | apevec wants to merge | 14:47 |
| marios | rlandy: gimme couple mins still in this call (looking) | 14:49 |
| marios | rlandy: yeah i mean ykarel test looks good https://review.opendev.org/c/openstack/tripleo-heat-templates/+/857182/2#message-566911cde7b0a87f0259463c1b87ccd1a2f30f2b | 14:50 |
| marios | rlandy: so lets merge | 14:50 |
| rlandy | marios: thanks | 14:54 |
| *** jpena|off is now known as jpena | 14:55 | |
| rlandy | jm1: bhagyashris|ruck: marios: logging vexx ticket | 15:03 |
| bhagyashris|ruck | rlandy, ok | 15:04 |
| bhagyashris|ruck | rlandy, jm1 leaving for the day... nothing major to watch | 15:05 |
| rlandy | bhagyashris|ruck: thanks - have a goo dnight | 15:05 |
| jm1 | bhagyashris|ruck: have a nice evening :) | 15:06 |
| dasm | pojadhav: i'm gonna try and i'm gonna see if i can help you with that: https://review.opendev.org/c/openstack/tripleo-ci/+/856051 | 15:06 |
| pojadhav | dasm, ack thank you! | 15:13 |
| marios | off in few rlandy need something ? | 15:19 |
| rlandy | marios: should be ok - thank you | 15:21 |
| *** dviroel is now known as dviroel|lunch | 15:21 | |
| rlandy | have a good night | 15:21 |
| dasm | pojadhav: i found the issue with https://review.opendev.org/c/openstack/tripleo-ci/+/856051 | 15:24 |
| dasm | please see comments | 15:24 |
| marios | thanks you too rlandy | 15:28 |
| *** marios is now known as marios|out | 15:28 | |
| jm1 | ykarel: finally managed to look through our failures for c9 master. it looks like we are hitting intermittent failures as everyday, nothing unusual. did not see your container failure. at least for this job it does not seem to be persistent https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&skip=0 | 15:33 |
| jm1 | ykarel: "usual" nowadays means dozens of overcloud deployment failures in each way possible per day | 15:35 |
| ykarel | jm1, so it's not specific to master nor c9, as seen also for c8 wallaby | 15:35 |
| ykarel | yep i see most of the time it failing for galera | 15:35 |
| jm1 | ykarel: yeah i checked the logs you provided and as much as i hate to write that, my guess is it is intermittent | 15:35 |
| ykarel | jm1, yes intermittent but looks like product bug | 15:36 |
| ykarel | so worth investigating/tracking | 15:36 |
| jm1 | ykarel: thing is, if we would track every intermittent bug that hits use three times, then we would spam our cix board. yesterday i had to trigger c9 wallaby fs64 dozens of times (20? 30?) until we got a promotion. it failed on different intermittent failures each and every damn time. | 15:39 |
| jm1 | ykarel: for now we focus on the really nasty intermittent failures. we have currently 6 bugs which are waiting for reports/cix cards https://hackmd.io/dKeK6zo9R66heikGyCb4NA | 15:41 |
| jm1 | ykarel: but feel free to add intermittent bugs which you see more often. i only have the two bug reports you sent and did not see it anywhere else so far | 15:42 |
| jm1 | ykarel: but didn't rlandy file a bug already? | 15:43 |
| rlandy | sorry rlandy is on other channel | 15:44 |
| rlandy | ykarel: agree worth tracking | 15:44 |
| rlandy | just didn't get to file that bug | 15:44 |
| rlandy | yet | 15:44 |
| rlandy | also - OVB is in bad shape right now | 15:45 |
| rlandy | starting with mirrors on vess | 15:45 |
| rlandy | vexx | 15:45 |
| rlandy | first thing we logged out today | 15:45 |
| rlandy | will get to deploy failures after that | 15:45 |
| ykarel | jm1, i am not following those jobs, just seen on two of my patches, so felt like it's more generic then it seems | 15:47 |
| ykarel | if those jobs report to logstash, there might get better picture of frequency | 15:47 |
| ykarel | the intermittent failures can sometime be product bugs so atleast it's good to track | 15:48 |
| ykarel | so if the errors here are not related to what seen on other intermittent fails then it might be seperate issue | 15:49 |
| chandankumar | see ya! | 15:52 |
| jm1 | ykarel: agree. but we have to automate the log collection. its not manageable at the moment, its flooding ruck/rovers | 15:55 |
| rlandy | frenzyfriday|lunch: ^^ what's our status on RDO elastic recheck? | 15:55 |
| *** frenzyfriday|lunch is now known as frenzyfriday | 15:55 | |
| jm1 | ykarel: but lucky us, frenzyfriday is working on that :) | 15:55 |
| frenzyfriday | rlandy, no much progress this week. It is deployed on the server but graph is still messed up | 15:56 |
| jm1 | rlandy: having a break now. integration jobs are running, but dont expect any promotions. usual suspects. | 15:57 |
| dasm | jm1: wrt logs, there is opensearch which we can incorporate into log collection | 15:58 |
| rlandy | jm1: it's fine | 16:01 |
| rlandy | you're in good shape promotion-wise | 16:01 |
| rlandy | waiting to hear back from vexx | 16:01 |
| rlandy | https://review.opendev.org/c/zuul/zuul-jobs/+/857730 | 16:02 |
| rlandy | should help if it gets through | 16:02 |
| rlandy | frenzyfriday: sorry - joining | 16:02 |
| jparker_ | I'm attempting to use tripleo-quickstart to deploy 1ctlr_2comp_1supp with freeipa against master-tripleo-ci, when it gets to ansible task of creating the freeipa script on the node it fails due to ansible_distribution_major_version, does anyone happen to know if I am missing necessary parameters for deployment? | 16:09 |
| jparker_ | https://paste.opendev.org/show/bpmeJOq6R9071zO6zaLv/ | 16:10 |
| jparker_ | sorry here is a better format of what i am running instead of single line: https://paste.opendev.org/show/bXqbZNUW0Mr73JSekeuv/ | 16:12 |
| rlandy | lunch brb | 16:23 |
| rlandy | rcastillo: ^^ we have a taker for the virt deploy | 16:24 |
| rlandy | jparker_: hi ... | 16:26 |
| rlandy | jparker_: do you need this for debug? | 16:27 |
| rlandy | might be easier if we start a job run and hold a node for you | 16:27 |
| jparker_ | preferably would like to come up with something I can do locally, its an effort for NOVA QE to start adding additional upstream sooner for nova features/deployment and several features will require some interfacing with barbican to test | 16:29 |
| rlandy | jparker_: ansible_distribution_major_version shoudl be picked up from the node | 16:29 |
| rlandy | can it not find the var? | 16:29 |
| rlandy | ie: var not defined or it fails on that particular node | 16:30 |
| jparker_ | yea its var undefined, specifically AnsibleUndefinedVariable: 'ansible_distribution_major_version' is undefined | 16:30 |
| rlandy | you can add the definition ... | 16:31 |
| rlandy | but it should be found | 16:31 |
| rlandy | https://github.com/openstack/tripleo-quickstart/search?p=2&q=ansible_distribution_major_version | 16:31 |
| rlandy | example | 16:31 |
| rlandy | rcastillo: ^^ can you help out here, please | 16:32 |
| rlandy | jparker_: that config of 1ctlr_2comp_1supp is usaually an OVB deloyment | 16:33 |
| rlandy | so I am not sure how the virt system will deal with it | 16:33 |
| rlandy | now really lunch - biab | 16:36 |
| *** dviroel|lunch is now known as dviroel | 16:43 | |
| *** amoralej is now known as amoralej|off | 16:58 | |
| *** jpena is now known as jpena|off | 17:00 | |
| rlandy | wallaby c8 should promote | 17:14 |
| jm1 | rlandy: will create bug report for ykarel, have some time now | 18:15 |
| rlandy | jm1: k - working with the vexxhost folks atm | 18:21 |
| *** rlandy is now known as rlandy|mtg | 18:28 | |
| jm1 | ykarel, rlandy: container creation issue during overcloud deploy https://bugs.launchpad.net/tripleo/+bug/1989606 | 18:28 |
| rlandy|mtg | jm1: thanks - will look after meeting | 18:28 |
| jm1 | rlandy|mtg: nothing to see there what you do not know already ;) | 18:33 |
| rcastillo | sorry, had an emergency | 18:40 |
| rcastillo | here now... | 18:40 |
| rcastillo | rlandy|mtg: jparker_ looking | 18:40 |
| jparker_ | rcastillo: it looks like the supplemental vm is not getting the same configuration setup as the other nodes, none of the centos ci repos are present and ansible is not installed on the ipa node | 18:52 |
| rcastillo | jparker_: yeah that's what I was thinking, we don't do supplemental with quickstart.sh afaik | 18:53 |
| rcastillo | maybe check whether deploy_supplemenal_node is set, and set it by hand otherwise | 18:59 |
| rcastillo | https://opendev.org/openstack/tripleo-quickstart/src/commit/1a70b73087551956392a04f9d9e0724ccb7597df/roles/libvirt/setup/meta/main.yml#L19 | 18:59 |
| jparker_ | rcastillo: ack, it should be coming from https://opendev.org/openstack/tripleo-quickstart/src/commit/1a70b73087551956392a04f9d9e0724ccb7597df/config/general_config/ipa.yml, but let me see if I am overriding it someohw | 19:02 |
| jm1 | rlandy|mtg: out for today, jobs are still running | 19:03 |
| * jm1 out for today, have a nice evening folks 🥂 | 19:03 | |
| rlandy|mtg | jm1: thanks - have a good night | 19:04 |
| *** rlandy|mtg is now known as rlandy | 19:23 | |
| rlandy | train rerun: https://review.rdoproject.org/r/c/testproject/+/31165 | 20:11 |
| * dviroel need to go afk - haircut | 20:26 | |
| *** dviroel is now known as dviroel|afk | 20:26 | |
| * dasm => offline | 21:27 | |
| *** dasm is now known as dasm|off | 21:27 | |
| dasm|off | o/ | 21:27 |
| *** rlandy is now known as rlandy|bbl | 22:01 | |
| *** dviroel|afk is now known as dviroel | 22:12 | |
| dviroel | fips enabled images are now starting, next step is to investigate failure on pre playbook, probably related to fips itself | 22:52 |
| *** dviroel is now known as dviroel|afk | 22:52 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!