*** rlandy|bbl is now known as rlandy|out | 00:02 | |
*** ysandeep|out is now known as ysandeep | 04:53 | |
*** ysandeep is now known as ysandeep|ruck | 04:55 | |
*** akahat|ruck is now known as akahat | 05:42 | |
ysandeep|ruck | ykarel: good morning o/ when you get a chance could you please take a look at https://bugs.launchpad.net/tripleo/+bug/1980333 , neutron is returning 504 during network provision - all ovb jobs are impacted. | 06:32 |
---|---|---|
ykarel | ysandeep|ruck, can you also run the same job in downstream to rule out infra involvement, reading comments that looks suspecious | 06:34 |
ykarel | will check more in some time on this | 06:34 |
ysandeep|ruck | ykarel, ack | 06:35 |
ysandeep|ruck | fyi.. noticed few tracebacks but not sure if they are related: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/93419a0/logs/undercloud/var/log/extra/errors.txt.gz | 06:36 |
ysandeep|ruck | ykarel, fyi.. the scheduled run of upstream jobs on downstream failed due to node_failure.. I have rechecked them - https://review.rdoproject.org/r/c/tripleo-downstream-trigger-nested-virt/+/43823 | 06:42 |
ysandeep|ruck | hmm.. last few run failed due to node_failure - https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-master%09&skip=0 | 06:43 |
ysandeep|ruck | if it this time - I will check with infra incase something wrong with nodepool centos image | 06:43 |
*** amoralej|off is now known as amoralej | 07:20 | |
*** ysandeep|ruck is now known as ysandeep|lunch | 07:28 | |
*** jpena|off is now known as jpena | 07:42 | |
ykarel | ysandeep|lunch, okk if it still fails with node failure | 07:43 |
ykarel | likely it's same issue that was yesterday in rdo for centos 9 image | 07:43 |
akahat | marios, o/ | 08:30 |
akahat | marios, whenever you are free we can have chat about mixed deployment | 08:31 |
marios | akahat: o/ sure | 09:07 |
marios | akahat: in 10 mins ok for you? | 09:07 |
marios | akahat: sent you invite | 09:09 |
*** ysandeep|lunch is now known as ysandeep|ruck | 09:22 | |
akahat | marios, ack. received. | 09:28 |
ysandeep|ruck | ykarel: yes it failed again, what was the issue in rdo yesterday for centos9 image? | 09:29 |
ykarel | ysandeep|ruck, <fbo> cloud-centos-9 no longer boot "[ 0.268001] usbserial: usb_serial_init - returning with error -19" | 09:33 |
marios | akahat: ack | 09:33 |
marios | sec | 09:33 |
ykarel | ysandeep|ruck, just back from lunch, still checking the issue, now feel like it's not related to infra | 09:34 |
ysandeep|ruck | ykarel, we need same for downstream: https://softwarefactory-project.io/r/c/config/+/25393 | 09:35 |
ykarel | yeap | 09:35 |
ysandeep|ruck | ykarel, thanks proposing fix | 09:36 |
marios | https://issues.redhat.com/browse/TRIPLEOCI-1039 | 09:36 |
marios | akahat: ^^ | 09:36 |
ykarel | ysandeep|ruck, just to confirm, you checked downstream also fails with same boot failure? | 09:37 |
ykarel | if yes then same fix should work | 09:38 |
ykarel | anyway it's good to get latest images | 09:38 |
marios | https://logserver.rdoproject.org/58/43558/10/check/tripleo-ci-centos-8-9-mixed-os/1085ea0/zuul-info/inventory.yaml | 09:43 |
ysandeep|ruck | ykarel, yes tested same issue: | 09:49 |
ysandeep|ruck | [ 0.566265] usbserial: usb_serial_init - registering generic driver failed | 09:49 |
ysandeep|ruck | [ 0.567387] usbserial: usb_serial_init - returning with error -19 | 09:49 |
ykarel | ack | 09:49 |
ysandeep|ruck | ykarel, https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/418137 similiar fix | 09:51 |
ykarel | k | 09:52 |
marios | https://logserver.rdoproject.org/58/43558/10/check/tripleo-ci-centos-8-9-mixed-os/1085ea0/logs/quickstart_files/hosts | 10:00 |
*** pojadhav is now known as pojadhav|afk | 10:20 | |
*** rlandy|out is now known as rlandy | 10:30 | |
rlandy | ysandeep|ruck: hello - how are things? | 10:32 |
rlandy | want to sync? | 10:32 |
rlandy | ysandeep|ruck: gate blockers should be cleared now | 10:33 |
ysandeep|ruck | good morning, ack for sync.. let me grab my headphones | 10:33 |
rlandy | ysandeep|ruck, https://meet.google.com/bjk-ksqm-fqf?pli=1&authuser=0 | 10:35 |
rlandy | when ready | 10:35 |
*** ysandeep|ruck is now known as ysandeep|brb | 10:51 | |
*** ysandeep|brb is now known as ysandeep | 11:05 | |
*** pojadhav|afk is now known as pojadhav | 11:15 | |
frenzy_friday | hey folks, review time | 11:16 |
rlandy | jm1[m]: can you look at the review list from June 30 | 11:21 |
rlandy | there is one patch from Sagi | 11:21 |
rlandy | we are not sure what to do here | 11:22 |
rlandy | left some notes | 11:22 |
*** dviroel|afk is now known as dviroel | 11:26 | |
jm1 | rlandy: yes, i added it to the review list yesterday because its open for a long time and we did not want it to get lost. its not urgent, so lets wait for sagi to answer the questions | 11:26 |
jm1 | will ping him on sunday about it, he is off today | 11:27 |
rlandy | ysandeep: so, we merged the master base image change yesterday | 11:35 |
rlandy | did that impact OVB? | 11:35 |
rlandy | if not, we can merge the name change and criteria | 11:35 |
rlandy | marios: frenzy_friday: ^^ fyi | 11:35 |
ysandeep | rlandy, master ovb are currently failing on network provision - we are running same fs001 in downstream to narrow down the issue (if infra related). | 11:36 |
rlandy | ysandeep: ok - if you can confirm it's not container related, we can merge | 11:37 |
rlandy | let me know | 11:37 |
rlandy | ysandeep: marios confirmed we can stop running victoria | 11:41 |
rlandy | rcastillo: ^^ you can put in a patch to remove the line | 11:41 |
rlandy | and components | 11:41 |
rlandy | to save resorces | 11:42 |
ysandeep | rlandy, ack | 11:42 |
marios | thanks ysandeep rlandy | 11:47 |
ysandeep | rlandy marios rcastillo: https://review.rdoproject.org/r/c/config/+/43828 | 11:53 |
ysandeep | reviewbot, please add in review list https://review.rdoproject.org/r/c/config/+/43828 | 11:53 |
reviewbot | I have added your review to the Review list | 11:53 |
rlandy | ysandeep: you have two +2s there | 11:56 |
rlandy | can merge when ready | 11:56 |
ysandeep | ack, as soon as zuul reports | 11:56 |
rlandy | we will need the to remove trigger | 11:57 |
rlandy | getting | 11:57 |
rlandy | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/upstream.yaml#L176 | 11:58 |
ysandeep | rlandy, we still have openstack-periodic-integration-stable3 there - and ussuri is gone | 11:59 |
rlandy | ysandeep; rdo had some jobs there | 12:00 |
rlandy | that is why | 12:00 |
jm1 | rlandy: gimme a minute 🙈 | 12:00 |
rlandy | ie: on ussuri | 12:00 |
rlandy | jm1: no worries | 12:00 |
rlandy | ping when ready | 12:00 |
ysandeep | ack.. so for stopping our jobs from running i think removing templates should be enough for now | 12:01 |
ykarel | ysandeep, downstream tests not needed, i figured out the issue | 12:01 |
ykarel | it's not infra related | 12:01 |
ykarel | me testing fix in https://review.rdoproject.org/r/c/testproject/+/43803 | 12:01 |
ykarel | updating lp | 12:01 |
ysandeep | ykarel++ nice | 12:01 |
ykarel | ysandeep, there is also a tooling issue related to container updates | 12:02 |
ykarel | if you remember that was seen in past too | 12:02 |
ykarel | and also need to check if network component was force promoted | 12:02 |
ysandeep | rlandy, jm1 akahat ^^ was network component force promoted? | 12:03 |
rlandy | ysandeep: no | 12:03 |
rlandy | tetsproject rerun | 12:03 |
ysandeep | ack | 12:04 |
ykarel | commented https://bugs.launchpad.net/tripleo/+bug/1980333/comments/5 | 12:04 |
ysandeep | rlandy, zuul reported back - could you please +w https://review.rdoproject.org/r/c/config/+/43828 | 12:04 |
ysandeep | ykarel, thanks reading | 12:04 |
ykarel | rlandy, which jobs? | 12:04 |
amoralej | we are bumping tempest in wallaby in https://review.rdoproject.org/r/c/rdoinfo/+/43785 | 12:08 |
amoralej | let me know if anything explodes | 12:08 |
amoralej | hopefully not :) | 12:08 |
ysandeep | amoralej, Needed for downstream import? | 12:08 |
amoralej | dunno, tbh, mkopec requested it | 12:09 |
amoralej | i'm not sure if it comes from downstream requirement | 12:09 |
amoralej | kopecmartin, ^ | 12:09 |
ysandeep | next week we are stopping import for 17.0 - just wondering if this is the right time - because if anything explore that mean we risk downstream dates. | 12:09 |
ysandeep | cc rlandy ^^ | 12:10 |
kopecmartin | we want to update tempest because of the import next week, the newer version contains fixes for some sshable issues and we want to get them downstream | 12:10 |
rlandy | ysandeep: amoralej: pls don;t do anything that is not absolutely needed for 17 right now | 12:10 |
rlandy | we have to get all patches onto import by july 6th | 12:11 |
rlandy | after that - go wild | 12:11 |
amoralej | rlandy, from kopecmartin comment, we want that for 17 | 12:12 |
ysandeep | amoralej, kopecmartin we are not running any tripleo jobs in check here: https://review.rdoproject.org/r/c/rdoinfo/+/43785 | 12:12 |
* ysandeep recall we used to trigger some tripleo ci specific jobs in rdoinfo patches | 12:13 | |
amoralej | ysandeep, only in master | 12:13 |
amoralej | but you can run jobs with depends on | 12:14 |
amoralej | mmmm | 12:14 |
amoralej | i've removed +W | 12:14 |
ysandeep | I am not sure depends-on work with rdo-info patches, ykarel ^^ have you ever tried depends-on with rdo-info patch on testproject? | 12:16 |
amoralej | ysandeep, i can provide a temporary repo with the new packages | 12:16 |
amoralej | so that we can create a dnm patch to the release config | 12:16 |
amoralej | i'm pretty sure we did that in the past | 12:16 |
ysandeep | ack that will do | 12:16 |
* ykarel reads back | 12:16 | |
ykarel | ysandeep, yes amoralej right it would work, depends-on on rdo-jobs where you define the custom jobs | 12:18 |
ykarel | that how it used to test in past | 12:18 |
ysandeep | by any chance someone have an example from past? | 12:19 |
ykarel | ysandeep, https://review.rdoproject.org/r/c/rdoinfo/+/43460 | 12:19 |
ykarel | u can use same | 12:20 |
amoralej | ah, i was thinking in something different | 12:21 |
amoralej | but ok | 12:21 |
amoralej | i was thinking in patching tripleo-quickstart release config for wallaby | 12:21 |
amoralej | and run patches on tripleo-ci or testproject or whatever with depends-on on that one | 12:22 |
amoralej | so that we may have anything | 12:22 |
amoralej | but i'm find in doing that too | 12:22 |
amoralej | ykarel, btw, there is anything that prevents us to merge https://review.rdoproject.org/r/c/rdo-jobs/+/40247 ? | 12:23 |
ykarel | amoralej, looking | 12:23 |
amoralej | we are not doing anything wrong there, right? | 12:23 |
amoralej | even add the jobs in experimental pipeline for changes in wallaby.yml | 12:24 |
amoralej | it'd be fine from my side | 12:24 |
amoralej | or even in check | 12:24 |
amoralej | np | 12:24 |
amoralej | my issue is that we don't run all the scenarios there, that's why i was thinking in the other approach | 12:24 |
rlandy | sorry - was on 1-1 | 12:29 |
rlandy | reading back | 12:29 |
amoralej | ykarel, ysandeep https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439 | 12:29 |
rlandy | ysandeep: https://review.rdoproject.org/r/c/config/+/43828 w+'ed | 12:29 |
*** ysandeep is now known as ysandeep|ruck | 12:30 | |
rlandy | amoralej: ysandeep: re: "from kopecmartin comment, we want that for 17" if needed for 17 ok - pls go ahead with above | 12:31 |
ysandeep|ruck | rlandy, thanks.. we are just discussing strategies to test that patch - so that it won't haunt us later. | 12:32 |
amoralej | ysandeep|ruck, https://zuul.opendev.org/t/openstack/status#848439 with that at least we'll have a couple of patches on centos9-wallaby | 12:33 |
amoralej | but it'd be good to run all relevant jobs with a depens-on with the tripleo-quickstart one | 12:34 |
rlandy | thanks | 12:34 |
ykarel | amoralej, wrt https://review.rdoproject.org/r/c/rdo-jobs/+/40247, yes should be fine to those jobs in tags update in stable branches, but good to rename like other rdoinfo jobs | 12:35 |
ykarel | can add those in experimental pipeline | 12:35 |
ysandeep|ruck | amoralej++ thanks I think your approach will work great, I can use your patch as depends on - But I think we need to add priority there.. | 12:35 |
amoralej | ykarel, let me check what naming convention we used ... | 12:36 |
ykarel | amoralej, and wrt https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439, need to add priority higher for that repo | 12:36 |
ysandeep|ruck | amoralej, ykarel I will update https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439 for priority , will also update the periodic version release file. | 12:37 |
amoralej | damn | 12:38 |
amoralej | ysandeep|ruck, ok | 12:38 |
amoralej | thanks | 12:38 |
ysandeep|ruck | ykarel, thanks for your comment on that network provision issue, so we need to wait for neutron fix to merge - I see its +wed already | 12:40 |
ysandeep|ruck | and we need to deal with container updates issue separately - to make sure all containers are properly updated. | 12:42 |
ykarel | amoralej, sorry i recalled wrong, namings looks right | 12:45 |
rlandy | ysandeep|ruck; on second thoughts, you're right - let's keep stable-2 line as it will be wallaby | 12:47 |
rlandy | when zed cuts | 12:47 |
ysandeep|ruck | yes | 12:48 |
ykarel | ysandeep|ruck, yes to avoid leaking commits | 12:48 |
ysandeep|ruck | ykarel, amoralej updated https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439 for testing, I have set priority as 1, so that it win in all cases. | 12:54 |
amoralej | yep, i saw | 12:54 |
amoralej | same priority as default dlrn repos, so it should be fine | 12:54 |
* ysandeep|ruck running testproject | 12:55 | |
rlandy | hmmm ... still have ntp issues on 17 | 12:56 |
rlandy | can look after 1-1 | 12:56 |
rlandy | ysandeep|ruck: ^^ | 12:57 |
ysandeep|ruck | ack, will check in few | 12:57 |
ysandeep|ruck | rlandy, 17 is not failing on ntp, other legit issue: https://sf.hosted.upshift.rdu2.redhat.com/logs/7f/7fb5373a0bb4270186e544be1f041de0da9c635c/openstack-periodic-integration-rhos-17-rhel9/periodic-tripleo-ci-rhel-9-standalone-rhos-17/7002d62/logs/undercloud/var/log/extra/errors.txt | 13:21 |
ysandeep|ruck | ERROR keystone TypeError: set() got an unexpected keyword argument 'dead_retry' | 13:21 |
* ysandeep|ruck reporting bug | 13:21 | |
ysandeep|ruck | Tengu, ^^ this should go to security right? jobs are failing during keystone_bootstrap during deployment. | 13:24 |
rlandy | ysandeep|ruck: lol - at least it wasn't us | 13:25 |
rlandy | ysandeep|ruck: weird you did not see that on wallaby | 13:26 |
ysandeep|ruck | rlandy, I suspect someone merged a downstream patch | 13:26 |
ysandeep|ruck | without merging in wallaby first | 13:27 |
* ysandeep|ruck will debug more after reporting bug | 13:27 | |
rcastillo | o/ | 13:31 |
*** rcastillo is now known as rcastillo|rover | 13:31 | |
*** pojadhav is now known as pojadhav|afk | 13:34 | |
rlandy | rcastillo|rover: hey | 13:39 |
rlandy | rcastillo|rover: welcome to the fun | 13:40 |
rcastillo|rover | what's going on? | 13:40 |
rlandy | ysandeep|ruck has it under control as usual | 13:41 |
rlandy | rhos-17 on rhel-9 issue with ntp above | 13:42 |
rlandy | master ovb falures | 13:42 |
rlandy | but I'll let you guys sync | 13:42 |
rlandy | rcastillo|rover: new rr hackmd https://hackmd.io/pRC9TDaoQLWqRSQioTUl5g | 13:42 |
rcastillo|rover | I'm reading through it atm | 13:42 |
rlandy | ysandeep|ruck: rcastillo|rover: logging out soon - need anything before I go? | 13:44 |
ysandeep|ruck | rlandy, all good | 13:45 |
ysandeep|ruck | rcastillo|rover, lets sync | 13:45 |
rcastillo|rover | rlandy: enjoy your time off | 13:45 |
rcastillo|rover | ysandeep|ruck: ack | 13:45 |
rlandy | ysandeep|ruck: rcastillo|rover: thanks for holding things together | 13:46 |
rlandy | see you monday | 13:46 |
rcastillo|rover | ysandeep|ruck: meet.google.com/xiq-mzub-uuy whenever you're ready | 13:46 |
rcastillo|rover | rlandy: o/ | 13:46 |
ysandeep|ruck | rcastillo|rover, https://review.opendev.org/c/openstack/neutron/+/848396 | 13:49 |
ysandeep|ruck | rcastillo|rover, https://bugzilla.redhat.com/show_bug.cgi?id=2103117 | 13:53 |
Tengu | '/13 | 13:57 |
Tengu | ysandeep|ruck: keystone is security, yep. | 13:57 |
marios | no one on happy friday o/ lemme know if you want to join i dropped i was foreveralone there | 14:01 |
ysandeep|ruck | rcastillo|rover, https://review.rdoproject.org/r/c/rdoinfo/+/43785 | 14:25 |
*** dasm|off is now known as dasm | 14:26 | |
dasm | o/ | 14:26 |
ysandeep|ruck | amoralej, I will be eod soon, fyi.. I am testing the tempest bump here: https://review.rdoproject.org/r/c/testproject/+/42693 - still running | 14:29 |
ysandeep|ruck | rcastillo|rover, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43832 | 14:37 |
amoralej | ysandeep|ruck, why do we need force_periodic? | 14:39 |
ysandeep|ruck | rcastillo|rover, https://review.rdoproject.org/r/c/testproject/+/38348 | 14:41 |
ysandeep|ruck | rcastillo|rover, https://review.rdoproject.org/r/c/testproject/+/38348 | 14:44 |
ysandeep|ruck | rcastillo|rover, https://review.rdoproject.org/r/c/testproject/+/32054 | 14:49 |
ysandeep|ruck | amoralej, hey o/ because we are running periodic job in check - we need to add force_periodic - so that it pick promotion- release file one and not the check one. | 14:50 |
ysandeep|ruck | only needed for periodic job | 14:51 |
amoralej | ysandeep|ruck, but for general gating, we don't want to use periodic but promoted content, right? | 14:51 |
ysandeep|ruck | amoralej, correct | 14:51 |
amoralej | i mean for rdoinfo gate | 14:51 |
amoralej | ok | 14:51 |
amoralej | thx | 14:51 |
ysandeep|ruck | for rdoinfo gate - we should test with our current-tripleo - so yeah no force_periodic | 14:52 |
ysandeep|ruck | rcastillo|rover, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-rhos-17/18a2cea/logs/undercloud/var/log/extra/rpm-list.txt | 14:59 |
ysandeep|ruck | last good job had python3-dogpile-cache-1.1.5-1.el9osttrunk.noarch | 15:00 |
dasm | arxcruz: i'm gonna need your attention on: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43815 | 15:06 |
dasm | you're the only person knowing about golang, and the author of the code. | 15:06 |
dasm | arxcruz: probably no one else is going to look into these changes :( | 15:07 |
arxcruz | dasm sure, may i do some tests first? :) | 15:07 |
dasm | arxcruz: do whatever you need. | 15:08 |
dasm | what kind of tests? like unittests? | 15:08 |
dasm | or manual tests? | 15:08 |
arxcruz | dasm i'll basically run it on a local cockpit environment | 15:09 |
dasm | k | 15:09 |
*** amoralej is now known as amoralej|off | 15:20 | |
*** dviroel is now known as dviroel|lunch | 15:25 | |
*** ysandeep|ruck is now known as ysandeep|dinner | 15:43 | |
*** jpena is now known as jpena|off | 15:59 | |
*** dviroel|lunch is now known as dviroel | 16:45 | |
ysandeep|dinner | rcastillo|rover, leaving for the day bbye o/ | 17:31 |
*** ysandeep|dinner is now known as ysandeep|out | 17:31 | |
dviroel | rcastillo|rover: hey, need help with something? | 17:31 |
rcastillo|rover | ysandeep|out: o/ | 17:32 |
rcastillo|rover | dviroel: mostly waiting on results, seems things are going ok | 17:32 |
dviroel | rcastillo|rover: ack, gate looks good | 17:32 |
dviroel | lot of changes to merge - one failure (mirror related) | 17:33 |
rcastillo|rover | yeah, still backed up from yesterday's blocker I think | 17:33 |
dviroel | yeah | 17:33 |
dviroel | hey rcastillo|rover, do you plan to use this change https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/847419 ? | 18:26 |
dviroel | if not, i will abandon | 18:26 |
rcastillo|rover | dviroel: go ahead and abandon, I'll try something different next week | 18:27 |
dviroel | ack | 18:29 |
* jm1 out for today. have a nice weekend, tripleo ci :) | 19:05 | |
dviroel | rcastillo|rover: hey - did you saw that https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-tht&project=openstack/tripleo-heat-templates ? | 20:33 |
dviroel | rcastillo|rover: filed a LP here https://bugs.launchpad.net/tripleo/+bug/1980552 | 21:06 |
* dasm => weekekd | 21:07 | |
dasm | Talk to you on Tuesday! Take care team! | 21:07 |
*** dasm is now known as dasm|off | 21:07 | |
dviroel | o/ | 21:08 |
*** dasm|off is now known as dasm | 21:09 | |
dviroel | dasm: back already? | 21:09 |
*** dasm is now known as Guest3965 | 21:10 | |
Guest3965 | for a few secs. i'm registering my username | 21:10 |
*** Guest3965 is now known as dasm | 21:12 | |
*** dasm is now known as dasm|off | 21:13 | |
*** dasm|off is now known as dasm | 21:13 | |
*** dasm is now known as dasm|off | 21:13 | |
dasm|off | ok, works :) | 21:13 |
dasm|off | o/ | 21:13 |
* dviroel going afk | 21:40 | |
*** dviroel is now known as dviroel|afk | 21:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!