Friday, 2022-07-01

*** rlandy|bbl is now known as rlandy|out00:02
*** ysandeep|out is now known as ysandeep04:53
*** ysandeep is now known as ysandeep|ruck04:55
*** akahat|ruck is now known as akahat05:42
ysandeep|ruckykarel: good morning o/ when you get a chance could you please take a look at https://bugs.launchpad.net/tripleo/+bug/1980333 , neutron is returning 504 during network provision - all ovb jobs are impacted.06:32
ykarelysandeep|ruck, can you also run the same job in downstream to rule out infra involvement, reading comments that looks suspecious06:34
ykarelwill check more in some time on this06:34
ysandeep|ruckykarel, ack06:35
ysandeep|ruckfyi.. noticed few tracebacks but not sure if they are related: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/93419a0/logs/undercloud/var/log/extra/errors.txt.gz 06:36
ysandeep|ruckykarel, fyi.. the scheduled run of upstream jobs on downstream failed due to node_failure.. I have rechecked them - https://review.rdoproject.org/r/c/tripleo-downstream-trigger-nested-virt/+/4382306:42
ysandeep|ruckhmm.. last few run failed due to node_failure - https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-master%09&skip=0 06:43
ysandeep|ruckif it this time - I will check with infra incase something wrong with nodepool centos image06:43
*** amoralej|off is now known as amoralej07:20
*** ysandeep|ruck is now known as ysandeep|lunch07:28
*** jpena|off is now known as jpena07:42
ykarelysandeep|lunch, okk if it still fails with node failure07:43
ykarellikely it's same issue that was yesterday in rdo for centos 9 image07:43
akahatmarios, o/08:30
akahatmarios, whenever you are free we can have chat about mixed deployment08:31
mariosakahat: o/ sure 09:07
mariosakahat: in 10 mins ok for you? 09:07
mariosakahat: sent you invite 09:09
*** ysandeep|lunch is now known as ysandeep|ruck09:22
akahatmarios, ack. received.09:28
ysandeep|ruckykarel: yes it failed again, what was the issue in rdo yesterday for centos9 image?09:29
ykarelysandeep|ruck, <fbo> cloud-centos-9 no longer boot "[    0.268001] usbserial: usb_serial_init - returning with error -19"09:33
mariosakahat: ack 09:33
mariossec09:33
ykarelysandeep|ruck, just back from lunch, still checking the issue, now feel like it's not related to infra09:34
ysandeep|ruckykarel, we need same for downstream: https://softwarefactory-project.io/r/c/config/+/2539309:35
ykarelyeap09:35
ysandeep|ruckykarel, thanks proposing fix09:36
marioshttps://issues.redhat.com/browse/TRIPLEOCI-103909:36
mariosakahat: ^^ 09:36
ykarelysandeep|ruck, just to confirm, you checked downstream also fails with same boot failure?09:37
ykarelif yes then same fix should work09:38
ykarelanyway it's good to get latest images09:38
marioshttps://logserver.rdoproject.org/58/43558/10/check/tripleo-ci-centos-8-9-mixed-os/1085ea0/zuul-info/inventory.yaml09:43
ysandeep|ruckykarel, yes tested same issue:09:49
ysandeep|ruck[    0.566265] usbserial: usb_serial_init - registering generic driver failed09:49
ysandeep|ruck[    0.567387] usbserial: usb_serial_init - returning with error -1909:49
ykarelack09:49
ysandeep|ruckykarel, https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/418137 similiar fix09:51
ykarelk09:52
marioshttps://logserver.rdoproject.org/58/43558/10/check/tripleo-ci-centos-8-9-mixed-os/1085ea0/logs/quickstart_files/hosts10:00
*** pojadhav is now known as pojadhav|afk10:20
*** rlandy|out is now known as rlandy10:30
rlandyysandeep|ruck: hello - how are things?10:32
rlandywant to sync?10:32
rlandyysandeep|ruck: gate blockers should be cleared now10:33
ysandeep|ruckgood morning, ack for sync.. let me grab my headphones10:33
rlandyysandeep|ruck, https://meet.google.com/bjk-ksqm-fqf?pli=1&authuser=010:35
rlandywhen ready10:35
*** ysandeep|ruck is now known as ysandeep|brb10:51
*** ysandeep|brb is now known as ysandeep11:05
*** pojadhav|afk is now known as pojadhav11:15
frenzy_fridayhey folks, review time11:16
rlandyjm1[m]: can you look at the review list from June 3011:21
rlandythere is one patch from Sagi11:21
rlandywe are not sure what to do here11:22
rlandyleft some notes11:22
*** dviroel|afk is now known as dviroel11:26
jm1rlandy: yes, i added it to the review list yesterday because its open for a long time and we did not want it to get lost. its not urgent, so lets wait for sagi to answer the questions11:26
jm1will ping him on sunday about it, he is off today11:27
rlandyysandeep: so, we merged the master base image change yesterday11:35
rlandydid that impact OVB?11:35
rlandyif not, we can merge the name change and criteria11:35
rlandymarios: frenzy_friday: ^^ fyi11:35
ysandeeprlandy, master ovb are currently failing on network provision - we are running same fs001 in downstream to narrow down the issue (if infra related).11:36
rlandyysandeep: ok - if you can confirm it's not container related, we can merge11:37
rlandylet me know 11:37
rlandyysandeep: marios confirmed we can stop running victoria11:41
rlandyrcastillo: ^^ you can put in a patch to remove the line11:41
rlandyand components11:41
rlandyto save resorces11:42
ysandeeprlandy, ack11:42
mariosthanks ysandeep rlandy 11:47
ysandeeprlandy marios rcastillo: https://review.rdoproject.org/r/c/config/+/4382811:53
ysandeepreviewbot, please add in review list https://review.rdoproject.org/r/c/config/+/4382811:53
reviewbotI have added your review to the Review list11:53
rlandyysandeep: you have two +2s there11:56
rlandycan merge when ready11:56
ysandeepack, as soon as zuul reports11:56
rlandywe will need the to remove trigger11:57
rlandygetting11:57
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/upstream.yaml#L17611:58
ysandeeprlandy, we still have openstack-periodic-integration-stable3 there - and ussuri is gone11:59
rlandyysandeep; rdo had some jobs there12:00
rlandythat is why12:00
jm1rlandy: gimme a minute 🙈12:00
rlandyie: on ussuri12:00
rlandyjm1: no worries12:00
rlandyping when ready12:00
ysandeepack.. so for stopping our jobs from running i think removing templates should be enough for now12:01
ykarelysandeep, downstream tests not needed, i figured out the issue12:01
ykarelit's not infra related12:01
ykarelme testing fix in https://review.rdoproject.org/r/c/testproject/+/4380312:01
ykarelupdating lp12:01
ysandeepykarel++ nice 12:01
ykarelysandeep, there is also a tooling issue related to container updates12:02
ykarelif you remember that was seen in past too12:02
ykareland also need to check if network component was force promoted12:02
ysandeeprlandy, jm1 akahat ^^ was network component force promoted?12:03
rlandyysandeep: no12:03
rlandytetsproject rerun12:03
ysandeepack12:04
ykarelcommented https://bugs.launchpad.net/tripleo/+bug/1980333/comments/512:04
ysandeeprlandy, zuul reported back - could you please +w https://review.rdoproject.org/r/c/config/+/4382812:04
ysandeepykarel, thanks reading12:04
ykarelrlandy, which jobs?12:04
amoralejwe are bumping tempest in wallaby in https://review.rdoproject.org/r/c/rdoinfo/+/4378512:08
amoralejlet me know if anything explodes12:08
amoralejhopefully not :)12:08
ysandeepamoralej, Needed for downstream import?12:08
amoralejdunno, tbh, mkopec requested it12:09
amoraleji'm not sure if it comes from downstream requirement12:09
amoralejkopecmartin, ^12:09
ysandeepnext week we are stopping import for 17.0 - just wondering if this is the right time - because if anything explore that mean we risk downstream dates.12:09
ysandeepcc rlandy ^^12:10
kopecmartinwe want to update tempest because of the import next week, the newer version contains fixes for some sshable issues and we want to get them downstream12:10
rlandyysandeep: amoralej: pls don;t do anything that is not absolutely needed for 17 right now12:10
rlandywe have to get all patches onto import by july 6th12:11
rlandyafter that  - go wild12:11
amoralejrlandy, from kopecmartin comment, we want that for 1712:12
ysandeepamoralej, kopecmartin we are not running any tripleo jobs in check here: https://review.rdoproject.org/r/c/rdoinfo/+/43785 12:12
* ysandeep recall we used to trigger some tripleo ci specific jobs in rdoinfo patches12:13
amoralejysandeep, only in master12:13
amoralejbut you can run jobs with depends on12:14
amoralejmmmm12:14
amoraleji've removed +W12:14
ysandeepI am not sure depends-on work with rdo-info patches, ykarel ^^ have you ever tried depends-on with rdo-info patch on testproject? 12:16
amoralejysandeep, i can provide a temporary repo with the new packages12:16
amoralejso that we can create a dnm patch to the release config12:16
amoraleji'm pretty sure we did that in the past12:16
ysandeepack that will do12:16
* ykarel reads back12:16
ykarelysandeep, yes amoralej right it would work, depends-on on rdo-jobs where you define the custom jobs12:18
ykarelthat how it used to test in past12:18
ysandeepby any chance someone have an example from past?12:19
ykarelysandeep, https://review.rdoproject.org/r/c/rdoinfo/+/4346012:19
ykarelu can use same12:20
amoralejah, i was thinking in something different12:21
amoralejbut ok12:21
amoraleji was thinking in patching tripleo-quickstart release config for wallaby12:21
amoralejand run patches on tripleo-ci or testproject or whatever with depends-on on that one12:22
amoralejso that we may have anything12:22
amoralejbut i'm find in doing that too12:22
amoralejykarel, btw, there is anything that prevents us to merge https://review.rdoproject.org/r/c/rdo-jobs/+/40247 ?12:23
ykarelamoralej, looking12:23
amoralejwe are not doing anything wrong there, right?12:23
amoralejeven add the jobs in experimental pipeline for changes in wallaby.yml12:24
amoralejit'd be fine from my side12:24
amoralejor even in check12:24
amoralejnp12:24
amoralejmy issue is that we don't run all the scenarios there, that's why i was thinking in the other approach12:24
rlandysorry - was on 1-112:29
rlandyreading back12:29
amoralejykarel, ysandeep https://review.opendev.org/c/openstack/tripleo-quickstart/+/84843912:29
rlandyysandeep: https://review.rdoproject.org/r/c/config/+/43828 w+'ed12:29
*** ysandeep is now known as ysandeep|ruck12:30
rlandyamoralej: ysandeep: re: "from kopecmartin comment, we want that for 17" if needed for 17 ok - pls go ahead with above12:31
ysandeep|ruckrlandy, thanks.. we are just discussing strategies to test that patch - so that it won't haunt us later.12:32
amoralejysandeep|ruck, https://zuul.opendev.org/t/openstack/status#848439 with that at least we'll have a couple of patches on centos9-wallaby12:33
amoralejbut it'd be good to run all relevant jobs with a depens-on with the tripleo-quickstart one12:34
rlandythanks12:34
ykarelamoralej, wrt https://review.rdoproject.org/r/c/rdo-jobs/+/40247, yes should be fine to those jobs in tags update in stable branches, but good to rename like other rdoinfo jobs12:35
ykarelcan add those in experimental pipeline12:35
ysandeep|ruckamoralej++ thanks I think your approach will work great, I can use your patch as depends on - But I think we need to add priority there..12:35
amoralejykarel, let me check what naming convention we used ...12:36
ykarelamoralej, and wrt https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439, need to add priority higher for that repo12:36
ysandeep|ruckamoralej, ykarel I will update https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439 for priority , will also update the periodic version release file.12:37
amoralejdamn12:38
amoralejysandeep|ruck, ok12:38
amoralejthanks12:38
ysandeep|ruckykarel, thanks for your comment on that network provision issue, so we need to wait for neutron fix to merge - I see its +wed already12:40
ysandeep|ruckand we need to deal with container updates issue separately - to make sure all containers are properly updated.12:42
ykarelamoralej, sorry i recalled wrong, namings looks right12:45
rlandyysandeep|ruck; on second thoughts, you're right - let's keep stable-2 line as it will be wallaby12:47
rlandywhen zed cuts12:47
ysandeep|ruckyes12:48
ykarelysandeep|ruck, yes to avoid leaking commits12:48
ysandeep|ruckykarel, amoralej updated https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439 for testing, I have set priority as 1, so that it win in all cases.12:54
amoralejyep, i saw12:54
amoralejsame priority as default dlrn repos, so it should be fine12:54
* ysandeep|ruck running testproject 12:55
rlandyhmmm ... still have ntp issues on 1712:56
rlandycan look after 1-112:56
rlandyysandeep|ruck: ^^12:57
ysandeep|ruckack, will check in few12:57
ysandeep|ruckrlandy, 17 is not failing on ntp, other legit issue: https://sf.hosted.upshift.rdu2.redhat.com/logs/7f/7fb5373a0bb4270186e544be1f041de0da9c635c/openstack-periodic-integration-rhos-17-rhel9/periodic-tripleo-ci-rhel-9-standalone-rhos-17/7002d62/logs/undercloud/var/log/extra/errors.txt 13:21
ysandeep|ruckERROR keystone TypeError: set() got an unexpected keyword argument 'dead_retry'13:21
* ysandeep|ruck reporting bug13:21
ysandeep|ruckTengu, ^^ this should go to security right? jobs are failing during keystone_bootstrap during deployment.13:24
rlandyysandeep|ruck: lol  - at least it wasn't us13:25
rlandyysandeep|ruck: weird you did not see that on wallaby13:26
ysandeep|ruckrlandy, I suspect someone merged a downstream patch 13:26
ysandeep|ruckwithout merging in wallaby first13:27
* ysandeep|ruck will debug more after reporting bug13:27
rcastilloo/13:31
*** rcastillo is now known as rcastillo|rover13:31
*** pojadhav is now known as pojadhav|afk13:34
rlandyrcastillo|rover: hey13:39
rlandyrcastillo|rover: welcome to the fun13:40
rcastillo|roverwhat's going on?13:40
rlandyysandeep|ruck has it under control as usual13:41
rlandyrhos-17 on rhel-9 issue with ntp above13:42
rlandymaster ovb falures13:42
rlandybut I'll let you guys sync13:42
rlandyrcastillo|rover: new rr hackmd https://hackmd.io/pRC9TDaoQLWqRSQioTUl5g13:42
rcastillo|roverI'm reading through it atm13:42
rlandyysandeep|ruck: rcastillo|rover: logging out soon - need anything before I go?13:44
ysandeep|ruckrlandy, all good13:45
ysandeep|ruckrcastillo|rover, lets sync13:45
rcastillo|roverrlandy: enjoy your time off13:45
rcastillo|roverysandeep|ruck: ack13:45
rlandyysandeep|ruck: rcastillo|rover: thanks for holding things together13:46
rlandysee you monday13:46
rcastillo|roverysandeep|ruck: meet.google.com/xiq-mzub-uuy whenever you're ready13:46
rcastillo|roverrlandy: o/13:46
ysandeep|ruckrcastillo|rover, https://review.opendev.org/c/openstack/neutron/+/84839613:49
ysandeep|ruckrcastillo|rover, https://bugzilla.redhat.com/show_bug.cgi?id=210311713:53
Tengu'/1313:57
Tenguysandeep|ruck: keystone is security, yep.13:57
mariosno one on happy friday o/ lemme know if you want to join i dropped i was foreveralone there14:01
ysandeep|ruckrcastillo|rover, https://review.rdoproject.org/r/c/rdoinfo/+/4378514:25
*** dasm|off is now known as dasm14:26
dasmo/14:26
ysandeep|ruckamoralej, I will be eod soon, fyi.. I am testing the tempest bump here: https://review.rdoproject.org/r/c/testproject/+/42693 - still running14:29
ysandeep|ruckrcastillo|rover, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4383214:37
amoralejysandeep|ruck, why do we need force_periodic?14:39
ysandeep|ruckrcastillo|rover, https://review.rdoproject.org/r/c/testproject/+/3834814:41
ysandeep|ruckrcastillo|rover, https://review.rdoproject.org/r/c/testproject/+/3834814:44
ysandeep|ruckrcastillo|rover, https://review.rdoproject.org/r/c/testproject/+/3205414:49
ysandeep|ruckamoralej, hey o/ because we are running periodic job in check - we need to add force_periodic - so that it pick promotion- release file one and not the check one.14:50
ysandeep|ruckonly needed for periodic job14:51
amoralejysandeep|ruck, but for general gating, we don't want to use periodic but promoted content, right?14:51
ysandeep|ruckamoralej, correct14:51
amoraleji mean for rdoinfo gate14:51
amoralejok14:51
amoralejthx14:51
ysandeep|ruckfor rdoinfo gate - we should test with our current-tripleo - so yeah no force_periodic 14:52
ysandeep|ruckrcastillo|rover, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-rhos-17/18a2cea/logs/undercloud/var/log/extra/rpm-list.txt 14:59
ysandeep|rucklast good job had python3-dogpile-cache-1.1.5-1.el9osttrunk.noarch15:00
dasmarxcruz: i'm gonna need your attention on: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4381515:06
dasmyou're the only person knowing about golang, and the author of the code.15:06
dasmarxcruz: probably no one else is going to look into these changes :(15:07
arxcruzdasm sure, may i do some tests first? :) 15:07
dasmarxcruz: do whatever you need.15:08
dasmwhat kind of tests? like unittests?15:08
dasmor manual tests?15:08
arxcruzdasm i'll basically run it on a local cockpit environment 15:09
dasmk15:09
*** amoralej is now known as amoralej|off15:20
*** dviroel is now known as dviroel|lunch15:25
*** ysandeep|ruck is now known as ysandeep|dinner15:43
*** jpena is now known as jpena|off15:59
*** dviroel|lunch is now known as dviroel16:45
ysandeep|dinnerrcastillo|rover, leaving for the day bbye o/17:31
*** ysandeep|dinner is now known as ysandeep|out17:31
dviroelrcastillo|rover: hey, need help with something?17:31
rcastillo|roverysandeep|out: o/17:32
rcastillo|roverdviroel: mostly waiting on results, seems things are going ok17:32
dviroelrcastillo|rover: ack, gate looks good17:32
dviroellot of changes to merge - one failure (mirror related)17:33
rcastillo|roveryeah, still backed up from yesterday's blocker I think17:33
dviroelyeah17:33
dviroelhey rcastillo|rover, do you plan to use this change https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/847419 ?18:26
dviroelif not, i will abandon18:26
rcastillo|roverdviroel: go ahead and abandon, I'll try something different next week18:27
dviroelack18:29
* jm1 out for today. have a nice weekend, tripleo ci :)19:05
dviroelrcastillo|rover: hey - did you saw that https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-tht&project=openstack/tripleo-heat-templates ?20:33
dviroelrcastillo|rover: filed a LP here https://bugs.launchpad.net/tripleo/+bug/198055221:06
* dasm => weekekd21:07
dasmTalk to you on Tuesday! Take care team!21:07
*** dasm is now known as dasm|off21:07
dviroelo/21:08
*** dasm|off is now known as dasm21:09
dviroeldasm: back already?21:09
*** dasm is now known as Guest396521:10
Guest3965for a few secs. i'm registering my username21:10
*** Guest3965 is now known as dasm21:12
*** dasm is now known as dasm|off21:13
*** dasm|off is now known as dasm21:13
*** dasm is now known as dasm|off21:13
dasm|offok, works :)21:13
dasm|offo/21:13
* dviroel going afk21:40
*** dviroel is now known as dviroel|afk21:40

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!