Friday, 2022-03-18

*** chem is now known as Guest245404:46
frenzyfridayfolks, pls add to your review list : https://review.rdoproject.org/r/c/config/+/39289, https://review.rdoproject.org/r/c/config/+/39444, 05:54
reviewbotI have added your review the to Review list05:54
reviewbotI have added your review the to Review list05:54
*** arxcruz is now known as arxcruz|rover07:47
*** jpena|off is now known as jpena07:48
matbudviroel|afk: hey yep looking thx08:01
marioschkumar|rover: o/ all ok need a hand with sthing? 08:16
marioschkumar|rover: did the mirror issues get fixed ?08:16
marioso/ tripleo ci please grateful for reviews when you next have time https://review.opendev.org/c/openstack/tripleo-ci/+/831543 08:33
arxcruz|rovermarios did we had some update on tripleo that now the endpoints are returning as ipv6 ?08:46
arxcruz|rovermarios btw, chkumar|rover is on pto today 08:46
mariosarxcruz|rover: ah .. and i forgot they switched 08:54
mariosarxcruz|rover: which job? 08:54
arxcruz|rovermarios i notice some tempest failures this morning on wallaby and it seems to be related to now the endpoint is returning ipv6 instead of ipv6 and tempest fails to connect to the endpoint 08:55
arxcruz|roveri saw some patches a few hours ago related to ipv6, i'm investigating 08:55
mariosi am not aware of any switch to default to ipv6 08:55
arxcruz|roverpatches on tht 08:55
mariosah well then it may be it arxcruz|rover 08:55
mariosask the authors? 08:56
mariosbut file the bug for starters... 08:56
arxcruz|rovermarios i'm holding a node now to investigate further 08:56
mariosk 08:56
marioslet me know if i can help with something arxcruz|rover 08:56
arxcruz|roverif it's the case, i'll open the bug yes :) 08:56
arxcruz|roverok08:56
marioseverything else ok gates look reasonable08:56
mariosneed a hand with something? promotions ok ?08:56
marioswho is your ruck? 08:57
mariosah its poja but out today arxcruz|rover 09:00
mariosso you should probably renick to arxcruz|ruck ? 09:00
*** arxcruz|rover is now known as arxcruz|ruck09:00
arxcruz|ruckyeah, i'm solo today 09:00
arxcruz|ruckshould be arxcruz|ruckandsuffering 09:00
marioswell let me know if you want me to jump in on something09:00
mariosis it bad? 09:00
arxcruz|rucki mean, ruckandrovering 09:00
marios:D09:01
arxcruz|ruckmarios well, a bunch of jobs fail in all branches with different errors, there is this tempest failure that seems to be related to ipv6 change, but it might be because it did not get the latest tht with the ipv6 changes, so i'm rerunning and holding a node to debug 09:01
arxcruz|ruckothers random failures, still investigating 09:01
mariosarxcruz|ruck: where .. on cockpit it looked like just one fail today for check/gate 09:03
mariosyou mean rdo ? 09:03
arxcruz|ruckmarios periodic 09:03
arxcruz|ruckhttps://logserver.rdoproject.org/46/38646/16/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4237e4e/logs/undercloud/var/log/tempest/stestr_results.html.gz09:03
mariosarxcruz|ruck: ack i see quite a few fail on openstack-periodic-integration-main in https://review.rdoproject.org/zuul/status 09:04
mariosso may be ongoing mirror issue possibly? 09:05
mariosor this new thing you found09:05
marioswith tempest/v609:05
arxcruz|ruckthe ones i was able to check, is related to tempest/v6 09:05
arxcruz|rucki mean, my guess 09:06
arxcruz|rucki'm waiting the node get ready so i can investigate more 09:07
arxcruz|ruckmaybe was random tempest failure, i can't tell for sure right now 09:07
marioswell luckily we aren't on fire for upstream promotions arxcruz|ruck 09:08
marioseverything is within 3 days old09:08
marioswould be nice to get a master promote before the weekend that is from 15th 09:09
arxcruz|ruckmarios well, not exactly, we had to remove some jobs from promotion criteria 09:09
mariosarxcruz|ruck: yes09:09
mariosarxcruz|ruck: had a quick look d/stream doesn't look terrible... have a tempest fail on baremetal but i only see one so no bug yet (but looks different to the upstream tempest fail) fyi 09:17
marioshttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-16.2/326d0d5/logs/undercloud/var/log/tempest/stestr_results.html.gz09:17
marioshmmm 17 is a bit more red checking now09:17
mariosarxcruz|ruck: do you have hackmd/etherpad/?09:20
mariosarxcruz|ruck: there is a new bug d/stream i want to file it for you but i want to check if we have existing issue 09:20
marios2022-03-17 15:58:09.963387 | primary | ERROR: No matching distribution found for ansible-config-template (unavailable)09:20
marioshave you seen a bug? ^ 09:20
marioscant find one in launchpad so filing09:22
arxcruz|ruckhttps://hackmd.io/wjL3Tpu8RhqiFWdKoGTOSw09:26
arxcruz|ruckmarios ^09:26
arxcruz|ruckhackmd 09:26
arxcruz|ruckstill working on it for today 09:26
mariosarxcruz|ruck: added a note there for now 09:35
mariosarxcruz|ruck: http://pastebin.test.redhat.com/1038119 09:35
mariosarxcruz|ruck: you can copy paste that into a bug later as needed ^^^09:35
mariosarxcruz|ruck: but will have to be bugzilla09:35
marioslets discuss that when rlandy|PTO is around (is she out today? I thought only yesterday)09:36
arxcruz|ruckmarios it shows on pto calendar yesterday and today 09:56
mariosarxcruz|ruck: k09:57
mariosarxcruz|ruck: going afk for a bit... will be back in half hour ish 10:13
*** rlandy|PTO is now known as rlandy10:26
rlandyarxcruz|ruck; hey ... 10:26
rlandyreading back10:27
arxcruz|ruckrlandy hi 10:27
arxcruz|ruckI thought you were off today 10:27
rlandyyesterday10:28
arxcruz|ruckin the pto calendar was today too, but anyway, welcome 10:29
arxcruz|ruckrlandy is it possible to kill a job ?10:33
arxcruz|ruckrlandy https://review.rdoproject.org/zuul/stream/1d1e3ed861ee4720887444f2752d9e6d?logfile=console.log10:33
arxcruz|ruck2022-03-16 20:24:08.016352 | 10:34
arxcruz|ruck2022-03-16 20:24:08.016478 | TASK [ovb-manage : Delete extra node security groups]10:34
arxcruz|ruckcheck the date10:34
arxcruz|ruckit's stucked there10:34
arxcruz|rucks/job/pipeline 10:34
rlandyarxcruz|ruck: not one job - just the whole line10:35
arxcruz|ruckrlandy how can we do it for this, the openstack-periodic-integration-main pipeline is blocked 42 hours 10:35
rlandyarxcruz|ruck: https://meet.google.com/hcw-qfec-szs?pli=1&authuser=010:36
rlandy83b81e7b7271bc2aea10bdd7c4492ff524522f4e10:46
rlandy--newrev10:48
mariosarxcruz|ruck: o/ 10:54
mariosmiss me? ;)10:54
*** dviroel|afk is now known as dviroel10:58
dviroelo/10:58
mariosoh 11:03
mariosis it st patricks day? 11:03
* marios grab 3 pints of something cold and pass to dviroel and arxcruz|ruck 11:03
dviroelwas yesterday I think11:06
rlandyarxcruz|ruck: https://opendev.org/openstack/puppet-keystone/commit/c2525c6e63c236b470bd935496875545f91e2b2611:07
mariosrlandy: oh you're not out today then 11:09
rlandymarios: hey11:09
rlandyI'm here11:09
jpodivindviroel: regarding the failing job, it is indeed issue with forgotten backports, and matbu is already on it. 11:11
dviroeljpodivin: ack, thanks jpodivin o/11:13
arxcruz|ruckrlandy just checking again now, it is two different issues 11:15
arxcruz|rucki'll open two bugs 11:15
rlandyarxcruz|ruck: ack - pls open bug and we'll go from there11:16
rlandyfrenzyfriday: hey  - can you edit https://review.rdoproject.org/r/c/config/+/3944411:22
rlandymove the when and add the line space11:23
rlandymarios and I can rereview now11:23
arxcruz|ruckrlandy https://bugs.launchpad.net/tripleo/+bug/196552511:46
frenzyfridayrlandy, checking11:49
mariosarxcruz|ruck: o/ join us? 11:50
rlandyarxcruz|ruck: https://meet.google.com/vzm-nrah-qqf?authuser=011:50
rlandyarxcruz|ruck: ^^11:51
rlandypls join us11:51
dviroelarxcruz|ruck: reading https://bugs.launchpad.net/tripleo/+bug/1965525 - which ipv6 error? so many errors there12:46
arxcruz|ruckdviroel i'm updating the log give me a sec 12:46
arxcruz|ruckit doesn't seems to be related to ipv6 12:47
dviroelarxcruz|ruck: i am looking  into similar issues, always loosing connection with DB12:47
arxcruz|ruckdviroel change the bug description for  Tripleo-CI Master and Wallaby jobs failing with keystone lost mysql connection 12:48
arxcruz|ruckthis is happening in both 12:48
arxcruz|ruckrlandy ^12:48
arxcruz|ruckwallaby and master ahve the same issue12:48
dviroelarxcruz|ruck: nice, i will add a comment with some finding there. I have a guess12:48
rlandyarxcruz|ruck: pls also changemilestone - we ar on yoga 312:49
rlandynot xena12:49
rlandyarxcruz|ruck: thank you - the split is easier to read now12:49
arxcruz|ruckdone 12:49
dviroelwe are actually in zed-1, but we don't have zed there yet12:50
* arxcruz|ruck frenetically pressing the refreshing button to see dviroel comments12:53
dviroelarxcruz|ruck: didn't started yet :P - sorry12:54
rlandyarxcruz|ruck: are you on internal chat?12:57
rlandypls join #rhos-delivery12:57
arxcruz|ruckrlandy i'm there now, i was on rhos-hos and sec-interop 13:01
rlandyarxcruz|ruck: cool - see you there now13:01
dviroelarxcruz|ruck: https://bugs.launchpad.net/tripleo/+bug/1965525/comments/313:09
arxcruz|ruckdviroel do you think this is related?13:11
dviroelarxcruz|ruck: maybe, it is related to socker disconnection, mysql13:11
arxcruz|ruckdviroel the patch add two variables, but it keeps the default values, it should not be a problem 13:14
arxcruz|rucki mean, at this point, i don't know tbh 13:14
arxcruz|ruckdviroel hmmm, maybe it is indeed a problem13:15
arxcruz|ruckcheck 13:15
arxcruz|ruckhttps://opendev.org/openstack/tripleo-heat-templates/commit/267f5ca3719a0b7725b53ed054260ea596c3cd4913:15
arxcruz|ruckdviroel it seems memccached wasn't enabled for ipv6 13:15
arxcruz|ruckthis patch was 1 day ago 13:15
rlandyarxcruz|ruck: k - downstream bug is created and CIX'ed13:16
arxcruz|ruckrlandy ack 13:16
arxcruz|rucki saw the notifications 13:16
dviroelarxcruz|ruck: ack - we might be missing something wrt to configuration maybe - we need to bring these guys to the bug13:16
rlandyarxcruz|ruck: dviroel: if https://opendev.org/openstack/tripleo-heat-templates/commit/6f8a5bc1ef16f9df5653c36bcf351026a49ea913 is related, same as the puppet review we looked at earlier13:19
rlandyyou can pass to the DF13:20
*** dasm|off is now known as dasm13:22
dasmo/13:22
rlandydviroel: ok - let me create those13:23
rlandyzed things13:23
rlandyfor launchpad13:23
dviroelrlandy: ah ok, thanks13:26
* rlandy needs to remember how :)13:27
dviroelcreate Series I think]13:29
rlandydone13:36
rlandyadding milestones13:36
rlandydviroel: yoga release is march 3013:36
rlandymarios: hey - need some former-ptl help here13:37
rlandycreating zed milestones in launchpad13:37
mariosrlandy: o/ whats up 13:38
rlandymilestone creation is failing on tripleo zed-113:38
rlandysame pattern used for yoga13:38
rlandytripleo yoga-1 was fine13:38
rlandywhat am I missing?13:38
mariosdid you create the z series? 13:39
marioshttps://launchpad.net/tripleo/+addseries rlandy 13:39
mariosrlandy: ah you did13:39
marioshttps://launchpad.net/tripleo/zed 13:39
mariosrlandy: so make a milestone there? ^^ or that fails? 13:40
mariosrlandy: "Create milestone" 13:40
rlandymarios: that step is failing on the name13:40
rlandybut it's the same pattern13:40
mariosrlandy: let me see13:41
mariosrlandy: date for z 1 so i don't have to lookup ?13:41
marioshttps://releases.openstack.org/zed/schedule.html 13:43
mariosrlandy: i made a zed-1 13:43
mariosdviroel: ^^ 13:43
rlandymarios: so the name you used was just zed-1?13:44
rlandyand it came out as tripleo zed-1?13:44
mariosrlandy: also i am going to close out the X milestones13:44
rlandyyep13:45
mariosrlandy: so they don't show up in the list of 'set milestone'13:45
mariosprobably need to make a 'release' let me also ping slagle just for his info13:45
rlandymarios: and yoga 1 and 213:45
rlandyok13:45
rlandymarios: should I switch zed to current dev focus?13:46
mariosrlandy: sure 13:46
mariosrlandy: k i am going to stop now13:46
mariosrlandy: let me know if you need me to do something? 13:46
mariosbut let me close out the x stuff sec 13:47
mariostheni'll stop13:47
mariosrlandy: k the x are gone from the list of 'set milestone' 13:48
rlandymarios: k- zed is now the current13:49
mariosour work here is done ;)13:49
rlandythanks13:51
arxcruz|ruckbrb. 10 min 14:00
rlandyugh master is failing everything14:03
rlandy: broken pipe\\nerror while running runtime: exit status 1",  issye14:03
rlandyhappens when cloud is busy14:03
rlandyarxcruz|ruck: dequeuing ussuri14:08
rlandyfailing anyways14:08
rlandywill re-enquue later14:08
rlandy2022-03-18 09:18:01 | Error: Error downloading packages:14:10
rlandy2022-03-18 09:18:01 |   ansible-core-2.12.3-1.el9.x86_64: Cannot download, all mirrors were already tried without success14:10
rlandyugh14:10
rlandyscenario01214:10
dviroelarxcruz|ruck: new one created: https://bugs.launchpad.net/tripleo/+bug/1965540 - will create skiplist14:12
rlandyarxcruz|ruck' testprojecting the failed master tests14:14
rlandyhttps://review.rdoproject.org/r/c/testproject/+/3625514:24
rlandyarxcruz|ruck: ^^ testprojecting failed jobs in master and stable 114:24
arxcruz|ruckrlandy checking 14:26
dviroelrlandy: arxcruz|ruck https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/83432114:31
dviroelrlandy: arxcruz|ruck I didn't add jobs list, it is on component line atm, but will probably hit integration after that - so we can avoid editting this skip? wdyt?14:32
arxcruz|ruckyup 14:32
dviroelor should we always add job names?14:32
rlandydviroel: jobs names14:38
rlandyif it shows up integration line - then all14:38
rlandyrekicking image build on wallaby c914:39
dviroelrlandy: ack, adding job names 14:40
rlandyarxcruz|ruck: ^^14:42
rlandywill rekick ovb is the image build jobs pass there14:42
arxcruz|ruckrlandy dviroel ack 14:42
rlandyum - two ruck rovers is oficially not working now14:42
dviroelhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/834321 - updated14:44
arxcruz|ruckrlandy https://bugs.launchpad.net/tripleo/+bug/196554614:52
arxcruz|ruckrlandy can we merge dviroel skiplist soon? I'm creating a new one to skip the testbasicops 14:53
arxcruz|ruckso we don 't have merge conflicts 14:53
arxcruz|ruckrlandy was you who put the testproject for master right?14:55
arxcruz|ruckperiodic-tripleo-ci-centos-9-standalone-full-tempest-api-master will fail the basicops test 14:55
arxcruz|ruckmarios https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/834321 please, quickly review :) 14:56
rlandyarxcruz|ruck: ack14:58
rlandyarxcruz|ruck: ^^ w+'ed14:59
arxcruz|ruckrlandy ack, i'm creating the basicops one 14:59
mariosarxcruz|ruck: checking15:02
arxcruz|ruckmarios rlandy already +w :) 15:03
mariosack arxcruz|ruck lgtm anyway voted15:04
rlandyhttps://logserver.rdoproject.org/60/39960/9/check/periodic-tripleo-centos-9-buildimage-overcloud-full-wallaby/c433725/build-err.log15:12
rlandyhmm - issue here15:12
rlandyand more mirror fun ...15:14
rlandy2022-03-18 15:06:25.317 | Errors during downloading metadata for repository 'baseos':15:14
rlandy2022-03-18 15:06:25.317 |   - Curl error (52): Server returned nothing (no headers, no data) for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Empty reply from server]15:14
rlandy2022-03-18 15:06:25.317 |   - Curl error (7): Couldn't connect to server for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Failed to connect to mirrors.centos.org port 443: Connection refused]15:14
rlandy2022-03-18 15:06:25.317 |   - Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http&countme=1 [OpenSSL SSL_connect: Connection reset by peer in connection to mirrors.centos.org:443 ]15:14
rlandy2022-03-18 15:06:25.317 |   - Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [OpenSSL SSL_connect: Connection reset by peer in connection to mirrors.centos.org:443 ]15:14
rlandy2022-03-18 15:06:25.321 | Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (7): Couldn't connect to server for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Failed to connect to mirrors.centos.org port 443: Connection refused]15:14
arxcruz|ruckrlandy https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/83433015:16
rlandyarxcruz|ruck: ^^ always failing?15:17
rlandyjust this once?15:17
arxcruz|ruckrlandy yes 15:17
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master15:17
arxcruz|ruckrlandy it fails twice as far as i can see, but different issue 15:18
arxcruz|rucki mean, it fails both on metadata, but one wasn't able to connect, the second connect but got error15:18
arxcruz|rucki added on the bug 15:18
rlandyarxcruz|ruck: ok - running again now -if it fails will merge15:18
arxcruz|ruckrlandy ok15:19
arxcruz|ruckweird, i did not saw this last passing one 15:19
arxcruz|rucktempest.scenario.test_network_v6.TestGettingAddress also fails on featureset020, i'll wait if it will fail again just in case 15:20
* dviroel lunch15:23
*** dviroel is now known as dviroel|lunch15:23
rlandyarxcruz|ruck: hmm.. scenraio01215:47
rlandythat's our next failure15:47
rlandyon tempest15:47
rlandymaster and wallaby15:47
arxcruz|ruckrlandy scen012 fails now with timeout on vexx host 15:49
arxcruz|rucknot tempest 15:49
rlandyok15:50
arxcruz|ruckrlandy re-kicking scen012 15:51
arxcruz|ruckscen001 requires john's patch 15:52
rlandyarxcruz|ruck: ack - will rekick when that merges15:57
arxcruz|ruckreckciekd scen012 already 15:58
*** dviroel|lunch is now known as dviroel16:10
mariosrlandy: arxcruz|ruck: going in a bit need something? 16:18
rlandymarios: should be fine - have a good weekend16:18
arxcruz|ruckmarios rlandy same here, but i'll continue to monitor the master jobs and rekicking if required 16:19
rlandyhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4038016:20
rlandyrechecked16:20
arxcruz|ruckrlandy should i rekick fs002 ?>16:21
arxcruz|ruckget this16:21
arxcruz|ruck2022-03-18 15:06:23.430 | error: Curl error (7): Couldn't connect to server for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Failed to connect to mirrors.centos.org port 443: Connection refused] (https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http).16:21
arxcruz|rucknot sure if rekick will work 16:21
mariosstill mirror stuff? but this is rdo still arxcruz|ruck ? or also upstream? 16:21
arxcruz|ruckmarios rdo periodic job 16:21
mariosk 16:21
rlandyhttps://review.rdoproject.org/r/c/testproject/+/3566316:23
rlandywallaby? rerunning already16:24
rlandyoh master16:24
rlandycan add to job16:24
arxcruz|ruckok 16:24
*** jpena is now known as jpena|off16:48
marioshave a nice weekend tripleo-ci o/16:49
*** marios is now known as marios|out16:49
dviroelmarios|out: you too o/16:54
rlandyarxcruz|ruck: hey - sorry - finally out of morning meetings17:36
rlandyarxcruz|ruck: how long are you still around?17:36
rlandywill follow hackmd17:37
arxcruz|ruckrlandy i'm already done for the day, but i'll still keep monitoring the jobs for a while 17:37
rlandyarxcruz|ruck: thanks  - have a good weekend17:37
rlandysee you monday17:37
rlandylunhc - brb17:38
*** dviroel_ is now known as dviroel17:46
dasmbrb18:12
rlandydasm: rcastillo; dviroel: pls review https://review.opendev.org/c/openstack/tripleo-ci/+/831543 for marios20:49
rlandyI have a small -1 there20:49
rlandybut the rest could use eyes20:49
dasmchecking20:49
dasmrlandy: reviewed. I found few other leftovers.21:05
rlandydasm++ 21:05
rlandythanks for looking21:05
rcastilloI'm not clear on the difference between the standalone jobs that are filtered with the "branches: " directive vs the -$release jobs21:09
rcastilloah, just realized monday is a holiday in mx, so I'll be out then21:14
dasmrcastillo: happy Monday's holiday!21:21
dasmrcastillo: i'm actually going through entire dependency tree of our jobs. That's a huge web with a lot of intricacies :)21:22
rlandyhappy holiday21:33
* dviroel going out - have a good weekeng team o/21:36
dasmdviroel: \o have a good one!21:36
*** dviroel is now known as dviroel|out21:37
rlandysee you all on monday!22:28

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!