Tuesday, 2022-07-05

*** ysandeep|out is now known as ysandeep|ruck04:02
*** ysandeep|ruck is now known as ysandeep|ruck|afk04:38
*** ysandeep|ruck|afk is now known as ysandeep|ruck05:08
jm1hello team :)05:56
ysandeep|rucko/ hello everyone05:58
ysandeep|ruckarxcruz: could you please add in your review list: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/84847206:11
*** ysandeep|ruck is now known as ysandeep|ruck|afk06:18
*** amoralej|off is now known as amoralej06:19
*** ysandeep|ruck|afk is now known as ysandeep|ruck07:11
bhagyashrismarios, hi, i am facing freeze graph issue here https://review.opendev.org/c/openstack/tripleo-ci/+/848260 could you plz help me when you have free time.07:30
mariosbhagyashris: k will have a look in a bit07:34
*** kopecmartin_ is now known as kopecmartin07:39
bhagyashristhanks07:49
mariosbhagyashris: commented @ https://review.opendev.org/c/openstack/tripleo-ci/+/848260/11#message-4907e167356e5c0361cddd7356a5096cc5640106 one suggestion to try first (branches:) 08:11
*** ysandeep|ruck is now known as ysandeep|lunch08:15
*** ysandeep|lunch is now known as ysandeep|ruck10:16
*** rlandy|out is now known as rlandy10:24
rlandyysandeep|ruck: hello10:24
rlandyysandeep|ruck: let's sync when you are free10:25
ysandeep|ruckrlandy: good morning o/ meet.google.com/nct-ibka-ccq10:26
rlandyjoining10:27
arxcruzHey guys, I have a reunion in my daugther school, i'll be back in 2 hours +-10:57
*** arxcruz is now known as arxcruz|brb10:59
rlandyamoralej: sorry for the delay - sending response to ci.centos thread11:17
rlandymarios: pls see email - signed updated11:17
amoralejack, thanks rlandy 11:17
mariosrlandy: thank you 11:21
rlandyfrenzy_friday: bhagyashris: need to chat with you about 17.1 component line11:24
frenzy_fridayrlandy, sure11:24
rlandyfrenzy_friday: bhagyashris: ok to in 15 mins?11:24
rlandyjust answering a mail11:24
frenzy_fridayyep, sure11:25
rlandyrcastillo|rover: when you get in - we need to chat re; virt jobs in PSI/vexxhost11:27
*** dviroel|biab is now known as dviroel11:29
rlandyfrenzy_friday: bhagyashris: https://meet.google.com/ijn-jius-owf?pli=1&authuser=011:36
*** amoralej is now known as amoralej|lunch11:54
*** arxcruz|brb is now known as arxcruz12:03
rlandyysandeep|ruck: putting in the compute component reruns12:09
ysandeep|ruckack.. fyi.. tripleo/network/tempest is already in rerun12:10
ysandeep|ruckrlandy, remove fs035 from wallaby c8 criteria: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43872 , I will drop the job run in a separate patch.12:22
ysandeep|ruckreviewbot: please add in reviewlist: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4387212:24
reviewbotI have added your review to the Review list12:24
rlandymarios: dviroel: rcastillo|rover: ^^ fyi12:30
rlandysuggesting to remove fs035 from wallaby c812:30
rlandywe usually skip promote this anyways12:30
rlandyin next rr shift, we need to do an evaluation of fs03512:30
rlandyto see if we can get it stable12:30
rlandymaybe with fewer tempest tests moved to another ipv6 job12:31
ysandeep|ruckrlandy, Question - 17.1 - are we doing only on rhel 9 or rhel-8 also?12:31
rlandymarios: do you need that fs035 job for mixed-rhel?12:31
*** amoralej|lunch is now known as amoralej12:31
rlandyysandeep|ruck: assume only rhel-912:31
rlandygood question though12:31
rlandywe may need 8 for mixed rhel12:32
rlandybut starting with 912:32
rlandybhagyashris: pojadhav: is soniya around today?12:33
rlandyakahat: ^^ anyone seen her?12:33
pojadhavrlandy, soniya is on pto12:33
akahatrlandy, no.12:33
rlandyoh oops yes12:33
rlandypojadhav++ thanks for reminder12:33
rlandyI see calendar12:33
pojadhavrlandy, :)12:33
ysandeep|ruckrlandy: okay, I think we should reduce our footprint for el8 for wallaby/17.12:33
ysandeep|ruckwe are running complete line on 17/812:33
dviroelrlandy: ack 12:34
pojadhavrlandy, i think she will be available by friday this week.12:34
ysandeep|ruckand if 17.1 is not on rhel-8, continuing running complete line for 17.0 make less sense12:34
rlandyysandeep|ruck: if we need rhel-8 for 17.1, my suggestion would be to run one rhel-8 line12:34
rlandyack12:34
rlandyysandeep|ruck: delivery is a bit slammed now with imports12:35
rlandywill confirm that on monday's prod chain call when jason is back12:35
rlandygood question12:35
ysandeep|ruckyeah so are we tracking promotions :)12:35
rlandyysandeep|ruck: but agree we are not keeping up two full rhel-8 lines12:35
rlandyonly one12:35
rlandyor some smaller version12:36
ysandeep|ruckack12:36
rlandymy guess is we whole sale switch rhel-8 to 17.212:36
rlandy17.112:36
rlandyonce 9 is stable12:36
mariosrlandy: o/ no i have a new fs for mixed rhe12:37
rlandymarios: awesome - any objection to https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43872?12:38
mariosrlandy: nope voted12:41
ysandeep|ruckrlandy, marios could one of you please +w ^^12:43
mariosysandeep|ruck: k 12:43
ysandeep|ruckthanks I was tracking c8 wallaby promotion12:43
rlandymario beat me to it :)12:44
ysandeep|ruckrlandy, https://review.rdoproject.org/r/c/rdo-jobs/+/43835 could you review - remove ipa job from train12:45
ysandeep|ruckand https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43860 12:45
rlandychecking12:49
rlandydone12:50
ysandeep|ruckrlandy: thanks!12:51
*** ysandeep|ruck is now known as ysandeep|brb12:52
*** ysandeep|brb is now known as ysandeep|ruck13:01
rlandyysandeep|ruck: rcastillo|rover: atm there are two wallaby patches in the gate13:03
rlandyrest are master13:03
ysandeep|rucklast time rush :)13:04
ysandeep|ruckwe can rekick the tripleo component one more time before eod13:04
ysandeep|ruckonce those patch which are currently in gate merges13:05
ysandeep|ruckpatches*13:06
rlandyyep in our afternoon time13:14
*** dasm|off is now known as dasm13:15
dasmo/13:15
rlandynow there are a bunch13:15
rlandyall tripleo13:15
bhagyashrisarxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm, jm1, marios, ysandeep|ruck rcastillo|rover 13:25
bhagyashrisTripleo CI community meeting in 5 mins13:25
bhagyashrishttps://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2022-07-05-Community-Call @line 2913:25
dasmarxcruz: o/ have you had an opportunity to review https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43815 ?13:26
dasmi don't see any comments from you13:26
dasmarxcruz: unfortunately you're the only one who chose golang, so you're the only one who can review this ^^13:27
dasmarxcruz: btw. i like how golang works with coroutines13:27
arxcruzdasm shoot, forgot that, i'll take a look 13:30
bhagyashrisrlandy, pojadhav ^13:31
bhagyashrisrcastillo|rover, 13:31
arxcruzdasm is nice for parallelism, thaat's the reason it's faster than python :) 13:31
arxcruzwork with threads in python is a pita 13:31
dasmarxcruz: you could use async too :)13:31
dasmit's similar between python and golang13:32
arxcruzdasm yup indeed, but i think at that time requests did not have async 13:32
bhagyashrisTengu, ^13:32
dasmack13:32
arxcruzi don't recall exactly why i did not use async 13:32
Tenguuho13:32
Tengujoining, sorry.13:32
rlandyrcastillo|rover: ysandeep|ruck: want to hold on and sync before ysandeep|ruck EoD?14:03
ysandeep|ruckyes.. I will be eod shortly so lets sync14:03
rcastillo|roverrlandy: yeah14:04
arxcruzdasm wow, looks nice the output, and very quick!14:08
dasm\( ゚ヮ゚)/14:10
dasmarxcruz: can i get +2 +W from you?14:11
dasmlike i said -- you're the only one knowing golang here ^^14:11
dasmor admitting to that :P14:12
Tengugolang?14:12
* Tengu hides and wants to forget he actually did an app in golang14:12
dasmTengu: we have a tool written in golang which fetches some results: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4381514:12
dasmTengu: if you want to :)14:13
Tenguwho had such an idea ^^'14:13
dasmarxcruz ;)14:13
arxcruzdasm sure 14:13
arxcruzTengu was that a bad idea ?14:14
* Tengu renames arxcruz to Horcrux14:14
dasmarxcruz: right now we have a bus factor 1 ;)14:14
dasmTengu: xD14:14
arxcruzdasm we need to update the mariadb script 14:15
Tengumore seriously, I can't review it. my golang is really limited and it was about a year ago -.-'.14:15
ysandeep|ruckrcastillo|rover, https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/component-pipeline-wallaby-centos9.yaml#L43-L5214:20
rcastillo|roverysandeep|ruck: rlandy https://review.rdoproject.org/r/c/testproject/+/4388114:25
arxcruzjm1 frenzy_friday https://ustraveldocs.com/de/de-niv-visafeeinfo.asp14:27
dasmarxcruz: you mean this update? https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4378014:28
arxcruzdasm yes 14:28
ysandeep|ruckrcastillo|rover, rlandy https://logserver.rdoproject.org/53/21853/93/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/3977293/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz14:35
rcastillo|roverrlandy: did you want to talk about libvirt on vexx/psi?14:39
*** ysandeep|ruck is now known as ysandeep|out14:39
rlandyrcastillo|rover: yeah - let's chat for when you get off rr14:39
rcastillo|roveralright14:40
rlandyhttps://meet.google.com/htc-yquz-kap?pli=1&authuser=014:40
rlandyrcastillo|rover: ^^14:40
rlandyrcastillo|rover: nodeset: single-centos-9-big-node14:45
rlandycloud-centos-9-stream-big14:46
dasmarxcruz: thanks for +2. Any objections before +W? I don't think anyone else is gonna review this code15:00
Tengumarios: fyi: https://review.opendev.org/c/openstack/tripleo-ansible/+/848754  will replace the t-h-t patch. No big change on your side though I think, unless CI is using tripleo_bootstrap to install needed packages.15:03
mariosTengu: ack noted15:03
* dviroel lunch15:07
*** dviroel is now known as dviroel|lunch15:07
pojadhavrlandy, rcastillo|rover, dviroel|lunch: please review when free https://review.rdoproject.org/r/q/topic:ooo_c7_teardown+owner:pojadhav15:36
pojadhavneed reviews : https://review.opendev.org/c/openstack/tripleo-quickstart/+/84383715:36
reviewbotI have added your review to the Review list15:36
pojadhavneed reviews : https://review.rdoproject.org/r/q/topic:ooo_c7_teardown+owner:pojadhav15:37
reviewbotI have added your review to the Review list15:37
arxcruzdasm no, it's fine do you want me to +w ?15:55
dasmarxcruz: yeah, go for it15:56
dasmarxcruz: btw, there is another one, which requires your attention: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/84847215:57
rlandyrcastillo|rover: ugh - gate reset16:02
*** marios is now known as marios|out16:03
rcastillo|roverrlandy: T-7 hrs, maybe we get that through?16:04
rlandygate is getting longer16:05
rlandywhatever is in when we kick the last tripleo  line promo is in16:05
rlandyugh  baremetal just updated16:06
rlandykicking that16:06
rcastillo|roverok we can juggle one more16:07
dasmo/ i see this: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/843870/4 has +2 +W but it's not merged. Should we rekick zuul on that?16:12
rlandylunch brb16:18
*** dviroel|lunch is now known as dviroel16:22
*** amoralej is now known as amoralej|off16:46
rcastillo|roverwallaby common should promote16:50
dviroelrlandy, rcastillo|rover: hey, we need manilaclient promotion for wallaby. It is ok to kick a clients component line now, instead of waiting until tomorrow 3:00 UTC? 16:52
dviroelor maybe testproject wallaby ones16:52
rcastillo|roverbetter to tp the wallaby ones I think16:52
dviroelyeah, the rest can wait until tomorrow16:53
rlandydviroel: ack - sure16:58
rcastillo|roverwallaby tempest component seems to have passed, just needs to report16:58
rlandydviroel: juts don;t kick the whole line16:59
* rlandy looks16:59
dviroelrcastillo|rover: rdoinfo patch just merged, we have a new release 2.6.4 - so need to promote from consistent to component-ci-testing 16:59
dviroelrlandy: rcastillo|rover: running wallaby only16:59
dviroelhttps://review.rdoproject.org/zuul/status#3635616:59
rlandytempest should also promote17:01
rcastillo|rovercoffee brb17:05
rlandycompute promoted17:05
rlandyugh - scenario001 failure on 1717:08
rcastillo|roverwallaby network job passed as well17:54
rlandyrcastillo|rover: ack - happy dance17:59
rlandygoing to run the promo job there17:59
rlandyrcastillo|rover: kicking the rhel-9 tripleo component18:21
rlandyrcastillo|rover: rekciked common and tempest18:41
rcastillo|roverhuh, why? I'm lost there18:41
rcastillo|roveroh wait, you mean downstream?18:42
rlandytempest is running now18:43
rlandyjust common18:43
rlandyrev'ed since we promoted18:43
rlandydownstream - only tripleo18:43
rcastillo|roverah, I see now18:44
dviroelhum, fs001-clients-wallaby hitting mirror issues (2nd time already)19:52
dviroelError: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried19:52
rlandydviroel; ack ... rcastillo|rover will rerun20:08
rcastillo|roverdviroel: not sure if it's just clients or all image builds, we'll know when tripleo runs20:08
rlandybaremetal will also need a rerun20:08
rlandyfs00120:08
rlandyrekicked and common sc 0320:10
dviroelack, i have a testproject for fs001-clients ready to recheck - just waiting a little bit20:10
rcastillo|roverrlandy: build image info showing up on tripleo component as well20:27
dviroelrcastillo|rover: mirrors too?20:28
rcastillo|roverdviroel: yes, baseos20:28
dviroelyeah, same repo too20:28
dviroelcarloss: ^ seems to be happening more often20:29
carlossoh20:30
dviroelrcastillo|rover:  Downloading successful, but checksum doesn't match. Calculated: d4e650ef6b39755ab61cc4410cd2d445ae215bb42a8d73737deb6fbcfcfcd9e1120d51a0a2d6ad520b9526aa8dff04b71ba395a6f0ee3d7a759182ad1c2bd719(sha512)  Expected: 839d326a640e52e1e2ccc47dfcf8ea15ceb1254afba21e7aac706fff4def27c6936eb477f16fb3065b3433abfb87ff20b26b39ebec76b85a434588a85e240ac4(sha512)20:31
dviroelrcastillo|rover: maybe the repos is on sync process20:31
dviroelchecksum not matching20:31
rcastillo|roverhttp://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/20:38
rcastillo|roverhasn't updated since the 27th, unless I'm looking at the wrong mirror20:39
dviroelyeah, there is no new packages too20:47
rlandyugh - tripleo rhel-9 failed node provision21:01
rlandyrerunning21:01
rlandytempest will promote21:04
dviroelyey21:04
rlandyrcastillo|rover: does the dashboard work for you?21:05
rlandydashboard-ci.tripleo.org failing to connect21:06
rlandydviroel: ^^?21:06
rlandynot reachable21:06
dviroelsame here21:06
rcastillo|roversame here21:06
rcastillo|roverreview bot went offline as well21:07
rcastillo|roversame node?21:07
rcastillo|roverhttps://rdo.vexxhost.ca/ down for anyone else?21:12
rlandyugh21:14
rlandyperfect timing :)21:14
rlandylooks like nodes are starting to regenerate21:15
dviroel:(21:19
rcastillo|roveroh boy21:19
rlandywe just have to wit it out21:23
rlandywait21:23
rlandyor flat promote21:24
rcastillo|roverrlandy: ack let's see if it comes back quickly21:25
rlandyrcastillo|rover: should - rechecked my one job21:28
rlandyanything you see missing?21:28
rlandydashboard is back21:28
rcastillo|roverrlandy: nothing yet, just the node_failures21:28
rlandywe have 1.5 hours21:33
rlandywe may need to just skip promote in a few21:35
rcastillo|roveryeah especially if we're still getting this build image issue21:40
dviroelfeatureset001-clients-wallaby got a node21:40
rlandystack create failed21:43
rlandyrcastillo|rover: if we can't get these jobs to pass OVB, I'm just skip promoting them21:49
rlandywe21:49
rlandywe will catch the issues on the donwstream side21:50
rcastillo|roverrlandy: ack21:50
rlandyprep'ing the skip patch21:52
dviroelclients component seems to be safe to skip fs001, the difference between last promoted content and ci-testing is only python-manilaclient rpm. And we don't have manila on ovb jobs 21:52
dviroelrlandy: ^21:52
rlandydviroel: ack - thanks ... common is missing scen 00321:53
rlandythat should pass21:53
rlandyso it's baremetal and tripleo21:53
* dviroel biab + dinner21:58
*** dviroel is now known as dviroel|afk21:59
* dasm leaves. ttyt22:02
*** dasm is now known as dasm|off22:02
rcastillo|roverback in 1522:03
rlandyrcastillo|rover: dviroel|afk: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43888 DNM: Temp remove criteria to promote wallaby22:23
rlandywe have half an hour22:24
rcastillo|roverrlandy: LGTM, do we just need to run the promote jobs with depends-on?22:29
rlandyin progress now22:29
rlandyrcastillo|rover: I didn;t force promote network22:29
rlandyit promotde 5 hours ago22:29
rlandyand there is a failure in dlrn22:29
rlandycan try that one later if need be22:30
rcastillo|roverrlandy: ack, these four should be fine, they were passing22:30
rlandybaremetal, clients, tripleo, common22:30
rlandylet's try for those22:30
rlandydamage is done22:31
rcastillo|roveryup22:33
rcastillo|rovereverything's still going through downstream still right? so it ain't all that bad22:33
rlandyrcastillo|rover: yeah - we'll catch it there if need be22:35
rlandybut we have to promote rhel-922:35
rlandywaiting for tripleo component  to promote22:36
rlandyso we can rekick that line22:36
rcastillo|roverrlandy: anything need watching? heading out otherwise23:23
rlandyrcastillo|rover: think it's ok - updating rr notes23:23
rlandyjust waiting for rhel-9 tripleo component to complete23:23
rlandyso I can rekick integration line23:23
*** dviroel|afk is now known as dviroel23:27
dviroelrlandy: anything that needs attention?23:46
dviroelrlandy: thanks for promoting the remaining components23:47
rlandydviroel: think we are ok23:47
rlandyjust rekicking rhel-9 line now23:47
rlandytripleo component promoted23:47
rlandyall wallaby promoted23:47
dviroelgreat 23:48
rlandydviroel: should be fine23:48
rlandyovbs are in rerun now23:48
rlandyI skip promoted them23:48
rlandyto get in under the wire23:48
dviroelrlandy: ok, have a good night, see you tomorrow o/23:49
rlandygood night23:49
*** dviroel is now known as dviroel|out23:50

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!