Monday, 2021-04-19

*** artom has quit IRC00:12
*** stevebaker has quit IRC00:53
*** stevebaker has joined #opendev00:56
*** tkajinam has quit IRC01:06
*** tkajinam has joined #opendev01:07
*** ykarel__ has joined #opendev03:51
*** redrobot0 has joined #opendev04:07
*** redrobot has quit IRC04:11
*** redrobot0 is now known as redrobot04:11
*** vishalmanchanda has joined #opendev04:53
*** ykarel_ has joined #opendev05:02
*** ykarel__ has quit IRC05:05
*** marios has joined #opendev05:06
*** paladox has quit IRC05:09
*** paladox has joined #opendev05:18
*** hamalq has joined #opendev05:20
*** hamalq has quit IRC05:24
*** bhagyashris|away is now known as bhagyashris05:33
*** whoami-rajat_ has joined #opendev05:38
*** slaweq has quit IRC06:16
*** slaweq has joined #opendev06:19
*** dpawlik7 is now known as dpawlik06:22
*** ralonsoh has joined #opendev06:32
*** fressi has joined #opendev06:33
*** eolivare has joined #opendev06:34
*** sboyron has joined #opendev06:36
*** fressi has quit IRC06:38
*** amoralej|off is now known as amoralej06:40
*** fressi has joined #opendev06:43
*** fressi has quit IRC07:15
*** fressi has joined #opendev07:17
*** andrewbonney has joined #opendev07:20
*** fressi has quit IRC07:36
*** fressi has joined #opendev07:42
*** rpittau|afk is now known as rpittau07:44
*** dtantsur|afk is now known as dtantsur07:54
*** jpena|off is now known as jpena07:56
*** tosky has joined #opendev08:03
*** vishalmanchanda has quit IRC08:22
*** fressi has quit IRC08:41
*** HeroicHitesh has joined #opendev08:43
*** ykarel_ is now known as ykarel|lunch08:46
*** fressi has joined #opendev08:48
*** HeroicHitesh has quit IRC08:52
*** vishalmanchanda has joined #opendev08:58
*** fressi has quit IRC08:59
*** fressi has joined #opendev09:00
*** ysandeep is now known as ysandeep|afk09:01
*** hamalq has joined #opendev09:22
*** roman_g has joined #opendev09:24
*** hamalq has quit IRC09:26
*** ykarel|lunch is now known as ykarel09:41
*** iurygregory has joined #opendev10:17
*** fressi has quit IRC10:18
*** fressi has joined #opendev10:19
*** amotoki has joined #opendev10:23
*** ysandeep|afk is now known as ysandeep10:24
*** marios is now known as marios|afk10:55
*** avass has quit IRC11:00
*** fressi has quit IRC11:03
*** fressi has joined #opendev11:26
*** jpena is now known as jpena|lunch11:34
*** HeroicHitesh has joined #opendev11:36
*** amoralej is now known as amoralej|lunch12:00
*** avass has joined #opendev12:09
*** ricolin has quit IRC12:15
*** paladox has quit IRC12:34
*** paladox has joined #opendev12:35
*** jpena|lunch is now known as jpena12:37
*** fressi has quit IRC12:38
*** marios|afk is now known as marios|ptg12:45
*** d34dh0r53 has joined #opendev12:46
*** smcginnis has joined #opendev12:54
*** amoralej|lunch is now known as amoralej12:59
*** mlavalle has joined #opendev13:01
fungiconfig-core: not urgent, i know a lot of folks are doing ptg things, but a second reviewer on https://review.opendev.org/783613 would be appreciated so we can continue making progress on enabling debian-bullseye nodes13:03
*** fressi has joined #opendev13:07
*** ykarel has quit IRC13:12
*** lpetrut has joined #opendev13:12
*** brinzhang has quit IRC13:14
*** brinzhang has joined #opendev13:15
*** lbragstad_ is now known as lbragstad13:17
*** HeroicHitesh has quit IRC13:29
fungiinfra-root: heads up, zoom just closed out multiple ptg conference rooms unexpectedly, so some teams are migrating to meetpad. will be keeping a close eye on the server resource graphs in case we need to add a jvb server13:35
fungitraffic's definitely starting to pick up on meetpad,jvb01,jvb0213:38
fungibut all within sane levels for the moment13:38
fungilooks like established tcp connections on meetpad.o.o has jumped up over 300, hopefully that's not a sign of trouble13:39
fungiload average on jvb01 is up around 7 now, but it has 8 cpus13:43
fungijvb01 load is up over 9 now so the server may struggle a bit13:47
fungi02 is nearly unused13:47
fungifoundation events staff have worked out the zoom problem13:49
fungialso load on jvb02 is dropping agani13:49
hrwin Kolla we moved to meeting on my account (Business one, no time limit)13:49
fungiagain13:49
fungihrw: yep, these were supposed to be set up that way too, but were misconfigured initially as unlicensed13:49
fungior that's the backchannel details i got at any rate13:50
hrwhappens13:51
*** artom has joined #opendev13:53
clarkbfungi: good to hear meetpad was keeping up13:58
clarkbI'll take a look at that change shortly13:58
fungiwell, i don't know for sure it was keeping up, but nobody mentioned trouble with it13:58
fungiat least not yet13:58
clarkbI'd like to proceed with https://review.opendev.org/c/opendev/system-config/+/786487 this morning but will check on the airship release first14:00
clarkbfungi: looks like that change needs a rebase or a followup change to update osuosl arm64 provider too? The only other thought I've got is we should be careful about builder disk space (I think we may be ok as we semi recently cleaned up an older fedora image iirc)14:04
*** gothicserpent has quit IRC14:06
clarkbI +2'd it if you think we should land it without osuosl then go ahead and +a and we can add that provider in a followu14:09
fungiyeah, i'll check disk space on the builders in a few14:12
fungimaybe noonedeadpunk or zigo are interested in writing the followup change to add bullseye nodes in the osuosl arm64 environment14:12
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels  https://review.opendev.org/c/openstack/project-config/+/78361314:13
zigofungi: Hi !14:13
zigoarm64, no... :)14:13
zigoamd64 yes14:13
fungilooks like noonedeadpunk just did14:14
*** gothicserpent has joined #opendev14:14
zigofungi: FYI, I'm slowly getting puppet-openstack fixed for Debian...14:15
fungicool!14:15
zigoIt of course works for me (tm) but having it working in the gate is something else.14:16
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels  https://review.opendev.org/c/openstack/project-config/+/78361314:17
openstackgerritDmitriy Rabotyagov proposed openstack/project-config master: Add Debian bullseye wheel cache publish jobs  https://review.opendev.org/c/openstack/project-config/+/78363314:18
noonedeadpunkwell, now just did:) needed to rebase first14:21
* clarkb looks again14:21
clarkbnoonedeadpunk: that diff looks odd to me. I expected just the addition of the image to the osuosl cloud, not the entire osuosl cloud to be added. Also I think you have to add it to nb03.opendev.org file for the image upload side14:22
clarkbaha the inter ps diff is what was weird14:23
noonedeadpunkclarkb: are comparing diff between patchsets? as since I rebased, it would show diff of adding cloud14:23
clarkbdiffing against base looks better but still need to add to nb03.opendev.org.yaml14:24
clarkbnoonedeadpunk: yup14:24
noonedeadpunkin nb03.opendev.org it seems you use anchors for osuosl14:24
*** fressi has quit IRC14:24
noonedeadpunkso no need to specificaly define it I guess?14:24
noonedeadpunkie L89 `diskimages: *arm64_diskimages`14:25
clarkbya for some reason I thought we didn't because we used different flag, but it appears we did14:25
clarkbI guess we just talked about possibly being able to drop uefi and didn't go for it as an explicit flag14:25
*** fressi has joined #opendev14:28
clarkbsounds like the airship release is still going but they hope to be done this morning. I half expected that after we helped them with branch stuff on firday14:40
clarkbI'll wait for them to say they are done before unWIPing the zk04 change14:40
*** ykarel has joined #opendev14:47
*** fressi has quit IRC14:48
clarkbnoonedeadpunk: fwiw I had expected the purple diff color when diffing between ps7 and 9. I should do more research into how gerrit determines that diff state14:59
clarkbfungi: I do wonder if we should go ahead and land https://review.opendev.org/c/opendev/system-config/+/786466 since those gitea upgrades have been pretty stable15:02
clarkbI'm around if you think we should just go for it15:02
noonedeadpunkclarkb: oh, well, yes, that's weird... As between 7 and 8 it's just parent change in commit msg15:02
noonedeadpunkand when select 7 and 9 huge thing appears....15:03
clarkbnoonedeadpunk: ya and sometimes gerrit uses purple to show you this is a delta from external changes15:03
noonedeadpunkI think I never saw that....15:03
clarkbit is really useful when it does that, but now I wonder how it determines that data (maybe it processes that in the background and it needs some time to update the change state)15:04
fungiclarkb: after the relmgt session (which is running over) ends i'm free to watch it15:06
*** ralonsoh_ has joined #opendev15:06
*** ralonsoh has quit IRC15:06
clarkbfungi: cool I'm free as well if you are comfortable and can help watch it I think you can +A?15:06
fungier, well, security-sig actually started 5 minutes ago15:06
clarkber +A when ready15:06
fungi(schedules are hard)15:06
*** ralonsoh_ has quit IRC15:07
*** ralonsoh has joined #opendev15:08
openstackgerritMerged openstack/project-config master: Add Debian Bullseye nodepool images and wheels  https://review.opendev.org/c/openstack/project-config/+/78361315:08
*** d34dh0r53 has quit IRC15:11
*** ykarel has quit IRC15:12
*** lpetrut has quit IRC15:12
*** lpetrut has joined #opendev15:14
*** smcginnis has quit IRC15:19
clarkbfungi: the other thing I'm looking at right now is the proposed set of gerrit account cleanups I pushed to review on friday. I think this gets closer to potentailly being noticed by people but I tried to look at each of those conflicts and using the audit output determine if it was reasonably safe to retire the account and cleanup external ids. I think for that set we can retire them and then wait15:21
clarkba week and see if anyone complains15:21
clarkbI've also not heard back from either of the two third party ci services I emaield about cleaning their accounts up15:21
clarkbfungi: have you seen a response from them (you were cc'd so just double checking I haven't missed it). If not I'll plan to do those cleanups today as well (no account retirements there just external id manipulation)15:22
*** d34dh0r53 has joined #opendev15:25
fungiyeah, i've seen no replies15:36
*** amoralej is now known as amoralej|off15:41
*** lpetrut has quit IRC15:41
fungiclarkb: i've approved 786466 just now. will try to keep an eye on it but also switching to diversity-wg session in a few minutes15:42
*** fressi has joined #opendev15:46
*** fressi has quit IRC15:46
*** fressi has joined #opendev15:47
clarkbfungi: thanks! I can keep an eye on it too, juist wanted to make sure it wasn't me alone15:51
*** brinzhang_ has joined #opendev15:52
*** fressi has quit IRC15:53
*** brinzhang has quit IRC15:55
*** marios|ptg is now known as marios|brb16:04
*** artom has quit IRC16:04
*** artom has joined #opendev16:05
*** ysandeep is now known as ysandeep|away16:10
*** hamalq has joined #opendev16:12
*** artom has quit IRC16:13
*** hamalq_ has joined #opendev16:15
*** marios|brb is now known as marios|afk16:15
*** hamalq has quit IRC16:19
*** HeroicHitesh has joined #opendev16:32
*** marios|afk is now known as marios16:34
clarkbI've asked luca to weigh in on the zuul status plugin change16:41
clarkb(via gerrit slack)16:41
*** marios is now known as marios|out16:42
fungithanks, having some closure there would be nice16:45
*** HeroicHitesh has quit IRC16:48
clarkbfungi: I've remembered the other thing I wanted to help with: the project reparenting16:50
fungioh, yep16:52
fungiso elod pushed some project-config changes for that16:52
fungiwe could try merging the first one in a bit and see what happens with the deploy job16:53
openstackgerritMerged opendev/system-config master: Upgrade gitea to 1.13.7  https://review.opendev.org/c/opendev/system-config/+/78646616:53
clarkbfungi: yup, I think I already +2'd it on friday too16:56
clarkbI did maths on it and want to say it was ~29 real project updates16:56
clarkbwhich seems reasonable for a first pass16:56
clarkbthe gitea updates should run shortly16:56
*** jpena is now known as jpena|off16:59
clarkbgitea01 has just updated and at first glance it looks good17:02
clarkbfungi: https://review.opendev.org/c/openstack/project-config/+/786735 is that first project-config change I think17:04
*** HeroicHitesh has joined #opendev17:04
*** marios|out has quit IRC17:07
*** rpittau is now known as rpittau|afk17:11
*** dtantsur is now known as dtantsur|afk17:15
*** ralonsoh has quit IRC17:15
*** artom has joined #opendev17:18
*** andrewbonney has quit IRC17:20
clarkbthe upgrade has finished on all 8 servers. I'm happy with the results from what I can see17:21
fungiyeah no problems yet at least17:22
clarkbif openstack is ready for https://review.opendev.org/c/openstack/project-config/+/786735 maybe we try that next17:25
*** artom has quit IRC17:35
TheJulia#ironic now Done until Later17:39
TheJulia#ironic Next APAC friendly session w/Virtual Media support validation17:40
clarkbTheJulia: wrong window :)17:41
TheJuliaDOH!17:41
fungiheh17:44
*** artom has joined #opendev17:45
fungiokay, i've approved 78673517:46
fungilooks like infra-prod-service-nodepool timed out on the current hourly run...17:48
fungiwonder if nb03 is down again17:48
fungimmm, i can ssh to it over ipv4 as well as ipv6 so nope17:49
fungithe ansible run for it is still in progress17:50
fungiinstall-docker on nb01.opendev.org looks like it took over an hour and failed17:51
fungii think17:51
clarkbthat uses the upstream package repo, I wonder if we aren't as noopy as we would hope around that17:52
fungiit was the TASK [install-docker : Install docker] step, took 1:16:22.84017:52
fungier, i guess that's elapsed time, so 1:16:22.840-0:11:08.119 from the previous step is ~1h5m14s?17:53
fungianyway, far too long17:54
clarkband the job timeout is ~30 minutes iirc17:55
openstackgerritMerged openstack/project-config master: Move projects under meta-config acl  https://review.opendev.org/c/openstack/project-config/+/78673517:56
fungihttp://paste.openstack.org/show/804613/17:59
fungii'm reading that correctly, yeah?17:59
fungimanage-projects is running now for 78673517:59
fungiit's currently on TASK [gerrit : Run manage-projects] as of 17:59:1418:00
clarkbfungi: yes I believe that is a correct reading of the timestamps if no other tasks for nb01.opendev.org ran between18:01
fungii wonder how we can tail the manage-projects log these days18:02
fungiclarkb: yeah, that was directly quoting adjacent tasks from the log18:02
funginothing between them18:02
fungioutput paused for over an hour from what i can see18:02
fungimanage-projects finished18:03
clarkbwow I guess the bulk of that job time really was in gitea18:04
clarkbshould be able to do many more than 29 at a time18:04
fungiyup, log shows it updating the ones i expect, based on spot-checking18:04
fungi`git show refs/meta/config` in ~gerrit2/review_site/git/openstack/cinder.git/ indicates the inheritFrom entry was added as intended18:06
fungielod: ^ looks like you're all set on that first change, if you want to test things when you have time18:06
*** hamalq_ has quit IRC18:08
*** hamalq has joined #opendev18:08
*** paladox_ has joined #opendev18:10
clarkbI'm starting to work on cleaning up the externalid conflicts related to those two CI accounts18:13
clarkbI sent them email a week ago with the proposed plan and told them to tell us if this was a problem. There have been no responses so I guess I'm going for it18:14
*** eolivare has quit IRC18:15
clarkbthe first set is done18:28
*** paladox_ has quit IRC18:37
*** sboyron has quit IRC18:37
*** paladox_ has joined #opendev18:39
*** paladox_ has quit IRC18:41
*** paladox_ has joined #opendev18:42
*** paladox_ has quit IRC18:51
*** paladox_ has joined #opendev18:52
clarkband the second is done now. Logs have been uploaded to the regular location. I'm working on a rerun of consistency checks now18:55
clarkb275 conflicts now (down from 281)19:03
clarkb#status log Cleaned up external account id conflicts with two third party CI accounts. The involved parties were emailed a week ago with the proposed plan. No objection to that plan was received so cleanup proceeded today.19:06
openstackstatusclarkb: finished logging19:06
fungiawesome, thanks!19:07
fungiprogress19:07
clarkbthis was a bit more involved and by hand than previous rounds as the fixups required a bunch of steps against currently active accounts rather than just shutting things down (the conflicts in this case were between human accounts and CI accounts not multiple human accounts)19:09
clarkbbut all of the same underlying tools were used19:09
*** TiagoPrimini has joined #opendev19:11
clarkbfungi: have you seen my more dangerous list yet?19:12
clarkbfungi: review:~clarkb/gerrit_user_cleanups/notes/proposed-cleanups.20210416 this was the result of my just going through and deciding if one or another of each conflict can be deleted with some amount of safety19:13
*** slaweq has quit IRC19:13
clarkba lot of that was based on last activity reports from the gerrit api for reviews and change uploads. I skipped over accounts like the stackalytics account which may just be used to query the api19:13
clarkbalso if anyone knows if the stackalytics account is still used that would be great (or who to talk to about it these days)19:14
fungiclarkb: i saw you created that list, meaning to look through it sometime today19:14
clarkband for accounts that showed more recent activity I figured we would try and reach out to them to make sure we aren't breaking some workflow they have19:14
clarkbsimilar to the third party ci account cleanups I just did19:15
clarkbfungi: thanks!19:16
*** paladox has quit IRC19:28
*** slaweq has joined #opendev19:32
*** HeroicHitesh has quit IRC19:34
*** whoami-rajat_ is now known as whoami-rajat19:59
*** vishalmanchanda has quit IRC20:02
*** _mlavalle_1 has joined #opendev20:03
*** mlavalle has quit IRC20:06
*** DSpider has joined #opendev20:19
*** slaweq_ has joined #opendev20:22
*** fressi has joined #opendev20:23
clarkbianw: I read luca's comment here https://gerrit-review.googlesource.com/c/plugins/zuul-results-summary/+/303262 to mean we are good to land that change20:34
clarkbthen we can update our images with that and get the jeepyb fixes in as well \o/20:34
fungiexcellent20:37
*** guilhermesp has quit IRC20:37
*** mattmceuen has quit IRC20:37
*** jmorgan has quit IRC20:37
*** CeeMac has quit IRC20:37
*** paladox_ has quit IRC20:37
*** seongsoocho has quit IRC20:37
*** walshh_ has quit IRC20:38
*** ildikov has quit IRC20:38
*** clayg has quit IRC20:38
*** whoami-rajat has quit IRC20:38
*** diablo_rojo_phon has quit IRC20:38
*** erbarr has quit IRC20:38
*** CeeMac has joined #opendev20:38
*** clayg has joined #opendev20:38
*** mattmceuen has joined #opendev20:38
*** jmorgan has joined #opendev20:38
*** walshh_ has joined #opendev20:38
*** ildikov has joined #opendev20:38
*** mnasiadka has quit IRC20:38
*** whoami-rajat has joined #opendev20:38
*** guilhermesp has joined #opendev20:38
*** mnaser has quit IRC20:38
*** erbarr has joined #opendev20:39
*** mnasiadka has joined #opendev20:39
*** mnaser has joined #opendev20:39
*** paladox_ has joined #opendev20:39
*** seongsoocho has joined #opendev20:40
*** paladox_ is now known as paladox20:42
*** roman_g has quit IRC20:45
clarkbI think I may have approval/submit abilities there, but having never done that before don't want to overstep. I'll let ianw take a look when the au day starts20:45
clarkb(it looks like we need to +1 verify it to make it mergeable, not sure if we typically do that by hand on that repo or not)20:46
clarkbfungi: do you think we should look at landing more of those openstack project reparenting changes or do we want to have the release team (elod) check stuff out first before proceeding? I guess if things need editing not needing to rework hundreds of project configs is a good thing21:08
fungiyeah, i'd rather have a thumbs-up before we end up needing to revert more of them21:10
clarkbwfm21:10
ianwclarkb: np, let's merge21:59
ianwsubmitted; we'll just need to modify the other change to copy the .jar now22:01
clarkbianw: and it looks like we do manually +1 verify on that repo22:01
openstackgerritIan Wienand proposed opendev/system-config master: Handle zuul-summary-results as .jar / per-project config  https://review.opendev.org/c/opendev/system-config/+/77811622:03
ianwclarkb: yeah; the polymer unit testing bits and how to plugin them in were just too much for me to get my head around :)  but contributions welcome22:04
ianwi've basically relied on system-config testing and it's screenshots for any changes22:04
*** smcginnis has joined #opendev22:05
clarkbcool, I just noticed I had the ability to toggle those votes, but wasn't sure if it was intended they be used :)22:06
ianwyep, verify and submit away :)22:06
ianwfungi: ahh, sorry I'd missed the bullseye addition to dib jobs getting in, did that make it in a release?22:06
ianw... to answer my own question, not yet22:07
clarkboh whoops did we land the image additions too soon?22:07
fungiianw: oh, i didn't think to check :/22:07
fungishould we revert, or release?22:08
ianwit looks like there we just a couple of tweaks to files to get the job working that the builders will need22:08
ianwi can release, but we'll need to push and update to nodepool as well to get new builder images22:09
fungiahh, yeah22:11
fungii'm happy to push a revert and a revert of the revert if that's cleaner22:11
ianwi'm just seeing if dib has anything important outstanding to merge before a release22:11
fungiand sorry again about that, i didn't realize that's what the change was waiting on22:12
fungii suppose we could switch to deploying dib master in our nodepool images if we wanted22:12
fungithere's likely a reason we don't22:12
fungiwe'd still have to do something to trigger a new nodepool image build/publish cycle regardless22:13
*** elod has quit IRC22:14
ianwok, nothing critical; i'll push a release22:14
*** whoami-rajat has quit IRC22:17
clarkblooks like we have had a couple of post failures recently. At least one of them is an upload to ovh error. Checking the url that failed does load in my browser and that works right now22:34
clarkbI guess keep an eye on it and remove the ovh upload target if it persists or worsens?22:35
*** calcmandan has quit IRC22:49
*** calcmandan has joined #opendev22:49
*** tosky has quit IRC23:23
*** stevebaker has quit IRC23:25
*** stevebaker has joined #opendev23:25
*** DSpider has quit IRC23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!