Wednesday, 2022-03-30

*** rlandy is now known as rlandy|out00:36
*** mazzy509881292958085946 is now known as mazzy5098812929580859402:08
*** mazzy509881292958085945 is now known as mazzy5098812929580859402:26
*** soniya29 is now known as soniya29|rover03:41
*** ysandeep|out is now known as ysandeep04:26
*** marios is now known as marios|ruck05:06
frickleroh, fun, new setuptools have reached our docs builds
*** arxcruz is now known as arxcruz|out07:15
*** jpena|off is now known as jpena07:37
*** ysandeep is now known as ysandeep|lunch07:59
*** pojadhav is now known as pojadhav|afk08:08
*** pojadhav|afk is now known as pojadhav08:39
*** ysandeep|lunch is now known as ysandeep08:58
*** soniya29|rover is now known as soniya29|rover|lunch09:05
ChayanPHi everyone, I am new to open source and can you please suggest some beginner friendly issues 09:57
ChayanPto contribute to09:57
*** marios|ruck is now known as marios|ruck|lunch10:01
*** soniya29|rover|lunch is now known as soniya29|rover10:07
*** ysandeep is now known as ysandeep|afk10:09
*** rlandy|out is now known as rlandy10:22
*** marios|ruck|lunch is now known as marios|ruck10:30
fungiChayanP: that will depend a lot on what you want to work on10:35
ChayanPI was hoping todo some programming related work10:37
fungibut on what sort of project? is there any in particular that you use, for example? you'll usually have more luck trying to fix bugs in software you already use10:39
ChayanPI have not used open stack before but was hoping to learn more  cloud infra I do not have any specific project in mind, but would be willing to learn the project10:55
fungiif it's openstack you're interested in, they have a landing page for interested contributors here:
fungithere's also a general openstack contributor guide for code and documentation contributions here:
*** soniya29|rover is now known as soniya29|rover|brb11:09
*** dviroel|out is now known as dviroel11:17
*** ysandeep|afk is now known as ysandeep11:18
*** soniya29|rover|brb is now known as soniya29|rover11:42
mgagnefungi: can iweb cloud be disabled? it's not going offline right now as previously discussed but we have some issues that will impact the API so it might be better to disabled completely for now.11:44
fungimgagne: we were hoping to keep it up through the next few hours until the release is finished. mass tag pushing is underway as i type this11:49
fungibut if it's not usable then sure, it's better we remove it from our configuration11:49
mgagnefungi: there are unfortunately issues outside our control right now.11:51
mgagneI can try to give you a workaround, issue is with domain/DNS.11:51
fungimgagne: it looks like the release activity, while resulting in a lot of builds, is going quickly and not consuming as much of our available quota as i expected, so taking iweb out of the mix shouldn't have any impact i don't think11:51
fungii'll push up a change to disable booting nodes there for now11:52
mgagneanything already booted should work11:52
mgagnebut might not be able to boot new nodes since API won't be reachable11:53
mgagneOr if you can "hack" the thing:11:53
opendevreviewJeremy Stanley proposed openstack/project-config master: Temporarily disable launching in iWeb
opendevreviewJeremy Stanley proposed openstack/project-config master: Revert "Temporarily disable launching in iWeb"
fungimgagne: no worries, i'll emergency approve 835822 and wip 835823 until we hear from you11:55
mgagnefungi: thanks, sorry about that. it's unfortunately currently outside our control.11:56
fungino need to apologize! we appreciate the help and the heads-up11:57
mgagnefungi: issue seems to be fixed or was fixed a couple minutes ago. there might still be some hiccups but I think DNS TTL was less than 30 minutes so things should recover relatively fast.12:10
fungimgagne: ahh, okay, so should i cancel the removal?12:10
mgagneif you can reach the APIs from your systems, IMO, it should be fine12:11
fungiyep, looks like they're resolving correctly for us. i'll abandon the removal change and its revert12:12
fungithanks again!12:12
*** ysandeep is now known as ysandeep|afk12:20
fungiclarkb: any idea if the current situation with logstash being broken could account for jobs spending a couple of minutes on "TASK [submit-logstash-jobs : Submit logstash processing jobs to log processors]"?12:20
fungiunfortunately, since the releasenotes jobs are intentionally serialized, that's contributing to very slow burndown in the tag pipeline12:33
fungifortunately, we probably don't need to hold up the final release announcement waiting for them to complete12:34
*** ysandeep|afk is now known as ysandeep12:58
opendevreviewJeremy Stanley proposed opendev/system-config master: Drop tap-as-a-service tarballs site redirect
Clark[m]fungi: looks like the logstash gearman queue is quite large. I suspect that makes it slow to submit new jobs. I think you can restart the geard on logstash.o.o to reset the queue to improve that. We'll lose the backlog but we're aiming to shut it all down soon anyway13:29
fungithanks! i'll give that a shot13:29
fungiand done. i wasn't sure if geard held that in memory or persisted it to disk across restarts13:30
fungihopefully that speeds things up13:31
Clark[m]It is in memory13:31
fungithe other thing i'm noticing is that, with the releasenotes jobs serialized via semaphore, there seems to be a several minute delay between when one completes and when the next one starts. i guess that's zuul waiting to issue the node request until the semaphore is freed up13:32
fungialso once you're at your computer, 835829 cleans up something we noticed in cross-checking the release13:33
Clark[m]Looks like it is failing on the setuptools issue. I wonder if that impacts my planned gitea upgrade for later today. We can land a change similar to tripleo's workarounds to address that.13:36
Clark[m]fungi: are changes like that important enough to deviate from typical morning routine? Looks like the release in general is ok?13:37
Clark[m]Just trying to figure out if I'm taking the kids to school this morning or opening the laptop early :)13:42
fungiClark[m]: not urgent, no13:45
marios|rucko/ folks can someone please check this when you have some time (updates ops for our oftc chan) thank you 13:50
fungimarios|ruck: approved, sorry we missed that one13:55
dpawlikhey fungi. So according to your comment you suggest to put the information into this document:  am I right?13:57
fungimarios|ruck: i suspect the removals will still need to be done by one of the people in your admins list talking to chanserv, since i think accessbot avoids touching entries which aren't in its configuration. if they're still in there after the new additions show up, i can help rlandy or chandankumar out with the necessary syntax13:59
fungidpawlik: into which can be updated through edits to
fungier, through edits to
opendevreviewMerged openstack/project-config master: Update channel ops for oooq (tripleo ci) channel
dpawlikfungi: ack. Will add it there14:15
dpawlikfungi: just FYI, we added more log files to be scraped,  looks so far ok. Also good that the instance has been resized to flavor with more vcpus -14:17
marios|ruckfungi: ok thank you noting to check that 14:20
*** pojadhav is now known as pojadhav|afk14:31
*** ysandeep is now known as ysandeep|out14:37
fungimarios|ruck: looks like it got applied, if you want to double-check the access list chanserv has for it14:51
marios|ruckthank you fungi 14:57
*** marios|ruck is now known as marios15:00
clarkbyes the setuptools + tox + docs/linters situation in system-config will prevent the gitea upgrade from landing.15:03
clarkbfungi: ^ any idea if we hvae a workaround change yet? If not I guess I'll push one for packages =?15:03
*** dviroel is now known as dviroel_15:04
*** dviroel_ is now known as dviroel15:04
fungii'm not aware of a workaround change yet, will check gerrit just to be sure15:04
funginothing for "is:open project:opendev/system-config"15:05
fungiso unless there's a different workaround, i think it's not been proposed yet15:05
clarkbok working on it15:06
mariosfungi: looks like you're right it has kept the previous entries (chanserv) 15:06
fungiyeah, any of the folks with admin permission for that channel should be able to remove the old entries15:07
mariosfungi: ack thank you will dig more into it15:07
fungior i can help with it, but my dance card is already pretty full on release day15:07
mariosfungi: ack :) thanks for help15:07
*** marios is now known as marios|ruck15:10
opendevreviewClark Boylan proposed opendev/system-config master: Workaround setuptools 61.0.0 package auto discovery problems
clarkbI think ^ may do what we want15:11
clarkbfungi: any idea if the logstash stuff sped up after the geard restart?15:12
fungii'll have to check. does ansible give us a per-task duration by any chance?15:13
fungithough if it's a significant speedup i can probably tell just by watching the log stream for the current build15:13
fungiyeah, looks like it helped!15:14
clarkbI think ansible gives yo ua timestamp when each task completes so yo ucan take a delta between two task completions15:14
fungi2022-03-30 15:14:13.546087 | TASK [submit-logstash-jobs : Submit logstash processing jobs to log processors]15:16
fungi2022-03-30 15:14:14.787645 | localhost | ok15:16
fungione second instead of several minutes15:16
clarkblooks like  835849 is going to be happy in the gate. If we land that we can land the redirect removal15:24
clarkbre the redirect removal at this point nothing should really be using Do we know what is/was using it?15:24
fungiclarkb: where is involved?15:28
clarkbfungi: isn't that what thos eredirects are for?15:28
fungitarballs site15:28
clarkboh this is docs nevermind15:28
clarkbah tarballs15:28
funginot docs either, no15:28
clarkbI didn't realize we'd ever done redirects for tarballs15:29
clarkbI don't think we've done those for recent renames15:29
clarkbmaybe it has been on a case by case basis15:29
fungiyeah, all the stuff which had been publishing tarballs to which wasn't an official openstack project got moved to or other namespaces15:29
clarkb has a +1 now15:29
fungisomeone (me? i don't remember now) scripted up all the file moves in afs15:30
clarkbah it was for the openstack org trimming15:30
clarkbnot general renaming15:30
clarkbif we land 835849 we can land the redirect change and then probably upgrade gitea as well15:34
clarkbWas going to wait on a complete all clear for the relesae before doing that bu I think we are close if not already there15:34
fungiit's all clear. we're just waiting for the last of the releasenotes jobs to finish but they're going to be hours still and aren't all that critical (merging another change to any branch of an affected project will update them anyway)15:38
opendevreviewMerged opendev/system-config master: Workaround setuptools 61.0.0 package auto discovery problems
fungiokay, i'm headed out to an appointment but should be back in an hour or so15:40
clarkbcool I just +a'd the redirect chagne and can approve the gitea change when you get back15:41
fungiyep, will jump back in here as soon as i can15:42
clarkbno rush15:42
*** dviroel is now known as dviroel|lunch15:44
*** marios|ruck is now known as marios|out16:20
clarkbnb01 and nb02 have filled their /opt. I'm going to see about cleaning them up16:26
*** jpena is now known as jpena|off16:28
clarkbthere wer ea number of leaked intermediate vhd files on both hosts. I've cleared those out. I'll stop the daemons one at a time and clear out /opt/dib_tmp next16:31
corvusinfra-root: i got an error from gerrit pushing a zuul tag:
corvusi checked and the tag i pushed is on a commit that is in a fresh clone of zuul from gitea16:33
clarkbcorvus: that looks a lot like the thing we put the no thin git review flag in place for16:33
clarkblet me double check what that does16:33
clarkbcorvus: it sets git push --no-thin16:34
clarkbif it is the same issue then it has to do with the negotiation between cgit and jgit not agreeing on whether sufficient info was provded during the push16:34
corvusah.  shall i try again with that?16:34
clarkbya I think you can try with --no-thin and that forces your C client to send jgit a lot more info and hopefully that is enough to sort it out16:35
corvusdone; success16:35
clarkbthere has been a long open bug upstream about it and it sound slike some people maybe understand it but it isn't easy to fix?16:35
clarkbwe have an lp bug from like 2013 about it on our side too16:36
corvusshould be `git push --harder`  ;)16:36
clarkbI see the tag and it is on master16:37
clarkbso ya I suspect it was just the negotiation16:37
clarkbI've got nb01's daemon stopped while I clear out /opt/dib_tmp then I'll reboot to clear any monts that may have been stale and start services again16:41
clarkband then I'll repeat with nb0216:41
opendevreviewMerged opendev/system-config master: Drop tap-as-a-service tarballs site redirect
clarkbI went ahead and put nb01 and nb02 in the emergency file tooso that ansible won't restart things while I wait for the disk cleanup to complete16:52
fungiokay, i'm caught up and it doesn't seem like there are any new fires16:56
clarkbnothing too hot anyway.16:56
fungialso i should start a backup prune today, we've been getting warnings since late last week that i've been putting off dealing with16:56
clarkbah I haven't set up the infra-root inbox on the laptop yet so missed that16:56
clarkbOnce nodepool builders are cleaned up I can approve the gitea upgrade unless you want to get it started sooner16:57
fungii'm happy to approve the upgrade change now if you're good with that16:58
clarkbyup I think so. There are a few release note jobs for openstack to complete but this shouldn't impact that16:59
*** dviroel|lunch is now known as dviroel17:00
clarkbnb01 is rebooting now17:05
fungiborg pruning is in progress in a root screen session on
clarkbis up and running services again. I pulled it out of the emergecny file. On to nb02 now17:07
clarkbcorvus: the zuul status dashboard for 835307 shows an estimated completion time despite one of the jobs not having started yet17:15
clarkbI'm not sure that is accurate? the unstarted job takes about 40 minutes too iirc17:16
clarkbI guess it shows time to completion for the running jobs which may be helpful17:18
corvusperhaps it's too simplistic and needs to be updated to account for unstarted jobs.  i'm not sure.17:19
fungiyeah, all the queued changes in the tag pipeline also show a projected completion time even though they don't have any builds in progress yet17:21
clarkbnb02 is done and out of the emergency file again17:36
opendevreviewMerged opendev/system-config master: Update gitea to v1.16.5
opendevreviewMerged opendev/system-config master: Clarify the purpose of this repository
clarkbWill be a few minutes before the gitea deploy happens due to the hourly jobs still in progress. I'll moitor it18:17
clarkbChrome version 100 is out now. There was some concern that various websites wouldn't work due to user agent parsing and behavior setting18:21
clarkbkeep an eye out for any reports that our applications have trouble I guess18:21
fungiwill do, thanks18:31
clarkbgitea01 has updated if anyone wants to take a look, but my quick check indicates it is happy18:43
fungiyep, saw that just hit, working fine for me so far18:46
fungicloning from it and browsing18:46
clarkball 8 are done now19:03
clarkbhaven't seen anything unexpected19:03
fungiborg prune completed19:19
fungi#status log Pruned backups on in order to free up some storage, info in /opt/backups/prune-2022-03-30-17-05-53.log19:20
opendevstatusfungi: finished logging19:20
*** dviroel is now known as dviroel|out21:16
*** rlandy is now known as rlandy|bbl22:11
*** ianw_pto is now known as ianw22:24

Generated by 2.17.3 by Marius Gedminas - find it at!