Monday, 2022-09-26

*** ysandeep|out is now known as ysandeep05:42
*** jpena|off is now known as jpena07:37
*** lbragstad2 is now known as lbragstad08:01
opendevreviewJames Page proposed openstack/project-config master: charms: add new ceph subteam
*** ysandeep is now known as ysandeep|afk08:25
*** marios is now known as marios|call09:01
*** soniya29 is now known as soniya29|afk09:30
*** ysandeep|afk is now known as ysandeep09:50
opendevreviewJames Page proposed openstack/project-config master: charms: add new ceph subteam
opendevreviewMerged openstack/project-config master: Update link to zuul gating docs
*** ysandeep is now known as ysandeep|afk11:20
*** soniya29|afk is now known as soniya2911:20
*** ysandeep|afk is now known as ysandeep11:38
*** marios|call is now known as marios12:01
*** ysandeep is now known as ysandeep|afk12:40
*** ysandeep|afk is now known as ysandeep13:08
*** dasm|off is now known as dasm13:28
*** ysandeep is now known as ysandeep|out14:10
*** dviroel is now known as dviroel|lunch15:10
*** marios is now known as marios|out15:29
clarkblooks like sphinx made a few releases over the weekend . Ithink at least one of the aerlier ones broke nodepool doc building and appears better now15:31
clarkbfyi if you see any sphinx 5.2.0 errors15:31
fungithanks! and yeah, i suspected that was the cause for those15:32
fungiclarkb: with "" in my /etc/hosts this url from my browser history is working (once i bypass the cert warning of course):
clarkbfungi: you had to copy the archive to the appropriate location too right? Any feedback on whether or not we can use a simpler dir structure for that? The need for parsing info out of the url sort of informed my decisions there but it seems clunky. However if it works...15:58
fungiclarkb: yeah, i'm updating the etherpad with the notes about that right now. i think the location is fine, though i did need to create the parent dir16:00
fungiwhich we probably should add to ansible16:01
fungii'll do a new revision with that shortly16:04
mnasiadkafungi: - are we waiting here for something?16:11
mnasiadkaah, probably for
clarkbI can probably review those later today16:13
fungiwell, also ianw may simply have not felt confident enough to approve it when he was about to step out the door on holiday16:13
clarkbya I think if the testing looks good we can land the prod update and worst case we pause those builds. But I need to catch up on a few other things first16:13
fungi858606 adds testing for it, which is necessary but can probably be approved in parallel. clarkb's additional eyes on those would be appreciated though, since we accidentally approved some broken configs in similar places in recent weeks16:14
clarkbfirst on my agenda is fixing nodepool functional testing log collection. I think my change late last week to gather libvirt logs exposed a bug in this16:15
opendevreviewMerged zuul/zuul-jobs master: test-requirements: bump to Ansible 2.8
*** dviroel|lunch is now known as dviroel16:20
fungiclarkb: actually, i take that back, my testing was flawed (i had initially tried overriding before realizing the initial change only adds a vhost for
mnasiadkaclarkb, fungi: thanks, need this for Kolla RL9 arm images support, but not that urgent that we can't followup with arm support later ;)16:26
fungicurrently getting a 403 forbidden from apache on the test server, but i can't figure out why apache isn't logging anything (not even successful requests)16:27
clarkbfungi: the initial change should have server aliases for all of the domains16:27
clarkbthere is only one vhost that serves everything.16:27
fungioh, indeed i see that now16:28
fungibut yeah, logging doesn't seem to be happening in either case16:28
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: zuul_azure_storage_upload: rename
clarkbfungi: are you sure you aren't hitting the old server then?16:29
fungitcpdump on the test server confirms i'm hitting it with my browser16:30
fungimaybe ${APACHE_LOG_DIR} isn't expanding the way we want it to16:30
clarkbfwiw I reproduce what you see16:30
fungithe current lists.o.o apache vhost configs do also use ${APACHE_LOG_DIR}16:31
fungiso it's probably not that16:31
fungiunless we're missing a module or config somewhere16:32
clarkbthere are also logs in the log dir16:32
fungiyeah, the current ones are just zero-byte16:33
clarkbright, they also rotated just a few minutes ago16:33
clarkbbut the older ones don't have any additional useful stuff either16:33
fungidisk filled up16:33
* fungi sighs16:33
fungi/dev/xvda1       37G   37G     0 100% /16:34
fungi/var/log/unbound.log is almost 17gb16:34
fungii guess we set the verbosity very high on it for jobs16:35
clarkbfungi: :( also the archive files don't actually exist16:35
clarkbso apache is doing the correct thing16:35
fungii'll truncate it and reboot the server16:35
clarkbwe just don't see logs becuase the disk is full16:35
opendevreviewMerged zuul/zuul-jobs master: zuul_azure_storage_upload: rename
fungii overwrote /var/log/unbound.log with /dev/null and rebooted16:36
fungii'll check the file paths for the archives in a sec, i probably just got the destination wrong16:37
fungi/dev/xvda1       37G   20G   16G  57% /16:37
clarkbfungi: they are symlinks16:37
fungithat looks better16:37
clarkband the targets don't exist16:37
fungioh! yep i see it16:38
fungiwe need to rewrite the symlinks since they weren't created as relative16:38
fungithe way public vs private archives work in mm2 is that everything goes into the private archives, and then anything which is set to have a "public" archive gets a symlink to the private one16:39
fungii'll need to noodle on the best way to redo the symlinks16:39
clarkband will apache rules need fiddling to deal with symlinks to other dirs?16:40
fungiquite possibly16:40
*** jpena is now known as jpena|off16:41
opendevreviewMerged zuul/zuul-jobs master: Fix CORS and endpoint in AWS log upload
clarkbre the cert expiry warning that went out this morning. My browser sees a current cert. I suspect that we've got stale apache workers on the host16:56
clarkbyes there is one older one. I think we can manually restart apache to address this.16:57
clarkbnow is probably not a good time for that, but I can do this later today16:57
fungisounds like a plan, and yeah i saw the notification earlier and suspected that was the cause17:02
clarkbmnasiadka: one small thing on we should go ahead and address before we build images since the turn around time on those is quite large. Otherwise I think we can land that once fixed17:15
fungiclarkb: okay, a rewrite of the symlinks was sufficient, now returns content from the held server for me17:17
clarkbfungi: cool testing here too17:17
clarkbfungi: I get a 503 hitting the root of the server now, but agree that url works17:18
clarkbfungi: I think maybe containers weren't restarted when you rebooted the server?17:19
clarkbyes they are all exited17:19
clarkbI'm going to up them17:19
*** dviroel is now known as dviroel|dr_appt17:20
clarkbhrm its still broken17:22
clarkbnevermind now it works. I guess it may have needed time to spin up17:23
clarkbfungi: so ya that looks good to me17:23
fungiyeah, they take a few to come up17:23
fungiokay, so need to create /var/lib/mailman/web-data/mm2archives and need to make docker-compose start the containers at boot?17:23
clarkbwell docker-compose should start them on boot automatically I thought17:24
clarkbit should only not start them on boot if you've deleted the containers. but maybe the restart policy comes into play here in a way I didn't know about17:24
clarkbya apparently we need to add restart policies for this17:25
fungiokay, the additions to the migrate script example in the etherpad have been tested now and i've used that exact code to relocate and relink all the remaining sites/archives on the held server17:36
fungiclarkb: should i include that script in 851248 at this point?17:37
clarkbfungi: can't hurt17:39
clarkbwill help us see the updates to the script more easily than in etherpad too (git diff is a better record of that)17:39
fungiclarkb: in our other compose files we seem to just set restart: always (except the jitsi-meet ones which default to unless-stopped, and gerrit where we restart: no)17:44
fungiis always what we want for these, do you think?17:44
opendevreviewDmitriy Rabotyagov proposed opendev/system-config master: Add Ceph Quincy mirror
noonedeadpunko/ 17:45
noonedeadpunkI know that afs is quite limited in space, but decided to push patch first and decide on how feasible it is to land it later (always can abandon)17:46
noonedeadpunkFor balancing used diskspace I'd suggest to drop ceph-nautilus, but it's currently referenced here: Tough I'm quite unsure about how much sense bionic does make today...17:47
noonedeadpunkgiven that release has been set to U17:47
fungi says that the other ceph reprepro mirrors take less than 10gb each and in most cases closer to 1gb (though we give them a 50gb quota just in case)17:49
fungiafs01.dfw is around 85% of 4tb used, so adding this probably won't make that much of an impact17:50
opendevreviewDmitriy Rabotyagov proposed opendev/system-config master: Add Ceph Quincy mirror
opendevreviewJeremy Stanley proposed opendev/system-config master: Add a mailman3 list server
funginoonedeadpunk: probably the bigger hurdle is someone finding time to create the volume for that. i can try to get to it later today if there's consensus17:52
noonedeadpunkfungi: well, I can't say there's some huge rush - it's nt blocking gates at least17:53
noonedeadpunk*not blocking17:54
fungii voted +1 on it for now since it will break if it lands before the volume has been added for it17:54
noonedeadpunkok, great, thanks!17:54
opendevreviewJames E. Blair proposed zuul/zuul-jobs master: Add prefix to azure log upload
fungiclarkb: running mysqldump on the held node (with the full production data set for all ml sites imported) identically to how we're doing backups in the cronjob installed on it seems to work without error for me. took 1m51.350s to complete18:10
clarkbfungi: excellent. I would've expected similar errors about packet sizes if we had gotten the config wrong18:11
fungian outstanding work item is that i need to shorten nonmember_rejection_notice for and, and also member_moderation_notice for openstack-stable-maint@lists.openstack.org18:13
fungii've added it to the todo list at the bottom of the etherpad for now so i don't forget18:14
clarkband then we should probably rerun the migration just to eb sure that doesn't expose anything new18:14
clarkbor do you think that is overkill? I guess worst case it confirms we've shortened sufficiently18:15
funginah, not overkill at all. we've fixed several things since the last import18:15
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Clarify that stage-output doesn't actually write to zuul_output_dir
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Clarify that stage-output doesn't actually write to zuul_output_dir
opendevreviewMerged zuul/zuul-jobs master: Add prefix to azure log upload
opendevreviewMerged openstack/diskimage-builder master: Add Rocky 9 ARM64 functional test
clarkbfungi: some linter doesn't like the construction of your for loop in the migrate script. However, I think it is faulty and we should ignore that19:13
clarkbthe way you've written it is more readable :/19:13
clarkbinfra-root with ianw out this week and frickler and corvus not typically participating in the meetings I'm wondering if we should cancel tomorrow's meeting? Or maybe have a much condensed agenda and focus on jaeger and mm3 migration stuff?19:14
fungii like readable19:47
fungibut i'll see what i can do19:47
fungibashate is a bit naive as linters go19:47
fungisystem-config-run-mirror-x86 also had a random disconnect on that buildset19:51
fungi'E010 The "do" should be on same line as for' seems like a very short-sighted concern, as it's not solvable when your loop condition is longer than the line length limit19:54
opendevreviewJeremy Stanley proposed opendev/system-config master: Add a mailman3 list server
opendevreviewJeremy Stanley proposed opendev/system-config master: Ignore bashate E010
fungiclarkb: 859311 needs to be rebased on the newer 859172 revision in order to exercise it, right?20:05
clarkbfungi: its the other way around20:06
clarkb859172 is based on 85931120:07
fungigah, i always read gerrit's "relation chain" upside down20:07
fungi859172 doesn't seem to run a functional-openstack job though? what's it running that's based on that abstract job?20:08
clarkbI think it is because the image build job timed out20:09
clarkbthere is a little note that two jobs were skipped20:09
fungioh, okay that makes sense. i assumed it was being skipped because of file filters, but that makes more sense20:33
clarkbI'm thinking I'll do an abbreviated agenda for tomorrow's meeting due to having fewer people20:35
*** dasm is now known as dasm|off21:35

Generated by 2.17.3 by Marius Gedminas - find it at!