| opendevreview | Merged opendev/system-config master: Add robots.txt to Zuul web https://review.opendev.org/c/opendev/system-config/+/914989 | 00:09 |
|---|---|---|
| *** elodille1 is now known as elodilles | 04:41 | |
| opendevreview | Thierry Carrez proposed opendev/irc-meetings master: Move Large Scale SIG meeting to 3rd wednesday https://review.opendev.org/c/opendev/irc-meetings/+/916875 | 09:45 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add new repository for NebulOuS testing data https://review.opendev.org/c/openstack/project-config/+/916876 | 09:46 |
| opendevreview | Merged opendev/irc-meetings master: Move Large Scale SIG meeting to 3rd wednesday https://review.opendev.org/c/opendev/irc-meetings/+/916875 | 11:35 |
| *** blarnath is now known as d34dh0r53 | 15:10 | |
| fungi | headed to lunch, back soon | 15:19 |
| clarkb | fungi: when you get back I've just drafted https://etherpad.opendev.org/p/fJz3-CUD_fGqj4pImgK0 to send to debian. Does that look like a correctly formatted bug report to you? | 15:22 |
| clarkb | the person that ended up submitting the upstream fix actually seems to be the debian maintainerso maybe this will be an easy one | 15:29 |
| frickler | clarkb: there seems to be a reference mismatch, [2] vs. [1], otherwise that lgtm | 15:37 |
| clarkb | fixed thanks | 15:52 |
| *** tobias-urdin4 is now known as tobias-urdin | 15:57 | |
| clarkb | we can probably go ahead and merge https://review.opendev.org/c/opendev/system-config/+/915289 today then see if we successfully update the clouds? | 16:01 |
| clarkb | for a glean release I think I will tag e4c80ba337e11ce815bd31ed06a64b5cdb218ed5 as 1.24.0 | 16:11 |
| clarkb | and ya looks like I need to go to backups for gpg stuff so this might be a bit | 16:21 |
| frickler | clarkb: fungi: is there any objection to me creating the debian-ceph-reef afs volume? or is there a different issue with https://review.opendev.org/c/opendev/system-config/+/916552 ? | 16:22 |
| clarkb | frickler: after discussion yesterday I have fewer concerns. It seems like ceph as part of IBM may actually be producing packages unlike previuosly. | 16:23 |
| frickler | +1 for glean==1.24.0 | 16:23 |
| fungi | clarkb: the request seems fine to me, looks like you've seen https://www.debian.org/Bugs/Reporting and followed the relevant syntax. see also https://www.debian.org/Bugs/Developer#tags for tags you might want to set, such as "patch" and "upstream" and perhaps "fixed-upstream" | 16:36 |
| fungi | glean 1.24.0 at e4c80ba sounds good | 16:38 |
| fungi | frickler: 916552 is fine by me, feel free to add the volume for it | 16:38 |
| clarkb | fungi: something like that for tags? | 16:48 |
| fungi | clarkb: yeah, that syntax looks correct to me | 16:54 |
| clarkb | fungi: cool and just to be clear the pseudo headers are pseudo because they go in the message body not actual header list? | 16:55 |
| fungi | exactly | 16:55 |
| fungi | i like that debian's bts is fully usable via e-mail, but it does take some getting used to | 16:55 |
| clarkb | cool I'll get the sent out momentarily | 16:55 |
| fungi | in general the debian community prefers bug reports generated by the reportbug utility, but that's mainly just so that newcomers don't have to worry about whether they've got the syntax correct | 16:56 |
| fungi | also, avoid including an html part if you can | 16:57 |
| clarkb | oh I think this mua may send both | 16:59 |
| clarkb | oh well | 16:59 |
| clarkb | I already sent it. Any idea how long it will take to show up in the bug list? | 16:59 |
| fungi | give it 30 minutes, there's greylisting and such in between to keep spammers to a minimum | 17:00 |
| clarkb | got it | 17:00 |
| fungi | that's around how long it usually takes my messages to the bts to get reflected anyway. sometimes as much as an hour | 17:01 |
| clarkb | I will attempt to practice patience | 17:02 |
| fungi | the debian bts was developed by gen-x'ers ;) | 17:03 |
| fungi | we don't know the meaning of "instant gratification" | 17:03 |
| fungi | after discussing with clarkb, i've pushed the signed tag for glean 1.24.0 at e4c80ba337e11ce815bd31ed06a64b5cdb218ed5 | 17:10 |
| clarkb | thanks! | 17:10 |
| fungi | no sweat | 17:10 |
| clarkb | we should periodically check nodepool iamge builds over the next few days to make sure there isn't anything obviously wrong iwht it that CI missed | 17:10 |
| fungi | good reminder | 17:13 |
| slittle | Help! storyboard.openstack.org won't let me save my new story | 17:15 |
| slittle | Why is 'Save Changes' grey's out? | 17:15 |
| fungi | slittle: there's a bug if you type or paste the full name of the repository for the initial task. try backspacing and selecting it from the typeahead prediction | 17:16 |
| fungi | the only other typical reasons i can think of are when a required field hasn't got any content | 17:17 |
| slittle | I added a dozen tasks. does the paste bug apply to all of them? | 17:18 |
| fungi | oh, the initial story creation will throw errors if you try to add multiple tasks at creation time. better to create the story with just the first default task and then add more once that's done | 17:19 |
| fungi | otherwise i think it ends up trying to add them to the database in parallel and steps on its own table locks | 17:20 |
| slittle | starting over ... | 17:21 |
| slittle | still no joy. Can I cut-n-paste the title/descriptions | 17:22 |
| fungi | should be able to | 17:22 |
| slittle | nope. didn't work | 17:22 |
| fungi | i'll try adding a story to a test project and see if something has broadly broken the server | 17:22 |
| corvus | other infra-roots seeing mailman error emails? | 17:23 |
| slittle | I assume it's ok to NOT click 'Private', and/or 'Vulnerability' | 17:23 |
| fungi | slittle: yes | 17:23 |
| corvus | hrm these emails i'm seeing are bounces from april 17 | 17:25 |
| fungi | corvus: cronspam with "Can't connect to server on '127.0.0.1'" ? | 17:25 |
| corvus | yeah | 17:25 |
| fungi | looks like it tried to deliver to root@localhost for a week and then bounced | 17:25 |
| corvus | so maybe whatever the event was has recovered now (was that a db maintenance time?) and a stuck queue just got dislodged? | 17:26 |
| corvus | ah, 1 week timeout | 17:26 |
| slittle | Ah. Starting from the 'StarlingX' view, create story does not work. Stating from the 'StarlingX/root' project view, create story works. | 17:26 |
| fungi | corvus: that may coincide with the mariadb container upgrade merging/deploying | 17:27 |
| fungi | slittle: was that via the | 17:27 |
| corvus | yeah, that would make sense. so we can probably ignore and revisit if we get more tomorrow :) | 17:28 |
| fungi | "+ create new..." drop-down in the top-left corner | 17:28 |
| fungi | ? | 17:28 |
| slittle | yes. My first failed attempt was from the 'Stories' view filtered on 'stx' | 17:29 |
| fungi | the "+ Add story" to the top-right when looking at a single project prepopulates the project name on an initial task, which is the main difference i thino | 17:30 |
| fungi | think | 17:30 |
| slittle | From the filtered 'Story' view, both '+ Create Story' in upper right, and 'Create-New' -> 'Story' in upper left failed | 17:31 |
| fungi | the "+ Create new..." drop-down in the top-left corner should be contextless regardless of what page you're on, i'll test that | 17:32 |
| fungi | i was able to create https://storyboard.openstack.org/#!/story/2011111 that way just now | 17:34 |
| fungi | there's some flakiness with the javascript for the text input fields where you need to make sure the blue "glow" border highlight appears around the field after you click on it, or for some reason the input validation doesn't realize you've entered anything | 17:35 |
| fungi | i started typing the project name into the project field, selected the one i wanted from the typeahead prediction drop-down once i entered enough of it, then clicked on the title and entered "new story" which also auto-copied into the initial task description, then i put dummy text in the description field and the "save" button became clickable | 17:37 |
| fungi | corvus: yep, https://review.opendev.org/915183 "Upgrade Mailman's MariaDB to 10.11" deployed around 2024-04-17 17:20 utc | 17:39 |
| fungi | and the last cronspam i got about database unreachability was at "Date: Wed, 17 Apr 2024 17:21:01 -0000" | 17:40 |
| fungi | so almost certainly related, and the error address we have configured is just broken so the failures sat in exim's queue for a week and bounced back to the root alias | 17:41 |
| clarkb | I thought we had updated the error address so that it would go to our root addrs? | 17:42 |
| clarkb | maybe that was for some other component and this is still missing the update | 17:42 |
| fungi | the messages are to "root@localhost" but looks like exim's local_domains list doesn't include "localhost" | 17:43 |
| fungi | it does include "@" which is a placeholder for the hostname, but doesn't include other /etc/hosts entries going to the interfaces i expect | 17:44 |
| clarkb | ah so maybe the config we added was not aligned with exim | 17:44 |
| fungi | on our other servers we just set local_domains = @ so it's probably a question of where mailman is configured to send those errors | 17:46 |
| fungi | looks like those errors are coming from django, implying it's something we need to set there | 17:47 |
| clarkb | yes I think we did set it | 17:49 |
| clarkb | playbooks/roles/mailman3/files/web-settings.py sets it I think | 17:49 |
| clarkb | hrm though that is the admin list so maybe not | 17:49 |
| fungi | indeed, we have root@localhost in docker/mailman/web/mailman-web/settings.py and also playbooks/roles/mailman3/files/web-settings.py implying it's copied from upstream's default | 17:50 |
| clarkb | SERVER_EMAIL in that file may also be related? | 17:51 |
| fungi | we do start the container with SERVE_FROM_DOMAIN=lists.opendev.org which should be trickling into the hostname value and thus SERVER_EMAIL | 17:53 |
| fungi | implying that SERVER_EMAIL isn't actually being used for this | 17:53 |
| fungi | i think SERVER_EMAIL is getting used in the from and sender, but the message gets addressed to the entries in the ADMINS list | 17:54 |
| clarkb | got it | 17:55 |
| fungi | i expect we could replace 'root@localhost' there with SERVER_EMAIL except that it's defined earlier in the script so we'd need to rearrange a little | 17:55 |
| fungi | which is another deviation from the upstream copy then | 17:56 |
| clarkb | you mean do root@$SERVER_EMAIL ? | 17:57 |
| fungi | no, i mean SERVER_EMAIL | 17:58 |
| fungi | SERVER_EMAIL = 'root@{}'.format(hostname) | 17:58 |
| clarkb | oh I see | 17:58 |
| fungi | hostname = os.environ.get('SERVE_FROM_DOMAIN', 'localhost.local') | 17:58 |
| fungi | so in our case SERVER_EMAIL contains the value "root@lists.opendev.org" | 17:59 |
| clarkb | ya so if we define that var earlier then we can reuse it. A divergence but probably worthwhile. Or we can just redefine things to make the diff a bit cleaner | 18:02 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Override upstream ADMINS address for mailman https://review.opendev.org/c/opendev/system-config/+/916940 | 18:03 |
| clarkb | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069781 | 18:14 |
| fungi | hah, i had just gone looking for it myself | 18:14 |
| fungi | comparing timestamps, looks like it took about 11 minutes to arrive and get processed | 18:16 |
| fungi | but probably didn't send notifications immediately | 18:16 |
| fungi | corvus: 916940 should stop trying to send those errors you spotted to an unreachable address, so that we get them ~immediately rather than a week after the fact | 20:26 |
| corvus | fungi: oh nice | 20:32 |
| clarkb | I need to do a school run in about 25 minutes so now is probably not a great time for me, but should we try and roughly schedule a time to land https://review.opendev.org/c/opendev/system-config/+/916847 to upgrade the gitea db servers? | 20:47 |
| clarkb | CI testing shows general compatibility with the newer version (as expected) so only the upgrade itself is untested but we've done a few of those now | 20:48 |
| fungi | i'll still be around for a few more hours if you want to do it today | 20:53 |
| fungi | otherwise my meetings tomorrow are done at 17:00 utc | 20:54 |
| clarkb | probably best to do it tomorrow. | 20:54 |
| opendevreview | Merged opendev/system-config master: Override upstream ADMINS address for mailman https://review.opendev.org/c/opendev/system-config/+/916940 | 21:12 |
| clarkb | now we need to stop the database and see where the email goes :) | 21:59 |
| fungi | heh | 21:59 |
| clarkb | I think the ubuntu jammy images will rebuild in about 4 hours | 22:04 |
| clarkb | so we'll be able to check those still boot with new glean son | 22:04 |
| clarkb | *soon | 22:05 |
| clarkb | I've started looking at codesearch for xenial things and while there are a lot of xenial strings out there very few of them seem to be related to zuul configs (on the master branches of projects at least) | 22:10 |
| clarkb | This is encouraging. For all the workload stuff I think we can pretty safely ignore that for nodepool and mirror cleanup | 22:10 |
| clarkb | a big exception to this is ozj so I think starting with ozj cleanup may be a good first step | 22:11 |
| clarkb | is there any concern with removing the python wheel caching for xenial before we cleanup much else? I think this is likely to be ok due to pip fallibg back to building from sdist in a worst case | 22:12 |
| clarkb | but also fi we break jobs now we can just turn them off since the node is going away soon enough | 22:12 |
| fungi | yeah, seems fine to do in that order | 22:20 |
| clarkb | heh as soon as I start looking at the ozj stuff I'm finding where all the threads go. | 22:29 |
| opendevreview | Clark Boylan proposed opendev/glean master: Update zuul config to drop xenial jobs https://review.opendev.org/c/opendev/glean/+/916952 | 22:34 |
| clarkb | for example ^ | 22:34 |
| clarkb | ya I'm finding the threads now. Its like a buncho f software zombies all holding onto references to old xenial stuff | 22:39 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Drop use of python35 job templates https://review.opendev.org/c/openstack/project-config/+/916953 | 22:41 |
| fungi | and the stable/unmaintained branches are what's really going to kill us | 22:42 |
| clarkb | ya my naive initial search really was only the tiniest tip of an iceberg | 22:44 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Stop publishing Xenial wheel mirror/cache content https://review.opendev.org/c/openstack/project-config/+/916954 | 22:50 |
| clarkb | ok topic:drop-ubuntu-xenial has a few more changes now. Unfortunately, this is really only scratching the surface, but I get the sense that this will be a chip away at it slowly kind of task | 22:55 |
| clarkb | I think xenial more so than centos and opensuse and buster is going to basically be: we do a reasonable amount of cleanup then we start force merging the removal of project templates and nodesets and central jobs | 23:01 |
| clarkb | because I'm just looking at one template (openstack-python35-jobs) and suspect it is all over the place. But there are similar templates for charms and for nodejs and for this and for that | 23:02 |
| opendevreview | Clark Boylan proposed opendev/glean master: Update zuul config to drop xenial jobs https://review.opendev.org/c/opendev/glean/+/916952 | 23:06 |
| clarkb | fungi: in ^ the py27 job continues to fail because python2 isn't installed. Do we not have the tox-py27 job configured to actually install python2? that seems like an oversight | 23:16 |
| clarkb | but maybe we can just set a job var and have it work | 23:16 |
| fungi | i think it relied on 2.7 being preinstalled on the platform, yes | 23:17 |
| clarkb | looking at openstack-tox-py27 it pins to bionic | 23:18 |
| clarkb | I can probably do that here for now | 23:18 |
| clarkb | then when we drop bionic we probably don't need py27 support in glean anymore | 23:18 |
| opendevreview | Clark Boylan proposed opendev/glean master: Update zuul config to drop xenial jobs https://review.opendev.org/c/opendev/glean/+/916952 | 23:19 |
| clarkb | ya that worked. Good enough for nwo | 23:31 |
| clarkb | I also like "good enough for who it is for" | 23:31 |
| clarkb | not a fan of airplane mechanic brother's line: "its not going to the moon" | 23:32 |
| fungi | yeah, that's a disturbing phrase in context | 23:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!