| opendevreview | Merged openstack/project-config master: Use vexxhost/project-config in vexxhost tenant https://review.opendev.org/c/openstack/project-config/+/931998 | 00:00 |
|---|---|---|
| tkajinam | E: Failed to fetch https://mirror.sjc3.raxflex.opendev.org/ubuntu/dists/focal/main/binary-amd64/Packages 403 Forbidden [IP: 65.17.193.187 443] | 10:15 |
| tkajinam | I've seen a few 403 from package mirrors today. there might be some issues with a few specific mirrors | 10:16 |
| tkajinam | I've seen 403 from the mirror 5 times for far | 10:36 |
| tkajinam | hmm looks like this specific mirror is still broken. and we see more failures caused by it since more people start their day | 11:16 |
| priteau | Hello. Anyone know when https://review.opendev.org/c/openstack/project-config/+/931631 will be deployed? The new image doesn't seem to be available yet. | 12:16 |
| mnasiadka | clarkb: It seems the locale patch in DIB did not help - I'll do some debug tomorrow to find out what's wrong now ;-) https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_acc/925581/59/check/kolla-ansible-centos9s/accafe5/primary/logs/ansible/bootstrap-servers | 12:27 |
| Clark[m] | tkajinam it looks like afs crashed or is no longer serving content there as the entire root is empty except for robots.txt. I can't look closer until I get kids to school though. | 13:55 |
| Clark[m] | priteau it depends on when images are built which is on a more staggered schedule these days. The code in your jobs should treat that as a cache and fetch the newer images directly if they are not available though | 13:56 |
| Clark[m] | Grafana has an image build dashboard that will show you when things were last built | 13:56 |
| Clark[m] | mnasiadka: maybe there is extra stuff that needs to be installed in addition to configuring rpm to build that locale when installing packages? As mentioned previously I'm not really clued into how that platform handles locales and am still hoping someone who is takes a closer look | 13:58 |
| mnasiadka | I will tomorrow and report back ;-) | 14:02 |
| Clark[m] | mnasiadka: I guess the other thing to check is the image was the newly built one. We capture that in the zuul info log files somewhere I think | 14:21 |
| fungi | my isp has been down again all morning (latest estimate is they'll have it working again by 18:00 utc), so i'm trying to work over a phone tether and don't have access to much besides irc and e-mail for the moment | 14:28 |
| opendevreview | Merged openstack/project-config master: Enable nodepool-in-zuul for opendev tenant https://review.opendev.org/c/openstack/project-config/+/931999 | 14:30 |
| corvus | do we happen to know what image format rax-flex wants? | 14:43 |
| corvus | we don't specify it in clouds.yaml; does that mean it's qcow2? | 14:43 |
| opendevreview | James E. Blair proposed opendev/system-config master: Install clouds.yaml on zuul-launcher https://review.opendev.org/c/opendev/system-config/+/932087 | 14:44 |
| Clark[m] | corvus: I think it does default to qcow2 | 14:47 |
| corvus | cool, that'll make for easier testing. i've got patches staged to add rax-flex to the zuul-launcher, so we can test the image upload there | 14:49 |
| corvus | Clark: fungi can you review https://review.opendev.org/931996 and https://review.opendev.org/932087 ? | 14:53 |
| clarkb | ya dmesg reports afs is very unhappy: afs: Cannot open cache file (code -30). Trying to continue, but AFS accesses may return errors or panic the system | 14:53 |
| corvus | they should be quick, and i'd like to get that deployment going | 14:53 |
| clarkb | also io errors like Buffer I/O error on dev dm-1, logical block 1070, async page read | 14:53 |
| corvus | clarkb: where are the afs errors? | 14:53 |
| clarkb | corvus: dmesg is where I found the one above | 14:53 |
| clarkb | on mirror.sjc3.raxflex.opendev.org | 14:54 |
| corvus | i mean server (and what volume are you looking at) | 14:54 |
| corvus | oh ok client side | 14:54 |
| fungi | aha, so not afs-wide, sounds like there might have been an iscsi outage impacting that cinder volume | 14:54 |
| fungi | on the client where the cache resides | 14:54 |
| clarkb | oh yup it seems to be specific to the mirror node. When I checked earlier I checked other mirrors were returning content too and they were | 14:54 |
| corvus | is this affecting one or all volumes on that server? | 14:55 |
| fungi | maybe a quick reboot will sort it, might have to stop the daemon and blow away the cache dir | 14:55 |
| corvus | ah i see, it's at least the root mirror volume, so no accessing any mirrors | 14:56 |
| clarkb | corvus: I think just vdd which I'm trying to figure out what that maps to | 14:56 |
| clarkb | oh you mean afs volumes ya its the whole thing as far as mirroring goes | 14:56 |
| clarkb | vdd isn't in fstab so that must be the cinder volume? lvs/pvs doesn't return any info | 14:57 |
| corvus | yeah, but with a name like "main-openafs" i'm pretty sure that's lvm | 14:57 |
| corvus | so maybe lvm is in a broken state? | 14:58 |
| clarkb | ya and they remounted ro looks like. So maybe things are sad enough that even lvm can't report things back? | 14:58 |
| corvus | seems like a good guess to me | 14:58 |
| clarkb | in that case should we try a reboot and see if things come back cleaner? | 14:59 |
| corvus | i wonder if we rescan devices/partitions if lvm would report info | 14:59 |
| clarkb | oh that is a good idea | 14:59 |
| corvus | partprobe /dev/vdd did not output happy messages | 15:00 |
| corvus | i'm on team reboot now | 15:00 |
| clarkb | corvus: do you want to do the typing for that or should I? | 15:00 |
| clarkb | (and I guess if it doesn't come up clean we set max-servers to 0 for now) | 15:00 |
| corvus | clarkb: i can | 15:01 |
| clarkb | thanks | 15:01 |
| corvus | kpartx -u /dev/vdd also was unhappy ftr | 15:01 |
| corvus | i suppose there's a chance it got reattached as a different device... 🤷 | 15:02 |
| clarkb | I want to say we use uuids for the setup so that would hopefully emit errors about the device not being present rather than being confused? But ya that seems theoretically possible | 15:02 |
| clarkb | no dev mapper mounts at all on reboot so its still not finding the lvm stuffon the disk I think | 15:04 |
| clarkb | I/O error, dev vdd, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 | 15:05 |
| clarkb | that seems pretty damning :/ | 15:05 |
| clarkb | I'll get max-servers set to 0 as this is unlikely to resolve quickly | 15:05 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Set raxflex max-servers to 0 https://review.opendev.org/c/openstack/project-config/+/932090 | 15:07 |
| corvus | should we just launch a new mirror? or do we want to practice recovery skills? | 15:07 |
| clarkb | now to do that manually. I'll put nl01 in the emergency file and then edit the config manually too | 15:07 |
| clarkb | corvus: I was wondering about that. I think we shouldn't delete the old mirror so that rackspace can investigate and ensure there isn't a systemic issue to address. But on our side spinning up a new mirror is likely a good course of action | 15:08 |
| fungi | so the cinder volume is present and the logical volumes on it get detected at boot but the disk is still reporting i/o errors? | 15:08 |
| mthebeau | Hi, I have an odd low pritority request. The gerrit server and review.opendev.org interprets the text "Candidat" on the first line of a file as an image. Here is an example review: https://review.opendev.org/c/starlingx/vault-armada-app/+/932089 How would I report something like that? | 15:08 |
| fungi | if the problem is just the cinder volume, can't we detach it and attach a new one? that would basically be one of the steps in creating another server regardless | 15:08 |
| clarkb | fungi: I don't think the lvm partition scheme is detected at all | 15:09 |
| clarkb | fungi: we're trying to read sector 0 and failing | 15:09 |
| fungi | okay, but still the cinder volume, sounds like. should be able to detach it and attach a new one, unless the hypervisor host has totally lost its access to the storage network or something | 15:10 |
| clarkb | fungi: oh ya that is another option for recovery | 15:10 |
| fungi | like i said, if you built a replacement server you'd still need to create an attach a cinder volume and so the lvm setup to create the volumes for apache and openafs caches regardless | 15:11 |
| clarkb | mthebeau: I suspect that string corresponds to the image/x-quicktime format and whatever file type detection library gerrit is using interprets it that way. The best place to report that is probably in the gerrit bug tracker | 15:11 |
| clarkb | mthebeau: https://issues.gerritcodereview.com/issues?q=status:open | 15:11 |
| clarkb | one of us can probably file the issue as well. It helps you've got the example change up that we can refer to. Do it happen if the file contains additional text after the string starts that way? | 15:12 |
| corvus | swapping the volume sounds good to me; clarkb are you able to do that? | 15:13 |
| clarkb | corvus: yes I should be able to figure that out this morning | 15:13 |
| mthebeau | Thanks clarkb | 15:13 |
| clarkb | as far as process for that goes I should probably remove the existing mounts from fstab, detach the existing cinder volume, attach a new cinder volume then go through the lvm and fs creation process then readd the mounts and maybe reboot for good measure? | 15:14 |
| clarkb | and i guess we need to stop apache and openafs so they stop writing to the root fs? | 15:14 |
| corvus | isn't the root fs fine? | 15:15 |
| clarkb | corvus: it is but we don't need it to fill up with content that will be masked by the new mounts | 15:15 |
| corvus | oh they're writing to mountpoints | 15:15 |
| corvus | yeah; probably worth remembering to do a rm on those before we cover them | 15:15 |
| clarkb | looks like /var/cache/apache2 is empty but /var/cache/openafs is not | 15:16 |
| clarkb | I'll put this node in the emergency file list so that we don't accidentally restart things | 15:16 |
| clarkb | node is in the emergency file and openafs-client and apache2 are stopped | 15:20 |
| fungi | that process sounds right. you may also need ansible re-run to recreate subdirectories under those volumes, i don't recall | 15:21 |
| clarkb | the old fstab entries are commented out now to go inspect what oepnstack sees | 15:23 |
| clarkb | dfcb0b9c-cef6-45e6-aab5-c23b4e26659b is the current volume and cinder reports it as being attached as vdd (just to confirm what we've surmised so far) | 15:26 |
| opendevreview | Merged openstack/project-config master: Set raxflex max-servers to 0 https://review.opendev.org/c/openstack/project-config/+/932090 | 15:27 |
| corvus | fungi: clarkb any chance you could review https://review.opendev.org/931996 and https://review.opendev.org/932087 real quick? | 15:30 |
| clarkb | sure | 15:30 |
| mthebeau | clarkb, I created the issue here: https://issues.gerritcodereview.com/issues/372628496 It seems that the only required text is "Candidat" at the beginning of the first line. The original review where it was noticed was here: https://review.opendev.org/c/starlingx/election/+/931990 | 15:30 |
| corvus | i can help with the recovery while those are working through the system :) | 15:30 |
| mthebeau | More or less text after that "Candidat" is not impactful | 15:31 |
| clarkb | mthebeau: thanks. I suspect that the detection will have to use additional heuristics to make a best effort (like checking file extension and the prefix and if they disagree take the conservative approach, or scanning for binary data after the magic number) | 15:32 |
| clarkb | I have detached the old volume from the mirror node (I was owrried this would not work but it did and /dev/vdd is gone) | 15:34 |
| clarkb | new volume is attached and /dev/vdd is back | 15:35 |
| clarkb | now I'm going to need a few minutes to go refresh our block device management setup | 15:36 |
| clarkb | we have a mirror_volumes.sh script in system-config which I'll use once I figure out the correct invocation | 15:37 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/launch/src/opendev_launch/mirror_volumes.sh it has a nice example in the help output. Perfect and thank you to whoever added that | 15:38 |
| clarkb | tonyb I suspect | 15:38 |
| clarkb | the script doesn't appear to mount -a so I can run it and then clean up /var/cache/openafs | 15:39 |
| clarkb | corvus: with openafs-client stopped it should be safe to just completely rm /var/cache/openafs right? I mean it says cache in the path :) | 15:40 |
| corvus | yep | 15:40 |
| clarkb | cool I'm just going to delete the two cache dirs since the script recreates the mountpoint if they are not precreated | 15:41 |
| clarkb | script is done running, cache mount points were created and fstab was updated | 15:45 |
| clarkb | lvs shows data now | 15:45 |
| clarkb | I'm going to mount -a then start apache2 and openafs-client again | 15:45 |
| clarkb | mount: (hint) your fstab has been modified, but systemd still uses the old version; use 'systemctl daemon-reload' to reload. | 15:46 |
| clarkb | its been a while since I found a systemd behavior particularly problematic. | 15:46 |
| clarkb | reset the timer | 15:46 |
| clarkb | it did mount things, but I'll do the daemon reload and rerun to make sure it is happy | 15:47 |
| clarkb | starting openafs client is slow... | 15:48 |
| clarkb | https://mirror.sjc3.raxflex.opendev.org/ looks good. Do we want to reboot the server for good measure before reseting max-servers back to 32? | 15:50 |
| clarkb | probably a good idea to catch any errors I may have introduced with boot time startup of all this stuff | 15:50 |
| clarkb | I'm going to grab something to drink and if no one objects by the time that is done I'll rebioot | 15:51 |
| clarkb | rebooting nwo | 15:52 |
| clarkb | server is back and https://mirror.sjc3.raxflex.opendev.org/ still looks good | 15:55 |
| clarkb | any objections to putting that cloud region back into service? | 15:55 |
| corvus | clarkb: lgtm | 15:56 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add some documentation about the mirror volume config script https://review.opendev.org/c/opendev/system-config/+/932092 | 15:57 |
| corvus | clarkb: ^ i didn't find *any* reference to that mirror volume config script in our docs. that isn't nearly enough information (because i don't actually know what to write), but that's at least a start | 15:57 |
| clarkb | ++ I'll take a look once I'm caught up elsewhere | 15:58 |
| corvus | it would probably be good for someone who has actually set up a mirror to update the docs to explain how it's done :) | 15:58 |
| clarkb | so nl01 has max-servers set to 32. maybe I got the node wrong in the emergecny file? | 15:58 |
| clarkb | in any case I think this is already in service :/ and I just need to revert the chagne to set it to 0 then after the revert lands remove emergency file entries | 15:58 |
| corvus | looks right to me; maybe some racing | 15:59 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Revert "Set raxflex max-servers to 0" https://review.opendev.org/c/openstack/project-config/+/932093 | 15:59 |
| * clarkb writes down a todo list so that updating docs and emailing rackspace with qusetions about the old volume don't get forgotten | 16:00 | |
| opendevreview | James E. Blair proposed opendev/system-config master: Install clouds.yaml on zuul-launcher https://review.opendev.org/c/opendev/system-config/+/932087 | 16:06 |
| corvus | clarkb: i missed a small thing on that change ^ | 16:06 |
| clarkb | corvus: looking I'm going to dobule check that group isn't used for deploying a whole nodepool or something ( Idon't think it is0 | 16:07 |
| clarkb | not finding evidence of host selection in plays based on that. Must only be used for var access? | 16:08 |
| clarkb | +2 from me based on ^ | 16:09 |
| corvus | yeah, that was the conclusion i came to as well, so i decided that was the cleanest way to do that | 16:10 |
| corvus | the launchers and builders have their own groups used for host targeting in playbooks | 16:11 |
| clarkb | corvus: for the mirror docs update the two main pieces of info that come to mind are the size of the volume (200GB minimum with 100gb each for apache and openafs) and then the command invocation from the script help is probably worht including too. I assume you'd prefer me to make that update but let me know if you'd like to keep pushing it along | 16:15 |
| corvus | clarkb: yeah, i think if we add that it should be fairly complete; i say you go for it :) | 16:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add some documentation about the mirror volume config script https://review.opendev.org/c/opendev/system-config/+/932092 | 16:24 |
| clarkb | done | 16:24 |
| clarkb | corvus: fungi can you review https://review.opendev.org/c/openstack/project-config/+/932093 so that I can undo the emergency file entries? | 16:25 |
| fungi | yay! i can now that my internet's back on | 16:29 |
| fungi | approved | 16:29 |
| clarkb | thank you | 16:30 |
| clarkb | I'm going to work on an email now | 16:30 |
| corvus | huh, i thought i did, but it turns out i just clicked the link to open it and then the browser sat there... | 16:31 |
| fungi | and thanks for fixing the raxflex mirror, sorry i was inaccessible | 16:35 |
| opendevreview | Merged openstack/project-config master: Revert "Set raxflex max-servers to 0" https://review.opendev.org/c/openstack/project-config/+/932093 | 16:39 |
| opendevreview | Merged opendev/project-config master: Add image build pipelines https://review.opendev.org/c/opendev/project-config/+/932000 | 16:40 |
| corvus | i made 2 pipelines there... i think we could combine them. but for now, we might want to keep them separate, just to have a little more control. | 16:41 |
| clarkb | email sent | 16:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup python 3.10 bullseye images https://review.opendev.org/c/opendev/system-config/+/932102 | 16:51 |
| clarkb | finally we can drop the bullseye image builds \o/ | 16:52 |
| opendevreview | Merged opendev/system-config master: Configure zuul-launcher to use its logging config file https://review.opendev.org/c/opendev/system-config/+/931996 | 16:53 |
| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Add raxflex provider and debian-bullseye image https://review.opendev.org/c/opendev/zuul-jobs/+/932103 | 16:53 |
| corvus | okay that's the actual fun change ^ (feel free to review now, but i'll approve once all the pre-reqs are confirmed in place) | 16:53 |
| clarkb | corvus: is the name overlap with what nodepool is building going to cause su to potentially boot the new niz label using a nodepool image or nodepool boot the nodepool label with a zuul image? | 16:56 |
| clarkb | also should we use the same flavor for better comparisons between the two? | 16:56 |
| clarkb | gp.0.4.8 is what we use on the nodepool side for the flavor | 16:57 |
| corvus | clarkb: both systems use the external image ids they store for deciding what to boot, so no problem there | 16:57 |
| corvus | i thought i did set it to use gp.0.4.8? | 16:57 |
| clarkb | corvus: oh there is an extra level of indication | 16:58 |
| clarkb | s/indication/indirection/ | 16:58 |
| clarkb | normal == gp.0.4.8 | 16:58 |
| clarkb | I'm not sure I understand why that is ncessary if the flavor only maps a name to a flavor-name. Are there additional attributes we expect the flavor objects to carry? | 16:59 |
| corvus | yep, that way we can map the different flavors on different clouds to one zuul flavor | 16:59 |
| clarkb | ah I see | 17:00 |
| corvus | (in nodepool we have to do that for every cloud plus every label, so it's n(cloud*label) mappings. in zuul it will be n(cloud) mappings) | 17:00 |
| clarkb | its the sort of thing that would be intuitive with multiple clouds from the start but less so with one. Anyway makes sense to me | 17:06 |
| fungi | sounds like we should also take raxflex back out of nodepool temporarily in the near future and recreate our tenant networks there (which i guess will also mean reattaching the mirror server, so probably a change of ip address?) | 17:06 |
| clarkb | fungi: it uses a floating ip so shouldn't affect that. Only the private ip would change and thats not a big deal | 17:07 |
| fungi | ah, yeah i guess we should be able to retain and move that fip | 17:07 |
| clarkb | once the hourly nodepool job ends I'll clean up the emergency file (this way I avoid any unexpected race conditions between that job and things merging earlier | 17:08 |
| clarkb | emergency file is updated. I note that jvb01 is in there with a note from tonyb about upgrades. I think the upgrades occurred and we might be able to clean that up now? | 17:14 |
| clarkb | ya only jvb02 is in inventory so its a noop | 17:14 |
| clarkb | wow I'm just noticing now the level of spam in the gerrit issue tracker... | 17:19 |
| corvus | (fyi i made a fix to the less-than-ideal config error zuul returned on that change: https://review.opendev.org/932106 ) | 17:20 |
| corvus | i'm starting to use the add-filter magnifying-glass icon thingy on the status page a bit more. as long as there's something in gate, i'm finding it pretty easy to just use that to add a filter for system-config. if there isn't something in gate, it's a bit harder. | 17:22 |
| opendevreview | Merged opendev/system-config master: Install clouds.yaml on zuul-launcher https://review.opendev.org/c/opendev/system-config/+/932087 | 17:37 |
| corvus | i think we need to restart the schedulers, web, and launcher to see the new connection, so i'm going to do that now | 18:24 |
| corvus | okay, amusing error due to mixture of speculative / non-speculative there | 18:37 |
| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Add raxflex provider and debian-bullseye image https://review.opendev.org/c/opendev/zuul-jobs/+/932103 | 18:38 |
| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Enable the debian-bullseye image build job https://review.opendev.org/c/opendev/zuul-jobs/+/932119 | 18:38 |
| corvus | that may need to be a two-step process for now; we might want to see if we can change that in zuul | 18:38 |
| clarkb | the restart is necessary for config validation since the scedhulers do that right? | 18:49 |
| corvus | clarkb: yes, but also, they generate the config that the launchers use, so even a force-merge wouldn't work | 19:02 |
| corvus | we should probably put some file matchers on those image build jobs :) | 19:21 |
| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Add file matchers to image build jobs https://review.opendev.org/c/opendev/zuul-jobs/+/932132 | 19:25 |
| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Add file matchers to image build jobs https://review.opendev.org/c/opendev/zuul-jobs/+/932132 | 19:29 |
| corvus | clarkb fungi ^ that might speed things up if you have a sec :) | 19:57 |
| fungi | lookin' | 19:58 |
| fungi | lgtm | 19:59 |
| fungi | i think one reviewer is enough for that, it's trivial and isn't going to break anything for now even if it's missing something, we just need to remember to expand the list of file patterns if we add relevant bits outside those directories | 20:00 |
| clarkb | sorry lunch went long | 20:16 |
| opendevreview | Merged opendev/zuul-jobs master: Add raxflex provider and debian-bullseye image https://review.opendev.org/c/opendev/zuul-jobs/+/932103 | 20:21 |
| opendevreview | Merged opendev/zuul-jobs master: Add file matchers to image build jobs https://review.opendev.org/c/opendev/zuul-jobs/+/932132 | 20:21 |
| corvus | 2024-10-10 20:21:49,461 ERROR zuul.Launcher: keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url | 20:22 |
| corvus | hrm. that's a little confusing. | 20:23 |
| corvus | on account of it is in the file | 20:23 |
| corvus | oh derp | 20:28 |
| opendevreview | James E. Blair proposed opendev/system-config master: Fix raxflex connection entry for zuul-launcher https://review.opendev.org/c/opendev/system-config/+/932147 | 20:30 |
| corvus | clarkb fungi ^ that fixes an oops. i'm going to make that edit on the launcher to keep moving. | 20:31 |
| clarkb | approved | 20:33 |
| clarkb | arg restarted firefox to pick up some updates and didn't realize I had a second window open so it saved the wrong set of tabs on exit | 20:35 |
| clarkb | good way to clear house on open tabs I guess | 20:35 |
| opendevreview | James E. Blair proposed opendev/zuul-jobs master: Revert "Add raxflex provider and debian-bullseye image" https://review.opendev.org/c/opendev/zuul-jobs/+/932148 | 20:46 |
| corvus | okay, i've run into a zuul bug that needs fixing; i think we should yank ^ until it's fixed to avoid looping on builds | 20:46 |
| corvus | (good news! image build job is running in the image build pipeline! :) | 20:46 |
| gouthamr | o/ need help to check if shubham.kumar.yadav369<at>gmail<dot>com has been subscribed to the openstack-discuss list? they have questions on the list, but, unfortunately doesn't respond to the list .. :( | 20:47 |
| clarkb | I guess mm3 doesn't tell you who the list admins are anymore? | 20:52 |
| clarkb | gouthamr: fungi and a couple of others have been managing that list but I don't recall who they all are at this point | 20:52 |
| gouthamr | ah; no i couldn't tell on the interface clarkb | 20:53 |
| clarkb | corvus: +2 from me I'll let you decide on if it needs a fast approval | 20:54 |
| clarkb | gouthamr: the only email I see on the thread from them did go to the list. So if anyone responded I think they responded off list | 20:58 |
| clarkb | which would then perpetuate via reply buttons in clients. | 20:58 |
| clarkb | but it is possible that fungi or someone else moderated through the original email earlier today | 20:58 |
| clarkb | gouthamr: X-Mailman-Rule-Hits: nonmember-moderation <- from the headers on the email that did make it to the list. I think that does indicate they are not a list member | 20:59 |
| gouthamr | clarkb: ah! their initial email goes to the list, but then they start replying off-list.. so i thought i'd drop them from the CC if they're a list subscriber to prevent it :) | 21:00 |
| gouthamr | thanks for sharing that tip! i'll watch for that next time | 21:00 |
| corvus | https://review.opendev.org/932150 is the zuul change needed to continue launcher work (that should explain the issue we ran into) | 21:47 |
| corvus | 5 lines to fix, 195 to test | 21:47 |
| opendevreview | Merged opendev/system-config master: Fix raxflex connection entry for zuul-launcher https://review.opendev.org/c/opendev/system-config/+/932147 | 22:50 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: Proposed new Ironic core structure https://review.opendev.org/c/openstack/project-config/+/931991 | 22:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!