Monday, 2022-10-24

opendevreviewIan Wienand proposed opendev/system-config master: launch-node : make into a small package
*** rlandy is now known as rlandy|out00:26
opendevreviewMerged opendev/system-config master: Abstract name of bastion host for testing path
opendevreviewIan Wienand proposed opendev/system-config master: launch-node : make into a small package
opendevreviewIan Wienand proposed opendev/system-config master: launch-node : make into a small package
opendevreviewMerged opendev/system-config master: bootstrap-bridge: use abstracted hostname
*** Guest0 is now known as osmanlicilegi04:11
opendevreviewMerged opendev/system-config master: Convert production playbooks to bastion host group
ianw^ infra-prod-bootstrap-bridge still working after that (
ianwtomorrow i'll start a new bridge node and i think we're in a position to transition05:11
*** marios is now known as marios|ruck05:13
opendevreviewMerged opendev/system-config master: Run a base test against "old" bridge
*** jpena|off is now known as jpena07:39
*** rlandy|out is now known as rlandy10:34
*** dviroel|out is now known as dviroel11:46
*** rlandy is now known as rlandy|brb12:38
*** rlandy|brb is now known as rlandy12:51
*** dasm|off is now known as dasm13:28
*** dasm is now known as dasm|rover13:28
*** rcastillo_ is now known as rcastillo13:30
fungi#status log Pruned backups on, reducing volume utilization by 36% (from 93% to 57%)13:32
opendevstatusfungi: finished logging13:32
clarkbthank you15:09
fungiclarkb: not urgent, but when you have a chance i believe you had some feedback in irc for which hasn't made it into review comments15:13
clarkbhrm I thought I posted them. Maybe I did it on the ACL change only15:14
clarkbya I think one of the comments from the acls applies here so I've posted that. Otherwise I Think this is good15:16
fungiawesome, thanks15:19
*** dviroel is now known as dviroel|lunch15:21
clarkbre the python image build stuff. Someone left a comment on my pip bug that they do similar but instead install everything using --user and then copy $USER/.local to the prod iamge. This hsa the benefit of not needing to copy the wheel files too15:22
clarkbI havne't been able to come up with a good reason why that wouldn't work. But it is a much bigger/different change for us. In particular we expect things to install globally so would hae to update an image at a time if we do that15:23
clarkbinfra-root is a node cleanup change for gitea-lb01 and jvb02. Once they are out of the inventory we can ask the clouds to delete them15:46
*** marios|ruck is now known as marios|out15:49
fricklerseems pypi is going bonkers once again. devstack gate fails like e.g. and also in ironic16:10
frickleractually the whole buildset pretty much exploded
JayFthe output was reading like a new pip release that turned on be_even_more_breaky=true or something lol16:11
clarkboof. That does indeed look like the stale fallbck index data16:11
clarkbJayF: if this is the issue we know about its on the server side. Pypi is fronted with a CDN. If that CDN fails to fetch necessary info from the backend it fallback to a backup backend. The problem is the backup backend isn't kept up to date and you get indexes from months ago16:12
JayFah, that makes sense16:13
clarkbOpenStack in particular is affected because constraints (for master) are kept up to date with recent releases16:13
clarkbI think it would be great if someone from openstack would file a bug arguing that it is better to fail upfront than serve stale data. The reason is who knows how many security bugs you are installing transparently if you don't use constraints or a similar mechanism16:14
clarkbBut I don't have the energy to argue with them that a fundamental piece of their architecture is falwed16:14
JayFI might know the right ears to bend about that.16:14
clarkbwhen we first ran into this it took us a fair bit of time to even convince them there was a problem. Once we convinced them there was a problem days later they discovered the backup backed had run out of disk a while back and that was why it was super stale...16:16
clarkbyou can try running an HTTP PURGE (non standard) request against the project indexes that have this issue which is supposed to tell the CDN to repopulate from the backend (part of the problem here is once it caches from the stale backend it will keep serving that bad data aiui)16:18
clarkb`curl -X PURGE` for frickler's example16:19
*** dviroel|lunch is now known as dviroel16:19
fungii got a socket timeout pulling a package from pypi at home a few hours ago16:19
fungiand shows a significant error spike on the graphs16:20
corvusi'm looking into some nodepool stuff and happened to notice that we are getting a lot of "No valid host was found" errors from nova on inmotion16:30
corvusi don't plan to do anything else with that.  just an fyi in case someone wants to look into tuning that cloud some more.16:30
corvus(the graph shows we're pretty consistently topping out at 50% of our expected capacity)16:31
fungithanks, sounds like we probably leaked some instances or something16:35
clarkbthe last time we had that in inmotion the problem was related to the placement service not being able to allocate to full capacity because of leaked record for nodes long since deleted. I don't recall the exact thing16:35
clarkbmelwitt helped with it last time. And iirc it was a known nova/placement issue16:35
clarkb link to that page was purple not blue so probably related16:36
melwitt++ I was about to grab that ^ to link you16:36
clarkbI can probably take a look at that a bit later today16:38
clarkbJust need to work through a bit more of my todo list first :)16:38
corvuscool.  that sounds consistent with what i'm seeing.16:39
clarkbrelated infra-root is a draft email I'm working on after discussions happened at the ptg (and elsewhere) that we/I were not included in. If you think that this email is a good productive step I'd like to send it out today16:39
opendevreviewArtem Goncharov proposed openstack/project-config master: Add post-review pipeline
clarkbgtema: ^ I think that needs to be rebased onto the latest patchset of its parent17:01
opendevreviewArtem Goncharov proposed openstack/project-config master: Add post-review pipeline
gtemathks clarkb, haven't noticed now17:02
*** dviroel is now known as dviroel|biab18:56
*** dviroel|biab is now known as dviroel19:40
clarkbthe meeting agenda has grown quite long. But please add important topics to it and we'll do our best to get through them20:04
opendevreviewIldiko Vancsa proposed openstack/project-config master: Add starlingx-gerritbot config
ianwis there a storyboard container?20:52
clarkbianw: yes but I'm not sure how much exercise it gets20:54
ianwi guess also bugzilla probably isn't an option -- reminded me of 20:55
ianwat least we're not starting from a base of sending patches via email to each other :)20:55
fungipython 3.11.0 was officially tagged an hour or so ago21:10
fungi(compiling now)21:12
clarkbfungi: a good time to remind people to review so that we have room for python3.11 images (though I think you have reviewed them?)21:21
clarkbok infra meeting agenda will go out in about 40 minutes or so. Please add any other updates nowish21:21
fungii have nothing new to add21:28
opendevreviewMerged opendev/grafyaml master: Fix default handling with newer voluptious
opendevreviewMerged opendev/grafyaml master: Update Docker images to python 3.10
ianwclarkb: ^ thanks, i think i reviewed all those21:31
opendevreviewMerged opendev/statusbot master: Update container images to python3.10
ianwi'm planning on getting on with the bridge update today.  i've abstracted everything i can think of relating to that21:33
*** rlandy is now known as rlandy|out21:42
clarkbianw: are there more changes that need reviewing related to that?21:47
ianwnot at the moment, i'm at the point of bringing up the server22:04
*** dasm|rover is now known as dasm22:13
*** dasm is now known as dasm|off22:14
ianwclarkb:  400: Client Error for url: ..., Bad networks format22:56
ianwis that one you saw?22:56
clarkbianw: yes I think so. I used my old venv to work around that iirc22:56
clarkbthough I used a paramiko venv so maybe not22:57
clarkbianw: where are you launching from?22:57
ianwok, on that topic; is a place we could codify that a bit more22:57
clarkbis it the /opt path that the README points too? Ihave/had a change to update that as that location is stale22:57
ianwbastion host22:57
clarkboh wait I think it was corvus  that hit that. I launched in vexxhost beacuse that is where gitea is hosted22:58
ianwbut just command line, i can create an env with a older openstacksdk :/22:58
clarkbI think corvus  has a venv that should work22:58
corvus~root/corvus-venv is what i used22:59
*** dviroel is now known as dviroel|out23:00
opendevreviewIan Wienand proposed opendev/system-config master: launch-node : make into a small package
ianwthanks, that is working.  that venv has 0.41 which i pinned in ^23:03
clarkbre Gerrit 3.6 I've just responded to a thread that indicates conflicts with checking might be broken in 3.623:29
clarkb(somethingto be aware of as we plan to upgrade, hopefully it isn't actually broken)23:29
opendevreviewIan Wienand proposed opendev/ master: Add
opendevreviewIan Wienand proposed opendev/system-config master: Switch testing bridge name to
opendevreviewIan Wienand proposed opendev/system-config master: Add to inventory
opendevreviewIan Wienand proposed opendev/system-config master: Add to inventory
opendevreviewIan Wienand proposed opendev/system-config master: Switch testing bridge name to

Generated by 2.17.3 by Marius Gedminas - find it at!