Wednesday, 2023-05-03

opendevreviewAshutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX
*** dmellado0 is now known as dmellado05:03
jrosseris codesearch unhappy? gives me `Hound is not ready.`05:33
ianwjrosser: you might have caught it on a daily reindex; should be back now06:09
ianwwe reload it daily if there's new projects (c.f.
jrosserianw: ah yes, it’s back now.06:21
*** amoralej|off is now known as amoralej07:17
*** jpena|off is now known as jpena07:40
noonedeadpunkfwiw after gitea is really fast07:54
*** amoralej is now known as amoralej|lunch11:03
opendevreviewMerged opendev/system-config master: tools/
*** amoralej|lunch is now known as amoralej12:13
fungiianw: based on i feel like maybe we should just stop relying on ansible-lint, the maintainer seems to feel that intentionally breaking the tool and its dependencies is a good development practice, so we're probably better off going without it13:57
fungior an alternate way of reading it is that they think everyone should pin every last transitive dependency, which is fairly incompatible with our workflows13:59
fungiafter all, this is python, not javascript14:00
*** dviroel_ is now known as dviroel14:28
clarkbnoonedeadpunk: thank you for confirming it looks better to you as well. I did some checking myself and I suspect that this was the underlying issue15:10
clarkbFedora email sent15:13
clarkbfungi: I think if you have time to review for the gitea upgrade today may be a good day for that. That said this still remains non urgent and we can punt it a bit15:23
fungii agree today seems good. planning to pick up about a quarter cubic meter of sand in a few minutes and do a bit of seawall repair, but will be around most of the afternoon after that15:31
clarkbfungi: cool I gess feel free to do review when you can (and don't forget to check the held node) and we can merge it when we are able to monitor 15:32
fungiyeah, i'm going through it now15:32
fungiseems like the highlights are new nodejs version, dropping auth from the org logo task, disabling the new actions feature, and merging some template updates15:33
clarkbyup that sounds about right. I also drop the no_log on the org logo task which is maybe my biggest concern now. It should be safe because we don't auth so anything that is exposed is already public15:35
clarkbbut double check that15:35
fungiheh, just noticed one of our screenshots has a typo in its name... gitea-org-expore.png15:35
opendevreviewJeremy Stanley proposed opendev/system-config master: Fix mistyped gitea screenshot image name
fungiscreenshots indicate org logos are still working as designed and the actions tab isn't appearing15:37
fungiclarkb: is that task which no longer does no_log and it lgtm15:44
fungialso doesn't include the user and password parameters to the uri call, as intended15:45
clarkbthank youfor checking15:45
fungiare we clear for me to approve the change then?15:45
clarkbI think so. Did you interact with the held node at all?15:46
clarkbI think that would be the only other thing to double check. Just poke around to make sure nothign pops up as a problem15:46
fungiyeah, i poked around some15:46
fungiapproved 877541, i should be able to go buy some bags of sand and get back before it's time to deploy15:48
clarkbI'm around all morning too15:59
clarkband afternoon as well15:59
JayFA blogpost I wrote a while ago about OpenStack testing metrices just got published -> -- consider it a note of appreciation for a lot of the work you all do16:03
clarkbJayF: nice I'll have to take a look16:04
fungiokay, back from the hardware store. i discovered the hard way that the maximum capacity on those flat-bed carts is ~16 bags of sand. at 20 it wouldn't move16:35
fungithanks for the heads up on the article JayF!16:35
corvusJayF: ++16:36
corvusfungi: you have to *buy* sand?16:37
corvusi'm going to restart the zuul web processes16:37
fungicorvus: yes, it's mind-bogglingly sad that, living on a sandbar, i still have to source sand commercially if i want to legally fill holes on my property16:37
fungii feel weird enough paying for dirt and rocks, but at least there isn't much of that here normally so it sort of makes sense16:38
corvusfungi: i'm assuming currents erode the sand from your property and deposit them at the location where the sand is collected and put in bags and shipped to the local hardware store.16:39
JayFYou just have to firmly, but politely, ask the ocean to stop stealing your property. If it's extra stubborn, you might have to show it your deed.  /s16:40
fungicorvus: pretty much. though in this specific case it moved the sand from directly behind my seawall and deposited it farther into my yard in a manner that's hard to pile back up16:41
fungiso i figure if i keep dumping sand behind the seawall, eventually my yard will get taller?16:42
fungiif nothing else, i consider it a hard-learned lesson in hydraulic systems16:43
corvus#status log restarted zuul web components16:43
opendevstatuscorvus: finished logging16:43
fungithanks corvus!@16:44
opendevmeetfungi: Error: "!!1!11eleven" is not a valid command.16:44
fungieven opendevmeet is getting in on the fun16:45
*** amoralej is now known as amoralej|off16:45
clarkbfungi: your screenshot typo fix failed on I'm suddenly concerned today is going to become debug docker-compose day16:47
clarkbI suspect the gitea upgrade change failed on the same thing16:47
clarkbI'm going to take a break before I dive down that rabbit hole16:50
clarkbbut before I do apparently new docker-compose is all golang and not python (so pip install gets you old stuff which I guess is starting to go stale wee)16:52
clarkbI suspect our options are either 1) switch back to distro docker-compose or 2) install golang docker-compose however the recommend you do that16:53
fungithat is a rather... opaque error16:55
fungii guess requests changed?16:56
clarkbwhich is also rather surprising16:56
fungier, wait that's urllib3 not stdlib's urllib16:56
fungilast new version was several days ago16:57
fungion sunday16:57
fungiwhich was a point release for a major v2 release earlier last week16:58
fungithat is not a short changelog16:58
funginew requests 2.30.0 an hour ago16:59
clarkbbut also I just pushed updated gitea 1.19.2 changes which passed16:59
clarkboh maybe that is the trigger then. We were using older requests which had older urllib3?16:59
fungi"Dependencies - ⚠️ Added support for urllib3 2.0."16:59
clarkbok so our options are 1) go back to distro docker-compose 2) install golang docker-compose 3) pin urllib3/requests17:00
fungiso yeah, maybe a recheck will "just work"17:00
clarkboh I'm thinking new requests is what broke us since urllib3 has been around long enough to have noticed earlier? But ya lets do a recheck and see17:00
clarkbfwiw I think we can do 1) and probably should at this point since everything using stop_grace_period is on focal or newer and that has docker-compose that supports the newer feature17:01
clarkbthe downside to this is cleaning things up will be weird since pip installs to /usr/local/bin which is earlier in path than /usr/bin which is where the distro package will put things17:01
clarkbI'm going to propose a revert of the pip docker-compose change and we can use that for testing at least17:02
clarkboh what isn't clear to me is if we can use an older docker-compose against services running under newer docker-compose17:04
clarkbdoes that mean the migration would be something like docker-compose down using newer docker-compose from pip, then remove pip docker-compose then docker-compose up with old docker compose?17:05
clarkbThat is going to be "fun"17:05
fungiyeah, i think it's new requests/urllib3 which is the problem. the failed build installed requests-2.30.0-py3-none-any.whl with urllib3-2.0.1-py3-none-any.whl
clarkboh I just rechecked it17:06
clarkbI guess we can expect that to fail then17:07
fungilikely so, but an extra data point doesn't hurt17:07
fungithis is also likely to hit other projects, i have a feeling, we just happened to be the first to stub our toes on it17:08
clarkbok I think that downgrading docker-compose isn't necessarily safe (ugh)17:09
fungihere's a just-opened issue related to the update, though i don't think it's what we're hitting:
clarkbI'm going to push a change up that installs docker-compose from both system packages and distro packages and then see what jobs hat runs. Then maybe hold some of those test nodes to test downgrading since that is one options we've got17:10
clarkbHowever, I'm beginning to think the best option may be to roll forward since I expect that is safe17:10
clarkbrolling forward means figuring out how to install docker-compose the modern way however thati s done now17:10
clarkbok jammy docker-compose is the same as latest pip docker-compose17:11
clarkbso this is only an issue for everything pre jammy (which is most stuff but still)17:11
opendevreviewClark Boylan proposed opendev/system-config master: Install docker-compose from the distro too
opendevreviewClark Boylan proposed opendev/system-config master: Install docker-compose from the distro too
clarkbpicked four services to force failures on and hold nodes for17:19
clarkb for grabbing newer docker-compose17:20
*** jpena is now known as jpena|off17:21
fungithe bug bounty approach on urllib3's issues is interesting:
clarkbholds in place for those four services. Now to make changes installing docker-compose the modern way and another to pin python-requests17:23
clarkbwe will have many options17:23
opendevreviewClark Boylan proposed opendev/system-config master: Pin python requests when installing docker-compose
clarkbwow so the new docker compose doesn't even really replace old docker compose17:41
clarkbswitching is a ton of work. I feel a little better about htis now knowing its a mountain of work no matter what we do :)17:41
*** Guest83 is now known as diablo_rojo17:41
fungidoesn't replace... as in has entirely new syntax/configuration?17:43
opendevreviewClark Boylan proposed opendev/system-config master: Install modern docker-compose from github releases
clarkbfungi: the config files are shared but not sure about the cli stuff other than the base command changes.17:49
clarkb882173 will give us more insight17:49
clarkbpinning python requests seems to be owrking17:50
clarkbbut as noted in the commit message that is probably only good as a short term woarkaround. We should plan for something else going forward17:50
opendevreviewClark Boylan proposed opendev/system-config master: Pin python requests when installing docker-compose
opendevreviewClark Boylan proposed opendev/system-config master: Install docker-compose from the distro too
clarkbI think 882171 is going to work (I had to fix an issue with the grafana job that appears unrelated we'll see if this latest patchset passes)18:10
clarkbThen I rebased 882169 on top of it because it was failing completely due to theissue we are trying to work around due to path precedence. I want things to deploy then fail in testinfra so that we can ssh in and see if downgrading to older docker-compose will even work18:11
fungimakes sense, yep18:14
clarkbI'm beginning to think that what we will probbaly end up wanting to do is merge 882171 then switch to the modern compose plugin. That too will need testing for the upgrade path as I'm not sure what happens if we docker-compse up -d then docker compose down18:16
clarkbwhich is basically what our automated upgrade path would be if we take no special action18:16
clarkbAnd I'm suspecting that simply because I expect docker-compose -> docker compose to be more of a seamless transition than docker-compose 1.29.2 -> docker-compose 1.25.018:16
clarkbbut we'll test it all I only have hunches at this point18:17
clarkbthe naive switch to docker compose seems to have worked for etherpad at least. I only updated a few of the roles because I wanted to see if it would work18:17
clarkbbut none of that is testing the docker-compose to docker compose transition yet18:18
fungiyeah, specifically concerned with a container that was upped with old docker-compose not downing correctly with new docker-compose?18:34
fungihopefully docker is handling the state maintenance in which case it would be fine18:35
fungibut i really have only a tiny grasp of all of this18:35
clarkbyes exactly. Because our current running state for services is things are up using docker-compose 1.29.2 (actually need to audit this). If we change out for docker-compose from the distro or docker compose the plugin what will happen is we update the tool  then run down and up -d using the new tool18:36
clarkbI suspect that docker-compose 1.29.2 -> docker compose the plugin will be handled more gracefully than docker-compose 1.29.2 -> docker 1.25.0 but testing should happen for all of this18:38
clarkbalso we may not use down and up -d everywhere? we probably need to audit for stop and start in some places too :/18:38
clarkbthe other idea I had just now is maybe as part of 882171 we want to move docker-compose into a virtualenv18:40
clarkbthat isolates it a bit more and makes the workaround less hacky18:40
clarkbthen we can take our time sorting out the better long term option18:40
clarkbI'm about to take a break for lunch and may even go for a walk or something as activity like that tends to be good for sorting through these problems18:41
fungiyeah, i'm pondering it while i unload and spread sand18:46
clarkbon 882169's held paste node I susccessfully added a paste, ran /usr/bin/docker-compose down, confirmed containers stopped with docker ps -a, then ran /usr/bin/docker-compose up -d and checked my paste was still there and made a new paste19:56
clarkbthat gives me some hope this won't be totally broken if we switch docker-compose versions like this19:56
clarkbthat held node had a 1.29.2 /usr/local/bin/docker-compose default docker-compose and a 1.25.0 /usr/bin/docker-compose from the distro19:57
fungiyeah, seems promising20:00
clarkblooking at I'll also want to test one or all of zuul-preview/meetpad/insecure-ci-registry20:01
clarkbI don't think any of the nodes I held on the first pass are bionic though. I'm going to run through the other three I did hold some of which are jammy which is 1.29.2 for both install methods and should be totally fine and others focal whihc will help confirm the paste results20:02
clarkbI do think that and then possibly following that up with a move to a virtualenv are reaosnable short term workarounds. Not great but willing to compromise there20:03
clarkbheld etherpad shows this transition working on jammy too as expected20:06
clarkbthe zk node is on focal and harder to verify as a human but four letter commands work after doing a down then up with the older 1.25.0 docker-compose20:09
clarkbNow to check gerrit and then push a new change to add more failures to bionic nodes20:09
clarkbthere was maybe some slowness on the gerrit node when running up -d. Could just be the node though20:12
clarkbit looks like downgrading docker-compose to the focal version is safe. Jammy versions is the same and works as well (as it should)20:13
opendevreviewClark Boylan proposed opendev/system-config master: Install docker-compose from the distro too
clarkbok holds in place to check those older services on that ps20:17
opendevreviewClark Boylan proposed opendev/system-config master: Install modern docker-compose from github releases
clarkbI think moving to modern docker-compose is going to be more of a challenge. In particular some of the output strings have changed whihc we use to detect thing like whether or not new images are pulled20:32
clarkbdoable I think just more effort20:32
opendevreviewClark Boylan proposed opendev/system-config master: Only install docker-compose from the distro
ianwfungi: yeah i guess with was a semver major release ... why not just do a "if type(str)" though and not break every other install.  the results of it are useful, but it's certainly hard to work with ... if you uncap ansible-lint we've had many regressions on new releases.  but if we cap it, now we're also broken by other libraries in the ecosystem...20:50
opendevreviewClark Boylan proposed opendev/system-config master: Rebuild gitea images
opendevreviewClark Boylan proposed opendev/system-config master: Add more docker-compose installation comments
opendevreviewClark Boylan proposed opendev/system-config master: Fix the testinfra hostname for the registry node
clarkbinfra-root going from docker-compose 1.29.2 running containers and trying to down them with docker-compose 1.17.1 on bionic fails.21:23
clarkbI think this is because the container names change21:23
clarkbI was able to down them with 1.29.2 then up them with 1.17.121:23
clarkb1.29.2 is able to down the 1.17.1 up'd containers. Which gives weight to the idea that rolling forward is generally safe but rolling back isn't21:24
clarkbidea time: we're landing the requests pin as a temporary workaround. We follow up with 882169 after removing the forced failures. This will install the distro package alongside pip installed docker-compose but distro docker-compose won't actually do anything at this point21:28
fungiwe can presumably lift that once we're off bionic?21:29
clarkboh thats a good point its only like 3 services on the old thing21:29
clarkbso maybe order of operations should be 1) workaround with requests pin 2) finish upgrades for those services 3) land 882169 4) remove pip installed docker-compose 5) automatic conversion by ansible things21:29
clarkbThen a very long term 6) move things to docker compose plugin21:30
opendevreviewClark Boylan proposed opendev/system-config master: Update gitea to 1.19.3
opendevreviewClark Boylan proposed opendev/system-config master: DNM intentional gitea failure to hold a node
fungisounds good to me, yep21:59
opendevreviewClark Boylan proposed opendev/system-config master: Install docker-compose from the distro
clarkbI used my held focal paste node to test docker-compose 1.25.0 up'd containers getting down'd by `docker compose` installed via apt-get install docker-compose-plugin from the docker deb repo. This seems to work22:17
clarkbthe format of things chagnes which is potentially awkward for transitioning stuff though22:17
clarkbbut in generaly seems to be functional despite containers changing names22:17
clarkbinterestingly it seems we can downgrade as well22:18
clarkbfrom `docker compose` up'd containers to docker-compose 1.25.0 down command22:18
clarkbso ya I think this gets significantly better if we get up to focal or newer22:19
clarkbThe changes I have pushed should reflect steps 1) 3) 4) and 5) of the above list. Steps 2 and 6 will need separate efforts22:23
clarkbafter all this trouble:
clarkbthey yanked the release22:37
fungifunny not funny22:47
corvusi think we should declare a work holiday whenever there's a requests release.  it would probably be more productive for everyone in the long run.23:07
fungiwhere are you headed with that huge backpack? oh, there was a requests release23:17
opendevreviewMerged opendev/system-config master: Pin python requests when installing docker-compose
opendevreviewMerged opendev/system-config master: Rebuild gitea images
opendevreviewMerged opendev/system-config master: Add more docker-compose installation comments

Generated by 2.17.3 by Marius Gedminas - find it at!