| opendevreview | Ashutosh Sarode proposed openstack/project-config master: Add Harbor app to StarlingX https://review.opendev.org/c/openstack/project-config/+/881962 | 04:02 |
|---|---|---|
| *** dmellado0 is now known as dmellado | 05:03 | |
| jrosser | is codesearch unhappy? gives me `Hound is not ready.` | 05:33 |
| ianw | jrosser: you might have caught it on a daily reindex; should be back now | 06:09 |
| ianw | we reload it daily if there's new projects (c.f. https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/codesearch/tasks/main.yaml#L74) | 06:10 |
| jrosser | ianw: ah yes, it’s back now. | 06:21 |
| *** amoralej|off is now known as amoralej | 07:17 | |
| *** jpena|off is now known as jpena | 07:40 | |
| noonedeadpunk | fwiw after https://review.opendev.org/c/opendev/system-config/+/882083 gitea is really fast | 07:54 |
| *** amoralej is now known as amoralej|lunch | 11:03 | |
| opendevreview | Merged opendev/system-config master: tools/upstream-wheel-audit.py https://review.opendev.org/c/opendev/system-config/+/879239 | 12:13 |
| *** amoralej|lunch is now known as amoralej | 12:13 | |
| fungi | ianw: based on https://github.com/ansible/ansible-compat/issues/258 i feel like maybe we should just stop relying on ansible-lint, the maintainer seems to feel that intentionally breaking the tool and its dependencies is a good development practice, so we're probably better off going without it | 13:57 |
| fungi | or an alternate way of reading it is that they think everyone should pin every last transitive dependency, which is fairly incompatible with our workflows | 13:59 |
| fungi | after all, this is python, not javascript | 14:00 |
| *** dviroel_ is now known as dviroel | 14:28 | |
| clarkb | noonedeadpunk: thank you for confirming it looks better to you as well. I did some checking myself and I suspect that this was the underlying issue | 15:10 |
| clarkb | Fedora email sent | 15:13 |
| fungi | thanks! | 15:14 |
| clarkb | fungi: I think if you have time to review https://review.opendev.org/c/opendev/system-config/+/877541 for the gitea upgrade today may be a good day for that. That said this still remains non urgent and we can punt it a bit | 15:23 |
| fungi | i agree today seems good. planning to pick up about a quarter cubic meter of sand in a few minutes and do a bit of seawall repair, but will be around most of the afternoon after that | 15:31 |
| clarkb | fungi: cool I gess feel free to do review when you can (and don't forget to check the held node) and we can merge it when we are able to monitor | 15:32 |
| fungi | yeah, i'm going through it now | 15:32 |
| fungi | seems like the highlights are new nodejs version, dropping auth from the org logo task, disabling the new actions feature, and merging some template updates | 15:33 |
| clarkb | yup that sounds about right. I also drop the no_log on the org logo task which is maybe my biggest concern now. It should be safe because we don't auth so anything that is exposed is already public | 15:35 |
| clarkb | but double check that | 15:35 |
| fungi | heh, just noticed one of our screenshots has a typo in its name... gitea-org-expore.png | 15:35 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Fix mistyped gitea screenshot image name https://review.opendev.org/c/opendev/system-config/+/882161 | 15:37 |
| fungi | screenshots indicate org logos are still working as designed and the actions tab isn't appearing | 15:37 |
| fungi | clarkb: https://97af450d4496db857e42-e03b399dba5d60e417a0a61de1589ca2.ssl.cf5.rackcdn.com/877541/9/check/system-config-run-gitea/3f848e5/bridge99.opendev.org/ara-report/results/535.html is that task which no longer does no_log and it lgtm | 15:44 |
| fungi | also doesn't include the user and password parameters to the uri call, as intended | 15:45 |
| clarkb | thank youfor checking | 15:45 |
| fungi | are we clear for me to approve the change then? | 15:45 |
| clarkb | I think so. Did you interact with the held node at all? | 15:46 |
| clarkb | I think that would be the only other thing to double check. Just poke around to make sure nothign pops up as a problem | 15:46 |
| fungi | yeah, i poked around https://158.69.65.228:3081 some | 15:46 |
| fungi | approved 877541, i should be able to go buy some bags of sand and get back before it's time to deploy | 15:48 |
| clarkb | I'm around all morning too | 15:59 |
| clarkb | and afternoon as well | 15:59 |
| JayF | A blogpost I wrote a while ago about OpenStack testing metrices just got published -> https://www.gresearch.co.uk/blog/article/invisible-work-of-openstack-testing-matrices/ -- consider it a note of appreciation for a lot of the work you all do | 16:03 |
| clarkb | JayF: nice I'll have to take a look | 16:04 |
| fungi | okay, back from the hardware store. i discovered the hard way that the maximum capacity on those flat-bed carts is ~16 bags of sand. at 20 it wouldn't move | 16:35 |
| fungi | thanks for the heads up on the article JayF! | 16:35 |
| corvus | JayF: ++ | 16:36 |
| corvus | fungi: you have to *buy* sand? | 16:37 |
| corvus | i'm going to restart the zuul web processes | 16:37 |
| fungi | corvus: yes, it's mind-bogglingly sad that, living on a sandbar, i still have to source sand commercially if i want to legally fill holes on my property | 16:37 |
| fungi | i feel weird enough paying for dirt and rocks, but at least there isn't much of that here normally so it sort of makes sense | 16:38 |
| corvus | fungi: i'm assuming currents erode the sand from your property and deposit them at the location where the sand is collected and put in bags and shipped to the local hardware store. | 16:39 |
| JayF | You just have to firmly, but politely, ask the ocean to stop stealing your property. If it's extra stubborn, you might have to show it your deed. /s | 16:40 |
| fungi | corvus: pretty much. though in this specific case it moved the sand from directly behind my seawall and deposited it farther into my yard in a manner that's hard to pile back up | 16:41 |
| fungi | so i figure if i keep dumping sand behind the seawall, eventually my yard will get taller? | 16:42 |
| fungi | if nothing else, i consider it a hard-learned lesson in hydraulic systems | 16:43 |
| corvus | #status log restarted zuul web components | 16:43 |
| opendevstatus | corvus: finished logging | 16:43 |
| fungi | thanks corvus!@ | 16:44 |
| fungi | s/@// | 16:44 |
| corvus | corvus!@#$@!# | 16:44 |
| fungi | !!!1!11eleven | 16:44 |
| opendevmeet | fungi: Error: "!!1!11eleven" is not a valid command. | 16:44 |
| fungi | even opendevmeet is getting in on the fun | 16:45 |
| *** amoralej is now known as amoralej|off | 16:45 | |
| clarkb | fungi: your screenshot typo fix failed on https://zuul.opendev.org/t/openstack/build/5f17fa02f34248398bc2cae09e1aa97a/log/job-output.txt#21189 I'm suddenly concerned today is going to become debug docker-compose day | 16:47 |
| fungi | argh | 16:47 |
| clarkb | I suspect the gitea upgrade change failed on the same thing | 16:47 |
| clarkb | I'm going to take a break before I dive down that rabbit hole | 16:50 |
| clarkb | but before I do apparently new docker-compose is all golang and not python (so pip install gets you old stuff which I guess is starting to go stale wee) | 16:52 |
| clarkb | I suspect our options are either 1) switch back to distro docker-compose or 2) install golang docker-compose however the recommend you do that | 16:53 |
| fungi | that is a rather... opaque error | 16:55 |
| fungi | i guess requests changed? | 16:56 |
| clarkb | which is also rather surprising | 16:56 |
| fungi | er, wait that's urllib3 not stdlib's urllib | 16:56 |
| fungi | last new version was several days ago | 16:57 |
| fungi | on sunday | 16:57 |
| fungi | which was a point release for a major v2 release earlier last week | 16:58 |
| fungi | https://github.com/urllib3/urllib3/blob/main/CHANGES.rst | 16:58 |
| fungi | that is not a short changelog | 16:58 |
| fungi | new requests 2.30.0 an hour ago | 16:59 |
| clarkb | but also I just pushed updated gitea 1.19.2 changes which passed | 16:59 |
| clarkb | oh maybe that is the trigger then. We were using older requests which had older urllib3? | 16:59 |
| fungi | "Dependencies - ⚠️ Added support for urllib3 2.0." | 16:59 |
| fungi | https://requests.readthedocs.io/en/latest/community/updates/#release-history | 16:59 |
| clarkb | ok so our options are 1) go back to distro docker-compose 2) install golang docker-compose 3) pin urllib3/requests | 17:00 |
| fungi | so yeah, maybe a recheck will "just work" | 17:00 |
| clarkb | oh I'm thinking new requests is what broke us since urllib3 has been around long enough to have noticed earlier? But ya lets do a recheck and see | 17:00 |
| clarkb | fwiw I think we can do 1) and probably should at this point since everything using stop_grace_period is on focal or newer and that has docker-compose that supports the newer feature | 17:01 |
| clarkb | the downside to this is cleaning things up will be weird since pip installs to /usr/local/bin which is earlier in path than /usr/bin which is where the distro package will put things | 17:01 |
| clarkb | I'm going to propose a revert of the pip docker-compose change and we can use that for testing at least | 17:02 |
| clarkb | oh what isn't clear to me is if we can use an older docker-compose against services running under newer docker-compose | 17:04 |
| clarkb | ugh | 17:04 |
| clarkb | does that mean the migration would be something like docker-compose down using newer docker-compose from pip, then remove pip docker-compose then docker-compose up with old docker compose? | 17:05 |
| clarkb | That is going to be "fun" | 17:05 |
| fungi | yeah, i think it's new requests/urllib3 which is the problem. the failed build installed requests-2.30.0-py3-none-any.whl with urllib3-2.0.1-py3-none-any.whl https://68381f75b18edb044fec-f99336677cbcb935a72dddbf18215860.ssl.cf2.rackcdn.com/882161/1/check/system-config-run-gitea/5f17fa0/bridge99.opendev.org/ara-report/results/472.html | 17:06 |
| clarkb | oh I just rechecked it | 17:06 |
| clarkb | I guess we can expect that to fail then | 17:07 |
| fungi | likely so, but an extra data point doesn't hurt | 17:07 |
| fungi | this is also likely to hit other projects, i have a feeling, we just happened to be the first to stub our toes on it | 17:08 |
| clarkb | ok I think that downgrading docker-compose isn't necessarily safe (ugh) | 17:09 |
| fungi | here's a just-opened issue related to the update, though i don't think it's what we're hitting: https://github.com/psf/requests/issues/6439 | 17:10 |
| clarkb | I'm going to push a change up that installs docker-compose from both system packages and distro packages and then see what jobs hat runs. Then maybe hold some of those test nodes to test downgrading since that is one options we've got | 17:10 |
| clarkb | However, I'm beginning to think the best option may be to roll forward since I expect that is safe | 17:10 |
| clarkb | rolling forward means figuring out how to install docker-compose the modern way however thati s done now | 17:10 |
| clarkb | ok jammy docker-compose is the same as latest pip docker-compose | 17:11 |
| clarkb | so this is only an issue for everything pre jammy (which is most stuff but still) | 17:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install docker-compose from the distro too https://review.opendev.org/c/opendev/system-config/+/882169 | 17:15 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install docker-compose from the distro too https://review.opendev.org/c/opendev/system-config/+/882169 | 17:18 |
| clarkb | picked four services to force failures on and hold nodes for | 17:19 |
| clarkb | https://github.com/docker/compose#linux for grabbing newer docker-compose | 17:20 |
| *** jpena is now known as jpena|off | 17:21 | |
| fungi | the bug bounty approach on urllib3's issues is interesting: https://github.com/urllib3/urllib3/issues | 17:22 |
| clarkb | holds in place for those four services. Now to make changes installing docker-compose the modern way and another to pin python-requests | 17:23 |
| clarkb | we will have many options | 17:23 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Pin python requests when installing docker-compose https://review.opendev.org/c/opendev/system-config/+/882171 | 17:29 |
| clarkb | wow so the new docker compose doesn't even really replace old docker compose | 17:41 |
| clarkb | switching is a ton of work. I feel a little better about htis now knowing its a mountain of work no matter what we do :) | 17:41 |
| *** Guest83 is now known as diablo_rojo | 17:41 | |
| fungi | doesn't replace... as in has entirely new syntax/configuration? | 17:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install modern docker-compose from github releases https://review.opendev.org/c/opendev/system-config/+/882173 | 17:49 |
| clarkb | fungi: the config files are shared but not sure about the cli stuff other than the base command changes. | 17:49 |
| clarkb | 882173 will give us more insight | 17:49 |
| clarkb | pinning python requests seems to be owrking | 17:50 |
| clarkb | but as noted in the commit message that is probably only good as a short term woarkaround. We should plan for something else going forward | 17:50 |
| fungi | yes | 17:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Pin python requests when installing docker-compose https://review.opendev.org/c/opendev/system-config/+/882171 | 18:08 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install docker-compose from the distro too https://review.opendev.org/c/opendev/system-config/+/882169 | 18:09 |
| clarkb | I think 882171 is going to work (I had to fix an issue with the grafana job that appears unrelated we'll see if this latest patchset passes) | 18:10 |
| clarkb | Then I rebased 882169 on top of it because it was failing completely due to theissue we are trying to work around due to path precedence. I want things to deploy then fail in testinfra so that we can ssh in and see if downgrading to older docker-compose will even work | 18:11 |
| fungi | makes sense, yep | 18:14 |
| clarkb | I'm beginning to think that what we will probbaly end up wanting to do is merge 882171 then switch to the modern compose plugin. That too will need testing for the upgrade path as I'm not sure what happens if we docker-compse up -d then docker compose down | 18:16 |
| clarkb | which is basically what our automated upgrade path would be if we take no special action | 18:16 |
| clarkb | And I'm suspecting that simply because I expect docker-compose -> docker compose to be more of a seamless transition than docker-compose 1.29.2 -> docker-compose 1.25.0 | 18:16 |
| clarkb | but we'll test it all I only have hunches at this point | 18:17 |
| clarkb | the naive switch to docker compose seems to have worked for etherpad at least. I only updated a few of the roles because I wanted to see if it would work | 18:17 |
| clarkb | but none of that is testing the docker-compose to docker compose transition yet | 18:18 |
| clarkb | wee | 18:18 |
| fungi | yeah, specifically concerned with a container that was upped with old docker-compose not downing correctly with new docker-compose? | 18:34 |
| fungi | hopefully docker is handling the state maintenance in which case it would be fine | 18:35 |
| fungi | but i really have only a tiny grasp of all of this | 18:35 |
| clarkb | yes exactly. Because our current running state for services is things are up using docker-compose 1.29.2 (actually need to audit this). If we change out for docker-compose from the distro or docker compose the plugin what will happen is we update the tool then run down and up -d using the new tool | 18:36 |
| clarkb | I suspect that docker-compose 1.29.2 -> docker compose the plugin will be handled more gracefully than docker-compose 1.29.2 -> docker 1.25.0 but testing should happen for all of this | 18:38 |
| clarkb | also we may not use down and up -d everywhere? we probably need to audit for stop and start in some places too :/ | 18:38 |
| clarkb | the other idea I had just now is maybe as part of 882171 we want to move docker-compose into a virtualenv | 18:40 |
| clarkb | that isolates it a bit more and makes the workaround less hacky | 18:40 |
| clarkb | then we can take our time sorting out the better long term option | 18:40 |
| clarkb | I'm about to take a break for lunch and may even go for a walk or something as activity like that tends to be good for sorting through these problems | 18:41 |
| fungi | yeah, i'm pondering it while i unload and spread sand | 18:46 |
| clarkb | on 882169's held paste node I susccessfully added a paste, ran /usr/bin/docker-compose down, confirmed containers stopped with docker ps -a, then ran /usr/bin/docker-compose up -d and checked my paste was still there and made a new paste | 19:56 |
| clarkb | that gives me some hope this won't be totally broken if we switch docker-compose versions like this | 19:56 |
| clarkb | that held node had a 1.29.2 /usr/local/bin/docker-compose default docker-compose and a 1.25.0 /usr/bin/docker-compose from the distro | 19:57 |
| fungi | yeah, seems promising | 20:00 |
| clarkb | looking at https://etherpad.opendev.org/p/opendev-bionic-server-upgrades I'll also want to test one or all of zuul-preview/meetpad/insecure-ci-registry | 20:01 |
| clarkb | I don't think any of the nodes I held on the first pass are bionic though. I'm going to run through the other three I did hold some of which are jammy which is 1.29.2 for both install methods and should be totally fine and others focal whihc will help confirm the paste results | 20:02 |
| clarkb | I do think that https://review.opendev.org/c/opendev/system-config/+/882171 and then possibly following that up with a move to a virtualenv are reaosnable short term workarounds. Not great but willing to compromise there | 20:03 |
| clarkb | held etherpad shows this transition working on jammy too as expected | 20:06 |
| clarkb | the zk node is on focal and harder to verify as a human but four letter commands work after doing a down then up with the older 1.25.0 docker-compose | 20:09 |
| clarkb | Now to check gerrit and then push a new change to add more failures to bionic nodes | 20:09 |
| clarkb | there was maybe some slowness on the gerrit node when running up -d. Could just be the node though | 20:12 |
| clarkb | it looks like downgrading docker-compose to the focal version is safe. Jammy versions is the same and works as well (as it should) | 20:13 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install docker-compose from the distro too https://review.opendev.org/c/opendev/system-config/+/882169 | 20:16 |
| clarkb | ok holds in place to check those older services on that ps | 20:17 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install modern docker-compose from github releases https://review.opendev.org/c/opendev/system-config/+/882173 | 20:31 |
| clarkb | I think moving to modern docker-compose is going to be more of a challenge. In particular some of the output strings have changed whihc we use to detect thing like whether or not new images are pulled | 20:32 |
| clarkb | doable I think just more effort | 20:32 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Only install docker-compose from the distro https://review.opendev.org/c/opendev/system-config/+/882178 | 20:34 |
| ianw | fungi: yeah i guess with was a semver major release ... why not just do a "if type(str)" though and not break every other install. the results of it are useful, but it's certainly hard to work with ... if you uncap ansible-lint we've had many regressions on new releases. but if we cap it, now we're also broken by other libraries in the ecosystem... | 20:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rebuild gitea images https://review.opendev.org/c/opendev/system-config/+/882180 | 20:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add more docker-compose installation comments https://review.opendev.org/c/opendev/system-config/+/882182 | 21:13 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix the testinfra hostname for the registry node https://review.opendev.org/c/opendev/system-config/+/882183 | 21:20 |
| clarkb | infra-root going from docker-compose 1.29.2 running containers and trying to down them with docker-compose 1.17.1 on bionic fails. | 21:23 |
| clarkb | I think this is because the container names change | 21:23 |
| clarkb | I was able to down them with 1.29.2 then up them with 1.17.1 | 21:23 |
| clarkb | 1.29.2 is able to down the 1.17.1 up'd containers. Which gives weight to the idea that rolling forward is generally safe but rolling back isn't | 21:24 |
| clarkb | idea time: we're landing the requests pin as a temporary workaround. We follow up with 882169 after removing the forced failures. This will install the distro package alongside pip installed docker-compose but distro docker-compose won't actually do anything at this point | 21:28 |
| fungi | we can presumably lift that once we're off bionic? | 21:29 |
| clarkb | oh thats a good point its only like 3 services on the old thing | 21:29 |
| clarkb | so maybe order of operations should be 1) workaround with requests pin 2) finish upgrades for those services 3) land 882169 4) remove pip installed docker-compose 5) automatic conversion by ansible things | 21:29 |
| clarkb | Then a very long term 6) move things to docker compose plugin | 21:30 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea to 1.19.3 https://review.opendev.org/c/opendev/system-config/+/877541 | 21:42 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM intentional gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 21:42 |
| fungi | sounds good to me, yep | 21:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Install docker-compose from the distro https://review.opendev.org/c/opendev/system-config/+/882169 | 22:07 |
| clarkb | I used my held focal paste node to test docker-compose 1.25.0 up'd containers getting down'd by `docker compose` installed via apt-get install docker-compose-plugin from the docker deb repo. This seems to work | 22:17 |
| clarkb | the format of things chagnes which is potentially awkward for transitioning stuff though | 22:17 |
| clarkb | but in generaly seems to be functional despite containers changing names | 22:17 |
| clarkb | interestingly it seems we can downgrade as well | 22:18 |
| clarkb | from `docker compose` up'd containers to docker-compose 1.25.0 down command | 22:18 |
| clarkb | so ya I think this gets significantly better if we get up to focal or newer | 22:19 |
| clarkb | The changes I have pushed should reflect steps 1) 3) 4) and 5) of the above list. Steps 2 and 6 will need separate efforts | 22:23 |
| clarkb | after all this trouble: https://pypi.org/project/requests/2.30.0/ | 22:37 |
| clarkb | they yanked the release | 22:37 |
| fungi | hah | 22:47 |
| fungi | funny not funny | 22:47 |
| corvus | i think we should declare a work holiday whenever there's a requests release. it would probably be more productive for everyone in the long run. | 23:07 |
| fungi | where are you headed with that huge backpack? oh, there was a requests release | 23:17 |
| opendevreview | Merged opendev/system-config master: Pin python requests when installing docker-compose https://review.opendev.org/c/opendev/system-config/+/882171 | 23:32 |
| opendevreview | Merged opendev/system-config master: Rebuild gitea images https://review.opendev.org/c/opendev/system-config/+/882180 | 23:47 |
| opendevreview | Merged opendev/system-config master: Add more docker-compose installation comments https://review.opendev.org/c/opendev/system-config/+/882182 | 23:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!