| clarkb | I think testinfra should be fairly quick the early part of the job is slow because it does two passes of gitea setup for idempotency checking | 00:00 |
|---|---|---|
| clarkb | I wonder if this might indicate gitea 1.22.5 is slow though | 00:00 |
| clarkb | or maybe we just got a slow node. A slow node seems more likely | 00:01 |
| fungi | yeah, we barely made it to post-run in the nick of time | 00:01 |
| opendevreview | Merged opendev/system-config master: Update Gitea to 1.22.5 https://review.opendev.org/c/opendev/system-config/+/937574 | 00:02 |
| fungi | yay! | 00:02 |
| fungi | unfortunately the hourly jobs kicked off a few minutes ago | 00:03 |
| clarkb | ya it will be another 15 minutes or so until we start upgrading | 00:04 |
| clarkb | the promote job for the image succeeded | 00:04 |
| clarkb | the sshd log and replication log files on review02 have one extra day compared to httpd and error logs | 00:10 |
| clarkb | I'm suddenly concerned that Gerrit's log pruning might only work for half of the logs :/ tomorrow shoudl give us a better indication though | 00:10 |
| clarkb | could also just be a rotation timing thing | 00:10 |
| fungi | likely | 00:14 |
| fungi | deploying now | 00:20 |
| clarkb | it does the load balancer first which means there is a small lag until we update the backends | 00:21 |
| clarkb | gitea09 just restarted its services | 00:22 |
| fungi | Powered by Gitea Version: v1.22.5 | 00:23 |
| fungi | lgtm | 00:23 |
| clarkb | a test clone is happy as is the web ui on a quick load | 00:23 |
| fungi | so far everything's checking out | 00:24 |
| clarkb | yup when the whole cluster is done I'm going to push my update to the podman stuff which we can use to check replication | 00:24 |
| fungi | perfect | 00:24 |
| clarkb | I've been saving that update just for this purpose | 00:24 |
| clarkb | bah my ssh agent just unloaded my keys | 00:25 |
| fungi | so close | 00:25 |
| clarkb | I've reloaded them | 00:26 |
| clarkb | I have them on timers to help me remember its time to stop working. Though these days early dinner beacuse kids tends to do a good job | 00:26 |
| clarkb | 09-11 are done now | 00:27 |
| clarkb | halfway there | 00:27 |
| clarkb | the job succeeded and checks against the web ui for all 6 lgtm. I'm pushing my change update now | 00:31 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea containers to use journald logging https://review.opendev.org/c/opendev/system-config/+/937657 | 00:31 |
| fungi | confirmed | 00:31 |
| clarkb | git fetch origin refs/changes/57/937657/2 worked for me where origin is https://opendev.org/opendev/system-config (fetch). Git show against FETCH_HEAD also has the new content | 00:32 |
| clarkb | so I think replication is working too | 00:33 |
| fungi | yeah, findable with some effort at https://opendev.org/opendev/system-config/commit/d9e8844 | 00:34 |
| clarkb | anything else you think we should check? | 00:35 |
| fungi | nope, cloning and stuff worked as did browsing around the various gitea webui features | 00:35 |
| fungi | i think we're all set | 00:36 |
| fungi | reprepro db rebuild for ubuntu-ports is still underway, i'll probably just check back in on it when i wake up tomorrow | 00:37 |
| clarkb | sounds good | 00:37 |
| opendevreview | Rafal Lewandowski proposed openstack/diskimage-builder master: Prevent from overwriting grub defaults if no variables are set https://review.opendev.org/c/openstack/diskimage-builder/+/937684 | 13:52 |
| opendevreview | Rafal Lewandowski proposed openstack/diskimage-builder master: Prevent from overwriting grub defaults if no variables are set https://review.opendev.org/c/openstack/diskimage-builder/+/937684 | 13:53 |
| fungi | ubuntu-porrts reprepro db build still in progress | 14:53 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Refactor check for new container images https://review.opendev.org/c/opendev/system-config/+/937655 | 15:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use docker-compose for container execs in gitea https://review.opendev.org/c/opendev/system-config/+/937717 | 16:08 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Run containers on Noble with docker compose and podman https://review.opendev.org/c/opendev/system-config/+/937641 | 16:08 |
| clarkb | I'm trying to stick all of the podman and docker compose related changes under topic:podman-prep so far I think we've found viable alternatives to behavior differences that accomodate things in backward/forward compatible ways | 16:09 |
| clarkb | mnasiadka: frickler: any reason for me to keep my raxflex ephemeral device test node or should I clean it up? | 16:10 |
| fungi | epel mirror is back to updating again after my change deployed, the epel 7 and 8 cleanup dropped utilization by half from 80gb to 40gb according to grafana | 16:50 |
| clarkb | thank you for sorting that out | 16:53 |
| fungi | so other than finishing the repair on ubuntu-ports and decommissioning the old non-stream centos volume (also opensuse?), all the other mirrors are updating regularly now | 16:55 |
| fungi | also the non-stream centos and opensuse mirrors are at 0 utilization so they're not taking up space in afs, just cluttering our grafana dashboard/reporting | 16:56 |
| clarkb | https://9cdcc73dee173a4574a8-d3d57836e2611e65f8d99a9aeb37f4da.ssl.cf1.rackcdn.com/937641/8/check/system-config-run-gitea/1f492b1/bridge99.opendev.org/ara-report/results/279.html is the next thing to sort out. It looks like docker compose as docker-compose has broken those execs (and that likely implies the database backup execs are also broken) | 16:58 |
| clarkb | I wonder if using -T means you don't get the env vars passed through or something | 16:59 |
| fungi | that would be odd | 16:59 |
| fungi | but not all that surprising i suppose | 16:59 |
| clarkb | ya it could also be a quoting issue I suppose | 16:59 |
| clarkb | I still have a held paste server with a mariadb container so I'm going to try and figure out what is going on there | 17:00 |
| clarkb | not exactly 1:1 but hopefully close enough to figure out | 17:00 |
| clarkb | I can definitely reproduce the behavior | 17:02 |
| clarkb | if I exec bash then run `mysql -uroot -ppasswrd` it works | 17:04 |
| clarkb | if I `exec bash -c 'mysql -uroot -ppasswrd'` it fails | 17:05 |
| clarkb | I suspect the issue may be the shim | 17:10 |
| clarkb | ya if I run things without the shim I get saner behavior so somethign with variable replacement and quoting in the shim | 17:11 |
| clarkb | ya the quotes are getting chomped | 17:18 |
| clarkb | and we're using the bash -c so that we can rely on the env vars for the secrets | 17:19 |
| clarkb | not sure what the best way to address this is. Maybe quote the entire string passed to the container in such a way that we preserve quoting? | 17:19 |
| clarkb | maybe alias would work better | 17:20 |
| clarkb | alias docker-compose='docker compose' does seem to fix my problems on the held node. But aliases are per user loaded in the bashrc which makes me wonder if this will work for cron jobs | 17:24 |
| fungi | nested wrappers might also work, but would be messy | 17:24 |
| clarkb | well nested wrappers is sort of what we already have | 17:24 |
| fungi | yeah, i mean yet another level of nesting | 17:24 |
| clarkb | docker-compose shim -> docker compose plugin -> container bash -> mysql | 17:25 |
| clarkb | fungi: I guess I'm not understanding how another wrapper would help | 17:25 |
| clarkb | the problem seems to be with bash processing of $@ and $* stripping out quoting in ways that are incompatible between the docker-compose shim and running docker compose directly | 17:26 |
| fungi | wherever the quotes are getting chomped, put that in its own script instead of quoting within the existing script, but i'm also not following where exactly it's breaking | 17:26 |
| fungi | ah, yes so dedicated wrapper for the desired command instead of relying on parameter expansion could be another way | 17:27 |
| clarkb | `docker compose -f /etc/lodgeit-compose/docker-compose.yaml exec mariadb bash -c 'mysql -u root -ppassword -e"USE lodgeit;"'` <- this works. But if you run throug through my naive docker-compose shim it executes `docker compose-f /etc/lodgeit-compose/docker-compose.yaml exec mariadb bash -c mysql -u root -ppassword -e'USE lodgeit;'` and fails | 17:27 |
| clarkb | the problem is bash -c needs everything after it to be considered input to the command argument. But instead it gets just bash -c 'mysql' and -u 'root' etc as separate args to bash | 17:28 |
| clarkb | maybe bash has a way to feed in a command that takes everything after a symbol as command input | 17:28 |
| clarkb | -s will read commands from STDIN | 17:30 |
| fungi | that might work, then use a heredoc | 17:30 |
| clarkb | do heredocs work in crontab entries? | 17:30 |
| fungi | oh, this is being embedded in a cronjob? | 17:31 |
| clarkb | fungi: the failure in the job I linked was not in a cronjob but we do a very similar thing for mysql backups which are in cronjobs | 17:31 |
| fungi | aha, you're trying to redo the database backups to use podman instead of docker | 17:31 |
| fungi | sorry, i was deep in other things and missing some of the context | 17:32 |
| clarkb | and also fix the gitea queries that failed in the job. They are similar but one runs in a ansible shell/command context and the other in a cronjob context | 17:32 |
| clarkb | I've found that alias docker-compose='docker compose' works well but I worry about ensuring the alias is in use in different contexts | 17:32 |
| clarkb | similarly a heredoc and -s would also probably work but maybe not for cronjob entries | 17:32 |
| fungi | in theory any command that's in a crontab entry could be put into its own script and then cron just invokes the script | 17:32 |
| clarkb | I guess for cron you just write a script that runs shell with a heredoc | 17:32 |
| fungi | looking at the bash manpage, it seems $* and $@ work differently when written as "$*" and "$@" as far as word-splitting during expansion | 17:41 |
| clarkb | ok I have heredoc with -s working with the shim for the non cron case | 17:41 |
| clarkb | which implies we can make it work with teh cron case by having cron run a shell script | 17:42 |
| clarkb | fungi: oh let me test that too | 17:42 |
| fungi | yeah, read the "special parameters" subsection in the manpage, other than it not being as straightforward as one might assume, i'm struggling to get the exact implications from it | 17:43 |
| clarkb | experimentally "$@" is what I want | 17:44 |
| fungi | that said, putting things in a separate script seems more straightforward and maintainable than relying on inherently confusing bash special-casing magic | 17:45 |
| clarkb | ya though making the shim more likely to work is also good for interactive usage | 17:46 |
| fungi | agreed, of course | 17:46 |
| clarkb | I'll update the shim to quote $@ and see if that is sufficient to make these commands happy and if not we're doing the other options anyway and if yes we can still refactor to make it easier to undersatnd | 17:46 |
| clarkb | I worrying the quoting in this: "USE gitea; UPDATE user SET avatar = '\''{{ item }}'\'', use_custom_avatar = 1 WHERE name = '\''{{ item }}'\''"' will still explode | 17:48 |
| clarkb | but maybe not | 17:48 |
| fungi | also maybe stick a comment adjacent referring to interpreter magic so when someone needs to make adjustments and stuff confusingly breaks they know where to start digging in the documentation | 17:49 |
| fungi | every time i try to find something in the bash manpage i feel like i'm stepping out into the wilderness without a map and compass | 17:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Run containers on Noble with docker compose and podman https://review.opendev.org/c/opendev/system-config/+/937641 | 17:51 |
| clarkb | fungi: ^ somethign like that | 17:51 |
| clarkb | oh I should add a keywork to grep in the bash manpage too | 17:51 |
| fungi | "special parameters" yeah | 17:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Run containers on Noble with docker compose and podman https://review.opendev.org/c/opendev/system-config/+/937641 | 17:52 |
| clarkb | done | 17:52 |
| fungi | that looks good, thanks! | 17:52 |
| clarkb | re the string above that I'm worried about I'm not even sure I undersatnd the quoting without hte extra layer of indirection :) | 17:52 |
| fungi | it's like somebody scattered toothpicks and pepper on the command | 17:53 |
| clarkb | echo'ing it the '\'' become single 's which makes sense from a ok that is what we are trying to execute standpoint but not from a how does '\'' escape to ' | 17:54 |
| clarkb | looks like to escape a ' within a bash string with 's you have to wrap the \' around 's? | 17:57 |
| fungi | embedding one language's syntax inside another is a recipe for madness, yes | 17:57 |
| fungi | oh, hey, the big reindex step for ubuntu-ports finished! | 17:58 |
| fungi | time to do the other steps | 17:58 |
| clarkb | oh I see 'string1''string2' -> "string1string2" so its literally just ending and starting strings based on what is most convenient for quoting | 17:59 |
| clarkb | I feel like I probably knew that once upon a time then my brain evicted it as unnecesary | 17:59 |
| clarkb | however that makes me more hopeful that the shim thing might actually work? I dunno | 17:59 |
| fungi | ah, yeah was there an outer set of ' quotes omitted in your example? if so, that's what's going on | 18:00 |
| clarkb | yes | 18:00 |
| fungi | yeah, it was missing the opening ' but i see the closing ' is still there | 18:00 |
| clarkb | so them maybe this magical quotes will do the right thing here since we're abusing string quoting and concatenation throughout | 18:00 |
| clarkb | s/them/then/ | 18:00 |
| fungi | hopefully-repaired reprepro database files are being copied back into afs now, which will presumably take a while as i'm overwriting nearly 4gb of data | 18:05 |
| clarkb | this has been the biggest problem I've run into so far though since it may require us to be very careful in how we do things to maintain backward and forward compatibility | 18:05 |
| clarkb | heredocs feel like the ultimate in fixing that though so we should hve a fallback (in particular because you can have them not interpolate variables and also avoid quoting/unquoting madness) | 18:05 |
| fungi | cache invalidation, naming things, and bash parameter quoting | 18:06 |
| clarkb | ha | 18:06 |
| clarkb | I'm glad I started poking at this with some of the more complicated services though. Its good to find these corner cases early before we decide whther or not we're going to commit to this | 18:06 |
| clarkb | one upside is that once a service transitions to docker compose we can stop using the shim entirely within that service which may resimplify things | 18:07 |
| fungi | oh, the db write into afs took a lot less time than reading it out of afs originally did | 18:08 |
| fungi | trying to refresh the mirror now | 18:09 |
| fungi | grr, back to the same error | 18:11 |
| clarkb | :/ | 18:13 |
| fungi | Internal error of the underlying BerkeleyDB database: | 18:14 |
| fungi | Within checksums.db subtable pool at get: BDB0075 DB_PAGE_NOTFOUND: Requested page not found | 18:14 |
| fungi | There have been errors! | 18:14 |
| clarkb | fwiw I ran the lodgeit db backup script and redirected into a file (it normally sends out to borg if configured to backup with borg) and that worked with the "$@" so it may just be the extra db manipulation we do for avatar images that have problems if the last patchset doesn't work | 18:14 |
| clarkb | fungi: are we out of quota and maybe short writing? | 18:14 |
| fungi | no, nearly 100gb of headroom before we hit quota in that volume | 18:15 |
| clarkb | I wonder if the inputs are corrupt so we get a corrupt db? seems like reprepro is built specifically to handle such cases of bad repos though | 18:17 |
| clarkb | I haven't heard anything form mnasiadka or frickler so I'm going to go ahead and clean up my jammy test node in raxflex | 18:24 |
| clarkb | It is easy to recreate | 18:24 |
| fungi | mmm, running reprepro with the db directory overridden to my local copy doesn't seem to be hitting the same errors, at least | 18:25 |
| fungi | though it is hitting different errors that are more tractable (packages with incorrect checksums that need cleaning up) | 18:25 |
| clarkb | looking through collected logs from test nodes the switch from syslog to journald for logging our conatiners seems to work and continues to write out to /var/log/containers too | 18:59 |
| clarkb | the only other real concern I have si that we might fill the disk more quicky with journald filling up? I'm not sure what the rotation policy is on that | 19:00 |
| clarkb | reading journald.conf's manpage it should police itself on maximum disk usage | 19:04 |
| clarkb | using up to 10% of the filesystem it lives on which seems reasonable and makes me less concerned about triple accounting those logs instead of double accounting them | 19:05 |
| clarkb | (previously was syslog + /var/log/containers, now we're adding journald to the mix) | 19:05 |
| clarkb | I suspect that a step -1 here is going to be switching to journald for all of our services, then updating docker exec commands to use docker-compose commands to avoid the conatiner name change problem. Then see what additional corner cases we can fine | 19:06 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use docker-compose for container execs in gitea https://review.opendev.org/c/opendev/system-config/+/937717 | 19:21 |
| clarkb | with ^ I think gitea might work with both old docker-compose and new docker compse | 19:21 |
| clarkb | oh wait there is one more issue | 19:24 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use docker-compose for container execs in gitea https://review.opendev.org/c/opendev/system-config/+/937717 | 19:36 |
| clarkb | good news is docker compose is quite a bit richer with what it can do. Bad news is docker-compose is pretty simple so we're left finding workable middle grounds while we try to be backward and forward compatible | 19:39 |
| clarkb | I did leave a todo for where wecan improve things once on docker compose though | 19:39 |
| fungi | ubuntu-ports mirror update is still in progress, i'm popping out for a bit to grab early dinner before i go back to paperwork | 20:00 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Use docker-compose for container execs in gitea https://review.opendev.org/c/opendev/system-config/+/937717 | 20:18 |
| *** elodilles is now known as elodilles_pto | 20:51 | |
| fungi | ubuntu-ports mirror update seems to be about halfway through the alphabet | 21:48 |
| fungi | argh. *now* i'm getting "Disk quota exceeded" | 22:44 |
| fungi | aborting and i'll bump up the quota a bit | 22:44 |
| clarkb | if it makes you feel better I finally got the system-config-run-gitea job passing with docker compose and podman and then the other jobs I was testing with docker compose failed due to rate limits... | 22:45 |
| clarkb | but the good news is I think this is backward and forward compatible as written | 22:46 |
| fungi | okay, quota on ubuntu-ports has been increased from 850gb to 1tb for now (regular ubuntu volume quota is 1.2tb with 1.1tb in use, for comparison) | 22:48 |
| fungi | the good news is it seems to have effectively resumed where it left off | 22:53 |
| clarkb | all of the prep changes I've written so far pass CI now. Trying to get the WIP change to pass showing the forward compatibility of the changes. if we get that I think we can start evaluating some of these prep changes and landing them. In particular they shold be roughly equivalent and therefore safe to use | 23:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!