| opendevreview | Lukas Kranz proposed zuul/zuul-jobs master: Remove get_md5 parameter from stat module. https://review.opendev.org/c/zuul/zuul-jobs/+/921637 | 06:01 |
|---|---|---|
| *** ykarel_ is now known as ykarel | 07:01 | |
| clarkb | looks like the cloud launcher playbook fix was landed. Has anyone checked if that has allowed things to run successfully? I'll be checking shortly myself | 15:08 |
| clarkb | I'm also starting to put an openmetal bootstrap todo list together here: https://etherpad.opendev.org/p/openmetal-cloud-bootstrapping as I'm slightly worreid with everything else going on and then me popping out for almost a week in a few days I may not get through it all myself | 15:09 |
| frickler | clarkb: looks like it is still failing, seems we need to mark /home/zuul/src/opendev.org/opendev/ansible-role-cloud-launcher/.git as safe instead | 15:12 |
| clarkb | I wondered about that. I found evidence elsewhere that marking the top level dir was fine btu maybe it depends on the action (clone vs other commands?) | 15:13 |
| clarkb | frickler: thanks for checking I can update shortly | 15:13 |
| frickler | clarkb: I also created a test account on openmetal last week to test some basic functionality, that looked good to me | 15:16 |
| frickler | I'm wondering whether we'd also (or primarily?) want to use that cloud for nested-virt labels, given that we have more control over it than for other clouds and we seem to have more than enough RAM available | 15:17 |
| clarkb | yes I think that is an option availbale to us as we spin things up | 15:17 |
| frickler | or in the other direction discuss how much CPU overbooking we could/want to do | 15:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Mark .git dirs safe when marking ansible roles safe for git https://review.opendev.org/c/opendev/system-config/+/921685 | 15:18 |
| clarkb | frickler: ^ something like that? | 15:18 |
| frickler | also do we want to use local storage or boot from volume? didn't check yet how it is set up by default | 15:18 |
| clarkb | I think the way it worked previously was that it was "local" but all backed by ceph? rather than explicit boot from volume | 15:20 |
| clarkb | ya I know the cloud storage is aggregated by ceph and I can't find any evidence in our nodepool configs that we were doing explicit boot from volume so instead I think it was transparent to us that ceph was backing the nodes | 15:21 |
| clarkb | fwiw I really don't like boot from volume for a number of reasons. Primary one for nodepool use cases is that when nodes fail to delete we can't delete the image tied to its volume. But other reasons include rescuing may or may not work and if it does work you have to know the magic incantation to set the appropriate microversion | 15:22 |
| clarkb | I think boot from volume is a really nice idea on paper, but the implementation leaves a lot to be desired | 15:22 |
| frickler | well I actually meant root volume on ceph vs. root volume on local storage. seems the cloud is now set up to do the former, which all 7 hosts running a ceph OSD, so some kind of converged setup, which may have performance issues. we can start with that anyway, but may want to reconsider if we see issues | 15:28 |
| clarkb | got it. I think that was how it was configured previously. We've definitely seen iops issues in some clouds but I don't think the previous inmotion one was one. ++ to keeping an eye on it though | 15:33 |
| frickler | I'm also not sure whether running nodepool instances on the controllers is something we'd want to do at all. running the mirror node there should be fine, though | 15:40 |
| clarkb | any opinion on whether or not we should use the default ubuntu qcow2 (converted to raw) image or the kvm optimized one? | 15:41 |
| clarkb | frickler: ya I think we can just add more nodes to the host aggregate if we make that decision. But again we did that before and it was mostly ok expcet for where it conflicted with the mirror node | 15:41 |
| clarkb | It looks like they preseed the cloud with ubuntu images now but I like the idea of uploading a more current one ourselves while we get started | 15:42 |
| clarkb | curiously noble doesn't have a kvm optimized kernel image option | 15:45 |
| clarkb | I went with the kvm opitmized option for jammy though | 15:45 |
| fungi | i wonder if that was a kernel backport for jammy and noble's just doesn't need optimizing for kvm due to being newer | 15:51 |
| clarkb | the horizon create an image wizard doesn't explain the difference between private, shared, community, and public... Also there is no progress meter | 16:10 |
| clarkb | I would've had to look all that up anyway if using the cli so not a huge deal but a missed opportunity maybe | 16:10 |
| opendevreview | Merged opendev/system-config master: Mark .git dirs safe when marking ansible roles safe for git https://review.opendev.org/c/opendev/system-config/+/921685 | 16:11 |
| tonyb | Now we wait ~10.5 hours and see if it ^^ worked | 16:13 |
| clarkb | ya or if I end up being fast (I doubt I will be) landing an update to the bastion vars to configure the new cloud may also trigger it sooner | 16:14 |
| clarkb | but I have a meeting this morning then need to get some exercise before it is too hot and also have to prep for tomorrows meetings... so maybe unlikely I'll move that quickly. We'll see | 16:14 |
| clarkb | good reminder to update the meeting agenda with anything you'd like to add. I can add the openmetal stuff when I get around to it later today | 16:15 |
| tonyb | I can work through the bootstrap list while I wait for runs of the mediawiki stuff | 16:15 |
| clarkb | tonyb: cool I'm currently working on the image uploads myself | 16:16 |
| clarkb | jammy is uploading and then I'll do noble next | 16:16 |
| tonyb | Okay cool. | 16:16 |
| clarkb | creating the new accounts and updating secrets files and all that is probably the next thing. I was going to cross check against what we have already in place for inmotion and do something similar | 16:17 |
| tonyb | Okay I can look at that. | 16:17 |
| frickler | I'm also concerned at bit about data for our cloud being sent to datadog, at least I received an invite from them but don't intend to pick that up. did openmetal do that or did we opt into it? (I did see openmetal does offer that as an option in their dashboard) | 16:20 |
| clarkb | frickler: I think that was done for us | 16:21 |
| clarkb | we can definitely followup and ask if disabling that is an option. I agree that it seems unlikely we'll use it | 16:21 |
| fungi | i also doubt we'll use it, on the other hand i don't think we do anything secret in there so other than the risk of exposing things like admin credentials for that specific cloud or introducing unexpected backdoors, it's probably not actively harmful | 16:47 |
| fungi | on balance, i agree disabling it if possible is the safer option, but if it's very important to the donors that it stay enabled i don't strongly object | 16:48 |
| clarkb | I'm beginning to wonderif this image upload is going to fail. Its been almost an hour. We shall see | 17:02 |
| tonyb | clarkb: That does seem to be excessive :/ | 17:03 |
| frickler | I mostly upload images from somewhere local to the target cloud to avoid slow uplinks | 17:05 |
| clarkb | it just finished so I was worried unnecessarily | 17:05 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM Testing did docker/moby break something? https://review.opendev.org/c/opendev/system-config/+/921693 | 17:07 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Urlencode git url in prepare-workspace-git https://review.opendev.org/c/zuul/zuul-jobs/+/921694 | 17:09 |
| frickler | fyi you can do ". /opt/omi-cli/bin/activate" and ". /etc/kolla/admin-openrc.sh" and then use osc locally on the .213 server | 17:12 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add an opendev specific build of mediawiki https://review.opendev.org/c/opendev/system-config/+/921321 | 17:27 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 17:27 |
| tonyb | frickler, fungi: If either of you have time can you have a look at: 920760: Add golang based docker compose tool. | https://review.opendev.org/c/opendev/system-config/+/920760 | 17:28 |
| tonyb | With the mediawiki install I'm working on I have 2 apache processes "outer" on the host, and "inner" inside the container. I did this as I thought it'd give us a place to add additional filters incase of bad actors. I'm starting to wonder if this isn't adding complexity for little to no value. Can we just expose the apache running in the container to the world? | 17:32 |
| tonyb | Thoughts? | 17:32 |
| clarkb | we could. I think we do similar with at least one other service (meetpad with nginx) | 17:32 |
| clarkb | and since it is php you basiclly have to have an apache in the container right? | 17:33 |
| fungi | or use something like fastcgi | 17:33 |
| tonyb | I believe so. I *think* in theory there is a setup that'd let FPM or other accelerators work with apache on the host and PHP/mediawiki in a container | 17:34 |
| clarkb | I think we can keep it simple and just go with we know works. Even if that means two apaches | 17:35 |
| tonyb | Okay | 17:35 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add an opendev specific build of mediawiki https://review.opendev.org/c/opendev/system-config/+/921321 | 17:56 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 17:56 |
| bbezak | Hi. I've got a question how ansible is installed on zuul nodes. ansible-galaxy is not installed? Or maybe it is in different location. (on the current node it is ansible 2.15.12 and it should have ansible-galaxy): | 18:10 |
| bbezak | https://zuul.opendev.org/t/openstack/build/d446eeb3c47a44e2b00106e62f67502b/console | 18:10 |
| bbezak | [Errno 2] No such file or directory: b'ansible-galaxy' | 18:10 |
| bbezak | I'd like to use newer community.docker collection due too https://github.com/ansible-collections/community.docker/issues/868 | 18:10 |
| bbezak | and I don't want to invoke another ansible inside current ansible executor (however that is also an option) | 18:10 |
| clarkb | bbezak: managing collections and roles via galaxy is not possible in the root zuul ansible env. You can expose thigns through the repos involved in the job, but thats it. Otherwise you need to use a nested ansible | 18:13 |
| bbezak | that's what I thought, thx clarkb | 18:14 |
| clarkb | we use nested ansible with quite a bit of success doing testing of the deployment configuration management for services like zuul and gerrit and so on. | 18:14 |
| bbezak | yeah, us too in Kolla world | 18:14 |
| bbezak | however here it is simple playbook that was running in periodics job to push kolla images to registry, so it wasn't need to be nested | 18:15 |
| clarkb | bbezak: if the newer version is available in ansible 9 you could opt into using that for the jobs | 18:18 |
| clarkb | the default is still 8 but 9 is now available. | 18:18 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 18:20 |
| bbezak | it looks like it is in ansible 10 clarkb - https://github.com/ansible-community/ansible-build-data/blob/main/10/CHANGELOG-v10.md#included-collections | 18:24 |
| clarkb | ya the issue mentioend it might be going into a 9 release too | 18:40 |
| clarkb | zuul doesn't have 10 yet | 18:40 |
| fungi | bbezak: also, to be absolutely clear, zuul doesn't install ansible onto job nodes (your job could install a copy of ansible on them, but that would be a nested ansible like clarkb described) | 18:40 |
| fungi | the ansible which interprets your job playbooks is in a venv on the zuul executor servers, invoked from an isolated sandbox container | 18:41 |
| clarkb | infra-root the noble image appears to have uploaded but then it also auto logged my session out. The file size looks right though so I'm going to assume it succeeded then logged me out. | 18:41 |
| clarkb | if we discover the image isn't functional later we can always do it again from a local location for higher bw | 18:42 |
| clarkb | with that stuff done I'm going to pop out now for a bike ride. I'll be back to work on the infra meeting agenda and contineu with openmetal stuff | 18:44 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 18:57 |
| bbezak | thx for clarification fungi | 19:14 |
| bbezak | indeed clarkb - community.docker could be bumped in next minor ansible 9 too, but not yet though - https://github.com/ansible-community/ansible-build-data/blob/main/9/CHANGELOG-v9.md#changed-collections https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#ansible-community-changelogs | 19:20 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 19:32 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 19:54 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: prepare-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 20:15 |
| clarkb | as a heads up I have a dentist appointment 2 hours before our meeting tomorrow. I should make it back home with plenty of time to spare judging on previous dentist visit timing. | 20:54 |
| tonyb | https://etherpad.opendev.org/p/openmetal-cloud-bootstrapping#L17 This just `openstack user create ... ` x 2 right? | 20:55 |
| fungi | just as long as you haven't grown any extra teeth recently | 20:55 |
| tonyb | We don't need the accounts in a seperate project etc | 20:55 |
| fungi | tonyb: we normally use two projects so that they can have separate quotas and other resources | 20:55 |
| clarkb | tonyb: yes in separate projects please | 20:55 |
| tonyb | Okay | 20:56 |
| clarkb | and for chosen passwords we tend to use long alnum strings because specail chars and config management don't always play nice | 20:56 |
| fungi | e.g. deleting all server instances nodepool can see won't break the mirror server | 20:56 |
| clarkb | I do pwgen -s 32 1 or similar iirc | 20:56 |
| tonyb | Okay | 20:56 |
| clarkb | I've done a first pass editing the meeting agenda. fungi I almost certainly used the wrong terminology for the verp stuff. Happy to amend that if you point me at the correct terms | 21:04 |
| fungi | clarkb: looks good enough to me | 21:05 |
| clarkb | cool I'll take it | 21:05 |
| fungi | the actual names for the config options are sort of all over the place anyway | 21:05 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 21:44 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 22:45 |
| opendevreview | Tony Breeds proposed opendev/system-config master: DNM: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 22:45 |
| clarkb | meeting agenda has been sent | 23:01 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!