15:01:57 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:01:57 <opendevmeet> Meeting started Tue Sep 27 15:01:57 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:57 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:57 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:02:02 <noonedeadpunk> #topic rollcall
15:02:06 <jrosser_> o/ hello
15:02:09 <noonedeadpunk> \o/
15:02:17 <damiandabrowski> hi!
15:02:28 <ThiagoCMC> jrosser_, you mean, without Inspector containers?
15:03:46 <ThiagoCMC> i.e., no "ironic-inspector_hosts:" declared in openstack_deploy dir...?
15:04:42 <noonedeadpunk> #topic office hours
15:05:18 <noonedeadpunk> So, recently I've worked a bit on improving testing for cinder volumes. Eventually, we did not test if it was working or not in aio at all
15:05:43 <noonedeadpunk> as we never run any tempest scenario that would use bfv or jsut attach volumes
15:06:40 <noonedeadpunk> At the same time I'm not sure how we really want to run this as we still want to check ephemerals and reduce jobs timing
15:06:47 <noonedeadpunk> (or at least not increasing it
15:07:08 <noonedeadpunk> I pushed couple of patches here https://review.opendev.org/q/topic:osa%252Ftest_volumes
15:07:46 <noonedeadpunk> Next thing I looked into is ansible 2.13. We're suuuper close now, but now a bit blocked by not having infra mirrors for moder ceph
15:08:11 <jrosser_> how is that connected?
15:08:16 <noonedeadpunk> By now we were installing octopus and jrosser_'s patch for pacific was never merged.
15:08:42 <noonedeadpunk> jrosser_: well, it's connected to ceph-ansible steable-7.0 that tries to install quincy by default
15:08:49 <noonedeadpunk> *stable-7.0
15:09:19 <jrosser_> oh i see - and we need 7.0 for the newer ansible to be ok?
15:09:30 <damiandabrowski> btw. what issues we had with bfv? I assume there's a reason why you're so motivated to cover it with tempest now :D
15:09:40 <noonedeadpunk> and stable-7.0 does install our config_template collection rather their own fork
15:10:48 <noonedeadpunk> damiandabrowski: it was not working?:) Denys was trying to spin up aio_ceph and cinder-volume was not starting at all because of backend naming started with rbd
15:11:03 <NeilHanlon> i'm looking into that ansible python / rocky / ceph thing as well as getting back to my integration for rocky 9. re-downloaded the VM and hoping to get it merged this week
15:11:26 <jrosser_> i'm sure we came across that backend naming thing a loooong time ago for rbd
15:11:29 <damiandabrowski> ah ok
15:11:53 <noonedeadpunk> oh yes we did. But we never fixed aio as it never bothered us (as we didn't test it)
15:12:01 <jrosser_> ahha of course
15:12:12 <noonedeadpunk> so that was motivation :)
15:12:47 <NeilHanlon> tests?! who needs tests
15:12:52 <NeilHanlon> tests only cause problems :D
15:12:58 <noonedeadpunk> NeilHanlon: well, as of now I've rejected our bug, as it can be quite simply worked around (and in fact it's not our bug). But in general ansible might be worth fixing...
15:13:38 <NeilHanlon> agreed, just not sure "what" needs fixing just yet.. in any case that one I'm taking back to rocky
15:14:07 <noonedeadpunk> NeilHanlon: another question to you (or your team) - maybe you're also itnerested in helping out MariaDB to build/have packages and repo for CentOS/Rocky... https://jira.mariadb.org/browse/MDEV-28842
15:14:48 <noonedeadpunk> As when I talked to monty - they said there's no interest from ppl to have that
15:14:48 <NeilHanlon> totally. i will comment on that issue with support and offer to help :)
15:16:08 <noonedeadpunk> I can help out with Rocky 9 if needed - I jsut downloaded images and was about to spawn VMs to see  what's wrong.
15:16:57 <NeilHanlon> iirc from zuul, ansible in the virtualenv  was unable to find libselinux to disable it
15:17:38 <jrosser_> this is something that was changed in recent ansibles, how selinux is handled
15:17:45 <noonedeadpunk> I wonder that maybe it does need libselinux binary to realize that it's not needed?:)
15:17:56 <jrosser_> specifically how the python<>.so interface is done
15:18:05 <noonedeadpunk> yeah, they do use c bindings now
15:18:33 <NeilHanlon> that lib should definitely be.. available, since the package is installed on the system
15:18:47 <noonedeadpunk> jrosser_: there was issue regardign your patch that fixed keystone bootstrap when proxy was used. It's fixed now with https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/859232 but not sure if this doesn't break anything for you (and idp stuff).
15:18:59 <jrosser_> ah yes i saw that
15:19:03 <noonedeadpunk> and Andrew was not around to ask to test out.
15:19:11 <jrosser_> i think andrew might be back next week
15:19:31 <noonedeadpunk> But I _think_ it should be fine
15:19:51 <jrosser_> i could try that easily with an AIO on our internal instance, thats behind a proxy
15:20:12 <noonedeadpunk> NeilHanlon: maybe it also goes back to ansibles' discovery that is failing
15:20:47 <noonedeadpunk> (and python discovery that does not happen for $reason)
15:21:20 <jrosser_> theres been quite some discussion of ironic recently too - we're really not in great shape there and it would be good if we can make things more working out-of-the-box for Z
15:21:25 <noonedeadpunk> but not sure - did not have chance to check out. Though I believe we do have couple of users by now that try or do use Rocky with OSA
15:22:17 * noonedeadpunk still haven't got to ironic deployment so quite clueless
15:22:49 <jrosser_> well, ironic itself is hugely configurable and pluggable
15:23:26 <jrosser_> and the existing os_ironic has one particular view about how the deployment should be (single tenant, metal deploy, some others, ......)
15:23:53 <jrosser_> and none of those things are documented in the role and i think jamesdenton mentioned making some docs which should cover that
15:24:21 <jrosser_> stuartgr has done an LXC deployment here and there is really a lot that doesnt work
15:24:46 <noonedeadpunk> that's sad to hear :( but indeed role before did not have enough love...
15:25:12 <jrosser_> good news is we have it kind of working now, but there are a bunch more patches needed
15:25:21 <noonedeadpunk> also to reming about PTG doc - here's etherpad:
15:25:24 <noonedeadpunk> #link https://etherpad.opendev.org/p/osa-antelope-ptg
15:25:27 <jrosser_> including for IPMI serial consoles working in horizon
15:26:12 <jrosser_> i would really like to get this in better shape for Z release if there is time
15:27:53 <noonedeadpunk> I'd love to help - but can help only with reviews now.
15:28:29 <noonedeadpunk> If look optimistically at ours internal backlog - I'm not sure if we get to it until Antelope... Likely yes, but not 100%
15:29:22 <noonedeadpunk> What I'd really to get progress on - internal SSLs. As a lot was already done (read - concept) but I'm not sure where and why this has stuck.
15:29:28 <jrosser_> well it's kind of similar here - we can work on ironic patches as we are deploying it
15:29:51 <jrosser_> SSL has got stuck because it was really being driven by james gibson and sadly he has moved to another job now
15:29:56 <jrosser_> and i don't have any replacement
15:30:06 <noonedeadpunk> I guess we can take it and move forward if it's the only reason
15:30:13 <opendevreview> Kevin Carter proposed openstack/openstack-ansible master: feat: Add skyline deployment capability  https://review.opendev.org/c/openstack/openstack-ansible/+/859446
15:30:27 <jrosser_> there is not really any technical issue other than needed to have time to work on it
15:30:36 <prometheanfire> elk logstash container too small at 8G?
15:31:40 <noonedeadpunk> and we're quite happy with the concept? as we haven't landed even blueprint for $reason
15:33:03 <jrosser_> well it is complex
15:33:22 <jrosser_> the transition from backends being http -> https across and upgrade is pretty difficult
15:33:30 <noonedeadpunk> yeah, migration is quite complex indeed
15:33:31 <jrosser_> and understanding is needed by whoever takes it on
15:34:07 <jrosser_> well the code makes the transition simple for an operator
15:34:08 <cloudnull> o/ jrosser_ I PR'd your role for skyline into the openstack-ansible repo. any thoughts on seeing that role brought into the os namespace?
15:34:11 <jrosser_> but the mechanism is complex
15:34:27 <jrosser_> cloudnull: that would be fine - just lacking time from my POV
15:34:33 <noonedeadpunk> cloudnull: the only issue I see is that skyline is not official project yet
15:34:41 <noonedeadpunk> not sure if that should tighten us though
15:35:00 <cloudnull> ¯\_(ツ)_/¯
15:35:01 <jrosser_> not sure they have branches or releases yet really which we'd have to be careful about
15:35:19 <noonedeadpunk> it's still "emerging"
15:35:34 <jrosser_> but the velocity seems high so having it at least as "experimental" would be great
15:35:36 <cloudnull> works i n dev 😉
15:35:57 <jrosser_> personally i am stuck with horizon because of federation
15:36:11 <jrosser_> and i did the role to bootstrap interest from others
15:36:12 <cloudnull> my only complaint is that all the VMs need a volume, but besides that its really nice alt to horizon
15:37:18 <jrosser_> cloudnull: i think CI might be interesting, because the "yarn" (?) bits needed many Gb of ram
15:37:38 <prometheanfire> heh, tested in chrome only it seems
15:37:40 <noonedeadpunk> I'm kind of fine bringing i as experimental
15:37:54 <cloudnull> All of the static files are shipped with the python bits so we dont have to build the Node things.
15:38:32 <noonedeadpunk> but I;m not sure they're following u-c and requirements yet.
15:38:43 <noonedeadpunk> but yeah, good as experimental
15:39:04 <noonedeadpunk> should we gain some interest during ptg/operational hours or jsut agree to add it?
15:39:10 <cloudnull> the current skyline role doesn't do any of the node building. it probably would be a good thing to add so that we can force a rebuild of static files if required?
15:39:55 <cloudnull> I'm in no rush to see it merged, just was committing things I have running in my environments.
15:40:09 <cloudnull> also trying to highlight jrosser_ excellent work on that role :D
15:40:11 <jrosser_> i think my first patches did have the node build stuff so it should be there in the history
15:40:42 <jrosser_> i converted the skyline "install from source" instructions pretty verbatim into ansible iirc
15:41:06 <cloudnull> I'm going to take another pass at trying to make it work in apache
15:41:15 <noonedeadpunk> I'd personally hate to mess up with npm...
15:41:17 <cloudnull> at the moment its nginx or bust
15:41:38 <jrosser_> yeah, i didnt dive deep enough into making the equivalent reverse proxy setup
15:41:46 <cloudnull> I sadly cant make it work in uwsgi 😢
15:41:49 <noonedeadpunk> I also wonder if we should make skyline mutualy exclusive with horizon
15:42:17 <noonedeadpunk> or we should have some acls on haproxy
15:42:31 <cloudnull> noonedeadpunk +1 probably a good idea. in that initial pr i used 8443 for ssl and 9999 for non-ssl (9999 is their default)
15:44:32 <noonedeadpunk> btw we have couple of backports to stable branches - so if anybody have time for reviews - super welcome
15:45:12 <jrosser_> cloudnull: https://github.com/jrosser/openstack-ansible-os_skyline/commit/82b1f5a5e6eff9df441c96677e0aa6d578bc8552#diff-7ae20663f88c2ee2e49e28cecf7c0eeb99efdb53ec0faf27c0a50ce3dcaf2370
15:45:35 <jrosser_> i expect this would benefit from doing all the npm-ness on the repo container
15:45:44 <jrosser_> only wants to be done once as it's super heavyweight
15:46:41 <cloudnull> for sure!
15:47:10 <NeilHanlon> noonedeadpunk: any idea on how to actually get into this DIB image? I tried injecting my ssh key but I get denied lol
15:47:32 <cloudnull> I found I didn't need to actually run that when doing the deployment as all the generated node bits are part of the skyline-console repo. however, having the option to rebuild is probably a really good thing to have.
15:47:40 <jrosser_> ah interesting
15:48:16 <cloudnull> yeah for my local builds I just pip installed my way to success.
15:48:21 <jrosser_> i wonder if that is deliberate, or just a .gitignore accident :)
15:48:34 <cloudnull> however, who knows how that will shape up as they get closer to an actual relesae
15:48:47 <cloudnull> yeah exactly my thoughts too
15:49:01 <noonedeadpunk> NeilHanlon: I think it depends on set of elements you've used to build it. As under normal conditions some cloud-init element should be explicitly included
15:49:55 <noonedeadpunk> we can also ask infra folks to create a hold for this patch/job and they can place a key to the VM spawned by zuul
15:50:06 <jrosser_> do you need config-drive?
15:50:37 <jrosser_> iirc there is glean instead of cloud-init?
15:50:48 <noonedeadpunk> oh. pf. yes
15:51:02 <noonedeadpunk> though I thought it's only for latest fedora...
15:51:14 <noonedeadpunk> But maybe it's indeed since 34...
15:51:40 <NeilHanlon> hrm yeah I see glean in the bootup logs
15:54:18 <NeilHanlon> ... i think the documentation on config drives is just a loop
15:56:22 <jrosser_> should be `openstack server create --use-config-drive ....`
15:57:01 <NeilHanlon> yeah was trying to launch in qemu but seems I'd need to make my own ISO with the right data.. probably easier to just do in openstack
16:00:26 <noonedeadpunk> #endmeeting