Thursday, 2024-01-25

jrossernoonedeadpunk: disabling tempest means we don’t get a public network in AIO07:20
jrosserthat sounds like a good use for openstack_resources - but interesting question if creating that network should be in the tempest role, or somewhere else07:22
noonedeadpunkjrosser: yeah, I'm due to create a playbook for using the role independently08:17
noonedeadpunkhaven't looked there yet though08:17
noonedeadpunkI would leave tempest "as is", or well, as is but with https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/88974108:18
noonedeadpunknot to break expectations too much08:18
noonedeadpunkAs there're quite some assumptions in tempest that not worth moving outside of it I guess08:18
noonedeadpunkI'm also thinking if we can indeed improve images download part in some follow-up08:20
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: WIP - Bootstrapping playbook  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90217809:52
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add role to install and run sonobouy k8s validation tests  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90605409:52
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add playbook to run functional test of magnum capi driver  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90636109:52
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add hook playbook install and test magnum capi driver  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90636309:52
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: WIP - Bootstrapping playbook  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90217809:54
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add role to install and run sonobouy k8s validation tests  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90605409:54
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add playbook to run functional test of magnum capi driver  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90636109:54
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add hook playbook install and test magnum capi driver  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90636309:54
andrewbonneynoonedeadpunk: re: proxy protocol #906447, we haven't needed to add the VIP to the allowed addresses, but I note that Zed doesn't carry the management_address/ansible_host patch. We originally had to override it for that reason10:40
noonedeadpunkok, so that's valid patch in fact?10:45
andrewbonneyI'm not sure. In that case I'd have thought you'd still want the host management addresses rather than the VIP. It may be the patch covers a different case, but it's not one I've encountered10:49
noonedeadpunkyeah, I thought actually the same. And there were couple of cases already where haproxy used VIP for connections rather then it's mgmt ip11:01
opendevreviewMerged openstack/openstack-ansible-ceph_client master: Align extra conf files mode  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/90603013:01
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Add variable to prevent tempest installation  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/90664113:03
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: WIP - Bootstrapping playbook  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90217813:03
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add role to install and run sonobouy k8s validation tests  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90605413:04
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add playbook to run functional test of magnum capi driver  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90636113:04
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add hook playbook install and test magnum capi driver  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90636313:04
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_magnum master: Add job to test Vexxhost cluster API driver  https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/90519913:04
opendevreviewMerged openstack/openstack-ansible-os_glance master: Fix iteration over backends config  https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/90604813:15
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_glance stable/2023.2: Fix iteration over backends config  https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/90649113:22
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client stable/2023.2: Align extra conf files mode  https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/90649213:23
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-os_tempest master: Add variable to prevent tempest installation  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/90664116:11
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: WIP - Bootstrapping playbook  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90217816:12
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add role to install and run sonobouy k8s validation tests  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90605416:12
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add playbook to run functional test of magnum capi driver  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90636116:12
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-ops master: Add hook playbook install and test magnum capi driver  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/90636316:12
admin1what is the oldest osa i can install ? 16:38
admin1that you guys know of still works ( due to repos and packages still being available ) 16:39
mgariepywhat version do you need?16:42
admin1ocata 16:44
admin1though i feel the rbac rules have been the same in all openstack versions until the  more recent ones .. this is to  test some rbac rules 16:45
mgariepyhmm. that would be a challenge i guess.16:45
jrosseryou would really have to try it, and fix it as you go16:46
mgariepyrabbitmq and galera will probablement need some work.16:46
admin1this is for rbac rules testing especially on keystone  16:46
admin1so i don't even need nova or neutron to be up 16:47
admin1on that note, what is the oldest we know might work out of the box16:47
jrosseri don't think ocata supports a metal deploy either which might be the most likley to work16:48
jrosserthough i think that might not be so true for older releases16:49
jrosserlxc images are probably going to need some effort/fixing too for that old16:49
jrosserwallaby CI looks broken16:51
jrosserbut xena looks ok https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fopenstack-ansible&branch=stable%2Fxena&pipeline=periodic&skip=016:51
admin1thanks jrosser for the link .. now I know how to check :) 16:55
admin1in which tag might we be able to use the magnum capi driver ? 17:13
admin1then i have to upgrade a lot of clusters to that version .. 17:13
admin1and what versions can check-pick it .. 17:13
jrosseradmin1: i have it running in a lab running antelope17:18
jrosserbut i am happy to apply *tons* of patches there to make it work17:19
jrosserif you want an "out of the box" experience i think for OSA it will be caracal /2024.117:21
jrosserhaving said that it would really be very nice if there was some testing of this beyond myself beforehand17:21
admin1i have around 10 diff versions of clusters on diff tags where i can apply to test ( in prod ) 17:31
admin1i have 28.0.1 running where i can test 17:32
admin1i need some guidiance on the patches to apply/cherry-pick and how to apply them 17:33
jrosseradmin1: it all starts from here https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/90519917:38
spatelI am running capi in production with kolla-ansible and recently I took trove to production. its fun.. 18:00
spatelOSA it little complicated when I think of doing trove and capi stuff 18:01
jrosserwell I take the long path18:01
jrosserit is not comparing equivalent architectures really18:02
jrosserand to make a useful CI job is really quite large effort18:04
noonedeadpunknot sure what is hard in trove with osa though18:06
noonedeadpunklike it's running in background for years now here18:06
noonedeadpunkBUT, trove is jsut broken when it comes to clustering18:07
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Slighly simplify primary node redeployment  https://review.opendev.org/c/openstack/openstack-ansible/+/90675018:07
noonedeadpunkit's known not to work18:07
noonedeadpunkspatel: I assume you're having separate rabbitmq cluster for trove?18:07
spatelNo18:10
spatelI have everything running on same rabbitMQ for all openstack components.. 18:10
spatelI did some networking magic do trove-guest-agent can talk to controller node.. 18:11
spatel:)18:11
noonedeadpunkyou not it's not really safe, do you ?:)18:12
noonedeadpunkin terms - given you can escape docker (which you can from time to time), with this rabbitmq cluster you can actually intercept any messages 18:13
jrosserthat is 8-O18:13
noonedeadpunkAnd there were quite nice showcases on how to gain admin with that back in the days18:13
noonedeadpunklike all you need is to have rabbitmq access18:14
noonedeadpunkand you pass that basically to tenant vms18:14
jrosserthis is kind of what I mean about architecture18:14
noonedeadpunkyeah18:14
jrosseryou can make anything work in a simple environment like devstack18:14
jrosseror if you just bridge/route “all the networks”18:15
noonedeadpunkand this is totally fine for small basement project to run for years18:15
jrosserbut that is a truly gigantic attack / risk surface18:15
jrosserthe reason things are sometimes “difficult” with OSA is that the architecture aims to prevent bad things happening, by design18:16
spatelWhy and how someone gain access ?18:16
jrosserbut sadly some services like trove are not built really with real environments in mind18:16
noonedeadpunkwhy is really a good question :D18:17
spatelTrove VM going to run in service project and nobody can access VM. 18:17
spatelEnd user only get access of DB. that person has no access of VM or SSH or anything 18:17
noonedeadpunkYou gain access to rabbitmq to this VM18:17
noonedeadpunkAnd kinda access to it resides inside a VM18:18
noonedeadpunkAnd you give some kind of access to VM to users18:18
spatelHow they will get access of rabbitMQ from where? you are saying from Trove VM ?18:18
noonedeadpunkAnd in some DBs you can run commands with mysql queries18:19
noonedeadpunkhttps://paste.openstack.org/show/bSzr1UA2aa77dXAmZjYx/18:19
spatelYou are saying mysql privilege escalation method.. (someone use mysql query to run shell level of command right? )18:20
noonedeadpunkSo it's jsut matter to escape docker container :D18:20
noonedeadpunkYeah, and you can get root to the DB in trove18:20
noonedeadpunkwhatever18:20
noonedeadpunkLike I still think we have that for a good reason: https://docs.openstack.org/openstack-ansible-os_trove/latest/configure-trove.html#use-stand-alone-rabbitmq18:21
spatelSame goes to Octavia :)18:21
noonedeadpunkno, not at all18:21
spatelsomeone can exploit haproxy and gain access of shell and access using lb-mgmt-net :)18:21
noonedeadpunkyou don't have rabbit talking on this network18:21
noonedeadpunkand through rabbit you can intercept/inject whatever you want18:22
noonedeadpunkIt's alike admin access to your cloud18:22
spatelNot rabbitMQ but person can gain access of system or exploit octavia-manager service... 18:22
noonedeadpunkhow?18:22
spatelSmart dude can exploit anything :) 18:22
noonedeadpunkmeh18:22
jrossernot really18:22
johnsomSpatel no, haproxy is in a network namespace that has no access to the lb-mgmt-net18:23
noonedeadpunkmTLS there for reason:)18:23
jrosserconnection is octavia to amphora18:23
jrosserother way round -> good design18:23
jrossertrove is all backwards18:23
noonedeadpunkand that ^18:23
spatelindeed.. 18:23
spatelbut lets say if we deploy dedicated rabbitMQ then we need to wire it up with all other trove components 18:24
johnsomTrove is a very different design with issues18:24
spateljohnsom welcome john (octavia keyword must brought you here )18:24
noonedeadpunkyes. but at least you're not giving control to rest of your openstack infrastructure - it's more or less isolated network without and isolated empty rabbit18:26
spatelI don't understand how does isolated rabbitMQ will work.. 18:27
spatelTrove components and trove-guest-agent will talk to isolated rabbitMQ  great!! but how openstack other components will talk to trove ?18:28
jrosserit’s a separate cluster that you *do* allow the trove parts to see18:28
jrosserso new interface on trove container and extra rabbit container, to put this in an OSA context18:28
jrosserand then that is a provider network18:29
spatelLet me create isolated rabbitMQ with trove and give it a try.. I am very interested in that design 18:29
jrosserthis all started with “architecture” for capi18:29
noonedeadpunkThey don't need to18:29
spatelnoonedeadpunk ?18:30
noonedeadpunkServices between each other talk only via API18:30
noonedeadpunkRabbit is used only for inside of the service18:30
noonedeadpunkALSO18:30
jrosserit’s totally possible to just throw a k8s up randomly and make it work as quick as you can18:30
spatelReally? trove doesn't use RabbitMQ to talk to openstack components.. 18:30
noonedeadpunkIIRC Trove does have a setting for using other messaging url for it's VMs18:30
noonedeadpunkNothing use rabbitmq for cross-component18:30
noonedeadpunkCross-component only API18:30
noonedeadpunkNova -> Cinder = API, Octavia -> Nova = API, etc18:31
noonedeadpunkSo you can (and maybe even should) have a rabbitmq cluster per service18:31
noonedeadpunkOr at least I guess that's how helm does18:31
noonedeadpunkor anyway - quite some ppl just have small 1 rabbitmq cluster per service18:32
noonedeadpunkLike Mirantis had most of their deployments this way18:32
noonedeadpunkRabbit fails - scrap it, just 1 service affected18:32
spatelI can create single node rabbitMQ for trove running on small VM :)18:32
noonedeadpunkdrop/create container and you're good to go again18:32
spatelits not going to kill Trove 18:32
spatelYes.. let me try that method and see.. 18:33
spatelIts a good point.. 18:33
noonedeadpunkI guess it's we're jsut lazy of doing big cluster for rabbit by default.18:35
noonedeadpunkBut it's totally possible to do plenty of small ones as well18:35
spatelDo you guys running 3 node cluster for rabbit or small pieces 18:47
noonedeadpunkWe run just 1 big but kinda lazy to change that18:52
spatelhaha19:03
noonedeadpunkI mean - that would involve training, change of monitoring, rewriting plenty of docs... meh19:06
noonedeadpunkBUT, for Trove we have separate cluster19:06
spatelI will deploy or try separate cluster19:39
spatelnoonedeadpunk around?20:19
spatelI have question related Ceph S3 implementation 20:20
spatelHow do you expose S3 to Public network? 20:20
noonedeadpunkthrough haproxy?20:20
spatelAre you using some kind of LB ?20:20
noonedeadpunkthough different one20:20
admin1via haproxy 20:20
admin1i use it via haproxy overrides20:21
noonedeadpunk(not same hardware as for control planes)20:21
admin1yes20:21
admin1its totally external ceph 20:21
noonedeadpunkbut using same role20:21
admin1the only thing that ties together is the haproxy and keystone20:21
spatelDo you run rgw service on dedicated nodes or with shared nodes 20:21
spatelI have dedicated Ceph cluster (not attached to openstack)20:21
spatelI am thinking to run rgw on same mon nodes (I have dedicated 3 nodes for mon) 20:25
noonedeadpunkwell.... that depends on traffic20:29
noonedeadpunkI think we run rgw together with haproxy and mds20:29
noonedeadpunkbut not with mon/mgr20:29
spatelnoonedeadpunk got it.. 20:31
spatelnoonedeadpunk how do you wire it up with keystone ?20:31
noonedeadpunkthrough public network20:32
noonedeadpunkor well "public"20:32
spatelI meant how does ceph talk to keystone 20:32
spatelDoes ceph has option to integrate with keystone?20:32
noonedeadpunkyes, sure20:32
jrosserspatel: you can study the osa AIO for all of this info :)20:33
noonedeadpunkhttps://docs.ceph.com/en/latest/radosgw/keystone/20:33
spateljrosser ofc.. just trying to get info as fast as possible :) 20:33
jrosserwe run rgw on the mgmt network and have extra haproxy20:34
jrosserthen rgw>keystone on the mgmt network/internal vip20:34
spatelI am first time deploying rgw so not sure how to architecture also I have no idea how much traffic it will bring 20:34
jrosserbut loads of ways to do it really20:34
spatelGot it. in-short make them reachable 20:35
spateldoes horizon has GUI or something to create rgw account / bucket etc.. 20:35
jrosseryou need to run the swift compatibility to make that work20:36
spatelNow swift coming in picture 20:37
jrosserso openstack cli / horizon uses Swift API against rgw to do all those things20:37
jrosserjust to be clear, you don’t need the openstack swift service20:37
spatelHmmm20:37
spatelokie!! 20:37
jrosseryou expose a swift compatible api from rgw, as well as s320:37
spatellet me google it and understand the workflow.. 20:38
jrosserwe did s3 static sites with it recently and that was interesting20:38
spatelstatic sites?20:39
admin1you setup ceph to use keystone for auth .. and then     you add the object storage service to endpoints .. then it will appear as object storage in horizon  .. and then can use the APIs  .. so ceph is completley indepdent of osa ,  only haproxy and keystone  and horizon is used 20:39
admin1static sites work good :) 20:39
admin1upto zed release,   i use to have my endpoints as id.domain.com,  s3.domain.com etc ..  then after zed, something changed in the way we use overrides , so now not anymore in new ones20:40
jrosserneed a bit of thinking about if to enable the multi tenancy setting in rgw20:40
jrosserthat made static sites a little tricky for us but we made it work eventually20:41
admin1spatel, be prepared to get logged out of horizon countless times until you get the keystone settings correct :D20:41
spatelsorry but what is static sites means? 20:41
admin1it means you run your whole webite ( css, html, java,  SPA) direct from s3 20:42
jrosserhttps://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html20:42
spateloh20:42
spateladmin1 what do you means countless logout? that is because of integration with ceph?20:43
admin1well, when you do the keystone setting in ceph , ceph will authenticate via keystone .. so if any setting is not correct, when you click the object storage link in horizon and then it will log you out 20:52
admin1you will know it when you see it :) 20:52
spatelDo I need to integrate entire ceph with keystone or just rgw service?20:53
spatelI am running cephadm 20:53
admin1i have one all in all cluster  ..  3 nodes ..      where each node is ceph everything ( mon, osd, mgr) and openstack everything ( compute ,controller, network ) 20:53
spateldo you have any doc or blog for rgw integration with keystone 20:53
admin1no but there is some youtube video 20:54
jrosserthe osa code does this20:54
admin1it does ? 20:54
admin1even if the ceph is external ? 20:54
jrosseryes20:54
admin1it never did in mine 20:54
admin1and i have like half a dozen different osa + ceph running 20:54
admin1what setting does it 20:54
admin1how can it go into cephadm docker container and make it work ? 20:55
admin1new ones = cephadmin where the docker does not allow ssh 20:55
spatelI have dedicated ceph cluster running cephadm :(20:55
admin1spatel, i have multiple of those ..  dedi ceph clusters via cephadm20:56
admin1but it works20:56
jrosserhttps://github.com/openstack/openstack-ansible/blob/master/playbooks/ceph-rgw-keystone-setup.yml20:56
admin1jrosser, this does not touch cephadm20:56
jrosserosa does not understand cephadm, never has20:56
spatel:(20:58
jrosserI think I gave you the keystone setup you need20:58
spatelcephadm is very cool and I am deploying all my deploying using cephadm 20:58
jrossernot the rgw settings for keystone20:58
jrosserthere are parts on both sides20:58
spatelI can take example and try to integrate with it 20:58
admin1spatel => https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/CRYFR777VKQLYLXPSNJVBOKZRGXLXFV6/20:59
admin1i use this as reference 20:59
admin1in the new haproxy .. 28.0.1 for example, how do I override endpoint such that keystone is on https://id.domain.com/  (  21:01
admin1old method was => haproxy_horizon_service_overrides:  haproxy_frontend_raw:  - acl cloud_keystone hdr(host) -i id.domain.com .. - use_backend keystone_service-back if cloud_keystone21:02
admin1this method not work since zed if i recall21:03
spateladmin1 This is helpful 21:04
spatelspecially for cephadm 21:04
admin1yes .. all new ones are cephadm based21:11
admin1spatel, do you have blog for trove ? 21:33
admin1is there postgresl ? 21:33
admin1postgres rabbit and redis will suppliment k8s cluster nicely 21:34
spatelTrove support all kind of DB.. mysql, postgres, mangodb, redis etc...21:34
spatelI have plan to create trove blog.. but I want to try one more experiment which is dedicated rabbitMQ node for trove as per today discussion 21:35
spatelcalling you.. hope its not later there 21:36
admin1i am in another call 21:36
admin1give me  5 mins 21:37
admin1will call back21:37
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/87879422:02

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!