| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Gather generic masakari facts https://review.opendev.org/c/openstack/openstack-ansible/+/880606 | 04:30 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Gather generic masakari facts https://review.opendev.org/c/openstack/openstack-ansible/+/880607 | 04:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Gather generic masakari facts https://review.opendev.org/c/openstack/openstack-ansible/+/880608 | 04:30 |
| jrosser | morning | 05:58 |
| jrosser | we should totally describe a small home lab / office lab config in the docs | 05:59 |
| jrosser | the network setup is really not clear and easy to make a big mess with multi-homing / reverse-path-lookup trouble | 06:00 |
| jrosser | why does `playbooks/os-neutron-install.yml --tags neutron-config --limit neutron_server` do a bunch of python_venv_build stuff, thats odd | 06:27 |
| noonedeadpunk | NeilHanlon: yup, that's correct | 07:11 |
| noonedeadpunk | CI is not failing as we pull latest from zuul, but if you deploy a sandbox it will get version from ansible-role-requirements | 07:12 |
| noonedeadpunk | I think we have some bumps for reviews | 07:12 |
| noonedeadpunk | because we do include role and have tag always... So it's running tasks that also have tag always inside python_venv_build... | 07:13 |
| jrosser | those `always` tasks depend on this being defined https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/neutron_install.yml#L74 | 07:25 |
| noonedeadpunk | oh, wait, we import_role there | 07:26 |
| jrosser | :) | 07:26 |
| noonedeadpunk | without tags.... | 07:26 |
| jrosser | well i just tried to update some neutron server config stuff and it all blew up in a wierd way | 07:27 |
| noonedeadpunk | this really drives me up the wall | 07:27 |
| noonedeadpunk | Oh.... so despite it should run only with tag neutron-install, since imports are (now?) processed before role.... | 07:28 |
| noonedeadpunk | /o\ | 07:28 |
| noonedeadpunk | I bet replacing with include will just sort that out in this case | 07:28 |
| noonedeadpunk | I feel like we'd need to do couple of things in nearest future - replace all modules with fqdn names and with that thoroughfully review what and why we are using for imports/includes and tags | 07:30 |
| jrosser | yeah - and i think we have to write it in our developer docs | 07:30 |
| jrosser | because its such a mess | 07:31 |
| noonedeadpunk | I'm not sure I have full understanding/overview of all possible corner cases to be frank | 07:34 |
| noonedeadpunk | And obviously smth has changed with behaviour quite lately IMO | 07:35 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Add example networking-generic-switch user role for Arista switch https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/880797 | 07:37 |
| noonedeadpunk | damiandabrowski: would be awesome to land this: https://review.opendev.org/q/topic:bump_osa+status:open | 07:48 |
| jrosser | noonedeadpunk: maybe we need some test case include/import code actually in the repo | 08:06 |
| jrosser | like self contained examples that show things | 08:06 |
| noonedeadpunk | we actually need to test tags I assume. | 08:06 |
| noonedeadpunk | after cleaning facts | 08:06 |
| noonedeadpunk | but again I'm not sure how to do that without reincarnating test repo | 08:07 |
| noonedeadpunk | as majority of roles will fail without keystone and infra being present | 08:07 |
| jrosser | i really meant just standalone manual things | 08:09 |
| jrosser | just to demonstrate what is / is not supposed to happen | 08:09 |
| jrosser | because i'm also pretty much not having a full understanding either | 08:09 |
| noonedeadpunk | ah | 08:11 |
| noonedeadpunk | Well, the thing is that right now we still launching such things with functional jobs/tests repo | 08:12 |
| noonedeadpunk | but maybe we could expand vartests scenario or smth | 08:13 |
| noonedeadpunk | Hm, looking at https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/test-vars-overrides.yml I wonder what point of having this at all... | 08:15 |
| damiandabrowski | noonedeadpunk: regarding https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/880781 | 09:03 |
| damiandabrowski | i tried using meta: clear_facts to fix this issue and it doesn't help. clearing facts for localhost and haproxy_all does not help and clearing facts for all hosts breaks playbook execution | 09:04 |
| noonedeadpunk | and https://paste.openstack.org/show/bLHdaI0NjQy229dZ9gZQ/ is not working either I assume? | 09:05 |
| damiandabrowski | nope :/ | 09:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Do not limit IP prefix for DHCP rule https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/880804 | 09:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Change default CIDR for security_group https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/880544 | 09:51 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Add example networking-generic-switch user role for Arista switch https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/880797 | 09:56 |
| admin1 | i need some direction .. i have disabled internal ssl in horizon, so if i do http://<internal-ip-of-horizon> , they all load fine all the time .. but when I do the same via https://domain.com ( going via haproxy) , i 503 Service Unavailable No server is available to handle this request. | 10:46 |
| admin1 | 3 controllers, so 3 horizon containers .. doing http:// directly to the 172.29.236 never fails | 10:46 |
| jrosser | admin1: 503 service unavailable from haproxy means it does not think the backend is up | 10:48 |
| admin1 | horizon back in haproxy stats shows all green | 10:48 |
| admin1 | image paste: https://pasteboard.co/vW6VJR7Gr2eE.png | 10:49 |
| jrosser | tbh i am not really understanding "disabled internal ssl" at all | 10:51 |
| noonedeadpunk | but internal ip goes also through haproxy | 10:51 |
| noonedeadpunk | are you sure that DNS is correct ?:) | 10:51 |
| jrosser | admin1: right now there is really no SSL at all haproxy<>horizon unless you have done something really specific to enable that | 10:52 |
| admin1 | haproxy horizon section -> https://gist.githubusercontent.com/a1git/90d9b40c4d6f6313854abab6746b4270/raw/cd5c373c1f7c4c01fcf87f944a4fb4bd0dd3184e/gistfile1.txt | 10:53 |
| jrosser | but what do you mean "disabled internal ssl in horizon" | 10:54 |
| noonedeadpunk | I think for internal vip | 10:54 |
| admin1 | horizon_enable_ssl: false | 10:54 |
| admin1 | http://horizon-mgmt-lxc-ip was redirectiing to https:// , removed that one so that they work on the mgmt network on direct 80 | 10:55 |
| noonedeadpunk | I'm not sure though horizon should be working through internal IP at all to be frank... | 10:55 |
| admin1 | http://172 ip just works fine | 10:55 |
| admin1 | it is what haproxy is proxying, | 10:55 |
| admin1 | but in my case, its 503 and i have not been able to figure out why | 10:56 |
| noonedeadpunk | I think it might be STS | 10:56 |
| jrosser | but the apache server in the horizon container needs to be told if the user beyond haproxy is accessing as https or not i think? | 10:56 |
| jrosser | it's not as simple as just saying it is or is not https to haproxy | 10:56 |
| noonedeadpunk | It should detect by X-Forwarded-Proto | 10:57 |
| noonedeadpunk | that's part of frontend config | 10:57 |
| damiandabrowski | and its logic was a bit weird...I fixed it a couple of days ago in master | 10:58 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879514 | 10:58 |
| noonedeadpunk | But in case https is disabled, hsts should be disabled as well | 10:58 |
| jrosser | but here https://github.com/openstack/openstack-ansible-os_horizon/blob/stable/zed/templates/openstack_dashboard.conf.j2#L9 | 10:58 |
| noonedeadpunk | um... how that's gonna work at all... behind haproxy... | 10:59 |
| jrosser | on Zed, all through the logic that configures this there is `horizon_enable_ssl` and `horizon_external_ssl` and those are not the same thing | 10:59 |
| jrosser | horizon_enable_ssl is basically bogus and legacy thing for putting your own cert on the horizon container afaik | 11:00 |
| noonedeadpunk | we're redirecting http>https on haproxy level... | 11:00 |
| noonedeadpunk | AH, if you've disabled tls and did not run horizon role then yeah, it will jsut redirect to 443 that's not listening anymore | 11:01 |
| admin1 | i am happy if only haproxy is doing ssl termination and internal services are on non-https .. this means only 3 servers ( controllers ) are doing the ssl transaction instead of 20x3 = 60 containers each doing their own ssl transaction | 11:01 |
| jrosser | admin1: which release are you using? | 11:02 |
| noonedeadpunk | this part completely looks legacy and not needed to me https://github.com/openstack/openstack-ansible-os_horizon/blob/stable/zed/templates/openstack_dashboard.conf.j2#L9-L16 | 11:02 |
| admin1 | 6.0.1 had no issues.. i upgraded to 6.1.0 and getting this issue | 11:02 |
| jrosser | Zed? please forget everything about 60 containers doing ssl | 11:03 |
| jrosser | none of this is relevant at all until antelope | 11:03 |
| admin1 | zed :) | 11:03 |
| jrosser | what we work on today in master branch is not part of Zed release | 11:03 |
| admin1 | i destroyed the containers are rebuit, had no effect | 11:04 |
| admin1 | i can rm -rf the /etc/ansible/roles, run it again, destory the repo servers and try to force a build | 11:04 |
| noonedeadpunk | I don't really see differences in 26.0.1 and 26.1.0 that could affect that | 11:06 |
| noonedeadpunk | I'm completely lost now | 11:08 |
| jrosser | likewise | 11:09 |
| noonedeadpunk | so http://<internal-ip-of-horizon> - works, https://<public-fqdn-of-horizon> - does not, https://<public-ip-of-horizon> - ? | 11:09 |
| jrosser | admin1: tbh this sounds like you have some incorrect override made | 11:09 |
| jrosser | also i never access horizon on the internal endpoint so could not say if this does/doesnt work | 11:10 |
| admin1 | variables for this: https://gist.githubusercontent.com/a1git/c41ec803bf7a6964e3c7b7a2acab388e/raw/bf073a4f04011e4f9df4cd7bf78402f02508d4f5/gistfile1.txt | 11:10 |
| admin1 | i did ssh deploy -D 6000, where 6000 is socks, then use foxyproxy to have browser use 6000 for all browsing = can access all internal ips and see how it works /behaves | 11:11 |
| admin1 | i can validate that it works: https://pasteboard.co/lJHAUC47Y1Xp.png | 11:12 |
| jrosser | i am not sure that using `horizon_enable_ssl: false` is valid thing to do | 11:15 |
| noonedeadpunk | admin1: I think you will need to override haproxy_horizon_service | 11:16 |
| noonedeadpunk | As horizon_enable_ssl is indeed not used for haproxy config | 11:16 |
| noonedeadpunk | So basically https://opendev.org/openstack/openstack-ansible/src/branch/stable/zed/inventory/group_vars/haproxy/haproxy.yml#L227-L228 should be just False | 11:17 |
| jrosser | ? | 11:17 |
| jrosser | but what about the frontend? | 11:17 |
| noonedeadpunk | But horizon_enable_ssl I guess assume no ssl will be used at all? | 11:17 |
| noonedeadpunk | but I'm not really understanding what is trying to be achieved right now | 11:18 |
| noonedeadpunk | If fully disable tls for horizon - then that's the way? | 11:18 |
| admin1 | it disables the internal redirection from http://horizon-mgmt-lxc-ip to https://horizon-mgmg-lxc-ip which just does not load in the browser at all | 11:18 |
| jrosser | admin1: can you be more precise please - this is very confusing | 11:20 |
| jrosser | do you mean the redirect in haproxy, or one in the horizon apache? | 11:20 |
| admin1 | ok .. let me walk through the scenairo step by step .. 1. cluster upgraded from 6.0.1 -> 6.1.0 .. https://cloud stopped to work (horizon) .. checked haproxy stats .. all green .. logged into each of the horizon srevers and did ss -plant .. all listening on 80 and 443, no errors .. .. tried curl http://ip .. it redirected to https://ip | 11:24 |
| admin1 | .. did curl https://ip gives curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number .. my thought was that https is using some wierd cert so haproxy <-> horizon is not working, so pasted here .. jrosser gave me the link | 11:24 |
| admin1 | https://github.com/openstack/openstack-ansible-os_horizon/blob/stable/zed/templates/openstack_dashboard.conf.j2#L9 which allows me to disable the internal https redirection .. did that, could access and login to each/all of horizon dashboard without fail and issues .. but https://cloud ( via haproxy) still give 503 | 11:24 |
| jrosser | i was not telling you to use that variable | 11:25 |
| jrosser | i intended to show what was configuring the redirect in the horizon apache | 11:25 |
| admin1 | at least that helped me to know that horizon is not in error and i can login and work properly and the issue lies in haproxy -> horizon | 11:26 |
| admin1 | otherwise 503 Service Unavailable via haproxy and "172.29.239.156 sent an invalid response. ERR_SSL_PROTOCOL_ERROR" via https would have pointed to non-existent issues on the horizon side | 11:27 |
| jrosser | have you looked at the generated apache config? | 11:28 |
| jrosser | i don't think i can help here really | 11:30 |
| admin1 | https://gist.githubusercontent.com/a1git/be2f283b53361bacae6345820d6a98a6/raw/d780106e23390d93dd6dd93543f94417469be110/gistfile1.txt | 11:30 |
| jrosser | best to make an AIO with a competely standard config and compare what you see there | 11:30 |
| admin1 | i don't see an issue here | 11:30 |
| admin1 | in the apache .. as horizon works perfectly fine in this | 11:31 |
| jrosser | but you say this `curl https://ip gives curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number .. my thought was that https is using some wierd cert so haproxy <-> horizon is not working` | 11:32 |
| jrosser | thats the IP of the horizon container? | 11:32 |
| admin1 | yes | 11:32 |
| noonedeadpunk | could you be using tlsv1.1? | 11:33 |
| admin1 | by default 80 -> 443 is enabled for internal ip in apache.conf, and curl gives that .. https:// in the browser also does not work | 11:33 |
| noonedeadpunk | or some override for that? | 11:33 |
| jrosser | admin1: there is no redirect in the apache config you pasted | 11:33 |
| admin1 | yes, because i removed it .. | 11:33 |
| noonedeadpunk | we have protocols defined here https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/ssl.yml#L19 | 11:34 |
| admin1 | noonedeadpunk, this is the generated config -> https://gist.githubusercontent.com/a1git/90d9b40c4d6f6313854abab6746b4270/raw/cd5c373c1f7c4c01fcf87f944a4fb4bd0dd3184e/gistfile1.txt | 11:34 |
| noonedeadpunk | admin1: yes, so VIP must be accessed through SSL both internal and external according to this | 11:34 |
| jrosser | the curl error is what happens when you point a browser at something on port 443 which is actually http, not https | 11:35 |
| noonedeadpunk | I think more vice versa? | 11:35 |
| noonedeadpunk | ah, no, you're right | 11:36 |
| admin1 | i can try to rm -rf the /etc/haproxy, remove the ssl override and re-run the haproxy playbook again | 11:37 |
| jrosser | imho if the horizon apache config was generated with a redirect then that says that the config is wrong | 11:37 |
| jrosser | if the variables are set up properly to make that backend be http, then the redirect would not be there | 11:37 |
| admin1 | but i pasted the whole variables and the generated configs | 11:37 |
| noonedeadpunk | I think this kinda all heading to some weird direction | 11:37 |
| admin1 | i have no overrides for ssl or haproxy | 11:38 |
| noonedeadpunk | Redirect is useless imo, since haproxy still uses 80 port | 11:38 |
| noonedeadpunk | redirect close to never should get uitlized, except direct access to horizon backends? | 11:38 |
| jrosser | right - but it totally confuses when you have socks tunnel into mgmt network then try to debug with a browser | 11:38 |
| noonedeadpunk | yeah. as I said, I don't think horizon even supposed to be working on internal network | 11:39 |
| admin1 | how else to check if horizon is running good .. curl gives curl: (35) error:0A00010B:SSL routines::wrong version number | 11:39 |
| jrosser | ^ you are accessing something as https which is not actually https | 11:39 |
| jrosser | port 443 is serving plain http in that case | 11:40 |
| admin1 | you mean for the curl ? | 11:40 |
| jrosser | yes | 11:40 |
| jrosser | just becasue it is on port 443 does not mean it is https | 11:40 |
| jrosser | only if the web server config makes it be https | 11:40 |
| admin1 | if you guys have any zed, login to any horizon container .. ss -plant you will see apache on 80 and 443 .. if you do curl -I http://ip:80 , it will redirect to https://ip/auth/login/?next=/ if you do curl on that ( which horizon is asking ) it will give that error | 11:41 |
| admin1 | so is the default behavior wrong ? | 11:41 |
| noonedeadpunk | um, apache on https is wrong, yes, but it never used by haproxy | 11:41 |
| noonedeadpunk | so it should not cause issues by default | 11:41 |
| noonedeadpunk | let me quickly add horizon to sandbox deployment | 11:43 |
| admin1 | i am going to rm -rf my /etc/ansible/roles and rerun the playbooks | 11:43 |
| noonedeadpunk | Not sure how this can help, but feel free | 11:44 |
| admin1 | noonedeadpunk, have to try to solve it somehow | 11:44 |
| admin1 | prod affected :( | 11:45 |
| noonedeadpunk | doing random action unlikely to help | 11:47 |
| noonedeadpunk | how one old belgian friend said - less haste more speed | 11:47 |
| admin1 | well, i submitted my variables, generated configs, also screenshots that horizon is in fact working , but got no ideas .. so have to try something .. | 11:48 |
| noonedeadpunk | admin1: ok, so I don't have redirect in apache by default | 11:48 |
| admin1 | are you in 6.1.0 ? | 11:48 |
| jrosser | i don't either | 11:48 |
| admin1 | i have one is 26.0.1 and i have redirect there also | 11:49 |
| admin1 | in* | 11:49 |
| noonedeadpunk | I still not understanding what exact setup of horizon you want. What I got is that you're trying to solve issue that horizon gets SSL error when accessing public FQDN | 11:49 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Use include instead of import for conditional tasks https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/874949 | 11:49 |
| admin1 | when I do https://cloud.domain.com, i want horizon to load | 11:50 |
| noonedeadpunk | But not with checking what's wrong there but trying to disable SSL at all? | 11:50 |
| jrosser | admin1: again can you be precise, do you mean the redirect is in the apache config like this https://github.com/openstack/openstack-ansible-os_horizon/blob/9c07e79890692cb477005cf34139fd20d4417be7/templates/openstack_dashboard.conf.j2#L10-L15 | 11:50 |
| jrosser | admin1: or do you mean that with wget/curl you get a 302 regardless of the apache config | 11:50 |
| jrosser | these are not the same thing, but i have no idea which you mean | 11:50 |
| admin1 | jrosser, i do not have that | 11:51 |
| jrosser | what?! :) | 11:51 |
| admin1 | but curl http:// redirects to https:// | 11:51 |
| jrosser | ok ;) | 11:51 |
| noonedeadpunk | ok, that's correct, as this is what haproxy should do | 11:52 |
| jrosser | ^ i think he means inside the horizon container? | 11:52 |
| admin1 | yes | 11:52 |
| jrosser | sooo much confusion | 11:52 |
| admin1 | inside horizon | 11:52 |
| noonedeadpunk | ++ | 11:52 |
| admin1 | when you guys login to a horizon container , then do curl http://ip, what does it return ? | 11:53 |
| admin1 | curl -I ( to get the headers) | 11:53 |
| admin1 | it will return https://<internal-ip>/auth/login | 11:53 |
| noonedeadpunk | what if you try setting horizon_external_ssl: True? | 11:55 |
| admin1 | i will try again .. i have external_lb_vip_address -> cloud.domain.com .. .. when i do https://cloud.domain.com it gives 503 .. but if I check the stats page, all backend is green .. when I directly call the http://horizon-intenral-ip/ i am able to login fine | 11:56 |
| noonedeadpunk | But likely horizon_enable_ssl should be left as default... | 11:57 |
| noonedeadpunk | but horizon_external_ssl should be true by default | 11:57 |
| noonedeadpunk | Unless you have override for openstack_external_ssl | 11:58 |
| noonedeadpunk | so basically `horizon_enable_ssl` should be true as well I guess | 11:58 |
| admin1 | i have haproxy_user_ssl_cert and haproxy_user_ssl_key | 11:59 |
| admin1 | ok | 11:59 |
| jrosser | noonedeadpunk: are you sure? `horizon_enable_ssl` is backend ssl | 11:59 |
| noonedeadpunk | Um.... I don't see that in paste... | 11:59 |
| admin1 | line 8 and 9 | 12:00 |
| admin1 | is the only lines i have for ssl | 12:00 |
| noonedeadpunk | jrosser: I'm not... Yeah, probably just default | 12:00 |
| admin1 | so i am missing some required vars like horizon_external_ssl | 12:01 |
| damiandabrowski | jrosser: there's some information about that in commit msg: https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879515/4 | 12:01 |
| noonedeadpunk | admin1: horizon_external_ssl is default to openstack_external_ssl | 12:02 |
| noonedeadpunk | that is True by default https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/all.yml#L71 | 12:02 |
| noonedeadpunk | and yes, jrosseris right, horizon_external_ssl should be true while horizon_enable_ssl should be false | 12:03 |
| noonedeadpunk | aha, so I was looking at new "fixed" code | 12:04 |
| jrosser | yeah is all different (and odd) in Zed | 12:04 |
| noonedeadpunk | so override admin1has regarding horizon_enable_ssl: false is correct one | 12:04 |
| jrosser | imho the whole business of using socks to a browser looking at the horizon container is not valid | 12:05 |
| damiandabrowski | horizon_enable_ssl should be enabled by default in Zed and it's kind of strange but okay as long as horizon_external_ssl is True | 12:05 |
| jrosser | really? | 12:05 |
| noonedeadpunk | well, redirect won't be added at least | 12:05 |
| noonedeadpunk | and certs wont' be copied as well... | 12:05 |
| damiandabrowski | from https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879515 commit message: | 12:06 |
| jrosser | even without the redirect in the apache config (lets be precise!) there is still a 302 if you curl port 80 in the horizon container | 12:06 |
| damiandabrowski | This patch does not change current behavior in gating as backend TLS | 12:06 |
| damiandabrowski | works only with horizon_external_ssl=False(while it's set to True by | 12:06 |
| damiandabrowski | default). | 12:06 |
| damiandabrowski | so i don't mean that backend TLS will work if you set horizon_enable_ssl=True in Zed | 12:07 |
| damiandabrowski | I'm just saying you can have horizon_enable_ssl=True, backend tls will still be disabled(if you have horizon_external_ssl=True) and everything should work fine | 12:07 |
| noonedeadpunk | OK, I think goal#1 admin1 is to get rid of the redirect in apache by setting up horizon_external_ssl and horizon_enable_ssl "correctly" | 12:07 |
| jrosser | noonedeadpunk: try curl -v http://<ip>:80 in your horizon container | 12:08 |
| noonedeadpunk | so external one should be true, enable - dunno, aio works with default | 12:08 |
| noonedeadpunk | jrosser: I have bare metal sandbox... But it redirect to http://172.29.236.100/auth/login/?next=/ which is fine? | 12:09 |
| noonedeadpunk | but yes, it's 302 | 12:11 |
| noonedeadpunk | though it looks like valid one, and not to https | 12:11 |
| jrosser | https://paste.opendev.org/show/bWCJbfhDFv2UGh71zVpd/ | 12:11 |
| noonedeadpunk | https://paste.opendev.org/show/bT5hWF4Tc3okutnvxfbD/ | 12:12 |
| noonedeadpunk | ok, it should not redirect to https | 12:12 |
| damiandabrowski | noonedeadpunk: basically in Zed you should have two redirects i think | 12:12 |
| damiandabrowski | https://10.6.0.153 -> http://10.6.0.153/auth/login/?next=/ -> https://10.6.0.153/auth/login/?next=/ | 12:12 |
| damiandabrowski | so https -> http -> https | 12:13 |
| damiandabrowski | which is strange but works | 12:13 |
| damiandabrowski | IIRC it was fixed in https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/879514/5 | 12:13 |
| noonedeadpunk | jrosser: what do you have in /etc/apache2/sites-enabled/ ? | 12:13 |
| jrosser | https://paste.opendev.org/show/bOHqweRX25roNJOzlu8T/ | 12:14 |
| noonedeadpunk | brrrrrr | 12:14 |
| noonedeadpunk | so the only thing that can redirect this way is horizon itself | 12:15 |
| jrosser | yes | 12:15 |
| noonedeadpunk | or default vhost which I assume is absent | 12:15 |
| damiandabrowski | i need to leave for now, I can help later this evening if needed as I spent plenty of time on horizon's redirections recently... | 12:16 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Move online data migrations to post-restart step https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/880210 | 12:17 |
| noonedeadpunk | jrosser: but again, it should be "fine" once you access through haproxy | 12:17 |
| jrosser | yes it should | 12:17 |
| jrosser | this is why i think just hacking at things making it work on br-mgmt is not a good plan | 12:18 |
| jrosser | particularly if you want internal vip to be http | 12:18 |
| jrosser | or different from external anyway | 12:18 |
| admin1 | i am re-doing it .. maybe it will fix it | 12:19 |
| noonedeadpunk | I still think that original issue admin1 could have was realted to HSTS until things god messed up | 12:19 |
| noonedeadpunk | *got | 12:19 |
| noonedeadpunk | as then haproxy will result in error as well - not sure if 503 or not though | 12:20 |
| noonedeadpunk | it will show all backends as green, but until security policies are met - it won't let you through to them | 12:21 |
| noonedeadpunk | So I'd try setting `haproxy_security_headers_csp_report_only: true` and revert all other changes to be frank | 12:22 |
| admin1 | the reports, where does it log to noonedeadpunk, journalctl ? | 12:24 |
| noonedeadpunk | Um, browser console iirc | 12:25 |
| jrosser | this could also be some problem with the certificate to generate a 503 | 12:26 |
| jrosser | when the backends are up | 12:26 |
| noonedeadpunk | yeah, could be... | 12:27 |
| noonedeadpunk | but no, csp is blocked by browser iirc, so it won't be 503? | 12:28 |
| noonedeadpunk | yeah, and internal VIP should not work exactly due to CSP | 12:30 |
| noonedeadpunk | as we paste there public VIP rules | 12:30 |
| noonedeadpunk | also in case of some redirect loop in apache I think haproxy should mark backends as down. as we're checking for /auth/login/ and expect 200 | 12:33 |
| noonedeadpunk | 301/302 will mark backends as down | 12:34 |
| noonedeadpunk | Will it?:) | 12:34 |
| noonedeadpunk | nah, all 2xx and 3xx are valid | 12:35 |
| noonedeadpunk | so indeed it could be 302 to https :( | 12:36 |
| admin1 | i found this in haproxy .. [19/Apr/2023:12:37:05.466] base-front-1~ base-front-1/<NOSRV> -1/-1/-1/-1/0 503 237 - - SC-- 523/1/0/0/0 0/0 "GET /auth/login/?next=/ HTTP/1.1" | 12:38 |
| jrosser | you've deployed master branch | 12:38 |
| jrosser | thats not Zed | 12:38 |
| admin1 | doh ! | 12:38 |
| jrosser | bingo | 12:38 |
| jrosser | simplest answer is almost the most likley | 12:38 |
| jrosser | *always | 12:38 |
| admin1 | i rm -rf /etc/ansible , checkout 6.1.0 and re-ran the haproxy playbook thinking of this possibility | 12:39 |
| jrosser | 6.1.0 ? | 12:39 |
| admin1 | 26.1.0 | 12:39 |
| jrosser | did you bootstrap-ansible.sh ? | 12:39 |
| admin1 | yes :D | 12:39 |
| admin1 | i need to rm -rf the /etc/ansible from all controller and re-run it again i guess | 12:40 |
| admin1 | to completely remove the old one ? | 12:40 |
| jrosser | the haproxy `base` stuff is only present in master and will be in antelope | 12:40 |
| noonedeadpunk | only on deploy host | 12:40 |
| * noonedeadpunk was wondering how you've spotted that... | 12:40 | |
| noonedeadpunk | Well, this also means we have stuff broken in master ?:) | 12:41 |
| noonedeadpunk | or well, for master we have some patches that might be covering this... | 12:41 |
| jrosser | i think it's because dropping master on top of Zed leaves the old config for port 443 which would have shown the horizon backends up | 12:41 |
| admin1 | i suspected and i did a ps aux | grep openstack on all servers, and everything showed 26.1.0 .. but somehow haproxy is master :( | 12:42 |
| jrosser | but at the same time drops the base config which actually serves port 443 but has no backends | 12:42 |
| jrosser | or something like that | 12:42 |
| jrosser | admin1: no, not really | 12:42 |
| noonedeadpunk | yeah, so we might be lacking upgrade path right now | 12:42 |
| jrosser | there is not anything in the haproxy role that defines that `base` should exist | 12:42 |
| jrosser | thats all in the openstack-ansible repo variables | 12:42 |
| noonedeadpunk | or well, it could be master one day | 12:43 |
| jrosser | you have used master branch of the openstack-ansible repo at some point, and that has written config into /etc/haproxy/conf.d/ | 12:43 |
| admin1 | if i checkout a new branch based on 26.1.0, then remove the /etc/ansible, bootstrap all .. and then rm -rf /etc/haproxy from controllers and ran haproxy again, it should get me back to the default of 26.1.0 for haproxy ? | 12:43 |
| jrosser | and no matter what you do, how many playbooks you run, that is not going to get removed unless you fix it by hand | 12:43 |
| noonedeadpunk | I think it won't as there's nothing that would remove base | 12:43 |
| jrosser | this is why just delete/recreate containers / run playbooks without debugging is wasted time | 12:44 |
| noonedeadpunk | admin1: rm -rf /etc/haproxy/conf.d/ | 12:44 |
| admin1 | yes , that is what i am saying | 12:44 |
| noonedeadpunk | and then re-run haproxy role | 12:44 |
| admin1 | rm -rf /etc/haproxy* and re-run haproxy playbook should recreated the new configs right ? | 12:44 |
| noonedeadpunk | yes | 12:44 |
| admin1 | ok | 12:44 |
| admin1 | another thing i noticed .. out of 3 controllers. i from haproxy disabled 2 and enabled only 1 .. what i noticed was out of 6 tries, openstack page loaded 1time, 4 times was 503 and final time was a HUGE openstack logo with broken css .. all from the same single active backend | 12:46 |
| noonedeadpunk | wasn't that during playbook run when vip is failovering across controllers? | 12:50 |
| admin1 | i had stats page setup wtih pass to access .. it was after the run and everything stable, i disabled each backend 1 by 1 and enabled only 1 to check if its a specific container among the 3 that was causing 503 | 12:51 |
| admin1 | and i found out that even on same backend, with the remaining 2 being on MAINT, the result was not consistent | 12:51 |
| admin1 | 503 - works .. 503 .. 503 .. 503 .. broen css -- | 12:52 |
| admin1 | broken* | 12:52 |
| admin1 | hmm.. now i am thinking . what if someone forgot to do -b 26.1.0 and just checkout master as 26.1.0 ( i have zsh and it shows heads/26.1.0 ) in the branch | 12:56 |
| admin1 | would ps ax | grep openstack will still show 26.1.0 ( my local branch name) in the venv ? | 12:57 |
| admin1 | jrosser noonedeadpunk thanks .. fixed | 13:07 |
| admin1 | just for my understanding, what is the base all about ? are we adding multi domain support of some sort ? | 13:07 |
| jrosser | \o/ | 13:07 |
| admin1 | or white label type of options | 13:07 |
| jrosser | there is a large refactoring of the haproxy config in Antelope | 13:08 |
| jrosser | and the `base` config brings together letsencrypt, security.txt and horizon (all the things on port 443) in a much neater and more obvious way than they are at the moment | 13:09 |
| jrosser | it is also more extensible to make the kind of thing you have done with everything on port 443 very easy in the future | 13:09 |
| admin1 | that config is working fine in my sandbox .. | 13:10 |
| admin1 | everything neat and on ssl | 13:10 |
| jrosser | well like i say there will be significant changes in Antelope | 13:11 |
| admin1 | if only there could be a sub_filter to change api and domain urls on the fly, we can whitelist a single horizon to multiple domains :D | 13:11 |
| admin1 | all endpoints get sub_filter to diff domains as their own custom cloud .. | 13:11 |
| jrosser | haproxy cannot do any rewriting, if thats what you mean | 13:13 |
| admin1 | it can .. rspirep .. | 13:13 |
| admin1 | http-response replace-header now a days | 13:14 |
| admin1 | i meant , someday to see some white label features | 13:14 |
| admin1 | back to my question .. if I see this, /openstack/venvs/nova-26.1.0/ , does it mean nova is on 26.1.0 .. or is that 26.1.0 whatever my branch name is ? | 13:16 |
| admin1 | how do i validate that my install is not on master .. ( except haproxy) everything was(is) working just fine .. | 13:17 |
| noonedeadpunk | admin1: it's whatever venv_tag or openstack_release is | 13:19 |
| noonedeadpunk | it can be overwriten | 13:19 |
| admin1 | i do not have those overrides | 13:20 |
| noonedeadpunk | by default it's identified using pbr basically https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/bootstrap-ansible.sh#L150 | 13:21 |
| admin1 | if i checkout tag 26.1.0 and make my local branch 1.1.1, it will not show 1.1.1. there right ? | 13:21 |
| noonedeadpunk | no, it will show only version that was bootstrapped | 13:22 |
| admin1 | ok . then phew ! .. | 13:22 |
| admin1 | we have a habit of running haproxy first to ensure it binds to all the correct ports etc, so only this got master and others are safe and in the version they should be | 13:23 |
| noonedeadpunk | so if you do like cd /etc/ansible/roles/haproxy; git checkout master - version won't be affected | 13:23 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Bump OpenStack-Ansible Xena https://review.opendev.org/c/openstack/openstack-ansible/+/880478 | 13:28 |
| opendevreview | Merged openstack/openstack-ansible stable/xena: Gather generic masakari facts https://review.opendev.org/c/openstack/openstack-ansible/+/880608 | 13:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/wallaby: Bump erlang versions https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/876708 | 13:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/wallaby: Switch rabbitmq repo back to packagecloud https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/880324 | 13:41 |
| opendevreview | Neil Hanlon proposed openstack/openstack-ansible stable/zed: bump openstack_hosts role to resolve openvswitch3.1 problem on Rocky https://review.opendev.org/c/openstack/openstack-ansible/+/880826 | 13:42 |
| opendevreview | Merged openstack/openstack-ansible master: Drop `else` condition in the container_skel_load loop https://review.opendev.org/c/openstack/openstack-ansible/+/878696 | 14:04 |
| noonedeadpunk | damiandabrowski: jrosser I have a small problem with https://review.opendev.org/c/openstack/openstack-ansible/+/871189 and it's problem not regarding content, but how we want to merge things efficiently. | 14:15 |
| noonedeadpunk | and it's related to https://bugs.launchpad.net/openstack-ansible/+bug/2007296 | 14:16 |
| noonedeadpunk | So basically, what I'm planning to do, is create a directory for each group, instead of having simple files | 14:17 |
| noonedeadpunk | and move content of repo_packages https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/defaults/repo_packages there to cover this bug | 14:17 |
| noonedeadpunk | I'm thinking of creating a follow up patch, that would move haproxy to it's own file, group_vars that were present to another and repo_packages content to it's own, so that bump script was able to find the file by expected filename/path | 14:18 |
| noonedeadpunk | But I think I would need to have 2 patches to move just created haproxy configs to the directory, and then move repo_packges? | 14:20 |
| spatel | folks! we don't have release notes for 26.1.0? | 15:28 |
| spatel | https://docs.openstack.org/releasenotes/openstack-ansible/zed.html | 15:29 |
| noonedeadpunk | Looks like we didn't manage to write any | 15:32 |
| noonedeadpunk | At very least we had couple of bugs closed there | 15:32 |
| noonedeadpunk | but yeah. don't see any release notes :( | 15:33 |
| damiandabrowski | noonedeadpunk: "what I'm planning to do, is create a directory for each group" | 16:16 |
| damiandabrowski | do we really need to have directory for each group? I don't think we have that many variables defined | 16:16 |
| jrosser | the files are edited automatically by the version bump tool | 16:17 |
| noonedeadpunk | ^ this | 16:18 |
| jrosser | it is much much easier to write them out completely without having to worry about * other vars along for the ride | 16:18 |
| damiandabrowski | where can i find this tool? | 16:18 |
| damiandabrowski | https://github.com/noonedeadpunk/osa-bump-bot | 16:20 |
| damiandabrowski | are we talking about this one? | 16:20 |
| noonedeadpunk | https://github.com/noonedeadpunk/osa_cli_releases | 16:20 |
| noonedeadpunk | btw we should update links here https://docs.openstack.org/openstack-ansible/latest/contributor/periodic-work.html#osa-cli-tooling | 16:21 |
| noonedeadpunk | or better move utilities in code... | 16:21 |
| noonedeadpunk | well, regardless, it's very bad idea to edit yaml files that has some content, and especially since we need to render them with jinja | 16:22 |
| noonedeadpunk | like there's no way of them to be co-located with other variables | 16:22 |
| noonedeadpunk | so my question was mostly related to how orginize our work to kinda not block each other and at the same time not to push patches that will jsut re-arrage things... | 16:24 |
| damiandabrowski | to me it totally looks like it should be "fixed" in a bump tool, but if you both think that creating separate directories for each group then i trust you | 16:28 |
| damiandabrowski | i guess it would be the best if we can merge https://review.opendev.org/c/openstack/openstack-ansible/+/871189 now | 16:28 |
| damiandabrowski | yesterday it already had two +2s but I applied few really minor changes suggested by noonedeadpunk | 16:28 |
| noonedeadpunk | yeah and then we also have https://review.opendev.org/c/openstack/openstack-ansible/+/879085/15 | 16:29 |
| damiandabrowski | yes, I'll respond to your comment in a minute, i need to check one thing | 16:30 |
| jrosser | for all things like that don't we need to duplicate them in group_vars/all | 16:31 |
| damiandabrowski | i think if default value of the variable has some logic that may be required | 16:32 |
| damiandabrowski | or at least in group_vars, it doesn't need to be defined for all hosts in my opinion | 16:36 |
| damiandabrowski | we already have glance_default_store defined in inventory/group_vars/glance_all | 16:40 |
| damiandabrowski | so i think we can just add these lines to glance_all file | 16:40 |
| damiandabrowski | https://paste.openstack.org/show/bvVpcWgqxSLSj08vhPrT/ | 16:40 |
| jrosser | i do not think we can override things from vars/ | 16:41 |
| jrosser | like the one starting _ | 16:41 |
| noonedeadpunk | can or should? | 16:42 |
| noonedeadpunk | as we obviously can | 16:42 |
| noonedeadpunk | But most likely should not | 16:42 |
| noonedeadpunk | we can not override ones, that are in debian/redhat, since they will get overloaded on vars include | 16:43 |
| noonedeadpunk | but vars/main.yml can be overriden | 16:43 |
| jrosser | but role vars are higher precedence than group_vars? | 16:44 |
| jrosser | which is why we "reflect" them generally through defaults/ and prefix with _ as a reminder | 16:44 |
| noonedeadpunk | ah, yes, right, over group_vars they have prescedence | 16:45 |
| jrosser | yes sorry by override i didnt mean -e, that obviously wins | 16:45 |
| noonedeadpunk | oh, well, even over play vars.. huh | 16:45 |
| noonedeadpunk | hm, then it looks like https://opendev.org/openstack/openstack-ansible-ceph_client/src/branch/master/vars/main.yml#L19-L46 should not be there | 16:46 |
| noonedeadpunk | anyway | 16:46 |
| jrosser | ultimately it depends where we use it | 16:46 |
| jrosser | and if ever it is to be overridden | 16:47 |
| damiandabrowski | but haproxy config doesn't know anything about role defaults or variables as haproxy role is imported outside the role | 16:47 |
| jrosser | it's ok if we don't describe it in defaults/main.yml imho | 16:47 |
| noonedeadpunk | well, we have release notes like this https://opendev.org/openstack/openstack-ansible-ceph_client/src/branch/master/releasenotes/notes/move-gnocchi-component-118ae07fce3562e1.yaml | 16:47 |
| noonedeadpunk | like it makes sense to be defined per group then | 16:48 |
| noonedeadpunk | or maybe not as it's filtered anyway by groups... | 16:48 |
| damiandabrowski | additionally, only _glance_available_stores is defined in role vars and it only refers to other variables so it's not relevant IMO | 16:48 |
| noonedeadpunk | meh | 16:48 |
| jrosser | right - and i think there is confusion because we're in the mind of saying to users "put that in user_*.yml" and that it doesnt matter | 16:48 |
| jrosser | but then trying to use group_vars to be selective rather than global, all this then really does matter | 16:49 |
| noonedeadpunk | well, it's just impossible to do some things without group_vars | 16:49 |
| jrosser | i still think we shouldnt override things in main repo group vars which are notionally private in role vars | 16:50 |
| noonedeadpunk | yes, agree here | 16:50 |
| noonedeadpunk | I'd say we should make this variable public first | 16:50 |
| jrosser | yep | 16:51 |
| noonedeadpunk | and then override it's default in group_vars | 16:51 |
| jrosser | otherwise there is 8-O with the vars precedence and surprises | 16:51 |
| noonedeadpunk | so patch to glance is needed | 16:51 |
| damiandabrowski | so do i understand it correctly? add https://paste.openstack.org/show/bvVpcWgqxSLSj08vhPrT/ to inventory/group_vars/glance_all and move _glance_available_stores from role vars to role defaults? | 16:54 |
| noonedeadpunk | well, you can do glance_available_stores: "{{ _glance_available_stores }}" in role defaults | 16:57 |
| damiandabrowski | is there any reason not to do `glance_available_stores: "{{ [ glance_default_store ] + glance_additional_stores }}"` ? | 16:58 |
| noonedeadpunk | and in group_vars you don't actually need to re-define glance_additional_stores | 16:58 |
| damiandabrowski | why? | 16:58 |
| noonedeadpunk | well, you can move that to defaults as is | 16:58 |
| damiandabrowski | as I said before, haproxy role is imported outside glance role so it doesn't have any knowledge about glance role defaults | 16:59 |
| damiandabrowski | so glance_additional_stores has to be defined in group_vars if its used in haproxy service definition | 16:59 |
| noonedeadpunk | well, I meant you can do like glance_use_uwsgi: "{{ ('ceph' not in [ glance_default_store | default('') ] + glance_additional_stores | default([]) ) }}" | 17:01 |
| noonedeadpunk | so we don't maintain defaults in 2 places | 17:01 |
| noonedeadpunk | while keeping the logic | 17:02 |
| noonedeadpunk | well, as long as we won't add ceph to default store | 17:02 |
| damiandabrowski | yeah i can if you think it's better, but haproxy_backend_ssl will be really hard to read then | 17:02 |
| damiandabrowski | it's already quite complex | 17:02 |
| damiandabrowski | haproxy_backend_ssl: "{{ (glance_use_uwsgi | default(True)) | ternary((glance_backend_ssl | default(openstack_service_backend_ssl)), False) }}" | 17:02 |
| noonedeadpunk | well, you can move glance_use_uwsgi outside of haproxy logic and jsut get it defined as well | 17:03 |
| noonedeadpunk | then it will be kinda same - aproxy_backend_ssl: "{{ (glance_use_uwsgi | ternary((glance_backend_ssl | default(openstack_service_backend_ssl)), False) }}" | 17:04 |
| damiandabrowski | i have literally 5 minutes left, I'll try to upload required changes before i leave | 17:05 |
| noonedeadpunk | so you just skip defining glance_additional_stores as we know ceph is not there by default | 17:05 |
| noonedeadpunk | and it's not in glance_default_store | 17:05 |
| noonedeadpunk | or well, glance_default_store is already in group_vars | 17:06 |
| noonedeadpunk | so need for default there | 17:06 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_glance master: Move _glance_available_stores to defaults https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/880872 | 17:15 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Add support for TLS backends https://review.opendev.org/c/openstack/openstack-ansible/+/879085 | 17:16 |
| damiandabrowski | hopefully i didn't make any mistake there | 17:17 |
| damiandabrowski | se you tomorrow guys | 17:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Update release name to Antelope https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/880761 | 18:09 |
| psymin | for a test deployment, are internal_lb_vip_address and external_lb_vip_address required? | 19:03 |
| jrosser | psymin: both are always required and should never be the same | 19:04 |
| psymin | my goal at the moment is to have a playbook configured that will deploy to the three servers, even if it isn't what I want. Then recreate the targets, change the playbook config, and redeploy. | 19:04 |
| psymin | If those IPs are necessary, do I first need to configure haproxy or will ansible do that for me with the config info? | 19:05 |
| jrosser | that is all done for you | 19:07 |
| psymin | nice | 19:07 |
| jrosser | how many controllers are you using? | 19:07 |
| psymin | I'm having some terminology confusion. I'd like to use three controllers, but would make due with one. I'll look up which openstack services make up the "controller" | 19:08 |
| jrosser | api services / database / message queues etc | 19:09 |
| jrosser | if you only define one, then I think you need to manually assign the internal/external vip to your interfaces | 19:09 |
| psymin | which network should the vips be on? | 19:10 |
| jrosser | if you have multiple controllers then keepalived is responsible for the IPs for H/A | 19:11 |
| psymin | Do I need to manually configure keepalived before deploying or does anisble do that? | 19:12 |
| jrosser | psymin: keepalived is all done for you | 19:27 |
| psymin | nice | 19:27 |
| jrosser | psymin: regarding which network the bios should be on, internal should be on the mgmt network | 19:28 |
| jrosser | external really depends on your use case | 19:28 |
| jrosser | not bios :( … vips I mean | 19:28 |
| psymin | whew, thanks for the clarification :) | 19:29 |
| * jrosser autocorrect | 19:29 | |
| jrosser | I think it’s important to think clearly about two different things | 19:29 |
| psymin | I assume I should choose an IP for external_lb_vip_address that is on an interface and network that can route to the internet? | 19:29 |
| jrosser | one, you are deploying the cloud and want all its components to talk to each other via the internal vip with sufficient security/whatever | 19:30 |
| jrosser | two, you are the user of the cloud accessing the dashboard on some ip that is the external vip | 19:31 |
| jrosser | and as an extension of that, as a user you make some vm which need an IP on whatever network is appropriate for your situation | 19:31 |
| psymin | currently I can only access the machines using the management IPs | 19:31 |
| psymin | which is acceptable to me at the moment | 19:32 |
| jrosser | that’s ok | 19:33 |
| psymin | I doubt we'll access the dashboard from the external IP | 19:35 |
| jrosser | you will, that’s how it works | 19:35 |
| psymin | we'd use wireguard to connect to the private network and from there to the dashboard | 19:35 |
| psymin | ahh | 19:35 |
| jrosser | external means “external side of haproxy” | 19:35 |
| jrosser | which may or may not be “external” in terms of your network | 19:36 |
| jrosser | it could be on your lan | 19:36 |
| psymin | for safety, can that external side of haproxy be the management network which isn't routable to the internet? | 19:36 |
| jrosser | I’m not sure how to answer that | 19:37 |
| psymin | I probably am thinking of the network incorrectly. | 19:38 |
| jrosser | as it would be radically different depending if you were building a cloud with actual internet ip on the outside | 19:38 |
| jrosser | or if you make something to run test workloads internally behind your firewall for example | 19:38 |
| psymin | Any time we'll need an actual IP I believe we'll manually map that through in the router to the destination service. | 19:38 |
| jrosser | it’s totally use case dependant, and one of the biggest reasons there’s no right answer | 19:38 |
| psymin | one usage case is to run the mail server. No public IPs bound to the server, but ports forwarded in the router to it. (not currently on openstack) | 19:39 |
| jrosser | i think a diagram is good | 19:41 |
| psymin | can you suggest a linux-friendly diagram program :) | 19:42 |
| jrosser | your cloud has an outside (haproxy external vip and your vm networking) | 19:42 |
| jrosser | and it has an inside, haproxy internal vip, mgmt, storage m, vxlan etc | 19:42 |
| jrosser | with haproxy sat right at the center of that | 19:42 |
| psymin | imagining this diagram, would the network that I ssh to a target from be considered "outside" ? | 19:45 |
| jrosser | I think that depends where on the diagram you would put yours melt when running playbooks/doing admin on the hosts | 19:46 |
| jrosser | could be vpn, or your existing company network, basically how you’d be comfortable managing hosts or network gear today | 19:47 |
| jrosser | then as a user of the cloud, accessing the dashboard, run terraform or whatever you’d place yourself somewhere a bit different maybe, as in most cases the admin of the cloud does that for some customers/users | 19:48 |
| jrosser | and finally you have your sevices, like your email server that might be accessible from the whole internet | 19:49 |
| jrosser | hopefully thinking about it like this clarifies the purpose of the different networks like mgmt and external vip a bit | 19:49 |
| psymin | I'll try to lay out what we're hoping for in a diagram. | 19:50 |
| psymin | on a slight tangent since work pressure is ramping up, how bad of an idea would it be to use multi node AIO? | 19:52 |
| jrosser | imho that is advanced level openstack-ansible | 19:54 |
| psymin | okay, if it were all up to you, and you had three servers, how would you set it up? | 19:54 |
| psymin | usage case being one mail server and no customers | 19:54 |
| noonedeadpunk | psymin: I think I would split services by servers. | 20:59 |
| psymin | noonedeadpunk, so one controller, one storage, one compute? | 21:00 |
| noonedeadpunk | No, not really. So, let's say nova-api and glance on node01 and node02, cinder-api and keystone on node02 and node03 and placement with neutron on node01 and node03 | 21:01 |
| noonedeadpunk | all of them are part of galera and rabbitmq clusters, as well as all used for VMs (nova-compute) | 21:02 |
| noonedeadpunk | There're more services to consider though, but I guess you've got the gist | 21:02 |
| noonedeadpunk | but idea is to have 2 copies of everything spread instead of 3 copies like suggested by docs | 21:03 |
| noonedeadpunk | except galera and rabbit that must have 3 copies for quorum | 21:03 |
| -opendevstatus- NOTICE: The Etherpad service on etherpad.opendev.org will be offline for the next 90 minutes for a server replacement and operating system upgrade | 21:57 | |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Bump OpenStack-Ansible Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/880479 | 23:05 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
