| *** chandankumar is now known as chkumar|rover | 05:27 | |
| jrosser | morning | 09:16 |
|---|---|---|
| noonedeadpunk | o/ | 09:23 |
| noonedeadpunk | so... mirrors are still desynced... | 09:57 |
| jrosser | odd that noone else has trouble | 09:59 |
| noonedeadpunk | and I know why | 10:05 |
| noonedeadpunk | check this devstack job, that succeeded https://zuul.opendev.org/t/openstack/build/1b37b0e31dd14aa398a71377ee27eb4b/log/job-output.txt#2264 | 10:06 |
| noonedeadpunk | And I'm not sure how much projects event test against centos | 10:07 |
| noonedeadpunk | s/event/even | 10:07 |
| *** dviroel|out is now known as dviroel | 10:48 | |
| admin1 | if i create a port-binding ( sr-iov) .. and then add a new sr-iov host, can that port be used.. or i have to create a new port when a new host is added ? | 10:48 |
| jrosser | noonedeadpunk: at some point maybe we make Rocky the first class RHEL-alike distro and make centos experimental | 10:57 |
| noonedeadpunk | well, I think we even said that during PTG | 10:57 |
| jrosser | but without running the jobs we will break conditionals for sure | 10:58 |
| jrosser | perhaps this would be a good thing to look at early this cycle | 10:58 |
| jrosser | trying to get rid of anything centos specific so that test on Rocky cover mostly both | 10:58 |
| noonedeadpunk | Just check these 2 https://opendev.org/opendev/system-config/commit/e76e0089d1c4a7fe781027feb7977ae874899443 and https://opendev.org/opendev/system-config/commit/cc2dd16d3a7194a4185ad6e1da854cb4fde01b1c | 10:59 |
| jrosser | grrr | 10:59 |
| noonedeadpunk | I think, that for Rocky there're no infra mirrors btw | 11:00 |
| noonedeadpunk | yeah, https://zuul.opendev.org/t/openstack/build/1cc5dd9cb91c4d7eb17512207cce15c7/log/logs/etc/host/yum.repos.d/rocky.repo.txt | 11:00 |
| noonedeadpunk | so not sure how good is that ^ | 11:01 |
| noonedeadpunk | should we propose yet another revert on the latest revert? :D | 11:02 |
| admin1 | in case of using sriov agent , when does privsep daemon starts ? .. is it after a vm has initiated, or always running ? .. i have 1 node where its running and works fine, in another node, vms fail to deploy and the diff is in working one, privsep process is also running.. in non working one, only sr-iov agent is running .... | 12:31 |
| admin1 | just checking if anyone else is also running sr-iov | 12:31 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron master: Create separate lock path for neutron-ovn-metadata-agent https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/868415 | 12:53 |
| opendevreview | James Denton proposed openstack/openstack-ansible master: Start 2023.1 Antelope development https://review.opendev.org/c/openstack/openstack-ansible/+/867954 | 12:58 |
| noonedeadpunk | jamesdenton: fwiw I've narrowed down the patch that breaks things | 13:50 |
| noonedeadpunk | ah. But I see you found how to fix that | 13:50 |
| noonedeadpunk | As things become failing on https://review.opendev.org/c/openstack/neutron/+/861649 | 13:51 |
| noonedeadpunk | But I'm not sure if that patch actually helps.... | 13:57 |
| noonedeadpunk | jamesdenton: yeah, it looks like you're right | 14:07 |
| spatel | OVN driver for octavia is so awesome! :) | 14:17 |
| spatel | atleast for basic use | 14:17 |
| spatel | I am using it with kolla but i going to try it out with OSA and blog it - https://satishdotpatel.github.io/openstack-kolla-deploy-octavia-ovn/ | 14:17 |
| noonedeadpunk | yeah. we should add ovn support to octavia | 14:22 |
| noonedeadpunk | shouldn't be too tricky tbh | 14:22 |
| spatel | No.. it just work out of box if you have OVN deployment | 14:29 |
| spatel | I will get that going in my charisma break.. | 14:29 |
| noonedeadpunk | well, I think you only need to install octavia "plugin" for that | 14:30 |
| spatel | yep | 14:40 |
| spatel | noonedeadpunk what do you think about multi-region deployment of OSA? | 14:47 |
| spatel | How do you guys deploying multiple openstack and linking them with single keystone? | 14:47 |
| noonedeadpunk | I'm currently not. But I think it should be quite posisble. Though would suggest some keystone federation. And likely not keystone to keystone... | 14:48 |
| noonedeadpunk | But maybe some ldap or smth | 14:49 |
| spatel | currently i am using LDAP for all my deployment (single identity) | 14:49 |
| noonedeadpunk | As if you have a keystone just in one region and others do connect to to this keystone - it's bad idea | 14:49 |
| spatel | I was thinking how about single horizon/keystone style deployment where end user select region from GUI | 14:50 |
| noonedeadpunk | Then I don't think there's any issue with that. Well, except you'de need to mess up with env.d a bit | 14:50 |
| spatel | I am just exploring all option that how public cloud company manager this kind of deployment. | 14:50 |
| noonedeadpunk | Eventually for that usecase - you don't even need to have multiregion | 14:50 |
| noonedeadpunk | You can have 2 standalone regions | 14:50 |
| noonedeadpunk | And just configure horizon accordingly | 14:51 |
| spatel | hmm.. without shared keystone? | 14:51 |
| noonedeadpunk | I did that couple of times | 14:51 |
| noonedeadpunk | yup | 14:51 |
| spatel | how.. how? | 14:51 |
| noonedeadpunk | For horizon it doesn't matter | 14:51 |
| spatel | but horizon get endpoint from keystone right? | 14:51 |
| noonedeadpunk | Um, kind of, but you can explicitly provide independant keystones as a regions | 14:52 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/defaults/main.yml#L259-L261 | 14:52 |
| noonedeadpunk | So you still will have a drop-down there with selection of regions, but it will connect to 2 independant keystones based on the selection | 14:53 |
| noonedeadpunk | Nasty part, is that user credentials will be different if there's no federation | 14:53 |
| noonedeadpunk | But if it's ldap - meh, do that | 14:53 |
| spatel | very interesting.. | 14:54 |
| spatel | How about federation? how its different compare to shared keystone? | 14:55 |
| noonedeadpunk | Franqly, once folks also did some nasty hooks to horizon, to make switching between regions even nicer, as with such setup when you switch regions you will be redirected to the login page (as you've not auth to another keystone). | 14:55 |
| noonedeadpunk | Well, I think shared keystone means that if link between regions goes down - you have problems | 14:55 |
| spatel | But you can have multi-master replication to avoid that :) but anyway upgrade etc would be tricky.. | 15:00 |
| spatel | shared is terrible in anyway. | 15:01 |
| noonedeadpunk | well, if you're going to have 3 regions - then might be. But then maybe you want AZs, not regions? | 15:14 |
| spatel | AZ is just logical grouping of servers/racks etc.. correct | 15:16 |
| noonedeadpunk | well, depends | 15:16 |
| noonedeadpunk | but regions are as well, kind of? | 15:16 |
| noonedeadpunk | but yes, you also have services per region | 15:16 |
| noonedeadpunk | and AZs share control plane | 15:17 |
| jamesdenton_ | o/ | 15:17 |
| jamesdenton_ | good day, all | 15:17 |
| noonedeadpunk | o/ | 15:17 |
| jamesdenton_ | you may not believe it, but when the children are home off school, the days tend to start... 2 hours earlier | 15:17 |
| noonedeadpunk | spatel: but if you bother yourself with stretching keystone, why not to stretch glance api and re-use same images | 15:18 |
| noonedeadpunk | jamesdenton_: oh, well, I do | 15:18 |
| noonedeadpunk | despite mine is too young, but when it's weekend he tends to wake up even earlier then during working days | 15:19 |
| jamesdenton_ | there's no sleeping in around here | 15:19 |
| spatel | noonedeadpunk we only use single image (single OS i meant) in glance :) | 15:20 |
| spatel | We don't give option to end users to pick images.. haha.. | 15:21 |
| spatel | jamesdenton_ did you use HP virtual connect or any experience? | 15:21 |
| jamesdenton_ | no experience | 15:22 |
| spatel | np | 15:24 |
| jamesdenton_ | oh spatel, you had a question yesterday that i missed. but i don't have the scrollback. do you recall? | 15:24 |
| spatel | I have OVN lab and it has eth1:physnet1 which i am using for flat networking, Now i want VLAN base provider so trying to use eth2 nic so how do i tell ovn use eth2:physnet2 for vlans ? | 15:25 |
| jamesdenton_ | ok | 15:26 |
| spatel | where should i add eth2 and map it in ovn for physnet2 ? | 15:26 |
| jamesdenton_ | how are you defining provider networks, in openstack_user_config.yml? | 15:26 |
| spatel | i don't have access of lab at present but you are saying just add in openstack_user_config and OSA will do all magic? | 15:27 |
| spatel | how does OSA map physnetX ? | 15:27 |
| spatel | i know in legacy way you can map entry in ml2 like eth1:physnet1,eth2:physnet2 etc.. | 15:28 |
| jamesdenton_ | yes | 15:28 |
| jamesdenton_ | one sec | 15:28 |
| spatel | does same way it apply to OVN :) | 15:28 |
| jamesdenton_ | https://paste.openstack.org/show/bj0tyuNGfbtDvwnhUKas/ | 15:29 |
| jamesdenton_ | playbooks will create the ovs bridges and connect network_interface | 15:29 |
| jamesdenton_ | and create the mapping and apply to ml2 config and ovn | 15:29 |
| spatel | Niceee! | 15:34 |
| spatel | Do we have this in OSA networking example :) | 15:35 |
| jamesdenton_ | i think so | 15:35 |
| spatel | what is this group for ? neutron-ovn-gateway ? | 15:35 |
| jamesdenton_ | definitely here: https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch.html, for OVS | 15:36 |
| jamesdenton_ | but it applies to LXB and OVN | 15:36 |
| spatel | it will make chassis gateway but which node? assuming network node but in ovn there is no network node correct? | 15:36 |
| jamesdenton_ | so, there's a patch in master/zed that splits out gateway from controller | 15:36 |
| jamesdenton_ | so you can have ovn-controllers (computes) and ovn-controller+ovn-gateway (dedicated network nodes) | 15:37 |
| jamesdenton_ | for example | 15:37 |
| jamesdenton_ | since the initial OVN implementation assumed all computes would also be gateways | 15:37 |
| spatel | yes that is what i am thinking.. all computes are network node.. | 15:38 |
| jamesdenton_ | they still can be | 15:38 |
| jamesdenton_ | but now they don't have to be | 15:38 |
| spatel | In short if you want dedicated network node then just install ovn-controller and set gateway flag | 15:38 |
| jamesdenton_ | so your grouping would be neutron-ovn-controller instead | 15:38 |
| jamesdenton_ | yeah, i don't recall how to trigger it, i think there's a new group | 15:38 |
| spatel | damn.. we need more docs :) | 15:39 |
| jamesdenton_ | some liberties were taken when we pulled OVN out of experimental status. And the consensus was... spatel is the only one using this | 15:39 |
| jamesdenton_ | so, let us know how it goes. | 15:39 |
| jamesdenton_ | :D | 15:39 |
| spatel | I am using ovn in small production environment until i get some expertise. so far i didn't hit any issue.. | 15:40 |
| jamesdenton_ | awesome | 15:41 |
| spatel | I have deployed OVN in some of my customer end with 30 to 40 node deployment and still no issue.. | 15:41 |
| jamesdenton_ | how's metadata? | 15:41 |
| jamesdenton_ | i think i saw your name on a bug | 15:41 |
| spatel | Yes.. i had and issue where meta service required restart and i didn't see any solution so i am doing cron restart (daily) | 15:43 |
| spatel | But in newer deployment i haven't see any issue related metadata.. | 15:44 |
| spatel | I believe they fixed issue after wallaby release | 15:44 |
| jamesdenton_ | oh cool. i have a deployment building now and need to check | 15:44 |
| jamesdenton_ | it was not working in my AIO earlier | 15:44 |
| spatel | Hmm! really.. i had no issue recently with meta | 15:45 |
| *** dviroel is now known as dviroel|afk | 15:46 | |
| spatel | I am building my home lab on Intel NUCs | 15:48 |
| spatel | I have 10 Intel NUC so it would be good for any kind of testing | 15:49 |
| jamesdenton_ | nice, i've put together some stuff to do a quick multi-node, but it does require an openstack undercloud | 15:50 |
| spatel | noonedeadpunk what do you mean openstack undercloud | 16:35 |
| jamesdenton_ | meaning my process deploys multi-node openstack cluster using VMs | 16:36 |
| jamesdenton_ | so i use my regular openstack cloud (lab) to deploy like... 8-9 instances (2x haproxy, 3x controller, 3x compute, 1x deploy) | 16:37 |
| jamesdenton_ | terraform+ansible | 16:37 |
| spatel | openstack on openstack :) | 16:41 |
| spatel | How many nodes you have in home lab (physical nodes ) running openstack? | 16:42 |
| jamesdenton_ | 3x controllers on ESX and 3x metal computes | 16:43 |
| spatel | I have 2 HP gen9 servers which and they are running VMware Host to create any kind of lab | 16:43 |
| jamesdenton_ | these are Gen9 as well, E5-2680 i think | 16:43 |
| jamesdenton_ | v4 | 16:43 |
| jamesdenton_ | they're just loaded ewith ram | 16:44 |
| spatel | How much electricity bills you are paying for those hardware? | 16:44 |
| jamesdenton_ | https://github.com/busterswt/mnaiov2, and inspired by https://github.com/openstack/openstack-ansible-ops/tree/master/multi-node-aio | 16:44 |
| jamesdenton_ | oh, i dunno... napkin math a while back was like, 50-70/mo if i had to guess, but likely lower | 16:45 |
| spatel | Nice!! MNAIO :) | 16:45 |
| spatel | I am waiting for Tesla solar panel after that my home will be all using green energy then i will run datacenter in my basement 24x7 | 16:46 |
| jamesdenton_ | so, i have them hooked up to some Emporia plugs, and it's showing about $30 as of 22-Dec, so maybe 40 by the EOM | 16:46 |
| spatel | Emporia plugs .. i need to look into that | 16:47 |
| spatel | What do you use for cooling in summer? | 16:48 |
| jamesdenton_ | i think they're called emporia smart plugs | 16:49 |
| jamesdenton_ | i don't do anything in particular... the gear is sitting in a large upstairs room in the cab, and isn't super loud and doesn't run too hot. it gets warm in that room but not too hot. ambient is maybe 85 in the summer w/ the servers on | 16:50 |
| jamesdenton_ | running that way for a couple of years now, no problems | 16:50 |
| spatel | Hmm! | 16:52 |
| *** dviroel|afk is now known as dviroel | 17:09 | |
| moha7 | Hi there; During deploying `repo-install.yml`, I get 3 mounting errors like this: | 17:25 |
| moha7 | fatal: [infra1_repo_container-b253b706]: FAILED! => {"changed": false, "cmd": "systemctl reload-or-restart $(systemd-escape -p --suffix=\"mount\" \"/var/www/repo\")" | 17:25 |
| moha7 | but it filly is finished witout any failed task! --> ok=66 changed=3 unreachable=0 failed=0 skipped=24 rescued=1 | 17:26 |
| moha7 | The playbook output: https://p.teknik.io/aIJDc | 17:26 |
| moha7 | finally* | 17:28 |
| moha7 | I think it leades to: infra1_galera_container-c57c8a02 : ok=10 changed=0 unreachable=0 failed=1 | 17:33 |
| moha7 | Running `setup-infrastructure.yml` exits with this error: https://p.teknik.io/xuK1H | 17:35 |
| moha7 | `openstack-ansible galera-install.yml -vvv` --> https://p.teknik.io/AIlqO | 17:39 |
| jrosser | moha7: see it says "rescued=1" | 17:41 |
| jrosser | you can read the explanation of that here https://opendev.org/openstack/ansible-role-systemd_mount/src/branch/master/tasks/systemd_mounts.yml#L75-L104 | 17:42 |
| moha7 | Ah, so it's expected and not the root cause of that Galera error | 17:44 |
| jrosser | yes its completely expected | 17:55 |
| jrosser | it is an unfortunate way for ansible to report the tasks as both failed and rescued | 17:55 |
| jrosser | which means the ultimate outcome was success | 17:55 |
| jrosser | moha7: i think that the galera playbook will be the first one that tries to adjust the loadbalancer state | 17:57 |
| jrosser | moha7: i would guess that haproxy is not running and the socket that the `haproxy` ansible module tries to connect to is not present | 18:00 |
| jrosser | check your haproxy service and the log to see what is happening | 18:00 |
| moha7 | I had som error in haproxy, so I run it manually first to fix it; here is my user_variables.yml file: http://ix.io/4ji5hmm | 18:03 |
| moha7 | I'm going to revert the VMs to fresh state and run `setup-infrastructure.yml` again.I | 18:03 |
| moha7 | some errors* | 18:03 |
| jrosser | well, you need to get it to a state where haproxy comes up cleanly with the playbookes | 18:04 |
| jrosser | and you can also run all of the playbooks individually like you did with the galera one | 18:05 |
| moha7 | The correct link to `user_variables.yml`: https://p.teknik.io/ps60L | 18:05 |
| jrosser | it would be just as good to re-run the haproxy role and fix up whatever variables you need to | 18:05 |
| jrosser | reverting the VM many times is not especially helpful | 18:05 |
| moha7 | As I use an official signed certificate, haproxy now starts well after adding `haproxy_ssl_self_signed_regen: true` to the user_variables file | 18:06 |
| jrosser | haproxy_keepalived_external_interface / haproxy_keepalived_internal_interface look unusual | 18:06 |
| jrosser | this is "internal and external virtual IP for the loadbalancer" | 18:07 |
| jrosser | internal is usually on the mgmt network | 18:07 |
| jrosser | external is usually on some outward facing network, or also another IP on the mgmt network depending if you want that to be accessible by your end users | 18:07 |
| jrosser | (you most likley dont) | 18:08 |
| jrosser | the mgmt network i mean, thats ideally private if you care about security of your deployment | 18:08 |
| moha7 | Oops, I'll move it to mgmt range | 18:08 |
| jrosser | haproxy_ssl_self_signed_regen is not really related to using a official certificate | 18:09 |
| jrosser | it might be a bug you found though if you started with an OSA self-signed one and wanted to change to an externally provided one, we may have some error there in switching? | 18:10 |
| jrosser | for a production deployment you might have some extra vlan that is on your controllers that the external endpoint is in, one that is accessible by your users | 18:11 |
| jrosser | but one that is different from the mgmt network | 18:11 |
| jrosser | consider how you would build a deployment where the external IP was on the internet, you would not want this to be the mgmt network | 18:11 |
| moha7 | you mean admin and interl endpoints on vlan1, and the external endpoint on the other on, vlan2? Then Where I should define this setting? | 18:17 |
| moha7 | internal* | 18:17 |
| jrosser | the host networking is entirely up to you | 18:18 |
| jrosser | openstack-ansible does not do this | 18:18 |
| jrosser | single interface / bond / whatever...... | 18:19 |
| moha7 | for a provided certificate, Does it need to any option in the user_variable file, other than these: haproxy_user_ssl_cert, haproxy_user_ssl_key, haproxy_user_ssl_ca_cert ? | 18:19 |
| jrosser | no that should be fine | 18:20 |
| jrosser | it is needed to do some thinking about how you want the network to be | 18:20 |
| jrosser | "my users are in vlan x / subnet y", "my openstack mgmt network is firewalled from that and on vlan z" | 18:21 |
| moha7 | You're right | 18:21 |
| moha7 | Going to deploy again with above updates; Thanks | 18:21 |
| jrosser | or "i don't care about security and everything can talk to everything else" | 18:21 |
| jrosser | only you can decide that and setup the host networking as you need | 18:21 |
| moha7 | I'm working on a Lab; But it will be deployed into a production env. in 3 next months | 18:22 |
| jrosser | right - so i would recomment making your lab as structurally representative of your production environment as you can | 18:22 |
| jrosser | same separation of trust / access even if you have to make that artifical in the lab | 18:23 |
| jrosser | then there are less surprises later | 18:24 |
| moha7 | +1 | 18:24 |
| jrosser | but i would always recommend starting with the "all-in-one" deployment just so that you have something simple and easy as a reference point | 18:24 |
| jrosser | then if your multinode lab deployment breaks you can go poke around the AIO to see how things should have turned out | 18:25 |
| *** dviroel is now known as dviroel|out | 20:15 | |
| spatel | is ceilometer still use to collect metrics from openstack components? | 20:36 |
| *** ianw is now known as ianw_pto | 23:43 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!