| *** mac189_ is now known as mac189 | 05:49 | |
| *** elenalindq_ is now known as elenalindq | 05:52 | |
| noonedeadpunk | I guess t->w should work though as well. Never tested that, but t->v worked and v->w was also quite straightforward | 06:46 |
|---|---|---|
| noonedeadpunk | well, not sure if nova will like that though | 06:46 |
| noonedeadpunk | (can't recall if it was W or X when they force fail because of rpc version missmatch | 06:57 |
| *** frenzy_friday is now known as frenzyfriday | 07:05 | |
| *** johnsom_ is now known as johnsom | 09:03 | |
| jrosser | the bump_osa patches fail on old centos-8 jobs | 09:06 |
| jrosser | i'm not around until tuesday next week to look at it but this topic needs to merge https://review.opendev.org/q/topic:%22osa%252Fremove-centos8%22+(status:open%20OR%20status:merged) | 09:07 |
| *** melwitt is now known as Guest1312 | 09:32 | |
| opendevreview | Merged openstack/openstack-ansible master: Collect the contents of /etc/ssh during zuul job log gathering https://review.opendev.org/c/openstack/openstack-ansible/+/825475 | 10:33 |
| *** melwitt is now known as Guest1320 | 10:50 | |
| opendevreview | Merged openstack/openstack-ansible master: Drop hosts file distribution tasks https://review.opendev.org/c/openstack/openstack-ansible/+/809521 | 10:57 |
| *** dviroel|out is now known as dviroel | 11:25 | |
| *** anbanerj is now known as frenzyfriday | 14:41 | |
| *** dviroel is now known as dviroel|lunch | 14:58 | |
| *** promethe- is now known as prometheanfire | 16:03 | |
| *** dviroel|lunch is now known as dviroel | 16:15 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Remove unused centos-8 functional job definitions https://review.opendev.org/c/openstack/openstack-ansible-tests/+/820674 | 16:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Remove unused centos-8 functional job definitions https://review.opendev.org/c/openstack/openstack-ansible-tests/+/820674 | 16:35 |
| MrClayPole | Hi, I've recently moved cinder-volume to my control nodes from my compute nodes. They previously ran on metal. I've removed them from openstack_user_variables.yml but they're still in the inventory. I've attempted to use the "inventory-manage.py" but this only remove hosts but I'm looking to remove a service from a host? What's the best way to do this? | 16:37 |
| noonedeadpunk | MrClayPole: I don't think it's implemented now in inventory-manage.py. We've recently added only fully removing group, but not specific host from group. It's good thing to implement though... | 16:39 |
| noonedeadpunk | In the meanwhile I think the only way is to manually edit /etc/openstack_deploy/openstack_inventory.json :( | 16:40 |
| MrClayPole | OK thanks, I though that would be the case but just wanted to check first | 16:40 |
| spatel | noonedeadpunk i have question for you. I have mounted 800T glusterfs in /var/lib/nova for shared storage for VMs but interesting thing nova placement still think you have local disk storage so showing disk size 70GB (how does nova report 800TB to placement?) | 17:10 |
| noonedeadpunk | spatel: are you sure that nova user has access to that storage? | 17:28 |
| spatel | yes.. and i can create vm also everything works fine.. only my placement showing low disk | 17:28 |
| spatel | may be it doesn't know we have bigger storage may be... | 17:28 |
| noonedeadpunk | hm, I was absolutely sure it does smth like getting size of mountpoint for /var/lib/nova... But I think I need to find code to verify that | 17:29 |
| spatel | noonedeadpunk damn it i know what is going on again :) | 17:30 |
| spatel | my bad.. | 17:30 |
| noonedeadpunk | huh? | 17:31 |
| spatel | my mount point issue its not mounted on couple of servers and by mistake i pick that server to verify :) | 17:32 |
| spatel | i can see proper reporting on other compute which has proper mounted filesystem | 17:32 |
| noonedeadpunk | oh, well :) | 17:33 |
| spatel | i think i need to write script to verify if mount point failed to mount then don't start nova service because otherwise it will use local disk to start VM | 17:33 |
| noonedeadpunk | well, if you'd use systemd-mount, then you can apply nova init overrides and make it dependant on mount | 17:34 |
| noonedeadpunk | at least we did that to mount cephfs to add space for huge images that are being converted | 17:36 |
| spatel | i am using /etc/fstab but i think you are correct i should move it to systemd-mount and make it dependent for nova | 17:36 |
| noonedeadpunk | and we have https://opendev.org/openstack/ansible-role-systemd_mount/ ;) | 17:37 |
| spatel | sweeeeet! i will use same | 17:38 |
| spatel | noonedeadpunk other question, i am building two new cloud in remote datacenter so is Xena is ready for production? | 17:39 |
| spatel | i want to upgrade wallaby -> xena in other datacenter soon | 17:39 |
| noonedeadpunk | I'd wait for 24.0.1 for sure which was should have been already released with https://review.opendev.org/c/openstack/openstack-ansible/+/825391 | 17:40 |
| noonedeadpunk | but we need to drop ci for centos-8 now to get this merged | 17:41 |
| spatel | sweet! i have 1 month in hand because currently we are buying servers and racking up so it will take little time so hope by the time its ready to rollout | 17:41 |
| noonedeadpunk | we already upgraded w->x and it was straigforward, but we pulled some patches from 24.0.1 manually | 17:41 |
| noonedeadpunk | wait, we upgraded v->x :P | 17:41 |
| spatel | lol | 17:42 |
| spatel | now i have 100G nic and trying to run iperf3 to test but only able to hit 40G.. damn it. | 17:46 |
| spatel | noonedeadpunk did you work on infiniband ? | 17:50 |
| spatel | i need some help to understand | 17:50 |
| noonedeadpunk | I did one day | 17:57 |
| noonedeadpunk | It brought really tons of troubles though... So using same mellanox cards in ethernet mode would be so much easier | 17:58 |
| noonedeadpunk | unless you need rdma ofc | 17:58 |
| noonedeadpunk | What I can recall that switching to connected mode improved IPoIB performance a lot comparing to default datagram | 18:00 |
| noonedeadpunk | I believe mostly because of MTU that is 4k tops in datagram | 18:01 |
| noonedeadpunk | but well, we had only 60g back then I believe | 18:02 |
| noonedeadpunk | (via ConnectX-3 Pro) | 18:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Remove CI jobs for centos-8 https://review.opendev.org/c/openstack/openstack-ansible/+/824570 | 18:05 |
| spatel | noonedeadpunk This cloud is for HPC use and RDMA will give better performance compare to IPoIB | 18:15 |
| spatel | Tell me if i am wrong here, my plan is to configure SRIOV on infiniband nic and attach vf to vm (vm can see rdma nic) | 18:16 |
| spatel | i didn't find any good document related how to build HPC on openstack (all i saw big high level concept but not good config based document) | 18:17 |
| spatel | most of people running HPC using ironic but in my case i want to use VMs | 18:18 |
| noonedeadpunk | I never had experience with HPC. Regarding rdma - yes, but it kind of require software that able to work with it | 18:19 |
| noonedeadpunk | sriov works nicely there though | 18:19 |
| noonedeadpunk | But what you can't do with ib interfaces - make bridge of them. it's smth to keep in mind. You won't do bridges though if looking for rdma | 18:20 |
| noonedeadpunk | but we used infiniband for storage and on computes only | 18:21 |
| noonedeadpunk | so sriov was applicable only on controllers when we needed to pass storage to containers (as sriov is the simplest way considering bridges or mac-vlan not an option). | 18:23 |
| mgariepy | the last ones that i knew were using IB on openstack did switch all the network to ethernet because at some point melanox was not providing patches for the ib in neutron. | 18:23 |
| noonedeadpunk | and ceph with rdma was in experimental state only, so it was all a mess | 18:23 |
| noonedeadpunk | for neutron we were using built-in 10G :p | 18:24 |
| mgariepy | lol nice :D haha | 18:24 |
| noonedeadpunk | but yes, IB was kind of pita if aim to use IPoIB | 18:24 |
| mgariepy | they were using the ib network directly but they deployed with some patches on top and then support dropped .. | 18:25 |
| noonedeadpunk | drivers were always failing to build against new kernels, this and that on every step, some weird lags from time to time... | 18:25 |
| noonedeadpunk | cards could get lost after reboot so power off/on was required for compute to find that... Likely it was partially because of rubbish hardware we used... | 18:27 |
| spatel | In my case we want to run MPI job on infiniband (no IP stuff) | 18:29 |
| spatel | for MPI job i would like to expose raw nic to vm via SRIOV. | 18:30 |
| noonedeadpunk | then I'd say it likely proper solution, as performance there should be great indeed | 18:30 |
| spatel | I have configured mallanox switch for SM (subnet manager) | 18:30 |
| spatel | now trying to understand how to map pkay (partition) to make with vlan etc.. that part i am having hard time to understand | 18:31 |
| noonedeadpunk | oh, SM was other PITA lol | 18:31 |
| spatel | pkey* | 18:31 |
| noonedeadpunk | well, it's jsut hex number | 18:32 |
| spatel | yes and how to map them with vlan etc | 18:33 |
| spatel | for multi-tenancy | 18:33 |
| noonedeadpunk | this pkey _is_ vlan | 18:33 |
| noonedeadpunk | but iirc everybody can use it kind of | 18:34 |
| noonedeadpunk | or well, on SM you define auth key. And every VM with that auth key can get all pkeys managed by SM | 18:34 |
| spatel | hmm | 18:35 |
| spatel | what is the use of pkey ? | 18:35 |
| noonedeadpunk | also you must somehow ensure that SM is not ran on any instance that has IB connected as they might conflict | 18:35 |
| noonedeadpunk | I think logical traffic separation? | 18:35 |
| spatel | I am running SM on Mallanox switch | 18:35 |
| spatel | ib sm enabled | 18:36 |
| noonedeadpunk | But nobody can stop from starting SM anywhere where OFED is installed :p | 18:36 |
| noonedeadpunk | they kind of run in HA then, but it's all messy a bit... | 18:36 |
| spatel | I have started standalone SM so i don't think it will allow anyone to join cluster | 18:37 |
| spatel | i am not running in HA mode | 18:37 |
| noonedeadpunk | And you have generated a uniqe key for it? | 18:38 |
| spatel | all i did on switch is > ib sm enabled | 18:38 |
| spatel | i have no idea about generating key etc.. | 18:39 |
| spatel | there was other command also called > ib sm-ha (i have only single switch so i thought i don't need HA ) | 18:39 |
| noonedeadpunk | then I guess it;'s default which means if somebody do `systemctl start opensm` it will likely join cluster with switch | 18:39 |
| spatel | i will give it a try and see if its impacting anything or not (This is not in production so i can mess with it) | 18:40 |
| spatel | where this pkey will get use? | 18:41 |
| noonedeadpunk | so pkey is a replacement for vlans. So what we did we were defining partition `Vxlan=0x87d2,ipoib: ALL=full;` and we had an isolated network and interface ib1.87d2 | 18:42 |
| spatel | ohhhh | 18:43 |
| noonedeadpunk | mgariepy: so for neturon vxlans we kind of used IB | 18:43 |
| noonedeadpunk | but to spawn interface I think either some service restart or echo to /sys/class/net/ib1/create_child was required | 18:44 |
| spatel | in my case if i create vm foo then how do vm know which pkey or partition i should be ? | 18:44 |
| mgariepy | huh ok so it was all supported well and integrated in neutron ? | 18:44 |
| spatel | or its part of neutron-agent job? | 18:44 |
| mgariepy | ;p | 18:45 |
| noonedeadpunk | nah... for vxlan all you need is IP on compute, and net node right?:) | 18:45 |
| noonedeadpunk | as for ml2 you jsut define IP thorugh which traffic would be incapsulated | 18:45 |
| noonedeadpunk | and we assigned IP to IB | 18:46 |
| mgariepy | if using ipoverib maybe ;) not sure mpi would work great on that tho. | 18:46 |
| spatel | Yes in IPoIB case that will work but in my case i don't want to configure IP on ib0 interface. i want to use RDMA | 18:46 |
| noonedeadpunk | spatel: have no idea. can't imagine how to do that | 18:46 |
| noonedeadpunk | all sr-iov devices would come with default pkey | 18:47 |
| spatel | let me mess with config and see. if solution is IPoIB then i should go with that | 18:47 |
| spatel | hmm! do i need to create pkey on SM or it will get auto generate if i don't do anything | 18:48 |
| noonedeadpunk | you need to define pkeys on SM | 18:48 |
| spatel | ok... let me do it.. (if all work out then i will blog that shit out because i didn't find any good document on internet :( ) | 18:49 |
| noonedeadpunk | but you for sure should define sm_key somewhere.... | 18:50 |
| noonedeadpunk | and use connected mode :) | 18:50 |
| spatel | I will take a look at also about security status | 18:50 |
| spatel | don't worry i will keep poking you until it all work out :) | 18:51 |
| noonedeadpunk | `This option specifies the SM's SM_Key (64 bits). This will effect SM authentication. Note that OpenSM version 3.2.1 and below used the default value '1' in a host byte order, it is fixed now but you may need this option to interoperate with old OpenSM running on a little endian machine.` | 18:51 |
| noonedeadpunk | so maybe not needed now, dunno.... | 18:52 |
| spatel | In my case i have Mallanox switch running SM so assuming they took care of security | 18:52 |
| noonedeadpunk | but I have some feeling, that pkey configuration would need to be done inside VM then... | 18:52 |
| noonedeadpunk | as you jsut passthrough device, which at that point knows nothing about pkeys | 18:53 |
| noonedeadpunk | I really have doubts they had any multi-tenancy in mind | 18:53 |
| spatel | that is where i am confused where should i put pkey ? | 18:53 |
| spatel | i think i should ask this question to mallanox mailing list or support | 18:54 |
| noonedeadpunk | in /etc/opensm/partitions.conf | 18:54 |
| mgariepy | you also need a network manager for IB i think. | 18:54 |
| noonedeadpunk | ifupdown handles IB nicely | 18:55 |
| noonedeadpunk | doesn't work at all with netplan though | 18:55 |
| spatel | there is a command on switch > ib sm virt enable | 18:55 |
| spatel | may be it will run SM per partition | 18:55 |
| mgariepy | not network/ subnet manager | 18:55 |
| mgariepy | ho. sm (subnet manager) | 18:56 |
| spatel | +1 | 18:56 |
| noonedeadpunk | yeah, so why I'm a bit confused now as I realized that we ran SM on ceph mons at first for HA. And only afterwards moved to switch | 18:57 |
| spatel | one more question currently i am using CentOS 8s based driver for all nic and they works fine. do i need to install MLNX OFED based driver? | 18:57 |
| noonedeadpunk | So maybe, jsut creating child is enough | 18:57 |
| noonedeadpunk | Well it contains tons of software you will likely need to have | 18:57 |
| spatel | only problem is i have to compile etc.. (it doesn't have RPM ) | 18:58 |
| noonedeadpunk | but watch out - on ubuntu there's cross dependency for IB drivers and ceph. So each time we were upgrading ofed - it was dropping all ceph packages | 18:58 |
| spatel | i was thinking i can try with OFED and if it work then i don't need to install compile lots of stuff | 18:58 |
| noonedeadpunk | mgariepy: btw, can you please review https://review.opendev.org/q/topic:"osa%252Fremove-centos8"+(status:open) ? | 18:59 |
| spatel | BRB | 19:01 |
| mgariepy | noonedeadpunk, looking | 19:01 |
| mgariepy | the link doens't work tho for ${REASON} | 19:02 |
| noonedeadpunk | because of quotes I guess... copy/paste should work though | 19:13 |
| *** Guest1320 is now known as melwitt | 19:18 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Remove CI jobs for centos-8 https://review.opendev.org/c/openstack/openstack-ansible/+/824567 | 19:58 |
| *** dviroel is now known as dviroel|out | 20:53 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Remove unused centos-8 functional job definitions https://review.opendev.org/c/openstack/openstack-ansible-tests/+/820674 | 20:58 |
| noonedeadpunk | seems https://review.opendev.org/c/openstack/ansible-role-qdrouterd/+/824537 is going to pass now :) | 21:06 |
| mgariepy | how comes it's not updated here via opendevreview when you push a new patch ? | 21:10 |
| krumelmonster | jrosser: Should we move the conversation here? | 21:53 |
| krumelmonster | I asked in #openstack how I'd configure letsencrypt/certbot for horizon | 21:54 |
| jrosser | krumelmonster: if i understand correctly, you’d set this to your fqdn rather than the ip https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.example#L115 | 21:57 |
| jrosser | ultimately it ends up here https://github.com/openstack/openstack-ansible-haproxy_server/blob/master/tasks/haproxy_ssl_letsencrypt.yml#L78 | 22:00 |
| jrosser | those two bits of config are tied together here https://opendev.org/openstack/openstack-ansible-haproxy_server/src/branch/master/defaults/main.yml#L224 | 22:03 |
| jrosser | to deploy that, re-run the haproxy playbook | 22:04 |
| jrosser | once that is done and working, the other thing that will be wrong is the service catalog entries which will still point to your old ip rather than the fqdn | 22:08 |
| opendevreview | Merged openstack/openstack-ansible-tests master: Remove unused centos-8 functional job definitions https://review.opendev.org/c/openstack/openstack-ansible-tests/+/820674 | 22:25 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org is being restarted briefly to apply a bugfix | 23:01 | |
| krumelmonster | jrosser: It didn't work on the first tries, I'll look into it further tomorrow. Thank you for your help. | 23:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!