| jamesdenton | :) | 00:11 |
|---|---|---|
| jamesdenton | cool, lemme know | 00:11 |
| *** macz_ has quit IRC | 00:12 | |
| *** macz_ has joined #openstack-ansible | 00:29 | |
| *** macz_ has quit IRC | 00:33 | |
| *** spatel has joined #openstack-ansible | 01:03 | |
| *** cshen has joined #openstack-ansible | 01:32 | |
| *** cshen has quit IRC | 01:37 | |
| *** spatel has quit IRC | 01:55 | |
| *** spatel has joined #openstack-ansible | 02:33 | |
| *** spatel has quit IRC | 02:37 | |
| ThiagoCMC | Is it okay to compress the Ceph pool of Glace, the "images", but not the Nova's "vms", neither Cinder's "volumes/backups"? If yes, will Ceph uncompress the image from Glance pool when I launch an Instance? | 03:31 |
| *** cshen has joined #openstack-ansible | 03:32 | |
| *** cshen has quit IRC | 03:37 | |
| ThiagoCMC | Wheee! My OpenStack with Ceph is finally working in my basement with gaming PCs! :-D | 04:20 |
| *** cshen has joined #openstack-ansible | 05:33 | |
| *** cshen has quit IRC | 05:37 | |
| *** evrardjp has quit IRC | 05:47 | |
| *** evrardjp has joined #openstack-ansible | 05:48 | |
| *** miloa has joined #openstack-ansible | 06:23 | |
| *** cshen has joined #openstack-ansible | 06:30 | |
| *** cshen has quit IRC | 06:34 | |
| *** SiavashSardari has joined #openstack-ansible | 06:47 | |
| *** cshen has joined #openstack-ansible | 07:30 | |
| *** cshen has quit IRC | 07:35 | |
| *** rpittau|afk is now known as rpittau | 07:37 | |
| *** cshen has joined #openstack-ansible | 08:12 | |
| *** luksky has joined #openstack-ansible | 08:15 | |
| *** spatel has joined #openstack-ansible | 08:25 | |
| *** spatel has quit IRC | 08:30 | |
| *** pto has quit IRC | 08:36 | |
| *** tosky has joined #openstack-ansible | 08:50 | |
| *** pto has joined #openstack-ansible | 08:55 | |
| *** andrewbonney has joined #openstack-ansible | 08:56 | |
| *** sep has quit IRC | 08:59 | |
| *** sep has joined #openstack-ansible | 09:01 | |
| *** jbadiapa has joined #openstack-ansible | 09:04 | |
| *** pto has quit IRC | 09:05 | |
| *** pto has joined #openstack-ansible | 09:06 | |
| *** mgautier has joined #openstack-ansible | 09:16 | |
| *** d34dh0r53 has quit IRC | 09:17 | |
| *** d34dh0r53 has joined #openstack-ansible | 09:20 | |
| *** luksky has quit IRC | 10:26 | |
| *** openstackgerrit has joined #openstack-ansible | 10:32 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Return calico to voting https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/702657 | 10:32 |
| *** pto has quit IRC | 10:45 | |
| *** pto has joined #openstack-ansible | 11:19 | |
| *** pto has quit IRC | 11:21 | |
| *** pto has joined #openstack-ansible | 11:21 | |
| *** spatel has joined #openstack-ansible | 11:26 | |
| *** spatel has quit IRC | 11:31 | |
| *** luksky has joined #openstack-ansible | 11:32 | |
| *** rfolco|ruck has joined #openstack-ansible | 11:36 | |
| *** SiavashSardari has quit IRC | 11:37 | |
| *** pto has quit IRC | 11:54 | |
| *** pto has joined #openstack-ansible | 11:55 | |
| *** klamath_atx has quit IRC | 12:59 | |
| *** klamath_atx has joined #openstack-ansible | 13:00 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/758751 | 13:22 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Rename nova_metadata_* variables https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/760149 | 13:23 |
| *** yann-kaelig has joined #openstack-ansible | 13:31 | |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_cloudkitty master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/764167 | 13:50 |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_designate master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/764168 | 13:52 |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_glance master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/764169 | 13:54 |
| *** sshnaidm is now known as sshnaidm|afk | 13:56 | |
| jrosser | i'm really still not sure about these tons of tox patch | 13:58 |
| jrosser | i used codesearch and there are really very many repos who have it set up just like the OSA ones | 13:59 |
| jrosser | so either theyre *all* wrong....... | 13:59 |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_gnocchi master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/764170 | 14:01 |
| noonedeadpunk | maybe it's scripted.... | 14:02 |
| *** pto has quit IRC | 14:02 | |
| noonedeadpunk | and made wrong assumption | 14:02 |
| noonedeadpunk | let we check with infra then | 14:02 |
| *** spatel has joined #openstack-ansible | 14:04 | |
| *** chandankumar has joined #openstack-ansible | 14:12 | |
| chandankumar | noonedeadpunk: jrosser please have a look at these patches https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/761019 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/761021 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/761021 | 14:14 |
| chandankumar | thanks! | 14:14 |
| rfolco|ruck | chandankumar, thanks | 14:15 |
| *** cshen has quit IRC | 14:15 | |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_heat master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/764173 | 14:19 |
| jrosser | chandankumar: i left a comment on the first one and the second looks like it has an unresolved comment already | 14:21 |
| chandankumar | rfolco|ruck: ^^ | 14:22 |
| chandankumar | jrosser: thanks! | 14:23 |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_horizon master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/764196 | 14:24 |
| *** rpittau is now known as rpittau|bbl | 14:29 | |
| *** sshnaidm|afk is now known as sshnaidm | 14:34 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Return calico to voting https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/702657 | 14:37 |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_ironic master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/764198 | 14:39 |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_keystone master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/764199 | 14:39 |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_magnum master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/764200 | 14:40 |
| openstackgerrit | wu.chunyang proposed openstack/openstack-ansible-os_manila master: Dep's should be restricted by tox-constraints https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/764201 | 14:40 |
| *** SiavashSardari has joined #openstack-ansible | 14:40 | |
| *** cshen has joined #openstack-ansible | 15:07 | |
| openstackgerrit | Rafael Folco proposed openstack/openstack-ansible-os_tempest stable/train: Switch tripleo job to content provider https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/761021 | 15:20 |
| *** SiavashSardari has quit IRC | 15:25 | |
| *** rpittau|bbl is now known as rpittau | 15:44 | |
| *** lemko7 has quit IRC | 15:48 | |
| *** lemko has joined #openstack-ansible | 15:48 | |
| *** luksky has quit IRC | 16:04 | |
| spatel | jamesdenton: are you there (tag me when you reply back, otherwise i will miss you mesg) | 16:19 |
| *** chandankumar is now known as raukadah | 16:30 | |
| jamesdenton | hi | 16:33 |
| jamesdenton | spatel ^^ | 16:33 |
| spatel | Hey | 16:33 |
| spatel | I am running Trex load test and numbers are very odd (general VM vs SR-IOV vm) | 16:34 |
| spatel | let me post result hold on | 16:34 |
| spatel | jamesdenton: https://pastebin.com/aRMRd4M6 | 16:34 |
| spatel | I have create 2 vm first one is standard VM using virtio and second one using SR-IOV | 16:35 |
| spatel | only difference is sriov can handle 100k more compare to standard virtio (that is very odd0 | 16:35 |
| spatel | I can't believe that | 16:36 |
| spatel | Do you think linux kernel or context switch creating bottleneck here | 16:36 |
| jamesdenton | trying to find my notes | 16:38 |
| spatel | jamesdenton: you should put them on blog :) | 16:38 |
| jamesdenton | well, if i felt confident they were accurate :D | 16:39 |
| jamesdenton | what did you command line syntax look like? | 16:39 |
| jamesdenton | my notes show sfr_delay_10_1G x10 (for 10G) for 120s, which was ~40,000cps and 2,000,000pps | 16:41 |
| *** irclogbot_0 has quit IRC | 16:41 | |
| spatel | ./t-rex-64 --cfg /etc/trex_cfg.yaml.SRIOV-PIN -f /tmp/udp-64.yaml -d 30 -m 3500 -c 10 -k 16 --nc | 16:41 |
| jamesdenton | but i was looking for packet loss there, which was 0.89% for sriov and 96% for virtio | 16:41 |
| spatel | what is sfr_delay_10_1G does? | 16:42 |
| jamesdenton | you do have ixgbevf loaded in the VM under test? | 16:42 |
| spatel | I am using udp packet with lowest value 64bytes | 16:42 |
| jamesdenton | i think sfr_delay_10_1G was just a mix | 16:42 |
| jamesdenton | it's been almost 2 yrs so i don't remember | 16:43 |
| -spatel- [root@sriov-dut-pinned ~]# ethtool -i ens4 | 16:43 | |
| -spatel- driver: ixgbevf | 16:43 | |
| jamesdenton | kk | 16:43 |
| jamesdenton | and cpu governor set to performance? | 16:43 |
| spatel | I didn't do any of those setting yet.. | 16:44 |
| spatel | Just simple create two vm vritio vs sriov | 16:44 |
| jamesdenton | k | 16:44 |
| spatel | i thought result should be 2x for sriov but not.. | 16:44 |
| spatel | i wonder if kernel is bottleneck | 16:44 |
| spatel | for testing i use metal machine and my trex hit 2mpps | 16:45 |
| spatel | that means my trex isn't bottleneck | 16:46 |
| *** irclogbot_1 has joined #openstack-ansible | 16:46 | |
| spatel | Do you think creating VF on nic reduce pipe speed or limit ? | 16:46 |
| spatel | curious how SRIOV spread bandwidth between all VF ? | 16:47 |
| jrosser | doesnt the interrupt have to be handed off to the vm by the hypervisor kernel though | 16:47 |
| jrosser | even for SRIOV | 16:47 |
| spatel | In SRIOV case host machine kernel out of the way so it should use any interrupt or context switching | 16:49 |
| spatel | now it comes to vm kernel who handling packets and how much switch it does to handle high pps. | 16:49 |
| spatel | I am wondering if my vm kernel is bottleneck here. | 16:50 |
| spatel | many folks saying SRIOV handle line-rate and it should handle 1mpps but in my case its not so how does people do their benchmark | 16:51 |
| spatel | I have tired to increase vcpu core on sriov VM from 8 to 16 and i gave me couple more thousand pps but not million | 16:51 |
| spatel | Trying to find way how VF handle NIC queueing.. because that is very important for performance | 16:52 |
| ThiagoCMC | spatel, have you tested the bare-metal itself (no virtualization, nothing, just the main OS), to see how much your hardware can handle? | 16:52 |
| spatel | bare-metal giving me 2million pps | 16:53 |
| spatel | fresh OS installed on bare-metal and without tuning i can hit 2mpps | 16:53 |
| ThiagoCMC | And the VM with SR-IOV? | 16:53 |
| ThiagoCMC | same OS...? | 16:54 |
| spatel | VM with SRIOV (8vcpu/8gb) only 334.92 Kpps | 16:54 |
| jamesdenton | i assume all cores are on the same numa as NIC? might be worth pinning to see if that helps some | 16:54 |
| spatel | same OS | 16:54 |
| spatel | Yes all 8cpu on NUMA 0 | 16:54 |
| jamesdenton | k | 16:54 |
| spatel | default openstack schedule them on same numa node | 16:55 |
| spatel | but let me verify again | 16:55 |
| jamesdenton | i think you're right, just not sure if it will bail if there aren't enough cpu avail | 16:55 |
| spatel | This comute node is totally empty in my lab and only single VM running | 16:56 |
| spatel | Let me show you interesting data, This is standard virtio VM and as i add more cpu for queue performance getting better - http://paste.openstack.org/show/800427/ | 16:56 |
| spatel | You can see first block using ethtool -L eth0 combined 1 | 16:57 |
| spatel | as i add more cpu for queue performance getting 2x better | 16:57 |
| spatel | Now i am looking how i can add more queue for SR-IOV Virtual-Function | 16:59 |
| *** d34dh0r53 has quit IRC | 17:00 | |
| spatel | jamesdenton: look like this is my issue, i have only 2 queue on SRIOV - VF nic - http://paste.openstack.org/show/800429/ | 17:02 |
| jrosser | on your bare metal host the queues and irq setup are mapped to the total number of cores you have | 17:02 |
| jrosser | like total in the whole host | 17:02 |
| jrosser | so that will look very different in a VM | 17:02 |
| *** d34dh0r53 has joined #openstack-ansible | 17:02 | |
| spatel | jrosser: yes.. mostly virtio vm has only single queue by default | 17:03 |
| spatel | This is very interesting thread - https://community.intel.com/t5/Ethernet-Products/Intel-NIC-82599-EB-enable-SR-IOV-and-multiqueue/td-p/387696 | 17:04 |
| spatel | VF supports 2 TX/RX queue pairs. | 17:04 |
| jamesdenton | https://community.intel.com/t5/Ethernet-Products/SR-IOV-multiqueue-support-with-ixgbe-ixgbevf-drivers/td-p/298350 | 17:04 |
| jamesdenton | hah yeah ok | 17:04 |
| spatel | i beat you :) | 17:05 |
| spatel | so we can't change this behavior | 17:05 |
| jamesdenton | you need some i40e | 17:05 |
| spatel | what is i40e ? | 17:06 |
| jamesdenton | the X710 family | 17:06 |
| spatel | let me check my bank account... hahaha | 17:07 |
| spatel | I totally with you. now its clear this is hardware limitation nothing to do with PPS | 17:07 |
| spatel | what if i create DPDK (may be DPDK give me full queue from host to VM) | 17:08 |
| *** luksky has joined #openstack-ansible | 17:09 | |
| jamesdenton | not sure, i did not pay attention to such things | 17:10 |
| jamesdenton | tried deploying dpdk w/ cx6 the other day but ran into some snags and didnt revisit | 17:10 |
| spatel | jamesdenton: i can quickly test that and i am going to do that.. | 17:12 |
| spatel | jamesdenton: when OVS+DPDK patch will get merge ? | 17:13 |
| spatel | otherwise i have to do all hand edit :( | 17:13 |
| jamesdenton | did you see my latest comment on that? | 17:14 |
| jamesdenton | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762729/5/vars/redhat.yml | 17:14 |
| spatel | reading... | 17:14 |
| spatel | on centOS i did binding manually | 17:15 |
| spatel | but that part we can handle with playbook right? | 17:16 |
| jamesdenton | If you're referring to this patch with the new repo, it has merged in master: https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/762661 | 17:16 |
| spatel | This is what i did on centOS - driverctl set-override 0000:06:00.1 vfio-pci | 17:16 |
| jamesdenton | oh right, yes, you have to do that manually as there is no service in place to facilitate that | 17:17 |
| spatel | can't we do that using playbook ? | 17:17 |
| jamesdenton | so if the dpdk service is an ubuntu-specific thing, then we need to add the 'when' clauses | 17:17 |
| jamesdenton | well, that won't survive a reboot | 17:18 |
| jamesdenton | which is why that dpdk service exists, more or less | 17:18 |
| spatel | really? | 17:18 |
| spatel | I think it can survive reboot.. if you want i can give it a try | 17:18 |
| jamesdenton | "The driver overrides created by | 17:18 |
| jamesdenton | driverctl are persistent across system reboots by default." so maybe im wrong | 17:19 |
| jamesdenton | it would be good to know, so yes, please try | 17:19 |
| spatel | i did reboot my compute multiple time and i didn't see any issue | 17:19 |
| jamesdenton | k | 17:19 |
| spatel | hold on let me try now | 17:19 |
| jamesdenton | i am taking my son to lunch, so will be back in a while | 17:19 |
| *** macz_ has joined #openstack-ansible | 17:19 | |
| jamesdenton | if you want to patch the playbooks, feel free | 17:20 |
| ThiagoCMC | Bon appetit! :-) | 17:20 |
| spatel | jamesdenton: currently i have this here - http://paste.openstack.org/show/800432/ | 17:20 |
| *** d34dh0r53 has quit IRC | 17:20 | |
| spatel | rebooting compute node now | 17:20 |
| *** miloa has quit IRC | 17:23 | |
| *** klamath_atx has quit IRC | 17:25 | |
| spatel | jamesdenton: compute node is back after reboot and i can see my NIC still bound to DPDK so it does survive reboot :) | 17:26 |
| ThiagoCMC | If it's declared at /etc/dpdk/interfaces, it should! =P | 17:27 |
| ThiagoCMC | DPDK is... Hard... | 17:27 |
| ThiagoCMC | Are you having fun with it spatel? lol | 17:28 |
| spatel | Lots of FUN and i think its pretty easy.. I didn't see any complication to run it | 17:28 |
| spatel | currently running in LAB but planning to use on production | 17:29 |
| spatel | if DPDK support multi-queue then it would be great | 17:29 |
| ThiagoCMC | It does | 17:30 |
| ThiagoCMC | I've never used it with OpenStack, only with plain QEMU hosts and OpenvSwitch. The OVS syntax to create bridges are so different | 17:30 |
| spatel | hmmm! let me create VM and run same load-test to find out how good its :) | 17:30 |
| *** mgautier has quit IRC | 17:37 | |
| jamesdenton | ThiagoCMC does /etc/dpdk/interfaces apply to centos? | 17:38 |
| jamesdenton | also spatel, the ansible ovs bond patches merged: https://github.com/ansible-collections/openvswitch.openvswitch | 17:39 |
| spatel | jamesdenton: centOS doesn't have this file /etc/dpdk/interfaces | 17:40 |
| spatel | +1 for bonding | 17:40 |
| jamesdenton | driverctl doesn't exist for 18.04, i see | 17:41 |
| spatel | in my case i can't use bonding because i have only 2 nic port :( | 17:41 |
| jamesdenton | right, ok | 17:41 |
| spatel | jamesdenton: i am thinking if i create multiple VF and bond them tether then i can utilize more queue from VF and give it to dpdk | 17:42 |
| jamesdenton | you can also bond VFs in the instance | 17:42 |
| jamesdenton | not sure if that will help | 17:42 |
| spatel | it will be complicated but should work | 17:42 |
| ThiagoCMC | jamesdenton, Oh, sorry about that... I don't know about DPDK on CentOS. | 17:44 |
| ThiagoCMC | Are you guys using "hw_qemu_guest_agent=yes" and "os_require_quiesce=yes" as Glance image properties for your OSA/Ceph deployments? | 17:46 |
| spatel | ThiagoCMC: i am not using that and not sure what those option for | 17:47 |
| ThiagoCMC | spatel, it explains here: https://docs.ceph.com/en/latest/rbd/rbd-openstack/ | 17:48 |
| jamesdenton | noonedeadpunk thx for the review. am i to take away that i should use the collections vs what is currently in place? | 17:48 |
| ThiagoCMC | But, there is a bug on that (apparently): https://bugs.launchpad.net/nova/+bug/1841160 | 17:48 |
| openstack | Launchpad bug 1841160 in OpenStack Compute (nova) "With libvirt/images_type = rbd, ephemeral instances silently ignore hw_qemu_guest_agent=yes" [Low,Confirmed] | 17:48 |
| noonedeadpunk | I think we are already using collections but old naming still works because of "routing" | 17:49 |
| jamesdenton | gotcha | 17:49 |
| noonedeadpunk | ie https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/master/tasks/service_setup.yml#L35 | 17:49 |
| jamesdenton | yes, ok | 17:49 |
| *** lemko8 has joined #openstack-ansible | 18:03 | |
| *** lemko has quit IRC | 18:03 | |
| *** lemko8 is now known as lemko | 18:03 | |
| *** luksky has quit IRC | 18:04 | |
| *** dave-mccowan has quit IRC | 18:04 | |
| *** dave-mccowan has joined #openstack-ansible | 18:11 | |
| *** jbadiapa has quit IRC | 18:14 | |
| *** sshnaidm has quit IRC | 18:23 | |
| *** sshnaidm has joined #openstack-ansible | 18:24 | |
| *** sshnaidm has quit IRC | 18:28 | |
| *** sshnaidm has joined #openstack-ansible | 18:28 | |
| *** rpittau is now known as rpittau|afk | 18:52 | |
| *** andrewbonney has quit IRC | 18:53 | |
| *** yann-kaelig has quit IRC | 19:12 | |
| *** macz_ has quit IRC | 19:31 | |
| *** macz_ has joined #openstack-ansible | 19:35 | |
| *** klamath_atx has joined #openstack-ansible | 19:41 | |
| admin0 | why does this task -- TASK [python_venv_build : Install python packages into the venv .. a part of setup-infra in the util container take a very very long time | 20:46 |
| admin0 | setup-infra fails with openstack command not found: https://gist.github.com/a1git/71affe8d6717f40fb16017a7a5ac7627 | 20:53 |
| admin0 | destroying utilty and re-creating it ends up in the same spot | 20:53 |
| *** rfolco|ruck has quit IRC | 21:08 | |
| admin0 | haproxy was dying for some reasons .. and the process just hung without giving a timeout | 21:24 |
| *** cshen has quit IRC | 21:52 | |
| *** cshen has joined #openstack-ansible | 22:16 | |
| *** klamath_atx has quit IRC | 22:32 | |
| *** klamath_atx has joined #openstack-ansible | 22:33 | |
| *** klamath_atx has quit IRC | 22:38 | |
| *** macz_ has quit IRC | 22:54 | |
| *** spatel has quit IRC | 23:11 | |
| *** tosky has quit IRC | 23:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!