Thursday, 2020-11-19

*** tosky has quit IRC00:14
*** spatel has joined #openstack-ansible01:19
*** spatel has quit IRC01:23
*** gyee has quit IRC01:38
*** macz_ has quit IRC02:08
*** cshen has joined #openstack-ansible02:30
*** cshen has quit IRC02:34
*** macz_ has joined #openstack-ansible03:20
*** d34dh0r53 has quit IRC03:24
*** macz_ has quit IRC03:25
*** d34dh0r53 has joined #openstack-ansible03:27
*** cshen has joined #openstack-ansible04:30
*** cshen has quit IRC04:35
*** ianychoi has joined #openstack-ansible04:36
*** rh-jlabarre has quit IRC04:58
*** macz_ has joined #openstack-ansible05:02
*** macz_ has quit IRC05:06
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
*** kleini has quit IRC05:35
*** macz_ has joined #openstack-ansible06:29
*** cshen has joined #openstack-ansible06:31
*** macz_ has quit IRC06:34
*** cshen has quit IRC06:35
*** cshen has joined #openstack-ansible06:37
*** cshen has quit IRC06:42
*** pcaruana has joined #openstack-ansible07:49
*** rpittau|afk is now known as rpittau08:03
*** cshen has joined #openstack-ansible08:10
*** andrewbonney has joined #openstack-ansible08:10
*** miloa has joined #openstack-ansible08:22
*** tosky has joined #openstack-ansible08:44
*** mmethot_ has quit IRC09:14
*** mmethot_ has joined #openstack-ansible09:14
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_service master: Add possibility to configure systemd sockets  https://review.opendev.org/76319409:16
*** luksky has joined #openstack-ansible10:18
*** d34dh0r53 has quit IRC10:21
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron stable/ussuri: Add neutron_keepalived_no_track variable  https://review.opendev.org/76333610:22
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron stable/train: Add neutron_keepalived_no_track variable  https://review.opendev.org/76333710:22
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron stable/train: Add neutron_keepalived_no_track variable  https://review.opendev.org/76333710:29
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron stable/train: Add neutron_keepalived_no_track variable  https://review.opendev.org/76333710:29
*** kleini has joined #openstack-ansible11:06
*** pto has quit IRC11:31
*** pto_ has joined #openstack-ansible11:31
*** pto_ has quit IRC11:44
*** pto has joined #openstack-ansible11:45
*** macz_ has joined #openstack-ansible11:46
*** macz_ has quit IRC11:51
*** pto has quit IRC12:35
*** pto has joined #openstack-ansible12:36
*** gshippey has joined #openstack-ansible12:40
*** rh-jlabarre has joined #openstack-ansible13:07
*** jbadiapa has joined #openstack-ansible13:09
*** rpittau is now known as rpittau|brb13:16
*** watersj has joined #openstack-ansible13:46
*** spatel has joined #openstack-ansible13:59
*** cshen has quit IRC14:10
fanfihi all, could someone help me please, how I can enable Image-Volume cache for ceph backend ? image_volume_cache_enabled = True14:12
*** pto has quit IRC14:14
noonedeadpunkI think you should just set `image_volume_cache_enabled: true` while defining backends for cinder14:14
*** pto has joined #openstack-ansible14:14
fanfioki, it will be easy than :) i will try this way14:16
fanfithx14:16
noonedeadpunkoh, I missed the part where you need extra tenant for that14:16
spateljrosser: noonedeadpunk can you check this - http://paste.openstack.org/show/800212/14:17
spatellook like i need flat network block back for octavia right?  or just add flat mechanism in neutron server?14:18
*** pto has quit IRC14:19
spatelI meant just add flat in this place on neutron server14:19
-spatel- tenant_network_types = vxlan,vlan,flat14:19
-spatel- type_drivers = flat,vlan,vxlan,local14:19
noonedeadpunkfanfi: well we create only cinder tenant which is used for service. ofc you can use it or create manually new one without admin privileges. after this you will be able to set `cinder_internal_tenant_project_id` and `cinder_internal_tenant_user_id` in cinder.conf with  cinder_cinder_conf_overrides14:20
noonedeadpunkspatel: I think we do have assumption that flat network is present and tbh I never tried doing otherwise14:22
spatelnoonedeadpunk: i thought i am using all vlan so i have removed flat type and now realized it need for octavia14:23
noonedeadpunkwell, maybe it's not really needed - not sure tbh....14:23
spatelOctavia use lbaas-mgmt on flat network.. look like14:24
spatelbased on error when its trying to create mgmt network and saying i don't have provider14:24
noonedeadpunkI have no idea, honestly.. Maybe it's possible to make it on vlan network instead...14:25
*** d34dh0r53 has joined #openstack-ansible14:25
spatelhttp://paste.openstack.org/show/800213/14:25
spatellet me add flat type in neutron and see if it make sense14:26
noonedeadpunkeventually you will need mapping then as well which is equal to fully returning of the section back14:27
spatelblock in openstack_user_config.yml ?14:27
noonedeadpunkyeah, as you need to tell neutron what flat network is on host/container14:28
spatelhmm14:31
gixxHello. Is it possible/viable to jump multiple releases at once? I.e. not upgrading incrementally until desired release. Running Queens at the moment, planning to upgrade to Ussuri and it'd be nice to not have to OS reinstall twice14:35
noonedeadpunkgixx: well, I was running direct rocky -> train upgrades, but I'd say it's something not tested, as Q->U really _a lot_ of things has changed and in upgrade script we are storing only relevant upgrade bits14:36
noonedeadpunkie galera upgrade might be tricky for instance14:37
noonedeadpunkso you should be really aware of what you're doing14:37
gixxWe're fine with doing some manual things to get it working14:37
noonedeadpunkbut techincally I think it's feasable.14:38
gixxDB upgrades for each project should be fine? As they build upon each other, all intermediate ones should be performed in order as I've interpreted it14:39
*** yolanda has quit IRC14:40
noonedeadpunkyeah, you're right, DB upgrades of project should be done properly, as they will try to perform all chain of migrations, so this should be ok14:40
noonedeadpunkthe thing that you should take care of is placement separation which is done in S->T and galera upgrade itself14:41
noonedeadpunkas there would be pretty solid jump -we use 10.4 in U14:41
gixxAlright thanks for the heads up14:41
noonedeadpunkand ofc you should check all of your overrides14:42
noonedeadpunkand probably add new as behaviour of services might change between releases14:42
gixxYeah, I'm sure we've done noooothing custom at all there..14:43
gixxHint hint custom cloudkitty and adjutant projects :) Will be nice to get rid of the custom stuff14:43
noonedeadpunkwell, we have cloudkitty role for a while and about to add adjutant :)14:44
noonedeadpunkhopefully for V14:44
gixxDid you get our role working eventually in master? Or did you have to scrap it all and start anew?14:44
gixxTo be honest I kind of forgot all about it, sorry14:44
noonedeadpunkwell, I did that patch:) https://review.opendev.org/#/c/756313/14:45
noonedeadpunkseems everybody is scared to approve it lol14:45
gixxlol14:45
noonedeadpunkbut it was passing and seemed working14:45
gixxI'm not proud if it at all. Getting py3 working in a Q deployment is not pretty at all14:46
noonedeadpunkadjutant has not tempest, so....14:46
gixxSo I'm sure it weren't up to standards at all14:46
noonedeadpunkeventually we're just using uwsgi where we can, and things really changed dramatically overall14:46
noonedeadpunklike having systemd_role and same across roles and synced db_setup and service_setup files14:47
noonedeadpunkIt still needs some work, but I hope it can be done on top of this patch...14:48
gixxThat's nice, makes things much simpler and cleaner14:48
noonedeadpunkI didn't have much time to work on it either14:49
noonedeadpunkso yeah:(14:49
*** rpittau|brb is now known as rpittau14:51
noonedeadpunkbtw cores, can we really merge this?:) https://review.opendev.org/#/c/756313/14:52
noonedeadpunkand iterate on it futher?14:53
noonedeadpunkas we can't do proper integrated testing until we got it merged (and can't set https://review.opendev.org/#/c/756310/ dependant on it either)14:54
gixxSo in short, there's some database changes in S->T we have to take into consideration and manually upgrade galera. Check all overrides as well as new defaults. The S->T placement separation, where can I find more information regarding it? Is it this (https://docs.openstack.org/placement/latest/admin/upgrade-to-stein.html)? So we have to upgrade nova to Stein release, perform the database split and then resume15:12
gixxwith rest of the upgrade15:12
*** tosky has quit IRC15:18
*** tosky has joined #openstack-ansible15:22
*** tosky has quit IRC15:26
*** macz_ has joined #openstack-ansible15:39
noonedeadpunkgixx: eventually in S and T we used same galera version but from Q to U it changed from 10.1 to 10.4. Regarding placement upgrade - yes, that's the right link. We had  https://review.opendev.org/#/c/664867/ and https://review.opendev.org/#/c/689145/ (but not limited to)15:46
noonedeadpunkthe thing is that we've dropped https://opendev.org/openstack/openstack-ansible-os_placement/src/branch/stable/train/templates/placement-migrate-db.rc.j2 in U15:46
spatelnoonedeadpunk: i found my octavia issue related flat networking, we don't need flat networking. This part of missing - http://paste.openstack.org/show/800222/15:47
*** tosky has joined #openstack-ansible15:50
*** miloa has quit IRC15:52
noonedeadpunkyeah, cool, as I thought you can choose network to be used:)15:53
spatelanyway all good, but that error message was little misleading. it should say flat provider is disabled15:59
spatelshouldn't*15:59
spateljamesdenton: are you around? had question related Trex packet generator16:00
jamesdentoni'm around now16:04
*** miloa has joined #openstack-ansible16:12
spatelI have Gen9 blades which has only two NIC card and i believe for Trex we need min 3 nic card, 1 for ssh and other two for DPDK RX and TX right?16:14
*** evrardjp has quit IRC16:17
spateljamesdenton: ^^16:19
jamesdentongood question.16:23
jamesdentoni would think so. but maybe iLo is OK?16:24
spatelthat is what i am thinking to use iLO for mgmt and use both nic for traffic16:31
*** miloa has quit IRC16:32
*** mgariepy has quit IRC16:34
*** gyee has joined #openstack-ansible16:34
*** mgariepy has joined #openstack-ansible16:36
jrosserspatel: sorry missed the octavia question - it does not need to be a flat network if you want to use vlan16:43
jrosserpersonally i would use vlan always then there is no need to mess with the config on the compute nodes to define the flat network16:44
*** gyee has quit IRC16:50
*** cshen has joined #openstack-ansible16:51
*** rpittau is now known as rpittau|afk16:52
*** cshen has quit IRC16:55
spateljrosser: totally it was my fault that i missed to define octavia parameter.16:59
*** d34dh0r53 has quit IRC17:02
jamesdentonspatel i recently deployed octavia w/o the br-lbaas bridge, just configured a lbaas-mgmt network as a vlan network (like a provider network) and made sure it was reachable from control plane net. worked OK17:03
*** macz_ has quit IRC17:05
*** gyee has joined #openstack-ansible17:05
*** d34dh0r53 has joined #openstack-ansible17:06
*** bverschueren has quit IRC17:06
*** bverschueren has joined #openstack-ansible17:08
spateljrosser: jamesdenton this method i am using to deploy Octavia with br-lbaas (james - do you using different way?) - https://satishdotpatel.github.io//openstack-ansible-octavia/17:08
jamesdentonspatel http://paste.openstack.org/show/799855/ https://ibb.co/K5NcCjZ17:11
jamesdentonalright, i have to step out. bbiaf17:12
spateljrosser: oh wait can you show me your openstack_user_config.yml ?17:14
spatelThis is what i have http://paste.openstack.org/show/800227/17:15
spatelyou are telling me i don't need br-lbaas block in openstack_user_config.yml?17:15
jrosserno, you have to have it there in order to assign IPs to eth14 on the ocavia containers if you do it like you are currently17:17
*** bverschueren has quit IRC17:18
*** bverschueren has joined #openstack-ansible17:18
*** klamath_atx has joined #openstack-ansible17:20
jrosserspatel: james has shown you a config where you could get rid of eth14 entirely on the octavia containers17:20
jrosseryou can choose which way you want to do it17:20
spateli am ok to remove br-lbaas block if that make sense and simplify deployment. ( trying to understand if i remove br-lbaas then how octavia container wire up with VLAN 62 in my case to go out)17:23
*** pcaruana has quit IRC17:25
spatelwhat is octavia_service_net_setup: True  does here?17:25
jrosserwhat james has done is use the default route on the container to directly get to the lbaas network17:26
jrosserso you have to configure your hardware router (hopefully firewall) to allow that traffic17:26
jrosserso 'it depends' if thats suitable or not17:26
spateljrosser: oh! i know what you saying.17:27
spateli have full control on my router and i can make it routable but let me stick to my way because i have same method in other datacenter so it will make documentation easy17:28
*** pcaruana has joined #openstack-ansible17:29
*** pcaruana has quit IRC17:31
*** pcaruana has joined #openstack-ansible17:32
*** mmercer has quit IRC17:35
*** macz_ has joined #openstack-ansible17:53
*** pcaruana has quit IRC17:54
*** macz_ has quit IRC18:08
*** gshippey has quit IRC18:14
ThiagoCMCHey guys, I'm wondering here about the minimum requirements for using letsencrypt with OSA. So far, I found two pages: https://docs.openstack.org/openstack-ansible-haproxy_server/ussuri/configure-haproxy.html and https://docs.openstack.org/openstack-ansible/ussuri/user/security/ssl-certificates.html18:34
admin0ThiagoCMC, provide the cert and it will work18:35
admin0private ip -> internal IP mapping can happen in haproxy , vyos router or physical router, so there is no fixed place where its done18:35
ThiagoCMCCan I just stick with the "haproxy_ssl_letsencrypt_*" vars and ignore the rest?18:35
ThiagoCMCSure...18:36
admin0i just generate it from zerossl ( gui based) and then provide to the right location18:36
admin0as most of the time, its internal or 1:1 is provided by router18:36
ThiagoCMCYeah, I have self-signed now but, some services complain about it, like Magnum and a few others18:36
admin0use zerossl :)18:36
admin0use a real cert18:36
ThiagoCMCHmm... LOL18:37
ThiagoCMCChecking it now!18:37
ThiagoCMC:-D18:37
admin0its the same free cert as letsencrypt .. just that you can use the GUI to generate it18:37
admin0so no need to install certbot etc in osa servers18:37
ThiagoCMCNice! Letsencrypt is so cool!18:37
ThiagoCMCBTW, the command `openstack coe cluster list --insecure` returns: "SSL exception connecting to https://10 (HTTP 500)" - I bet that's because of my creepy self-signed cert, right?18:38
ThiagoCMCMasakari also doesn't like it18:38
admin0having a real(free) ssl should be a pre-deployment requirement18:39
ThiagoCMCVery true.18:39
*** NewJorg has quit IRC18:42
*** andrewbonney has quit IRC18:45
*** NewJorg has joined #openstack-ansible18:49
*** cshen has joined #openstack-ansible18:51
*** macz_ has joined #openstack-ansible18:51
*** cshen has quit IRC18:56
openstackgerritMerged openstack/openstack-ansible master: Added Netplan examples for a Openstack Infrastructure node.  https://review.opendev.org/76009019:04
ThiagoCMCWhen I click at the above "gitweb" link, it shows "Not found". Link: https://review.opendev.org/gitweb?p=openstack%2Fopenstack-ansible.git;a=commitdiff;h=461071f1baedad7851eecb78428e750cd752d4d919:14
ThiagoCMCJust me?19:14
*** macz_ has quit IRC19:19
*** macz_ has joined #openstack-ansible19:37
*** cshen has joined #openstack-ansible19:55
*** watersj has quit IRC20:02
*** watersj has joined #openstack-ansible20:04
spateldo you guys use MTU 9000 for everything or just ceph storage network?20:21
*** luksky has quit IRC20:26
*** luksky has joined #openstack-ansible20:26
admin0everything possible :)20:36
admin0br-mgmt and br-vxlan also20:36
ThiagoCMCspatel, I just realized how slow is the RBD access from my compute nodes... Gonna try 9000 here!  =P - I might need a 2.5Gbps switch lol20:38
ThiagoCMCI'm wondering if there are multiqueue to access RBD block devices, or something else.20:39
spatelhow much slow your RBD access?20:39
spatelrun iperf test and see20:40
spatelI would definitely say use 9000 MTU for br-storage20:40
spateladmin0: i like to move to MTU 9000 but in my case sometime firewall other device comes in path and if they have MTU 1500 then it will be more work from them to break packets20:41
ThiagoCMCI'm working to get the numbers but, so far, from an instance with rbd-based virtual disc, the bonnie++ doesn't even work properly, kernel prints timeout and it's huge iowait. I missed something, for sure.20:51
ThiagoCMCMaybe some image properties... lol20:52
*** jbadiapa has quit IRC20:52
mgariepyThiagoCMC, use fio.20:57
ThiagoCMCmgariepy, I just saw some messages at ceph mail list telling to use fio!  :-D20:58
ThiagoCMCOn it!20:58
ThiagoCMCThanks!20:58
mgariepyhaha ;D20:58
spatelwhat kind of SSD you have, that is very important because i used some cheap consumer SSD and found that was the bottleneck21:03
spatellater i replaced all those SSD with enterprise SSD and boom!!21:03
ThiagoCMCHmm... I have Samsung SSDs, for Ceph DBs21:04
spatelwhat model?21:04
spatelI do have Samsung also21:04
spatelif its EVO or PRO then don't expect any performance21:04
ThiagoCMCEVO 86021:05
ThiagoCMChehe21:05
spatelgood luck :)21:06
ThiagoCMCWell, I can "rbd ls" from the compute node bare-metal, I'm going to map the volumes and run some tests outside of the virtual instance to see.21:06
spatelyou need DCT Series21:06
ThiagoCMCI see21:06
ThiagoCMCThanks for the tip!21:06
ThiagoCMCIt's waaaay too slow from the QEMU Instance.21:06
ThiagoCMCI'll see what I can get from the compute node itself.21:07
spatelfirst make sure you don't have network bottleneck (and then try new SSD)21:07
ThiagoCMCEven the "apt install fio" within the QEMU instance is taking a long time21:07
ThiagoCMCok21:07
spatelWe have hired Ceph consultant to fix our issue and they said use Samsumg PM863a SSDs21:09
ThiagoCMCThat sounds expensive21:09
ThiagoCMCMy cloud is at my basement lol21:10
spatelthen you are good with EVO :)21:10
spateli thought you are running some enterprise cloud..21:10
spatelin that case EVO should be good man!!21:10
ThiagoCMCI was, at me previous job. But now I just want to build my own21:10
ThiagoCMCSure! I believe that I should get faster access with what I have21:11
ThiagoCMCI'll compare the rbd from the compute node itself, with virtualization libvirt<->rbd<->virtio-scsi in the mix.21:11
spatelwhen you saying slow (what is the definition of slow )21:12
spatelyour VM disk located on ceph right?21:12
ThiagoCMCyep21:13
ThiagoCMCLike, 99 iowait when running "apt install fio", and it took a long time to finish21:13
spateldo you have  (glance image-update --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi ) ?21:13
spateldo you have 1G network?21:14
ThiagoCMCYep, Gigalan network, and I didn't changed the image properties!21:15
ThiagoCMCGonna try it now.21:15
admin0ThiagoCMC,  "fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75"21:17
ThiagoCMCadmin0, awesome!!! Thank you!21:17
admin0first ran on atl when testing that qnap over nfs and iscsi21:18
ThiagoCMCOh, damn... I remember that!  lol21:18
admin0i had all the fio commands and their output saved in onenote ..21:18
admin0TBW for samsung is not that much .. 1200TB21:27
admin0compared to intel ones, with 10200 TB21:28
ThiagoCMCDamn21:28
ThiagoCMCadmin0, have you used the fio against the block device itself? I want to map Ceph's volume at the compute node, using "rbd map" thing and benchmark agains, let's say "/dev/rbd0"21:31
admin0i have not .. i did it for the instance ephemeral disk ( via ceph) and cinder ( via nfs, and iscsi)21:32
ThiagoCMCok21:32
admin0unless the stakeholders say lets benchmark /dev/rbd0 from kvm directly and decide if we use openstack or not, i don't see a value in getting that benchmark .. we just end up with 2 numbers :)21:33
admin0your time is better used by getting magnum, octavia, trove, ovn to work :D21:34
ThiagoCMCLOL21:34
ThiagoCMCI'll get ther!21:34
ThiagoCMCMaybe "fio --filename=/dev/rbd0 ... "21:34
ThiagoCMCWhat about that Tacker thing, are you guys using it?21:35
admin0nope21:36
*** luksky has quit IRC21:41
*** watersj has quit IRC21:47
*** luksky has joined #openstack-ansible21:54
*** spatel has quit IRC22:16
*** cshen has quit IRC22:42

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!