| djhankb | Ok friends, thanks again for everyone's help in answering my questions - I'm about 95% on my migration and am stuck on an issue with RabbitMQ. I am unable to load my console in Horizon nor get the console URL using the openstack CLI. Nova loses connection to RabbitMQ and RabbitMQ sprays 100's of "channel exception not_found" errors: http://paste.openstack.org/show/789045/ I'm using Hyper-V and the Cloudbase FreeRDP plugin - of which I | 03:36 |
|---|---|---|
| djhankb | 'm not sure if that is the problem or if its something else. I cannot load the console log either FWIW. | 03:36 |
| *** raukadah is now known as chkumar|rover | 04:55 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:34 | |
| *** udesale has joined #openstack-ansible | 05:49 | |
| *** viks___ has joined #openstack-ansible | 05:54 | |
| *** errr has quit IRC | 05:56 | |
| *** errr has joined #openstack-ansible | 05:57 | |
| *** southquist has joined #openstack-ansible | 06:20 | |
| *** jstephan has joined #openstack-ansible | 07:05 | |
| *** southquist has quit IRC | 07:09 | |
| *** hwoarang has quit IRC | 07:36 | |
| *** hwoarang has joined #openstack-ansible | 07:37 | |
| *** Adri2000 has joined #openstack-ansible | 07:37 | |
| *** jbadiapa has joined #openstack-ansible | 07:39 | |
| *** pcaruana has joined #openstack-ansible | 07:43 | |
| openstackgerrit | Erik Berg proposed openstack/openstack-ansible-repo_server stable/rocky: Exclude repo_build_global_links_path from sync https://review.opendev.org/705390 | 07:48 |
| *** southquist has joined #openstack-ansible | 07:53 | |
| *** cshen has joined #openstack-ansible | 07:55 | |
| *** kopecmartin has joined #openstack-ansible | 08:00 | |
| *** rpittau|afk is now known as rpittau | 08:05 | |
| *** luksky has joined #openstack-ansible | 08:09 | |
| *** mathlin has quit IRC | 08:11 | |
| *** tosky has joined #openstack-ansible | 08:29 | |
| *** miloa has joined #openstack-ansible | 08:47 | |
| *** DanyC has joined #openstack-ansible | 08:50 | |
| mnaser | jrosser, noonedeadpunk, guilhermesp: i'm trying out using `strategy: free` for certain plays that dont need any ordering | 08:54 |
| mnaser | i think it will help a lot with bigger scale deploys | 08:54 |
| *** yolanda has joined #openstack-ansible | 09:03 | |
| openstackgerrit | Erik Berg proposed openstack/openstack-ansible-repo_server stable/rocky: Exclude repo_build_global_links_path from sync https://review.opendev.org/705390 | 09:29 |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible master: DNM - POC - Collect logs https://review.opendev.org/702676 | 09:29 |
| *** sshnaidm|off is now known as sshnaidm | 09:30 | |
| *** maharg101 has joined #openstack-ansible | 09:48 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-openstack_hosts master: Disable loading nf_nat_ipv4 in new kernels https://review.opendev.org/705403 | 09:49 |
| openstackgerrit | Erik Berg proposed openstack/openstack-ansible-repo_server master: lsyncd: use relative paths in exclude statement https://review.opendev.org/705404 | 09:51 |
| *** gshippey has joined #openstack-ansible | 09:54 | |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-os_keystone stable/train: Replace Module https://review.opendev.org/705408 | 09:59 |
| mnaser | gshippey: has that change worked well for you, cause we dont really test it in gate | 10:05 |
| gshippey | i've got some other patches that go on top of it I also need to backport in - that make it work | 10:05 |
| mnaser | gshippey: ok cool, so unfortunately that one is broken? :P | 10:06 |
| gshippey | That patch on its own will create brokenness, https://review.opendev.org/#/c/702920/, https://review.opendev.org/#/c/702714/, https://review.opendev.org/#/c/703462/ fix the brokenness | 10:08 |
| gshippey | could use a review on this one https://review.opendev.org/#/c/703462/ | 10:11 |
| mnaser | gshippey: done :) | 10:25 |
| *** jbadiapa has quit IRC | 10:49 | |
| jrosser | o/ morning i have a stack of patches which finishes here to make tempest install on stable/train sensible https://review.opendev.org/#/c/705336/ | 11:09 |
| jrosser | https://review.opendev.org/#/c/703572/ and this stack of 3 are needed https://review.opendev.org/#/c/704978/ | 11:10 |
| *** cshen has quit IRC | 11:29 | |
| *** bhyrted has joined #openstack-ansible | 11:37 | |
| bhyrted | how to integrate role such as cloudkitty into openstack-ansible train? | 11:38 |
| noonedeadpunk | jrosser: I think we also need cherry-pick https://review.opendev.org/#/c/704946 later? | 11:38 |
| noonedeadpunk | bhyrted: I think we have cloudkitty role (not really well tested though). | 11:38 |
| bhyrted | https://docs.openstack.org/openstack-ansible-os_cloudkitty/latest/ | 11:39 |
| bhyrted | hmm looks like it, i need to learn then how to integrate it | 11:39 |
| bhyrted | so i can deploy it on containers | 11:39 |
| * noonedeadpunk checking | 11:39 | |
| jrosser | noonedeadpunk: yes it'll to be a revert of https://review.opendev.org/#/c/704978/ as i've had to put n-v on debian too now | 11:39 |
| *** rpittau is now known as rpittau|bbl | 11:42 | |
| noonedeadpunk | Ok, I think we're missing env.d defenition for cloudkitty.... | 11:51 |
| *** cshen has joined #openstack-ansible | 11:51 | |
| *** luksky has quit IRC | 11:52 | |
| jrosser | noonedeadpunk: we're not using the integrated tests for cloudkitty, would be good to fix that up which would resolve a lot of that | 11:56 |
| noonedeadpunk | Yeah, it was in todo list, but so stuck with manila( | 11:57 |
| *** cshen has quit IRC | 11:58 | |
| mnaser | noonedeadpunk: thats weird, the ubuntu thing for kernel 5.x | 12:01 |
| mnaser | its somehow still breaking for me on an ubuntu machine | 12:01 |
| jrosser | is this ceph? | 12:01 |
| mnaser | jrosser: no, its openstack_hosts trying to load `nf_nat_ipv4` when it doesn't exist in kernel >5.x anymore | 12:02 |
| noonedeadpunk | humm... mnaser maybe it's not bumped yet? | 12:02 |
| mnaser | ohhhh | 12:02 |
| mnaser | yes you're right | 12:02 |
| mnaser | thats probably it | 12:02 |
| jrosser | we had 0% success with ceph OSD on a 5.x kernel, really didnt try very hard to understand what was going on though, just rolled back to HWE kernel | 12:03 |
| noonedeadpunk | Yeah, it's in https://review.opendev.org/#/c/704252/1/ansible-role-requirements.yml | 12:03 |
| noonedeadpunk | which is not merged... | 12:03 |
| *** jbadiapa has joined #openstack-ansible | 12:03 | |
| mnaser | slow morning | 12:04 |
| mnaser | jrosser: that's weird, ceph is pretty non-kernel-y ? | 12:04 |
| jrosser | you'd have thought :) | 12:04 |
| jrosser | it was to get ECC working on AMD rome which it moans about an awful lot otherwise | 12:04 |
| bhyrted | how about this env.d file https://opendev.org/openstack/openstack-ansible-os_cloudkitty/src/branch/master/extras/env.d/cloudkitty.yml ? | 12:05 |
| mnaser | jrosser: ah i see, i really think rome gets a lot of its benefits on much more modern kernels :x | 12:06 |
| noonedeadpunk | bhyrted: yep, exactly | 12:06 |
| noonedeadpunk | bhyrted: wait for a minute - I'll write you down what to do with it | 12:07 |
| bhyrted | mnaser: i think there is no nf_nat_ipv4 however i think nf_nat exist, we just override it using openstack_host_kernel_modules | 12:07 |
| mnaser | bhyrted: we're fixing that upstream so no one has to override things locally for the future :) | 12:08 |
| bhyrted | however we got less pefformance on ceph with 5.x kernel than hwe | 12:08 |
| bhyrted | <noonedeadpunk>: thanks | 12:08 |
| bhyrted | <mnaser> thats true ;-) | 12:08 |
| *** luksky has joined #openstack-ansible | 12:17 | |
| *** cshen has joined #openstack-ansible | 12:19 | |
| noonedeadpunk | bhyrted: so I think you should create file /etc/openstack_deploy/env.d/cloudkitty.yml with the following content: http://paste.openstack.org/show/789062/ | 12:19 |
| bhyrted | many thanks ! | 12:19 |
| noonedeadpunk | anfter that in /etc/openstack_deploy/opnestack_user_config.yml define cloudkitty_hosts: *infrastructure_hosts | 12:22 |
| bhyrted | thanks, will check on that soon | 12:24 |
| openstackgerrit | Arx Cruz proposed openstack/openstack-ansible master: DNM - POC - Collect logs https://review.opendev.org/702676 | 12:25 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Fix federation scenario assurances os_user usage https://review.opendev.org/703462 | 12:26 |
| *** cshen has quit IRC | 12:29 | |
| *** DanyC has quit IRC | 12:32 | |
| *** DanyC has joined #openstack-ansible | 12:33 | |
| ioni | bhyrted, are you sure that the perfomance hit isn't because on kernel 5 the mitigation for intel are on and maybe on your old kernel, they weren't on or were missing? | 12:38 |
| bhyrted | @ioni i think the problem was with lsi / avago / ... adapter ;-), anyway this is a hdd storage so we didnt expect it to be good performance at all ;-) | 12:40 |
| *** udesale_ has joined #openstack-ansible | 12:55 | |
| *** udesale has quit IRC | 12:57 | |
| *** rh-jelabarre has joined #openstack-ansible | 12:58 | |
| *** cshen has joined #openstack-ansible | 13:13 | |
| *** rpittau|bbl is now known as rpittau | 13:34 | |
| *** miloa has quit IRC | 13:54 | |
| *** ahosam has joined #openstack-ansible | 14:07 | |
| *** miloa has joined #openstack-ansible | 14:10 | |
| *** ahosam has quit IRC | 14:19 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Disable loading nf_nat_ipv4 in new kernels https://review.opendev.org/705403 | 14:29 |
| *** chkumar|rover is now known as raukadah | 14:37 | |
| *** jbadiapa has quit IRC | 15:02 | |
| *** zbr has quit IRC | 15:30 | |
| *** zbr has joined #openstack-ansible | 15:33 | |
| *** zbr has quit IRC | 15:33 | |
| *** zbr has joined #openstack-ansible | 15:34 | |
| *** jstephan has quit IRC | 15:35 | |
| *** ahosam has joined #openstack-ansible | 15:41 | |
| *** ahosam has quit IRC | 15:43 | |
| *** electrofelix has joined #openstack-ansible | 15:47 | |
| gundalow | European Ansible Contributors Summit announcement is live! https://groups.google.com/forum/#!topic/ansible-outreach/sLte90d5hdc | 16:00 |
| *** jbadiapa has joined #openstack-ansible | 16:03 | |
| *** gyee has joined #openstack-ansible | 16:04 | |
| *** snadge has quit IRC | 16:04 | |
| spotz | Thanks for posting that gundalow | 16:09 |
| gundalow | You're welcome | 16:10 |
| *** southquist has quit IRC | 16:10 | |
| *** nicolasbock has joined #openstack-ansible | 16:10 | |
| gundalow | spotz: would you possibly be able to send that round the relevant OpenStack email lists? | 16:13 |
| spotz | I don't think there'd be an issue with that or we could repoost to the tripleo and kolla channels, I think that's it for using ansible but I might be wrong there. cloudnull? mnaser? | 16:15 |
| cloudnull | I'd probably circulate to the mailing list, a blerb in the channels would be nice but it'd probably get lost in the general chatter | 16:16 |
| spotz | cloudnull: But aare those the only 3 projects you can think of for tagging? | 16:25 |
| spotz | They're the only ones I've worked with that used it that I can think of | 16:25 |
| *** cshen has quit IRC | 16:26 | |
| cloudnull | osa, tripleo, kolla, bifrost ? | 16:27 |
| cloudnull | infra too | 16:27 |
| openstackgerrit | Douglas Mendizábal proposed openstack/ansible-role-atos-hsm master: Add HA (load balancing) support https://review.opendev.org/700018 | 16:27 |
| *** sshnaidm is now known as sshnaidm|afk | 16:27 | |
| *** luksky has quit IRC | 16:32 | |
| *** udesale_ has quit IRC | 16:32 | |
| *** miloa has quit IRC | 16:33 | |
| *** rpittau is now known as rpittau|afk | 16:33 | |
| noonedeadpunk | jrosser: are you around? Decided that maybe it's worth discussing https://review.opendev.org/#/c/674653/ | 16:38 |
| jrosser | hello | 16:38 |
| jrosser | yeah i think we're probably talking circularly | 16:38 |
| noonedeadpunk | yeah, so the thing is, that ganesha deployment is not included anywhere, except manila role. And as included playbook plays against ceph-nfs group - it should just be not executed when group is not defined | 16:40 |
| noonedeadpunk | So maybe I have some mistake in this logic... | 16:40 |
| noonedeadpunk | And pretty ready to optimize it in case you see how | 16:41 |
| jrosser | well it's more of an ordering/expectations thing | 16:41 |
| noonedeadpunk | yeah, agree. I was pretty following logic of rgw deployments which is not included in ceph-install.yml | 16:42 |
| jrosser | ah well rgw is special becasue it depends on keystone | 16:42 |
| noonedeadpunk | because I don't know who may want to deploy ganesha without manila but only with ceph | 16:42 |
| jrosser | thats why it comes very late in the sequence | 16:42 |
| noonedeadpunk | but cephfs is included into ceph-install | 16:43 |
| jrosser | does that also need something like keystone? | 16:44 |
| jrosser | really my only nit was that running an os_<thing> playbook would touch your ceph stuff | 16:44 |
| jrosser | it might be a reasonable expectation that os_manila only touches the manila service | 16:44 |
| jrosser | thats basically all i was questioning | 16:45 |
| noonedeadpunk | hm, yeah probably you're right.... | 16:45 |
| jrosser | for the case when you want to do manual things running individual playbooks | 16:45 |
| noonedeadpunk | where do you think it would be more appropriate place to include? I think it don't need keystone | 16:45 |
| jrosser | things are a little messy right now as ceph-install has a whole bunch of stuff that could be broken out like the new nfs stuff is, and rgw | 16:46 |
| jrosser | so it either goes in ceph-install.yml, or in here https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/setup-infrastructure.yml#L25 | 16:47 |
| noonedeadpunk | yeah, probably setup-infrastrucutre if even better.... | 16:48 |
| jrosser | perhaps just putting it into setup-infrastructure as...... ^ yes | 16:48 |
| jrosser | then no surprises when running the manila playbook | 16:48 |
| noonedeadpunk | like initially I've added to os_manila because I wanted to add some extra condition but then just didn't found which one.... | 16:49 |
| noonedeadpunk | *find | 16:49 |
| jrosser | if the manila role docs say you have to define the nfss group when you want it to be deployed then i think it's ok | 16:50 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Add manila tempest testing https://review.opendev.org/674653 | 16:51 |
| spotz | ok shared | 16:51 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Exposed config for logstash elasticsearch endpoints https://review.opendev.org/705483 | 17:01 |
| *** spatel has joined #openstack-ansible | 17:06 | |
| *** spatel has quit IRC | 17:12 | |
| jrosser | mnaser: we had another rabbitmq incident - heat api and engine failing to connect to rabbitmq and then restarting, each time creating a new queue and you end up with a huge number of queues followed by total failure of rabbit | 17:32 |
| jrosser | this same thing has happened twice now, seems to have been heat both times | 17:32 |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-ansible | 17:34 | |
| jrosser | looks very similar to this https://bugs.launchpad.net/heat/+bug/1599104 | 17:35 |
| openstack | Launchpad bug 1599104 in OpenStack Heat "Number of heat Rabbitmq queues is growing from failover to failover" [Medium,In progress] - Assigned to Rico Lin (rico-lin) | 17:35 |
| *** jbadiapa has quit IRC | 17:40 | |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose Elasticsearch data path configuration https://review.opendev.org/704770 | 17:42 |
| openstackgerrit | Duncan Martin Walker proposed openstack/openstack-ansible-ops master: Exposed config for logstash elasticsearch endpoints https://review.opendev.org/705483 | 17:42 |
| *** DanyC has quit IRC | 17:45 | |
| *** DanyC has joined #openstack-ansible | 17:45 | |
| *** drifterza has joined #openstack-ansible | 17:47 | |
| *** errr has quit IRC | 17:56 | |
| *** errr has joined #openstack-ansible | 17:57 | |
| openstackgerrit | Merged openstack/ansible-role-uwsgi master: Run uwsgi tasks only when uwsgi_services defined https://review.opendev.org/704944 | 17:59 |
| *** spatel has joined #openstack-ansible | 18:00 | |
| *** electrofelix has quit IRC | 18:06 | |
| *** DanyC has quit IRC | 18:09 | |
| *** cshen has joined #openstack-ansible | 18:23 | |
| *** cshen has quit IRC | 18:32 | |
| *** drifterza has quit IRC | 18:33 | |
| *** luksky has joined #openstack-ansible | 18:37 | |
| mnaser | jrosser: we ran into that same issue too. noonedeadpunk had a screen running for hooooours deleting those :/ | 19:13 |
| jrosser | I did some rabbitmqctl | xargs goo | 19:14 |
| jrosser | mnaser: do you think this is a heat bug? | 19:15 |
| jrosser | seems suspicious it’s heat making a ton of queues both times for me | 19:15 |
| mnaser | jrosser: I don’t really know, I think it is because it should be reusing queues tbh | 19:16 |
| noonedeadpunk | afaik we've found some bug wich was already resolved? | 19:48 |
| *** gshippey has quit IRC | 19:50 | |
| *** irclogbot_0 has quit IRC | 20:05 | |
| *** irclogbot_0 has joined #openstack-ansible | 20:05 | |
| jrosser | noonedeadpunk: looking at that bug there doesnt appear to be a resolution, it was auto-closed after being inactive | 20:06 |
| jrosser | unless theres something not mentioned on the LP bug? | 20:06 |
| noonedeadpunk | ah, yes, it was rabbitmq exipration policy | 20:08 |
| noonedeadpunk | I think we can set in by osa actually | 20:08 |
| noonedeadpunk | That it probably smth I've forgot to place patch for | 20:09 |
| jrosser | do you think this is something that the other services do internally and heat is missing that when the queues are created? | 20:09 |
| noonedeadpunk | I think other services may clean up queus when they start. Not sure though - never looked into that part of code | 20:10 |
| noonedeadpunk | as then they could clean still relevant queues.... | 20:11 |
| noonedeadpunk | Anyway it's solution despite being pretty hacky one | 20:12 |
| noonedeadpunk | I think I've cleaned up things manually like this https://gist.github.com/noonedeadpunk/4a22b2dcc3b22fcc295445ed18c66247 | 20:12 |
| noonedeadpunk | (xargs wasn't working properly for some reason) | 20:13 |
| noonedeadpunk | if it's still relevant for you | 20:13 |
| jrosser | oh thats better than mine | 20:16 |
| jrosser | we just deleted all the queues, it was just a lab and the whole thing was utterly broken | 20:16 |
| jrosser | iirc i used two steps and generated a .txt file of the queues | 20:17 |
| jrosser | then xargs to iterate that and delete them | 20:17 |
| *** fresta has quit IRC | 20:22 | |
| *** fresta has joined #openstack-ansible | 20:24 | |
| noonedeadpunk | My xargs just drop bunch of queues and they just exits leaving the rest untouched.... Dunno what I did wrong - maybe just it has some buffer which I exceed... | 20:39 |
| *** shananigans has joined #openstack-ansible | 20:50 | |
| bjoernt | @jrosser when are you planning to cut 19.0.10 ? | 20:57 |
| openstackgerrit | Shannon Mitchell proposed openstack/openstack-ansible-os_heat stable/rocky: Fix keystone endpoint for heat servers https://review.opendev.org/705555 | 20:58 |
| jrosser | bjoernt: it's waiting for this i think https://review.opendev.org/#/c/705447/ | 20:58 |
| bjoernt | ok thanks | 20:59 |
| jrosser | could i get another review here please https://review.opendev.org/#/c/704978/ | 21:11 |
| *** nicolasbock has quit IRC | 21:29 | |
| *** bjolo has quit IRC | 21:47 | |
| *** snadge has joined #openstack-ansible | 22:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Download Magnum images once https://review.opendev.org/704575 | 22:33 |
| *** DanyC has joined #openstack-ansible | 22:36 | |
| *** spatel has quit IRC | 22:56 | |
| *** shananigans has quit IRC | 23:05 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!