| *** macza has quit IRC | 00:04 | |
| *** markvoelker has quit IRC | 00:05 | |
| *** hwoarang has quit IRC | 00:11 | |
| *** hwoarang has joined #openstack-ansible | 00:13 | |
| spotz | johnsom cloudnull - I know in the past if something was broken in docs, infra or docs fixed something and then we needed to make another update at least somewhere in the docs for all the outstanding changes to get merged in properly | 00:20 |
|---|---|---|
| *** markvoelker has joined #openstack-ansible | 00:26 | |
| cloudnull | it seems we're just missing the deploy guide job template, when comparing to the other branches, though Im not 100% sure thats all we need | 00:29 |
| johnsom | Might be worth just moving that under the main docs structure... | 00:34 |
| *** aedc has quit IRC | 00:35 | |
| openstackgerrit | Cam J. Loader (cjloader) proposed openstack/openstack-ansible-openstack_hosts master: [WIP] UCA repo fix for upgrades https://review.openstack.org/636797 | 00:55 |
| cjloader | Initial thoughts, I'll continue in the AM. | 00:56 |
| cjloader | good night all | 00:56 |
| *** hwoarang has quit IRC | 01:11 | |
| *** hwoarang has joined #openstack-ansible | 01:13 | |
| *** gyee has quit IRC | 01:15 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_glance master: Cleanup files and templates using smart sources https://review.openstack.org/588959 | 01:18 |
| *** eumel8 has quit IRC | 01:19 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_glance master: Cleanup files and templates using smart sources https://review.openstack.org/588959 | 01:20 |
| jamesdenton | If there are any cores around, a peek at https://review.openstack.org/#/c/636757/ and https://review.openstack.org/#/c/622216/ would be appreciated! | 01:21 |
| *** DanyC has quit IRC | 01:27 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server stable/rocky: ppc64le: fix package list gathering https://review.openstack.org/636752 | 01:28 |
| *** DanyC has joined #openstack-ansible | 01:32 | |
| *** ThiagoCMC has joined #openstack-ansible | 01:36 | |
| *** DanyC has quit IRC | 01:36 | |
| ThiagoCMC | Guys, I'm trying to install Trove, via `openstack-ansible os-trove-install.yml` but, the following error appear: "fatal: [vosctrl-1_trove_api_container-c6424daf]: FAILED! => {"changed": false, "msg": "shade is required for this module"}" | 01:37 |
| ThiagoCMC | Any idea? | 01:37 |
| ThiagoCMC | I tried to locate share, like `updated ; locate shade | grep -I python`, it's here! | 01:37 |
| ThiagoCMC | I'm using OVS/Rocky, Ubuntu 18.04. | 01:38 |
| ThiagoCMC | If I comment out os-trove-install.yml, the `openstack-ansible setup-everything.yml` works just fine! Rocky up and running with Ceph. | 01:40 |
| ThiagoCMC | Just Trove is failing... | 01:40 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server stable/queens: ppc64le: fix package list gathering https://review.openstack.org/636753 | 01:53 |
| *** ArchiFleKs has quit IRC | 02:07 | |
| *** ArchiFleKs has joined #openstack-ansible | 02:21 | |
| jamesdenton | ThiagoCMC I'm gonna try and run thru a deployment. Haven't deployed Trove before. | 02:50 |
| ThiagoCMC | jamesdenton, thanks! | 02:50 |
| jamesdenton | do you have any more logs you can share? | 02:50 |
| ThiagoCMC | Sure, I can run it again... | 02:51 |
| ThiagoCMC | Also | 02:51 |
| ThiagoCMC | I have a very small playbook, that uses `openstack-ansible`, using Ansible's module "os_network", to create a Neutron net, then a subnet. | 02:51 |
| jamesdenton | k | 02:52 |
| ThiagoCMC | When I run `openstack-ansible my-small-playbook-create-neutron-net.yml`, it fails too. | 02:52 |
| ThiagoCMC | Same error | 02:52 |
| ThiagoCMC | I remember that I did this in the past... Werid... | 02:52 |
| ThiagoCMC | jamesdenton, here is the log: http://paste.openstack.org/show/745062/ | 02:54 |
| ThiagoCMC | I tried: `time openstack-ansible os-trove-install.yml -vvv` | 02:54 |
| jamesdenton | thx | 02:55 |
| ThiagoCMC | I also have "/opt/ansible-runtime/lib/python2.7/site-packages/shade-1.29.0.dist-info | 02:55 |
| ThiagoCMC | Machine `./scripts/bootstrap-ansible.sh` worked okay (I did a `git pull` today) but, it can't find shade. | 02:56 |
| ThiagoCMC | Thank you! =) | 02:56 |
| ThiagoCMC | brb | 02:57 |
| *** ThiagoCMC has quit IRC | 02:57 | |
| *** ThiagoCMC has joined #openstack-ansible | 02:58 | |
| cloudnull | evevnings | 03:05 |
| cloudnull | jamesdenton done | 03:06 |
| jamesdenton | my hero | 03:06 |
| cloudnull | ThiagoCMC same camp, I've never given trove a run. | 03:07 |
| jamesdenton | churnin' thru the repo build at the moment | 03:08 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_glance master: Cleanup files and templates using smart sources https://review.openstack.org/588959 | 03:11 |
| jamesdenton | +2s for everyone! | 03:16 |
| cloudnull | indeed! | 03:17 |
| *** markvoelker has quit IRC | 03:27 | |
| *** markvoelker has joined #openstack-ansible | 03:27 | |
| *** markvoelker has quit IRC | 03:32 | |
| ThiagoCMC | cloudnull, but this isn't a trove only problem that I'm seeing. | 03:38 |
| ThiagoCMC | Even a playbook with 1 TASK, using Ansible's os_network module, can't find python shade! :-/ | 03:38 |
| ThiagoCMC | I can see that python shade is installed... But, can't use it. | 03:39 |
| jamesdenton | ThiagoCMC I am having a similar failure at the same task, except mine is "openstacksdk is required for this module" | 03:52 |
| jamesdenton | My env is a little different, tho | 03:52 |
| jamesdenton | When I hop into the container i can replicate that error, but if i activate the venv i can successfully import the module | 03:53 |
| jamesdenton | Ok ThiagoCMC, if you're up for it, try patching tasks/trove_service_network.yml with this: https://pasted.tech/pastes/ba9f9d27d9089c9b7c9fe74531c406c806164aa6 | 04:05 |
| jamesdenton | basically, I added the delegate and the vars | 04:05 |
| jamesdenton | with that, os-trove-install completed on my end and the services are up | 04:06 |
| jamesdenton | Something you may want to make sure you do beforehand is to have the networks created - specifically "dbaas_service_net", unless you override it | 04:07 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/queens: Only implement policy.json if an override is configured https://review.openstack.org/630637 | 04:08 |
| jamesdenton | There is also another bug out there that may help manipulate public vs private endpoint: https://bugs.launchpad.net/openstack-ansible/+bug/1738529 | 04:09 |
| openstack | Launchpad bug 1738529 in openstack-ansible "trove install fails" [Medium,Confirmed] | 04:09 |
| * cloudnull after a long day of bug fixing its time to update the home lab to the head of master! | 04:12 | |
| cloudnull | maybe i'll try and give trove a spin | 04:12 |
| jamesdenton | weeee | 04:12 |
| jamesdenton | I mean, the APIs are responsive. Clientside you may have to use --insecure. It wants to hit the publicURL even though the openrc is setup for internalURL. | 04:19 |
| jamesdenton | Anyway, i think that's enough for tonight. | 04:19 |
| ThiagoCMC | jamesdenton, thank you! I'll try that very soon | 04:26 |
| ThiagoCMC | G'Night! | 04:27 |
| cloudnull | later jamesdenton | 04:27 |
| jamesdenton | see ya | 04:27 |
| ThiagoCMC | cloudnull, hey man, Manila is coming for Stein, right? | 04:27 |
| cloudnull | yes vollman was working on that role | 04:28 |
| cloudnull | i believe its usable ? | 04:28 |
| cloudnull | though Ive not given it a spin | 04:28 |
| *** markvoelker has joined #openstack-ansible | 04:28 | |
| ThiagoCMC | I really need to give it a try! | 04:29 |
| cloudnull | https://review.openstack.org/#/c/611930 | 04:29 |
| cloudnull | that's the pr that's in flight | 04:29 |
| ThiagoCMC | Specially if it have Ceph support! | 04:29 |
| cloudnull | I believe that's the primary target | 04:29 |
| ThiagoCMC | Niiiice! Ceph backend is there! | 04:30 |
| ThiagoCMC | :-O | 04:30 |
| ThiagoCMC | Do you know if it's on master branch already? | 04:30 |
| cloudnull | no that pr is the one that will make it work | 04:30 |
| ThiagoCMC | Oh, okay... Thanks! | 04:30 |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Change openstack-dev to openstack-discuss in setup.cfg https://review.openstack.org/632430 | 04:31 |
| cloudnull | you'd have to pull in that patch `git clone https://git.openstack.org/openstack/openstack-ansible-os_manila /etc/ansible/roles/os_manila; cd /etc/ansible/roles/os_manila; git fetch https://git.openstack.org/openstack/openstack-ansible-os_manila refs/changes/30/611930/17 && git checkout FETCH_HEAD` | 04:31 |
| cloudnull | that'd get you the role and that patch | 04:32 |
| cloudnull | I think you'd still need to create the play for it too, | 04:32 |
| ThiagoCMC | O_O | 04:32 |
| ThiagoCMC | I'll definitely try it! | 04:32 |
| ThiagoCMC | Only for Stein? | 04:32 |
| ThiagoCMC | Or might work on stable/rocky as well? | 04:32 |
| cloudnull | https://review.openstack.org/#/c/612055/4/test-install-manila.yml | 04:33 |
| cloudnull | it might work for rocky | 04:33 |
| cloudnull | but i'd suspect stein | 04:34 |
| cloudnull | it might be good to reach out to vollman, might have some suggestions | 04:34 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Ensure create a volume from image https://review.openstack.org/632519 | 04:35 |
| ThiagoCMC | Thank you! | 04:35 |
| *** udesale has joined #openstack-ansible | 04:36 | |
| ThiagoCMC | Do you guys knows if it is possible to extend an attached (Cinder) volume, with Ceph backend? It's Libvirt + RBD at the Compute Nodes... And then, just run `resizefs` inside of the Instance? | 04:42 |
| *** hwoarang has quit IRC | 04:47 | |
| *** hwoarang has joined #openstack-ansible | 04:50 | |
| *** hwoarang has quit IRC | 04:56 | |
| *** hwoarang has joined #openstack-ansible | 04:56 | |
| *** markvoelker has quit IRC | 05:02 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Make keepalived configuration future-proof https://review.openstack.org/634209 | 05:13 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add venv build host to the group vars for integrated deployments https://review.openstack.org/636830 | 05:35 |
| cloudnull | ThiagoCMC I honestly dont know if you can grow a ceph volume | 05:35 |
| *** shyamb has joined #openstack-ansible | 05:39 | |
| *** shyamb has quit IRC | 05:44 | |
| *** markvoelker has joined #openstack-ansible | 05:58 | |
| *** macza has joined #openstack-ansible | 06:01 | |
| *** hamzaachi has joined #openstack-ansible | 06:01 | |
| *** shyamb has joined #openstack-ansible | 06:01 | |
| ThiagoCMC | cloudnull, ok, no worries... =) | 06:04 |
| *** macza has quit IRC | 06:05 | |
| prometheanfire | cloudnull: I imagine you tested? | 06:25 |
| *** markvoelker has quit IRC | 06:32 | |
| *** hamzaachi has quit IRC | 06:34 | |
| *** gokhani has quit IRC | 07:24 | |
| *** markvoelker has joined #openstack-ansible | 07:28 | |
| *** Adri2000 has quit IRC | 07:28 | |
| *** Adri2000 has joined #openstack-ansible | 07:29 | |
| *** macza has joined #openstack-ansible | 07:32 | |
| *** macza has quit IRC | 07:36 | |
| *** hamzaachi has joined #openstack-ansible | 07:43 | |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_hosts master: Trivial: Fix the pep8 warning https://review.openstack.org/616526 | 07:52 |
| ThiagoCMC | cloudnull, is it possible to update the endpoints and haproxy_keepalived_external_vip_cidr, from IP (which is behind a NAT and breaks spice console), to a FQDN (name-based)? | 07:52 |
| ThiagoCMC | Or do I have to re-deploy everything from scratch? hehe | 07:53 |
| *** markvoelker has quit IRC | 08:01 | |
| *** DanyC has joined #openstack-ansible | 08:05 | |
| *** DanyC has quit IRC | 08:09 | |
| *** gkadam has joined #openstack-ansible | 08:10 | |
| *** gkadam is now known as gkadam-brb | 08:11 | |
| *** shyamb has quit IRC | 08:12 | |
| *** hamzaachi has quit IRC | 08:13 | |
| *** hamzaachi has joined #openstack-ansible | 08:13 | |
| *** gkadam-brb is now known as gkadam | 08:20 | |
| *** phasespace has joined #openstack-ansible | 08:21 | |
| *** electrofelix has joined #openstack-ansible | 08:43 | |
| *** tosky has joined #openstack-ansible | 08:47 | |
| *** CeeMac has joined #openstack-ansible | 08:57 | |
| *** kopecmartin|off is now known as kopecmartin | 08:57 | |
| *** shyamb has joined #openstack-ansible | 08:58 | |
| *** markvoelker has joined #openstack-ansible | 08:58 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron master: Use the new services names for sfc https://review.openstack.org/622216 | 09:18 |
| fnpanic | hi | 09:31 |
| *** markvoelker has quit IRC | 09:31 | |
| gshippey | anyone got any input on this https://bugs.launchpad.net/openstack-ansible/+bug/1755821 | 09:37 |
| openstack | Launchpad bug 1755821 in openstack-ansible "config_template fails to parse template if it contains a comment with leading spaces" [High,Confirmed] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard) | 09:37 |
| *** DanyC has joined #openstack-ansible | 09:43 | |
| *** DanyC has quit IRC | 09:47 | |
| *** DanyC has joined #openstack-ansible | 09:50 | |
| odyssey4me | gshippey hmm, I guess we need to decide what we should do about it - either fail more elegantly, or accept the input but clear any leading whitespace | 09:50 |
| *** gkadam has quit IRC | 09:51 | |
| *** gkadam has joined #openstack-ansible | 09:51 | |
| *** DanyC has quit IRC | 09:57 | |
| gshippey | I'll give the second option a go | 09:59 |
| *** DanyC has joined #openstack-ansible | 10:01 | |
| *** aedc has joined #openstack-ansible | 10:03 | |
| *** aedc has quit IRC | 10:10 | |
| *** aedc has joined #openstack-ansible | 10:10 | |
| *** ArchiFleKs has quit IRC | 10:19 | |
| *** markvoelker has joined #openstack-ansible | 10:28 | |
| *** shyamb has quit IRC | 10:29 | |
| *** ArchiFleKs has joined #openstack-ansible | 10:30 | |
| *** shyamb has joined #openstack-ansible | 10:33 | |
| *** jbadiapa has quit IRC | 10:42 | |
| *** shyamb has quit IRC | 10:42 | |
| *** mkuf has quit IRC | 10:51 | |
| *** udesale has quit IRC | 10:58 | |
| *** markvoelker has quit IRC | 11:01 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Add the deployment guide job to our standard templates https://review.openstack.org/636780 | 11:32 |
| *** asettle has joined #openstack-ansible | 11:41 | |
| *** priteau has joined #openstack-ansible | 11:43 | |
| *** shyamb has joined #openstack-ansible | 11:44 | |
| *** mkuf has joined #openstack-ansible | 11:51 | |
| openstackgerrit | Georgina Shippey proposed openstack/ansible-config_template master: Remove whitespace before comments https://review.openstack.org/636935 | 11:55 |
| *** markvoelker has joined #openstack-ansible | 11:58 | |
| *** CeeMac_ has joined #openstack-ansible | 11:59 | |
| *** CeeMac has quit IRC | 12:03 | |
| jamesdenton | mornin | 12:12 |
| *** markvoelker has quit IRC | 12:26 | |
| *** priteau has quit IRC | 12:36 | |
| *** priteau has joined #openstack-ansible | 12:38 | |
| *** zenirc369 has joined #openstack-ansible | 12:39 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/rocky: Add heat user to heat domain admin role https://review.openstack.org/636630 | 12:45 |
| *** udesale has joined #openstack-ansible | 12:45 | |
| *** shyamb has quit IRC | 12:52 | |
| *** shyamb has joined #openstack-ansible | 12:54 | |
| *** shyamb has quit IRC | 13:05 | |
| *** shyamb has joined #openstack-ansible | 13:06 | |
| *** priteau has quit IRC | 13:13 | |
| *** jbadiapa has joined #openstack-ansible | 13:17 | |
| vollman | morning jamesdenton | 13:17 |
| vollman | Could you take a look at odyssey4me comment here https://review.openstack.org/#/c/633277/5/vars/ubuntu.yml when you have a minute? | 13:18 |
| jamesdenton | that plugin needs to be installed everywhere, AFAIK. | 13:19 |
| vollman | ok thanks | 13:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron stable/rocky: Add support for dns_domain_ports api extension https://review.openstack.org/636368 | 13:29 |
| *** shyamb has quit IRC | 13:33 | |
| *** vakuznet has joined #openstack-ansible | 13:36 | |
| jamesdenton | Any tempest expertise around? | 13:39 |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_cinder master: Avoid distro installing unused services https://review.openstack.org/633276 | 13:47 |
| cloudnull | mornings | 13:54 |
| *** priteau has joined #openstack-ansible | 13:55 | |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_nova master: Avoid distro installing unused services https://review.openstack.org/633275 | 13:56 |
| *** PTO has joined #openstack-ansible | 13:56 | |
| guilhermesp | cloudnull: o/ | 13:57 |
| cloudnull | o/ | 13:57 |
| PTO | @cloudnull do you have a minute? | 13:59 |
| cloudnull | sure what's up ? | 14:00 |
| PTO | Its regarding the elk_metrics_6x repo from ops. I have tried you improvements and struck a problem | 14:01 |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_neutron master: Avoid distro installing unused services https://review.openstack.org/633277 | 14:02 |
| cloudnull | what's going on ? | 14:02 |
| PTO | The elasticsearch does not start, as its denied memory allocation: https://pastebin.com/VMTbM8q7 | 14:03 |
| mnaser | hi all | 14:04 |
| cloudnull | o/ mnaser hows it ? | 14:04 |
| PTO | Im not a jedi in lxc. Is there missing a container parameter (cgroups?) which allows memory allocation | 14:04 |
| guilhermesp | mornings mnaser | 14:04 |
| mnaser | cloudnull, guilhermesp: morning | 14:04 |
| mnaser | my wrist is braced up so operating at 20% speed hah | 14:04 |
| mnaser | sadness | 14:05 |
| cloudnull | PTO looks like - [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] | 14:05 |
| guilhermesp | mnaser: =( | 14:05 |
| cloudnull | maybe something on the local system needs to be adjusted | 14:05 |
| cloudnull | mnaser that sounds terrible | 14:05 |
| cloudnull | you ok ? | 14:05 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client master: Use in-repo GPG keys https://review.openstack.org/636711 | 14:05 |
| mnaser | could be worse i guess, seems like everyone i talk to has ended up with one of those ganglion cysts | 14:05 |
| mnaser | so i just have to get it removed but until that happens ill deal with the frustration of not being able to type as fast, hah | 14:06 |
| cloudnull | damn. well I hope its not painful | 14:07 |
| mnaser | nah, but the bracing means its hard to use a mouse and i cant get things done as fast | 14:07 |
| jrosser | cloudnull: that looks similar to my trouble with elasticsearch, the heap size too large w.r.t host ram size | 14:08 |
| mnaser | i need every second of my time =P | 14:08 |
| mnaser | vollman: thanks for all your patience on clearing up the issue to get your manila stuff merged | 14:13 |
| vollman | mnaser: np. Do you know if there is a way to add multiple Depends-On to a changeset? | 14:14 |
| cloudnull | jrosser: ah, so maybe we just need to be more conservative ? | 14:14 |
| mnaser | vollman: yep, just multiple lines of Depends-On | 14:14 |
| cloudnull | PTO have you tried adjusting the heap size ? | 14:14 |
| vollman | mnaser: awesome thx | 14:14 |
| mnaser | vollman: an example https://review.openstack.org/#/c/627782/ | 14:15 |
| jrosser | cloudnull: this was the patch that really helped me out on 64G nodes https://github.com/openstack/openstack-ansible-ops/commit/6017fc0e89fbb3ffdadc62773e34c4069b3d4584 | 14:15 |
| PTO | cloudnull: I have tried changing /etc/security/limits.conf and with ulimit - it still refuses to start. Is there any lxc container privileges/config? | 14:15 |
| jrosser | the halfmem heap size is pretty much half the physical ram, so if you have anything else large running it can all get deadlocked and not start elasticsearch | 14:16 |
| cloudnull | PTO what distro / version are you running | 14:16 |
| PTO | ubuntu 16.04 | 14:17 |
| cloudnull | ok. so that should be covered in the systemd unit. | 14:20 |
| cloudnull | could be a redharring though, can you try setting the jvm heap size to something smaller, per jrosser suggestion? | 14:20 |
| PTO | cloudnull: Just testing the lower heap | 14:20 |
| cloudnull | cool | 14:20 |
| PTO | changing the jvm.options -Xms and -Xmx enough? | 14:21 |
| PTO | https://pastebin.com/G4kiu7nq | 14:23 |
| PTO | [1]: memory locking requested for elasticsearch process but memory is not locked | 14:23 |
| jrosser | "Unable to lock JVM Memory: error=12, reason=Cannot allocate memory" <- there needs to be sufficient free RAM to cover at least what the heap size is | 14:24 |
| jrosser | i found that as we went up through the ELK versions elasticsearch got more and more aggressive with memory usage | 14:24 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-rsyslog_client master: Templatize rsyslog configuration files https://review.openstack.org/624805 | 14:24 |
| PTO | free -h on the host gives 50G free mem | 14:24 |
| cloudnull | the memory locking can be disabled too | 14:25 |
| cloudnull | which will allow elasticsearch to swap, while not really recommended, its totally doable | 14:25 |
| PTO | Question is - should memory locking be disabled or allowed on the container | 14:25 |
| cloudnull | simply turn it off in the /etc/elasticsearch/elasticsearch.yml | 14:26 |
| cloudnull | but there's a var to do it in the playbooks too | 14:26 |
| cloudnull | PTO IDK, maybe we should have memory locking only enforced if deploying on baremetal? | 14:26 |
| cloudnull | I'm open to suggestions | 14:27 |
| PTO | Good question :-) | 14:28 |
| cloudnull | try the disabling memory locking in the config and starting it | 14:28 |
| cloudnull | also is elasticsearch running on a dedicated node ? | 14:28 |
| cloudnull | or is this something like an infra host? | 14:29 |
| *** PTO_ has joined #openstack-ansible | 14:31 | |
| jrosser | my experience has been that having the elasticsearch/kibana container on a 64G infra node was the first point that the stock memory allocation breaks | 14:32 |
| *** PTO has quit IRC | 14:32 | |
| jrosser | then if you leave then long enough dedicated 64G log nodes get wedged up on restart because the combination of elasticsearch and logstash together use enough RAM it won't allow elasticsearch to restart | 14:33 |
| jrosser | if you have loads of ram this isnt an issue, but where the h_mem value is approx 50% of the host ram it's going to break | 14:34 |
| cloudnull | so should we default to something like .25 & .125 respectively ? | 14:34 |
| jrosser | i've wound the elastic/kibana one down to 10G on a 64G host | 14:35 |
| jrosser | so 1/8th would do fine there i think | 14:35 |
| cloudnull | ok | 14:35 |
| jrosser | afaik those are only really marshalling queries and collating the responses from the data nodes | 14:36 |
| cloudnull | correct | 14:36 |
| PTO_ | i tested with 16G on a 192G host - it boots if i manually set ulimit -n 65536 and then run sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch -v | 14:36 |
| cloudnull | PTO_ can you look in the systemd unit file for elasticsearch, do you see "LimitNOFILE" | 14:39 |
| fnpanic | hi | 14:40 |
| fnpanic | maybe someone has an idea little quick | 14:40 |
| fnpanic | added a ad domain | 14:40 |
| fnpanic | then adding the admin user with role admin to this domain | 14:41 |
| fnpanic | keystone is saying this all the time Policy identity:list_users failed scope check. The token used to make the request was project scoped but the | 14:41 |
| cloudnull | fnpanic is this using rocky? | 14:42 |
| fnpanic | policy requires ['system'] scope. | 14:42 |
| fnpanic | yeah | 14:42 |
| fnpanic | with pike no problem | 14:42 |
| fnpanic | :-( | 14:42 |
| cloudnull | I think that was similar to the bug we were talking about yesterday, errr was looking at. -cc odyssey4me | 14:42 |
| * cloudnull goes back to look | 14:42 | |
| fnpanic | openstack role add --domain domain-id --user admin-id admin-role-id | 14:43 |
| cloudnull | https://bugs.launchpad.net/openstack-ansible/+bug/1808543 | 14:43 |
| openstack | Launchpad bug 1808543 in openstack-ansible "Keystone Federation cannot complete SP node setup on stable/rocky" [Undecided,Confirmed] - Assigned to Jesse Pretorius (jesse-pretorius) | 14:43 |
| fnpanic | this command i used for adding the admin | 14:43 |
| fnpanic | i switched to ldap for testing to rule out ssl errors :-) | 14:44 |
| fnpanic | mhhh not sure if this is related | 14:45 |
| fnpanic | maybe it is me doing something wrong | 14:46 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Change the q_mem and h_mem to lower and upper limits https://review.openstack.org/636975 | 14:52 |
| cloudnull | PTO_ jrosser ^ | 14:52 |
| cloudnull | PTO_ mind giving that a spin? | 14:52 |
| *** electrofelix has quit IRC | 14:53 | |
| fnpanic | so noone has an idea? | 14:58 |
| cloudnull | fnpanic sorry, I'm at a total loss | 14:59 |
| fnpanic | total loss? | 14:59 |
| cloudnull | are you seeing any errors within keystone? | 15:00 |
| fnpanic | no | 15:00 |
| fnpanic | only the warning i posted | 15:00 |
| odyssey4me | fnpanic if you're getting a scope error, then that's keystone doing the right thing - it has nothing to do with ssl, but instead has to do with the policies being applied and what you're trying to do... if you'd like to understand why it's failing then better to ask the keystone folks, but you'll have to unbreak your config first | 15:00 |
| cloudnull | oh, is this an issue with the keystone policy file ? | 15:01 |
| cloudnull | no something that is coming back from ad? | 15:01 |
| odyssey4me | cloudnull no, the action violates the policy | 15:01 |
| odyssey4me | not a problem with the file - a problem with the action according to the current policy | 15:01 |
| cloudnull | interesting | 15:03 |
| * cloudnull TIL | 15:03 | |
| odyssey4me | in other words, the openstack cli is spitting out a legitimate error which was fed back from keystone itself | 15:03 |
| fnpanic | ok | 15:03 |
| fnpanic | this is the reply from the openstack client | 15:04 |
| fnpanic | An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-c6eaaaa8-90df-4031-ad46-3189b693ea07) | 15:04 |
| fnpanic | and keystone shows this in the logs | 15:04 |
| odyssey4me | IIRC when you add a domain via LDAP, you have to have a domain by the right name that exists in keystone first - that's why we have https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_ldap_setup.yml#L16 | 15:05 |
| odyssey4me | then you put down the domain-specific conf file, like https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_ldap_setup.yml#L35 | 15:05 |
| odyssey4me | from there on, for role assignments and stuff - I honestly can't remember how that all works... perhaps best to ask the keystone folks | 15:05 |
| fnpanic | mhh but the domain is in place after i run the playbook with the config file | 15:05 |
| *** phasespace has quit IRC | 15:06 | |
| fnpanic | so openstack domain list works | 15:06 |
| fnpanic | and displays the domain | 15:06 |
| fnpanic | it was created after the playbook run os-keystone-install | 15:07 |
| odyssey4me | fnpanic looking through https://www.slideshare.net/JessePretorius/openstack-keystone-with-ldap now to hopefully remind me how it all works | 15:07 |
| odyssey4me | nope, not really :/ | 15:08 |
| odyssey4me | can you list users? | 15:08 |
| fnpanic | yes but only from default domain | 15:09 |
| fnpanic | when i do the openstack user list | 15:09 |
| fnpanic | woeks | 15:09 |
| fnpanic | openstack user list --domain new-domain | 15:09 |
| fnpanic | i get this error | 15:09 |
| fnpanic | An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-c6eaaaa8-90df-4031-ad46-3189b693ea07) | 15:10 |
| fnpanic | and in keystone journal i see this | 15:10 |
| odyssey4me | fnpanic right, so that means there's something wrong in the domain-specific config | 15:10 |
| fnpanic | now i see this default default] Could not find domain: cgm.ag.: DomainNotFound: Could not find domain: cgm.ag. | 15:11 |
| fnpanic | but i can list the domains | 15:13 |
| fnpanic | openstack domain list | 15:13 |
| fnpanic | shows it | 15:13 |
| odyssey4me | yes, but that's the domain in SQL - not from LDAP | 15:14 |
| fnpanic | ??? | 15:14 |
| fnpanic | now i am list | 15:14 |
| fnpanic | lost | 15:14 |
| fnpanic | the sql domain gets listed | 15:15 |
| fnpanic | and the ldap config is there | 15:15 |
| fnpanic | was created by the playbook | 15:15 |
| mnaser | honestly i'm always lost so thats okay fnpanic :) | 15:15 |
| odyssey4me | yes, but that domain is just an entry in the keystone SQL database - it is not the result of interacting with LDAP | 15:16 |
| fnpanic | ok | 15:17 |
| fnpanic | got it | 15:17 |
| fnpanic | under [identity] | 15:17 |
| fnpanic | is see | 15:17 |
| fnpanic | driver = sql | 15:18 |
| fnpanic | domain_config_dir = /etc/keystone/domains | 15:18 |
| fnpanic | domain_specific_drivers_enabled = True | 15:18 |
| odyssey4me | fnpanic ok, what's the config in the domain-specific entry | 15:18 |
| odyssey4me | it should show ldap there | 15:18 |
| fnpanic | in the conf the driver is ldap | 15:19 |
| odyssey4me | ok, that's good then | 15:19 |
| fnpanic | so what would be debug steps? | 15:20 |
| odyssey4me | in https://www.slideshare.net/JessePretorius/openstack-keystone-with-ldap slide 7 is an example conf - do you have all the bits shown there, or near to all of them? | 15:20 |
| fnpanic | yeah | 15:22 |
| *** udesale has quit IRC | 15:22 | |
| PTO_ | cloudnull: Sorry for the delay. Had a meeting. The perameter is there and is correct, LimitNOFILE=65536. I dont know why it complains when i launch it with systemd. Nothing in the journalctrl log | 15:23 |
| *** gkadam has quit IRC | 15:30 | |
| PTO_ | cloudnull: I dont think i have permissions to review the change | 15:30 |
| PTO_ | cloudnull: nvm. Just forgot where the function were | 15:31 |
| openstackgerrit | Marc Gariépy (mgariepy) proposed openstack/openstack-ansible master: Add python-setuptools to bootstrap-ansible.sh https://review.openstack.org/636992 | 15:40 |
| openstackgerrit | Jacob Wagner proposed openstack/openstack-ansible-ops master: Add ability to deploy designate with BIND9 servers https://review.openstack.org/635611 | 15:46 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_nova master: Add nova_user_pip_packages variable https://review.openstack.org/635579 | 15:55 |
| PTO_ | I have to leave for today. Thanks for the help. I will try your proposed changes @cloudnull | 16:06 |
| *** PTO_ has quit IRC | 16:06 | |
| cloudnull | have a good one PTO_. | 16:09 |
| *** hamzaachi has quit IRC | 16:12 | |
| *** ArchiFleKs has quit IRC | 16:13 | |
| mgariepy | fnpanic, what version of openstack are you using ? | 16:15 |
| fnpanic | rocky | 16:15 |
| fnpanic | i just copied a conf for keystone from my testcloud | 16:16 |
| fnpanic | which was pike | 16:16 |
| fnpanic | works like a charm | 16:16 |
| fnpanic | now i need to dig deeper what is the issue | 16:16 |
| mgariepy | i've seen the policy stuff, on queens. but with an old-ish version. | 16:17 |
| mgariepy | whuich tag ? | 16:17 |
| *** ArchiFleKs has joined #openstack-ansible | 16:17 | |
| mgariepy | https://github.com/openstack/openstack-ansible-os_keystone/commit/fc3d2fe4b6df67bd28f94097c81f71bb78518340 | 16:19 |
| mgariepy | do you have this patch ? | 16:19 |
| fnpanic | rocky 18.1.2-16 | 16:20 |
| mgariepy | ldap auth is not really tested in the gates, it would help to get it covered at some point but i do lack time like everyone. | 16:21 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: Enable functional deployment of FWaaS v2 https://review.openstack.org/636757 | 16:22 |
| fnpanic | mgariepy: yepp | 16:23 |
| fnpanic | have it | 16:23 |
| mgariepy | is it an upgrade or a clean install? | 16:24 |
| *** macza has joined #openstack-ansible | 16:27 | |
| mgariepy | fnpanic, can you contact the ldap server ? with ex: ldapsearch or something ? | 16:31 |
| mgariepy | from the keystone container** | 16:34 |
| cloudnull | anyone want to give this a nudge https://review.openstack.org/#/c/635997/ | 16:38 |
| *** markvoelker has joined #openstack-ansible | 16:38 | |
| mgariepy | done cloudnull | 16:40 |
| cloudnull | tyvm | 16:41 |
| spotz | cloudnull: nudge completed | 16:44 |
| *** trident has quit IRC | 16:46 | |
| *** gyee has joined #openstack-ansible | 16:55 | |
| ThiagoCMC | Morning! =P | 16:57 |
| ThiagoCMC | I'm a big fan of IPv6! I'm wondering here why OSA uses "0.0.0.0" to bind all services, instead of "::". | 16:57 |
| ThiagoCMC | With "::" it would make it IPv6-Ready in a flash of a second! | 16:57 |
| ThiagoCMC | ;-) | 16:57 |
| ThiagoCMC | Especially if you have DNS up and running to reach your cloud... | 16:58 |
| ThiagoCMC | I have deployed IPv6-Only OpenStack clouds years ago, there was only 1 problematic daemon, memcached but, I believe that it's fixed by now. | 16:59 |
| noonedeadpunk | Hi folks. I've got a question. Why do we need this block of code https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-cinder-install.yml#L165 when the same is placed inside cinder role itself over there https://github.com/openstack/openstack-ansible-os_cinder/blob/master/tasks/cinder_db_setup.yml#L48 ? | 17:01 |
| noonedeadpunk | Isn't it a dublicate task? | 17:01 |
| noonedeadpunk | odyssey4me: seems that you're the one, who may know the answer for this ^ :) | 17:04 |
| *** sreejithp has joined #openstack-ansible | 17:05 | |
| *** ianychoi has joined #openstack-ansible | 17:06 | |
| odyssey4me | noonedeadpunk https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-cinder-install.yml#L165 will enact the migrations once all the cinder environments are upgraded - the role task is just a fallback, just in case it gets missed somehow | 17:11 |
| odyssey4me | unfortunately I couldn't figure out any other way to ensure all cinder environments were upgraded before running the db contraction... I have ideas for improvement, but no time to try them out. | 17:12 |
| *** markvoelker has quit IRC | 17:12 | |
| *** zenirc369 has quit IRC | 17:15 | |
| *** kopecmartin is now known as kopecmartin|off | 17:16 | |
| odyssey4me | noonedeadpunk given that we're now using a more modern ansible, we could probably change that up to use include_role/tasks_from or something like that to deduplicate the code and keep the stuff in the role to be less confusing | 17:20 |
| ThiagoCMC | Do you guys knows how to workaround this bug: https://bugs.launchpad.net/cloud-init/+bug/1705346 ? | 17:21 |
| openstack | Launchpad bug 1705346 in cloud-init "Cloud-Init fails to deal with SWAP and Ephemeral if virtio-scsi is enabled" [Medium,Confirmed] | 17:21 |
| ThiagoCMC | I reported it back in 2017, still a bug today... | 17:21 |
| jamesdenton | will a recheck stop running tests, or do you have to wait it out? | 17:28 |
| cloudnull | jamesdenton you have to wait it out | 17:34 |
| cloudnull | though you could commit another change to cause it to restart | 17:34 |
| jamesdenton | gotcha, thank you! | 17:35 |
| jamesdenton | cloudnull can you take a look at https://review.openstack.org/#/c/626222/? The file is missing for me | 17:38 |
| odyssey4me | cloudnull should a oneshot service show as enabled? | 17:39 |
| cloudnull | yes | 17:39 |
| *** archklikk has joined #openstack-ansible | 17:39 | |
| cloudnull | jamesdenton sure | 17:39 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_neutron master: ovs: create bridge_mappings in openvswitch_agent.ini https://review.openstack.org/626222 | 17:40 |
| cloudnull | jamesdenton rebased it, but the file is still missing | 17:41 |
| cloudnull | which i think would mean its already been merged | 17:41 |
| cloudnull | or at least the content of the change has been merged | 17:41 |
| jamesdenton | yeah, interesting | 17:41 |
| *** DanyC has quit IRC | 17:43 | |
| *** DanyC has joined #openstack-ansible | 17:43 | |
| jamesdenton | The commit message and the file don't match. This was supposed to modify a task and not a template | 17:44 |
| noonedeadpunk | odyssey4me: oh, ok, got it. Just catching db migrations failure on brand new setup, seems like still facing this bug https://bugs.launchpad.net/cinder/+bug/1806156 | 17:46 |
| openstack | Launchpad bug 1806156 in Cinder "shared_targets_online_data_migration fails when cinder-volume service not running" [Undecided,Confirmed] | 17:46 |
| * jrosser boggles at magnum | 17:47 | |
| noonedeadpunk | And it's also failing for existing ones. Has anyone faced with the same problem on rocky? ^ | 17:47 |
| *** DanyC has quit IRC | 17:48 | |
| odyssey4me | I don't see how containerising the service helps at all there, as commented by codylab at the end there. | 17:48 |
| noonedeadpunk | odyssey4me it doesn't:) I believe, that he just got need_online_data_migrations cached | 17:54 |
| *** errr has joined #openstack-ansible | 17:59 | |
| ThiagoCMC | About the Neutron Trunk option. Is it supposed to use the "br-vlan" bridge? | 18:04 |
| noonedeadpunk | and it seems, that the problem started since this commit https://review.openstack.org/#/c/614617/ - at least I've started to face with it since this commit | 18:05 |
| *** markvoelker has joined #openstack-ansible | 18:09 | |
| jamesdenton | ThiagoCMC It just depends on the provider mapping. It may use br-vlan, or could use a physical interface in the event of host_bind_override | 18:11 |
| *** aedc has quit IRC | 18:11 | |
| *** priteau has quit IRC | 18:13 | |
| jamesdenton | odyssey4me can i get your blessing on this again? https://review.openstack.org/#/c/636757/ | 18:15 |
| *** priteau has joined #openstack-ansible | 18:17 | |
| *** aedc has joined #openstack-ansible | 18:19 | |
| jamesdenton | ThiagoCMC i think you'll find how it works here: https://goo.gl/RoqY2e | 18:20 |
| *** priteau has quit IRC | 18:22 | |
| *** DanyC has joined #openstack-ansible | 18:29 | |
| *** archklikk has quit IRC | 18:29 | |
| *** DanyC has quit IRC | 18:33 | |
| *** shardy has quit IRC | 18:40 | |
| *** markvoelker has quit IRC | 18:42 | |
| *** vakuznet has quit IRC | 18:43 | |
| *** priteau has joined #openstack-ansible | 18:45 | |
| *** macza has quit IRC | 18:46 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: [DNM] Convert dynamic includes to static imports https://review.openstack.org/637026 | 18:47 |
| *** priteau has quit IRC | 18:49 | |
| spotz | jamesdenton: you're good to go on https://review.openstack.org/#/c/636757/3 | 18:51 |
| *** hamzaachi has joined #openstack-ansible | 18:52 | |
| jamesdenton | thanks spotz! | 18:54 |
| *** macza has joined #openstack-ansible | 18:55 | |
| *** rpsene has quit IRC | 18:55 | |
| spotz | NP:) | 18:56 |
| *** archklikk has joined #openstack-ansible | 19:04 | |
| *** archklikk has quit IRC | 19:05 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Change the q_mem and h_mem to lower and upper limits https://review.openstack.org/636975 | 19:10 |
| openstackgerrit | Merged openstack/openstack-ansible master: Add option to disable container builds on specific hosts https://review.openstack.org/635997 | 19:15 |
| *** macza_ has joined #openstack-ansible | 19:18 | |
| *** macza has quit IRC | 19:18 | |
| *** macza has joined #openstack-ansible | 19:18 | |
| *** henriqueof has joined #openstack-ansible | 19:19 | |
| *** sreejithp_ has joined #openstack-ansible | 19:22 | |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_manila master: Basic working os_manila role https://review.openstack.org/611930 | 19:23 |
| *** sreejithp has quit IRC | 19:24 | |
| guilhermesp | mnaser: https://bugs.launchpad.net/openstack-ansible/+bug/1815902 has the PR backported to rocky https://github.com/openstack/openstack-ansible-galera_server/commit/5e78067f409a438411c5c563811ac6edba358229? | 19:30 |
| openstack | Launchpad bug 1815902 in openstack-ansible "Error galera_server : Download extra packages" [Undecided,New] | 19:30 |
| guilhermesp | oh, duplicated | 19:32 |
| *** markvoelker has joined #openstack-ansible | 19:39 | |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_neutron master: Avoid distro installing unused services https://review.openstack.org/633277 | 19:42 |
| *** gshippey has quit IRC | 19:49 | |
| *** aedc has quit IRC | 19:55 | |
| *** KeithMnemonic has joined #openstack-ansible | 20:02 | |
| cloudnull | guilhermesp i backported those yesterday | 20:04 |
| cloudnull | im not sure if they've merged | 20:04 |
| *** dave-mccowan has joined #openstack-ansible | 20:05 | |
| *** hamzaachi has quit IRC | 20:10 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Enable functional deployment of FWaaS v2 https://review.openstack.org/636757 | 20:12 |
| *** markvoelker has quit IRC | 20:12 | |
| guilhermesp | cloudnull: yep merged | 20:16 |
| guilhermesp | seems that it is not getting the value of the dictionary | 20:16 |
| guilhermesp | https://github.com/openstack/openstack-ansible-galera_server/blob/5e78067f409a438411c5c563811ac6edba358229/tasks/galera_install_download_extra_packages.yml#L25 | 20:16 |
| guilhermesp | but the syntax seems ok | 20:16 |
| *** dave-mccowan has quit IRC | 20:28 | |
| *** dave-mccowan has joined #openstack-ansible | 20:35 | |
| *** aedc has joined #openstack-ansible | 20:43 | |
| *** henriqueof has quit IRC | 20:48 | |
| * jrosser facepalm | 20:50 | |
| jrosser | root@infra1:~# lsof -K | grep pypi | grep inotify | wc -l | 20:50 |
| jrosser | 67161 | 20:50 |
| jrosser | so thats why the repo server sync no longer works | 20:50 |
| jrosser | in fact its super gross becasue it consumes all the inotify allowance across thew whole infra host | 20:51 |
| cloudnull | oh wow! | 20:59 |
| cloudnull | so pypi-server is doing that? | 20:59 |
| *** Nick_A has joined #openstack-ansible | 21:01 | |
| spotz | That's just wrong.... | 21:06 |
| jrosser | looks that way | 21:07 |
| *** markvoelker has joined #openstack-ansible | 21:09 | |
| *** priteau has joined #openstack-ansible | 21:15 | |
| *** priteau has quit IRC | 21:19 | |
| jrosser | trouble with this is that running out of inotify prevents systemd from restarting any services | 21:27 |
| -openstackstatus- NOTICE: Jobs are failing due to ssh host key mismatches caused by duplicate IPs in a test cloud region. We are disabling the region and will let you know when jobs can be rechecked. | 21:30 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts stable/rocky: Increase inotify watch limit https://review.openstack.org/637056 | 21:35 |
| jrosser | cloudnull: how much systemd stuff is there in queens? wondering how far back that ^^ needs tobackport | 21:36 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-openstack_hosts stable/queens: Increase inotify watch limit https://review.openstack.org/637059 | 21:41 |
| jrosser | it's xenial for Q&R so the same dnsmasq issue will exist on both | 21:41 |
| *** markvoelker has quit IRC | 21:42 | |
| cloudnull | jrosser yes it should be just q and r | 21:46 |
| *** dave-mccowan has quit IRC | 21:49 | |
| *** tosky has quit IRC | 22:05 | |
| *** tosky has joined #openstack-ansible | 22:05 | |
| -openstackstatus- NOTICE: The test cloud region using duplicate IPs has been removed from nodepool. Jobs can be rechecked now. | 22:13 | |
| *** dave-mccowan has joined #openstack-ansible | 22:22 | |
| *** sreejithp_ has quit IRC | 22:39 | |
| *** markvoelker has joined #openstack-ansible | 22:39 | |
| ThiagoCMC | jamesdenton, thank you! Looks like an awesome book! :-D | 22:53 |
| *** markvoelker has quit IRC | 23:12 | |
| *** aedc has quit IRC | 23:29 | |
| *** aedc has joined #openstack-ansible | 23:33 | |
| *** phasespace has joined #openstack-ansible | 23:40 | |
| *** aedc has quit IRC | 23:44 | |
| *** aedc has joined #openstack-ansible | 23:44 | |
| *** tosky has quit IRC | 23:51 | |
| *** macza has quit IRC | 23:56 | |
| *** aedc has quit IRC | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!