| *** markvoelker has joined #openstack-ansible | 00:03 | |
| *** markvoelker has quit IRC | 00:07 | |
| *** dave-mccowan has quit IRC | 00:11 | |
| *** bgmccollum has quit IRC | 00:12 | |
| *** gyee has quit IRC | 00:15 | |
| *** cshen has joined #openstack-ansible | 00:15 | |
| *** luksky has quit IRC | 00:15 | |
| *** bgmccollum has joined #openstack-ansible | 00:17 | |
| *** cshen has quit IRC | 00:20 | |
| NobodyCam | humm nope not getting a eth0 attress | 00:29 |
|---|---|---|
| NobodyCam | address evenb | 00:29 |
| *** cmart has quit IRC | 00:31 | |
| *** markvoelker has joined #openstack-ansible | 00:35 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: Pass ceph_mons as a list of inventory hosts https://review.openstack.org/644267 | 00:47 |
| *** markvoelker has quit IRC | 00:49 | |
| *** nurdie has joined #openstack-ansible | 01:01 | |
| *** chhagarw has joined #openstack-ansible | 01:02 | |
| *** nurdie has quit IRC | 01:05 | |
| *** chhagarw has quit IRC | 01:07 | |
| cloudnull | NobodyCam did restarting the lxc-dnsmasq process help? | 01:13 |
| *** cmart has joined #openstack-ansible | 01:24 | |
| *** shyamb has joined #openstack-ansible | 01:26 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Pass ceph_mons as a list of inventory hosts https://review.openstack.org/644268 | 01:33 |
| *** dave-mccowan has joined #openstack-ansible | 01:51 | |
| *** nicolasbock has quit IRC | 02:01 | |
| *** cshen has joined #openstack-ansible | 02:15 | |
| *** cshen has quit IRC | 02:20 | |
| *** cshen has joined #openstack-ansible | 02:21 | |
| *** dave-mccowan has quit IRC | 02:23 | |
| *** cmart has quit IRC | 02:24 | |
| *** cshen has quit IRC | 02:26 | |
| *** shyam89 has joined #openstack-ansible | 02:50 | |
| *** shyamb has quit IRC | 02:53 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.openstack.org/643864 | 02:55 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_neutron master: [WIP] Adds DPDK support for Open vSwitch https://review.openstack.org/590810 | 03:01 |
| *** nurdie has joined #openstack-ansible | 03:02 | |
| openstackgerrit | Merged openstack/openstack-ansible-haproxy_server master: Variable to set HTTP keepalive mode https://review.openstack.org/642293 | 03:03 |
| *** nurdie has quit IRC | 03:06 | |
| *** shyam89 has quit IRC | 03:08 | |
| *** shyamb has joined #openstack-ansible | 03:11 | |
| *** shyamb has quit IRC | 03:52 | |
| *** shyamb has joined #openstack-ansible | 03:52 | |
| *** shyamb has quit IRC | 03:58 | |
| *** raukadah is now known as chandankumar | 04:00 | |
| *** chhagarw has joined #openstack-ansible | 04:08 | |
| *** udesale has joined #openstack-ansible | 04:14 | |
| *** cshen has joined #openstack-ansible | 04:22 | |
| *** cshen has quit IRC | 04:26 | |
| *** nurdie has joined #openstack-ansible | 05:02 | |
| *** sm806 has quit IRC | 05:20 | |
| *** sm806 has joined #openstack-ansible | 05:20 | |
| *** udesale has quit IRC | 05:31 | |
| *** shyamb has joined #openstack-ansible | 05:39 | |
| *** shyamb has quit IRC | 05:41 | |
| *** shyamb has joined #openstack-ansible | 05:41 | |
| *** miloa has joined #openstack-ansible | 06:17 | |
| *** ivve has joined #openstack-ansible | 06:19 | |
| *** nurdie has quit IRC | 06:30 | |
| *** shyamb has quit IRC | 06:32 | |
| *** shyamb has joined #openstack-ansible | 06:36 | |
| *** mkuf has joined #openstack-ansible | 06:50 | |
| *** markvoelker has joined #openstack-ansible | 06:51 | |
| *** DanyC has joined #openstack-ansible | 07:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Bring back the --serial flag alive https://review.openstack.org/643947 | 07:06 |
| *** shyamb has quit IRC | 07:13 | |
| *** pcaruana has joined #openstack-ansible | 07:14 | |
| *** shyamb has joined #openstack-ansible | 07:16 | |
| *** tosky has joined #openstack-ansible | 07:25 | |
| *** DanyC has quit IRC | 07:30 | |
| *** pcaruana has quit IRC | 07:33 | |
| *** cshen has joined #openstack-ansible | 07:34 | |
| *** pcaruana has joined #openstack-ansible | 07:34 | |
| *** kopecmartin|off is now known as kopecmartin | 07:36 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Beautify Generate test-list file shell task https://review.openstack.org/644499 | 07:41 |
| *** DanyC has joined #openstack-ansible | 07:42 | |
| *** DanyC has quit IRC | 07:48 | |
| *** shyamb has quit IRC | 07:53 | |
| *** hamzaachi has joined #openstack-ansible | 08:03 | |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: Add the ability to set the rabbitmq bind address https://review.openstack.org/644379 | 08:04 |
| fnpanic | good morning | 08:04 |
| *** luksky has joined #openstack-ansible | 08:07 | |
| *** udesale has joined #openstack-ansible | 08:09 | |
| *** luksky has quit IRC | 08:16 | |
| *** phasespace has joined #openstack-ansible | 08:22 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest stable/rocky: Bring back the --serial flag alive https://review.openstack.org/644505 | 08:22 |
| *** priteau has joined #openstack-ansible | 08:37 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest stable/rocky: Bring back the --serial flag alive https://review.openstack.org/644505 | 08:40 |
| *** shyamb has joined #openstack-ansible | 08:41 | |
| *** ygk_12345 has joined #openstack-ansible | 08:45 | |
| ygk_12345 | odyssey4me: hi :) | 08:45 |
| ygk_12345 | odyssey4me: need some help from you bro :) | 08:45 |
| ygk_12345 | can anyone help me cinder in OSA | 08:48 |
| ygk_12345 | *with cinider | 08:48 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Beautify Generate test-list file shell task https://review.openstack.org/644499 | 08:50 |
| ygk_12345 | can anyone help me please with cinder drivers in rocky 18.1.4 version ? | 08:50 |
| ygk_12345 | i cant find drivers for Dell equilogic | 08:53 |
| ygk_12345 | the cinder-volume is failing becos of that | 08:53 |
| *** kukacz has quit IRC | 09:11 | |
| *** jbadiapa has joined #openstack-ansible | 09:13 | |
| *** DanyC has joined #openstack-ansible | 09:14 | |
| *** DanyC has quit IRC | 09:18 | |
| *** luksky has joined #openstack-ansible | 09:19 | |
| *** kukacz has joined #openstack-ansible | 09:28 | |
| *** kukacz has quit IRC | 09:29 | |
| *** kukacz has joined #openstack-ansible | 09:30 | |
| *** shyamb has quit IRC | 09:31 | |
| *** shyamb has joined #openstack-ansible | 09:34 | |
| CeeMac | morning | 09:38 |
| *** kukacz has quit IRC | 09:39 | |
| *** kukacz has joined #openstack-ansible | 09:39 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/rocky: Set tempest concurrency to int only https://review.openstack.org/643789 | 09:43 |
| *** shyamb has quit IRC | 09:45 | |
| *** electrofelix has joined #openstack-ansible | 09:47 | |
| *** jbadiapa has quit IRC | 09:59 | |
| *** DanyC has joined #openstack-ansible | 10:01 | |
| odyssey4me | ygk_12345 nothing is instrumented directly into OSA for that, but if you can figure out the right combination of python package and .conf entries, then it'd be easy to implement - if you can register a bug with the details then someone should be able to put it all together for you | 10:02 |
| ygk_12345 | odyssey4me: its fixed now | 10:02 |
| odyssey4me | ygk_12345 oh really? what did you need to do? | 10:03 |
| ygk_12345 | odyssey4me: i was using worng driver. eqx drivers r no longer supported | 10:03 |
| *** nurdie has joined #openstack-ansible | 10:06 | |
| *** ygk_12345 has left #openstack-ansible | 10:08 | |
| *** SmearedBeard has joined #openstack-ansible | 10:10 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Apply constraints when building the venv https://review.openstack.org/620331 | 10:10 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Use the repo if available https://review.openstack.org/620339 | 10:10 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Add toggle for python wheel build process https://review.openstack.org/644391 | 10:10 |
| *** nurdie has quit IRC | 10:10 | |
| *** jbadiapa has joined #openstack-ansible | 10:12 | |
| odyssey4me | jrosser mnaser there's a bit of low hanging fruit here if you guys have some people to throw at it... of course anyone else is also welcome to it: https://bugs.launchpad.net/openstack-ansible/+bug/1820830 | 10:22 |
| openstack | Launchpad bug 1820830 in openstack-ansible "Replace custom netloc filters with sensible native urlsplit filter" [Undecided,New] | 10:22 |
| odyssey4me | s/sensible/ansible/ - damned autocucumber :p | 10:22 |
| mkuf | hi there, i'd like to enable ssl for my galera cluster. setting 'galera_use_ssl: true' works fine and a self-signed cert is created but the generated galera-ca.pem isn't deployed to any container while running the os-* plays. Any idea, what i'm doing wrong? Testing on a 17.1.7 AIO. | 10:25 |
| *** rgogunskiy has quit IRC | 10:27 | |
| *** jbadiapa has quit IRC | 10:29 | |
| jrosser | mkuf: https://github.com/openstack/openstack-ansible-galera_client/blob/6b705298a53dee58bd0a3cdddaa6956a6a1a799b/tasks/galera_client_ssl.yml#L16 | 10:29 |
| *** cshen has quit IRC | 10:42 | |
| *** jbadiapa has joined #openstack-ansible | 10:43 | |
| *** nicolasbock has joined #openstack-ansible | 10:45 | |
| *** cshen has joined #openstack-ansible | 10:52 | |
| *** udesale has quit IRC | 10:53 | |
| *** SmearedBeard has quit IRC | 10:58 | |
| *** nicolasbock has quit IRC | 10:59 | |
| mkuf | jrosser: looking at main.yml, galera_client_ssl.yml should run when 'galera_use_ssl: true' | 10:59 |
| mkuf | i can't find any other playbook that references galera_client other than utility-install.yml https://github.com/openstack/openstack-ansible/blob/master/playbooks/utility-install.yml#L47 | 11:00 |
| mkuf | running utility-install.yml pulls the galera-ca into the container as expected | 11:00 |
| jrosser | you are right, the galera_client role is basically not used any more so I don't beleive the functionality you need is present | 11:04 |
| *** priteau has quit IRC | 11:08 | |
| *** stuartgr has quit IRC | 11:22 | |
| *** nicolasbock has joined #openstack-ansible | 11:24 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Moved compilemessages to handlers https://review.openstack.org/639960 | 11:27 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Set appropriate branch key in _horizon_translations_pull https://review.openstack.org/639978 | 11:27 |
| *** stuartgr has joined #openstack-ansible | 11:28 | |
| *** cshen has quit IRC | 11:30 | |
| *** cshen has joined #openstack-ansible | 11:44 | |
| chandankumar | odyssey4me: evrardjp dropped the email regarding ansible role & action plugin packaging http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003976.html | 11:53 |
| nicolasbock | I found why the opensuse distro checks are failing in tempest: | 11:53 |
| nicolasbock | The tempest version we ship in the rocky branch doesn't set the default `--concurrency` command line option which leads to a `None`, which can't be parsed to an int. | 11:54 |
| nicolasbock | I'll push an update to our repo | 11:54 |
| chandankumar | jrosser: odyssey4me serial patch backported to rocky https://review.openstack.org/#/c/644505/ | 11:57 |
| *** dave-mccowan has joined #openstack-ansible | 11:58 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest stable/rocky: Bring back the --serial flag alive https://review.openstack.org/644505 | 12:04 |
| *** luksky has quit IRC | 12:07 | |
| *** jbadiapa has quit IRC | 12:11 | |
| chandankumar | jrosser: stestr pinning patch is no longer needed https://review.openstack.org/#/c/643550/ as the requriements install for tempest is fixed by odyssey4me | 12:18 |
| *** luksky has joined #openstack-ansible | 12:20 | |
| *** markvoelker has quit IRC | 12:24 | |
| *** luksky has quit IRC | 12:29 | |
| *** SmearedBeard has joined #openstack-ansible | 12:33 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] AIO - Create HAProxy self-signed certificates during host prep https://review.openstack.org/644555 | 12:37 |
| jrosser | chandankumar: do we abandon that then https://review.openstack.org/#/c/643550/ ? | 12:41 |
| chandankumar | jrosser: yes abandoning | 12:41 |
| *** luksky has joined #openstack-ansible | 12:42 | |
| *** brtknr has joined #openstack-ansible | 12:44 | |
| *** ansmith has joined #openstack-ansible | 12:50 | |
| evrardjp | jrosser: curious about that patch 644555, you want to drop the generation from other places? | 12:51 |
| evrardjp | I think it would be wise :) | 12:51 |
| jrosser | evrardjp: you mean remove from the haproxy role and so on? | 12:53 |
| *** canori01 has joined #openstack-ansible | 12:53 | |
| jrosser | supplying the vars in user_variables should suppress haproxy making its own | 12:54 |
| jrosser | i was also seeing the questions before about galera where we only have a half job now that galera_client isnt used anymore, so yes it would be good to overhaul the galera role | 12:55 |
| evrardjp | jrosser: great | 12:56 |
| evrardjp | :) | 12:56 |
| evrardjp | something I never got the chance to tackle | 12:56 |
| jrosser | evrardjp: i also did this which is the counterpart https://review.openstack.org/#/c/641445/ | 12:56 |
| *** django has quit IRC | 12:56 | |
| kaiokmo | mnaser: ping | 12:58 |
| *** rgogunskiy has joined #openstack-ansible | 12:59 | |
| mkuf | jrosser: bummer :/ so there's no official way to deploy the ca-cert with osa? | 13:00 |
| jrosser | mkuf: there used to be, the galera role was in two parts, the server part optionally created the certificates and the client part distributed them | 13:01 |
| jrosser | but the galera_client role is not used any more | 13:02 |
| *** udesale has joined #openstack-ansible | 13:02 | |
| *** ansmith_ has joined #openstack-ansible | 13:02 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] AIO - Create HAProxy self-signed certificates during host prep https://review.openstack.org/644555 | 13:02 |
| jrosser | mkuf: but asyou see i am working on some more general certificate features now ^^ which will help fix this eventually | 13:03 |
| *** ansmith has quit IRC | 13:05 | |
| *** SmearedBeard has quit IRC | 13:05 | |
| *** dmsimard has quit IRC | 13:07 | |
| guilhermesp | morning | 13:07 |
| *** dmsimard has joined #openstack-ansible | 13:07 | |
| chandankumar | guilhermesp: \o/ | 13:08 |
| chandankumar | guilhermesp: all the patches are merged now for rocky tempest we are all good now | 13:08 |
| guilhermesp | nice patches chandankumar ! unblock a bunch of my PR, including this one https://review.openstack.org/#/c/642614/ | 13:08 |
| guilhermesp | reviews from the core team would be really appreciate! Mistral is already running in one of our rocky production envs | 13:09 |
| guilhermesp | also, cores, I'm about to create the cycle highlights for the last cycle.. any important highlights you'd like to mention ? | 13:12 |
| *** SmearedBeard has joined #openstack-ansible | 13:13 | |
| *** fresta has quit IRC | 13:13 | |
| *** fresta has joined #openstack-ansible | 13:14 | |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: WIP Prepare for manual execution https://review.openstack.org/640348 | 13:15 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest stable/rocky: Bring back the --serial flag alive https://review.openstack.org/644505 | 13:17 |
| *** jbadiapa has joined #openstack-ansible | 13:18 | |
| mkuf | jrosser: i'll have to find another way then, at least for now. thanks for the insight. | 13:23 |
| jrosser | mkuf: if you are able to help work on the code, that would be great | 13:23 |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: WIP User guide https://review.openstack.org/644570 | 13:28 |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: WIP Prepare for manual execution https://review.openstack.org/640348 | 13:30 |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: WIP User guide https://review.openstack.org/644570 | 13:31 |
| evrardjp | we should discuss when we branch today | 13:32 |
| evrardjp | guilhermesp: great -- I am jsut prepping a release which is something you could base yourself on | 13:32 |
| guilhermesp | that'd be nice evrardjp | 13:33 |
| evrardjp | that's good timing | 13:35 |
| mkuf | jrosser: i'm definitely keen to dig into it, but for the next few weeks i'm blocked with the deployment of our private cloud :/ | 13:36 |
| noonedeadpunk | hi everyone | 13:42 |
| fnpanic | hi | 13:43 |
| noonedeadpunk | it seems, that smth went wrong with galera role for centos | 13:43 |
| *** marst has joined #openstack-ansible | 13:44 | |
| noonedeadpunk | or it was just a temporary glitch... | 13:45 |
| mnaser | kaiokmo: bonjour | 13:50 |
| mnaser | odyssey4me: would you by any chance have an example of "before" and "after | 13:50 |
| mnaser | for the urlsplit stuff | 13:50 |
| *** BjoernT has joined #openstack-ansible | 13:53 | |
| kaiokmo | mnaser: I'm facing some issues regarding magnum deployment on rocky (tag 18.1.4) | 13:56 |
| cloudnull | mornings | 13:56 |
| kaiokmo | service setup is failing because of https://github.com/openstack/openstack-ansible-os_magnum/blob/stable/rocky/tasks/magnum_service_setup.yml#L121-L134 | 13:57 |
| guilhermesp | what kind of failures are you seeing kaiokmo ? | 13:58 |
| kaiokmo | after some hacking I added ignore_errors=True to the task and performed the task manually using the CLI | 13:58 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Moved compilemessages to handlers https://review.openstack.org/639960 | 13:59 |
| kaiokmo | because of this failure the service was not able to authenticate with keystone (using the trustee) | 13:59 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Set appropriate branch key in _horizon_translations_pull https://review.openstack.org/639978 | 14:00 |
| kaiokmo | guilhermesp: the mentioned task is failing because it can't find the user "trustee_domain_admin", defined here https://github.com/openstack/openstack-ansible-os_magnum/blob/stable/rocky/defaults/main.yml#L120 | 14:00 |
| evrardjp | noonedeadpunk: still didnt' get the chance to have a look at yoru patch | 14:00 |
| odyssey4me | mnaser sure, I can add that into the bug if you like | 14:02 |
| noonedeadpunk | evrardjp: no problem:) Jesse just reviewed it. I did a big mistake and place 2 patches at once regarding the same role - I should avoid this for sure in the future:) | 14:03 |
| evrardjp | that's how you learn | 14:03 |
| guilhermesp | kaiokmo: can't you list the trustee_domain_admin using cli? did you check if was created? | 14:03 |
| noonedeadpunk | Just still can not find balance between dividing into small patches and not forgetting to place them after previous gets merged | 14:04 |
| evrardjp | the amount of open bugs is very large | 14:05 |
| kaiokmo | guilhermesp: yup, it was created and without the admin role | 14:05 |
| kaiokmo | this is the exception output from the module: https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/openstack/os_user_role.py#L151 | 14:06 |
| jrosser | mnaser: we started looking at the urlsplit stuff already - sorry forgot to assign the bug | 14:06 |
| guilhermesp | hum yeah but should have kaiokmo did you assign the role manually? | 14:06 |
| kaiokmo | after consecutive failures I hacked into ignoring errors on the task and I did assign the role manually after playbook execution | 14:08 |
| odyssey4me | mnaser hopefully that's better? https://bugs.launchpad.net/openstack-ansible/+bug/1820830 | 14:12 |
| openstack | Launchpad bug 1820830 in openstack-ansible "Replace custom netloc filters with ansible native urlsplit filter" [Undecided,New] - Assigned to Georgina Shippey (gshippey) | 14:12 |
| odyssey4me | jrosser ^ | 14:12 |
| tosky | jrosser: re https://review.openstack.org/#/c/642787/ - you are right, I was waiting for https://review.openstack.org/#/c/642041/ to be merged (and if anyone wants to approve it, it will unlock 3 or 4 other os_sahara older patches :) | 14:13 |
| odyssey4me | mnaser I see that gshippey has picked it up - thanks jrosser ! | 14:14 |
| *** gshippey has joined #openstack-ansible | 14:18 | |
| *** alvinstarr has joined #openstack-ansible | 14:19 | |
| *** nurdie has joined #openstack-ansible | 14:20 | |
| *** SmearedBeard has quit IRC | 14:21 | |
| tosky | thanks guilhermesp :) | 14:23 |
| odyssey4me | looks like centos is having a bad percona day: http://logs.openstack.org/03/637503/2/check/openstack-ansible-functional-distro_install-centos-7/84f46f5/job-output.txt.gz#_2019-03-19_13_25_16_864510 | 14:24 |
| *** BjoernT has quit IRC | 14:24 | |
| noonedeadpunk | Yep, also noticed it, but after recheck everything worked for me. Seems like some floating issue | 14:26 |
| chandankumar | odyssey4me: is there a way to remove dependency from this var keystone_service_internaluri_insecure in os_tempest | 14:26 |
| chandankumar | ? | 14:26 |
| chandankumar | http://codesearch.openstack.org/?q=keystone_service_internaluri_insecure&i=nope&files=.yml&repos= | 14:26 |
| chandankumar | at most of the places it is used as a false | 14:26 |
| chandankumar | or better replacing it with something tempest_specific? | 14:27 |
| odyssey4me | chandankumar yeah, os_tempest could easily have a new var which defaults to the value of that var - although I'd suggest a different name, because that name is terrible | 14:29 |
| odyssey4me | using a negative for a var name isn't great, although it's a bit semantic | 14:29 |
| *** cmart has joined #openstack-ansible | 14:30 | |
| odyssey4me | I'd suggest something like 'tempest_keystone_interface_secure: {{ not ((keystone_service_internaluri_insecure | default(false)) | bool) }}' as a new default for os_tempest, then have all os_tempest use it | 14:31 |
| chandankumar | odyssey4me: sure doing that | 14:31 |
| *** phasespace has quit IRC | 14:32 | |
| odyssey4me | ugh, it looks like we're hitting all kinds of issues for centos with the percona repositories today :/ | 14:43 |
| odyssey4me | mnaser I've been wondering whether the use of the percona tooling shouldn't perhaps be changed to an opt-in. | 14:44 |
| odyssey4me | Any thoughts on that jrosser ? | 14:44 |
| noonedeadpunk | And creation date of repodata is today... | 14:44 |
| odyssey4me | Or an alternative could be to make it default to use it if the galera cluster size >1, given that the primary purpose of it is for SST's. | 14:45 |
| jrosser | odyssey4me: well - did you see this https://review.openstack.org/#/c/643798/ | 14:46 |
| odyssey4me | That way it wouldn't bother our gating for any repo except the galera repo tests. | 14:46 |
| *** BjoernT has joined #openstack-ansible | 14:46 | |
| jrosser | i had a long look at that and so did mnaser too, one of the xtrabackup bits seems missing and it's really unclear how to fix | 14:47 |
| odyssey4me | oof - that's a jump to 10.3 which failed hard | 14:47 |
| jrosser | it's missing a replication script as far as i can see (this is by no means my area of expertise) | 14:47 |
| masterpe | What is the best sollution with openstack-ansible when for example setup-hosts fails, to start again. Do you can do a --start-at-task option? | 14:49 |
| odyssey4me | masterpe run the whole thing again - I don't think we've done enough to make retry files or start-at-task work nicely | 14:50 |
| noonedeadpunk | You may safely start setup-hosts again | 14:50 |
| odyssey4me | masterpe alternatively, setup-hosts is just an include of some other plays, so you can run any of those individual playbooks | 14:51 |
| *** rgogunskiy has quit IRC | 14:51 | |
| odyssey4me | jrosser mine neither, where's a cloudnull when you need one :p | 14:53 |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: WIP User guide https://review.openstack.org/644570 | 14:53 |
| masterpe | too bad, I just was running a setup-hosts and after running for 4 hours it failed. | 14:55 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: Do not configure openstack policies by default https://review.openstack.org/640301 | 14:56 |
| *** luksky has quit IRC | 14:58 | |
| odyssey4me | masterpe that sounds like excessvely long | 14:58 |
| cloudnull | odyssey4me whats up? | 15:04 |
| odyssey4me | jrosser mnaser cloudnull hmm, look at the note at the top of https://mariadb.com/kb/en/library/percona-xtrabackup-overview/ | 15:04 |
| odyssey4me | it looks like it's time to switch from xtrabackup to mariabackup | 15:05 |
| cloudnull | https://mariadb.com/kb/en/library/mariabackup/ | 15:05 |
| cloudnull | ftw? | 15:05 |
| odyssey4me | also https://severalnines.com/blog/database-backups-comparing-mariadb-mariabackup-and-percona-xtrabackup | 15:06 |
| odyssey4me | it looks like we've been using mariadb 10.1 since ocata, so in theory we've got a broken backup implementation for all current branches... I guess the SST's seem to be working though | 15:11 |
| *** cmart has quit IRC | 15:11 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Fix deployment and tests (correct URIs, plugins) https://review.openstack.org/642041 | 15:16 |
| jrosser | odyssey4me: oh good find - background is that there have been a bunch of tests breaking with galera internal errors | 15:21 |
| jrosser | so that patch to bump to 10.3 was i guess an attempt to bring in some bug fixes | 15:22 |
| odyssey4me | jrosser yep, and perhaps now we're seeing why those bugs are coming up - we're using a bad software companion | 15:24 |
| spotz | :( | 15:24 |
| odyssey4me | the good news is that if we switch to using mariabackup, we ditch one more external repo's involvement - which is helpful to improve stability | 15:25 |
| spotz | And adds a touch of speed | 15:26 |
| *** SmearedBeard has joined #openstack-ansible | 15:29 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] AIO - Create HAProxy self-signed certificates during host prep https://review.openstack.org/644555 | 15:30 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] AIO - Create HAProxy self-signed certificates during host prep https://review.openstack.org/644555 | 15:32 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use requirements.yml for managing os_tempest deps https://review.openstack.org/644600 | 15:36 |
| chandankumar | evrardjp: ^^ short term gap solution | 15:37 |
| evrardjp | odyssey4me: I tested SST with 10.1 | 15:41 |
| evrardjp | it works | 15:42 |
| evrardjp | but I agree on the fact it would be better for backups | 15:42 |
| evrardjp | to use mariabackup | 15:42 |
| evrardjp | afaik | 15:42 |
| evrardjp | I mean for simplicity | 15:43 |
| *** luksky has joined #openstack-ansible | 15:51 | |
| brtknr | Any Ansible reviewers able to take a look at this: https://github.com/ansible/ansible/pull/53757 | 15:52 |
| brtknr | Its been sitting there for a few days... | 15:53 |
| tosky | what is the general policy for trivial patches like https://review.openstack.org/616521, https://review.openstack.org/616235 and https://review.openstack.org/614116 ? Fine with them? Not needed? Single-reviewer approval or not? | 15:54 |
| *** hamzy has quit IRC | 15:55 | |
| *** BjoernT has quit IRC | 15:57 | |
| masterpe | odyssey4me: are there some documentation to speedup the process? | 15:57 |
| masterpe | The cluster has 3 controllers, 2 compute nodes and 3 storage nodes | 15:58 |
| *** phasespace has joined #openstack-ansible | 15:58 | |
| NobodyCam | Good Morning OSA Folks | 15:58 |
| masterpe | The deploynode had 15 cpu cores, so I use --forks=15 on it. | 15:58 |
| odyssey4me | masterpe well, you may wish to look into what tasks took so long and then try to figure out why - a small cluster like that should not be taking so long unless your internet connection is terrible, or your compute power is not great... in which case you may wish to tune down the number of processes used by services | 15:59 |
| *** miloa has quit IRC | 15:59 | |
| odyssey4me | oh, and don't use forks above 10 unless you've tuned sshd to allow more than 10 connections (which is the default) | 15:59 |
| odyssey4me | there's a bug about that in OSA's history somewhere | 15:59 |
| NobodyCam | would anyone have an idea as to why A lxc container would not get a eth0 assigned? | 15:59 |
| odyssey4me | and a doc note | 15:59 |
| mnaser | #startmeeting openstack_ansible_meeting | 16:01 |
| openstack | Meeting started Tue Mar 19 16:01:37 2019 UTC and is due to finish in 60 minutes. The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:01 |
| *** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)" | 16:01 | |
| openstack | The meeting name has been set to 'openstack_ansible_meeting' | 16:01 |
| guilhermesp | o/ | 16:02 |
| mnaser | So on record I’m in favour of Mariabackup | 16:02 |
| kaiokmo | o/ | 16:02 |
| chandankumar | \o/ | 16:02 |
| *** BjoernT has joined #openstack-ansible | 16:02 | |
| *** BjoernT has quit IRC | 16:04 | |
| *** cyberpear has joined #openstack-ansible | 16:04 | |
| mnaser | #topic office hours | 16:05 |
| *** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)" | 16:05 | |
| mnaser | jrosser: odyssey4me ^^ about mariabackup | 16:05 |
| mnaser | or we can also do good ol rsync | 16:05 |
| chandankumar | is it mariabackup or mariadbbackup? | 16:07 |
| nicolasbock | o/ | 16:07 |
| *** yetiszaf has joined #openstack-ansible | 16:09 | |
| jamesdenton | o/ | 16:09 |
| spotz | o/ | 16:10 |
| jrosser | o/ hello | 16:12 |
| *** BjoernT has joined #openstack-ansible | 16:12 | |
| evrardjp | o/ | 16:14 |
| evrardjp | mnaser: I have submitted the request for our first alpha of stein. | 16:14 |
| evrardjp | I would like to freeze things again for RC soon. | 16:15 |
| evrardjp | when are we ready to branch ? | 16:15 |
| evrardjp | god, topic office hours is confusing me, should discuss about having office hours, or is this office hours? | 16:15 |
| guilhermesp | and for the record, I'm going to do the cycle highlights | 16:15 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] AIO - Create HAProxy self-signed certificates during host prep https://review.openstack.org/644555 | 16:16 |
| *** SmearedBeard has quit IRC | 16:17 | |
| *** jbadiapa has quit IRC | 16:17 | |
| *** hamzy has joined #openstack-ansible | 16:18 | |
| evrardjp | guilhermesp: thanks there | 16:18 |
| mnaser | It’s just office hours at this point. | 16:19 |
| mnaser | I think we should branch after release imho... | 16:20 |
| evrardjp | after release of openstack upstream projects? | 16:20 |
| evrardjp | I mean other projects, upstream is kinda weird here | 16:21 |
| chandankumar | on RDO side, we have started building packages for stein | 16:21 |
| evrardjp | mnaser: fine for me | 16:21 |
| evrardjp | it means less backporting | 16:21 |
| chandankumar | https://trunk.rdoproject.org/centos7-stein/report.html | 16:22 |
| mnaser | i think after stein has been released in terms of the tarballs from releases.o.o | 16:23 |
| *** laurni1 has joined #openstack-ansible | 16:23 | |
| mnaser | that way we consume something that fully works | 16:23 |
| spotz | Don't we usually cut like 2 weeks after everyone else? | 16:24 |
| mnaser | we could do that too as well | 16:24 |
| mnaser | the thing is i feel that our code is still moving often and backports might be missed, we're getting much more stable | 16:25 |
| evrardjp | spotz: we haven't done that at previous release | 16:25 |
| evrardjp | spotz: we don't have that mandatory requirement anymore | 16:25 |
| mnaser | so we could evaluate 2 weeks after and see what happens, but i think nailing down a very stable release and then pushing it out is better | 16:25 |
| evrardjp | it was useful to wait a little tbh | 16:25 |
| spotz | evrardjp: But it was a sensible one:) | 16:25 |
| *** cshen has quit IRC | 16:25 | |
| mnaser | that way we dont have to backport stuff like stein support for $some_distro or so | 16:25 |
| spotz | +2 | 16:25 |
| evrardjp | well here I am concerned that we'll branch very late | 16:25 |
| evrardjp | and then release maybe shortly after | 16:25 |
| *** nurdie has quit IRC | 16:26 | |
| evrardjp | but we have a few months now, so we will have time | 16:26 |
| *** SmearedBeard has joined #openstack-ansible | 16:26 | |
| evrardjp | mnaser: I would like to propose branching when others are doing it, so we can bump things properly. No problem with requirements either | 16:26 |
| evrardjp | and it takes time for the dust to settle when branching | 16:26 |
| evrardjp | "oh we forgot this" | 16:27 |
| mnaser | i'm curious on what's the concern of us release a month after openstack is out considering it sa deployment project | 16:27 |
| mnaser | tripleo releases well after once they have everything nailed down | 16:27 |
| spotz | That's how I thougt we were still doing it, weeks vs months though | 16:28 |
| jrosser | we should put a bit of effort now into housekeeping - like the very old galera version | 16:29 |
| evrardjp | mnaser: it's no problem for releasing late | 16:29 |
| evrardjp | I am just concerned about branching late. | 16:29 |
| evrardjp | I would prefer branching when all the others have branched | 16:30 |
| evrardjp | or most of the others have branched | 16:30 |
| mnaser | maybe i'm not following but i'm not sure why we cant just branch and then release afterwards. if we branch when nova branched but our mariadb is still pushing 10.1 | 16:30 |
| mnaser | and now we gotta backport a whole collection of patches to get a *stable* branch to use 10.3 | 16:30 |
| evrardjp | I am confused -- when do you want to branch? | 16:31 |
| *** ivve has quit IRC | 16:31 | |
| evrardjp | If it's after the official release of other projects, it means during a time we'll be testing those project's master branch, which might be less stable than their newly created stable branch | 16:32 |
| mnaser | imho i dont think our biggest problem is projects breaking us, it's more us breaking us | 16:33 |
| mnaser | so i don't know if that's as much of a risk | 16:33 |
| mnaser | i would like others to chime in though | 16:33 |
| *** BjoernT has quit IRC | 16:33 | |
| openstackgerrit | Nicolas Bock proposed openstack/openstack-ansible stable/rocky: [WIP] Default to setting --concurrency until SUSE package is merged https://review.openstack.org/644612 | 16:33 |
| spotz | Stable is a relative term but I think we need to wait until say nova says we're good, then have the time to make sure we're good before we say we're good:) | 16:34 |
| evrardjp | that's the thing, when nova branches, master will be for master dev | 16:34 |
| evrardjp | I don't see the backport as massively painful. | 16:35 |
| guilhermesp | I agree with spotz and evrardjp ... it is even safer and clear if we are breaking us | 16:35 |
| jrosser | iirc for R we had to backport a ton of stuff. odyssey4me did a really great job of keeping track of what was going into master and backporting the necessary stuff. | 16:36 |
| evrardjp | I agree on branching late | 16:36 |
| evrardjp | but at a point we should say stop | 16:36 |
| evrardjp | I am merely asking when | 16:36 |
| evrardjp | for me, when other project branch is a good time, because it's clear | 16:37 |
| mnaser | but i dont know if that's enough time for us to clean up our current state | 16:37 |
| jrosser | mnaser: you got a todo list so we know what done looks like? | 16:37 |
| mnaser | jrosser: from the top of my head -- updating our tests to use integrated repo for all roles, mariadb improvements, upgrade tests addition | 16:38 |
| evrardjp | these are big things | 16:38 |
| mnaser | we also have a big issue with upgrades because we now use the in-distro packaging for rabbitmq | 16:39 |
| mnaser | in master | 16:39 |
| mnaser | and 'upstream' in backports | 16:39 |
| mnaser | err | 16:39 |
| mnaser | 'upstream' as in upstream rabbitmq in stable | 16:39 |
| mnaser | which will result in a downgrade during upgrades... no bueno | 16:39 |
| *** jbadiapa has joined #openstack-ansible | 16:39 | |
| mnaser | and the python_venv_build clean up work, in order to eliminate the repo container by default too | 16:40 |
| mnaser | i think those a pretty critical for release if we want to put a proper stamp of approval. i'm open if people disagree | 16:40 |
| evrardjp | I am not disagreeing with the plan, I just don't have resources to put in so big changes | 16:40 |
| mnaser | indeed.. we can reduce the # of things | 16:41 |
| tosky | technically openstack-ansible is "cycle-trailing", like other deployment systems, so there is a bit of time after the branching of the managed projects | 16:41 |
| mnaser | i just think being 'goal' based rather than 'time' based might be better approach. | 16:41 |
| *** kopecmartin has quit IRC | 16:42 | |
| evrardjp | tosky: 3 months iirc | 16:42 |
| guilhermesp | yes 3 months | 16:42 |
| evrardjp | https://releases.openstack.org/reference/release_models.html#cycle-trailing | 16:42 |
| guilhermesp | https://docs.openstack.org/project-team-guide/release-management.html#trailing-the-common-cycle | 16:42 |
| evrardjp | god we have duplicate docs there -- that's bad. | 16:43 |
| *** tstrul has joined #openstack-ansible | 16:43 | |
| guilhermesp | heheheheh | 16:43 |
| *** laurni1 has quit IRC | 16:47 | |
| *** udesale has quit IRC | 16:48 | |
| tstrul | Hello, I just upgraded from Ocata to Pike and I have duplicate hypervisor records (every hypervisor is seen twice in openstack hypervisor list), overview page in horizon is stuck as well... It looks like its related to this bug: https://bugs.launchpad.net/openstack-ansible/+bug/1736731. looks like an open bug? how can i fix this? Thank you! | 16:49 |
| openstack | Launchpad bug 1736731 in openstack-ansible "os_nova might create a duplicate cell1" [High,Confirmed] - Assigned to Jean-Philippe Evrard (jean-philippe-evrard) | 16:49 |
| jrosser | on a totally unscientific grep we still have 7 repos containing policy.json.j2 - finishing off the smart sources work feels important too | 16:50 |
| mnaser | yep, what jrosser brings up too | 16:51 |
| mnaser | i think we need to start working on this and see where we land | 16:51 |
| jrosser | every time i look through gate run logs the service journals are full of config deprecation warnings too | 16:53 |
| guilhermesp | ok so we have a growing list with things that should be fixed/improved before we come up branching... like finishing the smart sources and fixing mariadb stuff but not a decision when to branch after the others right? | 16:53 |
| *** luksky has quit IRC | 16:54 | |
| mnaser | jrosser, guilhermesp: i agree, maybe we should setup a hack day | 16:59 |
| guilhermesp | agreed mnaser | 16:59 |
| jrosser | that would be time better spend than another bug sqaush day currently | 17:00 |
| *** hamzaachi has quit IRC | 17:01 | |
| *** cmart has joined #openstack-ansible | 17:01 | |
| guilhermesp | hacking stuff around could avoid the need of bug squash days :P | 17:02 |
| mnaser | #endmeeting | 17:03 |
| *** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC" | 17:03 | |
| openstack | Meeting ended Tue Mar 19 17:03:50 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:03 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-03-19-16.01.html | 17:03 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-03-19-16.01.txt | 17:03 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2019/openstack_ansible_meeting.2019-03-19-16.01.log.html | 17:03 |
| mnaser | okay.. maybe it would be good to have an etherpad to track these release goals? | 17:04 |
| guilhermesp | that's the skel https://etherpad.openstack.org/p/osa-trail-goals mnaser | 17:05 |
| mnaser | guilhermesp: this would probably be for stein :p | 17:05 |
| guilhermesp | oh sorry | 17:05 |
| guilhermesp | well, will be saved for trail tho | 17:05 |
| guilhermesp | https://etherpad.openstack.org/p/osa-stein-goals | 17:05 |
| *** hamzaachi has joined #openstack-ansible | 17:05 | |
| *** luksky has joined #openstack-ansible | 17:07 | |
| tstrul | Sorry guys i didnt notice i interrupted the meeting, any chance someone could take a look on the issue described above? | 17:08 |
| guilhermesp | Just added some items that we discussed today. Feel free to expand | 17:08 |
| NobodyCam | Good Morning OSA folks | 17:09 |
| NobodyCam | can someone point me to were OSA generates the network list for guests | 17:10 |
| chandankumar | mnaser: evrardjp can we priotize systemd-nspawn stuff for next release? I want to reuse the kolla containers for tempest ? | 17:10 |
| NobodyCam | I am attempting to add the octavia service but the lxc guest is only getting loop back interface created | 17:10 |
| mnaser | chandankumar: i don't think anyone has the time to work on that :\ | 17:12 |
| mnaser | evrardjp works on OSA on personal time, most those involved in nspawn aren't working on the project much | 17:12 |
| evrardjp | yeah, and nspawn is not my prio on my personal time | 17:13 |
| chandankumar | mnaser: if someone guide me, I will get os_tempest working on nspawn | 17:13 |
| jrosser | even then nspawn != kolla container? | 17:13 |
| mnaser | we have to get all of nspawn working for the integrated gate, we cant just.. nspawn for os_tempest | 17:13 |
| mnaser | the roles don't really care if you run them inside containers, baremetal or whatever | 17:13 |
| chandankumar | jrosser: I think we can just extract docker container image tarball, it will play nicely with nspawn | 17:14 |
| * chandankumar needs to find the blog post | 17:14 | |
| chandankumar | *that | 17:14 |
| evrardjp | I am curious, what's wrong with current tempest? | 17:15 |
| evrardjp | the role is not working nice? | 17:15 |
| mnaser | if you want to use containers ansible is probably not the solution | 17:15 |
| chandankumar | evrardjp: currently nothing wrong with os_tempest, just an assumption if everthing is running in container why not tempest from container | 17:16 |
| evrardjp | well it can be | 17:16 |
| evrardjp | I guess these are two different things | 17:16 |
| evrardjp | depends on what you are looking for | 17:16 |
| evrardjp | :) | 17:17 |
| chandankumar | here is the things , starting of last year, on Tripleo Side, we containerized all stuff, so we make tempest containerized and runnign it from there | 17:17 |
| chandankumar | tempest_Container helped one thing: donot need to install any tempest plugin everything is already there | 17:18 |
| chandankumar | we done all the stuff there, then exit from container | 17:19 |
| * chandankumar will look at nspawn stuff first | 17:20 | |
| jrosser | evrardjp: could you take a look at this? http://logs.openstack.org/55/644555/5/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/bb6d82b/job-output.txt.gz#_2019-03-19_16_35_05_190687 | 17:21 |
| jrosser | and then the subsequent FATAL: There is some really very odd stuff indeed in that data structure | 17:21 |
| chandankumar | evrardjp: do we have any job running nspawn container ? | 17:21 |
| evrardjp | jrosser: I will open ara for that | 17:22 |
| *** cshen has joined #openstack-ansible | 17:23 | |
| evrardjp | I don't have lynx eyes | 17:23 |
| jrosser | evrardjp: like this 'os_sahara': {'sahara_plugin_pip_packages': [' ', '$', "'", '(', ')', '*', ',', '-', '.', '1', '\\', '^', '_', 'a', 'b', 'c', 'e', 'g', 'h', 'i', 'l', 'm', 'n', 'p', 'r', 's', 't', 'u', 'x', '{', '|', '}'] | 17:24 |
| evrardjp | ohoh | 17:24 |
| evrardjp | I know this | 17:24 |
| *** gyee has joined #openstack-ansible | 17:24 | |
| evrardjp | I will have a look | 17:24 |
| *** priteau has joined #openstack-ansible | 17:24 | |
| chandankumar | evrardjp: mnaser https://review.openstack.org/#/c/644600/ any thoughts on this change! | 17:24 |
| jrosser | cool :) thankyou i was just 8-O | 17:25 |
| evrardjp | I am just in a meeting | 17:25 |
| evrardjp | when it's done I can check with you | 17:25 |
| *** cshen has quit IRC | 17:29 | |
| *** BjoernT has joined #openstack-ansible | 17:31 | |
| evrardjp | ok so | 17:32 |
| evrardjp | I am back | 17:32 |
| jrosser | i think we merged some sahara code very recently if that is related | 17:32 |
| evrardjp | this happened when py_pkgs was finding the elements in the list and we had a var in things | 17:32 |
| *** luksky has quit IRC | 17:32 | |
| evrardjp | yeah I will check there | 17:33 |
| evrardjp | https://github.com/openstack/openstack-ansible-os_sahara/blob/master/defaults/main.yml#L157 | 17:33 |
| evrardjp | that's the issue | 17:33 |
| evrardjp | https://github.com/openstack/openstack-ansible-os_sahara/commit/ed7ccad8987e89953e349c9dc9cd4b30232f9425 | 17:34 |
| evrardjp | 7 days ago | 17:34 |
| evrardjp | let me fix that for you | 17:34 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_sahara master: Make the _pip_packages understood by py_pkgs https://review.openstack.org/644617 | 17:38 |
| evrardjp | jrosser: & | 17:38 |
| evrardjp | woops | 17:38 |
| evrardjp | I meant ^ | 17:38 |
| jrosser | ah i see | 17:38 |
| evrardjp | ok so | 17:38 |
| evrardjp | next ? :D | 17:38 |
| mnaser | fwiw, i have it on my list to start using integrated repo for testing | 17:39 |
| evrardjp | if you check, all the chars you see in this line are basically coming from a list made from the string in that var. | 17:39 |
| evrardjp | chandankumar: reviewing https://review.openstack.org/#/c/644600/ now | 17:39 |
| evrardjp | I think it's nice to give this for user friendliness. | 17:40 |
| evrardjp | thanks chandankumar | 17:41 |
| evrardjp | mnaser: great :) | 17:41 |
| evrardjp | mnaser: FYI this was what was done long ago :p | 17:41 |
| mnaser | evrardjp: yeah, but really materializing it, and getting rid of role tests and use integrated for all of them | 17:41 |
| mnaser | the switch has taken quite a while :) | 17:41 |
| mnaser | so just really committing to the switch | 17:41 |
| jrosser | mnaser: do you have a plan for things like openstack_hosts and the other lower level things which have a bunch of unit tests? | 17:42 |
| evrardjp | the integrated brings a lot of things, but I am really looking forward to really benefit from zuul here | 17:42 |
| evrardjp | not have a single test case and reduce coverage | 17:42 |
| evrardjp | but instead leverage the integrated repo, and have the right testing based on roles | 17:42 |
| mnaser | jrosser: i haven't though about it that much tbh, i've bene thinking in scope of services mostly | 17:42 |
| evrardjp | I am enclined to have a multi node test mnaser | 17:42 |
| tosky | evrardjp: uhm, but according the logs of the jobs, the list of packages was correctly populated, or it seemed so - what did I miss? | 17:43 |
| evrardjp | next to that | 17:43 |
| mnaser | evrardjp: i would *love* that once we have all jobs using it and our jobs are stable, that'd be awesome | 17:43 |
| evrardjp | tosky: it was just fine for the role :) | 17:43 |
| evrardjp | tosky: the problem is for the integrated which still leverages py_pkgs lookup. We want to phase that lookup out though | 17:43 |
| evrardjp | in the future, that might be just fine | 17:43 |
| evrardjp | but right now we still have python wheel building somewhere | 17:44 |
| evrardjp | mnaser: dealing with hardware is slower and less reliable, but I believe this has a positive value | 17:44 |
| tosky | so there is no way to have a dynamic list like that until py_pkgs is replaced? | 17:44 |
| evrardjp | at least in periodics. | 17:44 |
| *** hamzaachi has quit IRC | 17:44 | |
| mnaser | when i say stable as in like, things like how today xtrabackup is failing | 17:44 |
| mnaser | multinode jobs eat up a lot of expensive upstream resources | 17:45 |
| evrardjp | tosky: depends -- if you rename that not ending with _pip_packages it would work, but then the pip package is not built in the repo | 17:45 |
| mnaser | i think our 'internal' bill for the donated hardware is something like $70-80k per month | 17:45 |
| evrardjp | so it would mean your pip install would need to not be frozen | 17:45 |
| tosky | evrardjp: and I really need a list of pip packages there | 17:45 |
| evrardjp | mnaser: not surprised | 17:46 |
| evrardjp | mnaser: this is why I believe running this in a weekly periodic would be fine | 17:46 |
| evrardjp | (skip early in periodic not matching day x) | 17:46 |
| mnaser | we'd need someone to actually look atthem | 17:46 |
| evrardjp | or alternatively, really build a different periodic pipeline | 17:46 |
| mnaser | dunno if anyone is looking at our periodics, used to be odyssey4me .. but yeah | 17:46 |
| tosky | evrardjp: if this change (not using py_pkgs) is planned for the not-far future, like during this cycle, re-applying the patch would not be complicated | 17:47 |
| evrardjp | yeah I used to do it at some point too, but I was PTL at that time :p | 17:47 |
| evrardjp | mnaser: :D | 17:47 |
| mnaser | i see what u did thur | 17:47 |
| mnaser | :P | 17:47 |
| evrardjp | hihi | 17:47 |
| evrardjp | tosky: what's the hold on that? | 17:47 |
| evrardjp | is there really a big problem on that? | 17:48 |
| jrosser | the big problem really is that the integrated build is totally wedged | 17:50 |
| mnaser | jrosser: again?! | 17:50 |
| mnaser | porque | 17:50 |
| jrosser | because of the sahara vars thing, we need that to merge | 17:50 |
| tosky | evrardjp: no, I'm not blocking the patch. I simply frown every time I have to duplicate an information that does not need it | 17:50 |
| tosky | mnaser: because I tried to be smart. Now let's merge this patch | 17:51 |
| mnaser | tosky: yeah, that's not pretty | 17:51 |
| mnaser | no, your thing is valid overall | 17:51 |
| mnaser | the reason it broke intergrated is because ... we're not running integrated :-) | 17:51 |
| chandankumar | evrardjp: awesome, thanks! we will solve the setup.cfg stuff later based on ML reply. | 17:51 |
| evrardjp | tosky: you might have a different name for some pip packages too, I am glad you are consistent at least | 17:52 |
| evrardjp | jrosser: no I meant on the fact to NOT want static list | 17:52 |
| evrardjp | It is a good enough stop gap for now | 17:52 |
| evrardjp | I think it could work if we make that an item in a list though | 17:52 |
| tosky | evrardjp: the names of the plugin packages are meant to follow that pattern | 17:53 |
| logan- | i don't think py_pkgs does any jinja interpretation at all | 17:53 |
| evrardjp | tosky: the problem, for me, is that we should leverage the right yaml parsing too. | 17:53 |
| tosky | but anyway, it does not matter, let's unblock the gates | 17:53 |
| logan- | what you could do is change sahara_plugins_base to a calculated var based on the _pip_packages | 17:53 |
| evrardjp | logan-: it used to not do it at least | 17:53 |
| evrardjp | logan-: that's what I said above :D | 17:53 |
| logan- | evrardjp: apologies :) | 17:53 |
| *** BjoernT has quit IRC | 17:54 | |
| evrardjp | logan-: the problem is that we probably want to have the pip packages built somewhere | 17:54 |
| logan- | yeah | 17:54 |
| evrardjp | --isolated could work, or other things | 17:54 |
| evrardjp | but anyway, wrote the workaround for now | 17:54 |
| evrardjp | we'll remove all of that in the future so... | 17:54 |
| *** chhagarw has quit IRC | 17:56 | |
| jrosser | anyway all that aside this seems to be working locally https://review.openstack.org/#/c/644555/ | 17:57 |
| jrosser | so interested to hear ideas for what we need to do moving certs being randomly generated in the roles to being pushed out from the deploy host | 17:57 |
| evrardjp | I like this idea | 17:59 |
| evrardjp | I am not sure what you ask :D | 17:59 |
| *** BjoernT has joined #openstack-ansible | 18:00 | |
| guilhermesp | regarding the cycle highlights, what items the team consider most import to point out ? Im just editing evrardjp 's pr https://review.openstack.org/#/c/644575/1/deliverables/stein/openstack-ansible.yaml | 18:01 |
| evrardjp | don't edit the PR :) | 18:01 |
| evrardjp | put it on top :D | 18:01 |
| evrardjp | well | 18:01 |
| evrardjp | I mean they are for two different purposes | 18:01 |
| evrardjp | but it's fine I guess? | 18:01 |
| guilhermesp | apologies for the use of incorrect word :) hehehhe | 18:01 |
| logan- | jrosser: i don't have strong feelings either way really, but it does diverge our test environment from the regular deployment somewhat. what do we gain from generating the cert earlier and should we apply it in a way that affects all self-signed deploys, not just the gate? | 18:05 |
| *** chandankumar is now known as raukadah | 18:05 | |
| evrardjp | logan-: isn't that the goal? | 18:07 |
| evrardjp | jrosser: ^ | 18:07 |
| *** priteau has quit IRC | 18:07 | |
| *** ivve has joined #openstack-ansible | 18:07 | |
| logan- | well right now it is in bootstrap-host which is only used for building aios | 18:07 |
| evrardjp | I am confused, I thought the only difference here is we have plays that generate the things | 18:07 |
| evrardjp | oh | 18:07 |
| evrardjp | yeah I would convert that to a play | 18:08 |
| evrardjp | or something in setup-hosts | 18:08 |
| evrardjp | if there is a file on localhost, then use it, else generate it by delegating to localhost? | 18:09 |
| logan- | and I guess the reason for the change is you then have the self signed CA cert generated early enough that you could add it to the trust stores on all of the hosts using jrosser's other patch right? | 18:09 |
| evrardjp | but setup hosts is quite early | 18:09 |
| evrardjp | but yeah not THAT early compared to bootstrap aio :D | 18:09 |
| *** bgmccollum has quit IRC | 18:09 | |
| odyssey4me | o/ apologies for missing the meeting - I was in a conf call | 18:10 |
| logan- | whats driving the need for the cluster-wide trust? is it related to the recent heat auth issues? | 18:10 |
| evrardjp | odyssey4me: no worries, we discussed releasing , we didn't get to the bugs chat. | 18:11 |
| evrardjp | so not important | 18:11 |
| evrardjp | hahah | 18:11 |
| * evrardjp removes the bug triage hat | 18:11 | |
| logan- | o/ odyssey4me | 18:11 |
| *** bgmccollum has joined #openstack-ansible | 18:12 | |
| jrosser | logan-: for magnum you need the ca cert in the magnum container if it’s self signed / corporate ca | 18:12 |
| logan- | jrosser got it | 18:12 |
| jrosser | And the same is conceivably true for keystone for auth with AD | 18:12 |
| evrardjp | jrosser: all of those run after setup_hosts | 18:13 |
| evrardjp | so it can totally be done there | 18:13 |
| evrardjp | it would also distribute certs to ALL hosts | 18:13 |
| evrardjp | including containers | 18:13 |
| jrosser | It depends where you define the vars surely? Yes everywhere for user_variables, wherever you like for group_vars | 18:14 |
| jrosser | evrardjp: you are right that my patch so far only addresses AIO, I will rework it to cover all self signed setups | 18:18 |
| odyssey4me | are you discussing where to generate certs to get distributed? | 18:19 |
| evrardjp | I am not sure what you meant there with the user_variables and group_vars, but I trust you :) | 18:19 |
| evrardjp | odyssey4me: yes, something that came up in the past | 18:19 |
| odyssey4me | I'm guessing the most sensible place is the openstack_hosts role/playbook? | 18:19 |
| odyssey4me | that's the earliest one that touches everything | 18:19 |
| evrardjp | Uniform way of doing so by generating on the localhost if doesn't exist, and distribute. | 18:19 |
| evrardjp | And yeah that's what we said odyssey4me :D | 18:19 |
| evrardjp | glad I am not becoming too outdated :D | 18:20 |
| evrardjp | glad you are not either ') | 18:20 |
| jrosser | odyssey4me: openstack_hosts for which bit? Sorry I’m confused :) | 18:20 |
| odyssey4me | it should be easy enough to do something that delegates to a 'service_setup_host' to cater for the deployment host by default, or any other delegated host if need be for special circumstances | 18:21 |
| jrosser | Needing pyOpenSSL has already made it complicated | 18:22 |
| odyssey4me | jrosser for distribution at the very least, but similar to the way the service setup is done in each role, this could work in much the same way for the generation | 18:22 |
| *** tstrul has quit IRC | 18:22 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Cleanup SUSE testing https://review.openstack.org/644626 | 18:23 |
| odyssey4me | ie add pyOpenSSL to the requirements on the deployment host, and delegate the cert generation to a service setup host (which is by default the deployment host), then do distribution from the service setup host | 18:23 |
| odyssey4me | if the service setup host != localhost, then fetch the generated certs from the service setup host and store them on the deployment host before distributing | 18:24 |
| odyssey4me | well, to me that would make sense - and it caters to situations where doing cert generation on the deployment host is either not possible or not desirable | 18:24 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Cleanup SUSE testing https://review.openstack.org/644626 | 18:26 |
| guilhermesp | odyssey4me jrosser any thought about cycle-highlights? if there is piece of information around that I can summarize and put together would be nice | 18:26 |
| evrardjp | odyssey4me: we should stop those convoluted things int he future -- only one path -- build on localhost | 18:27 |
| evrardjp | instead of pyopenssl, let's just use a shell script | 18:27 |
| evrardjp | YES I said that | 18:27 |
| evrardjp | :D | 18:27 |
| evrardjp | if you don't have openssl isntalled on your machine, well... you're pretty much doomed | 18:28 |
| evrardjp | :D | 18:28 |
| evrardjp | "live a modern life, install openSSL" | 18:28 |
| * jrosser mutters about the ancient version on centos | 18:28 | |
| evrardjp | "get out of your cavern, install openSSL" | 18:28 |
| evrardjp | jrosser: fair | 18:28 |
| evrardjp | but isn't centos7 recent enough? | 18:29 |
| jrosser | Not for pyopenssl >= 0.15 | 18:29 |
| noonedeadpunk | not for openssl | 18:29 |
| odyssey4me | oh, so the issue here is that pyopenssl doesn't work with the openssl version available on centos? | 18:31 |
| evrardjp | even openssl? | 18:31 |
| evrardjp | I mean pyopenssl I get it | 18:31 |
| evrardjp | but woot if we can't trust openssl on centos7... where are we :D | 18:31 |
| odyssey4me | well gosh darn it | 18:31 |
| jrosser | You need to install it from pip, not a distro package | 18:31 |
| jrosser | And it’s already in the ansible-runtime venv anyway at a suitable version | 18:32 |
| evrardjp | but we surely don't need pyopenssl to generate certs | 18:32 |
| evrardjp | oh | 18:32 |
| evrardjp | yeah | 18:32 |
| evrardjp | then good | 18:32 |
| evrardjp | we can delegate to localhost | 18:32 |
| odyssey4me | guilhermesp if I can finish the work to switch from repo build to have individual role python builds, then that would be a highlight | 18:32 |
| evrardjp | odyssey4me: that's not a cycle highlights, that's a herculean highlight | 18:32 |
| odyssey4me | lol | 18:32 |
| guilhermesp | evrardjp: lol heheh | 18:33 |
| odyssey4me | jrosser ok, so using the deployment host and its venv for the cert generation is ideal - what's blocking that? | 18:33 |
| *** kopecmartin has joined #openstack-ansible | 18:34 | |
| jrosser | odyssey4me: oh nothing :) that’s what my patch does right now, with the exception of being able to elect an alternate host | 18:34 |
| odyssey4me | jrosser ok, so the only env I can think of that needs an alternative host is yours - does your need it? | 18:34 |
| jrosser | No because I have the deploy host in a trusted place | 18:35 |
| jrosser | I would rather the certs were tbere | 18:35 |
| odyssey4me | of course if whoever doesn't want to use localhost, they can also generate certs whatever way they please and put them on to the deployment host afterwards | 18:35 |
| jrosser | Absolutely | 18:35 |
| odyssey4me | ok, then forget an alternative host, I say | 18:35 |
| jrosser | This is just default behaviour in absence of any real certs from the user | 18:35 |
| odyssey4me | the docs are simple - by default we do this. if you don't like it, then generate them wherever you please and put the result here | 18:36 |
| jrosser | Yep | 18:36 |
| tosky | uhm, the fix for os_sahara failed - missing packages (qpress), unrelated to the patch | 18:36 |
| jrosser | But as fall out I hope we can address the Galera certs, heat / magnum CA and so on | 18:37 |
| odyssey4me | tosky yep, know issue - something's gone funky with the centos mirrors... there was some discussion about what to do about it earlier, because it seems we shouldn't be using percona xtrabackup any more | 18:37 |
| odyssey4me | tosky evrardjp I'm super close to getting rid of py_pkgs... but yes, https://review.openstack.org/644617 is currently valid | 18:40 |
| odyssey4me | jrosser it sounds like you have it all in-hand... :) | 18:41 |
| NobodyCam | can someone assist me in figuring out what I've configured wrong with regard to Octavia network setup? | 18:41 |
| evrardjp | odyssey4me: have you ever doubted? :D | 18:41 |
| jrosser | Hah not so sure about that :) | 18:41 |
| *** hamzaachi has joined #openstack-ansible | 18:42 | |
| logan- | jrosser: why do we need pyopenssl? just for the ansible modules? | 18:42 |
| evrardjp | woot | 18:42 |
| evrardjp | http://mirror.dfw.rax.openstack.org:8080/percona/release/7/RPMS/x86_64/qpress-11-1.el7.x86_64.rpm: [Errno 14] HTTP Error 416 - Requested Range Not Satisfiable | 18:42 |
| evrardjp | Trying other mirror. | 18:42 |
| jrosser | logan-: yes that’s right, for the ansible modules | 18:43 |
| jrosser | And in fact a bunch of it needs to be done with command: anyway because the modules aren’t useful enough | 18:43 |
| logan- | imo just shell out and skip the modules | 18:43 |
| logan- | yup | 18:43 |
| evrardjp | jrosser: which is why I said, if necessary , skip the modules | 18:43 |
| logan- | the modules are pretty incomplete last i looked at them | 18:43 |
| evrardjp | logan-: ahahah great minds think alike :D | 18:43 |
| evrardjp | http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2019-03-19.log.html#t2019-03-19T18:27:28 | 18:44 |
| odyssey4me | jrosser evrardjp mnaser FYI, with regards to branching early/late - we used to have to care because of the requirements/upper-constraints changes pushing to all our repositories... I think that's no longer a thing... I think we can afford to wait to branch until the upstream projects have released | 18:44 |
| logan- | lol evrardjp | 18:44 |
| odyssey4me | if we are worried about the role tests, we can pin them at the release sha if need be | 18:44 |
| odyssey4me | afk for a bit | 18:46 |
| jrosser | NobodyCam: have you looked in the multi-node-aio code in the ops repo, that is setup for Octavia iirc | 18:47 |
| evrardjp | odyssey4me: I thought it still was | 18:47 |
| evrardjp | that's the reason | 18:48 |
| evrardjp | odyssey4me: yeah we can freeze things everywhere depends on needs | 18:48 |
| evrardjp | I just need to be informed of what we'll do :D | 18:48 |
| odyssey4me | evrardjp well, if we're desperately keen to get the role tests switched to use the integrated build then that's not far off - the only things missing are special words to enable a few features | 18:49 |
| odyssey4me | eg: more affinity, different storage back-ends, etc | 18:50 |
| *** jbadiapa has quit IRC | 18:50 | |
| odyssey4me | it's really just working through each role, looking at current tests, and figuring out how to make that happen in the integrated build | 18:50 |
| evrardjp | yeah | 18:51 |
| odyssey4me | I am hoping to have a little time to get that done before the release | 18:51 |
| evrardjp | carry a template | 18:51 |
| odyssey4me | the patterns are already there, it's just a case of working through the roles I think | 18:51 |
| NobodyCam | hi jrosser, I have and got a little confused, I am attempting to configure as vlan, I edited default/master.yml with the correct bridge name , vlan segment, network type but the octavia container is getting no eth devices other then LO | 18:51 |
| odyssey4me | anyway, I'll progress the python build work again tomorrow | 18:52 |
| odyssey4me | then move on to the test changes once the pattern is finalised for the python builds | 18:52 |
| odyssey4me | afk for a bit | 18:53 |
| NobodyCam | jrosser: I'm happy to share my user_config if that would help | 18:55 |
| NobodyCam | http://paste.openstack.org/show/WkWAngY0IpgVhI36t8fD/ | 18:56 |
| NobodyCam | all the over containers appear to be working / getting ip's | 18:57 |
| NobodyCam | but for Octavia all I get is: | 18:58 |
| NobodyCam | OpenStack01_octavia_server_container-649878a5 RUNNING 1 onboot, openstack - - | 18:58 |
| NobodyCam | if I cat the ansible_facts there are no eth devices at all | 19:00 |
| NobodyCam | http://paste.openstack.org/show/JhTky5DPCr2zBXPV6MKo/ | 19:01 |
| jrosser | Ooh our ceph/keystone patch merged \o/ https://github.com/ceph/ceph/commit/affb7d396f76273e885cfdbcd363c1882496726c | 19:02 |
| *** cshen has joined #openstack-ansible | 19:03 | |
| NobodyCam | can you point me to where the eth device list is generated? | 19:04 |
| *** DanyC has quit IRC | 19:04 | |
| jrosser | NobodyCam: https://github.com/openstack/openstack-ansible-lxc_container_create/blob/master/templates/container-interface.ini.j2 | 19:11 |
| logan- | jrosser: thats awesome! | 19:11 |
| *** tstrul has joined #openstack-ansible | 19:13 | |
| jrosser | logan-: we did 60gbits/sec 700 objects/sec and it crushed keystone completely, with that patch there’s no noticeable keystone load | 19:13 |
| *** cmart has quit IRC | 19:14 | |
| NobodyCam | jrosser: Thank you | 19:15 |
| jrosser | NobodyCam: I’m struggling on a mobile to provide a better link but you should find the config files that are dropped by that template | 19:15 |
| NobodyCam | yea, I not getting anything for that container, which is making me think it's something in my config | 19:16 |
| NobodyCam | not even the default lxcbr0 / eth0 interface | 19:16 |
| logan- | jrosser: yep the situation described in your PR is something i've seen with rgw too :) really happy to see that change in | 19:18 |
| odyssey4me | NobodyCam sounds like the container hasn't been started? | 19:18 |
| NobodyCam | it is running, just no eth devices: | 19:18 |
| NobodyCam | OpenStack01_octavia_server_container-649878a5 RUNNING 1 onboot, openstack - - | 19:18 |
| NobodyCam | this is a cat of the ansible_facts file: http://paste.openstack.org/show/JhTky5DPCr2zBXPV6MKo/ | 19:19 |
| jrosser | logan-: we’ve got some WIP on adding intel QAT support too, which lots of throughput the rgw cpu is 90% doing md5 and sha | 19:21 |
| *** cshen has quit IRC | 19:21 | |
| jrosser | So for the addition of a QAT card you can get the same through with a much much smaller cpu | 19:21 |
| logan- | nice! | 19:21 |
| odyssey4me | jrosser great work on that, it will benefit the broader openstack community too :) | 19:21 |
| jrosser | Yeah, upstream first :) | 19:22 |
| *** cshen has joined #openstack-ansible | 19:22 | |
| jrosser | Magic sauce is of course getting all that rgw I/O connected to VM efficiently | 19:22 |
| *** DanyC has joined #openstack-ansible | 19:24 | |
| *** electrofelix has quit IRC | 19:27 | |
| *** DanyC has quit IRC | 19:28 | |
| odyssey4me | alright folks, do we have any conclusions about what to do with the failing centos builds due to the broken percona things? | 19:28 |
| *** cshen has quit IRC | 19:30 | |
| *** tstrul has quit IRC | 19:31 | |
| jrosser | odyssey4me: I’ve not looked at that centos fail specifically, but did dig around mnaser original update patch. Getting an idea of what’s involved in the 10.3 update would be good. | 19:32 |
| mnaser | yes odyssey4me | 19:32 |
| mnaser | i think we're going to move towards nosql | 19:33 |
| mnaser | so we can make OSA more webscale | 19:33 |
| mnaser | alright. i'm done being silly. i'm all for switching to mariabackup. i can make my patch do that. | 19:33 |
| odyssey4me | jrosser cloudnull evrardjp mnaser guilhermesp I have a stack up for review to remove the need for repo build - it only does glance for now, but transforms glance to no longer require repo build and to handle its own constraints: https://review.openstack.org/#/q/topic:bp/python-build-simplification+status:open | 19:33 |
| odyssey4me | if those are all acceptable patches, then I think we can start to move things over to use that model and ditch repo build | 19:34 |
| mnaser | !!!!!! | 19:34 |
| openstack | mnaser: Error: "!!!!!" is not a valid command. | 19:34 |
| mnaser | million times yes | 19:34 |
| odyssey4me | the one is failing because of the unrelated percona things | 19:34 |
| odyssey4me | it's all using depends-on to ensure it merges in an order that doesn't cause problems | 19:35 |
| NobodyCam | hum so looking at lxc_container_config.yml interface config happens after Execute first script, which for me fails because I am not getting a eth0 so yum update fails | 19:35 |
| odyssey4me | oh yes, one more thing is it also gets rid of the need to deploy pip.conf | 19:35 |
| odyssey4me | oh, I left logan- out of the list, apologies logan- :) | 19:35 |
| mnaser | odyssey4me: ok let me get galera_server fixed up now so we can review that stack | 19:36 |
| logan- | nice odyssey4me! its finally happening! | 19:37 |
| odyssey4me | awesome, I'll be happy to get the other AIO services done too - but then will hope to have the rest of the roles done by helpers in the community | 19:37 |
| mnaser | i can get someone to copy/paste the process across other roles really | 19:37 |
| odyssey4me | it'll only be https://review.openstack.org/620340 that needs doing across the roles, so it's pretty straightforward | 19:38 |
| guilhermesp | nice odyssey4me looking the stack | 19:39 |
| *** cshen has joined #openstack-ansible | 19:41 | |
| *** cmart has joined #openstack-ansible | 19:50 | |
| *** cshen has quit IRC | 19:53 | |
| *** cmart has quit IRC | 19:57 | |
| mnaser | odyssey4me, jrosser: qpress is in the percona repos | 19:59 |
| mnaser | and its needed to get compression | 19:59 |
| *** cmart has joined #openstack-ansible | 19:59 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Correct notification driver https://review.openstack.org/635681 | 19:59 |
| odyssey4me | mnaser no alternatives | 19:59 |
| odyssey4me | native alternatives, I mean | 19:59 |
| mnaser | odyssey4me: well, we can either _not_ use compression, or use rsync | 20:00 |
| mnaser | i think tripleo uses rsync | 20:00 |
| odyssey4me | rsync works, it's just a lot slower | 20:00 |
| mnaser | or we can use mariabackup without compression | 20:01 |
| odyssey4me | cloudnull any thoughts? | 20:01 |
| mnaser | which tbh im not super opposed to, its not like we're pushing database over WAN links | 20:01 |
| mnaser | the CPU and I/O hit of compression and decompression has resulted in controllers being destroyed in my experience | 20:01 |
| mnaser | the transfer goes fast but once qpress starts going, it eats all iops | 20:02 |
| odyssey4me | I would personally go for mariabackup and no compression, but this change will definitely need a reno. | 20:02 |
| mnaser | http://galeracluster.com/documentation-webpages/sst.html | 20:02 |
| mnaser | apparently rsync is the fastest | 20:02 |
| odyssey4me | Of course there would also need to be something done with the upgrade process. | 20:02 |
| mnaser | it probably will naturally just restart and resync using the new system so it's not as bad as it might come out to be i think | 20:03 |
| odyssey4me | oh ,I think xtrabackup was used to prevent the donor block | 20:03 |
| mnaser | yeah | 20:03 |
| mnaser | that is pretty impactful | 20:03 |
| mnaser | rsync also features the rsync-wan modification, which engages the rsync delta transfer algorithm. However, given that this makes it more I/O intensive, you should only use it when the network throughput is the bottleneck, which is usually the case in WAN deployments. | 20:04 |
| odyssey4me | well, it seems to me that mariabackup without compression is the best for this use-case | 20:04 |
| mnaser | yea i think so | 20:04 |
| mnaser | we can kill it by default, leave the option open | 20:05 |
| cloudnull | I'd be +1 on mariabackup without compression | 20:05 |
| mnaser | but no longer pull down qpress and stuff | 20:05 |
| mnaser | hmm | 20:05 |
| mnaser | im a bit torn if we should let it be a configurable option if we cant support it by default | 20:05 |
| mnaser | i.e. turning on compression breaks | 20:05 |
| mnaser | i feel like we should drop that as an option if turning it on breaks, thoughts? | 20:06 |
| *** SmearedBeard has quit IRC | 20:06 | |
| cloudnull | or we have a code path that grabs qpress from percona when needed? | 20:06 |
| cloudnull | but that seems like a lot of extra code to carry | 20:06 |
| odyssey4me | if sst mechanism == mariadbbackup, then disable compression by default | 20:06 |
| mnaser | yeah it is | 20:06 |
| mnaser | i was hoping to kill adding that repo ever | 20:06 |
| cloudnull | meh, I say disable it and note that its disabled for a reason | 20:06 |
| odyssey4me | agreed | 20:06 |
| cloudnull | odyssey4me+1 | 20:06 |
| mnaser | ok, leaving an option to enable it with a note to say "you need to go add qpress, in some way" | 20:07 |
| mnaser | in the reno | 20:07 |
| odyssey4me | the only concern is the upgrade path, really - can we go from mariadb-10.1 + xtrabackup + qpress -> mariadb-10.3 + mariabackup | 20:07 |
| mnaser | i dont see why not, because qpress should already be there, so if it needs to decompress, itll be able to | 20:08 |
| mnaser | and if qpress is not there, its probably greenfield deployment and compression is disabled | 20:08 |
| mnaser | then remains xtrabackup -> mariabackup working.. that.. i don't know. upgrade jobs are working though | 20:08 |
| mnaser | so we can use that to iterate on the behaviour.. but, we need to make that jump either way | 20:09 |
| cloudnull | ++ | 20:09 |
| * cloudnull will go apply mariaback to my local environment to see how it all burns down | 20:10 | |
| odyssey4me | https://jira.mariadb.org/browse/MDEV-17600?focusedCommentId=118612&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-118612 | 20:10 |
| *** tstrul has joined #openstack-ansible | 20:10 | |
| mnaser | yeah looks like its deprecated AND the use case is WAN replication | 20:11 |
| mnaser | which reaaaally isn't our use case imho | 20:11 |
| odyssey4me | ok, so yeah - we should really be killing compression anyway | 20:11 |
| mnaser | ok, even better | 20:12 |
| odyssey4me | there's a bunch of stuff here which is good for operators, but I don't think we need to implement any of it in automation: https://mariadb.com/kb/en/library/using-encryption-and-compression-tools-with-mariabackup/ | 20:13 |
| odyssey4me | that might be better suited to a doc note in the ops docs | 20:13 |
| odyssey4me | going afk for the night - cheers all | 20:14 |
| guilhermesp | take care odyssey4me | 20:15 |
| mnaser | later odyssey4me | 20:15 |
| mnaser | dis gun be good | 20:23 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-galera_server master: Bump version and use mariabackup https://review.openstack.org/643798 | 20:23 |
| mnaser | +23, -194 | 20:23 |
| mnaser | aww yeah | 20:23 |
| cloudnull | mnaser trying your pr in the lab now | 20:25 |
| cloudnull | :) | 20:25 |
| cloudnull | worked in dev :) | 20:29 |
| NobodyCam | manually added eth0 config to my container attempting to run setup-hosts again, not sure this will work | 20:30 |
| *** tosky has quit IRC | 20:30 | |
| *** hamzaachi has quit IRC | 20:36 | |
| *** tstrul has quit IRC | 20:37 | |
| *** ansmith_ has quit IRC | 20:39 | |
| cloudnull | https://pasted.tech/pastes/aba7a0a3eff265cdeb79183a39225fcd330ff353 | 20:45 |
| *** hamzy has quit IRC | 20:46 | |
| *** luksky has joined #openstack-ansible | 20:54 | |
| mnaser | cloudnull: which: no mariabackup in (/usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin) | 21:03 |
| mnaser | i wonder what provides that | 21:03 |
| *** priteau has joined #openstack-ansible | 21:07 | |
| mnaser | hmm centos is breaking with that error | 21:07 |
| mnaser | but ubuntu is breaking with | 21:07 |
| mnaser | Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.1.0.3' --datadir '/var/lib/mysql/' --parent '5354' --binlog '/var/lib/mysql/mariadb-bin' --binlog-index '/var/lib/mysql/mariadb-bin.index': 1 (Operation not permitted) | 21:07 |
| mnaser | ah there we go | 21:09 |
| mnaser | mariadb-backup package | 21:09 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-galera_server master: Bump version and use mariabackup https://review.openstack.org/643798 | 21:10 |
| *** simon-AS559 has joined #openstack-ansible | 21:10 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-galera_server master: Bump version and use mariabackup https://review.openstack.org/643798 | 21:12 |
| mnaser | cloudnull: do you recall why we uninstall galera-server to do a rolling upgrade? | 21:13 |
| cloudnull | I do not. | 21:14 |
| mnaser | looks like we have `galera_server_upgrade_packages_remove` and as part of the pre upgrade step.. we uninstall everything there | 21:14 |
| NobodyCam | Fyi: things seem to be working after I manually created the networking for eth0 on my container. I ended up adding /etc/sysconfig/network and manually creating the network file for eth0, I also created eth0.hwaddr, eth0.ini and .sh files in /var/lib/lxc/<container_name> | 21:17 |
| NobodyCam | seems systems-networking is default, | 21:18 |
| jrosser | NobodyCam: did you use a —limit when creating he container? | 21:19 |
| NobodyCam | no | 21:19 |
| NobodyCam | my issue appears to be something with default lxc networking not working on newly created container | 21:20 |
| *** BjoernT has quit IRC | 21:21 | |
| *** BjoernT has joined #openstack-ansible | 21:23 | |
| NobodyCam | sidenote: everything seemed to work if I created a lxc container manually, (with default config) | 21:23 |
| jrosser | Well, if it’s not creating the config files for the Octavia containers then for some reason the data must be missing | 21:24 |
| jrosser | I’d be running the lxc_container_create play with -vvv next to see the data passed to the ansible modules | 21:25 |
| NobodyCam | it created the OpenStack interfaces, but those are not created until after yum update / and install systems-networking is run, but with out a default eth0 interface That was failing | 21:26 |
| NobodyCam | oh I also had to comment out the echo run cmd in opt/container-setup.sh | 21:28 |
| mnaser | woot. seems to have passed by adding the package | 21:33 |
| NobodyCam | is there a quick wy to reread ip if I run --clear-ip | 21:38 |
| *** goldenfri has joined #openstack-ansible | 21:48 | |
| *** cshen has joined #openstack-ansible | 21:49 | |
| *** cshen has quit IRC | 21:54 | |
| mnaser | jrosser, cloudnull, odyssey4me: https://review.openstack.org/#/c/643798/ as discussed | 21:56 |
| jrosser | mnaser: looks reasonable, is this “check experimental” time? | 22:03 |
| mnaser | jrosser: i guess we could do that if we need to, but that test does deploy a whole cluster | 22:03 |
| jrosser | Yeah, and I guess currently we’re broken anyway...... | 22:04 |
| cloudnull | it worked in my dev environment :) | 22:07 |
| jrosser | Yes actually we should push on | 22:08 |
| *** pcaruana has quit IRC | 22:08 | |
| jrosser | Master is blocked on the Sahara py_pkgs patch | 22:08 |
| jrosser | In turn blocked on mariadb | 22:08 |
| *** DanyC has joined #openstack-ansible | 22:24 | |
| *** ansmith_ has joined #openstack-ansible | 22:25 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server master: Bump version and use mariabackup https://review.openstack.org/643798 | 22:50 |
| *** priteau has quit IRC | 22:54 | |
| *** simon-AS559 has quit IRC | 23:00 | |
| *** marst has quit IRC | 23:00 | |
| *** luksky has quit IRC | 23:21 | |
| *** DanyC has quit IRC | 23:27 | |
| *** cshen has joined #openstack-ansible | 23:50 | |
| *** cshen has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!