*** maharg101 has joined #openstack-ansible | 00:04 | |
*** maharg101 has quit IRC | 00:08 | |
*** macz_ has quit IRC | 00:30 | |
*** dave-mccowan has quit IRC | 01:24 | |
*** jamesdenton has quit IRC | 01:33 | |
*** jamesden_ has joined #openstack-ansible | 01:34 | |
*** maharg101 has joined #openstack-ansible | 02:04 | |
*** maharg101 has quit IRC | 02:09 | |
*** cp- has quit IRC | 02:33 | |
*** cp- has joined #openstack-ansible | 02:38 | |
*** cp- has quit IRC | 02:43 | |
*** cp- has joined #openstack-ansible | 02:45 | |
*** maharg101 has joined #openstack-ansible | 04:05 | |
*** maharg101 has quit IRC | 04:10 | |
*** macz_ has joined #openstack-ansible | 04:21 | |
*** macz_ has quit IRC | 04:25 | |
*** mgariepy has quit IRC | 04:53 | |
*** mgariepy has joined #openstack-ansible | 04:53 | |
*** evrardjp has quit IRC | 05:33 | |
*** evrardjp has joined #openstack-ansible | 05:33 | |
*** yasemind has joined #openstack-ansible | 05:40 | |
yasemind | hi, good morning, when i run the openstack-ansible os-zun-install.yml command it is given an error http://paste.openstack.org/show/801807/, i checked container and server have PyMSQL library. can you help me? | 05:49 |
---|---|---|
yasemind | i think it didnt create databases in mysql automatically, but why ? | 05:54 |
yasemind | Create database for service is failed | 05:56 |
*** ahsen has joined #openstack-ansible | 06:03 | |
*** pto has joined #openstack-ansible | 06:09 | |
*** pto has joined #openstack-ansible | 06:10 | |
*** miloa has joined #openstack-ansible | 06:46 | |
*** gyee has quit IRC | 06:46 | |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: Change format of rows https://review.opendev.org/c/openstack/openstack-ansible/+/771659 | 06:57 |
*** jamesgibo has joined #openstack-ansible | 07:15 | |
*** klamath_atx has quit IRC | 07:18 | |
*** klamath_atx has joined #openstack-ansible | 07:19 | |
*** dotnetted has quit IRC | 07:19 | |
*** dotnetted has joined #openstack-ansible | 07:19 | |
*** luksky has joined #openstack-ansible | 08:03 | |
*** jamesden_ has quit IRC | 08:05 | |
*** jamesdenton has joined #openstack-ansible | 08:05 | |
*** maharg101 has joined #openstack-ansible | 08:06 | |
*** rpittau|afk is now known as rpittau | 08:11 | |
*** maharg101 has quit IRC | 08:12 | |
jrosser | yasemind: it would be really useful if you could say which branch of openstack-ansible you are deploying | 08:15 |
*** tosky has joined #openstack-ansible | 08:16 | |
noonedeadpunk | o/ | 08:17 |
jrosser | morning | 08:20 |
jrosser | looks like i need to fix openstack-ansible-tests for the new pip in order to land the other changes | 08:21 |
jrosser | the linters job runs from the tests repo but takes global-requirement-pins from the integrated repo | 08:21 |
*** andrewbonney has joined #openstack-ansible | 08:24 | |
*** maharg101 has joined #openstack-ansible | 08:47 | |
openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Use setuptools constraint from global-requirements-pins rather than u-c https://review.opendev.org/c/openstack/openstack-ansible-tests/+/771770 | 08:49 |
openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Update pip/setuptools/wheel to latest version https://review.opendev.org/c/openstack/openstack-ansible/+/770284 | 08:50 |
yasemind | jrosser we use ussuri version on our openstack system, but we use stable/victoria for zun | 09:02 |
*** ahsen has quit IRC | 09:05 | |
jrosser | yasemind: when you take a role from stable/victoria and try to use it on ussuri you will need to fix up anything which we have changed between those releases | 09:06 |
jrosser | in particular we have changed the way the db setup is done, so that is going to need some variables overriding | 09:06 |
jrosser | so for ussuri the db setup is expected to happen on the galera host https://github.com/openstack/openstack-ansible-os_zun/blob/stable/ussuri/defaults/main.yml#L106 | 09:08 |
jrosser | but we changed this in victoria to the utility host https://github.com/openstack/openstack-ansible-os_zun/blob/stable/victoria/defaults/main.yml#L116-L117 | 09:08 |
jrosser | so you will need to override the variables in the os_zun victoria defaults/main.yml to make it do the db setup on the first galera host | 09:09 |
jrosser | put these overrides in your user_variables.yml | 09:09 |
*** ahsen has joined #openstack-ansible | 09:11 | |
*** yasemind has quit IRC | 09:31 | |
*** ahsen has quit IRC | 09:31 | |
openstackgerrit | Merged openstack/openstack-ansible-haproxy_server stable/ussuri: Python3 supported version of hatop https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/768739 | 09:32 |
*** maharg102 has joined #openstack-ansible | 09:34 | |
*** Anthraxs has joined #openstack-ansible | 09:34 | |
Anthraxs | Hello, I'm having an odd issue. I'm trying to use SSL certificates on my AIO installation and the haproxy-install.yml playbook fails with The client lacks sufficient authorization error. I'm using train and I'm trying to use "distro" installation for certobt since certbot auto also trows an error that is deprecated | 09:36 |
Anthraxs | my domain name does have propper DNS and it does points to the correct IP... | 09:36 |
*** tosky has quit IRC | 09:36 | |
*** tosky_ has joined #openstack-ansible | 09:36 | |
*** maharg101 has quit IRC | 09:37 | |
dotnetted | jrosser - you were right with "smells like network trouble" heh - I had both (internal|external)_lb_vip_address on the br-mgmt interface and it was using external_lb_vip_address to communicate with xinetd on galera container - This was not whitelisted in only_from - Dropping the external IP from br-mgmt fixed it - Thanks again | 09:39 |
*** tosky_ is now known as tosky | 09:43 | |
Anthraxs | This is the error that I'm receiving if I use the default configuration (certbot-auto) | 09:44 |
Anthraxs | https://pastebin.com/dYtHNSG5 | 09:44 |
jrosser | Anthraxs: we did a huge overhaul of the letsencrypt support in later releases than train | 09:46 |
*** jamesdenton has quit IRC | 10:00 | |
*** jamesdenton has joined #openstack-ansible | 10:01 | |
*** Anthraxs has quit IRC | 10:14 | |
*** partlycloudy has quit IRC | 10:16 | |
*** yasemind has joined #openstack-ansible | 10:17 | |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible-base and OpenStack collections https://review.opendev.org/c/openstack/openstack-ansible/+/771780 | 10:19 |
*** partlycloudy has joined #openstack-ansible | 10:19 | |
*** ahsen has joined #openstack-ansible | 10:28 | |
noonedeadpunk | So, we finally get 22.0.0! | 10:33 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Release Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/771786 | 10:37 |
*** yasemind has quit IRC | 10:43 | |
jrosser | we are really so close to dropping the functional tests entirely from openstack-ansible tests | 10:56 |
openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_neutron master: Prevent neutron-l3-agent killing keepalived on restart https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/771791 | 10:56 |
*** ahsen has quit IRC | 10:56 | |
jrosser | just os_neutron being the major user of these now | 10:56 |
jrosser | would be really really good to drop the functional tests as it's a huge pain to maintain the tests repo | 10:56 |
noonedeadpunk | and keystone as well | 10:59 |
noonedeadpunk | wiht uw scenario | 10:59 |
noonedeadpunk | But I think we can move it to integrated pretty easily | 10:59 |
*** yasemind has joined #openstack-ansible | 11:04 | |
noonedeadpunk | hm wondering why all neutron functional tests failed... | 12:12 |
jrosser | something wierd is happening, look at this too https://review.opendev.org/c/openstack/openstack-ansible-tests/+/771770 | 12:12 |
jrosser | i needed to fix the 'cannot install setuptools' problem for the linters job | 12:13 |
jrosser | and somehow * is now broken | 12:13 |
noonedeadpunk | yeah, and on setup with auth failure | 12:13 |
noonedeadpunk | maybe smth is broken with keystone... | 12:14 |
jrosser | you've seen some neutron jobs doing the same? | 12:14 |
noonedeadpunk | yeah for https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/771791 | 12:14 |
noonedeadpunk | https://ab9b79f9491924810fc6-88bfc8d18e6df19b9cb6dbfc3489a6ab.ssl.cf1.rackcdn.com/771791/1/check/openstack-ansible-ovs-ubuntu-bionic/e5b3e7d/ for example | 12:14 |
jrosser | i was thinking my pip change to the tests repo was at fault but it can't be that | 12:14 |
noonedeadpunk | nah, it feels more global then that | 12:15 |
jrosser | we've just bumped all the SHA on master | 12:15 |
jrosser | i was looking in the keystone log, nothing obvious | 12:15 |
jrosser | though it's still upset about service tokens, as far as i can see they are just warnings though | 12:16 |
noonedeadpunk | I can imagine upstream project can drop this functionality now... | 12:16 |
*** jamesdenton has quit IRC | 12:18 | |
*** jamesdenton has joined #openstack-ansible | 12:19 | |
noonedeadpunk | however intagrated tests pass, so unlikely... | 12:23 |
noonedeadpunk | *integrated | 12:23 |
jrosser | so this is either some unintended consequence from the osa-new-pip changed which have already merged, or theres a bug/change come in with the SHA bump | 12:25 |
jrosser | noonedeadpunk: i am wondering about this https://github.com/openstack/tempest/commit/cd0bbbdad37a31248d479ef78df948da0a1e850e | 12:34 |
yasemind | thank you for your helping jrosser and noonedeadpunk , i can install zun and it works | 12:34 |
jrosser | yasemind: thats great news | 12:34 |
noonedeadpunk | backport does not though. Feels I missed smth | 12:35 |
noonedeadpunk | or on U service was deployed differently | 12:35 |
*** yasemind has quit IRC | 12:40 | |
*** yasemind has joined #openstack-ansible | 12:40 | |
yasemind | noonedeadpunk yeah we use ussuri openstack system, but we use victoria for zun. | 12:42 |
*** jamesgibo has quit IRC | 12:43 | |
jrosser | yasemind: did you see the settings for user-role-requirements? https://docs.openstack.org/openstack-ansible/latest/reference/configuration/extending-osa.html#extend-osa-roles | 12:44 |
andrewbonney | noonedeadpunk: are you talking about the zun backport? I've got a pending patch but waiting on kuryr backport first | 12:44 |
jrosser | you can use that to make your change to the role version of os_zun persist | 12:44 |
jrosser | yasemind: if you do a upgrade at any point, re-running bootstrap-ansible would undo anything you've done by hand to change the os_zun branch to victoria | 12:45 |
*** jamesgibo has joined #openstack-ansible | 12:45 | |
jrosser | User 621d7ce0c33c4fd3ae93421a8574926f has no access to domain default _validate_domain_scope | 12:56 |
yasemind | jrosser yes we updated user-role-requirements, we did stable/victoria | 12:58 |
*** rh-jelabarre has joined #openstack-ansible | 13:02 | |
noonedeadpunk | andrewbonney: yeah, was talking about https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/771547 | 13:03 |
andrewbonney | Ok. Once https://review.opendev.org/c/openstack/kuryr/+/771596 merges I've got a patch for the openstack-ansible repo | 13:03 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Ensure kuryr repo is available within CI images https://review.opendev.org/c/openstack/openstack-ansible/+/771608 | 13:05 |
noonedeadpunk | ah, ok, I see! | 13:05 |
noonedeadpunk | thaks! | 13:05 |
*** tosky has quit IRC | 13:06 | |
*** klamath_atx has quit IRC | 13:06 | |
*** tosky has joined #openstack-ansible | 13:06 | |
noonedeadpunk | jrosser: that's weird kind of... are we creating extra default in functional tests? | 13:16 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: [doc] Release Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/771809 | 13:18 |
openstackgerrit | Merged openstack/openstack-ansible stable/victoria: Disable repeatedly failing zun tempest test https://review.opendev.org/c/openstack/openstack-ansible/+/771548 | 13:20 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: [doc] Release Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/771809 | 13:22 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: [doc] Clenaup heading page https://review.opendev.org/c/openstack/openstack-ansible/+/771810 | 13:25 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/train: [doc] Cleanup heading page https://review.opendev.org/c/openstack/openstack-ansible/+/771810 | 13:26 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: [doc] Cleanup heading page https://review.opendev.org/c/openstack/openstack-ansible/+/771811 | 13:29 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Ensure kuryr repo is available within CI images https://review.opendev.org/c/openstack/openstack-ansible/+/771608 | 13:31 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/ussuri: Ensure kuryr repo is available within CI images https://review.opendev.org/c/openstack/openstack-ansible/+/771608 | 13:32 |
jrosser | noonedeadpunk: i was thinking something like that yes, some difference between tempest vars we set in functional vs. AIO | 13:35 |
mgariepy | jamesdenton, are you around ? | 13:43 |
openstackgerrit | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: Add configuration for zun-wsproxy service https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/769143 | 13:45 |
*** owalsh has quit IRC | 13:48 | |
*** owalsh has joined #openstack-ansible | 14:08 | |
openstackgerrit | Merged openstack/openstack-ansible master: Limit threads and processes for Senlin in AIO configuration https://review.opendev.org/c/openstack/openstack-ansible/+/771256 | 14:08 |
jamesdenton | mgariepy hello there | 14:22 |
jamesdenton | i was gone. but now i am back. | 14:23 |
openstackgerrit | Gaudenz Steinlin proposed openstack/openstack-ansible master: Use TCP mode for console if SSL is configured https://review.opendev.org/c/openstack/openstack-ansible/+/574153 | 14:24 |
*** macz_ has joined #openstack-ansible | 14:25 | |
mgariepy | jamesdenton, hey how are you doing ? | 14:27 |
mgariepy | do you have a deployment with ovs firewall? | 14:27 |
jamesdenton | no, not anymore. we are still using iptables_hybrid | 14:28 |
mgariepy | ok. | 14:28 |
jamesdenton | there was an ugly bug that was only recently fixed (maybe) | 14:29 |
mgariepy | what was it ? | 14:29 |
jamesdenton | one sec | 14:29 |
jamesdenton | https://bugs.launchpad.net/neutron/+bug/1732067 | 14:29 |
openstack | Launchpad bug 1732067 in neutron "openvswitch firewall flows cause flooding on integration bridge" [High,In progress] - Assigned to LIU Yulong (dragon889) | 14:29 |
*** macz_ has quit IRC | 14:30 | |
admin0 | Your system is not supported by certbot-auto anymore -- this is for ubuntu focal 20.04 .. | 14:35 |
admin0 | getting that on haproxy_ssl_letsencrypt_enable: true | 14:35 |
admin0 | 14:35 | |
frickler | this may be related to the keystone/tempest issues openstack/tempest master: Fix project/domain scope in dynamic_creds https://review.opendev.org/c/openstack/tempest/+/771817 | 14:38 |
mgariepy | i do have some issues with ovsf firewall it's forgetting some flows.. | 14:38 |
mgariepy | haha | 14:38 |
*** dave-mccowan has joined #openstack-ansible | 14:39 | |
jamesdenton | as in, they are dropping out? | 14:41 |
jrosser | frickler: ah awesome, thanks for the link to that, it references the tempest commit that i was suspicious about earlier | 14:41 |
mgariepy | when ovsdb and ovs-vswitchd are restarted (like when there is an upgrade of pkg) neutron readd all the flows. | 14:42 |
mgariepy | but not the remote-sec-group one. | 14:42 |
mgariepy | is there some doc if i want to build the venv for a specific verison/sha ? | 14:43 |
mgariepy | i vaguely remember there was but cannot find it :/ | 14:43 |
jamesdenton | maybe updating this? https://github.com/openstack/openstack-ansible/blob/master/ansible-role-requirements.yml | 14:44 |
jamesdenton | no | 14:44 |
*** sshnaidm|ruck is now known as sshnaidm|afk | 14:49 | |
mgariepy | jamesdenton, here is the output of the flows when ovsdb/vswitchd are restarted: http://paste.openstack.org/show/801819/ | 14:51 |
*** yasemind has quit IRC | 14:54 | |
openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Move neutron pip packages from constraints to requirements https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/770276 | 15:02 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: L3 agent cleanup_on_shutdown https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/771826 | 15:11 |
*** spatel has joined #openstack-ansible | 15:11 | |
*** fridtjof[m] has quit IRC | 15:20 | |
*** ioni has quit IRC | 15:20 | |
*** csmart has quit IRC | 15:20 | |
*** manti has quit IRC | 15:20 | |
*** irclogbot_2 has quit IRC | 15:21 | |
*** irclogbot_0 has joined #openstack-ansible | 15:21 | |
noonedeadpunk | spatel: are in correct channel?:) | 15:27 |
*** rh-jelabarre has quit IRC | 15:27 | |
spatel | damn it :( | 15:28 |
spatel | sorry | 15:28 |
spatel | I need to change my IRC client | 15:28 |
spatel | hard to know in what channel i am sitting in. | 15:29 |
noonedeadpunk | ok, pool id - what option is that? | 15:29 |
spatel | My question was how does osa generate pool id? | 15:29 |
spatel | let me show you | 15:29 |
noonedeadpunk | aha in designate.conf | 15:29 |
spatel | yes in designate.conf has pool_id = 794ccc2c-d751-44fe-b57f-8894c9f5c842 | 15:30 |
spatel | you have to use same pool_id in pools.yaml otherwise designate doesn't understand what pool you are talking abou | 15:30 |
noonedeadpunk | uh, nasty | 15:30 |
spatel | very ugly | 15:31 |
*** rh-jelabarre has joined #openstack-ansible | 15:31 | |
spatel | we can use "designate-manage pool show_config" command to find ID and use it to generate pools.yamal | 15:31 |
*** manti has joined #openstack-ansible | 15:31 | |
spatel | noonedeadpunk: this document saved my life, how to handle pool_id - https://www.runscripts.com/support/guides/tools/openstack/designate-and-external-dns | 15:32 |
noonedeadpunk | I'm pretty sure I was not doing all of that... | 15:32 |
spatel | I spent 24 hour to debug and finally found issue is related to pool_id :) | 15:32 |
noonedeadpunk | but I used zookeeper for coordination and that's why I might be ok... | 15:33 |
spatel | may be newer version has this pool_id condition. i had no issue with older version of designate | 15:33 |
noonedeadpunk | or might be that, yes | 15:33 |
spatel | i had no issue related pool id in ussuri deployment | 15:33 |
noonedeadpunk | yeah I was just doing designate-manage pool update | 15:34 |
spatel | I did that too but didn't help | 15:34 |
spatel | until unless you add id in pools.yaml it doesn't work | 15:35 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Adds handler for copying policy.json to the right place https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/744149 | 15:36 |
spatel | This is what it looks now - http://paste.openstack.org/show/801831/ | 15:36 |
spatel | with id | 15:36 |
spatel | i will try to run some test to find some workaround | 15:37 |
noonedeadpunk | uh, designate needs some love for sure | 15:37 |
jrosser | that is the default pool id isnt it | 15:38 |
noonedeadpunk | they have no mention of it in docs https://docs.openstack.org/designate/victoria/admin/pools.html | 15:38 |
spatel | in roles we have templates/designate.conf.j2:pool_id = {{ designate_pool_uuid }} | 15:38 |
noonedeadpunk | jrosser: which is hardcoded in defaults/main.yml | 15:38 |
jrosser | well, brace yourself.... https://codesearch.opendev.org/?q=794ccc2c-d751-44fe-b57f-8894c9f5c842 | 15:38 |
noonedeadpunk | o_O | 15:39 |
spatel | :) | 15:39 |
noonedeadpunk | that is ridiculous | 15:39 |
jrosser | defaults/main.yml maybe but its actually config default inside designate | 15:39 |
spatel | totally | 15:39 |
jrosser | so whats this, new behaviour in victoria | 15:40 |
spatel | i had no issue in previous version related pool_id but suddenly with victoria i hit that bug and fix was add id in pools.yaml | 15:41 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add dessignate pool uuid to secrets https://review.opendev.org/c/openstack/openstack-ansible/+/771833 | 15:42 |
noonedeadpunk | I'd expect to see that in docs.. But seems I would need to look through commits instead then | 15:43 |
spatel | all i am seeing this in commit - https://review.opendev.org/c/openstack/designate/+/751245 | 15:45 |
andrewbonney | Before I forget, we hit an issue after a ussuri upgrade with designate not consuming nova/neutron notifications | 15:47 |
*** b1tsh1ft3r has joined #openstack-ansible | 15:47 | |
andrewbonney | Might be some default wiring missing in the designate role | 15:47 |
* jrosser boggles at https://review.opendev.org/c/openstack/designate/+/751245 | 15:48 | |
noonedeadpunk | I can remember smth like that and I can recall some patches | 15:48 |
jrosser | really editing the initial db migration.....? | 15:48 |
noonedeadpunk | which is cherry-picked lol | 15:48 |
noonedeadpunk | and that means that already deployed designate won't ever be upgraded | 15:49 |
noonedeadpunk | (to this version) | 15:49 |
jrosser | right, but they will have the default id | 15:50 |
jrosser | rather than whatever might have been in the config | 15:50 |
jrosser | but even so, just looks super super odd adjusting the first migration in the series | 15:50 |
jrosser | not my area of expertise so maybe its ok | 15:51 |
noonedeadpunk | well looking at changes I think they should have preserved behaviour that was suggested from beginning aren't they? | 15:51 |
noonedeadpunk | they're just initializing config before doing migration | 15:51 |
noonedeadpunk | I faced the same at masakari not so far ago | 15:52 |
noonedeadpunk | when despite what have been set in config - it was ignored because at that step config was not read | 15:52 |
noonedeadpunk | so default value was used | 15:52 |
spatel | so this change sitting in master but not in victoria right? | 15:53 |
spatel | do you think i should also cherry pick for workaround ? | 15:54 |
noonedeadpunk | and we're doing wrong thing | 15:55 |
spatel | yes hardcoded uuid | 15:55 |
*** sshnaidm|afk is now known as sshnaidm|ruck | 15:56 | |
b1tsh1ft3r | So, i'm having some issues it seems with a recent upgrade from stein to train 20.2.1 | 15:56 |
b1tsh1ft3r | It would appear that originally, user_variables.yml had neutron_plugin_base defined | 15:56 |
b1tsh1ft3r | with only "qos" listed. I believe l3 and metering were a default included thing previously. | 15:56 |
b1tsh1ft3r | After the ugprade i've had problems with l3 agent flapping. I've gone back and since added | 15:56 |
b1tsh1ft3r | metering, firewall, l3-agent to the neutron_plugin_base and re-run the plays, however it would | 15:56 |
b1tsh1ft3r | seem that running 'openstack network agent list' now is showing dead l3 agents. | 15:56 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Generate designate_pool_uuid dynamically https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/771841 | 15:56 |
noonedeadpunk | spatel: can you chack ^ | 15:56 |
b1tsh1ft3r | I'm also not able to list routers in the horizon dashboard, or modify quotas of projects without generic errors. | 15:56 |
b1tsh1ft3r | It would appear that perhaps maybe rabbitmq traffic is lost from l3-agent to neutron server if that | 15:56 |
b1tsh1ft3r | even makes sense?! I'm not really seeing anything in the logs either. Running python 2.7 still across | 15:56 |
b1tsh1ft3r | the entire stack, but would like to upgrade. | 15:56 |
spatel | noonedeadpunk: that looks good, do you want me to run on my lab? | 15:57 |
noonedeadpunk | yeah, would be awesome, just ensure putting uuid into secrets and generating a value for it | 15:58 |
noonedeadpunk | b1tsh1ft3r: hey! Was you are the one who reported https://bugs.launchpad.net/openstack-ansible/+bug/1911482 ? | 15:58 |
openstack | Launchpad bug 1911482 in openstack-ansible "neutron-l3-agent broken after train upgrade" [Undecided,New] | 15:58 |
spatel | yes.. let me try | 15:58 |
*** macz_ has joined #openstack-ansible | 15:59 | |
*** ioni has joined #openstack-ansible | 15:59 | |
*** fridtjof[m] has joined #openstack-ansible | 15:59 | |
*** masterpe has joined #openstack-ansible | 15:59 | |
b1tsh1ft3r | noonedeadpunk co-worker of mine had filed it, but yes same problem. You were dead on about the inventory issue. | 15:59 |
*** csmart has joined #openstack-ansible | 15:59 | |
*** macz_ has quit IRC | 15:59 | |
*** macz_ has joined #openstack-ansible | 16:00 | |
noonedeadpunk | hm, l3-agent is probably wrong naming for the plugin? I think it's pretty much the list of supported options https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/defaults/main.yml#L328-L337 | 16:02 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Generate designate_pool_uuid dynamically https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/771841 | 16:07 |
noonedeadpunk | moreover fwaas is kind of obsoleted so you should consider your options if were using it | 16:08 |
b1tsh1ft3r | noonedeadpunk. Only using the following listed under neutron_plugin_base: router, metering, firewall, qos | 16:08 |
noonedeadpunk | aha, ok | 16:09 |
frickler | designate recentish changed a bug that essentially prevented starting with a different pool id, likely this is uncovering some other issues here now https://review.opendev.org/c/openstack/designate/+/751245 | 16:09 |
spatel | noonedeadpunk: do i need to add "id" in this section ? http://paste.openstack.org/show/801787/ | 16:10 |
noonedeadpunk | do you see anything useful in journalctl -u nautron-l3-agent ? | 16:10 |
frickler | also we should probably get rid of that default id being hardcoded | 16:10 |
noonedeadpunk | I think you don't | 16:10 |
spatel | cool | 16:10 |
spatel | running playbook now | 16:10 |
noonedeadpunk | frickler: yeah that's excatly why https://review.opendev.org/c/openstack/openstack-ansible/+/771833 was pushed at the first place | 16:11 |
b1tsh1ft3r | noonedeadpunk http://paste.openstack.org/show/JmzHwDhX8OcRQ2bqJqD4/ | 16:12 |
b1tsh1ft3r | neutron-server however is up | 16:12 |
jrosser | b1tsh1ft3r: have you checked the health of the rabbitmq cluster? | 16:15 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add dessignate pool uuid to secrets https://review.opendev.org/c/openstack/openstack-ansible/+/771833 | 16:17 |
spatel | noonedeadpunk: got this DB sync error, let me debug - http://paste.openstack.org/show/801836/ | 16:17 |
*** jamesgibo has quit IRC | 16:17 | |
spatel | I did drop container and database to re-build designate. | 16:19 |
spatel | I did run /openstack/venvs/designate-22.0.0.0b2.dev56/bin/designate-manage database sync on container didn't throw error but echo $? showing 1 | 16:22 |
spatel | it did created database in mysql but not sure why sync doesn't like it | 16:23 |
noonedeadpunk | andrewbonney: I guess that was the fix for designate notifications https://opendev.org/openstack/openstack-ansible-os_nova/commit/881620bd6476d008b218721b70807deca451b8c6 | 16:23 |
noonedeadpunk | uh | 16:23 |
andrewbonney | Yeah, similar for neutron | 16:23 |
*** jamesgibo has joined #openstack-ansible | 16:23 | |
*** jamesgibo has quit IRC | 16:24 | |
noonedeadpunk | spatel: and you have default_pool_id set in config? | 16:24 |
spatel | yes | 16:24 |
spatel | i used your patch | 16:24 |
spatel | user_secret.yaml i added uuid | 16:25 |
spatel | and made changes to default/main.yaml and template | 16:25 |
noonedeadpunk | I'm wondering if it just doesn't like it... what if you set it to 794ccc2c-d751-44fe-b57f-8894c9f5c842 ?:)))) | 16:25 |
spatel | noonedeadpunk: this is the error i am getting in logs - http://paste.openstack.org/show/801837/ | 16:26 |
frickler | noonedeadpunk: how do you identify the current pool id? that isn't clear to me from the upgrade reno | 16:27 |
noonedeadpunk | I think we're just defining random default one? | 16:27 |
noonedeadpunk | not sure, I made smth stupid? | 16:27 |
spatel | UUID is just uniq ID and it could be anything to make pool uniq | 16:28 |
spatel | let me revert my patch and test hold on | 16:28 |
b1tsh1ft3r | jrosser Yes, rabbitmq is up and running and functioning as expected. checking cluster_status returns all ok and up | 16:30 |
mgariepy | fun !! https://bugs.launchpad.net/neutron/+bug/1912651 | 16:31 |
openstack | Launchpad bug 1912651 in neutron "ovs flows are not readded when ovsdb-server/ovs-vswitchd are restarted." [Undecided,New] | 16:31 |
noonedeadpunk | frickler: ah, I got your qestion. Well, it was always hardcoded to the provided one. So if user has overriden it - then he does not need any futher actions or has it defined somewhere, so it should not be an issue | 16:31 |
spatel | why don't we let user define that UUID so it will be same no matter | 16:32 |
noonedeadpunk | once it's generated it will be the same? | 16:32 |
*** gyee has joined #openstack-ansible | 16:32 | |
spatel | that is true | 16:32 |
noonedeadpunk | we do that way with literally all uuids and passwords | 16:32 |
spatel | noonedeadpunk: i got same error after reverting patch | 16:33 |
noonedeadpunk | its' just that during upgrade it would be generated in case it's not defined in secrets.yml | 16:33 |
spatel | look like i need some clean up and re-deploy | 16:33 |
noonedeadpunk | you can try dropping tables from designate database | 16:33 |
noonedeadpunk | or just drop database designate; create database designate; | 16:33 |
spatel | let me try that | 16:34 |
noonedeadpunk | and re-run migration | 16:34 |
*** jamesgibo has joined #openstack-ansible | 16:34 | |
frickler | noonedeadpunk: I commented on your patch in the meantime to make it clearer, but I guess that's o.k., then | 16:35 |
noonedeadpunk | well, we probably can write some upgrade hook | 16:37 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add designate pool uuid to secrets https://review.opendev.org/c/openstack/openstack-ansible/+/771833 | 16:40 |
*** jamesgibo has quit IRC | 16:40 | |
openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Use global service variables https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/769944 | 16:45 |
jrosser | noonedeadpunk: i'm not really sure how we can work around the functional tests being broken | 16:50 |
jrosser | tempest is broken for us, so one thing i can see is we could pin this https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L61 temporarily until tempest master is working again | 16:51 |
jrosser | in the functional tests our SHA bumps do not apply, which is why they are broken but the integrated repo tests are still OK | 16:51 |
noonedeadpunk | how great that you made workaround for tempest for the new resolver... | 16:52 |
jrosser | well hmm, i guess its not totally helpful for this though | 16:52 |
noonedeadpunk | let's then set tempest_git_install_branch to 26.0.0 in test-vars.yml | 16:53 |
noonedeadpunk | oh, wait... | 16:54 |
noonedeadpunk | I think it was bumped in u-c? | 16:54 |
noonedeadpunk | Are we using them? | 16:54 |
jrosser | os_tempest installs master branch from defaults/main.yml | 16:55 |
jrosser | like tip of master, in a functional test | 16:55 |
noonedeadpunk | which won't be the case with new resolver? | 16:56 |
noonedeadpunk | or... | 16:56 |
noonedeadpunk | uh... | 16:56 |
jrosser | yeah | 16:56 |
jrosser | the resolver isnt an issue | 16:56 |
jrosser | it's broken auth behaviour in tempest either way | 16:56 |
jrosser | good idea with test-vars,i'll try that | 16:56 |
noonedeadpunk | so, what I was thinking about - if we have it in u-c what not to rely on u-c what tempest version to install? I guess we use tempest version from u-c for stable releases? | 16:57 |
jrosser | we do yes | 16:57 |
noonedeadpunk | why we don't use the same loginc for functional tests theb | 16:57 |
noonedeadpunk | *then | 16:57 |
noonedeadpunk | just grab the version that is present in u-c reither installing from source-source | 16:58 |
noonedeadpunk | or I'm just confused... | 16:58 |
jrosser | we'd have to comment this out https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L58-L61 | 16:58 |
spatel | noonedeadpunk: Your patch failing everytime on db sync (if i revert patch it pass ) | 16:58 |
noonedeadpunk | we're having so awful weather change today... In the morning it was -22C, now it's -5, in 2 hours it's going to be -2 and tomorrow in the morning +3 or +4. So it's really hard to think for me today :( | 16:59 |
spatel | let me do small experiment | 16:59 |
noonedeadpunk | spatel: well.... | 16:59 |
noonedeadpunk | at least we know that | 16:59 |
spatel | i wonder if we need UUID in - - - - formate like (794ccc2c-d751-44fe-b57f-8894c9f5c842) | 17:00 |
spatel | i used random string with your patch | 17:00 |
noonedeadpunk | jrosser: ha!@ | 17:00 |
noonedeadpunk | just try dropping this out https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-vars.yml#L360 | 17:01 |
spatel | let me try in - - formate | 17:01 |
jrosser | noonedeadpunk: oh awesome, good spot | 17:01 |
noonedeadpunk | and if you define literally 794ccc2c-d751-44fe-b57f-8894c9f5c842 in secrets.yml does migration work? | 17:01 |
spatel | trying test | 17:02 |
openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Use setuptools constraint from global-requirements-pins rather than u-c https://review.opendev.org/c/openstack/openstack-ansible-tests/+/771770 | 17:05 |
spatel | noonedeadpunk: ready? 794ccc2c-d751-44fe-b57f-8894c9f5c842 in user_secrets.yml PASS playbook | 17:06 |
noonedeadpunk | god damn it | 17:07 |
spatel | let me change some digits and see if it pass again make sure its not hardcoded somewhere | 17:07 |
*** b1tsh1ft3r has quit IRC | 17:09 | |
*** b1tsh1ft3r has joined #openstack-ansible | 17:10 | |
spatel | I changed last 2 digit of UUID and it works not let me change UUID format to reduce some (-) and see | 17:13 |
spatel | s/not/now | 17:13 |
noonedeadpunk | hm, and what has scripts/pw-token-gen.py generated for you? | 17:14 |
spatel | damn it i changed to 794ccc2c-d751-44fe-b57f-8888 and it Failed | 17:15 |
spatel | look like this string has to be in fix numbers and dashes | 17:15 |
noonedeadpunk | I think it should be just in uuid format | 17:15 |
noonedeadpunk | like any uuidgen | 17:15 |
spatel | noonedeadpunk: i didn't used scripts/pw-token-gen.py, manually put string in user_secrets.yml | 17:16 |
noonedeadpunk | and I guess we should generate it as well, but now I'm not 100% sure | 17:16 |
noonedeadpunk | aha, I see | 17:16 |
spatel | Yes we need to use uuidgen :) | 17:16 |
spatel | let me generate uuidgen and try that out | 17:16 |
noonedeadpunk | I think it should be using it https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/pw-token-gen.py#L179-L181 | 17:17 |
spatel | let me try scripts/pw-token-gen.py to see generate secret | 17:18 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add designate pool uuid to secrets https://review.opendev.org/c/openstack/openstack-ansible/+/771833 | 17:19 |
noonedeadpunk | frickler: tried to update reno and kind of upgrade hook ^ | 17:19 |
noonedeadpunk | worth testing it though.... | 17:19 |
spatel | noonedeadpunk: ./scripts/pw-token-gen.py works | 17:23 |
noonedeadpunk | ok, nice | 17:23 |
spatel | look like just need little documentation around so people understand its not password but uuidgen | 17:24 |
noonedeadpunk | we rely on secret ending actually | 17:24 |
noonedeadpunk | which we are mising generally... | 17:24 |
noonedeadpunk | btw token, key and password are also different | 17:25 |
noonedeadpunk | it's generally as much as this description https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/pw-token-gen.py#L42-L45 | 17:25 |
spatel | noonedeadpunk: everything works fine, able to create zone/record etc.. | 17:26 |
noonedeadpunk | ecept missing uuid :( | 17:26 |
spatel | ah | 17:27 |
spatel | we don't need this patch right because you already included it another patch - https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/771841 | 17:29 |
spatel | sorry may be i am wrong. we do need that | 17:29 |
*** dave-mccowan has quit IRC | 17:31 | |
spatel | noonedeadpunk: how do i apply this patch in production, what is the best way to handle this in prod so next time don't create issue during upgrade | 17:38 |
noonedeadpunk | you needto keep designate_pool_uuid in secrets.yml | 17:41 |
noonedeadpunk | I guess it's the main point. also right now you might want to apply override in designate to have default_pool_id which is commented out now | 17:42 |
*** rpittau is now known as rpittau|afk | 17:42 | |
noonedeadpunk | actually I think we need to backport 771841 except pasrt of removing of the uuis | 17:42 |
noonedeadpunk | *uuid | 17:42 |
spatel | i need to remove hardcoded UUID from roles by hand | 17:42 |
noonedeadpunk | nah, it would just be overriden | 17:43 |
spatel | what about pool_id which you have remove ? | 17:43 |
spatel | or it doesn't matter | 17:44 |
spatel | let me run test | 17:44 |
spatel | how does this file override when it has hardcoded id - /etc/ansible/roles/os_designate/templates/designate.conf.j2 | 17:45 |
spatel | default_pool_id = '794ccc2c-d751-44fe-b57f-8894c9f5c842' | 17:46 |
*** poopcat has quit IRC | 17:48 | |
*** poopcat has joined #openstack-ansible | 17:50 | |
openstackgerrit | Ebbex proposed openstack/ansible-role-systemd_service master: Make systemd.service more consistent https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/771869 | 17:52 |
*** miloa has quit IRC | 17:55 | |
*** cp- has quit IRC | 17:59 | |
*** maharg102 has quit IRC | 18:04 | |
*** cp- has joined #openstack-ansible | 18:08 | |
openstackgerrit | Marc GariƩpy proposed openstack/openstack-ansible master: Fix lib/modules path for focal release https://review.opendev.org/c/openstack/openstack-ansible/+/771870 | 18:12 |
*** b1tsh1ft3r has quit IRC | 18:13 | |
*** cp- has quit IRC | 18:14 | |
admin0 | anyone has centralized logging ? | 18:19 |
admin0 | what is being used | 18:19 |
admin0 | also what do you guys use for monitoring | 18:20 |
*** jamesgibo has joined #openstack-ansible | 18:21 | |
*** jamesgibo has quit IRC | 18:23 | |
*** jamesgibo has joined #openstack-ansible | 18:24 | |
noonedeadpunk | graylog / zabbix?:) | 18:27 |
*** cp- has joined #openstack-ansible | 18:30 | |
*** poopcat has quit IRC | 18:31 | |
*** poopcat has joined #openstack-ansible | 18:32 | |
*** jamesgibo has quit IRC | 18:32 | |
jrosser | oh no not another one https://github.com/openstack/requirements/blob/378d1f4adfbf8c067b8a802075b5805156170057/upper-constraints.txt#L122 | 18:36 |
jrosser | "horizon works as a library for horizon plugins" <- thats unfortunate | 18:38 |
*** jamesdenton has quit IRC | 18:43 | |
*** jamesdenton has joined #openstack-ansible | 18:43 | |
spatel | admin0: graylog | 18:46 |
spatel | what do you guys using to ship journactl logs to graylog? | 18:47 |
spatel | I am planning to use syslog hook which is easy to implement | 18:47 |
*** strattao has joined #openstack-ansible | 18:49 | |
*** andrewbonney has quit IRC | 18:55 | |
admin0 | i have graylog, but the journale2gelf dies ( when used from ops git) | 18:56 |
admin0 | so checking if anyone has a successful "recent" implementation | 18:57 |
admin0 | how is openstack lb set -status error dangerous ? | 19:01 |
admin0 | just wondering | 19:01 |
admin0 | people would immediately delete it after that | 19:02 |
admin0 | no one wants to see errors in their list | 19:02 |
admin0 | sorry .. wrong channel | 19:06 |
admin0 | :) | 19:06 |
kleini | spatel: http://paste.openstack.org/show/801846/ <- this is my very dynamic version of pools.yaml definition. I have PowerDNS running on infra hosts. Maybe this can be helpful | 19:07 |
spatel | kleini: nice!! | 19:09 |
*** pto_ has joined #openstack-ansible | 19:10 | |
kleini | But be careful, I have issues with that. I don't get all PowerDNS instances on the infra hosts to fetch latest zones. But this is maybe caused for my deployment by adding an infra host after PowerDNS and some zones are deployed. The IN SOA records do not contain all PowerDNS instances. | 19:13 |
*** pto has quit IRC | 19:13 | |
spatel | I have external PowerDNS | 19:13 |
kleini | Generally I configured every designate-mdns to be master for every PowerDNS instance and every PowerDNS instance is one target. So I don't know, if this setup is correct to have all cross connected. | 19:14 |
spatel | In my case i am planning to use mdns to be master and my external powerDNS will be slave | 19:15 |
kleini | I did the same | 19:19 |
kleini | But I still get new zones into only one PowerDNS instance and not in all. That is my problem. | 19:19 |
spatel | hmm i will try and let you know if hit same issue | 19:26 |
spatel | currently i have single powerDNS | 19:26 |
*** MrClayPole has quit IRC | 19:27 | |
*** gixx has quit IRC | 19:28 | |
*** MrClayPole has joined #openstack-ansible | 19:28 | |
*** gixx has joined #openstack-ansible | 19:28 | |
dotnetted | During ceph-install.yml, a ceph volume is created on the target host but task [ceph-osd : wait for all osd to be up] runs inside infra1_ceph-mon_container which has no knowledge of this volume. This causes the ceph osd stat in the container to fail. What might I be doing wrong here? Thanks | 19:32 |
dotnetted | Is the idea for the container ceph to communicate w/ ceph on the host or should the device be mounted into the container for local access? | 19:33 |
*** dave-mccowan has joined #openstack-ansible | 19:54 | |
admin0 | dotnetted, you can also have the ceph up and then just use the conf to have osa talk to ceph | 19:55 |
admin0 | this way, you can work with ceph-ansible independently of osa | 19:56 |
dotnetted | is that the more common use case? | 19:56 |
dotnetted | looks like "brosky" asked the same thing back in april but left before figuring it out heh: http://104.130.124.113/irclogs/%23openstack-ansible/%23openstack-ansible.2020-04-24.log.html | 19:57 |
dotnetted | Except for the fact that I have "devices:" defined and the ceph-volume is being properly created on the target host | 19:58 |
*** spatel has quit IRC | 20:00 | |
*** jamesdenton has quit IRC | 20:06 | |
*** jamesdenton has joined #openstack-ansible | 20:06 | |
*** irclogbot_0 has quit IRC | 20:14 | |
*** tosky has quit IRC | 20:14 | |
admin0 | dotnetted, that would be a recommended use case .. this way, you will be able to use the best of both | 20:14 |
dotnetted | thanks for the info :) | 20:15 |
*** tosky has joined #openstack-ansible | 20:15 | |
*** irclogbot_2 has joined #openstack-ansible | 20:16 | |
mgariepy | restart mysql(all) ? | 20:41 |
mgariepy | :/ | 20:41 |
*** spatel has joined #openstack-ansible | 20:43 | |
mgariepy | https://github.com/openstack/openstack-ansible-galera_server/blob/master/handlers/main.yml#L73-L90 | 20:44 |
spatel | kleini: hey do you have multiple Slave DNS in options: ? | 20:55 |
spatel | I have more than 2 slave PowerDNS then i need to notify both of them right and only option to do that is define multiple DNS in options: | 20:56 |
spatel | ah! i found also_notifies: | 20:58 |
spatel | that is what we need | 20:58 |
jrosser | mgariepy: https://github.com/openstack/openstack-ansible/blob/master/playbooks/galera-install.yml#L44 ? | 21:13 |
*** owalsh has quit IRC | 21:26 | |
*** dotnetted has quit IRC | 21:44 | |
*** owalsh has joined #openstack-ansible | 21:44 | |
*** owalsh has quit IRC | 22:16 | |
admin0 | I have this command: neutron subnet-create --allocation-pool start=xxxx:yyyy:8011:0000:0000:0000:0000:1111,end=xxxx:yyyy:8011:ffff:ffff:ffff:ffff:ffff --ip-version 6 --no-gateway --host-route destination=::/0,nexthop=xxxx:yyyy:8011::1 --enable-dhcp --name subnet6-direct --dns-nameserver 2001:4860:4860::8888 UUID xxxx:yyyy:8011::/48 --- .. when i login to the vm, the gateway is not set to xxxx:yyyy:8011::1 but something else ( not even in | 22:20 |
admin0 | this range) .. what am I doing wrong ? | 22:20 |
spatel | kleini: i figured out how to send notification to multiple slave powerDNS | 22:21 |
spatel | admin0: your command looks good so that is strange if it set gateway something else. make sure you don't have any other DHCP running on same VLAN or SLAAC may be causing this issue | 22:24 |
*** owalsh has joined #openstack-ansible | 22:25 | |
admin0 | hmm.. did not thought of another dhcp server on that vlan .. | 22:27 |
admin0 | will check | 22:28 |
*** spatel has quit IRC | 22:37 | |
*** d34dh0r53 has quit IRC | 22:37 | |
*** d34dh0r53 has joined #openstack-ansible | 22:48 | |
*** maharg101 has joined #openstack-ansible | 23:26 | |
*** maharg101 has quit IRC | 23:31 | |
*** strattao has quit IRC | 23:38 | |
*** luksky has quit IRC | 23:51 | |
*** lemko has quit IRC | 23:51 | |
*** lemko6 has joined #openstack-ansible | 23:51 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!