| *** tosky has quit IRC | 00:02 | |
| *** gregwork has quit IRC | 01:05 | |
| *** hwoarang_ has quit IRC | 02:22 | |
| *** cshen has joined #openstack-ansible | 02:49 | |
| *** cshen has quit IRC | 02:54 | |
| *** cshen has joined #openstack-ansible | 03:25 | |
| *** cshen has quit IRC | 03:29 | |
| *** cshen has joined #openstack-ansible | 03:40 | |
| *** cshen has quit IRC | 03:46 | |
| *** mbuil has quit IRC | 04:08 | |
| *** stingrayza has quit IRC | 04:08 | |
| *** openstackgerrit has quit IRC | 04:08 | |
| *** goldenfri has quit IRC | 04:08 | |
| *** dmsimard has quit IRC | 04:08 | |
| *** Jeffrey4l has quit IRC | 04:08 | |
| *** cosmicsound has quit IRC | 04:08 | |
| *** tbarron has quit IRC | 04:08 | |
| *** django has quit IRC | 04:08 | |
| *** Nick_A has quit IRC | 04:08 | |
| *** nsmeds has quit IRC | 04:08 | |
| *** aspiers has quit IRC | 04:08 | |
| *** szaher has quit IRC | 04:08 | |
| *** waxfire has quit IRC | 04:08 | |
| *** jmccrory has quit IRC | 04:08 | |
| *** openstackstatus has quit IRC | 04:11 | |
| *** openstackstatus has joined #openstack-ansible | 04:12 | |
| *** ChanServ sets mode: +v openstackstatus | 04:12 | |
| *** hwoarang has joined #openstack-ansible | 04:20 | |
| *** dave-mccowan has quit IRC | 04:23 | |
| *** goldyfruit has joined #openstack-ansible | 04:27 | |
| *** goldyfruit has quit IRC | 04:58 | |
| *** mensis has quit IRC | 05:04 | |
| *** rgogunskiy has joined #openstack-ansible | 05:12 | |
| *** rgogunskiy has quit IRC | 05:17 | |
| *** raukadah is now known as chkumar|rover | 05:22 | |
| *** gokhani has joined #openstack-ansible | 05:39 | |
| *** elenalindq has joined #openstack-ansible | 06:07 | |
| *** rgogunskiy has joined #openstack-ansible | 07:03 | |
| *** pcaruana has joined #openstack-ansible | 07:17 | |
| *** cshen has joined #openstack-ansible | 07:54 | |
| *** cp- has quit IRC | 08:00 | |
| *** tosky has joined #openstack-ansible | 08:02 | |
| *** cshen has quit IRC | 08:06 | |
| *** cshen has joined #openstack-ansible | 08:09 | |
| *** cp- has joined #openstack-ansible | 08:15 | |
| tacco | ran into this issue https://bugs.launchpad.net/openstack-ansible/+bug/1807346 | 08:30 |
|---|---|---|
| openstack | Launchpad bug 1807346 in openstack-ansible "[heat] Installations fails during Update Q->R" [Undecided,New] - Assigned to Guilherme Steinmuller Pimentel (guilhermesp) | 08:30 |
| tacco | is there alreay a workaround for this? | 08:30 |
| tacco | or am i doing something wrong? | 08:31 |
| cshen | tacco: according to the bug ticket, do you have multi domain? | 08:32 |
| *** rpittau|afk is now known as rpittau | 08:34 | |
| tacco | yes | 08:34 |
| tacco | default and ldap for user auth | 08:34 |
| tacco | but default is still untouched | 08:34 |
| cshen | since the ticket is not closed, it seems that no workaround yet. | 08:35 |
| *** ivve has joined #openstack-ansible | 08:35 | |
| tacco | was working when i initialy deployed without multidomain and added multidomain later. | 08:35 |
| cshen | we only have 1 domain, so I can't judge too much. | 08:35 |
| tacco | ok. Thanks so we have to deploy initialy without multidomain and then one more shot wiht multidomain | 08:35 |
| cshen | probably single domain works, multi domain not. | 08:36 |
| masterpe | I disabled the task by commenting it out | 08:37 |
| tacco | masterpe: thanks.. will see what i can do and report back what my solution was.. because i don't want to hacky fix/patch the ansible roles.. | 08:38 |
| *** sshnaidm|off is now known as sshnaidm | 08:41 | |
| *** DanyC has joined #openstack-ansible | 08:46 | |
| noonedeadpunk | mornings | 08:48 |
| *** DanyC has quit IRC | 08:48 | |
| *** DanyC has joined #openstack-ansible | 08:48 | |
| noonedeadpunk | tacco: so for some reason bug is assigned to guilhermesp - maybe he will provide you with bug status... | 08:49 |
| *** DanyC has quit IRC | 08:51 | |
| *** shyamb has joined #openstack-ansible | 08:59 | |
| *** DanyC has joined #openstack-ansible | 09:02 | |
| tacco | thanks.. i've tryed "heat_service_in_ldap=True" to skip the jobs.. | 09:16 |
| tacco | works.. | 09:16 |
| tacco | but my question is now.. what user is required in ldap.. or what heat expects from ldap | 09:16 |
| tacco | will figure out.. :D | 09:17 |
| *** mugsie has quit IRC | 09:19 | |
| tacco | guilhermesp: *dodge* :D | 09:19 |
| *** mugsie has joined #openstack-ansible | 09:21 | |
| *** yolanda has quit IRC | 09:28 | |
| *** yolanda__ has joined #openstack-ansible | 09:28 | |
| *** cosmicsound has joined #openstack-ansible | 09:30 | |
| jrosser | tacco: the only way to fix that is to patch your ansible installation (or use a forked version of ansible). You need the two patches mentioned in the bug report https://github.com/ansible/ansible/pull/42899/files and https://github.com/ansible/ansible/pull/42913/files | 09:31 |
| jrosser | unfortunately it is very difficult to get fixes merged into ansible for versions which are as "old" as the ones we use in Q & R | 09:32 |
| tacco | i see so user-domain was missing on the os_user module right? | 09:43 |
| cshen | and also openstack.py | 09:44 |
| tacco | i see. will patch this. :D | 09:44 |
| tacco | and os_user_role.py m( | 09:44 |
| tacco | anyway thanks jrosser for pointing me there. | 09:44 |
| jrosser | it is horrible sadly | 09:45 |
| jrosser | tacco: depending on how you want to do it, there are instructions here about how to have a fork of ansible https://docs.openstack.org/openstack-ansible/latest/user/source-overrides/index.html | 09:46 |
| jrosser | if you patch it in place there is a risk that doing boostrap-ansible.sh will undo your changes | 09:46 |
| jrosser | noonedeadpunk: there is something wierd going on with horizon | 09:47 |
| noonedeadpunk | what do you mean? | 09:48 |
| jrosser | see what pymysql version it wants https://8b7bc396c22f90223dfa-906a8885044ac3b62edc724702c6de01.ssl.cf1.rackcdn.com/699123/1/check/openstack-ansible-deploy-aio_lxc-debian-buster/004e481/logs/ara-report/index.html | 09:48 |
| jrosser | vs this https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L491 | 09:49 |
| noonedeadpunk | looks like they've upgraded django... | 09:50 |
| *** jbadiapa has joined #openstack-ansible | 09:52 | |
| *** jftalta has joined #openstack-ansible | 09:55 | |
| *** Jeffrey4l has joined #openstack-ansible | 10:05 | |
| *** django has joined #openstack-ansible | 10:05 | |
| *** jmccrory has joined #openstack-ansible | 10:05 | |
| *** mbuil has joined #openstack-ansible | 10:06 | |
| *** stingrayza has joined #openstack-ansible | 10:07 | |
| *** szaher has joined #openstack-ansible | 10:07 | |
| *** irclogbot_1 has quit IRC | 10:08 | |
| *** irclogbot_1 has joined #openstack-ansible | 10:09 | |
| *** miloa has joined #openstack-ansible | 10:20 | |
| *** jcosmao has joined #openstack-ansible | 10:25 | |
| noonedeadpunk | jrosser: ok, so I see the severl outcomes with horizon | 10:25 |
| noonedeadpunk | First one is to drop mysql dependency. It's used only as a session storage, so we can probably switch just to mamcached or cookies https://docs.openstack.org/horizon/latest/admin/sessions.html | 10:25 |
| noonedeadpunk | The other one is to update u-c and motivate it by requirement of fresh django...... | 10:26 |
| jrosser | i do wonder if horizon-manage meddles directly in the database? | 10:27 |
| noonedeadpunk | actually we are the ones who distribute horizon-manage.... https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/templates/horizon-manage.py.j2 | 10:27 |
| noonedeadpunk | so yeah... | 10:28 |
| *** DanyC has quit IRC | 10:28 | |
| noonedeadpunk | Horizon's original manage py just don't use mysqlclient https://opendev.org/openstack/horizon/src/branch/master/manage.py | 10:28 |
| jrosser | oh! | 10:29 |
| *** DanyC has joined #openstack-ansible | 10:30 | |
| noonedeadpunk | So if we move to cookies or memcached backends we can drop out solid part of the role | 10:31 |
| *** aspiers has joined #openstack-ansible | 10:33 | |
| *** cshen has quit IRC | 10:33 | |
| noonedeadpunk | downside of memcahed that it eventually will logout user if memcahced server goes down. Downsides of cookies are described in docs... So currenty our deploy is pretty balanced... | 10:33 |
| *** shyamb has quit IRC | 10:34 | |
| noonedeadpunk | oh.... | 10:38 |
| noonedeadpunk | jrosser: I think we can try installing https://pypi.org/project/mysqlclient/ | 10:40 |
| noonedeadpunk | instead of PyMySQL | 10:40 |
| noonedeadpunk | but for distro install it looks like it has no packages... | 10:42 |
| tacco | jrosser: patch was already upstram in my ansible.. so didn't worked for me | 10:43 |
| noonedeadpunk | and it's missing in u-c | 10:43 |
| *** jftalta has quit IRC | 11:06 | |
| *** shyamb has joined #openstack-ansible | 11:10 | |
| *** rpittau is now known as rpittau|bbl | 11:22 | |
| *** tbarron has joined #openstack-ansible | 11:41 | |
| jrosser | noonedeadpunk: will memcached be an issue if we move to loadbalancer for that? i.e session will be stored only in 1 memcached out of N | 12:00 |
| noonedeadpunk | I think this will be an issue anyway, as afaik it's stored on 1 instance now as well | 12:02 |
| noonedeadpunk | and I think we won't be changing default behaviour due to mnaser comment | 12:03 |
| noonedeadpunk | but I'll document an option how to use haproxy | 12:03 |
| noonedeadpunk | ofc we can try and adopt mysqlclient package isntead | 12:04 |
| *** cshen has joined #openstack-ansible | 12:08 | |
| *** dmsimard has joined #openstack-ansible | 12:12 | |
| *** ansmith has quit IRC | 12:17 | |
| *** ivve has quit IRC | 12:22 | |
| *** pcaruana has quit IRC | 12:27 | |
| *** DanyC has quit IRC | 12:30 | |
| *** ivve has joined #openstack-ansible | 12:30 | |
| *** shyamb has quit IRC | 12:30 | |
| *** pcaruana has joined #openstack-ansible | 12:33 | |
| *** cshen has quit IRC | 12:35 | |
| *** cshen has joined #openstack-ansible | 12:47 | |
| *** DanyC has joined #openstack-ansible | 12:51 | |
| *** goldyfruit has joined #openstack-ansible | 13:09 | |
| *** rpittau|bbl is now known as rpittau | 13:09 | |
| *** rgogunskiy has quit IRC | 13:09 | |
| *** ansmith has joined #openstack-ansible | 13:25 | |
| *** goldyfruit has quit IRC | 13:37 | |
| *** dave-mccowan has joined #openstack-ansible | 14:09 | |
| guilhermesp | tacco: noonedeadpunk unfortunately i havent had a chance to look at it | 14:20 |
| *** cshen_ has joined #openstack-ansible | 14:20 | |
| *** cshen has quit IRC | 14:23 | |
| guilhermesp | thanks jrosser to point out the fixes that are present in the bug report | 14:28 |
| jrosser | i really am not sure there is anything we can do - unless those bugfixes are now merged to ansible and we bump the version? | 14:43 |
| jrosser | that would be easy | 14:44 |
| *** rgogunskiy has joined #openstack-ansible | 14:44 | |
| *** rgogunskiy has quit IRC | 14:49 | |
| *** chkumar|rover is now known as ignoreirc | 14:57 | |
| *** ignoreirc is now known as chkumar|rover | 14:58 | |
| *** KeithMnemonic has joined #openstack-ansible | 15:02 | |
| *** schwicht has quit IRC | 15:06 | |
| cjloader | hi | 15:07 |
| *** schwicht has joined #openstack-ansible | 15:07 | |
| cjloader | looks like we missed release dealine? | 15:07 |
| *** cshen_ has quit IRC | 15:15 | |
| noonedeadpunk | yeah, we did | 15:20 |
| *** jgwentworth is now known as melwitt | 15:21 | |
| *** ivve has quit IRC | 15:21 | |
| *** goldyfruit has joined #openstack-ansible | 15:23 | |
| guilhermesp | jrosser: they seemed to be merged? | 15:32 |
| jrosser | hmm yes, but if they are in ansible we use for R though? | 15:34 |
| jrosser | i think it's really an issue of needing those patches in this version https://github.com/openstack/openstack-ansible/blob/stable/rocky/scripts/bootstrap-ansible.sh#L26 | 15:35 |
| cjloader | are we not going to release this cycle now? | 15:38 |
| tacco | guilhermesp: noonedeadpunk my workaround for now was to comment line 145,146 and 147 from os_heat/task/heat_service_setup.yml | 15:40 |
| *** goldyfruit has quit IRC | 15:45 | |
| tacco | but i guess this means all ldap users are not able to use heat.. if i got it right. | 15:51 |
| jrosser | tacco: is it right you think those patches are merged into ansible 2.5.15? | 16:08 |
| *** chkumar|rover is now known as raukadah | 16:14 | |
| *** goldyfruit has joined #openstack-ansible | 16:20 | |
| *** gregwork has joined #openstack-ansible | 16:21 | |
| cjloader | are we still going to be able to release? noonedeadpunk jrosser | 16:24 |
| *** gyee has joined #openstack-ansible | 16:28 | |
| *** miloa has quit IRC | 16:37 | |
| jrosser | cjloader: the patch is in the releases repo but needs +W https://review.opendev.org/#/c/698730/ | 16:46 |
| cjloader | ah | 16:46 |
| jrosser | thats for RC4 and then we need someone to merge the actual release patch | 16:47 |
| * jrosser asks in #openstack-release | 16:48 | |
| jrosser | bingo, there it goes | 16:49 |
| cjloader | woohoo | 16:50 |
| cjloader | jrosser: we both rechecked | 17:03 |
| cjloader | lol | 17:03 |
| *** udesale has joined #openstack-ansible | 17:14 | |
| *** goldyfruit has quit IRC | 17:17 | |
| noonedeadpunk | cores, we're about to release train, kindly put your votes https://review.opendev.org/#/c/698736/ | 17:33 |
| noonedeadpunk | cjloader: jrosser thanks for re-checking it:) | 17:33 |
| admin0 | are there plans to set a release as LTS and keep up from upgrading for say 2 years or something | 17:35 |
| noonedeadpunk | admin0: I think it's worth asking tc... | 17:42 |
| noonedeadpunk | We actually just following global releasing cycles | 17:42 |
| jrosser | isnt that 2 year cycle more of a $GIANT-VENDOR approach? | 17:43 |
| *** goldyfruit has joined #openstack-ansible | 17:43 | |
| jrosser | ^ substitute hat color to suit | 17:43 |
| admin0 | my rocky to stein is failing .. so asking :D | 17:43 |
| admin0 | if it works, i will shut up again :) | 17:43 |
| jrosser | admin0: well it should work - is something specific broken? | 17:46 |
| *** goldyfruit has quit IRC | 17:58 | |
| *** DanyC has quit IRC | 17:58 | |
| *** rgogunskiy has joined #openstack-ansible | 18:00 | |
| *** rgogunskiy has quit IRC | 18:05 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:26 | |
| *** jcosmao has left #openstack-ansible | 18:28 | |
| admin0 | jrosser, this one .. galera-cluster-rolling-restart.yml fails with Wait for mariadb port 3306 to be available | 18:52 |
| admin0 | am i supposed to manually restart galera, or what is not clear | 18:53 |
| jrosser | I guess if 3306 is to be available then galera needs to be stopped, and isn’t? | 18:53 |
| admin0 | am I to manually stop galera in that container ? | 18:54 |
| admin0 | because here, the script just waits and waits and does not proceed | 18:54 |
| jrosser | the upgrade script? | 18:54 |
| *** rpittau is now known as rpittau|afk | 18:55 | |
| jrosser | fwiw I don’t use the upgrade shell script, always step through he manual steps in the documentation | 18:56 |
| jrosser | and check very carefully what’s happening particularly with galera and rabbitmq | 18:56 |
| jrosser | personally I would run whichever galera playbook it is as the docs describe, and if it fails try to debug and look at why the service hasn’t stopped, if that’s what a previous task tried to do | 18:58 |
| admin0 | jrosser, it is in the upgrading manually section | 18:58 |
| admin0 | 2nd last step | 18:58 |
| admin0 | fatal: [c3v_galera_container-300600e4 -> 172.29.239.164]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 172.29.237.49:3306"} == is how it ends | 18:59 |
| jrosser | right I read the code - just on my phone so this is awkward | 19:02 |
| jrosser | but it’s looking for 3306 to be up and running and accepting connections | 19:02 |
| jrosser | so the answer will be in the galera log in that container about why it’s not restarted | 19:02 |
| admin0 | [c3v_galera_container-300600e4 -> 172.29.239.164 == this 164 is my utility container | 19:25 |
| admin0 | strange | 19:25 |
| admin0 | in my inventory management, it does not list galeray in that IP | 19:25 |
| admin0 | in /etc/hosts of deploy also, its listed as utility | 19:26 |
| jrosser | yes the ansible delegates the task there | 19:28 |
| admin0 | oh | 19:28 |
| jrosser | see it fails to connect to 172.29.237.49 | 19:29 |
| jrosser | the utility container is guaranteed to have tcp connectivity to port 3306 of the galera host | 19:30 |
| jrosser | that’s not necessarily the case for the deployment host, hence the delegate | 19:30 |
| admin0 | ok | 19:30 |
| admin0 | looks like mariadb crashed in this node | 19:31 |
| *** gshippey has quit IRC | 19:31 | |
| admin0 | https://pastebin.com/sediTxNP | 19:31 |
| mnaser | jrosser: have you had a single successful rotating restart of rabbitmq .. ever? | 19:32 |
| admin0 | it failed in that playbook with always this error .. so i think nope :D | 19:32 |
| mnaser | no matter what i do, if we ever restart any rabbitmq, it just goes back up but the whole thing is messed up and messages dont actually clear through | 19:33 |
| admin0 | i have mariadb running on 2 serves, but failed on this one | 19:33 |
| admin0 | https://pastebin.com/Wq1e9amS -- looks like the last mariadb is also stuck on something | 19:34 |
| jrosser | mnaser: rabbitmq has been a total car crash every upgrade we have done | 19:38 |
| jrosser | like massive outage | 19:38 |
| mnaser | jrosser: EVER if we manaully restart it, nothing | 19:39 |
| mnaser | i upgraded to latest which is 3.8.2 | 19:39 |
| mnaser | nothing, still stuck, the only resolution, kill all vhosts, recreate them, delete all queues, restart all services | 19:39 |
| mnaser | as we're using k8s for some control plane components now, im very keen on running a one-replica-non-clustered rabbitmq with no persistence behind it | 19:39 |
| mnaser | so that a simple restart gets us a clean state which should eliminate all that stuff | 19:40 |
| mnaser | i even tried resetting the WHOLE entire cluster and rebuilding it from scratch. still not sending messages. | 19:40 |
| jrosser | I think OSA has wrong defaults somewhere - still notifications piling up I think | 19:40 |
| *** rgogunskiy has joined #openstack-ansible | 19:42 | |
| jrosser | mnaser: from memory on our S>T upgrade the whole rabbit cluster went nuts 100% on all cores | 19:44 |
| mnaser | jrosser: no the notifications thing is fixed | 19:44 |
| mnaser | the piling up of messages i sbecause when you restart agents | 19:44 |
| mnaser | their existing queues are dropped and they grab new ones for fanouts/etc | 19:44 |
| mnaser | so their old ones which still are part of the exchange eventually time out after 30 minutes or so | 19:45 |
| *** jbadiapa has quit IRC | 19:46 | |
| jrosser | basically we don’t have good enough monitoring of the rabbit queues and that let to a horrid upgrade time | 19:47 |
| mnaser | jrosser: if you restart a single rabbitmq node right now | 19:47 |
| mnaser | i guarantee you will be having a bad day | 19:48 |
| mnaser | and we're not the only ones hitting it | 19:48 |
| mnaser | some folks from blizzard are too | 19:48 |
| *** udesale has quit IRC | 19:52 | |
| jrosser | mnaser: i've also seen the inverse where services get wedged up and are broken wne you restart rabbitmq | 19:54 |
| jrosser | but they look like they are running ok | 19:54 |
| jrosser | i think keystone did that to us on the S>T | 19:54 |
| *** rgogunskiy has quit IRC | 19:55 | |
| *** gyee has quit IRC | 19:59 | |
| mnaser | jrosser: yep exactly that :\ | 20:07 |
| admin0 | https://pastebin.com/Wq1e9amS -- looks like the last mariadb is also stuck on something | 20:07 |
| admin0 | looks like one of my galera db won't start no matter what .. is it safe to lxc-destroy it and re-create it again ? | 20:10 |
| admin0 | during that last part of the upgrade process | 20:10 |
| *** rgogunskiy has joined #openstack-ansible | 20:17 | |
| *** gregwork has quit IRC | 20:34 | |
| *** rgogunskiy has quit IRC | 20:36 | |
| *** gyee has joined #openstack-ansible | 20:40 | |
| *** ansmith has quit IRC | 21:30 | |
| *** elenalindq has quit IRC | 21:34 | |
| *** cyberik has joined #openstack-ansible | 21:38 | |
| *** rgogunskiy has joined #openstack-ansible | 21:38 | |
| *** goldyfruit has joined #openstack-ansible | 21:44 | |
| *** rgogunskiy has quit IRC | 21:47 | |
| *** rgogunskiy has joined #openstack-ansible | 21:47 | |
| *** pcaruana has quit IRC | 21:55 | |
| *** ansmith has joined #openstack-ansible | 22:16 | |
| *** rgogunskiy has quit IRC | 22:35 | |
| *** openstackgerrit has joined #openstack-ansible | 23:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron stable/queens: l3 agent on network_hosts do dvr_snat, anywhere else dvr https://review.opendev.org/698878 | 23:18 |
| *** cosmicsound has left #openstack-ansible | 23:20 | |
| *** tosky has quit IRC | 23:43 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!