16:00:17 <tiffanie> #startmeeting openstack_ansible_meeting
16:00:20 <openstack> Meeting started Tue Jul 30 16:00:17 2019 UTC and is due to finish in 60 minutes.  The chair is tiffanie. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:24 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
16:00:39 <evrardjp> o/
16:00:45 <noonedeadpunk> o/
16:00:46 <guilhermesp> o/
16:00:55 <mnaser> hiya
16:01:03 <tiffanie> #topic office hours
16:01:05 <jrosser> o/
16:01:30 <evrardjp> nice seeing someone stepping up to do the meetings :)
16:02:10 <mnaser> logan-: are you around by any chance?
16:02:25 <mnaser> i wanted to discuss https://review.opendev.org/#/c/672835/
16:02:29 <mnaser> perhaps other people can chime in too
16:02:56 <mnaser> it didn't work by default, or it seems like it sets the fact on localhost and not on the host that it's runnign it on
16:02:57 <guilhermesp> ah yep, that's an important one
16:04:43 <mnaser> anyone is welcome to chime in because its more of an ansible-ity
16:05:24 <jrosser> mnaser: so is this not working? https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-playbooks/nova.yml#L145
16:05:38 <mnaser> oh i didn't notice that
16:05:48 <mnaser> HMM
16:06:13 <mnaser> should it be like
16:06:15 <mnaser> vars: ..
16:06:47 <noonedeadpunk> yeah, maybe it's the reason why it doesn't work...
16:07:21 <guilhermesp> mnaser: I can test it locally now bth
16:07:27 <guilhermesp> btw*
16:07:29 <mnaser> let me see
16:07:54 <mnaser> ill test it too
16:08:06 <jrosser> i think it's the multiline equivalent to - { role: foo_app_instance, dir: '/opt/a',  app_port: 5000 } ?
16:08:35 <noonedeadpunk> I think yes
16:08:51 <evrardjp> json is yaml
16:09:01 <evrardjp> what are we talking about? :)
16:09:21 <jrosser> mnaser: where was this coming up as undefined?
16:09:31 <mnaser> jrosser: checking now what the role is being ran with
16:10:21 <mnaser> ok rerunning
16:10:36 <mnaser> with a debug inside the role
16:10:47 <mnaser> mm
16:10:49 <mnaser> it did evaluate to true
16:10:56 * mnaser is confused now
16:11:30 <mnaser> https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/main.yml#L227-L239
16:11:34 <mnaser> this seems like it should incldue it
16:11:37 <mnaser> ok let me run it again..
16:12:20 <jrosser> mnaser: i wonder if things are run with a --limit that does not include localhost, maybe this is skipped? https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/common-playbooks/nova.yml#L35
16:12:51 <evrardjp> yeah limit could impact this
16:12:59 <mnaser> oh man in this case im running
16:13:02 <mnaser> -l nova_computes
16:13:07 <evrardjp> I think it's documented
16:13:31 <mnaser> this is quite annoyin boo
16:13:43 <jrosser> assert that locahost is in ansible_play_hosts?
16:13:53 <mnaser> hmm you know what
16:14:01 <mnaser> i wonder if the fact we serialize this into small bunches has to do with it
16:14:03 <mnaser> im not sure
16:14:10 * mnaser is currently rerunning playbook to verify
16:14:19 <evrardjp> if not we have some part of docs adding localhost: https://docs.openstack.org/openstack-ansible/latest/admin/openstack-operations.html#updating-the-node-with-the-new-configuration
16:14:19 <mnaser> a debug inside the role def showed that it's true..
16:14:30 <mnaser> this is a new deployment thing tho
16:15:43 <evrardjp> definitely wrongly documented here: https://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks.html#running-ad-hoc-ansible-plays
16:15:45 <evrardjp> still digging
16:17:20 <mnaser> im running it now and seeing if it runs or gets skipped
16:18:39 <mnaser> dafuq
16:18:41 <mnaser> it just ran
16:18:49 * mnaser hmms
16:19:37 <guilhermesp> how are running now mnaser ? just nova-install playbook?
16:19:45 <mnaser> yes -l nova_computes
16:19:49 <mnaser> or whatever the hos tgroup is
16:20:11 <evrardjp> so is there something wrong?
16:20:21 <logan-> o/ mnaser
16:20:35 <evrardjp> (except our docs is not explaining the use of limit)
16:20:51 <mnaser> well yeah, i get an exception in nova when i try to create a vol
16:21:44 <guilhermesp> yeah the scenario is like cinder with rbd and nova with local storage. So right now we get an exception when try to create an instance based on volume
16:21:52 * mnaser looks for it
16:22:38 <mnaser> http://paste.openstack.org/show/755123/
16:22:52 <mnaser> i guess thats an openstack bug
16:23:33 <logan-> thats related to the rbd_inuse fact? whats causing that exception? something thats getting put into nova.conf?
16:24:51 <guilhermesp> so should [cinder] session in nova conf points to rbd configs?
16:25:07 <mnaser> logan-: no, i dont think so, i think it's actually an openstack thing (or we're missing a config or something)
16:25:14 <logan-> oh gothca
16:25:15 <mnaser> this is a local storage + cinder-vol with ceph scenario
16:25:21 <logan-> yeah
16:25:36 <logan-> so the issue with that fact was limit being used w/o including localhost right?
16:25:37 <guilhermesp> maybe missing configs here? https://github.com/openstack/openstack-ansible-os_nova/blob/cb53967f85fca48e457df4062358675bff16c256/templates/nova.conf.j2#L90
16:25:41 * guilhermesp looks openstack docs
16:25:47 <mnaser> nope, it didnt have to do with the localhost fact
16:25:48 <logan-> (ie the fact is not bugged if im understanding correctly)
16:25:49 <mnaser> it was just fine
16:25:51 <logan-> ok cool
16:25:54 <mnaser> yes, fact is cool, my bad
16:25:57 <logan-> np
16:26:07 <mnaser> behaviour still not working but thats another thing
16:27:16 <mnaser> wait hm
16:27:28 <mnaser> i wonder if its rbd_secret_uuid missing
16:27:44 <evrardjp> ok thanks for clarification mnaser
16:27:58 <guilhermesp> https://www.irccloud.com/pastebin/Zslco2UF/
16:28:03 <guilhermesp> https://docs.openstack.org/nova/latest/configuration/sample-config.html
16:28:11 <guilhermesp> mnaser that's what you mean?
16:28:42 <guilhermesp> in [libvirt] sessin
16:28:52 <noonedeadpunk> so it's needed despite usage of local storage for nova
16:28:55 <mnaser> https://github.com/openstack/openstack-ansible-os_nova/blob/cb53967f85fca48e457df4062358675bff16c256/templates/nova.conf.j2#L253-L260
16:29:03 <mnaser> i think what we need to include is rbd_user and rbd_secret_uuid
16:29:22 <guilhermesp> probably
16:29:22 <mnaser> if there is any ceph at all
16:29:29 <mnaser> and then if a pool is defined, we add image type
16:29:32 <mnaser> let me push up a patch
16:29:49 <guilhermesp> cool, i can test right away
16:30:04 <jrosser> isnt that in the cinder example
16:30:10 <jrosser> i have that in my user_variables
16:30:40 <jrosser> unless its the same thing used somewhere else
16:31:35 <noonedeadpunk> So nova should have rbd_secret_uuid defined for cinder-volumes created in ceph
16:31:58 <noonedeadpunk> not only when nova has it's own storage in ceph
16:32:32 <noonedeadpunk> mnaser: but I'm not sure that rbd_user is also required.
16:32:57 <mnaser> noonedeadpunk: you think so? how would it know what user to use?
16:32:58 <noonedeadpunk> Since nova won't have rbd_user if it's storage is not in ceph
16:33:05 <mnaser> yes well
16:33:10 <noonedeadpunk> at least it shouldn't
16:33:15 <guilhermesp> that's similar with another issues I had with radosgw
16:33:27 <mnaser> # The RADOS client name for accessing rbd(RADOS Block Devices) volumes. For more
16:33:29 <mnaser> # information, refer to the documentation. (string value)
16:33:35 <logan-> cinder will pass the client name to nova
16:33:38 <guilhermesp> so probably is the lack of configs to nova to be able to access the volume pool
16:33:46 <logan-> you need to have the libvirt secret added though
16:33:57 <mnaser> i dont think its adding it correctly if it is
16:34:09 <mnaser> "auth_username": "cinder"
16:34:13 <mnaser> inside connection_info
16:34:18 <logan-> i believe there is a way to configure OSA to do this, I have been doing it for years (up to and including rocky currently
16:34:18 <mnaser> yet it was complaining about a missing auth_username
16:34:19 <logan-> )
16:34:20 <guilhermesp> yep, that's what I see in docs. I think we are only covering the scenario of everything backend with cehp
16:34:27 <mnaser> well let me show ya
16:34:28 <logan-> if it broke I think it must be post-rocky
16:34:31 <noonedeadpunk> mnaser: so our nova tend not to have acces to cinder pools at all
16:34:55 <noonedeadpunk> so +1 to logan
16:35:07 <guilhermesp> now we just set ceph configs when we have nova in ceph https://github.com/openstack/openstack-ansible-os_nova/blob/cb53967f85fca48e457df4062358675bff16c256/templates/nova.conf.j2#L253-L260 noonedeadpunk
16:36:13 <noonedeadpunk> guilhermesp: yeah, but I mean that nova has local storage it should only have rbd_secret_uuid to use cinder volumes that are placed on rbd
16:36:58 <openstackgerrit> Mohammed Naser proposed openstack/openstack-ansible-os_nova master: rbd: add var for inuse and enable setting secret  https://review.opendev.org/673578
16:36:59 <mnaser> logan-: noonedeadpunk guilhermesp ^ thoughts?
16:37:56 <guilhermesp> lgtm
16:37:59 <guilhermesp> did you test it locally mnaser ?
16:38:08 <mnaser> not yet
16:38:08 <guilhermesp> is it still broken  or?
16:38:13 <guilhermesp> ok
16:38:14 <mnaser> i didnt try that yet to see if it fixes it
16:38:18 <mnaser> but like wanted to have thoughts bout ti
16:38:34 <mnaser> ill cherry pick locally and run adn try
16:39:05 <noonedeadpunk> ah, nova_ceph_client is cinder username
16:39:25 <noonedeadpunk> btw, why don't we drop https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/defaults/main.yml#L389-L390 ?
16:39:39 <guilhermesp> mnaser: comneted
16:39:41 <noonedeadpunk> I mean drop default
16:39:47 <mnaser> because i dont know what might happen if we drop that for now :P
16:42:29 <strattao> if we're having rabbitmq issues (people have been manually making config updates and we're seeing some strange clustering problems), can't I just stop all the services, destroy the rabbitmq containers and re-run rabbitmq-install.yml?
16:42:49 <noonedeadpunk> yeah, ok, you're right and it's not topic of your patch....
16:42:52 <mnaser> youd have to recreate teh rabbitmq clusteers too but yeah
16:43:47 <strattao> so, the rabbitmq-install doesn't create the clusters? At what step does that happen in the installation process then?
16:44:30 <noonedeadpunk> strattao: you can also rabbitmq-install -e 'rabbitmq_upgrade=true'
16:45:26 <noonedeadpunk> this will recreate queues then
16:45:30 <strattao> noonedeadpunk : so I could run that without having to destroy the containers, right?
16:45:35 <noonedeadpunk> yep
16:45:46 <strattao> yeah, I like that approach better
16:46:07 <noonedeadpunk> it tend to help in case of rabbit weirdness
16:46:48 <noonedeadpunk> but for clean re-install of cluster destroying containers is the way to follow
16:48:16 <strattao> if I run with the rabbitmq_upgrade=true, is that non-disruptive or do I need to still shutdown other services first?
16:49:01 <noonedeadpunk> it's disruptive since will drop services queues at some point
16:49:33 <strattao> but the upgrade will handle stopping and starting those services for me, correct?
16:49:40 <noonedeadpunk> But most of the services can re-connect once rabbitmq will be up again
16:49:58 <noonedeadpunk> only few of them might stuck
16:50:22 <noonedeadpunk> no, it won't do anything with openstack services
16:50:33 <strattao> ok
16:50:51 <strattao> thx!
16:51:13 * guilhermesp testing mnaser's patch locally..
16:58:01 <guilhermesp> mnaser: I have an instance based on volume created now
16:58:19 <mnaser> yeah i just tested it in an environment too
16:58:21 <mnaser> and it worked for me too
16:58:25 <guilhermesp> but playbook broke here http://paste.openstack.org/show/755127/
16:58:32 <guilhermesp> did it happen ther etoo?
16:59:01 <mnaser> uh maybe you mis-copied?
16:59:06 <mnaser> 'nova_cinder_rbd_inuse | bool)'
16:59:08 <mnaser> there is an extra )
16:59:11 * guilhermesp checking
16:59:26 <guilhermesp> yep
17:04:30 <tiffanie> #endmeeting