16:00:59 #startmeeting openstack_ansible_meeting 16:01:00 Meeting started Tue Nov 20 16:00:59 2018 UTC and is due to finish in 60 minutes. The chair is mnaser. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:01:00 just checked all 3 has same error 16:01:01 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:01:04 The meeting name has been set to 'openstack_ansible_meeting' 16:01:11 #topic roll call 16:01:11 o/ 16:01:17 cloudnull: i will talk to you after this meeting :) 16:01:21 laggy bot 16:01:23 o/ 16:01:29 o/ 16:01:37 o/ 16:02:12 quiet one 16:02:15 o/ 16:02:16 okay so lets get started 16:02:38 #topic what happened since last meeting 16:02:43 not much, i think. we were mostly all having a great time in berlin :) i had a really good deployemnt tool feedback session 16:02:51 i had a few action items that i carried out 16:02:58 also, project on-boarding and project update were amazing 16:03:27 clearly there is an amazing amonut of interest in OSA, so let's all keep up on that awesome work :D super motivating 16:03:41 arxcruz, thanks for suggestion it makes sense, I will update once dependent patch finishes the output 16:04:12 other things: we're working on os_placement and hopefully we'll land that eventually 16:04:13 it's cool because we're kinda one of the leaders in that so it would be nice for OSA to be "FIRST!" 16:04:31 other than that, jrosser brought up arm64 so maybe want to expand on that 16:04:49 ++ 16:05:01 so, out of mostly curiosity i had a go on the new arm64 ci nodes 16:05:11 :) 16:05:24 and then entirely coincidentally nowster showed up here in the channel trying to do it for real 16:05:27 jrosser: if you need any support on arm stuff, i can get people your way :) 16:05:52 so we've been just poking at that a bit and seeing how far it gets 16:06:06 theres a very obvious and now fixed bug in haproxy 16:06:07 if you are going to be doing some work on that, i'd love to make an introduction to massimo who leads a lot of the open source / openstack stuff and he is a really really useful asset in terms of maybe even getting you resources to accomlpish the things you need 16:06:30 that would be great 16:06:47 wonderful, ill send that soon 16:06:54 the other thing is that the performance of the arm64 nodes isnt great, so we need someone to help take a look at that 16:07:10 i suspect the storage is slow 16:07:32 perfect, he'll actually be able to help with all that stuff 16:07:48 there are other ppl you might want to reach, like linaro too, should you require some help 16:08:03 for anyone who is interested here is the test patch https://review.openstack.org/#/c/618305/ 16:08:12 also, i might be able to get you some.. faster resources for local testing 16:08:16 will follow up more in that email 16:08:27 cool :) 16:09:12 anything else from last week? 16:09:21 late o/ 16:09:40 :) 16:10:07 so I think you noticed that Ive been working on os_placement 16:10:23 here's the pr https://review.openstack.org/#/c/618820/ 16:10:48 I'm still building the basics of the role, but I still need to create the tests 16:11:00 seems that the placement api works under apache + uswgi 16:11:25 mnaser: mentioned that there is also red hat packages for distro install 16:11:28 is that right? 16:11:43 yes, rdo apparently publishment placement packages already 16:12:07 at the moment I'm focusing in source install 16:12:19 but good to know 16:12:28 yep lets get that and we can improve/iterate 16:12:37 anything else or shall we move onto bug triage? 16:12:57 just a quick note about the summit -- odyssey4me and I were in fhaas training session (which was great!), if any folks in the channel here were in that session and want to follow up on some of the things we talked about in the all day session please reach out. 16:13:17 s/fhaas/fghaas/ you're welcome :p 16:13:21 ^ 16:13:23 :) 16:13:31 #topic bug triage 16:13:33 #link https://bugs.launchpad.net/openstack-ansible/+bug/1804204 16:13:35 Launchpad bug 1804204 in openstack-ansible "Quickstart: AIO in openstack-ansible ubuntu instructions" [Undecided,New] 16:14:05 yep 16:14:06 thats confirmed 16:14:28 valid 16:14:34 low-hanging-fruit 16:14:37 low ? 16:15:11 i'd say medium considering .. its broken 16:15:29 something like this came up in the project onboarding ++ Id say medium 16:15:43 asked if paul can push up a patch 16:16:01 if not we can figure things out :) 16:16:01 #link https://bugs.launchpad.net/openstack-ansible/+bug/1803444 16:16:02 Launchpad bug 1803444 in openstack-ansible "gnocchi playbook failing on 18.0.0" [Undecided,New] 16:16:37 include vs import_playbook black magic 16:16:52 does anyone knows why the above might occur? 16:17:52 seems related to ceph_client too 16:18:09 checking 16:18:11 wait is it because the include is include-ing it regardless? 16:18:30 because it looks like that deployer is using ceph 16:18:30 so that block should be entirely ignored 16:18:43 I tried to avoid that playbook for a while :p 16:18:45 * cloudnull reading 16:18:46 hm, never faced with this one... 16:19:05 mnaser: Just to let you know that I'm working with "Works On ARM" too (ie. Massimo). 16:19:22 nowster: oh awesome! 16:19:22 fatal: [infra1_gnocchi_container-7ed445f6]: FAILED! => {"msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"} 16:19:23 we should probably adapt the play to only run the appropriate role if a grouping of host was conditionally successfull 16:19:27 that fits perfectly then :) 16:19:29 is generally an issue with facts 16:19:47 like the OS was not found or there were no valid facts and ansible didnt regather 16:19:50 cloudnull: I agree on the ceph client part 16:20:20 yeah so i think no facts were gathered for that because no tasks actually ran on the gnocchi container 16:20:21 at least in that specific section that it was breaking on 16:20:28 which was the setup-things workaround 16:20:46 as for the "when" in the import playbook. 16:20:53 im not sure that actually works any longer. 16:20:55 we should maybe ask for specific runs 16:21:02 cloudnull: yeah my point exactly 16:21:09 cloudnull: well tbh I am not sure WHEN it worked :p 16:21:12 IDK if import_playbook allows for vars to be passed through 16:21:14 double pun inside! 16:21:22 cause an import isn't really meant to be conditional right? 16:21:24 cloudnull: it was never intended for that 16:21:28 ++ 16:21:29 mnaser: right 16:21:50 it worked with include because we took advantage of the overloaded include system 16:22:23 yeah 16:22:28 is there include_playbook ..? 16:22:31 nope 16:22:37 only import 16:22:39 so what we need to do is move that logic into os-gnocchi right? 16:22:40 but as I said 16:22:47 include will load whatever it finds while import is more like python. 16:22:48 we can move some part into the play 16:22:50 and include_role 16:22:55 ++ 16:23:07 or the meta to end the playbook early 16:23:16 (but I'd rather avoid that one) 16:23:47 for the sake of the bug, I'd like to see a real run of gnocchi 16:23:48 is there a quick 2 minute explanation 16:23:56 as to why this even happens? 16:23:58 to see if it happens again 16:24:09 I mean of the gnocchi playbook 16:24:11 also noonedeadpunk mentions he hasnt ran into that.. do you use ceph? 16:25:11 do we wanna dig into this again next week? 16:26:05 mnaser: that sounds reasonable 16:26:08 mnaser: yep, I'm running ceph 16:26:14 hi guys, I'm the author of this bug report and am available if you need any other info :) 16:26:15 noonedeadpunk: with gnocchi? 16:26:20 francois: oh great! 16:26:24 perfect! 16:26:25 Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Correct clustered gating job https://review.openstack.org/619033 16:26:36 francois: could you run just that gnocchi play again? 16:27:10 openstack-ansible os-gnocchi-install.yml 16:27:15 it should not fail :p 16:27:24 evrardjp: ok 16:27:27 yep. but, I'm not running swift... 16:27:28 if you put the log somewhere that would be nice 16:27:39 noonedeadpunk: that's the deal :) 16:27:51 And now I'm wondering, how I get it installed at all... 16:28:08 francois: because if it repeats itself with that case, it's not a facts problem, but a playbook problem 16:29:12 oh, we're including gnocchi role twice... 16:29:17 francois: can you please update that bug with the info 16:29:20 and we'll cotninue 16:29:29 #link https://bugs.launchpad.net/openstack-ansible/+bug/1803382 16:29:30 Launchpad bug 1803382 in openstack-ansible "unbound-client.yml failure depending on TERMCAP envvar content" [Undecided,New] 16:29:48 mnaser: yeah, will do that 16:29:53 i ran into that bug too 16:30:10 it happened when i was running something in screen 16:30:10 i had *no* idea why it was happening 16:30:10 also mine :) 16:30:13 but it didn't ahppen anymore in screen 16:30:25 this one looks pretty weird! 16:30:40 i have no idea why it honestly happens 16:30:55 I was told that jrosser had the same issue 16:31:10 we can probably say confirmed then 16:31:17 an easy workaround is simply deleting this variable from /etc/openstack_deploy/ansible_facts/localhost 16:31:36 I have the same issue too 16:31:37 we should unset said thing in the connection plugin 16:31:42 but I'm quite interested in how it could happen :) 16:31:55 i will put this as confirmed/high 16:31:59 running things in screen is kinda.. important. 16:32:03 tmux? 16:32:05 :p 16:32:11 + 16:32:13 never had an issue with tmux :p 16:32:16 hahaha 16:32:18 anyway 16:32:20 byobu rules 16:32:20 yeah 16:32:21 tmux for life! 16:32:22 important 16:32:47 francois: do you have sometime to dig a little bit into that one? 16:32:49 byobu = tmux lover too 16:32:51 I digged through a rabbit hole long ago with a rabbitmq issue of the env var that was passed 16:33:23 but we changed our connection plugin now, and we are not relying on the same behaviours anymore 16:33:32 so you're hitting an issue in ansible itself 16:33:38 probably 16:33:52 worth checking in the bugs there too 16:33:56 i put the workaround but yeah, i have no idea, we cant solve it now, just triaging :) 16:33:58 #link https://bugs.launchpad.net/openstack-ansible/+bug/1803347 16:34:00 Launchpad bug 1803347 in openstack-ansible "Documentation mentions requirement of bridges on compute nodes" [Undecided,New] - Assigned to Mohammed Naser (mnaser) 16:34:04 mnaser: yeah lgtm 16:34:07 hah i remember this 16:34:19 Yep, my question, sorry. 16:34:19 :) 16:34:22 confirmed low 16:34:42 #link https://bugs.launchpad.net/openstack-ansible/+bug/1803019 16:34:43 Launchpad bug 1803019 in openstack-ansible "Nova generates versioned_notifications but no one consume" [Undecided,New] - Assigned to KimMinsik (for-beatitudo) 16:35:26 Yeah, it's problem but more about Nova 16:35:32 in progress medium 16:35:33 I know the workaround to it 16:35:49 ah, ok 16:36:46 maybe documenting the workaround is good? 16:37:01 it's already a patch for it present... 16:37:44 yeah we can talk about it in the review if you prefer 16:38:12 not at all. But it's exactly my way of resolving this. 16:38:39 but I've just overrided nova.conf with extra params 16:38:56 yep, its nice the patch has 'native' feature so itd be good to review it 16:39:03 #link https://bugs.launchpad.net/openstack-ansible/+bug/1802742 16:39:04 Launchpad bug 1802742 in openstack-ansible "parameterized nova placement nginx error log level" [Undecided,New] 16:39:34 that seems like wishlist 16:40:16 and for ocata... 16:40:26 it probably still applies 16:40:27 wait it doesnt 16:40:29 we dropped nginx right?! 16:40:57 I think it's just plain uwsgi instead.. 16:40:59 yeah 16:41:02 you're right 16:41:50 meh confiremd/wishlist 16:41:56 #link https://bugs.launchpad.net/openstack-ansible/+bug/1802466 16:41:57 Launchpad bug 1802466 in openstack-ansible "keystone cannot generate keystone.log file" [Undecided,New] - Assigned to KimMinsik (for-beatitudo) 16:42:16 not everywhere, but for nova placement I don't know 16:42:32 placement is uwsgi 16:42:38 also thats cool, we got a fix for that too 16:42:49 +2d it 16:43:11 #link https://bugs.launchpad.net/openstack-ansible/+bug/1802195 16:43:12 Launchpad bug 1802195 in openstack-ansible "ceph-rgw installation/configuration lacks idempotency, configuration changes break the setup" [Undecided,New] 16:43:15 for the last one, I am worrying about container non container 16:44:05 evrardjp: how so? 16:44:56 fghaas: well it was for the keystone cannot generate, which is not the last one, I am a little slow on typing :p 16:45:08 sorry for being in 2 meetings at the same time :p 16:45:30 evrardjp: ack, thanks for clarifying :) 16:45:57 fghaas: is that an osa issue or ceph-ansible? 16:46:12 btw, about keystone - I don't have /var/log/keystone at all, as all log files in journalctl 16:46:23 it's an osa issue in that osa pulls in a state of ceph-ansible that was never meant for production. 16:46:49 I hope the commit messages explain that fairly well 16:47:29 I think that bug is only in New because the actual thing to be fixed is in Rocky, and the fix in master is Related 16:47:59 fghaas: so does that mean its resolved ? 16:48:37 I certainly like to think so, and haven't heard anyone scream otherwise 16:49:18 ok cool 16:49:21 noonedeadpunk: that sounds more likely this is why I am concered of said patch. 16:49:34 #link https://bugs.launchpad.net/openstack-ansible/+bug/1802132 16:49:35 Launchpad bug 1802132 in openstack-ansible "OpenStack Ansible dashboard not able to log in with admin credentials" [Undecided,New] 16:50:02 sounds like a faield deploy 16:50:20 asked to rerun horizon install 16:50:33 we'll review next week 16:50:34 #link https://bugs.launchpad.net/openstack-ansible/+bug/1802070 16:50:35 Launchpad bug 1802070 in openstack-ansible "Add resource_filter field in cinder.conf for non-admin user can retrieve volume & snapshot volume" [Undecided,New] - Assigned to KimMinsik (for-beatitudo) 16:51:22 https://review.openstack.org/#/c/616113/ 16:51:23 already a patch 16:51:41 lets run this by the cinder team 16:51:42 ill do that 16:51:46 just to kinda figure out what is going on 16:51:51 but it should be in progress 16:52:38 #link https://bugs.launchpad.net/openstack-ansible/+bug/1801656 16:52:39 Launchpad bug 1801656 in openstack-ansible "fix tox python3 overrides" [Undecided,New] 16:53:15 thanks for dealing with it mnaser 16:53:19 i guess we can keep that one 16:53:23 put it as confirmed 16:53:26 to track our py3 stuff 16:53:29 ok 16:53:31 fine for me 16:53:37 it's almost all done anyway, isn't it? 16:53:48 it is mostly done i think 16:53:49 returning back to cinder, I thought, that policies resolves mentioned problem.. 16:54:18 #topic open discussion 16:54:29 we can talk about the bugs outside the meeting, but it does seem like it resolves things, but best to run by cinder team 16:54:54 I don't have any topic for open discussion 16:54:59 jungleboyj and the rest of the team should be home by now 16:55:02 while we have a few minutes, got anything on side of arxcruz re: os-tempest? 16:55:08 except I hoped you had a nice trip back from the summit 16:55:17 oh yeah probably worth checking that direction 16:55:31 mnaser: we are starting to add a job to test python-tempestconf 16:55:39 Yep. I am. :-) 16:55:57 and from there, we will add new functionalities on python-tempestconf, like override settings, rpm package, etc 16:56:07 What is the problem? 16:56:28 Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Add libpython2.7 as a required package https://review.openstack.org/619040 16:56:41 about os_placement, I'd really appreciate reviews and suggestions to finish the work https://review.openstack.org/#/c/618820/ :) 16:56:47 cool, thats awesome arxcruz 16:56:49 please send reviews 16:57:09 mnaser: yup, evrardjp is already reviewing it :) 16:57:17 sweet 16:59:36 #endmeeting