09:01:06 #startmeeting powervm_driver_meeting 09:01:07 Meeting started Wed Jun 19 09:01:06 2019 UTC and is due to finish in 60 minutes. The chair is dikonoor. Information about MeetBot at http://wiki.debian.org/MeetBot. 09:01:08 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 09:01:10 The meeting name has been set to 'powervm_driver_meeting' 09:04:24 Waiting for members to join. 09:04:33 +1 09:05:06 #topic PowerVM CI 09:05:29 Actions from last week are these: 09:05:33 * amansi26 09:05:33 * amansi26 to open a devstack LP bug for the neutron l3 problem and 09:05:33 propose changeset for that. 09:05:33 * amansi26 and mujahidali continuing to debug the cause of tempest 09:05:33 testcase failure during a manual CI run. 09:06:33 amansi26: Thanks for joining. The tempest problem that was discussed about last week, I believe is resolved. 09:07:14 On this devstack defect > https://review.opendev.org/#/c/664797/ 09:07:18 dikonoor: we are still facing teardown and setupclass failure issues with tempest. 09:07:45 MujahidAli : Are these due to the connection errors? 09:07:51 LAst we checked these were. 09:08:11 There might be some connection errors 09:09:17 We tried adding retry logic to pypowervm/adapter.py> request method but that hasn't helped. amansi26 reported around 52 failures with this fix applied locally. 09:09:47 amansi26: Did you get a chance to look at the logs for these 52 failures to know what exactly is the cause of failure? 09:09:55 amansi26: applied the patch by dikonoor: but no luck 09:10:02 Are each of these failures caused due to connectionerrors or something else? 09:10:33 we have to check why this patch didn't work and try something else MujahidAli. 09:10:53 #action Check why the retry patch failed. 09:12:09 amnsi26 collected the n-cpu logs and tried matching the timestamp, unfortunately there was no log at the timestamp of failure. failure happened at 35th second and logs were there only for 31st and 39th second. 09:13:49 amansi26: we have to talk to chhagarw. She might have some insights 09:14:05 But this is top priority on our list. 09:14:30 Did you or MujahidAli get a chance to talk to someone on the devstack changeset? 09:14:52 https://review.opendev.org/#/c/664797/ 09:15:04 joined 09:15:05 Yes. I checked for the cinder services on the instance they were down 09:15:34 I pinged the issue on the devstack channel(qa)but no reply 09:15:54 chhagarw:Thanks for joining. Need some inputs from you on a connection error that the PowerVM CI has been running into for some time 09:16:54 chhagarw: this is a ssp based environment and we run tempest test cases as part of the CI , that runs these testcases against the lastest instance of devstack with PowerVM drivers. 09:17:35 Each run lasts for 4 hours and we see that we run into a connection error once or max twice during this run 09:18:27 Note here that the the VM running CI has tempest /devstack /PowerVM driver and pypowervm and the calls to NovaLink happens remotely..meaning pypowervm and pvm components are on different systems. 09:18:34 chhagarw: SSLError: SSLError for GET https://172.16.0.1:12443/rest/api/uom/SharedStoragePool/116d6f61-e012-39f5-bcb7-50551596f533?group=None: HTTPSConnectionPool(host='172.16.0.1', port=12443): Max retries exceeded with url: /rest/api/uom/SharedStoragePool/116d6f61-e012-39f5-bcb7-50551596f533?group=None (Caused by SSLError(SSLError('bad handshake: ZeroReturnError()',),)) 09:18:38 This is the exact error 09:23:33 @dikonoor we need to investigate further on this it looks like some Connection Timeout for connection drop error. 09:28:21 ok @chhagarw. 09:29:05 amansi26: as discussed , on the devstack stacking error, we have to take devstack without PowerVM driver and see if stacking works there 09:29:31 If stacking works there, then it's probably not a generic problem.. and could be specific to PowerVM driver. 09:29:37 Pease check this and get back. 09:30:02 #action amansi26 to install devstack latest without any PowerVM driver 09:32:18 #topic Out of tree driver 09:32:49 There are a bunch of changesets that have gone into nova, that need few changes in our OOT driver. 09:32:55 I have added the list at https://etherpad.openstack.org/p/powervm-oot-todos 09:33:32 arunman:ymadhavi:aarora06 need volunteers to prpose changesets 09:33:51 Precisely these: 09:33:52 09:33:52 [dikonoor] Changes from following changesets in in-tree drivers is currently out of sync at OOT: 09:33:52 https://github.com/openstack/nova/commit/aafd873e2176076609ff17d63137dd6f84ddf8d3#diff-98d1a06e0feb7e5ff43a21433afd9f34 09:33:52 https://github.com/openstack/nova/commit/08f3ae960623c94bdd997cacb3e81f04b4bbba69#diff-46d49d172cae6b899c1be3628dede5d1 09:33:55 https://github.com/openstack/nova/commit/8545ba2af7476e0884b5e7fb90965bef92d605bc#diff-5dc5ac3c2bf8010333f1752a1e70ce18 09:33:58 https://github.com/openstack/nova/commit/4afe8ea5a19e236b485da3132d66064017a479ec#diff-c0abaa2f898cb7044ae6f1b8bd0340d9 09:34:01 09:34:03 [dikonoor] Changesets that need revival: 09:34:05 https://review.opendev.org/#/c/627355/ <> https://review.opendev.org/#/c/617693/ 09:34:07 https://review.opendev.org/#/c/654979/ 09:35:58 Request all to please take a look. 09:36:07 So actions we have are : 09:36:20 #action amansi26 to install devstack without PowerVM driver 09:36:46 #action chhagarw to take a closer look at the connection errors causing disruption in PowerVM CI 09:37:40 #action vijayendra_IBM: prabhuraj: ymadahvi:arunman:aarora06 will take a look at the changesets where nova_powervm is out of sync with nova 09:37:55 I am ending the meeting 09:38:03 Thanks all for joining 09:38:04 #endmeeting