21:00:18 <mriedem> #startmeeting nova 21:00:19 <openstack> Meeting started Thu Apr 25 21:00:18 2019 UTC and is due to finish in 60 minutes. The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:23 <openstack> The meeting name has been set to 'nova' 21:00:30 <mriedem> hello 21:00:34 <takashin> o/ 21:00:48 <melwitt> o/ 21:00:49 <edleafe> \o 21:00:59 <mriedem> eric is running his side hustle transportation business so i'll be your chair 21:01:22 <mriedem> #link agenda https://wiki.openstack.org/wiki/Meetings/Nova 21:01:31 <mriedem> #topic Last meeting 21:01:38 <mriedem> #link Minutes from last meeting: http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-04-18-14.00.html 21:01:45 <mriedem> #link Specless bp for libvirt video device models https://blueprints.launchpad.net/nova/+spec/libvirt-video-device-models 21:01:49 <mriedem> This was approved today 21:01:56 <mriedem> #topic Release News 21:02:03 <mriedem> there is no release news 21:02:10 <mriedem> #topic Bugs (stuck/critical) 21:02:18 <mriedem> bugs: we got'em! 21:02:23 <mriedem> No Critical bugs 21:02:30 <mriedem> #link 77 new untriaged bugs (up 1 since the last meeting): https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 21:02:41 <mriedem> #link 11 untagged untriaged bugs (steady since the last meeting): https://bugs.launchpad.net/nova/+bugs?field.tag=-*&field.status%3Alist=NEW 21:02:45 <mriedem> #link bug triage how-to: https://wiki.openstack.org/wiki/Nova/BugTriage#Tags 21:02:50 <mriedem> #help need help with bug triage 21:02:55 <mriedem> Gate status 21:03:01 <mriedem> as most of you probably realize, it's nasty right now 21:03:07 <mriedem> #link https://bugs.launchpad.net/nova/+bug/1825435 TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow." 21:03:08 <openstack> Launchpad bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] 21:03:17 <mriedem> we have probably a 50% failure rate (or higher) on that bug right now 21:03:26 <mriedem> http://status.openstack.org/elastic-recheck/#1825435 21:03:27 <melwitt> do you think this bug is responsible for the really long delay before queuing too? 21:03:47 <mriedem> i'm not sure, that's something i've been wondering too but have been too afraid to ask clarkb/fungi 21:03:53 <melwitt> same 21:04:01 <mriedem> but yeah, would be nice if i didn't have to wait 6 hours to get test results on my debug patch 21:04:07 <mriedem> #help Trying to debug bug 1825435 for over a week now 21:04:09 <openstack> bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] https://launchpad.net/bugs/1825435 21:04:14 <mriedem> #link Trying to reproduce with logs https://review.opendev.org/#/c/654468/ 21:04:23 <melwitt> yeah, I've noticed it takes hours to even get on a CI node 21:04:31 <fungi> have no fear 21:04:34 <mriedem> i blame airship! 21:05:01 <mriedem> i know some node provider was taken out a week or two ago b/c of networking issues thrashing the retries 21:05:04 <mriedem> before the opendev rename 21:05:18 <fungi> i think the long queuing time is mostly because we have a bunch of providers turned off... because they run buggy software like openstack ;) 21:05:26 <mriedem> d'oh 21:05:31 <melwitt> wah wah.... 21:05:56 <fungi> no seriously though, i think we're looking at higher volume than usual for a pre-summit week 21:05:59 <mriedem> ok so hoping that the latest version of https://review.opendev.org/#/c/654468/ will get us a reproduce with a traceback that's usable 21:06:19 <mriedem> 3rd party CI 21:06:20 <melwitt> ++ 21:06:25 <mriedem> #link 3rd party CI status http://ciwatch.mmedvede.net/project?project=nova&time=7+days 21:06:39 <mriedem> it looks to me from my unscientific eye that most 3rd party CI has been crushed by the opendev rename 21:06:42 <fungi> and we *do* have some providers turned off because of network flakiness, and also running into some where they have rogue vms nova doesn't know about squatting ip addresses 21:06:46 <mriedem> and teams are slow to respond 21:07:10 <mriedem> fungi: you don't say... https://review.opendev.org/#/q/topic:bug/1820802+(status:open+OR+status:merged) 21:07:25 <mriedem> mayhap those providers would benefit from ^ 21:07:42 <fungi> yeah, we've had a few (shall we call them "proactive"?) third-party ci operators asking about refused git:// connections or missing refs/changes in their origin remotes 21:07:49 <melwitt> interesting 21:07:54 <fungi> and there have been lingering fixes worming their way into devstack too 21:08:11 <mriedem> ok, well it's good to know it's a known issue 21:08:35 <mriedem> if they are hitting these orphan vms then it'd be good to up the heat rate on that bug 21:08:41 <fungi> i doubt all the issues are really known, but i would give them until after the summit to work a lot of them out 21:08:54 <mriedem> ack 21:08:58 <fungi> (for the 3pci operators) 21:09:04 <mriedem> #topic Reminders 21:09:09 <mriedem> there are none listed... 21:09:21 <mriedem> i guess update the retrospective if you haven't yet 21:09:36 <mriedem> #link retrospective https://etherpad.openstack.org/p/nova-ptg-train-retrospective 21:09:53 <mriedem> looks like that's going to be a fast pow wow during the ptg 21:10:01 <mriedem> #topic Stable branch status 21:10:07 <mriedem> #link Stein regressions: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005242.html 21:10:12 <mriedem> i just sent an update about ^ 21:10:21 <mriedem> 3 of the 4 bugs have patches, one is approved 21:10:29 <mriedem> two are +2ed 21:10:38 <mriedem> the other doesn't have an active fix being worked 21:10:49 <mriedem> #link stable/stein: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/stein 21:10:55 <mriedem> #link stable/rocky: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/rocky 21:11:03 <mriedem> #link stable/queens: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/queens 21:11:15 <mriedem> #link stable/pike: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/pike 21:11:36 <mriedem> #link final nova release for pike (ready) https://review.openstack.org/#/c/652868/ 21:11:45 <mriedem> #link pike-em tagging (ready, but will need recheck once ^ merges) https://review.opendev.org/#/c/652869/ 21:11:57 <mriedem> i'm told the release team hit some tooling issues so the pike-em thing should be happening after that's fixed 21:12:09 <mriedem> #topic Sub/related team Highlights 21:12:18 <mriedem> Placement (cdent) - No meeting this week, cdent on vacation 21:12:25 <mriedem> API (gmann) - No separate updates. Spec reviews are in progress. 21:12:49 <mriedem> i'll use this opportunity to plug for reviews on gmann's API inconsistency cleanup spec https://review.opendev.org/#/c/603969/ 21:12:59 <mriedem> it's on the PTG agenda, but would be good for cores to have read it before then 21:13:14 <mriedem> #topic Stuck Reviews 21:13:25 <mriedem> there is nothing on the agenda - any stuck reviews to bring up here? 21:13:40 <mriedem> #topic Forum Planning 21:13:51 <mriedem> there are several forum sessions linked into the meeting agenda 21:14:00 <mriedem> get the app and start planning your schedule 21:14:07 <mriedem> #topic PTG Planning 21:14:13 <mriedem> #link PTG: Nova etherpad https://etherpad.openstack.org/p/nova-ptg-train 21:14:30 <mriedem> i know eric has been trying to shuffle things and has a rough schedule layed out at the bottom 21:14:49 <mriedem> i look forward to seeing his head explode when people talk waaayyy past their time 21:14:56 <mriedem> muwahahaha 21:15:00 <mriedem> #topic Open discussion 21:15:15 <mriedem> there is one thing on the agenda from efried i think 21:15:16 <mriedem> #link blueprint for integrating openstacksdk and replacing use of python-*client https://blueprints.launchpad.net/nova/+spec/openstacksdk-in-nova 21:15:22 <mriedem> he asks: 21:15:23 <mriedem> Does this need a spec? 21:15:28 <mriedem> Is this something we can get behind? 21:15:40 <mriedem> i read the description before the meeting and left a comment 21:16:03 <mriedem> tl;dr i'd like to get some more feedback from operators since this is going to affect them the most, and to know if other services are using the sdk like this 21:16:12 <mriedem> otherwise i see the benefit but i'm meh+.5 21:16:36 <mriedem> openstack service devs using the sdk would improve the sdk for all users, which is a great long-term benefit 21:16:36 <melwitt> yeah, I guess, 21:17:15 <melwitt> when I read that I wonder, what about what has been said before about replacing stuff in novaclient with openstacksdk 21:17:26 <melwitt> I'm hazy on remembering what those discussions were about 21:17:59 <mriedem> i don't remember talking about replacing things within novaclient with the sdk, 21:18:04 <melwitt> that is, if novaclient used openstacksdk, wouldn't we get that by default in nova 21:18:09 <mriedem> but we've talked about replacing osc's usage of novaclient with the sdk 21:18:20 <melwitt> ok, maybe that's what I'm thinking about 21:18:22 <mriedem> nova the server doesn't use novaclient :) 21:18:40 <melwitt> oh..... yeah :( ugh, sorry. 21:19:17 <mriedem> i think the overall pros are (1) drop nova's use of all the various clients and use one thing, (2) which allows you to configure just clouds.yaml for all the services nova uses rather than all the various glance/ironic/keystone/neutron/cinder/barbican config entries 21:20:07 <mriedem> but i'm not seeing either of those as being big motivators for me personally 21:20:29 <melwitt> yeah. I feel similarly, sounds like a fine idea but not sure how beneficial compared to other things 21:21:01 <mriedem> since we seem to be the only two here caring about this, i don't think we need to make a decision today and can talk about it more at the ptg 21:21:06 <melwitt> like openstacksdk instead of novaclient and getting parity with novaclient 21:21:12 <melwitt> in osc 21:21:31 <melwitt> yeah, agreed 21:21:33 <mriedem> yeah i think closing gaps on osc is more important 21:21:45 <mriedem> but that doesn't mean the people that want to work on this are going to work on that, so shrug 21:21:57 <melwitt> true 21:22:28 <mriedem> #action discuss https://blueprints.launchpad.net/nova/+spec/openstacksdk-in-nova with more people at the ptg 21:22:38 <mriedem> that's it for the agenda - anything else for open discussion? 21:23:10 <mriedem> alright we'll end the meeting then, see some of you next week 21:23:12 <mriedem> o/ 21:23:14 <mriedem> #endmeeting