21:00:18 <mriedem> #startmeeting nova
21:00:19 <openstack> Meeting started Thu Apr 25 21:00:18 2019 UTC and is due to finish in 60 minutes.  The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:23 <openstack> The meeting name has been set to 'nova'
21:00:30 <mriedem> hello
21:00:34 <takashin> o/
21:00:48 <melwitt> o/
21:00:49 <edleafe> \o
21:00:59 <mriedem> eric is running his side hustle transportation business so i'll be your chair
21:01:22 <mriedem> #link agenda https://wiki.openstack.org/wiki/Meetings/Nova
21:01:31 <mriedem> #topic Last meeting
21:01:38 <mriedem> #link Minutes from last meeting: http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-04-18-14.00.html
21:01:45 <mriedem> #link Specless bp for libvirt video device models https://blueprints.launchpad.net/nova/+spec/libvirt-video-device-models
21:01:49 <mriedem> This was approved today
21:01:56 <mriedem> #topic Release News
21:02:03 <mriedem> there is no release news
21:02:10 <mriedem> #topic Bugs (stuck/critical)
21:02:18 <mriedem> bugs: we got'em!
21:02:23 <mriedem> No Critical bugs
21:02:30 <mriedem> #link 77 new untriaged bugs (up 1 since the last meeting): https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
21:02:41 <mriedem> #link 11 untagged untriaged bugs (steady since the last meeting): https://bugs.launchpad.net/nova/+bugs?field.tag=-*&field.status%3Alist=NEW
21:02:45 <mriedem> #link bug triage how-to: https://wiki.openstack.org/wiki/Nova/BugTriage#Tags
21:02:50 <mriedem> #help need help with bug triage
21:02:55 <mriedem> Gate status
21:03:01 <mriedem> as most of you probably realize, it's nasty right now
21:03:07 <mriedem> #link https://bugs.launchpad.net/nova/+bug/1825435  TestRPC unit tests intermittently fail with "'>' not supported  between instances of 'NoneType' and 'datetime.datetime'" - maybe due to  "Fatal Python error: Cannot recover from stack overflow."
21:03:08 <openstack> Launchpad bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed]
21:03:17 <mriedem> we have probably a 50% failure rate (or higher) on that bug right now
21:03:26 <mriedem> http://status.openstack.org/elastic-recheck/#1825435
21:03:27 <melwitt> do you think this bug is responsible for the really long delay before queuing too?
21:03:47 <mriedem> i'm not sure, that's something i've been wondering too but have been too afraid to ask clarkb/fungi
21:03:53 <melwitt> same
21:04:01 <mriedem> but yeah, would be nice if i didn't have to wait 6 hours to get test results on my debug patch
21:04:07 <mriedem> #help Trying to debug bug 1825435 for over a week now
21:04:09 <openstack> bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] https://launchpad.net/bugs/1825435
21:04:14 <mriedem> #link Trying to reproduce with logs https://review.opendev.org/#/c/654468/
21:04:23 <melwitt> yeah, I've noticed it takes hours to even get on a CI node
21:04:31 <fungi> have no fear
21:04:34 <mriedem> i blame airship!
21:05:01 <mriedem> i know some node provider was taken out a week or two ago b/c of networking issues thrashing the retries
21:05:04 <mriedem> before the opendev rename
21:05:18 <fungi> i think the long queuing time is mostly because we have a bunch of providers turned off... because they run buggy software like openstack ;)
21:05:26 <mriedem> d'oh
21:05:31 <melwitt> wah wah....
21:05:56 <fungi> no seriously though, i think we're looking at higher volume than usual for a pre-summit week
21:05:59 <mriedem> ok so hoping that the latest version of https://review.opendev.org/#/c/654468/ will get us a reproduce with a traceback that's usable
21:06:19 <mriedem> 3rd party CI
21:06:20 <melwitt> ++
21:06:25 <mriedem> #link 3rd party CI status http://ciwatch.mmedvede.net/project?project=nova&time=7+days
21:06:39 <mriedem> it looks to me from my unscientific eye that most 3rd party CI has been crushed by the opendev rename
21:06:42 <fungi> and we *do* have some providers turned off because of network flakiness, and also running into some where they have rogue vms nova doesn't know about squatting ip addresses
21:06:46 <mriedem> and teams are slow to respond
21:07:10 <mriedem> fungi: you don't say... https://review.opendev.org/#/q/topic:bug/1820802+(status:open+OR+status:merged)
21:07:25 <mriedem> mayhap those providers would benefit from ^
21:07:42 <fungi> yeah, we've had a few (shall we call them "proactive"?) third-party ci operators asking about refused git:// connections or missing refs/changes in their origin remotes
21:07:49 <melwitt> interesting
21:07:54 <fungi> and there have been lingering fixes worming their way into devstack too
21:08:11 <mriedem> ok, well it's good to know it's a known issue
21:08:35 <mriedem> if they are hitting these orphan vms then it'd be good to up the heat rate on that bug
21:08:41 <fungi> i doubt all the issues are really known, but i would give them until after the summit to work a lot of them out
21:08:54 <mriedem> ack
21:08:58 <fungi> (for the 3pci operators)
21:09:04 <mriedem> #topic Reminders
21:09:09 <mriedem> there are none listed...
21:09:21 <mriedem> i guess update the retrospective if you haven't yet
21:09:36 <mriedem> #link retrospective https://etherpad.openstack.org/p/nova-ptg-train-retrospective
21:09:53 <mriedem> looks like that's going to be a fast pow wow during the ptg
21:10:01 <mriedem> #topic Stable branch status
21:10:07 <mriedem> #link Stein regressions: http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005242.html
21:10:12 <mriedem> i just sent an update about ^
21:10:21 <mriedem> 3 of the 4 bugs have patches, one is approved
21:10:29 <mriedem> two are +2ed
21:10:38 <mriedem> the other doesn't have an active fix being worked
21:10:49 <mriedem> #link stable/stein: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/stein
21:10:55 <mriedem> #link stable/rocky: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/rocky
21:11:03 <mriedem> #link stable/queens: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/queens
21:11:15 <mriedem> #link stable/pike: https://review.openstack.org/#/q/status:open+(project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/nova)+branch:stable/pike
21:11:36 <mriedem> #link final nova release for pike (ready) https://review.openstack.org/#/c/652868/
21:11:45 <mriedem> #link pike-em tagging (ready, but will need recheck once ^ merges) https://review.opendev.org/#/c/652869/
21:11:57 <mriedem> i'm told the release team hit some tooling issues so the pike-em thing should be happening after that's fixed
21:12:09 <mriedem> #topic Sub/related team Highlights
21:12:18 <mriedem> Placement (cdent) - No meeting this week, cdent on vacation
21:12:25 <mriedem> API (gmann) - No separate updates. Spec reviews are in progress.
21:12:49 <mriedem> i'll use this opportunity to plug for reviews on gmann's API inconsistency cleanup spec https://review.opendev.org/#/c/603969/
21:12:59 <mriedem> it's on the PTG agenda, but would be good for cores to have read it before then
21:13:14 <mriedem> #topic Stuck Reviews
21:13:25 <mriedem> there is nothing on the agenda - any stuck reviews to bring up here?
21:13:40 <mriedem> #topic Forum Planning
21:13:51 <mriedem> there are several forum sessions linked into the meeting agenda
21:14:00 <mriedem> get the app and start planning your schedule
21:14:07 <mriedem> #topic PTG Planning
21:14:13 <mriedem> #link PTG: Nova etherpad https://etherpad.openstack.org/p/nova-ptg-train
21:14:30 <mriedem> i know eric has been trying to shuffle things and has a rough schedule layed out at the bottom
21:14:49 <mriedem> i look forward to seeing his head explode when people talk waaayyy past their time
21:14:56 <mriedem> muwahahaha
21:15:00 <mriedem> #topic Open discussion
21:15:15 <mriedem> there is one thing on the agenda from efried i think
21:15:16 <mriedem> #link blueprint for integrating openstacksdk and replacing use of python-*client https://blueprints.launchpad.net/nova/+spec/openstacksdk-in-nova
21:15:22 <mriedem> he asks:
21:15:23 <mriedem> Does this need a spec?
21:15:28 <mriedem> Is this something we can get behind?
21:15:40 <mriedem> i read the description before the meeting and left a comment
21:16:03 <mriedem> tl;dr i'd like to get some more feedback from operators since this is going to affect them the most, and to know if other services are using the sdk like this
21:16:12 <mriedem> otherwise i see the benefit but i'm meh+.5
21:16:36 <mriedem> openstack service devs using the sdk would improve the sdk for all users, which is a great long-term benefit
21:16:36 <melwitt> yeah, I guess,
21:17:15 <melwitt> when I read that I wonder, what about what has been said before about replacing stuff in novaclient with openstacksdk
21:17:26 <melwitt> I'm hazy on remembering what those discussions were about
21:17:59 <mriedem> i don't remember talking about replacing things within novaclient with the sdk,
21:18:04 <melwitt> that is, if novaclient used openstacksdk, wouldn't we get that by default in nova
21:18:09 <mriedem> but we've talked about replacing osc's usage of novaclient with the sdk
21:18:20 <melwitt> ok, maybe that's what I'm thinking about
21:18:22 <mriedem> nova the server doesn't use novaclient :)
21:18:40 <melwitt> oh..... yeah :( ugh, sorry.
21:19:17 <mriedem> i think the overall pros are (1) drop nova's use of all the various clients and use one thing, (2) which allows you to configure just clouds.yaml for all the services nova uses rather than all the various glance/ironic/keystone/neutron/cinder/barbican config entries
21:20:07 <mriedem> but i'm not seeing either of those as being big motivators for me personally
21:20:29 <melwitt> yeah. I feel similarly, sounds like a fine idea but not sure how beneficial compared to other things
21:21:01 <mriedem> since we seem to be the only two here caring about this, i don't think we need to make a decision today and can talk about it more at the ptg
21:21:06 <melwitt> like openstacksdk instead of novaclient and getting parity with novaclient
21:21:12 <melwitt> in osc
21:21:31 <melwitt> yeah, agreed
21:21:33 <mriedem> yeah i think closing gaps on osc is more important
21:21:45 <mriedem> but that doesn't mean the people that want to work on this are going to work on that, so shrug
21:21:57 <melwitt> true
21:22:28 <mriedem> #action discuss https://blueprints.launchpad.net/nova/+spec/openstacksdk-in-nova with more people at the ptg
21:22:38 <mriedem> that's it for the agenda - anything else for open discussion?
21:23:10 <mriedem> alright we'll end the meeting then, see some of you next week
21:23:12 <mriedem> o/
21:23:14 <mriedem> #endmeeting