| *** slaweq has joined #openstack-nova | 00:11 | |
| *** slaweq has quit IRC | 00:15 | |
| *** amodi_ has quit IRC | 00:17 | |
| eandersson | What kind of impact does getting additional filtering / weighting data have on scheduler performance? | 00:19 |
|---|---|---|
| eandersson | e.g. flavor information | 00:19 |
| eandersson | It's easy enough to test with a few computes and VMs, but not sure how to test it on more realistic PROD scales. | 00:20 |
| *** ileixe has joined #openstack-nova | 00:51 | |
| *** jamesdenton has quit IRC | 00:57 | |
| *** jamesdenton has joined #openstack-nova | 00:57 | |
| *** markvoelker has joined #openstack-nova | 01:09 | |
| *** alex_xu has joined #openstack-nova | 01:11 | |
| *** markvoelker has quit IRC | 01:45 | |
| *** TxGirlGeek has joined #openstack-nova | 01:59 | |
| *** slaweq has joined #openstack-nova | 02:11 | |
| *** hongbin has joined #openstack-nova | 02:14 | |
| *** slaweq has quit IRC | 02:15 | |
| *** mschuppert has quit IRC | 02:33 | |
| *** mhen has quit IRC | 02:52 | |
| *** sapd__x has joined #openstack-nova | 02:57 | |
| *** psachin has joined #openstack-nova | 02:57 | |
| *** mhen has joined #openstack-nova | 03:00 | |
| *** remi_ness has quit IRC | 03:01 | |
| *** brault has joined #openstack-nova | 03:04 | |
| *** sapd__x has quit IRC | 03:08 | |
| *** brault has quit IRC | 03:09 | |
| *** remi_ness has joined #openstack-nova | 03:15 | |
| *** markvoelker has joined #openstack-nova | 03:18 | |
| *** ircuser-1 has joined #openstack-nova | 03:19 | |
| *** sapd__x has joined #openstack-nova | 03:25 | |
| *** whoami-rajat has joined #openstack-nova | 03:30 | |
| *** markvoelker has quit IRC | 03:34 | |
| *** markvoelker has joined #openstack-nova | 03:34 | |
| *** markvoelker has quit IRC | 03:39 | |
| *** TxGirlGeek has quit IRC | 03:39 | |
| *** markvoelker has joined #openstack-nova | 03:48 | |
| *** diga has joined #openstack-nova | 03:51 | |
| *** markvoelker has quit IRC | 03:54 | |
| *** markvoelker has joined #openstack-nova | 03:55 | |
| *** Kevin_Zheng has quit IRC | 04:08 | |
| *** slaweq has joined #openstack-nova | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** udesale has joined #openstack-nova | 04:23 | |
| *** tbachman has joined #openstack-nova | 04:29 | |
| *** sapd__x has quit IRC | 04:33 | |
| *** owalsh_ has joined #openstack-nova | 04:33 | |
| *** sridharg has joined #openstack-nova | 04:33 | |
| *** owalsh has quit IRC | 04:37 | |
| *** sapd__x has joined #openstack-nova | 04:45 | |
| *** wolverineav has joined #openstack-nova | 04:55 | |
| *** _pewp_ has joined #openstack-nova | 05:07 | |
| *** ratailor has joined #openstack-nova | 05:10 | |
| *** markvoelker has quit IRC | 05:39 | |
| *** wolverineav has quit IRC | 05:51 | |
| *** wolverineav has joined #openstack-nova | 05:55 | |
| *** hongbin_ has joined #openstack-nova | 05:59 | |
| *** brault has joined #openstack-nova | 06:00 | |
| *** hongbin has quit IRC | 06:02 | |
| *** hongbin_ has quit IRC | 06:02 | |
| *** brault has quit IRC | 06:05 | |
| *** macza has joined #openstack-nova | 06:06 | |
| *** wolverineav has quit IRC | 06:07 | |
| *** cfriesen has joined #openstack-nova | 06:07 | |
| *** slaweq has joined #openstack-nova | 06:11 | |
| *** udesale has quit IRC | 06:11 | |
| *** udesale has joined #openstack-nova | 06:12 | |
| *** markvoelker has joined #openstack-nova | 06:12 | |
| *** slaweq has quit IRC | 06:15 | |
| *** brinzhang has joined #openstack-nova | 06:18 | |
| *** belmoreira has quit IRC | 06:35 | |
| *** wolverineav has joined #openstack-nova | 06:41 | |
| *** wolverineav has quit IRC | 06:45 | |
| *** slaweq has joined #openstack-nova | 06:46 | |
| *** slaweq has quit IRC | 06:50 | |
| *** remi_ness has quit IRC | 06:51 | |
| *** Luzi has joined #openstack-nova | 06:53 | |
| *** udesale has quit IRC | 06:58 | |
| *** udesale has joined #openstack-nova | 06:58 | |
| *** maciejjozefczyk has quit IRC | 06:58 | |
| *** maciejjozefczyk has joined #openstack-nova | 07:02 | |
| *** slaweq has joined #openstack-nova | 07:06 | |
| *** ileixe has quit IRC | 07:09 | |
| *** pcaruana has joined #openstack-nova | 07:11 | |
| *** wolverineav has joined #openstack-nova | 07:13 | |
| *** slaweq has quit IRC | 07:15 | |
| *** dpawlik has joined #openstack-nova | 07:29 | |
| *** sapd1 has quit IRC | 07:29 | |
| *** sapd__x has quit IRC | 07:29 | |
| *** sapd1 has joined #openstack-nova | 07:29 | |
| *** sapd__x has joined #openstack-nova | 07:29 | |
| *** brault has joined #openstack-nova | 07:42 | |
| *** ccamacho has joined #openstack-nova | 07:42 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Allow run metadata api per cell https://review.openstack.org/624612 | 07:44 |
| *** maciejjozefczyk has quit IRC | 07:48 | |
| *** belmoreira has joined #openstack-nova | 07:49 | |
| *** slaweq has joined #openstack-nova | 07:58 | |
| *** markvoelker has quit IRC | 07:58 | |
| *** lpetrut has joined #openstack-nova | 08:01 | |
| *** markvoelker has joined #openstack-nova | 08:03 | |
| *** maciejjozefczyk has joined #openstack-nova | 08:04 | |
| maciejjozefczyk | jaypipes: hey. I did a little update on counters here: https://review.openstack.org/#/c/614167/ Please check that ;) Sorry for spamming a lot about this | 08:05 |
| *** moshele has joined #openstack-nova | 08:06 | |
| *** wolverineav has quit IRC | 08:08 | |
| *** rpittau has joined #openstack-nova | 08:09 | |
| *** markvoelker has quit IRC | 08:10 | |
| *** helenafm has joined #openstack-nova | 08:11 | |
| *** sapd1 has quit IRC | 08:15 | |
| *** sapd1_x has joined #openstack-nova | 08:15 | |
| *** sapd1 has joined #openstack-nova | 08:16 | |
| *** ratailor_ has joined #openstack-nova | 08:18 | |
| *** sapd__x has quit IRC | 08:19 | |
| *** ratailor has quit IRC | 08:21 | |
| *** jangutter has joined #openstack-nova | 08:24 | |
| *** _alastor_ has joined #openstack-nova | 08:24 | |
| *** _alastor_ has quit IRC | 08:29 | |
| *** markvoelker has joined #openstack-nova | 08:29 | |
| *** markvoelker has quit IRC | 08:30 | |
| *** markvoelker has joined #openstack-nova | 08:32 | |
| *** markvoelker has quit IRC | 08:33 | |
| *** markvoelker has joined #openstack-nova | 08:35 | |
| *** liuyulong has joined #openstack-nova | 08:37 | |
| *** ralonsoh has joined #openstack-nova | 08:43 | |
| *** liuyulong has quit IRC | 08:48 | |
| *** sridharg has quit IRC | 09:03 | |
| *** sridharg has joined #openstack-nova | 09:04 | |
| *** sridharg has quit IRC | 09:04 | |
| *** sridharg has joined #openstack-nova | 09:04 | |
| *** panda|off is now known as panda | 09:12 | |
| *** sridharg has quit IRC | 09:16 | |
| *** derekh has joined #openstack-nova | 09:32 | |
| *** owalsh_ is now known as owalsh | 09:34 | |
| *** markvoelker has quit IRC | 09:52 | |
| *** wolverineav has joined #openstack-nova | 09:55 | |
| *** ratailor__ has joined #openstack-nova | 09:56 | |
| *** ratailor_ has quit IRC | 09:58 | |
| *** wolverineav has quit IRC | 09:59 | |
| *** macza has quit IRC | 10:01 | |
| *** cfriesen has quit IRC | 10:04 | |
| *** ratailor__ has quit IRC | 10:05 | |
| *** diga has quit IRC | 10:16 | |
| *** ratailor has joined #openstack-nova | 10:19 | |
| *** awalende has joined #openstack-nova | 10:20 | |
| *** yan0s has joined #openstack-nova | 10:35 | |
| *** dtantsur|afk is now known as dtantsur | 10:39 | |
| *** diga has joined #openstack-nova | 10:40 | |
| *** erlon has joined #openstack-nova | 10:41 | |
| *** moshele has quit IRC | 10:42 | |
| *** ratailor_ has joined #openstack-nova | 10:58 | |
| *** macza has joined #openstack-nova | 10:59 | |
| *** diga has quit IRC | 11:00 | |
| *** ratailor has quit IRC | 11:02 | |
| *** macza has quit IRC | 11:04 | |
| *** Dinesh_Bhor has joined #openstack-nova | 11:08 | |
| *** udesale has quit IRC | 11:13 | |
| *** macza has joined #openstack-nova | 11:20 | |
| *** macza has quit IRC | 11:24 | |
| *** mvkr has quit IRC | 11:26 | |
| *** erlon has quit IRC | 11:29 | |
| *** erlon has joined #openstack-nova | 11:33 | |
| *** erlon has quit IRC | 11:36 | |
| *** erlon has joined #openstack-nova | 11:37 | |
| *** cdent has joined #openstack-nova | 11:46 | |
| *** sapd1_x has quit IRC | 11:47 | |
| *** moshele has joined #openstack-nova | 11:48 | |
| finucannot | bauzas: If you're about today, fancy pushing https://review.openstack.org/630154 and https://review.openstack.org/630138 ? | 11:48 |
| *** finucannot is now known as stephenfin | 11:49 | |
| *** mvkr has joined #openstack-nova | 11:56 | |
| *** ratailor__ has joined #openstack-nova | 11:57 | |
| *** ratailor_ has quit IRC | 11:59 | |
| *** awalende has quit IRC | 12:00 | |
| *** markvoelker has joined #openstack-nova | 12:01 | |
| *** ratailor__ has quit IRC | 12:02 | |
| *** awalende has joined #openstack-nova | 12:03 | |
| *** sridharg has joined #openstack-nova | 12:04 | |
| *** sapd1_x has joined #openstack-nova | 12:06 | |
| *** awalende has quit IRC | 12:08 | |
| *** mvkr has quit IRC | 12:10 | |
| *** mvkr has joined #openstack-nova | 12:10 | |
| *** sapd1_x has quit IRC | 12:26 | |
| *** awalende has joined #openstack-nova | 12:27 | |
| jaypipes | maciejjozefczyk: no worries! will look at it shortly. | 12:35 |
| *** brinzhang has quit IRC | 12:38 | |
| *** cdent has quit IRC | 12:40 | |
| *** erlon_ has joined #openstack-nova | 12:44 | |
| *** Dinesh_Bhor has quit IRC | 12:45 | |
| *** erlon has quit IRC | 12:48 | |
| *** macza has joined #openstack-nova | 12:49 | |
| *** macza has quit IRC | 12:54 | |
| *** awalende has quit IRC | 12:55 | |
| *** rpittau is now known as rpittau|lunch | 12:59 | |
| *** needssleep is now known as TheJulia | 13:02 | |
| *** awalende has joined #openstack-nova | 13:03 | |
| *** macza has joined #openstack-nova | 13:11 | |
| *** dave-mccowan has joined #openstack-nova | 13:14 | |
| *** macza has quit IRC | 13:16 | |
| sean-k-mooney | o/ | 13:28 |
| *** _alastor_ has joined #openstack-nova | 13:28 | |
| *** awaugama has joined #openstack-nova | 13:31 | |
| *** wolverineav has joined #openstack-nova | 13:31 | |
| *** _alastor_ has quit IRC | 13:33 | |
| *** wolverineav has quit IRC | 13:36 | |
| *** ttsiouts has joined #openstack-nova | 13:39 | |
| *** ttsiouts has quit IRC | 13:44 | |
| *** ttsiouts has joined #openstack-nova | 13:44 | |
| *** mriedem has joined #openstack-nova | 13:44 | |
| *** temka is now known as artom | 13:50 | |
| gibi | mriedem: hi! welcome back! I'm trying to find a slot for a hangouts call about the bandwidth patches and need your input. Possible slots this week: Wednesday 18:00 UTC (after cdent's placement hangouts), Thursday 17:00 UTC, Friday 17:00 UTC | 13:51 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix configure() called after DatabaseAtVersion fixture https://review.openstack.org/619723 | 13:51 |
| *** tetsuro has joined #openstack-nova | 13:51 | |
| gibi | mriedem: and one more not so good spot on Tuesday 17:00 UTC | 13:53 |
| *** takashin has joined #openstack-nova | 13:54 | |
| mriedem | any of those should work for me | 13:55 |
| efried | n-sch meeting in 10 minutes in #openstack-meeting-alt | 13:56 |
| *** fragatina has joined #openstack-nova | 13:56 | |
| gibi | mriedem: thanks | 14:00 |
| gibi | mriedem: will post the final time on the ML | 14:00 |
| openstackgerrit | caoyuan proposed openstack/nova stable/rocky: Update the description to make it more accuracy https://review.openstack.org/630681 | 14:05 |
| *** rpittau|lunch is now known as rpittau | 14:05 | |
| *** sapd1_x has joined #openstack-nova | 14:09 | |
| *** mmethot has joined #openstack-nova | 14:12 | |
| *** mchlumsky has joined #openstack-nova | 14:16 | |
| *** beekneemech is now known as bnemec | 14:19 | |
| *** lbragstad has joined #openstack-nova | 14:19 | |
| *** awalende has quit IRC | 14:20 | |
| *** markvoelker has quit IRC | 14:31 | |
| efried | belmoreira: Howdy. Are you around today? | 14:33 |
| *** markvoelker has joined #openstack-nova | 14:34 | |
| *** asmita has joined #openstack-nova | 14:38 | |
| asmita | gibi: Hi | 14:39 |
| *** ShilpaSD has joined #openstack-nova | 14:40 | |
| *** fragatina has quit IRC | 14:40 | |
| gibi | asmita: hu | 14:41 |
| gibi | asmita: hi | 14:41 |
| *** takashin has left #openstack-nova | 14:42 | |
| sean-k-mooney | adrianc: do you mind if i rebase the sriov patch series and start addressing some of stephenfin's nits | 14:42 |
| asmita | gibi: How do we use this mehod?[1]:https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/unit/utils.py#L30 | 14:43 |
| asmita | method* | 14:43 |
| sean-k-mooney | mriedem: are you ok with removing your procedual -2 in https://review.openstack.org/#/c/616120/ since the spec has been merged and blueprint approved? | 14:44 |
| mriedem | you don't | 14:44 |
| mriedem | asmita: ^ | 14:44 |
| mriedem | it's patching the mock library at import time to avoid adding tests which call invalid assert methods | 14:44 |
| mriedem | which used to be a problem, but is fixed in latest mock i believe | 14:44 |
| gibi | asmita: mriedem was faster. But I think the mock library already fixed to reject those wrong asserts | 14:45 |
| *** tetsuro has quit IRC | 14:45 | |
| gibi | asmita: so you don't even need that patch any more | 14:45 |
| mriedem | sean-k-mooney: done | 14:45 |
| sean-k-mooney | mriedem: thanks it need to be rebased and some nits addressed but i think the feature is effectivly complete. we just need to test it properly and clean it up at this point. | 14:46 |
| asmita | Okay.There is a test scenario in test_hypervisors.py file of novaclient which I need to implement.I need to raise exception in that test case..But because of the fixtures,I get a error as "No mock address".So how do I overcome this situation? | 14:48 |
| gibi | asmita: do you have a patch up on gerrit I can look at? | 14:49 |
| asmita | gibi: Not yet. | 14:51 |
| gibi | asmita: just push the not working code up so I can look at it | 14:51 |
| *** mlavalle has joined #openstack-nova | 14:54 | |
| *** psachin has quit IRC | 14:56 | |
| *** macza has joined #openstack-nova | 14:56 | |
| asmita | gibi: Please have a look.https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/unit/utils.py#L30 | 14:56 |
| *** sridharg has quit IRC | 14:56 | |
| asmita | gibi: https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/unit/utils.py#L30 | 14:57 |
| asmita | gibi : Sorry wron link | 14:57 |
| asmita | wrong link | 14:57 |
| asmita | gibi: paste.openstack.org/show/742306..Thanks. | 14:58 |
| gibi | asmita: looking... | 14:58 |
| tobias-urdin | any good way to actually health check novncproxy? we've had some fallouts where it stops working with "handler exception: [Errno 32] Broken pipe" on console requests | 14:59 |
| tobias-urdin | i guess i'll have to look into some doing some vnc client that does a handshake and tries to find some response that could be checked | 15:00 |
| *** sean-k-mooney has quit IRC | 15:00 | |
| gibi | asmita: this patch you pasted works for me, both py27 and py35 test passes for me on it | 15:04 |
| gibi | asmita: http://paste.openstack.org/show/742307/ | 15:05 |
| *** sean-k-mooney has joined #openstack-nova | 15:06 | |
| asmita | gibi :Thanks . | 15:07 |
| gibi | asmita: so I don't know why it doesn't work for you | 15:07 |
| asmita | gibi : My concern here is whether it is okay to mock.patch and raise exception,the way I have done? | 15:09 |
| *** cfriesen has joined #openstack-nova | 15:09 | |
| *** Luzi has quit IRC | 15:10 | |
| *** derekh has quit IRC | 15:11 | |
| *** dpawlik has quit IRC | 15:11 | |
| *** jangutter has quit IRC | 15:12 | |
| asmita | gibi : Thanks.I will look into it. | 15:13 |
| *** mrch_ has joined #openstack-nova | 15:14 | |
| *** xek has joined #openstack-nova | 15:15 | |
| gibi | amotoki: is the self.cs.hypervisors.search() you call and the novaclient.v2.hypervisors.HypervisorManager.search that you mocked the same thing? if they are the same then you are not testing anything (except the mock library) | 15:17 |
| gibi | amotoki: sorry | 15:17 |
| gibi | asmita: ^^ | 15:17 |
| *** markvoelker has quit IRC | 15:20 | |
| *** markvoelker has joined #openstack-nova | 15:20 | |
| *** markvoelker has quit IRC | 15:20 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Allow run metadata api per cell https://review.openstack.org/624612 | 15:21 |
| *** derekh has joined #openstack-nova | 15:21 | |
| mriedem | dansmith: the channel topic is pretty out of date, maybe we should just link to https://etherpad.openstack.org/p/nova-runways-stein ? | 15:22 |
| mriedem | "Current review runways: https://etherpad.openstack.org/p/nova-runways-stein" or something | 15:22 |
| dansmith | aye | 15:22 |
| dansmith | will update in a bit | 15:22 |
| *** ttsiouts has quit IRC | 15:22 | |
| mriedem | thanks | 15:22 |
| *** ttsiouts has joined #openstack-nova | 15:23 | |
| *** efried1 has joined #openstack-nova | 15:24 | |
| *** efried has quit IRC | 15:25 | |
| *** efried1 is now known as efried | 15:25 | |
| *** jangutter has joined #openstack-nova | 15:26 | |
| mriedem | easy runway bp change with a +2 https://review.openstack.org/#/c/624612/ - run meta-api per cell | 15:27 |
| openstackgerrit | Jan Gutter proposed openstack/os-vif master: Extend port profiles with datapath offload type https://review.openstack.org/572081 | 15:27 |
| belmoreira | efried: hi | 15:27 |
| *** ttsiouts has quit IRC | 15:28 | |
| efried | belmoreira: Greetings. Today in the nova-scheduler meeting we brought up a couple of patches that CERN had showed interest in. | 15:28 |
| efried | We were looking to find out if y'all could (re)deploy them to make sure they do what's intended and don't break the world. | 15:28 |
| efried | (jaypipes heads up) | 15:29 |
| *** _alastor_ has joined #openstack-nova | 15:29 | |
| efried | The first one was the placement traffic improvement series starting at https://review.openstack.org/#/c/615677/ | 15:29 |
| *** mmethot has quit IRC | 15:30 | |
| efried | IIRC, you had deployed it and demonstrated the expected improvements, which is great. | 15:30 |
| efried | but | 15:30 |
| *** mmethot_ has joined #openstack-nova | 15:30 | |
| efried | there was some concern about how it would behave in large ironic deployments. | 15:30 |
| efried | Late last week I made some updates to hopefully take care of that | 15:30 |
| efried | Last I recall, you were about to start deploying to your ironic nodes - not sure how far you got on that or whether you stalled it because of the perceived problems with the patches. | 15:31 |
| belmoreira | efried: that are great news | 15:31 |
| *** ChanServ sets mode: +o dansmith | 15:31 | |
| efried | Anyway, we were hoping you could redeploy the series in its current form to see if it's copacetic | 15:31 |
| *** dansmith changes topic to "Current runways: https://etherpad.openstack.org/p/nova-runways-stein -- This channel is for Nova development. For support of Nova deployments, please use #openstack." | 15:32 | |
| belmoreira | before the end of the year I was trying it with the ironic compute nodes. It was in our small test cell, and it was OK. Scalability issues will only be noticed in our prod instance. Was planning to do it next week. | 15:34 |
| *** _alastor_ has quit IRC | 15:34 | |
| belmoreira | will check the new code, thanks. | 15:34 |
| efried | belmoreira: Great. | 15:36 |
| efried | The other one was jaypipes: https://review.openstack.org/#/c/623558/ | 15:36 |
| efried | I don't know anything about this, will let him speak to it. | 15:36 |
| *** jangutter has quit IRC | 15:37 | |
| *** TxGirlGeek has joined #openstack-nova | 15:40 | |
| cfriesen | artom: have you made any progress with the resource-aware live migration stuff? I don't see any reviews up since the spec review. | 15:43 |
| *** mmethot_ has quit IRC | 15:43 | |
| *** mmethot_ has joined #openstack-nova | 15:43 | |
| sean-k-mooney | cfriesen: i haven't seen artom online downstream yet today so not sure if he is about yet | 15:43 |
| cfriesen | sean-k-mooney: thanks | 15:44 |
| belmoreira | efried: I didn't try jaypipes code because merge/dependencies issues. Ended up with a custom code that only gets the info from the nodes in the cell. Will revisit it next week. | 15:44 |
| efried | ight | 15:45 |
| jaypipes | belmoreira: https://review.openstack.org/#/c/623558/ commit message should be self-explanatory. It should dramatically increase performance of your scheduler due to not doing N queries and only doing 1 query per cell. | 15:45 |
| sean-k-mooney | cfriesen: i know he plans to work on it this sprint and once i finsih off the last few bits for the sriov migraton ill likely try and spend my time helping him with that work | 15:45 |
| cfriesen | sean-k-mooney: good to hear. I'm hoping to have some review cycles for this. | 15:46 |
| *** mmethot_ has quit IRC | 15:46 | |
| *** mmethot_ has joined #openstack-nova | 15:47 | |
| *** mmethot_ has quit IRC | 15:49 | |
| *** mmethot_ has joined #openstack-nova | 15:50 | |
| *** _alastor_ has joined #openstack-nova | 15:50 | |
| belmoreira | jaypipes: let me revisit again your patch. I wasn't able to use your code directly because it was relying in few things that are new to rocky and I got scared to push them to prod. But we cooked a light similar thing. This is the result in the schedule time: https://docs.google.com/document/d/1atA0wbUNBCdojiq8WyBKA0k8Ulyj91bzXDMPOIN9gRI | 15:52 |
| belmoreira | jaypipes: I think yours will improve it even more | 15:52 |
| *** _alastor_ has quit IRC | 15:54 | |
| belmoreira | jaypipes: after this what we observed is that now most of the time is because the placement call... But needs more investigation in our side. | 15:55 |
| belmoreira | jaypipes: will let you know when we start again our tests. Thanks for all this work | 15:56 |
| jaypipes | belmoreira: rock on :) | 15:57 |
| *** lpetrut has quit IRC | 15:58 | |
| *** igordc has joined #openstack-nova | 16:01 | |
| *** eharney has joined #openstack-nova | 16:02 | |
| *** moshele has quit IRC | 16:03 | |
| *** igordc has quit IRC | 16:04 | |
| *** _alastor_ has joined #openstack-nova | 16:11 | |
| adrianc | sean-k-mooney: working on the nits, PS will be uploaded by tommorow | 16:11 |
| adrianc | sean-k-mooney: left to deal with several on the indirect implementation commit and ill upload | 16:12 |
| sean-k-mooney | ok ill wait untill you have uploaded it so to rebase | 16:13 |
| artom | sean-k-mooney, I'm around | 16:13 |
| artom | cfriesen, nothing posted yet, working on it | 16:13 |
| adrianc | sean-k-mooney: thanks | 16:13 |
| artom | sean-k-mooney, just because I don't say anything, doesn't mean I'm not here ;) | 16:13 |
| cfriesen | artom: cool, thanks. | 16:13 |
| *** cdent has joined #openstack-nova | 16:14 | |
| *** mmethot has joined #openstack-nova | 16:18 | |
| *** pcaruana has quit IRC | 16:20 | |
| *** jangutter has joined #openstack-nova | 16:21 | |
| sean-k-mooney | artom: ah lurking in the backgound. sneaky :) | 16:21 |
| *** mmethot has quit IRC | 16:21 | |
| *** mmethot has joined #openstack-nova | 16:22 | |
| *** mmethot_ has quit IRC | 16:22 | |
| artom | sean-k-mooney, I mean, it's sort of what everyone does by default, no? | 16:23 |
| *** amodi has joined #openstack-nova | 16:24 | |
| *** fragatina has joined #openstack-nova | 16:25 | |
| *** eharney has quit IRC | 16:25 | |
| sean-k-mooney | artom: ya true but i usually say hi in the morning | 16:26 |
| artom | sean-k-mooney, maybe I'll start doing that :) | 16:27 |
| *** mmethot_ has joined #openstack-nova | 16:29 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject interface attach with QoS aware port https://review.openstack.org/570078 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject networks with QoS policy https://review.openstack.org/570079 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Create RequestGroup from neutron port https://review.openstack.org/625941 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Include requested_resources to allocation candidate query https://review.openstack.org/625942 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend RequestGroup object for mapping https://review.openstack.org/619527 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove port allocation during detach https://review.openstack.org/622421 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor PortResourceRequestBasedSchedulingTestBase https://review.openstack.org/624080 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Record requester in the InstancePCIRequest https://review.openstack.org/625310 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add pf_interface_name tag to passthrough_whitelist https://review.openstack.org/625311 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ensure that bandwidth and VF are from the same PF https://review.openstack.org/623543 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend NeutronFixture to return port with resource request https://review.openstack.org/630719 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Read port resource request from Neutron https://review.openstack.org/630720 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject server create with port having resource request https://review.openstack.org/630721 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject resize with port having resource request https://review.openstack.org/630722 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migrate with port having resource request https://review.openstack.org/630723 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject evacuate with port having resource request https://review.openstack.org/630724 | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject unshelve with port having resource request https://review.openstack.org/630725 | 16:29 |
| * gibi is trully sorry about the lenght of the series | 16:30 | |
| sean-k-mooney | hehe i takeit you have resovled all the merge conflits | 16:31 |
| gibi | sean-k-mooney: not just that, but make it so that the new external behavior is only active in a new microversion (which still needs to be added on the top) | 16:32 |
| gibi | mriedem: ^^ fyi | 16:32 |
| *** _mmethot_ has joined #openstack-nova | 16:32 | |
| *** mmethot_ has quit IRC | 16:32 | |
| *** mmethot has quit IRC | 16:33 | |
| sean-k-mooney | gibi: cool and for the old micro version dose it ignore the request or reject | 16:33 |
| gibi | sean-k-mooney: today it rejects. Ignoring the request leads to some hard to manage edge cases that I added to the etherpad | 16:33 |
| gibi | sean-k-mooney: like create a server with a new microversion so there is port allocation but then migrate it with an old microversion | 16:34 |
| sean-k-mooney | ok well we can disucss those on the call i guess | 16:34 |
| gibi | sean-k-mooney: yes, definitely | 16:34 |
| sean-k-mooney | on blance that might be safer but we need a failly big upgrade impact section to call it out | 16:35 |
| sean-k-mooney | did you choose a time by the way | 16:35 |
| gibi | sean-k-mooney: I still have to talk to mlavalle about it | 16:35 |
| gibi | sean-k-mooney: as the last proposed time, Tuesday, does not seem to work any more | 16:35 |
| mlavalle | gibi: will get back to you in ~30 mins | 16:36 |
| gibi | mlavalle: OK, no worries | 16:36 |
| sean-k-mooney | ok cool ill try and be there but i also dont need to be so ill keep an eye out on the ml | 16:36 |
| *** hongbin has joined #openstack-nova | 16:37 | |
| gibi | sean-k-mooney: i will post to the ML in any case | 16:37 |
| *** lpetrut has joined #openstack-nova | 16:37 | |
| *** mmethot_ has joined #openstack-nova | 16:37 | |
| *** _mmethot_ has quit IRC | 16:38 | |
| gibi | sean-k-mooney: btw if we go with the reject solution in nova, we can still turn off sending the resource request in neutron. That way we can get back the true legacy behavior, i.e. no resource request in the port et all | 16:38 |
| sean-k-mooney | gibi: im not quite sure how you neutron would know when to do that. are you suggesting a config opion for it? | 16:39 |
| gibi | sean-k-mooney: yes, it is config driven in this WIP patch https://review.openstack.org/#/c/627978/ | 16:40 |
| *** eharney has joined #openstack-nova | 16:40 | |
| gibi | sean-k-mooney: it turned out that neutron api extensions are not selectively loaded but loaded by the service plugin | 16:40 |
| gibi | sean-k-mooney: but everything in the neutron code is ready to make an api extension like our resource request selectively loaded | 16:41 |
| sean-k-mooney | ah cool. so if you have stien neutron and rocky nova or older you would set the config option to disable it | 16:42 |
| sean-k-mooney | gibi: well there are selectivly loaded in that you have to enable the service plugin or extentinon in the neutron.conf or in the agent/ml2_conf.ini | 16:42 |
| gibi | sean-k-mooney: if you have stein neutron and rocky nova then nova does not check the resource request at all. if you have stein neutron and stein nova, but you want to go back to the legacy behavior then you can turn off the extension in neutron | 16:43 |
| gibi | sean-k-mooney: as far as I know if you enable a service plugin then it loads every extension of that plugin | 16:43 |
| *** ttsiouts has joined #openstack-nova | 16:44 | |
| sean-k-mooney | ok so its not that granular in say enable ingress qos only you can either enable qos or not | 16:44 |
| gibi | sean-k-mooney: yes, as far as I understand | 16:45 |
| sean-k-mooney | and you get all the qos oplicys or none today. i honestly did not look quite that closely at least not in a few years | 16:45 |
| *** ade_lee has joined #openstack-nova | 16:45 | |
| gibi | sean-k-mooney: I've never looked into it deeply. I rely on rubasov's and lajoskatona's neutron experties | 16:47 |
| ade_lee | hi all -- we have a grenade test that keeps failing due to an error which is related to volume attachment. Is there anyone here that knows about how nova volume attach works and can help? | 16:48 |
| *** fragatina has quit IRC | 16:48 | |
| jangutter | moshel, lennyb: I suspect the OVS_HW_offload CI is having some issues. Would you be able to check it? | 16:49 |
| ade_lee | specifically, we're running into an error where we try to write to a volume before its ready -- | 16:50 |
| ade_lee | we do wait for cinder to show the volume as being "in-use", but that seems to be insufficient. | 16:51 |
| ade_lee | what would we wait for to ensure that the volume is ready for use? | 16:51 |
| ade_lee | and the instance knows about it. | 16:52 |
| *** mdbooth has joined #openstack-nova | 16:52 | |
| *** rpittau has quit IRC | 16:57 | |
| *** fragatina has joined #openstack-nova | 17:00 | |
| stephenfin | bauzas: I think you need to submit those gantt reviews by yourself, as it's just yourself and johnthetubaguy that have access. melwitt is OK with us retiring it | 17:01 |
| bauzas | stephenfin: WDYM ? | 17:01 |
| bauzas | fast-approving ? | 17:01 |
| stephenfin | yup | 17:01 |
| stephenfin | bauzas: Pretty much everyone else is no longer working on OpenStack https://review.openstack.org/#/admin/groups/253,members | 17:02 |
| sean-k-mooney | stephenfin: well you can submit the change and bauzas and johnthetubaguy can approve | 17:02 |
| mdbooth | Are folks aware that openstack-placement isn't listed in either requirements or test-requirements, btw? | 17:02 |
| *** helenafm has quit IRC | 17:02 | |
| stephenfin | mdbooth: Neither is neutron, I'd imagine | 17:02 |
| mdbooth | stephenfin: How's it supposed to be installed? | 17:03 |
| stephenfin | They're all applications rather than libraries. You don't list applications in requirements.txt | 17:03 |
| mdbooth | stephenfin: We pull in placement in functional | 17:03 |
| stephenfin | If you're DevStack, usually by cloning the repo locally? | 17:03 |
| mdbooth | Functional tests don't work | 17:03 |
| stephenfin | Oh, then that's different | 17:03 |
| sean-k-mooney | stephenfin: it should be in test-requiremetns if its needed to run the tests | 17:03 |
| stephenfin | sean-k-mooney: Yup. It shouldn't be required to run tests though, otherwise we now have integration tests | 17:04 |
| stephenfin | mdbooth: Then I can't offer any more info. Sorry for the noise :) | 17:04 |
| sean-k-mooney | stephenfin: we import the test fixture form placement now so we can share it between nova and placement i think | 17:05 |
| mdbooth | https://github.com/openstack/nova/blob/master/nova/tests/functional/fixtures.py#L19 | 17:05 |
| sean-k-mooney | cdent: ^ you would know better then i | 17:05 |
| * mdbooth wonders how it's working anywhere | 17:05 | |
| *** fragatina has quit IRC | 17:06 | |
| cdent | in the gate, it is happening a a result of tox-siblings | 17:06 |
| openstackgerrit | Jan Gutter proposed openstack/nova master: Convert vrouter legacy plugging to os-vif https://review.openstack.org/571325 | 17:06 |
| jangutter | mriedem: would you be able to remove your procedural -2 from ^^ (a tag has been made!) | 17:06 |
| mdbooth | Sounds cosy | 17:06 |
| sean-k-mooney | cdent: how would you run tox locally to get the same effect. just pip install placement after the fact? | 17:07 |
| cdent | one mo | 17:07 |
| cdent | sean-k-mooney, mdbooth : look at the functional job def in tox.ini | 17:08 |
| cdent | placement is installed explicitly for functional only because unit tests should not know about placement | 17:08 |
| cdent | from git master | 17:08 |
| cdent | tox-siblings makes sure in the gate that depends-on type things are respected | 17:08 |
| mriedem | jangutter: done | 17:08 |
| cdent | if you need that locally, then pip install the git ref | 17:08 |
| sean-k-mooney | cdent: mdbout is have import errors when runing placement | 17:08 |
| sean-k-mooney | * nova fucntional tests | 17:09 |
| cdent | mdbooth: got any more details than that? | 17:09 |
| jangutter | mriedem: thanks! (finally, the dominoes have started falling!) | 17:09 |
| *** sapd1_x has quit IRC | 17:10 | |
| mdbooth | cdent: Ah, ha! | 17:10 |
| mdbooth | I was running functional-py37 which looks like it's missing a line | 17:10 |
| cdent | mdbooth: it does indeed appear to be missing a line | 17:11 |
| mdbooth | py3{5,6} both have an explicit deps= referring to functional deps | 17:11 |
| cdent | i bet that's a bad merge | 17:11 |
| *** yan0s has quit IRC | 17:11 | |
| cdent | functiona-py37 was added around the same time the functional test changes were added | 17:11 |
| cdent | but functional-py37 is not in the gate | 17:11 |
| cdent | mdbooth: you happy to fix that as part of whatever else you are doing, or would like me to get it? | 17:12 |
| mdbooth | cdent: 'whatever I'm doing' will merge shortly after the decay of the last proton, so you go ahead | 17:12 |
| * cdent hugs mdbooth | 17:12 | |
| *** lpetrut has quit IRC | 17:14 | |
| *** pcaruana has joined #openstack-nova | 17:15 | |
| jangutter | mdbooth: you made me look up the current lower limit for proton decay. I think the only think that will beat that is COBOL code running in production. | 17:15 |
| mdbooth | jangutter: lol | 17:16 |
| gibi | mlavalle, sean-k-mooney, efried, mriedem: So the (hopefully) final timeslot for the bandwidth hangouts in Friday 17:00 UTC. Posted on the ML | 17:18 |
| gibi | http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001710.html | 17:18 |
| mlavalle | gibi: ack | 17:18 |
| efried | ack | 17:18 |
| * gibi needs to run away | 17:19 | |
| efried | gibi: Hangout link to be provided day-of? | 17:20 |
| *** dtantsur is now known as dtantsur|afk | 17:23 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Make functional-py37 job work like others https://review.openstack.org/630745 | 17:29 |
| *** itlinux has joined #openstack-nova | 17:33 | |
| *** lpetrut has joined #openstack-nova | 17:33 | |
| *** ccamacho has quit IRC | 17:36 | |
| *** ttsiouts has quit IRC | 17:37 | |
| *** ttsiouts has joined #openstack-nova | 17:38 | |
| stephenfin | sean-k-mooney: thoughts on https://review.openstack.org/#/c/621528/5/nova/network/linux_net.py@1455 ? | 17:41 |
| *** mvkr has quit IRC | 17:41 | |
| *** ttsiouts has quit IRC | 17:42 | |
| *** fragatina has joined #openstack-nova | 17:42 | |
| sean-k-mooney | exitcode 2 is file not found or interface not found in this case | 17:42 |
| sean-k-mooney | 254 im not sure and 0 is success | 17:42 |
| sean-k-mooney | what is that patch for | 17:43 |
| sean-k-mooney | oh its the privesep changes im not sure if that code is used as os-vif should be used even for nova-networks | 17:44 |
| stephenfin | sean-k-mooney: Aye, could be nova-network only code | 17:45 |
| sean-k-mooney | you mean unrelated to vif plugging | 17:45 |
| stephenfin | No I mean the file I'm linking to seems to be nova-network only code and unrelated to the neutron/os-vif code path | 17:46 |
| stephenfin | We don't use os-vif for nova-net, do we? | 17:46 |
| sean-k-mooney | yes we do | 17:46 |
| sean-k-mooney | os-vif should be used for both nova-networks and neutorn in the libvirt dirver | 17:47 |
| sean-k-mooney | this is the legacy linux bridge code path which i think is unused currently | 17:47 |
| sean-k-mooney | in any case you are coorect that its a change in behavior but i think exit codes [0,2,254] would have been the correct set to use in this case | 17:48 |
| jaypipes | efried: got a sec? | 17:52 |
| jaypipes | efried: re: https://review.openstack.org/#/c/617042/14/nova/compute/manager.py@555 | 17:52 |
| jaypipes | efried: I'm still super-confused why we need that. | 17:52 |
| jaypipes | efried: and why we can't just instantiate the reportclient variable on __init__ of the compute manager | 17:52 |
| jaypipes | efried: is this to minimize the number of ProviderTree caches or something? | 17:53 |
| *** derekh has quit IRC | 18:00 | |
| *** jangutter has quit IRC | 18:02 | |
| *** lpetrut has quit IRC | 18:03 | |
| jaypipes | efried: never mind... I see you've addressed it in https://review.openstack.org/#/c/620711/13 | 18:06 |
| *** panda is now known as panda|off | 18:12 | |
| mriedem | it's funny that an az description field has never been requested for the api | 18:16 |
| *** ttsiouts has joined #openstack-nova | 18:20 | |
| *** wolverineav has joined #openstack-nova | 18:22 | |
| jaypipes | mriedem: perhaps because nobody knows wtf an AZ is in Nova? | 18:25 |
| *** itlinux_ has joined #openstack-nova | 18:26 | |
| *** dave-mccowan has quit IRC | 18:26 | |
| *** itlinux_ has quit IRC | 18:28 | |
| *** itlinux has quit IRC | 18:29 | |
| *** igordc has joined #openstack-nova | 18:31 | |
| sean-k-mooney | jaypipes: its like a aws AZ right with all the semantics or a totally seperate fault domain and is tototally not a metadata key on an aggregate | 18:31 |
| efried | jaypipes: sorry was afk. Assume any remaining action items are denoted in review comments at this point? | 18:32 |
| *** dave-mccowan has joined #openstack-nova | 18:32 | |
| jaypipes | efried: yuupers. | 18:36 |
| jaypipes | yuppers | 18:36 |
| jaypipes | sean-k-mooney: lol, yeah right :) | 18:36 |
| efried | jaypipes: Thanks for the review. I'm pretty excited to get this series going. | 18:36 |
| * cdent will dance when it merges | 18:37 | |
| efried | cdent: tape that ^ | 18:37 |
| eandersson | Are there any actual dependencies on the new libvirt / qemu versions for nova-compute? | 18:40 |
| cdent | efried: no sir | 18:44 |
| *** ralonsoh has quit IRC | 18:45 | |
| *** moshele has joined #openstack-nova | 18:47 | |
| *** remi_ness has joined #openstack-nova | 18:47 | |
| *** remi_ness has quit IRC | 18:54 | |
| lennyb | jangutter, moshele. yes. internal environment . I will check it | 18:54 |
| moshele | lennyb: ? | 18:55 |
| *** mvkr has joined #openstack-nova | 18:56 | |
| mriedem | jaypipes: heh, well, all the more reason to allow a cloud operator to describe their AZs | 18:56 |
| mriedem | eandersson: what do you mean? the libvirt driver has a minimum required libvirt/qemu otherwise nova-compute won't start | 18:57 |
| mriedem | we bump that minimum every other release or so | 18:58 |
| mriedem | some features in the libvirt driver require newer versions of the binaries and those have conditional version checks on them | 18:58 |
| mriedem | e.g. file-backed memory | 18:58 |
| *** wolverineav has quit IRC | 19:04 | |
| *** pcaruana has quit IRC | 19:08 | |
| *** moshele has quit IRC | 19:22 | |
| *** remi_ness has joined #openstack-nova | 19:25 | |
| *** wolverineav has joined #openstack-nova | 19:29 | |
| *** wolverineav has quit IRC | 19:32 | |
| *** wolverineav has joined #openstack-nova | 19:32 | |
| *** ttsiouts has quit IRC | 19:41 | |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Add free for claimed, allocated devices https://review.openstack.org/616120 | 19:41 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Allow per-port modification of vnic_type and profile https://review.openstack.org/607365 | 19:42 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Add get_instance_pci_request_from_vif https://review.openstack.org/619929 | 19:42 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115 | 19:42 |
| *** ttsiouts has joined #openstack-nova | 19:42 | |
| *** jangutter has joined #openstack-nova | 19:42 | |
| *** igordc has quit IRC | 19:43 | |
| *** ttsiouts has quit IRC | 19:47 | |
| adrianc | sean-k-mooney: ^ | 19:47 |
| *** itlinux has joined #openstack-nova | 19:51 | |
| lennyb | jangutter, moshele. fixed and passed | 20:04 |
| *** itlinux has quit IRC | 20:04 | |
| *** erlon_ has quit IRC | 20:04 | |
| jangutter | lennyb: thanks very much! | 20:04 |
| *** remi_ness has quit IRC | 20:05 | |
| *** ttsiouts has joined #openstack-nova | 20:05 | |
| *** cfriesen has quit IRC | 20:07 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Share image membership with instance owner https://review.openstack.org/630769 | 20:36 |
| ade_lee | mriedem, hey - any idea who I could talk to about voume attachments? | 20:38 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Share snapshot image membership with instance owner https://review.openstack.org/630769 | 20:38 |
| mriedem | ade_lee: likely me | 20:38 |
| ade_lee | mriedem, so -- I have a gate job that is failing | 20:38 |
| ade_lee | because I'm trying to attach a volume and then use it - and there is some timing issue | 20:39 |
| ade_lee | whereby it looks like we're trying to use it before its fully attached. | 20:39 |
| ade_lee | we're waiting to see if the volume is "in-use", | 20:39 |
| *** whoami-rajat has quit IRC | 20:40 | |
| ade_lee | but thats checking if cinder thinks the volume is in use | 20:40 |
| ade_lee | and not if the instance thinks the volume is attached .. | 20:40 |
| ade_lee | so how/what do I wait on instead? | 20:40 |
| ade_lee | mriedem, the specific gate job that is failing is here -- http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/ | 20:41 |
| mriedem | that should be sufficient, | 20:41 |
| mriedem | nova tells cinder when the local connection happens for the guest vm, | 20:41 |
| mriedem | and that's what should change the volume status to in-use | 20:41 |
| smcginnis | Yeah, I didn't think we marked it as in-use until things were ready. | 20:41 |
| ade_lee | mriedem, smcginnis interesting -- any idea whats going on in the above test then? | 20:42 |
| smcginnis | Could it be that the local device is getting set wrong for some reason? http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/job-output.txt.gz#_2019-01-14_15_22_54_281033 | 20:42 |
| *** moshele has joined #openstack-nova | 20:42 | |
| ade_lee | note - the code we are executing is here -- https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/manager.py line 326 | 20:43 |
| mriedem | is something specifically waiting on /dev/vdb ? | 20:45 |
| mriedem | because nova, or at least the libvirt driver, totally ignores the user-requested device name for the attach | 20:45 |
| mriedem | so https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/manager.py#L328 is kind of pointless to pass the device kwarg | 20:45 |
| mriedem | https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/manager.py#L346 | 20:46 |
| mriedem | looks like it probably is | 20:46 |
| ade_lee | mriedem, well - that fails when we actually try to write to the device .. | 20:46 |
| smcginnis | There's only a /dev/vda | 20:46 |
| mriedem | this is the POST to attach the volume it looks like http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/job-output.txt.gz#_2019-01-14_15_21_32_860642 | 20:46 |
| ade_lee | mriedem, so this is the test -- https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py line 101 | 20:47 |
| mriedem | wtf this is nice http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_14_46_115508 | 20:47 |
| mriedem | unrelated though | 20:47 |
| mriedem | http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_20_47_770302 | 20:48 |
| mriedem | Jan 14 15:20:47.770302 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: INFO nova.virt.libvirt.driver [None req-c127b555-17c3-4ce7-9c75-b3df8b653f76 tempest-VolumeEncryptionTest-1809383190 tempest-VolumeEncryptionTest-1809383190] [instance: 1f45a2d6-3811-4b85-a13b-68c3eb605dea] Ignoring supplied device name: /dev/vdb | 20:48 |
| mriedem | says it's attaching it to /dev/vdb though | 20:49 |
| mriedem | http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_20_48_012939 | 20:49 |
| mriedem | Jan 14 15:20:48.012939 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: INFO nova.compute.manager [None req-c127b555-17c3-4ce7-9c75-b3df8b653f76 tempest-VolumeEncryptionTest-1809383190 tempest-VolumeEncryptionTest-1809383190] [instance: 1f45a2d6-3811-4b85-a13b-68c3eb605dea] Attaching volume ed7fc802-ba1c-4639-8723-47ed02bd11de to /dev/vdb | 20:49 |
| lyarwood | ade_lee: are you only seeing this with the _cryptsetup test? | 20:51 |
| mriedem | hmm, i wonder if it's connected at sda | 20:51 |
| mriedem | http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_20_51_033570 | 20:51 |
| ade_lee | lyarwood, seems so | 20:51 |
| mriedem | something sure enjoys calling the barbican secrets API a lot during this attach flow | 20:53 |
| lyarwood | ade_lee: if the _luks version of the test is passing I'd put money on there being an issue with how os-brick is wiring up the encrypted volume on the underlying host | 20:54 |
| mriedem | Jan 14 15:20:51.786765 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: DEBUG os_brick.encryptors.cryptsetup [None req-c127b555-17c3-4ce7-9c75-b3df8b653f76 tempest-VolumeEncryptionTest-1809383190 tempest-VolumeEncryptionTest-1809383190] opening encrypted volume /dev/sda {{(pid=343) _open_volume /usr/local/lib/python2.7/dist-packages/os_brick/encryptors/cryptsetup.py:109}} | 20:54 |
| mriedem | it's on sda but the test is looking for dev/vdb | 20:54 |
| lyarwood | that's on the compute host itself | 20:54 |
| mriedem | http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_20_51_786765 | 20:54 |
| mriedem | <target bus="virtio" dev="vdb"/> | 20:55 |
| ade_lee | mriedem, not sure if I understand what that means -- does that mean we asked for it to be on vdb and it put it on sda instead? | 20:56 |
| mriedem | maybe not, tempest is sshi'ing into the guest looking for it at vdb | 20:57 |
| mriedem | if the sda thing is host only, then ignore me | 20:57 |
| mriedem | lyarwood seems to know something about cryptsetup | 20:57 |
| mriedem | i remember something about that coming up in berlin too | 20:57 |
| lyarwood | unfortunately, yeah I'd like to drop support for it tbh | 20:58 |
| mriedem | L22 https://etherpad.openstack.org/p/BER-volume-encryption-forum | 20:58 |
| lyarwood | ade_lee: so os-brick currently hacks the device paths on the host to point the instance at the decrypted volume using the original encrypted device path. | 20:59 |
| ade_lee | lyarwood, so os-brick is setting it up sda instead of vdb -- and luks is setting it up on vdb? | 21:01 |
| smcginnis | I don't think so. /dev/vda should be the boot volume. | 21:02 |
| *** hamzy_ has joined #openstack-nova | 21:03 | |
| *** alanmeadows_ has joined #openstack-nova | 21:04 | |
| lyarwood | ade_lee: os-brick points /dev/disk/by-id/scsi-360000000000000000e00000000010001 to the decrypted device on the host, for some reason I can't see the actual commands in the logs anymore. | 21:05 |
| *** cdent has quit IRC | 21:07 | |
| *** tbachman_ has joined #openstack-nova | 21:09 | |
| *** tbachman has quit IRC | 21:09 | |
| *** tbachman_ is now known as tbachman | 21:09 | |
| *** alanmeadows has quit IRC | 21:11 | |
| *** mgoddard has quit IRC | 21:11 | |
| *** bzhao__ has quit IRC | 21:11 | |
| *** awaugama has quit IRC | 21:11 | |
| *** hamzy has quit IRC | 21:11 | |
| *** jroll has quit IRC | 21:11 | |
| *** alanmeadows_ is now known as alanmeadows | 21:11 | |
| *** awaugama has joined #openstack-nova | 21:11 | |
| *** jroll has joined #openstack-nova | 21:12 | |
| ade_lee | lyarwood, sorry - did you find anything? I'm looking at a couple runs -- one where it worked and one where it didn't - but not sure how to make heads or tails of it | 21:13 |
| lyarwood | ade_lee: yeah sorry I can't see anything obvious on the host / os-brick side of things tbh | 21:13 |
| *** trident has quit IRC | 21:13 | |
| lyarwood | ade_lee: /dev/disk/by-id/scsi-360000000000000000e00000000010001 should point to the decrypted device that in turn points to encrypted /dev/sda | 21:14 |
| ade_lee | lyarwood, well sdb right? | 21:15 |
| *** mgoddard has joined #openstack-nova | 21:15 | |
| *** efried has quit IRC | 21:16 | |
| *** trident has joined #openstack-nova | 21:16 | |
| *** efried has joined #openstack-nova | 21:17 | |
| lyarwood | ade_lee: nope, this is on the host sorry | 21:18 |
| lyarwood | ade_lee: like I said there's a load of hacks with this legacy encryptor to decrypt the volume on the host itself | 21:19 |
| ade_lee | lyarwood, it seems to work intermittently .. so not sure whats going on | 21:19 |
| *** mrjk has quit IRC | 21:19 | |
| lyarwood | ade_lee: hard to say but if we can't read/write to the device once it's connected within the instance then I'd start by looking at the devicemapper layer on the compute host itself. | 21:21 |
| lyarwood | ade_lee: if it's intermittent it might be a race between os-brick connecting another device while also setting up the decrypt devices for this volume. | 21:23 |
| ade_lee | lyarwood, would that show up in the logs somehow? | 21:24 |
| efried | mriedem: Doesn't deleting the compute service remove its node provider(s) from placement? | 21:25 |
| lyarwood | ade_lee: yeah, os-brick keeps reporting the device path as /dev/disk/by-id/scsi-360000000000000000e00000000010001 once each volume is connected on the host. | 21:27 |
| lyarwood | ade_lee: we use `scsi-360000000000000000e00000000010001` as part of the device name when using cryptsetup to decrypt | 21:28 |
| *** xek has quit IRC | 21:28 | |
| *** mrjk has joined #openstack-nova | 21:31 | |
| lyarwood | ade_lee: I need to head offline now but it does look like we are overloading the /dev/disk/by-id/scsi-360000000000000000e00000000010001 path during these tests somehow | 21:31 |
| lyarwood | ade_lee: do you have an env to hand btw? | 21:31 |
| lyarwood | ade_lee: I'd be interested to see what `qemu-img info /dev/vdb` has to say within the instance | 21:32 |
| ade_lee | lyarwood, no - I've just been trying to run in the gate .. | 21:32 |
| *** moshele has quit IRC | 21:32 | |
| lyarwood | ade_lee: kk, I can take another look at this during the morning in EMEA if it ends up in a bug somewhere. | 21:32 |
| ade_lee | lyarwood, thanks - I'll send you an email and/or open a bug | 21:33 |
| ade_lee | lyarwood, oh -- this is interesting .. | 21:34 |
| ade_lee | Jan 14 15:21:51.361034 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: WARNING os_brick.encryptors.luks [None req-b8bc15a0-3faf-41ee-bda7-65ede3c3e432 tempest-VolumeEncryptionTest-1809383190 tempest-VolumeEncryptionTest-1809383190] isLuks exited abnormally (status 1): Device /dev/disk/by-id/scsi-360000000000000000e00000000010001 is not a valid LUKS device. | 21:34 |
| ade_lee | Jan 14 15:21:51.361319 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: Command failed with code 22: Device /dev/disk/by-id/scsi-360000000000000000e00000000010001 is not a valid LUKS device. | 21:34 |
| ade_lee | Jan 14 15:21:51.361579 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: : ProcessExecutionError: Unexpected error while running command. | 21:34 |
| *** igordc has joined #openstack-nova | 21:35 | |
| lyarwood | ade_lee: as horrid as that is it's expected the first time we connect to an empty encrypted volume | 21:35 |
| ade_lee | lyarwood, ok :) | 21:35 |
| lyarwood | ade_lee: as c-vol hasn't had to write and thus encrypt any data to the volume | 21:35 |
| mriedem | efried: should | 21:35 |
| mriedem | efried: are you on master? | 21:35 |
| mriedem | and is the api configured to talk to placement? | 21:35 |
| lyarwood | ade_lee: so n-cpu comes along and ensures the LUKs / crypt headers are written to the volume before we use it | 21:36 |
| efried | mriedem: I found the code. It looks like for ironic deployments, it'll only delete the "first" compute node (whatever that means in practice). | 21:36 |
| efried | mriedem: cf https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/api/openstack/compute/services.py#L244-L247 and https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/objects/service.py#L308-L311 | 21:36 |
| mriedem | efried: sure enough | 21:37 |
| mriedem | bugify it | 21:37 |
| efried | mriedem: Well, a) I can't prove it (not having an ironic setup); but b) as jaypipes points out here, we might actually not want to do that: https://review.openstack.org/#/c/615677/17//COMMIT_MSG@30 | 21:38 |
| efried | but I can open a bug anyway :) | 21:39 |
| mriedem | efried: well, the "has instances on them" is caught here https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/api/openstack/compute/services.py#L232 | 21:40 |
| *** mrjk has quit IRC | 21:40 | |
| efried | mriedem: Yeah, I didn't see that pre-check, but knew placement would bounce the delete in any case if we got that far. | 21:41 |
| mriedem | yeah that's also a relatively recent validation | 21:41 |
| efried | mriedem: But do we really want to delete "potentially thousands" of compute node records from placement, assuming they're non-spawned? | 21:42 |
| *** mrjk has joined #openstack-nova | 21:42 | |
| mriedem | ironic is special since we use the ironic node uuid to set the compute node uuid | 21:42 |
| mriedem | so we can actually link those | 21:42 |
| mriedem | which is also relatively recent... | 21:42 |
| mriedem | rocky or queens | 21:42 |
| mriedem | otherwise if you delete the service record and the compute_nodes records, the providers in placement would be orphaned if you restart the compute service which creates new compute nodes records with new uuids | 21:43 |
| efried | mriedem: I saw code elsewhere to delete those orphans. | 21:43 |
| mriedem | or just simply forget to stop the nova-compute service... | 21:43 |
| efried | seems like a bass-ackwards way off doing it. | 21:43 |
| efried | s/ff/f/ | 21:43 |
| mriedem | says the guy that +2d the change https://review.openstack.org/#/c/554920/ | 21:44 |
| * efried wipes egg | 21:45 | |
| mriedem | so let's say we delete a compute service hosting no instances and we delete 1000 ironic compute nodes records and related resource providers, | 21:46 |
| mriedem | the next time you restart that compute service (or update_available_resources runs), nova-compute will just create 1000 new compute nodes records with uuids that match the ironic nodes that service is hosting, | 21:46 |
| mriedem | and create 1000 new resource providers with those same uuids | 21:46 |
| mriedem | if we don't delete the resource providers, the compute_nodes records are gone either way i think | 21:47 |
| mriedem | when the services table record is deleted | 21:47 |
| mriedem | because of this https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/db/sqlalchemy/api.py#L406 | 21:48 |
| efried | If I'm following, you're saying that deleting the service is going to cause deletion-and-recreation of all the ironic nodes even today? Just with different-than-optimal timing? | 21:48 |
| mriedem | now if for some reason that service is only hosting 800 of those ironic nodes, n-cpu will create 800 compute_nodes records and you'll have orphaned 200 resource providers in placement | 21:48 |
| mriedem | it will delete all of the compute_nodes table records for that service, yes | 21:49 |
| mriedem | and always did | 21:49 |
| mriedem | if some other compute service starts managing that other 200 nodes, then cool - their providers are already in placement | 21:49 |
| mriedem | but if not, you're reporting things to the scheduler that might not be managed anywhere | 21:49 |
| *** awaugama has quit IRC | 21:50 | |
| mriedem | the scheduler should filter those out but you could get NoValidHost if you're like CERN and have a real low config for how many placement allocation candidates you want to get back | 21:50 |
| efried | I will try to express this in the bug report. | 21:51 |
| *** eharney has quit IRC | 21:51 | |
| mriedem | i could be wrong of course, i don't have a multinode ironic deployment to play with | 21:52 |
| mriedem | but we could easily simulate this in functional tests | 21:52 |
| mriedem | i guess the bug is, in the case of ironic, we should either delete 0 or all providers in placement for the related compute nodes | 21:52 |
| mriedem | sure is fun needing nova to mirror everything to placement isn't it o-) | 21:53 |
| efried | mriedem: https://bugs.launchpad.net/nova/+bug/1811726 -- please see if I expressed this correctly, kthx | 21:58 |
| openstack | Launchpad bug 1811726 in OpenStack Compute (nova) "Deleting compute service only deletes "first" ironic node from placement" [Undecided,New] | 21:58 |
| mriedem | commented | 22:03 |
| mriedem | but i don't think that cleanup orphan code you pointed out is going to clean up orphans | 22:03 |
| mriedem | that code in update_available_resource is for nodes that the driver no longer reports (user deleted the ironic node in the ironic api) but nova was still tracking it | 22:04 |
| *** jangutter has quit IRC | 22:05 | |
| *** imacdonn has quit IRC | 22:06 | |
| *** imacdonn has joined #openstack-nova | 22:06 | |
| openstackgerrit | Jack Ding proposed openstack/nova master: [WIP] Flavor extra spec and image properties validation https://review.openstack.org/620706 | 22:07 |
| *** jangutter has joined #openstack-nova | 22:20 | |
| * efried needs to never again let vacation time accumulate | 22:23 | |
| *** jangutter has quit IRC | 22:26 | |
| *** rcernin has joined #openstack-nova | 22:31 | |
| *** itlinux has joined #openstack-nova | 22:47 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Share snapshot image membership with instance owner https://review.openstack.org/630769 | 22:52 |
| jaypipes | mriedem: unless I'm badly mistaken, if you attempt to delete a resource provider and the provider has allocations against it, that will fail. | 22:55 |
| efried | mriedem: When you get a chance, would you mind having a quick look at https://review.openstack.org/#/c/615677/17/nova/compute/resource_tracker.py@824 ? | 22:55 |
| efried | Perhaps you can explain a circumstance under which is_new_compute_node==True but the report client's provider tree cache has an entry for the "new" node. | 22:55 |
| mriedem | jaypipes: yes i believe that's a 409 | 22:55 |
| edleafe | git ls | 22:55 |
| jaypipes | mriedem: and we are catching that 409 in the DELETE /os-services/{id} API? | 22:55 |
| edleafe | doh! | 22:55 |
| efried | cause I'm sure I put that code there for a reason. In fact, I want to say it was the/a main thing in this patch. | 22:56 |
| mriedem | jaypipes: we don't attempt to delete services/computenodes/resoruce providers if the service host has instances on it | 22:56 |
| mriedem | jaypipes: https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/api/openstack/compute/services.py#L232 | 22:56 |
| jaypipes | mriedem: ack. | 22:57 |
| mriedem | we don't explicitly catch that 409 though no, | 22:57 |
| mriedem | it'd be a 500 if we raced there | 22:57 |
| jaypipes | right. | 22:57 |
| mriedem | but we also wouldn't delete the service record or compute nodes table entries, so you could retry | 22:57 |
| jaypipes | mriedem: which I'm fine with, really, since it's a data corruption error if that happened IMHO (no InstanceList.get_by_service() results but there ARE allocations is a corruption in my book) | 22:58 |
| mriedem | efried: replied inline | 23:00 |
| *** mdbooth_ has joined #openstack-nova | 23:02 | |
| efried | thanks | 23:03 |
| *** mchlumsky has quit IRC | 23:03 | |
| *** mdbooth has quit IRC | 23:04 | |
| *** mriedem is now known as mriedem_away | 23:07 | |
| *** efried has quit IRC | 23:08 | |
| *** mlavalle has quit IRC | 23:11 | |
| *** mmethot_ has quit IRC | 23:14 | |
| *** efried has joined #openstack-nova | 23:15 | |
| *** ttsiouts has quit IRC | 23:47 | |
| *** ttsiouts has joined #openstack-nova | 23:47 | |
| *** ttsiouts has quit IRC | 23:52 | |
| *** erlon_ has joined #openstack-nova | 23:53 | |
| *** hongbin has quit IRC | 23:57 | |
| *** markvoelker has joined #openstack-nova | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!