Tuesday, 2021-08-31

opendevreviewMerged openstack/nova master: [func test] neutron fixture for extended resource request  https://review.opendev.org/c/openstack/nova/+/79430600:52
opendevreviewMerged openstack/nova master: Detect port-resource-request-groups neutron API extension  https://review.opendev.org/c/openstack/nova/+/79361800:52
opendevreviewMerged openstack/nova master: Reject server create with extended resource req  https://review.opendev.org/c/openstack/nova/+/79361900:52
opendevreviewMerged openstack/nova master: Reject server operations with extended resource req  https://review.opendev.org/c/openstack/nova/+/79362000:53
opendevreviewMerged openstack/nova master: Add same_subtree field to RequestLevelParams  https://review.opendev.org/c/openstack/nova/+/79150301:35
opendevreviewMerged openstack/nova master: Bump min placement microversion to 1.36  https://review.opendev.org/c/openstack/nova/+/79150401:35
opendevreviewMerged openstack/nova master: Support same_subtree in allocation_canadidate query  https://review.opendev.org/c/openstack/nova/+/79150501:35
opendevreviewMerged openstack/nova master: [func test] refactor assertPortMatchesAllocation  https://review.opendev.org/c/openstack/nova/+/79245801:35
opendevreviewMerged openstack/nova master: [func test] refactor asserts in qos tests  https://review.opendev.org/c/openstack/nova/+/79893001:35
opendevreviewMerged openstack/nova master: [func test] ports with both bw and pps resources  https://review.opendev.org/c/openstack/nova/+/79239401:35
opendevreviewmelanie witt proposed openstack/nova master: Add logic to enforce local api and db limits  https://review.opendev.org/c/openstack/nova/+/71213905:34
opendevreviewmelanie witt proposed openstack/nova master: Enforce api and db limits  https://review.opendev.org/c/openstack/nova/+/71214205:34
opendevreviewmelanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits  https://review.opendev.org/c/openstack/nova/+/71214305:34
opendevreviewmelanie witt proposed openstack/nova master: Update limit APIs  https://review.opendev.org/c/openstack/nova/+/71270705:34
opendevreviewmelanie witt proposed openstack/nova master: Update quota sets APIs  https://review.opendev.org/c/openstack/nova/+/71274905:34
opendevreviewmelanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources  https://review.opendev.org/c/openstack/nova/+/71330105:34
opendevreviewmelanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit  https://review.opendev.org/c/openstack/nova/+/61518005:34
opendevreviewmelanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits  https://review.opendev.org/c/openstack/nova/+/71349805:34
opendevreviewmelanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage  https://review.opendev.org/c/openstack/nova/+/71349905:34
opendevreviewmelanie witt proposed openstack/nova master: Add reno for unified limits  https://review.opendev.org/c/openstack/nova/+/71527105:34
bauzasgood morning Nova07:03
* bauzas is eventually released from his PTOs :p07:03
gibibauzas: welcome back07:04
gibibauzas: have you managed to recharge your batteries?07:04
gthiemongeHi Folks, do you know some recent issues with placement? the Octavia gates have been failing since the end of last week, with some exceptions in the placement-api logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-placement-api.txt07:05
bauzasgibi: let me see07:05
bauzasgibi: 80% :p07:05
gibibauzas: that looks good :)07:05
gthiemongelogstash shows that it started on 2021-08-25 in many projects: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22placement.exception.ProjectNotFound%3A%20No%20such%20project%5C%2207:06
bauzasgibi: you as well ?07:06
bauzasgthiemonge: looking07:07
gibigthiemonge: looking07:07
gibibauzas: sure, I had a nice weekend with a good book07:08
gibigthiemonge: 2/3rd of the hits are actually successful job runs07:09
gibigthiemonge: are you sure this is the root cause of the failed ones?07:09
gthiemongegibi: yeah, for Octavia at least07:10
gibigthiemonge: could you link an octavia failre please?07:10
gthiemongegibi: yes sure07:12
gthiemongeoctavia worker logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-o-cw.txt07:12
bauzasgibi: do you know why we persist the project UUIDs ? for the unified-limits  or for the consumer types ?07:12
gthiemongeerror is at Aug 30 13:07:06.582519 with a "{'code': 500, 'created': '2021-08-30T13:07:05Z', 'message': 'No valid host was found. There are not enough hosts available.',"07:13
gibibauzas: we have projecti and user id for a log time in placement07:13
gthiemongenova scheduler: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-n-sch.txt07:13
gthiemongeat Aug 30 13:07:04.212245 there's a Internal Server Error07:13
gibibauzas: but I don't think placement uses it directly07:13
gibibauzas: so possible yes, for unified limits so you can group usages per project07:14
gibigthiemonge: thanks, looking07:14
gthiemongegibi: if that helps: it fails on Octavia when we request 2 server creations at the same time, it doesn't fail when creating one server07:15
gibigthiemonge: thanks, this will be a race07:15
gibiwe changed transactional behavior of allocation update  in https://review.opendev.org/q/topic:bp/support-consumer-types 07:16
bauzasgibi: surely for unified limits, but l looked at the series and nothing merged yet07:17
bauzasgibi: yeah, hence my confusion : we merged stuff for consumer types but none of them touched the projects table07:18
gibibauzas: here you can query usages per project https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages07:18
bauzasoh now I remember07:19
bauzasgibi: gotcha, thanks07:19
gibibauzas: but consumer_types changes how the allocation transaction handling is done07:19
gibibauzas: before consumer_type we had actually two transactions per allocation update, after consumer_types we have one07:19
gibithe duplicate project entry (due to a race) makes the new single transaction session invalid and later operations are failed in the same transaction hence the sqlalchemy.exc.InvalidRequestError: This session is in 'inactive' state, due to the SQL transaction being rolled back; no further SQL can be emitted within this transaction.07:21
gibithat is my running assumption now07:21
bauzasyup, I agree07:22
gibiI have to go offline for ~2 hours07:23
bauzasI need to filter 100+ emails but I guess you found something07:23
gibigthiemonge: could you please file a bug for placement about it?07:23
gthiemongegibi: yes, thanks!07:23
gibibauzas: https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/handlers/util.py#L56 this is the first exception07:25
gibiand that is the second Aug 30 13:07:04.208104 nested-virt-ubuntu-focal-ovh-bhs1-0026183953 devstack@placement-api.service[89049]: ERROR placement.fault_wrap     proj = project_obj.Project.get_by_external_id(ctx, project_id)07:26
gibisorry wrong buffer07:26
gibiso the second https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/handlers/util.py#L5807:26
gibithis is strange I don't think consumer types changed this exact logic07:26
gibihttps://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/util.py#b65 yepp the logic is the same in the from state https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/util.py#b6507:28
gthiemongegibi: https://storyboard.openstack.org/#!/story/200915907:29
gibibut yes, the Project.create() and the Project.get_by_external_id was two separate transaction before the consumer_types 07:30
gibithis is the start of the new and only transaction https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/allocation.py#42407:30
gibigthiemonge: thanks07:30
gibiright now I don't know how to solve this. as we need one big transaction but we should not stop due to a project creation race07:35
gibimaybe we need to re-try the whole transaction from the start if the race is caugth07:35
gibiand now I go offline, back around 11:30 CEST07:36
opendevreviewFederico Ressi proposed openstack/nova master: Write requests failure details to log  https://review.opendev.org/c/openstack/nova/+/80668307:42
opendevreviewFederico Ressi proposed openstack/nova master: [WIP] Investigate on Nova quota limits problem  https://review.opendev.org/c/openstack/nova/+/80668308:03
*** whoami-rajat is now known as Guest587908:10
*** whoami-rajat__ is now known as whoami-rajat08:10
opendevreviewMerged openstack/nova master: docs: admin/networking rename neutron_tunneled to neutron_tunnel  https://review.opendev.org/c/openstack/nova/+/80619708:37
*** akekane_ is now known as abhishekk09:02
* gibi is back09:37
gibibauzas, melwitt: I will try to add a reproduction test for https://storyboard.openstack.org/#!/story/200915909:46
bauzasack09:47
gibistephenfin, lyarwood: a quick re-review on the pps series would be much appreciated :) (starting here https://review.opendev.org/c/openstack/nova/+/800086 )09:49
lyarwoodack done09:54
kevinzkashyap: sbauza: Hi,  bother again :-) could you help to review this live migration patch?live migration on arm64 patch,   https://review.opendev.org/c/openstack/nova/+/763928, the comments has been addressed.  really appreciated! 09:58
kevinzbauzas: ^^,  thanks for reviewing10:01
gibilyarwood: thanks10:02
*** bhagyashris_ is now known as bhagyashris10:07
kashyapkevinz: Hi10:13
kashyapkevinz: Just catching up after PTO; will look today10:14
bauzaskevinz: ditto as kashyap, I'm just back from a long period of inactivity10:23
bauzasthis reminds me actually... core reviews would be appreciated for https://review.opendev.org/c/openstack/nova/+/80291810:24
bauzasand the series10:24
kevinzkashyap: bauzas: Thanks! Really appreciated :-)10:30
opendevreviewMerged openstack/nova master: nova-manage: Introduce volume show, refresh, get_connector commands  https://review.opendev.org/c/openstack/nova/+/80063410:54
opendevreviewMerged openstack/placement master: Refactor consumer type methods for readability  https://review.opendev.org/c/openstack/placement/+/80603510:54
opendevreviewMerged openstack/nova stable/wallaby: Reduce mocking in test_reject_open_redirect for compat  https://review.opendev.org/c/openstack/nova/+/80309210:55
gibi /1211:21
opendevreviewBalazs Gibizer proposed openstack/placement master: Add reproducer for Project creation race bug  https://review.opendev.org/c/openstack/placement/+/80673011:46
gibimelwitt, bauzas: ^^ it is the reproducer to the issue octavia folks reported today. I could not create a real race in the func test env so I cheated a but11:47
gibibit11:47
opendevreviewStephen Finucane proposed openstack/nova master: tests: Address nits for configurable-instance-hostnames series  https://review.opendev.org/c/openstack/nova/+/80673512:22
gibisean-k-mooney: what is the expected behavior of a sriov interface attach in an env where the PCI devices are part of numa0 but the instance cpus are part of numa1?12:24
sean-k-mooneyit depend on if the vm has a numa toplogy and the numa policy12:25
sean-k-mooneyif the vm has no numa toplogy its fine it should boot12:25
sean-k-mooneyif it it has cpu pinning it should not boot unless you use pci_numa_toplogy=prefer12:25
sean-k-mooneywhich will prefer a host where affinity can be acive but not enforce it on the host12:26
gibisean-k-mooney: what about pci_numa_toplogy=legacy?12:26
sean-k-mooneylegacy means if the device report numa info its enforced but if it does not it allowed regardless of the cpu selected12:27
sean-k-mooneyso legacy which is the default rquired numa affinity if we have numa info for the device and the cpus/memory12:27
gibiOK, so In my internal case the VM has topology due to cpu pinning and they not specify policy so they get the legacy one 12:28
sean-k-mooneyyep12:28
gibiand the pci device has numa info so the basically get strict affinity12:29
gibiwith hw:pci_numa_affinity_policy=preferred they would get best effort affinity12:29
sean-k-mooneyyep12:29
sean-k-mooneywe have test for this in whitebox by the way 12:29
sean-k-mooneyhttps://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute/test_sriov.py#L221-L43212:29
sean-k-mooneywe also have func tests for this i think in nova12:30
sean-k-mooneypreffer is basically currenlty implemtned as a weigher12:30
sean-k-mooneyi have a very long running todo to make prefer prefer numa nodes on the comptue too12:30
sean-k-mooneyits a simple chagne just never got around to do it12:31
sean-k-mooneyto implement prefer on the compute node we litrally just need an else on https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2201-L2205 that sorts in teh reverse order12:32
sean-k-mooneygibi: for backwards compatiablity reasons we made legacy the default but prefer is arguable a more user friendly default12:33
gibiack thanks12:33
gibisean-k-mooney: do we have the per port numa affinity supported by nova?12:34
gibii.e. the nova side of https://bugs.launchpad.net/neutron/+bug/188679812:34
sean-k-mooneyyes12:34
sean-k-mooneyi need to check which release but we do in wallaby 12:34
sean-k-mooneymaybe victoria12:34
gibiso for interface attach the user could set the policy on the port if the instace was booted without the flavor extra sepc12:35
gibispec12:35
sean-k-mooneyhttps://github.com/openstack/nova-specs/blob/master/specs/wallaby/implemented/port-scoped-sriov-numa-affinity.rst12:35
gibicool, thanks12:35
sean-k-mooneyya i think we covered that usecase when you added sriov attach support too12:35
sean-k-mooneyboth feature were added in wallaby and im 99% sure that we pass the posicy when trying to claim the pci device on attach and will reject it if the numa affintiy cant be achived12:37
sean-k-mooneybut you can relax the policy with the neutron port12:37
sean-k-mooneyso the vm can default to legacy or strict and then you can add a port with prefer12:37
gibicool. I do remember passing policy around in sriov attach impl12:39
sean-k-mooneyyep we do https://github.com/openstack/nova/blob/649b2452da07c25e1251c6185552278f91b7c517/nova/compute/manager.py#L7788-L779412:40
sean-k-mooneyhehe ya was just finding that in the code12:40
sean-k-mooneythat gets the flavor/image policy then we override that with the port policy if set https://github.com/openstack/nova/blob/649b2452da07c25e1251c6185552278f91b7c517/nova/network/neutron.py#L2023-L202612:43
sean-k-mooneyhttps://github.com/openstack/nova/blob/649b2452da07c25e1251c6185552278f91b7c517/nova/network/neutron.py#L208412:43
sean-k-mooneyhttps://github.com/openstack/nova/blob/649b2452da07c25e1251c6185552278f91b7c517/nova/network/neutron.py#L2162-L216412:43
sean-k-mooneygibi: i rememebr discussing the precednece of these two during the spec review12:44
opendevreviewsean mooney proposed openstack/nova master: Add 'hw:vif_multiqueue_enabled' flavor extra spec  https://review.opendev.org/c/openstack/nova/+/79235612:47
opendevreviewsean mooney proposed openstack/nova master: docs: Document virtio-net multiqueue  https://review.opendev.org/c/openstack/nova/+/79236212:47
opendevreviewsean mooney proposed openstack/nova master: Move 'hw:pmu', 'hw_pmu' parsing to nova.virt.hardware  https://review.opendev.org/c/openstack/nova/+/79236412:47
sean-k-mooneystephenfin: i have adressed my nits regarding the acidental cahnge form 403 to 500 and resolved the merge conflict otherwise this is as you left it12:48
sean-k-mooneyi think those should be good to merge now if people have time to review12:48
kashyapkevinz: Looked at it; I see you've also done real tests with it.  I don't see anything glaring in there.  Also the conditional is AArch64 specific.  Thx for reworking.12:59
bauzassean-k-mooney: correct me if I'm wrong, but I see a last revision of generic mdevs that is only due for fixing pep8 mypy issues because we were not explicitely returning None ?13:02
bauzasplus the fact you moved the documentation bit out of the last change into a specific WIP change, right?13:02
sean-k-mooneybauzas: yes13:02
sean-k-mooneystephenfin: was ok with adressing the doc update in a seperate change so suggested we split it out to not block the feature13:03
sean-k-mooneysince the doc update can happen after FF13:03
sean-k-mooneybauzas: effectivly what stephenfin  would like is if we have  a stuf file that explains vgpu are now implemented in terms of generic mdevs and then links to mdev doc13:04
sean-k-mooneybauzas: if we want to have specific vgpu docs we can keep them int vgpu doc13:04
sean-k-mooneybauzas: but ya i did not actully change any of your codes functionality form where you left it just the mypy update13:05
bauzassean-k-mooney: no worries at all, my question was just for understanding the touched bits, and to make sure I wasn't missing crucial bits13:08
bauzassean-k-mooney: thanks for having followed up while I was working on my tan13:09
sean-k-mooneyi ment to get to it sooner but i did not see the pep8 failure until recently13:09
bauzas(which was a difficult effort as I discovered to having traveled to the worst raining place in France apparently)13:09
bauzasa perfect Irish weather for the first week13:10
sean-k-mooneyhaha we hit 27+ a few days while you were off13:13
gibinow that is perfect ^^ :)13:13
kevinzkashyap: Thanks a lot! Good to hear that!13:14
sean-k-mooneyits been a pretty good summber all things considered in ireland this year at least on teh weather front13:14
bauzassean-k-mooney: are you honestly saying I got an irish weather while you were getting a mediterrean weather ?13:15
bauzaslife is unfair13:15
sean-k-mooneyvery much so13:15
bauzaspro-tip : we usually joke about french Britain rainy weather13:15
bauzasbut please reconsider the basque country as more wet than britain :)13:16
sean-k-mooneywith that said my tomatoes got hit by blight last week and are all starting to die but we still got some of the harvest13:16
sean-k-mooneyits the result of having heat then lots of humitity for 2 and ah half weeks13:17
bauzassean-k-mooney: hah, I'm jealous13:17
bauzaswe got a ridiculous harvest this year for our tomatoes13:17
bauzasbut, let me ask one question : any series to look at now besides kevinz's ask ?13:18
sean-k-mooneyi think we have got like 2-3 kg form 20-25 plants which is not greate but i know some have just had there entire lot of plants wiped out13:18
bauzasgibi: I guess I'm all done for you with your pps series, right?13:18
sean-k-mooneybauzas: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting13:18
opendevreviewFederico Ressi proposed openstack/nova master: [WIP] Investigate on Nova quota limits problem  https://review.opendev.org/c/openstack/nova/+/80668313:19
sean-k-mooneyi just put a few on the meeting adgenda13:19
sean-k-mooneybauzas: the review priortiy lable merged a few days ago so im suggeesting we start using that to track them13:19
gibibauzas: I have my reviews for the pps so you can focus on other things 13:19
bauzassean-k-mooney: excellent13:20
bauzasgibi: that's the question I'm asking, I don't know what other things to do13:20
opendevreviewFederico Ressi proposed openstack/nova master: [WIP] Investigate on Nova quota limits problem  https://review.opendev.org/c/openstack/nova/+/80668313:20
bauzasunified limits would get my interest, but we're two days from FF, and this is unfortunate I didn't had time to look at it before13:21
gibibauzas: things in NeedsCodeReview state are a good candidates https://launchpad.net/nova/+milestone/xena-313:21
bauzaswill find candidates, no worries13:21
gibiack13:21
sean-k-mooneyoh thats a good point13:22
gibiand now I remember to update the meeting agenda ... :D13:22
opendevreviewFederico Ressi proposed openstack/nova master: [WIP] Investigate on Nova quota limits problem  https://review.opendev.org/c/openstack/nova/+/80668313:23
opendevreviewMerged openstack/nova master: Provide the mdev class for every PCI device  https://review.opendev.org/c/openstack/nova/+/80291813:44
opendevreviewMerged openstack/nova master: Provide and use other RCs for mdevs if needed  https://review.opendev.org/c/openstack/nova/+/80323313:44
gibisean-k-mooney: thanks for the review priority links on the agenda. I move that to a new section right after the Release Planning topic13:47
sean-k-mooneygibi: ack that was also going to be somethign i brought up. shoudl it be in its own section so cool.13:48
gibigmann: if you have time, could you please check back to the instance hostname series https://review.opendev.org/c/openstack/nova/+/778550 I think it is ready to land13:54
gmanngibi: sure, +w. 14:03
gibigmann: thanks, +1 feature for Xena \o/14:03
gmann+114:03
opendevreviewBalazs Gibizer proposed openstack/nova master: Support interface attach / detach with new resource request format  https://review.opendev.org/c/openstack/nova/+/80008914:10
opendevreviewBalazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place  https://review.opendev.org/c/openstack/nova/+/79362114:11
opendevreviewBalazs Gibizer proposed openstack/nova master: [nova-manage]support extended resource request  https://review.opendev.org/c/openstack/nova/+/80206014:13
opendevreviewBalazs Gibizer proposed openstack/nova master: Reno for qos-minimum-guaranteed-packet-rate  https://review.opendev.org/c/openstack/nova/+/80504614:13
*** whoami-rajat__ is now known as whoami-rajat14:15
sean-k-mooneystephenfin: just an fyi im going to rebase https://review.opendev.org/c/openstack/nova/+/669411 and adress gibi comment, this looks ok to me so should be an easy win14:30
sean-k-mooneyill see how hard it is to add that final check after binding14:31
gibiI only skimmed that patch, but if you make it fresh then I can try to look at the 14:32
sean-k-mooneyits most of the way there if it misses its not the end of the world but also its small and low risk so if we can get it done then its a nice to have14:32
sean-k-mooneyim currently goign trhough the set of blueprint and patches that seam close and seeing if i can get them over the line14:33
sean-k-mooneybut this is lower on my list of priorities 14:33
gibiack14:35
opendevreviewMerged openstack/nova master: Remove module level caching  https://review.opendev.org/c/openstack/nova/+/80639414:52
spatelis anyone running DPDK on Intel X710 nic, i would like to buy so looking for some good feedback ?15:04
spatelcurrently i am using  Intel® 82599 10 Gigabit Ethernet  but it has limitation of rx/tx queue which is max=2 per VF 15:10
gibifyi nova weekly meeting starts in 10 minutes here in the channel15:50
sean-k-mooneyspatel: x710 used ot be the primaly nic that intel used for dpdk development so its fully supported but its a few years old now and they have proably moveed tothe 800 seriese they lauched last year15:51
sean-k-mooneyspatel: this is the dpdk docs for that nic https://doc.dpdk.org/guides/nics/i40e.html15:52
spateldo you know how many rx/tx queue does x710 support ?15:52
spatelx710 is older so may be cheaper for me to get it up and running with dpdk 15:52
sean-k-mooneyx710 is much much newer then 8259915:52
sean-k-mooneybut i know the queue count is actul configurable in the driver15:53
sean-k-mooneyim not sure what its set too by default for VFs15:53
spatel82599 has hard limit for 2 rx/tx queue per VF 15:53
sean-k-mooneyah Reserved number of Queues per VF (default 4)15:53
spatelLook like x710 has 16 per VF (default 4) but we can change it so that is good 15:53
sean-k-mooneyhttps://doc.dpdk.org/guides/nics/i40e.html#runtime-config-options15:54
spatelnic look like x710 is good choice for me in low cost 15:54
sean-k-mooneythe 72x revision are nic if you need 25G or think will upgrade at some point15:55
sean-k-mooneythey are not much more expensive and also are slightly more power effeinct15:55
spatel10G is more then enough, my whole requirement is to run DPDK and remove SRIOV dependency which doesn't support security-group and bonding 15:56
spatelhmm 15:56
spatellet me see what i can get because all depend on budget but good to know15:57
spatelsean-k-mooney i have successfully deploy SRIOV with OVN and its working great15:57
sean-k-mooneygood to know15:58
sean-k-mooneydo not use it for PF passhtouhg however15:58
sean-k-mooneywe need to remove that at some point15:58
spatelremove it from where? NOVA?15:58
sean-k-mooneyovn15:59
sean-k-mooneyit allows vnic_type=direct-physical but does not manage it properly15:59
spatelI have question for you how does OVN provide redundancy for network node? currently my compute nodes are network node so lets say i have 10 compute nodes then how OVN schedule gateway chassis?15:59
bauzas8 seconds before the meeting ;)16:00
bauzasso, wait a bit for the answer :p16:00
spatel:)16:00
sean-k-mooneythat proably better asked in neutron too16:00
bauzasgibi: you're late16:00
spatelthank you!16:00
gibi#startmeeting nova16:01
opendevmeetMeeting started Tue Aug 31 16:01:25 2021 UTC and is due to finish in 60 minutes.  The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
opendevmeetThe meeting name has been set to 'nova'16:01
gibiI was just reading the scrollback about the nics16:01
gibi:)16:01
gibio/16:01
gmanno/16:01
elodilleso/16:02
dansmitho/16:02
gibi#topic Bugs (stuck/critical) 16:02
gibiwe have one bug marked critical16:02
gibi     https://bugs.launchpad.net/nova/+bug/1940555 around SQLAlchemy URL handling and the fix is going through the gate: https://review.opendev.org/c/openstack/nova/+/80566316:02
bauzas\o16:03
gibiit is bounced since I wrote the agenda but I requeued it16:03
sean-k-mooneyack16:03
gibi#link 17 new untriaged bugs (+6 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New16:03
gibiI had no time to check the untriaged bugs in the last couple of days16:04
gibiwe have 0 bugs marked with xena-rc-potential tag #link https://bugs.launchpad.net/nova/+bugs?field.tag=xena-rc-potential16:04
gibiplease start marking release critical bugs with xena-rc-potential tag16:04
bauzastbc, those need to be regression bugs16:04
gibiyepp16:04
bauzasie. something that worked before Xena but now has issues with this release16:05
bauzaswe only have 2 weeks before RC1 in order to merge bugfixes that *aren't* regressions16:05
bauzasor those would be fixed in the next Yoga release16:05
gibiand backported :)16:05
sean-k-mooneyi dont think we currenlty have any critical bugs that would need the rc flag16:06
bauzasso, pings are appreciated besides classic feature review requests16:06
bauzaswe have an interesting race condition issue with placement16:06
gibiyepp I saved that for the gate issue topic16:06
gibilets go there16:06
bauzasthis isn't only impacting the gate, right?16:06
gibiit is a new race so so far we only have infor from the gate about it16:07
bauzasbut yeah, let's discuss this bug on the gate section 16:07
gibi#topic Gate status 16:07
gibiNova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure16:07
gibiWe had the allocation deletion conflict bug #link https://bugs.launchpad.net/nova/+bug/1836754 that hopefully resolved when we landed #link https://review.opendev.org/c/openstack/nova/+/688802 yesterday16:07
gibiWe have a project creation race in allocation update bug as well #link https://storyboard.openstack.org/#!/story/2009159 that is affecting the those gate jobs that are booting more than on VM at the same time. Octavia is heavily affected. I've pushed a reproducer #link https://review.opendev.org/c/openstack/placement/+/806730 but I don't know how to fix the actual bug.16:08
gibithis is the one bauzas mentioned above ^^16:08
bauzasif the race isn't occurring a lot of times, we should maybe just call the transaction again16:09
gibiin short if two VM for the same project is scheduled at the same time then the PUT /allocations call could cause a race in the project creation in placement16:09
gibibauzas: that was my first idea, but looking the code I'm not sure about the amount of surgery needed for this16:10
dansmithit must be the *first* two vms right?16:10
gibiif somebody has deeper sqlAlchemy knowledge than me then help is appreciated16:10
bauzasdansmith: it *should* be yeah16:10
gibidansmith: first two VMs for a project yes16:10
sean-k-mooneygibi: sorry im a bit confusted by the term project createion in a placment context16:10
dansmiththat definitely seems uncool16:10
bauzasI guess we could hit the problem often with multicreate16:11
gibisean-k-mooney: placement stores the project_id and user_id of each consumer so that usages can be aggregated per project16:11
bauzasie. server create --max 216:11
sean-k-mooneywhy does placement need to be project wawre beyond storigng the project/user id in the allocation?16:11
bauzassean-k-mooney: because we use it from an API query16:11
dansmithsean-k-mooney: it's relational so it has a table for it I think16:12
sean-k-mooneyyes im aware it there for usage16:12
bauzasyou can filter per project16:12
gibihttps://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages16:12
sean-k-mooneybut im not sure what resouce in the db we would need to create that would race16:12
bauzasthis was my question earlier this EU morning time :)16:12
gibisean-k-mooney: it is the row in the projects table16:12
dansmithsean-k-mooney: the first time that entry goes into the table for the project?16:12
gibiyes ^^16:12
bauzascorrect16:12
sean-k-mooneyright im wondering why we need a proejct tbale at all16:12
dansmith...because it's relational :)16:13
gibisean-k-mooney: for the /usages?project_id= query 16:13
sean-k-mooneyi assuem its just for forine key constriants16:13
sean-k-mooneydansmith: right but we dont actuly do that in other dbs in openstack16:13
dansmithwe could create the project first in a separate transaction, which is easy to retry, but that's definitely surgery16:13
dansmithsean-k-mooney: not because it's a good idea :)16:13
gibidansmith: actually I think that is what happened before consumer_types16:14
gibidansmith: project_id user_id and consumer creation was each in a separate transaction, then a single transaction created / updated the allocation 16:14
dansmithgibi: ah, so this is more fallout from that work?16:15
gibiit seems so16:15
dansmithokay16:15
gibidansmith: this changed, now the whole PUT /allocations is a single DB transaction16:15
dansmithfor something like the project record, it makes sense to do that in a separate transaction I think16:15
gibiand when that transaction founds a duplicate project it dies even if we catch the exception from sqla16:15
sean-k-mooneyi wonder if we shoudl drop those tables in the future and just store the uuids direcly in the allcoation table16:15
dansmithgibi: further explains the large-ness of the transaction being a problem I guess16:15
bauzasdansmith: that's why I nearly consider this as a regression16:16
dansmithbauzas: definitely sounds like a regression16:16
bauzaswe stepped into this mess in Xena16:16
opendevreviewMerged openstack/nova master: Parse extended resource request from the port data  https://review.opendev.org/c/openstack/nova/+/80008516:16
dansmithsean-k-mooney: only if we want it to be slower and more bloated I think :)16:16
gibithe reason this become a single transaction is that consumer_type can be updated during PUT /allocations and if that happens in a separate transaction then we leak the update16:17
sean-k-mooneywell right now we are going to have to do a select by uuid to get the id in the project/user tabel then join based on that to the allcoation16:17
bauzasI have no idea because I haven't looked at code yet, but is it crazy to switch back to the 2-step transactional model we had before ?16:17
dansmithgibi: but we're just recording projects for relational purposes, so even if the allocation put doesn't succeed, it's fine that we've created the project record right?16:17
sean-k-mooneyso im not sure it would be slow to just to a select againt the allcoation table directly with an index on the user_id/project_id columns16:17
bauzasie. get the projet or create it, first, then do the allocation update16:18
gibidansmith: yes, for project and user it is fine, for consumer_type it is not fine16:18
bauzasinto two transactions16:18
dansmithgibi: right16:18
gibiso we can move back to many transactions but then we need to figure out how to move the consumer_type update to the last big transaction16:19
dansmithyeah, which is surgery for sure16:19
dansmithwas melwitt the one that wrote the consumer type patches?16:19
dansmithmaybe we should get her consult16:19
gibiit was melwitt who updated it after cdent left16:19
dansmithwell, right16:20
gibiso she knows more16:20
dansmith$current_owner = 'melwitt' is what I meant :)16:20
gibiyeah16:20
gibiOK, I guess that is it for now about this bug I will ping melwitt about it and record this discussion inthe bug16:20
gibimoving on16:20
gibiis there any other nova bug (gate or not) we need to discuss?16:21
bauzasdid we put the hot potato into someone's hand who isn't there ? excellent outcome of this discussion :p16:21
opendevreviewMerged openstack/nova master: Transfer RequestLevelParams from ports to scheduling  https://review.opendev.org/c/openstack/nova/+/79150616:21
gibibauzas: I can still feel the responsibility until I can talk to melwitt 16:21
dansmithwasn't it in her hands to begin with? I think it's okay :P16:21
* bauzas is happy to be eventually back, then :p16:21
gibi:D16:21
gibiI see there is no other bug to talk about great :D16:22
bauzasmoving on16:22
gibiPlacement periodic job status #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly 16:22
gibithey are greeen16:22
gibiand last but not least be prepared that the gate will be slow this week due to Milestone 316:22
gibiwhich brings us to16:23
gibi#topic Release Planning16:23
gibiRelease tracking etherpad #link https://etherpad.opendev.org/p/nova-xena-rc-potential16:23
gibiFeature freeze is 3rd of Sept which is end of this week. 16:23
gibiAlso we have Milestone 3 end of this week that marks the last release of our client libs. Release patches: 16:23
gibinovaclient https://review.opendev.org/c/openstack/releases/+/80659216:23
gibiosc-placement https://review.opendev.org/c/openstack/releases/+/80658016:23
gibiI'm not tracking any open novaclient patches 16:23
gibibut for osc-placement the     https://review.opendev.org/c/openstack/osc-placement/+/804458 consumer_types support client patch would be nice16:24
gibithe client release should happen this week so we still have a bit of time16:24
bauzasI can review it16:25
gibiWe are tracking the release TODOs in #link https://etherpad.opendev.org/p/nova-xena-rc-potential16:25
gibibauzas: thanks, I'm also plannig to get to it16:25
gibithere are two open todos for the release16:25
gibiWe need to produce the cycle highlights 16:25
gibireview is there: #link https://review.opendev.org/c/openstack/releases/+/80075516:26
gibiI will do a bp status cleanup tomorrow and along that I will update the highlights patch. But you can help me with the highlight by bringing up features in the comments.16:26
opendevreviewStephen Finucane proposed openstack/nova master: api: Add support for 'hostname' parameter  https://review.opendev.org/c/openstack/nova/+/77855016:26
opendevreviewStephen Finucane proposed openstack/nova master: policy: Deprecate field from 'os-extended-server-attributes' policy  https://review.opendev.org/c/openstack/nova/+/80613116:26
opendevreviewStephen Finucane proposed openstack/nova master: tests: Speed up 'servers' API tests  https://review.opendev.org/c/openstack/nova/+/77873216:26
opendevreviewStephen Finucane proposed openstack/nova master: tests: Address nits for configurable-instance-hostnames series  https://review.opendev.org/c/openstack/nova/+/80673516:26
gibithe other todo is16:26
gibiWe need a volunteer to write the release notes prelude (deadline for the prelude is RC1)16:26
dansmithgosh I wonder who will do that16:27
* gibi turning the general direction of France :)16:27
* gibi turning to the general direction of France :)16:27
bauzashmmm16:27
* bauzas wonders who gibi is looking at16:27
gibi:D16:28
bauzasbut ok, I can volunteer for it16:28
gibithanks 16:28
gibianything else about the release before we jump to open bps?16:28
gibi#topic Review priorities 16:29
gibithis is a new topic as the gerrit patch enabling the new Review-Priority label has been landed16:30
gibiyou can query the label with something like https://review.opendev.org/q/project:openstack/nova+label:Review-Priority%252B116:30
gibithe doc about the label is here https://review.opendev.org/c/openstack/nova/+/79235716:31
gibianyhow as we are close the FF sean-k-mooney collected couple of bps from our list that we quickl agree on to mark as priority16:31
gibiinstance host name https://review.opendev.org/q/topic:%22bp%252Fconfigurable-instance-hostnames%22+(status:open%20OR%20status:merged)16:32
gibithis is basically approved 16:32
gibias far as I see16:32
bauzaslet me ask a stupid question16:32
bauzasdo we have a formal process on how to use this flag ?16:32
gmannor we can run this to filter already +W one #link  https://review.opendev.org/q/project:openstack/nova+Review-Priority:1+label:Verified%253D1++NOT+label:Workflow%253C%253D-116:32
bauzasI remember some change I reviewed a while ago16:32
gibibauzas: this is where we discussing that https://review.opendev.org/c/openstack/nova/+/79235716:32
bauzasbut did we get into some direction ?16:33
sean-k-mooneybauzas: yes16:33
gibiI see you have negative comments there :)16:33
sean-k-mooneywell it was +w'd so i assuems we had agreed16:33
bauzasI can see some consensus16:33
bauzasso I can rereview this16:33
gibisure I have to back to that too16:34
gibilets then just quickly check some of the open bps that close to lend and ignore the new label until we agree on the doc16:34
gibis/lend/land16:34
gibiso instance hostname is a basically done we can move on16:35
gibivhost multi queue flavor extra spec https://review.opendev.org/q/topic:%22bp%252Fmultiqueue-flavor-extra-spec%22+(status:open%20OR%20status:merged)16:35
sean-k-mooneywell i would prefer to use the new lable and adjust our useage if we need too in a follow up t be hosest but sure lets proced16:35
gibisean-k-mooney: I don't want to hold up the meeting agreeing on the usage as we have FF this week16:36
sean-k-mooney gibi sure we can move on16:36
gibiso the multique series is a simple one but still lack reviews16:36
sean-k-mooneyya tl;dr that is supporting a mutlique via the flaovr in addtion to the existing image property16:36
bauzasgibi: I was looking into the multiqueue series before the meeting16:37
sean-k-mooneyits pretty straigt forward but need reviews16:37
bauzasso I could mark it with Review-Prio flag :p16:37
gibibauzas: you can :P16:37
gibiif nobody else shows up I can read that too16:38
gibimoving on16:38
gibigeneric mdevs https://review.opendev.org/q/topic:%22bp%252Fgeneric-mdevs%22+(status:open%20OR%20status:merged)16:38
gibias far as I see the code is approved16:38
gibithe doc patch is open16:38
bauzasthanks all16:38
gibiwe can land doc only patches after FF so this is in good shape16:38
bauzasI can work on the doc change btw.16:38
bauzasbut my priority for this weeks is reviews.16:39
gibithat sounds good to me16:39
gibimoving on16:39
gibipps qos https://review.opendev.org/q/topic:%2522bp/qos-minimum-guaranteed-packet-rate%2522+(status:open+OR+status:merged)+project:openstack/nova16:39
gibithis is easy as we don't have to rush 16:39
gibithe neturon API extension enabling the new logic probably wont land in Xena16:39
gibi(but already works on in my env)16:40
sean-k-mooneyah ok so this is moving to yoga then?16:40
gibithe neutron impact seems so16:40
gibithe nova impact is good to land now but it wont be triggered if neutron does not have the new api extension16:41
gibiso it is dead code16:41
sean-k-mooneyi mean if its dead code its also going to have little or no impact on people deploying xena but sure16:41
gibiyes16:41
gibimoving on16:42
gibimelwitt updated the unified limits https://review.opendev.org/q/topic:%22bp%252Funified-limits-nova%22+(status:open%20OR%20status:merged) sries16:42
gibidansmith: you had comments before. Will you have time this week to look back to this?16:42
dansmithI saw her note about not reaching into limit internals but haven't gone back to look16:42
dansmithI will, but are we really expecting to land this? if we are, then I'll try to prioritize, but if not, I'd focus on other things that are time-critical16:43
dansmithseems like there are a lot of patches yet to land for thursday,16:43
dansmithand I know some have had some review, but I don't think all (but could be wrong)16:43
bauzasagreed with dansmith16:43
bauzasI'd love this series to merge, but this is very late in time16:44
gibidansmith: it was open for a long time so I guess it is not that time critical16:45
dansmithlots of odd gate fail on the bottom few, which I'm sure is not related to those patches, but just in terms of likelihood for merging all of that by thursday seems unlikely16:46
gibiI don't see that much thing to close to land in nova but I guess you have commitments outside of nova too16:46
dansmithyeah I meant other stuff16:46
dansmithwe should recheck the bottom three that are +W at least I guess16:47
gibiyepp16:47
dansmiththe API ones at the top of the set are some I want to look closer at, and I haven't looked at the tempest tests that are available yet16:47
gibiack. if you not get to it the it will be moved to yoga, no hard feelings :)16:48
gibimoving on 16:48
gibihere are the rest of the open bps https://launchpad.net/nova/+milestone/xena-316:48
gibi(some of the should be closed which is will do tomorrow)_16:48
gibiis there any of those somebody want to get attention?16:49
gmannremove tenant-id is still not in shape for Xena, we are still need all code up before we start merging first patch16:49
gmannmay be for Yoga16:49
gibigmann: thanks for the info. that is also a long time open series. 16:49
gmannyeah. 16:50
gmannwe are reviewing it on time but just all the code should be up before we start merging it16:50
gibiyeah, that is a big change in a single microversion. 16:50
gibiany other bp? 16:51
gibithe moving on16:51
gibi#topic PTG Planning 16:51
gibievery info is in the PTG etherpad #link https://etherpad.opendev.org/p/nova-yoga-ptg16:51
gibiplease add topics ^^16:51
gibiIf you see a need for a specific cross project section then please let me know16:51
gibiany question about the PTG?16:51
gibi#topic Stable Branches 16:52
gibiperiodic-stable jobs are not failing16:52
gibievery stable gate seem to be working and unblocked16:52
gibiEOM( elodilles )16:52
gibianything else about stable?16:52
gibi#topic Sub/related team Highlights16:53
elodillesnothing exciting compared to FF :X16:53
gibiLibvirt (bauzas)16:53
gibielodilles: :)16:53
bauzaswell, I pledge guilty, I was off for 3 weeks16:53
gibigood for you :)16:53
gibi#topic Open discussion 16:54
gibi(gibi) PTL election: as you probably saw from Yoga we will have a new PTL, bauzas. I'm not going anywhere and I will help to make the transition smooth.16:54
gibiwelcome bauzas :)16:54
bauzasthanks, just hoping I won't get a revolution, as in my country16:54
bauzasdon't cut my head, please.16:55
gibidont put your head near to sharp object :D16:55
sean-k-mooneyhehe. all hail the new hearder of cat :P16:55
* gibi need more cats16:55
gibianything else for today?16:56
bauzasfun fact, I have no cats but a dog16:56
bauzas(and 2 kids, but apparently we can't count them as animals)16:56
gmannbauzas: :)16:57
gibiwould you like to? :)16:57
gibiif nothing else then lets close this meeting16:58
bauzaswell, they bark, always ask for food and sleep16:58
gibi:D16:58
dansmithare we done here? :)16:58
bauzasI guess16:58
gibi#endmeeting16:58
opendevmeetMeeting ended Tue Aug 31 16:58:50 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:58
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-31-16.01.html16:58
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-31-16.01.txt16:58
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2021/nova.2021-08-31-16.01.log.html16:58
gibiI'm dropping of for today see you tomorrow o/17:00
gmanno/, GN17:01
gmannstephenfin: can you please check this review https://review.opendev.org/c/openstack/nova/+/80629418:09
opendevreviewMerged openstack/nova stable/ussuri: Centralize sqlite FK constraint enforcement  https://review.opendev.org/c/openstack/nova/+/78507019:02
opendevreviewMerged openstack/nova stable/ussuri: Add functional test for bug 1837995  https://review.opendev.org/c/openstack/nova/+/78507119:02
mlozahello, i given a user a reader role at system level but unable to list all projects and instances 20:05
opendevreviewMerged openstack/nova stable/ussuri: Dynamically archive FK related records in archive_deleted_rows  https://review.opendev.org/c/openstack/nova/+/78507220:14
opendevreviewMerged openstack/nova master: db: Handle parameters in DB strings  https://review.opendev.org/c/openstack/nova/+/80566320:47
opendevreviewMerged openstack/nova stable/victoria: Fix 1vcpu error with multiqueue and vif_type=tap  https://review.opendev.org/c/openstack/nova/+/80600421:15
opendevreviewMerged openstack/nova master: api: Add support for 'hostname' parameter  https://review.opendev.org/c/openstack/nova/+/77855021:15
opendevreviewRodrigo Barbieri proposed openstack/nova stable/ussuri: Fix 1vcpu error with multiqueue and vif_type=tap  https://review.opendev.org/c/openstack/nova/+/80681121:29
opendevreviewMerged openstack/nova master: policy: Deprecate field from 'os-extended-server-attributes' policy  https://review.opendev.org/c/openstack/nova/+/80613121:37
opendevreviewMerged openstack/nova master: tests: Speed up 'servers' API tests  https://review.opendev.org/c/openstack/nova/+/77873221:38
opendevreviewMerged openstack/nova master: tests: Address nits for configurable-instance-hostnames series  https://review.opendev.org/c/openstack/nova/+/80673521:38
opendevreviewMerged openstack/nova master: Support boot with extended resource request  https://review.opendev.org/c/openstack/nova/+/80008621:38
opendevreviewMerged openstack/nova master: Support move ops with extended resource request  https://review.opendev.org/c/openstack/nova/+/80008721:38

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!