| *** igordc has quit IRC | 00:08 | |
| *** sapd1_ has joined #openstack-nova | 00:15 | |
| *** TxGirlGeek has joined #openstack-nova | 00:16 | |
| *** dtantsur has joined #openstack-nova | 00:16 | |
| *** mdbooth_ has joined #openstack-nova | 00:18 | |
| *** lee1 has joined #openstack-nova | 00:19 | |
| *** sapd1 has quit IRC | 00:21 | |
| *** TxGirlGe_ has quit IRC | 00:21 | |
| *** dtantsur|afk has quit IRC | 00:21 | |
| *** szaher has quit IRC | 00:22 | |
| *** mdbooth has quit IRC | 00:22 | |
| *** mvkr has quit IRC | 00:22 | |
| *** lyarwood has quit IRC | 00:22 | |
| *** sorrison has quit IRC | 00:22 | |
| *** sorrison has joined #openstack-nova | 00:22 | |
| *** mvkr has joined #openstack-nova | 00:22 | |
| *** johanssone has quit IRC | 00:23 | |
| *** dustinc has quit IRC | 00:25 | |
| *** cz3 has quit IRC | 00:25 | |
| *** mnasiadka has quit IRC | 00:25 | |
| *** dustinc has joined #openstack-nova | 00:25 | |
| *** knikolla has quit IRC | 00:25 | |
| *** arne_wiebalck has quit IRC | 00:25 | |
| *** ildikov has quit IRC | 00:25 | |
| *** pas-ha has quit IRC | 00:25 | |
| *** andreaf has quit IRC | 00:25 | |
| *** CeeMac has quit IRC | 00:25 | |
| *** vdrok has quit IRC | 00:25 | |
| *** donnyd has quit IRC | 00:25 | |
| *** rajinir has quit IRC | 00:25 | |
| *** ab-a has quit IRC | 00:25 | |
| *** fyx has quit IRC | 00:25 | |
| *** logan- has quit IRC | 00:25 | |
| *** mnasiadka has joined #openstack-nova | 00:25 | |
| *** andreaf has joined #openstack-nova | 00:25 | |
| *** CeeMac has joined #openstack-nova | 00:25 | |
| *** pas-ha has joined #openstack-nova | 00:25 | |
| *** knikolla has joined #openstack-nova | 00:25 | |
| *** cz3 has joined #openstack-nova | 00:25 | |
| *** donnyd has joined #openstack-nova | 00:25 | |
| *** rajinir has joined #openstack-nova | 00:25 | |
| *** vdrok has joined #openstack-nova | 00:25 | |
| *** fyx has joined #openstack-nova | 00:25 | |
| *** arne_wiebalck has joined #openstack-nova | 00:26 | |
| *** johanssone has joined #openstack-nova | 00:26 | |
| *** ildikov has joined #openstack-nova | 00:26 | |
| *** rm_work has quit IRC | 00:27 | |
| *** StevenK has quit IRC | 00:27 | |
| *** logan- has joined #openstack-nova | 00:27 | |
| *** StevenK has joined #openstack-nova | 00:27 | |
| *** openstackstatus has quit IRC | 00:28 | |
| *** jamesdenton has quit IRC | 00:28 | |
| *** Jeffrey4l has quit IRC | 00:29 | |
| *** melwitt has quit IRC | 00:29 | |
| *** rm_work has joined #openstack-nova | 00:30 | |
| *** melwitt has joined #openstack-nova | 00:30 | |
| *** ab-a has joined #openstack-nova | 00:30 | |
| *** Jeffrey4l has joined #openstack-nova | 00:31 | |
| *** jamesdenton has joined #openstack-nova | 00:31 | |
| efried | Thanks alex_xu | 00:44 |
|---|---|---|
| *** TxGirlGeek has quit IRC | 00:49 | |
| *** brinzhang has joined #openstack-nova | 00:56 | |
| *** spatel has joined #openstack-nova | 01:00 | |
| *** spatel has quit IRC | 01:00 | |
| *** vishalmanchanda has quit IRC | 01:04 | |
| *** adriant has quit IRC | 01:10 | |
| *** adriant has joined #openstack-nova | 01:13 | |
| *** jhesketh has quit IRC | 01:13 | |
| *** jhesketh has joined #openstack-nova | 01:14 | |
| *** mdbooth_ has quit IRC | 01:40 | |
| *** mdbooth has joined #openstack-nova | 01:41 | |
| *** Liang__ has joined #openstack-nova | 01:48 | |
| *** brinzhang has quit IRC | 02:04 | |
| *** brinzhang has joined #openstack-nova | 02:05 | |
| *** brinzhang has quit IRC | 02:06 | |
| *** brinzhang has joined #openstack-nova | 02:07 | |
| openstackgerrit | Merged openstack/nova master: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/682523 | 03:00 |
| *** gyee has quit IRC | 03:03 | |
| *** zhurong has joined #openstack-nova | 03:03 | |
| *** damien_r has joined #openstack-nova | 03:06 | |
| *** mkrai has joined #openstack-nova | 03:28 | |
| *** jmlowe has joined #openstack-nova | 03:31 | |
| openstackgerrit | Qiu Fossen proposed openstack/nova-specs master: specify mac for creating instance https://review.opendev.org/700429 | 03:38 |
| *** psachin has joined #openstack-nova | 03:43 | |
| *** jmlowe has quit IRC | 03:43 | |
| *** jmlowe has joined #openstack-nova | 03:45 | |
| *** xek has joined #openstack-nova | 04:00 | |
| *** udesale has joined #openstack-nova | 04:15 | |
| *** tetsuro has joined #openstack-nova | 04:27 | |
| *** tetsuro has quit IRC | 04:34 | |
| *** tetsuro has joined #openstack-nova | 04:37 | |
| *** zhurong has quit IRC | 04:41 | |
| *** jmlowe has quit IRC | 04:45 | |
| *** zhurong has joined #openstack-nova | 05:07 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to server actions API https://review.opendev.org/706179 | 05:16 |
| *** artom has quit IRC | 05:27 | |
| *** adriant has quit IRC | 05:27 | |
| *** adriant has joined #openstack-nova | 05:27 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-nova | 05:34 | |
| *** udesale_ has joined #openstack-nova | 05:39 | |
| *** udesale has quit IRC | 05:42 | |
| *** sean-k-mooney has quit IRC | 05:53 | |
| *** Liang__ has quit IRC | 05:54 | |
| *** sean-k-mooney has joined #openstack-nova | 05:55 | |
| *** ratailor has joined #openstack-nova | 06:05 | |
| *** artom has joined #openstack-nova | 06:10 | |
| *** yedongcan has joined #openstack-nova | 06:16 | |
| *** ccamacho has quit IRC | 06:26 | |
| *** brinzhang has quit IRC | 06:32 | |
| *** links has joined #openstack-nova | 06:32 | |
| *** brinzhang has joined #openstack-nova | 06:32 | |
| *** brinzhang has quit IRC | 06:35 | |
| *** brinzhang has joined #openstack-nova | 06:36 | |
| *** yedongcan has quit IRC | 06:40 | |
| *** vishalmanchanda has joined #openstack-nova | 06:40 | |
| *** yedongcan has joined #openstack-nova | 06:40 | |
| *** zhurong has quit IRC | 06:42 | |
| openstackgerrit | Merged openstack/nova master: Avoid calling neutron for N networks https://review.opendev.org/705784 | 06:50 |
| *** sean-k-mooney has quit IRC | 07:01 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to servers actions API https://review.opendev.org/706179 | 07:02 |
| *** sean-k-mooney has joined #openstack-nova | 07:03 | |
| *** lpetrut has joined #openstack-nova | 07:09 | |
| *** lpetrut has quit IRC | 07:10 | |
| *** lpetrut has joined #openstack-nova | 07:10 | |
| *** yedongcan has quit IRC | 07:20 | |
| *** artom has quit IRC | 07:20 | |
| *** yedongcan has joined #openstack-nova | 07:22 | |
| *** artom has joined #openstack-nova | 07:23 | |
| *** artom has quit IRC | 07:24 | |
| *** artom has joined #openstack-nova | 07:24 | |
| *** iurygregory has joined #openstack-nova | 07:46 | |
| *** mkrai has quit IRC | 07:47 | |
| *** gentoora- has joined #openstack-nova | 07:50 | |
| *** gentoorax has quit IRC | 07:50 | |
| *** gentoora- is now known as gentoorax | 07:50 | |
| gibi | efried: yeah max 20 I t was more on the 12 - 15 side in my memory | 07:53 |
| gibi | efried, dansmith: I was avoiding cyborg integration code recently as I saw that both of you are actively handled that but now I feel you want me to read in. I hope I will not blow up things with my fresh oppinion | 07:54 |
| *** xek has quit IRC | 07:56 | |
| *** xek has joined #openstack-nova | 07:56 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Expose instance action event details out of the API https://review.opendev.org/694430 | 08:04 |
| *** tkajinam has quit IRC | 08:04 | |
| *** tesseract has joined #openstack-nova | 08:13 | |
| *** xek has quit IRC | 08:16 | |
| *** xek has joined #openstack-nova | 08:16 | |
| *** maciejjozefczyk has joined #openstack-nova | 08:21 | |
| *** amoralej|off is now known as amoralej | 08:23 | |
| *** mkrai has joined #openstack-nova | 08:23 | |
| *** brinzhang_ has joined #openstack-nova | 08:27 | |
| *** brinzhang_ has quit IRC | 08:28 | |
| *** brinzhang has quit IRC | 08:31 | |
| *** ccamacho has joined #openstack-nova | 08:33 | |
| *** tosky has joined #openstack-nova | 08:40 | |
| *** slaweq has joined #openstack-nova | 08:40 | |
| *** brinzhang has joined #openstack-nova | 08:46 | |
| *** brinzhang has quit IRC | 08:46 | |
| *** brinzhang has joined #openstack-nova | 08:46 | |
| *** ralonsoh has joined #openstack-nova | 08:51 | |
| *** ratailor has quit IRC | 09:01 | |
| *** abhishekk|away is now known as abhishekk | 09:13 | |
| *** brinzhang has quit IRC | 09:19 | |
| *** brinzhang has joined #openstack-nova | 09:19 | |
| ralonsoh | stephenfin, sorry for bothering you again... | 09:30 |
| ralonsoh | https://review.opendev.org/#/c/705760/2/nova/network/neutron.py@2615 | 09:30 |
| ralonsoh | I didn't realize this when I reviewed the patch | 09:31 |
| ralonsoh | but a FIP can NOT be attached to a port | 09:31 |
| ralonsoh | so this exception... is unnecessary. Actually we are still having some problems in the Neutron OVN CI | 09:32 |
| ralonsoh | https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/703537/7/check/neutron-ovn-tempest-ovs-release/7b55444/testr_results.html | 09:32 |
| stephenfin | damn. okay, let me fix that up | 09:32 |
| ralonsoh | stephenfin, let me first check the test | 09:33 |
| ralonsoh | I would like to confirm that in the logs | 09:33 |
| *** bbowen has quit IRC | 09:35 | |
| *** bbowen has joined #openstack-nova | 09:35 | |
| *** martinkennelly has joined #openstack-nova | 09:35 | |
| ralonsoh | stephenfin, https://7147d7e07a0a4e67ee4e-8dbb68c995c97f9fb5b4258bb26a4c49.ssl.cf2.rackcdn.com/703537/7/check/neutron-ovn-tempest-ovs-release/7b55444/controller/logs/screen-n-api.txt | 09:39 |
| ralonsoh | we are creating IPs without any port associated | 09:39 |
| *** brtknr has quit IRC | 09:40 | |
| *** ociuhandu has joined #openstack-nova | 09:42 | |
| *** ociuhandu has quit IRC | 09:42 | |
| *** ociuhandu has joined #openstack-nova | 09:44 | |
| *** ivve has joined #openstack-nova | 09:58 | |
| *** xek has quit IRC | 10:03 | |
| stephenfin | ralonsoh: What happens if I do 'list_ports(ids=['foo'])' and 'foo' doesn't exist? | 10:05 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't error out on floating IPs without associated ports https://review.opendev.org/706213 | 10:06 |
| stephenfin | I assume I'll just get an empty response, i.e. {'ports': []} | 10:06 |
| stephenfin | ralonsoh, gibi: That's the fix, btw ^ | 10:06 |
| ralonsoh | stephenfin, yes, you have an empty list | 10:08 |
| ralonsoh | show_port will return an exception | 10:08 |
| ralonsoh | list_ports don't | 10:08 |
| stephenfin | cool, that's what I'd assumed in that patch | 10:08 |
| *** slaweq_ has joined #openstack-nova | 10:13 | |
| *** slaweq has quit IRC | 10:15 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Rework how we check for extensions https://review.opendev.org/705792 | 10:16 |
| *** ratailor has joined #openstack-nova | 10:29 | |
| *** slaweq__ has joined #openstack-nova | 10:39 | |
| *** slaweq_ has quit IRC | 10:41 | |
| *** rpittau|afk is now known as rpittau | 10:45 | |
| *** slaweq__ is now known as slaweq | 10:46 | |
| *** mkrai has quit IRC | 10:52 | |
| *** udesale_ has quit IRC | 11:02 | |
| *** lee1 is now known as lyarwood | 11:05 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Rename _is_storage_shared_with to _is_path_shared_with https://review.opendev.org/693337 | 11:09 |
| lyarwood | stephenfin: ^ can you +W that again, had to rebase to clear the conflict. | 11:09 |
| stephenfin | done | 11:09 |
| lyarwood | ta | 11:10 |
| gibi | stephenfin: a floating ip without a port is an unused floating ip? | 11:11 |
| openstackgerrit | Guo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120 | 11:11 |
| stephenfin | gibi: Yup. We're using | 11:12 |
| stephenfin | ...it for the 'GET /os-floating-ips' API, and that describes itself as "Return(ing) a list of floating IPs allocated to a project." | 11:13 |
| gibi | cool, thanks | 11:13 |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Re-propose "Secure Boot support for KVM & QEMU guests" for Ussuri https://review.opendev.org/693844 | 11:24 |
| brinzhang | gibi, lyarwood: this is my spec's invalid link fix patch https://review.opendev.org/#/c/705933/, please review | 11:25 |
| brinzhang | gibi, lyarwood: https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1663456 this is the bug 1663456 fix, one is +2 by stephenfin | 11:27 |
| openstack | bug 1663456 in OpenStack Compute (nova) "Field 'updated_at' always 'None' when show aggregate" [Low,In progress] https://launchpad.net/bugs/1663456 - Assigned to Brin Zhang (zhangbailin) | 11:27 |
| *** ociuhandu has quit IRC | 11:35 | |
| *** ociuhandu has joined #openstack-nova | 11:37 | |
| *** ociuhandu has quit IRC | 11:42 | |
| *** tbachman has quit IRC | 11:43 | |
| *** brtknr has joined #openstack-nova | 11:54 | |
| *** ociuhandu has joined #openstack-nova | 11:58 | |
| *** amoralej is now known as amoralej|lunch | 12:00 | |
| *** yedongcan has left #openstack-nova | 12:07 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add server actions v82 samples test https://review.opendev.org/706251 | 12:09 |
| *** nicolasbock has joined #openstack-nova | 12:12 | |
| *** pcaruana has quit IRC | 12:13 | |
| gibi | stephenfin: fyi if you need I can reproduce the stestr bug https://github.com/mtreinish/stestr/pull/285 in a pretty stable way in nova | 12:16 |
| *** slaweq has quit IRC | 12:17 | |
| stephenfin | gibi: I had this https://review.opendev.org/#/c/700522/ | 12:17 |
| *** slaweq has joined #openstack-nova | 12:23 | |
| *** ociuhandu has quit IRC | 12:30 | |
| *** ociuhandu has joined #openstack-nova | 12:30 | |
| gibi | stephenfin: you won :) | 12:31 |
| gibi | stephenfin: is there a way I can help the stestr fix merged? | 12:31 |
| artom | stephenfin, left you a drive-by comment on https://review.opendev.org/#/c/705792/3 | 12:34 |
| *** ociuhandu has quit IRC | 12:35 | |
| openstackgerrit | Arnaud Morin proposed openstack/nova master: Force refresh of instance info cache on event https://review.opendev.org/702394 | 12:37 |
| *** spatel has joined #openstack-nova | 12:39 | |
| *** jcosmao has joined #openstack-nova | 12:42 | |
| *** spatel has quit IRC | 12:44 | |
| *** damien_r has quit IRC | 12:49 | |
| *** ociuhandu has joined #openstack-nova | 12:50 | |
| *** rpittau is now known as rpittau|bbl | 12:52 | |
| openstackgerrit | Martin Midolesov proposed openstack/nova master: Implementing graceful shutdown. https://review.opendev.org/666245 | 12:55 |
| *** b3nt_pin is now known as beagles | 12:59 | |
| *** ratailor has quit IRC | 13:00 | |
| *** jmlowe has joined #openstack-nova | 13:07 | |
| *** tbachman has joined #openstack-nova | 13:07 | |
| kashyap | stephenfin: Thanks for taking time to address your own nits here! https://review.opendev.org/#/c/693844/ | 13:07 |
| kashyap | stephenfin: For future, don't worry about it. I don't consider it "tedious" to adddress reviewer's feedback (and yours fall "within reason" almost always) :-) | 13:08 |
| kashyap | Appreciate it regardless. Ah, I see that in some places I've used single back ticks instead of the double back ticks | 13:08 |
| *** fyx has quit IRC | 13:13 | |
| *** slaweq_ has joined #openstack-nova | 13:14 | |
| gibi | efried: I cannot join to the today's nova meeting as I have a F2F meeting at the same time | 13:15 |
| *** guilhermesp has quit IRC | 13:16 | |
| *** damien_r has joined #openstack-nova | 13:16 | |
| *** slaweq has quit IRC | 13:16 | |
| *** Liang__ has joined #openstack-nova | 13:16 | |
| *** bcm has quit IRC | 13:20 | |
| *** gmann has quit IRC | 13:23 | |
| *** irclogbot_1 has quit IRC | 13:26 | |
| *** belmoreira has joined #openstack-nova | 13:27 | |
| *** nweinber has joined #openstack-nova | 13:27 | |
| *** cz3 has quit IRC | 13:28 | |
| *** irclogbot_0 has joined #openstack-nova | 13:30 | |
| *** rpittau|bbl is now known as rpittau | 13:30 | |
| *** links has quit IRC | 13:32 | |
| *** jmlowe has quit IRC | 13:50 | |
| stephenfin | gibi: Other than annoying mtreinish and lifeless, probably not :) | 13:51 |
| gibi | :) | 13:51 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Rework how we check for neutron extensions https://review.opendev.org/705792 | 13:51 |
| stephenfin | artom: oh yeah, done | 13:51 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix obj_what_changed for InstancePCIRequests ovo https://review.opendev.org/706266 | 13:52 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix Instance.obj_what_changed behavior for the "extra" field https://review.opendev.org/706267 | 13:52 |
| brinzhang | stephenfin: hi, I have a question with the cmd.test_policy, https://github.com/openstack/nova/blob/master/nova/tests/unit/cmd/test_policy.py#L119 | 13:54 |
| *** amoralej|lunch is now known as amoralej | 13:54 | |
| efried | gibi: specifically for that one patch the important thing was seeing how the steps taken to extract device profiles and convert them into resource requests correspond to the same steps for bandwidth resources. | 13:55 |
| efried | gibi: I put a patch above that one that moved some of that stuff around, would be nice to get your nod on that. | 13:55 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Selectively refresh instance.pci_requests during rollback https://review.opendev.org/706269 | 13:55 |
| brinzhang | stephenfin: I added a new policy to the os-instance-action https://review.opendev.org/#/c/706179/2/nova/policies/instance_actions.py@27, but while run the unit test in cmd.test_policy, it was failed, and the "passing_rules " cannot get the new rule 'os_compute_api:os-instance-actions:show_details | 13:55 |
| gibi | efried: ack, I will try to review those | 13:56 |
| brinzhang | stephenfin, eric: I saw this file was changed or add by you (from github), can you give me some idea? | 13:56 |
| brinzhang | efried | 13:56 |
| efried | me? | 13:59 |
| sean-k-mooney | you :) | 13:59 |
| *** cz3 has joined #openstack-nova | 14:00 | |
| gibi | dansmith: I spent time on the selective refresh of instance.pci_requests based on obj_what_changes(), see the my stab at it here https://review.opendev.org/#/c/706269/1/nova/compute/manager.py@8566 and in the parent patches | 14:00 |
| *** fyx has joined #openstack-nova | 14:01 | |
| *** guilhermesp has joined #openstack-nova | 14:01 | |
| brinzhang | efried: yeah, but it's in two years ago :) | 14:02 |
| *** gibi is now known as gibi_on_call | 14:02 | |
| efried | brinzhang: ping me after the nova meeting | 14:03 |
| brinzhang | efried: ok, I am looking into the "def _check_filter_rules()", but cannot get some useful info, need some help :) | 14:04 |
| *** jmlowe has joined #openstack-nova | 14:04 | |
| *** bcm has joined #openstack-nova | 14:06 | |
| efried | brinzhang: if this is a policy thing, I'm definitely not the best person to ask. Have you checked with gmann? | 14:06 |
| efried | johnthetubaguy would also be a good resource if he's around... | 14:06 |
| *** spatel has joined #openstack-nova | 14:10 | |
| openstackgerrit | sean mooney proposed openstack/nova-specs master: move implemented spec for train https://review.opendev.org/706276 | 14:14 |
| umbSublime | o/ | 14:18 |
| *** mriedem has joined #openstack-nova | 14:19 | |
| *** ociuhandu has quit IRC | 14:21 | |
| *** ociuhandu has joined #openstack-nova | 14:22 | |
| *** slaweq_ is now known as slaweq | 14:24 | |
| *** ociuhandu has quit IRC | 14:24 | |
| *** ociuhandu has joined #openstack-nova | 14:25 | |
| *** ccamacho has quit IRC | 14:25 | |
| dansmith | gibi_on_call: re: cyborg, know that I'm about +2 on the rest of the set aside from the second patch I've been deferring to efried on.. hopefully no major concerns as it's been a looong road :) | 14:29 |
| bauzas | efried: you're always using this acronym : NTF | 14:30 |
| bauzas | what's this ? | 14:30 |
| efried | bauzas: NumaTopologyFilter, sorry | 14:31 |
| bauzas | oh ah | 14:32 |
| bauzas | gdi | 14:32 |
| bauzas | thanks | 14:32 |
| *** ccamacho has joined #openstack-nova | 14:33 | |
| *** belmoreira has quit IRC | 14:38 | |
| artom | stephenfin, cheers! I'll do a proper review soon | 14:38 |
| *** mriosfer has joined #openstack-nova | 14:48 | |
| bauzas | sean-k-mooney: question for you | 14:49 |
| bauzas | sean-k-mooney: if I want 8VCPUs, 8GB of RAM and hw:numa_nodes=2 | 14:49 |
| bauzas | does my memory split in twice ? | 14:49 |
| *** spatel has quit IRC | 14:49 | |
| sean-k-mooney | yes and if you cant devidie the vcpus and ram an integage number of times it raise an error | 14:51 |
| sean-k-mooney | so if you didn 8VCPUs, 8GB of RAM and hw:numa_nodes=3 | 14:51 |
| sean-k-mooney | then its an error unless you also use the other args for asymetric numa toplogies | 14:52 |
| bauzas | what kind of error ? | 14:52 |
| bauzas | sean-k-mooney: it only tells that it raises an exception in https://docs.openstack.org/nova/latest/user/flavors.html#extra-specs | 14:54 |
| brinzhang | gmann: are you around? | 14:54 |
| sean-k-mooney | yes it raise an exception and the vm goes to error state | 14:55 |
| *** Sundar has joined #openstack-nova | 14:56 | |
| sean-k-mooney | bauzas: actully the vm wont go to error | 14:56 |
| sean-k-mooney | it will rais the excptionin the api before we create teh vm db record | 14:56 |
| bauzas | cool, that's what I hoped | 14:56 |
| bauzas | a synchronous 40x | 14:56 |
| sean-k-mooney | proably a 400 bad request | 14:56 |
| bauzas | because we also need to make it for the translation mechanism | 14:56 |
| sean-k-mooney | a 4xx for sure | 14:57 |
| bauzas | ideally, same exception | 14:57 |
| *** abhishekk is now known as abhishekk|out | 14:57 | |
| brinzhang | efried: how about now? :) | 14:57 |
| sean-k-mooney | ya so this is all handeled in the hardware.py module | 14:57 |
| sean-k-mooney | and its reused in several places | 14:57 |
| efried | brinzhang: for which? The spec review? | 14:58 |
| sean-k-mooney | bauzas: its all handeled by https://github.com/openstack/nova/blob/014c1ab864cec6213750c5ef57d8080ae8cd973d/nova/virt/hardware.py#L1742 | 14:58 |
| brinzhang | efried: I added a new policy to the os-instance-action https://review.opendev.org/#/c/706179/2/nova/policies/instance_actions.py@27, but while run the unit test in cmd.test_policy, it was failed, and the "passing_rules " cannot get the new rule 'os_compute_api:os-instance-actions:show_details | 14:59 |
| bauzas | sean-k-mooney: thanks, looking | 14:59 |
| bauzas | okay, will look | 14:59 |
| brinzhang | efried:https://github.com/openstack/nova/blob/master/nova/tests/unit/cmd/test_policy.py#L119 the check function is here | 14:59 |
| Sundar | efried, dansmith, sean-k-mooney: Re. https://review.opendev.org/#/c/631243/55/nova/accelerator/cyborg.py@86, I am adding checks for keystone exceptions here. However, if this 'get device profile' call fails, there will be no further calls to Cyborg for that instance. So, there is no point in adding redundant exception checks to other calls for | 15:00 |
| Sundar | creating ARQs, etc. Do you agree? | 15:00 |
| sean-k-mooney | bauzas: the importnt bit for you is https://github.com/openstack/nova/blob/014c1ab864cec6213750c5ef57d8080ae8cd973d/nova/compute/api.py#L925 in _validate_and_build_base_options | 15:00 |
| dansmith | Sundar: you should assume every call to another service could fail | 15:01 |
| bauzas | sean-k-mooney: anyway, I'll write something in the spec and you'll chime in | 15:01 |
| efried | brinzhang: I'm afraid I have no clue about this. I would have to dig in fresh, and I don't have time to do that right now. However, I think gmann might be able to address your questions much more easily. | 15:01 |
| sean-k-mooney | bauzas: cool | 15:01 |
| efried | brinzhang: not sure where he is... | 15:02 |
| bauzas | basically, my concern is 'I want 8 vCPUs, 7GB of RAM and hw:numa_nodes=2', accordingly the RAM has to be provided by two different memory pages that are in each NUMA nodes | 15:02 |
| bauzas | sean-k-mooney: ^ | 15:02 |
| bauzas | meaning that the translation mechanism will split this memory amount in twice (or more) | 15:02 |
| stephenfin | I want to use an unreleased feature in neutronclient in a nova patch. For 'Depends-On' to work, I need to to add 'python-neutronclient' to 'required-projects' in '.zuul.yaml', right? | 15:02 |
| sean-k-mooney | ya you dont need to wory about that really | 15:03 |
| brinzhang | efried: ok, I will wait him, and I pinged him another question too, I will wait for him ,thanks | 15:03 |
| sean-k-mooney | if the requst_spec has a numa_toplogy objct its a numa instnace and all of that has already been calualted | 15:03 |
| efried | bauzas: yes, that sounds correct to me | 15:03 |
| sean-k-mooney | so you just need to read the value form the numa toplogy object | 15:03 |
| efried | stephenfin: yes, that sounds correct. But we wouldn't want to merge that change. | 15:04 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs https://review.opendev.org/706295 | 15:04 |
| stephenfin | efried: Yup, just a WIP for now | 15:04 |
| efried | stephenfin: and that only works for zv3 jobs btw, not for legacy (IIUC) | 15:04 |
| sean-k-mooney | Sundar: im goign to redeploy with the latest patches are they all updated | 15:04 |
| Sundar | dansmith: Yes, but we are talking of failures like EndpointNotFound, MissingAuthPlugin, and Unauthorized. If the first call succeeds, is the second one going to fail for one of these reasons? | 15:05 |
| sean-k-mooney | you said you tought i was using an old set of cybog pataches when you looked at the error i got on monday but that was the latest set that were up at that point | 15:05 |
| Sundar | sean-k-mooney: Great, thanks | 15:05 |
| dansmith | Sundar: no, this is a distributed system.. you should never make that assumption | 15:05 |
| *** ociuhandu has quit IRC | 15:06 | |
| mriosfer | Sean: if host got two sockets , is recomended to "force" hw:numa_nodes=2 ? If vcpus are more 2 / 4 / 6 / 8 etc... | 15:06 |
| efried | Sundar: If you wanted to trap those failures (you can use the superclass ClientException) and convert them to DeviceProfileError("Couldn't communicate with cyborg!") I think that would be okay. | 15:06 |
| dansmith | Sundar: calls from nova to cyborg (and back) may be routed to different api endpoints on subsequent calls, in different states of broken, misconfigured, or transient upgrade | 15:06 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Use neutronclient's port binding APIs https://review.opendev.org/706295 | 15:06 |
| sean-k-mooney | mriosfer: that depens on your workloads but i do for larger vms | 15:06 |
| dansmith | efried: agreed | 15:06 |
| stephenfin | kashyap: np, I figured it would just be easier do it myself real quick. No skin off my back | 15:07 |
| mriosfer | Sean : larger VM from 8 vcpu? | 15:07 |
| mriosfer | or more | 15:07 |
| sean-k-mooney | for small vms of 4-8 CPUs i will often keep them with hw:numa_nodes=1 | 15:07 |
| *** ociuhandu has joined #openstack-nova | 15:07 | |
| sean-k-mooney | ya over 8 i tend to always split | 15:07 |
| openstackgerrit | Brian Rosmaita proposed openstack/nova master: Do not inherit cinder_ image properties https://review.opendev.org/706298 | 15:07 |
| sean-k-mooney | to hw:numa_nodes=2 | 15:07 |
| stephenfin | dansmith: That neutron fix yesterday was incomplete, btw. ralonsoh and I worked out the fix this morning if you care to look again https://review.opendev.org/#/c/706213/ | 15:08 |
| dansmith | stephenfin: I'm reading now | 15:08 |
| stephenfin | ta | 15:08 |
| sean-k-mooney | but that is based on host core counts of 10-16 cores per socket. if you have 32 cores per socket then you might adjust what you consider big | 15:08 |
| mtreinish | stephenfin: thanks for the reminder, I'll re-review that PR today or tomorrow | 15:08 |
| dansmith | stephenfin: where is the code that actually uses this? | 15:08 |
| stephenfin | dansmith: Just nova/api/openstack/compute/floating_ips.py | 15:09 |
| dansmith | stephenfin: "port_details" is not in that file | 15:09 |
| dansmith | do you mean it just passes through the structure to the api requester? | 15:10 |
| Sundar | efried, dansmith: Got it. Yes, I would handle service failures in distributed systems in general. I asked because I don't see these specific checks everywhere e.g. https://github.com/openstack/nova/blob/master/nova/image/glance.py . But, I agree it is better to do it everywhere. | 15:10 |
| stephenfin | dansmith: sure it is - line 40 | 15:10 |
| mtreinish | stephenfin: the only thing is we're in prep for 3.0.0 (which removes 2.7). So that PR probably won't get released until then which we're shooting for mid-March | 15:11 |
| dansmith | stephenfin: % grep -c port_details nova/api/openstack/compute/floating_ips.py | 15:11 |
| dansmith | 0 | 15:11 |
| stephenfin | dansmith: It's definitely there :) https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/floating_ips.py#L40 | 15:12 |
| dansmith | oh is this a recent change? /me rebases | 15:12 |
| sean-k-mooney | dansmith: stephenfin started usign it as part of the nova net removal | 15:13 |
| sean-k-mooney | so yes | 15:13 |
| dansmith | yeah I see | 15:13 |
| stephenfin | Yup, it was picked up in one of the "Stop using nova-network objects as containers" patches | 15:13 |
| sean-k-mooney | so floating ips are not ports and do not normally have port details | 15:13 |
| stephenfin | mtreinish: That's okay. To be honest, the more crucial fix is the subunit fix and I need lifeless for that | 15:13 |
| dansmith | okay just wanted to see what used it and that it was okay with None | 15:13 |
| sean-k-mooney | there is a neutorn extnion to expose the port_details of the prot that they are attached too when attached | 15:14 |
| sean-k-mooney | but its optional | 15:14 |
| sean-k-mooney | apparently | 15:14 |
| stephenfin | mtreinish: https://github.com/testing-cabal/subunit/pull/40 for reference (I think I referenced it the stestr PR) | 15:14 |
| sean-k-mooney | stephenfin: when did we deprecate teh proxy api | 15:17 |
| dansmith | stephenfin: so because this broke the neutron gate, I assume we have tempest tests that actually use this proxy api, right? | 15:18 |
| *** ociuhandu has quit IRC | 15:18 | |
| stephenfin | sean-k-mooney: nova/api/openstack/api_version_request.py tells me 2.36 | 15:18 |
| *** ociuhandu has joined #openstack-nova | 15:18 | |
| stephenfin | dansmith: correct. the reason we didn't see it was because we test using ML2-OVS or ML2-LB (I don't know which) in the gate, not ML2-OVN | 15:18 |
| sean-k-mooney | well it was only added to neutron in 2018 | 15:18 |
| sean-k-mooney | https://github.com/openstack/neutron-lib/commit/24a0877d1d234830a36794388bb342d3d91b9230 | 15:18 |
| sean-k-mooney | which is well after we had deprecated it | 15:19 |
| dansmith | stephenfin: yeah I get that. are the tests _testing_ the proxy or using it out of convenience? | 15:19 |
| sean-k-mooney | 2.36 was newton | 15:19 |
| *** tbachman has quit IRC | 15:19 | |
| stephenfin | sean-k-mooney: Yup, it's not the proxy API that was wrong, it was how I updated things to stop using nova-network o.vo's | 15:19 |
| *** lpetrut has quit IRC | 15:20 | |
| sean-k-mooney | stephenfin: what im saying is the proxy api should not conatin that info | 15:20 |
| sean-k-mooney | we deprecated itn in 2016 that extention was only added in 2018 | 15:20 |
| dansmith | sean-k-mooney: I'm guessing we had it in there before via other mechanisms | 15:21 |
| sean-k-mooney | so there should be noting relying on it for fip port details | 15:21 |
| sean-k-mooney | ya proably | 15:21 |
| dansmith | sean-k-mooney: looking at the older code I was originally looking at, it was still putting those things in there but in a different way | 15:21 |
| stephenfin | dansmith: That I don't know. Maybe ralonsoh knows since he had a patch up against neutron to disable the broken job yesterday | 15:21 |
| sean-k-mooney | the correct way to look this up before was you get all the ports attach to the vm then get the fip for each port | 15:21 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add SYSTEM_READER role to servers actions API https://review.opendev.org/706179 | 15:22 |
| ralonsoh | stephenfin, yes, we are marking this CI job as non-voting | 15:22 |
| stephenfin | sean-k-mooney: yeah, what dansmith said. We were getting this information by always making a second call to neutron for the ports | 15:22 |
| stephenfin | ralonsoh: Which job is it? dansmith is curious as to what's actually using these deprecated APIs | 15:22 |
| sean-k-mooney | stephenfin: yes which is the correct way to do it if the extention is not available | 15:22 |
| stephenfin | sean-k-mooney: I know. I made a mistake when updating things because I didn't know port_details was optional and figured I should use it | 15:23 |
| dansmith | ralonsoh: job, but specifically which test(s) | 15:23 |
| ralonsoh | dansmith, an example of CI job failing https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_db7/705779/1/check/neutron-ovn-tempest-ovs-release/db7becd/testr_results.html | 15:23 |
| stephenfin | In hindsight, it should have been a separate patch :( | 15:23 |
| ralonsoh | stephenfin, dansmith those ones with exception message | 15:24 |
| ralonsoh | "<class 'nova.exception.PortNotFound'>" | 15:24 |
| dansmith | ralonsoh: okay so some specifically for that api, but also some general ones it looks like | 15:24 |
| *** spatel has joined #openstack-nova | 15:24 | |
| stephenfin | dansmith: I assume you're suggesting that the general ones shouldn't be using the proxy APIs? | 15:24 |
| spatel | sean-k-mooney: morning | 15:24 |
| stephenfin | i.e. test_server_basic_ops | 15:25 |
| sean-k-mooney | stephenfin: ya i think the secnairo test shoudl be updated | 15:25 |
| spatel | Could you check this CPU policy for vm - http://paste.openstack.org/show/789225/ | 15:25 |
| dansmith | stephenfin: well, maybe, but it's also maybe interesting that if the alternative is very onerous ... that tells us something | 15:25 |
| *** tbachman has joined #openstack-nova | 15:25 | |
| sean-k-mooney | the other way is not really that hard. its just more api requests | 15:26 |
| stephenfin | dansmith: that everyone is probably ignoring us and using these deprecated APIs anyway? :) | 15:26 |
| dansmith | stephenfin: like, if the alternative is 20 calls to neutron and a bunch of local mapping, then that may be an indication that we kinda messed up | 15:26 |
| dansmith | stephenfin: indeed | 15:26 |
| * stephenfin votes to remove all previous microversions | 15:27 | |
| stephenfin | less users = less bugs = more time for stephenfin to eat popcorn | 15:27 |
| *** gmann has joined #openstack-nova | 15:28 | |
| sean-k-mooney | the old way is just neutron port list device-id=<vm uuid> then you loop over that calling /v2.0/floatingips?port_id=<port uuid>) | 15:29 |
| sean-k-mooney | so you get the port detalis form the orginal call to list all the ports on a vm and then map it to the fip from the second call | 15:30 |
| dansmith | yeah, this is exactly what I'm talking about :) | 15:30 |
| sean-k-mooney | if you know the fip you jsut wrok backward first do a fip show get the port and then do a port show | 15:30 |
| sean-k-mooney | port_details on the fip is a proxy api in neutorn to get info form a different resouce | 15:31 |
| sean-k-mooney | the added it as an optional exteion to make it 1 call | 15:31 |
| *** tbachman has quit IRC | 15:31 | |
| sean-k-mooney | but each ml2 driver has to advertise support and ovn did not | 15:32 |
| kashyap | stephenfin: Hehe, figured as much. (And I like the expression) | 15:32 |
| *** ccamacho has quit IRC | 15:36 | |
| *** ociuhandu has quit IRC | 15:39 | |
| *** ociuhandu has joined #openstack-nova | 15:40 | |
| Sundar | sean-k-mooney: Re. your message "that was the latest set that were up at that point", it got updated on Jan 10 to remove the specific exception that you posted: https://review.opendev.org/#/c/698846/2/cyborg/common/nova_client.py. Please rebase if your env is older. Sorry for the trouble. | 15:43 |
| *** eharney has quit IRC | 15:43 | |
| openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs https://review.opendev.org/552924 | 15:46 |
| sean-k-mooney | Sundar: i used the local.conf i posted | 15:46 |
| sean-k-mooney | which was pulling in form a new branch | 15:46 |
| *** ociuhandu has quit IRC | 15:46 | |
| sean-k-mooney | ill double check | 15:47 |
| openstackgerrit | Merged openstack/nova master: libvirt: Rename _is_storage_shared_with to _is_path_shared_with https://review.opendev.org/693337 | 15:47 |
| sean-k-mooney | strang it seam to have checkout out master... | 15:48 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Merge unnecessary 'NovaProxyRequestHandlerBase' separation https://review.opendev.org/705655 | 15:49 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove 'run_once' helper https://review.opendev.org/705656 | 15:49 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tox: Integrate mypy https://review.opendev.org/676208 | 15:49 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: mypy: Add type annotations to 'nova.pci' https://review.opendev.org/676209 | 15:49 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: mypy: Add nova.cmd, nova.conf, nova.console https://review.opendev.org/705657 | 15:49 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: mypy: Add type annotations to top-level modules https://review.opendev.org/705658 | 15:49 |
| sean-k-mooney | Sundar: im using enable_plugin cyborg https://review.opendev.org/openstack/cyborg refs/changes/61/703261/1 | 15:50 |
| sean-k-mooney | that should have checkout the specific version | 15:50 |
| stephenfin | bauzas: Care to do me the honour? https://review.opendev.org/#/c/703796/ | 15:52 |
| bauzas | sure | 15:52 |
| bauzas | sean-k-mooney: efried: others, I'm glad to say I'm eventually done with https://review.opendev.org/552924 | 15:53 |
| sean-k-mooney | Sundar: it looks like there is either a devstack bug or the cybog plugin is nit support the gitref | 15:53 |
| *** spatel has quit IRC | 15:53 | |
| sean-k-mooney | Sundar: https://github.com/openstack/devstack/blob/ae73b23373f8673722fac2fbd349fa2075fda8d6/doc/source/plugins.rst#plugin-interface | 15:53 |
| efried | bauzas: that's what YOU think :P | 15:53 |
| bauzas | honestly, thanks you all | 15:53 |
| bauzas | but I'm dead with this spec :p | 15:54 |
| bauzas | no battery left. | 15:54 |
| * sean-k-mooney thinks about sending bauzas some deathwish coffee | 15:54 | |
| *** mdbooth has quit IRC | 15:55 | |
| sean-k-mooney | bauzas: https://www.deathwishcoffee.com/ i have a friend that quite likes it | 15:55 |
| Sundar | sean-k-mooney: I think it is the latter: https://opendev.org/openstack/cyborg/src/branch/master/devstack/lib/cyborg#L85 | 15:55 |
| bauzas | sean-k-mooney: I've been told that an english tea is *always* black with milk | 15:56 |
| *** mdbooth has joined #openstack-nova | 15:56 | |
| bauzas | can't tell what an english coffee is then | 15:56 |
| sean-k-mooney | Sundar: ok i can just define CYBORG_BRANCH in my local.conf | 15:56 |
| sean-k-mooney | but you actully dont need to and should not clone the git repo in the plugin | 15:57 |
| sean-k-mooney | that is done for you by devstack | 15:57 |
| dansmith | sean-k-mooney: devstack or devstack-gate? | 15:58 |
| sean-k-mooney | bauzas: are you confusing me with an english man becasue as a irish eu citizen i coudl be offended by that. also irish <anything> usally meens add good wiskey | 15:59 |
| sean-k-mooney | *irish <beverage or food item> | 15:59 |
| sean-k-mooney | dansmith: devstack | 16:00 |
| dansmith | sean-k-mooney: the other projects have git_clone calls in their modules | 16:00 |
| *** gibi_on_call is now known as gibi | 16:00 | |
| sean-k-mooney | so in the gate they are disabled via a diffenert parmater | 16:00 |
| mnaser | ok, i'm absolutely hitting a wall. nova list --all-tenants returns only 11 vms, but i have 1297 vm_state='ACTIVE' instances. instance_mappings are correct in nova_api | 16:01 |
| sean-k-mooney | but the plugin interface specific was designed for devstack to do the clone of the pugin repo | 16:01 |
| mnaser | i double checked the db creds inside nova_api.cell_mappings | 16:01 |
| sean-k-mooney | anyway its not a big deal | 16:01 |
| mnaser | the crazier thing is running something like placement heal allocations only returns 11 instances only, which that is doing objects.InstaceList.... | 16:02 |
| *** redrobot has joined #openstack-nova | 16:02 | |
| mnaser | so its note even an API level problem, it's in the db layer (i assume objects.* API inside nova-manage commands talks directly to db, not via conductor?) | 16:02 |
| dansmith | mnaser: nova-manage does not talk to conductor | 16:03 |
| mnaser | ok, so that's what i thought as well | 16:04 |
| sean-k-mooney | nova manage will try and get the db infor form your nova.conf | 16:04 |
| sean-k-mooney | is it perhaps geting the wrong db info | 16:04 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Configure max-line-length to 88 https://review.opendev.org/706311 | 16:04 |
| mnaser | i used the mysql client to use the same credentials in nova.conf and i see all the instances | 16:05 |
| * stephenfin throws cat among pigeons | 16:05 | |
| dansmith | mnaser: one non-cell0 cell? | 16:05 |
| stephenfin | bauzas: thanks :) | 16:05 |
| mnaser | yes, only one non-cell0 cell | 16:05 |
| dansmith | mnaser: and the 11 instances you do see, are they from that cell or in cell0? | 16:06 |
| bauzas | sean-k-mooney: I've been told that by an irish man in Grenoblre | 16:06 |
| mnaser | i ddint check all of them but one of them is from that cell dansmith (aka inside `nova`) | 16:06 |
| * bauzas remembers a song | 16:06 | |
| dansmith | mnaser: is it complete? meaning, it's not returned as a shell instance with no real state information? | 16:07 |
| mnaser | oh good call | 16:07 |
| mnaser | nova show brings even the user_data | 16:07 |
| mnaser | so it sounds like its pulling it in from nova and not nova_api | 16:07 |
| dansmith | that's not the same | 16:07 |
| dansmith | when you see it in nova list, does it have vm_state? | 16:07 |
| mnaser | yes | 16:07 |
| dansmith | of active/ | 16:08 |
| mnaser | yep | 16:08 |
| mnaser | and those instances were all created post upgrade so its almost like the db just like.. only started reading specific records | 16:08 |
| mnaser | now to note this is a s=>t upgrade, but those have been pretty smooth : | 16:08 |
| dansmith | so I think you can rule out cells, cell_mappings, instance_mapping related issues | 16:09 |
| mnaser | well to me the weird thing is how objects.InstanceList which is called inside heal placement allocation.. only sees those 11 instances | 16:09 |
| mnaser | yet hypervisors are happily complaining like "66 vms on host but 65 in db" and things like that still happen | 16:10 |
| dansmith | yep, another reason I think you can rule out the cell infrastructure | 16:10 |
| mnaser | and if i hit a "Show" on a specific instance, i can retrieve it (just found out) | 16:10 |
| mnaser | even if i dont see it listed | 16:10 |
| mnaser | oh wth | 16:11 |
| dansmith | show finds instances in a pretty different way, | 16:11 |
| dansmith | but it also means that instance_mappings is correct | 16:11 |
| mnaser | it looks like a db migration has failed | 16:11 |
| mnaser | ..but it somehow KEPT going with the migrations? | 16:11 |
| dansmith | what migration failed? | 16:11 |
| mnaser | http://paste.openstack.org/show/789229/ | 16:12 |
| * dansmith wonders why people *ever* use paste.o.o | 16:12 | |
| dansmith | wow, that's interesting | 16:12 |
| mnaser | dansmith: pastebin.com sounds like it only contains leaks of *stuff* | 16:12 |
| mnaser | let me look at what migration is that | 16:12 |
| mnaser | so i think it failed but the version was bumped | 16:13 |
| dansmith | "add instances hidden" | 16:13 |
| mnaser | and then it kept going after | 16:13 |
| dansmith | yeah, that'd do it | 16:13 |
| *** ociuhandu has joined #openstack-nova | 16:13 | |
| dansmith | although, it doesn't use server_default so it's a little weird | 16:13 |
| dansmith | mnaser: does your instances table have a "hidden" column? | 16:13 |
| mnaser | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/migrate_repo/versions/399_add_instances_hidden.py <- for whoever is following and my notes | 16:14 |
| mnaser | yes i'm checking that now | 16:14 |
| dansmith | right | 16:14 |
| mnaser | i am gonna assume | 16:14 |
| mnaser | it does | 16:14 |
| dansmith | the thing is that default is not server_default, so I would expect you're not missing that | 16:14 |
| mnaser | tahts probably why new instances are ok | 16:14 |
| dansmith | mnaser: you're assuming or you checked? | 16:14 |
| mnaser | and old ones are not | 16:14 |
| mnaser | (iam going to check now) | 16:14 |
| mnaser | but that might make sense as to why the new instances (11) show up | 16:15 |
| dansmith | hidden defaults to false on the code side, which means if it exists, they should all default to hidden=False, | 16:15 |
| dansmith | and if it didn't exist, you should be getting query errors | 16:15 |
| mnaser | | hidden | tinyint(1) | YES | | NULL | | | 16:15 |
| dansmith | I think that's the right schema | 16:15 |
| mnaser | there is 57 instances with hidden=0 | 16:15 |
| mnaser | and the rest are null. | 16:15 |
| mnaser | id be update instances set hidden=0; would fix it | 16:16 |
| dansmith | right, and those NULLs should be presumed =False by the python side | 16:16 |
| dansmith | well, | 16:16 |
| dansmith | maybe to get you out of the jam, but... that's not what *should* be happening | 16:16 |
| mnaser | right yeah, you're right, hidden being null should default to false | 16:17 |
| mnaser | let me get out of my jam and i will look at the code | 16:17 |
| dansmith | oooh, | 16:17 |
| dansmith | this might be a bug | 16:17 |
| dansmith | a really bad one | 16:17 |
| dansmith | let it be known that mriedem may have made a mistake | 16:18 |
| mnaser | he left us with a fun one :( | 16:18 |
| dansmith | it's not a hard fix | 16:18 |
| *** READ10 has joined #openstack-nova | 16:19 | |
| dansmith | mnaser: is this the first s->t upgrade for you? | 16:19 |
| mnaser | dansmith: no, and i didnt hit the first time | 16:19 |
| mnaser | but i also dont think i hit that db migration issue that time too | 16:19 |
| *** dtantsur is now known as dtantsur|afk | 16:19 | |
| dansmith | oh, hrm | 16:19 |
| dansmith | the db migration doesn't seem to have left the schema incorrect though | 16:19 |
| mriedem | fake news | 16:21 |
| mnaser | lol | 16:21 |
| mnaser | dansmith: i think it acutally tried to do an update and set things hidden=0 | 16:21 |
| dansmith | mnaser: can you compare that bit of schema to one of your other clusters, and also look at the hidden values on those instances/ | 16:21 |
| mnaser | (this is a much bigger cloud) | 16:21 |
| mnaser | and my stupid update instances set hidden=0 gave me this | 16:21 |
| mnaser | ERROR 1180 (HY000): Got error 90 "Message too long" during COMMIT | 16:21 |
| dansmith | mnaser: okay it shouldn't be doing that | 16:21 |
| mnaser | so its almost like once it does the schema update, it bumps the version, and then tries to backfill default values | 16:21 |
| mnaser | let me check the value of hidden in the other train deployemnt | 16:22 |
| dansmith | yeah, that's not the intent of that migration for sure | 16:22 |
| openstackgerrit | Merged openstack/nova-specs master: FUP: Fixed the invalid index in References https://review.opendev.org/705933 | 16:22 |
| dansmith | maybe in some situations sqla-migrate actually tries to patch up data for you on existing tables, which is a real problem | 16:23 |
| mnaser | ok yeah | 16:23 |
| mnaser | the other deployment has hidden=0 for every single record | 16:23 |
| dansmith | and if so, this is a good example of why we don't do data transformation (intentionally) in schema migrations | 16:23 |
| dansmith | okay damn | 16:23 |
| *** TxGirlGeek has joined #openstack-nova | 16:23 | |
| dansmith | so we need to change that migration retroactively, and patch the load code to properly interpret the null as the default (which sqla does not do with default=) | 16:23 |
| *** psachin has quit IRC | 16:24 | |
| mnaser | so i think what happened is: schema updated and column added, db version bumped, backfill values to 0 | 16:24 |
| mnaser | and then i failed in the backfill, so the next run it just started from the next migration | 16:24 |
| dansmith | yeah, and on your big cloud, that timed out | 16:24 |
| dansmith | yeah, probably because the backfill wasn't technically a schema fail | 16:24 |
| dansmith | which is also scary | 16:24 |
| mnaser | i think your theory is best | 16:25 |
| mnaser | s/theory/solution/ | 16:25 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add MigrationTypeField https://review.opendev.org/706013 | 16:25 |
| dansmith | mnaser: can you file a bug for me to write this up against? | 16:25 |
| mnaser | sure | 16:25 |
| dansmith | thanks | 16:25 |
| mriedem | fwiw i count 5 non-api db schema migrations that add a new boolean column with default=False, | 16:26 |
| dansmith | mriedem: yup | 16:27 |
| mriedem | if it's actually backfilling, then maybe just not an issue for those b/c they are smaller tables | 16:27 |
| dansmith | mriedem: we've been doing this without realizing | 16:27 |
| dansmith | yu[ | 16:27 |
| dansmith | will need a poison test too | 16:27 |
| dansmith | like the others I have for alter operations | 16:27 |
| *** TxGirlGeek has quit IRC | 16:27 | |
| mriedem | i sort of remember this discussion with jaypipes on https://review.opendev.org/#/c/557958/ | 16:29 |
| mriedem | which uses server_default | 16:29 |
| *** slaweq_ has joined #openstack-nova | 16:29 | |
| *** N3l1x has joined #openstack-nova | 16:30 | |
| dansmith | I guess server_default would be one option, but it involves another schema change which I think is less good since we have to backport this | 16:30 |
| *** slaweq has quit IRC | 16:31 | |
| *** spatel has joined #openstack-nova | 16:31 | |
| mriedem | https://docs.sqlalchemy.org/en/13/core/metadata.html#sqlalchemy.schema.Column.params.default | 16:34 |
| mriedem | "which will be invoked upon insert if this column is otherwise not specified in the VALUES clause of the insert" | 16:34 |
| mriedem | didn't realize it would backfill though | 16:34 |
| dansmith | right, I always assume it fills on read too, but just insert | 16:35 |
| mriedem | https://docs.sqlalchemy.org/en/13/core/metadata.html#sqlalchemy.schema.Column.params.server_default - "A FetchedValue instance, str, Unicode or text() construct representing the DDL DEFAULT value for the column." | 16:35 |
| mriedem | so yeah, i guess never use default in migrations | 16:35 |
| dansmith | right | 16:35 |
| mriedem | but is default in the models OK? | 16:35 |
| mriedem | i thought it was | 16:35 |
| *** gyee has joined #openstack-nova | 16:35 | |
| dansmith | it is,m | 16:36 |
| mriedem | mnaser: just purge your db :) | 16:36 |
| dansmith | and I just tested that the model sync test does not fail you for having default on the column and not the migration | 16:36 |
| mnaser | mriedem: you're not wrong | 16:36 |
| mnaser | dansmith: https://bugs.launchpad.net/nova/+bug/1862205 | 16:36 |
| openstack | Launchpad bug 1862205 in OpenStack Compute (nova) "Instances not visible when hidden=NULL" [Undecided,New] | 16:36 |
| *** nweinber has quit IRC | 16:36 | |
| dansmith | mnaser: thanks | 16:37 |
| *** nweinber has joined #openstack-nova | 16:37 | |
| mnaser | dansmith: if we come up/discuss a solution, i'm happy to work out a fix, i have bigger clouds to upgrade ahead of this sooo | 16:37 |
| dansmith | mnaser: I've got a patch started, but yeah, will appreciate real testing for sure | 16:38 |
| mnaser | dansmith: ok cool, i can try and help reviewing to my ability and id be able to provide some feedback on a cloud with 1.5 times the # of records (but only in 2 weeks :X) | 16:39 |
| mriedem | i've still got core if you need to poke someone later | 16:39 |
| dansmith | cool | 16:39 |
| *** rpittau is now known as rpittau|afk | 16:40 | |
| melwitt | I can help with review too if needed | 16:41 |
| *** tbachman has joined #openstack-nova | 16:43 | |
| *** vishalmanchanda has quit IRC | 16:50 | |
| gibi | dansmith, efried, Sundar: I agree with the discussed direction in https://review.opendev.org/#/c/631243/55 See details in my review | 16:52 |
| gibi | efried: is there other patches in the series that need my eyes? I can spend some extra time on the series tomorrow if needed | 16:54 |
| *** tesseract has quit IRC | 16:54 | |
| *** eharney has joined #openstack-nova | 16:59 | |
| *** TxGirlGeek has joined #openstack-nova | 17:03 | |
| *** tbachman has quit IRC | 17:03 | |
| *** iurygregory has quit IRC | 17:07 | |
| *** READ10 has quit IRC | 17:08 | |
| openstackgerrit | Merged openstack/nova master: Don't error out on floating IPs without associated ports https://review.opendev.org/706213 | 17:12 |
| openstackgerrit | Merged openstack/nova master: nova-net: Update API reference guide https://review.opendev.org/703796 | 17:12 |
| openstackgerrit | Merged openstack/nova stable/queens: Add functional recreate test for bug 1852610 https://review.opendev.org/699705 | 17:12 |
| openstack | bug 1852610 in OpenStack Compute (nova) queens "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Low,In progress] https://launchpad.net/bugs/1852610 - Assigned to Matt Riedemann (mriedem) | 17:12 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove native LUKS compat code https://review.opendev.org/669121 | 17:16 |
| *** ivve has quit IRC | 17:22 | |
| efried | gibi: Well, I've been hoping that I would get the chance to review the series from top to bottom and be the second +2 (with dansmith) but I'm no longer sure I'm going to be able to get to it soon. So if you have time, your help would be much appreciated. But I would say you shouldn't push other stuff out of the way for it. | 17:25 |
| *** ociuhandu has quit IRC | 17:29 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix instance.hidden migration and querying https://review.opendev.org/706331 | 17:30 |
| dansmith | mnaser: mriedem melwitt ^ haven't run all tests yet, so it might break other things, but that's what I'm thinking as a backportable change | 17:31 |
| * dansmith notes that this is the M-club apparently | 17:31 | |
| melwitt | in da club | 17:31 |
| * sean-k-mooney ignore the british meaning of that https://www.urbandictionary.com/define.php?term=in%20the%20pudding%20club | 17:33 | |
| *** evrardjp has quit IRC | 17:33 | |
| dansmith | gross. | 17:34 |
| *** evrardjp has joined #openstack-nova | 17:34 | |
| *** tosky has quit IRC | 17:34 | |
| *** READ10 has joined #openstack-nova | 17:34 | |
| melwitt | lol wtf. never heard that before | 17:34 |
| sean-k-mooney | in the club to mean pregnant ya ist an old old way of saying that in polite company when gossiping | 17:35 |
| *** tbachman has joined #openstack-nova | 17:36 | |
| melwitt | interesting. here it means like at the dance club, in a bunch of rap songs | 17:36 |
| dansmith | as someone who feels like society favors those who procreate unfairly, I definitely see it as "a club" in the negative sense | 17:36 |
| sean-k-mooney | yes as i said it was used when gossiping about someone behind there back so it had a negitive over tone in many cases | 17:38 |
| dansmith | that's not really what I meant :) | 17:38 |
| melwitt | lol | 17:38 |
| mriedem | i'm in da costco club | 17:39 |
| dansmith | mnaser: how many instances in the database that failed, and how many in the one that didn't? | 17:39 |
| mriedem | if you need a years supply of toilet paper i'm your guy | 17:39 |
| sean-k-mooney | lol | 17:39 |
| melwitt | me too, Costco Executive Club | 17:39 |
| *** mlavalle has joined #openstack-nova | 17:43 | |
| *** ociuhandu has joined #openstack-nova | 17:44 | |
| melwitt | maybe this is too out there but I wonder if this might be related to the cell timeout gate bug (affecting only grenade jobs) | 17:47 |
| dansmith | the first query after upgrade works, so ... I don't think so | 17:48 |
| dansmith | and also, mnaser said it worked fine for his smaller cluster, which I'm sure is waaaay larger than a grenade setup with a tempest-run worth of instances in the db | 17:48 |
| melwitt | ok. I wondered if there could be something to do with the table rewriting in the background or something | 17:48 |
| dansmith | surely hope not :) but it seems like it was synchronous in mnaser's case | 17:49 |
| melwitt | because when I dug into that, it seemed like mysql just overloaded somehow and logging errors "reading communication packets" | 17:49 |
| melwitt | ok | 17:49 |
| *** ociuhandu_ has joined #openstack-nova | 17:52 | |
| *** martinkennelly has quit IRC | 17:53 | |
| *** ociuhandu has quit IRC | 17:55 | |
| *** ociuhandu_ has quit IRC | 17:57 | |
| sean-k-mooney | any table updates or data migration should be taking a full table write lock. reads should see the old data until the transaction completes | 17:59 |
| *** ociuhandu has joined #openstack-nova | 18:00 | |
| sean-k-mooney | at least if you are using the innodb schema for mysql | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| *** ociuhandu has quit IRC | 18:06 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix instance.hidden migration and querying https://review.opendev.org/706331 | 18:14 |
| *** jmlowe has quit IRC | 18:18 | |
| *** amoralej is now known as amoralej|off | 18:22 | |
| *** ralonsoh has quit IRC | 18:22 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix instance.hidden migration and querying https://review.opendev.org/706331 | 18:28 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Disallow specifying a default= on new Columns in schema changes https://review.opendev.org/706351 | 18:34 |
| dansmith | ^ makes it impossible to use Column(default=) in migrations after 399 | 18:34 |
| dansmith | efried: did you catch the earlier discussion about the relatively serious upgrade-corrupts-database bug in train? | 18:35 |
| efried | no | 18:35 |
| dansmith | efried: on largeish deployments, upgrading to Train can make all your instances hidden silently | 18:35 |
| dansmith | we're going to want to backport my fix and release a train update with definitive action at some point here soon | 18:36 |
| sean-k-mooney | dansmith: do we have that fix downstream by the way | 18:36 |
| dansmith | sean-k-mooney: we don't have it upstream yet, so..no. | 18:37 |
| sean-k-mooney | ok cause you know what day it is :) | 18:37 |
| efried | dansmith: do I need to review things? | 18:38 |
| dansmith | efried: you're welcome to.. I think we're looking for a test run, a report from mnaser and melwitt/mriedem are also going to review | 18:38 |
| dansmith | efried: https://review.opendev.org/#/c/706331/ | 18:38 |
| dansmith | just for YI if nothing else | 18:38 |
| efried | ack | 18:42 |
| mnaser | catching up on buffer... | 18:45 |
| mnaser | dansmith: ~220kish it didnt fail, ~340kish it failed, mariadb 10.3.something i believe | 18:45 |
| dansmith | mnaser: okay thanks. I guess if anyone asks me I'll say "somewhere around 330k" | 18:46 |
| mnaser | yeah and in this case it would be nice if we purged records so | 18:46 |
| dansmith | er, "300k" I mean | 18:46 |
| mnaser | important note that this doesnt mean "300k active instance" but "300k db records" | 18:47 |
| dansmith | yup | 18:47 |
| *** ivve has joined #openstack-nova | 18:47 | |
| *** jmlowe has joined #openstack-nova | 18:48 | |
| *** dpawlik has quit IRC | 18:50 | |
| mnaser | dansmith: i wonder if this might start a whack-a-mole of references to hidden=false or hidden=null | 18:51 |
| efried | mnaser: I was just leaving a comment to that effect. | 18:51 |
| *** Liang__ has quit IRC | 18:52 | |
| dansmith | mnaser: maybe except I think these are the only two places :) | 18:52 |
| efried | for now | 18:52 |
| mnaser | yeah, thing is i have no better solution | 18:52 |
| dansmith | not really, because everything else sits above the db layer | 18:52 |
| sean-k-mooney | we porably should be normalising that when we load if form the db | 18:52 |
| dansmith | this is what the db and object apis are *for* | 18:53 |
| dansmith | sean-k-mooney: we are, that's what I'm doing | 18:53 |
| sean-k-mooney | dansmith: in https://review.opendev.org/#/c/706331/ | 18:53 |
| dansmith | the second case is specifically doing a very targeted non-ORM query | 18:53 |
| dansmith | otherwise this would be a single place | 18:53 |
| *** mvkr has quit IRC | 18:54 | |
| *** jmlowe has quit IRC | 18:58 | |
| *** tosky has joined #openstack-nova | 18:59 | |
| *** imacdonn has joined #openstack-nova | 19:00 | |
| dansmith | mnaser: added comments to the test to mirror your self-discovery | 19:01 |
| dansmith | efried: added the hidden=False case to the test | 19:01 |
| dansmith | gonna wait to push it up so we can get a run on it | 19:01 |
| dansmith | efried: also added that ref to the commit message | 19:01 |
| efried | ack. dansmith when I re-add the default in the poison patch and run just that test, it works, but when I run the whole suite, the exception is weird | 19:02 |
| efried | AttributeError: hidden | 19:02 |
| dansmith | hmm, that's weird | 19:02 |
| mriedem | dansmith: question in that test | 19:02 |
| dansmith | I was running just one | 19:02 |
| dansmith | mriedem: replied | 19:05 |
| openstackgerrit | Merged openstack/nova stable/queens: Add functional recreate revert resize test for bug 1852610 https://review.opendev.org/699708 | 19:05 |
| openstack | bug 1852610 in OpenStack Compute (nova) queens "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Low,In progress] https://launchpad.net/bugs/1852610 - Assigned to Matt Riedemann (mriedem) | 19:05 |
| efried | dansmith: repro is: | 19:05 |
| efried | activate the venv | 19:05 |
| efried | fails with the expected message: stestr run -n nova.tests.unit.db.test_migrations.TestNovaMigrationsSQLite.test_walk_versions | 19:05 |
| efried | fails with AttributeError: stestr run -n nova.tests.unit.db.test_migrations | 19:05 |
| efried | ...venv is py36 | 19:06 |
| dansmith | efried: oh I was running the whole test_migrations module myself and wasn't seeing the error, but let me try again | 19:06 |
| dansmith | I'm on py37 | 19:06 |
| * efried creates py37 venv... | 19:07 | |
| efried | dansmith: zuul agrees with me | 19:07 |
| efried | but it also fails despite the migration having been 'fixed'. | 19:09 |
| dansmith | hrm | 19:10 |
| dansmith | what version of sqla is in your venv? | 19:10 |
| efried | I just repro'd with a fresh py37 env, so whatever's in requirements. | 19:11 |
| dansmith | can you look so I can compare? | 19:11 |
| efried | SQLAlchemy==1.3.13,sqlalchemy-migrate==0.13.0,sqlparse==0.3.0 | 19:11 |
| sean-k-mooney | tox -e py37 -- nova.tests.unit.db.test_migrations.TestNovaMigrationsSQLite.test_walk_versions | 19:11 |
| sean-k-mooney | passes form me with dans change | 19:11 |
| dansmith | okay I just did a fuuuul run and that failed like yours but doesn't fail for me on just the module | 19:12 |
| dansmith | so must be an ordering thing or something? | 19:12 |
| efried | or something. | 19:12 |
| mriedem | shouldn't column_default_banned default to False for everything before 399? | 19:12 |
| efried | I skimmed looking for other monkey patches of Column, but didn't see any at a glance. | 19:12 |
| mriedem | i guess the default maybe doesn't matter since the test is always passing a value | 19:12 |
| dansmith | mriedem: it does.. set it to zero and it will fail for 216_havana | 19:12 |
| mriedem | yeah it was the default in the fixture that i was thinking about - the default isn't used since the test always overwrites it | 19:13 |
| dansmith | correct | 19:13 |
| efried | if I'm being super picky, either remove the default or specify the kwarg name in the call. | 19:13 |
| dansmith | efried: you see that the original doesn't specify either I assume | 19:14 |
| efried | but that's so nittish for a test I won't even put it in the review | 19:14 |
| dansmith | the existing argument I mean | 19:14 |
| efried | hah, the number of times we've busted Sundar for following a precedent that was wrong... | 19:14 |
| dansmith | python2 used to not allow multi-line with statements so it had to be short | 19:14 |
| mriedem | only other comment on the actual fix is do you want a reno to advertise it since it's going to be backported | 19:14 |
| dansmith | mriedem: yep | 19:15 |
| mriedem | can i -1 for it to get stats!? | 19:15 |
| mriedem | after that i'd be +2 on the fix | 19:15 |
| mriedem | well i guess there is no test for the quota thing but i'll leave that up to you guys, | 19:16 |
| mriedem | in my new job tests are a low concern for people so i'm getting used to not asking for them. | 19:16 |
| sean-k-mooney | efried: dansmith ya i also can repoduce the error with "tox -e py37 -- nova.tests.unit.db.test_migration" so this is like due to incorrect mocking | 19:16 |
| dansmith | mriedem: I didn't because we kinda already test the non-null site, and didn't want to have to replicate the raw-sql create of a null-having record for that too, but I can | 19:16 |
| mriedem | up to you, like i said, any tests are good :) | 19:17 |
| mriedem | mnaser: now that you're getting to train does this mean you're going to start cross-cell resizing like a mad man? | 19:19 |
| mriedem | oh wait, did that land in ussuri? | 19:20 |
| mriedem | ah right it's available in ussuri, nvm me | 19:20 |
| dansmith | mriedem: he would need multiple cells to cross :) | 19:21 |
| mriedem | i know, it was implied as a nudge | 19:21 |
| mriedem | because there has been some vexxhost multi-cell chatter for awhile | 19:21 |
| mriedem | does cern still upgrade nova? | 19:21 |
| *** martinkennelly has joined #openstack-nova | 19:31 | |
| *** jmlowe has joined #openstack-nova | 19:32 | |
| dansmith | ah I see the problem | 19:37 |
| dansmith | I dunno why it doesn't always happen, but it's also not going to be an easy fix :/ | 19:37 |
| mnaser | mriedem: hah. yeah,ii think for cross-cell there's a few mountains to climb first | 19:39 |
| mnaser | like figuring out glance with multiple backends and nova cells with different ceph backends in each one | 19:40 |
| mnaser | and time | 19:40 |
| dansmith | mnaser: that's in the works fwiw | 19:40 |
| dansmith | glance has to do a thing first and then I plan to get on the nova side | 19:40 |
| dansmith | efried: mriedem: the reno for this should just have an item in "fixes:" ? or critical? or upgrade? | 19:46 |
| *** jmlowe has quit IRC | 19:47 | |
| mriedem | either fixes or upgrade, or both i guess? | 19:47 |
| mriedem | so i guess if you do upgrade, you can say if you haven't rolled to this point yet make sure you do first rather than like train GA, but if you have already upgraded to train GA and hit this issue, you can manually update the records (maybe after an archive/purge)? | 19:48 |
| mriedem | my guess is people that hit this will be looking for some guidance on what to do | 19:48 |
| dansmith | aight | 19:52 |
| *** openstackstatus has joined #openstack-nova | 19:56 | |
| *** ChanServ sets mode: +v openstackstatus | 19:56 | |
| melwitt | does anyone know if there's a similar issue with a column default when using alembic for migrations? | 20:03 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver. https://review.opendev.org/631245 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Enable hard/soft reboot with accelerators. https://review.opendev.org/697940 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Enable start/stop of instances with accelerators. https://review.opendev.org/699553 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Enable and use COMPUTE_ACCELERATORS trait. https://review.opendev.org/699554 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Bump compute rpcapi version and reduce Cyborg calls. https://review.opendev.org/704227 | 20:04 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999 | 20:04 |
| mriedem | melwitt: i'm not actually sure if it's sqlalchemy or sqlalchemy-migrate that is applying that default value to existing records, | 20:20 |
| mriedem | probably need to ask zzzeek | 20:20 |
| dansmith | there are two things here: first reading the null values and needing to do the defaulting is a SQLA thing, not related to alembic | 20:21 |
| dansmith | second is the application of the default to the existing rows, which could be different | 20:21 |
| melwitt | asking because the proposed consumer_types table in placement is specifying a default column value https://review.opendev.org/#/c/669170/10/placement/db/sqlalchemy/alembic/versions/422ece571366_add_consumer_types_table.py@83 | 20:21 |
| melwitt | ok | 20:21 |
| dansmith | melwitt: should be pretty easy to test.. I'm sure you have a devstack with that applied for your own testing | 20:21 |
| dansmith | melwitt: just create rows without the patch applied, then apply and roll over that migration and see if the field for existing rows is NULL or the default | 20:22 |
| melwitt | yes ... right ... | 20:22 |
| *** damien_r has quit IRC | 20:24 | |
| mriedem | bingo bango https://opendev.org/x/sqlalchemy-migrate/src/branch/master/migrate/changeset/schema.py#L594 | 20:25 |
| mriedem | dansmith: there is the proof | 20:25 |
| *** maciejjozefczyk has quit IRC | 20:25 | |
| mriedem | so you *could* use default= in schema migrations but you'd have to also set populate_default=False | 20:26 |
| dansmith | yeah, but no | 20:26 |
| dansmith | not sure what purpose that would serve | 20:26 |
| dansmith | if we shared Column() defs with migrations and models or something maybe | 20:26 |
| dansmith | but the models in sync test didn't even fail for me | 20:26 |
| *** eharney has quit IRC | 20:30 | |
| melwitt | I'm realizing there's a difference between 'default' and 'server_default'? the placement table is using 'server_default' | 20:31 |
| melwitt | https://stackoverflow.com/questions/14002631/why-isnt-sqlalchemy-default-column-value-available-before-object-is-committed#14013090 | 20:35 |
| zzzeek | melwitt / mriedem not reading everyhing but when you add a column to a database that has a default value and it's "not null", the DB adds that default. that is how you get a MySQL migration that is very slow for large tables btw | 20:35 |
| *** gmann has quit IRC | 20:36 | |
| *** damien_r has joined #openstack-nova | 20:37 | |
| *** gmann has joined #openstack-nova | 20:37 | |
| melwitt | zzzeek: is that true regardless of whether it's specified as a 'default' vs a 'server_default'? will 'server_default' also try to backfill in already existing records that do not have a value set? | 20:38 |
| zzzeek | melwitt: oh...server default only, sorrhy | 20:38 |
| zzzeek | melwitt: for "default" that is not a server default, I have no idea what sqlalhcemyt-migrate does | 20:38 |
| zzzeek | i'd be surprised if they use it, though, because the "add not null column / populate server default" is necessarily atomic | 20:38 |
| zzzeek | you can't do that from the client using a python-side default | 20:39 |
| melwitt | zzzeek: ok, it also does a backfill, this is the patch where it's being fixed if you're curious https://review.opendev.org/706331 | 20:39 |
| zzzeek | only if migrate takes the crazy insane step of making the column as nullable first, then populating, then not-nulling | 20:39 |
| melwitt | zzzeek: this is the link from migrate https://opendev.org/x/sqlalchemy-migrate/src/branch/master/migrate/changeset/schema.py#L594 | 20:39 |
| zzzeek | melwitt: wow, yuck :) | 20:40 |
| melwitt | lol | 20:40 |
| zzzeek | i hate migrate | 20:40 |
| zzzeek | b.c. you know that fails if the DB is live and new rows still getting added | 20:41 |
| melwitt | mnaser knows ;) | 20:41 |
| *** martinkennelly has quit IRC | 20:42 | |
| mnaser | yeah, i do | 20:47 |
| *** efried has quit IRC | 20:47 | |
| *** efried has joined #openstack-nova | 20:48 | |
| *** irclogbot_0 has quit IRC | 20:53 | |
| *** irclogbot_2 has joined #openstack-nova | 20:54 | |
| *** martinkennelly has joined #openstack-nova | 20:58 | |
| *** slaweq__ has joined #openstack-nova | 21:02 | |
| *** slaweq_ has quit IRC | 21:03 | |
| *** READ10 has quit IRC | 21:03 | |
| *** jmlowe has joined #openstack-nova | 21:12 | |
| sean-k-mooney | Sundar: i redeploy and have been able to boot a vm with the fake cyborg driver | 21:15 |
| sean-k-mooney | http://paste.openstack.org/show/789247/ | 21:16 |
| efried | nice | 21:16 |
| *** Sundar has quit IRC | 21:18 | |
| *** nweinber has quit IRC | 21:18 | |
| sean-k-mooney | i will start trying different life cycle operation and testing and recording info on placement allcoation, db dumps and the like | 21:18 |
| sean-k-mooney | i have a bash script i have written to automate this so ill script up a few test cases. and do some manually | 21:19 |
| sean-k-mooney | since i have created a cyborg flavor i shoudl be able to run some of the standard tempest tests with that flavor | 21:20 |
| *** eharney has joined #openstack-nova | 21:21 | |
| *** slaweq__ has quit IRC | 21:27 | |
| *** eharney has quit IRC | 21:30 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix instance.hidden migration and querying https://review.opendev.org/706331 | 21:39 |
| lifeless | stephenfin: hi, what do you need? mtreinish has commit rights on subunit | 21:39 |
| *** spatel has quit IRC | 21:50 | |
| *** jmlowe has quit IRC | 22:00 | |
| *** damien_r has quit IRC | 22:21 | |
| *** jmlowe has joined #openstack-nova | 22:24 | |
| *** martinkennelly has quit IRC | 22:27 | |
| efried | mnaser: did your +1 at PS3 here https://review.opendev.org/#/c/706331/ indicate that you had successfully tested this locally? | 22:30 |
| *** ivve has quit IRC | 22:31 | |
| *** jmlowe has quit IRC | 22:32 | |
| *** jmlowe has joined #openstack-nova | 22:35 | |
| *** mriedem has left #openstack-nova | 22:37 | |
| *** mriedem has joined #openstack-nova | 22:38 | |
| mriedem | dansmith: i've got to run but will take a look at the latest later tonight | 22:38 |
| *** mriedem has left #openstack-nova | 22:38 | |
| dansmith | efried: I think he's going to have to apply it at the time he does his next upgrade which might not be for a while, it sounded like | 22:41 |
| melwitt | dansmith: wouldn't your change unhide the instances for him today that are being incorrectly hidden? or are you saying he already fixed that via manual db update | 22:44 |
| dansmith | melwitt: he already fixed up his db, as I understand it | 22:44 |
| melwitt | gotcha | 22:45 |
| *** jmlowe has quit IRC | 22:45 | |
| *** jmlowe has joined #openstack-nova | 22:48 | |
| *** tkajinam has joined #openstack-nova | 22:54 | |
| *** jmlowe has quit IRC | 22:56 | |
| *** tkajinam has quit IRC | 22:57 | |
| *** tkajinam has joined #openstack-nova | 22:57 | |
| sean-k-mooney | dansmith: i can check the code but are we not storing the resouce requests form the cybrog device profile in the request spec? | 23:01 |
| sean-k-mooney | dansmith: im seeing "requested_resources": null in the request spec for the cyborg nova instance | 23:03 |
| sean-k-mooney | the embeded flavor has "accel:device_profile": "FakeDeviceProfile" | 23:04 |
| *** jmlowe has joined #openstack-nova | 23:04 | |
| sean-k-mooney | however since we are not storing the groups if you change the device profil after the fact and we migration and instance or did something else that would need us to call plamcnet wwe would have to go back to cyborg which could have changed | 23:05 |
| *** damien_r has joined #openstack-nova | 23:16 | |
| dansmith | sean-k-mooney: I think that's the idea | 23:18 |
| dansmith | sean-k-mooney: you live migrate, scheduler calls placement with a new set of resources constructed from the device profile and what cyborg told you when you asked | 23:18 |
| dansmith | sean-k-mooney: maybe we need to be doing something like examining the existing ARQs to generate those resource requests if the instance already exists? | 23:19 |
| sean-k-mooney | new arqs sure but we dont want to hard reboot or live migate and change form an nvida gpu to an intel fpga | 23:19 |
| sean-k-mooney | i think we need to be storing the groups retruned by cycborg when we instilly created the vm the same way we embed the flavor or image | 23:20 |
| dansmith | not sure how that would happen on a hard reboot, but obviously agree on live-migration, but that's why I'm saying maybe we should look at the device profile on boot, and look at our existing arqs on any other move operation when asking cyborg for the resources | 23:20 |
| *** jmlowe has quit IRC | 23:21 | |
| sean-k-mooney | dansmith: i guess hardreboot it would not | 23:21 |
| sean-k-mooney | we woudl just use the exising arq | 23:22 |
| dansmith | maybe we need to ask sundar if the dp can change in a predictable or restricted way | 23:22 |
| *** nweinber has joined #openstack-nova | 23:22 | |
| *** damien_r has quit IRC | 23:22 | |
| dansmith | I would expect the arqs can't be changed after they're allocated, and if we used those to base our new request on that would get us what we want | 23:22 |
| sean-k-mooney | looking at the client i only see a create and delete | 23:22 |
| sean-k-mooney | so i guess not | 23:22 |
| sean-k-mooney | ill check the api | 23:23 |
| dansmith | are you saying the device profile can too? | 23:23 |
| dansmith | can change in place Imean | 23:23 |
| sean-k-mooney | the device profile in osc can only be create or delete like a flavor | 23:23 |
| sean-k-mooney | im going to check the api to see if there is an update | 23:23 |
| sean-k-mooney | if they are imutable no issue | 23:24 |
| dansmith | ah okay I thought you were implying it *could* change but you were just assuming? | 23:24 |
| dansmith | yea | 23:24 |
| dansmith | I need to run off for a bit, but we can sync with sundar tomorrow | 23:24 |
| sean-k-mooney | ya no worries | 23:24 |
| sean-k-mooney | i was just going to write a function to pull the resouce request form the nova db | 23:24 |
| sean-k-mooney | form the request spec and compar them to the allcoations | 23:24 |
| sean-k-mooney | to ensure they matched up | 23:25 |
| *** CeeMac has quit IRC | 23:26 | |
| *** artom has quit IRC | 23:27 | |
| *** artom has joined #openstack-nova | 23:27 | |
| sean-k-mooney | ok device profiels cannot be updated. https://docs.openstack.org/api-ref/accelerator/v2/index.html#device-profiles but ARQs can be whcih kind of make sense | 23:31 |
| sean-k-mooney | you update ARQs with the host when binding or move operations | 23:31 |
| sean-k-mooney | you cant change the device profile name however | 23:32 |
| *** nweinber has quit IRC | 23:33 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix instance.hidden migration and querying https://review.opendev.org/706331 | 23:37 |
| dansmith | sean-k-mooney: cool | 23:37 |
| * dansmith & | 23:37 | |
| *** damien_r has joined #openstack-nova | 23:43 | |
| *** mriosfer has quit IRC | 23:44 | |
| *** nicolasbock has quit IRC | 23:44 | |
| *** damien_r has quit IRC | 23:45 | |
| *** brinzhang_ has joined #openstack-nova | 23:47 | |
| *** tbachman has quit IRC | 23:51 | |
| *** brinzhang has quit IRC | 23:51 | |
| *** N3l1x has quit IRC | 23:52 | |
| *** tosky has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!