Friday, 2021-03-05

openstackgerritmelanie witt proposed openstack/nova master: Add functional regression test for bug 1853009  https://review.opendev.org/c/openstack/nova/+/69501200:06
openstackbug 1853009 in OpenStack Compute (nova) ussuri "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard)00:06
openstackgerritmelanie witt proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker  https://review.opendev.org/c/openstack/nova/+/69518700:06
openstackgerritmelanie witt proposed openstack/nova master: Invalidate provider tree when compute node disappears  https://review.opendev.org/c/openstack/nova/+/69518800:06
openstackgerritmelanie witt proposed openstack/nova master: Prevent deletion of a compute node belonging to another host  https://review.opendev.org/c/openstack/nova/+/69480200:06
openstackgerritmelanie witt proposed openstack/nova master: Fix inactive session error in compute node creation  https://review.opendev.org/c/openstack/nova/+/69518900:06
*** tosky has quit IRC00:11
*** yoctozepto has quit IRC00:13
*** yoctozepto has joined #openstack-nova00:13
*** gryf has quit IRC00:18
*** gryf has joined #openstack-nova00:21
*** hamalq has quit IRC00:26
*** jamesdenton has quit IRC00:46
*** jamesden_ has joined #openstack-nova00:46
*** tbachman_ has joined #openstack-nova00:51
*** brinzhang0 has joined #openstack-nova00:54
*** tbachman has quit IRC00:54
*** tbachman has joined #openstack-nova00:54
*** tbachman_ has quit IRC00:57
*** brinzhang_ has quit IRC00:57
*** mlavalle has quit IRC01:19
*** hemanth_n has joined #openstack-nova01:44
*** masterpe has quit IRC01:48
*** ihti[m] has quit IRC01:49
*** masterpe has joined #openstack-nova02:20
*** k_mouza has joined #openstack-nova02:21
*** k_mouza has quit IRC02:26
*** songwenping_ has joined #openstack-nova02:38
*** swp20 has quit IRC02:41
*** martinkennelly has quit IRC02:42
*** irclogbot_3 has quit IRC02:52
*** irclogbot_2 has joined #openstack-nova02:55
*** gyee has quit IRC03:08
*** ircuser-1 has quit IRC03:11
*** psachin has joined #openstack-nova03:36
*** ircuser-1 has joined #openstack-nova03:51
*** tbachman has quit IRC03:56
*** tbachman has joined #openstack-nova03:57
*** k_mouza has joined #openstack-nova03:59
*** k_mouza has quit IRC04:04
*** ircuser-1 has quit IRC04:24
*** jamesden_ has quit IRC04:25
*** jamesdenton has joined #openstack-nova04:25
*** mkrai has joined #openstack-nova04:27
*** ircuser-1 has joined #openstack-nova04:28
*** ratailor has joined #openstack-nova04:37
openstackgerritYongli He proposed openstack/nova master: Smartnic support - cyborg drive  https://review.opendev.org/c/openstack/nova/+/77136204:53
openstackgerritYongli He proposed openstack/nova master: smartnic support - new vnic type  https://review.opendev.org/c/openstack/nova/+/77136304:53
openstackgerritYongli He proposed openstack/nova master: smartnic support  https://review.opendev.org/c/openstack/nova/+/75894404:53
*** ratailor_ has joined #openstack-nova04:59
*** ratailor__ has joined #openstack-nova05:02
*** ratailor has quit IRC05:03
*** ratailor_ has quit IRC05:06
*** dviroel has quit IRC05:10
*** rcernin has quit IRC06:00
*** vishalmanchanda has joined #openstack-nova06:24
*** amodi has quit IRC06:28
*** bnemec has quit IRC06:43
*** bnemec has joined #openstack-nova06:44
*** slaweq has joined #openstack-nova06:59
*** ralonsoh has joined #openstack-nova07:03
openstackgerritYongli He proposed openstack/nova master: Smartnic support - cyborg drive  https://review.opendev.org/c/openstack/nova/+/77136207:04
openstackgerritYongli He proposed openstack/nova master: smartnic support - new vnic type  https://review.opendev.org/c/openstack/nova/+/77136307:04
openstackgerritYongli He proposed openstack/nova master: smartnic support  https://review.opendev.org/c/openstack/nova/+/75894407:04
*** khomesh24 has joined #openstack-nova07:05
*** cz3 has quit IRC07:19
*** cz3 has joined #openstack-nova07:20
*** mkrai has quit IRC07:43
*** zzzeek has quit IRC07:52
*** zzzeek has joined #openstack-nova07:53
*** songwenping__ has joined #openstack-nova07:55
*** mkrai has joined #openstack-nova07:56
*** songwenping_ has quit IRC07:58
*** lpetrut has joined #openstack-nova08:00
*** dklyle has quit IRC08:01
*** zzzeek has quit IRC08:08
*** zzzeek has joined #openstack-nova08:09
*** mkrai has quit IRC08:18
*** rpittau|afk is now known as rpittau08:21
openstackgerritBalazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach  https://review.opendev.org/c/openstack/nova/+/77024608:24
*** luksky has joined #openstack-nova08:25
lyarwoodelod: https://review.opendev.org/c/openstack/nova/+/777209/1 - would you mind hitting this and the regression test below it for melwitt please.08:27
*** zzzeek has quit IRC08:40
*** zzzeek has joined #openstack-nova08:41
*** zzzeek has quit IRC08:42
*** zzzeek has joined #openstack-nova08:43
*** ociuhandu has joined #openstack-nova08:44
elodlyarwood: sure, looking08:47
yonglihealex_xu: gibi:  test cases fixed, resolve concerns.08:48
lyarwoodelod: thanks08:57
*** lucasagomes has joined #openstack-nova09:04
*** derekh has joined #openstack-nova09:14
bauzasgibi: stephenfin: the whole api db series is now +Wd09:23
bauzasbut,09:23
*** tosky has joined #openstack-nova09:23
bauzashave we forgotten to add wallaby db placeholders like we did for Rocky ? :scared_emoji_you'd_like_have:09:23
stephenfinDo we need to?09:24
stephenfinWe can just use the Victoria placeholders09:24
stephenfinWe don't have any DB migrations afaik09:24
bauzasstephenfin: well, in general, we create DB placeholders even if we don't have existing DB things09:25
bauzashttps://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions09:25
stephenfinI know. I'm questioning that :)09:26
bauzasstephenfin: the point is, if someone writes a DB migration in Xena and then we need to backport it in Wallaby but not in Victoria, how to do it ?09:26
bauzassurely, it's a corner case, but that's why we have placeholders for every release09:27
stephenfinHmm, that's a good point09:27
stephenfinI really doubt we'd want/need them but you can add them if you think it's a good idea?09:28
bauzasI can try09:28
stephenfinThey'll all be deleted when we switch to alembic in Xena anyway09:28
bauzasit's maybe just a bikeshed09:28
bauzasbut in case we have issues, then...09:28
bauzasstephenfin: have we already agreed on accepting to use alembic for nova db migrations in Xena ? :D09:29
stephenfinI'm going on the assumption that we have, yes :D09:29
openstackgerritMerged openstack/nova stable/victoria: Add regression test for bug 1914777  https://review.opendev.org/c/openstack/nova/+/77712109:29
openstackbug 1914777 in OpenStack Compute (nova) victoria "Possible race condition between n-cpu and n-api when deleting a building instance" [High,In progress] https://launchpad.net/bugs/1914777 - Assigned to melanie witt (melwitt)09:29
stephenfinIf not, I'm core on sqlalchemy-migrate. It'd be a shame if something were to...happen to it09:29
stephenfin"Nice DB migration library you got there..."09:29
stephenfin😈09:30
bauzas:)09:30
bauzasat least we should remember why we had concerns before09:30
stephenfinhttp://lists.openstack.org/pipermail/openstack-discuss/2021-February/020638.html09:30
bauzasstephenfin: add this to the xena ptg etherpad if you haven't done it already :)09:31
stephenfinlink?09:31
bauzassec09:31
bauzashttps://etherpad.opendev.org/p/nova-xena-ptg09:31
bauzashaven't you went to the nova meeting yesterday ? maaaaaan09:31
stephenfinI have a conflict :-(09:31
bauzas;)09:31
stephenfinNDSU mentoring meeting09:31
bauzaswhat's NSDU ?09:32
stephenfinNorth Dakota State University09:33
stephenfinStudents09:33
bauzashah09:34
bauzasI'd love to help some students here09:35
bauzasbut...09:35
bauzasfrench universities don't like to have remote mentors09:35
stephenfinthis year could be your year09:36
stephenfinit's all remote now :-\09:36
bauzasmeh09:37
bauzasit's... difficult, trust me09:37
bauzaseven for our company :)09:37
*** k_mouza has joined #openstack-nova09:42
*** Underknowledge has quit IRC09:48
*** Underknowledge1 has joined #openstack-nova09:49
*** Underknowledge1 is now known as Underknowledge09:49
stephenfingibi: URI qemu:///system does not support events: module 'nova.tests.unit.virt.libvirt.fakelibvirt' has no attribute 'VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED'10:09
stephenfinI'm seeing that in unit tests10:09
stephenfinLooks like you might want to add stuff to fakelibvirt, else mute the warning?10:10
*** zoharm has joined #openstack-nova10:17
*** dviroel has joined #openstack-nova10:18
*** tbachman has quit IRC10:30
*** tbachman has joined #openstack-nova10:31
*** jangutter has joined #openstack-nova10:41
*** jangutter has quit IRC10:43
*** jangutter has joined #openstack-nova10:43
openstackgerritLee Yarwood proposed openstack/nova master: nova-next: Start testing the q35 machine type  https://review.opendev.org/c/openstack/nova/+/70870110:44
lyarwood^ gibi / kashyap / stephenfin ; this should be ready for reviews now, just had a quick nit to resolve.10:44
*** jangutter_ has quit IRC10:45
kashyaplyarwood: Thanks for the sustained effort on it; will look today10:45
gibibauzas, stephenfin: I think we only need to add the wallaby db placeholders at the start of X as in X we need to differentiate between db changes that are X only from db changes that are backported to W10:51
gibistephenfin: regarding the unit test logs I will look into it10:52
stephenfinlyarwood: cool10:53
kashyaplyarwood: If you're respinning, can you please add the comment I noted in the review?  It'll be useful for those who don't dwell on machine types10:58
kashyapI don't insist; just that it avoids some head-scratching; I added my +1 regardless, FWIW.10:59
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add support for virtio-based input devices  https://review.opendev.org/c/openstack/nova/+/75655211:01
openstackgerritStephen Finucane proposed openstack/nova master: tests: Stop mocking 'nova.virt.libvirt.utils.get_arch'  https://review.opendev.org/c/openstack/nova/+/77887911:01
openstackgerritMerged openstack/nova stable/victoria: Handle instance = None in _local_delete_cleanup  https://review.opendev.org/c/openstack/nova/+/77720911:02
stephenfinlyarwood, gibi: Mind taking at look at ^ again? Took a while but I got the mocking nailed down eventually11:05
lyarwoodkashyap: ack I did note something about that in the commit but if I respin I'll add a note11:09
lyarwoodstephenfin: ack looking11:09
*** k_mouza has quit IRC11:10
gibistephenfin: done11:12
stephenfinty11:12
*** artom has quit IRC11:16
*** ociuhandu has quit IRC11:16
*** ociuhandu has joined #openstack-nova11:17
*** ociuhandu has quit IRC11:17
*** ociuhandu has joined #openstack-nova11:18
*** k_mouza has joined #openstack-nova11:22
*** lucasagomes has quit IRC11:24
*** lucasagomes has joined #openstack-nova11:28
*** ociuhandu has quit IRC11:29
openstackgerritBalazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach  https://review.opendev.org/c/openstack/nova/+/77024611:29
*** jangutter has quit IRC11:29
*** jangutter has joined #openstack-nova11:30
*** ociuhandu has joined #openstack-nova11:31
kashyaplyarwood: Unless I'm terribly blind, I see that you've added about the SATA bus aspect, but I can't find anything about the PCIe root ports. :)  But no prob11:33
stephenfinlyarwood: The patch to add '--block-device' to 'openstack server create' landed and I'm adding functional tests. It seems boot_index is not mandatory11:34
stephenfinIf I omit it, I guess it defaults to -1. The guest correctly boots from the image I'm specifying alongside (cirros)11:35
openstackgerritLee Yarwood proposed openstack/nova master: WIP zuul: Replace nova-grenade-multinode with grenade-multinode  https://review.opendev.org/c/openstack/nova/+/77888511:36
lyarwoodstephenfin: are you providing both --image and --block-device?11:36
stephenfinyes11:37
lyarwoodstephenfin: I think osc and novaclient fudge things in the past to always boot from the --image you provided11:37
lyarwoodstephenfin: in terms of the api behaviour11:37
lyarwoodstephenfin: I think n-api also ensures we boot from the image, let me check11:38
stephenfinThere's also additional weirdness. I was copying this behaviour https://github.com/openstack/python-novaclient/blob/master/novaclient/v2/servers.py#L775-L77911:39
stephenfinHowever, that seems broken. nova rejects those requests with 'Block Device Mapping is Invalid: Mapping image to local is not supported.'11:40
stephenfinopenstack server create ... --image cirros-0.5.1-x86_64-disk --block-device device_name=vdb,source_type=image,destination_type=volume,uuid=$UUID ...11:41
gibistephenfin: ooh that image to local error was one of my early nova contribution ;) I0c38134e2aacca83d98ed0b341f15c480afe14f111:41
stephenfinthat works just fine, so I'm going to change it to default destination type to volume when source type is image instead, since that seems more logical/correct11:42
* stephenfin validates11:42
stephenfingibi: we can't be blamed for the mistakes of our younger selves ;)11:42
gibiindeed11:43
* lyarwood reads11:43
stephenfinah, wait, I think I misunderstood that code11:43
lyarwoodright11:44
lyarwoodimage to local is valid11:44
lyarwoodI'm not sure what we end up doing to hit that11:44
lyarwooddo we provide an imageRef *and* image->local bdm?11:44
stephenfinI think that's transforming the '--image' request to a bdm dict11:45
* stephenfin runs with --debug11:45
lyarwoodyeah it should do and I think that needs to drop the imageRef as a result?11:45
lyarwoodwith imageRef we end up with an image based local bdm so I'm not sure why we would block this in the API11:46
lyarwoodmaybe because it skips imageRef11:46
stephenfinhttp://paste.openstack.org/show/803266/11:46
stephenfinthat's what the OSC code currently ends up generating11:46
stephenfinin response to http://paste.openstack.org/show/803267/11:47
lyarwoodstephenfin: and n-api rejects that?11:47
stephenfinnope, that's fine11:47
lyarwoodoh11:48
lyarwoodokay11:48
stephenfinif I add destination_type=local to the '--block-device' arg though11:48
stephenfinthe API rejects _that_11:48
lyarwoodoh right11:48
lyarwoodyeah11:48
lyarwoodthere can only be one11:48
lyarwoodI think11:48
stephenfinYeah, quite possibly11:49
stephenfinI'm also using the same image twice11:49
stephenfini don't know if that's an issue or not11:49
stephenfinhttp://paste.openstack.org/show/803268/11:49
lyarwoodhttps://github.com/openstack/nova/blob/c8a6f8d2e502e139c2212fb20cc0f89de816aeb1/nova/block_device.py#L193-L20611:49
stephenfinplus other differences as you can see11:49
stephenfinokay, so the logic should be...11:51
gibias far as I remember I blocked image to local mapping as it was not implemented (for other than for a single image)11:51
stephenfinif --image is specified, use destination_type=volume since this is an additional image11:51
lyarwooddestination_type shouldn't be optional right?11:52
lyarwoodhuh it is11:52
lyarwoodI thought it wasn't in the API11:52
lyarwoodso yeah default to volume if --image is provided11:52
lyarwoodand destination_type isn't11:52
stephenfinthey're all optional except UUID11:52
lyarwoodand reject the use of image as a destination_type if --image is already provided in the cli11:53
stephenfinof local you mean11:53
lyarwoodyeah sorry11:53
lyarwoodlocal11:53
openstackgerritBalazs Gibizer proposed openstack/nova master: Add device event constants to fakelibvirt  https://review.opendev.org/c/openstack/nova/+/77888711:53
stephenfin--image + --block-device source_type=image,destination_type=local ==> FAIL11:53
lyarwoodI can add a note in the api docs about this11:54
stephenfinactually, for OSC I might just leave this to the server11:54
stephenfinas you suggested in the review11:54
lyarwoodah damn past me11:54
stephenfinsimply using a sensible destination_type (volume) would be good11:54
*** ociuhandu has quit IRC11:54
lyarwoodbut that makes sense11:54
stephenfinenough11:54
lyarwoodyeah11:54
*** ociuhandu has joined #openstack-nova11:56
*** ociuhandu has quit IRC12:00
*** derekh has quit IRC12:13
*** k_mouza has quit IRC12:16
openstackgerritLee Yarwood proposed openstack/nova master: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode  https://review.opendev.org/c/openstack/nova/+/77888512:26
lyarwood^ tosky / gmann ; I'm not sure what you both think about this approach but it could allow us to finish our move to zuulv3 before M312:27
lyarwoodtl;dr - move everything to grenade-multinode and add coverage for ceph later once that multinode base job is finished12:27
*** tkajinam has quit IRC12:35
*** tkajinam has joined #openstack-nova12:35
*** ratailor__ has quit IRC12:36
*** belmoreira has joined #openstack-nova12:40
*** k_mouza has joined #openstack-nova12:52
*** k_mouza has quit IRC12:52
*** k_mouza has joined #openstack-nova12:53
sean-k-mooneylyarwood: that wont actully remove grenade unless we remove it form the integrated-gate-compute template12:55
sean-k-mooneyit wil still be added by integrated-gate-compute and it will no longer use the irrelevant-files list12:55
sean-k-mooneyso what you really need to do is also remove the integrated-gate-compute template12:56
sean-k-mooneyoh never mind12:56
sean-k-mooneyhttps://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L442-L454 the patch to remvoe grenade form that has been merged12:57
*** whoami-rajat has joined #openstack-nova12:58
*** tbachman has quit IRC12:58
*** tbachman has joined #openstack-nova12:58
*** xek has joined #openstack-nova13:01
gmannyeah that is already done13:05
gmannlyarwood: ack, thanks, I will check during weekend or on Monday13:05
*** ociuhandu has joined #openstack-nova13:06
lyarwoodsean-k-mooney: it's already removed from that template13:06
lyarwoodsean-k-mooney: yeah13:06
toskylyarwood: that's entirely up to what you (nova) think it's better13:07
lyarwoodI'd rather avoid duplication and move to zuulv3 now instead of waiting to get the ceph coverage in place13:08
openstackgerritLee Yarwood proposed openstack/nova master: zuul: Remove nova-dsvm-multinode-base  https://review.opendev.org/c/openstack/nova/+/77890813:09
openstackgerritLee Yarwood proposed openstack/nova master: WIP zuul: Create and use nova-libvirt-irrelevant-files  https://review.opendev.org/c/openstack/nova/+/77890913:09
*** artom has joined #openstack-nova13:10
stephenfinlyarwood: Low priority, but you may like 'Either a URI-style path (\'file:\\\\{path}\') to a JSON file '13:11
stephenfinwhoops13:11
stephenfinhttps://review.opendev.org/c/openstack/python-openstackclient/+/77891013:12
* stephenfin gets back to secure boot13:12
*** bbowen has quit IRC13:12
lyarwoodstephenfin: nice idea13:13
stephenfinI stole it https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html13:13
stephenfin(search for --block-device-mappings)13:13
*** jamesdenton has quit IRC13:18
*** jamesden_ has joined #openstack-nova13:18
*** khomesh24 has quit IRC13:18
sean-k-mooneyspeaking of osc https://review.opendev.org/c/openstack/python-openstackclient/+/77857113:18
openstackgerritLee Yarwood proposed openstack/nova master: zuul: Create and use nova-libvirt-irrelevant-files  https://review.opendev.org/c/openstack/nova/+/77890913:18
sean-k-mooneythat is for vdpa13:18
sean-k-mooneystephenfin: care to take a look i dont know if i need a release note for that or anything else13:19
sean-k-mooneystephenfin: i just did then minium i needed to not have to use curl13:19
stephenfinsean-k-mooney: will do13:20
stephenfinthat's merged in neutron now?13:20
sean-k-mooneyyep13:20
stephenfincool13:20
stephenfinwill review fully after lunch but a small release note would be A+13:21
stephenfinand a link to the neutron change that added it (for context)13:21
* stephenfin does lunch13:21
sean-k-mooneysure ill see if i can do that later today13:21
*** ociuhandu has quit IRC13:26
*** ociuhandu has joined #openstack-nova13:26
*** tbachman has quit IRC13:27
*** tbachman has joined #openstack-nova13:28
openstackgerritMerged openstack/nova master: tests: Stop mocking 'nova.virt.libvirt.utils.get_arch'  https://review.opendev.org/c/openstack/nova/+/77887913:30
*** ociuhandu has quit IRC13:31
*** jangutter has quit IRC13:33
*** jangutter has joined #openstack-nova13:34
toskylyarwood: in that patch, maybe you can also emove nova-dsvm-multinode-base13:41
openstackgerritBalazs Gibizer proposed openstack/nova master: Move instance power state check to _detach_with_retry  https://review.opendev.org/c/openstack/nova/+/77891813:41
lyarwoodtosky: I've done that in the following change13:42
*** hemanth_n has quit IRC13:43
*** jangutter has quit IRC13:43
*** jangutter has joined #openstack-nova13:44
*** tbachman has quit IRC13:46
*** tbachman has joined #openstack-nova13:46
openstackgerritSylvain Bauza proposed openstack/nova master: Placeholders for DB migration backports to Wallaby  https://review.opendev.org/c/openstack/nova/+/77892313:55
bauzasgibi: stephenfin: sorry, was at some gym, here is the wallaby db placeholders change ^13:55
bauzaslemme look when we merged the other ones13:55
gibibauzas: thanks13:55
*** efried has quit IRC13:56
bauzasgibi: IIRC, we wanted to have the placeholders *before* any other db migration for a cycle13:56
gibiyes13:56
gibiso if we merge it at the start of the cycle then we are goo13:56
gibid13:56
bauzaslike, you write a db migration for wallaby, you need to have the placeholders *before*13:56
bauzassure, but then we are lucky to not have any merged db migration during this cycle13:57
bauzasmy point is, we should backport this sooner than later and write another change for reserving the xena ones early in the cycle13:57
bauzasdon't we have this in the ptl guide or something like it ?13:57
bauzashttps://review.opendev.org/c/openstack/nova/+/686411 is the Train one13:58
gibibauzas: post release check list https://wiki.openstack.org/wiki/Nova/ReleaseChecklist13:58
gibilinked in the https://docs.openstack.org/nova/latest/contributor/ptl-guide.html13:59
bauzasthe above change was merged on oct 4th13:59
bauzas(2019)13:59
bauzasand the train timeline was https://releases.openstack.org/train/schedule.html13:59
gibiaccording to git it was included from ussuri14:00
sean-k-mooneybauzas: so for xena  i think stephenfin  wants to move to alemnbic14:00
sean-k-mooneyso that might be different in the future14:00
bauzasgibi: /me facepalms14:00
sean-k-mooneynot sure if we still need to do reservations the same way14:00
bauzasso, yeah, this was merged early in the cycle, not late14:00
gibibauzas: hm, I think ti was merged after RC1 and therefore after we branched out stable/train14:00
bauzassean-k-mooney: we'll open the xena cycle before the PTG, we need the placeholders sooner than later14:01
bauzasgibi: sure, but at this point in time, the ussuri placeholders change was merged when we did the master branch to ussuri14:01
sean-k-mooneybauzas: sure just pointing out thye may not be used14:01
gibibauzas: correct14:01
sean-k-mooneythat said they dont hurt to have really14:01
gibibauzas: so it is only in ussuri but not in train14:01
bauzasgibi: correct, that's a PEBKAC when I said Train14:02
bauzasoh wai14:02
gibisean-k-mooney: it is like we don't know yet how the  alembic migration will look like so I think we have to do the safe move and add reservation14:02
bauzasman, this is confusing14:02
bauzashttps://review.opendev.org/c/openstack/nova/+/686411/2/nova/tests/functional/db/api/test_migrations.py14:02
bauzasthis was merged just after the branch but we name them train migrations14:03
bauzasso, hence my confusion, placeholders go last14:03
bauzasand just after we cut14:03
gibiyeah it is like placeholder for migration backported from ussuri to train14:03
gibiso both name can be correct :)14:03
bauzasgibi: so you were right, we should hold until the branhc14:03
sean-k-mooneygibi: yep we should for that reason and because whilte stephenfin want to do the move next cycle something might come up which means we cant finish it14:03
sean-k-mooneyso we need to prepare for backport in anycase14:04
gibisean-k-mooney: correct14:04
bauzassean-k-mooney: gibi: like I said to stephenfin, before discussing it at the PTG, I'd like to dig why we didn't like that before14:04
sean-k-mooneyalembic14:07
bauzasyup14:07
sean-k-mooneyit was manly because we did not feel the effort of the move was worth it similar to removing mox14:07
bauzasI recall this14:07
bauzasbut IIRC there were other concerns14:07
bauzasthat's what I'd like to dif14:08
sean-k-mooneybut like mox that ment the nova team was signing up to maintian sqlalchemy-migrate indefinetly14:08
bauzasdig*14:08
bauzassure, but this prove to work ?14:08
bauzaswe would just pull another dependency we loose control14:08
bauzasover their releases14:08
sean-k-mooneybauzas: we are using a dep that neutron uses already14:08
bauzasI know14:09
sean-k-mooneybauzas: and the maintiner of sqlachmey asked us to move14:09
bauzasbut that just increases the potential flaw, it doesn't prove it works14:09
sean-k-mooneyit does prove it works they have used it for years14:09
sean-k-mooneyim pretty sure neutron moved beofre we created placment14:10
bauzasoh yes, I remember14:15
sean-k-mooneyneutron dont seam to have placeholder migration anymore https://github.com/openstack/neutron/tree/b04dfcee192cf7ddcb69854e760739fdcfa7839e/neutron/db/migration/alembic_migrations/versions14:15
bauzasI actually never remember neutron using sqalchemy-migrate14:16
bauzaseven when they were named quantum14:16
sean-k-mooneyand they have tooling to auto genrate the migrations14:16
sean-k-mooneybauzas: i tought quantum did but maybe not14:16
*** jangutter has quit IRC14:16
*** ociuhandu has joined #openstack-nova14:16
bauzaswell, i played with quantum with the Folsom release, and I don't think they were using migrate14:16
bauzasbut that's been a while14:17
*** jangutter has joined #openstack-nova14:17
sean-k-mooneyactully hevan used it alredy14:17
sean-k-mooneyso i guess not14:17
bauzasthey were incubated during Essex, so I didn't use it in my prod14:17
bauzascan't speak for Essex14:17
bauzasoh gosh, remembrance times...14:18
sean-k-mooneylooking at the eol tags14:20
*** claudiub has joined #openstack-nova14:20
sean-k-mooneyup to fulsom they had not migration in repo14:20
sean-k-mooneyso the first migration they added was grizzly and that used alembic14:21
sean-k-mooneyso for as long as they had migghration yes they used alembic to handel it14:21
bauzasglad to know my memory haven't vanished :)14:21
bauzashasn't*14:22
sean-k-mooneyif it woks well enough for neutron which allows exptions to also have addtional db tables i think it should be flexably enough ot use in nova14:22
bauzasagain, this is not news14:22
sean-k-mooneythe main issue for use will be learning how to write them correctly and review them14:22
bauzasthat's my point14:22
bauzasneutron uses it for ages, but by the times we discussed this for nova, we nacked the idea a couple of times, so I just want to refresh my memory before jumping into conclusions14:23
sean-k-mooneysure but for me continuing to use it is like continuting ot use mox or python214:23
sean-k-mooneysure they work but it woudl be better to simply the dendencis fo openstack an converg on alembic14:24
*** ociuhandu has quit IRC14:33
*** spatel has joined #openstack-nova14:34
openstackgerritsean mooney proposed openstack/nova master: support per port numa policies with sriov  https://review.opendev.org/c/openstack/nova/+/77379214:37
*** ociuhandu has joined #openstack-nova14:39
*** efried has joined #openstack-nova14:49
*** efried has quit IRC14:54
*** efried has joined #openstack-nova14:59
*** jamesden_ is now known as jamesdenton15:03
*** lpetrut has quit IRC15:03
*** rpittau is now known as rpittau|afk15:03
melwittlyarwood, elod: thank you for the reviews :) fyi the stable/ussuri changes for the same bug, I stacked them on top of this other bug (2 patches) fix to eliminate merge conflicts. I figured it was a useful fix for stable anyways https://review.opendev.org/c/openstack/nova/+/777216 and https://review.opendev.org/c/openstack/nova/+/77721715:09
*** mkrai has joined #openstack-nova15:12
*** amodi has joined #openstack-nova15:12
*** ociuhandu has quit IRC15:29
*** ociuhandu has joined #openstack-nova15:33
*** ociuhandu has quit IRC15:36
*** ociuhandu has joined #openstack-nova15:36
claudiubHello, so, I have been looking a bit into the NUMA node 0 overcommit issue I was having. As a reminder, I had ~400 instaHello, so, I have been looking a bit into the NUMA node 0 overcommit issue I was having. As a reminder, I had ~400 instances in NUMA node 0, while NUMA node 1 was mostly empty.15:40
claudiubTurns out, there seems to be an issue when live-migrate / host-evacuate-live is called with a target destination.15:40
claudiubIt seems that the CPU allocation and the memory allocation ratio is ignored in this case, leading to the overcommit, and even OOM, apparently.15:41
claudiubSo, from what I saw, the live_migrate task will eventually call check_can_live_migrate_destination on the destination node (https://github.com/openstack/nova/blob/c8a6f8d2e502e139c2212fb20cc0f89de816aeb1/nova/compute/manager.py#L7931),15:41
claudiubwhich will then call _live_migration_clain and then self.rt.live_migration_clain. In there, a new claims.MoveClaim is created (https://github.com/openstack/nova/blob/c8a6f8d2e502e139c2212fb20cc0f89de816aeb1/nova/compute/resource_tracker.py#L298),15:41
stephenfinYou mean e.g. 'openstack server migrate --live HOST SERVER' ?15:41
claudiubhich will then call hardware.numa_fit_instance_to_host to get the new instance_topology (https://github.com/openstack/nova/blob/c8a6f8d2e502e139c2212fb20cc0f89de816aeb1/nova/compute/claims.py#L138), which will then set it in self.claimed_numa_topology. In the hardware.numa_fit_instance_to_host call,15:41
claudiubbecause the given limits is None, there is no check that CPU / memory is not overcommited in numa cell 0, so it gets scheduled there, since it's first match. https://github.com/openstack/nova/blob/5b52db409b00e6257e83598efa0d8528ebe380e0/nova/virt/hardware.py#L116815:42
claudiubAnd indeed, the limits are None, from what I've seen in the extra logging I've added.15:42
claudiubstephenfin: yep. :)15:42
stephenfinHow about if you do 'openstack server migrate --live-migration --host HOST SERVER'?15:42
stephenfin'--live HOST' bypasses the scheduler which means the NUMATopologyFilter never triggers to populate limits15:43
stephenfin(it's deprecated and we're removing it in the next version of OSC, fwiw)15:43
stephenfinin fact the patch to do so merged yesterday :)15:43
*** dklyle has joined #openstack-nova15:43
claudiubstephenfin: sorry, it was openstack server migrate --live --host dest_host --block-migration inst_uuid15:46
stephenfinah, sorry, okay, that means this is likely an issue so15:46
stephenfinclaudiub: I won't have time to look at this in depth today (on RFE review/rework duty pending feature freeze) but I could help work on this the end of next week15:47
stephenfinany chance you could open a bug?15:47
claudiubalso, I saw that the NUMATopologyFilter was populating the limits. Why is that? Couldn't that just be initialized in the HostState anyways? It's not dependent on a request anyways.15:47
claudiubstephenfin: will do. :)15:48
claudiubI was wondering if, on the nova-compute side, if we get a check_can_live_migrate_destination call from somewhere and we don't get any limits, couldn't we just use the CPU / mem allocation ratios in the nova.conf file as the limits?15:49
stephenfinI assume we could. I think the reason we package the limits up is so we can check them in the scheduler (since we don't want to use the limits of the nova-scheduler service)15:50
claudiubok then, will open up that issue, and I'll check how it goes if we use the nova.conf cpu / mem allocation ratios as limits if no limits are given. :)15:53
openstackgerritBalazs Gibizer proposed openstack/nova master: Consolidate device detach error handling  https://review.opendev.org/c/openstack/nova/+/77897815:54
stephenfinsounds good15:54
stephenfinlyarwood: Can you look at my two replies here before I pushed the rebased series? https://review.opendev.org/c/openstack/nova/+/77568915:54
sean-k-mooneyclaudiub: the limit not being poplulated is the corrct behaivor15:55
lyarwoodstephenfin: yup can do, just need to grab a tea first15:55
stephenfinack15:55
sean-k-mooneyif you do not have hw:cpu_policy=dedicated or hw:mem_page_size set then we dont really expect it to be set15:55
sean-k-mooneyalthough there is a sperate bug fix that is relevent15:55
sean-k-mooneyclaudiub: https://review.opendev.org/c/openstack/nova/+/76961415:56
sean-k-mooneyclaudiub: the limits are depended on the request at least for cpus15:57
elodmelwitt: the functional job failure seems legit in the patch: https://review.opendev.org/c/openstack/nova/+/777218/15:58
sean-k-mooneyclaudiub: none of the behviaor your are discibing is new to me15:58
*** zoharm has quit IRC15:58
sean-k-mooneyclaudiub: this is why i said added hw:numa_nodes with no other numa parmateters is not valid15:58
elodmelwitt: otherwise it would be clean & OK o:)15:59
melwittelod: oh, sorry I completely missed that and only saw the pep8 cherry pick fail. thanks for pointing it out, I will fix :)15:59
sean-k-mooneyclaudiub: the limits are not in general related to the nova.conf allocation ratios16:00
gibilyarwood: fyi, now I think I fixed every comment in https://review.opendev.org/c/openstack/nova/+/770246/ with two follow up patch16:00
elodmelwitt: np :)16:00
sean-k-mooneyclaudiub: well there are 2 limits object https://github.com/openstack/nova/blob/db666e2118972e501637141e48164a94f9bead54/nova/objects/numa.py#L273 and https://github.com/openstack/nova/blob/db666e2118972e501637141e48164a94f9bead54/nova/objects/request_spec.py#L1076-L108716:03
lyarwoodgibi++ awesome16:03
gibilyarwood: you had very good comments about simplifying error handling16:03
claudiuboh ok. I only saw the host_state.limits here: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L11216:04
sean-k-mooneythat being none or not has nothing to do with your stacking problem16:06
sean-k-mooneyif you dont have hw:cpu_polciy=dedicated or hw:mem_page_size16:06
sean-k-mooneyyou have not told nova to claim any reslouce form the numa nodes16:07
sean-k-mooneyso it will never decrement the capsity16:07
sean-k-mooneytherefor as long as the vm can fit on the numa node e.g. does not exceed the reouce avaiable on the node it will be accpeted16:07
claudiubsean-k-mooney: indeed, my current stacking problem is not caused by the numa topology filter, since that is skipped if you specify the destination host on live migration. Will try live-migration with instances having that flavor extra_spec to see if it still persists.16:08
openstackgerritBalazs Gibizer proposed openstack/nova master: Move the guest.get_disk test to test_guest  https://review.opendev.org/c/openstack/nova/+/77715116:08
sean-k-mooneyclaudiub: the same logic is used on the compute hosts16:08
claudiub(with the destination host specified)16:08
sean-k-mooneywe should be blocking all flaovr that only have hw:numa_node at the api level but we can do inter extraspec validation currenlty16:09
lyarwoodstephenfin: LGTM, sorry about that, two pointless things to block that on previously16:09
* lyarwood isn't sure how he missed the caps caching16:09
openstackgerritBalazs Gibizer proposed openstack/nova master: Enable mypy on libvirt/guest.py  https://review.opendev.org/c/openstack/nova/+/77715516:09
lyarwoodgibi: you're welcome! :)16:10
sean-k-mooneyclaudiub: well to be percise using a flaovr with hw:numa_node and no other numa prarmater with an image that also does not set themis invalid16:10
sean-k-mooneyclaudiub: the fact that you have to chekc the flavor and image is why we cand do the valdiation at flavor create time16:10
gibilyarwood: thanks :)16:10
openstackgerritBalazs Gibizer proposed openstack/nova master: Follow up type hints for a634103  https://review.opendev.org/c/openstack/nova/+/77715916:11
sean-k-mooneyclaudiub: we should technically  be blocking it here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L76016:11
sean-k-mooneyby adding a check here https://github.com/openstack/nova/blob/a65e4201cc03f89d37296ddb803934c5a7977a71/nova/virt/hardware.py#L194216:13
sean-k-mooney numa_get_constraints should raise an excption in this case or default the mem_page_size=any16:14
*** psachin has quit IRC16:17
sean-k-mooneyclaudiub: i was going to defaut the pagezie request to any there fyi so that it would at least blance based on memory usage16:24
*** mkrai has quit IRC16:29
*** mkrai_ has joined #openstack-nova16:29
*** ociuhandu has quit IRC16:29
*** ociuhandu has joined #openstack-nova16:32
*** ociuhandu has quit IRC16:36
*** ociuhandu has joined #openstack-nova16:42
*** mkrai_ has quit IRC16:47
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add firmware auto-configuration support  https://review.opendev.org/c/openstack/nova/+/77667816:58
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add tests for insecure UEFI configuration  https://review.opendev.org/c/openstack/nova/+/77736716:58
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Stop passing around virt_type, caps  https://review.opendev.org/c/openstack/nova/+/77568916:58
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Add missing type hints  https://review.opendev.org/c/openstack/nova/+/77568816:58
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Clarify purpose of 'Host.supports_*' properties  https://review.opendev.org/c/openstack/nova/+/77873916:59
openstackgerritStephen Finucane proposed openstack/nova master: hardware: Start parsing 'os_secure_boot'  https://review.opendev.org/c/openstack/nova/+/68262816:59
openstackgerritStephen Finucane proposed openstack/nova master: compute: Report COMPUTE_SECURITY_UEFI_SECURE_BOOT  https://review.opendev.org/c/openstack/nova/+/77667916:59
*** lucasagomes has quit IRC16:59
openstackgerritStephen Finucane proposed openstack/nova master: scheduler: Translate secure boot requests to trait  https://review.opendev.org/c/openstack/nova/+/77668016:59
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Switch to libvirt's firmware auto-selection  https://review.opendev.org/c/openstack/nova/+/77569116:59
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Wire up 'os_secure_boot' property  https://review.opendev.org/c/openstack/nova/+/77668116:59
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Report secure boot support to scheduler  https://review.opendev.org/c/openstack/nova/+/77569016:59
openstackgerritStephen Finucane proposed openstack/nova master: tests: Add functional tests for UEFI, secure boot  https://review.opendev.org/c/openstack/nova/+/77668216:59
openstackgerritStephen Finucane proposed openstack/nova master: tests: Remove duplicated 'start_compute' helper  https://review.opendev.org/c/openstack/nova/+/77668316:59
openstackgerritStephen Finucane proposed openstack/nova master: doc: Document UEFI secure boot feature  https://review.opendev.org/c/openstack/nova/+/77668416:59
stephenfinlyarwood: That's mostly merge conflict resolution. I'm working on undoing the firmware auto-selection changes now but it should free up everything below it to merge, at least17:00
stephenfinjust FYI17:00
stephenfinbauzas also17:00
lyarwoodstephenfin: thanks, can we chat about the auto selection stuff on Monday just so I'm sure I get the direction?17:01
stephenfinsure17:01
lyarwoodstephenfin: I'll also hit everything up to that first thing on Monday regardless17:01
stephenfinin short though, I'm just going to drop that patch and go back to generating everything manually17:01
stephenfinhttps://that.guru/blog/uefi-secure-boot-in-libvirt/17:01
stephenfinthat might help explain what it was supposed to do and why we can't use it (scroll to the bottom for that)17:02
lyarwoodkk I thought so, I couldn't get my head around all of the details in the bug17:02
openstackgerritBalazs Gibizer proposed openstack/nova master: Add device event constants to fakelibvirt  https://review.opendev.org/c/openstack/nova/+/77888717:05
*** belmoreira has quit IRC17:10
kashyapstephenfin: That's quite some speed there.17:20
kashyapI mean, in reworking the series.17:20
* kashyap will look with proper energy on Mon17:20
*** mjturek has quit IRC17:23
*** k_mouza has quit IRC17:24
*** ociuhandu_ has joined #openstack-nova17:39
*** ociuhandu_ has quit IRC17:40
*** ociuhandu has quit IRC17:40
*** ociuhandu has joined #openstack-nova17:41
*** mlavalle has joined #openstack-nova17:47
*** jawad_axd has joined #openstack-nova17:50
*** dklyle has quit IRC17:52
*** dklyle has joined #openstack-nova17:52
openstackgerritmelanie witt proposed openstack/nova stable/ussuri: Add regression test for bug 1914777  https://review.opendev.org/c/openstack/nova/+/77721817:52
openstackbug 1914777 in OpenStack Compute (nova) victoria "Possible race condition between n-cpu and n-api when deleting a building instance" [High,In progress] https://launchpad.net/bugs/1914777 - Assigned to melanie witt (melwitt)17:52
openstackgerritmelanie witt proposed openstack/nova stable/ussuri: Handle instance = None in _local_delete_cleanup  https://review.opendev.org/c/openstack/nova/+/77721917:52
*** mjturek has joined #openstack-nova17:55
*** irclogbot_2 has quit IRC18:03
*** irclogbot_3 has joined #openstack-nova18:05
*** k_mouza has joined #openstack-nova18:10
*** k_mouza has quit IRC18:15
*** gyee has joined #openstack-nova18:17
*** irclogbot_3 has quit IRC18:24
*** irclogbot_0 has joined #openstack-nova18:28
*** admin0 has joined #openstack-nova18:31
admin0hi all  openstack server show $uuid => hostId -- how do i check what hostId is which hypervisor ?18:32
sean-k-mooneyas a non admin you cant intntionally18:33
sean-k-mooneyifyou are an admin you can check it differently18:33
sean-k-mooneyhostid is a hash of the hostname salted by your poject id18:34
sean-k-mooneyOS-EXT-SRV-ATTR:hypervisor_hostname should be avaiable as an admin18:35
*** ralonsoh has quit IRC18:35
sean-k-mooneyhostid: "An ID string representing the host. This is a hashed value so will not actually look like a hostname, and is hashed with data from the project_id, so the same physical host as seen by two different project_ids, will be different. It is useful when within the same project you need to determine if two instances are on the same or different physical hosts for the purposes of18:36
sean-k-mooneyavailability or performance."18:36
sean-k-mooneyfrom the api ref ^18:36
sean-k-mooneyadmin0: hopefully that helps. its considerd a security issue if vms can determin what host they are running on directly18:38
sean-k-mooneyassuming they can use that info to target another vm to the same host18:38
sean-k-mooneythat is why the hostid is hashed and different proejct will see different values for the same host18:39
admin0oh .. got it18:39
*** ociuhandu has quit IRC18:42
*** jawad_axd has quit IRC18:44
openstackgerritsean mooney proposed openstack/nova master: support per port numa policies with sriov  https://review.opendev.org/c/openstack/nova/+/77379218:49
*** ociuhandu has joined #openstack-nova18:54
*** ociuhandu has quit IRC18:58
*** spatel has quit IRC18:58
*** spatel_ has joined #openstack-nova18:59
*** xek has quit IRC19:04
*** jawad_axd has joined #openstack-nova19:05
*** elod has quit IRC19:12
*** takamatsu has quit IRC19:13
*** elod has joined #openstack-nova19:13
*** kevinbenton has quit IRC19:15
*** k_mouza has joined #openstack-nova19:25
*** takamatsu has joined #openstack-nova19:27
dansmithhas anyone noticed compute nodes getting 500 errors from placement?19:27
dansmithhttps://zuul.opendev.org/t/openstack/build/19989232c4b94c848894c6e54d9fa49c/log/logs/subnode-2/screen-n-cpu.txt#920419:27
dansmithoh you know,19:28
dansmiththat might be during the upgrade part where the service is down19:28
*** k_mouza has quit IRC19:30
melwittdansmith: I haven't noticed that before but can see the corresponding request doesn't show up in the placement logs, so that seems to support what you're thinking that it's when the service is down? I didn't know how/if the service can 500 while it is maybe partially up/in a state where it would not log the request in placement. timestamps match too19:35
dansmithmelwitt: because of the TLS proxies I expect19:36
dansmithmelwitt: after that in the log, conductor is down for a while when it's trying to update its service record, which I think is the key indicating we're in the upgrade phase19:36
dansmiththis is the left-behind node of the multinode upgrade19:36
melwittoh I didn't realize that job ran with TLS enabled19:36
dansmithwell, I guess I was assuming it did19:37
melwittit might, I had thought only nova-next did19:37
dansmithoh, actually, what I mean is with the apache->uwsgi->nova bit19:38
dansmithtechnically that's not the tls19:38
dansmithlots of jobs do, this is the base tempest integrated job: https://zuul.opendev.org/t/openstack/build/266f30d8a31944ca8e391c2d67402582/log/controller/logs/local_conf.txt#3419:39
dansmithnot just nova-next19:39
melwittoh, TIL. cool19:40
sean-k-mooneythe tls proxy?19:41
sean-k-mooney i think thats enabled by default19:41
melwittI was thinking of this jazz, since I've worked with it before https://github.com/openstack/nova/blob/master/.zuul.yaml#L229-L23519:41
sean-k-mooneybut that is quite different form tls everywhere as we normally thinkg of it19:41
dansmithright19:41
dansmithmelwitt: yeah I think it's on in more base jobs we inherit from19:42
dansmithmelwitt: that nova-next job might not because it's multinode or something, but it's on in a lot of places now19:42
dansmith"might not get it from inheritance" I mean19:42
melwittI see, cool19:42
sean-k-mooneyhttps://github.com/openstack/devstack/blob/master/.zuul.yaml#L48519:43
sean-k-mooneyits on by default in the base devstack job19:43
sean-k-mooneywhich menas it on in all the tempest jobs by default https://github.com/openstack/tempest/blob/master/zuul.d/base.yaml#L2-L319:44
sean-k-mooneymelwitt: i think setting it in nova-next is likely legacy form when that was zullv2 based19:46
sean-k-mooneymelwitt: i dont think the tls_proxy was enabeld in the legacy jobs19:46
*** jawad_axd has quit IRC19:52
*** jawad_axd has joined #openstack-nova19:57
*** slaweq has quit IRC20:17
admin0hi guys . my openstack ansible is stuck here: my nova is stuck at  raise exceptions.VersionNotFoundError(", "migrate.exceptions.VersionNotFoundError: \"Database schema file with version 82 doesn't exist.\""]}20:19
admin0any ideas where to get this version 82 from ?20:19
*** jamesdenton has quit IRC20:49
*** jamesden_ has joined #openstack-nova20:49
*** jawad_axd has quit IRC20:55
*** hoonetorg has quit IRC20:57
*** hoonetorg has joined #openstack-nova21:01
*** k_mouza has joined #openstack-nova21:10
*** hamalq has joined #openstack-nova21:14
*** k_mouza has quit IRC21:15
*** jawad_axd has joined #openstack-nova21:17
*** jawad_axd has quit IRC21:31
*** spatel_ has quit IRC22:41
openstackgerritmelanie witt proposed openstack/nova master: Differentiate between InstanceNotFound and ConstraintNotMet  https://review.opendev.org/c/openstack/nova/+/77530922:54
*** iurygregory has quit IRC22:57
*** iurygregory has joined #openstack-nova23:22
*** elod has quit IRC23:25
*** hamalq has quit IRC23:44
*** k_mouza has joined #openstack-nova23:52
*** k_mouza has quit IRC23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!