*** luksky has quit IRC | 00:07 | |
rouk | old qemu - > migrate to 3.1 machine type - > upgrade qemu - > update machine type - > slowly let people trickle over is the migration plan till nova can handle this, i guess. | 00:08 |
---|---|---|
*** tosky has quit IRC | 00:09 | |
*** johnthetubaguy has quit IRC | 00:16 | |
*** johnthetubaguy has joined #openstack-nova | 00:22 | |
*** hamalq has quit IRC | 00:56 | |
*** macz_ has joined #openstack-nova | 00:58 | |
*** macz_ has quit IRC | 01:03 | |
*** rouk has quit IRC | 01:09 | |
*** martinkennelly has quit IRC | 01:14 | |
*** ircuser-1 has joined #openstack-nova | 01:26 | |
*** zhanglong has joined #openstack-nova | 01:34 | |
*** tbachman has quit IRC | 01:39 | |
*** tbachman has joined #openstack-nova | 01:41 | |
*** jamesdenton has quit IRC | 02:13 | |
*** jamesden_ has joined #openstack-nova | 02:13 | |
*** rcernin has quit IRC | 02:28 | |
openstackgerrit | xinyu wang proposed openstack/nova-specs master: add instance join/leave server group https://review.opendev.org/c/openstack/nova-specs/+/782353 | 02:30 |
*** rcernin has joined #openstack-nova | 02:39 | |
*** hemanth_n has joined #openstack-nova | 03:07 | |
openstackgerrit | xinyu wang proposed openstack/nova master: join/leave server group https://review.opendev.org/c/openstack/nova/+/782356 | 03:25 |
openstackgerrit | xinyu wang proposed openstack/nova master: join/leave server group https://review.opendev.org/c/openstack/nova/+/782356 | 03:35 |
*** zhanglong has quit IRC | 03:51 | |
*** khomesh24 has joined #openstack-nova | 04:03 | |
*** vishalmanchanda has joined #openstack-nova | 04:09 | |
*** viks____ has quit IRC | 04:20 | |
*** links has joined #openstack-nova | 04:22 | |
*** mkrai has joined #openstack-nova | 04:23 | |
*** gyee has quit IRC | 04:26 | |
*** whoami-rajat_ has joined #openstack-nova | 04:57 | |
*** mgariepy has quit IRC | 05:14 | |
*** kinpaa12389 has joined #openstack-nova | 05:14 | |
*** mgariepy has joined #openstack-nova | 05:14 | |
*** jamesden_ has quit IRC | 05:25 | |
*** jamesdenton has joined #openstack-nova | 05:25 | |
*** rcernin has quit IRC | 05:25 | |
*** rcernin has joined #openstack-nova | 05:26 | |
*** zhanglong has joined #openstack-nova | 05:31 | |
*** jamesdenton has quit IRC | 06:20 | |
*** jamesden_ has joined #openstack-nova | 06:21 | |
*** links has quit IRC | 06:24 | |
*** ralonsoh has joined #openstack-nova | 06:26 | |
openstackgerrit | Josephine Seifert proposed openstack/nova stable/ussuri: Add config parameter 'live_migration_scheme' to live migration with tls guide https://review.opendev.org/c/openstack/nova/+/782126 | 06:30 |
*** manuvakery1 has joined #openstack-nova | 06:30 | |
*** Luzi has joined #openstack-nova | 06:34 | |
*** kinpaa12389 has quit IRC | 06:36 | |
*** links has joined #openstack-nova | 06:44 | |
*** ratailor has joined #openstack-nova | 06:47 | |
*** zhanglong has quit IRC | 06:51 | |
*** dklyle has quit IRC | 06:55 | |
*** mkrai has quit IRC | 07:04 | |
*** mkrai_ has joined #openstack-nova | 07:05 | |
*** lpetrut has joined #openstack-nova | 07:11 | |
*** xarlos has quit IRC | 07:15 | |
*** slaweq_ has joined #openstack-nova | 07:15 | |
*** zhanglong has joined #openstack-nova | 07:23 | |
*** links has quit IRC | 07:23 | |
*** mkrai_ has quit IRC | 07:23 | |
*** links has joined #openstack-nova | 07:26 | |
*** zhanglong has quit IRC | 07:33 | |
*** kinpaa12389 has joined #openstack-nova | 07:37 | |
*** pawan-gupta has joined #openstack-nova | 07:39 | |
*** mkrai_ has joined #openstack-nova | 07:43 | |
*** rcernin has quit IRC | 07:46 | |
*** slaweq_ is now known as slaweq | 07:53 | |
*** whoami-rajat_ is now known as whoami-rajat | 08:03 | |
*** zhanglong has joined #openstack-nova | 08:10 | |
*** gokhani has joined #openstack-nova | 08:13 | |
*** rpittau|afk is now known as rpittau | 08:13 | |
Luzi | gibi, elod, lyarwood: the next backport (part 2 of 4) https://review.opendev.org/c/openstack/nova/+/782126 | 08:14 |
*** khomesh24 has quit IRC | 08:16 | |
kinpaa12389 | Hi, | 08:18 |
kinpaa12389 | I am trying to create snapshot from server. In this operation, I have added expiration value in keystone.conf and time.sleep() in snapshot() | 08:18 |
kinpaa12389 | Due to token expiry value in keystone.conf, glanceclient receive 401 error. How do I debug this 401 error in keystone ? | 08:18 |
kinpaa12389 | Enabled all logs of keystone, still not able to see place from where this error is thrown to glanceclient ? | 08:18 |
kinpaa12389 | keys=root, access, warnings, keystone, cc, radius, keystonemiddleware, keystoneauth, oslo_messaging, ldap, amqp, amqplib, sqlalchemy | 08:18 |
kinpaa12389 | above are keys in /etc/keystone/logging.conf | 08:18 |
kinpaa12389 | ~ | 08:18 |
*** rcernin has joined #openstack-nova | 08:23 | |
*** andrewbonney has joined #openstack-nova | 08:24 | |
lyarwood | Luzi: LGTM but could you cherry-pick -x $commit_from_stable_victoria https://docs.openstack.org/project-team-guide/stable-branches.html#proposing-fixes | 08:38 |
kashyap | Is it the fix to the LM w/ TLS guide? | 08:38 |
kashyap | Ah, yep | 08:39 |
lyarwood | yes | 08:39 |
kashyap | Mornin, folks | 08:39 |
lyarwood | morning | 08:39 |
openstackgerrit | Lee Yarwood proposed openstack/nova stable/ussuri: libvirt: Skip encryption metadata lookups if secret already exists on host https://review.opendev.org/c/openstack/nova/+/765770 | 08:41 |
openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: Add regression test for bug #1895696 https://review.opendev.org/c/openstack/nova/+/752487 | 08:45 |
openstack | bug 1895696 in Cinder "unable to boot instance from encrypted volume created from a glance image of an encrypted volume" [Undecided,New] https://launchpad.net/bugs/1895696 | 08:45 |
openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: Create volume attachment during boot from volume in compute https://review.opendev.org/c/openstack/nova/+/752488 | 08:45 |
openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: compute: Skip cinder_encryption_key_id check when booting from volume https://review.opendev.org/c/openstack/nova/+/752489 | 08:45 |
*** zhanglong has quit IRC | 08:46 | |
*** ociuhandu has joined #openstack-nova | 08:51 | |
openstackgerrit | Josephine Seifert proposed openstack/nova stable/ussuri: Add config parameter 'live_migration_scheme' to live migration with tls guide https://review.opendev.org/c/openstack/nova/+/782126 | 08:52 |
Luzi | lyarwood, ^ is this what you meant? | 08:54 |
lyarwood | yup ack'd thanks | 08:54 |
*** ociuhandu has quit IRC | 08:59 | |
*** lucasagomes has joined #openstack-nova | 09:02 | |
*** derekh has joined #openstack-nova | 09:02 | |
*** tosky has joined #openstack-nova | 09:04 | |
openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: compute: Lock by instance.uuid lock during swap_volume https://review.opendev.org/c/openstack/nova/+/758734 | 09:08 |
lyarwood | some quick and easy docs changes ahead of rc if anyone has time https://review.opendev.org/c/openstack/nova/+/779479 & https://review.opendev.org/c/openstack/nova/+/779446 | 09:09 |
* lyarwood owes stephenfin a few docs reviews in return | 09:09 | |
*** ociuhandu has joined #openstack-nova | 09:09 | |
*** ociuhandu has quit IRC | 09:09 | |
*** martinkennelly has joined #openstack-nova | 09:12 | |
*** ociuhandu has joined #openstack-nova | 09:20 | |
openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Use absolute path during qemu img rebase https://review.opendev.org/c/openstack/nova/+/780790 | 09:21 |
*** zoharm has joined #openstack-nova | 09:21 | |
*** vishalmanchanda has quit IRC | 09:28 | |
*** zhanglong has joined #openstack-nova | 09:33 | |
*** luksky has joined #openstack-nova | 10:07 | |
*** tosky has quit IRC | 10:13 | |
*** admin0 has quit IRC | 10:14 | |
*** tosky has joined #openstack-nova | 10:14 | |
*** ociuhandu has quit IRC | 10:16 | |
*** zhanglong has quit IRC | 10:20 | |
*** ociuhandu has joined #openstack-nova | 10:22 | |
gibi | does somebody remembers why we forbid reparenting RPs in placement? I think we allow null -> parent_uuid update but we reject parent_uuid -> new_parent_uuid update. I would need local reparenting, moving an RP to another parent within the same PR tree | 10:28 |
*** ociuhandu has quit IRC | 10:28 | |
gibi | I can see that moving an RP to one tree to the other could break things in the allocations | 10:28 |
bauzas | gibi: because it would need some reshape, maybe ? | 10:34 |
bauzas | :) | 10:35 |
gibi | bauzas: yeah if there are allocations then moving an RP could invalidate them | 10:35 |
gibi | in the generic case | 10:35 |
bauzas | gibi: afair, we only accept to modify the RPs by some reshape method | 10:35 |
bauzas | like we did for vGPUs and others | 10:35 |
bauzas | if we modify the parent, then the resources would also be modified | 10:36 |
gibi | does reshape allows reparenting? | 10:36 |
bauzas | gibi: yup | 10:36 |
bauzas | you provide new inventories | 10:36 |
gibi | hm, then reshape it is | 10:36 |
bauzas | gibi: context ? | 10:36 |
gibi | bauzas: we have a bug in neutron that creates a slightly wrong RP tree for qos | 10:37 |
bauzas | gibi: ahah | 10:38 |
*** ociuhandu has joined #openstack-nova | 10:38 | |
gibi | the expected tree would be computeRP <- neutron agent RP <- deviceRP | 10:38 |
gibi | but after a bugfix in Ussuri it was changed to | 10:38 |
bauzas | gibi: related to the ML thread I saw ? | 10:38 |
gibi | computeRP <- neutron agent RP | 10:39 |
gibi | computeRP <- device RP | 10:39 |
bauzas | I see | 10:39 |
gibi | bauzas: which thread? We just figured this problem out yesterday | 10:39 |
bauzas | sec | 10:39 |
gibi | it does not cuase any scheduling issue at the momemnt, this is why it was hidden so long | 10:39 |
bauzas | title is '[ops] Bandwidth problem on computes' | 10:40 |
gibi | bauzas: nope, that is some physical bandwidht issue | 10:40 |
bauzas | ah yeah, just looked | 10:40 |
gibi | so the above bug does not case any issue toady as the agent RP is basically unused | 10:40 |
gibi | but I have a request to start tracking OVS packet processing capacity as a resource | 10:41 |
gibi | and as today the ovs agent has a 1-1 relationship with the ovs agent, the ovs agent RP would be a good place for that new resource | 10:41 |
bauzas | okay I see the problem | 10:42 |
lucasagomes | hi, can someone please take a look at https://review.opendev.org/c/openstack/nova/+/776419, https://review.opendev.org/c/openstack/nova/+/776944 and https://review.opendev.org/c/openstack/nova/+/776934 ? | 10:42 |
gibi | I mean the ovs agent has 1-1 relationship with OVS itself | 10:42 |
gibi | bauzas: so first I would like to fix the bug | 10:42 |
lucasagomes | These are small patches that will prevent the nova gate from breaking when we change the default network backend in DevStack to OVN next cycle | 10:42 |
gibi | and then introduce new resources | 10:42 |
gibi | lucasagomes: I don't expect that these patches will be merged before the RC1 and the stable/wallaby is cut. | 10:43 |
lucasagomes | gibi, ah fair enough ok | 10:43 |
gibi | lucasagomes: but kick me after the cut and I will find somebody to review them | 10:44 |
gibi | bauzas: so one way to fix it is to re-parent the device RP | 10:44 |
lucasagomes | gibi, that sounds good, thanks much! | 10:44 |
gibi | bauzas: under the agent RP as it was before Ussuri | 10:44 |
gibi | bauzas: anyhow I will look into the reshape way now | 10:44 |
gibi | that is a good tip | 10:45 |
gibi | s/tip/suggestion/ | 10:46 |
*** rcernin has quit IRC | 10:46 | |
*** mkrai_ has quit IRC | 10:47 | |
gibi | bauzas: hm, so /reshape moves inventory and allocation between RPs but it does not move RPs between parents | 10:49 |
gibi | so I could create a new device RP under the agent and move the inventory and the allocations from the old device RP to the new device RP and then delete the old device RP | 10:50 |
gibi | this could work from placement perspectiv | 10:51 |
gibi | e | 10:51 |
gibi | neutron port does store the uuid of the RP the port is allocation from. So that would need to change too | 10:52 |
bauzas | gibi: you provide a new tree with the reshape API | 10:53 |
gibi | bauzas: really? I don't see RPs provided in the API | 10:54 |
gibi | I see inventories and allocations | 10:54 |
*** macz_ has joined #openstack-nova | 10:55 | |
*** macz_ has quit IRC | 11:00 | |
*** yoctozepto has quit IRC | 11:02 | |
gibi | bauzas: the vgpu reshape also create the pgpu RPs _before_ the placement reshape call https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8414 | 11:03 |
*** yoctozepto has joined #openstack-nova | 11:03 | |
*** artom has joined #openstack-nova | 11:04 | |
bauzas | gibi: sure, but we provided back the new provider tree by this method https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8380 | 11:05 |
bauzas | gibi: so, we provide a new tree and then we look at the allocations | 11:05 |
bauzas | if there are some of them, then we call the reshape API | 11:05 |
bauzas | https://github.com/openstack/nova/blob/3de7fb7c327db348d04d15d4cd3c4f811a336126/nova/virt/libvirt/driver.py#L8102 | 11:06 |
bauzas | s/allocs/inventories | 11:06 |
bauzas | https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/reshape-provider-tree.html#proposed-change | 11:08 |
bauzas | gibi: every 60 secs (by default), the virt driver passes back the provider tree | 11:08 |
bauzas | so it can change it | 11:09 |
*** rcernin has joined #openstack-nova | 11:09 | |
bauzas | but if the virt driver wants to modify the tree by punting new inventories and allocations from a RP to another one, then it needs to tell placement to move them | 11:09 |
bauzas | gibi: I can understand your concern, you don't want to move inventories or allocations, but just moving the parent | 11:10 |
*** vishalmanchanda has joined #openstack-nova | 11:11 | |
bauzas | but then I guess that the resources would be modified | 11:12 |
bauzas | gibi: say, for example, with https://docs.openstack.org/placement/latest/user/provider-tree.html#filtering-by-tree | 11:12 |
*** elod has quit IRC | 11:13 | |
bauzas | you would then get different candidates | 11:13 |
bauzas | that's why we need to be sure that we won't have different resources | 11:13 |
gibi | sure if I change the structure of the tree then the in_tree a_c queries will change | 11:14 |
gibi | regardless if I change the tree with reshape or just going in the db and change the parent_uuid of an RP | 11:15 |
gibi | as far as I see if I base the fix to reshape then the process is create new dev RP, rehaspe inv and alloc from old dev RP, change neutron ports to point to the new dev RP, deled old dev RP | 11:16 |
gibi | I think this would work | 11:16 |
gibi | but it creates a new RP instead of move the existing RP | 11:17 |
gibi | so it is more complicated that what I need | 11:17 |
*** rcernin has quit IRC | 11:17 | |
*** rcernin has joined #openstack-nova | 11:18 | |
*** elod has joined #openstack-nova | 11:21 | |
*** ociuhandu has quit IRC | 11:22 | |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add a resource limits guide https://review.opendev.org/c/openstack/nova/+/781433 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add a real-time guide https://review.opendev.org/c/openstack/nova/+/781434 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Remove duplicate TPM extra spec info https://review.opendev.org/c/openstack/nova/+/781435 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Remove duplicated PCI passthrough extra spec info https://review.opendev.org/c/openstack/nova/+/781436 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add SEV guide https://review.opendev.org/c/openstack/nova/+/781437 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add CPU models guide https://review.opendev.org/c/openstack/nova/+/781438 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Change formatting of hypervisor config guides https://review.opendev.org/c/openstack/nova/+/781439 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Add libvirt misc doc https://review.opendev.org/c/openstack/nova/+/781440 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Clarify host-model, host-passthrough differences https://review.opendev.org/c/openstack/nova/+/782410 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Fold in MDS security flaw doc https://review.opendev.org/c/openstack/nova/+/782411 | 11:24 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: api: Improve extra spec validator help texts https://review.opendev.org/c/openstack/nova/+/782412 | 11:24 |
*** rcernin has quit IRC | 11:25 | |
*** ociuhandu has joined #openstack-nova | 11:29 | |
gibi | bauzas: another issue. I need to keep the name of the device RP as nova depends on it | 11:41 |
gibi | and RP name needs to b unique | 11:41 |
bauzas | yup | 11:41 |
gibi | in placement | 11:41 |
bauzas | in general, we use (parent)_(name) | 11:41 |
gibi | so the create a new device RP way does not work | 11:41 |
sean-k-mooney | is this in relation to something speciric? | 11:42 |
bauzas | I wonder | 11:42 |
bauzas | sean-k-mooney: see above | 11:42 |
sean-k-mooney | im trying to read back but what i have got sofar is moving bandwith inventories via a reshape? | 11:43 |
bauzas | gibi: maybe you should just create a new reshape method | 11:43 |
bauzas | gibi: and then modify the names | 11:43 |
bauzas | in there | 11:43 |
bauzas | like : does the RP have the wrong parent UUID ? yes => ReshapeNeeded | 11:44 |
gibi | bauzas: ohh the RP name is settable, so we can use a temporary name | 11:44 |
bauzas | then, ReshapeNeeded will be provided so you could say 'ah, ReshapeNeeded, so create a new RP, modify the existing name, and then use this name for my new RP' | 11:44 |
sean-k-mooney | gibi: its setable only one i think | 11:45 |
sean-k-mooney | gibi: eg i dont think you can modify it after you create teh RP | 11:45 |
gibi | sean-k-mooney: parent (null-> nonnull) and name is settable | 11:45 |
gibi | as far as I see | 11:45 |
bauzas | gibi: this way, you wouldn't change the RPs after the compute service is restarted | 11:45 |
sean-k-mooney | there used to be a limiation around this but im tryign to recall what it was | 11:45 |
gibi | bauzas: these RPs are not handled by nova, so this reshape will happen in neutron :) | 11:46 |
bauzas | ah, shit, right | 11:46 |
gibi | bauzas: but besides that yes | 11:46 |
* bauzas needs to lunhc | 11:46 | |
sean-k-mooney | allowing RPs to be reparented would be a much cleaner approch | 11:47 |
sean-k-mooney | i have asked for this in the past | 11:47 |
sean-k-mooney | to avoid the need to reshap | 11:47 |
bauzas | no | 11:48 |
bauzas | because you need to modify them just after restarting the service | 11:48 |
bauzas | and not after | 11:48 |
bauzas | or candidates would be different | 11:48 |
sean-k-mooney | bauzas: we have disucssed this in the past and i have nver been satifed with the answeres as to why changeing the parent is not allowed | 11:48 |
gibi | sean-k-mooney: agree reparenting in my specific case would be cleaner, a generic reparenting is dangerous in the other hand | 11:49 |
gibi | like moving an RP from one tree to other | 11:50 |
sean-k-mooney | i dont think its any more dangours then a reshap | 11:50 |
gibi | that would definitely break existing allocations | 11:50 |
sean-k-mooney | it may or it may not | 11:50 |
sean-k-mooney | there are several cases where it would not | 11:50 |
sean-k-mooney | gibi: setting the parent is what i was tinking of before not the name by the way | 11:51 |
sean-k-mooney | e.g. where we could only do it once | 11:51 |
gibi | sean-k-mooney: yeah, parent can only be set from null to non null | 11:51 |
sean-k-mooney | gibi: the bandwith RPs shoudl really be using the pci device addres not the netdev name by the way | 11:52 |
sean-k-mooney | i stongly advocated for not using the netdev name in the RP name when this was being first added | 11:52 |
gibi | sean-k-mooney: nova uses PCI address, neutron uses netdev name | 11:52 |
sean-k-mooney | right which is problematic | 11:53 |
gibi | as neutron handleds these RPs it got the netdev name and nova does a translation. | 11:53 |
sean-k-mooney | right i know | 11:53 |
sean-k-mooney | but i did not wnat it to use the netdev name in neutron at all | 11:53 |
gibi | thats a wide change ^^ | 11:54 |
gibi | as neturon used that even before qos | 11:54 |
sean-k-mooney | correct which has been broken for sriov for a long time | 11:54 |
sean-k-mooney | by the way i was not suggestign that neutron stops usign devname eveywhere | 11:55 |
sean-k-mooney | that is a sperate thing | 11:55 |
sean-k-mooney | just that we model device in placment with teh pci address | 11:55 |
*** zoharm has quit IRC | 11:55 | |
sean-k-mooney | when we start modeling sriov vfs in placmnet we will need to use the PF pci address as part of the name for the VF inventores | 11:56 |
*** brinzhang has quit IRC | 11:56 | |
sean-k-mooney | other but that will be challing with qos | 11:56 |
*** brinzhang has joined #openstack-nova | 11:57 | |
sean-k-mooney | we shoudl not have different behavour with nic vs other pci devices so i dont see using the devname as a valid option for nova reporting pci device in the future | 11:57 |
gibi | sean-k-mooney: I agree that when we model the PFs in placement we need to do it right. and make the QoS related RPs aligned. | 11:59 |
*** admin0 has joined #openstack-nova | 12:02 | |
*** tobias-urdin has joined #openstack-nova | 12:10 | |
*** ociuhandu_ has joined #openstack-nova | 12:12 | |
*** ociuhandu has quit IRC | 12:13 | |
*** ociuhandu has joined #openstack-nova | 12:13 | |
gibi | sean-k-mooney: it is not impossile rename the device RPs in placement to use pci address, we just need to add some netdev -> PCI address translation code in the neturon sriov agent, and keep conditional logic in nova to look for old netdev names and new pci address names for an extra cycle to support rolling upgrade | 12:13 |
sean-k-mooney | how did the RP get move by the way | 12:14 |
sean-k-mooney | you mentioned a bugfix? | 12:14 |
sean-k-mooney | i assume that has been reverted? | 12:14 |
sean-k-mooney | and yes that is one way to adress that in the future | 12:15 |
gibi | sean-k-mooney: this fixed a bug https://review.opendev.org/c/openstack/neutron/+/696600 but broke the tree | 12:15 |
sean-k-mooney | the other way woul dbe to add a kind fo symlink or alias feature to placment | 12:15 |
sean-k-mooney | so we can refer to the same RP with different names | 12:15 |
sean-k-mooney | ah i rememebr this | 12:16 |
*** ociuhandu_ has quit IRC | 12:17 | |
sean-k-mooney | gibi: what is currently broken by having them under the compute node RP | 12:17 |
gibi | sean-k-mooney: if you upgrade from Train to Ussuri in a way that you had QoS configured already in Train then the Ussuri neutron will error out | 12:18 |
gibi | if you deploy a new Ussuri then no visible problem seen | 12:18 |
sean-k-mooney | got it | 12:18 |
sean-k-mooney | so we dont have min bandwith testing in grenade | 12:18 |
gibi | this is why the problem went undetected for so long | 12:18 |
sean-k-mooney | well neutron does not | 12:19 |
gibi | I can even think that schduling works after the upgrade | 12:19 |
gibi | but i have to reproduce it and try | 12:19 |
sean-k-mooney | i dont think it woudl break schduling | 12:19 |
gibi | new deployments are not affected just have a wrong tree sturcture | 12:19 |
sean-k-mooney | nova is not currently relying on the structure | 12:19 |
sean-k-mooney | i proposed using that stucure because i wante dot model other networking requirments | 12:20 |
sean-k-mooney | like make port count support by a vswtich | 12:20 |
sean-k-mooney | or trait on a agent provider for offloads or network type supprot e.g. vxlan | 12:20 |
sean-k-mooney | then evneutally add that to the query with same subtree | 12:21 |
*** hemna has quit IRC | 12:21 | |
sean-k-mooney | but we did nto have same subtree at the time so the agent RPs are not actuly adding any benifit currently | 12:21 |
*** hemna has joined #openstack-nova | 12:21 | |
sean-k-mooney | anyway so the fix for this would go into neutron right | 12:22 |
gibi | yes and yes | 12:22 |
gibi | and the whole think came out as I started looking into modeling OVS packet processing capacity on the ovs agent RP | 12:22 |
*** hemna has quit IRC | 12:23 | |
sean-k-mooney | as in packet per second? | 12:23 |
gibi | yepp | 12:23 |
sean-k-mooney | you an i both know that that depends on the type of packet you use and the ip pipline/match rules | 12:23 |
*** hemna has joined #openstack-nova | 12:23 | |
sean-k-mooney | its very hard to get that right without a lot of testing on your excat hardware | 12:24 |
sean-k-mooney | i assume neutron will be dumb and just have the operator say the capastity is X in a config like bandwith | 12:24 |
gibi | I'm not well educated in OVS | 12:24 |
gibi | yes the inventory would be config driven like bandwidth | 12:24 |
sean-k-mooney | gibi: the pps number is differnt for vlan network vs vxlan network on the same ovs on the same host | 12:25 |
sean-k-mooney | e.g. l3 tunneled networks use more cpu cycle to decap and encap then vlan | 12:25 |
sean-k-mooney | in the case of ovs-dpdk vm to vm traffic on the same host uses more cycle then vm to physical network | 12:26 |
sean-k-mooney | since vm to vm can levgerage as many hardware offloads | 12:26 |
sean-k-mooney | so depening on the direction of traffic and the overlay use the capsity will change | 12:27 |
*** ociuhandu has quit IRC | 12:27 | |
sean-k-mooney | you can obvioulsy deploy and test for this but it also will change based on other factors | 12:27 |
sean-k-mooney | such as security group rules implemnte in contrack and the traffic profile. | 12:27 |
sean-k-mooney | e.g. setting up and tearing downs lots of tcp connection requires all the inital packets to go through contrack to do the state tracking | 12:28 |
sean-k-mooney | so the pps will be lower with lots of new connection being established vs steady state | 12:29 |
sean-k-mooney | multicast/broadcast vs unicast also is a factor | 12:29 |
sean-k-mooney | gibi: so im glad figuring that out will be the operators problem | 12:29 |
*** ociuhandu has joined #openstack-nova | 12:33 | |
lyarwood | https://review.opendev.org/c/openstack/nova/+/768466 - anyone able to review this change introducing a nova-live-migration-ceph job? Everything has finally merged so we should be good to go now. | 12:33 |
lyarwood | I'm still working on the grenade change ontop of it | 12:34 |
gibi | sean-k-mooney: thanks for the info, I don't have this deep networking knowledge, so I appreciate these details | 12:36 |
*** hemanth_n has quit IRC | 12:37 | |
gibi | sean-k-mooney: I agree that configuring this will be really deployment specific | 12:37 |
* sean-k-mooney im just happy i dont need to deal with rfc2544 in my current role | 12:38 | |
*** ociuhandu has quit IRC | 12:38 | |
openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Simplify device_path check in _detach_encryptor https://review.opendev.org/c/openstack/nova/+/778463 | 12:38 |
gibi | sean-k-mooney: as far as I understand some of our big deployers tend use a unified network setup, like all vlan. Also the these deployers tend to do deep performance testing with their config, so I assume they will figure out a pps number for their specific deployment and traffic patterns. | 12:38 |
sean-k-mooney | i worked with other teams at intel that worked on ovs and vsperf in opnfv | 12:38 |
sean-k-mooney | gibi: yep they do | 12:39 |
sean-k-mooney | but for public clouds this gets harder as you dont know the workload | 12:39 |
gibi | yes | 12:39 |
sean-k-mooney | i assume as part of this there is an enforcemnt elemnt | 12:39 |
sean-k-mooney | e.g. a max pps qos policy | 12:40 |
sean-k-mooney | i think tc can enforce that | 12:40 |
gibi | yeah the max pps is already proposed https://review.opendev.org/c/openstack/neutron-specs/+/779940/1/specs/wallaby/qos_pps_rule.rst | 12:40 |
gibi | at the moment I'm looking into min pps only for the scheduling decision but later there might be enforcement for that as well | 12:41 |
sean-k-mooney | good because we dont currenlty allow enforcing that in nova flavor extra specs with quota:vif_* and i kind of want to kill those | 12:41 |
sean-k-mooney | well we can do what we do for bandwith today | 12:42 |
sean-k-mooney | e.g. set min = max | 12:42 |
sean-k-mooney | and just schulde on min and enfroce max | 12:42 |
sean-k-mooney | which sice we dont over subscibe means we could effectivly enforce min | 12:42 |
gibi | yes, that is possible here too | 12:43 |
sean-k-mooney | i dont think ovs support min pps | 12:43 |
sean-k-mooney | even for sriov that is a challange | 12:43 |
gibi | three are some differences with BW. The packet rate inventory seems to be directionless in nature | 12:45 |
* bauzas is back | 12:45 | |
gibi | and in case of OVS it more belongs to the softswitch level than to the bridge level | 12:45 |
gibi | and for SRIOV packet rate is not really a limiting factor | 12:46 |
gibi | afaik | 12:46 |
sean-k-mooney | the same is true for pps form an ovs point of view | 12:46 |
sean-k-mooney | bridges are meaningless for the most part | 12:46 |
sean-k-mooney | gibi: it kind fo still is | 12:47 |
sean-k-mooney | even with sriov 64b packets will limit the bandwith fo the nic | 12:47 |
sean-k-mooney | typeically they can handel 1 or 2 VF at that rate but not multiple | 12:47 |
gibi | sean-k-mooney: interesting, I did not know that | 12:48 |
gibi | so the pps will be the limit before the bandwidth if there is a lot of VFs and the packets are small? | 12:49 |
sean-k-mooney | normlaly the north south driction can handel full line rate at 64b packets for 10G thats about 14.4mpps | 12:49 |
sean-k-mooney | so it depens where the limit will come in would be vm to vm traffic on the same host | 12:49 |
sean-k-mooney | the other place its a proble is prot mirroing | 12:50 |
sean-k-mooney | our running vf in trusted mode | 12:50 |
sean-k-mooney | setting a vf to trusted mode consumes a lot of bandwith ignorign the security implications espcially if there are multipel of them | 12:51 |
sean-k-mooney | it forcs the nic to copy a subset of packets to multiple vfs | 12:51 |
*** mgoddard has quit IRC | 12:53 | |
gibi | thanks, these make sense. This also points to the direction that keeping bandwidth and packet rate inventory configurable as it is really deployment and traffic pattern dependent | 12:53 |
sean-k-mooney | gibi: you tend to hit the pcie bottelneck before the pps bottelneck in most cases but if you wnat to get full line rate with sriov and small packet i know our nfv folks had to a lot fo tuning | 12:53 |
gibi | good to know | 12:54 |
sean-k-mooney | https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/network_functions_virtualization_planning_and_configuration_guide/part-sriov-nfv-configuration#sect-configuring-sriov | 12:55 |
sean-k-mooney | they are using isolcpus=1-19,21-39 on the kernel in part 4 | 12:55 |
sean-k-mooney | 6.2.4 | 12:55 |
gibi | yeah we also separate tenat cpus from host cpus and from ovs cpus | 12:57 |
gibi | with isolcpu | 12:57 |
sean-k-mooney | really i dont think people should use that on non realtime hosts | 12:57 |
*** ociuhandu has joined #openstack-nova | 12:57 | |
sean-k-mooney | its dprecated upstream in the kernel and its kind fo a blunt hammer | 12:57 |
gibi | isolcpu helps avouding host cpu load to affect the pinned VMs | 12:58 |
sean-k-mooney | in many cases you can achive the same thing without using it | 12:58 |
sean-k-mooney | it does yes | 12:58 |
sean-k-mooney | but people enable it and then deploy unpinned vms on the host | 12:58 |
gibi | cgroups would be the alternative I guess | 12:58 |
sean-k-mooney | yes tuned has support | 12:59 |
sean-k-mooney | we wanted to entirely remove supprot for isolcpus form our product and move people to use tuned instead | 12:59 |
sean-k-mooney | but we have not been able to convice the nfv dfg to do that yet | 12:59 |
sean-k-mooney | https://github.com/redhat-performance/tuned/blob/master/profiles/cpu-partitioning/cpu-partitioning-variables.conf | 13:00 |
sean-k-mooney | you can use tuned to configure the cgoups for you including isolating the cores form irqs | 13:00 |
sean-k-mooney | thats what we use by default for non realtime hosts now | 13:01 |
*** ociuhandu has quit IRC | 13:02 | |
gibi | I see | 13:03 |
sean-k-mooney | in the cgroup way you can prevent the kernel schduling vms to the core indepently of balancing the vms on cores | 13:04 |
sean-k-mooney | isolcpus does both | 13:05 |
sean-k-mooney | so you can use the tuned/cgrop method with floating vms too not jsut pinned | 13:05 |
sean-k-mooney | but isolcpu does a few things that you cant quite do form userspace fully or at least with tuned at this point so they still use and support itn for nfv usecases | 13:06 |
gibi | I see. I think we only support pinned vms by default | 13:06 |
sean-k-mooney | that both complicates and simplcies your life :) | 13:06 |
gibi | yeah | 13:06 |
gibi | soo telco | 13:06 |
gibi | :) | 13:07 |
sean-k-mooney | complciates becasue numa and simplfies because only numa | 13:07 |
gibi | yeah | 13:07 |
sean-k-mooney | if you dont have to deal with peopel mixing numa and non numa vms its nice | 13:07 |
*** ociuhandu has joined #openstack-nova | 13:08 | |
gibi | yes we sort of force numa with pinned cpus and huge pages | 13:08 |
*** ociuhandu has quit IRC | 13:08 | |
*** ociuhandu has joined #openstack-nova | 13:10 | |
*** ociuhandu has quit IRC | 13:15 | |
openstackgerrit | Merged openstack/nova master: Dynamically archive FK related records in archive_deleted_rows https://review.opendev.org/c/openstack/nova/+/773834 | 13:19 |
*** lpetrut has quit IRC | 13:22 | |
*** ociuhandu has joined #openstack-nova | 13:26 | |
*** hemanth_n has joined #openstack-nova | 13:29 | |
bauzas | folks, if you need to discuss with kashyap, he's off IRC as he has an Internet issue with his ISP | 13:29 |
sean-k-mooney | i did have one thing that came up last night | 13:31 |
*** ociuhandu has quit IRC | 13:31 | |
sean-k-mooney | stars here http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-03-22.log.html#t2021-03-22T19:34:19 | 13:32 |
sean-k-mooney | basically qemu change the cpu flags that are set in a specific cpu model and if you dont use version machine types that will break live migration | 13:32 |
sean-k-mooney | we may or may not be able to work around that in nova via the migration xml | 13:33 |
sean-k-mooney | but the end effect is that eypc-ibrs when the vm was booted on the source host nolgere results in the same cpu feature enabel if you boot a vm on the same host or a different one with the same libvirt xml | 13:34 |
sean-k-mooney | wehn you are using unversion machine types | 13:34 |
bauzas | sean-k-mooney: kashyap dunno when he will be back | 13:34 |
sean-k-mooney | its not supper urgent at least for me | 13:35 |
*** belmoreira has joined #openstack-nova | 13:35 | |
*** ratailor has quit IRC | 13:35 | |
sean-k-mooney | but rouk has already confrim that setting an explit machine type in nova or using kashyap recent patch to allow removing feature flags both cannot fix the issue | 13:35 |
sean-k-mooney | since neither ake effect on live migration | 13:36 |
sean-k-mooney | it wont break use downstream since we use versioned machine types but it will break anyone that uses an unversioned machine type after a qemu update | 13:36 |
sean-k-mooney | that include our downstream customer if they set an unversion machine type in the nova config or in the glance image | 13:37 |
*** hemanth_n has quit IRC | 13:39 | |
*** ociuhandu has joined #openstack-nova | 13:42 | |
*** ociuhandu has quit IRC | 13:47 | |
*** ociuhandu has joined #openstack-nova | 13:50 | |
*** sapd1 has joined #openstack-nova | 13:51 | |
*** ociuhandu has quit IRC | 13:52 | |
*** ociuhandu has joined #openstack-nova | 13:52 | |
*** mgoddard has joined #openstack-nova | 13:57 | |
*** mlavalle has joined #openstack-nova | 13:59 | |
*** ociuhandu has quit IRC | 14:04 | |
*** ociuhandu has joined #openstack-nova | 14:10 | |
*** ociuhandu has quit IRC | 14:15 | |
*** ociuhandu has joined #openstack-nova | 14:27 | |
lyarwood | gibi / stephenfin ; https://review.opendev.org/c/openstack/nova/+/733627 - could either of you look at this sometime this week, wsgi fix when launched by mod_wsgi | 14:30 |
gibi | lyarwood: ack | 14:33 |
lyarwood | thanks | 14:33 |
*** ociuhandu has quit IRC | 14:35 | |
stephenfin | done | 14:39 |
gibi | stephenfin won | 14:43 |
*** dklyle has joined #openstack-nova | 14:47 | |
*** Luzi has quit IRC | 14:53 | |
bauzas | gibi: taking the semaphore for bug triaging | 14:54 |
gibi | bauzas: given | 14:54 |
bauzas | I see very old open bugs that aren't triaged | 14:55 |
bauzas | like https://bugs.launchpad.net/nova/+bug/1463631 | 14:55 |
openstack | Launchpad bug 1463631 in grenade "60_nova/resources.sh:106:ping_check_public fails intermittently" [Undecided,Confirmed] | 14:55 |
bauzas | any reason to leave them in such state ? | 14:55 |
*** whoami-rajat has quit IRC | 15:03 | |
bauzas | lyarwood: hmm, very interesting corner case https://bugs.launchpad.net/nova/+bug/1920886 | 15:13 |
openstack | Launchpad bug 1920886 in OpenStack Compute (nova) "ImageNotFound error occurs after live migration" [Undecided,New] | 15:14 |
*** mkrai has joined #openstack-nova | 15:17 | |
gibi | bauzas: https://bugs.launchpad.net/nova/+bug/1463631 has a not too old comment from lyarwood that he saw it again | 15:22 |
openstack | Launchpad bug 1463631 in grenade "60_nova/resources.sh:106:ping_check_public fails intermittently" [Undecided,Confirmed] | 15:22 |
lyarwood | sorry was flooded with pings downstream | 15:24 |
* lyarwood reads | 15:24 | |
lyarwood | my god, a well written bug report | 15:25 |
lyarwood | I think I might faint | 15:25 |
*** lpetrut has joined #openstack-nova | 15:26 | |
*** macz_ has joined #openstack-nova | 15:30 | |
*** mkrai has quit IRC | 15:32 | |
*** mkrai has joined #openstack-nova | 15:32 | |
lyarwood | gibi / bauzas ; re the grenade bug, yeah I couldn't make any sense of that, we could drop nova if you don't think we can help | 15:35 |
*** ebbex has quit IRC | 15:35 | |
*** dosaboy has quit IRC | 15:35 | |
*** nautik has quit IRC | 15:35 | |
*** owalsh has quit IRC | 15:35 | |
*** owalsh has joined #openstack-nova | 15:36 | |
*** lpetrut has quit IRC | 15:37 | |
*** martinkennelly has quit IRC | 15:37 | |
*** purplerbot has quit IRC | 15:37 | |
*** dosaboy has joined #openstack-nova | 15:37 | |
*** frickler has quit IRC | 15:37 | |
*** ebbex has joined #openstack-nova | 15:37 | |
*** irclogbot_0 has quit IRC | 15:37 | |
*** frickler has joined #openstack-nova | 15:37 | |
gibi | lyarwood: I've just run a logstash query http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Couldn't%20ping%20server%5C%22 and in the last 7 days we had 5 hits so this is active | 15:38 |
*** masterpe has quit IRC | 15:38 | |
kashyap | bauzas: Thanks! My network has been super flaky; just reading the scrollback | 15:39 |
*** irclogbot_1 has joined #openstack-nova | 15:39 | |
openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject requests to commit intermediary snapshot of an inactive instance https://review.opendev.org/c/openstack/nova/+/781138 | 15:39 |
kashyap | sean-k-mooney: Hi, looking at the chat log | 15:39 |
*** martinkennelly has joined #openstack-nova | 15:40 | |
kashyap | Assuming I'm still connected here, I'm getting a "We’re having trouble finding that site." | 15:40 |
*** adriant7 has joined #openstack-nova | 15:44 | |
*** rouk has joined #openstack-nova | 15:45 | |
bauzas | lyarwood: ack, will put the grenade bug to be invalid for nova | 15:45 |
*** xek has quit IRC | 15:45 | |
*** adriant has quit IRC | 15:46 | |
*** adriant7 is now known as adriant | 15:46 | |
rouk | sean-k-mooney: so yeah, updating cpu_map nor machine type worked. | 15:46 |
*** xek has joined #openstack-nova | 15:46 | |
rouk | im just building images with qemu 3.1 i guess now as the last ditch fix. | 15:46 |
*** rouk has quit IRC | 15:48 | |
*** irclogbot_1 has quit IRC | 15:49 | |
*** rouk has joined #openstack-nova | 15:51 | |
*** mkrai has quit IRC | 15:51 | |
*** kinpaa12389 has quit IRC | 15:52 | |
*** irclogbot_0 has joined #openstack-nova | 15:53 | |
*** ociuhandu has joined #openstack-nova | 15:56 | |
*** nautik has joined #openstack-nova | 15:57 | |
openstackgerrit | Ruby Loo proposed openstack/nova master: Allow plus sign in flavor ids https://review.opendev.org/c/openstack/nova/+/782545 | 15:58 |
*** LinPeiWen has quit IRC | 16:06 | |
openstackgerrit | Ruby Loo proposed openstack/nova master: Allow plus sign in flavor ids https://review.opendev.org/c/openstack/nova/+/782545 | 16:09 |
*** gokhani has quit IRC | 16:11 | |
*** masterpe has joined #openstack-nova | 16:20 | |
*** __ministry has joined #openstack-nova | 16:38 | |
*** links has quit IRC | 16:55 | |
openstackgerrit | Merged openstack/nova master: Initialize global data separately and run_once in WSGI app init https://review.opendev.org/c/openstack/nova/+/733627 | 16:56 |
*** __ministry has quit IRC | 16:59 | |
*** lucasagomes has quit IRC | 17:00 | |
*** iurygregory has quit IRC | 17:02 | |
*** belmoreira has quit IRC | 17:07 | |
*** jangutter_ has joined #openstack-nova | 17:08 | |
*** ociuhandu_ has joined #openstack-nova | 17:10 | |
*** jangutter has quit IRC | 17:12 | |
*** ociuhandu has quit IRC | 17:14 | |
*** ociuhandu_ has quit IRC | 17:15 | |
*** rpittau is now known as rpittau|afk | 17:26 | |
*** iurygregory has joined #openstack-nova | 17:29 | |
*** sapd1 has quit IRC | 17:29 | |
*** derekh has quit IRC | 18:03 | |
*** andrewbonney has quit IRC | 18:12 | |
*** vishalmanchanda has quit IRC | 18:17 | |
*** hamalq has joined #openstack-nova | 18:18 | |
rouk | sean-k-mooney: nova patch wise, see a simpler way to make it work? | 18:31 |
sean-k-mooney | the only thing i can think of is | 18:34 |
sean-k-mooney | to update teh migrate xml | 18:34 |
sean-k-mooney | to example exactil the feature that are in use on the source node | 18:34 |
sean-k-mooney | and skip the cpu compare | 18:34 |
sean-k-mooney | with a workaround flag | 18:34 |
sean-k-mooney | but i have not looked at writing that | 18:35 |
sean-k-mooney | i ping kashyap about this a little eairler to see if he had any ideas but you tried most of the ones i tought would be strait forward | 18:36 |
rouk | that sounds like more work than removing a commit from qemu | 18:36 |
sean-k-mooney | since they didnt work nothing that is quick an simple | 18:36 |
sean-k-mooney | ya it is | 18:36 |
rouk | ill get it cooking then. | 18:37 |
*** hemna has quit IRC | 18:39 | |
*** hemna has joined #openstack-nova | 18:50 | |
*** manuvakery1 has quit IRC | 18:51 | |
*** dtantsur is now known as dtantsur|afk | 18:55 | |
*** dtantsur|afk is now known as dtantsur|afk|afk | 18:55 | |
*** dtantsur|afk|afk is now known as dtantsur|afk | 18:55 | |
*** hemna has quit IRC | 19:25 | |
*** hemna has joined #openstack-nova | 19:26 | |
melwitt | gmann: do you think such a change like https://review.opendev.org/c/openstack/nova/+/782545 represents an api change that needs a new microversion? | 19:53 |
gmann | melwitt: yeah, it change 400 to 200 which is interop issue. microversion is needed for such changes | 19:56 |
melwitt | gmann: ack, thanks | 19:56 |
gmann | melwitt: this is same as allowing more char in keypair name, - https://blueprints.launchpad.net/nova/+spec/allow-special-characters-in-keypair-name | 20:09 |
*** pawan-gupta has quit IRC | 20:15 | |
melwitt | gmann: a-ha, thanks for the example! | 20:19 |
*** sean-k-mooney has quit IRC | 20:24 | |
*** jmlowe has quit IRC | 20:39 | |
*** ociuhandu has joined #openstack-nova | 20:39 | |
*** jmlowe has joined #openstack-nova | 20:40 | |
*** rcernin has joined #openstack-nova | 21:01 | |
*** ociuhandu has quit IRC | 21:05 | |
*** ociuhandu has joined #openstack-nova | 21:11 | |
*** rcernin has quit IRC | 21:16 | |
*** ociuhandu has quit IRC | 21:16 | |
*** jangutter has joined #openstack-nova | 21:22 | |
*** jangutter_ has quit IRC | 21:26 | |
*** ralonsoh has quit IRC | 21:36 | |
*** fnordahl has quit IRC | 21:37 | |
*** rcernin has joined #openstack-nova | 21:46 | |
*** slaweq has quit IRC | 21:52 | |
*** slaweq has joined #openstack-nova | 21:54 | |
*** rcernin has quit IRC | 22:10 | |
*** rcernin has joined #openstack-nova | 22:10 | |
openstackgerrit | Merged openstack/nova stable/ussuri: Add config parameter 'live_migration_scheme' to live migration with tls guide https://review.opendev.org/c/openstack/nova/+/782126 | 22:46 |
*** hoonetorg has joined #openstack-nova | 22:53 | |
*** slaweq has quit IRC | 23:05 | |
*** slaweq has joined #openstack-nova | 23:06 | |
*** jamesden_ has quit IRC | 23:16 | |
*** jamesdenton has joined #openstack-nova | 23:16 | |
*** luksky has quit IRC | 23:22 | |
*** johnthetubaguy has quit IRC | 23:36 | |
*** johnthetubaguy has joined #openstack-nova | 23:38 | |
*** macz_ has quit IRC | 23:40 | |
*** johnthetubaguy has quit IRC | 23:49 | |
*** hamalq has quit IRC | 23:55 | |
*** johnthetubaguy has joined #openstack-nova | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!