| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Use a per-instance cephx identity for CephFS shares https://review.opendev.org/c/openstack/nova/+/1001900 | 03:22 |
|---|---|---|
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Extract share management from ComputeManager https://review.opendev.org/c/openstack/nova/+/999349 | 03:22 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | 03:22 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | 03:22 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | 05:34 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix the test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/c/openstack/nova/+/1003910 | 05:53 |
| Uggla | gouthamr "but i don't know if Uggla gibi thought that wasn't necessary when they wrote this up", If I remember well nothing was really specified about it. I set something just to highlight the share was somewhere managed by nova. | 09:43 |
| *** sfinucan is now known as stephenfin | 10:55 | |
| opendevreview | Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410 | 12:26 |
| opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug 2162835 https://review.opendev.org/c/openstack/nova/+/999977 | 13:17 |
| opendevreview | ribaudr proposed openstack/nova master: Fix CPU pinning reset during evacuate periodic race https://review.opendev.org/c/openstack/nova/+/999978 | 13:17 |
| opendevreview | Kamil Sambor proposed openstack/nova master: Replace StaticallyDelayingWrapper with futurist DelayingExecutor https://review.opendev.org/c/openstack/nova/+/997410 | 13:19 |
| *** EugenMayer4401809 is now known as EugenMayer440180 | 13:33 | |
| *** EugenMayer4401808 is now known as EugenMayer440180 | 15:06 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix parallel evacuations pushing instance to ERROR https://review.opendev.org/c/openstack/nova/+/1003994 | 15:17 |
| gibi | this is a nasty one ^^ | 15:17 |
| gibi | (all hail the automatic evacuation engines hammering the nova-api with parallel evac requests) | 15:17 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix the test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/c/openstack/nova/+/1003910 | 15:30 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | 15:30 |
| gmaan | gibi: regarding the test_migrate_disk_and_power_off_crash_finish_revert_migration failure, I think I am going in same direction as you commented on the ashish patch. consider the graceful_shutdown_timeout in polling the instance state and if execution is more for this test then we can reduce the graceful_shutdown_timeout for this test to make graceful shutdown fast. | 15:36 |
| gmaan | I am testing the polliing fix first and try to reduce the timeout. | 15:37 |
| gmaan | i tried to find the ashish change but could not due to its topic, i tried to check if any effort to run this test in threading mode and only searched eventlet-removal gerrit topic | 15:38 |
| gmaan | ashigupt: ^^ and updated the topic for your change | 15:43 |
| gmaan | dansmith: there is one bug for the task tracking leaking the live migration task on destination, its fix it ready if you hav time to look :https://review.opendev.org/c/openstack/nova/+/1003386/1 | 15:53 |
| dansmith | gmaan: ack I'm trying to focus on the manila FFE today since we can fix bugs later | 15:54 |
| gmaan | sure | 15:54 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: Fix the test_migrate_disk_and_power_off_crash_finish_revert_migration https://review.opendev.org/c/openstack/nova/+/1003910 | 16:32 |
| ashigupt | gmaan, ack thankyou | 16:36 |
| gmaan | ashigupt: let me test the fix on my change and then I can update you, something you can check on monday | 16:37 |
| ashigupt | gmaan: sure thank you | 16:41 |
| opendevreview | Ghanshyam Maan proposed openstack/nova master: [func test]Catch hanging task at graceful shutdown https://review.opendev.org/c/openstack/nova/+/1003266 | 16:51 |
| dansmith | melwitt: gouthamr I'm a bit unahppy with the double-nested RPC-to-DB calls in the reconcile patch | 18:01 |
| dansmith | we can get a list of all the instances on the current host without doing the list all by host.. and can't we add another query to the object to get all mappings for a list of uuids? that would turn it from n^2 queries to 2 | 18:03 |
| dansmith | because we're also doing a share object lookup for every migration we find, even old long-since-confirmed migrations forever and ever it seems | 18:05 |
| dansmith | if we had one "get all share mappings for list-of-uuids" call we could do that once with all the uuids we're looking for, from both the instance list and migration list | 18:06 |
| * gouthamr guess so, orients.. | 18:08 | |
| dansmith | like, let's say we have 500 instances on this host, none of which use shares currently.. and 10k migrations over the last several years, few of which are still on this host and none of which used shares.. we're going to do 10500 RPC calls to DB queries at startup and every hour until the cows come home right? | 18:11 |
| dansmith | for operators that do slide puzzle migrations, every instance running in the system will have gone through every host eventually and those will all have confirmed migration records for any host they've ever been on AFAIK | 18:12 |
| dansmith | (sorry n^2 queries above is not right of course, it's just n^2 iterations) | 18:14 |
| gouthamr | dansmith: ack, yeah no this falls on that scale - although RPC can handle it, its inefficient.. so, get all instance UUIDs on this host, union in the instance UUIDs from the migration list, then one ShareMappingList.get_by_instance_uuids for that whole set... and iterate that instead... | 18:14 |
| dansmith | cc Uggla since you reviewed too ^ | 18:14 |
| * gouthamr #TIL new term: slide puzzle migrations - that's exactly what people do | 18:14 | |
| sean-k-mooney | on startup we pass a list of all the isntance to the driver via a new process instnace on startup up method after thehost start is done | 18:14 |
| sean-k-mooney | *after the host startup stable uuid check is done | 18:15 |
| dansmith | sean-k-mooney: right we already have the list of instances but this adds another | 18:15 |
| sean-k-mooney | rigfht i jsut removed one duplciate list all isntance on host on startup this cycle | 18:16 |
| sean-k-mooney | and got down to doing it only once | 18:16 |
| sean-k-mooney | so if we can not go back to 2 that would be nice. | 18:16 |
| dansmith | gouthamr: I'm not sure RPC can handle it TBH.. if we have 500 nodes in a cell and all of them are hammering the MQ conductors and DB with over 10k requests every time the periodic runs, or storming them at cluster restart.. that's not going to go well | 18:16 |
| sean-k-mooney | im not really followign the full converstion i just saw scary numbers | 18:17 |
| dansmith | oh actually it's worse | 18:18 |
| dansmith | because then for each share we again query all the instances using that share without a host filter I guess | 18:18 |
| gouthamr | let me look, yeah - i guess i can piggy back on "get_uuids_by_host" - pass it down to the helper that'll get invoked at init, and periodically | 18:18 |
| sean-k-mooney | gouthamr: what are you actully trying to do | 18:19 |
| sean-k-mooney | is this for https://review.opendev.org/c/openstack/nova/+/1001916/19/nova/compute/share_management.py#588 | 18:19 |
| gmaan | ditto, even in native threading mode, RPC is not so cheap and can put other RPC requests in queue. these numbers are really scary :) | 18:19 |
| dansmith | sean-k-mooney: yes | 18:20 |
| dansmith | gmaan: agree | 18:20 |
| gouthamr | sean-k-mooney: yeah, i'm trying to identify migrations that have leaked manila access rules, find them and reap them periodically | 18:20 |
| dansmith | gmaan: native threading will actually let n-cpu hammer the other services even harder | 18:20 |
| gmaan | yeah | 18:20 |
| sean-k-mooney | ok so 1 call to manilla for all share access rules for the current host 1 2 calls to the db 1 for the isntance on this host and one for migrations? | 18:21 |
| dansmith | sean-k-mooney: that's what I suggested but apparently we can't do that? | 18:21 |
| sean-k-mooney | oh ok | 18:22 |
| sean-k-mooney | so to that atht then we woudl neeed to reconstcut it ofrm the nova side | 18:23 |
| sean-k-mooney | what about not makign it a perodic and provide a heal share allocations command liek we did for placement allcoaitns in nova-manage | 18:23 |
| sean-k-mooney | this repair shoudl nto be needed constantly right | 18:23 |
| gouthamr | yeah unfortunately, there's no "give me all access rules for this client" sorta call available in manila.. it's "give me all access rules for this share".. | 18:24 |
| sean-k-mooney | so if we provide a nova-manage command for it operators can use it in the event of a bug | 18:24 |
| sean-k-mooney | and there is no give me all the shares with assess rules for this host command | 18:24 |
| gouthamr | yes | 18:26 |
| sean-k-mooney | ok so host on the share is the hostnaem of the thing exproting the share | 18:27 |
| gouthamr | yes | 18:27 |
| sean-k-mooney | just looking at https://docs.openstack.org/api-ref/shared-file-system/#id17 | 18:27 |
| sean-k-mooney | ack | 18:27 |
| gouthamr | (not compute host) | 18:27 |
| dansmith | sean-k-mooney: weren't you trying to get us to stop doing things like reading proc and sysfs directly in code? | 18:28 |
| dansmith | using nova/filesystem? | 18:28 |
| sean-k-mooney | yes ideally | 18:28 |
| sean-k-mooney | manily ot have a central place that handels the resouce busy issues | 18:28 |
| sean-k-mooney | and make it more testable | 18:28 |
| dansmith | looks like no such handler for proc in there, but this code reads /proc/mounts directly, so probably should go there.. although I'm surprised there's no os module for that | 18:29 |
| sean-k-mooney | the place we had the speic issue that bit us was reading the online state fo cpu cores | 18:30 |
| sean-k-mooney | that was randomly returnign a device busy sortly before/after onlining a core | 18:30 |
| sean-k-mooney | as it was still goting though the state machine in the kernel | 18:30 |
| sean-k-mooney | im not sure if proc would have the same issue | 18:31 |
| dansmith | sean-k-mooney: right this code would not run on macos and so if it's naked in tests will break that compatibility we just achieved | 18:31 |
| sean-k-mooney | since its not actully reading form hardware | 18:31 |
| gouthamr | sean-k-mooney: on the self-healing at the host level vs giving the operator means.. we thought about this earlier.. draining a access needs host-local state (my_shared_fs_storage_ip/CONF.host + "is the share still hard mounted here").. operators could disable this periodic check though and let this only occur at startup (like other optional periodic tasks) | 18:31 |
| sean-k-mooney | oh well yes that too | 18:31 |
| dansmith | yeah making this not self-heal is bad, IMHO | 18:32 |
| dansmith | and requiring cron running nova-manage on compute nodes is no bueno | 18:32 |
| sean-k-mooney | im not agaisnt self healing i just suggested providing a command because we evnetull moved form healign the allcoation automaticly to a commnd | 18:32 |
| sean-k-mooney | but ya i assume its like cider where we neded lcoal state to do that form brick | 18:33 |
| dansmith | yeah but the command has to be run on each compute node, not centrally | 18:33 |
| gouthamr | technically, on macos, you're only running unit tests? | 18:33 |
| sean-k-mooney | dansmith: yes like with volumes | 18:33 |
| sean-k-mooney | wich i also agree is kind of a pain so we can likely park that | 18:33 |
| sean-k-mooney | so the prople is how to do that without melting the rpc bus | 18:34 |
| sean-k-mooney | *problem | 18:34 |
| gouthamr | (i mocked the /proc/mounts calls .. i run unit tests locally on mac, and yes, this would _have_ to be mocked regardless) | 18:34 |
| sean-k-mooney | it sound like a direct remotable obejct method on InstanceList that retuns the list of instnace that have attachment on a given host with the atachmetn info is going to be needed | 18:36 |
| gouthamr | (ese: https://review.opendev.org/c/openstack/nova/+/1001916/19/nova/tests/unit/compute/test_compute_mgr.py#4435) | 18:36 |
| sean-k-mooney | so one rpc that give you only the instnace for a given host that have attachmets with the attachment info | 18:36 |
| sean-k-mooney | dansmith: was the concer only with instnace the were moved sepcificly? | 18:36 |
| sean-k-mooney | doing that filtering on the python side seam like a bad time but we shoudl be able to do it semi effeictly in sql | 18:38 |
| dansmith | gouthamr: mm, nope functional should be working (I'm running now to see if something has regressed) | 18:39 |
| sean-k-mooney | we would need to join on the migration table compute nodes tabel and isntance table and isntance sares table but we shoudl be abel to say give me all instnace that migrated to/from this host that had share attachmetns | 18:40 |
| dansmith | sean-k-mooney: and yeah sounds like maybe a single query method that returns "any share mapping we might care about" would be the best option.. we have a ShareMappingList already which is the right place to put it I thinjk | 18:40 |
| gouthamr | ack, i ran functional too on a mac.. /me is running it again after collapsing the queries | 18:40 |
| sean-k-mooney | ya if its the mappitn we care about that sound better to me as well | 18:40 |
| sean-k-mooney | i was not sure if we wanted the mappitns or the instnace or both | 18:41 |
| dansmith | sean-k-mooney: I'm going to commit comments on the concerns.. would be nice to have your weight applied | 18:41 |
| dansmith | sean-k-mooney: mappings AFAIK | 18:41 |
| gouthamr | yes, mappings | 18:41 |
| sean-k-mooney | sure but as the questions i have been asking imply i have not been folloing the overall work closely | 18:42 |
| melwitt | yeah so it would be get all share mappings for a list of instance uuids rather than doing one by one right | 18:42 |
| sean-k-mooney | but i agree the scalaiblity of the current approch is concerning | 18:42 |
| gouthamr | yeah.. the data model was forcing me with some of these calls in the code.. I'm working on your suggestions to stitch (InstanceList + MigrationList + ShareMappingList) and get a list to parse... one remotable RPC that does all this would help cleanly for sure. but, can i do that in Indri? or would you prefer to have it done now? | 18:42 |
| melwitt | adding a ShareMappingList.get_by_instance_uuids() would be simple enough. do we need more than that? | 18:45 |
| dansmith | gouthamr: what you just described is what I just said we should do in my comment just now | 18:45 |
| sean-k-mooney | or ShareMappingList.get_by_migration_host ? | 18:46 |
| dansmith | gouthamr: I don't think we can/should merge this the way it is, but it would suck to have to punt it all to indri as well.. since you'll have to add RPC changes it won't be backportable either | 18:46 |
| dansmith | melwitt: instance uuids is not enough and it's not even the likely larger list | 18:47 |
| gouthamr | yeah no.. i wasn't looking to backport this stuff | 18:47 |
| gouthamr | its super important for me though to make forward progress on the cold migration :( it makes all of this partially usable before we fix up the hard block on other instance actions that you folks pointed out at the PTG | 18:47 |
| gouthamr | the backport was just going to be for: https://review.opendev.org/c/openstack/nova/+/1001900/ | 18:48 |
| dansmith | yup | 18:48 |
| gouthamr | s/backport/backport request :) | 18:48 |
| * gouthamr reads comments on gerrit | 18:49 | |
| gouthamr | ty for this discussion and the comments! | 18:49 |
| sean-k-mooney | gouthamr: i have not looked at https://review.opendev.org/c/openstack/nova/+/1001900 in detail but just looking at the tile im not sure that is backportable | 18:53 |
| gouthamr | sean-k-mooney: oh, read on :P yes, we're moving auth, but it doesn't break anything running until you choose | 18:53 |
| sean-k-mooney | it might be but we have to supprot n-2 compatiblity | 18:54 |
| gouthamr | sean-k-mooney: this will be supplemented with an OSSN/operator guidance on how the move will occur | 18:54 |
| sean-k-mooney | so we need atuth to work with a patched souce node and unpatch dest | 18:55 |
| sean-k-mooney | as an example | 18:55 |
| sean-k-mooney | its too late on a friday for me to really ingest the details | 18:55 |
| gouthamr | sean-k-mooney: it will; on the unpatched destination, we'll use a "shared credential" until that's patched some day | 18:55 |
| gouthamr | sean-k-mooney: yes :) ty for the brainstorm! have a great weekend | 18:56 |
| dansmith | sean-k-mooney: it's really not ideal, I've already raised the concerns | 18:57 |
| sean-k-mooney | ack, im not really sure that shareing a cephx id is a problme | 18:58 |
| dansmith | sean-k-mooney: the virtiofs implementation in the tree is sort of "so leaky that it doesn't break upgrades" which is not good but also means that we _can't_ do automatic cleanup there anyway | 18:58 |
| sean-k-mooney | but i woudl have to read the bug to knwo why it woudl be | 18:58 |
| dansmith | sean-k-mooney: it's a hardening thing | 18:58 |
| sean-k-mooney | sure but for nova rbd driver we mostly use the same cephx key | 18:59 |
| sean-k-mooney | technially it can vary per host | 18:59 |
| sean-k-mooney | but i dont really knwo of any installer that does that | 18:59 |
| sean-k-mooney | we just use a single nova key on all compute nodes | 18:59 |
| sean-k-mooney | so using shared ceph keyrings between host is not generally a sefutiy issue but i dont really have an object to hardening | 19:00 |
| dansmith | I'm not sure our bad behavior is a justification to do it badly.. but those ceph shares are less user-visible than these I think | 19:01 |
| dansmith | and the current implementation sort of can't even track which hosts should and should not have access to a share as a result | 19:01 |
| sean-k-mooney | ack as i said i have no real issue with hardenign | 19:01 |
| sean-k-mooney | jsut notign that it not the same as nova provisend rbd voluems | 19:01 |
| sean-k-mooney | i think for cidner we are given a cephx key in teh conenciton info to ues | 19:02 |
| sean-k-mooney | possibel per voluem? | 19:02 |
| sean-k-mooney | and as i said ther is nothign prventign you minting a per host key for images_type=rbd today | 19:02 |
| sean-k-mooney | but ya as you said if you have escaped to the host your already have other compomise vector to worry about | 19:03 |
| * gouthamr had stepped away into a terminal | 19:05 | |
| sean-k-mooney | for the peroupus of share attachment the client of the sheare was desiged to be nova rather then the instnace | 19:07 |
| sean-k-mooney | the same way as nova is the sincel client for ceph voleusm for images_type=rbd | 19:07 |
| dansmith | that's how the code is designed now, but I think that was probably wrong | 19:07 |
| sean-k-mooney | it depend on your pesertive of who own the expore the end user or nova | 19:08 |
| sean-k-mooney | if its nova which is howe we desgiend it its correct | 19:08 |
| dansmith | treating all of nova as a single unit is a bad idea, whether it be shares, volumes, access to the MQ or DB, etc.. in 2010 I think that was a lot more normal but today it seems verging on irresponsible to me | 19:08 |
| gouthamr | ++ | 19:08 |
| sean-k-mooney | if its the end user then the cephex key shoudl be in barbican liek cinder encyption keys | 19:08 |
| sean-k-mooney | right as i said im oke with hardenign but that feels mroe like a security feature then soemthign we would activly backport | 19:09 |
| gouthamr | cephfs != rbd, for RBD, cephx is one of several layers. For a CephFS share, the cephx access rule is the access-control layer.. | 19:10 |
| dansmith | it is and you could surely argue that it's not backport-worthy IMHO | 19:10 |
| gouthamr | even there though, we could make a better choice than having a single shared credential, imo.. | 19:10 |
| sean-k-mooney | gouthamr: we dont require it to be shared in nova today taht up to the installer | 19:11 |
| melwitt | so are we thinking a separate patch for the ShareMappingList.get_share_ids_for_host() to go in front of the reconcile patch? | 19:11 |
| gouthamr | melwitt: i'll fold it in.. | 19:11 |
| melwitt | to try and minimize churn on the reconcile? ok | 19:11 |
| gouthamr | sean-k-mooney: ack, yes, but in this case, it's hardcoded so worse :( | 19:11 |
| gouthamr | melwitt: it was the last patch in the series, so i can fix it up in isolation :) | 19:12 |
| gouthamr | dansmith: i'm exploring psutil.disk_partitions.. i see its being used by quobyte: https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/volume/quobyte.py#L112 | 19:13 |
| dansmith | gouthamr: aha, nice | 19:14 |
| gouthamr | dansmith: it solves my problem of not hemorrhaging on an inaccessible hard mount | 19:14 |
| dansmith | so realistically here.. unfortunately monday is a holiday for many of us | 19:16 |
| dansmith | I wish Uggla was around but I assume merging this on tuesday is pretty dang late | 19:16 |
| dansmith | I almost wish melwitt had been less of a responsible person and sent the bottom patches to the gate already :) | 19:16 |
| * gouthamr is grinning, but also sad | 19:17 | |
| sean-k-mooney | so realsticlly , we cut rc1 without this next week, we can still merge this on master it just will be for next cycel if we reapprove the spec | 19:17 |
| sean-k-mooney | and we can repprose that already | 19:18 |
| dansmith | I have no plans for this weekend other than recharging my brokenness.. I'm not really up for reviewing tomorrow but I guess I could come back later today if it happens that it's all ready | 19:18 |
| dansmith | but it's feeling pretty rushed for sure | 19:19 |
| melwitt | I can also review if updated today | 19:20 |
| melwitt | and I can check up on it over the weekend | 19:21 |
| gouthamr | from my (probably biased) vantage point, the reconciliation of rules because we're trying not to wedge instances on cold migration is good hardening.. cold migration will be used no doubt if we shipped this even without the reconciliation - but, we're doing this right and i wouldn't do it differently.. | 19:21 |
| gouthamr | I know of a few <redacted> clouds that would be happy to have this feature land in H.. but they're all really waiting for a day their can hot-attach cephfs and call it a day - it's killing their cloud right now when person 100001 is coming to them to enable this, and they have to tell them they'd have to roll out their own tenant-scoped ceph cluster | 19:22 |
| dansmith | gouthamr: we also know the reconciliation will not be backportable if it becomes more of a problem than you think | 19:23 |
| gouthamr | yes | 19:23 |
| melwitt | yeah, that's the problem, reconcile would not be backportable | 19:23 |
| dansmith | IME instances wedged in the middle of migrations is not that uncommon so I'm nervous about having no path for cleanup for something users can trigger | 19:23 |
| gouthamr | ack; i'll flush this out and test it and have something for your eyes in a bit.. the psutil change was the simplest :) but, the share mapping object update, i'll deep-test | 19:24 |
| melwitt | that is my experience also. maybe the biggest complaint about nova is stuff stuck in a state requiring manual database intervention to recover | 19:25 |
| dansmith | yup | 19:25 |
| dansmith | gouthamr: okay then I'll take a break for a bit so I can be available for a bit later | 19:25 |
| gouthamr | ack, thank you dansmith! | 19:25 |
| gouthamr | same, melwitt and sean-k-mooney! | 19:25 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | 20:07 |
| opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Only load same-host instances when checking share usage https://review.opendev.org/c/openstack/nova/+/1004057 | 20:07 |
| * gouthamr is watching gate, but got all comments he think | 20:39 | |
| gouthamr | s | 20:39 |
| dansmith | melwitt: looks okay to me | 21:10 |
| gouthamr | dansmith: do you mind me folding follow up comments into a single commit post these merging | 21:16 |
| gouthamr | like a low-risk follow up that'll land next week? | 21:16 |
| dansmith | which comments? | 21:16 |
| gouthamr | you're right about "is_mounted()".. it's a mount path that we're check | 21:17 |
| gouthamr | checking* | 21:17 |
| dansmith | oh yeah that's why I said "not critical to fix right now" | 21:17 |
| gouthamr | ++ ack, ty will watch for melwitt's comments as well | 21:18 |
| * gouthamr steps away briefly | 21:18 | |
| melwitt | will be looking | 21:39 |
| gmaan | dansmith: thanks for review on task tracking fix, I replied to your commment https://review.opendev.org/c/openstack/nova/+/1003386/comments/25784b40_6a5b7aad | 21:42 |
| gmaan | if we are ok to do RPC versioning bupm, | 21:42 |
| gmaan | * RPC version bump, then i can add a new RPC cast to end the task which will not impact the dest rollback for old compute mixed env | 21:43 |
| gmaan | I was thinking to handle that generacally in part-3 | 21:43 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!