| *** brinzhang has joined #openstack-cinder | 00:03 | |
| *** brinzhang_ has quit IRC | 00:06 | |
| *** _hemna has joined #openstack-cinder | 00:15 | |
| *** ianychoi has quit IRC | 00:18 | |
| *** ianychoi has joined #openstack-cinder | 00:20 | |
| *** tejdeep has quit IRC | 00:46 | |
| *** Liang__ has joined #openstack-cinder | 00:59 | |
| *** _hemna has quit IRC | 01:14 | |
| *** eharney has quit IRC | 01:15 | |
| *** imacdonn has quit IRC | 01:18 | |
| *** imacdonn has joined #openstack-cinder | 01:18 | |
| *** _erlon_ has quit IRC | 02:05 | |
| *** brinzhang_ has joined #openstack-cinder | 02:08 | |
| *** brinzhang has quit IRC | 02:12 | |
| *** zzzeek has quit IRC | 02:16 | |
| *** stakeda has joined #openstack-cinder | 02:17 | |
| *** zzzeek has joined #openstack-cinder | 02:19 | |
| *** deiter has quit IRC | 02:21 | |
| *** n-saito has joined #openstack-cinder | 02:27 | |
| *** dklyle has quit IRC | 02:34 | |
| *** david-lyle has joined #openstack-cinder | 02:35 | |
| *** tkajinam_ has joined #openstack-cinder | 02:57 | |
| *** tkajinam has quit IRC | 02:59 | |
| *** m75abrams has joined #openstack-cinder | 03:05 | |
| *** tkajinam__ has joined #openstack-cinder | 03:07 | |
| *** tkajinam_ has quit IRC | 03:09 | |
| *** psachin has joined #openstack-cinder | 03:30 | |
| *** dviroel has quit IRC | 03:33 | |
| *** abhishekk has joined #openstack-cinder | 03:34 | |
| *** gkadam has joined #openstack-cinder | 03:50 | |
| *** gkadam has quit IRC | 03:50 | |
| *** whoami-rajat has joined #openstack-cinder | 03:52 | |
| *** udesale has joined #openstack-cinder | 03:54 | |
| *** sapd1_x has joined #openstack-cinder | 03:57 | |
| *** sapd1_x has quit IRC | 04:07 | |
| *** tejdeep has joined #openstack-cinder | 04:12 | |
| *** pcaruana has joined #openstack-cinder | 04:27 | |
| *** jmccrory has quit IRC | 04:52 | |
| *** Luzi has joined #openstack-cinder | 04:55 | |
| *** jmccrory has joined #openstack-cinder | 05:04 | |
| *** _hemna has joined #openstack-cinder | 05:11 | |
| *** pcaruana has quit IRC | 05:12 | |
| *** jmccrory has quit IRC | 05:14 | |
| *** jmccrory has joined #openstack-cinder | 05:14 | |
| *** _hemna has quit IRC | 05:15 | |
| *** boxiang has joined #openstack-cinder | 05:25 | |
| *** pcaruana has joined #openstack-cinder | 05:25 | |
| *** jmccrory has quit IRC | 05:41 | |
| *** ruffian_sheep has joined #openstack-cinder | 05:46 | |
| *** ruffian_sheep has quit IRC | 05:59 | |
| *** tejdeep has quit IRC | 06:02 | |
| *** jmccrory has joined #openstack-cinder | 06:11 | |
| *** baojg has joined #openstack-cinder | 06:26 | |
| *** baojg has quit IRC | 06:27 | |
| *** baojg has joined #openstack-cinder | 06:28 | |
| *** jawad_axd has joined #openstack-cinder | 06:38 | |
| *** tosky has joined #openstack-cinder | 06:41 | |
| *** raghavendrat has joined #openstack-cinder | 06:42 | |
| raghavendrat | is whoami-rajat: online? | 06:43 |
|---|---|---|
| whoami-rajat | raghavendrat: Hi | 06:43 |
| raghavendrat | this is regarding https://review.opendev.org/#/c/657585/; whenever you get some time, could you please review & share your feedback | 06:44 |
| *** brinzhang has joined #openstack-cinder | 06:48 | |
| whoami-rajat | raghavendrat: sure | 06:56 |
| raghavendrat | thanks | 06:56 |
| *** sahid has joined #openstack-cinder | 06:59 | |
| *** bhagyashris_ has joined #openstack-cinder | 07:04 | |
| *** rcernin has quit IRC | 07:04 | |
| *** tesseract has joined #openstack-cinder | 07:05 | |
| *** dpawlik has joined #openstack-cinder | 07:07 | |
| *** _hemna has joined #openstack-cinder | 07:12 | |
| *** _hemna has quit IRC | 07:16 | |
| *** gfidente has joined #openstack-cinder | 07:29 | |
| gfidente | geguileo in the upgrade from queens to train | 07:30 |
| gfidente | is there a specific order in which it is best to upgrade the cinder services? | 07:30 |
| geguileo | gfidente: is this a rolling upgrade? | 07:30 |
| gfidente | geguileo no I think it's going to be like it was for newton to queens | 07:31 |
| gfidente | so we'd basically run all the db schema updates and jump the packages to the newest version | 07:32 |
| gfidente | one node at a time | 07:32 |
| geguileo | gfidente: I assume this is going to be a FFU | 07:34 |
| gfidente | geguileo yeah that | 07:34 |
| gfidente | I think the assumption is that upgrading api/scheduler on controlplane nodes | 07:34 |
| gfidente | one by one | 07:34 |
| gfidente | then move on to -volume | 07:34 |
| gfidente | will magically get everything back to work on the newer version | 07:35 |
| gfidente | assuming service is unavailable or unreliable until all -volume instances are upgraded too | 07:35 |
| geguileo | gfidente: in Queens we added a "bad" online migration script | 07:36 |
| geguileo | gfidente: that requires it to be run with cinder-volumes running | 07:36 |
| geguileo | gfidente: so we need to run the cinder-manager online-data-migrations command with cinder-volumes running | 07:37 |
| geguileo | gfidente: and then we can stop them and continue the FFU normally | 07:37 |
| gfidente | geguileo and volume needs to stay at the queens level | 07:37 |
| gfidente | until the onlinemigrations finished | 07:38 |
| gfidente | ? | 07:38 |
| geguileo | gfidente: yeah, until the quees online migrations are completed | 07:38 |
| gfidente | wait, this is to upgrade to train | 07:38 |
| geguileo | gfidente: then we can go through the cycle of upgrading to N+1, db sync, online migration, upgrading to N+2, etc | 07:38 |
| gfidente | yeah and can -volume instances be upgraded last, jumping to train | 07:39 |
| gfidente | in case they are on dedicated nodes | 07:39 |
| gfidente | ? | 07:39 |
| geguileo | iirc on FFU we don't start the services until we have upgraded everything to N+3 | 07:40 |
| geguileo | so the order is not really relevant, since everything will have been stopped | 07:40 |
| gfidente | geguileo yeah but if the services are on different nodes it might happen | 07:41 |
| gfidente | that api/scheduler get to the newer version before volume is upgraded | 07:41 |
| gfidente | I assume the N / N-3 won't work? | 07:41 |
| gfidente | I mean, it's not reliable? | 07:41 |
| geguileo | the db-sync and online migration happens on a single node iirc | 07:42 |
| geguileo | so you only install on the intermediary versions on 1 node | 07:42 |
| geguileo | to be able to run the migrations | 07:42 |
| geguileo | and then on the rest of the nodes you just installed the N+3 version | 07:43 |
| geguileo | iirc | 07:43 |
| gfidente | yeah and assuming on the api nodes you jump db to N+3 | 07:43 |
| *** raghavendrat has quit IRC | 07:43 | |
| gfidente | but still have -volume running at the N level, is that a situation in which service is supposedly available | 07:43 |
| gfidente | or not until -volume gets to N+3 too? | 07:44 |
| geguileo | No, you have stopped all the services in FFU | 07:44 |
| geguileo | So you have NOTHING running at N, N+1, N+2 | 07:44 |
| gfidente | geguileo that is the point I was trying to make, not if -volume is on a separate node | 07:44 |
| geguileo | you had running N, you stop, do db-online-migration in N, then install N+1 in 1 node, do db-sync, do online-migration, install N+2 in 1 node, db-sync, online-migration, install N+3 on 1 node, db-sync (no online-migration here), install N+3 on all nodes, run N+3 on all nodes | 07:45 |
| geguileo | gfidente: it is irrelevant where the services are installed | 07:46 |
| geguileo | gfidente: you have 1 node that will be in charge of DB changes | 07:46 |
| geguileo | gfidente: and then you will install N+3 on ALL of them with the DB ready | 07:46 |
| geguileo | gfidente: so you will be running N+3 on all of them, regardless of the order at which they start | 07:46 |
| geguileo | gfidente: I have never run FFU, but that was the design I heard last | 07:47 |
| gfidente | geguileo no if -volume is on a different node | 07:48 |
| geguileo | gfidente: why not? | 07:48 |
| gfidente | geguileo we might end up in a situation where certain nodes (controlplane) hosting api/scheduler are at train level, and got the db migrated | 07:48 |
| gfidente | but -volume instances are all still up and running at the queens level | 07:48 |
| geguileo | gfidente: that should never happen | 07:49 |
| geguileo | gfidente: because FFU should stop all services before doing the FFU | 07:49 |
| gfidente | geguileo it is, but it's not doing it on all nodes | 07:49 |
| geguileo | regardless of where the services are running | 07:49 |
| geguileo | gfidente: then that's a bug | 07:49 |
| geguileo | XD | 07:49 |
| gfidente | for example it doesn't stop nova-computes when upgrading the controlplane | 07:49 |
| gfidente | ok but I think I get the point, all services must be upgraded to the train level for service to be restored | 07:50 |
| gfidente | a different question now | 07:50 |
| gfidente | and thanks a lot for helping clarifying this | 07:50 |
| geguileo | gfidente: all services OF THE SAME COMPONENT must be upgraded at the same time | 07:51 |
| gfidente | yeah of the same component of course, all cinder services in this case | 07:51 |
| geguileo | gfidente: and it is recommended that you don't run different components with a difference of more than 1 release | 07:51 |
| gfidente | so the different question is | 07:51 |
| geguileo | gfidente: btw, I just detected a bug in cinder's online-db-migrations | 07:52 |
| * geguileo facepalms | 07:52 | |
| gfidente | wait a sec the other question | 07:52 |
| geguileo | a bug that we've been carrying since Rocky | 07:52 |
| gfidente | can a nova guest be hard rebooted with cinder/api down, if it had any cinder volume attached? | 07:52 |
| * gfidente doesn't think so | 07:53 | |
| geguileo | gfidente: that's a Nova question... | 07:53 |
| gfidente | geguileo right | 07:53 |
| geguileo | gfidente: I don't know if they detach the volumes on hard reboot or if they let them as they were | 07:53 |
| gfidente | lyarwood ^^ ? :D | 07:54 |
| geguileo | gfidente: even if the detach them, depending on the release they could have the connection info on their DB and use it to attach again without contacting Cinder | 07:54 |
| lyarwood | geguileo / gfidente ; we detach on hard reboot | 07:56 |
| lyarwood | so reading back, no we can't hard reboot with c-api down | 07:57 |
| gfidente | lyarwood so to add up on your discussion yesterday, guests using cinder volumes can only be rebooted if cinder is available | 07:57 |
| gfidente | did I get it right? | 07:57 |
| gfidente | yeah that | 07:57 |
| gfidente | thank guys | 07:57 |
| geguileo | lyarwood: you call os-brick to detach, but do you also call Cinder to remove the export/map? | 07:57 |
| lyarwood | geguileo: yes we do now | 07:58 |
| lyarwood | geguileo: prior to queens we didn't | 07:58 |
| geguileo | lyarwood: good to know, that's probably the best thing to do :-) | 07:58 |
| lyarwood | geguileo: yeah, helps reset connection_info etc | 07:58 |
| *** e0ne has joined #openstack-cinder | 07:58 | |
| whoami-rajat | geguileo: Hi, so i really don't know why this is happening, whenever i run all tests (tox -epy27), and i set | 08:05 |
| whoami-rajat | 1) persistence -> db : all tests pass | 08:05 |
| whoami-rajat | 2) persistence -> memory : hit the following traceback http://paste.openstack.org/show/754790/ | 08:05 |
| whoami-rajat | the persistence is set in test_base and no changes are made to dbms tests. | 08:06 |
| geguileo | whoami-rajat: looks like some test is trying to access a volume that doesn't exist | 08:06 |
| whoami-rajat | geguileo: yep, but test_base is only running 2 tests with memory persistence and clearing all class attributes at teardown, don't understand why it affects the dbms tests. the same error i hit before when trying out memory pers. | 08:08 |
| geguileo | whoami-rajat: I can help you find the issue | 08:09 |
| geguileo | whoami-rajat: how can I get the code that is failing? | 08:09 |
| whoami-rajat | geguileo: should i update the PS, i refactored helper methods into a separate file. | 08:10 |
| geguileo | whoami-rajat: if you update the PS with what's failing I can have a look and help you figure out what's wrong (it could be a cinderlib bug) | 08:11 |
| whoami-rajat | geguileo: sure, will update in few secs | 08:11 |
| openstackgerrit | Rajat Dhasmana proposed openstack/cinderlib master: Fix: Snapshot creation with volume types https://review.opendev.org/670887 | 08:12 |
| whoami-rajat | geguileo: ^ done | 08:12 |
| geguileo | whoami-rajat: thanks, I'll have a look in 10 minutes, as soon as I finish fixing an online data migration bug I've just detected | 08:13 |
| whoami-rajat | geguileo: oh ok. sure, i think i've a doubt regarding that too, hah | 08:15 |
| *** tkajinam__ has quit IRC | 08:18 | |
| openstackgerrit | yenai proposed openstack/cinder master: Fix creation from raw image failed when qemu-img is not installed https://review.opendev.org/653694 | 08:21 |
| *** gfidente has quit IRC | 08:25 | |
| geguileo | whoami-rajat: interesting. It only fails in Python 3. Python 2 work fine... | 08:26 |
| whoami-rajat | geguileo: oh, these are different from the results of my local env but the test failure reason is same. | 08:28 |
| openstackgerrit | yenai proposed openstack/cinder master: Add host level for volume_get_all_by_host https://review.opendev.org/668345 | 08:28 |
| openstackgerrit | yenai proposed openstack/cinder master: Fix creation from raw image failed when qemu-img is not installed https://review.opendev.org/653694 | 08:28 |
| openstackgerrit | Bhaa Shakur proposed openstack/cinder master: Zadara VPSA: Move to API access key authentication https://review.opendev.org/670715 | 08:32 |
| openstackgerrit | Gorka Eguileor proposed openstack/cinder master: Fix online data migrations https://review.opendev.org/672466 | 08:32 |
| geguileo | whoami-rajat: I get the error on Python 2 as well XD Time to debug it | 08:38 |
| openstackgerrit | yenai proposed openstack/cinder master: Change volume_host to backup_host in backup rpcapi https://review.opendev.org/665878 | 08:38 |
| openstackgerrit | yenai proposed openstack/cinder master: Change volume_host to backup_host in backup rpcapi https://review.opendev.org/665878 | 08:38 |
| whoami-rajat | geguileo: hah, cause gate isn't running the tests that are failing locally :( | 08:40 |
| geguileo | whoami-rajat: whaaaat? :-O | 08:41 |
| whoami-rajat | geguileo: oh wait, my bad | 08:41 |
| whoami-rajat | geguileo: i did a typo while searching the test, sorry | 08:41 |
| geguileo | you gave me a scare XD | 08:41 |
| geguileo | jaja, np, I did the same thing on Monday | 08:42 |
| whoami-rajat | geguileo: oh, still gate is executing tests differently :/ | 08:45 |
| whoami-rajat | geguileo: i've one more query to ask if i'm not wasting much of your time :) | 08:46 |
| openstackgerrit | yenai proposed openstack/cinder master: Speed up starting cinder-backup https://review.opendev.org/657543 | 08:49 |
| openstackgerrit | yenai proposed openstack/cinder master: Speed up starting cinder-backup https://review.opendev.org/657543 | 08:49 |
| geguileo | whoami-rajat: sure | 08:51 |
| whoami-rajat | geguileo: thanks | 08:51 |
| *** altlogbot_0 has quit IRC | 08:51 | |
| *** irclogbot_3 has quit IRC | 08:51 | |
| *** altlogbot_1 has joined #openstack-cinder | 08:52 | |
| *** irclogbot_0 has joined #openstack-cinder | 08:52 | |
| whoami-rajat | geguileo: so while upgrading suppose STEIN-> TRAIN, we run online migrations of stein and then migration scripts of train, is there a way to run online data migration of TRAIN before the migration scripts? | 08:52 |
| openstackgerrit | yenai proposed openstack/cinder master: Correct the exception msg of ImageUnacceptable https://review.opendev.org/653649 | 08:52 |
| geguileo | whoami-rajat: no, the process to upgrade from stein to train is: stein online data migrations and then train db sync | 08:53 |
| geguileo | whoami-rajat: but we have a online-data-migration bug in Cinder Rocky, Stein, and Train | 08:54 |
| geguileo | whoami-rajat: https://bugs.launchpad.net/cinder/+bug/1837703 | 08:54 |
| openstack | Launchpad bug 1837703 in Cinder "Cinder Rocky, Stein, and Train are running wrong online-data-migrations" [Undecided,In progress] - Assigned to Gorka Eguileor (gorka) | 08:54 |
| geguileo | whoami-rajat: I just proposed a fix for it in master, and we'll have to backport it to Stein and Rocky | 08:54 |
| whoami-rajat | geguileo: oh yes, thanks for fixing this. i think we never faced the issue cause two migrations never overlapped | 08:56 |
| geguileo | whoami-rajat: the problem we also had is that in Queens we had a terrible data migration that required cinder-volume services to be running | 08:57 |
| geguileo | whoami-rajat: and since we didn't remove as we should in Rocky, Stein, and Train, we end up requiring this incorrect behavior when upgrading to Stein and Train as well | 08:58 |
| whoami-rajat | geguileo: oh, so i've implemented a migration to change volume_type_id(nullable=False) but it requires all volumes and snapshots to migrate to not None types, sadly both migrations are part of same patch and grenade is failing, so i need to implement the db migration in the next release, got it. | 08:58 |
| geguileo | whoami-rajat: sorry, I didn't get it... | 08:59 |
| openstackgerrit | yenai proposed openstack/cinder master: Correct the exception msg of ImageUnacceptable https://review.opendev.org/653649 | 09:00 |
| geguileo | whoami-rajat: snapshot don't have volume types | 09:00 |
| geguileo | whoami-rajat: and the "proper" way to implement the setting of this new type on the loading of Volume OVOs when they detect that they don't have a volume type | 09:01 |
| geguileo | whoami-rajat: not on a migration | 09:01 |
| whoami-rajat | geguileo: yeah, i agree the data migrations shouldn't be travelling to next release, can we also add an upgrade check in the patch you proposed. | 09:02 |
| geguileo | whoami-rajat: and then have an online-data-migration on that same patch that actually sets it | 09:02 |
| geguileo | whoami-rajat: upgrade check? | 09:02 |
| whoami-rajat | geguileo: yep https://github.com/openstack/cinder/blob/master/cinder/cmd/status.py | 09:03 |
| geguileo | whoami-rajat: I was wrong, the snapshots do have volume type references (bad memory) | 09:03 |
| whoami-rajat | geguileo: the snapshot table do have a volume_type_id field. | 09:03 |
| geguileo | whoami-rajat: yup, I had forgotten | 09:04 |
| whoami-rajat | geguileo: so to brief out my query, https://review.opendev.org/#/c/639180/23/cinder/db/sqlalchemy/migrate_repo/versions/133_make_vol_type_col_non_nullable.py | 09:05 |
| whoami-rajat | this particular migration is failing the grenade on gate cause the online migrations that needs to be performed are part of the same patch | 09:05 |
| whoami-rajat | this script should be implemented in next release IIUC | 09:05 |
| geguileo | whoami-rajat: that code should be a check in the status command | 09:07 |
| geguileo | whoami-rajat: not part of the DB | 09:07 |
| geguileo | migration | 09:07 |
| *** lpetrut has joined #openstack-cinder | 09:07 | |
| whoami-rajat | geguileo: oh ok. but in the next release right? | 09:08 |
| geguileo | whoami-rajat: I think so | 09:09 |
| *** awalende has joined #openstack-cinder | 09:10 | |
| whoami-rajat | geguileo: ok. thanks for the help! | 09:10 |
| geguileo | whoami-rajat: I'm debugging the cinderlib thing now, but I'll have a look later to see if I should add the check you suggested to the data migration patch | 09:14 |
| openstackgerrit | ye proposed openstack/cinder master: This fix let the delete err-info more precisely https://review.opendev.org/671996 | 09:15 |
| whoami-rajat | geguileo: since you mentioned that online migrations of that release shouldn't go into the next release, this sounds similar to the need of upgrade check. | 09:16 |
| *** davidsha has joined #openstack-cinder | 09:16 | |
| geguileo | whoami-rajat: since the migration should have been removed in Rocky we would need the check since rocky (if we end up needing it) | 09:17 |
| whoami-rajat | geguileo: thanks for looking into the issue, i'm getting to know cinderlib from docs and testing and will try to help if i find something. | 09:18 |
| whoami-rajat | geguileo: but the base framework is implemented in stein and seemingly shouldn't be backported. hmm. i think the upgrade check should be a separate patch | 09:19 |
| geguileo | whoami-rajat: oh, then I would just backport it to stein | 09:20 |
| whoami-rajat | geguileo: yep, the online migration removal could move to rocky and the upgrade check to stein, (since we allowed upgrade checks to be backported that were implemented in train) | 09:21 |
| whoami-rajat | geguileo: but i'm really not sure, jungleboyj and smcginnis would provide better input here than me :) | 09:22 |
| geguileo | whoami-rajat: looks like the problems are being introduced by some of your tests... | 09:23 |
| geguileo | whoami-rajat: 'i'll try to figure out why now | 09:23 |
| geguileo | whoami-rajat: we'll want to backport the check for sure :-) | 09:23 |
| *** Liang__ has quit IRC | 09:25 | |
| whoami-rajat | geguileo: yes, when set to memory persistence, my tests are causing the issues in other tests but i'm not really sure how :( | 09:26 |
| geguileo | whoami-rajat: I now have an easy way to reproduce it just running 1 of your tests and then a DB test ;-) | 09:27 |
| *** Kuirong has quit IRC | 09:37 | |
| *** boxiang has quit IRC | 09:42 | |
| whoami-rajat | geguileo: oh yes that failed, thanks for the heads up, now i can debug better :) | 09:42 |
| *** lpetrut has quit IRC | 09:47 | |
| *** Kuirong has joined #openstack-cinder | 09:49 | |
| *** m75abrams has quit IRC | 09:53 | |
| *** trident has quit IRC | 10:06 | |
| *** ganso has quit IRC | 10:06 | |
| *** ganso has joined #openstack-cinder | 10:07 | |
| *** trident has joined #openstack-cinder | 10:08 | |
| *** lpetrut has joined #openstack-cinder | 10:09 | |
| *** brinzhang has quit IRC | 10:10 | |
| *** boxiang has joined #openstack-cinder | 10:18 | |
| *** sahid has quit IRC | 10:24 | |
| *** abhishekk has quit IRC | 10:24 | |
| *** Kuirong has quit IRC | 10:35 | |
| *** dpawlik has quit IRC | 10:39 | |
| geguileo | whoami-rajat: I figured out what the problem is... It is not your fault :-( | 10:45 |
| geguileo | whoami-rajat: the issue is in cinderlib test code, and it appears in your patch because test_base goes before test_dbms | 10:46 |
| geguileo | whoami-rajat: working on a patch to fix this | 10:47 |
| *** carloss has joined #openstack-cinder | 10:49 | |
| whoami-rajat | geguileo: oh ok. i tried the debug run and got error during conversion of snapshot ovo to dict ( dict(snap._ovo)) , glad the problem is tracked :) | 10:49 |
| *** sahid has joined #openstack-cinder | 10:54 | |
| *** bhagyashris_ has quit IRC | 10:55 | |
| geguileo | whoami-rajat: renaming the test file from test_base to test_this_sucks should make the tests run (I haven't tried), but I'm looking to fix it right | 10:56 |
| whoami-rajat | geguileo: yup, that works. IIUC the catch here is to run the tests after dbms, i tried running memory tests before dbms and the same failure occurred. | 11:00 |
| geguileo | whoami-rajat: exactly that's when the problem happens | 11:01 |
| geguileo | whoami-rajat: it's because memory uses DB class which replaces some OVO methods | 11:01 |
| whoami-rajat | geguileo: oh, can we use cinderlib with multiple plugins simultaneously or only one plugin should be configured at a time? | 11:03 |
| geguileo | whoami-rajat: that's precisely the problem, cinderlib is only meant to run with 1 persistence plugin | 11:03 |
| geguileo | whoami-rajat: but in the tests we have to work around that limitation to test the different ones we have | 11:04 |
| openstackgerrit | Pawel Kaminski proposed openstack/os-brick master: connectors/nvme: Wait utill nvme device show up in kernel https://review.opendev.org/672031 | 11:05 |
| whoami-rajat | geguileo: ok. so we need to optimize the teardown code in memory tests to leave cinderlib in original state after the test run? | 11:07 |
| geguileo | whoami-rajat: yah, I have that code now | 11:07 |
| geguileo | whoami-rajat: but in one of the multiple tests I run I had an error, so I was trying to reproduce it to see what went wrong there... | 11:07 |
| *** dpawlik has joined #openstack-cinder | 11:08 | |
| whoami-rajat | geguileo: great :) | 11:08 |
| *** boxiang has quit IRC | 11:10 | |
| *** boxiang has joined #openstack-cinder | 11:11 | |
| *** udesale has quit IRC | 11:15 | |
| *** m75abrams has joined #openstack-cinder | 11:18 | |
| *** deiter has joined #openstack-cinder | 11:29 | |
| *** irclogbot_0 has quit IRC | 11:33 | |
| *** irclogbot_3 has joined #openstack-cinder | 11:34 | |
| openstackgerrit | hjy proposed openstack/cinder master: Add MacroSAN cinder driver https://review.opendev.org/612311 | 11:38 |
| geguileo | whoami-rajat: finally found the other issue... you'll have to change in your code the tearDown from "= {}" to ".clear()" and I'll have to change it as well in the test_memory as well | 11:51 |
| whoami-rajat | geguileo: wow :D | 11:51 |
| geguileo | whoami-rajat: yeah, since there can only be 1 plugin the volumes and other dicts are class attributes | 11:51 |
| geguileo | whoami-rajat: so we are not actually cleaning them when we set them to = {} and when you instantiate another memory persistance test they come back to live | 11:52 |
| whoami-rajat | geguileo: i'm still seeing the same failure :( | 11:59 |
| geguileo | whoami-rajat: no, no, that's additionally | 11:59 |
| geguileo | whoami-rajat: I'm writing the commit message for the proper fix | 11:59 |
| geguileo | whoami-rajat: but even with that fix you'll still need to make that .clear() modification | 12:00 |
| whoami-rajat | geguileo: oh, ok | 12:00 |
| *** jsquare has quit IRC | 12:00 | |
| *** deiter has quit IRC | 12:00 | |
| whoami-rajat | geguileo: thanks for fixing this, it was a good learning experience :) | 12:00 |
| *** jsquare has joined #openstack-cinder | 12:01 | |
| geguileo | whoami-rajat: for me this was hell XDXDXD | 12:01 |
| geguileo | nasty bugs | 12:01 |
| whoami-rajat | geguileo: lol, hope i don't find any bug while implementing the volume type code in cinderlib | 12:03 |
| whoami-rajat | s/find/encounter | 12:03 |
| *** zul has joined #openstack-cinder | 12:04 | |
| geguileo | whoami-rajat: well, you know were to find me if you do ;-) | 12:05 |
| openstackgerrit | Gorka Eguileor proposed openstack/cinderlib master: Fix cleanup of persistence tests https://review.opendev.org/672508 | 12:06 |
| whoami-rajat | geguileo: glad that you're VERY active ! | 12:06 |
| geguileo | whoami-rajat: ^ try rebasing your patch on top of that one and making the "clear()" change | 12:06 |
| jungleboyj | whoami-rajat: ++ | 12:06 |
| whoami-rajat | geguileo: Thanks!! | 12:07 |
| tosky | hi! When would be a good time to discuss about the porting of the remaining legacy jobs to native Zuul v3 jobs? | 12:08 |
| *** rosmaita has joined #openstack-cinder | 12:09 | |
| geguileo | tosky: probably in today's cinder IRC meeting | 12:09 |
| tosky | should I add a point to the agenda? | 12:09 |
| geguileo | tosky: yup just add it to https://etherpad.openstack.org/p/cinder-train-meetings | 12:10 |
| geguileo | tosky: thanks! | 12:11 |
| *** deiter has joined #openstack-cinder | 12:15 | |
| *** dviroel has joined #openstack-cinder | 12:20 | |
| openstackgerrit | Rajat Dhasmana proposed openstack/cinderlib master: Fix: Snapshot creation with volume types https://review.opendev.org/670887 | 12:22 |
| *** mriedem has joined #openstack-cinder | 12:35 | |
| *** coreycb has quit IRC | 12:45 | |
| *** coreycb has joined #openstack-cinder | 12:47 | |
| *** eharney has joined #openstack-cinder | 12:49 | |
| *** e0ne has quit IRC | 12:57 | |
| *** e0ne has joined #openstack-cinder | 12:57 | |
| *** Kuirong has joined #openstack-cinder | 13:01 | |
| *** jawad_axd has quit IRC | 13:01 | |
| *** awalende has quit IRC | 13:04 | |
| *** awalende has joined #openstack-cinder | 13:05 | |
| *** awalende has quit IRC | 13:09 | |
| *** deiter has quit IRC | 13:19 | |
| *** awalende has joined #openstack-cinder | 13:34 | |
| openstackgerrit | Bhaa Shakur proposed openstack/cinder master: Zadara VPSA: Move to API access key authentication https://review.opendev.org/670715 | 13:34 |
| openstackgerrit | Rafael Weingärtner proposed openstack/cinder master: Add mv to return volume type name and ID with volumes https://review.opendev.org/666886 | 13:37 |
| *** awalende has quit IRC | 13:38 | |
| *** ganso has quit IRC | 13:39 | |
| openstackgerrit | Rafael Weingärtner proposed openstack/cinder master: Add mv to return volume type name and ID with volumes https://review.opendev.org/666886 | 13:41 |
| *** enriquetaso has joined #openstack-cinder | 13:42 | |
| openstackgerrit | Rafael Weingärtner proposed openstack/cinder master: Add mv to return volume type name and ID with volumes https://review.opendev.org/666886 | 13:47 |
| *** ganso has joined #openstack-cinder | 14:01 | |
| *** Luzi has quit IRC | 14:14 | |
| *** lennyb has quit IRC | 14:18 | |
| *** raghavendrat has joined #openstack-cinder | 14:18 | |
| raghavendrat | hi hemna: and jungleboyj: this is regarding https://review.opendev.org/#/c/657585/; whenever you get some time, could you please review & share your feedback | 14:20 |
| jungleboyj | raghavendrat: Ok. Will do. | 14:21 |
| raghavendrat | thanks | 14:21 |
| *** dpawlik has quit IRC | 14:25 | |
| *** mriedem has quit IRC | 14:29 | |
| *** dpawlik has joined #openstack-cinder | 14:33 | |
| *** raghavendrat has quit IRC | 14:34 | |
| *** psachin has quit IRC | 14:41 | |
| *** raghavendrat has joined #openstack-cinder | 14:42 | |
| raghavendrat | thanks jungleboyj: for providing +2 | 14:43 |
| jungleboyj | raghavendrat: Welcome. :-) | 14:43 |
| *** jmlowe has quit IRC | 14:45 | |
| *** david-lyle is now known as dklyle | 14:45 | |
| *** dpawlik has quit IRC | 14:48 | |
| *** ag-47 has joined #openstack-cinder | 14:48 | |
| ag-47 | Hello everyone | 14:49 |
| jungleboyj | Howdy. | 14:50 |
| *** raghavendrat has quit IRC | 14:51 | |
| ag-47 | i'm having a probleme each time i launch cinder-volume-usage-audit, it creates a Aborted connection on the mysql logs, it is not critical but did anyone had the same problem | 14:52 |
| ag-47 | i'm on rocky with ubuntu 18.04 | 14:56 |
| *** sahid has quit IRC | 14:59 | |
| *** jmlowe has joined #openstack-cinder | 15:00 | |
| *** mriedem has joined #openstack-cinder | 15:09 | |
| jungleboyj | I have not heard of that problem before. | 15:10 |
| jungleboyj | Is there any indication of what is causing the abort? Are we maybe not closing the connection properly? | 15:10 |
| eharney | it looks likely that that tool just doesn't do any teardown of the db connections | 15:11 |
| jungleboyj | eharney: That was what I was thinking. | 15:11 |
| openstackgerrit | Eric Harney proposed openstack/cinder master: Prevent double-attachment race in attachment_reserve https://review.opendev.org/671370 | 15:20 |
| eharney | finally got unit tests in working order for this ^ i'd love some reviews since this is a pretty nasty bug | 15:21 |
| *** tejdeep has joined #openstack-cinder | 15:42 | |
| openstackgerrit | CY Chiang proposed openstack/cinder stable/stein: doc: Fix rbd driver marked support multiattach https://review.opendev.org/672552 | 15:50 |
| *** boxiang has quit IRC | 15:50 | |
| *** boxiang has joined #openstack-cinder | 15:50 | |
| *** Liang__ has joined #openstack-cinder | 15:54 | |
| openstackgerrit | Gorka Eguileor proposed openstack/cinder master: Fix online data migrations https://review.opendev.org/672466 | 15:56 |
| geguileo | whoami-rajat: ^ Added the status checks like we discussed | 15:57 |
| whoami-rajat | geguileo: we can't add checks for queens now :( | 15:59 |
| *** Liang__ is now known as LiangFang | 15:59 | |
| geguileo | whoami-rajat: these are checks that make sense here as well | 15:59 |
| geguileo | whoami-rajat: they are not for queens, they are for Stein | 15:59 |
| geguileo | an later | 15:59 |
| whoami-rajat | geguileo: oh wait, | 16:00 |
| whoami-rajat | geguileo: yep, realized it later, | 16:00 |
| geguileo | so what we are doing is detecting if you try to upgrade from queens or later to train without running the online data migrations from queens | 16:00 |
| whoami-rajat | geguileo: oh ok. it's a case for FFU | 16:01 |
| geguileo | whoami-rajat: also if someone tries to do the upgrade manually | 16:01 |
| geguileo | whoami-rajat: like directly jumping from queens to stein, or something like that | 16:01 |
| whoami-rajat | geguileo: are we planning to backport the removal of migrations to rocky? | 16:02 |
| geguileo | whoami-rajat: only to where the status tool exists | 16:02 |
| geguileo | whoami-rajat: if it didn't exist in Rocky, then we won't backport that part to Rocky | 16:03 |
| *** Kuirong has quit IRC | 16:03 | |
| whoami-rajat | geguileo: okay, wanted to know that only. Thanks for including the upgrade check :) | 16:04 |
| geguileo | whoami-rajat: it was a good suggestion :-) | 16:04 |
| openstackgerrit | Eric Harney proposed openstack/cinder-specs master: Volume Rekey spec https://review.opendev.org/664973 | 16:07 |
| *** mvkr_ has quit IRC | 16:10 | |
| *** pcaruana has quit IRC | 16:13 | |
| *** mvkr_ has joined #openstack-cinder | 16:16 | |
| *** m75abrams has quit IRC | 16:19 | |
| whoami-rajat | eharney: could you take a look at rosmaita comment here https://review.opendev.org/#/c/651480/ ? | 16:20 |
| *** ag-47 has quit IRC | 16:27 | |
| *** tejdeep has quit IRC | 16:53 | |
| *** davidsha has quit IRC | 17:01 | |
| *** mmethot_ is now known as mmethot | 17:02 | |
| jungleboyj | geguileo: You still around? | 17:04 |
| geguileo | jungleboyj: yeah, I'm around :- | 17:04 |
| geguileo | ) | 17:04 |
| jungleboyj | Hey. So, pots and I got a note from a customer seeing multipathing issues. | 17:05 |
| jungleboyj | The problem sounds familiar. | 17:05 |
| geguileo | jungleboyj: what was the problem? was it FC or iSCSI/ | 17:06 |
| geguileo | ? | 17:06 |
| jungleboyj | After migrating a VM from one node to another there are faulty paths left on the source. | 17:06 |
| jungleboyj | iSCSI | 17:06 |
| geguileo | jungleboyj: but the migration completed? | 17:06 |
| jungleboyj | https://www.irccloud.com/pastebin/hvSzZWv9/ | 17:06 |
| hemna | mep | 17:08 |
| geguileo | jungleboyj: it could be that Nova ignored some disconnect issues from OS-Brick | 17:08 |
| jungleboyj | This is back on Pike. | 17:08 |
| jungleboyj | I feel like we have made changes in that code path since. | 17:08 |
| geguileo | jungleboyj: they should see some tracebacks in the hk-m1-comp-node22 nova compute logs | 17:08 |
| geguileo | jungleboyj: I have made a few, but depends on the release they are running | 17:09 |
| jungleboyj | Ok. | 17:09 |
| geguileo | jungleboyj: I would first look at the compute logs and why it wasn't properly detached | 17:10 |
| geguileo | jungleboyj: and I think at some point we fixed something in Nova as well | 17:10 |
| geguileo | but I really don't recall what was it... | 17:10 |
| jungleboyj | Ok. I will get more info from the customer. | 17:11 |
| smcginnis | geguileo: All of those os-brick fixes got backport back to pike, right? | 17:11 |
| smcginnis | jungleboyj: Do they have the latest os-brick release for pike? ^ | 17:11 |
| *** LiangFang has quit IRC | 17:12 | |
| jungleboyj | That is a good question. I will check on that as well. | 17:13 |
| geguileo | smcginnis: : : I'm not 100% sure... But I think most of them did | 17:13 |
| geguileo | jungleboyj: it could also be related to this Nova bug I was thinking of https://review.opendev.org/#/c/551302/ | 17:14 |
| jungleboyj | That wasn't backported to pike. | 17:15 |
| geguileo | jungleboyj: It's also important to know what if the open iscsi version has support for disabling automatic scans | 17:15 |
| jungleboyj | We could do that though if it looks like the same issue. | 17:15 |
| geguileo | jungleboyj: if disabling automatic scans is not supported, then it could be that they don't see a backtrace in the logs, and the issue is a race condition (which the manual scans prevent) | 17:16 |
| jungleboyj | geguileo: Thanks for the pointers. I will send them the info and see what they say. | 17:17 |
| jungleboyj | smcginnis: What do we want to do about the MacroSAN driver? | 17:17 |
| jungleboyj | I see that they have been pushing patches up. But if their CI is down now ... | 17:18 |
| geguileo | jungleboyj: they can tell if their open-iscsi version supports manual scans by checking one of the nodes and seeing if it has the 'node.session.scan' parameter | 17:18 |
| * geguileo thinks the issue is probably that their iscsi version doesn't have that feature | 17:19 | |
| jungleboyj | It is SuSE Open Cloud 8 | 17:19 |
| geguileo | jungleboyj: I know SuSE has an open iscsi package that supports manual scans, because one of hemna's customers has been having trouble with it, but I don't know what your customer is running... | 17:22 |
| jungleboyj | Ok. I will have them check that as well. | 17:22 |
| hemna | SOC8 is fairly old now | 17:27 |
| hemna | I'm not sure what version of the open-iscsi utils that is | 17:27 |
| hemna | geguileo: what version of open-iscsi added the manual scan feature ? | 17:27 |
| geguileo | hemna: I don't remember, I added it a while, while back... | 17:28 |
| geguileo | Damn, the last patch is from March 2017! | 17:29 |
| hemna | soc8 has open-iscsi 2.0.876 | 17:30 |
| geguileo | hemna: it seems like it was in 2.0.875 | 17:30 |
| geguileo | hemna: then if they are on that version it should have manual scans and os-brick should be using them | 17:30 |
| hemna | jungleboyj: have to see what version you have | 17:30 |
| *** tejdeep has joined #openstack-cinder | 17:31 | |
| pots | geguileo: are there any specific guidelines for setting up multipath.conf for use with os-brick? | 17:31 |
| jungleboyj | Ok, just collected up those questions and are sending them. | 17:31 |
| geguileo | pots: not really | 17:31 |
| jungleboyj | hemna: geguileo Thanks for the help. | 17:31 |
| *** gnufied has quit IRC | 17:31 | |
| geguileo | pots: it should work with almost anything | 17:32 |
| hemna | pots: it should work with friendly names on or off if that's what you might be asking | 17:33 |
| pots | good guess :) | 17:34 |
| pots | thanks | 17:34 |
| *** awalende has joined #openstack-cinder | 17:34 | |
| hemna | geguileo: fwiw, one of my qa guys tested the iscsi iface0 workaround and got it to work | 17:35 |
| openstackgerrit | Jay Bryant proposed openstack/cinder stable/rocky: Revert "Declare multiattach support for HPE MSA" https://review.opendev.org/670086 | 17:35 |
| hemna | geguileo: you da man | 17:35 |
| hemna | jungleboyj: revert? | 17:35 |
| hemna | ah ok | 17:35 |
| jungleboyj | hemna: Yep. | 17:35 |
| * hemna read the patch | 17:35 | |
| jungleboyj | It was an update now that the commit message is accurate. | 17:36 |
| *** e0ne has quit IRC | 17:37 | |
| *** awalende has quit IRC | 17:39 | |
| *** pcaruana has joined #openstack-cinder | 17:41 | |
| *** tpsilva has joined #openstack-cinder | 17:41 | |
| *** henriqueof has joined #openstack-cinder | 17:47 | |
| *** ag-47 has joined #openstack-cinder | 17:49 | |
| ag-47 | response for message [17:10] : i launch the tool in debug mode so i had this as information related to the db connection | 17:53 |
| ag-47 | MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | 17:53 |
| ag-47 | Other than this i didn't see anything weird | 17:54 |
| *** KeithMnemonic has joined #openstack-cinder | 17:54 | |
| *** ag-47 has quit IRC | 18:05 | |
| *** ag-47 has joined #openstack-cinder | 18:13 | |
| *** gary_perkins_ has quit IRC | 18:19 | |
| *** gary_perkins has joined #openstack-cinder | 18:20 | |
| *** ag-47 has quit IRC | 18:21 | |
| *** jmlowe has quit IRC | 18:33 | |
| *** henriqueof has quit IRC | 18:40 | |
| *** tosky has quit IRC | 18:42 | |
| *** tesseract has quit IRC | 18:49 | |
| *** lpetrut has quit IRC | 18:50 | |
| *** henriqueof has joined #openstack-cinder | 19:13 | |
| *** jmlowe has joined #openstack-cinder | 19:22 | |
| *** e0ne has joined #openstack-cinder | 19:25 | |
| *** vishalmanchanda has quit IRC | 19:32 | |
| *** e0ne has quit IRC | 19:46 | |
| TheJulia | I've got a devstack change I've been trying to make and I'm seeing a weird failure with cinder's grenade scripting, has anyone ever seen http://logs.openstack.org/66/666766/2/check/neutron-grenade/699f904/logs/grenade.sh.txt.gz#_2019-07-24_12_56_19_758 before? | 19:48 |
| *** e0ne has joined #openstack-cinder | 19:57 | |
| *** henriqueof has quit IRC | 20:01 | |
| *** eharney has quit IRC | 20:05 | |
| *** e0ne has quit IRC | 20:19 | |
| jungleboyj | Looking. | 20:29 |
| jungleboyj | smcginnis: rosmaita whoami-rajat I have RSVPd for the PTG in Shanghai, I guessed we would get maybe 15 people based on responses. | 20:29 |
| rosmaita | jungleboyj: that seems like a pretty decent turnout | 20:30 |
| whoami-rajat | jungleboyj: great | 20:30 |
| jungleboyj | Half of what we would normally get. We had 15 or 30 in Denver. | 20:30 |
| *** deiter has joined #openstack-cinder | 20:35 | |
| BLZbubba | i'm seeing a bandwidth limit with cinder-iscsi, about 400MB/sec | 20:38 |
| BLZbubba | iirc, tgtd uses a single socket and doesn't support MC/S - am I the only one seeing this limit? | 20:39 |
| BLZbubba | i remember a couple years ago i set up 4 ip aliases and multipath to the same target, and it went like 4x faster | 20:40 |
| *** hemna has quit IRC | 20:50 | |
| *** hemna has joined #openstack-cinder | 20:50 | |
| *** eharney has joined #openstack-cinder | 20:56 | |
| *** e0ne has joined #openstack-cinder | 21:04 | |
| *** pcaruana has quit IRC | 21:05 | |
| *** henriqueof has joined #openstack-cinder | 21:07 | |
| *** imacdonn has quit IRC | 21:16 | |
| BLZbubba | duh, i should just use the nvmeof version | 21:25 |
| *** henriqueof has quit IRC | 21:27 | |
| *** e0ne has quit IRC | 21:30 | |
| jungleboyj | TheJulia: Generally that error message means that the Instance didn't finish booting. So, if it boot from volume something went wrong there. It started pinging. Not sure why SSH didn't come up. | 21:36 |
| *** e0ne has joined #openstack-cinder | 21:39 | |
| *** e0ne has quit IRC | 21:46 | |
| *** mchlumsky has quit IRC | 21:59 | |
| *** harlowja has quit IRC | 22:45 | |
| *** tkajinam has joined #openstack-cinder | 22:57 | |
| *** carloss has quit IRC | 23:15 | |
| *** rcernin has joined #openstack-cinder | 23:30 | |
| *** Kuirong has joined #openstack-cinder | 23:46 | |
| openstackgerrit | Merged openstack/cinder stable/rocky: Revert "Declare multiattach support for HPE MSA" https://review.opendev.org/670086 | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!