Tuesday, 2020-08-11

*** sapd1 has joined #openstack-cinder00:18
*** sapd1 has quit IRC00:39
*** dsariel has joined #openstack-cinder00:40
*** zhanglong has joined #openstack-cinder00:41
*** hemna has quit IRC01:01
*** hemna has joined #openstack-cinder01:01
*** _pewp_ has quit IRC01:01
*** _pewp_ has joined #openstack-cinder01:02
*** hemna has quit IRC01:17
*** n-saito has joined #openstack-cinder01:18
*** hemna has joined #openstack-cinder01:18
openstackgerrithaixin proposed openstack/cinder master: Fix volume copy logic  https://review.opendev.org/74548501:36
*** carloss has quit IRC01:56
*** zhanglong has quit IRC02:04
*** ganso has quit IRC02:04
*** zhanglong has joined #openstack-cinder02:06
*** ganso has joined #openstack-cinder02:09
*** sapd1 has joined #openstack-cinder02:09
*** zhanglong has quit IRC02:10
*** zhanglong has joined #openstack-cinder02:12
*** zhanglong has quit IRC02:21
*** hamalq_ has quit IRC02:21
*** zhanglong has joined #openstack-cinder02:22
*** rcernin has quit IRC02:37
*** zhanglong has quit IRC02:40
*** zhanglong has joined #openstack-cinder02:42
*** rcernin has joined #openstack-cinder02:48
*** zhanglong has quit IRC03:00
*** zhanglong has joined #openstack-cinder03:01
*** tobberydberg_ has quit IRC03:04
*** irclogbot_2 has quit IRC03:05
*** hemna has quit IRC03:05
*** hemna has joined #openstack-cinder03:06
*** irclogbot_2 has joined #openstack-cinder03:06
*** sapd1 has quit IRC03:08
*** tobberydberg has joined #openstack-cinder03:10
*** Liang__ has joined #openstack-cinder03:28
*** psachin has joined #openstack-cinder03:37
*** Liang__ has quit IRC03:40
*** Liang__ has joined #openstack-cinder03:41
*** yaawang_ has quit IRC03:48
*** yaawang_ has joined #openstack-cinder03:49
*** rcernin has quit IRC03:54
*** Liang__ has quit IRC04:07
*** Liang__ has joined #openstack-cinder04:09
*** sonivi has joined #openstack-cinder04:15
*** rcernin has joined #openstack-cinder04:21
*** sonivi has quit IRC04:27
*** zhanglong has quit IRC04:36
*** sonivi has joined #openstack-cinder04:37
*** sonivi has joined #openstack-cinder04:39
*** m75abrams has joined #openstack-cinder04:56
*** abdysn has joined #openstack-cinder05:21
*** udesale has joined #openstack-cinder05:29
*** vishakha has joined #openstack-cinder05:53
*** sonivi has quit IRC05:58
*** zhanglong has joined #openstack-cinder06:18
*** udesale has quit IRC06:51
*** udesale has joined #openstack-cinder06:52
*** Abdallahyas has joined #openstack-cinder06:56
*** abdysn has quit IRC06:59
*** Lucas_Gray has joined #openstack-cinder07:12
*** yedongcan has joined #openstack-cinder07:32
*** tosky has joined #openstack-cinder07:41
*** Wryhder has joined #openstack-cinder07:44
*** Lucas_Gray has quit IRC07:45
*** Wryhder is now known as Lucas_Gray07:45
openstackgerritSharath Kacham proposed openstack/cinder master:     [STORWIZE]-Code changes to handle groups with replication_enabled or     snapshot_enabled     Closes-Bug: #1890776  https://review.opendev.org/74529307:46
openstackbug 1890776 in Cinder "[STORWIZE]-Code changes to handle groups with replication_enabled or snapshot_enabled" [Undecided,In progress] https://launchpad.net/bugs/1890776 - Assigned to Sharath Kacham (sharathkacham)07:46
*** Abdallahyas has quit IRC07:54
*** abdysn has joined #openstack-cinder07:55
*** e0ne has joined #openstack-cinder08:02
*** brinzhang_ has quit IRC08:09
*** yedongcan has quit IRC08:10
*** sapd1 has joined #openstack-cinder08:17
*** priteau has joined #openstack-cinder08:26
*** isakgicu has joined #openstack-cinder08:39
isakgicuHello everyone, I'm wondering if I remove my current PV and VG, replace my current physical disks and recreate PV and VG, will my cinder work properly ?08:41
isakgicuor I should do some additional steps to make it work ?08:42
*** priteau has quit IRC08:42
*** psachin has quit IRC08:54
geguileothat should work if you restart cinder-volume08:58
*** priteau has joined #openstack-cinder08:58
*** rcernin has quit IRC09:11
*** pcaruana has quit IRC09:12
*** martinkennelly has joined #openstack-cinder09:23
*** pcaruana has joined #openstack-cinder09:24
*** psachin has joined #openstack-cinder09:31
*** rcernin has joined #openstack-cinder09:33
*** tkajinam has quit IRC09:36
*** sapd1 has quit IRC09:36
*** markvoelker has joined #openstack-cinder09:45
*** zhanglong has quit IRC09:45
*** Lucas_Gray has quit IRC09:49
*** udesale_ has joined #openstack-cinder09:49
*** markvoelker has quit IRC09:49
*** udesale has quit IRC09:52
*** zhanglong has joined #openstack-cinder09:54
*** rcernin has quit IRC09:56
*** n-saito has quit IRC10:04
*** priteau has quit IRC10:11
*** yaawang_ has quit IRC10:13
*** Liang__ has quit IRC10:31
CeeMaclooking at adding a new cinder backend, presumably the cinder-volume service will need restarting to pick that up, will that break attachments to instances running on that node?10:36
CeeMactrying to work out the correct operational procedure and if I need to schedule rolling host evacuations10:37
CeeMacon rocky using zadara cinder volume driver10:37
e0neCeeMac: volume service restart should be safe10:43
e0neCeeMac: you can set it to 'disable' state to make it more safer, so it won't afffect create/delete/attach operations during service restart10:44
*** vishakha has quit IRC10:50
*** markvoelker has joined #openstack-cinder10:52
*** markvoelker has quit IRC10:56
CeeMacthanks e0ne11:08
CeeMace0ne: i'm not familiar with the process to set it to 'disable' is that something done through openstack cli?11:10
CeeMacnvm found it :)11:15
e0neCeeMac: AFAIR, it should be done via cinder-manage but I need to re-check it11:18
*** rcernin has joined #openstack-cinder11:31
*** markvoelker has joined #openstack-cinder11:35
*** markvoelker has quit IRC11:40
*** Lucas_Gray has joined #openstack-cinder11:40
*** zhanglong has quit IRC11:58
*** martinkennelly has quit IRC11:58
*** martinkennelly has joined #openstack-cinder11:58
*** zhanglong has joined #openstack-cinder12:00
*** martinkennelly has quit IRC12:00
*** martinkennelly has joined #openstack-cinder12:00
*** carloss has joined #openstack-cinder12:11
CeeMacanother semi-related question.  In cinder.conf backends are defined in [], but also have a volume_backend_name12:23
CeeMacare these 1:1 ratio12:23
CeeMacand how do they relate?12:23
CeeMacI've seen in the 'host' of the volume service list host@backend12:24
CeeMacand in the UI its show as host@backend#backendName12:24
CeeMacis it possible to have host@backen#backendName1, host@backend#backendName2 etc?12:25
CeeMace.g if we have one storage controller with multiple pools, each pool would be configured as a backend, but it would be good to name them so they're referenced as the same storage controller12:26
CeeMacin the []12:26
CeeMacif that makes sense12:26
*** isakgicu has quit IRC12:28
whoami-rajatCeeMac, the value after # is the pool name, host@backend@pool12:30
*** pcaruana has quit IRC12:30
CeeMacwhoami-rajat: and can you define multiple pools for the same backend?12:31
CeeMacor is it 1:1?12:31
whoami-rajatCeeMac, one backend can have multiple pools if the driver allows it and reports it12:32
*** enriquetaso has joined #openstack-cinder12:32
CeeMacis the pool name pulled from the storage then or taken from the volume_backend_name value?12:33
CeeMacmy backend and pool both have the same name, so i've made it a bit confusing for myself12:33
whoami-rajatwhich backend are you using? and is it reporting the pool info to cinder scheduler ?12:37
CeeMacand I dont know, how would i tell?12:39
whoami-rajatcheck cinder-scheduler logs, and check what capabilities are being published there12:40
CeeMacok, i'll take a look, thanks12:40
*** zhanglong has quit IRC12:43
*** pcaruana has joined #openstack-cinder12:44
*** eharney has quit IRC12:54
*** Luzi has joined #openstack-cinder12:55
*** sonivi has joined #openstack-cinder12:56
*** e0ne_ has joined #openstack-cinder12:58
*** e0ne has quit IRC12:58
*** e0ne has joined #openstack-cinder12:59
*** Luzi has quit IRC13:00
*** e0ne_ has quit IRC13:03
*** vishalmanchanda has joined #openstack-cinder13:04
*** eharney has joined #openstack-cinder13:07
*** dave-mccowan has joined #openstack-cinder13:11
soniviHi, I mock whole class in testcase instead of specific function, is there any way to get/associated the functionality of assert_called_with() with our mock object13:12
*** pcaruana has quit IRC13:26
*** dave-mccowan has quit IRC13:29
*** eharney has quit IRC13:29
*** eharney has joined #openstack-cinder13:41
*** ociuhandu_ has joined #openstack-cinder13:46
*** ociuhandu has quit IRC13:47
*** sapd1 has joined #openstack-cinder13:53
*** sonivi has quit IRC14:00
*** pcaruana has joined #openstack-cinder14:00
*** smcginnis has quit IRC14:01
*** abdysn has quit IRC14:03
*** smcginnis has joined #openstack-cinder14:05
*** ChanServ sets mode: +o smcginnis14:05
*** rcernin has quit IRC14:08
*** m75abrams has quit IRC14:11
*** rcernin has joined #openstack-cinder14:18
*** rcernin has quit IRC14:24
*** rcernin has joined #openstack-cinder14:27
*** rcernin has quit IRC14:31
CeeMacwhoami-rajat: I'm not seeing anything of particular interest inthe cinder-scheduler logs sadly14:32
whoami-rajatCeeMac, do you see any msg like http://paste.openstack.org/show/796739/ ? search for 'Received volume service update from '14:34
CeeMacno, you need debug enabled for that to show?14:35
CeeMacit mostly consists of 'Deleted 0 expired messages'14:35
whoami-rajatyeah, set debug=True in cinder.conf [DEFAULT] section14:36
CeeMacdoes that require a restart?14:36
CeeMaci can never remember14:36
whoami-rajatyep, config changes require restart of services14:40
CeeMacwhoami-rajat: http://paste.openstack.org/show/796740/14:42
smcginnisLogging level can be changed at runtime. Either by reloading the config (SIGHUP I think?) or via API call.14:47
* CeeMac goes to look up some words14:53
whoami-rajatCeeMac, i think the driver needs to report the pool stats, something like https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/dell_emc/unity/adapter.py#L619-L65814:53
CeeMacthanks whoami-rajat I'll get back to the vendor for advise14:54
CeeMacbasically i need to add a second pool and not break the existing one14:54
*** vishakha has joined #openstack-cinder14:55
CeeMacfor the zadara cinder driver you explicitly define the pool id in the backend config so I'm guessing it won't be doing any dynamic reporting14:55
*** sapd1 has quit IRC14:55
*** dsariel has quit IRC14:57
CeeMacis it possible to make a particular backend only available to certain projects?15:04
rosmaitaCeeMac: indirectly, you create a private volume-type and then make it available to those projects15:06
*** brtknr has left #openstack-cinder15:09
CeeMacah, i saw something about that, just reading through the docs again15:09
*** dsariel has joined #openstack-cinder15:09
CeeMacso, set a particular backend-name on the backends to be 'private', create a new 'private' vol-type and link this to the appropriate backend-name?15:10
CeeMacgreat, thanks15:11
rosmaitathen use this API to give the projects access to that volume-type: https://docs.openstack.org/api-ref/block-storage/v3/#volume-type-access-types-action-types-os-volume-type-access15:12
CeeMacah, thats not in the openstack cli yet then?15:13
rosmaitanot sure15:13
CeeMacok, i'll need to brush my api-fu then15:13
rosmaitait's in the cinderclient15:14
rosmaitacinder help | grep ' type-'15:15
CeeMacopenstack volume type set --project <project> <volume-type> might work?15:16
rosmaitawell, what do you know15:17
CeeMaci'll take that as a yes? :D15:17
CeeMaci'll give it a go, I'm drafting up some process in my test environment15:18
CeeMacthanks for the help everyone :D15:18
rosmaitakeep us posted15:18
CeeMacwill do15:20
CeeMacI'll need to catch up with storage guy in the morning15:20
*** rcernin has joined #openstack-cinder15:39
*** psachin has quit IRC15:41
*** rcernin has quit IRC15:44
*** sapd1 has joined #openstack-cinder16:06
*** hamalq has joined #openstack-cinder16:10
*** hamalq_ has joined #openstack-cinder16:11
rosmaitasmcginnis: have you had any time to look at your devstack patch https://review.opendev.org/#/c/743566/ ?16:15
*** hamalq has quit IRC16:16
smcginnis"msg": "[Errno 2] No such file or directory: 'tox': 'tox'",16:17
rosmaitayeah, i don't get it16:18
smcginnisThat's later, Looks like the actual failure is a MySQL access problem.16:18
smcginnisERROR 1698 (28000): Access denied for user 'root'@'localhost'16:18
rosmaitaoh, i missed that16:18
smcginnisMe too. I usually start at the bottom and work my way up, but sometimes that's not the root error.16:19
rosmaitado you think the cinder bindep is somehow messing up the "regular" mysql install?16:19
rosmaitaas far as i can tell, your patch is the only one hitting this post failure16:19
smcginnisIt would appear so.16:19
smcginnisWhich makes me think we probably have something in our local bindep.txt that is interfering.16:20
smcginnisMaybe a different version of MySQL?16:20
clarkbsmcginnis: rosmaita tools/test-setup.sh is the other place to look iirc16:21
smcginnisclarkb: The patch in question is a change to devstack that just adds the installation of bindep packages from the repo root.16:22
clarkboh neat thats happening16:23
clarkbin that case likely bindep contents then ya16:23
smcginnisBut we do have an issue in the cinder tools/test_setup.sh for newer mysql versions, so that's a good reminder.16:23
smcginnisrosmaita: Not sure if it is related yet, but we will need https://review.opendev.org/#/c/737364/16:23
rosmaitaman, it's like the mail, bugs keep coming, and coming, and coming ...16:24
smcginnisUnless you're the USPS. :/16:24
smcginnisBoth jobs look like they are installing mysql-server-5.7, so doesn't look like that is the issue in this case.16:25
*** Lucas_Gray has quit IRC16:29
*** hamalq_ has quit IRC16:29
*** hamalq has joined #openstack-cinder16:30
*** Lucas_Gray has joined #openstack-cinder16:31
rosmaitai wonder why it's only causing a problem on that job16:33
smcginnisHmm, the only additional package the patch is getting installed is lsscsi. I don't see a difference that would explain why it's failing with that.16:33
smcginnisNot even like it is pulling in other dependencies.16:33
*** udesale_ has quit IRC16:34
lseki> Logging level can be changed at runtime. Either by reloading the config (SIGHUP I think?) or via API call16:37
lsekitested here and both `systemctl reload devstack@c-vol` and `kill -HUP <c-vol PID>` works16:37
smcginnisThanks lseki!16:37
lseki^ CeeMac whoami-rajat smcginnis16:37
lsekinp :-)16:38
lsekis/c-vol/c-sch if reloading c-sch configs16:38
*** zzzeek has quit IRC16:47
openstackgerritMerged openstack/cinder master: Make test-setup.sh compatible with mysql8  https://review.opendev.org/73736416:51
smcginnisrosmaita: This is looking like an issue: https://opendev.org/openstack/devstack/src/branch/master/lib/databases/mysql#L92-L9516:54
smcginnisSo maybe something has already set the password?16:55
rosmaitai don't know, the first time 'mysqladmin' appears in the job-output.txt is whe it tries to set the password and fails16:56
*** tosky has quit IRC16:59
smcginnisIt preseeds the configuration in https://opendev.org/openstack/devstack/src/branch/master/lib/databases/mysql#L151-L15717:00
smcginnisAfter that, on failing setup: mysql-server is already the newest version (5.7.31-0ubuntu0.18.04.1).17:00
smcginnisPassing setup: The following NEW packages will be installed:  libevent-core-2.1-6 libnuma1 mysql-client-5.7 mysql-client-core-5.7   mysql-server mysql-server-5.7 mysql-server-core-5.717:00
smcginnisBut that still doesn't explain why the only difference in the cinder bindep step is rather than nothing, it solely installs lsscsi.17:01
*** sfernand has joined #openstack-cinder17:02
*** dsariel has quit IRC17:02
rosmaitalet's put up a DNM changing the cinder bindep.txt to have only lsscsi in it, make your patch depend on it, and go to lunch17:03
smcginnisI do see where mysql-server is being installed earlier. It looks like maybe devstack collects all plugin bindep entries and does one install. That would probably make sense.17:04
smcginnisSo maybe since the config hasn't been pre-seeded, even though it is not connecting and trying to do any DB yet, it's too late without that config preconfigured for it.17:04
smcginnisI guess that's a good idea to sanity check that assumption. I will put up a DNM that just drops mysql and see how that goes.17:05
rosmaitaok, that's a better idea (dropping mysql only17:05
rosmaitathanks, i'll check back later -- i have a dog here who's letting me know that noon was an hour ago17:05
openstackgerritSean McGinnis proposed openstack/cinder master: DNM: Test mysql install order issue with bindep patches  https://review.opendev.org/74571117:06
*** sapd1 has quit IRC17:07
*** e0ne has quit IRC17:07
*** Lucas_Gray has quit IRC17:26
CeeMaclseki: thanks! I've added that to my notes :D17:29
lsekinp :-D17:30
*** hamalq has quit IRC17:31
CeeMacin the interests of understanding things better so i can not break things.  When a volume is created it retains the host/backend information in the os-vol-host-attr:host as far as I can see.  Presumably, if one were to change the backend configuration name, or the backend-name, this would have a detrimental impact to the volume and any instances it is attached to?17:35
CeeMacjust trying to plan in terms of backend expansion, growth, migration etc17:36
CeeMacto maintain consist naming and to move to volume groups etc17:38
*** e0ne has joined #openstack-cinder18:03
openstackgerritkatari manoj kumar proposed openstack/cinder master: [storwize]: optimize create_flashcopy_to_consistgrp  https://review.opendev.org/74571918:06
*** e0ne has quit IRC18:15
openstackgerritGirish Chilukuri proposed openstack/cinder master: [Storwize]:check_vdisk_fc_mappings method is not  deleting the clone fc maps.  https://review.opendev.org/74466418:21
*** andrebeltrami has joined #openstack-cinder18:30
*** e0ne has joined #openstack-cinder18:34
*** e0ne has quit IRC18:35
*** e0ne has joined #openstack-cinder18:49
*** e0ne has quit IRC18:57
*** dsariel has joined #openstack-cinder19:02
*** rcernin has joined #openstack-cinder19:41
*** vishakha has quit IRC19:44
*** rcernin has quit IRC19:45
lsekiCeeMac: hmm are you intending to rename the back end stanza names?20:10
lsekior the `volume_backend_name`s perhaps?20:21
*** dsariel has quit IRC20:27
CeeMaclseki: either or both I guess.  I wasn't fully aware of the configuration when I set the environment up so its not really implemented in a scalable way.  The volume_backend_name is set to match a specific storage target as is the configuration name in the [backend], where really, as I see things now, this should be more generic so I can have multiple backends with the same volume_backend_name for scalability20:33
CeeMacand let the scheduler place the volumes on the best backend20:33
CeeMacthen I can swap backends in and out easier if required20:33
CeeMacalso me and storage guy both get ocd when it comes to matching names across the stack, so moving from backend1 to a new backend2, then deleting backend1 would freak us both out lol20:34
CeeMacjust trying to plan the best migration path20:35
lseki> moving from backend1 to a new backend2, then deleting backend1 would freak us both out20:51
lsekiI see. I'm not sure if there's a way to avoid this, though...20:51
lsekiI believe you'll need to create volume types that match to the backends by some more generic specs (and not their volume_backend_name)20:51
lsekibut for the pre-existing volumes, I guess you'll have to retype them one by one20:51
-openstackstatus- NOTICE: The openstackgerrit IRC bot (gerritbot) will be offline for a short period while we redeploy it on a new server20:51
*** openstackgerrit has quit IRC20:52
CeeMacYes, that's pretty much the conclusion I had come to also20:53
CeeMacI'll probably end up deploying new backends with generic name and type, migrate / retype all the volumes from the original backend then delete and recreate them in the new way20:55
CeeMacI guess there is no "rebalance" api to automigrate volumes within the new backend types to even out the spread?20:56
lsekinot that I'm aware of20:59
lsekiI think other folks already left today, you might ask them for advices again tomorrow21:01
CeeMacSure, thanks21:02
*** vishalmanchanda has quit IRC21:14
*** martinkennelly has quit IRC21:30
*** Reepicheep has quit IRC21:33
*** zzzeek has joined #openstack-cinder21:38
smcginnisCeeMac: Yeah, no automatic thing for that. You will need to retype to where you want things to go.21:58
*** geguileo has quit IRC21:59
smcginnisIf you define a volume type that isn't tied to the backend name, you could probably do a simple bash script that retypes from the current one to the new type. Then the scheduler can handle distributing where the volume goes.21:59
smcginnisNot sure if that fits for your use case or not, but an option to consider.21:59
*** geguileo has joined #openstack-cinder22:00
*** ociuhandu_ has quit IRC22:05
*** gregwork has joined #openstack-cinder22:45
*** rcernin has joined #openstack-cinder22:45
*** tkajinam has joined #openstack-cinder22:54
*** logan- has quit IRC23:05
*** Reepicheep has joined #openstack-cinder23:18
*** carloss has quit IRC23:21

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!