21:00:27 <oneswig> #startmeeting scientific-sig
21:00:28 <openstack> Meeting started Tue Apr 14 21:00:27 2020 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:31 <openstack> The meeting name has been set to 'scientific_sig'
21:00:39 <oneswig> Hi
21:01:50 <martial> Hello friends
21:01:56 <oneswig> Hey martial
21:02:00 <oneswig> #chair martial
21:02:02 <openstack> Current chairs: martial oneswig
21:02:04 <martial> Hey Stig :)
21:02:06 <oneswig> how are you?
21:02:43 <martial> doing well, enjoying a ton of work so the usual :)
21:04:08 <oneswig> Strangely work for some is much, much higher!
21:04:23 <oneswig> Keeping me busy, for sure.
21:04:34 <martial> oh yeah
21:04:44 <martial> it is crazy here too
21:06:32 <oneswig> awfully quiet today...
21:06:33 <martial> things going well otherwise? Seems like it is going to be another slow SIG meeting
21:06:46 <rbudden> hello :)
21:06:55 <oneswig> Well yes, I think so.
21:06:58 <oneswig> hey Bob
21:07:41 <rbudden> hey Stig, how’s it goin?
21:07:59 <oneswig> Bored :-)
21:09:42 <oneswig> Any news on that issue with Cinder, Nova and backends in various AZs?
21:10:25 <rbudden> yes, we’ve got the multi AZ pieces all working for the most part
21:10:53 <rbudden> a few weird nuances ATM, but nothing that appears to be a show stopper for our future setup
21:12:12 <oneswig> Luxury :-) I've been having fun with giving some cranky old hardware a second life.
21:13:25 <rbudden> the strangest is that since Nova through CLI/Horizon has no way of passing a Cinder volume type, it appears that Cinder selects the right backend based on AZ, but assigns it ‘__DEFAULT__’ as the type instead of the choice it made
21:14:22 <oneswig> I've seen that volume type before... pretty annoying and that's without AZs in the mix
21:15:16 <rbudden> it’s been interesting digging into some of this, i’m not sure if many others have done multi AZ setups w/Nova, Cinder, and Neutron
21:15:41 <rbudden> it’s almost like there needs to be a default volume type per AZ
21:16:16 <smcginnis> The workaround in horizon is to create the volume first, then use that to create a new instance. Unfortunately.
21:16:18 <rbudden> or have Nova be smart enough to pass the volume type
21:16:34 <oneswig> Hi smcginnis, thanks for that :-)
21:16:43 <rbudden> yes, that does work
21:17:05 <rbudden> sadly, it requires our users to actually read the documentation to know to peform that step ;)
21:17:31 <rbudden> we’ve used the same workaround on the CLI for the moment as well
21:17:56 <oneswig> The inevitable follow-up question is why?  Seems like the sort of usability issue the UC ought to have been chipping away at.
21:18:25 <smcginnis> If you have default_availability_zone set in cinder.conf, that should prevent "__DEFAULT__" from being used. But that only helps for that one type.
21:18:53 <oneswig> Is it that the APIs don't support this level of specification, and can't be extended to add it?
21:19:02 <rbudden> if you have a default set, then you can never get a nova boot to pick a cinder volume type in the non-default AZ
21:19:25 <rbudden> cinder-api effectively applies the default volume type before it sends the request to the scheduler
21:19:34 <smcginnis> oneswig: In the past, we did push for nova being able to send that to cinder in the volume creation request, but the team had been acitively trying to get away from orchestrating non-Nova API workflows. So it was shot down multiple times.
21:20:46 <smcginnis> One suggestion I gave in the past was to add the smarts to horizon, so internally it could orchestrate creating the volume, then using it for the instance. Not sure if there is a reason why that couldn't be done.
21:21:11 <oneswig> Thanks for the helpful context
21:21:19 <rbudden> FWIW this problem also exists with just Cinder CLI outside of Nova
21:21:34 <smcginnis> No great answers, but at least some background.
21:21:44 <rbudden> yep, appreciate the info
21:21:51 <smcginnis> rbudden: Not sure what you mean. You can specify volume type from the Cinder CLI.
21:22:33 <rbudden> right, i’m saying if you want to create a new volume based on AZ and not volume type, you get the same experience as with the Nova boot (i.e. Nova passing ‘None’ as the volume type)
21:23:40 <rbudden> so if you do a ‘cinder create —availability-zone $az …’ without specifying a type, you traverse the same code path Nova does where the Cinder scheduler does pick the right volume type, and creates the volume, but ends up with type being ‘__DEFAULT__’ regardless of which backend it landed on
21:23:47 <smcginnis> Yeah, everything is driven off of volume types to determine where to create volumes. So if the type isn't given, it will always use the default volume type and place it wherever that goes.
21:24:32 <rbudden> yep, that’s what it looks like. it does appear to have enough logic that if no default is set, it works, with the above issue
21:24:42 <rbudden> operationally it works, so that makes me happy
21:25:15 <oneswig> rbudden: so you have users following the two-stage workflow?
21:25:29 <rbudden> no, not at the moment
21:25:32 <rbudden> we’re trying to avoid that
21:25:41 <rbudden> this workaround is in our TDS
21:26:04 <rbudden> our current production cloud is single AZ and single Cinder backend, so it doens’t have these
21:26:06 <rbudden> issues
21:26:38 <rbudden> when we merge our two clouds we will end up with at least one AZ per buliding, maybe more, and at least one Cinder volume backend per building
21:29:29 <rbudden> the main idea is to prevent the need (and peformance loss) of doing Cinder cross-az-attach
21:30:01 <smcginnis> Looking at the cinder code, I think you are right that there is a bug in how --availability-zone gets passed in.
21:33:37 <oneswig> Interesting to know.
21:38:23 <rbudden> so yeah, that’s where i’m at with things ATM. been fun digging through code though :)
21:39:18 <oneswig> rbudden: It's always fun, until you have to do it because it's preventing you from doing what you really wanted to do...
21:40:22 <rbudden> haha, it’s still fun then as long as you eventually find a solution in a timely fashion ;)
21:40:56 <rbudden> luckily right now the entire point of our TDS is to flesh out these issues now
21:41:50 <oneswig> Done by the book!
21:42:21 <rbudden> we’re still hacking through Monasca/Ceilosca/CloudKitty
21:42:30 <rbudden> @oneswig we may pick your brain some more if you don’t mind
21:42:36 <oneswig> Any luck with tying together Monasca and CloudKitty?  I think Pierre got some useful pointers
21:42:45 <rbudden> yes, we’re getting closer
21:43:36 <rbudden> still a disconect or two, but I’d have to defer to Jonathan on much of that work
21:43:43 <oneswig> rbudden: brain-picking more than welcome, what's left of it...
21:43:54 <rbudden> I need to catch up to him in the Telemetry department
21:46:16 <rbudden> we’ve appreaciate the help thus far!
21:47:19 <oneswig> You're welcome - in return I'd be interested hearing if you finesse things with the AZs in due course :-)
21:47:53 <rbudden> absolutely
21:51:57 <martial> Bob, be ready to present :)
21:52:26 <rbudden> already have some plans to present when our new setup is opperational
21:52:42 <oneswig> Present here or somewhere fancier? :-)
21:52:58 <rbudden> was hoping Germany Summit, but who knows these days ;)
21:53:33 <oneswig> The SIG beers in Berlin last time - have to do that again...
21:54:05 <rbudden> maybe once this COVID-19 is al over i think i need to replan that long lost trip to visit you Stig
21:54:25 <rbudden> i’ve never been to Germany, but it’s been on my bucket list
21:54:31 <rbudden> *Crosses fingers*
21:55:09 <oneswig> Any time, rbudden, except for now alas
21:56:06 <martial> Kind of hoping Berlin will be a go too
21:56:54 <oneswig> Likewise!
21:58:07 <oneswig> OK, nearly at the hour. AOB?
21:58:50 <oneswig> We are planning for our first CentOS 8-based deployment, maybe kicking off next week all going to plan.
21:59:16 <rbudden> very nice. i’ll be interested in how this goes!
21:59:42 <rbudden> we’ll be toying with CentOS 8 at some point in our TDS as well, but not on the immediate horizon
21:59:46 <oneswig> Me too. It's been well tested upstream so I'm hoping relatively smoothly.
22:00:02 <oneswig> Ceph Octopus too
22:00:15 <oneswig> On that happy note, final comments?
22:00:37 <oneswig> #endmeeting