16:00:07 #startmeeting cinder 16:00:08 Meeting started Wed Dec 10 16:00:07 2014 UTC and is due to finish in 60 minutes. The chair is thingee. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:12 The meeting name has been set to 'cinder' 16:00:15 hello all 16:00:20 hi! 16:00:27 hi 16:00:30 hi 16:00:31 hi 16:00:37 hello 16:00:44 today's agenda is pretty full 16:00:45 hi 16:00:46 #link https://wiki.openstack.org/wiki/CinderMeetings 16:00:48 hi 16:00:49 hi 16:01:06 hi 16:01:15 Quick note to reviewers for Cinder, drivers get priority this milestone 16:01:15 \o 16:01:19 #link https://etherpad.openstack.org/p/cinder-kilo-priorities 16:01:22 o/ 16:01:27 Hi 16:01:32 it's important for core to help out and help see a driver through or two 16:01:39 hi 16:01:40 hi 16:01:49 I would like to see these merged this week 16:01:59 even if you're not core, please help 16:02:02 hello 16:02:06 thingee: What's your take on Tempest tests? 16:02:11 o/ 16:02:27 Personally I'm not inclined to review if they don't pass the full Volume tests and show it 16:02:27 jgriffith: they're great 16:02:33 jgriffith: ah heh 16:02:33 thingee: ok 16:02:36 thingee: thanks 16:02:49 jgriffith: sorry not trying to be smart. I wasn't sure what you were talking about 16:02:56 thingee: :) 16:02:56 is it OK to add a driver to the K-1 drivers list? 16:03:06 https://etherpad.openstack.org/p/cinder-kilo-priorities ^^ 16:03:24 jgriffith: they should definitely pass the full volume tests in tempest. that was a requirement as mentioned in the email 16:03:29 o/ 16:03:36 jgriffith: so you'd prefer that drivers faked things because dependencies don't provide all required functions yet? 16:03:38 anyone can edit the etherpad if a driver actually isn't ready to be reviewed 16:03:41 Unit tests for the Dell driver are going in today. 16:03:54 Swanson: thanks 16:04:04 ok lets start with the first item 16:04:04 thingee, what if the driver is ready to be reviewed ? 16:04:10 flip214: No 16:04:13 thingee, but not on the list 16:04:19 jordanP: if it's not on the list, add it 16:04:24 right 16:04:35 #topic Abstract interface for volume drivers 16:04:39 mkoderer: here? 16:05:03 #link https://review.openstack.org/#/c/127455/ 16:05:11 so that's the review that's sitting ^ 16:05:36 I want to know what the rest of the folks think of drivers having to adopt this 16:05:39 #link https://review.openstack.org/#/c/138661/2/cinder/volume/drivers/rbd.py 16:05:41 +1 16:05:45 that's an example of the rbd driver with it 16:05:52 discuss 16:05:56 o/ 16:06:20 morning 16:06:22 thingee: what would the timeframe be? 16:06:26 i was curious what the impact will be for drivers that inherit from the iscsi/san driver base class 16:06:36 eikke: to adopt it? none yet. 16:06:38 I think that's a nice feature-detecting implementation 16:06:51 o/ 16:06:55 patrickeast: explain 16:06:59 flip214: +1 16:07:02 Timeline would be best-effort for Kilo, I think 16:07:10 should all new drivers in K use it? 16:07:12 well the rbd example shows one that uses just the volume driver directly 16:07:45 patrickeast: I think it would be the same? 16:07:57 it's the core features that alll drivers need to have today 16:08:04 patrickeast: your driver should inherit the features from the parent, and then you would need to add any other extra features you need 16:08:18 ok got it 16:08:43 so IMO, adopting this shouldn't be difficult. I think just like third party CI we need good communication and reasonable time 16:09:04 seems like an easy enough change 16:09:28 if we agree with this change, we can communicate to maintainers by the end of kilo 16:09:37 that's two milestones... 16:09:41 anyone disagree? 16:09:46 Just FYI I gave it a -1 because it introduces merge conflicts for items in progress 16:09:47 +1 16:09:52 +1 sounds easy enough 16:09:57 thingee, +1 16:10:21 does anyone disagree with this change and approach? 16:10:23 +1 (agree) 16:10:24 +1 16:10:30 no 16:10:38 jgriffith: merge conflicts happen - if something beats you to the merge then you have to fix up IMO - adding -1 seems a bit unfair 16:10:50 DuncanT: not unfair to me 16:10:53 * jungleboyj came in late. We have to go back and look. 16:10:53 wouldnt it make sense to only merge this after K-1 closed? 16:10:54 ok, I will handle communication once this is merged in k-1. 16:10:58 DuncanT: feel free to ignore my -1 16:11:03 DuncanT: i was thinking the same thing, merge conflicts can get resolved at merge time 16:11:19 eikke: sure 16:11:41 Ok I just wanted to understand if the rest of the team agreed with this approach and should we continue to move forward 16:11:43 thanks 16:11:45 eikke: +1 16:11:59 ok so we'll merge after k-1 16:12:02 * jgriffith agrees with the apparoach for sure 16:12:06 #topic CI for zone drivers? 16:12:13 jnrao: here? 16:12:16 yes 16:12:28 DuncanT: I know you had thoughts on this from the summit too 16:12:43 i would like to know, is there any framework in place for us to start CI for zone drivers? 16:13:01 jnrao, what specifically do you mean by zone drivers ? 16:13:19 thingee: My only thoughts were 'we should totally do this' 16:13:23 the fibre channel zone manager drivers (cisco and brocade) ? 16:13:32 Whjyes 16:13:33 yes 16:13:46 fibre channel switches/networks have something called zones for those that don't know 16:14:00 jnrao, the problem now is that the CI system can't do FC 16:14:01 it's a FC-specific security construct 16:14:08 because libvirt doesn't support FC NPIV 16:14:17 ic 16:14:22 it's possible 16:14:25 but it's a major pain 16:14:36 #idea should cinder have third-party ci for FC zone drivers? 16:14:41 we have to do a PCI passthrough for our HBA's in our local CI in order to test our FC volume drivers 16:14:46 hemna_, we have CI doing FC 16:14:46 the problem is that nothing in openstack manages FC networks, so the job falls on cinder 16:14:53 asselin, yes I know 16:15:26 sorrison, I like the idea of having CI for the FCZM and it's drivers and the lookup services. 16:15:31 *sigh* 16:15:33 so 16:15:40 but it's on brocade and cisco to do it. 16:15:47 hemna_: +1 16:16:01 hemna_: makes sense to me 16:16:03 hemna_, agree +1 16:16:07 hemna_: +1, although I'd argue it should also fall on Vendors that consume it to a point 16:16:13 sorrison, not only do they need to have tests for the drivers, but they will also need an FC array to do the complete tests. 16:16:21 I think that all the people with a stake in FC should definitely be looking at CI - not as a requirement yet, but as something that would make things better for sure 16:16:26 I was looking out for core team to provide some guidelines for FC vendors like us do the CI. 16:16:26 jgriffith, yah, we are doing some tests with it, 16:16:27 I agree with jgriffith -- it seems unlikely that cisco and brocade will start caring about cinder 16:16:37 hemna_: in other words, if you want an FC driver in OpenStack you're part of it 16:16:53 jgriffith, yup. we do testing with the FCZM and one of the drivers now. 16:17:02 hemna_: and like anything, you're going to need CI that demonstrates you work on both 16:17:02 but we don't have both cisco and brocade switches 16:17:07 so we are only testing with 1 currently 16:17:10 jnrao: Do you have the facilities to run CI? 16:17:15 we test FCZM too 16:17:15 hemna_: cuz that's the beauty of FC, it's not all the same necessarily 16:17:21 we are setting up CI for cinder. 16:17:40 we're in planning stages for fc ci 16:17:42 So we need some guidelines on how to do the CI with zone drivers. 16:17:46 All vendors send a device to jnrao to test ;) 16:17:51 hehe 16:18:06 jnrao, do you have an FC enabled array? 16:18:17 i mean we do only for Brocade switch gear. 16:18:22 not for all. 16:18:33 has anyone used the cisco driver? 16:18:34 jnrao, I'd start by looking at which tempest tests would cover the zone driver 16:18:40 akerr, not us yet. 16:18:44 jgriffith: i'm surprised you're not volunteering to test FC :) 16:18:50 avishay: :) 16:18:51 we have cisco switch 16:18:51 asselin :ok. 16:18:53 lol 16:19:00 jgriffith, avishay: :) 16:19:12 * jungleboyj volunteers jgriffith 16:19:15 avishay: won't be long before I have to I'm afraid 16:19:16 Cisco fixed a bug recently but there's still one more issue they are looking at 16:19:18 jnrao, attaches, detaches, live migration 16:19:22 jnrao, then having ci run those tests is just configuration + tempest regex 16:19:26 * jgriffith slaps jungleboyj upside the head 16:19:31 jnrao, copy volume <--->image 16:19:32 jgriffith: i'm sure 16:19:39 * jungleboyj blocks it with ninja like skills. 16:19:55 * jgriffith runs away 16:20:02 OK :) 16:20:07 May I suggest moving to the next point? 16:20:19 if we get all storage drivers with CI by K that would be great 16:20:19 only attach and detach only for now. 16:20:25 avishay, +1 16:20:30 flip214: +1, its almost lunch time 16:20:39 FC zone drivers are bonus for now IMO 16:20:42 akerr: no, the kids want dinner! 16:20:51 How many zone manager drivers do we have at this point? 16:20:53 hemna_: ^ 16:20:55 thingee, 2 16:21:00 cisco, brocade 16:21:00 * bswartz gave up hope of eating lunch on wednesdays 16:21:05 but if we are doing CI 16:21:15 we need to do CI for the driver and the lookup services for each vendor 16:21:26 avishay: +1 That is the first goal. 16:21:37 each vendor's support with the FCZM is broken up into 2 components. the driver and the lookup service. 16:21:38 hemna_: ugh 16:21:45 hemna_:agree that we need CI for driver and lookup 16:21:50 hemna_: is there zone driver-specific code in the storage driver? 16:21:56 avishay, no 16:21:58 well 16:22:06 avishay: no. 16:22:10 wonder if we should take a look at how ZM was designed 16:22:13 there is a decorator on the initialize_connection and terminate_Connection methods for FC enabled drivers. 16:22:16 think there's a better way 16:22:25 is the interface into the lookup service not standard? Can we claim compliance with both if we CI with only 1? 16:22:29 so if the FCZM is configured it gets called, w/o the volume manager being involved. 16:22:33 perhaps we can reasonable and knowing that the zone manager works with one FC volume driver? 16:22:40 ie turn it into a target, so to speak 16:22:44 jgriffith: sorry if I missed this earlier. what happened to your FC driver?:) 16:22:46 people, we're at article 2 of SEVEN on the list, with >⅓ of the time gone. 16:22:58 xyang1: haven't written it yet :) 16:23:14 thingee, we can take this offline in the cinder channel if you like. 16:23:15 do IBM, EMC, and HP use different ZM drivers in their CIs? 16:23:17 xyang1: but it's relatively light I think 16:23:18 so we can move ahead ? 16:23:21 hemna_: yes please 16:23:25 jnrao: is that fine? 16:23:39 avishay, we just use 1 currently. we don't have both vendor's switches at the moment. 16:23:42 xyang1: no pun intended... FC == light :) 16:24:10 #agreed nothing. need further discussion 16:24:10 avishay: We are still working on getting that working. 16:24:14 #topic NFS/posix backup drivers again 16:24:15 hemna_: i meant if IBM uses the brocade, and HP uses the cisco, for example, then at least both ZM drivers are exercised 16:24:22 avishay, true 16:24:23 tbarron, kfox1111: hello 16:24:24 avishay: +1 16:24:25 hi 16:24:25 avishay, gotcha 16:24:31 hi 16:24:33 alright lets all be nice here 16:24:43 avishay, I think xyang1 has cisco switches. fwiw. 16:24:43 jgriffith: good, as long as you are planning to write one:) 16:24:46 we have the posix backup driver 16:24:47 #link https://review.openstack.org/#/c/82996/ 16:24:59 Yup. I believe we found a common solution. 16:25:00 and the nfs backup driver 16:25:02 #link 16:25:03 so Kevin and I have met and agreed on an approach that avoids duplication of functionality 16:25:11 #link https://review.openstack.org/#/c/138234 16:25:15 hemna_, avishay: yes, we tested it. there's one more issue they need to fix before we retest 16:25:24 I'm documenting here: https://review.openstack.org/#/c/130858/ 16:25:30 jnrao, please join the #openstack-cinder channel to continue the FCZM discussion after the meeting 16:25:43 Duncan recommended making the code from the swift driver more generic so it could be shared. 16:25:45 * DuncanT was very pleased to see the chunking base class 16:26:05 #link https://review.openstack.org/#/c/139737/ 16:26:14 I've pulled most of the swift code out to a base class, and added parts of the posix driver to support big chunks. 16:26:23 #agreed common solution of making swift driver more generic so it can be shared 16:26:36 I've yet to port the posix driver to it, but I'm shooting for Friday. It should be a very small driver at that point. 16:26:39 +1 16:27:02 ok, we might have to target to k-2 in review priority 16:27:03 so the posix driver chunks the volume and stores the chunks in separate files? 16:27:05 at this point 16:27:07 Then generic NFS driver - which needs to manage its own mounts - will inherit from posix 16:27:07 The nfs driver can then build on it, doing the mount/config. 16:27:15 is that fine? 16:27:30 avishay: yes. 16:27:30 sorry lost connection. 16:27:40 We avoid duplication of function. 16:27:46 thingee: sounds reasonable. 16:27:49 sounds like a good plan 16:27:53 alright great 16:27:56 sounds good to me 16:27:56 I am a little worried that swift changes may make merging hard. 16:28:01 thingee: CI For zone drivers: I will catch up with you later on this topic. 16:28:06 kfox1111, tbarron: anything else? 16:28:09 avishay: yes 16:28:14 if there arn't any for that driver, before then, it won't be a problem though. 16:28:19 are there any schedualed? 16:28:28 * tbarron is a bit worried about delay since we are coupled to lots of changes 16:28:45 backup drivers? I'm not sure. there are no more volume drivers accepted after k-1 is done 16:29:04 kfox1111: look at the multithreading for Swift backup driver change 16:29:22 tbarron. ok. so at least one. :/ 16:29:33 thingee: if we already have a review in process we're OK though, right? 16:29:35 kfox1111: it's slow moving last I checked 16:29:53 The multi-threading is proving a pain to get right 16:29:55 tbarron: the requirement for k-1 just applies to volume drivers. not back up drivers 16:29:58 I'd like to get https://review.openstack.org/#/c/130858/ approved at this point (bp spec) 16:30:09 thingee: ack 16:30:12 And probably wants to be multi-process to get significant wins 16:30:22 tbarron: that's a good thing 16:30:24 Since threading in python is painfully limitted 16:30:37 need DuncanT and eharney to look at the bp-spec 16:30:58 I will revise with kfox1111's comments from this morning 16:31:07 ok I think this going a good direction 16:31:11 #topic Discuss implementation direction of PureISCSIDriver CHAP support changes 16:31:17 patrickeast: you're up 16:31:22 it has a plan of record for NFS, posix, and chunking 16:31:25 #link https://review.openstack.org/#/c/137175/ 16:31:27 hi, so for https://review.openstack.org/#/c/137175/ there was some contention about the direction 16:31:29 Thanks all. gtg. 16:31:36 kfox1111: thank you 16:31:39 and it was recommended to bring it up for discussion at the meeting 16:31:51 the basic issue is that we don’t have a good place to store the CHAP secret 16:32:14 the latest version of the changes use a new db table to store them 16:32:41 i’ve put together a little bit more info and some answers to common questions here https://etherpad.openstack.org/p/pureiscsi-chap-faq 16:32:43 * thingee scans the change 16:33:15 patrickeast: so first, I owe you for your patience and explaining the issue multiple times for me :) 16:33:35 jgriffith: np 16:34:06 patrickeast, so FWIW, we had the same issue a while back 16:34:12 patrickeast: Personally I don't mind the table entry, even the file is *ok* now that I understand the problem better. So I'm good with whichever method the rest of the team deems better 16:34:25 we ended up finding a solution on our array to store the CHAP creds. 16:34:36 hemna_: yea unfortunately we don’t have that ability 16:34:43 hemna_: well, very easily 16:34:47 i think it's credible that Cinder should store this info as a general idea 16:34:57 hehe patrickeast I didn't say ours was easy :P 16:35:05 hemna_: lol 16:35:23 some drivers use cinder.conf 16:35:26 which has it's own issues 16:35:30 So one thing we need to thnk about is sec people flip when this isn't stored encrypted 16:35:43 yea :( 16:35:43 We'll have to address that at some point... maybe later in K? 16:36:03 i wonder if you can get around this issue a different way by working like LIO does 16:36:12 eharney: I was just waffling on that idea. 16:36:13 maybe instead of another table, add a driver-specific column in the volume table, 255 char, driver can put whatever they want? 16:36:14 heh and if you put the encryption keys in cinder.conf, then how secure are we really? 16:36:20 just generate them at attach time? 16:36:36 avishay: interesting.... 16:36:46 I kinda like that... driver-metadata column 16:36:49 avishay, like....metadata :P 16:36:51 avishay: _1 16:36:52 errr... table 16:36:56 avishay: the problem is that we need to have access even when the volume table is potentially empty (sans the one being attached) 16:36:57 avishay: +1 16:37:00 We already have two fields in the table for that 16:37:01 seems overkill to add a new table for one driver 16:37:02 hemna_: durrr, yea :) 16:37:03 provider-* 16:37:06 hemna_: admin metadata even 16:37:11 patrickeast: it'd be an independent table 16:37:12 oh metadata God :) 16:37:16 oooh 16:37:17 avishay, yah that's what I was thinking. admin metadata 16:37:35 as long as it isn’t tied to a volume it will probably work 16:37:43 thats been the biggest issue 16:37:44 btw, what's up with the admin metadata that's there? 16:37:45 ? 16:38:00 patrickeast, but you need the CHAP to attach the volume. 16:38:02 patrickeast: nope, wouldn't be tied to a vol at all 16:38:05 why can't it be tied to a volume ? 16:38:05 is storing the chap credentials long term actually required? 16:38:18 eharney: it is for some people :( 16:38:41 so for our backend we don’t have volume specific credentials, they are specific to a host (more explaination in the FAQ i posted) 16:38:44 is it? you can't generate them at attach time and handle it dynamically then? 16:38:54 patrickeast, ours is the same way 16:39:04 ahh 16:39:05 eharney: that's what I was thinking 16:39:08 eharney: trust me, that horse is dead and I beat it for a week :) 16:39:17 jgriffith: ok :) 16:39:18 Many drivers put in cinder.conf now - whatever alternative solution we go for needs to be more secure than that IMO to be worth changing - a field in the db is not inherently more secure than a database table 16:39:24 eharney, the problem is, you need credentials for the iscsi session to the host the first attach. 16:39:32 s/than a database table/a conf file/ 16:39:33 every attach after that needs the same credentials 16:39:52 hemna_: yeah, different scheme, LIO does creds per-volume 16:40:04 eharney: +1, same with SF 16:40:04 got it 16:40:16 makes life much easier 16:40:24 DuncanT: yea i agree, the benefit is that you can have different credentials for each host instead of a globally used set of credentials 16:40:24 how far off would using something like barbican be? 16:40:32 ok so I heard a new table and metadata. 16:40:35 DuncanT: this was a feature specified by a customer 16:40:37 ameade_: I probably wouldn't even consider it TBH 16:40:47 metdadata won't work for patrickeast because that's tied to a volume 16:40:57 thingee: I like the idea of a Vendor-Metadata table personally 16:41:04 patrickeast: Customers specify all sorts of crazy things, I like to get the reasons *why* they want a thing 16:41:07 thingee: and the idea of encrypting the keys in the db 16:41:09 ameade_: I agree, it should be considerated 16:41:13 jgriffith, yah, we could have used that ages ago, if we had it. 16:41:31 so i guess if we have buy-in for a new table we can hash out the details in the review/or in openstack-cinder, don’t want to take up too much of the meetin time 16:41:31 #idea vendor-metadata table 16:41:35 Doesn't seem crazy 16:41:37 vendor-metadata is what, one table per vendor? 16:41:41 thingee: I would like to have a look/culling of all the metadata tables we have currently though 16:41:51 seems we've become a bit "metadata happy" 16:41:52 On record per vendor even? 16:41:59 DuncanT: yeah 16:42:08 DuncanT: err...no 16:42:12 * DuncanT votes for calling it other than meta-data 16:42:14 normalize it to a vendor table, and FK that to the metadata table? 16:42:14 DuncanT: so one table, multiple entries 16:42:15 patrickeast: I would like to take it offline to understand the point eharney made about getting the details on attach time. or point me to the answer on the faq 16:42:25 then you could have multiple entries in the table from different vendors 16:42:26 but yes sounds like we have an approach to consider 16:42:34 thingee, +1 16:42:35 columns like: Vendor-Name, Vendor-ID, Driver.... 16:42:39 thingee: np, i’m happy to explain it in length ;) 16:42:40 jgriffith, yah 16:42:46 and of course K/V's 16:43:12 #topic Discuss database cleanup status and priorities 16:43:16 e0ne: you're up 16:43:17 hi all 16:43:18 my only concern is that maybe we have too much of that already. And maybe admin meta is a good place for this particular problem? 16:43:23 #link https://etherpad.openstack.org/p/cinder-kilo-db-cleanup 16:43:26 Oh...gues we're done on that 16:43:26 K/Vs make indexing near impossible though, so if there are going to be many entries we need something better 16:43:30 hi all. i don't want to take a lot of time 16:43:35 i just want to prioritize these things. 16:43:41 there are a lot of blueprints which could be outdated or important 16:43:43 jgriffith, isn't the admin meta data tied to a volume though ? 16:43:43 DuncanT: FK in Volume column maybe? 16:43:58 also, there are no much perfermance impact from the db side when we'ge got 10000+ records and more then 1000 existing volumes 16:44:15 hemna_: yeah :( 16:44:32 DuncanT: actually, indexes doesn't solve all problems 16:44:40 a lot of time takes to data transfer between db <> api <> client 16:45:11 e.g. 14ms for select in db bacome about 250 from python side:( 16:45:20 e0ne: No, but a brute force of search of many K/V records is always worse 16:45:26 e0ne: i've seen the same thing 16:45:56 DuncanT: i'll provide some profiler result with indexes 16:46:04 it will winn about 2-3ms: 16:46:38 we need to optimize data transfering too 16:46:49 the most important thing IMO is to reduce the number of DB accesses. we saw the osprofiler example with like 30 DB accesses for create_volume? 16:47:01 e0ne: The win will totally depend on the number of records, number of K/Vs in each record etc 16:47:02 we select too many data when we don't need ir 16:47:04 and storing heartbeats there, etc. 16:47:07 s/ir/it 16:47:15 avishay, +1 16:47:22 avishay: +1, started cleaning some of those up... particularly update, followed by get :( 16:47:29 DuncanT: i tested with 100k+ records 16:47:41 jgriffith: +1 16:48:39 from what i've seen we've gotten rid of the O(N) operations (like one DB lookup per volume). scale seems to be OK i think. 16:48:49 heartbits and deleted record is a really big problem on big env 16:49:01 e0ne: so you're original question was to prioritize 16:49:15 deleted records in quota reservations bit us recently 16:49:17 your* 16:49:23 12 million of the damn things 16:49:29 thingee: yes. and verify existing blueprints 16:49:31 doh 16:50:04 anyone want to work with e0ne on this? 16:50:08 DuncanT: :-) 16:50:17 I'd like to volunteer 16:50:20 myself 16:50:29 rushiagr: thanks! 16:50:45 DuncanT: do they ever get deleted, or are they just marked as deleted? 16:50:49 I'd recommend just carry conversations in #openstack-cinder and people might over read them and chime in. 16:50:56 e0ne: :) 16:51:06 #topic Discuss HA issues mentioned by Ops during the summit 16:51:10 hi all 16:51:13 #link https://etherpad.openstack.org/p/kilo-crossproject-ha-integration 16:51:15 i've always wondered why deleted records stick around rather than getting deleted 16:51:15 avishay: There's nothing in cinder to delete them at the moment, you need to implement something outside of cinder 16:51:22 #link https://etherpad.openstack.org/p/kilo-crossproject-ha-integration 16:51:26 this is from the summit session 16:51:46 there are multiple cinder issues mentioned, one of them is no recovery in case of failure mid-task 16:51:54 akerr: audit, billing, record-keeping etc etc 16:52:01 our operators are constantly encountering this, so I'm motivated to work on it 16:52:17 dulek: +1 from me 16:52:18 I can offer my full time + maybe another our developer to drive this topic through Kilo cycle, I would just need some guidance where and how to start 16:52:20 dulek: curious, is anybody else in OpenStack solving this? 16:52:30 DuncanT: gotcha thanks 16:52:48 jgriffith: taskflow? Glance is integrating with in Kilo. 16:52:48 dulek: great. i think there might be some discussions that happened at the summit that might overlap with this. I have not read this over myself though to verify atm 16:52:50 dulek: ie is there any project that recovers from failure mid-action 16:52:56 is the HA wanted on a per-service level? 16:52:57 jgriffith, wasn't one portion of this looking at removing the local locks in the volume manager? 16:53:02 dulek: Glance is a trivial non-interesting case IMO 16:53:16 because the easy solution is to put some services in a VM and have *that* HA via pacemaker... 16:53:26 jgriffith, and we had spoke about this in the meetup in paris to add 'ing' state checks in the API to prevent actions being taken 16:53:34 flip214: Well, still you lose your request 16:53:59 Also you have no possibility to safely close the service (for upgrade for example) 16:54:02 flip214: that's not really the problem being discussed, it's more about rpc dropping on the floor mid-stream or db getting out of sync or stuck 16:54:11 flip214: we need some monitoring/healthcheck api for pacemaker 16:54:24 flip214: solutions like pacemaker and A/A nodes are in existence 16:54:26 I don't want to waste meeting time so maybe someone can volunteer to explain me status of that offline? ML replies were a little too vague for me to understand anything. 16:54:29 dulek: I'll make an action item for myself to review this and see if some of the other stuff in progress takes care of the concerns. I'll provide a blueprint/spec when necessary. 16:54:49 #action thingee to review HA etherpad to see what overlaps 16:54:56 dulek: so for the sake of record keeping, I think it's a GREAT thing to work on 16:55:08 dulek: just found the problems to be stated a bit too vaguely 16:55:19 thingee: if you have any questions, i wrote most of the comments in the etherpad 16:55:29 jgriffith: +1 16:55:29 avishay: thanks! 16:55:41 thingee: Can you answer on the ML then when you finish? 16:55:51 e0ne: you also wanted to kickoff a HA-related topic today, right? 16:55:59 jgriffith: usually that's how it is. I think it's important to point dulek to the work in progress that are smaller chunks to work towards this 16:56:13 dulek: sure 16:56:17 thingee: +1, that's what I need to start helping. :) 16:56:26 dulek: i'm ready to start it almost anytime:) 16:56:33 dulek: ok I need to move onto the next topic, sorry 16:56:38 thingee: yeah, I'd appreciate that pointer as well :) Maybe link them on the etherpad 16:56:47 #topic Third Party CI updates 16:56:50 asselin: go go go! 16:56:59 * jgriffith keeps loosing track of thangp and harlowja_away 's work :( 16:57:03 hi, real quick: 16:57:05 Would like to invite Cinder third party ci "operators" to third party ci meeting: https://wiki.openstack.org/wiki/Meetings/ThirdParty 16:57:09 #link https://wiki.openstack.org/wiki/Meetings/ThirdParty 16:57:19 would be great to have more participation 16:57:27 Meeting time change is proposed on mailing list. Vote here: http://lists.openstack.org/pipermail/openstack-dev/2014-December/052546.html 16:57:33 go vote for new meeting time! 16:57:43 #link http://lists.openstack.org/pipermail/openstack-dev/2014-December/052546.html 16:57:46 lastly, Asselin is proposing an in-tree 3rd party ci solution. Idea well received in 3rd party ci & infra meetings. Please review spec and/or let me know your interest. 16:57:57 #link Gerrit topic "thirdpartyci" query: https://review.openstack.org/#/q/topic:thirdpartyci,n,z 16:58:49 asselin: we're interested in this 16:59:20 asselin: are the times UTC? 16:59:22 you can ping me in the -cinder channel, or comment on spec. 16:59:26 flip214, yes 16:59:34 +1 for A 16:59:34 eikke, great :) 16:59:35 /whois asselin 16:59:55 rushiagr, Ramy Asselin 17:00:00 :-) 17:00:05 rushiagr: who is John Galt! 17:00:13 that's a rather manual way of answering whois. 17:00:16 thanks everyone! 17:00:19 #endmeeting