18:00:04 #startmeeting keystone 18:00:05 Meeting started Tue Jul 5 18:00:04 2016 UTC and is due to finish in 60 minutes. The chair is stevemar. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:08 The meeting name has been set to 'keystone' 18:00:09 reminder for ajayaa, amakarov, ayoung, breton, browne, crinkle, claudiub, davechen, david8hu, dolphm, dstanek, edmondsw, gyee, henrynash, hogepodge, htruta, jamielennox, joesavak, jorge_munoz, knikolla, lbragstad, lhcheng, marekd, MaxPC, morgan, nkinder, notmorgan, raildo, rodrigods, rderose, roxanaghe, samleon, samueldmq, shaleh, stevemar, tjcocozz, tsymanczyk, topol, vivekd, wanghong, xek 18:00:09 o/ 18:00:16 o/ 18:00:27 o/ 18:00:28 hello 18:00:36 o/ 18:00:41 \o 18:00:41 o/ 18:00:43 agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting 18:01:12 o/ 18:01:19 \o 18:01:21 o/ 18:01:22 give it a minute or so, lazy cores not showing up >.< 18:01:22 \o 18:01:25 oh there they are 18:01:42 hi 18:01:54 everyone ready to talk about specs! 18:01:55 (laxy core, present and correct) 18:02:09 o/ 18:02:26 hi everyone o/ 18:02:30 quick first topic before the spec talk 18:02:36 #topic api-ref sprint 18:02:44 #link https://etherpad.openstack.org/p/keystone-api-sprint 18:02:47 all the details are there 18:03:28 basically, once https://review.openstack.org/#/c/337805/ merges, then http://developer.openstack.org/api-ref-identity-v3.html will pull from the keystone repo 18:03:49 only trouble is, those are out of date and don't match what we've been doing here: http://specs.openstack.org/openstack/keystone-specs/ :( 18:04:34 I wanted to have a day or two to sprint on getting the APIs from our specs repo migrated over to the API site in the keystone repo 18:04:38 out-of-day? 18:04:44 v3 or v2? 18:04:48 gyee: out of day? 18:04:58 s/day/date/ 18:05:09 stevemar: a sprint is a good idea 18:05:24 gyee: yes, previously those APIs were maintained by the docs team, but that didn't scale 18:05:46 so now each project is keeping an api-ref directory in their project repo 18:05:54 see: https://github.com/openstack/keystone/tree/master/api-ref/source 18:06:21 there is some content missing, i've outline what is missing in the TODO section of the etherpad: https://etherpad.openstack.org/p/keystone-api-sprint 18:06:58 stevemar: bug friday work? 18:07:04 stevemar: when do you plan on sprinting? 18:07:21 dstanek: the dates are on the etherpad, july 13 and 14th 18:07:27 stevemar, we still care about WADLs and XSTs from V2 world? 18:07:42 gyee: there are no more WADLs or XSTs or any of that 18:07:53 oh good! 18:07:57 gyee: the pieces that need to be filled are all v3 related 18:08:38 stevemar: those dates work for me 18:08:38 dstanek: if you're interested, add your name to the volunteers list :) 18:08:43 will do 18:09:02 i'll publish a hangouts link as the day approaches 18:09:20 it'll be a quick way to get reviews and commits, i'll focus on those that day 18:09:44 if something doesn't make sense, then ask me in -keystone :) 18:09:50 sounds good 18:09:52 next topic! specs 18:10:08 #topic discuss open keystone specs for newton 18:10:40 we have the keystone feature proposal freeze happening next week: http://releases.openstack.org/newton/schedule.html 18:10:50 err sorry 18:10:59 we have the spec freeze *THIS* week 18:11:06 o/ 18:11:17 so specs have to be merged this week, or they are bumped to ocata 18:11:31 start bugging cores 18:11:50 #topic specs related to federation 18:11:59 enhance mapping (dolph) https://review.openstack.org/#/c/324055/ 18:12:00 federated query API (ayoung) https://review.openstack.org/#/c/313604/ 18:12:00 specify project id (amakarov) https://review.openstack.org/#/c/323499/ 18:12:23 dolphm's spec has seen a lot of positive feedback 18:12:42 ideally i want only one of these to be approved, since i think they are trying to solve the same issue 18:12:54 enhance mapping doesn't seem to solve custom project Id, as far as I can tell 18:13:06 ayoung and amakarov are not here to defend :( 18:13:18 dolphm: can you speak to that point? ^ 18:13:39 it allows you specify a project name, and if that isn't created it does on dmeand 18:13:40 I did a quick review this morning and had a few concerns 18:13:40 demand 18:13:41 iirc 18:13:44 gyee: ^ 18:13:59 gyee: the custom project ID was (IIRC) so federating projects could happen 18:14:10 gyee: are your concerns blockers? 18:14:21 But amarakov was interested in more than just federated, right? 18:14:25 samueldmq, stevemar, that's for mirroring projects, same ID, same name, same everything 18:14:42 shaleh, right, basically, replicating projects 18:14:53 creating with a custom project id was a cheap version of replication - i'm still not convinced we should allow that 18:15:03 jamielennox: ++ 18:15:07 gyee: If any of those projects or roles do not exist, they must be created by Keystone automatically. 18:15:11 gyee: what jamielennox said 18:15:12 gyee: from the spec ^ 18:15:14 jamielennox, sure, but that's a different argument 18:15:39 gyee: why? if we don't need it then we don't need to cover the spec 18:15:50 gyee: the mapping enhancements that dolph proposed will create the project automatically if it's not there 18:16:02 stevemar: exactly 18:16:09 #link https://review.openstack.org/#/c/324055/ 18:16:12 stevemar, that's very different than asking for the same project ID :-) 18:16:17 as for dolphm's spec I like the idea of it but I share ayoung's concern over testability and maintainability 18:16:18 so those specs are unrelated 18:16:27 i really don't like the idea of supporting replication through the api 18:16:40 there be dragons there and i don't like dragons 18:16:42 there is a lot of room for oops and not a lot of great ways to test it. 18:16:47 dstanek: agreed 18:16:51 dstanek: ++ 18:17:27 I am merely pointing out that the two are unrelated 18:17:34 dolphm: as an operator making map files how do you intend for them to ensure they are not over/under specing? 18:17:38 gyee: they are not as unrelated as you think 18:17:40 so we are not confused 18:17:42 dstanek: shaleh: which spec has dragons? all 3 result in some means of "replication" 18:17:50 gyee: they are both trying to solve the case of federating keystones 18:18:08 stevemar, no, the replicate project one is very different 18:18:15 stevemar: amarakov was not necessarily using federation. 18:18:28 shaleh: "over / under speccing" as in granting or not granting enough authorization? 18:18:30 stevemar: he just has multiple datacenters trying to use the same data 18:18:38 dolphm, does mapping allowed replicating by ID? 18:18:46 dolphm: 'specify project id (amakarov) https://review.openstack.org/#/c/323499/' moreso than yours, but it's possible yours can be abused too? 18:18:46 dolphm: yes. or mistaking the projects named, etc. 18:19:12 dolphm: as you point out in the spec when an oops is found the clean up can be non-trivial. 18:19:33 shaleh, question for you later...don't disappear 18:19:35 gyee: i only specified it for project names, but you could extend it to project IDs if there was a use case 18:19:53 dolphm: while I do not 100% agree with ayoung's spec I like the problems he raises 18:19:58 dolphm, if we extend it to project IDs then we can close the other one 18:20:00 ayoung: welcome 18:20:00 ayoung: k 18:20:07 that's precisely what the other one wants 18:20:53 and i'd still be -1 18:21:00 luddites. 18:21:05 myself included 18:21:16 -1 for what? 18:21:24 don't specifiy project id! 18:21:33 duuuude 18:21:33 anywhere! 18:21:39 dolphm: I would be positive on your spec if we included some helping mechanism like ayoung proposes. 18:21:43 jamielennox, I like that 18:21:48 treat ids like inodes 18:22:01 how often do you specify anything by inode id 18:22:21 gyee: do any other APIs allow to specify ids? 18:22:41 jamielennox, what we are offering is choice and flexibility, didn't ops said galera replication falls over after certain number of masters? 18:22:44 anyway personal opinion, dolphm to fix spec to commit this cycle - others not 18:22:56 TODO list: 1. Kille domains. 2. Make projects hierarchical. 3. Make project name a URL 18:23:08 ayoung: not there yet :-) 18:23:22 shaleh, I've beeen *there* for 4+ years now 18:23:35 ayoung, unless you enjoying typing the entire url via cURL 18:23:37 ayoung: no, I mean we are talking about other specs at the moment 18:23:49 gyee, no problem 18:23:53 its called a hyperlink 18:24:01 new fangled thing 18:24:06 think it is going to catch on 18:24:10 lets make ayoung type in his LDAP DN every time he authenticates 18:24:11 :) 18:24:24 gyee, nah, I want LDAP to convert to using URLs too 18:24:37 ayoung: can the snark :P 18:24:45 X500 notation is gah 18:24:49 dolphm: in classic ops there is a testing env before rolling a change out to prod. I do not see a sensible way to validate the mapping change in testing env. 18:25:05 stevemar, all this is, strange to say, that I like and support dolphm's spec 18:25:23 shaleh, if you don't validate mapping during testing, you are asking for trouble :-) 18:25:24 ayoung: \o/ hallelujah 18:25:50 stevemar, I think that, however, we are going to need all those other things I posted only "seemingly" humorously 18:26:00 we need to kill, or at least, de-emphasize domains 18:26:14 proposal: dolph’s speec gets the nod, the others are bumped 18:26:14 we need to have the ability to have users be self adminig for common tasks 18:26:32 henrynash: i was just writing that 18:26:39 you can self admin today, just tweak the policies 18:26:41 esp, man, esp 18:26:51 we need to be able to ahve a project on keystone-over-there mirrored in this keystone for federation, but the unique namiong thing gets in the way 18:26:53 gyee: are your concerns about dolphm's spec blockers? 18:26:55 ayoung did you check with henrynash on your domain bashing? I think he needs them for something 18:27:11 topol, nah he is having them forced on him 18:27:15 gyee: ? 18:27:19 to aboid the the strict naming thing 18:27:19 stevemar, no, if you guys added id mapping, I am happy 18:27:23 (ducks) 18:27:26 okay, done 18:27:42 dolphm's spec gets the nod, provided he does some clean up of the current iteration 18:27:46 henrynash is a pragmatist, and working within the policies of the current Amdminstration 18:27:47 others are bumped for now 18:27:56 #topic specs related to service-to-service communication 18:27:56 stevemar, list of "opthers" 18:28:01 * ayoung had network issue 18:28:09 ayoung: federated query API (ayoung) https://review.openstack.org/#/c/313604/ and specify project id (amakarov) https://review.openstack.org/#/c/323499/ 18:28:29 ayoung: OK to continue to next topic? 18:28:33 stevemar, yeah, I can live with that 18:28:38 s2s is important 18:28:39 ayoung: ty 18:28:45 jamielennox: you're on the hot seat 18:28:45 stevemar, no need to bump project id, we just need to merge it with dolphm's 18:28:49 and I I think jamielennox 's approach is good 18:29:02 fg 18:29:03 gyee: do a follow on patch then 18:29:19 stevemar, yes sir 18:29:23 so, i think the service users spec got nacked by the security group and i kind of agree with them 18:29:40 the reservations spec still needs a better name and a lot more work 18:29:42 * topol me too :-) 18:30:01 i'm hoping to get some time with people at the midcylce to hash out some more details and make sure everyone understands it 18:30:02 jamielennox, operational tokens 18:30:26 hopefully have code examples too 18:30:46 jamielennox: i left some comments there, i'm not sure if there's an operation that uses the token multiple times right now 18:30:47 jamielennox, a reservation, as you origianlly described it, is a rule to transition 18:30:57 jamielennox: so you're aiming for Ocata ? 18:31:01 from the token a user origianlly requests, 18:31:08 to permissions for a specified operation. 18:31:13 samueldmq: i would think it's newton 18:31:14 So abandon one, and i want to work with people on the reservations one over the next few months and land early ocata 18:31:22 okay 18:31:34 i think of it as capability tokens but whatever, i've been trying to avoid saying token 18:31:38 jamielennox: can you abandon the correct one 18:31:42 the idea of a "reservation" though, is that you go to glance and create the reservation. Lets drop that word 18:31:47 * topol reservations always reminds me of Dave Cheritons famous paper on Leases. If anyone besides me and ayoung remember that work 18:31:52 it really does not reflect your current design 18:31:58 topol, oooh 18:32:02 Lease.... 18:32:13 topol, can you find a link? 18:32:18 sure 18:32:30 #link http://web.stanford.edu/class/cs240/readings/89-leases.pdf 18:32:35 we have trust, oauth, and now reservations, goody 18:32:37 topol: ^ 18:32:41 well since one is abandoned and the other is targeting ocata, i don't see a reason to continue this topic 18:32:56 1989 18:32:59 http://web.stanford.edu/class/cs240/readings/89-leases.pdf 18:33:00 jamielennox: are you OK with that? 18:33:03 2nd place 18:33:03 stevemar: ++ 18:33:11 stevemar: yep 18:33:12 gyee, this is why we wanted unified delegation 18:33:20 they are all different use cases 18:33:25 okay, next topic, none of this is landing in newton 18:34:11 #topic specs related to HMT 18:34:20 henrynash: where's the popcorn? 18:34:26 gyee makes a good point listing all the options that makes our consumability difficult 18:34:33 ok, so https://review.openstack.org/#/c/332940/ is the key 18:34:45 request last week was to get more feedback 18:35:02 So...while I think this review is 100% sane and something we should support, not allowing the nested naming under projects is criminal 18:35:05 henrynash, lets do V4 and get it over with :-) 18:35:28 so far have feedback from 2 operators in the the UK (once called datacentred,who host revenue and customs), plis one other 18:35:37 gyee, or just accept the relaxation of the rule in V3. We are making people's lives difficult with no reason. 18:35:40 But...all that asid 18:35:41 e 18:36:03 lets get Henry's work going so we can continue to confuse people with domains...which are still not supported in any service other than Keystone 18:36:06 could we goto the TC and get the rule relaxed? 18:36:26 topol, no, because4 our local TC rep is one of the luddites keeping us here 18:36:30 so far thefeedback is supportive of the concept, although I agree 2 feedbacks does not give us a lot of cover 18:36:36 topol, afaik, TC's job is to cut ribbons 18:36:36 i still think when combined with things like domain specific backends and domain specific roles this is going to get difficult 18:36:37 topol: i see no reason they wouldn't say to keep API compat 18:36:37 * topol one time mulligan? 18:37:18 topol, here's how I see it 18:37:27 we can punt and say "aok, diomains can be allowed to do this 18:37:36 and in doing so, we've kept the letter of the law 18:37:42 effectively cr3eaing a feature that is unusable 18:37:56 ayoung: ? 18:38:07 and we do that vbecause the fear of relasing the naming within "proejctes" which, as averyone know, are still usually called tenants 18:38:12 will break something 18:38:25 henrynash, nested domains 18:38:33 henrynash, nested domains are legal and useless 18:38:48 henrynash, we should never have even included the conceopt of domains 18:38:57 we should have made projects hierarchical from the get go 18:39:03 ayoung: namespaces are needed 18:39:10 we should have made IdPs into domains 18:39:17 ayoung: agreed, we should have done that for projects 18:39:18 we did neither fo those things and made a mess 18:39:32 henrynash, hierarchical projects *should* have been our namespaces 18:39:39 ajnd henrynash I won't hold this up 18:39:59 but, prcatically, these are not going to fit in with an tyhning today 18:40:06 right now, it is all about quota 18:40:08 ayoung, I disagree, that's a long argument we need to have over beers :-) 18:40:11 and quoata is not a domain option 18:40:12 kk, let's just fix the "mess" if we can or leave with it 18:40:15 gyee, I blame you 18:40:20 indeed, we have a lot of simialar concepts sparsely used 18:40:21 we can't rollback doamins ... 18:40:24 gyee, acatull;y, I blame me 18:40:35 ayoung: and sicne domains are projects, you DO get quotes here 18:40:40 so...why do we want this? 18:40:43 the reasons for domains are well stated 18:40:47 henrynash, not really 18:41:00 henrynash, quoats are assigned on projects *under* domains today 18:41:29 so if I get a domain scoped token, and iti s under another domain it will not fit under current quota 18:41:37 ayoung: since they were designed before projects could ast as domains, now we have that, this can be repalces 18:41:39 ayoung, you can get a project scoped token 18:41:44 with the is_domain on it 18:41:49 gyee, the reasons are solid, the fact that they were not done as projects was the mistake 18:41:51 right? 18:41:59 ayoung: ++ 18:42:03 rodigods: ++ 18:42:12 so you can enforce this in the policy files 18:42:41 today, we can set quota for a "domain" using, the project acting as domain since mitaka 18:42:55 what we need is a way to do openstack --user-domain-name=somedomain --proejct=name=test and have that work when there are two projects named test 18:43:07 and to set "domain" quotas, just need to add the is_domain in the policy rule 18:43:07 the rest of this is avoiduing that discussion 18:43:08 you can't get a project-scoped token with is_domain set to True today, can you? 18:43:15 irrelevant 18:43:38 domains are not a concept we can have outside of Keystone today. We need nested projects for the rest of the world, not for Keystone 18:43:48 this *is* namespaceing 18:44:02 and we missed the opportunity to do it right...years agbo 18:44:04 ago 18:44:09 hell, we inherited it 18:44:13 ayoung: we;ve had that discussion. And teh concensus was it was too risky. WE are not green field any more. We have to work within the echos of our past deicsions 18:44:34 henrynash, that does not mean making useless mechanism 18:44:35 s 18:44:41 and nested domains are going to be just that 18:44:42 from a UX perspective to perform this will require a user to use --os-domain-name parent/project instead of --os-project-name which i feel is also weird 18:44:55 (and we should say "project acting as a domain" as little as possible to the rest of the world) 18:45:02 jamielennox, that is what everything should be a URL 18:45:06 jamielennox: ++ 18:45:17 when I get a scoped token, I should pass in a URL 18:45:32 and only that URL, I might add 18:45:49 i don't see this being resolved any time soon, i am more than happy to give this a FFE after the midcycle, i'm thinking it'll be better discussed in person? 18:45:59 seriaously, If I were doing this from scratch: BASIC_AUTH /passwrod GET https://url/down/to/project and dopne 18:45:59 stevemar: agreed 18:46:09 but..regardless...will nested domains buy us anything 18:46:12 ? 18:46:23 ayoung: yes, but lets defere to in-person 18:46:24 can anywon besides henrynash say that it will? 18:46:40 ayoung (it’s nice to be special) 18:46:45 henrynash, sorry to do this to you, becuase I know you are trying to get things done 18:47:07 ayoung: but if I can’t convince more than myslef, then we shouldn’t do it 18:47:09 ayoung: it's fine, i appreciate your opinion on this 18:47:15 ayoung: henrynash: does this solve/hurt reseller? 18:47:26 dstanek: it aids reseller big time 18:47:44 dstanek: but I didn’t wnat to complicate the spec with that 18:47:45 dstanek: yeah, it would be essential for reseller 18:47:49 dstanek, this help a lot the reseller use case, but I think we need a couple more of work on it 18:48:01 but it is a huge step 18:48:03 henrynash: i just asked because ayoung was asking what else it buys us 18:48:04 henrynash, I think it limits reseller to one level, if I understand how things are going to be done. 18:48:15 say..Verizon to momandpop 18:48:29 mom and pop need to be able to create domains under momandpop. 18:48:40 ayoung: no it can be multi-level (but again, a discssio for another day) 18:48:44 stevemar: next 18:48:47 and ther we run into the unique naming thing/information hiding issues that were so central to Reserller years ago 18:48:51 is amakarov here? 18:49:09 #topci spec for RBAC support 18:49:14 #topic spec for RBAC support 18:49:17 he's not 18:49:19 (amakarov) https://review.openstack.org/#/c/325326/ 18:49:41 i haven't reviewed it yet, the title confuses me, i thought we already have rbac support :P 18:49:55 stevemar, termie suggested this back in Vancouver 18:50:16 this is basically to have non-bearer tokens... 18:50:17 move policy enforcement into Keystone instead of "token validation" in Keystone, policy in middleware 18:50:24 this wont fly 18:50:26 jst fyi 18:50:43 other projects have to opt in/buy in. i am doubtful this will happen 18:50:44 on this spec, I like the idea to improve the oslo.policy support 18:50:58 ….by having keystone give the final say on whether user has sufficient roles at the point of policy enforcement 18:50:58 not that it would be good/bad 18:51:12 notmorgan: agreed. I do not see Nova going for this after the work they have put in recently. 18:51:16 from my understanding of it we need to discuss it a lot more first 18:51:17 i don't see how this eliminates bearer tokens, but i've old just browsed the spec 18:51:21 notmorgan, so...it could actually be done without their support 18:51:29 jamielennox: ++ 18:51:38 dstanek: it doesn't 18:51:52 since the call to validate a token is in Keystone middleware, it would jsut require adding more data into the call, which could, in theory, be validated later 18:51:53 i was hoping he'd be at the midcycle but i don't thinks o 18:52:06 isn’t that in the pre-amble? 18:52:21 ayoung: but enforcement has to happen based on results from the data in db. 18:52:25 notmorgan, it would be kindof like the suggesting that we split policy 18:52:28 because it would tie into policy ideas for reservations 18:52:35 so, we can't enforce in middleware 18:52:41 at the moment 18:52:45 notmorgan, just hte "role" part would be in middleware 18:52:53 not the the "and the proejct matches" part 18:53:00 stevemar: I can’t imagine we can apprive this for Newton…given the scope of this 18:53:11 henrynash: +++++ 18:53:12 instead we say "assumign the proejct matches, would this be allowed" 18:53:13 this is certainly a large endeavor 18:53:26 i thought a tenant of the openstack architecture was to push out the policy on purpose so that identity wasn't hit when it didn't need to be 18:53:33 and then policy would do "and the proejct matches" inside the code in the API call, aftermiddleware 18:53:36 henrynash: i thought i read that somewhere in the spec 18:53:38 dstanek +++ 18:53:38 henrynash: for sure not newton, but no reason we can't aim for ocata and start the work now 18:53:45 dstanek, it is hit for token validation anyway 18:53:50 ayoung: current architecture does not work that way 18:53:59 if you wanted to optimize, you could do this: 18:54:18 i'm going to ask for reviews to look at the spec, but i honestly don't think it can make newton 18:54:19 call keystone with token and proejct Id. Response comes back with a list of operations that are allowed 18:54:24 we are cutting newton-2 in 7 days 18:54:31 and this is moving a mountain 18:54:36 it really is not 18:54:36 stevemar, ++ 18:54:45 dstanek: line 89: The current architecture (with role assignment control and policy enforcement 18:54:46 separated) forces us to use bearer tokens 18:54:47 so we are hitting keystone on every authz call? performance will be fun 18:54:49 it might be triggering a landslide... 18:54:55 ayoung: :) 18:55:01 tsunami! 18:55:15 where’s that boat I had handy 18:55:17 ayoung: there is now a tool in oslo.policy which does what you ask 18:55:19 ayoung: something i'm not comfortable with in the middle of the cycle, things like this need to land *early* 18:55:29 like crazy early 18:55:31 stevemar, I think all it would allow is adding in a flag to allow returing a list of allowed operations for the given endpoint 18:55:39 shaleh, I know. I wrote it. 18:55:58 stevemar: talking about it now, preparing for O is not a bad idea though. But the spec is still a little rough. 18:56:16 shaleh: yep, also why O is a better candidate here 18:56:29 really quickly, i want to talk about henry's last spec 18:56:40 stevemar, I think we should discuss at the midcycle. We might be able to do this with minimal impact. WOuld be Beta quality whenver it lands 18:56:40 #topic keystone-manage migration-complete step for rolling upgrades 18:56:50 https://review.openstack.org/337680 18:56:50 ayoung: i'm OK with that 18:56:52 henrynash, I think I was too harsh on nested domains 18:57:01 ayoung: np 18:57:02 I think that I was conflating two issues 18:57:15 This is really to fix bug https://bugs.launchpad.net/keystone/+bug/1596500 as well as prevent the same happening with https://review.openstack.org/#/c/328447/, but given questions about our rolling upgrade support, I wrote a short spec for it. 18:57:15 Launchpad bug 1596500 in OpenStack Identity (keystone) "Passwords created_at attribute could remain unset during rolling upgrade" [Undecided,New] - Assigned to Henry Nash (henry-nash) 18:57:22 and I think that nested domains, for reseller, while it would impose some limitations, is a legitimate way to work it 18:57:39 henrynash: I like this. It sounds very reasonable. 18:58:22 dolphm: I think you had (many) concerns about rolling upgrdaes? 18:58:24 henrynash, just so I have it here on the record...I am not opposed to nested domains, and support it for HMT. 18:58:34