21:00:16 #startmeeting crossproject 21:00:17 courtesy ping for Qiming TravT gordc dirk mriedem SergeyLukjanov 21:00:17 courtesy ping for daemontool jroll boris-42 redrobot flaper87 rhochmuth 21:00:17 courtesy ping for fungi flwang dims vipul johnthetubaguy rakhmerov 21:00:17 courtesy ping for docaedo stevemar mtreinish bswartz adam_g adrian_otto 21:00:18 Meeting started Tue Feb 16 21:00:16 2016 UTC and is due to finish in 60 minutes. The chair is dhellmann. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:18 courtesy ping for zigo Piet sdake mugsie sheeprine thinrichs 21:00:19 courtesy ping for jklare loquacities smelikyan Daisy skraynev odyssey4me 21:00:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:21 The meeting name has been set to 'crossproject' 21:00:21 courtesy ping for catherineD dhellmann dprince hyakuhei notmyname devkulkarni 21:00:33 o/ 21:00:35 o/ 21:00:35 o/ 21:00:36 ohai! 21:00:38 o/ 21:00:40 hello :) 21:00:43 Hello 21:00:44 thingee asked me to lead the meeting until he's on a stable wifi connection, which should happen shortly 21:00:44 #chair thingee 21:00:45 Current chairs: dhellmann thingee 21:00:47 hi all 21:00:52 o/ 21:01:02 dhellmann: may I have my name added to the list of courtesy ping too ? :) 21:01:27 o/ 21:01:28 samueldmq : yes, I'll make a note of that (I'm not sure if thingee has a separate list, I got this one from last week's meeting) 21:01:42 #action thingee to add samueldmq to the courtesy p ing list 21:01:45 #undo 21:01:46 Removing item from minutes: 21:01:48 #action thingee to add samueldmq to the courtesy ping list 21:01:52 o/ 21:02:00 #link https://wiki.openstack.org/wiki/Meetings/CrossProjectMeeting our agenda for today 21:02:12 #topic Team announcements (horizontal, vertical, diagonal) 21:02:13 who has announcements? 21:02:40 #info release team: final non-client library releases coming up Feb 22-26 21:02:45 #info release team: feature freeze for Mitaka coming up Feb 29-4 21:02:49 dhellmann: add me to courtesy reminders too 21:02:51 #link http://releases.openstack.org/mitaka/schedule.html the release schedule 21:02:59 #action thingee add nikhil to courtesy ping list 21:03:12 dhellmann: thanks 21:03:25 Ops mid-cycles going on 21:03:27 sup dhellmann 21:03:29 last week we had some discussion about quota; I went back to nova-last to ask for an interested party to participate. alaski said he would keep an eye on that. 21:03:40 oh crossproject roger 21:03:42 s/nova-last/nova-land/ 21:03:45 cool 21:03:45 bknudson_ : tell all the ops we said "hi" 21:03:59 I'm not there! also, maybe it's done already 21:04:04 hi, sdake :-) 21:04:04 we'd have the x-prj spec up later this week 21:04:05 dhellmann: I'm not sure whether this is worth mentioning here or not: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086706.html 21:04:14 bknudson_ : oh, well 21:04:26 mtreinish : sure, that's what this section of the meeting is for 21:04:31 but it's removing a ML that used to be there, which seems sorta cp 21:04:39 #info the openstack-qa mailing list is officially shut down 21:04:52 I didn't know about http://status.openstack.org/openstack-health/#/g/build_queue/periodic 21:05:12 mtreinish : do you have a link to your blog post about the dashboard handy? 21:05:26 o/ 21:05:28 dhellmann: this one: http://blog.kortar.org/?p=279 21:05:40 that's the one 21:05:53 #link http://blog.kortar.org/?p=279 mtreinish wrote up a nice description of the new openstack health dashboard 21:06:24 what other cp teams do we have represented today? docs? stable? 21:07:18 cdent : sorry, I missed your comment earlier. is there some place we need to make a note about alaski's participation in that (I missed last week's meeting) 21:07:40 infra, but we don't have any cross-project announcements this week afaik 21:07:41 dhellmann: no, don't think so. as long as the people driving the x-spec are aware it's good 21:08:08 fungi : ack, sorry to leave you out 21:08:13 cdent : sounds good 21:08:17 o/ 21:08:20 hi, thingee 21:08:35 hi 21:08:36 it sounds like that covers the announcements, unless thingee has anything? 21:08:37 * thingee unstable connection. will probably drop off in 22 mins 21:08:48 nothing from me 21:08:54 ok, let's move on then 21:08:55 #topic A Common Policy Scenario Across All Projects 21:09:00 #link https://review.openstack.org/#/c/245629/ 21:09:13 jamielennox: ^ 21:09:21 aha, I was just going to ask who is driving this 21:09:40 So basically i was looking to get a broader cross project view on the above spec 21:10:02 dhellmann: is this user policy? or somethign else 21:10:17 more out of how services grew than anything else we've been stuck in an admin or non-admin kind of policy for a long time now 21:10:27 sballe: yes, it's related to policy definitions within the applications for access control, etc. 21:10:49 and from a keystone perspective we keep trying to add new security features that really aren't a lot of help when operators have to use the admin role to get anything done 21:11:22 the goal of the spec is to define a scenario, a couple of roles that everyone can agree on and would implement by default 21:11:28 jamielennox : I see some comments in recent reviews about names of things, which makes me think this is not so controversial but needs some time to think about little details. is that your understanding of the state of the spec, or do you think there might be uncovered issues? 21:11:51 s/uncovered/as yet uncovered/ 21:11:56 this would not prevent any operator from defining their own roles and policy as they do now, but it would significantly increase the security of those deployments who don't want to change policy files 21:12:05 I think there's also some concern about size/extent, dhellmann 21:12:08 * notmorgan is lurking (sorry) 21:12:13 dhellmann: so there is some questioning of naming (as always) 21:12:32 but the other question that dolphm and i have been going back and forward on and could use some broader scrutiny is how extensive we want this to be 21:12:47 sorry, joined just a second ago 21:12:58 initially the plan was to just provide some basic roles and improve on the current scenario 21:13:01 jamielennox : a phased approach does have some appealing aspects 21:13:22 the spec sort of grew from there to doing fairly fine grained roles 21:13:48 either work and the appeal of doing this in one go is not having to update every project's policy.json multiple times 21:13:56 jamielennox: I got a few cinder cores to weigh in and they are all saying the proposed roles are more complicated than they need to be but agree that there should be more than just admin and non admin 21:14:46 yeah - i like the new project / read only roles, but not the more permjssion style roles 21:14:48 diablo_rojo: that's part of the intent - it's more roles than any one user / deployer needs, but covers all the bases for everyone 21:14:51 seemed to me that you don't *have* to descend into the deep end of the role configurations in that spec right away, or even ever. but, they seemed like nice options. 21:15:01 diablo_rojo: so we asked for feedback at the tokyo summit and have talked to operators since and this seemed to cover 90% of their use cases, we're definetly looking at this from an ops perspective rather than de 21:15:07 by convention, there's probably a role defined for what you need 21:15:25 with the per manager roles i think we would cover everything 21:15:45 don't orgs already have their idea about what their policy is? Maybe our efforts to polish policy.json should rather go into re-using existing orgs' roles and policies? 21:15:46 elmiko: i'd want to split the granular (api managers and per capabilities roles in particular) into a separate spec if we're going to go halfway on anything 21:15:52 instead* of going halfway 21:15:53 dolphm: ++ 21:16:05 breton: that is exactly where this conversation stems from 21:16:13 dolphm: i think that's fair and most likely sensible 21:16:41 i don't have an objections to the fine grained stuff, but i see it as optional to the end-user/deployer 21:16:42 breton: yes, we want to polish policy - the problem is we want all services to polish their policy in a consistent way 21:16:54 elmiko: absolutely is optional, as proposed 21:17:05 breton: hence defining a scenario that they would all aim for rather than everyone make up their own roles 21:18:40 essentially i want people to take the spec back to their ops and project teams and comment on whether the proposed roles are sufficient as it'd be nice to not need to take this through to summit 21:19:02 ok, that's a good call to action for this one 21:19:15 does anyone else have any questions to raise right now? 21:19:23 dolphm: do you have a different timeline there? 21:19:29 dolphm: or action? 21:19:38 nope, that's my goal 21:19:49 i'd like to propose the resulting policy to at least keystone and nova 21:19:55 i've done keystone, but i went all the way with it 21:20:08 if we're going to break the spec down, i'll do two separate patch sequences 21:20:37 for example, my current patch sequence for keystone starts here: https://review.openstack.org/#/c/274143/ 21:20:47 dolphm: given the nature of keystone i actually don't mind if it goes per manager roles, but obviously on top of the basic ones and not necessarily something we recommend to all projects 21:20:54 i add an identity_admin, then managers, then observers, then per-capability roles 21:21:20 #info the goal is to review the proposed roles before the summit 21:21:26 #action everyone please review the common policy spec with your teams 21:21:32 jamielennox: still want to split the effort apart then. my goal in working keystone and nova is to illustrate the impact of this spec to all the services under big tent 21:21:45 dolphm: ok 21:21:53 implementing in other projects should be low hanging fruit from there 21:22:11 except for project's like ironic :P where there's a bigger hurdle to climb first (using policy at all) 21:22:30 can we put us back on the agenda for a fortnight? 21:22:38 ahh, that's 2 weeks 21:22:54 we don't use the metric system here 21:22:57 I have a question around implementation of this 21:22:58 jamielennox : sure, you can do that by editing the wiki page 21:23:13 dhellmann: yep, just aiming to give a timeframe to the project teams 21:23:15 i'll be afk for this meeting in 2 weeks 21:23:30 what level/aspects of services are expected to adhere to this granular sample policy file? 21:23:32 nikhil: ask! 21:23:44 \whois dolphm 21:23:49 (sorry the first one was to avoid last min topic switch) 21:23:50 nikhil: what do you mean by level/aspects ? 21:23:53 diablo_rojo: =) 21:23:59 dolphm: Lol sorry :) 21:24:05 yeah, I will take glance as example 21:24:08 nikhil: Dolph Mathews @ Rackspace 21:24:21 oops, diablo_rojo ^ 21:24:25 :) 21:24:33 is it merely roles that are elaborated or 21:25:04 do we have a this sample deployment that points to roles for say "protected properties" that may be used 21:25:05 ? 21:25:25 dunno, this may be on the edge of what we want to provide and what we can actually provide 21:25:26 nikhil: elaborated is a good way to put it 21:25:37 gotcha 21:25:51 dolphm: jamielennox: I also have a question :) 21:25:59 samueldmq: ask away! 21:26:07 protip: never ask for permission to ask questions 21:26:11 the per capability roles are the finest we provide right ? 21:26:22 samueldmq: yes, as currently proposed by the spec 21:26:36 so, isn't it going to be hard to an operator to sync all those fined grained roles inside keystone 21:26:47 as projects add/remove APIs 21:26:48 there isn't anything finer unless you're talking about per-resource RBAC (like this user has this capability on this particular VM, regardless of the VM's tenancy) 21:27:01 samueldmq: you don't need to create any of these roles in keystone 21:27:02 are we willing to provide tools to help them in that task ? 21:27:04 samueldmq: not one 21:27:21 dolphm: don't they come from the token ? 21:27:26 samueldmq: and yes, we could totally provide a openstack role gotta-create-them-all 21:27:40 samueldmq: the roles a user actually has are expressed in the token, yes 21:27:51 samueldmq: the roles a user needs to have is expressed in policy 21:28:15 dolphm: yes, I am in the most complex case, where one uses all the per-capability roles in his org 21:28:26 this change would make policy read like: admin OR identity_admin OR catalog_manager OR endpoint_manager OR endpoint_create 21:28:31 so he could make use of such a tool to load the roles in his keystone 21:28:39 samueldmq: sure, why not 21:28:47 dolphm: nice 21:29:02 I was just making sure about this, I like the idea (and you know!) 21:29:12 :) 21:29:25 dolphm: thanks 21:29:47 we do have another topic for today, are we ready to move on? 21:30:01 or for a read-only operation like "list endpoints", policy would effectively be: admin, observer, identity_admin, identity_observer, catalog_manager, endpoint_manager, list_endpoints 21:30:11 we can move on if there's another topic 21:30:24 dolphm : I don't want to cut you short, I thought we'd hit a lull 21:30:29 dhellmann: ++ 21:30:38 dhellmann: i'm done 21:30:46 ok, we can come back during open discussion if anyone thinks of other questions 21:30:49 #topic Support for 4-byte unicode for naming volume, snapshot, instance etc. 21:30:55 sheel & jgregor, you're up 21:30:59 Hello 21:31:04 dhellmann: yes we are there 21:31:25 can you give a brief introduction of the issue? it sounds like a mysql configuration change? 21:31:32 #link https://review.openstack.org/#/c/280371/ 21:31:42 So as of right now MySQL does not support 4 byte unicode 21:31:51 ok, so here we want to discuss for support for 4 byte unicode characters in different components.. 21:31:52 keystone doesn't configure mysql as far as I know 21:32:02 bknudson_: ++ 21:32:02 we just use whatever it's got 21:32:20 jgregor : it doesn't support them in any configuration at all? 21:32:31 * dhellmann was drafted to chair at the last minute and didn't read the spec ;-) 21:32:52 dhellmann: Not currently 21:33:20 From reading the spec I believe it's DB/table/column charset problem? By default it's not 4-byte? 21:33:24 So this was brought up in a Cinder meeting, and it was pointed out there that this could be cross project worthy 21:33:36 By default it is not 21:33:44 dulek: right 21:33:50 We would need to update the char set to utf8mb4 21:34:00 we have had problems in the past where we didn't set the charset on the columns... 21:34:01 yes, I think we would want all databases to be configured with this setting the same way 21:34:12 hm. i worry we're going to run into charset utf-8 issues again. 21:34:16 is that something we do within the schemas in each project? 21:34:22 but it's not an unreasonable thing to ask. 21:34:33 dhellmann: in most cases we specify now, iirc, utf-8 21:34:41 notmorgan: What issues were you running into? 21:34:43 notmorgan : where? 21:34:51 keystone does this: 'ALTER DATABASE %s DEFAULT CHARACTER SET utf8' 21:34:57 dhellmann: in the migrations. 21:35:11 jgregor: we have had many issues with the check to make sure things are utf-8 in keystone 21:35:14 maybe someone from nova can comment, but I thought they were trying to get away from huge impact migration scripts to support rolling upgrades without downtime 21:35:16 and all our tables do mysql_engine='InnoDB', mysql_charset='utf8' 21:35:20 jgregor: it's openstack code, not mysql issues. 21:35:39 dhellmann: yes, that's an open question, discussed on the spec 21:35:41 yeah, no consistency here is making the code ugly in a lot of places 21:35:44 dhellmann: on new tables and on initial creats 21:36:02 dhellmann: It's would still be possible to make such migration while keeping the rolling-upgrades guidelines. 21:36:08 dhellmann: but yes all tables would need an alter. 21:36:09 It's just a lot of additional work. 21:36:10 there are these corner cases to be handled in different places that if missed are resulting into 5xx responses 21:36:14 is the setting table-wide, or column-specific? 21:36:18 and I'd say that really bad 21:36:21 dhellmann: table wide in mysql 21:36:23 that's* 21:36:28 notmorgan : thanks 21:36:46 i've never tried a column specific charset. 21:36:55 nor looked to see if it even exists 21:37:00 nikhil: there's two problems/goals, right? One is to trap the exception that is currently causing the 500 and return a 400 instead. The other is to make it possible to use the wide charsets. We shouldn't conflate those two. 21:37:01 Dumb Q I don't see answered in the spec, what languages make use of 4byte Unicode and can't be represented in 3 bytes? 21:37:18 notmorgan: bknudson_: ++ keystone migration https://github.com/openstack/keystone/blob/5b868ab849d931b24c8832075fc4d36aef54550b/keystone/common/sql/migrate_repo/versions/067_kilo.py#L36-L37 21:37:24 cdent: That is correct 21:37:34 Kiall: Some example in different projects where we notify problem handling 4 byte unicodes: 21:37:43 Nova: https://bugs.launchpad.net/nova/+bug/1545729 21:37:43 Launchpad bug 1545729 in OpenStack Compute (nova) "Use 4 byte unicode for entity names in mysql" [Undecided,Invalid] - Assigned to Sheel Rana (ranasheel2000) 21:37:44 Keystone: https://bugs.launchpad.net/keystone/+bug/1545736 21:37:45 Cinder: https://bugs.launchpad.net/cinder/+bug/1542413 21:37:45 Glance : https://blueprints.launchpad.net/glance/+spec/ 21:37:46 heat : https://bugs.launchpad.net/heat/+bug/1545740 21:37:46 Launchpad bug 1545736 in OpenStack Identity (keystone) "keystone role create failed when 4 byte unicode character is provided in name field" [Wishlist,Triaged] - Assigned to Sheel Rana (ranasheel2000) 21:37:47 Launchpad bug 1542413 in Cinder "Unable to use 4-bytes unicode when creating volume, volume_type, CG, etc" [Medium,Confirmed] - Assigned to Jacob Gregor (jgregor) 21:37:48 Launchpad bug 1545740 in heat "support for unicode character handling in stack-create operation" [Undecided,New] - Assigned to Sheel Rana (ranasheel2000) 21:37:48 Kiall: Chinese, Japanese, Korean 21:37:53 oh gosh 21:38:03 cdent: and in glance we'd this issue were we are changing the responses for 500s to 4xx but that's just 21:38:26 pain as it's merely redressing issues for two types of DBs (one being MySQL) 21:38:27 nikhil: cdent: yes but atleast in glance things are handled gracefully 21:38:30 jgregor: thanks, I somehow thought those were covered in 3byte. 21:38:36 * bswartz notices new topic 21:38:46 as we support more DBs I think we are going to run into multiple of such issues 21:38:53 sheel: sure, but after a ton!!! of work 21:38:59 +1 for 4-byte utf-8 support 21:38:59 and making the code ugly as hell 21:39:11 nikhil: yep 21:39:11 and catching stuff where it shouldn't necessarily be 21:39:24 I would prefer it if keystone didn't have to care about this. 21:39:40 bknudson_: agreed 21:39:41 ++ 21:39:47 same goes for glance 21:40:04 jgregor: are you sure they are not in utf8 3byte? 21:40:38 mugsie: some of them are not there in 3 bytes... 21:40:39 https://www.drupal.org/node/1314214 21:40:57 Another q.. Has anyone measured the performance impact of switching a large e.g. instances table over? 21:41:21 Kiall : the impact of the migration itself, or of running with it that way after the migration? 21:41:22 reminds me of during the vancouver summit when we discovered that putting a 4-byte codepoint in a pad caused etherpad to go into a mad crash-and-restart loop 21:41:25 so, it's some kanji not all that are affected. 21:41:36 fungi : yep 21:41:41 not all. 21:41:42 I understood it the issue in mysql was that tables consumed much more disk/memory 21:41:50 Running with it after migration 21:41:53 Kiall: Well, online-schema-upgrades guidelines Nova have would block such migration. It would require an approach with a table copy probably. 21:42:12 any performance concerns would be related to exploded resource consumption 21:42:15 Kiall: Ah, okay, my asnwer assumed migration itself. 21:42:35 does it expand utf to fixed-width? 21:42:39 fwiw infra basically went through the necessary mysql changes to switch etherpad tables to 4-byte unicode encoding/collation, and the commands in the spec look like what i remember us doing 21:42:42 (mysql) 21:42:51 fungi : cool, that's good to know 21:43:37 We seem to agree that this is a lot of work for any project. Is it really worth it? 21:44:11 dulek: that's what i was asking when the spec was proposed 21:44:13 what's the work involved? 21:44:28 i couldn't get a clear use case, aside from "i want to use 4-byte characters" 21:44:33 so, for keystone, i don't see a huge reason why we wouldn't want this. it would be a change to create a new migration and update our starting one (we're not on the rolling migrations stuff yet) 21:44:41 but, it is a chunk of work regardless. 21:44:50 I think it's worth it -- it's a matter of correctness 21:45:03 yeah, I think if we have users or potential uses who can't put their language into the database, we should fix that. 21:45:08 for nova or other serivices that are on the rolling schema thing, it's a ton of work. 21:45:09 *potential users 21:45:10 bknudson_: In case of Nova and Cinder migrations will get really complicated due to rolling upgrades support. 21:45:11 bswartz: You want to use emojis for your share names, don't you. 21:45:17 bswartz: +1 21:45:29 ¬_¬ 21:45:37 dulek : yeah, I think we're going to need more detail about how to make those scenarios work before approving the spec 21:45:46 let's say the user configured mysql correctly, does openstack code force 3-byte? 21:45:54 openstack server create "(╯°□°)╯︵ ┻━┻)" 21:45:56 bknudson_: our migrations would override it 21:45:56 bknudson_ : oh, wow, I hope not 21:46:03 bknudson_: but python shouldn't care 21:46:05 stevemar: hah! 21:46:12 dhellmann: Sure, I can try to PoC something with sheel for Cinder to make amount of work measurable. 21:46:14 python has supported wide utf-8 since 2.2 21:46:16 iirc 21:46:24 stevemar : I think you mean terminate not create 21:46:28 https://www.python.org/dev/peps/pep-0261/ i think 21:46:38 dhellmann: oh right, silly me 21:46:39 the issue is with explicit charset declaration in migrations 21:46:40 :) 21:46:56 and mysql's use of utf8 to _not_ meant 4byte. to get that you have to be explicit 21:46:57 dulek: that sounds like a good plan 21:47:00 dulek: yep 21:47:07 cdent: like mysql_charset='utf8' ? 21:47:10 yes 21:47:13 oops!! 21:47:22 we did that to prevent some other problem! 21:47:28 of course 21:47:36 cdent : yeah, I wonder if we can build some tools to automate checks for that 21:47:51 dhellmann: just don't do it in the migrate way oslo_db does for charset utf-8 21:48:04 so it checks before a new migration, so you get wedged :( 21:48:17 notmorgan : yeah :-( 21:48:25 notmorgan: that was fun.. Or not. 21:49:03 sheel & jgregor: is it clear what your next steps are? 21:49:28 dhellmann: Yep 21:49:35 can we just store strings in binary? 21:49:42 dhellmann: yeah. I think measurement of actual work required for supporting it 21:49:51 bknudson_ : gzip or bz2? 21:50:08 jgregor, sheel: I can support you on the Cinder PoC from rolling upgrades perspective. 21:50:09 dhellmann: base64 encode! 21:50:13 dhellmann: everything! 21:50:16 notmorgan :-) 21:50:23 * bswartz barfs 21:50:27 dhellmann: including after we base64 the base64 encoded encoded string string 21:50:33 dulek: Sounds great 21:50:35 dulek: yes, that would work.. 21:50:57 notmorgan: ++ to get it as small as possible 21:51:04 :-) 21:51:36 ok, sounds good 21:51:42 #topic Open discussion 21:51:59 * notmorgan pushes soapbox back under the table. 21:52:07 we have a few minutes, if we have any late announcements or need to revisit something we've already discussed 21:52:55 going once 21:53:07 going twice 21:53:24 SOLD! 21:53:27 ok, then, I think we'll call that done 21:53:33 huzzah 21:53:34 thanks for joining us everyone 21:53:40 thx 21:53:48 #endmeeting