11:00:44 <oneswig> #startmeeting scientific-wg
11:00:45 <openstack> Meeting started Wed Aug 16 11:00:44 2017 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.
11:00:46 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
11:00:48 <openstack> The meeting name has been set to 'scientific_wg'
11:00:56 <oneswig> Hello
11:01:02 <martial_> Hello Stig
11:01:08 <oneswig> #chair martial_
11:01:09 <openstack> Current chairs: martial_ oneswig
11:01:11 <oneswig> Morning martial_
11:01:11 <zz9pzza> Hi :)
11:01:16 <verdurin> Afternoon.
11:01:20 <enolfc> Hi
11:01:23 <oneswig> Hi zz9pzza verdurin enolfc
11:01:24 <daveholland> hello!
11:01:28 <oneswig> and daveholland!
11:01:44 <oneswig> #link agenda for today (such as it is) https://wiki.openstack.org/wiki/Scientific_working_group#IRC_Meeting_August_16th_2017
11:02:00 <priteau> Hello
11:02:11 <oneswig> Main item of interest this week is the second edition of the book
11:02:13 <martial_> my, quite a good turnout of people I do not remember before. Welcome all
11:02:14 <oneswig> Hi priteau
11:02:44 <oneswig> Indeed, welcome everyone
11:03:00 <oneswig> #topic Scientific OpenStack book revision
11:03:02 <daveholland> @oneswig possibly for that (or for AOB) I would like to ask people's opinions and experiences about reporting "availability" of an entire Openstack platform (not just nagios)
11:03:30 <oneswig> daveholland: good topic, lets keep time for AOB
11:04:00 <daveholland> TY
11:04:18 <oneswig> So, the book was created for Supercomputing 2016 and handed out by various people at the conference
11:04:27 <oneswig> I think it was really popular and well received
11:04:42 <oneswig> I've seen a number of people since with copies of it
11:04:52 <oneswig> So this is a great way we can make a difference
11:05:22 <oneswig> The plan is to revise / improve / extend the book for SC2017
11:05:28 * johnthetubaguy enjoyed reading the previous edition
11:05:44 <zz9pzza> hi john congrats btw
11:05:45 <oneswig> Hi johnthetubaguy, good to see you
11:05:50 <martial_> #link Working document to organize resources https://etherpad.openstack.org/p/scientific-computing-book-update-2017
11:05:50 <oneswig> :-)
11:06:09 <oneswig> thanks martial_
11:06:23 <oneswig> I missed last week's meeting, can you recap the discussion from the US timezone?
11:06:33 <oneswig> Or should I just read the etherpad :-)
11:07:29 <martial_> sorry, copying data, but basically
11:07:38 <oneswig> OK, there's some very useful contributions offered there.
11:07:38 <martial_> Mr Hoge said "With time running out, real-time update from Kathy regarding the book. Deadline is Sept 22"
11:08:20 <zz9pzza> Dave has been grabbed by someone, "Sangar multi-tenant Lustre integration ?" I will talk with him later as my English skills are minimal
11:08:40 <martial_> a few people offered to work on updates of previous chapters
11:09:02 <martial_> and we have --so far-- a short number of new papers to to add
11:09:29 <oneswig> #link document source is here https://github.com/openstack/scientific-wg
11:09:58 <oneswig> martial_: one of the hoped-for contributions is a section on federation, right?
11:10:02 <enolfc> oneswig: who's contributing to the federated identity chapter?
11:10:23 <oneswig> enolfc: well volunteered? :-)
11:10:38 <oneswig> How is EGI federation going BTW?
11:10:50 <martial_> oneswig: I tried to reach out to Dr Bohn but he will not be able to do one, so there is a possibility about the ORC
11:10:51 <enolfc> oneswig: I guess I have volunteered :)
11:11:22 <oneswig> I believe there was jmlowe from Indiana University looking to make a contribution.
11:11:36 <enolfc> EGI is now ready to move to federated identity with OpenID Connect and planning to do so in production in the next months
11:11:36 <zioproto> hello, sorry I guess I am 10 minutes late
11:11:37 <oneswig> thanks enolfc, that'd be great to have your input
11:11:45 <noggin143> I'll have a chat with Jose. We could certainly review the federation chapter but may not have time to write it.
11:11:46 <oneswig> Hi zioproto, welcome
11:11:46 <martial_> and Mr Fernandez just did volunteer :)
11:12:01 <enolfc> I can also check with INDIGO colleagues as I see you also mention it in the current text
11:12:30 <martial_> welcome zioproto
11:12:34 <oneswig> Hi noggin143, thanks, that would be a great help.  What tends to happen is a body of text is written followed by 2-3 case studies.  A case study is typically half a page with a picture
11:13:03 <oneswig> Perhaps that might not be too onerous for Jose?
11:13:09 <noggin143> That sounds OK.
11:13:45 <noggin143> A sample configuration on how to do federated id with Edugain and Active Directory
11:13:53 <oneswig> I recall much of the existing text came from the user story zioproto put together last year
11:13:55 <noggin143> we could do that.
11:14:17 <oneswig> noggin143: that would be brilliant.
11:14:59 <oneswig> martial_: do you recall the ID federation jmlowe was discussion - was it Globus via OIDC?
11:15:22 <martial_> that sounds right
11:15:58 <oneswig> martial_: who can we approach wrt a closing paragraph on ORC?
11:16:17 <oneswig> Or should we wait for results first there?
11:16:17 <martial_> enol is involved with it
11:16:26 <enolfc> oneswig, nogging143: we can have 3 use cases, Edugain, AD and EGI
11:16:27 <oneswig> Ah, great, of course.
11:17:27 <enolfc> oneswig: I can provide the ORC text
11:17:39 <oneswig> Ideally we'd have diverse approaches in the case studies.  Any keycloak users out there?  I recall that was discussed positively
11:18:00 <oneswig> Thanks enolfc, appreciated
11:18:29 <daveholland> not sure who added Sanger Lustre integration to the etherpad (it wasn't us!) but as I'm presenting it at LAD in October then - given guidance on number of words/diagrams, expected audience, etc - that might be do-able. Is it too niche?
11:19:08 <martial_> Enol: I missed the last ORC meeting but I will be able to join the next one. We can add this conversation to the agenda for the next meeting
11:19:11 <oneswig> daveholland: Not at all.  We have a whole chapter on filesystems IIRC, this would make a great case study
11:19:25 <martial_> daveholland: I would love to see this paper
11:19:26 <zz9pzza> We are going to spend a chunk of time going over things and writing it up as a paper ish could be done.
11:19:27 <daveholland> OK, I will have to read the current version :)
11:19:42 <enolfc> martial_: sure, I think will start joining again next week also
11:20:01 <martial_> as we get closer, we are going to need to make sure we have reviewers also
11:20:04 <oneswig> Can those who volunteered add their names + details to the end of the etherpad so we can keep track?  thanks
11:20:40 <verdurin> martial_: given everything else I'm doing at the moment, I'm unlikely to be able to submit new material, but happy to be a reviewer
11:21:05 <martial_> maybe we should request for the deadline to be made shorter (vs the official 22nd) to get a couple reviewers per chapter
11:21:09 <oneswig> Thanks verdurin, that would be appreciated
11:21:58 <martial_> the 22nd is a Friday so our meeting is likely the 19th that week ... 2 days to review, 2 days to fix
11:22:07 <oneswig> martial_: We'll probably need all the time we can, but definitely want to avoid a massive body of unreviewed documentation come the 21st
11:23:18 <oneswig> Excellent - any more to cover here?
11:23:40 <oneswig> Thanks for everyone's volunteering :-)
11:23:57 <oneswig> #topic OpenStack London
11:24:27 <oneswig> Not much defined here but for those who are planning to attend OpenStack Days UK on 26th September, we have a slot booked for the WG.
11:24:41 <oneswig> I was wondering how we should fill it...
11:24:55 <martial_> how long are you talking about?
11:25:06 <zz9pzza> Dave has a hurdy gurdy.
11:25:31 <martial_> (that seems to be about the same time as the ORC meeting?)
11:25:33 <oneswig> It's probably an hour, might be 40 mins.  Will need to check
11:25:51 <oneswig> zz9pzza: Thanks for your input :-)
11:26:04 <noggin143> I'll be there too.
11:26:09 <martial_> 10 minutes intro to the group, a stig-sentation follow by a couple lighting talks?
11:26:29 <noggin143> but I think I'm talking in another slot.
11:26:51 <oneswig> noggin143: at the same time?  We can't have that.  I haven't seen timing but I'll try to get things shuffled if so
11:27:05 <noggin143> don't know the schedule yet
11:27:24 <oneswig> martial_: That's a good model, has worked previously.
11:27:40 <noggin143> +1 for the lightning talks
11:28:25 <martial_> stig: you can make a tease about the meeting in Sydney as well :)
11:28:44 <oneswig> Do we have volunteers for talks?
11:28:55 <oneswig> I might know one or two interested people
11:29:17 <zz9pzza> Tim or Pete are doing the lustre talk as a separate slot.
11:29:26 <oneswig> zz9pzza: daveholland: Is Tim C presenting?  Ah OK
11:30:13 <oneswig> I'll sound out some people
11:30:44 <martial_> sounds like it is going to be a great pesentation, will it be streamed?
11:30:46 <oneswig> We can also offer to people on the day, but that's uncertain
11:31:07 <oneswig> martial_: The talks were filmed last year, so I would expect so
11:32:17 <oneswig> OK, was there anything more on OSD UK?  Nothing here
11:32:37 <oneswig> #topic Open Research Clouds
11:32:50 <oneswig> How is it going?
11:33:14 <martial_> I missee the last meeting, maybe enol can let us know?
11:33:50 <enolfc> I also missed last meeting
11:34:13 <enolfc> but from EGI we have the logistics ready for having the congress in Amsterdam
11:34:17 <martial_> okay, well the last meeting I attended, things were still in definition
11:34:21 <enolfc> registration is already open
11:34:30 <martial_> working groups were forming'
11:34:34 <oneswig> enolfc: It's end September, right?
11:34:34 <enolfc> Now there is some discussion about sponsorship
11:34:40 <enolfc> yes, 27-29
11:35:14 <enolfc> I plan to have AAI wg starting to meet from next week
11:35:28 <martial_> #link http://www.openresearchcloud.org/orc-events/amsterdam-september-2017/
11:35:28 <oneswig> enolfc: can you link to the EGI page for it?
11:35:30 <enolfc> when some EGI people is back from holiday
11:35:40 <oneswig> Ah, thanks :-)
11:35:49 <enolfc> https://indico.egi.eu/indico/event/3414/
11:35:56 <oneswig> Intriguing fish
11:36:48 <enolfc> We now need to start moving things in the other wg, and finalise agenda
11:36:52 <oneswig> Who should attend?
11:37:12 <martial_> here are the latest happenings (had to find the link) https://docs.google.com/document/d/1NdcuKlSKnQU_RqWI3lcgpJWwg6a0qhaJra-wN8zCqg8/edit
11:38:25 <oneswig> OK, thanks martial_ enolfc - any more to add here?
11:38:53 <martial_> oneswig: as to who should attend, I think anybody that can contribute to the effort
11:39:06 <enolfc> martial_ +1
11:39:17 <enolfc> oneswig: no more updates from me so far
11:39:31 <martial_> oneswig: there was a request to contribute from the point of view of scientific computing as well
11:39:57 <martial_> oneswig: so it is going to depend on the narrative of that conversation
11:40:09 <martial_> I think Tim mentioned he might be able to make it
11:40:19 <martial_> I am possibly going to be there as well
11:40:30 <oneswig> martial_: The timing is possible for me, however my level of contribution is uncertain
11:41:06 <oneswig> OK - next item?
11:41:13 <martial_> Khalil requested a small presentation on the state of scientific computing and why an ORC would be welcome
11:41:20 <martial_> oneswig: go ahead
11:41:27 <oneswig> #topic AOB
11:41:39 <oneswig> daveholland: take it away mr music
11:41:46 <martial_> I wanted to share this link ... a cool program I discussed with Flanders
11:42:00 <martial_> #link http://superuser.openstack.org/articles/openstack-mentor-mentee-program/
11:42:43 <daveholland> ta :) so, we are happy measuring e.g. "is the API up" with nagios. But we are dancing around a measurement for "availability" for an OpenStack platform. e.g. if one user cannot deploy a large flavour instance because of hypervisor memory fragmentation, that's a showstopper for him, but the platform is still available.
11:42:48 <martial_> and for those who have not seen it so far ...
11:42:50 <daveholland> anyone got thoughts, experiences, opionions?
11:42:52 <martial_> #link http://superuser.openstack.org/articles/scientific-wg-update/
11:43:31 <zz9pzza> What happens if say I can't create new volumes but everything else is working ...
11:43:33 <johnthetubaguy> daveholland: did you look at using rally to regular test if you can create an instance?
11:43:35 <noggin143> For those interested in nested quotas, I have posted the CERN model proposal to https://openstack-in-production.blogspot.ch/2017/07/nested-quota-models.html. Happy to discuss if there are other proposals too.
11:44:07 <daveholland> @johnthetubaguy an "end to end" nagios test (maybe using heat) is on my radar, not thought of rally, I'd mentally got that as a post-deploy benchmark - am I wrong?
11:44:09 <noggin143> At CERN, we run Rally to give the availability. A sample set of workloads runs constantly, creating VMs, attaching volumes etc. It works great.
11:44:44 <oneswig> noggin143: is it on CERN ops github?
11:44:44 <johnthetubaguy> I knew someone told me about that before, must have been noggin143
11:45:16 <daveholland> OK I will take a look. (it will inflate the headline "12345 instances launched to date" number, but hey)
11:45:17 <noggin143> old presentation is at https://indico.cern.ch/event/407658/contributions/968467/attachments/1126441/1608329/150330_rally_deployment_cern.pdf for details.
11:45:30 <zz9pzza> Thanks
11:45:35 <noggin143> It's great for the VMs/hour graphs too :-)
11:45:43 <daveholland> @noggin143 thanks!
11:46:25 <johnthetubaguy> noggin143: quota wise, I guess that is going to come up at the PTG, I might try sync up with that effort again, are other folks there to discuss that already?
11:46:29 <oneswig> I thought this was pretty cool from Nectar: https://status.rc.nectar.org.au/capacity/
11:46:47 <oneswig> It's not directly related, but plugs the gap of revealing the available capacity on a private cloud
11:47:14 <noggin143> For quota, I'd passed the information to Sean Dague who was looking for the 'rulesets'.
11:47:26 <daveholland> @oneswig that's something else we've been asked for, in the context of a small host aggregate/ceph-backed flavour for "pets"
11:47:36 <johnthetubaguy> noggin143: cool, he is leading that discussion AFAIK
11:48:40 <johnthetubaguy> the placement API should help some with capacity queries: https://docs.openstack.org/nova/latest/user/placement.html#rest-api
11:49:47 <johnthetubaguy> #link https://developer.openstack.org/api-ref/placement/ might be the better link
11:50:15 <priteau> johnthetubaguy: What is the default access policy to discover capacity?
11:50:18 <oneswig> This autogenerated documentation is really good johnthetubaguy
11:50:46 <johnthetubaguy> oneswig: its largely hand written now, to get it better, ups and downs
11:50:58 <johnthetubaguy> priteau: its admin only API at the moment
11:51:10 <oneswig> In which case, I should say this hand-written documentation is really consistent :-0
11:51:38 <daveholland> even an admin API is fine for populating metrics to produce a user-visible graph/number
11:51:41 <priteau> johnthetubaguy: That's a sound default, but it would be nice if most academic/community clouds allowed users to query it
11:52:21 <johnthetubaguy> priteau: its got a policy rule you can change, as usual, I think, that should just be the default
11:52:30 <johnthetubaguy> daveholland: +1
11:53:04 <johnthetubaguy> I think its probably better to build graphs like the nectar ones from that info
11:53:21 <johnthetubaguy> it used to be crazy hard to extract that data, should be easier / more portable now
11:54:26 <noggin143> Placement stuff looks promising... time to throw away the nasty local legacy database scripts...
11:55:07 <priteau> +1
11:55:57 <oneswig> johnthetubaguy: are you looking at the resource provider inventories here?
11:56:40 <johnthetubaguy> oneswig: that is one way, you can also query what resource providers have space for a particular flavor you are interested in
11:56:58 <johnthetubaguy> well, its a resource request, rather than a flavor, but there is a mapping
11:57:10 <oneswig> and in response you'd get a 'yes' or 'I can accommodate 800 of those'?
11:57:42 <johnthetubaguy> I think you get a list of candidates where it can fit, with info on how many you could fit in each candidate (not sure how obvious the second part is)
11:58:31 <oneswig> johnthetubaguy: sounds very good for this case (and not at all what a public cloud would want to share)
11:58:42 <johnthetubaguy> the real answer is probably need to add some APIs to make life easier, but I think its possible without
11:59:13 <oneswig> OK we are nearly upon the hour - any more to add?
11:59:15 <daveholland> lots to think about and try out. thanks all
11:59:15 <johnthetubaguy> yeah, its currently not exposed externally by default (I would recommend)
12:00:13 <oneswig> OK time to close the meeting - thanks everyone, very useful discussion
12:00:29 <martial_> very good conversation, the meeting notes are going to be fun :)
12:00:36 <oneswig> #endmeeting