21:01:24 #startmeeting nova 21:01:25 Meeting started Thu Apr 25 21:01:24 2013 UTC. The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:01:26 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:01:28 The meeting name has been set to 'nova' 21:01:28 Howdy! 21:01:30 who's here? 21:01:35 o/ 21:01:37 -o/ 21:01:39 o/ 21:01:41 o/ 21:01:41 hi 21:01:44 Hi 21:01:48 people! 21:01:49 * Vek tries to decide... 21:02:00 me 21:02:05 #link https://wiki.openstack.org/wiki/Meetings/Nova 21:02:20 #topic havana blueprints 21:02:21 I'm virtually here (at best) :P 21:02:30 #link http://lists.openstack.org/pipermail/openstack-dev/2013-April/007788.html 21:02:39 I posted that message earlier this week, please take a moment to read it if you haven't 21:02:41 presnet 21:02:52 Morning 21:02:57 The most important thing at this point in the cycle is that we get a roadmap put in place 21:03:04 \o 21:03:06 you can find the roadmap in progress here: 21:03:15 #link https://blueprints.launchpad.net/nova/havana 21:03:27 So, if there's something you plan to work on, please file a blueprint 21:03:37 or multiple blueprints if it's complex enough to need to be broken up into stages 21:03:56 alot of stuff is still up for discussion isn't it? especially the orchestration/conductor pieces... 21:04:12 to get it on the havana roadmap, you will need to set the assignee, propose it for havana, and select a milestone that you feel is realistic for when you can have it completed 21:04:29 harlowja: yes, lots will be under discussion, we just need to take our best shot at a roadmap 21:04:43 ok, roadmap seems hard when its still under discussion :-p 21:04:45 but ok 21:04:45 the reason we do all this is to communicate amongst ourselves, as well as to the larger community what it is we're working on 21:04:56 yeh, the starting point should be all the stuff which has concensus 21:04:57 russellb: so for the migration cleanup and the object stuff, I'm happy to go off and file those blueprints now, if that's cool 21:04:58 lots of blueprints are filed but not on the roadmap, those are the ones in flux a bit, and that's ok 21:05:09 dansmith: yes please do 21:05:17 I was waiting until today for some undefined and silly reason, I guess 21:05:17 which I think there is a lot from summit, so best to take russellb's approach and get that all in 21:05:20 I'm on vacation this week, I'll catch up with mine next week 21:05:26 mikal: sounds good 21:05:37 my goal is to have a good first cut at this done in about 1.5 weeks from now 21:05:58 a couple other comments on the blueprint process ... 21:06:11 dansmith: Is the migration one for the general cleanup and unifying all the migration code stuff and cleaning up evactuate etc? 21:06:13 at the end of the release cycle, the blueprint list is effectively our feature list, and what we use to build release notes 21:06:30 as you are reviewing, please be looking for feature patches without a blueprint, and ask for one to be filed 21:06:31 cburgess_: yeah, and unifying all of that to conductor, as we discussed 21:06:46 dansmith: please can we finish discussion on that before it suddenly appears 21:07:02 another review thing ... when reviewing patches that have a blueprint, please make sure it is on the roadmap before approving 21:07:11 harlowja: appearing as a placeholder can happen before we hash out all the details 21:07:15 ok 21:07:20 if something needs to be ACKed for the roadmap, you can ask me, or anyone on nova-drivers 21:07:41 and it can either be approved, or they may ask for a discussion to make sure there is concensus on the ML first 21:08:02 sounds good 21:08:04 ok, that's all my notes on blueprint process. any questions on the process? also feel free to discuss specific blueprints if you'd like. 21:08:10 we can also discuss them in open discussion at the end. 21:08:26 hi all 21:08:52 rmk and vishy were looking various nova-network => mutnauq upgrade things, any idea if either of them are going to put in something for deprecating nova-network? 21:09:16 sdague: that's a good point, we need something for that. 21:09:21 long live nova-network! 21:09:36 sdague: it's unclear what code work is needed ... it's basically "figure out the migration path" 21:10:01 yeh, true, though I imagine some of that upgrade code will need to be in tree 21:10:06 yep 21:10:16 but it's worth not loosing as a thing 21:10:17 i'll add that to my notes for something to chase down 21:10:18 Yeah, its likely they'll need to configure multiple API endpoints for example 21:10:21 agreed 21:10:25 sdague: I think that may have been cburgess actually 21:10:46 and hi yes I am here 21:10:48 rmk: ok, it was day 4, so I was a little fuzzy by then :) 21:10:48 one of the problems is that no one person has taken clear ownership over this effort 21:11:06 sdague: no worries, I was a zombie too by then 21:11:10 a few people volunteered to do some necessary research 21:11:30 so maybe i'll take it for now, at least to track who is researching what 21:11:34 and keep it moving 21:11:37 russellb: I think you could consider me+garyk as the owners 21:11:47 vishy: ok that would be awesome 21:12:02 vishy: i'll file a nova blueprint and set you as the assignee (since we can have only one) 21:12:17 vishy: do you have any update on the blue print to coordinate the schedulign of nova and cinder using current rpc mechanism? 21:12:26 vishy: do you want to file a blueprint for it? 21:12:54 vishy: not sure what we'd call it ... perhaps it's just deprecate-nova-network, and it's just whatever we need to complete to feel good about doing that 21:13:23 senhuang: i'd like to see that also, or maybe u can create one? 21:14:09 harlowja: i have one for unified resource placement module, which is more like a centralized approach that sits on top of nova and cinder 21:14:26 senhuang: so there u go, how can we make that start to appear :-p 21:14:30 ok, let's cover a few other topics, then we can come back to specific blueprints at the end 21:14:41 #topic feature patch proposal deadline 21:14:49 #link http://lists.openstack.org/pipermail/openstack-dev/2013-April/007823.html 21:14:55 I put out this proposal on the list earlier this week 21:15:10 basically if we're going to reject patches for coming in too late, i want to communicate that clearly ahead of time 21:15:21 that seems to be ok with me 21:15:23 i picked a date out of the air, which was 2 weeks ahead of the official feature freeze 21:15:23 I think it's a good call 21:15:25 +1000 21:15:28 +1 21:15:30 +1 21:15:32 +1 21:15:40 +1 FIrm dates are great we can all work toward that and plan. 21:15:43 +1 21:15:46 +1 21:15:47 ok cool. 21:15:54 it would be good to record your support on the ML thread 21:15:59 sure 21:16:00 russellb: gee, I think that has enough support 21:16:13 I didn't know Dan got a thousand votes. 21:16:18 just might want to be some wording when this happens to not piss people off, haha 21:16:29 if anyone has an objection let me know ... otherwise I'll work on making sure it's communicated effectively 21:16:56 harlowja: well that's why i'm trying to establish it as early as possible, so it's not a surprise ... unless it's your fault for not paying attention 21:17:05 dripton: as you know, a bunch of +1's don't add up to a +2 :) 21:17:12 russellb: agreed 21:17:24 dripton: it's binary, so it's only 8 votes 21:17:26 dansmith: aw, harsh man, they do to me, depending on who the +1s are from :) 21:17:41 sdague: lol 21:17:46 sdague: nice 21:18:07 russellb: heh, well, I was trying to downplay my obvious over-vote for that, given that everyone else only allocated +1 for it :) 21:18:16 alright, next topic, maybe we can finally disagree on something 21:18:20 #topic baremetal split 21:18:28 #link http://lists.openstack.org/pipermail/openstack-dev/2013-April/007979.html 21:18:35 #link https://wiki.openstack.org/wiki/BaremetalSplitRationale 21:19:01 drum roll 21:19:04 nm, ha 21:19:05 please review that, ponder it, and make your opinion known if you have one 21:19:08 I don't even know what this is yet but yes 21:19:16 yeah, it was just posted today 21:19:20 so people need time to review it 21:19:25 but i wanted to make sure awareness was raised. 21:19:27 devananda: ^ 21:19:30 honestly bare metal doesn't fit well at all in nova 21:19:31 yep, thanks 21:19:39 i believe it's still waiting moderation on the -tc list, too 21:19:41 OpenStack Bare Metal (Ironic) 21:19:42 russellb: does a split count as a new incubation request for the TC? 21:19:45 i'm a fan of the codename. 21:19:54 mikal: yes, it will be a TC issue to consider 21:19:55 I've already said this, but I'm +1 on the Ironic proposal 21:20:05 i'd say in the current way baremetal works, it would be good to move it out of nova into it's own project. 21:20:11 mikal: but I'd like to drive more broad consensus before the TC votes on it or anything 21:20:11 mikal: as far as i've been told, this would bypass incubation and follow the same path Cinder took 21:20:16 me to, it will be interesting to see how it plays out, hopefully good in the end 21:20:17 russellb: agreed on the code name. 21:20:23 The name I'm impartial to but +as_many_as_possible to splitting it out 21:20:43 The use case is just so vastly different than dealing with VMs 21:20:45 mikal: since the code's already in a core project 21:21:14 but what we *don't* want is for this to be seen as a fast track method to being an incubated project 21:21:22 aside from being cleaner, it gives the project a much better chance of success since it'll be able to adapt to vastly different bare metal needs 21:21:27 as in, instead of starting off on your own, try to grow in an existing project and split off, because it's easier/quicker 21:21:28 russellb: indeed 21:21:40 devananda: will this bare metal project also support one VM is spawned on the bare metal and possesses all the resources exclusively? 21:21:43 rmk: yeah, agreed, i'm pretty sold on the idea 21:21:54 russellb: It seems to me this was more a result of discovering that it didn't fit well than anything intentional 21:22:16 rmk: oh sure, not accusing that here, i'm just worried about precedent, and abuse of the pattern elsewhere 21:22:16 senhuang: i'm not sure i follow your question, but i think the answer is yes, it already does 21:22:20 rmk: I'm skeptical of that devananda guy and his intentions 21:22:25 Heh 21:22:25 dansmith: totally 21:22:27 :) 21:22:29 he's pretty sketchy 21:22:35 long hair and everything 21:22:39 heh 21:22:44 I think it can't hurt to run it quickly through the TC, and perhaps truncate the incubation process 21:22:52 russellb: as long as it's clear the reason for the split, i think it shouldn't become a pattern. 21:22:53 dansmith: he also bites his dogs! 21:22:57 yeh, well during the review cycle it was clear it fit oddly 21:23:01 mikal: *gasp* 21:23:02 it's probably easier to figure out if anyone objects to this 21:23:11 i think i saw him with a trench coat on, too 21:23:11 because it seems like we couldn't possibly be in more agreement 21:23:13 which can't be good 21:23:22 senhuang: take a look at https://github.com/tripleo/incubator/blob/master/README.md and see if that answers some of your questions. it's a special case for baremetal 21:23:31 devananda: okay. cool. thanks! 21:23:34 yep, so everyone here seems happy with it ... please continue comments on list 21:23:40 mikal: only when necessary! 21:23:45 even +1s on the list are quite helpful to showing consensus 21:24:18 anything else on bare metal? 21:24:23 devananda: are the ntt people working with u, might be useful to have there advice, since they are likely using the code that exists and might know more about the migration path? 21:24:35 harlowja: very little 21:24:39 we would have to go through a deprecation path for this 21:24:50 devananda: hmmm, interesting 21:24:52 just like any other drastic change like this ... a deprecation cycle, migration path 21:25:05 since they handed the project to us iat the grizzly sumit, they've been pretty quiet 21:25:17 arata sends in patches and bug fixes often 21:25:31 devananda: it would be nice to try to draw them in on the new effort 21:25:34 and the ISI folks added tilera/pdu support right after FF ended 21:25:48 yep, that landed for havana 21:25:48 sdague: indeed. mikyung was in the session when we talked about this 21:25:54 and didn't voice any objection 21:26:03 which is about as much assent as i have gotten from them :) 21:26:09 ya, wonder if they can provide more input, since my guess is ntt, isi are using it more than at least y! is 21:26:37 so there feedback would seem pretty valueable if u can draw said feedback out of them 21:26:50 i'll try to ping them directly // off list and see 21:27:04 ok cool, one more quick topic, then we can go to open discussion 21:27:06 #topic bugs 21:27:21 we also have usage for it in some dev/ops. one comment I got from them is that it would be nice to have a common api to provision bare metal and vm.. 21:27:21 let's not let all this fun future work let us forget about bugs :) 21:27:25 #link http://webnumbr.com/untouched-nova-bugs 21:27:36 34 untriaged bugs right now 21:27:58 not *terrible*, but we should be watching, as we probably have bug reports coming in as people try out grizzly 21:28:16 russllb: could you response to my comment on but https://bugs.launchpad.net/nova/+bug/1049249 21:28:19 Launchpad bug 1049249 in nova "Remove plugging of internal classes from configuration" [High,Confirmed] 21:28:29 *everyone* can help with triage, if you're not sure how, see this page: 21:28:30 #link http://webnumbr.com/untouched-nova-bugs 21:29:00 senhuang: i guess it's valid as long as that option exists 21:29:15 senhuang: the compromise/solution has been moving things to entrypoints with short names 21:29:26 russellb: okay. i probably need more guide on this bug.. 21:29:32 so it'll still be pluggable, technically, but not so obviously directly exposed in the config file 21:30:11 any other bug comments or questions? 21:30:33 if we all triage just a couple this week, we'll be in much better shape (that goes for most weeks, really ..) 21:30:34 yes 21:30:47 should I report bug for each race condition? 21:30:52 before adding UC? 21:31:01 more races, eck :( 21:31:08 less races=) 21:31:09 russellb: the URL link form webnumbr shows only one bug? am I misunderstanding the function of that link? 21:31:19 morganfainberg: it showed 34 for me? 21:31:28 boris-42: the glass is half-full, good attitude 21:31:28 boris-42: you've got a blueprint for all the UC stuff though right? 21:31:32 I'd just use that 21:31:42 agreed, the bp is fine ... 21:31:53 technically you can associate bugs to a blueprint 21:32:00 but ... no need to create extra work unnecessarily 21:32:09 yeh, but I think the bp is fine, the uc work is a well known quantity 21:32:09 if you *want* to create bugs, it's fine with me, but you don't have to 21:32:14 * russellb nods 21:32:18 russellb: something off in my web-browser then. *shrug* 21:32:21 morganfainberg: it shows 38 for me, but only displays one of them due to windowing. Says "1 of 38" 21:32:37 dripton: i think it's an extension going dumb on my browser. 21:32:43 boris-42: has there been any thought on the races that are at a level above the DB? 21:32:53 read/write/modify types 21:33:01 you can also look at https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 21:33:08 not quite the same, but close enough 21:33:44 russellb what do you prefer adding bugs or not for races?) I just don't now is there any reason for spamming bug tracker?) 21:33:53 just use the bp 21:34:00 russellb ok 21:34:27 #topic open discussion 21:34:30 harlowja: afaik, no one's really looking at those races above db/api -- except for quotas 21:34:40 devananda: durn 21:34:43 harlowja: if you have some in mind, please open bug reports and tag with 'db' 21:34:50 harlowja I think that there is also a lot of races=) 21:34:53 kk, i have a few, do orphaned resources count 21:34:57 boris-42: 100% agreed 21:34:58 yeh, even better is tests 21:35:05 yay tests. 21:35:12 <3 tests 21:35:16 then we can keep them from showing up again 21:35:50 Btw I have some question about pci devices... 21:36:07 There is already about 4 BP for that! 21:36:21 One task but 4 BP =) 21:36:47 yes that area is a bit of a mess 21:37:01 a good example of what happens when people don't talk enough 21:37:09 probably we should chose one of that 21:37:18 yeah. 21:37:18 There is also problems with USB 21:37:38 if we are using host dev pci passthrough in libvirt 21:37:51 sdague agreed on tests, but if u just look at pieces of the nova code, u can spot these things, but maybe its just my mindset 21:37:55 then we are able to passthrough only USB controllers 21:37:56 I have a question on the nova's api extension 21:38:13 oh thats a good one, senhuang what happened to the nova extension refactoring effort thingy 21:38:21 harlowja: then bring tests to the table that demonstrate it 21:38:29 We should make a little bit different operation for USBs 21:38:30 sdague: sounds good :) 21:38:51 challenge accepted :-p 21:38:59 harlowja: i am still researching on how to do api extension for nova 21:39:01 boris-42: i think you're on the right track by going back and documenting your design 21:39:14 boris-42: then hopefully everyone interested can discuss if it meets their requirements 21:39:24 boris-42: and the people most involved with the libvirt driver can provide feedback as well 21:39:28 boris-42: then we can move forward 21:39:28 senhuang: just in general how to do them, or how to make the api extension mechanism easier? 21:39:40 re: API extension framework I'd appreciate any feedback on https://review.openstack.org/#/c/27276/ - would like to get that moving soon as we can't really do much v3 API work until at least some basic support is in. 21:39:52 for the later i thought there was https://etherpad.openstack.org/NovaAPIExtensionFramework 21:39:54 harlowja: on general how to do it 21:39:57 ya, with chris 21:40:08 cyeoh: i'll look it over today/tomorrow and give you feedback 21:40:10 senhuang: i can try to help, its not so clear 21:40:15 russellb Yes I am working on it, so there should be some changes in my approach to provide PCI passthrough and USB passthrough in one common approach=) 21:40:16 harlowja: that is pretty cool! thanks 21:40:25 morganfainberg: thanks! 21:40:26 senhuang: ya cyeoh just pointed that out 21:40:26 boris-42: ok 21:41:29 cyeoh: that is awesome. i will look at it 21:42:39 any other topics for the day? 21:42:53 please ping me anytime over the next week if you need assistance getting blueprints in shape 21:42:59 any one have time to take at look at this blue print? 21:43:19 https://blueprints.launchpad.net/nova/+spec/unified-rpm 21:43:20 btw is there some info about no downtime migrations? 21:43:42 boris-42: i wasn't in that session, so not sure what was discussed 21:43:59 a lot was discussed about versioned db objects 21:44:01 russellb: i'd like to hear how conductor can become something like orchestration, or understand at least from a design perspective how, just so i can accomodate it in my wiki and blueprints and such 21:44:11 IMO, that led to being able to do no-downtime-db-upgrades 21:44:17 senhuang: yeah, i saw that one ... it seems like there's not clear consensus on how to approach that problem yet. 21:44:25 or no downtime service upgrades for all services, including db and conductor 21:44:40 so, are there any BP up for versioned db objects? 21:45:05 russellb: maybe we could first provide a well-defined api extension and achieve consensus on the api? 21:45:09 harlowja: I started talking about this on a thread on the ML today. I feel like we've got some work combining code paths for migrate/live-migrate/resize/evacuate, and part of that will be moving it to conductor. that's the first step here. 21:45:22 how does moving it to conductor work? 21:45:24 senhuang: has there been a ML thread on this? 21:45:32 devananda: well, we're focused on internal objects first, 21:45:35 russellb: not yet 21:45:41 devananda: then versioning those and isolating schema changes along with that 21:45:55 russellb: i hear lots of moving to conductor but never have seen how it would get there or even if it makes sense there, shouldn't that be part of the discussion? 21:45:56 devananda: I think we don't expect to really hit the big versioning parts until post-H, realistically 21:45:57 senhuang: ok, that's where we need to discuss it basically. I think having a proposal for what the API would look like would aid that discussion. 21:45:59 russellb: i am in discussions with garyk on the api 21:46:02 dansmith: yes. i mean, is that plan written up anywhere? 21:46:10 senhuang: but it's not clear that there is even agreement that it should be a new separate thing, so we need to work through that. 21:46:13 devananda: the internal objects one? 21:46:36 devananda: I just filed a blueprint, and we talked about it on IRC a bit, but I need to fill out the blueprint with more details yet 21:46:37 dansmith: i could see maybe 3 BPs covering internal versioned objects, then rpc forwards/backwards compat, then db stuff 21:46:42 harlowja: it has been discussed a whole bunch, and we had a session dedicated to cleaning up these code paths where we discussed it further. 21:46:45 dansmith: ack 21:47:00 russellb: where's the docs on how that will be done? 21:47:02 devananda: yeah, I've got three now, actually, almost exactly like that.. still need to make the deps between then and fill out the details 21:47:03 senhuang: things like that really need to get to general concensus on the mailing list first 21:47:18 dansmith: great :) 21:47:23 boris-42: ^ 21:47:26 russellb: that is true. i think i still need to provide more details 21:47:32 senhuang: ok, sounds good 21:47:43 sdague: yep. that is probably a good idea 21:47:54 russellb: can u possible re-explain to me how said stuff fits in the conductor 21:48:16 russellb: on the conductor-does-evacuate stuff, i'd like Heat considered in that situation too 21:48:27 devananda: me too, or at least a common library for this stuff 21:48:30 which nova uses 21:48:39 russellb: as in, if there's an auto scaling group, how that'll be impacted by conductor doing things automagically 21:48:39 devananda: you realize "evacuate" isn't what it sounds like, right? :) 21:48:40 we shouldn't keep on reimplementing custom workflows 21:48:45 devananda: sure. note that we're *not* talking about doing anything automatically 21:48:54 russellb: ahh. didn't realize that :) 21:49:00 nothing automagic, all triggered from the outside like it si now 21:49:05 devananda: I don't want conductor doing automagic stuff, btw 21:49:09 sdague: i /think/ i know what it does 21:49:19 this is just about turning a bunch of separate code paths doing much of the same thing into common code where it makes sense 21:49:20 devananda: I want heat doing all the orchestration of failover type stuff from the outside entirely 21:49:29 and also, in the cases of migration and such, we have compute nodes telling each other what to do 21:49:30 dansmith: ++ 21:49:36 we don't want any compute nodes telling any other compute nodes what to do at all 21:49:41 devananda: the current "evacuate" action restarts vms that were on a machine that it toast 21:49:44 so the solution is to move most of the processing up a layer 21:49:57 sdague: cool. that's what i thought 21:50:14 russellb: so thats great, but thats not explaining how conductor will do it, thats just explaining the general principle right, how does conductor change to be said thing, especially when i am planning something that can do said tasks also, likely something that will use this heat library 21:50:23 harlowja: dude, chill 21:50:33 :) 21:50:41 trying, just i don't get it :-p 21:50:51 harlowja: i think more formal state tracking is good, but we need a place for it to plug in cleanly, and we don't have that yet 21:50:53 i agree on the general principles :-) 21:50:55 so this code cleanup is step 1 21:51:18 confused, isn't making the foundation correct step #1? 21:51:21 have to look at the steps it will take to get to where you want to go, and i really believe these are the first steps 21:51:26 then moving said code cleanup onto said foundation 21:51:57 we have to do it in logical steps, not an enormous patch set that rewrites everything 21:52:08 i understand what you want, and i think we can get there 21:52:09 isn't that possible with a good foundation? 21:52:09 harlowja: it seemed pretty clear in the room that it was a pretty good incremental cleanup 21:52:09 ... in stages 21:52:27 i still think it can be done in stages, with a good common library first though, not secondary :-p 21:52:28 I really think it's unrelated to orchestration 21:52:48 sdague: how so? 21:52:53 harlowja: hey, nothing prevents you from proposing alternative incremental code 21:53:05 sdague: and thats where its a waste of time to do 21:53:07 true :) 21:53:17 so here's the thing ... 21:53:18 we have code 21:53:20 a whole bunch of it 21:53:38 we want to do some cleanup, combining, and slight restructuring of that code as an incremental step 21:53:40 but I think rehashing consolodating the migration paths isn't very useful, because it's a huge win 21:54:03 from a verification stand point especially, as right now the whole live migrate path isn't testing at all in gate 21:54:10 which is the suck 21:55:28 but at the end of the day, code talks. bring something more clean, and more useful to the table, and it will be evaluated. 21:55:46 everyone interested in doing the work for the unified migrations path seems to be in agreement, 21:55:51 so I don't think we really have a problem here 21:55:56 sdague: i have code for the foundation of this, haha 21:56:15 would be good to see it on the list 21:56:18 i don't think i have seen it yet 21:56:28 harlowja: propose it and we'll handle it in gerrit 21:56:32 it was shown in the summit :) 21:56:39 dude, I have piles of code :) that doesn't mean it's more useful for openstack in all situations. 21:56:41 yeah, but lots of people weren't at the summit 21:56:51 i also had a session conflict (rpc stuff, and i'm one of the few maintainers of that code) 21:57:15 ok, will send that out again 21:57:19 k 21:57:25 I have to run 21:57:27 anything else? just a few minutes left in our time. 21:57:30 k, later dansmith 21:57:43 just i'd like to really understand clearly how the future of conductor fits into the plans, even incrementally 21:57:58 i honestly am not sure how to better explain it 21:58:21 write it down? 21:58:25 draw pictures? 21:58:37 we can continue out-of-meeting 21:58:39 thanks everyone! 21:58:41 #endmeeting