10:31:47 <dguitarbite> #startmeeting training-labs
10:31:48 <openstack> Meeting started Wed Jan  6 10:31:47 2016 UTC and is due to finish in 60 minutes.  The chair is dguitarbite. Information about MeetBot at http://wiki.debian.org/MeetBot.
10:31:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
10:31:52 <openstack> The meeting name has been set to 'training_labs'
10:32:57 <rluethi> so, discuss pending patches?
10:33:19 <egallen> yes
10:33:55 <rluethi> hey, egallen!
10:34:21 <rluethi> I looked at your patch this morning. LGTM, but needs a "-y".
10:34:35 <rluethi> Otherwise, yum will hang there.
10:35:31 <dguitarbite> Hello, good to see you.
10:35:32 <dguitarbite> my net is not reliable! rluethi can you host the meeting?
10:36:03 <rluethi> dguitarbite: I am not sure you can switch the host mid-meeting, but I'll do my best.
10:36:03 <dguitarbite> hi again
10:36:34 <dguitarbite> rluethi: no worries. Ill try my best. lets continue
10:36:38 <rluethi> dguitarbite: I added a comment/question re: one of your liberty updates.
10:36:49 <dguitarbite> rluethi: checking
10:37:28 <dguitarbite> #link https://review.openstack.org/#/c/252297/
10:37:58 <dguitarbite> rluethi: fair enough, Ill update the patch.
10:38:20 <dguitarbite> anything else for Liberty patch?
10:38:53 <rluethi> dguitarbite: the races in private/public network config are still there, of course.
10:39:19 <rluethi> When I tried to look into it, I stumbled over other issues that needed fixing first.
10:39:31 <rluethi> (host-side code)
10:39:55 <dguitarbite> Yes, I see some patches from your side. I wonder if they are more important than Liberty release urgency wise/
10:41:41 <rluethi> The i386 is not urgent at all. Well, I only have a month or so to test with a Google Compute Engine. It would be nice to find out why it is extremely slow.
10:41:50 <rluethi> No relation to Liberty, though.
10:42:28 <rluethi> I have some other patches pending. One bug fix that may be necessary for Liberty.
10:42:31 <dguitarbite> rluethi: Not sure but I think google compute engine uses something like containers/zero-vm/thread based encapsulation
10:43:21 <rluethi> dguitarbite: It is orders of magnitude slower than a VirtualBox environment on a laptop.
10:43:30 <dguitarbite> anything important for liberty patch? I want to discuss on the upcoming Fedora support with egallen.
10:43:32 <dguitarbite> rluethi: I agree.
10:43:49 <dguitarbite> rluethi: I think we should take that topic off meeting.
10:44:07 <rluethi> dguitarbite: It is on the agenda :)
10:44:21 <egallen> dguitarbite: I’m testing Liberty
10:44:34 <rluethi> anyhow, wrt to Liberty:
10:44:39 <egallen> do I’ve to maintain a kilo release also ?
10:45:05 <rluethi> if you try to run additional scripts on nodes that you set up previously, you will run into problems.
10:45:46 <rluethi> Basically, once you start init for a new node, you cannot run any scripts on earlier nodes.
10:45:46 <dguitarbite> egallen: Its your call, but I strongly recommend starting with Liberty
10:46:03 <egallen> OK, I start with Liberty
10:46:29 <dguitarbite> rluethi: I'm confused.
10:47:18 <dguitarbite> egallen: If you could get the state of CentOS support where it installs the OS and makes it ready for * Openstack Release, then I can help you out with adding Liberty support.
10:47:19 <rluethi> dguitarbite: you can do this: init controller, scripts controller, init compute1, scripts compute 1. But after init compute1, you can no longer run a script on controller.
10:47:42 <dguitarbite> rluethi: Unless ofcourse via. ssh, but yes you are right.
10:48:23 <rluethi> dguitarbite: the problem is that the host-side scripts only set the ssh configuration once, during init.
10:48:59 <dguitarbite> rluethi: is it really very important?
10:49:10 <dguitarbite> rluethi: I would focus working on the python ports for these features.
10:49:29 <rluethi> dguitarbite: If you have to touch the controller when you add a storage node, it might be.
10:50:07 <rluethi> dguitarbite: I have a patch for it, though.
10:51:33 <dguitarbite> rluethi: I have solved/answered a lof of these questions and have good progress with python port.
10:51:33 <dguitarbite> rluethi: no need for touching controller. Let me take a look once and tell you if we need to add this feature.
10:51:33 <dguitarbite> rluethi: Ok, if you have already finished working on this.
10:52:18 <rluethi> egallen: IMO the next steps would be to move the yum_* scripts into a scripts/centos subdirectory, and then add the equivalent of apt_upgrade.sh and apt_pre-download.sh.
10:52:21 <dguitarbite> then send the patch. I just meant to say that if the features are complex and if we could live without them, its a better idea to push them in the python port.
10:52:52 <dguitarbite> #topic CentOS/Fedora Support
10:52:59 <egallen> rluethi: I’m preparing yum_* scripts
10:53:53 <egallen> I’ll make an other specific commit
10:53:58 <dguitarbite> egallen: Can you send a patch which creates the CentOS/* folder and the common helper scripts and then another patch which starts work on adding Liberty support.
10:53:59 <rluethi> egallen: excellent. I suppose you want to submit them with a separate patch.
10:54:25 <rluethi> not CentOS/, centos/.
10:54:33 <egallen> OK
10:54:36 <rluethi> no uppercase letters in paths.
10:54:37 <egallen> Yes I prepare a separate patch, step by step
10:54:53 <dguitarbite> for the Liberty patch, we could have similar approach for collaborating like the current one.
10:55:03 <dguitarbite> rluethi: yes, small letter paths
10:55:30 * dguitarbite Pinky on shift is really naughty ;)
10:56:19 <rluethi> since we are not breaking any existing centos install, we don't have to do the whole liberty port in one huge patch like we do with ubuntu.
10:56:53 <dguitarbite> rluethi: egallen its upto you guys, I am fine with both approaches.
10:57:43 <rluethi> it's egallen's call.
10:58:31 <dguitarbite> anything else on this topic?
10:58:35 <egallen> I’ll start with Liberty
10:58:36 <rluethi> what I would not recommend writing a huge patch in private and then having to rework it all over.
10:58:48 <dguitarbite> #infor Egallen is working on CentOS support.
10:59:16 <dguitarbite> #info CentOS support with start with Liberty release.
10:59:29 <dguitarbite> #info Egallen is working on CentOS support.
10:59:36 * dguitarbite typos!!!
10:59:43 <dguitarbite> #topic Python Port
11:00:00 <dguitarbite> #link https://github.com/dguitarbite/labs-POC
11:00:56 <dguitarbite> Sorry, did I change the topic too early?
11:01:23 <rluethi> fine with me.
11:02:03 <dguitarbite> Python port is being worked on a github repository, we decided to do this as opposed to a feature branch for faster progress.
11:02:52 <dguitarbite> rluethi: I am working on adding libvirt units and I plan to get libvirt/kvm part done very soon. I would say in the upcoming 2-3 days.
11:03:13 * rluethi is not holding his breath
11:03:16 <dguitarbite> rluethi: I have a question regarding dependency for ssh. I would like to use paramiko
11:03:29 <dguitarbite> but it needs pip to be installed.
11:03:52 <rluethi> I don't see how we could possibly use pip.
11:03:59 <dguitarbite> #link http://www.paramiko.org/
11:04:43 <dguitarbite> rluethi: so I assume its a strong no from your side.
11:04:56 <dguitarbite> rluethi: I will write a wrapper to invoke openSSH via. the terminal
11:05:00 <rluethi> dguitarbite: maybe we'll find a way.
11:05:11 <dguitarbite> I have a solution for not using ssh at all for this
11:05:38 <dguitarbite> I would like to implement a simple TCP socket based program to handle interaction with the guest machine.
11:05:41 <rluethi> If we can avoid installing additional software on the users system, at least outside of our directory, it should be fine.
11:05:47 <dguitarbite> This would eliminate use of ssh alltogather :)
11:06:23 <dguitarbite> rluethi: ok, but this may take some time, esp. for testing so we can use the wrapper around ssh for the time being.
11:07:36 <dguitarbite> #info dguitarbite will implement a simple TCP socket based client/server protocol for managing interaction with guests.
11:08:25 <rluethi> we will still need it to work on Windows, mind you.
11:08:59 <dguitarbite> rluethi: that is the reason for this topic :) to eliminate "ssh" for windows.
11:09:00 <rluethi> if we can't write batch scripts, we need to find a solution for Python on Windows.
11:09:15 <rluethi> ssh was never needed for windows.
11:09:35 <rluethi> Windows uses a different mechanism that doesn't require users to install ssh.
11:09:57 <rluethi> ssh is only assumed to be there for Linux and OS X, and that is a safe assumption.
11:10:10 <dguitarbite> rluethi: also I found something interesting in disttools! We may not need to use py2exe for generating windows executables :D
11:10:14 <dguitarbite> #link http://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation
11:10:26 <dguitarbite> s/distools/setuptools
11:11:23 <dguitarbite> rluethi: I do not want to setup a file share for windows. So ill write a small python deamon running on the guest OS which will allow us to send scripts and run them on the guest without SSH or file system sharing
11:12:06 <rluethi> and how do you get the script into the VM?
11:12:18 <rluethi> I mean the daemon.
11:12:22 <dguitarbite> using sockets!
11:12:37 <dguitarbite> ahh, we install it via. preseed/ks files
11:13:07 <rluethi> Maybe we should have multiple PoCs.
11:13:25 <rluethi> One for architecture, others for trying out mechanisms like this.
11:13:36 <dguitarbite> rluethi: dont worry about this, I will work on this after we finish the python port
11:13:54 <rluethi> That is impossible.
11:13:58 <dguitarbite> till then I am planning to do a 1:1 clone (feature/dependency wise)
11:14:10 <rluethi> The python port cannot be finished before these issues are solved.
11:14:13 <dguitarbite> ssh for mac/linux and fire share for windows
11:14:35 <dguitarbite> rluethi: that is not a issue. I just wanted to be more pythonic rather than writing a wrapper around commands in the userspace.
11:14:38 <rluethi> ah, okay, a clone is great.
11:15:13 <rluethi> Well, due to our mission, we do some things in a less Pythonic way.
11:15:13 <dguitarbite> yes, but there is no simple command to do this clone ;) ... that ironically makes it more fun.
11:15:33 <dguitarbite> rluethi: yes, that is right. It hurts sometimes to make this sacrifise though.
11:15:43 <dguitarbite> s/this/these/
11:15:57 <rluethi> I kinda like the fact that I can write out a vbm.log or virsh.log that devs and users can try on the shell.
11:16:21 <dguitarbite> yes
11:17:02 <dguitarbite> another point I wanted to discuss is designing the configuration files and API for the python port. I assume we need to discuss this over a weekend of hacking.
11:17:15 <rluethi> If we wanted to go all Pythonic, we could also stop using the CLI clients on the guest side.
11:17:41 <dguitarbite> rluethi: CLI clients ?? Meaning the lib folders etc?
11:17:44 <rluethi> I'd say keep it simple for now and worry about it laster.
11:18:12 <rluethi> dguitarbite: I mean calling the neutron, openstack and nova clients from the shell rather than some API.
11:18:32 <dguitarbite> rluethi: that would be against the mission since we have to mimic install-guides.
11:19:39 <dguitarbite> rluethi: I meant to ask about config/ section. We should take this oppurtunity to re-write the VM confs., things like scripts.*cluster etc. into better ones like JSON or YAML.
11:19:49 <rluethi> right. we are building something that should be useful for instruction and training, not a demonstration of the Pythonic way.
11:20:37 <dguitarbite> rluethi: yes, that is true. I would not consider your example to be an argument w.r.t. pythonic way of doing things.
11:21:22 <dguitarbite> rluethi: What do you think about configuration files? Shall we discuss this in more detail later on? I think it would make it easier to configure our system and also make it more flexible.
11:21:27 <rluethi> dguitarbite: Like I said, _that_ is something I wouldn't worry about. I _know_ that we will find a decent solution for config files.
11:21:51 <dguitarbite> rluethi: ok, so we keep that as a new feature after merging python support then.
11:22:25 <dguitarbite> Im afraid that after merging python support, we may be overwhelmed by a new variety of bugs for a good part of N release.
11:22:37 <rluethi> What I care most about is knowing that we have a tried solution for every technical (and legal) problem.
11:23:03 <dguitarbite> rluethi: alright, its clearer for me now :)
11:23:13 <dguitarbite> lets give a few minutes for other topics.
11:23:37 <dguitarbite> #topic Any other buiz...
11:23:55 <rluethi> just real quick, I'd like to return to nested virtualization.
11:24:05 <dguitarbite> yes, tell me?
11:24:28 <dguitarbite> Do you mean this for deploying training clusters on public clouds?
11:24:50 <rluethi> if anyone can make an educated guess why our cluster build is extremely slow on Google Compute Engine, and what we could do to improve it, I'd be grateful.
11:25:23 <rluethi> I have a VM with 6.5 GB RAM, 2 cores.
11:25:35 <rluethi> basedisk build is 7 or 9 hours.
11:25:43 <rluethi> cluster build takes several days.
11:26:01 <rluethi> it looks entirely CPU-bound.
11:26:46 <rluethi> okay, so nested virtualization is slow, there is no hardware virtualization support.
11:27:28 <rluethi> but an i386 virtualized debian running inside a VirtualBox is a much, much faster _host_ system.
11:28:01 <dguitarbite> wow!
11:28:01 <rluethi> it shouldn't have hardware virtualization support, either, so where's the speed difference coming from?
11:28:37 <dguitarbite> I cannot say the exact reasons. But my educated guess would be with googles hypervisor in their cloud
11:29:08 <dguitarbite> I think they dont use virtualization like KVM/XEN ... they have their own thread based container system which is similar to a stripped down hypervisor
11:29:42 <dguitarbite> I could be wrong too ... but that is IMHO the reason for bad performance for our work load because its optimized for running applications not entire VMs in there.
11:30:01 <rluethi> I suspect it might be something obvious that I'm overlooking.
11:30:18 <rluethi> anyhow, time is up
11:30:22 <dguitarbite> And I believe, there is nothing we could do to fix it from our side unless we run the scripts directly Google Compute Instances.
11:30:30 <dguitarbite> rluethi: yes you are right.
11:30:35 <dguitarbite> #endmeeting