17:03:17 <johngarbutt> #startmeeting XenAPI
17:03:18 <openstack> Meeting started Wed Jan 16 17:03:17 2013 UTC.  The chair is johngarbutt. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:03:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:03:21 <openstack> The meeting name has been set to 'xenapi'
17:03:29 <johngarbutt> Hi all
17:03:38 <johngarbutt> #topic actions from last meeting
17:03:38 <matelakat> hi
17:03:39 <toanster> hello
17:03:41 <Mr_T> howdy
17:03:56 <bobba_> Morning!
17:04:10 <johngarbutt> one action was for bobball to check about any limits on the xenstore stuff
17:04:23 <johngarbutt> I think it was 2k for any xenstore value?
17:04:51 <BobBall> That's right - it's a page of 4k but with a bunch of headers, then we use a round value of 2048 bytes
17:05:06 <johngarbutt> right
17:05:18 <Mr_T> Thanks again, BobBall.
17:05:30 <johngarbutt> #info 2k limit on size of xenstore value
17:05:45 <BobBall> hth!
17:06:04 <johngarbutt> the other think was to discuss resetting passwords and looking towards cloud-init rather than xenapi specific agent
17:06:15 <johngarbutt> I think we scheduled that for 17:30
17:06:27 <johngarbutt> #topic blueprints
17:06:37 <johngarbutt> anyone got any blueprint worries?
17:07:08 <johngarbutt> mikal: is config drive done now? blueprint says in progress?
17:07:31 <johngarbutt> There are the same review requests from last week I think
17:08:02 <johngarbutt> sounds like nothing more on this one?
17:08:19 <johngarbutt> #topic docs
17:08:42 <johngarbutt> I updated the hypervisor wiki page to note that config drive hit trunk
17:09:09 <johngarbutt> #topic QA
17:09:41 <johngarbutt> any more updates on gating trunk?
17:10:36 <johngarbutt> #topic open discussion
17:10:45 <johngarbutt> anything people want to raise?
17:11:39 <johngarbutt> #info I got XCP iso running on virtual box, then openstack running using devstack, makes for a neat dev environment
17:12:36 <johngarbutt> is this meeting time still good with people? is it still useful?
17:12:37 <BobBall> It's been a quiet week I think
17:12:56 <johngarbutt> I am keen to keep doing this weekly, so we have a good time to raise things when they come up
17:12:59 <guitarzan> I can bring up force detach if you like
17:12:59 <guitarzan> :)
17:13:10 <zykes-> any news on san stufF ? :p
17:13:16 <toanster> yes, it is extremely helpful to me, i am still learning the space though
17:13:30 <johngarbutt> guitarzan: fire away
17:14:09 <matelakat> guitarzan: is it related to volumes?
17:14:11 <guitarzan> it's the same request... we (rackspace) would really like to be able to do it
17:14:16 <guitarzan> matelakat: yeah
17:14:46 <johngarbutt> Oh, I see, today you might wait until a VM reboots before it happens, no way to force it?
17:15:02 <guitarzan> yes
17:15:16 <guitarzan> and I'm not even sure the support guys can always get it to happen during a reboot
17:15:32 <guitarzan> but I'm not entirely clear on that case
17:15:35 <BobBall> Guests with PV tools installed? Doesn't the detach happen from XS immediately? or is it OpenStack that's postponing it becuase it's not sure it can happen?
17:15:49 <johngarbutt> you get disk in use from XenServer
17:15:49 <guitarzan> BobBall: xen won't detach because it's in use by the pv guests
17:16:03 <matelakat> I guess it's mounted, right?
17:16:14 <BobBall> oh - of course.  The detach is negotiated with the guest, yes.
17:16:18 <guitarzan> mounted, part of a raid, lvm, someone used the wrong side of the pillow
17:16:59 <BobBall> guitarzan, Just for confirmation, if you run a vbd-unplug with force=true, does that succeed?
17:17:07 <johngarbutt> it is one of those things that might be kernel specific I guess, due to pvops code
17:17:07 <guitarzan> BobBall: someone was supposed to get me your email address and I was going to send you a couple of emails
17:17:26 <guitarzan> BobBall: I'm not sure... I didn't know about force=true!
17:17:29 <BobBall> guitarzan, I'm bob.ball@citrix.com - feel free to send me as many emails as you like
17:17:31 <johngarbutt> do we have the bug for this?
17:18:00 <guitarzan> BobBall: thanks!
17:18:18 <guitarzan> one other interesting xen issue is we've seen pbd-unplug not actually remove the iscsi connection
17:18:24 <johngarbutt> does nova have a force detach?
17:18:26 <guitarzan> BobBall: that's the other one I needed to ask you
17:18:32 <guitarzan> johngarbutt: I'm not sure
17:18:56 <johngarbutt> I guess the current code goes into a retry loop checking to see if the volume gets detached
17:18:56 <guitarzan> I think clay added it, but I think that was just the cinder side
17:19:11 <johngarbutt> I guess the nice thing is to add a force remove (admin only operation?)
17:19:23 <johngarbutt> or is it users that get worried?
17:19:28 <BobBall> guitarzan, I think that the iSCSI connection is managed by the SR - so since the PBD could in theory be re-plugged as long as the SR is active then that connection is expected to stay?  johngarbutt - is that right?
17:19:45 <guitarzan> well, they ran an sr forget as well
17:19:50 <guitarzan> this has only happened a few times
17:19:52 <BobBall> oh - drat!
17:20:10 <johngarbutt> it certainly should get tidied up
17:20:13 <guitarzan> it's not exactly an unconfirmed bug, but I don't know how to reproduce it
17:20:15 <johngarbutt> #link https://bugs.launchpad.net/nova/+bug/1030108
17:20:18 <uvirtbot> Launchpad bug 1030108 in nova "Detaching a volume from a XenAPI instance fails if XenAPI thinks it is in use" [Medium,Confirmed]
17:20:41 <BobBall> So you've seen some cases where the SR doesn't exist in XS any more, yet the low level iscsi connection is still active?
17:20:42 <johngarbutt> I think that is when I became aware of the issues here
17:21:10 <guitarzan> BobBall: yes, very infrequently
17:21:21 <guitarzan> johngarbutt: got it
17:21:37 <guitarzan> that bug specifies the "remove from queue" half, but I think a force is more wanted
17:22:00 <johngarbutt> yep, we have a force, but bad things might happen to the data in the volume
17:22:05 <BobBall> guitarzan, That's not ideal! We'll have to look into possible causes for that
17:22:10 <zykes-> johngarbutt: san?
17:22:38 <guitarzan> BobBall: it would be great if we could explain it at least. it's possible that it happened during some manual sr/pbd cleanup after failed resizes and such
17:23:49 <johngarbutt> zykes: no real news there I am afraid
17:23:51 <BobBall> guitarzan, Can we get a copy of the SMlog from a system that this has occured on?  There should be some logging of when we try and log out of the iscsi session
17:24:00 <zykes-> johngarbutt: doh
17:24:39 <guitarzan> BobBall: I will look, but it's probably logrotated away by now
17:24:46 <johngarbutt> #action BobBall and guitarzan to get some logs about occasional iSCSI issue
17:24:47 <guitarzan> if I see another one, I will definitely grab logs
17:25:22 <BobBall> Great, thanks.  Ideally grab a bugtool so that other possible logs are also contained (e.g. messages, xensource.log) and the xapi DB to check state etc
17:25:58 <BobBall> Is this something that you fall over quickly when it happens? (i.e. because you can't reconnect the iscsi target to another host or similar?)
17:26:07 <johngarbutt> cool, so that covers most things
17:26:30 <guitarzan> BobBall: no, we just found this doing some backend volume clean/audit
17:26:36 <johngarbutt> any more for any more (before we move on to passwords without an agent
17:27:22 <BobBall> Hmmm okay.  Well let's take this offline - I'll have a bit of a think and a dig to refresh myself on how we handle the iscsi sessions and hopefully we'll get an occurence soon we can debug
17:28:12 <guitarzan> great, thanks
17:29:09 <johngarbutt> #topic password reset in a post xenapi agent world
17:29:23 <johngarbutt> hi, so there were some extra folks going to join us
17:29:59 <johngarbutt> who is in the channel and interested?
17:30:28 <johngarbutt> alexpilotti: hows things?
17:30:32 <johngarbutt> pvo: hows tings?
17:30:55 <alexpilotti> johngarbutt: hey
17:30:58 <johngarbutt> #link https://blueprints.launchpad.net/nova/+spec/get-password
17:31:14 <johngarbutt> so looks like there is a patch from vish to make the old and new work together
17:31:26 <johngarbutt> #link https://review.openstack.org/#/c/19746/
17:31:48 <johngarbutt> basically, if you set a password with the agent, you can get the encrypted password using the new nova call
17:32:11 <johngarbutt> alexpilotti: what is the plan with hyper-v and cloud-init?
17:32:28 <johngarbutt> or rather cloud-init on windows
17:32:43 <alexpilotti> johngarbutt: 1) Implementing the patch that Vish proposed
17:32:56 <alexpilotti> using the metadata service
17:33:01 <johngarbutt> post to metadata service?
17:33:05 <alexpilotti> yep
17:33:14 <johngarbutt> #link https://gist.github.com/4008762
17:33:17 <alexpilotti> 2) doing it via guest / host communication
17:33:42 <alexpilotti> letting the user choosing between 1 and 2
17:33:47 <johngarbutt> so how do you kick off the getting of the password, is it just on VM create?
17:34:15 <alexpilotti> that's the safest way
17:34:30 <alexpilotti> this way the password don't travel unencrypted on any channel
17:34:43 <alexpilotti> *doesn't
17:34:53 <johngarbutt> I think the xenapi side requirement is that we can reset the password without a reboot, at the users request
17:35:08 <alexpilotti> that's what I want to do as well
17:35:12 <johngarbutt> I know the user currently specifies the password, but I think we are OK with it being generated
17:35:19 <johngarbutt> as you say, much safer
17:35:28 <alexpilotti> here's BTW the API from nova.api.metadata import password; password.set_password(context, ecrypted_pass)
17:35:45 <alexpilotti> to be used on the host side to set the password
17:35:51 <johngarbutt> right, makes sense, thanks
17:36:10 <johngarbutt> so I guess you want the hypervisor system so you don't need the metadata service
17:36:26 <alexpilotti> I'm thinking on transforming cloud-init from a simple "inizialize and exit" service to a service that listens all the time for requests
17:36:42 <alexpilotti> which is similar to the way in which it works on Azure, for example
17:36:58 <johngarbutt> interesting. where would the requests come from in your case?
17:36:59 <alexpilotti> setting the password is the most obvious scenario
17:37:16 <johngarbutt> I guess puppet kick might be another, eventually
17:37:18 <alexpilotti> with a Nova extension, like the one you are already supporting
17:37:46 <johngarbutt> I mean, how does cloud-init get communicated to?
17:38:01 <johngarbutt> does it poll metadata service? does config drive get updated?
17:38:11 <alexpilotti> I'd like to have an sbstract API on top of each hypervisor's communication channel
17:38:33 <alexpilotti> let's say a simple value get / set semantyc
17:38:37 <matelakat> So it's more like an agent :-)
17:38:39 <johngarbutt> OK, I think that makes sense
17:38:53 <johngarbutt> have you seen the agent api in xenapi, one sec
17:39:08 <alexpilotti> with also some event listening channel
17:39:18 <matelakat> y, the whole stuff remonds me to the xenapi agent.
17:39:27 <alexpilotti> johngarbutt: can you sned me a link?
17:39:39 <johngarbutt> #link https://github.com/openstack/nova/blob/master/nova/virt/xenapi/agent.py
17:39:42 <matelakat> *reminds
17:39:45 <pvo> here now… sorry had something run over.
17:39:52 <alexpilotti> if we can avoid to reivent the well is even better :-)
17:39:57 <alexpilotti> * wheel, lol
17:40:08 <johngarbutt> and on the server side: #link https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/agent
17:40:22 <johngarbutt> if __name__ == "__main__":    XenAPIPlugin.dispatch(        {"version": version,        "key_init": key_init,        "password": password,        "resetnetwork": resetnetwork,        "inject_file": inject_file,        "agentupdate": agent_update})
17:40:34 <johngarbutt> hmm, that is a bit specific I guess
17:41:41 <alexpilotti> so agent.py runs on the guest?
17:41:53 <johngarbutt> yes
17:41:59 <johngarbutt> no
17:42:00 <johngarbutt> sorry
17:42:10 <johngarbutt> it runs on the hypervisor as an API extension
17:42:16 <johngarbutt> along with this code #link https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/xenstore.py
17:42:23 <johngarbutt> it is called from nova-compute
17:42:31 <matelakat> Johngarbutt: do you have a link to the agent source code?
17:42:35 <johngarbutt> basically there is a shared key value store
17:43:00 <pvo> is it still here? https://launchpad.net/openstack-guest-agents
17:43:10 <pvo> or moved to github?
17:43:33 <johngarbutt> http://wiki.openstack.org/GuestAgent
17:43:39 <pvo> https://github.com/rackspace/openstack-guest-agents-unix
17:43:51 <johngarbutt> thats the one, cheers
17:44:39 <matelakat> alexpilotti: does it help?
17:44:56 <johngarbutt> bit information overload, lets set back a little
17:45:00 <alexpilotti> looking!
17:45:37 <johngarbutt> password generate on guest, sent to use encrypted via metadata service: done by nova, cloud-init still needs to support
17:46:09 <johngarbutt> password generate is triggered by VM start
17:46:15 <alexpilotti> got it
17:46:17 <johngarbutt> at the same time as key injection
17:46:31 <matelakat> johngarbutt: thanks, that helps, I almost lost the context.
17:46:34 <johngarbutt> what we need is:
17:46:42 <johngarbutt> 1) method to kick off password reset
17:46:58 <comstud> (yes, github code is current for agent)
17:47:07 <johngarbutt> 2) method to send encrypted generated password when not using metadata service
17:47:45 <johngarbutt> you could do 1 by making cloud init poll the metadata service
17:47:51 <alexpilotti> 1) I'd go with an extension like "reset-password" or similar
17:48:10 <johngarbutt> I think vish already almost added that
17:48:13 <alexpilotti> and use vishy's get-password to retrieve it
17:48:15 <johngarbutt> clear-password or something?
17:48:25 <comstud> i think cloud-init is not a daemon today, correct?
17:48:30 <comstud> ie, just runs once at boot
17:48:34 * comstud is not sure
17:48:35 <alexpilotti> comstud: correct
17:48:38 <johngarbutt> right, good point
17:48:55 <alexpilotti> comstud: and smoser doesn't like the idea of transforming it into an agent
17:49:00 <johngarbutt> I like the idea of just re-triggering cloud-init in certain cases, and an agent would do that I guess
17:49:13 <johngarbutt> right, that is worth knowing, thanks
17:49:24 <alexpilotti> my take is that cloud-init should become a full agent
17:49:34 <comstud> switching out the agent for cloud-init affects more than just password reset
17:49:40 <comstud> it also affects any networking changes that are made
17:49:54 <johngarbutt> good point
17:49:56 <comstud> right now we're able to tell the agent to re-configure networking
17:50:03 <alexpilotti> I'd also fifferentiate the command plugins
17:50:20 <BobBall> nice word!
17:50:32 <alexpilotti> lool
17:50:44 <alexpilotti> forget ny typos :-D
17:50:49 <johngarbutt> :-)
17:51:10 <alexpilotti> I'm notoriously a disaster in writing in English quickly :-)
17:51:29 <johngarbutt> I guess the other point is to get images that can work in more places at once
17:51:31 <alexpilotti> I should deploy more than 4 fingers probably ;-)
17:51:38 <alexpilotti> back to the idea
17:51:46 <alexpilotti> a command can have a property
17:52:01 <alexpilotti> that indicates if it can be executed only at boot
17:52:14 <alexpilotti> so for example, password reset could be triggered any time
17:52:24 <alexpilotti> network injection only at boot, etc
17:52:36 <pvo> alexpilotti: this is assuming its a daemon, right?
17:52:47 <johngarbutt> erm, I think we want network reset at other times too, but  I see what you mean
17:52:50 <alexpilotti> that's an option.
17:53:00 <alexpilotti> pvo ^
17:53:06 <pvo> alexpilotti: how would it get triggered at other times?
17:53:08 <alexpilotti> pvo: on WIndows it's a service
17:53:16 <pvo> ok
17:53:22 <comstud> he's proposing it become a daemon
17:53:32 <comstud> with certian commands marked as boot-time-only
17:53:50 <pvo> a Windows service is a daemon, no?
17:53:55 <matelakat> y
17:53:59 <comstud> johngarbutt: and yeah, network reconfiguration should be allowed at any time, IMO
17:54:00 <pvo> (admittedly, its been a while for me and windows)
17:54:08 <alexpilotti> I'd go with password-reset Nova API extension -> hypervisor sends message via comm channel to guest -> cloud-init agent triggers event -> command plugin execution
17:54:17 <comstud> this isn't windows 95 where we should have to reboot to get changes.
17:54:18 <comstud> :)
17:54:35 <johngarbutt> or windows ME
17:54:38 <johngarbutt> yew
17:54:40 <johngarbutt> anyways
17:54:41 <comstud> ;)
17:54:48 <alexpilotti> comstud: reemind me to put a flag to set the host on fire if somebody tries to deploy Win95 :-D
17:55:00 <comstud> heheh
17:55:04 <BobBall> I miss windows 95...
17:55:10 <BobBall> But I digress
17:55:31 <johngarbutt> I think we might need to separate the cloud-init and deamon thingy
17:55:32 <comstud> I have a windows 3.11 VM.. w/ IE 3.
17:55:32 <alexpilotti> BobBall: "de gustibus non disputandum est" :-)
17:55:46 <johngarbutt> could we say there is an agent that kicks cloud-init, for example
17:55:46 <matelakat> johngarbutt: +1
17:55:51 <pvo> johngarbutt: thats kinda what we did on our windows agent.
17:55:54 <pvo> its two services
17:55:59 <alexpilotti> johngarbutt: smoser anyway doesnt like the idea
17:56:08 <pvo> alexpilotti: is that the only reason to kill it?
17:56:12 <alexpilotti> johngarbutt: on WIndows, I'd prefer to deploy a single WIndows service
17:56:36 <comstud> I think we do just have an extra agent that can support extra abilities that cloud-init cannot provide
17:56:40 <matelakat> in XenAPI terms it means agent and configdrive living together?
17:56:42 <comstud> It's optional...
17:56:52 <alexpilotti> since cloudbase-init (aka cloud-init for Windows) is already running as a service
17:56:57 <comstud> if you don't have it in your VM.. you can't password-reset or reconfig networking without rebooting
17:57:06 <comstud> etc
17:57:10 <alexpilotti> comstud: correct
17:57:18 <johngarbutt> I am thinking, what if we start again, what would we build...
17:57:24 <johngarbutt> good point, it is optional
17:57:28 <alexpilotti> comstud: IMO "reset-password" shoud throw an Exception if the opration fails
17:57:42 <comstud> right
17:57:43 <alexpilotti> due to missing agent code
17:57:49 <johngarbutt> I guess cloud-init takes configdrive or metadata stuff and does things
17:57:57 <alexpilotti> johngarbutt: correct
17:58:09 <johngarbutt> ideally we would like to add another alternative: hypervisor transport
17:58:19 <johngarbutt> (such as xenstore)
17:58:27 <johngarbutt> the other thing is when do start cloud-init
17:58:39 <johngarbutt> that is either at boot, or maybe via an agent
17:58:42 <alexpilotti> My 2c are: configdrive/metadata RO, hypervisor transport RW
17:58:51 <johngarbutt> what the agent does today could move to a cloud-init plugin?
17:59:06 <johngarbutt> alexpilotti: good point
17:59:25 <johngarbutt> (do ping if there is another meeting now)
17:59:34 <johngarbutt> does that sound crazy or good?
17:59:39 <alexpilotti> johngarbutt: form a first check, your agent's design looks very similar to cloud-init and cloudbase-init
17:59:45 <johngarbutt> the agent cloud become part of cloud-init, maybe
17:59:55 <alexpilotti> smoser: ping
18:00:11 <comstud> yeah, that's what's been thrown around before
18:00:25 <alexpilotti> for sure we can at least write a common code base
18:00:26 <comstud> things that are RO and aren't supported by cloud-init directly can be cloud-init plugins
18:00:42 <alexpilotti> and if the argument cloud-init <> agent wins
18:00:51 <comstud> RW is separate agent still, comm via XenStore or whatever secure mechanism if you have to talk between hyp and guest
18:00:57 <alexpilotti> at least cloud-init and the agent become wrappers on the common code
18:01:18 <johngarbutt> well password stuff now means metadata can accept data
18:01:30 <alexpilotti> johngarbutt: I hate it
18:01:35 <alexpilotti> :-)
18:01:35 <johngarbutt> but the trigger from a two way channel is the agent
18:01:37 <smoser> alexpilotti, my ears were ringing.
18:01:47 <alexpilotti> smoser: hi!
18:02:19 <alexpilotti> smoser: we are discussing about the old cloud-init vs agent
18:02:29 <smoser> i just personally do not see much value in an agent that is tied to a hypervisor/cloud-platform.
18:02:38 <johngarbutt> we are over time here, we might want to nip onto #openstack-nova?
18:02:50 <alexpilotti> np
18:02:54 <smoser> sure
18:03:08 <johngarbutt> thanks all
18:03:17 <johngarbutt> see some of in the other channel
18:03:19 <alexpilotti> I suggest -dev, as it's a discussion that might attract more people there
18:03:25 <johngarbutt> good point
18:03:31 <johngarbutt> #endmeeting