15:01:04 <anteaya> #startmeeting third-party
15:01:05 <openstack> Meeting started Mon Feb 23 15:01:04 2015 UTC and is due to finish in 60 minutes.  The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:09 <openstack> The meeting name has been set to 'third_party'
15:01:29 <anteaya> if you are here for the third-party meeting, do say hello
15:01:37 <patrickeast> hello
15:01:43 <anteaya> hello patrickeast
15:01:49 <anteaya> how are you doing today?
15:02:03 <smcginnis> o/
15:02:08 <wznoinsk> hi
15:02:09 <asselin> hi
15:02:13 <anteaya> wow
15:02:16 <anteaya> a crowd
15:02:20 * anteaya checks her hair
15:02:28 <anteaya> how is everyone today?
15:02:32 <smcginnis> anteaya: The pressure is on. :
15:02:38 <smcginnis> :)
15:02:44 <patrickeast> i’m doing well
15:02:45 <anteaya> smcginnis: no kidding
15:02:50 <anteaya> patrickeast: glad to hear it
15:03:00 <asselin> very busy
15:03:06 <anteaya> how is everyone's system running, for those running systems?
15:03:10 <smcginnis> busy and cold.
15:03:10 <anteaya> asselin: I bet
15:03:18 <anteaya> smcginnis: cold? where are you?
15:03:32 <smcginnis> Minnesota. I think it was -12 on the drive in this morning.
15:03:39 * anteaya pictures everyone not her to be somewhere warmer
15:03:49 <asselin> mine took a turn for the worse today. pypi issues or something else is breaking devstack
15:03:50 <anteaya> smcginnis: ah okay, yeah you are cold too
15:04:03 <anteaya> asselin: what are you seeing?
15:04:12 <smcginnis> n-novnc?
15:04:44 <asselin> give me a sec
15:05:01 <anteaya> is everyone okay if we take some time to discuss asselin's issue?
15:05:08 <anteaya> it might help other folks too
15:05:16 <smcginnis> anteaya: Good with me.
15:05:22 <asselin> http://15.126.198.151/00/155300/2/check/3par-iscsi-driver-master-client-pip-eos10-dsvm/74edc1f/logs/devstacklog.txt.gz#_2015-02-23_14_23_52_617
15:05:53 <smcginnis> asselin: I started getting this this morning: TypeError: setup() takes at least 2 arguments (1 given)
15:05:58 <smcginnis> n-novnc failed to start
15:06:08 <smcginnis> asselin: Doesn't look related to yours though.
15:06:53 <anteaya> asselin: this looks like it couldn't find a wheel
15:06:57 <asselin> no...wonder ifyours is related to recent nova client changes
15:07:22 <asselin> smcginnis, there was a lot of stuff on the mailing list regarding that
15:07:45 * smcginnis goes to check the archive
15:08:00 <asselin> anteaya, yea, not sure why. and not sure why it isn't connecting to my pypi mirror.
15:08:45 <anteaya> is the pip wheel on your pypi mirror?
15:09:08 <asselin> anteaya, it seems so. I was able to run the same command manually and it worked
15:09:43 <kaisers> o/
15:09:43 <wznoinsk> asselin: a long shot, proxy settings ? (changed recently) ?
15:10:27 <asselin> http://paste.openstack.org/show/180602/
15:10:38 <asselin> actually, looking at it again, seems manual setup went to pypi
15:12:23 <anteaya> any chance pypi.python.org had a derp when you ran your tests before?
15:12:31 <anteaya> did this happen more than one time?
15:13:15 <anteaya> but you want this to use your pypi mirror?
15:13:15 <asselin> anteaya, yes, it's notoriously bad from my site, so I switched to a mirror. Maybe my mirror settings are no longer valid due to other pip changes
15:13:27 <anteaya> perhaps that is it
15:13:29 <asselin> anteaya, it's causing many test failures this morning
15:13:45 <anteaya> since it appears both your test log and your manual test hit pypi.python.org
15:14:05 <anteaya> many test failures this mornging <- for you or for more than you?
15:14:19 <asselin> for me
15:14:41 <anteaya> okay
15:14:53 <asselin> all red. I should sell my stocks
15:15:00 <anteaya> no kidding
15:15:15 <anteaya> can we look at your mirror settings then?
15:15:25 <anteaya> or can you and tell us what you see?
15:17:01 <anteaya> while we are waiting for asselin
15:17:29 <anteaya> did anyone else hit what smcginnis seems to have hit this morning/local time of day for you?
15:17:36 <asselin> http://paste.openstack.org/show/180604/
15:19:01 <smcginnis> anteaya: Regarding my issue, appears to be a legitimate bug and being addressed:
15:19:02 <anteaya> looks like it is hitting your extra-index-url without trying your index url
15:19:04 <smcginnis> https://bugs.launchpad.net/nova/+bug/1424532
15:19:05 <openstack> Launchpad bug 1424532 in OpenStack Compute (nova) "setup() takes at least 2 arguments (1 given)" [Undecided,In progress] - Assigned to Edward Hope-Morley (hopem)
15:19:12 <anteaya> smcginnis: awesome
15:19:32 <anteaya> asselin: what happens if you comment out extra-index-url in the config and try manually?
15:20:42 <asselin> fyi it worked here (recently) http://15.126.198.151/13/155313/4/check/3par-iscsi-driver-master-client-pip-eos10-dsvm/eb5a0aa/logs/devstacklog.txt.gz#_2015-02-23_14_36_02_165
15:20:46 <asselin> anteaya, will try
15:21:59 <asselin> anteaya, it worked
15:22:00 <anteaya> okay when it installs it successfully it doesn't tell us what url it used to get it
15:22:05 <anteaya> asselin: interesting
15:22:11 <asselin> anteaya, no it doesn't
15:22:18 <anteaya> hmmm
15:22:56 <asselin> but putting that line back and uninstall, and install, works too
15:23:41 <anteaya> so where are we now?
15:23:49 <anteaya> working or still troubleshooting?
15:24:31 <asselin> well...seems it's intermitnt. I'll kick off another job and hope this just goes away :/
15:24:40 <anteaya> ha ha ha
15:24:47 * anteaya hopes that about a lot of things
15:24:55 <asselin> :)
15:25:10 <anteaya> any one have anything else that is one their minds?
15:25:24 <anteaya> a question or stacktrace to discuss?
15:26:14 <anteaya> no?
15:26:25 <smcginnis> anteaya: I guess I can bring up one thing...
15:26:27 <anteaya> noone has anything else they want to talk about
15:26:32 <anteaya> smcginnis: go ahead
15:26:33 <patrickeast> nothin from me today
15:26:38 <anteaya> patrickeast: great
15:26:53 <smcginnis> Has anyone worked on making cinder FC testing easier. :)
15:27:02 <anteaya> FC?
15:27:17 <smcginnis> anteaya: Fibre channel.
15:27:18 * asselin remembers his promise to smcginnis
15:27:28 <anteaya> ah
15:27:37 <smcginnis> There are some unique challenges with it for testing.
15:27:46 <anteaya> what is the current status for fibre channel
15:27:51 <smcginnis> And granted, unique to cinder and probable no one else.
15:27:56 <smcginnis> anteaya: It works.
15:28:00 <anteaya> smcginnis: yes I remember from the fort collins meetup
15:28:15 <anteaya> smcginnis: so fibre channel works
15:28:21 <smcginnis> But the challenge is you need to do things a little differently to be able to pass through a FC adapter to the instance.
15:28:24 <anteaya> and it is being tested at all?
15:28:32 <smcginnis> So not as easy as just spinning up a cloud image.
15:28:46 <smcginnis> anteaya: It is being tested. But requires some special handling.
15:28:49 <asselin> anteaya, yes we have ours working (minus the recent devstack/pip wheel issue)
15:28:50 <anteaya> how is that being done, the passing through a FC adapter to the instance?
15:28:59 <smcginnis> asselin has done some good work I hear.
15:29:10 <smcginnis> And I'm not sure how it could be addressed widely.
15:29:12 <asselin> ok, here's the basics:
15:29:14 <anteaya> asselin: are you able to share how you are testing fibre channel with your setup?
15:29:26 <smcginnis> It may just be something that is a specific challenge for vendors that need to support it.
15:29:37 <asselin> anteaya, yes I plan to write it up...but I can go over it briefly now
15:29:42 <anteaya> asselin: thanks
15:29:45 * anteaya listens
15:29:53 <smcginnis> asselin: Sorry, didn't mean to put you on the spot.
15:30:07 <asselin> on the blade running openstck, there's a bit of setup.
15:30:20 <asselin> first you need to export the fc pci devices
15:31:20 <asselin> http://paste.openstack.org/show/180608/
15:31:33 <asselin> I save that in the /etc/environment on the BLADE
15:32:10 <asselin> then, you also need to setup the fc_passthrough public key on the blade's authorized keys list
15:32:44 <asselin> b/c the nova instance /jenkins slave/nodepool node will ssh into it's host blade to invoke the passthrough on itself
15:33:03 <asselin> (BTW, there may be other solutions, but this is what we've got working now)
15:33:14 <smcginnis> asselin: Thanks!
15:33:24 <smcginnis> For now at least I have custom scripting.
15:33:35 <smcginnis> I have an ESXi host with an image staged and snapshotted.
15:34:00 <smcginnis> My script just starts up the shutdown image, runs tests, shuts it back down and reverts it to the snapshot.
15:34:26 <smcginnis> git branch -a
15:34:31 <smcginnis> Oops :)
15:34:55 <asselin> also on the blade you need to enable pci stub: http://paste.openstack.org/show/180612/
15:36:20 <anteaya> thanks asselin
15:36:25 <anteaya> that is awesome
15:37:23 <anteaya> I'm thinking that smcginnis is looking for something and pasting it
15:37:45 <smcginnis> anteaya: No, sorry, nothing good to share right now. :]
15:37:51 <anteaya> smcginnis: ah okay
15:37:59 <anteaya> well that was great
15:38:01 <asselin> then, on the jenkins slave, before running the job, you inkode passthrough using this magic script: http://paste.openstack.org/show/180613/
15:38:04 <smcginnis> anteaya: Just thought since we didn't have another topic I would bring it up.
15:38:12 <anteaya> smcginnis: glad you did
15:38:22 * smcginnis bookmarks all of asselin's links
15:38:46 <smcginnis> Definitely not a trivial thing.
15:38:51 <asselin> bascially, ssh into the blade to invoke the nova command to get information about yourself like you nova id
15:38:51 * anteaya pictures that file called magic_script on asselin's env
15:40:11 <asselin> then, you have to create this passthrough xml file, upload it to the blade (that's whay you need a special private key authorized)
15:41:04 <asselin> also, I use the private ip's int he same subnet as the public ips, b/c I use the nova instance's private ip to figure out it's host's ip address
15:41:24 <asselin> then, for nodepool (yest there's more)
15:41:44 <asselin> I configure each blade as 2  separate nodepool providers
15:41:57 <asselin> the first is for non-FC, and second if for FC.
15:42:35 <asselin> the reason is b/c they have different max-servers, especially in the case of FC, b/c that is limited tot he number of HBAs on the blade itself.
15:42:57 <asselin> so i you only have 1 hba, then only 1 nova instance on that blade can run the Fc ci test
15:43:23 <asselin> if you have 4, then you can have 4 nova instances running FC tests in parallel
15:43:42 <smcginnis> asselin: I imagine this took quite a while to sort out!
15:43:56 <asselin> yes it did
15:44:17 <asselin> any questions? :)
15:44:22 <smcginnis> asselin: Thanks for sharing all the pointers. I'm going to have to dig through all this.
15:44:30 <anteaya> how many people are in the situation where they need to test fibre channel?
15:44:30 <smcginnis> asselin: I'm sure I'll have questions later. :)
15:44:40 <asselin> I need to document it more, but that's the jist of it.
15:44:43 <patrickeast> yea this is great stuff, i was a bit worried about adding FC support in L but it looks like this is a pretty good solution
15:44:43 <smcginnis> anteaya: There's at least a handful of vendors.
15:44:52 <anteaya> smcginnis: okay thanks
15:45:06 <anteaya> asselin: you do great work!
15:45:22 <smcginnis> There were maybe half a dozen at the cinder midcycle IIRC.
15:45:55 <anteaya> smcginnis: okay I do recall fibre channel is definitely a must have for some and a really special case
15:46:01 <patrickeast> i think pretty much all the main storage array vendors have (or will have) some sort of fc
15:46:09 * smcginnis feels special :)
15:46:15 <anteaya> patrickeast: okay good to know
15:46:21 <anteaya> smcginnis: good, because you are!
15:46:26 <anteaya> you are all special
15:46:34 * anteaya enjoys a warm fuzzy moment
15:46:41 <anteaya> :)
15:46:45 <asselin> maybe if nova provided a way to inject fc passthrough, then it would be easier
15:46:59 <smcginnis> asselin: +1
15:47:01 <anteaya> that is an interseting thought
15:47:12 <anteaya> has anyone approached nova about this in the past?
15:47:15 <patrickeast> yea i had been looking through https://wiki.openstack.org/wiki/Pci_passthrough looking to see what is available
15:47:25 * anteaya notes now is a poor time to approach nova about anything new
15:47:38 <asselin> b/c a lot of what the script does is to get nova details, and manually invoke passthrough on the hyperviser
15:47:50 <anteaya> #link https://wiki.openstack.org/wiki/Pci_passthrough
15:48:05 * asselin looks
15:48:16 <anteaya> so here is my thought
15:48:43 <anteaya> if noone has approached nova about adding this (I would consider it a feature expansion if not a new feature outright)
15:49:02 <anteaya> get some notes together about what generic approach works for multiple vendors
15:49:31 <anteaya> show up at a nova meeting and give them a heads up that you are working on this with the intent of asking for a design summit session in vancouver
15:49:52 <anteaya> you aren't asking for review time now just letting them know your intentions so they aren't surprised
15:49:52 <asselin> nice link....looks very promising!
15:50:10 <anteaya> then request a summit session to add the feature
15:50:25 <anteaya> you might get it, you might now but it helps to communicate your intention
15:50:41 <anteaya> and then work on a spec for L to get the code into nova
15:50:47 <anteaya> any thoughts?
15:50:55 <patrickeast> sounds like a good plan to me
15:50:55 <anteaya> have any of these steps been done already?
15:52:15 <anteaya> I'm taking that as no
15:52:23 <anteaya> we can mull on this for a bit
15:52:46 <anteaya> but I do think that is a way forward which has a hope of achieveing the goal
15:53:01 <asselin> looks like the code was written 2 years ago
15:53:54 <anteaya> is this code in nova?
15:54:01 <anteaya> or is it its own repo?
15:54:32 <asselin> https://github.com/yjiang5/pci_api.git
15:55:09 <anteaya> it isn't in stackforge
15:55:19 <asselin> (that's the link from the wiki page)
15:55:29 <anteaya> so would your code be an addition to this repo, asselin?
15:55:44 <asselin> anteaya, no, hopefully this would make my code go away
15:55:59 <anteaya> so just use this repo?
15:56:36 <asselin> anteaya, no...I wouldn't. that code may be merged into nova already, just the wiki is out of date.
15:56:42 <anteaya> ah
15:56:53 <anteaya> anyone willing to ask nova if that is the case?
15:57:48 <anteaya> going to be tough to find an answer then
15:58:27 <asselin> I can ask
15:58:34 <anteaya> asselin: thanks
15:58:36 <anteaya> :)
15:58:43 <anteaya> minute and a half left
15:58:52 <anteaya> anyone have anything else?
15:59:12 <anteaya> then I shall wrap this up
15:59:24 <anteaya> thanks everyone for your attendance and partcipation
15:59:29 <anteaya> see you next week
15:59:32 <anteaya> #endmeeting