15:02:28 <anteaya> #startmeeting third-party 15:02:29 <openstack> Meeting started Mon Jun 15 15:02:28 2015 UTC and is due to finish in 60 minutes. The chair is anteaya. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:02:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:02:33 <openstack> The meeting name has been set to 'third_party' 15:02:37 <anteaya> hello 15:02:45 <akerr> hello 15:02:45 <patrickeast> hey 15:02:55 <anteaya> akerr patrickeast how are you? 15:02:56 <Alexey_Nexenta> Hi 15:03:02 <IlyaG> Hi guys, I am looking for hardware requirements/references for CI infrastructure - can someone point me into a right direction? 15:03:03 <marcusvrn> Hi 15:03:04 <anteaya> Alexey_Nexenta: hello 15:03:12 <anteaya> IlyaG: I'm not a guy 15:03:28 <anteaya> what shall we discuss today? 15:03:53 <eantyshev> hello 15:04:25 <anteaya> #link https://wiki.openstack.org/wiki/VirtualSprints#OpenStack_Common-CI_Solution 15:04:33 <wznoinsk> hi 15:04:37 <anteaya> asselin_'s sprint is coming up 15:04:46 <anteaya> please consider attending if you can 15:05:03 <anteaya> we will need people to conduct patch reviews and tests 15:05:11 <anteaya> so everyone can be helpful 15:05:28 <anteaya> wznoinsk: do you want to go into any more detail about your experience 15:05:33 <anteaya> thanks for posting ot the mailing list 15:06:12 <anteaya> #link http://lists.openstack.org/pipermail/openstack-infra/2015-June/002851.html 15:06:15 <mmedvede> hi 15:06:17 <wznoinsk> anteaya: no problem 15:06:27 <anteaya> wznoinsk: care to expand a bit? 15:06:29 <krtaylor> o/ 15:06:31 <anteaya> what is pygerrit? 15:06:42 <anteaya> is that the name of your script you wrote? 15:07:08 <wznoinsk> anteaya: I wasn't able to drill down exactly what piece of the operation is causing the problem or how to reproduce it outside of a python script 15:07:35 <wznoinsk> pygerrit is python lib for gerrit: https://pypi.python.org/pypi/pygerrit/0.2.2 15:07:40 <anteaya> ah 15:07:47 <anteaya> is anyone else using pygerrit? 15:08:27 <anteaya> I'm not advocating for it, just curious 15:08:47 <anteaya> wznoinsk: it seems to be staying away from Control-Z is your recommended way forward? 15:09:43 <wznoinsk> it didn't affect my 'ssh ... gerrit stream-event' on the command line when I did ctrl+z so my python/pygerrit is one situation you want to avoid ctrl+z with... rather than avoid ctrl+z at all 15:10:08 <anteaya> ah okay thanks for that clarification 15:10:24 <anteaya> wznoinsk: any other thoughts worth sharing at this time? 15:11:28 <wznoinsk> I got this link from someone on -infra, can't remember who it was tho: https://gerrit-documentation.storage.googleapis.com/ReleaseNotes/ReleaseNotes-2.9.2.html 15:11:36 <wznoinsk> it seems like this is fixed in gerrit 2.9.2 15:12:04 <wznoinsk> anteaya: nope, that was the most interesting one 15:12:25 <anteaya> wonderful 15:12:35 <anteaya> thank you for helping us with this issue 15:13:40 <anteaya> if anyone is able to help review any puppet-openstackci patches this week 15:13:45 <anteaya> there are two right now 15:13:49 <anteaya> #link https://review.openstack.org/#/q/status:open+project:openstack-infra/puppet-openstackci+branch:master+topic:downstream-puppet,n,z 15:14:00 <anteaya> it would be great if you are able to take a look at them 15:14:23 <wznoinsk> on slightly other note, I think it would be beneficial for CI operators (especially the ones that have their accounts disabled) to attend these meetings, it's a opportunity to get the issues solved quicker than on ML I think, how do we attract them? ;-) 15:14:33 <anteaya> IlyaG: you had some questions 15:15:03 <anteaya> IlyaG: did this person leave? 15:15:28 <patrickeast> looks like it 15:15:33 <anteaya> wznoinsk: a very good point 15:15:41 <anteaya> wznoinsk: I'm open to suggestions 15:15:58 <anteaya> as one of the big things I need when re-enabling a system is trust 15:16:10 <anteaya> and open communication goes a long way to establishing that 15:16:31 <wznoinsk> it needs promoting this irc meeting during any disabled/naughty account discussino on ml 15:16:37 <anteaya> since if someone takes out our system, I have to trust they will pay more attention next time 15:16:51 <anteaya> wznoinsk: I'm open to that 15:17:03 <anteaya> I don't do it myself as it may appear to be self-serving 15:17:12 <anteaya> however I do not discourage others from doing so 15:17:35 <anteaya> I do link to common third party resources when I have the energy 15:17:54 <anteaya> but I do admit that sometimes I lag behind in linking all the things 15:18:12 <anteaya> I welcome help and support in providing links to all the things 15:18:30 <krtaylor> wznoinsk, system operators have to want to get involved, they just can't miss all the irc/email traffic on it if they are paying attention 15:19:23 <akerr> can the ML not inject a link into the footer of every message on a list-by-list basis? 15:19:30 <Alexey_Nexenta> anteaya: Ilya wanted to ask if you have any recommendation/requirements for CI hardware ? 15:19:43 <anteaya> akerr: I don't know I haven't investigated 15:19:44 <Alexey_Nexenta> I would forward it to him 15:19:54 <krtaylor> akerr, we use openstack-dev, it would have no way of knowing 15:19:56 <anteaya> akerr: I find footers tend to be ignored 15:20:02 <krtaylor> ++ 15:20:18 <anteaya> Alexey_Nexenta: ah okay thanks, are you working on the same ci system? 15:20:39 <Alexey_Nexenta> yes, working on Nexenta CI 15:20:43 <akerr> well somehow the third-part-announce emails have a very specific link in their footer, so it's able to figure out 15:20:45 <anteaya> Alexey_Nexenta: we don't have a list, it depends entirely on what you are planning on testing 15:21:41 <anteaya> akerr: the footer is the third party announce mailing list information 15:21:56 <anteaya> some of the conversations happen on -dev and -infra too 15:22:13 <anteaya> if folks want a busier footer for third party announce, I can look into it 15:22:34 <akerr> anteaya: true, but we when an account is disabled it goes through the announce email. Why not just also advertise this meeting there 15:22:40 <patrickeast> Alexey_Nexenta: you would be testing cinder, right? 15:22:49 <anteaya> akerr: I can look into it 15:22:52 <krtaylor> hm, for announce, it might make sense, ++ 15:22:56 <Alexey_Nexenta> patrickeast: yes, cinder drivers 15:23:20 <anteaya> #action anteaya to look into adding link to third party meetings in -announce ml footer 15:23:30 <Alexey_Nexenta> Our main problem is that we keep getting timeout errors 15:23:34 <patrickeast> Alexey_Nexenta: ok cool, same here, for fwiw i’ve found that any less than ~3 build slaves able to run tests and our system gets too far behind 15:23:55 <patrickeast> Alexey_Nexenta: with a single jenkins slave it ends up ~12-24 hours behind on build very quickly 15:24:10 <patrickeast> with 3 it can stay only maybe 2-3 behind (with 35min builds) 15:24:37 <patrickeast> so any hardware that can acommodate that amount of devstack vm’s and the controllers should work 15:24:38 <ameade> Alexey_Nexenta: A good optimization is to only run on patches that Jenkins has +1'd 15:25:06 <Alexey_Nexenta> ameade: yes, I think we only run those 15:26:33 <anteaya> Alexey_Nexenta: what is your current setup? 15:27:06 <hodos|2> anteaya: we use john griffiths sos-ci 15:27:23 <anteaya> hodos|2: how many other team members do you have in this meeting? 15:27:36 <hodos|2> anteaya: 3 threads, fuel over vsphere 15:27:41 <Alexey_Nexenta> anteaya: just us two 15:27:45 <anteaya> okay thanks 15:28:06 <anteaya> hodos|2: are you using zuul and jenkins then? 15:28:17 * anteaya is not familiar with jgriffith's setup 15:28:33 <hodos|2> anteaya: nope, just python + ansible 15:28:35 <patrickeast> it replaces zuul, gearman, jenkins, all that 15:28:42 <anteaya> oh okay 15:28:45 <anteaya> no idea then 15:28:55 <anteaya> for timeouts I suggest you talk to jgriffith 15:29:06 <anteaya> it isn't an infra supported setup 15:29:18 <hodos|2> anteaya: it seems to do the job, timeouts are happening on the different side 15:29:19 <patrickeast> what kind of timeouts are you guys getting? 15:30:02 <patrickeast> we get ssh timeouts on some of the tempest tests when our system is stressed (still looking into why its happening) 15:30:40 <hodos|2> patrickeast: we probably using slow setup for openstack cloud: we have all-in-one node, but i guess, it's the same: backend or network is too slow 15:31:25 <hodos|2> patrickeast: most of those are happening with compute_with_volumes 15:31:32 <patrickeast> have you adjusted any of the timeout settings? iirc there are like 5 different ones to play with depending on what is timing out 15:31:57 <wznoinsk> it may be too radical but you may look into containers instead of vm's and stack inside 15:32:28 <hodos|2> patrickeast: can you point on how to change those, cause i couldn't find any iinfo 15:32:56 <marcusvrn1> patrickeast: same here (ssh timeouts), boot_pattern intermittently fails by timeout. If you solve this problem, could you let me know? I'm still working on it too 15:33:16 <akerr> we change a lot of our timeouts to crazy high numbers... let me see if i can find the variables 15:33:53 <patrickeast> hodos|2: so the ones we change are env variables for devstack-gate and jenkins, which unfortunately are not going to be helpful for sos-ci 15:34:08 <patrickeast> hodos|2: but, at some point they should be ending up in the tempest config file 15:34:28 <patrickeast> or i suppose in the sos-ci code if it has any global timeout for test runs like jenkins does 15:35:10 <hodos|2> patrickeast: well, i meant tempest timeouts, but i couldn't find the variable names for local.conf 15:35:26 <patrickeast> marcusvrn1: will do, we have looked into them a bit in the past and didn’t figure it out, but we are going back in to deep dive this sprint 15:35:35 <marcusvrn1> hodos|2: those variables are in tempest.conf 15:36:02 <marcusvrn1> #link https://github.com/openstack/tempest/blob/master/etc/tempest.conf.sample 15:36:31 <akerr> In the [volume] section you can set build_timeout 15:36:36 <akerr> of tempest.conf 15:36:50 <akerr> you also might want to adjust [compute] build_timeout 15:37:00 <hodos|2> marcusvrn1: thanks, i thougt you're able to modify tempest.conf through local.conf with TEMPEST_ vars 15:37:10 <marcusvrn1> hodos|2: 15:37:17 <marcusvrn1> hodos|2: yep, you are able 15:37:26 <patrickeast> hodos|2: you can, just define those variables in a section for your tempest config 15:37:36 <akerr> add [[post-extra|$TEMPEST_CONFIG]] to local.conf to set tempest variables 15:37:53 <marcusvrn1> hodos|2: in the local.conf you have to add something like "[[post-extra|\$TEMPEST_CONFIG]]" and then add which sections and variables you want to add 15:38:32 <hodos|2> akerr marcusvrn1 patrickeast: thanks, will try 15:39:11 <marcusvrn1> patrickeast: thanks 15:39:25 <anteaya> have we reached a happy place? 15:40:00 <anteaya> anything more to say on this topic? 15:40:08 <wznoinsk> one thing to remember when using proxy is to specify no_proxy to have at least 127.0.0.1,ip of management interface when running commands that connect to keystone 15:40:47 <asselin_> o/ 15:40:52 <anteaya> hey asselin_ 15:41:08 <asselin_> anteaya, hi 15:41:53 <anteaya> Alexey_Nexenta hodos|2 thanks for coming and asking, I hope you have enough to move forward this week 15:42:24 <anteaya> does anyone have anything else they would like to discuss today? 15:42:55 <hodos|2> anteaya: yep, we're hoping to speed up our env, by removing bottlenecks in network, and providing SSDs to backend 15:43:13 <anteaya> hodos|2: okay sounds great 15:43:17 <wznoinsk> anteaya: I focus to get my CIs as good as possible, is not having CIs mentioned on mailing list enough to say they're ok assuming they follow the rules from wiki? 15:43:19 <Alexey_Nexenta> thanks for the help everyone, will try those advices 15:43:48 <anteaya> wznoinsk: we don't have any stamp that says "this is a good ci" 15:44:08 <anteaya> wznoinsk: the biggest thing I look for is if someone has a question is someone around to answer it 15:44:11 <anteaya> or will be soon 15:44:39 <wznoinsk> anteaya: ok, is there anything in works to have something like 'infra approved' ci kind of thing? 15:44:53 <anteaya> wznoinsk: the fact you show up to meetings regularly and help others when you can goes a long way for me to add my voice of confidence when someone has a question about your system 15:45:15 <anteaya> Alexey_Nexenta: welcome, I hope you find a solution 15:45:19 <anteaya> wznoinsk: no 15:45:28 <asselin_> wznoinsk, not infra approved, but close: https://github.com/rasselin/os-ext-testing 15:45:33 <anteaya> wznoinsk: as any system can go sideways at any time 15:45:55 <anteaya> wznoinsk: we have an infra supported workflowl 15:46:11 <krtaylor> wznoinsk, we'll have a CI monitoring dashboard, hopefully soon, that will help with whether "this is a good CI" question 15:46:11 <wznoinsk> anteaya: thanks for the link, that's what I was after 15:46:13 <anteaya> which is what asselin_ is helpfing folks with and is working on improving 15:46:24 <anteaya> wznoinsk: ah sorry I mis-understood 15:46:42 <asselin_> wznoinsk, this is the goal: http://specs.openstack.org/openstack-infra/infra-specs/specs/openstackci.html 15:47:06 <anteaya> wznoinsk: some folks are looking to have their system blessed so they can stop maintaining it, a turn it on a forget it solution 15:47:17 <anteaya> wznoinsk: sorry I mis-understood your question 15:48:01 <anteaya> anything else on this topic? 15:48:24 <anteaya> anyone with anything else to discuss today? 15:49:08 <anteaya> can anyone think of any reason why we shouldn't wrap up the meeting? 15:49:29 <anteaya> okey dokey 15:49:43 <anteaya> thanks everyone for your attendance and participation 15:49:46 <anteaya> have a good week 15:49:51 <anteaya> see you next monday 15:49:54 <anteaya> #endmeeting